7 Commits (main)

Author	SHA1	Message	Date
Dmitry S	534615dd50	(core) Update logging in sandbox code, and log tracebacks as single log messages. ... Summary: - Replace logger module by the standard module 'logging'. - When a log message from the sandbox includes newlines (e.g. for tracebacks), keep those lines together in the Node log message. Previously each line was a different message, making it difficult to view tracebacks, particularly in prod where each line becomes a separate message object. - Fix assorted lint errors. Test Plan: Added a test for the log-line splitting and escaping logic. Reviewers: georgegevoian Reviewed By: georgegevoian Differential Revision: https://phab.getgrist.com/D3956	11 months ago
Dmitry S	17569561bf	(core) Fix issue that ints would be imported with a trailing ".0" from Google Sheets. ... Summary: Whole numbers, when imported from Excel into a Text column show up without decimals (e.g. "300"), but when imported from Google Sheets show up with decimals (e.g. "300.0"). The decimals are hard for end-users to remove. Fix by treating whole numbers consistently as ints. Test Plan: Added a fixture reproducing the issue, and a test case. Reviewers: georgegevoian Reviewed By: georgegevoian Differential Revision: https://phab.getgrist.com/D3800	1 year ago
Yohan Boniface	7bd895ef42	Add BooleanConverter to map proper boolean cells to a Bool column ... Note that only proper boolean will be considered, but not integers nor truthy or falsy strings.	2 years ago
Yohan Boniface	8bc5c7d595	Fix columns with falsy cells wrongly parsed as dates (#276) ... Eg. before this commit, this table would result in Date columns: | A | B | | ----- | -- | | FALSE | 0 | For now, even FALSE is parsed as Numeric (not sure why we don't have a BooleanConverter).	2 years ago
Alex Hall	6c90de4d62	(core) Switch excel import parsing from messytables+xlrd to openpyxl, and ignore empty rows ... Summary: Use openpyxl instead of messytables (which used xlrd internally) in import_xls.py. Skip empty rows since excel files can easily contain huge numbers of them. Drop support for xls files (which openpyxl doesn't support) in favour of the newer xlsx format. Fix some details relating to python virtualenvs and dependencies, as Jenkins was failing to find new Python dependencies. Test Plan: Mostly relying on existing tests. Updated various tests which referred to xls files instead of xlsx. Added a Python test for skipping empty rows. Reviewers: georgegevoian Reviewed By: georgegevoian Differential Revision: https://phab.getgrist.com/D3406	2 years ago
Alex Hall	321019217d	(core) Lossless imports ... Summary: - Removed string parsing and some type guessing code from parse_data.py. That logic is now implicitly done by ValueGuesser by leaving the initial column type as Any. parse_data.py mostly comes into play when importing files (e.g. Excel) containing values that already have types, i.e. numbers and dates. - 0s and 1s are treated as numbers instead of booleans to keep imports lossless. - Removed dateguess.py and test_dateguess.py. - Changed what `guessDateFormat` does when multiple date formats work equally well for the given data, in order to be consistent with the old dateguess.py. - Columns containing numbers are now always imported as Numeric, never Int. - Removed `NullIfEmptyParser` because it was interfering with the new system. Its purpose was to avoid pointlessly changing a column from Any to Text when no actual data was inserted. A different solution to that problem was already added to `_ensure_column_accepts_data` in the data engine in a recent related diff. Test Plan: - Added 2 `nbrowser/Importer2` tests. - Updated various existing tests. - Extended testing of `guessDateFormat`. Added `guessDateFormats` to show how ambiguous dates are handled internally. Reviewers: georgegevoian Reviewed By: georgegevoian Differential Revision: https://phab.getgrist.com/D3302	2 years ago
Alex Hall	4d526da58f	(core) Move file import plugins into core/sandbox/grist ... Summary: Move all the plugins python code into the main folder with the core code. Register file importing functions in the same main.py entrypoint as the data engine. Remove options relating to different entrypoints and code directories. The only remaining plugin-specific option in NSandbox is the import directory/mount, i.e. where files to be parsed are placed. Test Plan: this Reviewers: paulfitz Reviewed By: paulfitz Subscribers: dsagal Differential Revision: https://phab.getgrist.com/D2965	3 years ago