8 Commits (77ef9df27da1390760c5a4d3616c429775c19dc4)

Author	SHA1	Message	Date
Dmitry S	8269c33d01	(core) When importing JSON, create columns of type Numeric rather than Int ... Summary: JSON import logic was creating columns of type Int when JSON contained integral values. This causes errors with large errors (e.g. millisecond timestamps), and Numeric is generally the more convenient and common default. Test Plan: TBD Reviewers: jarek, alexmojaki Reviewed By: jarek, alexmojaki Subscribers: jarek, alexmojaki Differential Revision: https://phab.getgrist.com/D3339	3 years ago
Alex Hall	321019217d	(core) Lossless imports ... Summary: - Removed string parsing and some type guessing code from parse_data.py. That logic is now implicitly done by ValueGuesser by leaving the initial column type as Any. parse_data.py mostly comes into play when importing files (e.g. Excel) containing values that already have types, i.e. numbers and dates. - 0s and 1s are treated as numbers instead of booleans to keep imports lossless. - Removed dateguess.py and test_dateguess.py. - Changed what `guessDateFormat` does when multiple date formats work equally well for the given data, in order to be consistent with the old dateguess.py. - Columns containing numbers are now always imported as Numeric, never Int. - Removed `NullIfEmptyParser` because it was interfering with the new system. Its purpose was to avoid pointlessly changing a column from Any to Text when no actual data was inserted. A different solution to that problem was already added to `_ensure_column_accepts_data` in the data engine in a recent related diff. Test Plan: - Added 2 `nbrowser/Importer2` tests. - Updated various existing tests. - Extended testing of `guessDateFormat`. Added `guessDateFormats` to show how ambiguous dates are handled internally. Reviewers: georgegevoian Reviewed By: georgegevoian Differential Revision: https://phab.getgrist.com/D3302	3 years ago
Edward Betts	d6e0e1fee3	Correct spelling mistakes	3 years ago
Dmitry S	64d9faed5a	(core) Fix import parsing from choking up on Python isdigit() surprises ... Summary: Python isdigit() returns true for unicode characters such as "²", which fail when used as an argument to int(). Instead, be explicit about only considering characters 0-9 to be digits. Test Plan: Added a test case which produces an error without this change. Reviewers: alexmojaki Reviewed By: alexmojaki Differential Revision: https://phab.getgrist.com/D3027	3 years ago
Dmitry S	26356fe588	(core) Fix bug with "maximum recursion depth exceeded" in imports. ... Summary: Our date-guessing logic analyzes text in full looking for date parts. This diff skip all that work when text is so long that we don't need to consider it to be a valid date. This is a quick fix. There are probably many other cases when we don't need to try hard to parse arbitrary text as dates. Test Plan: Added a fixture and test case that would trigger the error without the fix. Reviewers: paulfitz Subscribers: paulfitz Differential Revision: https://phab.getgrist.com/D2992	3 years ago
Alex Hall	4d526da58f	(core) Move file import plugins into core/sandbox/grist ... Summary: Move all the plugins python code into the main folder with the core code. Register file importing functions in the same main.py entrypoint as the data engine. Remove options relating to different entrypoints and code directories. The only remaining plugin-specific option in NSandbox is the import directory/mount, i.e. where files to be parsed are placed. Test Plan: this Reviewers: paulfitz Reviewed By: paulfitz Subscribers: dsagal Differential Revision: https://phab.getgrist.com/D2965	3 years ago
Alex Hall	16f297a250	(core) Simple Python 3 compatibility changes ... Summary: Changes that move towards python 3 compatibility that are easy to review without much thought Test Plan: The tests Reviewers: dsagal Reviewed By: dsagal Differential Revision: https://phab.getgrist.com/D2873	3 years ago
Paul Fitzpatrick	b82eec714a	(core) move data engine code to core ... Summary: this moves sandbox/grist to core, and adds a requirements.txt file for reconstructing the content of sandbox/thirdparty. Test Plan: existing tests pass. Tested core functionality manually. Tested docker build manually. Reviewers: dsagal Reviewed By: dsagal Differential Revision: https://phab.getgrist.com/D2563	4 years ago