5 Commits (6c90de4d62e91df0a954cd73e8b826ca275fb0d4)

Author	SHA1	Message	Date
Alex Hall	6c90de4d62	(core) Switch excel import parsing from messytables+xlrd to openpyxl, and ignore empty rows ... Summary: Use openpyxl instead of messytables (which used xlrd internally) in import_xls.py. Skip empty rows since excel files can easily contain huge numbers of them. Drop support for xls files (which openpyxl doesn't support) in favour of the newer xlsx format. Fix some details relating to python virtualenvs and dependencies, as Jenkins was failing to find new Python dependencies. Test Plan: Mostly relying on existing tests. Updated various tests which referred to xls files instead of xlsx. Added a Python test for skipping empty rows. Reviewers: georgegevoian Reviewed By: georgegevoian Differential Revision: https://phab.getgrist.com/D3406	2 years ago
Dmitry S	64d9faed5a	(core) Fix import parsing from choking up on Python isdigit() surprises ... Summary: Python isdigit() returns true for unicode characters such as "²", which fail when used as an argument to int(). Instead, be explicit about only considering characters 0-9 to be digits. Test Plan: Added a test case which produces an error without this change. Reviewers: alexmojaki Reviewed By: alexmojaki Differential Revision: https://phab.getgrist.com/D3027	3 years ago
Dmitry S	26356fe588	(core) Fix bug with "maximum recursion depth exceeded" in imports. ... Summary: Our date-guessing logic analyzes text in full looking for date parts. This diff skip all that work when text is so long that we don't need to consider it to be a valid date. This is a quick fix. There are probably many other cases when we don't need to try hard to parse arbitrary text as dates. Test Plan: Added a fixture and test case that would trigger the error without the fix. Reviewers: paulfitz Subscribers: paulfitz Differential Revision: https://phab.getgrist.com/D2992	3 years ago
Alex Hall	4d526da58f	(core) Move file import plugins into core/sandbox/grist ... Summary: Move all the plugins python code into the main folder with the core code. Register file importing functions in the same main.py entrypoint as the data engine. Remove options relating to different entrypoints and code directories. The only remaining plugin-specific option in NSandbox is the import directory/mount, i.e. where files to be parsed are placed. Test Plan: this Reviewers: paulfitz Reviewed By: paulfitz Subscribers: dsagal Differential Revision: https://phab.getgrist.com/D2965	3 years ago
Paul Fitzpatrick	b82eec714a	(core) move data engine code to core ... Summary: this moves sandbox/grist to core, and adds a requirements.txt file for reconstructing the content of sandbox/thirdparty. Test Plan: existing tests pass. Tested core functionality manually. Tested docker build manually. Reviewers: dsagal Reviewed By: dsagal Differential Revision: https://phab.getgrist.com/D2563	4 years ago