gristlabs_grist-core/sandbox/grist/imports
Dmitry S d5a4605d2a (core) Improve encoding detection for csv imports, and make encoding an editable option.
Summary:
- Using a sample of data was causing poor detection if the sample were
  cut mid-character. Switch to using line-based detection.
- Add a simple option for changing encoding. No convenient UI is offered
  since config UI is auto-generated, but this at least makes it possible to
  recover from bad guesses.
- Upgrades chardet library for good measure.

- Also fixes python3-building step, to more reliably rebuild Python
  dependencies when requirements3.* files change.

Test Plan:
Added a python-side test case, and a browser test that encodings can
be switched, errors are displayed, and wrong encodings fail recoverably.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3979
2023-08-24 09:50:52 -04:00
..
fixtures (core) Improve encoding detection for csv imports, and make encoding an editable option. 2023-08-24 09:50:52 -04:00
__init__.py (core) Switch excel import parsing from messytables+xlrd to openpyxl, and ignore empty rows 2022-05-12 14:43:21 +02:00
import_csv_test.py (core) Improve encoding detection for csv imports, and make encoding an editable option. 2023-08-24 09:50:52 -04:00
import_csv.py (core) Improve encoding detection for csv imports, and make encoding an editable option. 2023-08-24 09:50:52 -04:00
import_json_test.py (core) Run and test imports only in Python 3, upgrade openpyxl, fix weird date handling 2022-09-02 16:27:34 +02:00
import_json.py (core) When importing JSON, create columns of type Numeric rather than Int 2022-03-30 09:54:35 -04:00
import_utils.py Be more accepting with None value in headers candidate (#331) 2022-10-31 15:57:26 -04:00
import_xls_test.py (core) Fix issue that ints would be imported with a trailing ".0" from Google Sheets. 2023-02-26 15:24:15 -05:00
import_xls.py (core) Update logging in sandbox code, and log tracebacks as single log messages. 2023-07-18 11:21:25 -04:00
register.py (core) Switch excel import parsing from messytables+xlrd to openpyxl, and ignore empty rows 2022-05-12 14:43:21 +02:00
test_imports.py (core) Run and test imports only in Python 3, upgrade openpyxl, fix weird date handling 2022-09-02 16:27:34 +02:00