Dmitry S d5a4605d2a (core) Improve encoding detection for csv imports, and make encoding an editable option. ...

Summary:
- Using a sample of data was causing poor detection if the sample were
  cut mid-character. Switch to using line-based detection.
- Add a simple option for changing encoding. No convenient UI is offered
  since config UI is auto-generated, but this at least makes it possible to
  recover from bad guesses.
- Upgrades chardet library for good measure.

- Also fixes python3-building step, to more reliably rebuild Python
  dependencies when requirements3.* files change.

Test Plan:
Added a python-side test case, and a browser test that encodings can
be switched, errors are displayed, and wrong encodings fail recoverably.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3979

2023-08-24 09:50:52 -04:00

..

functions

REQUEST with unknown and undetectable encoding (#600)

2023-08-02 16:49:37 -04:00

imports

(core) Improve encoding detection for csv imports, and make encoding an editable option.

2023-08-24 09:50:52 -04:00

acl_formula.py

(core) Support for $ syntax in ACL rules

2022-11-09 16:33:11 +01:00

acl.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

action_obj.py

(core) External requests

2022-06-17 21:53:20 +02:00

action_summary.py

(core) Fix undo error for automatically removed rows, especially in summary tables

2022-09-09 22:15:45 +02:00

actions.py

(core) AddOrUpdateRecord user action

2022-02-03 16:22:51 -05:00

attribute_recorder.py

(core) Allow assistant to evaluate current formula

2023-07-24 21:59:00 +02:00

autocomplete_context.py

(core) Make Python tests pass in Python 3.11

2023-07-28 16:37:38 +02:00

codebuilder.py

(core) Use new asttokens.ASTText to support dollar signs inside f-strings

2023-08-23 13:39:28 +02:00

column.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

csv_patch.py

(core) Simple Python 3 compatibility changes

2021-06-22 17:13:17 +02:00

depend.py

(core) Implement trigger formulas (generalizing default formulas)

2021-06-25 22:53:07 -04:00

docactions.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

docmodel.py

(core) Comments

2022-10-17 13:38:21 +02:00

engine.py

(core) Allow assistant to evaluate current formula

2023-07-24 21:59:00 +02:00

fake_std_streams.py

(core) Use fake stdout/stderr when evaluating formulas

2023-06-14 18:24:27 -04:00

formula_prompt.py

(core) Use new asttokens.ASTText to support dollar signs inside f-strings

2023-08-23 13:39:28 +02:00

friendly_errors.py

(core) Upgrade friendly-traceback and its dependencies

2022-10-28 14:44:14 +02:00

gencode.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

grist.py

(core) Use MixedTypesKey for sort_by arg of lookupRecords to avoid errors in Python 3

2021-11-09 18:08:29 +02:00

identifiers.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

import_actions.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

lookup.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

main.py

(core) Allow assistant to evaluate current formula

2023-07-24 21:59:00 +02:00

match_counter.py

(core) move data engine code to core

2020-07-29 08:57:25 -04:00

migrations.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

moment.py

(core) New type conversion in the backend

2022-02-04 20:28:13 +02:00

objtypes.py

(core) Python optimizations to speed up data engine

2023-02-09 12:49:58 -05:00

parse_data.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

records.py

(core) Python optimizations to speed up data engine

2023-02-09 12:49:58 -05:00

relabeling.py

(core) Simple Python 3 compatibility changes

2021-06-22 17:13:17 +02:00

relation.py

(core) Polish and enable Reference List widget

2021-08-12 11:51:21 -07:00

runtests.py

python(tests): print logging when running tests in verbose mode

2022-09-20 17:22:28 +02:00

sandbox.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

schema.py

(core) reconcile webhook and widget description migrations

2023-05-15 11:56:15 -04:00

sort_specs.py

(core) Adding sort options for columns.

2021-11-03 15:31:39 +01:00

summary.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

table_data_set.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

table.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

test_acl_formula.py

(core) Support for $ syntax in ACL rules

2022-11-09 16:33:11 +01:00

test_acl_renames.py

(core) Update ACL resources/rules when tables/columns get renamed

2021-01-04 22:03:01 -05:00

test_actions.py

(core) move data engine code to core

2020-07-29 08:57:25 -04:00

test_codebuilder.py

(core) Use new asttokens.ASTText to support dollar signs inside f-strings

2023-08-23 13:39:28 +02:00

test_column_actions.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

test_completion.py

(core) Make Python tests pass in Python 3.11

2023-07-28 16:37:38 +02:00

test_default_formulas.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

test_depend.py

(core) Fix recursion error in invalidate_deps

2021-06-18 15:42:20 +02:00

test_derived.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

test_display_cols.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

test_docmodel.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

test_engine.py

(core) Make Python tests pass in Python 3.11

2023-07-28 16:37:38 +02:00

test_find_col.py

Correct spelling mistakes

2022-02-19 09:46:49 +00:00

test_formula_error.py

(core) Ignore leading whitespace in formulas, and strip out leading '=' sign users might add

2023-04-25 15:28:40 -04:00

test_formula_prompt.py

(core) Use new asttokens.ASTText to support dollar signs inside f-strings

2023-08-23 13:39:28 +02:00

test_formula_undo.py

(core) Mark column changing actions as indirect when adding data to empty column

2022-05-25 16:21:04 +02:00

test_functions.py

(core) Don't swallow TypeErrors in functions like SUM

2022-10-25 12:15:13 +02:00

test_gencode.py

(core) Make Python tests pass in Python 3.11

2023-07-28 16:37:38 +02:00

test_import_actions.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

test_lookups.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

test_match_counter.py

(core) Import Hashable from six.moves.collections_abc

2022-02-22 00:27:51 +02:00

test_migrations.py

(core) Simple Python 3 compatibility changes

2021-06-22 17:13:17 +02:00

test_moment.py

(core) New type conversion in the backend

2022-02-04 20:28:13 +02:00

test_record_func.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

test_recordlist.py

(core) Return RecordSet instead of list from property access when possible, to allow further property access

2022-04-05 18:05:00 +02:00

test_reflist_rel.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

test_relabeling.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

test_renames2.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

test_renames.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

test_replay.py

(core) Python optimizations to speed up data engine

2023-02-09 12:49:58 -05:00

test_requests.py

REQUEST with unknown and undetectable encoding (#600)

2023-08-02 16:49:37 -04:00

test_rules_grid.py

(core) Implementing row conditional formatting

2022-08-09 20:11:36 +02:00

test_rules.py

(core) Updating RawData views

2022-05-04 21:41:42 +02:00

test_side_effects.py

(core) Ensure formulas return something and don't assign to attributes of rec

2022-05-23 21:06:18 +02:00

test_sort_spec.py

(core) Adding sort options for columns.

2021-11-03 15:31:39 +01:00

test_summary2.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

test_summary_choicelist.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

test_summary_undo.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

test_summary.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

test_table_actions.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

test_table_data_set.py

(core) Simple Python 3 compatibility changes

2021-06-22 17:13:17 +02:00

test_temp_rowids.py

(core) Allow using negative rowIds to add records and refer to them in Reference values.

2020-12-15 16:39:56 -05:00

test_textbuilder.py

(core) Simple Python 3 compatibility changes

2021-06-22 17:13:17 +02:00

test_treeview.py

(core) move data engine code to core

2020-07-29 08:57:25 -04:00

test_trigger_formulas.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

test_twowaymap.py

(core) move data engine code to core

2020-07-29 08:57:25 -04:00

test_types.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

test_undo.py

(core) Fix undo error for automatically removed rows, especially in summary tables

2022-09-09 22:15:45 +02:00

test_urllib_patch.py

(core) Remaining Python 3 compatibility changes

2021-06-25 12:00:58 +02:00

test_user.py

(core) Adding new column in users table "ref" with unique identifier.

2022-10-04 15:19:28 +02:00

test_useractions.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

testsamples.py

(core) move data engine code to core

2020-07-29 08:57:25 -04:00

testscript.json

(core) Updating RawData views

2022-05-04 21:41:42 +02:00

testutil.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

textbuilder.py

(core) Remaining Python 3 compatibility changes

2021-06-25 12:00:58 +02:00

treeview.py

(core) move data engine code to core

2020-07-29 08:57:25 -04:00

twowaymap.py

(core) Simple Python 3 compatibility changes

2021-06-22 17:13:17 +02:00

tzdata.data

(core) apply some dependabot suggestions

2022-09-07 14:15:34 -04:00

urllib_patch.py

(core) Remaining Python 3 compatibility changes

2021-06-25 12:00:58 +02:00

user.py

(core) Adding new column in users table "ref" with unique identifier.

2022-10-04 15:19:28 +02:00

useractions.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

usercode.py

(core) Make Python tests pass in Python 3.11

2023-07-28 16:37:38 +02:00

usertypes.py

(core) Update logging in sandbox code, and log tracebacks as single log messages.

2023-07-18 11:21:25 -04:00

xmlrunner.py

(core) Run python unit tests again in python 3

2021-06-30 21:46:18 +02:00