Commit Graph

194 Commits

Author SHA1 Message Date
Jarosław Sadziński
101450262c (core) Support for $ syntax in ACL rules
Summary: Adding support for the "$" syntax in ACL rules.

Test Plan: Updated

Reviewers: georgegevoian, dsagal

Reviewed By: georgegevoian, dsagal

Differential Revision: https://phab.getgrist.com/D3692
2022-11-09 16:33:11 +01:00
Yohan Boniface
4ff5a2eaa7
Be more accepting with None value in headers candidate (#331)
We already filter out a line will only None values, and sometimes
Excel of LibreOffice mistakes the real number of columns adding
one or more that have no value at all.
2022-10-31 15:57:26 -04:00
Alex Hall
16a43edc2e (core) Upgrade friendly-traceback and its dependencies
Summary:
Upgrading the friendly-traceback package to include a fix that I specifically requested in https://github.com/friendly-traceback/friendly-traceback/issues/144 as a solution for the problem mentioned in https://grist.quip.com/HoSmAlvFax0j#MbTADAEcJb7 . Specifically, this shows a friendly explanation when using `len()` with a generator expression.

Also upgraded the dependencies `executing` and `stack_data` (which are mine) while I'm at it, although I don't expect this to really change anything.

Test Plan:
Existing tests. There was one test failure because of a new explanation about generic `Exception`s which I've suppressed.

Tested manually that the new explanation appears:

{F64605}

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3687
2022-10-28 14:44:14 +02:00
Alex Hall
aa88c156e6 (core) Don't swallow TypeErrors in functions like SUM
Summary: Math functions like SUM which call `_chain` were catching `TypeError`s raised by the iterable arguments themselves, e.g. `SUM(r.A / r.B for r in $group)` where `r.A / r.B` raises a `TypeError` would silently return wrong results. This diff narrows the `try/catch` to only check whether the argument is iterable as intended, but not catch errors from the process of iterating.

Test Plan: Added Python unit test.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3679
2022-10-25 12:15:13 +02:00
Alex Hall
89259371a5 (core) Respect sort_by in lookupOne, and allow reverse sorting
Summary:
Ensure that `lookupOne` (via `RecordSet.get_one`) pays attention to the `sort_by` parameter by picking the first of its sorted list of row IDs.

Allow specifying reverse sort order in `sort_by` by adding `"-"` before the column ID.

Suggested in https://grist.slack.com/archives/C0234CPPXPA/p1665756041063079

Test Plan: Extended Python lookup test

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3675
2022-10-24 20:10:08 +02:00
Dmitry S
0a8ce2178a (core) Add PHONE_FORMAT function using the phonenumberslight library
Summary:
Add phonenumberslite-8.12.57 to requirements
Implement PHONE_FORMAT function.

Test Plan: Added doctest test cases

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3665
2022-10-21 08:13:18 -04:00
Jarosław Sadziński
bfd7243fe2 (core) Comments
Summary:
First iteration for comments system for Grist.
- Comments are stored in a generic metatable `_grist_Cells`
- Each comment is connected to a particular cell (hence the generic name of the table)
- Access level works naturally for records stored in this table
-- User can add/read comments for cells he can see
-- User can't update/remove comments that he doesn't own, but he can delete them by removing cells (rows/columns)
-- Anonymous users can't see comments at all.
- Each comment can have replies (but replies can't have more replies)

Comments are hidden by default, they can be enabled by COMMENTS=true env variable.
Some things for follow-up
- Avatars, currently the user's profile image is not shown or retrieved from the server
- Virtual rendering for comments list in creator panel. Currently, there is a limit of 200 comments.

Test Plan: New and existing tests

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3509
2022-10-17 13:38:21 +02:00
Jarosław Sadziński
9628253fd8 (core) Adding new column in users table "ref" with unique identifier.
Summary:
There is a new column in users table called ref (user reference).
It holds user's unique reference number that can be used for features
that require some kind of ownership logic (like comments).

Test Plan: Updated tests

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian, paulfitz

Differential Revision: https://phab.getgrist.com/D3641
2022-10-04 15:19:28 +02:00
George Gevoian
cd64237dad (core) Allow duplicating tables from Raw Data page
Summary:
Adds a "Duplicate Table" menu option to the tables listed on
the Raw Data page. Clicking it opens a dialog that allows you to
make a copy of the table (with or without its data).

Test Plan: Python, server, and browser tests.

Reviewers: jarek, paulfitz

Reviewed By: jarek, paulfitz

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3619
2022-09-29 08:59:42 -07:00
Alex Hall
792565976a (core) Show example values in formula autocomplete
Summary:
This diff adds a preview of the value of certain autocomplete suggestions, especially of the form `$foo.bar` or `user.email`. The main initial motivation was to show the difference between `$Ref` and `$Ref.DisplayCol`, but the feature is more general.

The client now sends the row ID of the row being edited (along with the table and column IDs which were already sent) to the server to fetch autocomplete suggestions. The returned suggestions are now tuples `(suggestion, example_value)` where `example_value` is a string or null. The example value is simply obtained by evaluating (in a controlled way) the suggestion in the context of the given record and the current user. The string representation is similar to the standard `repr` but dates and datetimes are formatted, and the whole thing is truncated for efficiency.

The example values are shown in the autocomplete popup separated from the actual suggestion by a number of spaces calculated to:

1. Clearly separate the suggestion from the values
2. Left-align the example values in most cases
3. Avoid having so much space such that connecting suggestions and values becomes visually difficult.

The tokenization of the row is then tweaked to show the example in light grey to deemphasise it.

Main discussion where the above was decided: https://grist.slack.com/archives/CDHABLZJT/p1661795588100009

The diff also includes various other small improvements and fixes:

- The autocomplete popup is much wider to make room for long suggestions, particularly lookups, as pointed out in https://phab.getgrist.com/D3580#inline-41007. The wide popup is the reason a fancy solution was needed to position the example values. I didn't see a way to dynamically resize the popup based on suggestions, and it didn't seem like a good idea to try.
- The `grist` and `python` labels previously shown on the right are removed. They were not helpful (https://grist.slack.com/archives/CDHABLZJT/p1659697086155179) and would get in the way of the example values.
- Fixed a bug in our custom tokenization that caused function arguments to be weirdly truncated in the middle: https://grist.slack.com/archives/CDHABLZJT/p1661956353699169?thread_ts=1661953258.342739&cid=CDHABLZJT and https://grist.slack.com/archives/C069RUP71/p1659696778991339
- Hide suggestions involving helper columns like `$gristHelper_Display` or `Table.lookupRecords(gristHelper_Display=` (https://grist.slack.com/archives/CDHABLZJT/p1661953258342739). The former has been around for a while and seems to be a mistake. The fix is simply to use `is_visible_column` instead of `is_user_column`. Since the latter is not used anywhere else, and using it in the first place seems like a mistake more than anything else, I've also removed the function to prevent similar mistakes in the future.
- Don't suggest private columns as lookup arguments: https://grist.slack.com/archives/CDHABLZJT/p1662133416652499?thread_ts=1661795588.100009&cid=CDHABLZJT
- Only fetch fresh suggestions specifically after typing `lookupRecords(` or `lookupOne(` rather than just `(`, as this would needlessly hide function suggestions which could still be useful to see the arguments. However this only makes a difference when there are still multiple matching suggestions, otherwise Ace hides them anyway.

Test Plan: Extended and updated several Python and browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3611
2022-09-28 19:42:36 +02:00
Alex Hall
1864b7ba5d (core) Add BulkAddOrUpdateRecord action for efficiency
Summary:
This diff adds a new `BulkAddOrUpdateRecord` user action which is what is sounds like:

- A bulk version of the existing `AddOrUpdateRecord` action.
- Much more efficient for operating on many records than applying many individual actions.
- Column values are specified as maps from `colId` to arrays of values as usual.
- Produces bulk versions of `AddRecord` and `UpdateRecord` actions instead of many individual actions.

Examples of users wanting to use something like `AddOrUpdateRecord` with large numbers of records:

- https://grist.slack.com/archives/C0234CPPXPA/p1651789710290879
- https://grist.slack.com/archives/C0234CPPXPA/p1660743493480119
- https://grist.slack.com/archives/C0234CPPXPA/p1660333148491559
- https://grist.slack.com/archives/C0234CPPXPA/p1663069291726159

I tested what made many `AddOrUpdateRecord` actions slow in the first place. It was almost entirely due to producing many individual `AddRecord` user actions. About half of that time was for processing the resulting `AddRecord` doc actions. Lookups and updates were not a problem. With these changes, the slowness is gone.

The Python user action implementation is more complex but there are no surprises. The JS API now groups `records` based on the keys of `require` and `fields` so that `BulkAddOrUpdateRecord` can be applied to each group.

Test Plan: Update and extend Python and DocApi tests.

Reviewers: jarek, paulfitz

Reviewed By: jarek, paulfitz

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3642
2022-09-28 17:58:33 +02:00
Alex Hall
df65219729 (core) Remove messytables completely, particularly for excel imports
Summary: Mirror of https://github.com/gristlabs/grist-core/pull/289

Test Plan: In addition to the PR, tweaked one Excel fixture file and tests involving that and one other fixture.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3640
2022-09-28 17:15:42 +02:00
Yohan Boniface
54703e2794 Remove messytables dependency from xlsx import 2022-09-21 14:20:19 +02:00
Yohan Boniface
045227cb52
Update sandbox/grist/imports/import_csv.py
Co-authored-by: Alex Hall <alex.mojaki@gmail.com>
2022-09-20 17:34:59 +02:00
Yohan Boniface
ce31d1632d
Update sandbox/grist/imports/import_csv.py
Co-authored-by: Alex Hall <alex.mojaki@gmail.com>
2022-09-20 17:34:41 +02:00
Yohan Boniface
57c8f9f4fe csv importer: mimic messytables defaults for now 2022-09-20 17:28:46 +02:00
Yohan Boniface
2544736aa8 Applying review from @alexmojaki 2022-09-20 17:22:28 +02:00
Yohan Boniface
9bbf66e50e wip: remove dependency to messytables 2022-09-20 17:22:28 +02:00
Yohan Boniface
410cf61d94 python(tests): print logging when running tests in verbose mode 2022-09-20 17:22:28 +02:00
Yohan Boniface
83985ab3cf test(import_csv): highlight differences between passed and returned options in parse_file 2022-09-20 16:29:23 +02:00
Yohan Boniface
462b66b7ee Add tests to cover CSV parsed options 2022-09-20 15:44:08 +02:00
Alex Hall
b1f5e061c1
Merge pull request #279 from yohanboniface/boolean-converter
Add BooleanConverter to map proper boolean cells to a Bool column
2022-09-20 14:34:00 +02:00
Paul Fitzpatrick
b9441cf8fd (core) updates from grist-core 2022-09-12 09:14:22 -04:00
Yohan Boniface
7bd895ef42 Add BooleanConverter to map proper boolean cells to a Bool column
Note that only proper boolean will be considered, but not integers
nor truthy or falsy strings.
2022-09-10 07:07:45 +02:00
Alex Hall
56624c4a95 (core) Fix undo error for automatically removed rows, especially in summary tables
Summary:
Fixes a bug noted here: https://grist.slack.com/archives/C069RUP71/p1662564341132349

This bug could happen quite easily as follows:

1. Have a formula in a summary table such as `$group.amount`. Typically there's also a `SUM` but that's not essential.
2. Find a group with nonzero values of `amount`.
3. Delete all rows in that group in the source table. Typically that just means one row in a lonely group.
4. The summary table row is automatically deleted.
5. Try to undo. This raises an error about trying to update a non-existent summary table row.

I tried to account for this undo problem in https://phab.getgrist.com/D3489 by not saving the updated value for `$group` when it was found to be empty. The reason this was insufficient is that `$group.amount` is immediately invalidated anyway when the source row(s) are deleted (I think because that's just how dependency relations involving references work) *and* the calculated value of `$group.amount` changes even if `$group` doesn't. For example, `$group.amount` may have previously been `[100, 200]`. After deleting the rows, `$group.amount` becomes `[0, 0]`. Keeping `$group` unchanged prevents `$group.amount` from just being `[]`, but deleting the source rows means that the amounts become the numeric default `0` which is still a change. This change in value is then noted which leads to saving an undo action to update the summary table record. All this happens in step 3 above, and the summary record is only deleted after that point.

This diff removes that special handling for `group` and instead adds a more general fix to `action_summary.py`. This inserts undo actions for deleted rows at the beginning of the undo list rather than at the end, which was already done for deleted tables and columns.

Test Plan: Python tests

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3626
2022-09-09 22:15:45 +02:00
Yohan Boniface
8bc5c7d595
Fix columns with falsy cells wrongly parsed as dates (#276)
Eg. before this commit, this table would result in Date columns:

| A     | B |
| ----- | -- |
| FALSE | 0 |

For now, even FALSE is parsed as Numeric (not sure why we don't have
a BooleanConverter).
2022-09-09 15:13:34 -04:00
Paul Fitzpatrick
bde44323c5 (core) apply some dependabot suggestions
Summary:
This applies the set of dependabot suggestions that are currently
passing tests on grist-core. There are a lot more suggestions to
come, an unusual number are not passing tests because tests were
briefly broken.

The list of suggestions is extracted from:

https://api.github.com/repos/gristlabs/grist-core/pulls?search=status:success+state:open

And then applied using:

  yarn upgrade package1@version1 package2@version2 ....

After application, any new entries in package.json are pruned, leaving
just updated entries and yarn.lock changes.

Non-trivial code updates include:
 * A change related to axios typing
 * A change related to jquery dropping `size()` in favor of `length`

Test Plan: existing tests should pass

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3621
2022-09-07 14:15:34 -04:00
Alex Hall
0a2bc56938 (core) Test undo for all Python summary table tests
Summary:
Undo often leads to errors, especially with summary tables. One example is here: https://grist.slack.com/archives/C069RUP71/p1662564341132349

This diff simply decorates all relevant tests in 3 files testing summary tables with `@test_engine.test_undo`. This didn't catch any new bugs or reveal the problem in the thread above, but it seems good to have.

Test Plan: this

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3624
2022-09-07 18:02:02 +02:00
Alex Hall
42afb17e36 (core) Run and test imports only in Python 3, upgrade openpyxl, fix weird date handling
Summary:
Python 2 only needs to be supported for the sake of old documents and formulas. This doesn't apply to the separate sandboxes that parse files for imports. Using Python 3 only allows using newer libraries and library versions. In particular, the latest version of openpyxl doesn't support Python 2. This will also make it easier to make other similar changes in the future, such as replacing messytables with a modern library. See https://grist.slack.com/archives/C0234CPPXPA/p1661261829343999?thread_ts=1661260442.837959&cid=C0234CPPXPA

The latest openpyxl is better at handling a particular edge case with broken dates in Excel, but still doesn't quite do what we want, so we monkeypatch it. Discussion: https://grist.slack.com/archives/C02EGJ1FUCV/p1661440851911869?thread_ts=1661154219.515549&cid=C02EGJ1FUCV

Setting `preferredPythonVersion` to '3' in SafePythonComponent ensures that JS always creates import sandboxes that use Python 3. Within Python, a module used by all imports will raise an error in Python 2. Python unit tests of imports are now only run in Python 3, using the `load_tests` protocol of `unittest`.

Test Plan: Mostly existing tests. Added another strange date to the Excel fixture.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3606
2022-09-02 16:27:34 +02:00
Alex Hall
1c43aed5dd (core) Upgrade sortedcontainers, python_dateutil, and html5lib for better Python 3.10 compatibility
Summary:
Based on https://github.com/gristlabs/grist-core/pull/251. It may not look like it, but there's very little going on in this diff:

- Tweak the DATEVALUE doctest for Python 2/3 compatibility.
- Mirrors the PR's changes to requirements3.txt in requirements.txt, i.e. make the same dependency upgrades in Python 2.
- Make the same upgrades in the thirdparty folder for the Python 2 nacl sandbox.

Test Plan: Updated one doctest for dateutil. Checked changelog of sortedcontainers. html5lib is only used by messytables and isn't actually relevant.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3609
2022-08-30 16:46:29 +02:00
Paul Fitzpatrick
63683f98cc (core) updates from grist-core 2022-08-26 17:29:25 -04:00
Yohan Boniface
ac9f2fbdd1 Update some python deps to prepare python>=3.10 compat
ABC classes have moved from collections to collections.abc since
python 3.3, and retro compat has ended with 3.8
2022-08-26 16:45:24 +02:00
Alex Hall
eac1f26f3e (core) More helpful messages when formula probably needs to use Table.all
Summary:
Raise an exception with a customised message for two cases when a user tries on operation directly on a table without `.all`:

1. For `Table.Col`, where `Col` is an existing column, suggest `Table.all.Col`. If `Col` doesn't exist as a column, fall back to the standard AttributeError.
2. When iterating directly over a table, e.g. `[r for r in Table]`, suggest looping over `Table.all` instead.

Test Plan: Added Python unit tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3593
2022-08-24 14:49:33 +02:00
Alex Hall
5f17dd0a06 (core) Convince grist-help that REQUEST is really unimplemented
Summary:
Change made to allow rebuilding functions in https://github.com/gristlabs/grist-help/pull/170

Deals the error described here: https://grist.slack.com/archives/C0234CPPXPA/p1660245534781629

Test Plan: Regenerating functions in grist-help works again.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3594
2022-08-23 19:25:27 +02:00
Alex Hall
42060df29a (core) Formula autocomplete improvements for references and lookups
Summary:
Makes the following improvements to formula autocomplete:

- When a user types `$RefCol` (or part of it), also show `$RefCol.VisibleCol` (replace actual column names) in the autocomplete even before the `.` is typed, to help users understand the difference between a raw reference/record and its visible column.
- When a user types a table name, show `.lookupOne` and `.lookupRecords` in the autocomplete, again even before the `.` is typed.
- For `.lookupRecords(` and `.lookupOne(`, once the `(` is entered, suggest each column name as a keyword argument.
- Also suggest lookup arguments involving compatible reference columns, especially 'reverse reference' lookups like `refcol=$id` which are very common and difficult for users.
- To support these features, the Ace editor autocomplete needs some patching to fetch fresh autocomplete options after typing `.` or `(`. This also improves unrelated behaviour that wasn't great before when one column name is contained in another. See the first added browser test.

Discussions:

- https://grist.slack.com/archives/CDHABLZJT/p1659707068383179
- https://grist.quip.com/HoSmAlvFax0j#MbTADAH5kgG
- https://grist.quip.com/HoSmAlvFax0j/Formula-Improvements#temp:C:MbT3649fe964a184e8dada9bbebb

Test Plan: Added Python and nbrowser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3580
2022-08-20 19:11:41 +02:00
Alex Hall
49cb51bac5 (core) Error explanations from friendly-traceback
Summary: Extend formula error messages with explanations from https://github.com/friendly-traceback/friendly-traceback. Only for Python 3.

Test Plan: Updated several Python tests. In general, these require separate branches for Python 2 and 3.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3542
2022-08-12 19:45:00 +02:00
Alex Hall
31f54065f5 (core) Use standard library uuid.uuidv4 when possible for better randomness
Summary: Using the `random` module in the Grist `UUID()` function is not cryptographically secure, and is only necessary for the old pynbox (Python 2) sandbox which doesn't support `os.urandom`. This diff uses the `uuid.uuidv4()` function from the Python standard library when possible, which is more secure, only falling back to the old implementation when necessary.

Test Plan: Added Python unit tests to check both implementations.

Reviewers: dsagal

Subscribers: paulfitz, dsagal

Differential Revision: https://phab.getgrist.com/D3578
2022-08-12 16:45:11 +02:00
Alex Hall
b416a5c4b1 (core) Fix error when updating summary table formulas after rename
Summary:
Formulas in summary tables were being associated with the source table for automatic updating. When a table/column was renamed such that the formula needed to update to match, it would look for a column with the same colId but in the source table. Such a column might not exist which would lead to an error, or if it existed then the update would be wrong.

This association was created while building formulas to display in the code view in a nested `_Summary` class, it didn't need to exist at all. So this diff simply prevents the association from being created.

User report and discussion: https://grist.slack.com/archives/C0234CPPXPA/p1659717322297019

Test Plan: Extended `TestSummary.test_table_rename` Python test.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3568
2022-08-11 12:51:11 +02:00
Jarosław Sadziński
9e4d802405 (core) Implementing row conditional formatting
Summary:
Conditional formatting can now be used for whole rows.
Related fix:
- Font styles weren't applicable for summary columns.
- Checkbox and slider weren't using colors properly

Test Plan: Existing and new tests

Reviewers: paulfitz, georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3547
2022-08-09 20:11:36 +02:00
Alex Hall
083a0ec000 (core) Allow using MIN and MAX functions with dates/datetimes
Summary:
The MIN and MAX functions for formulas previously only considered numbers, ignoring other types, including dates. An example of this being a problem is here: https://community.getgrist.com/t/last-field-circularreferror-what-is-it/1114/4 . Using `MIN` on a column of dates would return 0 (the default) which gets converted to 1970-01-01. Users have to use `min` instead, which is confusing, and doesn't work when some values are empty.

This diff lets the functions operate on date and datetime values. A mixture of dates and datetimes is allowed, even though these cannot usually be compared in Python. Mixing dates and numbers will raise an exception.

Test Plan: Extended doctests

Reviewers: jarek, paulfitz

Reviewed By: jarek

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3560
2022-08-09 13:34:22 +02:00
George Gevoian
771e1edd54 (core) Keep track of row counts per table
Summary: Displays a live row count of each table on the Raw Data page.

Test Plan: Browser tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3540
2022-08-03 08:13:33 -07:00
Paul Fitzpatrick
7078922a65 (core) ensure randomness works when sandbox is cloned from a checkpoint
Summary:
This calls a new `initialize` method on the sandbox before we start
doing calculations with it, to make sure that `random.seed()` has
been called. Otherwise, if the sandbox is cloned from a checkpoint,
the seed will have been reset.

The `initialize` method includes the functionality previously done
by `set_doc_url` since it is also initialization/personalization and
this way we avoid introducing another round trip to the sandbox.

Test Plan: tested with grist-core configured to use gvisor

Reviewers: georgegevoian, dsagal

Reviewed By: georgegevoian, dsagal

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3549
2022-07-27 14:59:27 -04:00
Alex Hall
938928f1b9 (core) Fix renaming columns when iterating over Table.all in comprehension
Summary:
Comprehensions iterating over `Table.all` like `[foo.bar for foo in Table.all]` led to an error when renaming the column `bar`. This diff fixes that so that renaming `bar` does the same thing as for a comprehension over `Table.lookupRecords()`. Note that `next(foo for foo in Table.all).bar` is still not supported, as the same is not supported for `Table.lookupRecords()` either.

Discussion: https://grist.slack.com/archives/C069RUP71/p1658360276762949

Test Plan: Parametrised existing Python test to test the same thing for both `all` and `lookupRecords`

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3538
2022-07-21 18:30:16 +02:00
Alex Hall
d221aeed24 (core) Update _grist_ACLResources.tableId when migration summary table IDs
Summary: https://grist.slack.com/archives/C0234CPPXPA/p1658146315294109

Test Plan: Updated test

Reviewers: dsagal, paulfitz

Reviewed By: dsagal, paulfitz

Differential Revision: https://phab.getgrist.com/D3535
2022-07-18 16:49:34 +02:00
Alex Hall
b8486dcdba (core) Nice summary table IDs
Summary:
Changes auto-generated summary table IDs from e.g. `GristSummary_6_Table1` to `Table1_summary_A_B` (meaning `Table1` grouped by `A` and `B`). This makes it easier to write formulas involving summary tables, make API requests, understand logs, etc.

Because these don't encode the source table ID as reliably as before, `decode_summary_table_name` now uses the summary table schema info, not just the summary table ID. Specifically, it looks at the type of the `group` column, which is `RefList:<source table id>`.

Renaming a source table renames the summary table as before, and now renaming a groupby column renames the summary table as well.

Conflicting table names are resolved in the usual way by adding a number at the end, e.g. `Table1_summary_A_B2`. These summary tables are not automatically renamed when the disambiguation is no longer needed.

A new migration renames all summary tables to the new scheme, and updates formulas using summary tables with a simple regex.

Test Plan:
Updated many tests to use the new style of name.

Added new Python tests to for resolving conflicts when renaming source tables and groupby columns.

Added a test for the migration, including renames in formulas.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3508
2022-07-14 12:09:56 +02:00
Alex Hall
29fb3360b6 (core) Update attributes of Table.all when column is renamed
Summary: Adds an InferenceTip which treats `Table.all` similarly to `Table.lookupRecords(...)`, so that `Table.all.foo` is changed to `Table.all.bar` when the column `foo` is renamed to `bar`.

Test Plan: Extended test for the `lookupRecords` case.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3521
2022-07-14 12:08:38 +02:00
Alex Hall
77775401fc (core) Don't clear widget options when changing column type
Summary: Previously, changing the type of a column would clear its widget options and conditional style rules by default, with a few exceptions to explicitly keep them. This diff reverses that behaviour, keeping the options by default.

Test Plan: Updated several existing tests, plus lots of manual testing.

Reviewers: cyprien

Reviewed By: cyprien

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3491
2022-07-11 14:27:44 +02:00
Alex Hall
0bdc82a170 (core) Automatically remove empty summary table rows
Summary: When the `getSummarySourceGroup` function (used by the `$group` column) finds that the group is empty, raise a new special exception `EmptySummaryRow`. The engine catches this exception, avoids saving a value to the cell, and removes the record.

Test Plan: Updated several Python tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3489
2022-07-08 18:56:41 +02:00
George Gevoian
a051830aeb (core) Show summary tables on Raw Data page
Summary:
Summary tables now have their own raw viewsection, and are shown
under Raw Data Tables on the Raw Data page.

Test Plan: Browser and Python tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3495
2022-07-06 09:41:48 -07:00
Alex Hall
8bab8c18fa (core) Clear error.__traceback__ to prevent memory leaks
Summary: When a formula raises an exception, we store that in the cell in memory. In Python 3, exceptions have a `__traceback__` attribute, which includes all the stack frames and local variables. This has huge memory leak potential. We already strategically format the exception when needed, we don't need to keep storing the actual traceback object.

Test Plan:
Manually tested that tracebacks are still sensible.

To check the effect on memory usage, made a simple test doc with 30k rows all containing an exception, and here's what ps aux says:

```
       %MEM    VSZ   RSS
before: 2.4 681996 588828
after:  1.6 499052 405712
```

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3505
2022-07-06 13:43:14 +02:00