Commit Graph

14 Commits

Author SHA1 Message Date
Alex Hall
9b36fb4dab (core) Fix error in sandbox when removing multiple summary source columns
Summary:
Fixes a very specific bug reported here: https://grist.slack.com/archives/C069RUP71/p1694630242765769

The error occurred when:

1. Removing multiple columns simultaneously
2. Those columns were sources of groupby columns for a summary table (so removing them meant recreating the summary table and thus deleting its columns)
3. There was a display column for one of the columns that got deleted (either directly or indirectly) which was set to be automatically removed since it was no longer needed, but this failed because the column was already deleted as part of earlier table removal.

I fixed this by making `apply_auto_removes` remove table records last, so removing the display column wouldn't be a problem.

That fixed the original error, but then I noticed that trying to undo the removal could lead to another error (under even more specific circumstances). It's hard to see exactly why, but I can see that just 3 `RemoveColumn` user actions generated over 100 doc actions and corresponding undo actions, hence the difficulty in narrowing the problem down. This is partly because removing a single column would recreate a summary table, only for that table to be immediately replaced again when another column was removed. Making the frontend send a single `[BulkRemoveRecord, _grist_Tables_column, ...] ` leads to a more efficient and sensible process with about half as many doc actions and no undo error. I think this alone would also solve the original error, but the data engine change seems more generally helpful and worth keeping.

Test Plan: Added a Python test and an nbrowser test

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4052
2023-09-27 16:23:49 +02:00
Dmitry S
534615dd50 (core) Update logging in sandbox code, and log tracebacks as single log messages.
Summary:
- Replace logger module by the standard module 'logging'.
- When a log message from the sandbox includes newlines (e.g. for tracebacks),
  keep those lines together in the Node log message.

  Previously each line was a different message, making it difficult to view
  tracebacks, particularly in prod where each line becomes a separate message
  object.

- Fix assorted lint errors.

Test Plan: Added a test for the log-line splitting and escaping logic.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3956
2023-07-18 11:21:25 -04:00
Alex Hall
56624c4a95 (core) Fix undo error for automatically removed rows, especially in summary tables
Summary:
Fixes a bug noted here: https://grist.slack.com/archives/C069RUP71/p1662564341132349

This bug could happen quite easily as follows:

1. Have a formula in a summary table such as `$group.amount`. Typically there's also a `SUM` but that's not essential.
2. Find a group with nonzero values of `amount`.
3. Delete all rows in that group in the source table. Typically that just means one row in a lonely group.
4. The summary table row is automatically deleted.
5. Try to undo. This raises an error about trying to update a non-existent summary table row.

I tried to account for this undo problem in https://phab.getgrist.com/D3489 by not saving the updated value for `$group` when it was found to be empty. The reason this was insufficient is that `$group.amount` is immediately invalidated anyway when the source row(s) are deleted (I think because that's just how dependency relations involving references work) *and* the calculated value of `$group.amount` changes even if `$group` doesn't. For example, `$group.amount` may have previously been `[100, 200]`. After deleting the rows, `$group.amount` becomes `[0, 0]`. Keeping `$group` unchanged prevents `$group.amount` from just being `[]`, but deleting the source rows means that the amounts become the numeric default `0` which is still a change. This change in value is then noted which leads to saving an undo action to update the summary table record. All this happens in step 3 above, and the summary record is only deleted after that point.

This diff removes that special handling for `group` and instead adds a more general fix to `action_summary.py`. This inserts undo actions for deleted rows at the beginning of the undo list rather than at the end, which was already done for deleted tables and columns.

Test Plan: Python tests

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3626
2022-09-09 22:15:45 +02:00
Alex Hall
0a2bc56938 (core) Test undo for all Python summary table tests
Summary:
Undo often leads to errors, especially with summary tables. One example is here: https://grist.slack.com/archives/C069RUP71/p1662564341132349

This diff simply decorates all relevant tests in 3 files testing summary tables with `@test_engine.test_undo`. This didn't catch any new bugs or reveal the problem in the thread above, but it seems good to have.

Test Plan: this

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3624
2022-09-07 18:02:02 +02:00
Alex Hall
b416a5c4b1 (core) Fix error when updating summary table formulas after rename
Summary:
Formulas in summary tables were being associated with the source table for automatic updating. When a table/column was renamed such that the formula needed to update to match, it would look for a column with the same colId but in the source table. Such a column might not exist which would lead to an error, or if it existed then the update would be wrong.

This association was created while building formulas to display in the code view in a nested `_Summary` class, it didn't need to exist at all. So this diff simply prevents the association from being created.

User report and discussion: https://grist.slack.com/archives/C0234CPPXPA/p1659717322297019

Test Plan: Extended `TestSummary.test_table_rename` Python test.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3568
2022-08-11 12:51:11 +02:00
Alex Hall
b8486dcdba (core) Nice summary table IDs
Summary:
Changes auto-generated summary table IDs from e.g. `GristSummary_6_Table1` to `Table1_summary_A_B` (meaning `Table1` grouped by `A` and `B`). This makes it easier to write formulas involving summary tables, make API requests, understand logs, etc.

Because these don't encode the source table ID as reliably as before, `decode_summary_table_name` now uses the summary table schema info, not just the summary table ID. Specifically, it looks at the type of the `group` column, which is `RefList:<source table id>`.

Renaming a source table renames the summary table as before, and now renaming a groupby column renames the summary table as well.

Conflicting table names are resolved in the usual way by adding a number at the end, e.g. `Table1_summary_A_B2`. These summary tables are not automatically renamed when the disambiguation is no longer needed.

A new migration renames all summary tables to the new scheme, and updates formulas using summary tables with a simple regex.

Test Plan:
Updated many tests to use the new style of name.

Added new Python tests to for resolving conflicts when renaming source tables and groupby columns.

Added a test for the migration, including renames in formulas.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3508
2022-07-14 12:09:56 +02:00
Alex Hall
0bdc82a170 (core) Automatically remove empty summary table rows
Summary: When the `getSummarySourceGroup` function (used by the `$group` column) finds that the group is empty, raise a new special exception `EmptySummaryRow`. The engine catches this exception, avoids saving a value to the cell, and removes the record.

Test Plan: Updated several Python tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3489
2022-07-08 18:56:41 +02:00
George Gevoian
a051830aeb (core) Show summary tables on Raw Data page
Summary:
Summary tables now have their own raw viewsection, and are shown
under Raw Data Tables on the Raw Data page.

Test Plan: Browser and Python tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3495
2022-07-06 09:41:48 -07:00
Jarosław Sadziński
f194d6861b (core) Updating RawData views
Summary:
- Better focus on the widget title
- Adding columns only to the current view section
- New popup with options when user wants to delete a page
- New dialog to enter table name
- New table as a widget doesn't create a separate page
- Removing a table doesn't remove the primary view

Test Plan: Updated and new tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3410
2022-05-04 21:41:42 +02:00
Cyprien P
0829ae67ef (core) Fixing hidden col missing after summary update
Summary:
Updating one summary groupby columns is handle by the
`update_summary_section`. That routine is in charge of creating a new
summary table and transfer all possible columns from old table to the
new one, so that the new table appear to change only the minimum to
users.

The problem that this diff address is that, the logic to decide what columns we need to keep, only applies to the visible
fields and ignore the hidden columns, causing all hidden columns to be removed. This diff, simply
changes that routine to address all columns.

the mutliple fixes on the test_summary2.py result from a change of
ordering of columns: the culprit is the `group` columns, which is by
default a hidden columns and that had to be explicitely added back to
the new summary table. It is now transferred automatically, like other
columns, which does cause a little change of ordering on the db. This
should not result in any display order changes for the users.

Recent related diff: https://phab.getgrist.com/D3351

Test Plan: Includes new test case; both python and nbrowser

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3393
2022-04-27 14:00:30 +02:00
Jarosław Sadziński
d2077bc486 (core) Improving experience when editing group-by column.
Summary:
Improving experience when editing group-by column:
- Disable column rename
- Allow changing most widget properties:
 - Color/Background
 - Number format
 - Date/DateTime format (but not the timezone)
 - All toggle options (for toggle column)
- Remove Edit button on Choice Edit
- Changing the underlying column should reset all those options back to the original column.

Test Plan: nbrowser

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3216
2022-01-18 14:31:31 +01:00
Alex Hall
02fd71d9bb (core) Nicer conversion from numeric to text
Summary: Expanded Text.do_convert for float values

Test Plan: Add python unit test test_numeric_to_text_conversion

Reviewers: jarek

Reviewed By: jarek

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3049
2021-09-30 14:16:24 +02:00
Dmitry S
e2226c3ab7 (core) Store formula values in DB, and include them into .stored/.undo fields of actions.
Summary:
- Introduce a new SQLiteDB migration, which adds DB columns for formula columns
- Newly added columns have the special ['P'] (pending) value in them
  (in order to show the usual "Loading..." on the first load that triggers the migration)
- Calculated values are added to .stored/.undo fields of user actions.
- Various changes made in the sandbox to include .stored/.undo in the right order.
- OnDemand tables ignore stored formula columns, replacing them with special SQL as before
- In particular, converting to OnDemand table leaves stale values in those
  columns, we should maybe clean those out.

Some tweaks on the side:
- Allow overriding chai assertion truncateThreshold with CHAI_TRUNCATE_THRESHOLD
- Rebuild python automatically in watch mode

Test Plan: Fixed various tests, updated some fixtures. Many python tests that check actions needed adjustments because actions moved from .stored to .undo. Some checks added to catch situations previously only caught in browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2645
2020-11-04 16:45:47 -05:00
Paul Fitzpatrick
b82eec714a (core) move data engine code to core
Summary:
this moves sandbox/grist to core, and adds a requirements.txt
file for reconstructing the content of sandbox/thirdparty.

Test Plan:
existing tests pass.
Tested core functionality manually.  Tested docker build manually.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2563
2020-07-29 08:57:25 -04:00