gristlabs_grist-core

Commit Graph

Author	SHA1	Message	Date
Dmitry S	534615dd50	(core) Update logging in sandbox code, and log tracebacks as single log messages. Summary: - Replace logger module by the standard module 'logging'. - When a log message from the sandbox includes newlines (e.g. for tracebacks), keep those lines together in the Node log message. Previously each line was a different message, making it difficult to view tracebacks, particularly in prod where each line becomes a separate message object. - Fix assorted lint errors. Test Plan: Added a test for the log-line splitting and escaping logic. Reviewers: georgegevoian Reviewed By: georgegevoian Differential Revision: https://phab.getgrist.com/D3956	1 year ago
Alex Hall	56624c4a95	(core) Fix undo error for automatically removed rows, especially in summary tables Summary: Fixes a bug noted here: https://grist.slack.com/archives/C069RUP71/p1662564341132349 This bug could happen quite easily as follows: 1. Have a formula in a summary table such as `$group.amount`. Typically there's also a `SUM` but that's not essential. 2. Find a group with nonzero values of `amount`. 3. Delete all rows in that group in the source table. Typically that just means one row in a lonely group. 4. The summary table row is automatically deleted. 5. Try to undo. This raises an error about trying to update a non-existent summary table row. I tried to account for this undo problem in https://phab.getgrist.com/D3489 by not saving the updated value for `$group` when it was found to be empty. The reason this was insufficient is that `$group.amount` is immediately invalidated anyway when the source row(s) are deleted (I think because that's just how dependency relations involving references work) and the calculated value of `$group.amount` changes even if `$group` doesn't. For example, `$group.amount` may have previously been `[100, 200]`. After deleting the rows, `$group.amount` becomes `[0, 0]`. Keeping `$group` unchanged prevents `$group.amount` from just being `[]`, but deleting the source rows means that the amounts become the numeric default `0` which is still a change. This change in value is then noted which leads to saving an undo action to update the summary table record. All this happens in step 3 above, and the summary record is only deleted after that point. This diff removes that special handling for `group` and instead adds a more general fix to `action_summary.py`. This inserts undo actions for deleted rows at the beginning of the undo list rather than at the end, which was already done for deleted tables and columns. Test Plan: Python tests Reviewers: dsagal Reviewed By: dsagal Subscribers: dsagal Differential Revision: https://phab.getgrist.com/D3626	2 years ago
Alex Hall	b8486dcdba	(core) Nice summary table IDs Summary: Changes auto-generated summary table IDs from e.g. `GristSummary_6_Table1` to `Table1_summary_A_B` (meaning `Table1` grouped by `A` and `B`). This makes it easier to write formulas involving summary tables, make API requests, understand logs, etc. Because these don't encode the source table ID as reliably as before, `decode_summary_table_name` now uses the summary table schema info, not just the summary table ID. Specifically, it looks at the type of the `group` column, which is `RefList:<source table id>`. Renaming a source table renames the summary table as before, and now renaming a groupby column renames the summary table as well. Conflicting table names are resolved in the usual way by adding a number at the end, e.g. `Table1_summary_A_B2`. These summary tables are not automatically renamed when the disambiguation is no longer needed. A new migration renames all summary tables to the new scheme, and updates formulas using summary tables with a simple regex. Test Plan: Updated many tests to use the new style of name. Added new Python tests to for resolving conflicts when renaming source tables and groupby columns. Added a test for the migration, including renames in formulas. Reviewers: georgegevoian Reviewed By: georgegevoian Differential Revision: https://phab.getgrist.com/D3508	2 years ago
Alex Hall	0bdc82a170	(core) Automatically remove empty summary table rows Summary: When the `getSummarySourceGroup` function (used by the `$group` column) finds that the group is empty, raise a new special exception `EmptySummaryRow`. The engine catches this exception, avoids saving a value to the cell, and removes the record. Test Plan: Updated several Python tests Reviewers: georgegevoian Reviewed By: georgegevoian Subscribers: dsagal Differential Revision: https://phab.getgrist.com/D3489	2 years ago
Jarosław Sadziński	f194d6861b	(core) Updating RawData views Summary: - Better focus on the widget title - Adding columns only to the current view section - New popup with options when user wants to delete a page - New dialog to enter table name - New table as a widget doesn't create a separate page - Removing a table doesn't remove the primary view Test Plan: Updated and new tests Reviewers: georgegevoian Reviewed By: georgegevoian Differential Revision: https://phab.getgrist.com/D3410	2 years ago
Alex Hall	437d30bd9f	(core) Log number of rows in user tables in data engine Summary: Adds a method Table._num_rows using an empty lookup map column. Adds a method Engine.count_rows which adds them all up. Returns the count after applying user actions to be logged by ActiveDoc. Test Plan: Added a unit test in Python. Tested log message manually. Reviewers: paulfitz Reviewed By: paulfitz Differential Revision: https://phab.getgrist.com/D3275	3 years ago
Dmitry S	f024aaaf5d	(core) Fix some bugs with ChoiceList in summary tables, and evaluation of lookups. Summary: Addresses several issues: - Error 'Cannot modify summary group-by column' when changing Text -> ChoiceList in the presence of summary tables. - Error 'ModifyColumn in unexpected position' when changing ChoiceList -> Text in the presence of summary tables. - Double-evaluation of trigger formulas in some cases. Fixes include: - Fixed verification that summary group-by columns match the underlying ones, and added comments to explain. - Avoid updating non-metadata lookups after each doc-action (early lookups generated extra actions to populate summary tables, causing the 'ModifyColumn in unexpected position' bug) - When updating formulas, do update lookups first. - Made a client-side tweak to avoid a JS error in case of some undos. Solution to reduce lookups is based on https://phab.getgrist.com/D3069?vs=on&id=12445, and tests for double-evaluation of trigger formulas are taken from there. Add a new test case to protect against bugs caused by incorrect order of evaluating #lookup columns. Enhanced ChoiceList browser test to check a conversion scenario in the presence of summary tables, previously triggering bugs. Test Plan: Various tests added or enhanced. Reviewers: alexmojaki Reviewed By: alexmojaki Subscribers: jarek Differential Revision: https://phab.getgrist.com/D3184	3 years ago
Dmitry S	e2226c3ab7	(core) Store formula values in DB, and include them into .stored/.undo fields of actions. Summary: - Introduce a new SQLiteDB migration, which adds DB columns for formula columns - Newly added columns have the special ['P'] (pending) value in them (in order to show the usual "Loading..." on the first load that triggers the migration) - Calculated values are added to .stored/.undo fields of user actions. - Various changes made in the sandbox to include .stored/.undo in the right order. - OnDemand tables ignore stored formula columns, replacing them with special SQL as before - In particular, converting to OnDemand table leaves stale values in those columns, we should maybe clean those out. Some tweaks on the side: - Allow overriding chai assertion truncateThreshold with CHAI_TRUNCATE_THRESHOLD - Rebuild python automatically in watch mode Test Plan: Fixed various tests, updated some fixtures. Many python tests that check actions needed adjustments because actions moved from .stored to .undo. Some checks added to catch situations previously only caught in browser tests. Reviewers: paulfitz Reviewed By: paulfitz Differential Revision: https://phab.getgrist.com/D2645	4 years ago
Paul Fitzpatrick	b82eec714a	(core) move data engine code to core Summary: this moves sandbox/grist to core, and adds a requirements.txt file for reconstructing the content of sandbox/thirdparty. Test Plan: existing tests pass. Tested core functionality manually. Tested docker build manually. Reviewers: dsagal Reviewed By: dsagal Differential Revision: https://phab.getgrist.com/D2563	4 years ago

9 Commits (d6ff2ec60b12e1030e21389fcd4a8a2d6e11426e)