Summary:
Loading all user data to run a migration is risky (creates more than usual
memory pressure), and almost never needed (only one migration requires it).
This diff attempts to run migrations using only metadata (_grist_* tables),
but retries if the sandbox tells it that all data is needed.
The intent is for new migrations to avoid needing all data.
Test Plan: Added a somewhat contrived unittest.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2659
Summary:
- Introduce a new SQLiteDB migration, which adds DB columns for formula columns
- Newly added columns have the special ['P'] (pending) value in them
(in order to show the usual "Loading..." on the first load that triggers the migration)
- Calculated values are added to .stored/.undo fields of user actions.
- Various changes made in the sandbox to include .stored/.undo in the right order.
- OnDemand tables ignore stored formula columns, replacing them with special SQL as before
- In particular, converting to OnDemand table leaves stale values in those
columns, we should maybe clean those out.
Some tweaks on the side:
- Allow overriding chai assertion truncateThreshold with CHAI_TRUNCATE_THRESHOLD
- Rebuild python automatically in watch mode
Test Plan: Fixed various tests, updated some fixtures. Many python tests that check actions needed adjustments because actions moved from .stored to .undo. Some checks added to catch situations previously only caught in browser tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2645
Summary:
This is a hacky solution that unfortunately relies on internal workings of ACE
autocomplete popups. I don't see a less hacky one if we stick with ACE
autocomplete.
Test Plan: Added a test case for links to test/nbrowser/Formulas.ts
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2610
Summary:
This makes it possible to serve a table or tables only to owners.
* The _grist_ACLResources table is abused (temporarily) such that rows of the form `{colId: '~o', tableId}` are interpreted as meaning that `tableId` is private to owners.
* Many websocket and api endpoints are updated to preserve the privacy of these tables.
* In a document where some tables are private, a lot of capabilities are turned off for non-owners to avoid leaking info indirectly.
* The client is tweaked minimally, to show '-' where a page with some private material would otherwise go.
No attempt is made to protect data from private tables pulled into non-private tables via formulas.
There are some known leaks remaining:
* Changes to the schema of private tables are still broadcast to all clients (fixable).
* Non-owner may be able to access snapshots or make forks or use other corners of API (fixable).
* Changing name of table makes it public, since tableId in ACLResource is not updated (fixable).
Security will require some work, the attack surface is large.
Test Plan: added tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2604
Summary:
- Disable a pointless timing check in Python that occasionally fails.
- Make the test of InvalidValues more robust by waiting for calculated values to load.
- Make Snapshots test more robust by waiting for an action that precedes a URL change.
Test Plan: These tests should fail less often when the tested logic is correct.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2609
Summary:
- Make suggestions less case-sensitive (not entirely case-insensitive, but
allow top-level suggestions to match in all-lowercase)
- Add function signatures to suggestions for Grist functions.
- Excel-like functions that are present but not implemented are no longer
offered as suggestions.
Test Plan:
Added a test case on python side, and a browser test case for how suggestions
are rendered and inserted.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2608
Summary:
- Change RECORD's dates_as_str default to False.
- Reimplement objtype encode_object/decode_object with less machinery.
- Implement encoding of dicts (with string keys).
- Make lists and dicts encode values recursively.
- Implement encoding/decoding in the client
- Decode automatically in plugins' fetchSelectedTable/Record, with an option to skip.
Test Plan: Tested manually, not sure what tests may be affected yet.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2593
Summary:
- Instead of sending an "UnmarshallableError" as an exception, introduce an
"Unmarshallable" type of value, represented as ['U', repr(value)]
- Unmarshallable values are rendered using a bluish text color, no longer a
pink background.
- Factor out ErrorDom to be simpler and cleaner.
- Add GristObjCode enum, and simplify related helpers.
- Use safe_repr() for when repr() itself fails
- Handle conversion errors using safe_repr() when str() fails
Test Plan: Added a test case based on a fixture covering a bunch of cases.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2584
Summary:
Examining a Record is currently difficult, because its columns are hard to
list (and to use), and CircularRef errors hard to avoid. The RECORD function
takes care of this mess to return a simple dictionary of values.
- Supports dates_as_iso=False flag to turn off the translation of date/datetime
objects to strings.
- Supports expand_refs=True flag to apply RECORD() to encountered values of
type Record, for a single level of nesting.
Test Plan: Added a unittest for RECORD()
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2582
Test Plan: Added a test case for this, which fails without the change.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2569
Summary:
this moves sandbox/grist to core, and adds a requirements.txt
file for reconstructing the content of sandbox/thirdparty.
Test Plan:
existing tests pass.
Tested core functionality manually. Tested docker build manually.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2563
