gristlabs_grist-core

Commit Graph

Author	SHA1	Message	Date
Alex Hall	a64fb105e3	(core) Use GristObjCode in CellValue Summary: Makes type checking a bit stronger Test Plan: it just has to compile Reviewers: jarek Reviewed By: jarek Differential Revision: https://phab.getgrist.com/D3065	3 years ago
George Gevoian	e1780e4f58	(core) Migrate import code from data engine to Node Summary: Finishing imports now occurs in Node instead of the data engine, which makes it possible to import into on-demand tables. Merging code was also refactored and now uses a SQL query to diff source and destination tables in order to determine what to update or add. Also fixes a bug where incremental imports involving Excel files with multiple sheets would fail due to the UI not serializing merge options correctly. Test Plan: Browser tests. Reviewers: jarek Reviewed By: jarek Differential Revision: https://phab.getgrist.com/D3046	3 years ago
Alex Hall	04e5d90f86	(core) Barely working reference lists in frontend Summary: This makes it possible to set the type of a column to ReferenceList, but the UI is terrible ReferenceList.ts is a mishmash of ChoiceList and Reference that sort of works but something about the CSS is clearly broken ReferenceListEditor is just a text editor, you have to type in a JSON array of row IDs. Ignore the value that's present when you start editing. I can maybe try mashing together ReferenceEditor and ChoiceListEditor but it doesn't seem wise. I think @georgegevoian should take over here. Reviewing the diff as it is to check for obvious issues is probably good but I don't think it's worth trying to land/merge anything. Test Plan: none Reviewers: dsagal Reviewed By: dsagal Subscribers: georgegevoian Differential Revision: https://phab.getgrist.com/D2914	3 years ago
Paul Fitzpatrick	6e15d44cf6	(core) start applying defenses for untrusted document uploads Summary: This applies some mitigations suggested by SQLite authors when opening untrusted SQLite databases, as we do when Grist docs are uploaded by the user. See: https://www.sqlite.org/security.html#untrusted_sqlite_database_files Steps implemented in this diff are: * Setting `trusted_schema` to off * Running a SQLite-level integrity check on uploads Other steps will require updates to our node-sqlite3 fork, since they are not available via the node-sqlite3 api (one more reason to migrate to better-sqlite3). I haven't yet managed to create a file that triggers an integrity check failure without also being detected as corruption by sqlite at a more basic level, so that is a TODO for testing. Test Plan: existing tests pass; need to come up with exploits to actually test the defences and have not yet Reviewers: dsagal Reviewed By: dsagal Differential Revision: https://phab.getgrist.com/D2909	3 years ago
Dmitry S	8d62a857e1	(core) Add ChoiceList type, cell widget, and editor widget. Summary: - Adds a new ChoiceList type, and widgets to view and edit it. - Store in SQLite as a JSON string - Support conversions between ChoiceList and other types Test Plan: Added browser tests, and a test for how these values are stored Reviewers: paulfitz Reviewed By: paulfitz Differential Revision: https://phab.getgrist.com/D2803	3 years ago
Dmitry S	526b0ad33e	(core) Configure more comprehensive eslint rules for Typescript Summary: - Update rules to be more like we've had with tslint - Switch tsserver plugin to eslint (tsserver makes for a much faster way to lint in editors) - Apply suggested auto-fixes - Fix all lint errors and warnings in core/, app/, test/ Test Plan: Some behavior may change subtly (e.g. added missing awaits), relying on existing tests to catch problems. Reviewers: paulfitz Reviewed By: paulfitz Differential Revision: https://phab.getgrist.com/D2785	3 years ago
Paul Fitzpatrick	9e8e895abd	(core) fix filters with many values when querying directly from db Summary: This fixes DocStorage.fetchQuery when the number of parameters exceeds the maximum that can be passed directly to sqlite. In this case, parameters are now stored and used from a temporary table. Problem first noticed via a use of DocStorage.fetchQuery by granular access controls. Access control should be optimized to make fewer such queries, but that is a separate issue. Test Plan: added tests Reviewers: dsagal Reviewed By: dsagal Differential Revision: https://phab.getgrist.com/D2772	3 years ago
Paul Fitzpatrick	d8df2404c2	(core) return to using meaningful SQL types for columns Summary: Previously in {{D1053}} we switched to using BLOB as the "type" for all columns, to prevent SQLite from casting data unexpectedly. This diff now returns to more meaningful types. We apply marshalling to values when being placed in a column where a cast might occur, to inhibit such casting. The benefit is that Grist documents become easier to interact with via regular database clients/libraries, which often rely on the column type more than a purely SQLite tool would. On column type conversion, we run all blobs in the column through a decode/encode cycle so if they no longer need to be marshalled they revert to native type. This could be optimized further, it is somewhat brute force. Test Plan: Updated tests and reference document Reviewers: dsagal Reviewed By: dsagal Differential Revision: https://phab.getgrist.com/D2755	4 years ago
Dmitry S	5b2de988b5	(core) Perform migrations of Grist schema using only metadata tables when possible. Summary: Loading all user data to run a migration is risky (creates more than usual memory pressure), and almost never needed (only one migration requires it). This diff attempts to run migrations using only metadata (_grist_* tables), but retries if the sandbox tells it that all data is needed. The intent is for new migrations to avoid needing all data. Test Plan: Added a somewhat contrived unittest. Reviewers: paulfitz Reviewed By: paulfitz Differential Revision: https://phab.getgrist.com/D2659	4 years ago
Dmitry S	e2226c3ab7	(core) Store formula values in DB, and include them into .stored/.undo fields of actions. Summary: - Introduce a new SQLiteDB migration, which adds DB columns for formula columns - Newly added columns have the special ['P'] (pending) value in them (in order to show the usual "Loading..." on the first load that triggers the migration) - Calculated values are added to .stored/.undo fields of user actions. - Various changes made in the sandbox to include .stored/.undo in the right order. - OnDemand tables ignore stored formula columns, replacing them with special SQL as before - In particular, converting to OnDemand table leaves stale values in those columns, we should maybe clean those out. Some tweaks on the side: - Allow overriding chai assertion truncateThreshold with CHAI_TRUNCATE_THRESHOLD - Rebuild python automatically in watch mode Test Plan: Fixed various tests, updated some fixtures. Many python tests that check actions needed adjustments because actions moved from .stored to .undo. Some checks added to catch situations previously only caught in browser tests. Reviewers: paulfitz Reviewed By: paulfitz Differential Revision: https://phab.getgrist.com/D2645	4 years ago
Paul Fitzpatrick	71519d9e5c	(core) revamp snapshot inventory Summary: Deliberate changes: * save snapshots to s3 prior to migrations. * label migration snapshots in s3 metadata. * avoid pruning migration snapshots for a month. Opportunistic changes: * Associate document timezone with snapshots, so pruning can respect timezones. * Associate actionHash/Num with snapshots. * Record time of last change in snapshots (rather than just s3 upload time, which could be a while later). This ended up being a biggish change, because there was nowhere ideal to put tags (list of possibilities in diff). Test Plan: added tests Reviewers: dsagal Reviewed By: dsagal Differential Revision: https://phab.getgrist.com/D2646	4 years ago
Paul Fitzpatrick	5ef889addd	(core) move home server into core Summary: This moves enough server material into core to run a home server. The data engine is not yet incorporated (though in manual testing it works when ported). Test Plan: existing tests pass Reviewers: dsagal Reviewed By: dsagal Differential Revision: https://phab.getgrist.com/D2552	4 years ago

12 Commits (fd957577d3b4a05c5314f6084a52ec4218793bea)