gristlabs_grist-core

Commit Graph

Author	SHA1	Message	Date
Alex Hall	fcbad1c887	(core) Add GET /attachments endpoint for listing attachment metadata Summary: Combines the code and behaviour of the existing endpoints `GET /records` (for the general shape of the result and the parameters for sort/filter/limit etc) and retrieving a specific attachment with `GET /attachments/:id` for handling fields specific to attachments. Test Plan: Added a DocApi test. Also updated one test to use the new endpoint instead of raw `GET /tables/_grist_Attachments/records`. Reviewers: cyprien Reviewed By: cyprien Subscribers: cyprien Differential Revision: https://phab.getgrist.com/D3443	2 years ago
George Gevoian	2fd8a34ff8	(core) Move Notifier to /ext Summary: This makes it possible to configure a SendGrid-based Notifier instance via a JSON configuration file. Test Plan: Tested manually. Reviewers: alexmojaki Reviewed By: alexmojaki Subscribers: paulfitz Differential Revision: https://phab.getgrist.com/D3432	2 years ago
Paul Fitzpatrick	0a61d740ba	(core) clear redis state for docapi tests Summary: DocApi tests occasionally fail in a confusing way due to redis state. This resets the redis state at the beginning of the test. Test Plan: existing tests pass Reviewers: alexmojaki Reviewed By: alexmojaki Subscribers: alexmojaki Differential Revision: https://phab.getgrist.com/D3424	2 years ago
George Gevoian	1e42871cc9	(core) Add attachment and data size usage Summary: Adds attachment and data size to the usage section of the raw data page. Also makes in-document usage banners update as user actions are applied, causing them to be hidden/shown or updated based on the current state of the document. Test Plan: Browser tests. Reviewers: jarek Reviewed By: jarek Subscribers: alexmojaki Differential Revision: https://phab.getgrist.com/D3395	2 years ago
Alex Hall	0beb2898cb	(core) Add flexibility to daily API usage limit Summary: Allow exceeding the daily API usage limit for a doc based on additional allocations for the current hour and minute. See the doc comment on getDocApiUsageKeysToIncr for details. This means that up to 5 redis keys may be relevant at a time for a single document. Test Plan: Updated and expanded 'Daily API Limit' tests. Reviewers: dsagal Reviewed By: dsagal Differential Revision: https://phab.getgrist.com/D3368	2 years ago
Alex Hall	a701b4bf13	(core) Remove expired attachments every hour and on shutdown Summary: Call ActiveDoc.removeUnusedAttachments every hour using setInterval, and in ActiveDoc.shutdown (which also clears said interval). Unrelated: small fix to my webhooks code which was creating a redis client on shutdown just to quit it. Test Plan: Tweaked DocApi test to remove expired attachments by force-reloading the doc, so that it removes them during shutdown. Extracted a new testing endpoint /verifyFiles to support this test (previously running that code only happened with `/removeUnused?verifyfiles=1`). Tested the setInterval part manually. Reviewers: paulfitz, dsagal Reviewed By: paulfitz Subscribers: dsagal Differential Revision: https://phab.getgrist.com/D3387	2 years ago
Alex Hall	09da815c0c	(core) Add /attachments/removeUnused DocApi endpoint to hard delete all unused attachments in document Summary: Adds methods to delete metadata rows based on timeDeleted. The flag expiredOnly determines if it only deletes attachments that were soft-deleted 7 days ago, or just all soft-deleted rows. Then any actual file data that doesn't have matching metadata is deleted. Test Plan: DocApi test Reviewers: paulfitz Reviewed By: paulfitz Subscribers: dsagal Differential Revision: https://phab.getgrist.com/D3364	2 years ago
Alex Hall	64369df4c3	(core) Add /attachments/updateUsed DocApi endpoint to soft delete all unused attachments in document Summary: Builds on https://phab.getgrist.com/D3352 Add DocStorage.scanAttachmentsForUsageChanges to do fancy JSON query to find all attachment metadata rows whose soft deletion status needs updating. Add ActiveDoc.updateUsedAttachments which uses the above and then applies the appropriate user action if needed to soft delete/undelete metadata rows. Add endpoint in DocApi calling ActiveDoc method. Test Plan: Added DocApi test Reviewers: paulfitz Reviewed By: paulfitz Differential Revision: https://phab.getgrist.com/D3357	2 years ago
Paul Fitzpatrick	c6d66e15bf	(core) configure typedoc for generating plugin api documentation Summary: This annotates the plugin api sufficiently to generate some documentation for it. See https://github.com/gristlabs/grist-help/pull/139 Contains some small code tweaks for things that caused typedoc some trouble. Test Plan: manual inspection of output Reviewers: jarek Reviewed By: jarek Differential Revision: https://phab.getgrist.com/D3342	2 years ago
Paul Fitzpatrick	24522e61ff	remove stray redis dependency, and upgrade node in tests (#173 ) * remove stray redis dependency in test * tweak handling of database connection between tests * upgrade node versions in tests, type guessing in node 10 has problems	3 years ago
Paul Fitzpatrick	de703343d0	(core) disentangle some server tests, release to core, add GRIST_PROXY_AUTH_HEADER test Summary: This shuffles some server tests to make them available in grist-core, and adds a test for the `GRIST_PROXY_AUTH_HEADER` feature added in https://github.com/gristlabs/grist-core/pull/165 It includes a fix for a header normalization issue for websocket connections. Test Plan: added test Reviewers: georgegevoian Reviewed By: georgegevoian Differential Revision: https://phab.getgrist.com/D3326	3 years ago
Paul Fitzpatrick	c4fb40e0bd	(core) add gvisor-based sandbox to jenkins test worker image Summary: This is preparatory work for running tests with the new sandbox in jenkins. * Makes a base image that is now shared by grist servers and jenkins workers. * Needed to allow jenkins to run `sudo runsc`. * Converged on port 2020 for ssh to workers and servers. Test Plan: added one runsc-based test and confirmed it was run Reviewers: dsagal Reviewed By: dsagal Differential Revision: https://phab.getgrist.com/D3029	3 years ago
Paul Fitzpatrick	6e15d44cf6	(core) start applying defenses for untrusted document uploads Summary: This applies some mitigations suggested by SQLite authors when opening untrusted SQLite databases, as we do when Grist docs are uploaded by the user. See: https://www.sqlite.org/security.html#untrusted_sqlite_database_files Steps implemented in this diff are: * Setting `trusted_schema` to off * Running a SQLite-level integrity check on uploads Other steps will require updates to our node-sqlite3 fork, since they are not available via the node-sqlite3 api (one more reason to migrate to better-sqlite3). I haven't yet managed to create a file that triggers an integrity check failure without also being detected as corruption by sqlite at a more basic level, so that is a TODO for testing. Test Plan: existing tests pass; need to come up with exploits to actually test the defences and have not yet Reviewers: dsagal Reviewed By: dsagal Differential Revision: https://phab.getgrist.com/D2909	3 years ago
Alex Hall	8a940676e9	(core) Generic tools for recording pycalls, deterministic mode. Summary: Replaces https://phab.getgrist.com/D2854 Refactoring of NSandbox: - Simplify arguments to NSandbox.spawn. Only half the arguments were used depending on the flavour, adding a layer of confusion. - Ensure the same environment variables are passed to both flavours of sandbox - Simplify passing down environment variables. Implement deterministic mode with libfaketime and a seeded random instance. - Include static prebuilt libfaketime.so.1, may need another solution in future for other platforms. Recording pycalls: - Add script recordDocumentPyCalls.js to open a single document outside of tests. - Refactor out recordPyCalls.ts to support various uses. - Add afterEach hook to save all pycalls from server tests under $PYCALLS_DIR - Make docTools usable without mocha. - Add useLocalDoc and loadLocalDoc for loading non-fixture documents Test Plan: Made a document with formulas NOW() and UUID() Compare two document openings in normal mode: diff <(test/recordDocumentPyCalls.js samples/d4W6NrzCMNVSVD6nWgNrGC.grist /dev/stdout) \ <(test/recordDocumentPyCalls.js samples/d4W6NrzCMNVSVD6nWgNrGC.grist /dev/stdout) Output: < 1623407499.58132, --- > 1623407499.60376, 1195c1195 < "B": "bd2487f6-63c9-4f02-bbbc-5c0d674a2dc6" --- > "B": "22e1a4fd-297f-4b86-91a2-bc42cc6da4b2" `export DETERMINISTIC_MODE=1` and repeat. diff is empty! Reviewers: paulfitz Reviewed By: paulfitz Differential Revision: https://phab.getgrist.com/D2857	3 years ago
Paul Fitzpatrick	9f234b758d	(core) freshen grist-core build Summary: * adds a smoke test to grist-core * fixes a problem with highlight.js failing to load correctly * skips survey for default user * freshens docker build Utility files in test/nbrowser are moved to core/test/nbrowser, so that gristUtils are available there. This increased the apparent size of the diff as "./" import paths needed replacing with "test/nbrowser/" paths. The utility files are untouched, except for the code to start a server - it now has a small grist-core specific conditional in it. Test Plan: adds test Reviewers: dsagal Reviewed By: dsagal Differential Revision: https://phab.getgrist.com/D2768	4 years ago

15 Commits (1c6f80f956dcf5e1522caa1be518a151f850c041)