gristlabs_grist-core

Commit Graph

Author	SHA1	Message	Date
Alex Hall	faec8177ab	(core) Use MetaTableData more Summary: Add more method overrides to MetaTableData for extra type safety. Use MetaTableData, MetaRowRecord, and getMetaTable in more places. Test Plan: Mostly it just has to compile. Tested manually that types are being checked more strictly now, e.g. by adding a typo to property names. Some type casting has also been removed. Reviewers: dsagal Reviewed By: dsagal Subscribers: dsagal Differential Revision: https://phab.getgrist.com/D3168	3 years ago
Alex Hall	561e32fb44	(core) Better logging in DocTriggers Summary: Added a helper to include lots of metadata in every logging call, added and converted many logging calls. Test Plan: Existing tests pass Reviewers: paulfitz Reviewed By: paulfitz Differential Revision: https://phab.getgrist.com/D3136	3 years ago
Alex Hall	e09e919016	(core) Ensure that large changes are processed in full by triggers (for webhooks) Summary: Uses the new alwaysPreserveColIds option for action summaries in Triggers.ts. Triggers.ts is now responsible for generating the summary to make it easy to pass this option. The value of the option is just all colIds mentioned in triggers configured in this document. Test Plan: Tested adding 200 rows to a subscribed table to ensure the events are not truncated. Also tests batching nicely. Reviewers: paulfitz Reviewed By: paulfitz Differential Revision: https://phab.getgrist.com/D3135	3 years ago
Alex Hall	4a70283292	(core) Webhook event queue on redis Summary: Push webhook events to redis queue with key based on docId. Remove events from redis after sending using LTRIM. Put failed events back on the end of the queue under normal circumstances. When the event queue gets too long: - Wait until it gets consumed before continuing. - Drop failed events (i.e. don't put them back on the end of the queue) - Limit webhook retries to 5 Test Plan: Tested that interactions with redis are as expected using redis MONITOR command. Reviewers: paulfitz Reviewed By: paulfitz Differential Revision: https://phab.getgrist.com/D3100	3 years ago
Alex Hall	276adc5f51	(core) Starting to make webhooks more robust Summary: - Puts events on a queue in memory and ensures they are sent in the order they were generated. - Makes the caller (Sharing.ts) wait until changed records have been fetched from the DB, but allows it to continue after while remaining work happens asynchronously. - Gathers all new webhook events into an array so they can be backed up to the queue on redis in a single command (in a future diff). - Uses changes in isReady to determine event type, no more 'existed before' The structure of the code has changed a lot, so I think the scope of the diff needs to stop here. Lots of work is still deferred in TODOs. Test Plan: Updated existing test. Actually dropped testing of retry on failures and slowness because it no longer made sense to keep that as part of the current test, so a new test for that will be added in a future diff. Reviewers: paulfitz Reviewed By: paulfitz Differential Revision: https://phab.getgrist.com/D3074	3 years ago
Alex Hall	8c1f8bc9a6	(core) Test webhooks Summary: Tests DocApi endpoints _subscribe and _unsubscribe, including various bad inputs. Tests that webhooks are sent to a test express server, with retrying on failure, filtered by event type, and waiting for isReadyColumn. Test Plan: this Reviewers: paulfitz Reviewed By: paulfitz Differential Revision: https://phab.getgrist.com/D3042	3 years ago
Dmitry S	fb583f303a	(core) Support 'new' row in anchor links. Summary: - Anchor links with row of 'new' could be created but weren't parsed or used correctly. This fixes it. - Also adds UIRowId type for row IDs which includes the special 'new' row. It's already been used in places as `number\|'new'`, this diff gives it a name usable in app/common (it doesn't touch another name, RowId, that's been available in app/client). Test Plan: Added a test assert for anchor links to new row Reviewers: alexmojaki Reviewed By: alexmojaki Differential Revision: https://phab.getgrist.com/D3039	3 years ago
Alex Hall	3c4d71aeca	(core) Initial webhooks implementation Summary: See https://grist.quip.com/VKd3ASF99ezD/Outgoing-Webhooks - 2 new DocApi endpoints: _subscribe and _unsubscribe, not meant to be user friendly or publicly documented. _unsubscribe should be given the response from _subscribe in the body, e.g: ``` $ curl -X POST -H "Authorization: Bearer 8fd4dc59ecb05ab29ae5a183c03101319b8e6ca9" "http://localhost:8080/api/docs/6WYa23FqWxGNe3AR6DLjCJ/tables/Table2/_subscribe" -H "Content-type: application/json" -d '{"url": "https://webhook.site/a916b526-8afc-46e6-aa8f-a625d0d83ec3", "eventTypes": ["add"], "isReadyColumn": "C"}' {"unsubscribeKey":"3246f158-55b5-4fc7-baa5-093b75ffa86c","triggerId":2,"webhookId":"853b4bfa-9d39-4639-aa33-7d45354903c0"} $ curl -X POST -H "Authorization: Bearer 8fd4dc59ecb05ab29ae5a183c03101319b8e6ca9" "http://localhost:8080/api/docs/6WYa23FqWxGNe3AR6DLjCJ/tables/Table2/_unsubscribe" -H "Content-type: application/json" -d '{"unsubscribeKey":"3246f158-55b5-4fc7-baa5-093b75ffa86c","triggerId":2,"webhookId":"853b4bfa-9d39-4639-aa33-7d45354903c0"}' {"success":true} ``` - New DB entity Secret to hold the webhook URL and unsubscribe key - New document metatable _grist_Triggers subscribes to table changes and points to a secret to use for a webhook - New file Triggers.ts processes action summaries and uses the two new tables to send webhooks. - Also went on a bit of a diversion and made a typesafe subclass of TableData for metatables. I think this is essentially good enough for a first diff, to keep the diffs manageable and to talk about the overall structure. Future diffs can add tests and more robustness using redis etc. After this diff I can also start building the Zapier integration privately. Test Plan: Tested manually: see curl commands in summary for an example. Payloads can be seen in https://webhook.site/#!/a916b526-8afc-46e6-aa8f-a625d0d83ec3/0b9fe335-33f7-49fe-b90b-2db5ba53382d/1 . Great site for testing webhooks btw. Reviewers: dsagal, paulfitz Reviewed By: paulfitz Differential Revision: https://phab.getgrist.com/D3019	3 years ago

8 Commits (ae6c857ac584d5fc4de58007df8091a13b098102)