Archives/gristlabs_grist-core
1
0
Fork 0
mirror of https://github.com/gristlabs/grist-core.git synced 2024-10-27 20:44:07 +00:00
Code Issues Projects Releases Wiki Activity
3dfe4be5f3
gristlabs_grist-core/app/server/lib
History
Alex Hall 3dfe4be5f3 (core) Remove DB transaction from webhook update, add mutex to all webhook endpoints 
Summary:
This removes problematic code that was holding a HomeDB transaction while applying user actions which could hang indefinitely, especially if the webhook queue is full as in https://grist.slack.com/archives/C05DBJ6LA1F/p1698159750945949.

The discussion about adding this code is here: https://phab.getgrist.com/D3821#inline-45054

The initial motivation was to roll back HomeDB changes if something went wrong while applying user actions, to avoid saving only part of the changes the user requested. I think it's actually fine to just allow such a partial save to happen - I don't see anything particularly undesirable about keeping an update to the webhook URL if other updates requested by the user didn't also get applied, as the fields don't affect each other.

The comment approving the transaction approach said "so we shouldn't end up leave the transaction hanging around too long" which has been falsified.

It looks like there was also some desire to prevent a mess caused by multiple simultaneous calls to this endpoint, which the transaction may have helped with a little, but didn't really seem like a solution. Comments in `Triggers.ts` also mention fears of race conditions when clearing (some of) the queue and the need for some locking. So I wrapped all webhook-related endpoints in a simple `Mutex` held by the `ActiveDoc` to prevent simultaneous changes. I *think* this is a good thing. These endpoints shouldn't be called frequently enough to create a performance issue, and this shouldn't affect actually sending webhook events when records are added/updated. And it does seem like interleaving calls to these endpoints could cause very weird problems.

Test Plan: Nothing yet, I'd like to hear if others think this is sensible.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D4111
2023-11-14 16:51:21 +02:00
..
AccessTokens.ts (core) add an access token mechanism to help with attachments in custom widgets 2022-07-19 11:55:18 -04:00
ACLFormula.ts Dump the rule for ACL formula warnings (#639) 2023-08-23 09:23:29 -04:00
ActionHistory.ts (core) Fix issue with lodash's map interpreting objects with length as array-like 2023-07-01 21:07:39 -04:00
ActionHistoryImpl.ts support other SQLite wrappers, and various hooks needed by grist-static (#516) 2023-05-23 15:17:28 -04:00
ActiveDoc.ts (core) Remove DB transaction from webhook update, add mutex to all webhook endpoints 2023-11-14 16:51:21 +02:00
ActiveDocImport.ts Split out new importFileAsNewTable method for grist-static (#564) 2023-07-12 15:57:02 +02:00
AppEndpoint.ts (core) Add new telemetry events 2023-11-01 10:49:33 -04:00
AppSettings.ts add an endpoint for doing SQL selects (#641) 2023-09-04 09:21:18 -04:00
Assistance.ts (core) updates from grist-core 2023-08-21 09:15:53 -04:00
Authorizer.ts (core) Revamp ForwardAuthLogin and unify with GRIST_PROXY_AUTH_HEADER 2023-11-07 16:30:49 -05:00
BrowserSession.ts Add native OIDC support in Grist #707 (#717) 2023-11-09 14:56:39 -05:00
checksumFile.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
Client.ts (core) Manage memory used for websocket responses to reduce the risk of server crashes. 2023-08-07 11:28:31 -04:00
Comm.ts (core) Revamp ForwardAuthLogin and unify with GRIST_PROXY_AUTH_HEADER 2023-11-07 16:30:49 -05:00
configureMinIOExternalStorage.ts Abort when MinIO bucket does not have versioning enabled #545 (#546) 2023-07-10 06:24:55 -04:00
dbUtils.ts (core) remove ormconfig.js from saas build; more sendgrid logging 2023-03-01 17:02:47 -05:00
DiscourseConnect.ts (core) Adding GristConnect login system 2022-05-18 20:28:25 +02:00
DocApi.ts (core) Remove DB transaction from webhook update, add mutex to all webhook endpoints 2023-11-14 16:51:21 +02:00
DocClients.ts (core) add a yarn run cli tool, and add a sqlite gristify option 2022-07-14 12:00:30 -04:00
DocManager.ts (core) Add new telemetry events 2023-11-01 10:49:33 -04:00
DocPluginData.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
DocPluginManager.ts (core) add an access token mechanism to help with attachments in custom widgets 2022-07-19 11:55:18 -04:00
DocSession.ts (core) User language switcher 2023-01-26 09:47:14 +01:00
DocSnapshots.ts Track saved version per hour, day, week, month, year, and number of times a version with parameter (#509) 2023-05-12 11:38:29 -04:00
DocStorage.ts add an endpoint for doing SQL selects (#641) 2023-09-04 09:21:18 -04:00
DocStorageManager.ts (core) dust off electron build a little bit 2022-07-29 11:19:26 -04:00
docUtils.d.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
docUtils.js Correct spelling mistakes 2022-02-19 09:46:49 +00:00
DocWorker.ts Remove attachments and system files from document download as template (#729) 2023-11-08 15:09:01 -05:00
DocWorkerMap.ts (core) Add unquarantine command to admin CLI 2022-08-15 13:04:55 -07:00
DocWorkerUtils.ts (core) Add support for auto-copying docs on signup 2023-09-06 15:12:08 -04:00
ExcelFormatter.ts (core) Currency from grist column is persistent when exporting to excel 2023-05-08 10:39:01 +02:00
ExpandedQuery.ts Correct spelling mistakes 2022-02-19 09:46:49 +00:00
Export.ts Option to export colId as header in CSV / XSLX instead of label (#688) (#692) 2023-10-15 20:17:43 -04:00
ExportCSV.ts Option to export colId as header in CSV / XSLX instead of label (#688) (#692) 2023-10-15 20:17:43 -04:00
ExportTableSchema.ts (core) Fix imports into reference columns, and support two ways to import Numeric as a reference. 2023-05-02 10:28:14 -04:00
ExportXLSX.ts add hooks for tweaking how downloads happen (for grist-static) (#665) 2023-09-09 14:50:32 -04:00
expressWrap.ts (core) Update dependencies 2023-10-11 17:36:58 -04:00
ExternalStorage.ts (core) Faster builds all around. 2022-07-04 10:42:40 -04:00
extractOrg.ts (core) support for bundling custom widgets with the Grist app 2023-10-27 17:00:10 -04:00
FileParserElement.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
filterUtils.ts (core) uncheck FullCopy special when copying/forking a document 2021-04-29 08:56:54 -04:00
FlexServer.ts (core) Revamp ForwardAuthLogin and unify with GRIST_PROXY_AUTH_HEADER 2023-11-07 16:30:49 -05:00
ForwardAuthLogin.ts (core) Revamp ForwardAuthLogin and unify with GRIST_PROXY_AUTH_HEADER 2023-11-07 16:30:49 -05:00
GoogleAuth.ts Add ws id and doc name params to POST /docs (#655) 2023-09-05 14:27:35 -04:00
GoogleExport.ts Add ws id and doc name params to POST /docs (#655) 2023-09-05 14:27:35 -04:00
GoogleImport.ts (core) Adding DELETE /api/docs/webhooks/queue endpoint to clear the queue 2022-12-01 12:23:19 +01:00
GranularAccess.ts Dump the rule for ACL formula warnings (#639) 2023-08-23 09:23:29 -04:00
GristServer.ts (core) Revamp ForwardAuthLogin and unify with GRIST_PROXY_AUTH_HEADER 2023-11-07 16:30:49 -05:00
gristSessions.ts (core) Add options to /status health-check endpoints to check DB and Redis liveness. 2023-10-02 14:41:04 -04:00
gristSettings.ts move getTemplateOrg method; enable template org in docker tests (#602) 2023-07-31 16:10:59 -04:00
guessExt.ts (core) Switch excel import parsing from messytables+xlrd to openpyxl, and ignore empty rows 2022-05-12 14:43:21 +02:00
hashingUtils.ts (core) Add Support Grist page and nudge 2023-07-04 17:36:59 -04:00
HashUtil.ts (core) add more detail to /compare endpoint 2020-09-18 16:31:29 -04:00
HostedMetadataManager.ts (core) Faster builds all around. 2022-07-04 10:42:40 -04:00
HostedStorageManager.ts (core) Persist forks in home db 2023-02-20 22:46:36 -05:00
httpEncoding.ts (core) External requests 2022-06-17 21:53:20 +02:00
IBilling.ts (core) Add basic activation page to grist-ee 2022-08-23 10:30:52 -07:00
IChecksumStore.ts (core) revamp snapshot inventory 2020-10-30 13:52:46 -04:00
ICreate.ts (core) Delete my account button 2023-09-27 14:49:23 +02:00
IDocStorageManager.ts (core) add a yarn run cli tool, and add a sqlite gristify option 2022-07-14 12:00:30 -04:00
idUtils.ts (core) Persist forks in home db 2023-02-20 22:46:36 -05:00
IElectionStore.ts (core) move some material to core that slipped through in a rebase 2020-07-23 11:29:05 -04:00
initialDocSql.ts (core) reconcile webhook and widget description migrations 2023-05-15 11:56:15 -04:00
INotifier.ts (core) Speed up and upgrade build. 2022-06-27 16:10:10 -04:00
ISandbox.ts (core) Faster builds all around. 2022-07-04 10:42:40 -04:00
IShell.ts (core) dust off electron build a little bit 2022-07-29 11:19:26 -04:00
ITestingHooks-ti.ts (core) Manage memory used for websocket responses to reduce the risk of server crashes. 2023-08-07 11:28:31 -04:00
ITestingHooks.ts (core) Manage memory used for websocket responses to reduce the risk of server crashes. 2023-08-07 11:28:31 -04:00
log.ts fix log.add under electron (#478) 2023-04-03 13:29:35 -04:00
LogMethods.ts (core) Faster builds all around. 2022-07-04 10:42:40 -04:00
manifest.ts (core) support for bundling custom widgets with the Grist app 2023-10-27 17:00:10 -04:00
MemoryPool.ts (core) Manage memory used for websocket responses to reduce the risk of server crashes. 2023-08-07 11:28:31 -04:00
MinimalLogin.ts A set of tweaks to simplify electron packaging (#421) 2023-02-13 15:52:17 -05:00
MinIOExternalStorage.ts Issue 359 support scaleway (#577) 2023-07-20 06:16:58 -04:00
NSandbox.ts (core) Update dependencies 2023-10-11 17:36:58 -04:00
NullSandbox.ts (core) Fix snapshot migrations 2023-05-23 15:50:15 -04:00
OIDCConfig.ts Add native OIDC support in Grist #707 (#717) 2023-11-09 14:56:39 -05:00
OnDemandActions.ts (core) Adds a UI panel for managing webhooks 2023-05-08 18:25:27 -04:00
PermissionInfo.ts (core) Fixing memos in record dependent rules. 2023-09-08 10:16:09 +02:00
Permit.ts (core) Add new Grist login page 2022-04-01 15:24:19 -07:00
places.ts A set of tweaks to simplify electron packaging (#421) 2023-02-13 15:52:17 -05:00
PluginEndpoint.ts (core) support for bundling custom widgets with the Grist app 2023-10-27 17:00:10 -04:00
PluginManager.ts (core) support for bundling custom widgets with the Grist app 2023-10-27 17:00:10 -04:00
ProcessMonitor.ts (core) Add new telemetry events 2023-11-01 10:49:33 -04:00
ProxyAgent.ts (core) Add logging of errors whenever ProxyAgent is used, and a test for it. 2023-05-17 10:21:53 -04:00
reportTimeTaken.ts (core) Faster builds all around. 2022-07-04 10:42:40 -04:00
Requests.ts REQUEST now supports POST (#588) 2023-07-30 15:13:43 -04:00
requestUtils.ts (core) Add welcomeQuestionsSubmitted telemetry event 2023-09-13 00:31:04 -04:00
RowAccess.ts (core) control the distribution of attachment metadata 2022-12-22 09:10:30 -05:00
SafePythonComponent.ts (core) Run and test imports only in Python 3, upgrade openpyxl, fix weird date handling 2022-09-02 16:27:34 +02:00
SamlConfig.ts (core) Fix delete user button for Google-only accounts 2023-10-18 10:52:58 -04:00
SandboxControl.ts (core) Update dependencies 2023-10-11 17:36:58 -04:00
sandboxUtil.ts (core) Update logging in sandbox code, and log tracebacks as single log messages. 2023-07-18 11:21:25 -04:00
sendAppPage.ts (core) Removing the new menu flag 2023-11-08 07:37:54 +01:00
ServerColumnGetters.ts (core) Filter rows based on linked widgets when exporting view 2023-07-26 21:49:52 +02:00
ServerLocale.ts (core) Converting server-side Comm.js to typescript 2022-06-07 15:47:17 -04:00
serverUtils.ts (core) Add timeouts to prevent ActiveDoc bad state during shutdown. 2023-09-26 15:32:49 -04:00
Sessions.ts (core) Converting server-side Comm.js to typescript 2022-06-07 15:47:17 -04:00
Sharing.ts (core) port some more test/browser tests to newer selenium 2023-04-12 13:00:53 -04:00
shortDesc.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
shutdown.js support other SQLite wrappers, and various hooks needed by grist-static (#516) 2023-05-23 15:17:28 -04:00
SqliteCommon.ts add an endpoint for doing SQL selects (#641) 2023-09-04 09:21:18 -04:00
SQLiteDB.ts add an endpoint for doing SQL selects (#641) 2023-09-04 09:21:18 -04:00
SqliteNode.ts add an endpoint for doing SQL selects (#641) 2023-09-04 09:21:18 -04:00
TableMetadataLoader.ts (core) open documents without blocking on data engine 2021-10-01 10:18:56 -04:00
TagChecker.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
Telemetry.ts (core) Add new telemetry events 2023-11-01 10:49:33 -04:00
TestingHooks.ts (core) Manage memory used for websocket responses to reduce the risk of server crashes. 2023-08-07 11:28:31 -04:00
TestLogin.ts (core) make the support account configurable, and allow listing public sites 2022-10-14 14:23:59 -04:00
Throttle.ts (core) Fix bugs with intervals 2022-08-25 12:38:36 -07:00
TimeQuery.ts (core) Adds a UI panel for managing webhooks 2023-05-08 18:25:27 -04:00
Triggers.ts (core) When a webhook is disabled, clear its queue 2023-11-07 15:48:35 +02:00
UnsafeNodeComponent.ts (core) Faster builds all around. 2022-07-04 10:42:40 -04:00
uploads.ts (core) Add support for auto-copying docs on signup 2023-09-06 15:12:08 -04:00
WidgetRepository.ts allow bundled widgets to be hidden from dropdown, and nested (#714) 2023-10-30 21:13:21 -04:00
WorkCoordinator.ts support other SQLite wrappers, and various hooks needed by grist-static (#516) 2023-05-23 15:17:28 -04:00
workerExporter.ts Option to export colId as header in CSV / XSLX instead of label (#688) (#692) 2023-10-15 20:17:43 -04:00