gristlabs_grist-core/app/server/lib
Paul Fitzpatrick e30d0fd5d0 (core) fix sync to s3 when doc is marked as dirty but proves to be clean
Summary:
This fixes a two problems:
 * A mistake in `KeyedMutex.runExclusive`.
 * Logic about saving a document to s3 when the document is found to match what is already there.

`HostedStorageManager.flushDoc` could get caught in a loop if a document was uploaded to s3 and then, without any change to it, marked as dirty.  Low level code would detect there was no change and skip the upload; but then the snapshotId could be unknown, causing an error and retries. This diff fixes that problem by discovering the snapshotId on downloads and tracking it. It also corrects a mutex problem that may have been creating the scenario. A small delay is added to `flushDoc` to mitigate the effect of similar problems in future. Exponential backoff would be good, but `flushDoc` is called in some situations where long delays would negatively impact worker shutdown or user work.

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2654
2020-11-10 08:12:31 -05:00
..
ActionHistory.ts (core) Store formula values in DB, and include them into .stored/.undo fields of actions. 2020-11-04 16:45:47 -05:00
ActionHistoryImpl.ts (core) When parsing ActionHistory into ActionGroups, avoid keeping many large actions in memory. 2020-09-19 20:45:33 -04:00
ActionSummary.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
ActiveDoc.ts (core) Store formula values in DB, and include them into .stored/.undo fields of actions. 2020-11-04 16:45:47 -05:00
ActiveDocImport.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
AppEndpoint.ts (core) back-end support for tables that are accessible only by owners 2020-09-14 18:05:27 -04:00
Authorizer.ts (core) support adding user characteristic tables for granular ACLs 2020-10-19 13:33:47 -04:00
BrowserSession.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
checksumFile.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
Client.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
Comm.js (core) move home server into core 2020-07-21 20:39:10 -04:00
dbUtils.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
DocApi.ts (core) support GRIST_WORKER_GROUP to place worker into an exclusive group 2020-11-02 15:46:46 -05:00
DocClients.ts (core) add some row-level access control 2020-10-12 11:17:37 -04:00
DocManager.ts (core) revamp snapshot inventory 2020-10-30 13:52:46 -04:00
DocPluginData.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
DocPluginManager.ts (core) make user role available in ActiveDoc methods 2020-09-02 14:46:15 -04:00
DocSession.ts (core) support adding user characteristic tables for granular ACLs 2020-10-19 13:33:47 -04:00
DocSnapshots.ts (core) fix sync to s3 when doc is marked as dirty but proves to be clean 2020-11-10 08:12:31 -05:00
DocStorage.ts (core) Store formula values in DB, and include them into .stored/.undo fields of actions. 2020-11-04 16:45:47 -05:00
DocStorageManager.ts (core) revamp snapshot inventory 2020-10-30 13:52:46 -04:00
docUtils.d.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
docUtils.js (core) move home server into core 2020-07-21 20:39:10 -04:00
DocWorker.ts (core) make user role available in ActiveDoc methods 2020-09-02 14:46:15 -04:00
DocWorkerMap.ts (core) support GRIST_WORKER_GROUP to place worker into an exclusive group 2020-11-02 15:46:46 -05:00
ExpandedQuery.ts (core) Store formula values in DB, and include them into .stored/.undo fields of actions. 2020-11-04 16:45:47 -05:00
expressWrap.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
ExternalStorage.ts (core) fix sync to s3 when doc is marked as dirty but proves to be clean 2020-11-10 08:12:31 -05:00
extractOrg.ts (core) add a deployment test for Import-from-URL, and fix underlying issue 2020-07-23 11:26:16 -04:00
FileParserElement.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
FlexServer.ts (core) support GRIST_WORKER_GROUP to place worker into an exclusive group 2020-11-02 15:46:46 -05:00
GranularAccess.ts (core) support access control on columns 2020-11-03 19:08:44 -05:00
GristServer.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
gristSessions.ts (core) support GRIST_WORKER_GROUP to place worker into an exclusive group 2020-11-02 15:46:46 -05:00
guessExt.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
HashUtil.ts (core) add more detail to /compare endpoint 2020-09-18 16:31:29 -04:00
HostedMetadataManager.ts (core) revamp snapshot inventory 2020-10-30 13:52:46 -04:00
HostedStorageManager.ts (core) fix sync to s3 when doc is marked as dirty but proves to be clean 2020-11-10 08:12:31 -05:00
IBilling.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
IChecksumStore.ts (core) revamp snapshot inventory 2020-10-30 13:52:46 -04:00
ICreate.ts (core) revamp snapshot inventory 2020-10-30 13:52:46 -04:00
IDocStorageManager.ts (core) revamp snapshot inventory 2020-10-30 13:52:46 -04:00
idUtils.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
IElectionStore.ts (core) move some material to core that slipped through in a rebase 2020-07-23 11:29:05 -04:00
IInstanceManager.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
ILoginSession.ts (core) move client code to core 2020-10-02 13:24:21 -04:00
INotifier.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
ISandbox.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
IShell.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
ITestingHooks-ti.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
ITestingHooks.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
log.ts (core) give instructions on using Grist with docker 2020-10-28 13:59:13 -04:00
manifest.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
NSandbox.ts (core) move data engine code to core 2020-07-29 08:57:25 -04:00
OnDemandActions.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
Permit.ts (core) move some material to core that slipped through in a rebase 2020-07-23 11:29:05 -04:00
places.ts (core) fix docker packaging after core shuffle 2020-07-22 14:45:42 -04:00
PluginEndpoint.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
PluginManager.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
requestUtils.ts (core) switch to newer download endpoint in client 2020-10-19 12:44:03 -04:00
SafePythonComponent.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
sandboxUtil.js (core) move home server into core 2020-07-21 20:39:10 -04:00
sendAppPage.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
ServerColumnGetters.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
serverUtils.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
Sessions.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
Sharing.ts (core) Store formula values in DB, and include them into .stored/.undo fields of actions. 2020-11-04 16:45:47 -05:00
shortDesc.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
shutdown.js (core) move home server into core 2020-07-21 20:39:10 -04:00
SQLiteDB.ts (core) revamp snapshot inventory 2020-10-30 13:52:46 -04:00
TagChecker.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
TestingHooks.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
Throttle.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
TimeQuery.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
UnsafeNodeComponent.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
uploads.ts (core) mitigate csrf by requiring custom header for unsafe methods 2020-10-08 14:19:25 -04:00
WorkCoordinator.ts (core) move home server into core 2020-07-21 20:39:10 -04:00