Archives/gristlabs_grist-core
1
0
Fork 0
mirror of https://github.com/gristlabs/grist-core.git synced 2024-10-27 20:44:07 +00:00
Code Issues Projects Releases Wiki Activity
e10067ff78
gristlabs_grist-core/app/common
History
Dmitry S d191859be7 (core) For exporting XLSX, do it memory-efficiently in a worker thread. 
Summary:
- Excel exports were awfully memory-inefficient, causing occasional docWorker
  crashes. The fix is to use the "streaming writer" option of ExcelJS
  https://github.com/exceljs/exceljs#streaming-xlsx-writercontents. (Empirically
  on one example, max memory went down from 3G to 100M)
- It's also CPU intensive and synchronous, and can block node for tens of
  seconds. The fix is to use a worker-thread. This diff uses "piscina" library
  for a pool of threads.
- Additionally, adds ProcessMonitor that logs memory and cpu usage,
  particularly when those change significantly.
- Also introduces request cancellation, so that a long download cancelled by
  the user will cancel the work being done in the worker thread.

Test Plan:
Updated previous export tests; memory and CPU performance tested
manually by watching output of ProcessMonitor.

Difference visible in these log excerpts:

Before (total time to serve request 22 sec):
```
Telemetry processMonitor heapUsedMB=2187, heapTotalMB=2234, cpuAverage=1.13, intervalMs=17911
Telemetry processMonitor heapUsedMB=2188, heapTotalMB=2234, cpuAverage=0.66, intervalMs=5005
Telemetry processMonitor heapUsedMB=2188, heapTotalMB=2234, cpuAverage=0, intervalMs=5005
Telemetry processMonitor heapUsedMB=71, heapTotalMB=75, cpuAverage=0.13, intervalMs=5002
```
After (total time to server request 18 sec):
```
Telemetry processMonitor heapUsedMB=109, heapTotalMB=144, cpuAverage=0.5, intervalMs=5001
Telemetry processMonitor heapUsedMB=109, heapTotalMB=144, cpuAverage=1.39, intervalMs=5002
Telemetry processMonitor heapUsedMB=94, heapTotalMB=131, cpuAverage=1.13, intervalMs=5000
Telemetry processMonitor heapUsedMB=94, heapTotalMB=131, cpuAverage=1.35, intervalMs=5001
```
Note in "Before" that heapTotalMB goes up to 2GB in the first case, and "intervalMs" of 17 seconds indicates that node was unresponsive for that long. In the second case, heapTotalMB stays low, and the main thread remains responsive the whole time.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3906
2023-06-01 12:06:48 -04:00
..
themes (core) Improve dark mode 2023-04-12 01:58:48 -04:00
ACLPermissions.ts (core) Implement checkbox for SchemaEdit permission in Access Rules UI. 2023-01-18 10:55:44 -05:00
ACLRuleCollection.ts (core) Implement checkbox for SchemaEdit permission in Access Rules UI. 2023-01-18 10:55:44 -05:00
ActionBundle.ts (core) deal with write access for attachments 2022-11-15 09:52:32 -05:00
ActionDispatcher.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
ActionGroup.ts (core) Add attachment and data size usage 2022-05-04 13:46:55 -07:00
ActionRouter.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
ActionSummarizer.ts (core) Adds a UI panel for managing webhooks 2023-05-08 18:25:27 -04:00
ActionSummary.ts Correct spelling mistakes 2022-02-19 09:46:49 +00:00
ActiveDocAPI.ts add support for conversational state to assistance endpoint (#506) 2023-05-08 14:15:22 -04:00
AlternateActions.ts (core) Adds a UI panel for managing webhooks 2023-05-08 18:25:27 -04:00
ApiError.ts (core) give more detailed reasons for access denied when memos are present 2021-02-15 17:02:24 -05:00
arrayToString.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
AssistancePrompts.ts add support for conversational state to assistance endpoint (#506) 2023-05-08 14:15:22 -04:00
AsyncCreate.ts (core) add a python3 button 2021-09-16 10:06:04 -04:00
AsyncFlow.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
asyncIterators.ts (core) Add AzureExternalStorage 2022-05-09 21:44:57 +02:00
AttachmentColumns.ts (core) control the distribution of attachment metadata 2022-12-22 09:10:30 -05:00
BaseAPI.ts (core) Adds a UI panel for managing webhooks 2023-05-08 18:25:27 -04:00
BasketClientAPI.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
BigInt.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
BillingAPI.ts (core) set up a trialing list and a call scheduled flag for Grist SaaS 2023-02-20 15:32:58 -05:00
BinaryIndexedTree.js Correct spelling mistakes 2022-02-19 09:46:49 +00:00
BrowserSettings.ts (core) Simple localization support and currency selector. 2021-08-26 13:36:49 -07:00
CircularArray.js (core) Moving client and common tests to core 2022-08-23 19:20:10 +02:00
ColumnFilterFunc.ts (core) Fix date filter for DateTime columns. 2023-01-18 10:18:15 +01:00
ColumnGetters.ts (core) Adding sort options for columns. 2021-11-03 15:31:39 +01:00
CommTypes.ts (core) Adds a UI panel for managing webhooks 2023-05-08 18:25:27 -04:00
csvFormat.ts (core) Add our very own implementation of tokenfield (aka pillbox, tag list) 2021-05-03 17:16:24 -04:00
CustomWidget.ts (core) Widget options api 2022-01-13 11:10:17 +01:00
declarations.d.ts (core) Guess date format during type conversion 2022-02-21 22:39:47 +02:00
delay.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
DisposableWithEvents.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
DocActions.ts (core) For exporting XLSX, do it memory-efficiently in a worker thread. 2023-06-01 12:06:48 -04:00
DocData.ts (core) Adds a UI panel for managing webhooks 2023-05-08 18:25:27 -04:00
DocDataCache.ts (core) Adds a UI panel for managing webhooks 2023-05-08 18:25:27 -04:00
DocLimits.ts (core) Keep track of row counts per table 2022-08-03 08:13:33 -07:00
DocListAPI.ts (core) Add API endpoint to get site usage summary 2022-05-16 11:16:19 -07:00
DocSnapshot.ts (core) Add AzureExternalStorage 2022-05-09 21:44:57 +02:00
DocumentSettings.ts (core) add a python3 button 2021-09-16 10:06:04 -04:00
DocUsage.ts (core) Keep track of row counts per table 2022-08-03 08:13:33 -07:00
emails.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
EncActionBundle.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
ErrorWithCode.ts (core) give more detailed reasons for access denied when memos are present 2021-02-15 17:02:24 -05:00
Features.ts (core) Add product for new personal plan 2022-07-26 11:33:23 -07:00
FilterState.ts (core) New date filter with a calendar view 2022-12-20 09:58:42 +01:00
Formula.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
getCurrentTime.ts (core) Adds dots menu to access rules page item 2023-01-12 09:40:45 +01:00
GranularAccessClause.ts (core) deal with write access for attachments 2022-11-15 09:52:32 -05:00
GristServerAPI.ts (core) Remove a bunch of dead code 2021-07-01 18:38:21 +02:00
gristTypes.ts (core) Comments 2022-10-17 13:38:21 +02:00
gristUrls.ts support other SQLite wrappers, and various hooks needed by grist-static (#516) 2023-05-23 15:17:28 -04:00
gutil.ts (core) Porting the AI evaluation script 2023-03-15 14:54:28 +01:00
hashingUtils.ts (core) Tweak telemetry 2023-05-19 13:06:15 -04:00
InactivityTimer.ts (core) Speed up and upgrade build. 2022-06-27 16:10:10 -04:00
Interval.ts (core) Fix bugs with intervals 2022-08-25 12:38:36 -07:00
isHiddenTable.ts (core) Fix exports to CSV/XLSX/etc when data is restricted by access rules 2022-09-02 10:59:59 -04:00
KeyedMutex.ts (core) fix sync to s3 when doc is marked as dirty but proves to be clean 2020-11-10 08:12:31 -05:00
KeyedOps.ts (core) limit retries of uploads to external store in tests 2022-06-06 16:19:41 -04:00
LocaleCodes.ts (core) Extending default locale list 2021-09-24 15:10:13 +02:00
Locales.ts Adding new flags for supported locales (#420) 2023-02-03 18:56:24 -05:00
LoginSessionAPI.ts (core) Improve dark mode 2023-04-12 01:58:48 -04:00
marshal.ts support other SQLite wrappers, and various hooks needed by grist-static (#516) 2023-05-23 15:17:28 -04:00
MemBuffer.js (core) move home server into core 2020-07-21 20:39:10 -04:00
NumberFormat.ts (core) Multi-column configuration 2022-10-17 09:51:19 +02:00
NumberParse.ts (core) Guess numeric formatting options 2022-03-03 21:32:03 +02:00
orgNameUtils.ts (core) show any error when creating initial team site with GRIST_SINGLE_ORG 2022-09-29 10:12:32 -04:00
parseDate.ts (core) Faster builds all around. 2022-07-04 10:42:40 -04:00
plugin.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
PluginInstance.ts Correct spelling mistakes 2022-02-19 09:46:49 +00:00
Prefs.ts (core) Focusing the creator panel on the table wiget for charts and custom views 2023-04-14 12:45:24 +02:00
RecentItems.js (core) Moving client and common tests to core 2022-08-23 19:20:10 +02:00
RefCountMap.ts (core) Fix an insidious bug in RefCountMap, manifesting as JS errors some time after import. 2020-11-06 09:24:56 -05:00
RelativeDates.ts (core) Set DateTime timezone during xlsx import 2023-05-24 11:39:49 -04:00
resetOrg.ts (core) Show usage banners in doc menu of free team sites 2022-05-26 15:01:35 -07:00
roles.ts (core) Add tip for "Add New" button 2023-01-16 16:50:42 -08:00
RowFilterFunc.ts (core) Download as CSV button on sections 2021-05-27 15:48:12 +02:00
schema.ts (core) reconcile webhook and widget description migrations 2023-05-15 11:56:15 -04:00
ShareAnnotator.ts (core) Avoid flagging support user as collaborator 2022-12-14 01:23:49 -05:00
sharing.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
SortFunc.ts (core) Fix the empty values last option in multicol sorting 2023-01-31 12:14:22 +01:00
SortSpec.ts (core) Adding sort options for columns. 2021-11-03 15:31:39 +01:00
StringUnion.ts (core) Allow configuring (mostly hiding) various little bits of UI 2022-05-27 14:32:05 +02:00
TableData.ts (core) Adds a UI panel for managing webhooks 2023-05-08 18:25:27 -04:00
TabularDiff.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
tagManager.ts (core) Record new user sign-ups 2022-03-12 14:34:46 -08:00
tbind.ts (core) Configure more comprehensive eslint rules for Typescript 2021-04-26 18:54:55 -04:00
Telemetry.ts (core) For exporting XLSX, do it memory-efficiently in a worker thread. 2023-06-01 12:06:48 -04:00
TestState.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
ThemePrefs-ti.ts (core) Improve dark mode 2023-04-12 01:58:48 -04:00
ThemePrefs.ts (core) Improve dark mode 2023-04-12 01:58:48 -04:00
Themes.ts (core) Add dark mode to user preferences 2022-09-05 19:17:32 -07:00
timeFormat.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
tpromisified.ts (core) Configure more comprehensive eslint rules for Typescript 2021-04-26 18:54:55 -04:00
Triggers-ti.ts (core) Adds a UI panel for managing webhooks 2023-05-08 18:25:27 -04:00
Triggers.ts (core) Adds a UI panel for managing webhooks 2023-05-08 18:25:27 -04:00
tsconfig.json (core) show package.json version when hovering on Grist icon in grist-core 2022-10-12 16:02:01 -04:00
tsvFormat.ts (core) move home server into core 2020-07-21 20:39:10 -04:00
UIRowId.ts (core) Support 'new' row in anchor links. 2021-09-24 09:01:10 -04:00
uploads.ts (core) Extending Google Drive integration scope 2021-10-01 10:47:12 +02:00
urlUtils.ts (core) remove metrics 2020-09-29 18:57:56 -04:00
UserAPI.ts (core) Adds a UI panel for managing webhooks 2023-05-08 18:25:27 -04:00
UserConfig.ts (core) Remove REPL code 2021-07-20 15:17:03 +02:00
ValueConverter.ts (core) Use visible column formatting when converting RefList to Text 2022-02-12 09:45:57 +02:00
ValueFormatter.ts (core) Fix imports into reference columns, and support two ways to import Numeric as a reference. 2023-05-02 10:28:14 -04:00
ValueGuesser.ts (core) Move guessing logic for column types to run in node once for all columns. 2022-05-19 12:49:51 -04:00
ValueParser.ts (core) Fix imports into reference columns, and support two ways to import Numeric as a reference. 2023-05-02 10:28:14 -04:00
WidgetOptions.ts Export table schema (#459) 2023-03-16 17:37:24 -04:00