4 Commits (a3467c35c66143abb86c4abb0d54ffabbf9a7112)

Author	SHA1	Message	Date
Dmitry S	d191859be7	(core) For exporting XLSX, do it memory-efficiently in a worker thread. ... Summary: - Excel exports were awfully memory-inefficient, causing occasional docWorker crashes. The fix is to use the "streaming writer" option of ExcelJS https://github.com/exceljs/exceljs#streaming-xlsx-writercontents. (Empirically on one example, max memory went down from 3G to 100M) - It's also CPU intensive and synchronous, and can block node for tens of seconds. The fix is to use a worker-thread. This diff uses "piscina" library for a pool of threads. - Additionally, adds ProcessMonitor that logs memory and cpu usage, particularly when those change significantly. - Also introduces request cancellation, so that a long download cancelled by the user will cancel the work being done in the worker thread. Test Plan: Updated previous export tests; memory and CPU performance tested manually by watching output of ProcessMonitor. Difference visible in these log excerpts: Before (total time to serve request 22 sec): ``` Telemetry processMonitor heapUsedMB=2187, heapTotalMB=2234, cpuAverage=1.13, intervalMs=17911 Telemetry processMonitor heapUsedMB=2188, heapTotalMB=2234, cpuAverage=0.66, intervalMs=5005 Telemetry processMonitor heapUsedMB=2188, heapTotalMB=2234, cpuAverage=0, intervalMs=5005 Telemetry processMonitor heapUsedMB=71, heapTotalMB=75, cpuAverage=0.13, intervalMs=5002 ``` After (total time to server request 18 sec): ``` Telemetry processMonitor heapUsedMB=109, heapTotalMB=144, cpuAverage=0.5, intervalMs=5001 Telemetry processMonitor heapUsedMB=109, heapTotalMB=144, cpuAverage=1.39, intervalMs=5002 Telemetry processMonitor heapUsedMB=94, heapTotalMB=131, cpuAverage=1.13, intervalMs=5000 Telemetry processMonitor heapUsedMB=94, heapTotalMB=131, cpuAverage=1.35, intervalMs=5001 ``` Note in "Before" that heapTotalMB goes up to 2GB in the first case, and "intervalMs" of 17 seconds indicates that node was unresponsive for that long. In the second case, heapTotalMB stays low, and the main thread remains responsive the whole time. Reviewers: jarek Reviewed By: jarek Differential Revision: https://phab.getgrist.com/D3906	1 year ago
Jakub Serafin	37347a79c0	(core) Currency from grist column is persistent when exporting to excel ... Summary: - when grist table is exported, currency is check and introduced in cell format in the form of "[currency symbol] [value]" (for example: zł 10000, $ 5000) . It's not what some cultures should display currences, but it's close enought - when no symbol is defined for the currency, currency 3 letters code is used instead - when currency is unknown, we are falling back to "$" Test Plan: - nbrowser test scenario added for that purpose, please check Currences.xlsx to see output format exported. Reviewers: georgegevoian Reviewed By: georgegevoian Differential Revision: https://phab.getgrist.com/D3886	1 year ago
Dmitry S	65013331a3	(core) Fix imports into reference columns, and support two ways to import Numeric as a reference. ... Summary: - When importing into a Ref column, use lookupOne() formula for correct previews. - When selecting columns to import into a Ref column, now a Numeric column like 'Order' will produce two options: "Order" and "Order (as row ID)". - Fixes exports to correct the formatting of visible columns. This addresses multiple bugs: 1. Formatting wasn't used, e.g. a Ref showing a custom-formatted date was still presented as YYYY-MM-DD in CSVs. 2. Ref showing a Numeric column was formatted as if a row ID (e.g. `Table1[1.5]`), which is very wrong. - If importing into a table that doesn't have a primary view, don't switch page after import. Refactorings: - Generalize GenImporterView to be usable in more cases; removed near-duplicated logic from node side - Some other refactoring in importing code. - Fix field/column option selection in ValueParser - Add NUM() helper to turn integer-valued floats into ints, useful for "as row ID" lookups. Test Plan: Added test cases for imports into reference columns, updated Exports test fixtures. Reviewers: georgegevoian Reviewed By: georgegevoian Differential Revision: https://phab.getgrist.com/D3875	1 year ago
Jarosław Sadziński	64710b60f3	(core) Moving nbrowser tests to grist-core ... Summary: Moving bulk of nbrowser tests to core. Some tests were split and only part of them were moved. Tests that are left are either: not suitable for grist-core (like billing) or are failing during browser tests (are not reliable). Four fixtures directory (uploads, docs, exports-csv/excel) where completely moved to grist-core and are linked as folders. Those changes allows to add an nbrowser test in grist-core or in the main test folder without any need to link it or link a fixture document. Other changes: - testrun.sh has been modified, now it runs tests from both folders (test and core/test), - TestServer used in grist-core is now adding sample orgs and users (kiwi and others), Test modified - SelectionSummary: now it is run on a bigScreen, it was failing randomly - Billing.ts: relative paths were used - DateEditor: added waitForServer - it was failing in browser mode - FrozenColumns, ImportFromGDrive, Printing: updated import paths - UserManager.ts: was split into two parts (it assumed limited products) - ViewLayoutResize.ts: this test is still in main repo, it is still failing in browser mode tests Test Plan: Existing Reviewers: paulfitz Reviewed By: paulfitz Subscribers: dsagal, paulfitz Differential Revision: https://phab.getgrist.com/D3664	2 years ago