spoffy/fix-import-error-s3-no-redis
dependabot/npm_and_yarn/express-4.20.0
latest_candidate
main
spoffy/webdriver-logs
dependabot/npm_and_yarn/webpack-5.94.0
dependabot/npm_and_yarn/dompurify-3.1.3
latest
dependabot/npm_and_yarn/elliptic-6.5.7
dependabot/npm_and_yarn/axios-1.7.4
dependabot/npm_and_yarn/micromatch-4.0.8
berhalak/build-test
ignore-alert
link-to-issue-templates
spoffy/rename-candidate-action-job
dependabot/npm_and_yarn/fast-xml-parser-4.4.1
spoffy/playwright
spoffy/grist-ee-defaults
dependabot/npm_and_yarn/ws-8.17.1
dependabot/npm_and_yarn/tar-6.2.1
dependabot/npm_and_yarn/braces-3.0.3
jordigh/native-arm64
paulfitz/preview
paulfitz/smoosh
test-server-reset
dsagal-readme-gvisor
readme-update-dec2023
paulfitz/bundle-widget-prep
jv-linkstate-bubbles-tooltips
jv-linkstate-bubbles-base
jv-bidirectional-tests
preview
bidirectional
chainlink-fix
alex/skip-fstrings-3.9
alex/upgrade-pyodide
alex/3.11-tests
alex/_importParsedFileAsNewTable
poc-engine-data-layer
poc-engine
sponsors-section
removing-missing-key-error
friendly-locale
messytables-requirements
add-page-name
markdown-cells
v1.1.12
v1.1.11
v1.1.10
v1.1.9
v1.1.8
v1.1.7
v1.1.6
v1.1.5
v1.1.4
v1.1.3
v1.1.2
v1.1.1
v1.1.0
v1.0.9
v1.0.8
v1.0.7
v1.0.6
v1.0.5
v1.0.4
v1.0.3
v0.7.9
v0.7.8
v0.7.7
v0.7.6
v0.7.5
v0.7.4
v0.7.1
v0.7.2
v0.7.3
v1.1.13
v1.1.14
v1.1.15
v1.1.16
v1.1.17
v1.1.18
${ noResults }
5 Commits (5705c37d02e39e2a4c9e34a8b15647e3605a4adf)
Author | SHA1 | Message | Date |
---|---|---|---|
Dmitry S | d191859be7 |
(core) For exporting XLSX, do it memory-efficiently in a worker thread.
Summary: - Excel exports were awfully memory-inefficient, causing occasional docWorker crashes. The fix is to use the "streaming writer" option of ExcelJS https://github.com/exceljs/exceljs#streaming-xlsx-writercontents. (Empirically on one example, max memory went down from 3G to 100M) - It's also CPU intensive and synchronous, and can block node for tens of seconds. The fix is to use a worker-thread. This diff uses "piscina" library for a pool of threads. - Additionally, adds ProcessMonitor that logs memory and cpu usage, particularly when those change significantly. - Also introduces request cancellation, so that a long download cancelled by the user will cancel the work being done in the worker thread. Test Plan: Updated previous export tests; memory and CPU performance tested manually by watching output of ProcessMonitor. Difference visible in these log excerpts: Before (total time to serve request 22 sec): ``` Telemetry processMonitor heapUsedMB=2187, heapTotalMB=2234, cpuAverage=1.13, intervalMs=17911 Telemetry processMonitor heapUsedMB=2188, heapTotalMB=2234, cpuAverage=0.66, intervalMs=5005 Telemetry processMonitor heapUsedMB=2188, heapTotalMB=2234, cpuAverage=0, intervalMs=5005 Telemetry processMonitor heapUsedMB=71, heapTotalMB=75, cpuAverage=0.13, intervalMs=5002 ``` After (total time to server request 18 sec): ``` Telemetry processMonitor heapUsedMB=109, heapTotalMB=144, cpuAverage=0.5, intervalMs=5001 Telemetry processMonitor heapUsedMB=109, heapTotalMB=144, cpuAverage=1.39, intervalMs=5002 Telemetry processMonitor heapUsedMB=94, heapTotalMB=131, cpuAverage=1.13, intervalMs=5000 Telemetry processMonitor heapUsedMB=94, heapTotalMB=131, cpuAverage=1.35, intervalMs=5001 ``` Note in "Before" that heapTotalMB goes up to 2GB in the first case, and "intervalMs" of 17 seconds indicates that node was unresponsive for that long. In the second case, heapTotalMB stays low, and the main thread remains responsive the whole time. Reviewers: jarek Reviewed By: jarek Differential Revision: https://phab.getgrist.com/D3906 |
1 year ago |
Dmitry S | 51ff72c15e |
(core) Faster builds all around.
Summary: Building: - Builds no longer wait for tsc for either client, server, or test targets. All use esbuild which is very fast. - Build still runs tsc, but only to report errors. This may be turned off with `SKIP_TSC=1` env var. - Grist-core continues to build using tsc. - Esbuild requires ES6 module semantics. Typescript's esModuleInterop is turned on, so that tsc accepts and enforces correct usage. - Client-side code is watched and bundled by webpack as before (using esbuild-loader) Code changes: - Imports must now follow ES6 semantics: `import * as X from ...` produces a module object; to import functions or class instances, use `import X from ...`. - Everything is now built with isolatedModules flag. Some exports were updated for it. Packages: - Upgraded browserify dependency, and related packages (used for the distribution-building step). - Building the distribution now uses esbuild's minification. babel-minify is no longer used. Test Plan: Should have no behavior changes, existing tests should pass, and docker image should build too. Reviewers: georgegevoian Reviewed By: georgegevoian Subscribers: alexmojaki Differential Revision: https://phab.getgrist.com/D3506 |
2 years ago |
George Gevoian | 4c5de16e2d |
(core) Include altSessionId in logs
Summary: Adds altSessionId to log output. Test Plan: Tested manually. Reviewers: paulfitz Reviewed By: paulfitz Differential Revision: https://phab.getgrist.com/D3355 |
3 years ago |
Paul Fitzpatrick | bb8cb2593d |
(core) support python3 in grist-core, and running engine via docker and/or gvisor
Summary: * Moves essential plugins to grist-core, so that basic imports (e.g. csv) work. * Adds support for a `GRIST_SANDBOX_FLAVOR` flag that can systematically override how the data engine is run. - `GRIST_SANDBOX_FLAVOR=pynbox` is "classic" nacl-based sandbox. - `GRIST_SANDBOX_FLAVOR=docker` runs engines in individual docker containers. It requires an image specified in `sandbox/docker` (alternative images can be named with `GRIST_SANDBOX` flag - need to contain python and engine requirements). It is a simple reference implementation for sandboxing. - `GRIST_SANDBOX_FLAVOR=unsandboxed` runs whatever local version of python is specified by a `GRIST_SANDBOX` flag directly, with no sandboxing. Engine requirements must be installed, so an absolute path to a python executable in a virtualenv is easiest to manage. - `GRIST_SANDBOX_FLAVOR=gvisor` runs the data engine via gvisor's runsc. Experimental, with implementation not included in grist-core. Since gvisor runs on Linux only, this flavor supports wrapping the sandboxes in a single shared docker container. * Tweaks some recent express query parameter code to work in grist-core, which has a slightly different version of express (smoke test doesn't catch this since in Jenkins core is built within a workspace that has node_modules, and wires get crossed - in a dev environment the problem on master can be seen by doing `buildtools/build_core.sh /tmp/any_path_outside_grist`). The new sandbox options do not have tests yet, nor does this they change the behavior of grist servers today. They are there to clean up and consolidate a collection of patches I've been using that were getting cumbersome, and make it easier to run experiments. I haven't looked closely at imports beyond core. Test Plan: tested manually against regular grist and grist-core, including imports Reviewers: alexmojaki, dsagal Reviewed By: alexmojaki Differential Revision: https://phab.getgrist.com/D2942 |
3 years ago |
Jarosław Sadziński | 08295a696b |
(core) Export to Excel and Send to drive
Summary: Implementing export to excel and send to Google Drive feature. As part of this feature few things were implemented: - Server side google authentication exposed on url: (docs, docs-s, or localhost:8080)/auth/google - Exporting grist documents as an excel file (xlsx) - Storing exported grist document (in excel format) in Google Drive as a spreadsheet document. Server side google authentication requires one new environmental variables - GOOGLE_CLIENT_SECRET (required) used by authentication handler Test Plan: Browser tests for exporting to excel. Reviewers: paulfitz, dsagal Reviewed By: paulfitz Differential Revision: https://phab.getgrist.com/D2924 |
3 years ago |