2020-05-20 04:50:46 +00:00
|
|
|
{
|
|
|
|
|
"name": "grist-core",
|
2023-11-20 15:20:13 +00:00
|
|
|
"version": "1.1.8",
|
2020-10-28 17:37:36 +00:00
|
|
|
"license": "Apache-2.0",
|
|
|
|
|
"description": "Grist is the evolution of spreadsheets",
|
|
|
|
|
"homepage": "https://github.com/gristlabs/grist-core",
|
|
|
|
|
"repository": "git://github.com/gristlabs/grist-core.git",
|
2020-05-20 04:50:46 +00:00
|
|
|
"scripts": {
|
(core) add machinery for self-managed flavor of Grist
Summary:
Currently, we have two ways that we deliver Grist. One is grist-core,
which has simple defaults and is relatively easy for third parties to
deploy. The second is our internal build for our SaaS, which is the
opposite. For self-managed Grist, a planned paid on-premise version
of Grist, I adopt the following approach:
* Use the `grist-core` build mechanism, extending it to accept an
overlay of extra code if present.
* Extra code is supplied in a self-contained `ext` directory, with
an `ext/app` directory that is of same structure as core `app`
and `stubs/app`.
* The `ext` directory also contains information about extra
node dependencies needed beyond that of `grist-core`.
* The `ext` directory is contained within our monorepo rather than
`grist-core` since it may contain material not under the Apache
license.
Docker builds are achieved in our monorepo by using the `--build-context`
functionality to add in `ext` during the regular `grist-core` build:
```
docker buildx build --load -t gristlabs/grist-ee --build-context=ext=../ext .
```
Incremental builds in our monorepo are achieved with the `build_core.sh` helper,
like:
```
buildtools/build_core.sh /tmp/self-managed
cd /tmp/self-managed
yarn start
```
The initial `ext` directory contains material for snapshotting to S3.
If you build the docker image as above, and have S3 access, you can
do something like:
```
docker run -p 8484:8484 --env GRIST_SESSION_SECRET=a-secret \
--env GRIST_DOCS_S3_BUCKET=grist-docs-test \
--env GRIST_DOCS_S3_PREFIX=self-managed \
-v $HOME/.aws:/root/.aws -it gristlabs/grist-ee
```
This will start a version of Grist that is like `grist-core` but with
S3 snapshots enabled. To release this code to `grist-core`, it would
just need to move from `ext/app` to `app` within core.
I tried a lot of ways of organizing self-managed Grist, and this was
what made me happiest. There are a lot of trade-offs, but here is what
I was looking for:
* Only OSS-code in grist-core. Adding mixed-license material there
feels unfair to people already working with the repo. That said,
a possible future is to move away from our private monorepo to
a public mixed-licence repo, which could have the same relationship
with grist-core as the monorepo has.
* Minimal differences between self-managed builds and one of our
existing builds, ideally hewing as close to grist-core as possible
for ease of documentation, debugging, and maintenance.
* Ideally, docker builds without copying files around (the new
`--build-context` functionality made that possible).
* Compatibility with monorepo build.
Expressing dependencies of the extra code in `ext` proved tricky to
do in a clean way. Yarn/npm fought me every step of the way - everything
related to optional dependencies was unsatisfactory in some respect.
Yarn2 is flexible but smells like it might be overreach. In the end,
organizing to install non-core dependencies one directory up from the
main build was a good simple trick that saved my bacon.
This diff gets us to the point of building `grist-ee` images conveniently,
but there isn't a public repo people can go look at to see its source. This
could be generated by taking `grist-core`, adding the `ext` directory
to it, and pushing to a distinct repository. I'm not in a hurry to do that,
since a PR to that repo would be hard to sync with our monorepo and
`grist-core`. Also, we don't have any licensing text ready for the `ext`
directory. So leaving that for future work.
Test Plan: manual
Reviewers: georgegevoian, alexmojaki
Reviewed By: georgegevoian, alexmojaki
Differential Revision: https://phab.getgrist.com/D3415
2022-05-12 15:24:48 +00:00
|
|
|
"start": "sandbox/watch.sh",
|
2023-06-20 16:49:57 +00:00
|
|
|
"start:debug": "NODE_INSPECT=1 sandbox/watch.sh",
|
2020-07-27 18:57:36 +00:00
|
|
|
"install:python": "buildtools/prepare_python.sh",
|
2022-01-07 17:06:04 +00:00
|
|
|
"install:python2": "buildtools/prepare_python2.sh",
|
|
|
|
|
"install:python3": "buildtools/prepare_python3.sh",
|
(core) add machinery for self-managed flavor of Grist
Summary:
Currently, we have two ways that we deliver Grist. One is grist-core,
which has simple defaults and is relatively easy for third parties to
deploy. The second is our internal build for our SaaS, which is the
opposite. For self-managed Grist, a planned paid on-premise version
of Grist, I adopt the following approach:
* Use the `grist-core` build mechanism, extending it to accept an
overlay of extra code if present.
* Extra code is supplied in a self-contained `ext` directory, with
an `ext/app` directory that is of same structure as core `app`
and `stubs/app`.
* The `ext` directory also contains information about extra
node dependencies needed beyond that of `grist-core`.
* The `ext` directory is contained within our monorepo rather than
`grist-core` since it may contain material not under the Apache
license.
Docker builds are achieved in our monorepo by using the `--build-context`
functionality to add in `ext` during the regular `grist-core` build:
```
docker buildx build --load -t gristlabs/grist-ee --build-context=ext=../ext .
```
Incremental builds in our monorepo are achieved with the `build_core.sh` helper,
like:
```
buildtools/build_core.sh /tmp/self-managed
cd /tmp/self-managed
yarn start
```
The initial `ext` directory contains material for snapshotting to S3.
If you build the docker image as above, and have S3 access, you can
do something like:
```
docker run -p 8484:8484 --env GRIST_SESSION_SECRET=a-secret \
--env GRIST_DOCS_S3_BUCKET=grist-docs-test \
--env GRIST_DOCS_S3_PREFIX=self-managed \
-v $HOME/.aws:/root/.aws -it gristlabs/grist-ee
```
This will start a version of Grist that is like `grist-core` but with
S3 snapshots enabled. To release this code to `grist-core`, it would
just need to move from `ext/app` to `app` within core.
I tried a lot of ways of organizing self-managed Grist, and this was
what made me happiest. There are a lot of trade-offs, but here is what
I was looking for:
* Only OSS-code in grist-core. Adding mixed-license material there
feels unfair to people already working with the repo. That said,
a possible future is to move away from our private monorepo to
a public mixed-licence repo, which could have the same relationship
with grist-core as the monorepo has.
* Minimal differences between self-managed builds and one of our
existing builds, ideally hewing as close to grist-core as possible
for ease of documentation, debugging, and maintenance.
* Ideally, docker builds without copying files around (the new
`--build-context` functionality made that possible).
* Compatibility with monorepo build.
Expressing dependencies of the extra code in `ext` proved tricky to
do in a clean way. Yarn/npm fought me every step of the way - everything
related to optional dependencies was unsatisfactory in some respect.
Yarn2 is flexible but smells like it might be overreach. In the end,
organizing to install non-core dependencies one directory up from the
main build was a good simple trick that saved my bacon.
This diff gets us to the point of building `grist-ee` images conveniently,
but there isn't a public repo people can go look at to see its source. This
could be generated by taking `grist-core`, adding the `ext` directory
to it, and pushing to a distinct repository. I'm not in a hurry to do that,
since a PR to that repo would be hard to sync with our monorepo and
`grist-core`. Also, we don't have any licensing text ready for the `ext`
directory. So leaving that for future work.
Test Plan: manual
Reviewers: georgegevoian, alexmojaki
Reviewed By: georgegevoian, alexmojaki
Differential Revision: https://phab.getgrist.com/D3415
2022-05-12 15:24:48 +00:00
|
|
|
"build:prod": "buildtools/build.sh",
|
|
|
|
|
"start:prod": "sandbox/run.sh",
|
2023-11-27 11:26:11 +00:00
|
|
|
"test": "GRIST_SESSION_COOKIE=grist_test_cookie GRIST_TEST_LOGIN=1 TEST_SUPPORT_API_KEY=api_key_for_support TEST_CLEAN_DATABASE=true LANGUAGE=en_US mocha ${DEBUG:+-b --no-exit} --slow 8000 $([ -z $DEBUG ] && echo --forbid-only) -g \"${GREP_TESTS}\" '_build/test/common/*.js' '_build/test/client/*.js' '_build/test/nbrowser/*.js' '_build/test/server/**/*.js' '_build/test/gen-server/**/*.js'",
|
|
|
|
|
"test:nbrowser": "TEST_SUITE=nbrowser TEST_SUITE_FOR_TIMINGS=nbrowser TIMINGS_FILE=test/timings/nbrowser.txt GRIST_SESSION_COOKIE=grist_test_cookie GRIST_TEST_LOGIN=1 TEST_SUPPORT_API_KEY=api_key_for_support TEST_CLEAN_DATABASE=true LANGUAGE=en_US mocha ${DEBUG:+-b --no-exit} $([ -z $DEBUG ] && echo --forbid-only) -g \"${GREP_TESTS}\" --slow 8000 -R test/xunit-file '_build/test/nbrowser/**/*.js'",
|
2023-06-27 06:11:08 +00:00
|
|
|
"test:client": "GRIST_SESSION_COOKIE=grist_test_cookie mocha ${DEBUG:+'-b'} '_build/test/client/**/*.js'",
|
|
|
|
|
"test:common": "GRIST_SESSION_COOKIE=grist_test_cookie mocha ${DEBUG:+'-b'} '_build/test/common/**/*.js'",
|
|
|
|
|
"test:server": "TEST_SUITE=server TEST_SUITE_FOR_TIMINGS=server TIMINGS_FILE=test/timings/server.txt GRIST_SESSION_COOKIE=grist_test_cookie mocha ${DEBUG:+'-b'} -R test/xunit-file '_build/test/server/**/*.js' '_build/test/gen-server/**/*.js'",
|
|
|
|
|
"test:smoke": "mocha _build/test/nbrowser/Smoke.js",
|
(core) add a `yarn run cli` tool, and add a `sqlite gristify` option
Summary:
This adds rudimentary support for opening certain SQLite files in Grist.
If you have a file such as `landing.db` in Grist, you can convert it to Grist format by doing (either in monorepo or grist-core):
```
yarn run cli -h
yarn run cli sqlite -h
yarn run cli sqlite gristify landing.db
```
The file is now openable by Grist. To actually do so with the regular Grist server, you'll need to either import it, or convert some doc you don't care about in the `samples/` directory to be a soft link to it (and then force a reload).
This implementation is a rudimentary experiment. Here are some awkwardnesses:
* Only tables that happen to have a column called `id`, and where the column happens to be an integer, can be opened directly with Grist as it is today. That could be generalized, but it looked more than a Gristathon's worth of work, so I instead used SQLite views.
* Grist will handle tables that start with an uncapitalized letter a bit erratically. You can successfully add columns, for example, but removing them will cause sadness - Grist will rename the table in a confused way.
* I didn't attempt to deal with column names with spaces etc (though views could deal with those).
* I haven't tried to do any fancy type mapping.
* Columns with constraints can make adding new rows impossible in Grist, since Grist requires that a row can be added with just a single cell set.
Test Plan: added small test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3502
2022-07-14 09:32:06 +00:00
|
|
|
"test:docker": "./test/test_under_docker.sh",
|
2022-10-03 15:13:37 +00:00
|
|
|
"test:python": "sandbox_venv3/bin/python sandbox/grist/runtests.py ${GREP_TESTS:+discover -p \"test*${GREP_TESTS}*.py\"}",
|
2022-12-15 09:58:12 +00:00
|
|
|
"cli": "NODE_PATH=_build:_build/stubs:_build/ext node _build/app/server/companion.js",
|
2022-12-27 18:35:03 +00:00
|
|
|
"lint": "eslint --cache --cache-strategy content .",
|
2022-12-27 18:57:55 +00:00
|
|
|
"lint:fix": "eslint --cache --cache-strategy=content --fix .",
|
|
|
|
|
"lint:ci": "eslint --max-warnings=0 .",
|
2023-01-17 20:54:41 +00:00
|
|
|
"generate:translation": "NODE_PATH=_build:_build/stubs:_build/ext node buildtools/generate_translation_keys.js",
|
|
|
|
|
"generate:schema:ts": "buildtools/update_schema.sh"
|
2020-05-20 04:50:46 +00:00
|
|
|
},
|
2020-10-28 17:37:36 +00:00
|
|
|
"keywords": [
|
|
|
|
|
"grist",
|
|
|
|
|
"spreadsheet",
|
|
|
|
|
"database"
|
|
|
|
|
],
|
|
|
|
|
"author": {
|
|
|
|
|
"name": "Grist Labs Inc.",
|
|
|
|
|
"email": "info@getgrist.com"
|
|
|
|
|
},
|
|
|
|
|
"private": false,
|
2020-05-20 04:50:46 +00:00
|
|
|
"devDependencies": {
|
2022-12-16 16:37:55 +00:00
|
|
|
"@babel/core": "7.18.5",
|
|
|
|
|
"@babel/eslint-parser": "7.18.2",
|
2021-08-26 21:06:55 +00:00
|
|
|
"@types/accept-language-parser": "1.5.2",
|
2020-07-21 13:20:51 +00:00
|
|
|
"@types/backbone": "1.3.43",
|
2021-04-02 23:11:27 +00:00
|
|
|
"@types/chai": "4.1.7",
|
|
|
|
|
"@types/chai-as-promised": "7.1.0",
|
2020-07-21 13:20:51 +00:00
|
|
|
"@types/content-disposition": "0.5.2",
|
2020-10-02 15:10:00 +00:00
|
|
|
"@types/diff-match-patch": "1.0.32",
|
2023-03-22 13:48:50 +00:00
|
|
|
"@types/dompurify": "2.4.0",
|
2020-07-21 13:20:51 +00:00
|
|
|
"@types/double-ended-queue": "2.1.0",
|
2023-10-11 21:03:02 +00:00
|
|
|
"@types/express": "4.17.17",
|
2021-04-02 23:11:27 +00:00
|
|
|
"@types/form-data": "2.2.1",
|
2020-07-21 13:20:51 +00:00
|
|
|
"@types/fs-extra": "5.0.4",
|
(core) Moving nbrowser tests to grist-core
Summary:
Moving bulk of nbrowser tests to core. Some tests were split and only part of them were moved.
Tests that are left are either: not suitable for grist-core (like billing) or are failing during browser tests (are not reliable).
Four fixtures directory (uploads, docs, exports-csv/excel) where completely moved to grist-core and are linked as folders.
Those changes allows to add an nbrowser test in grist-core or in the main test folder without any need to link it or link a fixture document.
Other changes:
- testrun.sh has been modified, now it runs tests from both folders (test and core/test),
- TestServer used in grist-core is now adding sample orgs and users (kiwi and others),
Test modified
- SelectionSummary: now it is run on a bigScreen, it was failing randomly
- Billing.ts: relative paths were used
- DateEditor: added waitForServer - it was failing in browser mode
- FrozenColumns, ImportFromGDrive, Printing: updated import paths
- UserManager.ts: was split into two parts (it assumed limited products)
- ViewLayoutResize.ts: this test is still in main repo, it is still failing in browser mode tests
Test Plan: Existing
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: dsagal, paulfitz
Differential Revision: https://phab.getgrist.com/D3664
2022-10-21 21:34:26 +00:00
|
|
|
"@types/http-proxy": "1.17.9",
|
2022-09-29 08:01:37 +00:00
|
|
|
"@types/i18next-fs-backend": "1.1.2",
|
2020-07-21 13:20:51 +00:00
|
|
|
"@types/image-size": "0.0.29",
|
|
|
|
|
"@types/js-yaml": "3.11.2",
|
2022-08-18 21:08:39 +00:00
|
|
|
"@types/jsdom": "16.2.14",
|
2022-07-27 20:20:14 +00:00
|
|
|
"@types/jsesc": "3.0.1",
|
2022-07-19 15:39:49 +00:00
|
|
|
"@types/jsonwebtoken": "7.2.8",
|
2020-07-21 13:20:51 +00:00
|
|
|
"@types/lodash": "4.14.117",
|
2021-10-01 13:45:27 +00:00
|
|
|
"@types/lru-cache": "5.1.1",
|
2023-03-22 13:48:50 +00:00
|
|
|
"@types/marked": "4.0.8",
|
2021-04-02 23:11:27 +00:00
|
|
|
"@types/mime-types": "2.1.0",
|
2022-12-21 15:11:25 +00:00
|
|
|
"@types/minio": "7.0.15",
|
2023-10-11 21:03:02 +00:00
|
|
|
"@types/mocha": "10.0.1",
|
2020-07-21 13:20:51 +00:00
|
|
|
"@types/moment-timezone": "0.5.9",
|
2023-09-26 20:37:27 +00:00
|
|
|
"@types/mousetrap": "1.6.2",
|
2023-10-11 21:03:02 +00:00
|
|
|
"@types/node": "18.11.9",
|
2022-11-30 12:04:27 +00:00
|
|
|
"@types/node-fetch": "2.6.2",
|
2020-07-21 13:20:51 +00:00
|
|
|
"@types/pidusage": "2.0.1",
|
2022-08-02 14:54:54 +00:00
|
|
|
"@types/plotly.js": "2.12.1",
|
2023-10-11 21:03:02 +00:00
|
|
|
"@types/proper-lockfile": "4.1.2",
|
2022-01-07 18:11:52 +00:00
|
|
|
"@types/qrcode": "1.4.2",
|
2020-07-21 13:20:51 +00:00
|
|
|
"@types/redlock": "3.0.2",
|
2021-08-16 15:11:17 +00:00
|
|
|
"@types/saml2-js": "2.0.1",
|
2023-10-11 21:03:02 +00:00
|
|
|
"@types/selenium-webdriver": "4.1.15",
|
2022-05-18 10:25:14 +00:00
|
|
|
"@types/sinon": "5.0.5",
|
2020-07-21 13:20:51 +00:00
|
|
|
"@types/sqlite3": "3.1.6",
|
|
|
|
|
"@types/tmp": "0.0.33",
|
|
|
|
|
"@types/uuid": "3.4.4",
|
(core) support python3 in grist-core, and running engine via docker and/or gvisor
Summary:
* Moves essential plugins to grist-core, so that basic imports (e.g. csv) work.
* Adds support for a `GRIST_SANDBOX_FLAVOR` flag that can systematically override how the data engine is run.
- `GRIST_SANDBOX_FLAVOR=pynbox` is "classic" nacl-based sandbox.
- `GRIST_SANDBOX_FLAVOR=docker` runs engines in individual docker containers. It requires an image specified in `sandbox/docker` (alternative images can be named with `GRIST_SANDBOX` flag - need to contain python and engine requirements). It is a simple reference implementation for sandboxing.
- `GRIST_SANDBOX_FLAVOR=unsandboxed` runs whatever local version of python is specified by a `GRIST_SANDBOX` flag directly, with no sandboxing. Engine requirements must be installed, so an absolute path to a python executable in a virtualenv is easiest to manage.
- `GRIST_SANDBOX_FLAVOR=gvisor` runs the data engine via gvisor's runsc. Experimental, with implementation not included in grist-core. Since gvisor runs on Linux only, this flavor supports wrapping the sandboxes in a single shared docker container.
* Tweaks some recent express query parameter code to work in grist-core, which has a slightly different version of express (smoke test doesn't catch this since in Jenkins core is built within a workspace that has node_modules, and wires get crossed - in a dev environment the problem on master can be seen by doing `buildtools/build_core.sh /tmp/any_path_outside_grist`).
The new sandbox options do not have tests yet, nor does this they change the behavior of grist servers today. They are there to clean up and consolidate a collection of patches I've been using that were getting cumbersome, and make it easier to run experiments.
I haven't looked closely at imports beyond core.
Test Plan: tested manually against regular grist and grist-core, including imports
Reviewers: alexmojaki, dsagal
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D2942
2021-07-27 23:43:21 +00:00
|
|
|
"@types/which": "2.0.1",
|
2022-06-04 04:12:30 +00:00
|
|
|
"@types/ws": "^6",
|
2022-12-16 16:37:55 +00:00
|
|
|
"@typescript-eslint/eslint-plugin": "5.29.0",
|
|
|
|
|
"@typescript-eslint/parser": "5.29.0",
|
2022-03-24 17:11:26 +00:00
|
|
|
"app-module-path": "2.2.0",
|
2020-10-02 15:10:00 +00:00
|
|
|
"catw": "1.0.1",
|
2021-04-02 23:11:27 +00:00
|
|
|
"chai": "4.2.0",
|
|
|
|
|
"chai-as-promised": "7.1.1",
|
2022-08-18 21:08:39 +00:00
|
|
|
"chance": "1.0.16",
|
(core) Speed up and upgrade build.
Summary:
- Upgrades to build-related packages:
- Upgrade typescript, related libraries and typings.
- Upgrade webpack, eslint; add tsc-watch, node-dev, eslint_d.
- Build organization changes:
- Build webpack from original typescript, transpiling only; with errors still
reported by a background tsc watching process.
- Typescript-related changes:
- Reduce imports of AWS dependencies (very noticeable speedup)
- Avoid auto-loading global @types
- Client code is now built with isolatedModules flag (for safe transpilation)
- Use allowJs to avoid copying JS files manually.
- Linting changes
- Enhance Arcanist ESLintLinter to run before/after commands, and set up to use eslint_d
- Update eslint config, and include .eslintignore to avoid linting generated files.
- Include a bunch of eslint-prompted and eslint-generated fixes
- Add no-unused-expression rule to eslint, and fix a few warnings about it
- Other items:
- Refactor cssInput to avoid circular dependency
- Remove a bit of unused code, libraries, dependencies
Test Plan: No behavior changes, all existing tests pass. There are 30 tests fewer reported because `test_gpath.py` was removed (it's been unused for years)
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3498
2022-06-27 20:09:41 +00:00
|
|
|
"esbuild-loader": "2.19.0",
|
2022-12-16 16:37:55 +00:00
|
|
|
"eslint": "8.18.0",
|
(core) Moving nbrowser tests to grist-core
Summary:
Moving bulk of nbrowser tests to core. Some tests were split and only part of them were moved.
Tests that are left are either: not suitable for grist-core (like billing) or are failing during browser tests (are not reliable).
Four fixtures directory (uploads, docs, exports-csv/excel) where completely moved to grist-core and are linked as folders.
Those changes allows to add an nbrowser test in grist-core or in the main test folder without any need to link it or link a fixture document.
Other changes:
- testrun.sh has been modified, now it runs tests from both folders (test and core/test),
- TestServer used in grist-core is now adding sample orgs and users (kiwi and others),
Test modified
- SelectionSummary: now it is run on a bigScreen, it was failing randomly
- Billing.ts: relative paths were used
- DateEditor: added waitForServer - it was failing in browser mode
- FrozenColumns, ImportFromGDrive, Printing: updated import paths
- UserManager.ts: was split into two parts (it assumed limited products)
- ViewLayoutResize.ts: this test is still in main repo, it is still failing in browser mode tests
Test Plan: Existing
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: dsagal, paulfitz
Differential Revision: https://phab.getgrist.com/D3664
2022-10-21 21:34:26 +00:00
|
|
|
"http-proxy": "1.18.1",
|
2022-12-16 17:10:19 +00:00
|
|
|
"i18next-scanner": "4.1.0",
|
2022-09-09 15:15:07 +00:00
|
|
|
"jsdom": "16.5.0",
|
2023-06-27 06:11:08 +00:00
|
|
|
"mocha": "10.2.0",
|
2023-10-11 21:03:02 +00:00
|
|
|
"mocha-webdriver": "0.3.1",
|
2020-05-20 04:50:46 +00:00
|
|
|
"moment-locales-webpack-plugin": "^1.2.0",
|
|
|
|
|
"nodemon": "^2.0.4",
|
2022-04-01 21:31:24 +00:00
|
|
|
"otplib": "12.0.1",
|
2023-10-11 21:03:02 +00:00
|
|
|
"proper-lockfile": "4.1.2",
|
2022-05-18 10:25:14 +00:00
|
|
|
"sinon": "7.1.1",
|
2020-05-20 04:50:46 +00:00
|
|
|
"source-map-loader": "^0.2.4",
|
2021-07-19 14:44:16 +00:00
|
|
|
"tmp-promise": "1.0.5",
|
2022-09-05 08:24:34 +00:00
|
|
|
"ts-interface-builder": "0.3.2",
|
(core) Speed up and upgrade build.
Summary:
- Upgrades to build-related packages:
- Upgrade typescript, related libraries and typings.
- Upgrade webpack, eslint; add tsc-watch, node-dev, eslint_d.
- Build organization changes:
- Build webpack from original typescript, transpiling only; with errors still
reported by a background tsc watching process.
- Typescript-related changes:
- Reduce imports of AWS dependencies (very noticeable speedup)
- Avoid auto-loading global @types
- Client code is now built with isolatedModules flag (for safe transpilation)
- Use allowJs to avoid copying JS files manually.
- Linting changes
- Enhance Arcanist ESLintLinter to run before/after commands, and set up to use eslint_d
- Update eslint config, and include .eslintignore to avoid linting generated files.
- Include a bunch of eslint-prompted and eslint-generated fixes
- Add no-unused-expression rule to eslint, and fix a few warnings about it
- Other items:
- Refactor cssInput to avoid circular dependency
- Remove a bit of unused code, libraries, dependencies
Test Plan: No behavior changes, all existing tests pass. There are 30 tests fewer reported because `test_gpath.py` was removed (it's been unused for years)
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3498
2022-06-27 20:09:41 +00:00
|
|
|
"typescript": "4.7.4",
|
|
|
|
|
"webpack": "5.73.0",
|
|
|
|
|
"webpack-cli": "4.10.0",
|
2021-04-26 22:49:02 +00:00
|
|
|
"why-is-node-running": "2.0.3"
|
2020-05-20 04:50:46 +00:00
|
|
|
},
|
|
|
|
|
"dependencies": {
|
2021-07-21 08:46:03 +00:00
|
|
|
"@googleapis/drive": "0.3.1",
|
|
|
|
|
"@googleapis/oauth2": "0.2.0",
|
2023-02-02 16:33:33 +00:00
|
|
|
"@gristlabs/connect-sqlite3": "0.9.11-grist.5",
|
2020-07-21 13:20:51 +00:00
|
|
|
"@gristlabs/express-session": "1.17.0",
|
2022-02-21 14:45:17 +00:00
|
|
|
"@gristlabs/moment-guess": "1.2.4-grist.1",
|
2020-07-21 13:20:51 +00:00
|
|
|
"@gristlabs/pidusage": "2.0.17",
|
2023-02-14 17:39:12 +00:00
|
|
|
"@gristlabs/sqlite3": "5.1.4-grist.8",
|
2020-10-02 15:10:00 +00:00
|
|
|
"@popperjs/core": "2.3.3",
|
2021-08-26 21:06:55 +00:00
|
|
|
"accept-language-parser": "1.5.0",
|
2023-07-13 14:00:56 +00:00
|
|
|
"ace-builds": "1.23.3",
|
2020-10-30 16:53:23 +00:00
|
|
|
"async-mutex": "0.2.4",
|
2022-09-07 15:28:53 +00:00
|
|
|
"axios": "0.21.2",
|
2020-07-21 13:20:51 +00:00
|
|
|
"backbone": "1.3.3",
|
2020-10-02 15:10:00 +00:00
|
|
|
"bootstrap": "3.3.5",
|
|
|
|
|
"bootstrap-datepicker": "1.9.0",
|
|
|
|
|
"bowser": "2.7.0",
|
2020-07-21 13:20:51 +00:00
|
|
|
"collect-js-deps": "^0.1.1",
|
2022-08-08 13:32:50 +00:00
|
|
|
"color-convert": "2.0.1",
|
(core) add a `yarn run cli` tool, and add a `sqlite gristify` option
Summary:
This adds rudimentary support for opening certain SQLite files in Grist.
If you have a file such as `landing.db` in Grist, you can convert it to Grist format by doing (either in monorepo or grist-core):
```
yarn run cli -h
yarn run cli sqlite -h
yarn run cli sqlite gristify landing.db
```
The file is now openable by Grist. To actually do so with the regular Grist server, you'll need to either import it, or convert some doc you don't care about in the `samples/` directory to be a soft link to it (and then force a reload).
This implementation is a rudimentary experiment. Here are some awkwardnesses:
* Only tables that happen to have a column called `id`, and where the column happens to be an integer, can be opened directly with Grist as it is today. That could be generalized, but it looked more than a Gristathon's worth of work, so I instead used SQLite views.
* Grist will handle tables that start with an uncapitalized letter a bit erratically. You can successfully add columns, for example, but removing them will cause sadness - Grist will rename the table in a confused way.
* I didn't attempt to deal with column names with spaces etc (though views could deal with those).
* I haven't tried to do any fancy type mapping.
* Columns with constraints can make adding new rows impossible in Grist, since Grist requires that a row can be added with just a single cell set.
Test Plan: added small test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3502
2022-07-14 09:32:06 +00:00
|
|
|
"commander": "9.3.0",
|
2020-10-02 15:10:00 +00:00
|
|
|
"components-jqueryui": "1.12.1",
|
2022-04-12 20:39:56 +00:00
|
|
|
"connect-redis": "3.4.0",
|
2022-05-18 10:25:14 +00:00
|
|
|
"cookie": "0.5.0",
|
2020-07-21 13:20:51 +00:00
|
|
|
"cookie-parser": "1.4.3",
|
|
|
|
|
"csv": "4.0.0",
|
2023-05-05 09:34:12 +00:00
|
|
|
"currency-symbol-map": "5.1.0",
|
2020-10-02 15:10:00 +00:00
|
|
|
"diff-match-patch": "1.0.5",
|
2023-03-22 13:48:50 +00:00
|
|
|
"dompurify": "3.0.0",
|
2020-07-21 13:20:51 +00:00
|
|
|
"double-ended-queue": "2.1.0-0",
|
2021-07-21 08:46:03 +00:00
|
|
|
"exceljs": "4.2.1",
|
2023-10-11 21:03:02 +00:00
|
|
|
"express": "4.18.2",
|
2022-09-12 13:34:28 +00:00
|
|
|
"file-type": "16.5.4",
|
2020-07-21 13:20:51 +00:00
|
|
|
"fs-extra": "7.0.0",
|
2021-10-15 09:31:13 +00:00
|
|
|
"grain-rpc": "0.1.7",
|
(core) move more tests to grist-core
Summary:
* Tie build and run-time docker base images to a consistent version (buster)
* Extend the test login system activated by GRIST_TEST_LOGIN to ease porting tests that currently rely on cognito (many)
* Make org resets work in absence of billing endpoints
* When in-memory session caches are used, add missing invalidation steps
* Pass org information through sign-ups/sign-ins more carefully
* For CORS, explicitly trust GRIST_HOST origin when set
* Move some fixtures and tests to core, focussing on tests that cover existing failures or are in the set of tests run on deployments
* Retain regular `test` target to run the test suite directly, without docker
* Add a `test:smoke` target to run a single simple test without `GRIST_TEST_LOGIN` activated
* Add a `test:docker` target to run the tests against a grist-core docker image - since tests rely on certain fixture teams/docs, added `TEST_SUPPORT_API_KEY` and `TEST_ADD_SAMPLES` flags to ease porting
The tests ported were `nbrowser` tests: `ActionLog.ts` (the first test I tend to port to anything, out of habit), `Fork.ts` (exercises a lot of doc creation paths), `HomeIntro.ts` (a lot of DocMenu exercise), and `DuplicateDocument.ts` (covers a feature known to be failing prior to this diff, the CORS tweak resolves it).
Test Plan: Manually tested via `buildtools/build_core.sh`. In follow up, I want to add running the `test:docker` target in grist-core's workflows. In jenkins, only the smoke test is run. There'd be an argument for running all tests, but they include particularly slow tests, and are duplicates of tests already run (in different configuration admittedly), so I'd like to try first just using them in grist-core to gate updates to any packaged version of Grist (the docker image currently).
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3176
2021-12-10 22:42:54 +00:00
|
|
|
"grainjs": "1.0.2",
|
2022-09-07 15:28:53 +00:00
|
|
|
"handlebars": "4.7.7",
|
2022-09-12 13:34:28 +00:00
|
|
|
"highlight.js": "10.7.3",
|
2022-06-17 18:49:18 +00:00
|
|
|
"http-proxy-agent": "5.0.0",
|
|
|
|
|
"https-proxy-agent": "5.0.1",
|
2020-10-26 14:45:31 +00:00
|
|
|
"i18n-iso-countries": "6.1.0",
|
2022-09-29 08:01:37 +00:00
|
|
|
"i18next": "21.9.1",
|
2023-10-11 21:03:02 +00:00
|
|
|
"i18next-http-middleware": "3.3.2",
|
2020-07-21 13:20:51 +00:00
|
|
|
"image-size": "0.6.3",
|
2022-09-07 15:28:53 +00:00
|
|
|
"jquery": "3.5.0",
|
2022-09-12 13:34:28 +00:00
|
|
|
"js-yaml": "3.14.1",
|
2022-07-27 20:20:14 +00:00
|
|
|
"jsesc": "3.0.2",
|
2022-07-19 15:39:49 +00:00
|
|
|
"jsonwebtoken": "8.3.0",
|
2020-07-21 13:20:51 +00:00
|
|
|
"knockout": "3.5.0",
|
2021-08-26 21:06:55 +00:00
|
|
|
"locale-currency": "0.0.2",
|
2022-09-09 15:15:07 +00:00
|
|
|
"lodash": "4.17.21",
|
2023-03-22 13:48:50 +00:00
|
|
|
"marked": "4.2.12",
|
2022-12-21 15:11:25 +00:00
|
|
|
"minio": "7.0.32",
|
2022-09-07 15:28:53 +00:00
|
|
|
"moment": "2.29.4",
|
|
|
|
|
"moment-timezone": "0.5.35",
|
2020-07-21 13:20:51 +00:00
|
|
|
"morgan": "1.9.1",
|
2020-10-02 15:10:00 +00:00
|
|
|
"mousetrap": "1.6.2",
|
2021-05-13 02:06:19 +00:00
|
|
|
"multiparty": "4.2.2",
|
2022-11-30 12:04:27 +00:00
|
|
|
"node-abort-controller": "3.0.1",
|
2022-09-07 15:28:53 +00:00
|
|
|
"node-fetch": "2.6.7",
|
2023-11-14 09:31:29 +00:00
|
|
|
"openid-client": "5.6.1",
|
2022-05-18 19:43:51 +00:00
|
|
|
"pg": "8.6.0",
|
(core) For exporting XLSX, do it memory-efficiently in a worker thread.
Summary:
- Excel exports were awfully memory-inefficient, causing occasional docWorker
crashes. The fix is to use the "streaming writer" option of ExcelJS
https://github.com/exceljs/exceljs#streaming-xlsx-writercontents. (Empirically
on one example, max memory went down from 3G to 100M)
- It's also CPU intensive and synchronous, and can block node for tens of
seconds. The fix is to use a worker-thread. This diff uses "piscina" library
for a pool of threads.
- Additionally, adds ProcessMonitor that logs memory and cpu usage,
particularly when those change significantly.
- Also introduces request cancellation, so that a long download cancelled by
the user will cancel the work being done in the worker thread.
Test Plan:
Updated previous export tests; memory and CPU performance tested
manually by watching output of ProcessMonitor.
Difference visible in these log excerpts:
Before (total time to serve request 22 sec):
```
Telemetry processMonitor heapUsedMB=2187, heapTotalMB=2234, cpuAverage=1.13, intervalMs=17911
Telemetry processMonitor heapUsedMB=2188, heapTotalMB=2234, cpuAverage=0.66, intervalMs=5005
Telemetry processMonitor heapUsedMB=2188, heapTotalMB=2234, cpuAverage=0, intervalMs=5005
Telemetry processMonitor heapUsedMB=71, heapTotalMB=75, cpuAverage=0.13, intervalMs=5002
```
After (total time to server request 18 sec):
```
Telemetry processMonitor heapUsedMB=109, heapTotalMB=144, cpuAverage=0.5, intervalMs=5001
Telemetry processMonitor heapUsedMB=109, heapTotalMB=144, cpuAverage=1.39, intervalMs=5002
Telemetry processMonitor heapUsedMB=94, heapTotalMB=131, cpuAverage=1.13, intervalMs=5000
Telemetry processMonitor heapUsedMB=94, heapTotalMB=131, cpuAverage=1.35, intervalMs=5001
```
Note in "Before" that heapTotalMB goes up to 2GB in the first case, and "intervalMs" of 17 seconds indicates that node was unresponsive for that long. In the second case, heapTotalMB stays low, and the main thread remains responsive the whole time.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3906
2023-06-01 13:09:50 +00:00
|
|
|
"piscina": "3.2.0",
|
2022-08-02 14:54:54 +00:00
|
|
|
"plotly.js-basic-dist": "2.13.2",
|
2020-10-02 15:10:00 +00:00
|
|
|
"popper-max-size-modifier": "0.2.0",
|
2023-10-17 18:14:54 +00:00
|
|
|
"popweasel": "0.1.20",
|
2023-10-24 21:36:53 +00:00
|
|
|
"prom-client": "14.2.0",
|
2022-01-07 18:11:52 +00:00
|
|
|
"qrcode": "1.5.0",
|
2020-10-02 15:10:00 +00:00
|
|
|
"randomcolor": "0.5.3",
|
2022-09-07 15:28:53 +00:00
|
|
|
"redis": "3.1.1",
|
2020-07-21 13:20:51 +00:00
|
|
|
"redlock": "3.1.2",
|
2022-09-09 15:15:07 +00:00
|
|
|
"saml2-js": "2.0.5",
|
2020-07-21 13:20:51 +00:00
|
|
|
"short-uuid": "3.1.1",
|
|
|
|
|
"tmp": "0.0.33",
|
2021-10-15 09:31:13 +00:00
|
|
|
"ts-interface-checker": "1.0.2",
|
2022-08-31 16:30:16 +00:00
|
|
|
"typeorm": "0.3.9",
|
2022-09-12 13:34:28 +00:00
|
|
|
"underscore": "1.12.1",
|
2020-07-21 13:20:51 +00:00
|
|
|
"uuid": "3.3.2",
|
2021-05-10 18:44:10 +00:00
|
|
|
"winston": "2.4.5",
|
2023-08-07 22:39:14 +00:00
|
|
|
"ws": "8.13.0"
|
2020-10-02 15:10:00 +00:00
|
|
|
},
|
|
|
|
|
"resolutions": {
|
2022-09-07 15:28:53 +00:00
|
|
|
"jquery": "3.5.0",
|
2023-02-02 16:33:33 +00:00
|
|
|
"ts-interface-checker": "1.0.2",
|
2023-02-14 17:39:12 +00:00
|
|
|
"@gristlabs/sqlite3": "5.1.4-grist.8"
|
2023-06-27 06:11:08 +00:00
|
|
|
},
|
|
|
|
|
"mocha": {
|
2023-07-04 11:23:04 +00:00
|
|
|
"require": [
|
|
|
|
|
"test/setupPaths",
|
|
|
|
|
"source-map-support/register",
|
|
|
|
|
"test/report-why-tests-hang",
|
|
|
|
|
"test/init-mocha-webdriver",
|
|
|
|
|
"test/split-tests",
|
|
|
|
|
"test/chai-as-promised"
|
|
|
|
|
]
|
2020-05-20 04:50:46 +00:00
|
|
|
}
|
|
|
|
|
}
|
