Commit Graph

108 Commits

Author SHA1 Message Date
Paul Fitzpatrick
bde44323c5 (core) apply some dependabot suggestions
Summary:
This applies the set of dependabot suggestions that are currently
passing tests on grist-core. There are a lot more suggestions to
come, an unusual number are not passing tests because tests were
briefly broken.

The list of suggestions is extracted from:

https://api.github.com/repos/gristlabs/grist-core/pulls?search=status:success+state:open

And then applied using:

  yarn upgrade package1@version1 package2@version2 ....

After application, any new entries in package.json are pruned, leaving
just updated entries and yarn.lock changes.

Non-trivial code updates include:
 * A change related to axios typing
 * A change related to jquery dropping `size()` in favor of `length`

Test Plan: existing tests should pass

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3621
2022-09-07 14:15:34 -04:00
Jarosław Sadziński
2438a63255 (core) Moving widget tests to core
Summary:
- Custom widget tests are now in grist-core
- Adding buildtools for grist-plugin-api.js

Test Plan: Existing tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3617
2022-09-06 17:17:14 +02:00
Paul Fitzpatrick
d7b3fb972c (core) upgrade typeorm so we can support newer postgres
Summary:
upgrade typeorm version, so Grist can run against newer versions of postgres.

Dusted off some old benchmarking code to verify that important queries don't get slower. They don't appear to, unlike for some intermediate versions of typeorm I tried in the past.

Most of the changes are because `findOne` changed how it interprets its arguments, and the value it returns when nothing is found. For the return value, I stuck with limiting its impact by emulating old behavior (returning undefined rather than null) rather than propagating the change out to parts of the code unrelated to the database.

Test Plan: existing tests pass; manual testing with postgres 10 and 14

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3613
2022-09-02 15:34:21 -04:00
Jarosław Sadziński
a52d56f613 (core) Moving client and common tests to core
Summary:
- Moved /test/client and /test/common to core.
- Moved two files (CircularArray and RecentItems) from app/common to core/app/common.
- Moved resetOrg test to gen-server.
- `testrun.sh` is now invoking common and client test from core.
- Added missing packages to core's package.json (and revealed underscore as it is used in the main app).
- Removed Coord.js as it is not used anywhere.

Test Plan: Existing tests

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3590
2022-08-23 19:20:10 +02:00
Paul Fitzpatrick
e95b215405 (core) updates from grist-core 2022-08-15 10:00:08 -04:00
Jarosław Sadziński
22ed7d3024 Skipping tests that can't be run on external server. 2022-08-12 17:23:22 +02:00
Jarosław Sadziński
9e4d802405 (core) Implementing row conditional formatting
Summary:
Conditional formatting can now be used for whole rows.
Related fix:
- Font styles weren't applicable for summary columns.
- Checkbox and slider weren't using colors properly

Test Plan: Existing and new tests

Reviewers: paulfitz, georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3547
2022-08-09 20:11:36 +02:00
Dmitry S
787b70f0f6 (core) Upgrade to latest plotly, and set automargin on Y-Axis
Test Plan: Only tested manually

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3543
2022-08-08 14:47:19 -04:00
George Gevoian
c54dde3dba (core) Populate doc title, description and thumbnail in app.html
Summary:
Fills in the title and description/thumbnail (for templates) in app.html if the
page being requested is for a document.

Test Plan: Tested manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3544
2022-07-27 13:57:59 -07:00
Paul Fitzpatrick
dd8d2e18f5 (core) add an access token mechanism to help with attachments in custom widgets
Summary:
With this, a custom widget can render an attachment by doing:
```
const tokenInfo = await grist.docApi.getAccessToken({readOnly: true});
const img = document.getElementById('the_image');
const id = record.C[0];  // get an id of an attachment
const src = `${tokenInfo.baseUrl}/attachments/${id}/download?auth=${tokenInfo.token}`;
img.setAttribute('src', src)
```

The access token expires after a few mins, so if a user right-clicks on an image
to save it, they may get access denied unless they refresh the page. A little awkward,
but s3 pre-authorized links behave similarly and it generally isn't a deal-breaker.

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3488
2022-07-19 11:55:18 -04:00
Paul Fitzpatrick
ec8ab598cb (core) add a yarn run cli tool, and add a sqlite gristify option
Summary:
This adds rudimentary support for opening certain SQLite files in Grist.

If you have a file such as `landing.db` in Grist, you can convert it to Grist format by doing (either in monorepo or grist-core):
```
yarn run cli -h
yarn run cli sqlite -h
yarn run cli sqlite gristify landing.db
```

The file is now openable by Grist. To actually do so with the regular Grist server, you'll need to either import it, or convert some doc you don't care about in the `samples/` directory to be a soft link to it (and then force a reload).

This implementation is a rudimentary experiment. Here are some awkwardnesses:
 * Only tables that happen to have a column called `id`, and where the column happens to be an integer, can be opened directly with Grist as it is today. That could be generalized, but it looked more than a Gristathon's worth of work, so I instead used SQLite views.
 * Grist will handle tables that start with an uncapitalized letter a bit erratically. You can successfully add columns, for example, but removing them will cause sadness - Grist will rename the table in a confused way.
 * I didn't attempt to deal with column names with spaces etc (though views could deal with those).
 * I haven't tried to do any fancy type mapping.
 * Columns with constraints can make adding new rows impossible in Grist, since Grist requires that a row can be added with just a single cell set.

Test Plan: added small test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3502
2022-07-14 12:00:30 -04:00
Dmitry S
dd2eadc86e (core) Speed up and upgrade build.
Summary:
- Upgrades to build-related packages:
  - Upgrade typescript, related libraries and typings.
  - Upgrade webpack, eslint; add tsc-watch, node-dev, eslint_d.

- Build organization changes:
  - Build webpack from original typescript, transpiling only; with errors still
    reported by a background tsc watching process.

- Typescript-related changes:
  - Reduce imports of AWS dependencies (very noticeable speedup)
  - Avoid auto-loading global @types
  - Client code is now built with isolatedModules flag (for safe transpilation)
  - Use allowJs to avoid copying JS files manually.

- Linting changes
  - Enhance Arcanist ESLintLinter to run before/after commands, and set up to use eslint_d
  - Update eslint config, and include .eslintignore to avoid linting generated files.
  - Include a bunch of eslint-prompted and eslint-generated fixes
  - Add no-unused-expression rule to eslint, and fix a few warnings about it

- Other items:
  - Refactor cssInput to avoid circular dependency
  - Remove a bit of unused code, libraries, dependencies

Test Plan: No behavior changes, all existing tests pass. There are 30 tests fewer reported because `test_gpath.py` was removed (it's been unused for years)

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3498
2022-06-27 16:10:10 -04:00
Alex Hall
9fffb491f9 (core) External requests
Summary:
Adds a Python function `REQUEST` which makes an HTTP GET request. Behind the scenes it:

- Raises a special exception to stop trying to evaluate the current cell and just keep the existing value.
- Notes the request arguments which will be returned by `apply_user_actions`.
- Makes the actual request in NodeJS, which sends back the raw response data in a new action `RespondToRequests` which reevaluates the cell(s) that made the request.
- Wraps the response data in a class which mimics the `Response` class of the `requests` library.

In certain cases, this asynchronous flow doesn't work and the sandbox will instead synchronously call an exported JS method:

- When reevaluating a single cell to get a formula error, the request is made synchronously.
- When a formula makes multiple requests, the earlier responses are retrieved synchronously from files which store responses as long as needed to complete evaluating formulas. See https://grist.slack.com/archives/CL1LQ8AT0/p1653399747810139

Test Plan: Added Python and nbrowser tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: paulfitz, dsagal

Differential Revision: https://phab.getgrist.com/D3429
2022-06-17 21:53:20 +02:00
Dmitry S
4f1cb53b29 (core) Converting server-side Comm.js to typescript
Summary:
- Add app/common/CommTypes.ts to define types shared by client and server.
- Include @types/ws npm package

Test Plan: Intended to have no changes in behavior

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3467
2022-06-07 15:47:17 -04:00
Paul Fitzpatrick
2f3cf59fc3 (core) updates from grist-core 2022-05-25 15:59:27 -04:00
Paul Fitzpatrick
fc0d9905d2 v0.7.9 2022-05-19 11:00:11 -04:00
Paul Fitzpatrick
3ddeb511e8 (core) include pg package in grist-core
Summary: Grist works well with postgres@11 and earlier, when the needed TYPEORM_* environment variables are set. This includes the package needed, for convenience.

Test Plan: manual

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3438
2022-05-18 16:30:22 -04:00
Jarosław Sadziński
0ab9e4a6a0 (core) Adding GristConnect login system
Summary:
New login system to allow simple SSO flow that is based on Discourse description that is available at:
https://meta.discourse.org/t/discourseconnect-official-single-sign-on-for-discourse-sso/13045

Test Plan: New core test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3418
2022-05-18 20:28:25 +02:00
Paul Fitzpatrick
e6983e9209 (core) add machinery for self-managed flavor of Grist
Summary:
Currently, we have two ways that we deliver Grist. One is grist-core,
which has simple defaults and is relatively easy for third parties to
deploy. The second is our internal build for our SaaS, which is the
opposite. For self-managed Grist, a planned paid on-premise version
of Grist, I adopt the following approach:

 * Use the `grist-core` build mechanism, extending it to accept an
   overlay of extra code if present.
 * Extra code is supplied in a self-contained `ext` directory, with
   an `ext/app` directory that is of same structure as core `app`
   and `stubs/app`.
 * The `ext` directory also contains information about extra
   node dependencies needed beyond that of `grist-core`.
 * The `ext` directory is contained within our monorepo rather than
   `grist-core` since it may contain material not under the Apache
   license.

Docker builds are achieved in our monorepo by using the `--build-context`
functionality to add in `ext` during the regular `grist-core` build:

```
docker buildx build --load -t gristlabs/grist-ee --build-context=ext=../ext .
```

Incremental builds in our monorepo are achieved with the `build_core.sh` helper,
like:

```
buildtools/build_core.sh /tmp/self-managed
cd /tmp/self-managed
yarn start
```

The initial `ext` directory contains material for snapshotting to S3.
If you build the docker image as above, and have S3 access, you can
do something like:

```
docker run -p 8484:8484 --env GRIST_SESSION_SECRET=a-secret \
  --env GRIST_DOCS_S3_BUCKET=grist-docs-test \
  --env GRIST_DOCS_S3_PREFIX=self-managed \
  -v $HOME/.aws:/root/.aws -it gristlabs/grist-ee
```

This will start a version of Grist that is like `grist-core` but with
S3 snapshots enabled. To release this code to `grist-core`, it would
just need to move from `ext/app` to `app` within core.

I tried a lot of ways of organizing self-managed Grist, and this was
what made me happiest. There are a lot of trade-offs, but here is what
I was looking for:

 * Only OSS-code in grist-core. Adding mixed-license material there
   feels unfair to people already working with the repo. That said,
   a possible future is to move away from our private monorepo to
   a public mixed-licence repo, which could have the same relationship
   with grist-core as the monorepo has.
 * Minimal differences between self-managed builds and one of our
   existing builds, ideally hewing as close to grist-core as possible
   for ease of documentation, debugging, and maintenance.
 * Ideally, docker builds without copying files around (the new
   `--build-context` functionality made that possible).
 * Compatibility with monorepo build.

Expressing dependencies of the extra code in `ext` proved tricky to
do in a clean way. Yarn/npm fought me every step of the way - everything
related to optional dependencies was unsatisfactory in some respect.
Yarn2 is flexible but smells like it might be overreach. In the end,
organizing to install non-core dependencies one directory up from the
main build was a good simple trick that saved my bacon.

This diff gets us to the point of building `grist-ee` images conveniently,
but there isn't a public repo people can go look at to see its source. This
could be generated by taking `grist-core`, adding the `ext` directory
to it, and pushing to a distinct repository. I'm not in a hurry to do that,
since a PR to that repo would be hard to sync with our monorepo and
`grist-core`. Also, we don't have any licensing text ready for the `ext`
directory. So leaving that for future work.

Test Plan: manual

Reviewers: georgegevoian, alexmojaki

Reviewed By: georgegevoian, alexmojaki

Differential Revision: https://phab.getgrist.com/D3415
2022-05-12 12:39:52 -04:00
Paul Fitzpatrick
05c89fb65a
add connect-redis for redis support (#183)
Redis is an optional dependency of Grist. When available, it can
be used for session storage, for supporting a pool of workers, and
for managing webhook delivery. This commit adds connect-redis to
package.json to simplify enabling Redis for the end user.
2022-04-12 16:39:56 -04:00
Paul Fitzpatrick
8367219cf1 v0.7.8 2022-04-07 10:03:28 -04:00
Paul Fitzpatrick
cce7e658ed v0.7.7 2022-04-04 10:03:10 -04:00
George Gevoian
6305811ca6 (core) Add new Grist login page
Summary:
Adds a new Grist login page to the login app, and replaces the
server-side Cognito Google Sign-In flow with Google's own OAuth flow.

Test Plan: Browser and server tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3332
2022-04-01 15:24:19 -07:00
Paul Fitzpatrick
24522e61ff
remove stray redis dependency, and upgrade node in tests (#173)
* remove stray redis dependency in test
* tweak handling of database connection between tests
* upgrade node versions in tests, type guessing in node 10 has problems
2022-03-28 15:43:47 -04:00
Paul Fitzpatrick
de703343d0 (core) disentangle some server tests, release to core, add GRIST_PROXY_AUTH_HEADER test
Summary:
This shuffles some server tests to make them available in grist-core,
and adds a test for the `GRIST_PROXY_AUTH_HEADER` feature added in
https://github.com/gristlabs/grist-core/pull/165

It includes a fix for a header normalization issue for websocket connections.

Test Plan: added test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3326
2022-03-24 15:11:32 -04:00
Paul Fitzpatrick
b2715ae9ef (core) forbid use of sqlite ATTACH except during VACUUM
Summary:
This calls sqlite3_limit(SQLITE_LIMIT_ATTACHED, 0) so that
if ever an `ATTACH` were snuck into an SQL query, it would be denied.
The limit needs to be waived when calling VACUUM since the implementation
of VACUUM uses ATTACH.

Test Plan: added test; existing tests should pass

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3316
2022-03-14 09:34:44 -04:00
Paul Fitzpatrick
00eb44fa00 v0.7.6 2022-03-09 10:11:30 -05:00
Paul Fitzpatrick
99ee495b4b v0.7.5 2022-02-28 16:52:44 -05:00
Alex Hall
f1002c0e67 (core) Regularly log data size in DocStorage.applyStoredActions using sqlite dbstat
Summary:
- Small cleanup: Make DocStorage implement OnDemandStorage, and remove unused execWithBackup
- Upgrade to new versions (.3) of @gristlabs/sqlite3 and connect-sqlite3 to use dbstat
- Add _logDataSize method which queries dbstat, adding up pgsize for tables loaded into the data engine
- Only complete _logDataSize every 5 minutes using new field _lastLoggedDataSize

Test Plan: Tested manually

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3277
2022-02-22 00:59:04 +02:00
Alex Hall
5b352211c4 (core) Guess date format during type conversion
Summary:
- Adds a dependency moment-guess (https://github.com/apoorv-mishra/moment-guess) to guess date formats from strings. However the npm package is missing source maps which leads to an ugly warning, so currently using a fork until https://github.com/apoorv-mishra/moment-guess/pull/22 is resolved.
- Adds guessDateFormat using moment-guess to determine the best candidate date format. The logic may be refined for e.g. lossless imports where the stakes are higher, but for now we're just trying to make type conversions smoother.
- Uses guessDateFormat to guess widget options when changing column type to date or datetime.
- Uses the date format of the original column when possible instead of guessing.
- Fixes a bug where choices were guessed based on the display column instead of the visible column, which made the guessed choices influenced by which values were referenced as well as completely broken when converting from reflist.
- @dsagal @georgegevoian This builds on https://phab.getgrist.com/D3265, currently unmerged. That diff was created first to alert to the change. Without it there would still be similar test failures/changes here as the date format would often be concretely guessed and saved as YYYY-MM-DD instead of being left as the default `undefined` which is shows as YYYY-MM-DD in the dropdown.

Test Plan: Added a unit test to `parseDate.ts`. Updated several browser tests which show the guessing in action during type conversion quite nicely.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal, georgegevoian

Differential Revision: https://phab.getgrist.com/D3264
2022-02-21 22:39:47 +02:00
George Gevoian
ba6ecc5e9e (core) Move user profile to new page and begin MFA work
Summary:
The user profile dialog is now a separate page, in preparation
for upcoming work to enable MFA. This commit also contains
some MFA changes, but the UI is currently disabled and the
implementation is limited to software tokens (TOTP) only.

Test Plan:
Updated browser tests for new profile page. Tests for MFAConfig
and CognitoClient will be added in a later diff, once the UI is enabled.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3199
2022-01-13 21:21:49 -08:00
Paul Fitzpatrick
5cdc7b2ea4 (core) freshen core README; support python3 in grist-core docker image
Summary:
This updates the grist-core README to list specific features of Grist,
to make it easier for a casual visitor to get a sense of its scope. Adds links
to some new resources (reviews, templates, grist v airtable post) that could
also help. Adds python3 to docker image so that templates work without fuss.

Test Plan: existing tests should pass

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal, anaisconce

Differential Revision: https://phab.getgrist.com/D3204
2022-01-08 18:27:20 -05:00
Paul Fitzpatrick
975eed8564 (core) fix browser check and favicon in grist-core
Summary:
A check for old browsers and a Grist favicon were not available in
grist-core, leaving harmless but distracting errors in logs. This
adds them.

Test Plan: checked manually

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3207
2022-01-05 18:15:05 -05:00
Paul Fitzpatrick
d99db8d016 (core) move more tests to grist-core
Summary:
 * Tie build and run-time docker base images to a consistent version (buster)
 * Extend the test login system activated by GRIST_TEST_LOGIN to ease porting tests that currently rely on cognito (many)
 * Make org resets work in absence of billing endpoints
 * When in-memory session caches are used, add missing invalidation steps
 * Pass org information through sign-ups/sign-ins more carefully
 * For CORS, explicitly trust GRIST_HOST origin when set
 * Move some fixtures and tests to core, focussing on tests that cover existing failures or are in the set of tests run on deployments
 * Retain regular `test` target to run the test suite directly, without docker
 * Add a `test:smoke` target to run a single simple test without `GRIST_TEST_LOGIN` activated
 * Add a `test:docker` target to run the tests against a grist-core docker image - since tests rely on certain fixture teams/docs, added `TEST_SUPPORT_API_KEY` and `TEST_ADD_SAMPLES` flags to ease porting

The tests ported were `nbrowser` tests: `ActionLog.ts` (the first test I tend to port to anything, out of habit), `Fork.ts` (exercises a lot of doc creation paths), `HomeIntro.ts` (a lot of DocMenu exercise), and `DuplicateDocument.ts` (covers a feature known to be failing prior to this diff, the CORS tweak resolves it).

Test Plan: Manually tested via `buildtools/build_core.sh`. In follow up, I want to add running the `test:docker` target in grist-core's workflows. In jenkins, only the smoke test is run. There'd be an argument for running all tests, but they include particularly slow tests, and are duplicates of tests already run (in different configuration admittedly), so I'd like to try first just using them in grist-core to gate updates to any packaged version of Grist (the docker image currently).

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3176
2021-12-10 18:33:07 -05:00
Paul Fitzpatrick
6e3ec298f2 (core) regularly freshen grist-core:latest docker image
Summary:
 * Keeps `grist-core:latest` docker image up to date with grist-core
   source, with a daily cron job.
 * Update yarn.lock file.
 * In passing, tag current version for a versioned release.

When a push to `main` branch passes tests, the result is placed in `latest_candidate` branch. The cron job will update docker up from `latest_candidate`, placing code used for image in `latest` branch.

Test Plan: tested in a grist-core fork

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3156
2021-11-29 14:06:05 -05:00
Jarosław Sadziński
3e661db38c (core) Adding schema validation for records endpoint
Summary:
Adding validation for api /records endpoint, that checks if the json payload is valid.
Modifying POST /records endpoint to allow creating blank or partial records.

Test Plan: Updated tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3061
2021-10-18 21:40:50 +02:00
Paul Fitzpatrick
b3b7410ede (core) open documents without blocking on data engine
Summary:
With this diff, when a user opens a Grist document in a browser, they will be able to view its contents without waiting for the data engine to start up. Once the data engine starts, it will run a calculation and send any updates made. Changes to the document will be blocked until the engine is started and the initial calculation is complete.

The increase in responsiveness is useful in its own right, and also reduces the impact of an extra startup time in a candidate next-generation sandbox.

A small unrelated fix is included for `core/package.json`, to catch up with a recent change to `package.json`.

A small `./build schema` convenience is added to just rebuild the typescript schema file.

Test Plan: added test; existing tests pass - small fixes needed in some cases because of new timing

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3036
2021-10-01 10:18:56 -04:00
George Gevoian
90d1340b64 (core) Add new packages to grist-core
Summary: Adds a few new NPM packages to grist-core.

Test Plan: Manually verified grist-core builds and runs.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3000
2021-08-27 10:20:10 -07:00
Paul Fitzpatrick
c561dad22d (core) lightly freshen the core readme, mentioning roadmap and forums etc.
Summary: This is a documentation update, and version bump on grist-core.

Test Plan: No code changes.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2982
2021-08-17 23:51:58 -04:00
Paul Fitzpatrick
54beaede84 (core) revive saml support and test against Auth0
Summary:
SAML support had broken due to SameSite changes in browsers. This
makes it work again, and tests it against Auth0 (now owned by Okta).

Logging in and out works.  The logged out state is confusing, and may
not be complete.  The "Add Account" menu item doesn't work.
But with this, an important part of self-hosting becomes easier.

SAML support works also in grist-core, for site pages, but there
is a glitch on document pages that I'll look into separately.

Test Plan: tested manually

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2976
2021-08-16 17:36:09 -04:00
Paul Fitzpatrick
bb8cb2593d (core) support python3 in grist-core, and running engine via docker and/or gvisor
Summary:
 * Moves essential plugins to grist-core, so that basic imports (e.g. csv) work.
 * Adds support for a `GRIST_SANDBOX_FLAVOR` flag that can systematically override how the data engine is run.
   - `GRIST_SANDBOX_FLAVOR=pynbox` is "classic" nacl-based sandbox.
   - `GRIST_SANDBOX_FLAVOR=docker` runs engines in individual docker containers. It requires an image specified in `sandbox/docker` (alternative images can be named with `GRIST_SANDBOX` flag - need to contain python and engine requirements). It is a simple reference implementation for sandboxing.
   - `GRIST_SANDBOX_FLAVOR=unsandboxed` runs whatever local version of python is specified by a `GRIST_SANDBOX` flag directly, with no sandboxing. Engine requirements must be installed, so an absolute path to a python executable in a virtualenv is easiest to manage.
   - `GRIST_SANDBOX_FLAVOR=gvisor` runs the data engine via gvisor's runsc. Experimental, with implementation not included in grist-core. Since gvisor runs on Linux only, this flavor supports wrapping the sandboxes in a single shared docker container.
 * Tweaks some recent express query parameter code to work in grist-core, which has a slightly different version of express (smoke test doesn't catch this since in Jenkins core is built within a workspace that has node_modules, and wires get crossed - in a dev environment the problem on master can be seen by doing `buildtools/build_core.sh /tmp/any_path_outside_grist`).

The new sandbox options do not have tests yet, nor does this they change the behavior of grist servers today. They are there to clean up and consolidate a collection of patches I've been using that were getting cumbersome, and make it easier to run experiments.

I haven't looked closely at imports beyond core.

Test Plan: tested manually against regular grist and grist-core, including imports

Reviewers: alexmojaki, dsagal

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D2942
2021-07-28 09:02:32 -04:00
Jarosław Sadziński
08295a696b (core) Export to Excel and Send to drive
Summary:
Implementing export to excel and send to Google Drive feature.

As part of this feature few things were implemented:
- Server side google authentication exposed on url: (docs, docs-s, or localhost:8080)/auth/google
- Exporting grist documents as an excel file (xlsx)
- Storing exported grist document (in excel format) in Google Drive as a spreadsheet document.

Server side google authentication requires one new environmental variables
- GOOGLE_CLIENT_SECRET (required) used by authentication handler

Test Plan: Browser tests for exporting to excel.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2924
2021-07-21 16:36:00 +02:00
Paul Fitzpatrick
1b760d83a9 (core) add missing tmp-promise dev package from core
Summary: In porting a js file to typescript, a dependency on tmp-promise was added - but core doesn't have that. This adds it, and also takes the opportunity to bump the grist-core version number.

Test Plan: manual (smoke test doesn't catch this)

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D2922
2021-07-19 11:45:30 -04:00
Dmitry S
869b2f00ec (core) Remove LoginSession, which was mainly serving situations that are no longer used.
Summary:
In the past, Cognito sign-ins were intended to give authorization to some AWS
services (like SQS); various tokens were stored in the session for this
purpose. This is no longer used. Profiles from Cognito now serve a limited
purpose: first-time initialization of name and picture, and keeping track of
which login method was used. For these remaining needs, ScopedSession is
sufficient.

Test Plan:
Existing test pass. Tested manually that logins work with Google and
Email + Password. Tested manually that on a clean database, name and picture
are picked up from a Google Login.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2907
2021-07-12 13:04:00 -04:00
Paul Fitzpatrick
6d2e8378cd (core) fix some tests for node v14
Test Plan: existing tests pass

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2814
2021-05-12 22:49:53 -04:00
Paul Fitzpatrick
37521a3612 (core) switch grist-core docker image to use node v14
Summary: As practice for upgrading node in regular grist images, this upgrades the grist-core image to v14.

Test Plan: manual

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2807
2021-05-10 15:23:45 -04:00
Dmitry S
fd73831b39 (core) Move report-why-tests-hang helper to core
Summary:
This helps forcibly end mocha tests when they hang, and print out
something that may help debug the situation.

Also add the generated static/bundle.css file to core/.gitignore.

Also, avoid using npm-packages-offline-cache when building core, by avoiding use of .yarnrc which turns it on.

Test Plan: Tested manually

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2788
2021-04-26 23:52:16 -04:00
Paul Fitzpatrick
9f234b758d (core) freshen grist-core build
Summary:
 * adds a smoke test to grist-core
 * fixes a problem with highlight.js failing to load correctly
 * skips survey for default user
 * freshens docker build

Utility files in test/nbrowser are moved to core/test/nbrowser, so that gristUtils are available there. This increased the apparent size of the diff as "./" import paths needed replacing with "test/nbrowser/" paths. The utility files are untouched, except for the code to start a server - it now has a small grist-core specific conditional in it.

Test Plan: adds test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2768
2021-04-03 09:41:06 -04:00
Paul Fitzpatrick
d8df2404c2 (core) return to using meaningful SQL types for columns
Summary:
Previously in {{D1053}} we switched to using BLOB as the "type" for all columns, to prevent SQLite from casting data unexpectedly.  This diff now returns to more meaningful types.  We apply marshalling to values when being placed in a column where a cast might occur, to inhibit such casting.

The benefit is that Grist documents become easier to interact with via regular database clients/libraries, which often rely on the column type more than a purely SQLite tool would.

On column type conversion, we run all blobs in the column through a decode/encode cycle so if they no longer need to be marshalled they revert to native type.  This could be optimized further, it is somewhat brute force.

Test Plan: Updated tests and reference document

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2755
2021-03-25 10:26:39 -04:00
Paul Fitzpatrick
71519d9e5c (core) revamp snapshot inventory
Summary:
Deliberate changes:
 * save snapshots to s3 prior to migrations.
 * label migration snapshots in s3 metadata.
 * avoid pruning migration snapshots for a month.

Opportunistic changes:
 * Associate document timezone with snapshots, so pruning can respect timezones.
 * Associate actionHash/Num with snapshots.
 * Record time of last change in snapshots (rather than just s3 upload time, which could be a while later).

This ended up being a biggish change, because there was nowhere ideal to put tags (list of possibilities in diff).

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2646
2020-10-30 13:52:46 -04:00
Paul Fitzpatrick
9287439e5a (core) give instructions on using Grist with docker
Summary: cleans up docker build and instructions.

Test Plan: docker image and instructions tested manually

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2648
2020-10-28 13:59:13 -04:00
Dmitry S
d7802bc7db (core) Support international addresses in the Billing form
Summary:
- When displaying, include the country code, and don't assume state is always present.
- When entering, include a country selector (defaulting to US), and
  make state/zip optional when non-US.
- Bring in an npm module with country codes.

Test Plan: Added a browser test case.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2647
2020-10-26 11:41:15 -04:00
Paul Fitzpatrick
4d3777578e (core) add Dockerfile for grist-core
Summary:
This adds a two-stage Dockerfile for grist-core. The first stage builds
Grist, and the second collects all files needed to run Grist.

The resulting image is about 600 MB which is quite a bit bigger
than it needs to be, but seems fine for now when the first goal is
to establish that people can open and edit Grist files on their
own infrastructure.

The image uses stock python rather than our sandboxed python for now.

Test Plan: manual

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2637
2020-10-12 15:45:22 -04:00
Paul Fitzpatrick
1654a2681f (core) move client code to core
Summary:
This moves all client code to core, and makes minimal fix-ups to
get grist and grist-core to compile correctly.  The client works
in core, but I'm leaving clean-up around the build and bundles to
follow-up.

Test Plan: existing tests pass; server-dev bundle looks sane

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2627
2020-10-02 13:24:21 -04:00
Paul Fitzpatrick
b82eec714a (core) move data engine code to core
Summary:
this moves sandbox/grist to core, and adds a requirements.txt
file for reconstructing the content of sandbox/thirdparty.

Test Plan:
existing tests pass.
Tested core functionality manually.  Tested docker build manually.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2563
2020-07-29 08:57:25 -04:00
Paul Fitzpatrick
5ef889addd (core) move home server into core
Summary: This moves enough server material into core to run a home server.  The data engine is not yet incorporated (though in manual testing it works when ported).

Test Plan: existing tests pass

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2552
2020-07-21 20:39:10 -04:00
Dmitry S
ad35f54b87 Update tsconfig files and switch to _build for outputs, for consistency with main grist repo 2020-05-22 02:14:28 -04:00
Dmitry S
ec182792be Initial config with a few files that build on client and server side. 2020-05-20 00:50:46 -04:00