Commit Graph

170 Commits

Author SHA1 Message Date
Dmitry S
f2f4fe0eca (core) Add LogMethods helper and use it for more JSON data in logs. Reduce unhelpful logging.
Summary:
- Sharing, Client, DocClients, HostingStorageManager all include available info.
- In HostingStorageManager, log numSteps and maxStepTimeMs, in case that helps
  debug SQLITE_BUSY problem.
- Replace some action-bundle logging with a JSON version aggregating some info.
- Skip logging detailed list of actions in production.

Test Plan: Tested manually by eyeballing log output in dev environment.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3086
2021-10-25 10:25:18 -04:00
Alex Hall
e900f39da3 (core) Log statistics about table sizes
Summary:
Record numbers of rows, columns, cells, and bytes of marshalled data for most calls to table_data_from_db

Export new function get_table_stats in the sandbox, which gives the raw numbers and totals.

Get and log these stats in ActiveDoc right after loading tables, before Calculate, so they are logged even in case of errors.

Tweak logging about number of tables, especially number of on-demand tables, to not only show in debug logging.

Test Plan: Updated doc regression tests, that shows what the data looks like nicely.

Reviewers: dsagal, paulfitz

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3081
2021-10-21 17:54:20 +02:00
Paul Fitzpatrick
dd0f1be117 (core) get all tests working under python3/gvisor
Summary:
This verifies that all existing tests are capable of running under python3/gvisor, and fixes the small issues that came up. It does not yet activate python3 tests on all diffs, only diffs that specifically request them.

 * Adds a suffix in test names and output directories for tests run with PYTHON_VERSION=3, so that results of the same test run with and without the flag can be aggregated cleanly.
 * Adds support for checkpointing to the gvisor sandbox adapter.
 * Prepares a checkpoint made after grist python code has loaded in the gvisor sandbox.
 * Changes how `DOC_URL` is passed to the sandbox, since it can no longer be passed in as an environment variable when using checkpoints.
 * Uses the checkpoint to speed up tests using the gvisor sandbox, otherwise a lot of tests need more time (especially on mac under docker).
 * Directs jenkins to run all tests with python2 and python3 when a new file `buildtools/changelogs/python.txt` is touched (this diff counts as touching that file).
 * Tweaks miscellaneous tests
   - some needed fixes exposed by slightly different timing
   - a small number actually give different results in py3 (removal of `u` prefixes).
   - some needed a little more time

The DOC_URL change is not the ultimate solution we want for DOC_URL. Eventually it should be a variable that gets updated, like the date perhaps. This is just a small pragmatic change to preserve existing behavior.

Tests are run mindlessly as py3, and for some tests it won't change anything (e.g. if they do not use NSandbox). Tests are not run in parallel, doubling overall test time.

Checkpoints could be useful in deployment, though this diff doesn't use them there.

The application of checkpoints doesn't check for other configuration like 3-versus-5-pipe that we don't actually use.

Python2 tests run using pynbox as always for now.

The diff got sufficiently bulky that I didn't tackle running py3 on "regular" diffs in it. My preference, given that most tests don't appear to stress the python side of things, would be to make a selection of the tests that do and a few wild cards, and run those tests on both pythons rather then all of them. For diffs making a significant python change, I'd propose touching buildtools/changelogs/python.txt for full tests. But this is a conversation in progress.

A total of 6886 tests ran on this diff.

Test Plan: this is a step in preparing tests for py3 transition

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3066
2021-10-18 17:44:15 -04:00
Jarosław Sadziński
3e661db38c (core) Adding schema validation for records endpoint
Summary:
Adding validation for api /records endpoint, that checks if the json payload is valid.
Modifying POST /records endpoint to allow creating blank or partial records.

Test Plan: Updated tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3061
2021-10-18 21:40:50 +02:00
Alex Hall
276adc5f51 (core) Starting to make webhooks more robust
Summary:
- Puts events on a queue in memory and ensures they are sent in the order they were generated.
- Makes the caller (Sharing.ts) wait until changed records have been fetched from the DB, but allows it to continue after while remaining work happens asynchronously.
- Gathers all new webhook events into an array so they can be backed up to the queue on redis in a single command (in a future diff).
- Uses changes in isReady to determine event type, no more 'existed before'

The structure of the code has changed a lot, so I think the scope of the diff needs to stop here. Lots of work is still deferred in TODOs.

Test Plan: Updated existing test. Actually dropped testing of retry on failures and slowness because it no longer made sense to keep that as part of the current test, so a new test for that will be added in a future diff.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3074
2021-10-15 20:01:15 +02:00
Alex Hall
9d1cc89dc9 (core) Strip invalid characters from table name in excel import
Summary: Add sanitizeWorksheetName function, pass result to library function addWorksheet where error was raised.

Test Plan: Added unit test for sanitizeWorksheetName function, updated a fixture document to use a messy table name.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3072
2021-10-11 17:47:12 +02:00
Alex Hall
a64fb105e3 (core) Use GristObjCode in CellValue
Summary: Makes type checking a bit stronger

Test Plan: it just has to compile

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3065
2021-10-11 14:11:32 +02:00
George Gevoian
62db263d1f (core) Add diff preview to Importer
Summary:
Updates the preview table in Importer to show a diff of changes
when importing into an existing table and updating existing records.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3060
2021-10-08 14:15:07 -07:00
Paul Fitzpatrick
d635c97686 (core) flesh out "View As" feature
Summary:
The users shown by the "View As" button are now drawn from more sources:
 * There are users the document is shared with. This has been rationalized, the behavior was somewhat erratic. If the user is not an owner of the document, the only user of this kind that will be listed is themselves.
 * There are users mentioned in any user attribute table keyed by Email. If name and access columns are present, those are respected, otherwise name is taken from email and access is set to "editors".
 * There are example users provided if there are not many other users available.

Test Plan: added and extended tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3045
2021-10-08 12:00:40 -04:00
Paul Fitzpatrick
07558dceba (core) avoid censorship for one client clobbering data for another client
Summary:
When filtering document updates to send to clients after a change,
censorship of individual cells was being applied to state shared
across the clients. This diff eliminates that shared state, and
extends testing of broadcasts to check different orderings.

Test Plan:
extends a test to tickle a reported bug, and gives
DocClients a knob to control message order needed to tickle
the bug reliably.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3064
2021-10-07 23:21:07 -04:00
Paul Fitzpatrick
df318ad6b3 (core) add a mac-specific sandbox for development
Summary:
docker is slow on macs, so use native sandbox-exec by
default for tests involving python3 on macs.

Test Plan: updated test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3068
2021-10-07 14:14:25 -04:00
George Gevoian
e1780e4f58 (core) Migrate import code from data engine to Node
Summary:
Finishing imports now occurs in Node instead of the
data engine, which makes it possible to import into
on-demand tables. Merging code was also refactored
and now uses a SQL query to diff source and destination
tables in order to determine what to update or add.

Also fixes a bug where incremental imports involving
Excel files with multiple sheets would fail due to the UI
not serializing merge options correctly.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3046
2021-10-04 10:27:00 -07:00
Dmitry S
7e07f0ce56 (core) For grist_sid*_status cookie, remember to set the path
Test Plan: Only tested manually that path is included.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3056
2021-10-04 09:13:47 -04:00
Paul Fitzpatrick
8853e095bb (core) fix core build, and make smoke test more effective
Summary:
This makes the `core` test operate on a directory outside the
jenkins workspace, so that packages in the workspace don't
interfere with the test and obscure errors.

It also includes a small type fix for the `core` build.

Test Plan: updating a test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3054
2021-10-01 13:34:47 -04:00
Dmitry S
1517dca644 (core) Implement DiscourseConnect to enable easy sign-in to community forum
Summary:
- Update cookie module, to support modern sameSite settings
- Add a new cookie, grist_sid_status with less-sensitive value, to let less-trusted subdomains know if user is signed in
- The new cookie is kept in-sync with the session cookie.
- For a user signed in once, allow auto-signin is appropriate.
- For a user signed in with multiple accounts, show a page to select which account to use.
- Move css stylings for rendering users to a separate module.

Test Plan: Added a test case with a simulated Discourse page to test redirects and account-selection page.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3047
2021-10-01 11:24:22 -04:00
Paul Fitzpatrick
b3b7410ede (core) open documents without blocking on data engine
Summary:
With this diff, when a user opens a Grist document in a browser, they will be able to view its contents without waiting for the data engine to start up. Once the data engine starts, it will run a calculation and send any updates made. Changes to the document will be blocked until the engine is started and the initial calculation is complete.

The increase in responsiveness is useful in its own right, and also reduces the impact of an extra startup time in a candidate next-generation sandbox.

A small unrelated fix is included for `core/package.json`, to catch up with a recent change to `package.json`.

A small `./build schema` convenience is added to just rebuild the typescript schema file.

Test Plan: added test; existing tests pass - small fixes needed in some cases because of new timing

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3036
2021-10-01 10:18:56 -04:00
Jarosław Sadziński
42910cb8f7 (core) Extending Google Drive integration scope
Summary:
New environmental variable GOOGLE_DRIVE_SCOPE that modifies the scope
requested for Google Drive integration.
For prod it has value https://www.googleapis.com/auth/drive.file which leaves
current behavior (Grist is allowed only to access public files and for private
files - it fallbacks to Picker).
For staging it has value https://www.googleapis.com/auth/drive.readonly which
allows Grist to access all private files, and fallbacks to Picker only when the file is
neither public nor private).
Default value is https://www.googleapis.com/auth/drive.file

Test Plan: manual and existing tests

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3038
2021-10-01 10:47:12 +02:00
Paul Fitzpatrick
383b8ffbf0 (core) add a tool for deleting a user
Summary:
This adds a `user:delete` target to the `cli.sh` tool. The desired user will be deleted from our database, from sendgrid, and from cognito.

There is code for scrubbing the user from team sites, but it isn't yet activated, I'm leaving finalizing and writing tests for it for follow-up.

Test Plan: tested manually

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3043
2021-09-29 12:08:23 -04:00
Paul Fitzpatrick
876a0298a2 (core) do not look at content of recent actions when loading documents
Summary:
This removes the need for any information drawn from the content of recent actions when loading a document.

The undo/redo system does need some facts about recent actions up front. But that system has an important restriction: only actions a particular client is known to have generated can be undone by that client.

So in this diff, as we store which client has performed an action, we also store the few pieces of metadata about that action that the undo/redo system needs: `linkId`, `otherId`, `rowIdHint`, `isUndo` fields. These are all small integers (or in one case a boolean).

An existing limitation is that information about which client has performed which action is stored in memory in the worker, and not persisted anywhere. This diff does not change that limitation, meaning that undos continue to not survive a worker transition. A reasonable way to deal with that would be to back the store with redis.

Test Plan: existing tests pass

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3044
2021-09-29 11:27:02 -04:00
Paul Fitzpatrick
0fffe918c1 (core) don't garble document url in SELF_HYPERLINK on forks
Summary: There was a bad regex processing the document url passed to the sandbox.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3048
2021-09-28 16:53:48 -04:00
Alex Hall
8c1f8bc9a6 (core) Test webhooks
Summary:
Tests DocApi endpoints _subscribe and _unsubscribe, including various bad inputs.

Tests that webhooks are sent to a test express server, with retrying on failure, filtered by event type, and waiting for isReadyColumn.

Test Plan: this

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3042
2021-09-28 01:13:54 +02:00
Dmitry S
fb583f303a (core) Support 'new' row in anchor links.
Summary:
- Anchor links with row of 'new' could be created but weren't parsed or used
  correctly. This fixes it.
- Also adds UIRowId type for row IDs which includes the special 'new' row. It's
  already been used in places as `number|'new'`, this diff gives it a name usable in app/common
  (it doesn't touch another name, RowId, that's been available in app/client).

Test Plan: Added a test assert for anchor links to new row

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3039
2021-09-24 09:01:10 -04:00
Alex Hall
3c4d71aeca (core) Initial webhooks implementation
Summary:
See https://grist.quip.com/VKd3ASF99ezD/Outgoing-Webhooks

- 2 new DocApi endpoints: _subscribe and _unsubscribe, not meant to be user friendly or publicly documented. _unsubscribe should be given the response from _subscribe in the body, e.g:

```
$ curl -X POST -H "Authorization: Bearer 8fd4dc59ecb05ab29ae5a183c03101319b8e6ca9" "http://localhost:8080/api/docs/6WYa23FqWxGNe3AR6DLjCJ/tables/Table2/_subscribe" -H "Content-type: application/json" -d '{"url": "https://webhook.site/a916b526-8afc-46e6-aa8f-a625d0d83ec3", "eventTypes": ["add"], "isReadyColumn": "C"}'
{"unsubscribeKey":"3246f158-55b5-4fc7-baa5-093b75ffa86c","triggerId":2,"webhookId":"853b4bfa-9d39-4639-aa33-7d45354903c0"}
$ curl -X POST -H "Authorization: Bearer 8fd4dc59ecb05ab29ae5a183c03101319b8e6ca9" "http://localhost:8080/api/docs/6WYa23FqWxGNe3AR6DLjCJ/tables/Table2/_unsubscribe" -H "Content-type: application/json" -d '{"unsubscribeKey":"3246f158-55b5-4fc7-baa5-093b75ffa86c","triggerId":2,"webhookId":"853b4bfa-9d39-4639-aa33-7d45354903c0"}'
{"success":true}
```

- New DB entity Secret to hold the webhook URL and unsubscribe key
- New document metatable _grist_Triggers subscribes to table changes and points to a secret to use for a webhook
- New file Triggers.ts processes action summaries and uses the two new tables to send webhooks.
- Also went on a bit of a diversion and made a typesafe subclass of TableData for metatables.

I think this is essentially good enough for a first diff, to keep the diffs manageable and to talk about the overall structure. Future diffs can add tests and more robustness using redis etc. After this diff I can also start building the Zapier integration privately.

Test Plan: Tested manually: see curl commands in summary for an example. Payloads can be seen in https://webhook.site/#!/a916b526-8afc-46e6-aa8f-a625d0d83ec3/0b9fe335-33f7-49fe-b90b-2db5ba53382d/1 . Great site for testing webhooks btw.

Reviewers: dsagal, paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3019
2021-09-23 14:35:39 +02:00
Paul Fitzpatrick
d5a7fb23fe (core) apply SchemaEdit flag to metadata changes in general
Summary:
A user without SchemaEdit permission was able to reorder pages, since
this changes _grist_Pages, and that table was left under control of
regular access rules.  This diff tightens things up, to require
SchemaEdit for all metadata edits.  The one remaining exception is
_grist_Attachments, which needs some reworking to play well with
granular access.

Test Plan: extended test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3025
2021-09-16 13:36:20 -04:00
George Gevoian
8a7edb6257 (core) Enable incremental imports
Summary:
The import dialog now has an option to 'Update existing records',
which when checked will allow for selection of 1 or more fields
to match source and destination tables on.

If all fields match, then the matched record in the
destination table will be merged with the incoming record
from the source table. This means the incoming values will
replace the destination table values, unless the incoming
values are blank.

Additional merge strategies are implemented in the data
engine, but the import dialog only uses one of the
strategies currently. The others can be exposed in the UI
in the future, and tweak the behavior of how source
and destination values should be merged in different contexts,
such as when blank values exist.

Test Plan: Python and browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3020
2021-09-16 09:15:54 -07:00
Paul Fitzpatrick
a543e5194a (core) add a python3 button
Summary: This adds a dropdown to the document settings model in staging/dev to set the python engine to Python2 or Python3. The setting is saved in `_grist_DocInfo.documentSettings.engine`.

Test Plan: tested manually for now - separate diff needed to add runsc to jenkins setup and make this testable

Reviewers: dsagal, alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3014
2021-09-16 10:06:04 -04:00
Paul Fitzpatrick
3e5a292cde (core) add tests for site deletion
Summary: This tests site deletion with and without a plan.

Test Plan: adding tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3017
2021-09-14 10:03:18 -04:00
Paul Fitzpatrick
ddcd08e147 (core) add a cli tool for deleting sites
Summary:
This adds a `site:delete` target to `cli.sh` for deleting sites. Sites should be specified by numeric org id, and for confirmation their name also needs to be given.

All the docs in the site are deleted permanently, and the workspaces, and the site, and the stripe customer (if any).

Test Plan: manual

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3015
2021-09-10 11:02:26 -04:00
George Gevoian
0717ee627e (core) Relocate export urls to /download/
Summary:
Moves CSV and XLSX export urls under /download/, and
removes the document title query parameter which is now
retrieved from the backend.

Test Plan: No new tests. Existing tests that verify endpoints still function.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3010
2021-09-02 09:36:33 -07:00
George Gevoian
ef5da42378 (core) Update export CSV and Excel endpoints
Summary:
The endpoints for exporting CSV and Excel are now under
/api/docs/:docId/ and are forwarded to a doc worker for export.

The Share Menu has been updated to use the new endpoints.

Test Plan: No new tests. Existing tests that verify endpoints work correctly.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3007
2021-08-31 10:47:24 -07:00
George Gevoian
a6e08883e0 (core) Simple localization support and currency selector.
Summary:
- Grist document has a associated "locale" setting that affects how currency is formatted.
- Currency selector for number format.

Test Plan: not done

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D2977
2021-08-26 13:36:49 -07:00
Paul Fitzpatrick
e492dfdb22 (core) add experimental support for python3 in staging
Summary:
This adds `runsc` and `python3` to the grist-server images. For deployments with GRIST_EXPERIMENTAL_PLUGINS=1 (dev + staging but not prod) a hack is added to use `python3` under `runsc` for documents with a special title (`activate-python3-magic` or similar).

This will simplify experiments on behavior of this configuration under realistic conditions.

Hopefully, before landing this, I'll be able to switch to storing a python flag in a document options cell being added by @georgegevoian in a parallel diff, since using the doc title is super hacky :-).

Test Plan: tested manually on worker built locally

Reviewers: dsagal, alexmojaki

Reviewed By: dsagal, alexmojaki

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D2998
2021-08-26 09:39:26 -04:00
Paul Fitzpatrick
b3636b97e2 (core) Report memos consistently for blocked actions involving schema
Summary:
Currently actions blocked early because they could modify the
schema (e.g. changing formulas) do not report memo information
(comments in relevant rules).  This diff fixes that by using
more of the same code path in the two situations.  It also
adds information about what type of action was blocked to
error messages.

Test Plan: extended a test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2995
2021-08-24 08:03:58 -04:00
Paul Fitzpatrick
f53ab2cb30 (core) forbids edits when "view as" user is a viewer and access rules are permissive
Summary:
Currently, if access rules are set to allow edits unconditionally,
and an owner does "View As" a user who is a viewer only, they will
be allowed to make edits.  This catches that condition and adds a
test.

Test Plan: added test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D2991
2021-08-20 16:04:28 -04:00
Dmitry S
00c1a0c688 (core) Log the time taken by decodeActionFromRow() operations.
Summary:
Decoding large actions is a plausible culprit for hogging CPU time for
certain documents. To begin with, log the time taken for this operation,
so that we can tell if it's a problem in practice.

Test Plan: Should not affect any current behaviors

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2989
2021-08-20 11:28:33 -04:00
Paul Fitzpatrick
c561dad22d (core) lightly freshen the core readme, mentioning roadmap and forums etc.
Summary: This is a documentation update, and version bump on grist-core.

Test Plan: No code changes.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2982
2021-08-17 23:51:58 -04:00
Paul Fitzpatrick
f9630b3aa4 (core) clean up a collection of small problems affecting grist-core
Summary:
 * Remove adjustSession hack, interfering with loading docs under saml.
 * Allow the anonymous user to receive an empty list of workspaces for
   the merged org.
 * Behave better on first page load when org is in path - this used to
   fail because of lack of cookie.  This is very visible in grist-core,
   as a failure to load localhost:8484 on first visit.
 * Mark cookie explicitly as SameSite=Lax to remove a warning in firefox.
 * Make errorPages available in grist-core.

This changes the default behavior of grist-core to now start off in
anonymous mode, with an explicit sign-in step available.  If SAML is not configured,
the sign-in operation will unconditionally sign the user in as a default
user, without any password check or other security.  The user email is
taken from GRIST_DEFAULT_EMAIL if set.  This is a significant change, but
makes anonymous mode available in grist-core (which is convenient
for testing) and makes behavior with and without SAML much more consistent.

Test Plan: updated test; manual (time to start adding grist-core tests though!)

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2980
2021-08-17 21:44:50 -04:00
Alex Hall
e6e792655b (core) Add /columns endpoint to DocApi
Summary: Add /columns endpoint to DocApi

Test Plan: Added test

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2981
2021-08-17 23:20:52 +02:00
Paul Fitzpatrick
54beaede84 (core) revive saml support and test against Auth0
Summary:
SAML support had broken due to SameSite changes in browsers. This
makes it work again, and tests it against Auth0 (now owned by Okta).

Logging in and out works.  The logged out state is confusing, and may
not be complete.  The "Add Account" menu item doesn't work.
But with this, an important part of self-hosting becomes easier.

SAML support works also in grist-core, for site pages, but there
is a glitch on document pages that I'll look into separately.

Test Plan: tested manually

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2976
2021-08-16 17:36:09 -04:00
Alex Hall
34e9ad3498 (core) Add /records endpoint to DocApi with GET, POST, and PATCH
Summary:
Applies simple data transformations to the existing /data API.

Mimics the Airtable API. Designed in https://grist.quip.com/RZh9AEbPaj8x/Doc-API#FZfACAAZ9a0

Haven't done deletion because it seems like less of a priority and also not fully designed.

Test Plan: Added basic server tests similar to the /data tests. Haven't tested edge cases like bad input.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D2974
2021-08-12 18:02:56 +02:00
Alex Hall
7f1f8fc9e6 (core) Linking summary tables grouped by list columns
Summary:
Prefix keys of `LinkingState.filterColValues` with `_contains:` when the source column is a ChoiceList or ReferenceList.

This is parsed out to make a boolean `isContainsFilter` which is kept in each value of `QueryRefs.filterTuples` (previously `filterPairs`).

Then when converting back in `convertQueryFromRefs` we construct `Query.contains: {[colId: string]: boolean}`.

Finally `getFilterFunc` uses `Query.contains` to decide what kind of filtering to do.

This is not pretty, but the existing code is already very complex and it was hard to find something that wouldn't require touching loads of code just to make things compile.

Test Plan: Added a new nbrowser test and fixture, tests that selecting a source table by summary tables grouped by a choicelist column, non-list column, and both all filter the correct data.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2940
2021-08-10 20:41:24 +02:00
Alex Hall
4d526da58f (core) Move file import plugins into core/sandbox/grist
Summary:
Move all the plugins python code into the main folder with the core code.

Register file importing functions in the same main.py entrypoint as the data engine.

Remove options relating to different entrypoints and code directories. The only remaining plugin-specific option in NSandbox is the import directory/mount, i.e. where files to be parsed are placed.

Test Plan: this

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D2965
2021-08-09 18:37:14 +02:00
Paul Fitzpatrick
4a23b964ed (core) update read access for exceptional sessions
Summary:
Exceptional sessions had lost full read access to documents; this
restores it.  Exceptional sessions are used for system actions or
while creating documents.

Test Plan: added test

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D2966
2021-08-05 16:49:23 -04:00
Jarosław Sadziński
4ca47878ca (core) Adding import from google drive to the home screen
Summary: Importing from google drive from home screen (also for anonymous users)

Test Plan: Browser tests

Reviewers: dsagal, paulfitz

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2943
2021-08-05 20:46:11 +02:00
Jarosław Sadziński
6ed1d8dfea (core) Adding google drive plugin as a fallback for url plugin
Summary:
When importing from url, user types a url for google spreadsheet,
Grist will switch to Google Drive plugin to allow user to choose file manualy.

Test Plan: Browser tests

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2945
2021-08-04 17:59:13 +02:00
Alex Hall
5aed22dc1e (core) Remove dead code for fetching snapshots
Summary: Deletes code which was previously only used by SharedSharing.ts, which was deleted in D2894

Test Plan: no

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2960
2021-08-04 15:42:31 +02:00
Paul Fitzpatrick
750c78763e (core) add python2 to gvisor Dockerfile, for use in making comparisons
Summary:
This adds python2 to the gvisor sandbox image.  It can be used instead
of the default python3 by setting PYTHON_VERSION to 2 (or calling run.py with python2).
This is useful for making side-by-side comparisons with code running python3.

Test Plan: manual

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D2957
2021-08-03 17:51:31 -04:00
Jarosław Sadziński
521bbd9ac1 (core) Improving error messages on file imports
Summary: Improving error messages that get returned from "Import from URL" plugin.

Test Plan: browser tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: dsagal, alexmojaki

Differential Revision: https://phab.getgrist.com/D2946
2021-08-02 17:30:59 +02:00
Paul Fitzpatrick
6b3ac07ca7 (core) process GristDocAPI calls from custom widgets in the client
Summary:
Processing these calls in the client, rather than passing them on
to the backend, means that access rules are more straightforward to
apply.

An unrelated fix is included to filter _grist_ tables when fetched
individually - metadata could leak through this path.

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2954
2021-07-30 10:41:32 -04:00
Paul Fitzpatrick
bb8cb2593d (core) support python3 in grist-core, and running engine via docker and/or gvisor
Summary:
 * Moves essential plugins to grist-core, so that basic imports (e.g. csv) work.
 * Adds support for a `GRIST_SANDBOX_FLAVOR` flag that can systematically override how the data engine is run.
   - `GRIST_SANDBOX_FLAVOR=pynbox` is "classic" nacl-based sandbox.
   - `GRIST_SANDBOX_FLAVOR=docker` runs engines in individual docker containers. It requires an image specified in `sandbox/docker` (alternative images can be named with `GRIST_SANDBOX` flag - need to contain python and engine requirements). It is a simple reference implementation for sandboxing.
   - `GRIST_SANDBOX_FLAVOR=unsandboxed` runs whatever local version of python is specified by a `GRIST_SANDBOX` flag directly, with no sandboxing. Engine requirements must be installed, so an absolute path to a python executable in a virtualenv is easiest to manage.
   - `GRIST_SANDBOX_FLAVOR=gvisor` runs the data engine via gvisor's runsc. Experimental, with implementation not included in grist-core. Since gvisor runs on Linux only, this flavor supports wrapping the sandboxes in a single shared docker container.
 * Tweaks some recent express query parameter code to work in grist-core, which has a slightly different version of express (smoke test doesn't catch this since in Jenkins core is built within a workspace that has node_modules, and wires get crossed - in a dev environment the problem on master can be seen by doing `buildtools/build_core.sh /tmp/any_path_outside_grist`).

The new sandbox options do not have tests yet, nor does this they change the behavior of grist servers today. They are there to clean up and consolidate a collection of patches I've been using that were getting cumbersome, and make it easier to run experiments.

I haven't looked closely at imports beyond core.

Test Plan: tested manually against regular grist and grist-core, including imports

Reviewers: alexmojaki, dsagal

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D2942
2021-07-28 09:02:32 -04:00