Commit Graph

205 Commits

Author SHA1 Message Date
Jarosław Sadziński
b80e56a4e1 (core) Custom Widget column mapping feature.
Summary:
Exposing new API in CustomSectionAPI for column mapping.

The custom widget can call configure method (or use a ready method) with additional parameter "columns".
This parameter is a list of column names that should be mapped by the user.
Mapping configuration is exposed through an additional method in the CustomSectionAPI "mappings". It is also available
through the onRecord(s) event.

This DIFF is connected with PR for grist-widgets repository https://github.com/gristlabs/grist-widget/pull/15

Design document and discussion: https://grist.quip.com/Y2waA8h8Zuzu/Custom-Widget-field-mapping

Test Plan: browser tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3241
2022-02-08 17:41:04 +01:00
Alex Hall
5d671bf0b3 (core) New type conversion in the backend
Summary: This is https://phab.getgrist.com/D3205 plus some changes (https://github.com/dsagal/grist/compare/type-convert...type-convert-server?expand=1) that move the conversion process to the backend. A new user action ConvertFromColumn uses `call_external` so that the data engine can delegate back to ActiveDoc. Code for creating formatters and parsers is significantly refactored so that most of the logic is in `common` and can be used in different ways.

Test Plan: The original diff adds plenty of tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3240
2022-02-04 20:28:13 +02:00
Paul Fitzpatrick
d8aacbe3b4 (core) AddOrUpdateRecord user action
Summary:
New user action as described in https://grist.quip.com/fZSrAnJKgO5j/Add-or-Update-Records-API, with options to allow most of the mentioned possible behaviours.

The Python code is due to Alex (as should be obvious from the u in behaviours).

Test Plan: Added a unit test.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3239
2022-02-03 16:22:51 -05:00
Alex Hall
fa9e6eee88 (core) Create an extra raw data widget when creating a table
Summary: This is the first step towards raw data views, merely adding metadata without any UI. Every 'normal' table now has a widget referenced by `rawViewSectionRef`. It has no parent view/page and cannot actually be viewed for now. The widget is created during the AddTable user action, and the migration creates a widget for existing tables.

Test Plan: Many tests had to be updated, especially tests that listed all view sections and/or fields.

Reviewers: jarek, dsagal

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3232
2022-02-01 21:19:30 +02:00
Paul Fitzpatrick
7440485ebe (core) run survey for new users only when a survey doc is set
Summary:
stop providing a default document id DOC_ID_NEW_USER_INFO for
surveying, and don't show survey if a document id is not available.

Test Plan: existing tests pass; grist-core checked

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3225
2022-01-20 15:50:35 -05:00
Dmitry S
215bb90e68 (core) Replace questionnaire for new users with a popup asking for just their primary use-case.
Summary:
- WelcomeQuestions implements the new popup.
- Popup shows up on any doc-list page, the first time the user visits one after
  signing up and setting their name.
- Submits responses to the same "New User Questions" doc, which has been
  changed to accept two new columns (ChoiceList of use_cases, and Text for
  use_other).
- Improve modals on mobile along the way.

Test Plan: Added browser tests and tested manually

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3213
2022-01-14 10:16:11 -05:00
George Gevoian
ba6ecc5e9e (core) Move user profile to new page and begin MFA work
Summary:
The user profile dialog is now a separate page, in preparation
for upcoming work to enable MFA. This commit also contains
some MFA changes, but the UI is currently disabled and the
implementation is limited to software tokens (TOTP) only.

Test Plan:
Updated browser tests for new profile page. Tests for MFAConfig
and CognitoClient will be added in a later diff, once the UI is enabled.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3199
2022-01-13 21:21:49 -08:00
Jarosław Sadziński
85ef873ce5 (core) Widget options api
Summary:
Adding configuration options for CustomWidgets.

Custom widgets can now store options (in JSON) in viewSection metadata.

Changes in grist-plugin-api:
- Adding onOptions handler, that will be invoked when the widget is ready and when the configuration is changed
- Adding WidgetAPI - new API to read and save a configuration for widget.

Changes in Grist:
- Rewriting CustomView code, and extracting code that is responsible for showing the iframe and registering Rpc.
- Adding Open Configuration button to Widget section in the Creator panel and in the section menu.
- Custom Widgets can implement "configure" method, to show configuration screen when requested.

Test Plan: Browser tests.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3185
2022-01-13 11:10:17 +01:00
Paul Fitzpatrick
5cdc7b2ea4 (core) freshen core README; support python3 in grist-core docker image
Summary:
This updates the grist-core README to list specific features of Grist,
to make it easier for a casual visitor to get a sense of its scope. Adds links
to some new resources (reviews, templates, grist v airtable post) that could
also help. Adds python3 to docker image so that templates work without fuss.

Test Plan: existing tests should pass

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal, anaisconce

Differential Revision: https://phab.getgrist.com/D3204
2022-01-08 18:27:20 -05:00
Dmitry S
e0fb281eba (core) When searching, use formatter.formatAny() to handle values of invalid type.
Summary:
This turns out necessary because ReferenceList columns are formatted
using the formatter of the associated visibleCol. This works correctly
in dedicated widgets, but in generic code (like SearchModel here), this
formatter needs to handle unexpected values (of type ReferenceList).

Without the fix, it produces JS errors when search reaches a
RefList:<Date> column.

A better fix would allow a formatter to know that it expects a ReferenceList,
AND to know how to format each value of it, but that's a bigger question
that's outside the scope of this fix.

Test Plan: Includes a browser test which reproduces the bug.

Reviewers: cyprien

Reviewed By: cyprien

Differential Revision: https://phab.getgrist.com/D3195
2021-12-21 15:35:40 -05:00
Alex Hall
d1a848b44a (core) Parse string cell values in Doc API and Imports
Summary:
- Adds a function `parseUserAction` for parsing strings in UserActions to `ValueParser.ts`
- Adds a boolean option `parseStrings` to use `parseUserAction` in `ActiveDoc.applyUserActions`, off by default.
- Uses `parseStrings` by default in DocApi (set `?noparse=true` in a request to disable) when adding/updating records through the `/data` or `/records` endpoints or in general with the `/apply` endpoint.
- Uses `parseStrings` for various actions in `ActiveDocImport`. Since most types are parsed in Python before these actions are constructed, this only affects references, which still look like errors in the import preview. Importing references can also easily still run into more complicated problems discussed in https://grist.slack.com/archives/C0234CPPXPA/p1639514844028200

Test Plan:
- Added tests to DocApi to compare behaviour with and without string parsing.
- Added a new browser test, fixture doc, and fixture CSV to test importing a file containing references.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3183
2021-12-17 15:40:58 +02:00
Jarosław Sadziński
1ae586cf42 (core) Adding Skip options when importing multiple tables.
Summary:
Adding new destination "Skip" for multiple table imports.
Selecting this destination skips the import and makes the preview grayed out.

Test Plan: New Tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3181
2021-12-13 19:07:33 +01:00
Paul Fitzpatrick
d99db8d016 (core) move more tests to grist-core
Summary:
 * Tie build and run-time docker base images to a consistent version (buster)
 * Extend the test login system activated by GRIST_TEST_LOGIN to ease porting tests that currently rely on cognito (many)
 * Make org resets work in absence of billing endpoints
 * When in-memory session caches are used, add missing invalidation steps
 * Pass org information through sign-ups/sign-ins more carefully
 * For CORS, explicitly trust GRIST_HOST origin when set
 * Move some fixtures and tests to core, focussing on tests that cover existing failures or are in the set of tests run on deployments
 * Retain regular `test` target to run the test suite directly, without docker
 * Add a `test:smoke` target to run a single simple test without `GRIST_TEST_LOGIN` activated
 * Add a `test:docker` target to run the tests against a grist-core docker image - since tests rely on certain fixture teams/docs, added `TEST_SUPPORT_API_KEY` and `TEST_ADD_SAMPLES` flags to ease porting

The tests ported were `nbrowser` tests: `ActionLog.ts` (the first test I tend to port to anything, out of habit), `Fork.ts` (exercises a lot of doc creation paths), `HomeIntro.ts` (a lot of DocMenu exercise), and `DuplicateDocument.ts` (covers a feature known to be failing prior to this diff, the CORS tweak resolves it).

Test Plan: Manually tested via `buildtools/build_core.sh`. In follow up, I want to add running the `test:docker` target in grist-core's workflows. In jenkins, only the smoke test is run. There'd be an argument for running all tests, but they include particularly slow tests, and are duplicates of tests already run (in different configuration admittedly), so I'd like to try first just using them in grist-core to gate updates to any packaged version of Grist (the docker image currently).

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3176
2021-12-10 18:33:07 -05:00
Dmitry S
8100272e9a (core) Update HelpScout beacon to work with embedded documentation articles.
Summary:
- Fix base href in HelpScout beacon when showing articles (in particular for Firefox)
- Show the 'Answers' tab normally except when reporting an error.
- Combine the "Give Feedback" and "Help Center" buttons into one that normally
  opens the beacon (with a link to Help Center and to Community Forum), and a
  smaller one that opens the Help Center site in a new tab.
- Update HELP_SCOUT_* env vars to use _V2 suffix, to allow them to coexist with
  code using the previous beacon.

Test Plan: Updated the browser test to check the new behavior.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3170
2021-12-09 22:22:55 -05:00
Alex Hall
faec8177ab (core) Use MetaTableData more
Summary:
Add more method overrides to MetaTableData for extra type safety.

Use MetaTableData, MetaRowRecord, and getMetaTable in more places.

Test Plan: Mostly it just has to compile. Tested manually that types are being checked more strictly now, e.g. by adding a typo to property names. Some type casting has also been removed.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3168
2021-12-07 17:09:58 +02:00
Paul Fitzpatrick
a94905dd0a (core) make sure forks with no changes are persisted
Summary:
This fixes a problem where a fork could be created, have no changes
made, and then (e.g. if worker rolled over) fail to open with a
`cannot create fork` error. Adds a test that fails priot to this diff.

Test Plan: added test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3162
2021-12-01 22:27:56 -05:00
Jarosław Sadziński
1425461cd8 (core) Exposing custom widgets on the UI
Summary:
Exposing custom widgets as a dropdown menu in custom section configuration panel.

Adding new environmental variable GRIST_WIDGET_LIST_URL that points to a
json file with an array of available widgets. When not present, custom widget menu is
hidden, exposing only Custom URL option.

Available widget list can be fetched from:
https://github.com/gristlabs/grist-widget/releases/download/latest/manifest.json

Test Plan: New tests, and updated old ones.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3127
2021-12-01 18:21:06 +01:00
Jarosław Sadziński
53bdd6c8e1 (core) Exposing more descriptive errors from exports
Summary:
Exports used to show generic message on error.
Adding error description to the message.

Test Plan: Updated tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3157
2021-11-30 17:26:32 +01:00
Paul Fitzpatrick
3055a11fb2 (core) set cookie response header more consistently
Summary:
The express-session middleware, in its regular configuration, will
only set a cookie response header at the beginninng of a session or
when the session contents have changed. It won't set the header if
only the expiration time is changed. This diff uses a dummy `alive`
field to nudge the middleware into setting the header consistently.

Test Plan: tested manually

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3153
2021-11-24 10:16:30 -05:00
George Gevoian
7fe4423a6f (core) Allow filtering hidden columns
Summary:
Existing filters are now moved out of fields
and into a new metadata table for filters, and the
client is updated to retrieve/update/save filters from
the new table. This enables storing of filters for
columns that don't have fields (notably, hidden columns).

Test Plan: Browser and server tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3138
2021-11-22 10:26:08 -08:00
George Gevoian
c6aa9b65d4 (core) Fix bug preventing importing of nested json files
Summary:
BulkAddRecord when finishing imports of nested JSON was throwing
an error due to unchecked access of referencing tables. This adds
a guard to prepare_new_values to handle such cases.

Imports happened to cause this to occur because the order that
imported tables are created/populated isn't aware of references
between tables, so it's possible for a reference column to
exist (momentarily) without a valid reference to another table.
These references are currently fixed after all imported tables are
created/populated.

Test Plan: Browser test.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3144
2021-11-18 17:06:03 -08:00
Paul Fitzpatrick
c7331e2453 (core) make document reloading cleaner
Summary:
Currently when reloading a document, we may have two sqlite connections
to the document for a small period of time. This diff removes that
overlap.

Test Plan: added test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3140
2021-11-16 16:46:46 -05:00
Alex Hall
561e32fb44 (core) Better logging in DocTriggers
Summary: Added a helper to include lots of metadata in every logging call, added and converted many logging calls.

Test Plan: Existing tests pass

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3136
2021-11-11 15:24:33 +02:00
Alex Hall
e09e919016 (core) Ensure that large changes are processed in full by triggers (for webhooks)
Summary:
Uses the new alwaysPreserveColIds option for action summaries in Triggers.ts.

Triggers.ts is now responsible for generating the summary to make it easy to pass this option. The value of the option is just all colIds mentioned in triggers configured in this document.

Test Plan: Tested adding 200 rows to a subscribed table to ensure the events are not truncated. Also tests batching nicely.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3135
2021-11-10 23:13:55 +02:00
Paul Fitzpatrick
302202b4fb (core) freshen tests for python3
Summary:
Recent python3 changes perturbed timing again, and a few more tests started failing.

Contains an unrelated correction for gvisor running under docker (a useful configuration on macs for debugging gvisor problems, but not supported by throttling code).

Test Plan: updated tests

Reviewers: dsagal, alexmojaki

Reviewed By: dsagal, alexmojaki

Differential Revision: https://phab.getgrist.com/D3129
2021-11-10 10:46:12 -05:00
George Gevoian
08b1286f4f (core) Add column matching to Importer
Summary:
The Importer dialog is now maximized, showing additional column
matching options and information on the left, with the preview
table shown on the right. Columns can be mapped via a select menu
listing all source columns, or by clicking a formula field next to
the menu and directly editing the transform formula.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3096
2021-11-09 12:30:52 -08:00
Jarosław Sadziński
96fa7ad562 (core) Error message on Duplicate Document
Summary: Fixing error message when user can't copy document.

Test Plan: Updated tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3130
2021-11-09 19:12:57 +01:00
Paul Fitzpatrick
58880d4818 (core) support setting python version of new docs with PYTHON_VERSION_ON_CREATION
Summary:
If PYTHON_VERSION_ON_CREATION is set in the environment, new documents will be created with a specific desired python version (2 or 3).

This diff commits to offering a choice of engine, so the engine for a document no longer starts to initialize until the document has been fetched and read. Staging (and dev, and testing) has been like this for a while.

Test Plan: added test; manual testing of forks/copies etc

Reviewers: dsagal, alexmojaki

Reviewed By: dsagal, alexmojaki

Differential Revision: https://phab.getgrist.com/D3119
2021-11-05 10:51:18 -04:00
Paul Fitzpatrick
877542225d (core) mark engine setting as experimental
Summary:
This marks the engine setting in document settings as experimental,
with a skull and cross-bones.

It also makes sure the setting is shown if PYTHON_VERSION_ON_CREATION
is set (this relates to a separate change to set the default python
version to 3).

Test Plan: manual

Reviewers: alexmojaki, dsagal

Reviewed By: alexmojaki, dsagal

Subscribers: anaisconce

Differential Revision: https://phab.getgrist.com/D3120
2021-11-04 20:17:24 -04:00
Paul Fitzpatrick
db34e1a6d9 (core) tweak throttling to work for gvisor/runsc
Summary:
Grist has, up to now, used a throttling mechanism that allows a sandbox free rein until it starts using above some threshold percentage of a cpu for some time - at that point, we start sending STOP and CONT signals on a duty cycle, with longer and longer STOPped periods until cpu usage is at a threshold. The general idea is to do short jobs quickly, while throttling long jobs (thus unfortunately making them even longer) in order to continue doing other short jobs quickly.

The runsc sandbox is not a single process, there are in fact 5 per sandbox in our setup. Runsc can work with kvm or ptrace. Kvm is not available to us, so we use ptrace. With ptrace, there is one process that is the appropriate one to duty cycle, and another that needs to receive a signal in order to yield. This diff adds the necessary machinery.

This is a conservative change, where I stick with our existing throttling mechanism and adapt it to the new sandbox. It would be reasonable to consider switching throttling. There's a lot the OS allows. We can set a quota for how much cpu a process can use within a given period, for example. However the overall behavior with that would be quite different to what we have, so feels like this would need more discussion.

The implementation contains use of a linux utility `pgrep` since portability is not important (runsc is only available on linux) and there's no node api for enumerating children of a process.

The diff contains some tweaks to `buildtools/contain.sh` to streamline experimenting with Grist and runsc on a mac. It is important for throttling that node and the sandbox processes are in the same process name space, if docker is in between them then some extra machinery is needed (a proxy throttler and a way to communicate with it) which I chose not to implement.

Test Plan: added test; a lot of manual testing

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3113
2021-11-04 17:23:43 -04:00
Paul Fitzpatrick
10a4cbb6bd (core) make document assignment endpoint available via /housekeeping api
Summary:
The /assign endpoint checks if a document is on the desired worker
and moves it if not. This is never done under regular operation, but
is useful when quarantining a misbehaving document.

The endpoint was failing to operate correctly if the requester did
not have access to the document. This diff makes the endpoint
accessible through a /housekeeping route, using the same pattern as
the /force-reload endpoint.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3109
2021-11-04 16:14:21 -04:00
Alex Hall
4a70283292 (core) Webhook event queue on redis
Summary:
Push webhook events to redis queue with key based on docId.

Remove events from redis after sending using LTRIM.

Put failed events back on the end of the queue under normal circumstances.

When the event queue gets too long:
- Wait until it gets consumed before continuing.
- Drop failed events (i.e. don't put them back on the end of the queue)
- Limit webhook retries to 5

Test Plan: Tested that interactions with redis are as expected using redis MONITOR command.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3100
2021-11-03 22:43:35 +02:00
Jarosław Sadziński
3c72639e25 (core) Adding sort options for columns.
Summary:
Adding sort options for columns.
- Sort menu has a new option "More sort options" that opens up Sort left menu
- Each sort entry has an additional menu with 3 options
-- Order by choice index (for the Choice column, orders by choice position)
-- Empty last (puts empty values last in ascending order, first in descending order)
-- Natural sort (for Text column, compares strings with numbers as numbers)
Updated also CSV/Excel export and api sorting.
Most of the changes in this diff is a sort expression refactoring. Pulling out all the methods
that works on sortExpression array into a single namespace.

Test Plan: Browser tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: dsagal, alexmojaki

Differential Revision: https://phab.getgrist.com/D3077
2021-11-03 15:31:39 +01:00
Dmitry S
c5db65d1d2 (core) Process new user info in /welcome/info post without waiting for it to be written to the collecting document
Summary:
The document collecting new user info
(https://docs.getgrist.com/doc/GristNewUserInfo) got very slow, taking 40+
seconds for cold open. Sign-up submissions had to wait this time to proceed to
next step, because they waited for the write to this doc, which was blocked on
the Calculate action to complete.

Two changes were made: one to remove all expensive columns and summaries in the
actual doc, so the doc is back to opening in single seconds, and times should
be acceptable now.

The second change is this diff: to avoid waiting for the write step, so that it
doesn't affect users even if it gets slow again.

Test Plan: Existing test continues to work with a minor reliability tweak.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3103
2021-10-31 13:54:31 -04:00
Paul Fitzpatrick
6c53f3e820 (core) add an option to action summarization to preserve columns entirely
Summary:
Action summaries by default will drop rows in bulk changes, keeping only a few of them as examples. This diff allows overriding that, or selectively preserving some columns in their entirety.

This is intended for use with webhooks.

Test Plan: added test

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3098
2021-10-28 21:52:19 -04:00
Dmitry S
f2f4fe0eca (core) Add LogMethods helper and use it for more JSON data in logs. Reduce unhelpful logging.
Summary:
- Sharing, Client, DocClients, HostingStorageManager all include available info.
- In HostingStorageManager, log numSteps and maxStepTimeMs, in case that helps
  debug SQLITE_BUSY problem.
- Replace some action-bundle logging with a JSON version aggregating some info.
- Skip logging detailed list of actions in production.

Test Plan: Tested manually by eyeballing log output in dev environment.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3086
2021-10-25 10:25:18 -04:00
Alex Hall
e900f39da3 (core) Log statistics about table sizes
Summary:
Record numbers of rows, columns, cells, and bytes of marshalled data for most calls to table_data_from_db

Export new function get_table_stats in the sandbox, which gives the raw numbers and totals.

Get and log these stats in ActiveDoc right after loading tables, before Calculate, so they are logged even in case of errors.

Tweak logging about number of tables, especially number of on-demand tables, to not only show in debug logging.

Test Plan: Updated doc regression tests, that shows what the data looks like nicely.

Reviewers: dsagal, paulfitz

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3081
2021-10-21 17:54:20 +02:00
Paul Fitzpatrick
dd0f1be117 (core) get all tests working under python3/gvisor
Summary:
This verifies that all existing tests are capable of running under python3/gvisor, and fixes the small issues that came up. It does not yet activate python3 tests on all diffs, only diffs that specifically request them.

 * Adds a suffix in test names and output directories for tests run with PYTHON_VERSION=3, so that results of the same test run with and without the flag can be aggregated cleanly.
 * Adds support for checkpointing to the gvisor sandbox adapter.
 * Prepares a checkpoint made after grist python code has loaded in the gvisor sandbox.
 * Changes how `DOC_URL` is passed to the sandbox, since it can no longer be passed in as an environment variable when using checkpoints.
 * Uses the checkpoint to speed up tests using the gvisor sandbox, otherwise a lot of tests need more time (especially on mac under docker).
 * Directs jenkins to run all tests with python2 and python3 when a new file `buildtools/changelogs/python.txt` is touched (this diff counts as touching that file).
 * Tweaks miscellaneous tests
   - some needed fixes exposed by slightly different timing
   - a small number actually give different results in py3 (removal of `u` prefixes).
   - some needed a little more time

The DOC_URL change is not the ultimate solution we want for DOC_URL. Eventually it should be a variable that gets updated, like the date perhaps. This is just a small pragmatic change to preserve existing behavior.

Tests are run mindlessly as py3, and for some tests it won't change anything (e.g. if they do not use NSandbox). Tests are not run in parallel, doubling overall test time.

Checkpoints could be useful in deployment, though this diff doesn't use them there.

The application of checkpoints doesn't check for other configuration like 3-versus-5-pipe that we don't actually use.

Python2 tests run using pynbox as always for now.

The diff got sufficiently bulky that I didn't tackle running py3 on "regular" diffs in it. My preference, given that most tests don't appear to stress the python side of things, would be to make a selection of the tests that do and a few wild cards, and run those tests on both pythons rather then all of them. For diffs making a significant python change, I'd propose touching buildtools/changelogs/python.txt for full tests. But this is a conversation in progress.

A total of 6886 tests ran on this diff.

Test Plan: this is a step in preparing tests for py3 transition

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3066
2021-10-18 17:44:15 -04:00
Jarosław Sadziński
3e661db38c (core) Adding schema validation for records endpoint
Summary:
Adding validation for api /records endpoint, that checks if the json payload is valid.
Modifying POST /records endpoint to allow creating blank or partial records.

Test Plan: Updated tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3061
2021-10-18 21:40:50 +02:00
Alex Hall
276adc5f51 (core) Starting to make webhooks more robust
Summary:
- Puts events on a queue in memory and ensures they are sent in the order they were generated.
- Makes the caller (Sharing.ts) wait until changed records have been fetched from the DB, but allows it to continue after while remaining work happens asynchronously.
- Gathers all new webhook events into an array so they can be backed up to the queue on redis in a single command (in a future diff).
- Uses changes in isReady to determine event type, no more 'existed before'

The structure of the code has changed a lot, so I think the scope of the diff needs to stop here. Lots of work is still deferred in TODOs.

Test Plan: Updated existing test. Actually dropped testing of retry on failures and slowness because it no longer made sense to keep that as part of the current test, so a new test for that will be added in a future diff.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3074
2021-10-15 20:01:15 +02:00
Alex Hall
9d1cc89dc9 (core) Strip invalid characters from table name in excel import
Summary: Add sanitizeWorksheetName function, pass result to library function addWorksheet where error was raised.

Test Plan: Added unit test for sanitizeWorksheetName function, updated a fixture document to use a messy table name.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3072
2021-10-11 17:47:12 +02:00
Alex Hall
a64fb105e3 (core) Use GristObjCode in CellValue
Summary: Makes type checking a bit stronger

Test Plan: it just has to compile

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3065
2021-10-11 14:11:32 +02:00
George Gevoian
62db263d1f (core) Add diff preview to Importer
Summary:
Updates the preview table in Importer to show a diff of changes
when importing into an existing table and updating existing records.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3060
2021-10-08 14:15:07 -07:00
Paul Fitzpatrick
d635c97686 (core) flesh out "View As" feature
Summary:
The users shown by the "View As" button are now drawn from more sources:
 * There are users the document is shared with. This has been rationalized, the behavior was somewhat erratic. If the user is not an owner of the document, the only user of this kind that will be listed is themselves.
 * There are users mentioned in any user attribute table keyed by Email. If name and access columns are present, those are respected, otherwise name is taken from email and access is set to "editors".
 * There are example users provided if there are not many other users available.

Test Plan: added and extended tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3045
2021-10-08 12:00:40 -04:00
Paul Fitzpatrick
07558dceba (core) avoid censorship for one client clobbering data for another client
Summary:
When filtering document updates to send to clients after a change,
censorship of individual cells was being applied to state shared
across the clients. This diff eliminates that shared state, and
extends testing of broadcasts to check different orderings.

Test Plan:
extends a test to tickle a reported bug, and gives
DocClients a knob to control message order needed to tickle
the bug reliably.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3064
2021-10-07 23:21:07 -04:00
Paul Fitzpatrick
df318ad6b3 (core) add a mac-specific sandbox for development
Summary:
docker is slow on macs, so use native sandbox-exec by
default for tests involving python3 on macs.

Test Plan: updated test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3068
2021-10-07 14:14:25 -04:00
George Gevoian
e1780e4f58 (core) Migrate import code from data engine to Node
Summary:
Finishing imports now occurs in Node instead of the
data engine, which makes it possible to import into
on-demand tables. Merging code was also refactored
and now uses a SQL query to diff source and destination
tables in order to determine what to update or add.

Also fixes a bug where incremental imports involving
Excel files with multiple sheets would fail due to the UI
not serializing merge options correctly.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3046
2021-10-04 10:27:00 -07:00
Dmitry S
7e07f0ce56 (core) For grist_sid*_status cookie, remember to set the path
Test Plan: Only tested manually that path is included.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3056
2021-10-04 09:13:47 -04:00
Paul Fitzpatrick
8853e095bb (core) fix core build, and make smoke test more effective
Summary:
This makes the `core` test operate on a directory outside the
jenkins workspace, so that packages in the workspace don't
interfere with the test and obscure errors.

It also includes a small type fix for the `core` build.

Test Plan: updating a test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3054
2021-10-01 13:34:47 -04:00
Dmitry S
1517dca644 (core) Implement DiscourseConnect to enable easy sign-in to community forum
Summary:
- Update cookie module, to support modern sameSite settings
- Add a new cookie, grist_sid_status with less-sensitive value, to let less-trusted subdomains know if user is signed in
- The new cookie is kept in-sync with the session cookie.
- For a user signed in once, allow auto-signin is appropriate.
- For a user signed in with multiple accounts, show a page to select which account to use.
- Move css stylings for rendering users to a separate module.

Test Plan: Added a test case with a simulated Discourse page to test redirects and account-selection page.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3047
2021-10-01 11:24:22 -04:00