Commit Graph

203 Commits

Author SHA1 Message Date
Alex Hall
d1a848b44a (core) Parse string cell values in Doc API and Imports
Summary:
- Adds a function `parseUserAction` for parsing strings in UserActions to `ValueParser.ts`
- Adds a boolean option `parseStrings` to use `parseUserAction` in `ActiveDoc.applyUserActions`, off by default.
- Uses `parseStrings` by default in DocApi (set `?noparse=true` in a request to disable) when adding/updating records through the `/data` or `/records` endpoints or in general with the `/apply` endpoint.
- Uses `parseStrings` for various actions in `ActiveDocImport`. Since most types are parsed in Python before these actions are constructed, this only affects references, which still look like errors in the import preview. Importing references can also easily still run into more complicated problems discussed in https://grist.slack.com/archives/C0234CPPXPA/p1639514844028200

Test Plan:
- Added tests to DocApi to compare behaviour with and without string parsing.
- Added a new browser test, fixture doc, and fixture CSV to test importing a file containing references.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3183
2021-12-17 15:40:58 +02:00
Alex Hall
9d62e67369 (core) Fix display of formatted dates in reference columns
Summary:
Reference/referencelist columns displaying date/datetime columns didn't show the formatting of that column, formatting them as ISO instead. One weird effect of this was that opening the editor suddenly changed the format because the editor formatted the dates correctly. You can see this in the checkin doc as an example.

This was discussed in https://grist.slack.com/archives/C0234CPPXPA/p1636482208111800. Here's the main point:

> both use the visible column formatter's formatAny. the editor uses the value from the visible column, which for a date column is a raw timestamp number. the cell display uses the value from the display column which is of type Any so the value is wrapped in a list starting with 'd'. the former gets formatted according to the formatting options, but the latter just gets formatted as ISO.

Probably a good solution to the broader problem is to ensure that the display column has the same type and widget options as the visible column. That seems potentially messy, so I did something easier: fix `DateFormatter` to accept encoded date/datetime objects. It still receives the correct widget options from the visible column as before but can handle the values from the display column. This might also have other uses in the future.

Test Plan:
- Fixed several tests which previously expected the buggy behaviour.
- Converted ValueFormatter.js tests to typescript and cleaned up the existing code slightly.
- Added tests for DateFormatter and DateTimeFormatter to the ValueFormatter test suite, which only tested numbers before.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3190
2021-12-16 23:24:39 +02:00
Jarosław Sadziński
1ae586cf42 (core) Adding Skip options when importing multiple tables.
Summary:
Adding new destination "Skip" for multiple table imports.
Selecting this destination skips the import and makes the preview grayed out.

Test Plan: New Tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3181
2021-12-13 19:07:33 +01:00
Paul Fitzpatrick
d99db8d016 (core) move more tests to grist-core
Summary:
 * Tie build and run-time docker base images to a consistent version (buster)
 * Extend the test login system activated by GRIST_TEST_LOGIN to ease porting tests that currently rely on cognito (many)
 * Make org resets work in absence of billing endpoints
 * When in-memory session caches are used, add missing invalidation steps
 * Pass org information through sign-ups/sign-ins more carefully
 * For CORS, explicitly trust GRIST_HOST origin when set
 * Move some fixtures and tests to core, focussing on tests that cover existing failures or are in the set of tests run on deployments
 * Retain regular `test` target to run the test suite directly, without docker
 * Add a `test:smoke` target to run a single simple test without `GRIST_TEST_LOGIN` activated
 * Add a `test:docker` target to run the tests against a grist-core docker image - since tests rely on certain fixture teams/docs, added `TEST_SUPPORT_API_KEY` and `TEST_ADD_SAMPLES` flags to ease porting

The tests ported were `nbrowser` tests: `ActionLog.ts` (the first test I tend to port to anything, out of habit), `Fork.ts` (exercises a lot of doc creation paths), `HomeIntro.ts` (a lot of DocMenu exercise), and `DuplicateDocument.ts` (covers a feature known to be failing prior to this diff, the CORS tweak resolves it).

Test Plan: Manually tested via `buildtools/build_core.sh`. In follow up, I want to add running the `test:docker` target in grist-core's workflows. In jenkins, only the smoke test is run. There'd be an argument for running all tests, but they include particularly slow tests, and are duplicates of tests already run (in different configuration admittedly), so I'd like to try first just using them in grist-core to gate updates to any packaged version of Grist (the docker image currently).

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3176
2021-12-10 18:33:07 -05:00
Alex Hall
6b448567c9 (core) Refactor more value parsing code into common
Summary:
Following discussion in https://phab.getgrist.com/D3164:

- Change createParser to accept docData and one or two metadata row IDs and let it extract the metadata, so it's more easily usable in the server.
- Change ViewFieldRec.valueParser observable to a function createValueParser.

Test Plan: Existing tests.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3172
2021-12-09 21:05:27 +02:00
Paul Fitzpatrick
4164d89b84 (core) annotate guests of other docs correctly in UserManager dialog
Summary:
This corrects the annotations of users who are not team members but
are guests of other documents. Their annotation was previously
blank, rather than showing as collaborators.

Test Plan: added test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3178
2021-12-09 12:57:52 -05:00
Dmitry S
7a6d726daa (core) Change datepicker in DateEditor to use moment format, show AltText in DateEditor
Summary:
- Rather than translate from moment format to that of bootstrap-datepicker, use
  the customization methods to format datepicker dates using moment directly.
- Fix issue with parseDate() when format includes tokens like Mo or Do
- Fix issue in parseDateTime() that could produce an off-by-one error in date
  depending on local timezone.
- When opening DateEditor, show AltText value if present.

- Add crossorigin=anonymous to scripts that were missing it (including
  bootstrap-datepicker), to ensure that errors from them are reported properly
  rather than as 'Script error.'

Test Plan:
Added test cases to parseDate() test for low-level fixes; added a
browser test for the fixed DateEditor behavior.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3169
2021-12-07 11:33:49 -05:00
Alex Hall
faec8177ab (core) Use MetaTableData more
Summary:
Add more method overrides to MetaTableData for extra type safety.

Use MetaTableData, MetaRowRecord, and getMetaTable in more places.

Test Plan: Mostly it just has to compile. Tested manually that types are being checked more strictly now, e.g. by adding a typo to property names. Some type casting has also been removed.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3168
2021-12-07 17:09:58 +02:00
Alex Hall
116fb15eda (core) Move most of the reference parsing code into common so that the server can use it
Summary: Refactoring in preparation for parsing strings from the API. The plan is that the API code will only need to do a server-side version of the code in ViewFieldRec.valueParser (minus ReferenceUtils) which is quite minimal.

Test Plan: Nothing extra here, I don't think it's needed. This stuff will get tested more in a future diff which changes the API.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3164
2021-12-07 01:19:18 +02:00
Jarosław Sadziński
1425461cd8 (core) Exposing custom widgets on the UI
Summary:
Exposing custom widgets as a dropdown menu in custom section configuration panel.

Adding new environmental variable GRIST_WIDGET_LIST_URL that points to a
json file with an array of available widgets. When not present, custom widget menu is
hidden, exposing only Custom URL option.

Available widget list can be fetched from:
https://github.com/gristlabs/grist-widget/releases/download/latest/manifest.json

Test Plan: New tests, and updated old ones.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3127
2021-12-01 18:21:06 +01:00
Alex Hall
064455b2f7 (core) Parsing pasted datetimes
Summary:
Add function parseDateTime which parses a string containing both date and time componenents, intended for parsing pasted strings.

Add DateTimeParser subclass of ValueParser.

Test Plan: Extended parseDate.ts and CopyPaste.ts tests.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3152
2021-11-25 00:49:01 +02:00
George Gevoian
7fe4423a6f (core) Allow filtering hidden columns
Summary:
Existing filters are now moved out of fields
and into a new metadata table for filters, and the
client is updated to retrieve/update/save filters from
the new table. This enables storing of filters for
columns that don't have fields (notably, hidden columns).

Test Plan: Browser and server tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3138
2021-11-22 10:26:08 -08:00
Alex Hall
0d460ac2d4 (core) Parsing pasted ChoiceLists
Summary:
Added ChoiceListParser capable of parsing JSON, CSVs, and other strings containing user-configured choices (e.g. separated by spaces)

I got a little carried away here. It works, and I can't think of any bugs, but it's complicated enough that there could be hidden edge cases or difficulties maintaining it in the future. The advantage of the current method is that it should work well for ambiguous or poorly formatted inputs, e.g. choices separated only by spaces or choices containing commas which are not escaped/quoted properly. The code can be vastly simplified if we don't try to support that and require that users paste proper JSON or CSVs.

Test Plan: Added a new file test/common/ChoiceListParser.ts with pure unit tests. Waiting for approval of the overall approach before adding to the nbrowser CopyPaste test.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3141
2021-11-20 19:00:36 +02:00
George Gevoian
59699bf446 (core) Add additional org items to AppHeader
Summary:
Adds links to manage team and go to billing account in
the org menu (opened by clicking the dropdown in the
top-left corner of Grist). Tweaks some wording of items
in both AppHeader and AccountWidget, and adds a link
to create a new team site to the Site Switcher in both
menus.

Also tweaks the UI of UserManager by adding
an animation when the manager is opened from the
doc access dialog.

Test Plan: Browser tests.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3121
2021-11-05 08:19:51 -07:00
Jarosław Sadziński
4ddc29fb40 (core) Adding fallback for currency symbol
Summary: Adding fallback for currency formatting on older browsers.

Test Plan: Existing tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3108
2021-11-04 22:39:34 +01:00
Dmitry S
cce679d928 (core) Add some polish to the billing page, particularly for sign-up.
Summary:
- Change "Continue" button to "Review" (we don't charge immediately,
  first show a review screen)
- Show more informative messages for certain failures with discount
  coupons.
- Focus form elements with error, or at least the part of the form
  containing an error.
- Auto-focus discount input box when it gets toggled on.
- Show warning about URL changes only when subdomain is changed.

Test Plan: Updated tests; tested focus and changed error messages manually.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3115
2021-11-04 14:23:53 -04:00
Alex Hall
1db138d7ac (core) Change timezone abbreviation parsing
Summary: Allows any timezone abbreviation associated with the given timezone, and simply ignores it. Previously only certain abbreviations worked and they were not unique so using them outside the US was broken.

Test Plan: Added parseDate tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3106
2021-11-03 16:40:41 +02:00
Jarosław Sadziński
3c72639e25 (core) Adding sort options for columns.
Summary:
Adding sort options for columns.
- Sort menu has a new option "More sort options" that opens up Sort left menu
- Each sort entry has an additional menu with 3 options
-- Order by choice index (for the Choice column, orders by choice position)
-- Empty last (puts empty values last in ascending order, first in descending order)
-- Natural sort (for Text column, compares strings with numbers as numbers)
Updated also CSV/Excel export and api sorting.
Most of the changes in this diff is a sort expression refactoring. Pulling out all the methods
that works on sortExpression array into a single namespace.

Test Plan: Browser tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: dsagal, alexmojaki

Differential Revision: https://phab.getgrist.com/D3077
2021-11-03 15:31:39 +01:00
Alex Hall
d63da496a8 (core) Value parsing for refs, parsing data entry for numbers
Summary:
Handle reference columns in ViewFieldRec.valueParser.

Extracted code for reuse from ReferenceEditor to look up values in the visible column. While I was at it, also extracted a bit of common code from ReferenceEditor and ReferenceListEditor into a new class ReferenceUtils. More refactoring could be done in this area but it's out of scope.

Changed NTextEditor to use field.valueParser, which affects numeric and reference fields. In particular this means numbers are parsed on data entry, it doesn't change anything for references.

Test Plan:
Added more CopyPaste testing to test references.

Tested entering slightly formatted numbers in NumberFormatting.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3094
2021-11-01 19:31:52 +02:00
Alex Hall
e58df5df5b (core) ValueParser for Date columns
Summary: Adds parseDateStrict function based on parseDate, uses it in DateParser subclass of ValueParser.

Test Plan:
Tweaked parseDate test to check parseDateStrict.

Extended test in CopyPaste to test parsing dates as well as numbers.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3088
2021-10-26 13:03:19 +02:00
Paul Fitzpatrick
f7c9919120 (core) annotate shares listed in UserManager for documents
Summary:
This gives more guidance to users when editing document shares in the UserManager dialog.

  * For a document on a team site, any shares with team members are marked `Team member`.
  * Shares that count as external collaborators are marked for documents on a team or personal site as `collaborator` (personal site) or `outside collaborator` (team site).
  * Collaborators are marked `1 of 2`, `2 of 2`, and then `limit exceeded`.
  * On a team site, links are offered for each collaborator to add them to the team. The links lead to a prefilled dialog for managing team membership which can be confirmed immediately, allowing the user to continue without interruption.
  * On a personal site, for the last collaborator and beyond, a link is added for creating a team. This isn't seamless since creating a team involves billing etc.

There's a small unrelated tweak in tests to remove a confusing import from `test/browser` in `test/server`.

One thing I didn't get to is checking if owner of doc is owner of site. If they aren't, they may try to add a member and be denied at that point - it would be more polite to change messaging earlier for them.

Test Plan: added and updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3083
2021-10-25 14:29:19 -04:00
Dmitry S
f2f4fe0eca (core) Add LogMethods helper and use it for more JSON data in logs. Reduce unhelpful logging.
Summary:
- Sharing, Client, DocClients, HostingStorageManager all include available info.
- In HostingStorageManager, log numSteps and maxStepTimeMs, in case that helps
  debug SQLITE_BUSY problem.
- Replace some action-bundle logging with a JSON version aggregating some info.
- Skip logging detailed list of actions in production.

Test Plan: Tested manually by eyeballing log output in dev environment.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3086
2021-10-25 10:25:18 -04:00
Alex Hall
99878c08ed (core) Add ValueParser, use when pasting
Summary:
Add ValueParser file, base class, and subclasses for column types. Only NumericParser is used for now.

Add valueParser field to ViewFieldRec.

Use valueParser when parsing pasted text data in Grid and Detail views.

Test Plan: Add test to nbrowser CopyPaste suite, copying into a numeric column with different currency and locale settings.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3082
2021-10-21 21:43:19 +02:00
George Gevoian
f2e11a5329 (core) Migrate to Stripe v8 + implement discount codes
Summary:
New plan signups now include a discount code field in
the signup form. If a valid discount code is entered, a
discount will be applied on the confirmation page.

Test Plan: Browser and server tests.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3076
2021-10-20 12:34:03 -07:00
Alex Hall
4894631ba4 (core) Generic number parsing functionality.
Summary:
Added NumberParse.ts, counterpart of NumberFormat.ts.

Contains generic functionality for parsing numbers formatted by Intl.NumberFormat, not tied to documents or anything.

This doesn't change any actual behaviour, applying this parsing when pasting/typing in numeric columns will be a separate diff.

Test Plan: New file with extensive unit tests.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3078
2021-10-19 23:19:13 +02:00
Paul Fitzpatrick
dd0f1be117 (core) get all tests working under python3/gvisor
Summary:
This verifies that all existing tests are capable of running under python3/gvisor, and fixes the small issues that came up. It does not yet activate python3 tests on all diffs, only diffs that specifically request them.

 * Adds a suffix in test names and output directories for tests run with PYTHON_VERSION=3, so that results of the same test run with and without the flag can be aggregated cleanly.
 * Adds support for checkpointing to the gvisor sandbox adapter.
 * Prepares a checkpoint made after grist python code has loaded in the gvisor sandbox.
 * Changes how `DOC_URL` is passed to the sandbox, since it can no longer be passed in as an environment variable when using checkpoints.
 * Uses the checkpoint to speed up tests using the gvisor sandbox, otherwise a lot of tests need more time (especially on mac under docker).
 * Directs jenkins to run all tests with python2 and python3 when a new file `buildtools/changelogs/python.txt` is touched (this diff counts as touching that file).
 * Tweaks miscellaneous tests
   - some needed fixes exposed by slightly different timing
   - a small number actually give different results in py3 (removal of `u` prefixes).
   - some needed a little more time

The DOC_URL change is not the ultimate solution we want for DOC_URL. Eventually it should be a variable that gets updated, like the date perhaps. This is just a small pragmatic change to preserve existing behavior.

Tests are run mindlessly as py3, and for some tests it won't change anything (e.g. if they do not use NSandbox). Tests are not run in parallel, doubling overall test time.

Checkpoints could be useful in deployment, though this diff doesn't use them there.

The application of checkpoints doesn't check for other configuration like 3-versus-5-pipe that we don't actually use.

Python2 tests run using pynbox as always for now.

The diff got sufficiently bulky that I didn't tackle running py3 on "regular" diffs in it. My preference, given that most tests don't appear to stress the python side of things, would be to make a selection of the tests that do and a few wild cards, and run those tests on both pythons rather then all of them. For diffs making a significant python change, I'd propose touching buildtools/changelogs/python.txt for full tests. But this is a conversation in progress.

A total of 6886 tests ran on this diff.

Test Plan: this is a step in preparing tests for py3 transition

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3066
2021-10-18 17:44:15 -04:00
Alex Hall
a64fb105e3 (core) Use GristObjCode in CellValue
Summary: Makes type checking a bit stronger

Test Plan: it just has to compile

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3065
2021-10-11 14:11:32 +02:00
George Gevoian
62db263d1f (core) Add diff preview to Importer
Summary:
Updates the preview table in Importer to show a diff of changes
when importing into an existing table and updating existing records.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3060
2021-10-08 14:15:07 -07:00
Paul Fitzpatrick
d635c97686 (core) flesh out "View As" feature
Summary:
The users shown by the "View As" button are now drawn from more sources:
 * There are users the document is shared with. This has been rationalized, the behavior was somewhat erratic. If the user is not an owner of the document, the only user of this kind that will be listed is themselves.
 * There are users mentioned in any user attribute table keyed by Email. If name and access columns are present, those are respected, otherwise name is taken from email and access is set to "editors".
 * There are example users provided if there are not many other users available.

Test Plan: added and extended tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3045
2021-10-08 12:00:40 -04:00
George Gevoian
e1780e4f58 (core) Migrate import code from data engine to Node
Summary:
Finishing imports now occurs in Node instead of the
data engine, which makes it possible to import into
on-demand tables. Merging code was also refactored
and now uses a SQL query to diff source and destination
tables in order to determine what to update or add.

Also fixes a bug where incremental imports involving
Excel files with multiple sheets would fail due to the UI
not serializing merge options correctly.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3046
2021-10-04 10:27:00 -07:00
Dmitry S
1517dca644 (core) Implement DiscourseConnect to enable easy sign-in to community forum
Summary:
- Update cookie module, to support modern sameSite settings
- Add a new cookie, grist_sid_status with less-sensitive value, to let less-trusted subdomains know if user is signed in
- The new cookie is kept in-sync with the session cookie.
- For a user signed in once, allow auto-signin is appropriate.
- For a user signed in with multiple accounts, show a page to select which account to use.
- Move css stylings for rendering users to a separate module.

Test Plan: Added a test case with a simulated Discourse page to test redirects and account-selection page.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3047
2021-10-01 11:24:22 -04:00
Paul Fitzpatrick
b3b7410ede (core) open documents without blocking on data engine
Summary:
With this diff, when a user opens a Grist document in a browser, they will be able to view its contents without waiting for the data engine to start up. Once the data engine starts, it will run a calculation and send any updates made. Changes to the document will be blocked until the engine is started and the initial calculation is complete.

The increase in responsiveness is useful in its own right, and also reduces the impact of an extra startup time in a candidate next-generation sandbox.

A small unrelated fix is included for `core/package.json`, to catch up with a recent change to `package.json`.

A small `./build schema` convenience is added to just rebuild the typescript schema file.

Test Plan: added test; existing tests pass - small fixes needed in some cases because of new timing

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3036
2021-10-01 10:18:56 -04:00
Jarosław Sadziński
42910cb8f7 (core) Extending Google Drive integration scope
Summary:
New environmental variable GOOGLE_DRIVE_SCOPE that modifies the scope
requested for Google Drive integration.
For prod it has value https://www.googleapis.com/auth/drive.file which leaves
current behavior (Grist is allowed only to access public files and for private
files - it fallbacks to Picker).
For staging it has value https://www.googleapis.com/auth/drive.readonly which
allows Grist to access all private files, and fallbacks to Picker only when the file is
neither public nor private).
Default value is https://www.googleapis.com/auth/drive.file

Test Plan: manual and existing tests

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3038
2021-10-01 10:47:12 +02:00
Paul Fitzpatrick
876a0298a2 (core) do not look at content of recent actions when loading documents
Summary:
This removes the need for any information drawn from the content of recent actions when loading a document.

The undo/redo system does need some facts about recent actions up front. But that system has an important restriction: only actions a particular client is known to have generated can be undone by that client.

So in this diff, as we store which client has performed an action, we also store the few pieces of metadata about that action that the undo/redo system needs: `linkId`, `otherId`, `rowIdHint`, `isUndo` fields. These are all small integers (or in one case a boolean).

An existing limitation is that information about which client has performed which action is stored in memory in the worker, and not persisted anywhere. This diff does not change that limitation, meaning that undos continue to not survive a worker transition. A reasonable way to deal with that would be to back the store with redis.

Test Plan: existing tests pass

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3044
2021-09-29 11:27:02 -04:00
Jarosław Sadziński
048c8ee165 (core) Extending default locale list
Summary: Adding more locale codes to support more countries in document settings

Test Plan: existing tests

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3018
2021-09-24 15:10:13 +02:00
Dmitry S
fb583f303a (core) Support 'new' row in anchor links.
Summary:
- Anchor links with row of 'new' could be created but weren't parsed or used
  correctly. This fixes it.
- Also adds UIRowId type for row IDs which includes the special 'new' row. It's
  already been used in places as `number|'new'`, this diff gives it a name usable in app/common
  (it doesn't touch another name, RowId, that's been available in app/client).

Test Plan: Added a test assert for anchor links to new row

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3039
2021-09-24 09:01:10 -04:00
Alex Hall
3c4d71aeca (core) Initial webhooks implementation
Summary:
See https://grist.quip.com/VKd3ASF99ezD/Outgoing-Webhooks

- 2 new DocApi endpoints: _subscribe and _unsubscribe, not meant to be user friendly or publicly documented. _unsubscribe should be given the response from _subscribe in the body, e.g:

```
$ curl -X POST -H "Authorization: Bearer 8fd4dc59ecb05ab29ae5a183c03101319b8e6ca9" "http://localhost:8080/api/docs/6WYa23FqWxGNe3AR6DLjCJ/tables/Table2/_subscribe" -H "Content-type: application/json" -d '{"url": "https://webhook.site/a916b526-8afc-46e6-aa8f-a625d0d83ec3", "eventTypes": ["add"], "isReadyColumn": "C"}'
{"unsubscribeKey":"3246f158-55b5-4fc7-baa5-093b75ffa86c","triggerId":2,"webhookId":"853b4bfa-9d39-4639-aa33-7d45354903c0"}
$ curl -X POST -H "Authorization: Bearer 8fd4dc59ecb05ab29ae5a183c03101319b8e6ca9" "http://localhost:8080/api/docs/6WYa23FqWxGNe3AR6DLjCJ/tables/Table2/_unsubscribe" -H "Content-type: application/json" -d '{"unsubscribeKey":"3246f158-55b5-4fc7-baa5-093b75ffa86c","triggerId":2,"webhookId":"853b4bfa-9d39-4639-aa33-7d45354903c0"}'
{"success":true}
```

- New DB entity Secret to hold the webhook URL and unsubscribe key
- New document metatable _grist_Triggers subscribes to table changes and points to a secret to use for a webhook
- New file Triggers.ts processes action summaries and uses the two new tables to send webhooks.
- Also went on a bit of a diversion and made a typesafe subclass of TableData for metatables.

I think this is essentially good enough for a first diff, to keep the diffs manageable and to talk about the overall structure. Future diffs can add tests and more robustness using redis etc. After this diff I can also start building the Zapier integration privately.

Test Plan: Tested manually: see curl commands in summary for an example. Payloads can be seen in https://webhook.site/#!/a916b526-8afc-46e6-aa8f-a625d0d83ec3/0b9fe335-33f7-49fe-b90b-2db5ba53382d/1 . Great site for testing webhooks btw.

Reviewers: dsagal, paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3019
2021-09-23 14:35:39 +02:00
Dmitry S
a974dd839c (core) Improve time-parsing for DateTime columns, which was unusable with some formats (e.g. tz-containing ones)
Test Plan: Added a test with lots of time-parsing cases.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3034
2021-09-21 10:02:59 -04:00
Alex Hall
1a8abdcd96 (core) recursiveMoveToCursorPos with new kinds of linking involving lists
Summary: Constructs a ClientQuery in a similar way to LinkingState to handle lists.

Test Plan: Extended SelectBySummary and SelectByRefList tests.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3030
2021-09-20 23:13:24 +02:00
Paul Fitzpatrick
7907467dbc (core) treat summary tables like formulas for access control purposes
Summary:
This unsets the `direct` flag for actions emitted when summary tables are updated. That means those actions will be ignored for access control purposes. So if a user has the right to change a source table, the resulting changes to the summary won't result in the overall action bundle being forbidden.

I don't think I've actually seen the use case that inspired this issue being filed. I could imagine perhaps a user forbidden from creating rows globally making permitted updates that could add rows in a summary (and it being desirable to allow that).

Test Plan: added tests

Reviewers: jarek

Reviewed By: jarek

Subscribers: dsagal, alexmojaki, jarek

Differential Revision: https://phab.getgrist.com/D3022
2021-09-16 18:44:50 -04:00
George Gevoian
8a7edb6257 (core) Enable incremental imports
Summary:
The import dialog now has an option to 'Update existing records',
which when checked will allow for selection of 1 or more fields
to match source and destination tables on.

If all fields match, then the matched record in the
destination table will be merged with the incoming record
from the source table. This means the incoming values will
replace the destination table values, unless the incoming
values are blank.

Additional merge strategies are implemented in the data
engine, but the import dialog only uses one of the
strategies currently. The others can be exposed in the UI
in the future, and tweak the behavior of how source
and destination values should be merged in different contexts,
such as when blank values exist.

Test Plan: Python and browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3020
2021-09-16 09:15:54 -07:00
Paul Fitzpatrick
a543e5194a (core) add a python3 button
Summary: This adds a dropdown to the document settings model in staging/dev to set the python engine to Python2 or Python3. The setting is saved in `_grist_DocInfo.documentSettings.engine`.

Test Plan: tested manually for now - separate diff needed to add runsc to jenkins setup and make this testable

Reviewers: dsagal, alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3014
2021-09-16 10:06:04 -04:00
Paul Fitzpatrick
3e5a292cde (core) add tests for site deletion
Summary: This tests site deletion with and without a plan.

Test Plan: adding tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3017
2021-09-14 10:03:18 -04:00
George Gevoian
0717ee627e (core) Relocate export urls to /download/
Summary:
Moves CSV and XLSX export urls under /download/, and
removes the document title query parameter which is now
retrieved from the backend.

Test Plan: No new tests. Existing tests that verify endpoints still function.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3010
2021-09-02 09:36:33 -07:00
Jarosław Sadziński
7e78ebfeeb (core) Subdomain can start from a number
Summary: Allowing subdomain names to start from a number.

Test Plan: Updated tests

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3009
2021-09-01 18:16:57 +02:00
George Gevoian
ef5da42378 (core) Update export CSV and Excel endpoints
Summary:
The endpoints for exporting CSV and Excel are now under
/api/docs/:docId/ and are forwarded to a doc worker for export.

The Share Menu has been updated to use the new endpoints.

Test Plan: No new tests. Existing tests that verify endpoints work correctly.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3007
2021-08-31 10:47:24 -07:00
Paul Fitzpatrick
8b1ad588e9 (core) make treatment of emails consistent across /access endpoints
Summary:
Access endpoints were supposed to provide display versions of emails,
but in fact only the org endpoint was doing so.  This brings the
workspaces and docs endpoints into line, and adds tests.

Full user information is tweaked slightly to return an anonymous
flag only when anonymous.  This was already anticipated in the
FullUser type.

Test Plan: extended test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2999
2021-08-27 09:45:25 -04:00
George Gevoian
a6e08883e0 (core) Simple localization support and currency selector.
Summary:
- Grist document has a associated "locale" setting that affects how currency is formatted.
- Currency selector for number format.

Test Plan: not done

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D2977
2021-08-26 13:36:49 -07:00
Alex Hall
9916a2d919 (core) Suggest correct table when converting to RefList
Summary: RecordSets now have new encoding and rendering analogous to Records: `['r', 'Table', [1, 2, 3]]` and `Table[[1, 2, 3]]`.

Test Plan: Added to nbrowser/TypeChange.ts.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2987
2021-08-20 23:04:48 +02:00
Dmitry S
e361a9fd94 (core) Fix a few issues with parsing of dates in DateEditor.
Summary:
- With a format like "DD-MM-YYYY" or "DD MMM YYYY", allow parsing dates
  with two digit year or numeric month (like "16-8-21").
- Interpret two-digit years in the same way for moment parsing and for
  bootstrap-datepicker.
- For partial inputs (like "8/16"), when a format is present, assume that
  provided parts cover the date, then month, then year (even for a format that
  starts with year).

Test Plan: Expanded a unittest

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D2985
2021-08-20 14:54:41 -04:00
Paul Fitzpatrick
9f25a96d18 (core) add a way to change subdomain in billing pages
Summary:
This adds an `updateDomain` billing task that allows editing
the subdomain (and the org name, which is also editable with
the address).

A warning is shown that changing the subdomain will mean that
saved links need updating.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2988
2021-08-19 21:22:41 -04:00
Alex Hall
e6e792655b (core) Add /columns endpoint to DocApi
Summary: Add /columns endpoint to DocApi

Test Plan: Added test

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2981
2021-08-17 23:20:52 +02:00
George Gevoian
79f6f605f8 (core) Polish and enable Reference List widget
Summary:
Adds Reference List as a widget type.

Reference List is similar to Choice List: multiple references can be added
to each cell through a similar editor, and the individual references
will always reflect their current value from the referenced table.

Test Plan: Browser tests.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz, jarek, alexmojaki, dsagal

Differential Revision: https://phab.getgrist.com/D2959
2021-08-12 11:51:21 -07:00
Alex Hall
34e9ad3498 (core) Add /records endpoint to DocApi with GET, POST, and PATCH
Summary:
Applies simple data transformations to the existing /data API.

Mimics the Airtable API. Designed in https://grist.quip.com/RZh9AEbPaj8x/Doc-API#FZfACAAZ9a0

Haven't done deletion because it seems like less of a priority and also not fully designed.

Test Plan: Added basic server tests similar to the /data tests. Haven't tested edge cases like bad input.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D2974
2021-08-12 18:02:56 +02:00
Alex Hall
7f1f8fc9e6 (core) Linking summary tables grouped by list columns
Summary:
Prefix keys of `LinkingState.filterColValues` with `_contains:` when the source column is a ChoiceList or ReferenceList.

This is parsed out to make a boolean `isContainsFilter` which is kept in each value of `QueryRefs.filterTuples` (previously `filterPairs`).

Then when converting back in `convertQueryFromRefs` we construct `Query.contains: {[colId: string]: boolean}`.

Finally `getFilterFunc` uses `Query.contains` to decide what kind of filtering to do.

This is not pretty, but the existing code is already very complex and it was hard to find something that wouldn't require touching loads of code just to make things compile.

Test Plan: Added a new nbrowser test and fixture, tests that selecting a source table by summary tables grouped by a choicelist column, non-list column, and both all filter the correct data.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2940
2021-08-10 20:41:24 +02:00
Jarosław Sadziński
4ca47878ca (core) Adding import from google drive to the home screen
Summary: Importing from google drive from home screen (also for anonymous users)

Test Plan: Browser tests

Reviewers: dsagal, paulfitz

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2943
2021-08-05 20:46:11 +02:00
Dmitry S
1605e18f66 (core) Enable auto triggering of Welcome Tour, and various improvements.
Summary:
- Add showGristTour preference, and trigger tour automatically.
- Tour is only triggered for new and anonymous users on a personal org, with
  edit permission.

- Automatically open the right panel at tour start.
- Don't show tours on mobile, since that's not ready (popups are cut off
  and can't be dismissed)
- Cancel previous tour if a new one is somehow started.
- Remove #repeat- trigger hash tags from the URL when the tour starts.
- Ensure Help Center popup is positioned even when left panel is collapsed.
- Polish up the content of the last two cards in the tour.

Test Plan: Added test case for triggering and opening right panel.

Reviewers: alexmojaki, paulfitz

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D2955
2021-07-30 14:17:54 -04:00
Alex Hall
73c4efa315 (core) Hide the GristDocTour table by default but reveal it when /p/GristDocTour is in the URL
Summary:
Adds 'GristDocTour' as a possible value of urlState().docPage

GristDoc checks for this and converts it to a normal view record ID

It also then sets a flag showGristDocTour=true which tells Pages.ts to show the page in the sidebar

Otherwise the page is 'hidden' in the sidebar in the same way it would be if blocked by ACL rules

This all feels very hacky, but I don't know this code well enough to know if there's a better way. Hopefully this behaviour is temporary.

Test Plan: Tested manually, not sure if this is worth an automated test at this stage

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz, dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D2953
2021-07-30 20:14:34 +02:00
George Gevoian
24fc3a2d00 (core) Redesign examples and templates UI
Summary:
The old Examples and Templates workspace is now
a page that pulls templates from a new public Grist Templates org.
The All Documents view will pull featured templates from that org, where
featured templates are simply pinned documents in Grist Templates. The
Examples and Templates page will also show the featured templates, as
well as the rest of the available templates organized by category. The
categories are equivalent to workspaces in Grist Templates, and are
generated dynamically.

Test Plan: Browser tests.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz, dsagal

Subscribers: dsagal, paulfitz, jarek

Differential Revision: https://phab.getgrist.com/D2930
2021-07-28 12:29:03 -07:00
Alex Hall
cd0c6de53e (core) Automatically or explicitly show document tours in the same way as example cards.
Summary:
Extracts code from showExampleCard into a generic function which is reused for document tours.

It handles reading and writing to user preferences for automatic showing and explicitly reopening.

Test Plan:
Manually tested that it automatically shows a tour just once and clicking to reopen works.

There's not much new functionality so there's little that needs testing. This is an initial version that's mostly internal and is likely to be polished for users in the future.

If I should still add tests, I'd like confirmation that the current behaviour is as desired.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2944
2021-07-27 18:35:48 +02:00
Alex Hall
15f1ef96fa (core) Allow creating custom document tours with a special table
Summary:
Like the welcome tour, a special URL hash triggers startDocTour which uses data from a table GristDocTour to construct the appropriate popups.

This is the basic version described in https://grist.quip.com/sN2RAHI2dchm/Document-tours

Test Plan:
Added a new nbrowser test which tests the data produced by makeDocTour. The general behaviour of the UI and popups has hardly changed so existing tests cover that well enough.

The new test uses a new fixture document which you can open to easily experience the tour.

Error cases where there's no valid document tour are not tested because that behaviour is likely to change significantly and this feature is still quite 'private'.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: jarek, dsagal

Differential Revision: https://phab.getgrist.com/D2938
2021-07-23 20:23:30 +02:00
Alex Hall
04e5d90f86 (core) Barely working reference lists in frontend
Summary:
This makes it possible to set the type of a column to ReferenceList, but the UI is terrible

ReferenceList.ts is a mishmash of ChoiceList and Reference that sort of works but something about the CSS is clearly broken

ReferenceListEditor is just a text editor, you have to type in a JSON array of row IDs. Ignore the value that's present when you start editing. I can maybe try mashing together ReferenceEditor and ChoiceListEditor but it doesn't seem wise.
I think @georgegevoian should take over here. Reviewing the diff as it is to check for obvious issues is probably good but I don't think it's worth trying to land/merge anything.

Test Plan: none

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D2914
2021-07-23 18:41:44 +02:00
Jarosław Sadziński
a07395855a (core) Fixing anchor link navigation.
Summary: Last document position was overwritting anchor link navigation.

Test Plan: Browser tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D2934
2021-07-23 11:02:07 +02:00
Jarosław Sadziński
08295a696b (core) Export to Excel and Send to drive
Summary:
Implementing export to excel and send to Google Drive feature.

As part of this feature few things were implemented:
- Server side google authentication exposed on url: (docs, docs-s, or localhost:8080)/auth/google
- Exporting grist documents as an excel file (xlsx)
- Storing exported grist document (in excel format) in Google Drive as a spreadsheet document.

Server side google authentication requires one new environmental variables
- GOOGLE_CLIENT_SECRET (required) used by authentication handler

Test Plan: Browser tests for exporting to excel.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2924
2021-07-21 16:36:00 +02:00
Alex Hall
1f6e693b6e (core) Remove REPL code
Summary: Remove repl.py, REPLTab.js, some wiring code, CSS, and a test in testscript.json.

Test Plan: NA

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2923
2021-07-20 15:17:03 +02:00
Cyprien P
693f2f6325 (core) Brings welcome tour and hide behind a flag
Summary:
This diff brings in the new welcome tour. It builds upon `client/ui/OnBoardingPopup` that was committed to that purposes. Per this diff,  the tour is accessible behind a flag and won't be visible to user: few caveats listed below needs to be adressed first.

This diff also brings few changes to onboarding module.
  - allow to refer to element with selector
     - usually dynamic selection of element sounds useful for when the
     element does not exist yet when the tour starts. But the actual
     reason when add it here, is to allow selecting the first cell.
     - if the selector yields undefined (missing element), the popup
     is simply skipped
  - got rid of the internal registry to link between popup contents
  and popup options. All is now define in the same interface. Registry
  overall felt overkill and not needed.
  - adds an option to show message as a simple modal that is centered
  on the screen

This diff also brings the new welcome tour and hide it behind a flag

CAVEATS that need to be addressed in follow up commit:
 - The url needs cleanup, #repeat-welcome-tour sticks to it and so even when navigating to home page. This could eventually become an issue: if user opens another document it would starts the onboarding tour again.
 - For now you have to manually make sure the right panel is opened with the Column tab selected before starting the tour.
  - On boarding tours were not designed with mobile support in mind. So probably a good idea to disable.
  - Backend support needs to be done (persistence of first time user).

Test Plan:
Updated `projects/OnBoardingPopup` and adds new `nbrowser/welcomeTour`
To launch the tour:
  - open any document
  - open manually the right panel and the field tab
  - append the flag `#repeat-welcome-tour` at the end of the url in the url bar and reload the page

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2917
2021-07-19 16:30:53 +02:00
Paul Fitzpatrick
997be24a21 (core) add docs.options column to home db to store doc description, icon, openMode
Summary:
Bundles some new document options into a JSON column.
The icon option is treated somewhat gingerly.  It is intended, at
least initially, to store an image thumbnail for a document as a
url to hand-prepared assets (for examples and templates), so it is
locked down to a particular url prefix to avoid opening the door to
mischief.

Test Plan: added test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D2916
2021-07-15 21:51:05 -04:00
George Gevoian
e5eeb3ec80 (core) Add 'user' variable to trigger formulas
Summary:
The 'user' variable has a similar API to the one from access rules: it
contains properties about a user, such as their full name and email
address, as well as optional, user-defined attributes that are populated
via user attribute tables.

Test Plan: Python unit tests.

Reviewers: alexmojaki, paulfitz, dsagal

Reviewed By: alexmojaki, dsagal

Subscribers: paulfitz, dsagal, alexmojaki

Differential Revision: https://phab.getgrist.com/D2898
2021-07-15 15:18:32 -07:00
Jarosław Sadziński
625fce5f65 (core) Refactoring google drive plugin
Summary:
Finishing implementation for google drive plugin.
- Refactoring plugin code to make it more robust and to follow grist ux
- Changing the way server hosts untrusted user content, from different domain to different port

Test Plan: Browser tests

Reviewers: dsagal, paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2881
2021-07-14 09:52:04 +02:00
George Gevoian
9592e3610b (core) Add 'value' to trigger formula autocomplete
Summary:
API signature for autocomplete updated to add column ID, which is
necessary for exposing correct types for 'value'.

Test Plan: Unit tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: jarek, alexmojaki

Differential Revision: https://phab.getgrist.com/D2896
2021-07-12 15:07:16 -07:00
Dmitry S
0e0528a2cc (core) Add some more special subdomains, not to be used for team sites
Test Plan: Existing tests should pass, no new ones needed

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2905
2021-07-12 09:51:20 -04:00
Paul Fitzpatrick
d0905b6224 (core) correct capitalization of AppSumo; add account management link
Summary:
Branding feedback from AppSumo found a capitalization problem. They also nudged us again to include a link back for the user to manage
their AppSumo account.

Test Plan: manual

Reviewers: dsagal, anaisconce

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D2901
2021-07-08 12:41:29 -04:00
Alex Hall
ea01ca814d (core) Remove a bunch of dead code
Summary: Removed test/aws/, most of app/server/lib/, 3 dirs in app/lambda/, corresponding tests, and more!

Test Plan: a lot of this is quite the opposite...

Reviewers: dsagal, paulfitz

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2894
2021-07-01 18:38:21 +02:00
Dmitry S
b537539b73 (core) Implement UI for trigger formulas.
Summary:
- Implement UI with "Apply to new records" and "Apply on record changes"
  checkboxes, and options for selecting which changes to recalculate on.
- For consistency, always represent empty RefList as None
- Fix up generated SchemaTypes to remember that values are encoded.

Included test cases for the main planned use cases:
- Auto-filled UUID column
- Data cleaning
- NOW() formula for record's last-updated timestamp.
- Updates that depend on other columns.

Test Plan: Added a browser test.

Reviewers: jarek

Reviewed By: jarek

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D2885
2021-06-29 10:24:16 -04:00
Dmitry S
a56714e1ab (core) Implement trigger formulas (generalizing default formulas)
Summary:
Trigger formulas can be calculated for new records, or for new records and
updates to certain fields, or all fields. They do not recalculate on open,
and they MAY be set directly by the user, including for data-cleaning.

- Column metadata now includes recalcWhen and recalcDeps fields.
- Trigger formulas are NOT recalculated on open or on schema changes.
- When recalcWhen is "never", formula isn't calculated even for new records.
- When recalcWhen is "allupdates", formula is calculated for new records and
  any manual (non-formula) updates to the record.
- When recalcWhen is "", formula is calculated for new records, and changes to
  recalcDeps fields (which may be formula fields or column itself).
- A column whose recalcDeps includes itself is a "data-cleaning" column; a
  value set by the user will still trigger the formula.
- All trigger-formulas receive a "value" argument (to support the case above).

Small changes
- Update RefLists (used for recalcDeps) when target rows are deleted.
- Add RecordList.__contains__ (for `rec in refList` or `id in refList` checks)
- Clarify that Calculate action has replaced load_done() in practice,
  and use it in tests too, to better match reality.

Left for later:
- UI for setting recalcWhen / recalcDeps.
- Implementation of actions such as "Recalculate for all cells".
- Allowing trigger-formulas access to the current user's info.

Test Plan: Added a comprehensive python-side test for various trigger combinations

Reviewers: paulfitz, alexmojaki

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2872
2021-06-25 22:53:07 -04:00
Paul Fitzpatrick
dca3abec1d (core) complete light sign-up flow for appsumo, and customize summaries
Summary:
Current appsumo sign-up flow doesn't reach the billing pages.
This diff nudges user on through that extra step.

It also tweaks plan summaries to say what special appsumo
features are in effect (member count prepaid for).

Test Plan: manual

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2882
2021-06-25 14:13:13 -04:00
Paul Fitzpatrick
36d5e7870e (core) streamline registration flow for new appsumo users
Summary:
This adds a new landing page for cognito sign-up, intended for
use by new appsumo users.

Their email address is pre-filled and locked down, and sign-up
is by entering a password.

The page is very crude compared to hosted cognito - especially
in error reporting! - but having the address filled in more
than makes up for that.

The flow does not quite connect with the new billing signup.
I think we can do that through the regular "welcome" process,
which will list the user's team site.  When the user visits
that site, we could detect that we are on a site with no
domain set yet and for which the user is a billing manager,
and trigger a visit to the appropriate billing page.

Test Plan: manual - hard to test through cognito email step

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2880
2021-06-25 10:47:10 -04:00
Paul Fitzpatrick
1af99e9567 (core) link AppSumo activations with stripe, and support upgrades/downgrades
Summary:
This links AppSumo sign-ups with Stripe subscriptions
and our billing pages. Different AppSumo tiers are supported by
different coupons on the standard plan. Configuration of this
is in stripe, and then cached in the database.

The front end is tweaked just enough to make completing a sign-up
possible. It is not yet friendly.

Not covered includes:
 * Streamlining landing page.
 * Making billing pages git clearer summaries of AppSumo states.
 * Making flow through Cognito as graceful as possible - default
   probably doesn't meet AppSumo requirements.
 * Disabling site on cancellation/refund.
 * Downgrades when more seats in use than lower tier allows.

Test Plan: api-level tests added. No front-end tests yet.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2878
2021-06-24 10:18:42 -04:00
George Gevoian
b94eb107d4 (core) Use individual choices for filtering choice lists
Test Plan: Wrote unit and browser tests that verify new behavior.

Reviewers: paulfitz, dsagal

Reviewed By: dsagal

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D2855
2021-06-11 09:34:06 -07:00
Paul Fitzpatrick
6f02987d10 (core) allow undos to be partial, if access control prohibits some part of them
Summary:
This is a somewhat experimental change, that will implement permitted parts of an undo if not all parts are permitted.  This is in preparation for trigger columns, where it may become common for a change in a record resulting in a change to an automatic change to another that the user cannot edit directly.  How to undo such an action is somewhat unclear.  One option is to undo the permitted parts, and then the triggers can rerun.

The general case is a bit of a can of worms, and feels adjacent to merging/rebasing etc.

Oh: it would probably be important in general to communicate to the user that an undo was partial, but this diff doesn't do that.  It would need some new plumbing.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2839
2021-06-10 18:26:11 -04:00
Jarosław Sadziński
96fee73b70 (core) Download as CSV button on sections
Summary: Adding "Download as CSV" button that exports filtred section data to csv

Test Plan: Browser tests

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2830
2021-05-27 15:48:12 +02:00
Dmitry S
d1c1416d78 (core) Add rules to eslint to better match our coding conventions.
Summary:
We used tslint earlier, and on switching to eslint, some rules were not
transfered. This moves more rules over, for consistent conventions or helpful
warnings.

- Name private members with a leading underscore.
- Prefer interface over a type alias.
- Use consistent spacing around ':' in type annotations.
- Use consistent spacing around braces of code blocks.
- Use semicolons consistently at the ends of statements.
- Use braces around even one-liner blocks, like conditionals and loops.
- Warn about shadowed variables.

Test Plan: Fixed all new warnings. Should be no behavior changes in code.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2831
2021-05-24 12:56:18 -04:00
Cyprien P
e3a957a715 (core) Fixes persistence of the aclAsUser_ parameters across navigation
Summary:
  - The parameter was inadvertently removed by userOverrideParams().
  - when passed a function to `urlState().setHref(...)` or `urlState().makeUrl(...)` it is important that the function does not mutate the state that it receives as argument.

Link to the related task: https://gristlabs.getgrist.com/doc/check-ins/p/5#a1.s9.r791.c19

Test Plan: Adds test of the persistence by slightly modifying existing nbrowser/AccessRules2 tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D2820
2021-05-19 09:51:38 +02:00
Jarosław Sadziński
5f182841b9 (core) Document keeps track of latest cursor position and latest editor value and is able to restore them when it is reloaded.
Summary: Grist document, when reloaded, is able to restore the latest cursor position and the editor state.

Test Plan: Browser test were created.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D2808
2021-05-18 10:36:59 +02:00
Dmitry S
dee487684e (core) Show invalid columns as an error when entering rules
Summary:
Auto-complete helps enter correct column names, and when incorrect ones are
entered, we now show an error and prevent saving the rules.

In an unrelated tweak, fix focusing of ACLFormula when clicking into scroll area.

Test Plan: Added a test case for showing invalid columns

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2815
2021-05-13 11:52:10 -04:00
Paul Fitzpatrick
d0d3d3d0c9 (core) discount indirect changes for access control purposes
Summary:
This diff discounts indirect changes for access control purposes.  A UserAction that updates a cell A, which in turn causes changes in other dependent cells, will be considered a change to cell A for access control purposes.

The `engine.apply_user_actions` method now returns a `direct` array, with a boolean for each `stored` action, set to `true` if the action is attributed to the user or `false` if it is attributed to the engine.  `GranularAccess` ignores actions attributed to the engine when checking for edit rights.

Subtleties:
 * Removal of references to a removed row are considered direct changes.
 * Doesn't play well with undos as yet.  An action that indirectly modifies a cell the user doesn't have rights to may succeed, but it will not be reversible.

Test Plan: added tests, updated tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2806
2021-05-12 11:26:21 -04:00
Dmitry S
8d62a857e1 (core) Add ChoiceList type, cell widget, and editor widget.
Summary:
- Adds a new ChoiceList type, and widgets to view and edit it.
- Store in SQLite as a JSON string
- Support conversions between ChoiceList and other types

Test Plan: Added browser tests, and a test for how these values are stored

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2803
2021-05-12 10:38:32 -04:00
Dmitry S
2f26f140c8 (core) Add our very own implementation of tokenfield (aka pillbox, tag list)
Summary:
- Supports auto-completion
- Supports various keyboard interactions to delete/select/copy-paste tokens
- Supports various mouse interactions to select and move tokens.
- Supports undo/redo while editing tokens.

Test Plan: stand-alone fixture and browser test verify the supported behaviors

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2789
2021-05-03 17:16:24 -04:00
Dmitry S
526b0ad33e (core) Configure more comprehensive eslint rules for Typescript
Summary:
- Update rules to be more like we've had with tslint
- Switch tsserver plugin to eslint (tsserver makes for a much faster way to lint in editors)
- Apply suggested auto-fixes
- Fix all lint errors and warnings in core/, app/, test/

Test Plan: Some behavior may change subtly (e.g. added missing awaits), relying on existing tests to catch problems.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2785
2021-04-26 18:54:55 -04:00
Paul Fitzpatrick
d64461cd81 (core) optimization: remove lodash/pullAt
Summary:
For a long array with removals proportional to that length,
lodash/pullAt becomes slow due to doing one splice per removal.
This diff swaps in an alternate implementation that doesn't become
quadratic.  On a 250k-row doc with a row-level access rule, this improves
initial page load for a viewer with access to half the rows from minutes
to seconds.

Test Plan: added test; did manual benchmarking

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2777
2021-04-15 21:18:47 -04:00
Dmitry S
8c2bea0f73 (core) Remove the aclUI=1 flag and add 'BETA' tag to the Access Rules page name.
Summary:
- Remove support for aclUI=1 URL parameter, making it the default.
- Add 'BETA' tag to the Access Rules link in side panel.
- Remove all mentions of aclUI=1 in tests.

Test Plan: Updated tests should pass

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2766
2021-03-25 18:10:00 -04:00
Paul Fitzpatrick
9d1bc5a518 (core) make AccessRules and FullCopies effective
Summary:
This allows `*SPECIAL:AccessRules` to give read access to the access rules to more users, and `*SPECIAL:FullCopies` to grant download/copy rights to more users.

This diff also changes forks to be owned by the user who forked them (previously they were an editor), since that feels more natural.

Test Plan: Added and updated tests.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2760
2021-03-25 15:05:26 -04:00
Dmitry S
e14488bcc8 (core) Add support for special ACL rules, for viewing rules and downloading documents.
Summary:
- Use special ACLResources of the form "*SPECIAL:<RuleType>" to represent
  special document-wide rules.
- Include default rules that give Read access to these resources to Owners only.
- Add UI with a checkbox to give access to everyone instead.
- Allow expanding the UI for advanced configuration.

- These rules don't actually have any behavior yet.

Test Plan: WIP

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2764
2021-03-25 10:28:05 -04:00
Paul Fitzpatrick
afb83a4ff1 (core) add OWNERS='owners', EDITOR='editors', VIEWER='viewers' to condition formulas
Summary: this adds constants for user access roles, to facilitate autocomplete.

Test Plan: updated tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2761
2021-03-19 18:20:33 -04:00
Dmitry S
b4c34cedad (core) Update UI for formula and column label/id in the right-side panel.
Summary:
- Update styling of label, id, and "derived ID from label" checkbox.
- Implement a label which shows 'Data Column' vs 'Formula Column' vs 'Empty Column',
  and a dropdown with column actions (such as Clear/Convert)
- Implement new formula display in the side-panel, and open the standard
  FormulaEditor when clicked.
- Remove old FieldConfigTab, of which now very little would be used.
- Fix up remaining code that relied on it (RefSelect)

Test Plan: Fixed old tests, added new browser cases, and a case for a new helper function.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2757
2021-03-17 01:35:56 -04:00
Dmitry S
6e844a2e76 (core) Use unicode-aware comparisons for user-visible strings.
Summary:
- Switch code that compares user strings to use localeCompare() based on Intl.Collator.
- Use en-US locale for now. (Ideally should be a document property.)
- Note that with this change, sorting is also becoming case-insensitive (which
  seems an improvement)

- Updated a sorted test fixture
- Updated a browser test with lots of unicode to expect different order.
- Added a bit of unicode to test ordering in Reference autocomplete dropdown.

Test Plan: Fixed / updated tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2758
2021-03-15 09:54:10 -04:00
Paul Fitzpatrick
a1a84d99c0 (core) alert user if they try to use rec in a column rule controlling read permission
Summary:
This particular combination of features is not built out - data will be
censored but changes to data will not.  So the user will now get an error
if they try to do it.  Existing rules of this kind will continue to
operate as before, and can be set via the api.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2751
2021-03-10 11:57:09 -05:00
Dmitry S
3f29baaded (core) Add a button and a tooltip to Access Rules page item, in View-As mode.
Summary:
- When in View-As mode, clicking the Access Rules page now shows a tooltip
  with a link to return to normal mode and open the Access Rules page.
- A "revert" button is shown next to the item with the same behavior.
- Implemented hoverTooltip() with various options. (It will have other uses.)
- Simplify creation of links based on UrlState:
  - Allow merging with previous urlState using a function
  - Add a helper function to merge in aclAsUser parameter.
  - Add setHref() method to UrlState

Test Plan:
Added test cases:
  - for tooltips generally in test/projects
  - for updating UrlState using a callback
  - for Access Rules tooltip and button behavior

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2749
2021-03-08 16:08:21 -05:00
Cyprien P
5e5bf3af9d (core) Makes the hex value editable in the color select
Summary:
 - reuses the textInput form the editableLabel module
 - adds a isValidHex utility function to gutil

Test Plan: - Adds test to the project test

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2744
2021-03-08 09:12:47 +01:00
Paul Fitzpatrick
7bd3b2499f (core) allow multiple rule sets for overlapping columns if they are all allows or all denies
Summary:
Previously, it was forbidden to have two rule sets with overlapping columns,
since that could introduce an dependency on order of evaluation without
the user having a way to control that order.  This diff permits such rule sets
if the are compatible in a very simple way -- all allows or all denies.
Anything more complicated (even if actually order independent) remains forbidden.

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2745
2021-03-04 11:22:09 -05:00