Commit Graph

576 Commits

Author SHA1 Message Date
Jarosław Sadziński
745ddc21ac (core) Expanding search textbox to full height.
Summary: Expanding search input field to full available height, to make the clickable area bigger.

Test Plan: Manual tests on browserstack

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3145
2021-11-18 23:48:07 +01:00
George Gevoian
05eb7afcb6 (core) Show ACL link in UserManager conditionally
Summary:
The 'Open Access Rules' link is now hidden unless the
UserManager is opened inside a document, and the resource
that users are being managed for is a document.

Test Plan: Browser tests.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3142
2021-11-17 08:43:14 -08:00
Paul Fitzpatrick
c7331e2453 (core) make document reloading cleaner
Summary:
Currently when reloading a document, we may have two sqlite connections
to the document for a small period of time. This diff removes that
overlap.

Test Plan: added test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3140
2021-11-16 16:46:46 -05:00
Alex Hall
561e32fb44 (core) Better logging in DocTriggers
Summary: Added a helper to include lots of metadata in every logging call, added and converted many logging calls.

Test Plan: Existing tests pass

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3136
2021-11-11 15:24:33 +02:00
Alex Hall
e09e919016 (core) Ensure that large changes are processed in full by triggers (for webhooks)
Summary:
Uses the new alwaysPreserveColIds option for action summaries in Triggers.ts.

Triggers.ts is now responsible for generating the summary to make it easy to pass this option. The value of the option is just all colIds mentioned in triggers configured in this document.

Test Plan: Tested adding 200 rows to a subscribed table to ensure the events are not truncated. Also tests batching nicely.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3135
2021-11-10 23:13:55 +02:00
Paul Fitzpatrick
302202b4fb (core) freshen tests for python3
Summary:
Recent python3 changes perturbed timing again, and a few more tests started failing.

Contains an unrelated correction for gvisor running under docker (a useful configuration on macs for debugging gvisor problems, but not supported by throttling code).

Test Plan: updated tests

Reviewers: dsagal, alexmojaki

Reviewed By: dsagal, alexmojaki

Differential Revision: https://phab.getgrist.com/D3129
2021-11-10 10:46:12 -05:00
George Gevoian
08b1286f4f (core) Add column matching to Importer
Summary:
The Importer dialog is now maximized, showing additional column
matching options and information on the left, with the preview
table shown on the right. Columns can be mapped via a select menu
listing all source columns, or by clicking a formula field next to
the menu and directly editing the transform formula.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3096
2021-11-09 12:30:52 -08:00
Jarosław Sadziński
96fa7ad562 (core) Error message on Duplicate Document
Summary: Fixing error message when user can't copy document.

Test Plan: Updated tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3130
2021-11-09 19:12:57 +01:00
Alex Hall
ecb30eebb8 (core) Parsing multiple values in reflists, parsing refs without table data in client
Summary:
Added a new object type code `l` (for lookup) which can be used in user actions as a temporary cell value in ref[list] columns and is immediately converted to a row ID in the data engine. The value contains the original raw string (to be used as alt text), the column ID to lookup (typically the visible column) and one or more values to lookup.

For reflists, valueParser now tries parsing the string first as JSON, then as a CSV row, and applies the visible column parsed to each item.

Both ref and reflists columns no longer format the parsed value when there's no matching reference, the original unparsed string is used as alttext instead.

Test Plan: Added another table "Multi-References" to CopyPaste test. Made that table and the References table test with and without table data loaded in the browser.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3118
2021-11-09 14:41:04 +02:00
Jarosław Sadziński
b6dd066b7f (core) Adding more space for text editor to avoid scrollbars
Summary:
For Windows and Ubuntu (on chrome) texteditor was showing scrollbars for
very long words. Adding more space to fix this issue.

Test Plan: manual

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3122
2021-11-09 12:36:40 +01:00
Alex Hall
45fc46070d (core) Fix converting column to ChoiceList when cells contain JSON arrays with non-string values
Summary: title

Test Plan: Tested manually, I don't think this needs an automated test. Made a text column with a value `[1, 2]` and converted the column to choice list. Previously this threw a JS error that `tag.trim` wasn't a function. Works now, suggests `1` and `2` as choices in the configuration.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3128
2021-11-09 12:53:20 +02:00
George Gevoian
59699bf446 (core) Add additional org items to AppHeader
Summary:
Adds links to manage team and go to billing account in
the org menu (opened by clicking the dropdown in the
top-left corner of Grist). Tweaks some wording of items
in both AppHeader and AccountWidget, and adds a link
to create a new team site to the Site Switcher in both
menus.

Also tweaks the UI of UserManager by adding
an animation when the manager is opened from the
doc access dialog.

Test Plan: Browser tests.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3121
2021-11-05 08:19:51 -07:00
Paul Fitzpatrick
58880d4818 (core) support setting python version of new docs with PYTHON_VERSION_ON_CREATION
Summary:
If PYTHON_VERSION_ON_CREATION is set in the environment, new documents will be created with a specific desired python version (2 or 3).

This diff commits to offering a choice of engine, so the engine for a document no longer starts to initialize until the document has been fetched and read. Staging (and dev, and testing) has been like this for a while.

Test Plan: added test; manual testing of forks/copies etc

Reviewers: dsagal, alexmojaki

Reviewed By: dsagal, alexmojaki

Differential Revision: https://phab.getgrist.com/D3119
2021-11-05 10:51:18 -04:00
Jarosław Sadziński
e8e614c584 (core) Formula UI redesign
Summary:
Redesigning column type section to make it more user-friendly. Introducing column behavior concept.
Column can be either:
- Empty Formula Column: initial state (user can convert to Formula/Data Column)
- Data Column: non formula column with or without trigger (with option to add trigger, or convert to formula)
- Formula Column: pure formula column, with an option to convert to data column with a trigger.

Test Plan: Existing tests.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3092
2021-11-05 13:07:30 +01:00
Paul Fitzpatrick
877542225d (core) mark engine setting as experimental
Summary:
This marks the engine setting in document settings as experimental,
with a skull and cross-bones.

It also makes sure the setting is shown if PYTHON_VERSION_ON_CREATION
is set (this relates to a separate change to set the default python
version to 3).

Test Plan: manual

Reviewers: alexmojaki, dsagal

Reviewed By: alexmojaki, dsagal

Subscribers: anaisconce

Differential Revision: https://phab.getgrist.com/D3120
2021-11-04 20:17:24 -04:00
Jarosław Sadziński
4ddc29fb40 (core) Adding fallback for currency symbol
Summary: Adding fallback for currency formatting on older browsers.

Test Plan: Existing tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3108
2021-11-04 22:39:34 +01:00
Paul Fitzpatrick
db34e1a6d9 (core) tweak throttling to work for gvisor/runsc
Summary:
Grist has, up to now, used a throttling mechanism that allows a sandbox free rein until it starts using above some threshold percentage of a cpu for some time - at that point, we start sending STOP and CONT signals on a duty cycle, with longer and longer STOPped periods until cpu usage is at a threshold. The general idea is to do short jobs quickly, while throttling long jobs (thus unfortunately making them even longer) in order to continue doing other short jobs quickly.

The runsc sandbox is not a single process, there are in fact 5 per sandbox in our setup. Runsc can work with kvm or ptrace. Kvm is not available to us, so we use ptrace. With ptrace, there is one process that is the appropriate one to duty cycle, and another that needs to receive a signal in order to yield. This diff adds the necessary machinery.

This is a conservative change, where I stick with our existing throttling mechanism and adapt it to the new sandbox. It would be reasonable to consider switching throttling. There's a lot the OS allows. We can set a quota for how much cpu a process can use within a given period, for example. However the overall behavior with that would be quite different to what we have, so feels like this would need more discussion.

The implementation contains use of a linux utility `pgrep` since portability is not important (runsc is only available on linux) and there's no node api for enumerating children of a process.

The diff contains some tweaks to `buildtools/contain.sh` to streamline experimenting with Grist and runsc on a mac. It is important for throttling that node and the sandbox processes are in the same process name space, if docker is in between them then some extra machinery is needed (a proxy throttler and a way to communicate with it) which I chose not to implement.

Test Plan: added test; a lot of manual testing

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3113
2021-11-04 17:23:43 -04:00
Paul Fitzpatrick
10a4cbb6bd (core) make document assignment endpoint available via /housekeeping api
Summary:
The /assign endpoint checks if a document is on the desired worker
and moves it if not. This is never done under regular operation, but
is useful when quarantining a misbehaving document.

The endpoint was failing to operate correctly if the requester did
not have access to the document. This diff makes the endpoint
accessible through a /housekeeping route, using the same pattern as
the /force-reload endpoint.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3109
2021-11-04 16:14:21 -04:00
Dmitry S
cce679d928 (core) Add some polish to the billing page, particularly for sign-up.
Summary:
- Change "Continue" button to "Review" (we don't charge immediately,
  first show a review screen)
- Show more informative messages for certain failures with discount
  coupons.
- Focus form elements with error, or at least the part of the form
  containing an error.
- Auto-focus discount input box when it gets toggled on.
- Show warning about URL changes only when subdomain is changed.

Test Plan: Updated tests; tested focus and changed error messages manually.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3115
2021-11-04 14:23:53 -04:00
Dmitry S
0dd4ad34f5 (core) Avoid an error when pasting data with first row shorter than the rest
Summary:
Pasting data like `A\nB\tC` was failing because the first row, used for
checking column type (to handle rich data), was failing on an undefined value.

Discovered while trying out the fix in https://phab.getgrist.com/D3110.

Test Plan: Tested manually. The case mentioned now works as expected.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3111
2021-11-04 10:15:56 -04:00
Alex Hall
dc2aee4be9 (core) Trim trailing whitespace when pasting plain text
Summary: title

Test Plan:
Tested manually. Blank rows at the end are no longer pasted. Pasting multiple columns separated by tabs can still have blank cells in some cells of the final rows.

I don't think this needs an automated test.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3110
2021-11-04 01:26:47 +02:00
Alex Hall
4a70283292 (core) Webhook event queue on redis
Summary:
Push webhook events to redis queue with key based on docId.

Remove events from redis after sending using LTRIM.

Put failed events back on the end of the queue under normal circumstances.

When the event queue gets too long:
- Wait until it gets consumed before continuing.
- Drop failed events (i.e. don't put them back on the end of the queue)
- Limit webhook retries to 5

Test Plan: Tested that interactions with redis are as expected using redis MONITOR command.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3100
2021-11-03 22:43:35 +02:00
Alex Hall
1db138d7ac (core) Change timezone abbreviation parsing
Summary: Allows any timezone abbreviation associated with the given timezone, and simply ignores it. Previously only certain abbreviations worked and they were not unique so using them outside the US was broken.

Test Plan: Added parseDate tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3106
2021-11-03 16:40:41 +02:00
Jarosław Sadziński
3c72639e25 (core) Adding sort options for columns.
Summary:
Adding sort options for columns.
- Sort menu has a new option "More sort options" that opens up Sort left menu
- Each sort entry has an additional menu with 3 options
-- Order by choice index (for the Choice column, orders by choice position)
-- Empty last (puts empty values last in ascending order, first in descending order)
-- Natural sort (for Text column, compares strings with numbers as numbers)
Updated also CSV/Excel export and api sorting.
Most of the changes in this diff is a sort expression refactoring. Pulling out all the methods
that works on sortExpression array into a single namespace.

Test Plan: Browser tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: dsagal, alexmojaki

Differential Revision: https://phab.getgrist.com/D3077
2021-11-03 15:31:39 +01:00
Alex Hall
0f946616b6 (core) Parsing pasting in reflist columns
Summary: Handle reflist columns in ViewFieldRec.parseValue

Test Plan: Reused section of test of reference columns

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3105
2021-11-03 14:37:08 +02:00
Cyprien P
ab7af2b2ef (core) Fix bars order in chart bars order when multiseries is on.
Summary:
 - Grouping series may result in series with inconsistent number of values. This can result in inconsistent ordering of the bars displayed by plotly.
 - This diff fixes it by consolidating grouped series by adding unll values for each missing xvalues in the series.

Here a is a minimal example of that bug:
{F36639}

Test Plan: Includes new nbrowser test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3085
2021-11-03 08:58:10 +01:00
Alex Hall
d63da496a8 (core) Value parsing for refs, parsing data entry for numbers
Summary:
Handle reference columns in ViewFieldRec.valueParser.

Extracted code for reuse from ReferenceEditor to look up values in the visible column. While I was at it, also extracted a bit of common code from ReferenceEditor and ReferenceListEditor into a new class ReferenceUtils. More refactoring could be done in this area but it's out of scope.

Changed NTextEditor to use field.valueParser, which affects numeric and reference fields. In particular this means numbers are parsed on data entry, it doesn't change anything for references.

Test Plan:
Added more CopyPaste testing to test references.

Tested entering slightly formatted numbers in NumberFormatting.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3094
2021-11-01 19:31:52 +02:00
Dmitry S
f0da3eb3b2 (core) Select previous search input when clicking the search icon
Summary:
Searching with the keyboard shortcut selected the previous search text in the
search box, but using the Search icon did not. A user reported it as an
inconvenience: having to manually delete the value before searching for a new
one.

Test Plan: Verified manually

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3102
2021-11-01 01:28:08 -04:00
Dmitry S
c5db65d1d2 (core) Process new user info in /welcome/info post without waiting for it to be written to the collecting document
Summary:
The document collecting new user info
(https://docs.getgrist.com/doc/GristNewUserInfo) got very slow, taking 40+
seconds for cold open. Sign-up submissions had to wait this time to proceed to
next step, because they waited for the write to this doc, which was blocked on
the Calculate action to complete.

Two changes were made: one to remove all expensive columns and summaries in the
actual doc, so the doc is back to opening in single seconds, and times should
be acceptable now.

The second change is this diff: to avoid waiting for the write step, so that it
doesn't affect users even if it gets slow again.

Test Plan: Existing test continues to work with a minor reliability tweak.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3103
2021-10-31 13:54:31 -04:00
Paul Fitzpatrick
6c53f3e820 (core) add an option to action summarization to preserve columns entirely
Summary:
Action summaries by default will drop rows in bulk changes, keeping only a few of them as examples. This diff allows overriding that, or selectively preserving some columns in their entirety.

This is intended for use with webhooks.

Test Plan: added test

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3098
2021-10-28 21:52:19 -04:00
Paul Fitzpatrick
35e18cc0ad (core) fix bug where sharing doc with everyone@ as viewer made it unlisted for site viewers
Summary:
Shares of the same role (e.g. viewer) at different levels could interact for a resource (e.g. a doc) shared with everyone@, potentially blocking the listing of that resource. This diff removes the interaction.

The permission of a user on a resource is calculated by finding all acl rules that link that resource to a group to which the user belongs, or to a group that has a subgroup to which the user belongs, etc, and then bitwise-or-ing the permissions on the acl rules. A later wrinkle was to allow public sharing via special users. A still later wrinkle was to avoid listing resources if they were only shared with the special everyone@ user, while allowing access to them if user has their full link. That wrinkle had a bug, where if e.g. a doc were shared with everyone@ as a viewer, and the org the doc was in was shared with someone@ as a viewer, and the doc inherited the org permissions via a workspace, then that doc would end up not being listed.

The fix is straightforward enough, but needs different code for postgres and sqlite, and is a bit verbose because we unwrap subgroups to a few levels rather than doing recursion (which looks cleaner but was slower in benchmarks).

Test Plan: added test that fails without this fix

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3095
2021-10-28 12:48:31 -04:00
Alex Hall
e58df5df5b (core) ValueParser for Date columns
Summary: Adds parseDateStrict function based on parseDate, uses it in DateParser subclass of ValueParser.

Test Plan:
Tweaked parseDate test to check parseDateStrict.

Extended test in CopyPaste to test parsing dates as well as numbers.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3088
2021-10-26 13:03:19 +02:00
Dmitry S
65e743931b (core) Convert CopySelection and tableUtil to typescript
Summary:
- This should make these easier to work with and make changes to.
- Removes one unused method.

Test Plan: No changes of behavior, existing tests should pass.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3091
2021-10-25 18:55:26 -04:00
Paul Fitzpatrick
f7c9919120 (core) annotate shares listed in UserManager for documents
Summary:
This gives more guidance to users when editing document shares in the UserManager dialog.

  * For a document on a team site, any shares with team members are marked `Team member`.
  * Shares that count as external collaborators are marked for documents on a team or personal site as `collaborator` (personal site) or `outside collaborator` (team site).
  * Collaborators are marked `1 of 2`, `2 of 2`, and then `limit exceeded`.
  * On a team site, links are offered for each collaborator to add them to the team. The links lead to a prefilled dialog for managing team membership which can be confirmed immediately, allowing the user to continue without interruption.
  * On a personal site, for the last collaborator and beyond, a link is added for creating a team. This isn't seamless since creating a team involves billing etc.

There's a small unrelated tweak in tests to remove a confusing import from `test/browser` in `test/server`.

One thing I didn't get to is checking if owner of doc is owner of site. If they aren't, they may try to add a member and be denied at that point - it would be more polite to change messaging earlier for them.

Test Plan: added and updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3083
2021-10-25 14:29:19 -04:00
Dmitry S
f2f4fe0eca (core) Add LogMethods helper and use it for more JSON data in logs. Reduce unhelpful logging.
Summary:
- Sharing, Client, DocClients, HostingStorageManager all include available info.
- In HostingStorageManager, log numSteps and maxStepTimeMs, in case that helps
  debug SQLITE_BUSY problem.
- Replace some action-bundle logging with a JSON version aggregating some info.
- Skip logging detailed list of actions in production.

Test Plan: Tested manually by eyeballing log output in dev environment.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3086
2021-10-25 10:25:18 -04:00
Jarosław Sadziński
8eeeae7fbf (core) Fixing scrollbars on ace editor for windows users.
Summary: Fixing formula editor scrollbars for windows users.

Test Plan: Manual

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3087
2021-10-25 10:56:43 +02:00
Alex Hall
99878c08ed (core) Add ValueParser, use when pasting
Summary:
Add ValueParser file, base class, and subclasses for column types. Only NumericParser is used for now.

Add valueParser field to ViewFieldRec.

Use valueParser when parsing pasted text data in Grid and Detail views.

Test Plan: Add test to nbrowser CopyPaste suite, copying into a numeric column with different currency and locale settings.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3082
2021-10-21 21:43:19 +02:00
Alex Hall
e900f39da3 (core) Log statistics about table sizes
Summary:
Record numbers of rows, columns, cells, and bytes of marshalled data for most calls to table_data_from_db

Export new function get_table_stats in the sandbox, which gives the raw numbers and totals.

Get and log these stats in ActiveDoc right after loading tables, before Calculate, so they are logged even in case of errors.

Tweak logging about number of tables, especially number of on-demand tables, to not only show in debug logging.

Test Plan: Updated doc regression tests, that shows what the data looks like nicely.

Reviewers: dsagal, paulfitz

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3081
2021-10-21 17:54:20 +02:00
George Gevoian
f2e11a5329 (core) Migrate to Stripe v8 + implement discount codes
Summary:
New plan signups now include a discount code field in
the signup form. If a valid discount code is entered, a
discount will be applied on the confirmation page.

Test Plan: Browser and server tests.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3076
2021-10-20 12:34:03 -07:00
Alex Hall
4894631ba4 (core) Generic number parsing functionality.
Summary:
Added NumberParse.ts, counterpart of NumberFormat.ts.

Contains generic functionality for parsing numbers formatted by Intl.NumberFormat, not tied to documents or anything.

This doesn't change any actual behaviour, applying this parsing when pasting/typing in numeric columns will be a separate diff.

Test Plan: New file with extensive unit tests.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3078
2021-10-19 23:19:13 +02:00
Paul Fitzpatrick
dd0f1be117 (core) get all tests working under python3/gvisor
Summary:
This verifies that all existing tests are capable of running under python3/gvisor, and fixes the small issues that came up. It does not yet activate python3 tests on all diffs, only diffs that specifically request them.

 * Adds a suffix in test names and output directories for tests run with PYTHON_VERSION=3, so that results of the same test run with and without the flag can be aggregated cleanly.
 * Adds support for checkpointing to the gvisor sandbox adapter.
 * Prepares a checkpoint made after grist python code has loaded in the gvisor sandbox.
 * Changes how `DOC_URL` is passed to the sandbox, since it can no longer be passed in as an environment variable when using checkpoints.
 * Uses the checkpoint to speed up tests using the gvisor sandbox, otherwise a lot of tests need more time (especially on mac under docker).
 * Directs jenkins to run all tests with python2 and python3 when a new file `buildtools/changelogs/python.txt` is touched (this diff counts as touching that file).
 * Tweaks miscellaneous tests
   - some needed fixes exposed by slightly different timing
   - a small number actually give different results in py3 (removal of `u` prefixes).
   - some needed a little more time

The DOC_URL change is not the ultimate solution we want for DOC_URL. Eventually it should be a variable that gets updated, like the date perhaps. This is just a small pragmatic change to preserve existing behavior.

Tests are run mindlessly as py3, and for some tests it won't change anything (e.g. if they do not use NSandbox). Tests are not run in parallel, doubling overall test time.

Checkpoints could be useful in deployment, though this diff doesn't use them there.

The application of checkpoints doesn't check for other configuration like 3-versus-5-pipe that we don't actually use.

Python2 tests run using pynbox as always for now.

The diff got sufficiently bulky that I didn't tackle running py3 on "regular" diffs in it. My preference, given that most tests don't appear to stress the python side of things, would be to make a selection of the tests that do and a few wild cards, and run those tests on both pythons rather then all of them. For diffs making a significant python change, I'd propose touching buildtools/changelogs/python.txt for full tests. But this is a conversation in progress.

A total of 6886 tests ran on this diff.

Test Plan: this is a step in preparing tests for py3 transition

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3066
2021-10-18 17:44:15 -04:00
Jarosław Sadziński
3e661db38c (core) Adding schema validation for records endpoint
Summary:
Adding validation for api /records endpoint, that checks if the json payload is valid.
Modifying POST /records endpoint to allow creating blank or partial records.

Test Plan: Updated tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3061
2021-10-18 21:40:50 +02:00
Alex Hall
276adc5f51 (core) Starting to make webhooks more robust
Summary:
- Puts events on a queue in memory and ensures they are sent in the order they were generated.
- Makes the caller (Sharing.ts) wait until changed records have been fetched from the DB, but allows it to continue after while remaining work happens asynchronously.
- Gathers all new webhook events into an array so they can be backed up to the queue on redis in a single command (in a future diff).
- Uses changes in isReady to determine event type, no more 'existed before'

The structure of the code has changed a lot, so I think the scope of the diff needs to stop here. Lots of work is still deferred in TODOs.

Test Plan: Updated existing test. Actually dropped testing of retry on failures and slowness because it no longer made sense to keep that as part of the current test, so a new test for that will be added in a future diff.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3074
2021-10-15 20:01:15 +02:00
Cyprien P
e3801a5eb9 (core) Fix pie sectors order according to section sort spec
Summary:
Plotly sorts pie charts sectors by default and that is overiding the
section ordering. This diff fixes that by passing setting .sort to
false (thus disabling reordering) when there is a sort spec going on.

Issue was reported by user: https://gristlabs.getgrist.com/k1f3bMzUvitZ/User-Feedback#a1.s3.r333.c19

Test Plan: Added nbrowser test

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3075
2021-10-15 08:38:48 +02:00
Dmitry S
c8da5c7356 (core) Fix Card view linked to summary to show card navigation buttons
Test Plan: Added a test case to verify that nav buttons are now shown and work.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3073
2021-10-14 12:26:54 -04:00
Jarosław Sadziński
67ec52365a (core) Showing links in text cells
Summary:
When there is a link in a text cell (and formula cells), it will be
rendered with a little clickable icon wrapped in the anchor tag
with a proper link. Only links that starts with https? will be
rendered as links.

Links are shown in a Text and Formula fields, inside a GridView,
CardView and in the Import preview dialog.

Test Plan: Browser tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: dsagal, alexmojaki

Differential Revision: https://phab.getgrist.com/D3070
2021-10-13 23:22:03 +02:00
Dmitry S
16eb158673 (core) Make ProfileDialog wait consistently for work to finish before closing.
Summary: - Added functionality to modal.ts to allow pending work to delay the closing of the dialog.

Test Plan: Added a test case that tickled a failure previously.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3071
2021-10-13 16:30:43 -04:00
Alex Hall
9d1cc89dc9 (core) Strip invalid characters from table name in excel import
Summary: Add sanitizeWorksheetName function, pass result to library function addWorksheet where error was raised.

Test Plan: Added unit test for sanitizeWorksheetName function, updated a fixture document to use a messy table name.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3072
2021-10-11 17:47:12 +02:00
Alex Hall
a64fb105e3 (core) Use GristObjCode in CellValue
Summary: Makes type checking a bit stronger

Test Plan: it just has to compile

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3065
2021-10-11 14:11:32 +02:00
George Gevoian
62db263d1f (core) Add diff preview to Importer
Summary:
Updates the preview table in Importer to show a diff of changes
when importing into an existing table and updating existing records.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3060
2021-10-08 14:15:07 -07:00
Paul Fitzpatrick
d635c97686 (core) flesh out "View As" feature
Summary:
The users shown by the "View As" button are now drawn from more sources:
 * There are users the document is shared with. This has been rationalized, the behavior was somewhat erratic. If the user is not an owner of the document, the only user of this kind that will be listed is themselves.
 * There are users mentioned in any user attribute table keyed by Email. If name and access columns are present, those are respected, otherwise name is taken from email and access is set to "editors".
 * There are example users provided if there are not many other users available.

Test Plan: added and extended tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3045
2021-10-08 12:00:40 -04:00
Paul Fitzpatrick
07558dceba (core) avoid censorship for one client clobbering data for another client
Summary:
When filtering document updates to send to clients after a change,
censorship of individual cells was being applied to state shared
across the clients. This diff eliminates that shared state, and
extends testing of broadcasts to check different orderings.

Test Plan:
extends a test to tickle a reported bug, and gives
DocClients a knob to control message order needed to tickle
the bug reliably.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3064
2021-10-07 23:21:07 -04:00
Paul Fitzpatrick
df318ad6b3 (core) add a mac-specific sandbox for development
Summary:
docker is slow on macs, so use native sandbox-exec by
default for tests involving python3 on macs.

Test Plan: updated test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3068
2021-10-07 14:14:25 -04:00
Jarosław Sadziński
26766fd4ab (core) Removing error styles from user messages
Summary:
Removing error styles from user messages. Only unexpected errors are styled with red icon and border.
Removing reportSuccess message - leaving it for another diff.

Test Plan: manual tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3063
2021-10-07 17:20:52 +02:00
Jarosław Sadziński
a2e066176c (core) Label editor for Choice/ChoiceList column editor
Summary:
Allowing a user to change labels' in Choice/ChoiceList entry editor. For updated
entries, renaming those values in all cells in the column.

Test Plan: Updated tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3057
2021-10-07 11:55:38 +02:00
Dmitry S
cf7a3153f9 (core) When hidden pages are present in the page list, allow removing them
Summary:
After an incomplete import, any GristHidden_* tables will show up in the page
list, but may not be removable if there is only one non-hidden table remaining.
Such tables should still be removable in this case.

Test Plan: Added a test case

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3058
2021-10-05 15:39:42 -04:00
Jarosław Sadziński
40ddb57dfc (core) Adding colors to toast notification
Summary:
Styling toast notification. Adding colors and icons.
In Grist, changed the default style for errors (will be shown in red), and a style for
Linked copied to clipboard (will be shown in Green).
All other colors are not used currently, left for another diff.

Test Plan: manual

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3053
2021-10-05 10:19:25 +02:00
Dmitry S
43a62e7254 (core) Fix autocomplete selection when options include emoji
Summary:
Searching for matching choices was using nativeCompare for a binary search,
while the list was sorted according to localeCompare. This was causing the
search to fail for some strings when the two orderings differ.

Test Plan: Tested manually

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3059
2021-10-04 22:58:00 -04:00
George Gevoian
e1780e4f58 (core) Migrate import code from data engine to Node
Summary:
Finishing imports now occurs in Node instead of the
data engine, which makes it possible to import into
on-demand tables. Merging code was also refactored
and now uses a SQL query to diff source and destination
tables in order to determine what to update or add.

Also fixes a bug where incremental imports involving
Excel files with multiple sheets would fail due to the UI
not serializing merge options correctly.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3046
2021-10-04 10:27:00 -07:00
Dmitry S
7e07f0ce56 (core) For grist_sid*_status cookie, remember to set the path
Test Plan: Only tested manually that path is included.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3056
2021-10-04 09:13:47 -04:00
Paul Fitzpatrick
8853e095bb (core) fix core build, and make smoke test more effective
Summary:
This makes the `core` test operate on a directory outside the
jenkins workspace, so that packages in the workspace don't
interfere with the test and obscure errors.

It also includes a small type fix for the `core` build.

Test Plan: updating a test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3054
2021-10-01 13:34:47 -04:00
Dmitry S
1517dca644 (core) Implement DiscourseConnect to enable easy sign-in to community forum
Summary:
- Update cookie module, to support modern sameSite settings
- Add a new cookie, grist_sid_status with less-sensitive value, to let less-trusted subdomains know if user is signed in
- The new cookie is kept in-sync with the session cookie.
- For a user signed in once, allow auto-signin is appropriate.
- For a user signed in with multiple accounts, show a page to select which account to use.
- Move css stylings for rendering users to a separate module.

Test Plan: Added a test case with a simulated Discourse page to test redirects and account-selection page.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3047
2021-10-01 11:24:22 -04:00
Paul Fitzpatrick
b3b7410ede (core) open documents without blocking on data engine
Summary:
With this diff, when a user opens a Grist document in a browser, they will be able to view its contents without waiting for the data engine to start up. Once the data engine starts, it will run a calculation and send any updates made. Changes to the document will be blocked until the engine is started and the initial calculation is complete.

The increase in responsiveness is useful in its own right, and also reduces the impact of an extra startup time in a candidate next-generation sandbox.

A small unrelated fix is included for `core/package.json`, to catch up with a recent change to `package.json`.

A small `./build schema` convenience is added to just rebuild the typescript schema file.

Test Plan: added test; existing tests pass - small fixes needed in some cases because of new timing

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3036
2021-10-01 10:18:56 -04:00
Jarosław Sadziński
42910cb8f7 (core) Extending Google Drive integration scope
Summary:
New environmental variable GOOGLE_DRIVE_SCOPE that modifies the scope
requested for Google Drive integration.
For prod it has value https://www.googleapis.com/auth/drive.file which leaves
current behavior (Grist is allowed only to access public files and for private
files - it fallbacks to Picker).
For staging it has value https://www.googleapis.com/auth/drive.readonly which
allows Grist to access all private files, and fallbacks to Picker only when the file is
neither public nor private).
Default value is https://www.googleapis.com/auth/drive.file

Test Plan: manual and existing tests

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3038
2021-10-01 10:47:12 +02:00
Cyprien P
a0c53f2b61 (core) Fix chart when x axis is set to a choice list column
Summary:
Bug reported by user: https://gristlabs.getgrist.com/doc/check-ins/p/3#a1.s7.r1183.c19p

Setting x axis to a column of type ChoiceList was breaking chart.

This diff fixes that by splitting the record into several records: one for each choice.

`test/nbrowser/ChartView1.ts` was becoming too big and long to run, so this diff introduces `test/nbrowser/ChartView2.ts` to add more test and `test/nbrowser/chartViewTestUtils.ts` to put all utilities or testing charts.

Test Plan: Adds new test.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3041
2021-10-01 10:03:27 +02:00
Paul Fitzpatrick
383b8ffbf0 (core) add a tool for deleting a user
Summary:
This adds a `user:delete` target to the `cli.sh` tool. The desired user will be deleted from our database, from sendgrid, and from cognito.

There is code for scrubbing the user from team sites, but it isn't yet activated, I'm leaving finalizing and writing tests for it for follow-up.

Test Plan: tested manually

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3043
2021-09-29 12:08:23 -04:00
Paul Fitzpatrick
876a0298a2 (core) do not look at content of recent actions when loading documents
Summary:
This removes the need for any information drawn from the content of recent actions when loading a document.

The undo/redo system does need some facts about recent actions up front. But that system has an important restriction: only actions a particular client is known to have generated can be undone by that client.

So in this diff, as we store which client has performed an action, we also store the few pieces of metadata about that action that the undo/redo system needs: `linkId`, `otherId`, `rowIdHint`, `isUndo` fields. These are all small integers (or in one case a boolean).

An existing limitation is that information about which client has performed which action is stored in memory in the worker, and not persisted anywhere. This diff does not change that limitation, meaning that undos continue to not survive a worker transition. A reasonable way to deal with that would be to back the store with redis.

Test Plan: existing tests pass

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3044
2021-09-29 11:27:02 -04:00
Paul Fitzpatrick
0fffe918c1 (core) don't garble document url in SELF_HYPERLINK on forks
Summary: There was a bad regex processing the document url passed to the sandbox.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3048
2021-09-28 16:53:48 -04:00
Alex Hall
8c1f8bc9a6 (core) Test webhooks
Summary:
Tests DocApi endpoints _subscribe and _unsubscribe, including various bad inputs.

Tests that webhooks are sent to a test express server, with retrying on failure, filtered by event type, and waiting for isReadyColumn.

Test Plan: this

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3042
2021-09-28 01:13:54 +02:00
Jarosław Sadziński
8684c9e930 (core) Adding traceback to trigger formulas
Summary:
Traceback is available on the Creator Panel in the formula editor. It is evaluated the same way as for normal formulas.
In case when the traceback is not available, only the error name is displayed with information that traceback is not available.
Cell with an error, when edited, shows the previous valid value that was used before the error happened (or None for new rows).
Value is stored inside the RaisedException object that is stored in a cell.

Test Plan: Created tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki, dsagal

Differential Revision: https://phab.getgrist.com/D3033
2021-09-27 17:12:39 +02:00
Jarosław Sadziński
048c8ee165 (core) Extending default locale list
Summary: Adding more locale codes to support more countries in document settings

Test Plan: existing tests

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3018
2021-09-24 15:10:13 +02:00
Alex Hall
52fd28815e (core) Raise syntax errors that Python can format nicely to show the location
Summary: Update _create_syntax_error_code to raise an error with similar arguments to the real arguments it already has, with our modifications.

Test Plan: Updated python unit tests

Reviewers: jarek, dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3040
2021-09-24 15:07:09 +02:00
Dmitry S
fb583f303a (core) Support 'new' row in anchor links.
Summary:
- Anchor links with row of 'new' could be created but weren't parsed or used
  correctly. This fixes it.
- Also adds UIRowId type for row IDs which includes the special 'new' row. It's
  already been used in places as `number|'new'`, this diff gives it a name usable in app/common
  (it doesn't touch another name, RowId, that's been available in app/client).

Test Plan: Added a test assert for anchor links to new row

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3039
2021-09-24 09:01:10 -04:00
Alex Hall
3c4d71aeca (core) Initial webhooks implementation
Summary:
See https://grist.quip.com/VKd3ASF99ezD/Outgoing-Webhooks

- 2 new DocApi endpoints: _subscribe and _unsubscribe, not meant to be user friendly or publicly documented. _unsubscribe should be given the response from _subscribe in the body, e.g:

```
$ curl -X POST -H "Authorization: Bearer 8fd4dc59ecb05ab29ae5a183c03101319b8e6ca9" "http://localhost:8080/api/docs/6WYa23FqWxGNe3AR6DLjCJ/tables/Table2/_subscribe" -H "Content-type: application/json" -d '{"url": "https://webhook.site/a916b526-8afc-46e6-aa8f-a625d0d83ec3", "eventTypes": ["add"], "isReadyColumn": "C"}'
{"unsubscribeKey":"3246f158-55b5-4fc7-baa5-093b75ffa86c","triggerId":2,"webhookId":"853b4bfa-9d39-4639-aa33-7d45354903c0"}
$ curl -X POST -H "Authorization: Bearer 8fd4dc59ecb05ab29ae5a183c03101319b8e6ca9" "http://localhost:8080/api/docs/6WYa23FqWxGNe3AR6DLjCJ/tables/Table2/_unsubscribe" -H "Content-type: application/json" -d '{"unsubscribeKey":"3246f158-55b5-4fc7-baa5-093b75ffa86c","triggerId":2,"webhookId":"853b4bfa-9d39-4639-aa33-7d45354903c0"}'
{"success":true}
```

- New DB entity Secret to hold the webhook URL and unsubscribe key
- New document metatable _grist_Triggers subscribes to table changes and points to a secret to use for a webhook
- New file Triggers.ts processes action summaries and uses the two new tables to send webhooks.
- Also went on a bit of a diversion and made a typesafe subclass of TableData for metatables.

I think this is essentially good enough for a first diff, to keep the diffs manageable and to talk about the overall structure. Future diffs can add tests and more robustness using redis etc. After this diff I can also start building the Zapier integration privately.

Test Plan: Tested manually: see curl commands in summary for an example. Payloads can be seen in https://webhook.site/#!/a916b526-8afc-46e6-aa8f-a625d0d83ec3/0b9fe335-33f7-49fe-b90b-2db5ba53382d/1 . Great site for testing webhooks btw.

Reviewers: dsagal, paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3019
2021-09-23 14:35:39 +02:00
Alex Hall
de76cc48d1 (core) Move cursor to new record in link target when selecting new record in link source
Summary: Update LinkingState._makeSrcCellGetter to account for 'new'

Test Plan: Extended test in RightPanelSelectBy.ts

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3037
2021-09-22 22:57:05 +02:00
Cyprien P
33f056a187 (core) Fix js errors when (un)summarizing underlying table of charts
Summary:
Used to throw js errors: `Resize must be passed a displayed plot div element.`

Summarizing (or unsummarizing) causes the ChartView view instance to
be replace by a new one in the view layout.
However, the problem is that the old view instance get disposed only
after the new view instance is added to the view layout.
This causes the old view layout to try to resize chart while chart dom
has been removed from the dom (which Plotly does not support).

This diff fixes it by checking the the chart dom elemnt is still in
the dom before making the plotly call to resize the chart.

TODO: It feels weird that the old view instance gets disposed after
the new one get added. Maybe we should check that also.

Test Plan: New test added.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3035
2021-09-22 17:04:52 +02:00
Dmitry S
a974dd839c (core) Improve time-parsing for DateTime columns, which was unusable with some formats (e.g. tz-containing ones)
Test Plan: Added a test with lots of time-parsing cases.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3034
2021-09-21 10:02:59 -04:00
Alex Hall
1a8abdcd96 (core) recursiveMoveToCursorPos with new kinds of linking involving lists
Summary: Constructs a ClientQuery in a similar way to LinkingState to handle lists.

Test Plan: Extended SelectBySummary and SelectByRefList tests.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3030
2021-09-20 23:13:24 +02:00
Dmitry S
cd241a633a (core) In make-copy dialog, ensure it's non-saveable while fetching workspaces after a switch of the destination org
Summary:
While switching the destination site in the Duplicate Document dialog, there
were times when it was saveable even though destination workspaces were still
being fetched. This sometimes causes a test failure, with the document getting
saved to a workspace from the previously-selected org.

Test Plan:
Tested manually; reproduced by adding a conditional artificial delay
in _updateWorkspaces helper.

Reviewers: paulfitz, georgegevoian

Reviewed By: paulfitz, georgegevoian

Differential Revision: https://phab.getgrist.com/D3032
2021-09-20 11:54:06 -04:00
Cyprien P
a1480faa09 (core) Changes X-AXIS to LABEL in the axis config when chart is a pie chart
Summary:
Also fixes issue with group data options when switching to pie chart.

Issue was that if the group data picker was on, switching to the pie
chart was not hiding it.

Test Plan: Adds more tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3028
2021-09-20 08:47:23 +02:00
Alex Hall
80bf4ea2a3 (core) Get first unicode character (e.g. 📝 ) for page label, not 1st byte
Summary:
📝

https://community.getgrist.com/t/feature-request-custom-icon-favicon-for-each-document/210/2

```
>>> json.dumps("📝")
'"\\ud83d\\udcdd"'
>>> json.dumps("")
'"\\u2728"'
```

```
> "📝 abc"[0]
'\uD83D'
> "📝 abc"[1]
'\uDCDD'
> Array.from("📝 abc")
['📝', ' ', 'a', 'b', 'c']
> Array.from("📝 abc")[0]
'📝'
```

Test Plan: no

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3026
2021-09-17 12:59:53 +02:00
Paul Fitzpatrick
7907467dbc (core) treat summary tables like formulas for access control purposes
Summary:
This unsets the `direct` flag for actions emitted when summary tables are updated. That means those actions will be ignored for access control purposes. So if a user has the right to change a source table, the resulting changes to the summary won't result in the overall action bundle being forbidden.

I don't think I've actually seen the use case that inspired this issue being filed. I could imagine perhaps a user forbidden from creating rows globally making permitted updates that could add rows in a summary (and it being desirable to allow that).

Test Plan: added tests

Reviewers: jarek

Reviewed By: jarek

Subscribers: dsagal, alexmojaki, jarek

Differential Revision: https://phab.getgrist.com/D3022
2021-09-16 18:44:50 -04:00
George Gevoian
e5ebc4668c (core) Disable mousedown on token delete button
Summary:
This change prevents dragging tokens when the cursor is over the
delete button. Now, trying to drag after mousedown over the delete
button will do nothing if the cursor is released outside the delete
button, and will delete if the cursor is released inside the delete
button.

Test Plan: Tested manually.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3024
2021-09-16 10:45:06 -07:00
Paul Fitzpatrick
d5a7fb23fe (core) apply SchemaEdit flag to metadata changes in general
Summary:
A user without SchemaEdit permission was able to reorder pages, since
this changes _grist_Pages, and that table was left under control of
regular access rules.  This diff tightens things up, to require
SchemaEdit for all metadata edits.  The one remaining exception is
_grist_Attachments, which needs some reworking to play well with
granular access.

Test Plan: extended test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3025
2021-09-16 13:36:20 -04:00
Cyprien P
4fcdd2ba07 (core) Fix y-axis blinking in chart view configuration
Summary:
This is a follow up diff for https://phab.getgrist.com/D3021.  Y-axis
draggable list used to blink when user changed either one of the x
axis or groupdata column.

This was due to the fact that all of theses axis are stored into the
same array and changing one of them changes the whole array even
though items relative to the y-axis actually were not changing.

This diff addresses this issue by 1) being carefull at not updating
the array of items when the changes do not impact y axis. And 2) by
adding a freeze observable allowing to freeze the draggable list of
y-axis while actions are being treated on the server.

Test Plan:
Catching such bug is hard, and given that it's only look and fill, maybe not worth the time and effort.

Tested manually though.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3023
2021-09-16 18:18:28 +02:00
George Gevoian
8a7edb6257 (core) Enable incremental imports
Summary:
The import dialog now has an option to 'Update existing records',
which when checked will allow for selection of 1 or more fields
to match source and destination tables on.

If all fields match, then the matched record in the
destination table will be merged with the incoming record
from the source table. This means the incoming values will
replace the destination table values, unless the incoming
values are blank.

Additional merge strategies are implemented in the data
engine, but the import dialog only uses one of the
strategies currently. The others can be exposed in the UI
in the future, and tweak the behavior of how source
and destination values should be merged in different contexts,
such as when blank values exist.

Test Plan: Python and browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3020
2021-09-16 09:15:54 -07:00
Paul Fitzpatrick
a543e5194a (core) add a python3 button
Summary: This adds a dropdown to the document settings model in staging/dev to set the python engine to Python2 or Python3. The setting is saved in `_grist_DocInfo.documentSettings.engine`.

Test Plan: tested manually for now - separate diff needed to add runsc to jenkins setup and make this testable

Reviewers: dsagal, alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3014
2021-09-16 10:06:04 -04:00
Cyprien P
2cf2088373 (core) New chart view axis conf with picker for each of X,Y and group by
Summary:
Chart view used to rely on the same view field configuration as used in any other widget.

This diff allows to explicitely select X-AXIS, Y-AXIS and group by column with column picker.

As charts supports several y-axis, we still use a draggable list to arrange them.

Diff also fix doc to the `insertPositions` function.

Test Plan: Updated the relevant test.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3021
2021-09-16 09:10:54 +02:00
Paul Fitzpatrick
3e5a292cde (core) add tests for site deletion
Summary: This tests site deletion with and without a plan.

Test Plan: adding tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3017
2021-09-14 10:03:18 -04:00
Dmitry S
70160fe01c (core) Rename page option 'Duplicate' to 'Duplicate Page', and shorten warning to be more noticeable
Test Plan: Changes too minor to affect tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3016
2021-09-10 11:25:24 -04:00
Paul Fitzpatrick
ddcd08e147 (core) add a cli tool for deleting sites
Summary:
This adds a `site:delete` target to `cli.sh` for deleting sites. Sites should be specified by numeric org id, and for confirmation their name also needs to be given.

All the docs in the site are deleted permanently, and the workspaces, and the site, and the stripe customer (if any).

Test Plan: manual

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3015
2021-09-10 11:02:26 -04:00
Dmitry S
b716a57e31 (core) Avoid loading the Grist frontend app more than once.
Summary:
This can happen thanks to some badly-behaved extensions (e.g. lingvanex), and
results in errors such as "Cannot redefine property: isInput".

Test Plan:
Tested manually that Grist fails to load in Chrome with Lingvanex
extension, and loads successfully with this change.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3013
2021-09-07 16:40:40 -04:00
George Gevoian
0717ee627e (core) Relocate export urls to /download/
Summary:
Moves CSV and XLSX export urls under /download/, and
removes the document title query parameter which is now
retrieved from the backend.

Test Plan: No new tests. Existing tests that verify endpoints still function.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3010
2021-09-02 09:36:33 -07:00
Cyprien P
cecebded1f (core) Fix bar chart weirdness when x axis has redundant values
Summary:
Bar chart was a bit broken when there were redundant values on the X axis: the bars’s height maps to the sum of all the corresponding y values, when the data that shows up on hover is only the last one.

It seems that plotly does not support redundant values in the x axis and in all Plotly examples (implementation relies on plotly) x values only have unique values.

This diff, fixes by making sure x axis has unique values. If user actually wants to plot groups, they'll have to use a summary charts.

Test Plan: tested manually

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3011
2021-09-01 20:58:29 +02:00
Jarosław Sadziński
7e78ebfeeb (core) Subdomain can start from a number
Summary: Allowing subdomain names to start from a number.

Test Plan: Updated tests

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3009
2021-09-01 18:16:57 +02:00
George Gevoian
a825115c04 (core) Limit number of errors shown on narrow screens
Summary: We still show up to 5 on regular-width screens.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3008
2021-08-31 12:15:31 -07:00
George Gevoian
ef5da42378 (core) Update export CSV and Excel endpoints
Summary:
The endpoints for exporting CSV and Excel are now under
/api/docs/:docId/ and are forwarded to a doc worker for export.

The Share Menu has been updated to use the new endpoints.

Test Plan: No new tests. Existing tests that verify endpoints work correctly.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3007
2021-08-31 10:47:24 -07:00
Alex Hall
29dd33a45c (core) Add other direction of linking by reflist
Summary:
Allows selecting by a reflist in another table. This generalises cursor-linking with a ref column, but now it's filter linking.

Added another case to LinkingState where the source column is a reflist to the target table, filtering by the id column.

Updated convertQueryFromRefs and related functions to handle this since the id column has no column ref. In this case the string 'id' is used instead of a number.

LinkingState also checks if the source value is a reflist and uses that as the list of filter values instead of a single-element list of the cell value.

Indirect linking also works, where the source and target columns both are both references to the same table. This was the plan for a source reflist and target ref column.
I was surprised to see it also works perfectly when both columns are reflists, and it filters rows where there's an intersection!

Adding rows to the target section using the selected source record for default values is iffy. When filtering by row ID, there's no column for defaults, so the new row disappears.
For a source reflist and target ref, the first value of the reflist is the default, which is okayish. When both are reflists, the full source reflist is the default for the target column.
This seems like a bit much but just using the first value seems a bit arbitrary when there's room for all of them?

While doing all this I noticed an unrelated bug which I fixed as I was refactoring. Previously cursor linking based on a reference column did not update the cursor in the link target
when the value of the selected reference cell changed. Now cursor linking uses a floating row model like most other cases to observe the value correctly.

Test Plan: Extended SelectByRefList test and fixture, added previously failing test to RightPanelSelectBy.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3004
2021-08-30 17:39:15 +02:00
Paul Fitzpatrick
08d0ee65d1 (core) tweak how search bar places cursor on sections
Summary: It is important for linking to be maintained, or user could be gravely misled about material in other sections.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3003
2021-08-30 09:26:28 -04:00
George Gevoian
3d085ff2d7 (core) Fix warning bug in Save Copy dialog
Summary:
The warning about workspace write access would still be shown if
a user picked a workspace they had write access to, and cleared the
Name field in the Save Copy dialog. This fixes the condition for
showing the warning to not show it in this case, and adds a placeholder
to the Name field when it is blank.

Test Plan: Browser test.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3002
2021-08-27 12:02:59 -07:00
Dmitry S
3edb3a6826 (core) Filter out pages for hidden views from the page list.
Summary:
- Filters out hidden pages from docModel.allPagesList (used for knowing default page, and for search iteration).
- Filters out hidden pages from TreeModel (uses raw tableData, so has a different way to filter).

Test Plan: WIP

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D2996
2021-08-27 13:51:57 -04:00
Paul Fitzpatrick
8b1ad588e9 (core) make treatment of emails consistent across /access endpoints
Summary:
Access endpoints were supposed to provide display versions of emails,
but in fact only the org endpoint was doing so.  This brings the
workspaces and docs endpoints into line, and adds tests.

Full user information is tweaked slightly to return an anonymous
flag only when anonymous.  This was already anticipated in the
FullUser type.

Test Plan: extended test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2999
2021-08-27 09:45:25 -04:00
George Gevoian
a6e08883e0 (core) Simple localization support and currency selector.
Summary:
- Grist document has a associated "locale" setting that affects how currency is formatted.
- Currency selector for number format.

Test Plan: not done

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D2977
2021-08-26 13:36:49 -07:00
Paul Fitzpatrick
e492dfdb22 (core) add experimental support for python3 in staging
Summary:
This adds `runsc` and `python3` to the grist-server images. For deployments with GRIST_EXPERIMENTAL_PLUGINS=1 (dev + staging but not prod) a hack is added to use `python3` under `runsc` for documents with a special title (`activate-python3-magic` or similar).

This will simplify experiments on behavior of this configuration under realistic conditions.

Hopefully, before landing this, I'll be able to switch to storing a python flag in a document options cell being added by @georgegevoian in a parallel diff, since using the doc title is super hacky :-).

Test Plan: tested manually on worker built locally

Reviewers: dsagal, alexmojaki

Reviewed By: dsagal, alexmojaki

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D2998
2021-08-26 09:39:26 -04:00
Alex Hall
7465af8ce8 (core) Port LinkingState.js to TypeScript
Summary: Converted LinkingState from constructor function to class.

Test Plan: no

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2997
2021-08-26 15:10:34 +02:00
Dmitry S
faa0d9988e (core) When changing a table for a page widget, unset widget-linking to avoid invalid values.
Summary:
Previously, using "Change Widget" allowed one to change the underlying table,
but would keep the linking settings. This could allow invalid settings which
would sometimes lead to JS errors. These manifested in production as
"UserError: Query error: n is not a function".

- Unset linking settings in this case, to avoid invalid values.
- In case invalid values are encountered (e.g. saved previously), treat them as
  unset, to avoid JS errors.
- If an error does occur, report it with a stack trace.

Also, for testing, added 'selectBy' option to gristUtils helpers for using page-widget-picker.

Test Plan: Added test cases for resetting linking, and for ignoring invalid link settings.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D2993
2021-08-25 12:53:46 -04:00
Paul Fitzpatrick
b3636b97e2 (core) Report memos consistently for blocked actions involving schema
Summary:
Currently actions blocked early because they could modify the
schema (e.g. changing formulas) do not report memo information
(comments in relevant rules).  This diff fixes that by using
more of the same code path in the two situations.  It also
adds information about what type of action was blocked to
error messages.

Test Plan: extended a test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2995
2021-08-24 08:03:58 -04:00
Cyprien P
e4633c293c (core) Hide filter bar for inactive widget in mobile mode
Summary:
The filter bar used to show in mobile mode while the widget was inactive as illustrated in this screen shot
{F31970}

This diff fixes it.
{F31971}

Test Plan: Manually tested.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D2958
2021-08-23 10:33:44 +02:00
Alex Hall
9916a2d919 (core) Suggest correct table when converting to RefList
Summary: RecordSets now have new encoding and rendering analogous to Records: `['r', 'Table', [1, 2, 3]]` and `Table[[1, 2, 3]]`.

Test Plan: Added to nbrowser/TypeChange.ts.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2987
2021-08-20 23:04:48 +02:00
Paul Fitzpatrick
f53ab2cb30 (core) forbids edits when "view as" user is a viewer and access rules are permissive
Summary:
Currently, if access rules are set to allow edits unconditionally,
and an owner does "View As" a user who is a viewer only, they will
be allowed to make edits.  This catches that condition and adds a
test.

Test Plan: added test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D2991
2021-08-20 16:04:28 -04:00
Dmitry S
e361a9fd94 (core) Fix a few issues with parsing of dates in DateEditor.
Summary:
- With a format like "DD-MM-YYYY" or "DD MMM YYYY", allow parsing dates
  with two digit year or numeric month (like "16-8-21").
- Interpret two-digit years in the same way for moment parsing and for
  bootstrap-datepicker.
- For partial inputs (like "8/16"), when a format is present, assume that
  provided parts cover the date, then month, then year (even for a format that
  starts with year).

Test Plan: Expanded a unittest

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D2985
2021-08-20 14:54:41 -04:00
Alex Hall
97cb8065d9 (core) Open links with different link keys in a new tab
Summary: Check urlState.params.linkParameters in needPageLoad

Test Plan: Added case to nbrowser test, tested manually

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2984
2021-08-20 19:54:52 +02:00
Alex Hall
54b932300b (core) Filter linking by reference list columns
Summary:
Use 'intersects' query operation when linking against a RefList column, otherwise the rest is the same as linking with a Ref column.

Add RefList columns to Select By options along with Ref columns.

Test Plan: Added new test and fixture similar to SelectBySummary

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2986
2021-08-20 17:52:47 +02:00
Dmitry S
572b59cc0c (core) Fix JS error when pasting data with merged cells.
Summary:
Pasting data with merged cells from Excel (or from HTML tables with colspan/rowspan),
we used to get "Cannot read property 'displayValue' of undefined".

Fix it by assuming that some cell values may be empty.

Test Plan: Added test case reproduces the failure without the fix, and passes with.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D2990
2021-08-20 11:45:05 -04:00
Dmitry S
00c1a0c688 (core) Log the time taken by decodeActionFromRow() operations.
Summary:
Decoding large actions is a plausible culprit for hogging CPU time for
certain documents. To begin with, log the time taken for this operation,
so that we can tell if it's a problem in practice.

Test Plan: Should not affect any current behaviors

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2989
2021-08-20 11:28:33 -04:00
Paul Fitzpatrick
9f25a96d18 (core) add a way to change subdomain in billing pages
Summary:
This adds an `updateDomain` billing task that allows editing
the subdomain (and the org name, which is also editable with
the address).

A warning is shown that changing the subdomain will mean that
saved links need updating.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2988
2021-08-19 21:22:41 -04:00
George Gevoian
d83d734b75 (core) Site Switcher and Other Sites
Summary:
A new section, Other Sites, will now be shown on the All Documents page when:

 - A user is on a personal site and has access to other team sites.
 - A user is on a public site with view access only.

In addition, a site switcher is now available by clicking
the site name in the top-left section of the UI next to the
Grist logo. It works much like the switcher in the Account
menu.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2979
2021-08-18 11:24:41 -07:00
Paul Fitzpatrick
c561dad22d (core) lightly freshen the core readme, mentioning roadmap and forums etc.
Summary: This is a documentation update, and version bump on grist-core.

Test Plan: No code changes.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2982
2021-08-17 23:51:58 -04:00
Paul Fitzpatrick
f9630b3aa4 (core) clean up a collection of small problems affecting grist-core
Summary:
 * Remove adjustSession hack, interfering with loading docs under saml.
 * Allow the anonymous user to receive an empty list of workspaces for
   the merged org.
 * Behave better on first page load when org is in path - this used to
   fail because of lack of cookie.  This is very visible in grist-core,
   as a failure to load localhost:8484 on first visit.
 * Mark cookie explicitly as SameSite=Lax to remove a warning in firefox.
 * Make errorPages available in grist-core.

This changes the default behavior of grist-core to now start off in
anonymous mode, with an explicit sign-in step available.  If SAML is not configured,
the sign-in operation will unconditionally sign the user in as a default
user, without any password check or other security.  The user email is
taken from GRIST_DEFAULT_EMAIL if set.  This is a significant change, but
makes anonymous mode available in grist-core (which is convenient
for testing) and makes behavior with and without SAML much more consistent.

Test Plan: updated test; manual (time to start adding grist-core tests though!)

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2980
2021-08-17 21:44:50 -04:00
Alex Hall
e6e792655b (core) Add /columns endpoint to DocApi
Summary: Add /columns endpoint to DocApi

Test Plan: Added test

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2981
2021-08-17 23:20:52 +02:00
Paul Fitzpatrick
54beaede84 (core) revive saml support and test against Auth0
Summary:
SAML support had broken due to SameSite changes in browsers. This
makes it work again, and tests it against Auth0 (now owned by Okta).

Logging in and out works.  The logged out state is confusing, and may
not be complete.  The "Add Account" menu item doesn't work.
But with this, an important part of self-hosting becomes easier.

SAML support works also in grist-core, for site pages, but there
is a glitch on document pages that I'll look into separately.

Test Plan: tested manually

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2976
2021-08-16 17:36:09 -04:00
Jarosław Sadziński
800731e771 (core) Initial page for custom widget
Summary: Adding a custom widget will show the initial page with some information instead of a blank page.

Test Plan: Existing tests

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D2978
2021-08-13 23:58:25 +02:00
George Gevoian
79f6f605f8 (core) Polish and enable Reference List widget
Summary:
Adds Reference List as a widget type.

Reference List is similar to Choice List: multiple references can be added
to each cell through a similar editor, and the individual references
will always reflect their current value from the referenced table.

Test Plan: Browser tests.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz, jarek, alexmojaki, dsagal

Differential Revision: https://phab.getgrist.com/D2959
2021-08-12 11:51:21 -07:00
Alex Hall
34e9ad3498 (core) Add /records endpoint to DocApi with GET, POST, and PATCH
Summary:
Applies simple data transformations to the existing /data API.

Mimics the Airtable API. Designed in https://grist.quip.com/RZh9AEbPaj8x/Doc-API#FZfACAAZ9a0

Haven't done deletion because it seems like less of a priority and also not fully designed.

Test Plan: Added basic server tests similar to the /data tests. Haven't tested edge cases like bad input.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D2974
2021-08-12 18:02:56 +02:00
George Gevoian
ba1e919d39 (core) Allow creating docs from the templates page
Summary:
Enables the options in the Add New menu when on the templates page
(p/templates).

Test Plan: Browser test.

Reviewers: paulfitz, alexmojaki

Reviewed By: paulfitz, alexmojaki

Subscribers: paulfitz, jarek, alexmojaki

Differential Revision: https://phab.getgrist.com/D2971
2021-08-11 09:59:32 -07:00
Alex Hall
7f1f8fc9e6 (core) Linking summary tables grouped by list columns
Summary:
Prefix keys of `LinkingState.filterColValues` with `_contains:` when the source column is a ChoiceList or ReferenceList.

This is parsed out to make a boolean `isContainsFilter` which is kept in each value of `QueryRefs.filterTuples` (previously `filterPairs`).

Then when converting back in `convertQueryFromRefs` we construct `Query.contains: {[colId: string]: boolean}`.

Finally `getFilterFunc` uses `Query.contains` to decide what kind of filtering to do.

This is not pretty, but the existing code is already very complex and it was hard to find something that wouldn't require touching loads of code just to make things compile.

Test Plan: Added a new nbrowser test and fixture, tests that selecting a source table by summary tables grouped by a choicelist column, non-list column, and both all filter the correct data.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2940
2021-08-10 20:41:24 +02:00
Alex Hall
4d526da58f (core) Move file import plugins into core/sandbox/grist
Summary:
Move all the plugins python code into the main folder with the core code.

Register file importing functions in the same main.py entrypoint as the data engine.

Remove options relating to different entrypoints and code directories. The only remaining plugin-specific option in NSandbox is the import directory/mount, i.e. where files to be parsed are placed.

Test Plan: this

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D2965
2021-08-09 18:37:14 +02:00
Jarosław Sadziński
5b92a43849 (core) Disabling plugin import sources menu options when import is disabled
Summary: Disabling import sources from add menu when a user is not allowed to import.

Test Plan: browser tests

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D2970
2021-08-09 16:54:12 +02:00
Jarosław Sadziński
0b11960fe7 (core) Disabling 'Add widget to page' for special pages
Summary: Disabling "Add widget to page" on special pages like acl or code

Test Plan: browser tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2969
2021-08-09 14:49:19 +02:00
George Gevoian
07bf250592 (core) Hide workspace in breadcrumbs if doc is unsaved
Summary:
Hides the workspace in the breadcrumbs menu if
the doc is unsaved and is not a fork. In practice,
this should usually be when an anonymous user creates a new
document.

Test Plan: Browser tests.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2967
2021-08-05 15:57:03 -07:00
Paul Fitzpatrick
4a23b964ed (core) update read access for exceptional sessions
Summary:
Exceptional sessions had lost full read access to documents; this
restores it.  Exceptional sessions are used for system actions or
while creating documents.

Test Plan: added test

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D2966
2021-08-05 16:49:23 -04:00
Jarosław Sadziński
4ca47878ca (core) Adding import from google drive to the home screen
Summary: Importing from google drive from home screen (also for anonymous users)

Test Plan: Browser tests

Reviewers: dsagal, paulfitz

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2943
2021-08-05 20:46:11 +02:00
Jarosław Sadziński
6ed1d8dfea (core) Adding google drive plugin as a fallback for url plugin
Summary:
When importing from url, user types a url for google spreadsheet,
Grist will switch to Google Drive plugin to allow user to choose file manualy.

Test Plan: Browser tests

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2945
2021-08-04 17:59:13 +02:00
Alex Hall
5aed22dc1e (core) Remove dead code for fetching snapshots
Summary: Deletes code which was previously only used by SharedSharing.ts, which was deleted in D2894

Test Plan: no

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2960
2021-08-04 15:42:31 +02:00
Paul Fitzpatrick
750c78763e (core) add python2 to gvisor Dockerfile, for use in making comparisons
Summary:
This adds python2 to the gvisor sandbox image.  It can be used instead
of the default python3 by setting PYTHON_VERSION to 2 (or calling run.py with python2).
This is useful for making side-by-side comparisons with code running python3.

Test Plan: manual

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D2957
2021-08-03 17:51:31 -04:00
Jarosław Sadziński
521bbd9ac1 (core) Improving error messages on file imports
Summary: Improving error messages that get returned from "Import from URL" plugin.

Test Plan: browser tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: dsagal, alexmojaki

Differential Revision: https://phab.getgrist.com/D2946
2021-08-02 17:30:59 +02:00
Dmitry S
1605e18f66 (core) Enable auto triggering of Welcome Tour, and various improvements.
Summary:
- Add showGristTour preference, and trigger tour automatically.
- Tour is only triggered for new and anonymous users on a personal org, with
  edit permission.

- Automatically open the right panel at tour start.
- Don't show tours on mobile, since that's not ready (popups are cut off
  and can't be dismissed)
- Cancel previous tour if a new one is somehow started.
- Remove #repeat- trigger hash tags from the URL when the tour starts.
- Ensure Help Center popup is positioned even when left panel is collapsed.
- Polish up the content of the last two cards in the tour.

Test Plan: Added test case for triggering and opening right panel.

Reviewers: alexmojaki, paulfitz

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D2955
2021-07-30 14:17:54 -04:00
Alex Hall
73c4efa315 (core) Hide the GristDocTour table by default but reveal it when /p/GristDocTour is in the URL
Summary:
Adds 'GristDocTour' as a possible value of urlState().docPage

GristDoc checks for this and converts it to a normal view record ID

It also then sets a flag showGristDocTour=true which tells Pages.ts to show the page in the sidebar

Otherwise the page is 'hidden' in the sidebar in the same way it would be if blocked by ACL rules

This all feels very hacky, but I don't know this code well enough to know if there's a better way. Hopefully this behaviour is temporary.

Test Plan: Tested manually, not sure if this is worth an automated test at this stage

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz, dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D2953
2021-07-30 20:14:34 +02:00
Paul Fitzpatrick
6b3ac07ca7 (core) process GristDocAPI calls from custom widgets in the client
Summary:
Processing these calls in the client, rather than passing them on
to the backend, means that access rules are more straightforward to
apply.

An unrelated fix is included to filter _grist_ tables when fetched
individually - metadata could leak through this path.

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2954
2021-07-30 10:41:32 -04:00
Alex Hall
1d1a9297f8 (core) Polish UI/UX of onboarding popups
Summary:
Replace Finish button with Previous and an X to close
Add keyboard shortcuts to tour popups
Change last Next button to Finish instead of disabling, can be triggered by Enter key.
Allow closing the tour and reopening in the same place.

Test Plan: only manual, need to confirm desired behaviour

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2950
2021-07-30 15:44:18 +02:00
Paul Fitzpatrick
d09c2f1b3d (core) tweak description of FullCopies special permission
Summary: This mentions fiddle mode in FullCopies special permission.

Test Plan: visual, existing tests pass

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2952
2021-07-30 06:18:41 -04:00
George Gevoian
931827b195 (core) Hide templates org in copy menu
Summary:
Temporarily adds a client-side check to hide the
Grist Templates org in the Save Copy menu. This will
be removed later once we update getOrgs to optionally
filter orgs that have no workspaces with write access.

Test Plan: Browser tests.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D2951
2021-07-29 16:37:32 -07:00
Alex Hall
aec360498c (core) Allow adding a single link at the bottom of a document tour card
Summary:
Checks for new special columns in GristDocTour: Link_Text, Link_URL, and Link_Icon.

No link is generated if Link_Text is blank or Link_URL cannot be parsed as a URL.

No icon is shown if Link_Icon is not the name of an icon in IconList.ts

Test Plan: Expanded tests, but they now assert things about HTML which may be brittle

Reviewers: georgegevoian, dsagal

Reviewed By: georgegevoian, dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D2947
2021-07-30 00:49:53 +02:00
George Gevoian
37eed2d3c2 (core) Fix staging test and tweak template line height
Summary:
Staging tests were using an organization named Test Grist,
while local was using one named test-grist. Both are now
named Test Grist, which should help keep things more consistent
between local and deployment test runs.

The inherited line height of template docs in icon view was
different on Windows, cutting off part of the last line of the
description. The description line height should now be fixed
to a reasonable value.

Test Plan: Manually tested CSS fix on a Windows machine.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D2948
2021-07-29 08:16:19 -07:00
George Gevoian
24fc3a2d00 (core) Redesign examples and templates UI
Summary:
The old Examples and Templates workspace is now
a page that pulls templates from a new public Grist Templates org.
The All Documents view will pull featured templates from that org, where
featured templates are simply pinned documents in Grist Templates. The
Examples and Templates page will also show the featured templates, as
well as the rest of the available templates organized by category. The
categories are equivalent to workspaces in Grist Templates, and are
generated dynamically.

Test Plan: Browser tests.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz, dsagal

Subscribers: dsagal, paulfitz, jarek

Differential Revision: https://phab.getgrist.com/D2930
2021-07-28 12:29:03 -07:00
Paul Fitzpatrick
bb8cb2593d (core) support python3 in grist-core, and running engine via docker and/or gvisor
Summary:
 * Moves essential plugins to grist-core, so that basic imports (e.g. csv) work.
 * Adds support for a `GRIST_SANDBOX_FLAVOR` flag that can systematically override how the data engine is run.
   - `GRIST_SANDBOX_FLAVOR=pynbox` is "classic" nacl-based sandbox.
   - `GRIST_SANDBOX_FLAVOR=docker` runs engines in individual docker containers. It requires an image specified in `sandbox/docker` (alternative images can be named with `GRIST_SANDBOX` flag - need to contain python and engine requirements). It is a simple reference implementation for sandboxing.
   - `GRIST_SANDBOX_FLAVOR=unsandboxed` runs whatever local version of python is specified by a `GRIST_SANDBOX` flag directly, with no sandboxing. Engine requirements must be installed, so an absolute path to a python executable in a virtualenv is easiest to manage.
   - `GRIST_SANDBOX_FLAVOR=gvisor` runs the data engine via gvisor's runsc. Experimental, with implementation not included in grist-core. Since gvisor runs on Linux only, this flavor supports wrapping the sandboxes in a single shared docker container.
 * Tweaks some recent express query parameter code to work in grist-core, which has a slightly different version of express (smoke test doesn't catch this since in Jenkins core is built within a workspace that has node_modules, and wires get crossed - in a dev environment the problem on master can be seen by doing `buildtools/build_core.sh /tmp/any_path_outside_grist`).

The new sandbox options do not have tests yet, nor does this they change the behavior of grist servers today. They are there to clean up and consolidate a collection of patches I've been using that were getting cumbersome, and make it easier to run experiments.

I haven't looked closely at imports beyond core.

Test Plan: tested manually against regular grist and grist-core, including imports

Reviewers: alexmojaki, dsagal

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D2942
2021-07-28 09:02:32 -04:00
Alex Hall
cd0c6de53e (core) Automatically or explicitly show document tours in the same way as example cards.
Summary:
Extracts code from showExampleCard into a generic function which is reused for document tours.

It handles reading and writing to user preferences for automatic showing and explicitly reopening.

Test Plan:
Manually tested that it automatically shows a tour just once and clicking to reopen works.

There's not much new functionality so there's little that needs testing. This is an initial version that's mostly internal and is likely to be polished for users in the future.

If I should still add tests, I'd like confirmation that the current behaviour is as desired.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2944
2021-07-27 18:35:48 +02:00
Alex Hall
15f1ef96fa (core) Allow creating custom document tours with a special table
Summary:
Like the welcome tour, a special URL hash triggers startDocTour which uses data from a table GristDocTour to construct the appropriate popups.

This is the basic version described in https://grist.quip.com/sN2RAHI2dchm/Document-tours

Test Plan:
Added a new nbrowser test which tests the data produced by makeDocTour. The general behaviour of the UI and popups has hardly changed so existing tests cover that well enough.

The new test uses a new fixture document which you can open to easily experience the tour.

Error cases where there's no valid document tour are not tested because that behaviour is likely to change significantly and this feature is still quite 'private'.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: jarek, dsagal

Differential Revision: https://phab.getgrist.com/D2938
2021-07-23 20:23:30 +02:00
Alex Hall
04e5d90f86 (core) Barely working reference lists in frontend
Summary:
This makes it possible to set the type of a column to ReferenceList, but the UI is terrible

ReferenceList.ts is a mishmash of ChoiceList and Reference that sort of works but something about the CSS is clearly broken

ReferenceListEditor is just a text editor, you have to type in a JSON array of row IDs. Ignore the value that's present when you start editing. I can maybe try mashing together ReferenceEditor and ChoiceListEditor but it doesn't seem wise.
I think @georgegevoian should take over here. Reviewing the diff as it is to check for obvious issues is probably good but I don't think it's worth trying to land/merge anything.

Test Plan: none

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D2914
2021-07-23 18:41:44 +02:00
Dmitry S
8d68c1c567 (core) Replace time zone selector with one based on the newer autocomplete.
Summary:
Flaky Dates test failures related to the use of JQuery autocomplete for time
zones, which wasn't working well.

This diff replaces that autocomplete (as well as a similar select box in
DocumentSettings) with our newer autocomplete, adding some select-box like
behavior.

Most of the behavior is factored out into ACSelect, which could be more
generally useful.

Adds an option to autocomplete to keep options ordered according to their
initial order.

Unrelated: fix up usage of MultiHolder in Drafts to avoid 'already disposed'
warnings.

Test Plan: Fixed several affected tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D2919
2021-07-23 08:02:05 -04:00
Jarosław Sadziński
a07395855a (core) Fixing anchor link navigation.
Summary: Last document position was overwritting anchor link navigation.

Test Plan: Browser tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D2934
2021-07-23 11:02:07 +02:00
Jarosław Sadziński
f8e4fe54ba (core) Fixing origin check during Google Authentication
Summary:
Fixing two bugs
- Google Auth Endpoint wasn't resolving protocol in a correct way
- Google Auth Popup was navigationg to endpoint url based on home url, which
  was diffent from current page origin

Test Plan: n/a

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D2937
2021-07-23 00:43:27 +02:00
Paul Fitzpatrick
95cc2eb282 (core) read document as owner in pre-fork mode, if have sufficient access to it
Summary:
This tweaks pre-fork mode to make the user's experience a bit more seamless.
Pre-fork mode is where the user has opened a document with intent to
fork it, but actual forking (with allocation of a new document id)
is postponed until they make their first change.

The tweak makes the user an owner for granular access purposes, if
forking is permitted.  So data visible only to owners because of
access rules will be visible to them.  As always, any edits would
go to a separate new copy.

A remaining tricky corner is what to do about "View As" functionality
on forks.  Fork sharing cannot be controlled, so the "Users -> View As"
functionality isn't available.  Perhaps the "Users" button on a fork
could encourage doing a save-copy and inviting users, or offer some
dummy users?  In any case, this diff doesn't change anything with
that corner.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2931
2021-07-21 14:52:31 -04:00
Jarosław Sadziński
291bcd17ff (core) Google auth endpoint has not responded with auth code
Summary:
Google Auth popup wasn't able to resolve origin from gristConfig.
Moving this reponsability to server side, where it gets calculated from initial request.

Test Plan: n/a

Reviewers: dsagal, paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2935
2021-07-21 20:18:04 +02:00
Jarosław Sadziński
08295a696b (core) Export to Excel and Send to drive
Summary:
Implementing export to excel and send to Google Drive feature.

As part of this feature few things were implemented:
- Server side google authentication exposed on url: (docs, docs-s, or localhost:8080)/auth/google
- Exporting grist documents as an excel file (xlsx)
- Storing exported grist document (in excel format) in Google Drive as a spreadsheet document.

Server side google authentication requires one new environmental variables
- GOOGLE_CLIENT_SECRET (required) used by authentication handler

Test Plan: Browser tests for exporting to excel.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2924
2021-07-21 16:36:00 +02:00
Jarosław Sadziński
9cc034f606 (core) Fixing bug on a fresh Choice and ChoiceList column
Summary: Adding a new element to a new Choice or ChoiceList column resulted in javascript error

Test Plan: Browser tests

Reviewers: georgegevoian, dsagal

Reviewed By: georgegevoian, dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D2932
2021-07-21 15:44:10 +02:00
Alex Hall
1f6e693b6e (core) Remove REPL code
Summary: Remove repl.py, REPLTab.js, some wiring code, CSS, and a test in testscript.json.

Test Plan: NA

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2923
2021-07-20 15:17:03 +02:00
Dmitry S
67aca9ccf6 (core) Fix several flaky tests.
Summary:
- Fix flaky SamplesWS test, which was occasionally clicking a disabled button
- Increase timeout on backupSqliteDatabase test, which sometimes times out
- Fix a little flakiness in ExportSection test.
- Fix flaky Billing test by adjusting behavior slightly.

In Billing, when re-fetching subscription (e.g. on navigating back), we now
unset it while the fetch is pending, so that billing pages show a spinner or
"Fetching..." messages. This also gives tests something to wait for.

Also adjusts Billing styles on the "Fetching..." messages to make them look
different from regular text.

Test Plan: Mainly making exising tests more robust. Billing changes exercised by existing tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2920
2021-07-19 11:47:14 -04:00
Cyprien P
693f2f6325 (core) Brings welcome tour and hide behind a flag
Summary:
This diff brings in the new welcome tour. It builds upon `client/ui/OnBoardingPopup` that was committed to that purposes. Per this diff,  the tour is accessible behind a flag and won't be visible to user: few caveats listed below needs to be adressed first.

This diff also brings few changes to onboarding module.
  - allow to refer to element with selector
     - usually dynamic selection of element sounds useful for when the
     element does not exist yet when the tour starts. But the actual
     reason when add it here, is to allow selecting the first cell.
     - if the selector yields undefined (missing element), the popup
     is simply skipped
  - got rid of the internal registry to link between popup contents
  and popup options. All is now define in the same interface. Registry
  overall felt overkill and not needed.
  - adds an option to show message as a simple modal that is centered
  on the screen

This diff also brings the new welcome tour and hide it behind a flag

CAVEATS that need to be addressed in follow up commit:
 - The url needs cleanup, #repeat-welcome-tour sticks to it and so even when navigating to home page. This could eventually become an issue: if user opens another document it would starts the onboarding tour again.
 - For now you have to manually make sure the right panel is opened with the Column tab selected before starting the tour.
  - On boarding tours were not designed with mobile support in mind. So probably a good idea to disable.
  - Backend support needs to be done (persistence of first time user).

Test Plan:
Updated `projects/OnBoardingPopup` and adds new `nbrowser/welcomeTour`
To launch the tour:
  - open any document
  - open manually the right panel and the field tab
  - append the flag `#repeat-welcome-tour` at the end of the url in the url bar and reload the page

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2917
2021-07-19 16:30:53 +02:00
George Gevoian
5b2666a88a (core) Enhance autocomplete and choice colors
Summary:
Choice columns can now add new choices directly
from the autocomplete menu. The autocomplete will
now highlight the first matching item, even if there are equally
ranked alternatives. No changes have been made to how the
autocomplete index is created, or how it scores items.

For choice and choice list columns, the filter menu will
now display values using their configured colors, similar to the
rest of the UI. Choice tokens throughout the UI now do a better
job of handling text overflow by showing an ellipsis whenever
there isn't enough space to show the full text of a choice.

Test Plan: Browser tests.

Reviewers: cyprien

Reviewed By: cyprien

Differential Revision: https://phab.getgrist.com/D2904
2021-07-16 09:10:51 -07:00
Paul Fitzpatrick
997be24a21 (core) add docs.options column to home db to store doc description, icon, openMode
Summary:
Bundles some new document options into a JSON column.
The icon option is treated somewhat gingerly.  It is intended, at
least initially, to store an image thumbnail for a document as a
url to hand-prepared assets (for examples and templates), so it is
locked down to a particular url prefix to avoid opening the door to
mischief.

Test Plan: added test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D2916
2021-07-15 21:51:05 -04:00
George Gevoian
e5eeb3ec80 (core) Add 'user' variable to trigger formulas
Summary:
The 'user' variable has a similar API to the one from access rules: it
contains properties about a user, such as their full name and email
address, as well as optional, user-defined attributes that are populated
via user attribute tables.

Test Plan: Python unit tests.

Reviewers: alexmojaki, paulfitz, dsagal

Reviewed By: alexmojaki, dsagal

Subscribers: paulfitz, dsagal, alexmojaki

Differential Revision: https://phab.getgrist.com/D2898
2021-07-15 15:18:32 -07:00
Dmitry S
6c114ef439 (core) Fix session handling when redirected to login when visiting a doc on a team site
Summary:
When redirecting to login, it's important to have a valid session set. This was
done by middleware that only applies to home pages. We need to set session to
live when redirecting in case of doc pages too.

Test Plan: Added a test case for fixed behavior by applying an existing case to doc pages too

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2915
2021-07-15 17:51:16 -04:00
Paul Fitzpatrick
1ce5e98996 (core) suspend a team site after an AppSumo refund
Summary:
This suspends service to a team site for which an AppSumo refund has been made, and nudges users to their free personal account.

I expect that a refund request would fail for a site where user is also paying us for extra seats.

Test Plan: tested manually

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2912
2021-07-15 09:14:12 -04:00
Paul Fitzpatrick
6e15d44cf6 (core) start applying defenses for untrusted document uploads
Summary:
This applies some mitigations suggested by SQLite authors when
opening untrusted SQLite databases, as we do when Grist docs
are uploaded by the user.  See:
  https://www.sqlite.org/security.html#untrusted_sqlite_database_files

Steps implemented in this diff are:
  * Setting `trusted_schema` to off
  * Running a SQLite-level integrity check on uploads

Other steps will require updates to our node-sqlite3 fork, since they
are not available via the node-sqlite3 api (one more reason to migrate
to better-sqlite3).

I haven't yet managed to create a file that triggers an integrity
check failure without also being detected as corruption by sqlite
at a more basic level, so that is a TODO for testing.

Test Plan:
existing tests pass; need to come up with exploits to
actually test the defences and have not yet

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2909
2021-07-14 18:34:27 -04:00
Jarosław Sadziński
625fce5f65 (core) Refactoring google drive plugin
Summary:
Finishing implementation for google drive plugin.
- Refactoring plugin code to make it more robust and to follow grist ux
- Changing the way server hosts untrusted user content, from different domain to different port

Test Plan: Browser tests

Reviewers: dsagal, paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2881
2021-07-14 09:52:04 +02:00
Dmitry S
0a0e320ca8 (core) Fix reporting of errors to HelpScout using "Report a problem" link.
Summary:
Error notifications include a "Report a problem" link, but the inclusion of
error message and stack trace was broken.

Test Plan: Tested manually and added a test case.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2911
2021-07-13 18:26:36 -04:00
Jarosław Sadziński
09b4aaa90e (core) Formatting date time editor
Summary: Formatting date and datetime editor. Changing styles for Timezone, DateFormat and TimeFormat section.

Test Plan: Existing tests

Reviewers: georgegevoian, dsagal

Reviewed By: georgegevoian, dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D2899
2021-07-13 09:22:39 +02:00
George Gevoian
5d461a4df2 (core) Show sigma next to summary table titles
Summary:
The sigma icon appears to the left of the table title
if the table is a summary table.

Test Plan: Updated browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D2895
2021-07-12 20:10:15 -07:00
George Gevoian
9592e3610b (core) Add 'value' to trigger formula autocomplete
Summary:
API signature for autocomplete updated to add column ID, which is
necessary for exposing correct types for 'value'.

Test Plan: Unit tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: jarek, alexmojaki

Differential Revision: https://phab.getgrist.com/D2896
2021-07-12 15:07:16 -07:00
Dmitry S
869b2f00ec (core) Remove LoginSession, which was mainly serving situations that are no longer used.
Summary:
In the past, Cognito sign-ins were intended to give authorization to some AWS
services (like SQS); various tokens were stored in the session for this
purpose. This is no longer used. Profiles from Cognito now serve a limited
purpose: first-time initialization of name and picture, and keeping track of
which login method was used. For these remaining needs, ScopedSession is
sufficient.

Test Plan:
Existing test pass. Tested manually that logins work with Google and
Email + Password. Tested manually that on a clean database, name and picture
are picked up from a Google Login.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2907
2021-07-12 13:04:00 -04:00
Dmitry S
f079ffdcb3 (core) Fix a log message about when a doc will close to be more accurate
Test Plan: Checked manually for a long-opening document that the time reported is correct.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2906
2021-07-12 09:51:54 -04:00
Dmitry S
0e0528a2cc (core) Add some more special subdomains, not to be used for team sites
Test Plan: Existing tests should pass, no new ones needed

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2905
2021-07-12 09:51:20 -04:00
Paul Fitzpatrick
4222f1ed32 (core) communicate with sandbox via standard pipes
Summary:
This switches to using stdin/stdout for RPC calls to the sandbox, rather than specially allocated side channels. Plain text error information remains on stderr.

The motivation for the change is to simplify use of sandboxes, some of which support extra file descriptors and some of which don't.

The new style of communication is made the default, but I'm not committed to this, just that it be easy to switch to if needed. It is possible I'll need to switch the communication method again in the near future.

One reason not to make this default would be windows support, which is likely broken since stdin/stdout are by default in text mode.

Test Plan: existing tests pass

Reviewers: dsagal, alexmojaki

Reviewed By: dsagal, alexmojaki

Differential Revision: https://phab.getgrist.com/D2897
2021-07-12 06:45:47 -04:00
George Gevoian
2f900f68f8 (core) Add color options to choice config UI
Summary:
Includes overhauled choice configuration UI for choice and choice list
columns based on the TokenField library. Features include rich copy
and paste support, keyboard shortcuts for token manipulation, and
drag-and-drop support for arrangement.

Configured choice colors are visible throughout the application, such
as in the autocomplete window for both choice and choice list cells, and
in table cells directly.

Choice cells in particular are now styled closer to choice list cells,
and render their contents as colored tokens. Choice cells now also
use the improved autocomplete component that choice lists use, with
some room for future improvement (e.g. allowing new choice items to be
added inline like in choice list's autocomplete).

Also includes a minor fix for choice list cells where right align
was not working.

Test Plan: Browser tests updated.

Reviewers: jarek, dsagal

Reviewed By: jarek, dsagal

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D2890
2021-07-09 12:07:38 -07:00
Dmitry S
11c944d794 (core) Accept values entered into Reference columns showing RowIDs without waiting.
Summary:
- Normally Reference columns can only be used for data entry once the target
  table has loaded. When it shows RowID, we shouldn't need to wait.
- Also, fix pasting values between cells of a RowID-showing column.

Test Plan: Added a test for entering data before data has loaded.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2902
2021-07-08 13:09:24 -04:00
Paul Fitzpatrick
d0905b6224 (core) correct capitalization of AppSumo; add account management link
Summary:
Branding feedback from AppSumo found a capitalization problem. They also nudged us again to include a link back for the user to manage
their AppSumo account.

Test Plan: manual

Reviewers: dsagal, anaisconce

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D2901
2021-07-08 12:41:29 -04:00
Cyprien P
b389ee7c23 (core) Adds new mechanism to generate on boarding popups
Summary:
Does the UI only no backend.

Follow up work:
  - Implement a way to remember when a user dimsmis the popups, so
    that we don't show her again.
  - After users clicks Finish adds a final popup saying  "You can repeat this tour from the Help Center" , and in help center home page, have a link "Repeat Grist welcome tour", which opens, say, https://docs.getgrist.com/doc/lightweight-crm#repeat-welcome-tour, where the hash part tells us to repeat the tour.

Test Plan: Tested in project/OnBoardingPopups

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2892
2021-07-02 17:54:25 +02:00
Alex Hall
ea01ca814d (core) Remove a bunch of dead code
Summary: Removed test/aws/, most of app/server/lib/, 3 dirs in app/lambda/, corresponding tests, and more!

Test Plan: a lot of this is quite the opposite...

Reviewers: dsagal, paulfitz

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2894
2021-07-01 18:38:21 +02:00
Alex Hall
84ddbc448b (core) Add test_replay for easily replaying data sent to the sandbox purely within python
Summary:
Run JS with a value for SANDBOX_BUFFERS_DIR, then run test_replay in python with the same value to replay just the python code.

See test_replay.py for more info.

Test Plan:
Record some data, e.g. `SANDBOX_BUFFERS_DIR=manual npm start` or `SANDBOX_BUFFERS_DIR=server ./test/testrun.sh server`.

Then run `SANDBOX_BUFFERS_DIR=server python -m unittest test_replay` from within `core/sandbox/grist` to replay the input from the JS.

Sample of the output will look like this:

```
Checking /tmp/sandbox_buffers/server/2021-06-16T15:13:59.958Z
True
Checking /tmp/sandbox_buffers/server/2021-06-16T15:16:37.170Z
True
Checking /tmp/sandbox_buffers/server/2021-06-16T15:14:22.378Z
True
```

Reviewers: paulfitz, dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2866
2021-06-30 16:56:09 +02:00
Dmitry S
b537539b73 (core) Implement UI for trigger formulas.
Summary:
- Implement UI with "Apply to new records" and "Apply on record changes"
  checkboxes, and options for selecting which changes to recalculate on.
- For consistency, always represent empty RefList as None
- Fix up generated SchemaTypes to remember that values are encoded.

Included test cases for the main planned use cases:
- Auto-filled UUID column
- Data cleaning
- NOW() formula for record's last-updated timestamp.
- Updates that depend on other columns.

Test Plan: Added a browser test.

Reviewers: jarek

Reviewed By: jarek

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D2885
2021-06-29 10:24:16 -04:00
Jarosław Sadziński
e180641c7d (core) Context menu for cards.
Summary:
User was not able to delete cards. This patch introduces a context menu for cards, analogous to the one available for rows on a GridView.

Changes:
- Row numbers on a GridView have the same icon as on columns to make context menu more discoverable.
- Context menu for rows and columns, when activated, didn't switch section in rare conditions (i.e. when the section had 2 or more columns selected, one of which had the same rowId as a column in the section that the user switched from).
- Card list layout and a single card layout has the same context menu as in a GridView, available by pressing the context menu button.

Test Plan: Browser tests

Reviewers: dsagal, paulfitz

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2870
2021-06-29 15:29:56 +02:00
Dmitry S
01cef034ed (core) A quick attempt to fix summarizing by reference columns
Test Plan: TBD

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2884
2021-06-28 15:05:55 -04:00
Dmitry S
a56714e1ab (core) Implement trigger formulas (generalizing default formulas)
Summary:
Trigger formulas can be calculated for new records, or for new records and
updates to certain fields, or all fields. They do not recalculate on open,
and they MAY be set directly by the user, including for data-cleaning.

- Column metadata now includes recalcWhen and recalcDeps fields.
- Trigger formulas are NOT recalculated on open or on schema changes.
- When recalcWhen is "never", formula isn't calculated even for new records.
- When recalcWhen is "allupdates", formula is calculated for new records and
  any manual (non-formula) updates to the record.
- When recalcWhen is "", formula is calculated for new records, and changes to
  recalcDeps fields (which may be formula fields or column itself).
- A column whose recalcDeps includes itself is a "data-cleaning" column; a
  value set by the user will still trigger the formula.
- All trigger-formulas receive a "value" argument (to support the case above).

Small changes
- Update RefLists (used for recalcDeps) when target rows are deleted.
- Add RecordList.__contains__ (for `rec in refList` or `id in refList` checks)
- Clarify that Calculate action has replaced load_done() in practice,
  and use it in tests too, to better match reality.

Left for later:
- UI for setting recalcWhen / recalcDeps.
- Implementation of actions such as "Recalculate for all cells".
- Allowing trigger-formulas access to the current user's info.

Test Plan: Added a comprehensive python-side test for various trigger combinations

Reviewers: paulfitz, alexmojaki

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2872
2021-06-25 22:53:07 -04:00
Paul Fitzpatrick
dca3abec1d (core) complete light sign-up flow for appsumo, and customize summaries
Summary:
Current appsumo sign-up flow doesn't reach the billing pages.
This diff nudges user on through that extra step.

It also tweaks plan summaries to say what special appsumo
features are in effect (member count prepaid for).

Test Plan: manual

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2882
2021-06-25 14:13:13 -04:00
Paul Fitzpatrick
36d5e7870e (core) streamline registration flow for new appsumo users
Summary:
This adds a new landing page for cognito sign-up, intended for
use by new appsumo users.

Their email address is pre-filled and locked down, and sign-up
is by entering a password.

The page is very crude compared to hosted cognito - especially
in error reporting! - but having the address filled in more
than makes up for that.

The flow does not quite connect with the new billing signup.
I think we can do that through the regular "welcome" process,
which will list the user's team site.  When the user visits
that site, we could detect that we are on a site with no
domain set yet and for which the user is a billing manager,
and trigger a visit to the appropriate billing page.

Test Plan: manual - hard to test through cognito email step

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2880
2021-06-25 10:47:10 -04:00
Alex Hall
305b133c59 (core) Remaining Python 3 compatibility changes
Summary: Biggest change is turning everything to unicode

Test Plan: The tests

Reviewers: dsagal, paulfitz

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2875
2021-06-25 12:00:58 +02:00
Paul Fitzpatrick
1af99e9567 (core) link AppSumo activations with stripe, and support upgrades/downgrades
Summary:
This links AppSumo sign-ups with Stripe subscriptions
and our billing pages. Different AppSumo tiers are supported by
different coupons on the standard plan. Configuration of this
is in stripe, and then cached in the database.

The front end is tweaked just enough to make completing a sign-up
possible. It is not yet friendly.

Not covered includes:
 * Streamlining landing page.
 * Making billing pages git clearer summaries of AppSumo states.
 * Making flow through Cognito as graceful as possible - default
   probably doesn't meet AppSumo requirements.
 * Disabling site on cancellation/refund.
 * Downgrades when more seats in use than lower tier allows.

Test Plan: api-level tests added. No front-end tests yet.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2878
2021-06-24 10:18:42 -04:00
Dmitry S
6240fd6982 (core) Fix reporting of errors when saving cells by clicking away, and deduplicate memos.
Test Plan: Enhanced the test case for memos to check these cases too (fails without this fix).

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2876
2021-06-23 08:50:42 -04:00
Cyprien P
7a0cd6c2b4 (core) Makes filter counts take other column filters into account
Summary:
Makes filter counts take other column filters into account.

 - Changes the summaries rows to reflect hidden rows:
    - hidden rows are added to the `Other Values` summary
    - show the unique number of other values as `Other Values (12)`
 - Also, adds a sort button to the column filter menu

Test Plan: Adds browser test.

Reviewers: paulfitz, jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D2861
2021-06-23 09:01:41 +02:00
Paul Fitzpatrick
cc04c6481a (core) add appsumo endpoints with stub implementations
Summary:
This adds appsumo /token and /notification endpoints, with some
tests.  The stub implementation is sufficient for AppSumo
activation to succeed (when exposed via port forwarding for testing).
It needs fleshing out:

 * Implement upgrade/downgrade/refund and stripe subscription.
 * Implement custom landing page and flow.

Test Plan: added tests

Reviewers: dsagal, georgegevoian

Reviewed By: dsagal

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D2864
2021-06-21 16:04:33 -04:00
Jarosław Sadziński
bdd4d3c46e (core) Freezing columns on a GridView
Summary:
User can freeze any number of columns, which will not move when a user scrolls grid horizontally.
Main use cases:
- Frozen columns don't move when a user scrolls horizontally
- The number of frozen columns is automatically persisted
- Readonly viewers see frozen columns and can modify them - but the change is not persisted
- On a small screen - frozen columns still moves to the left when scrolled, to reveal at least one column
- There is a single menu option - Toggle freeze - which offers the best action considering selected columns
- When a user clicks a single column - action to freeze/unfreeze is always there
- When a user clicks multiple columns - action is offered only where it makes sens (columns are near the frozen border)

Test Plan: Browser tests

Reviewers: dsagal, paulfitz

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2852
2021-06-18 12:22:13 +02:00
Jarosław Sadziński
698c9d4e40 (core) Readonly editors
Summary:
Grist should not prevent read-only viewers from opening cell editors since they usually provide much more information than is visible in a cell.

Every editor was enhanced with a read-only mode that provides the same information available for an editor but doesn't allow to change the underlying data.

Test Plan: Browser tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2842
2021-06-17 19:12:16 +02:00
Alex Hall
8a940676e9 (core) Generic tools for recording pycalls, deterministic mode.
Summary:
Replaces https://phab.getgrist.com/D2854

Refactoring of NSandbox:
- Simplify arguments to NSandbox.spawn. Only half the arguments were used depending on the flavour, adding a layer of confusion.
- Ensure the same environment variables are passed to both flavours of sandbox
- Simplify passing down environment variables.

Implement deterministic mode with libfaketime and a seeded random instance.
- Include static prebuilt libfaketime.so.1, may need another solution in future for other platforms.

Recording pycalls:
- Add script recordDocumentPyCalls.js to open a single document outside of tests.
- Refactor out recordPyCalls.ts to support various uses.
- Add afterEach hook to save all pycalls from server tests under $PYCALLS_DIR
- Make docTools usable without mocha.
- Add useLocalDoc and loadLocalDoc for loading non-fixture documents

Test Plan:
Made a document with formulas NOW() and UUID()
Compare two document openings in normal mode:

    diff <(test/recordDocumentPyCalls.js samples/d4W6NrzCMNVSVD6nWgNrGC.grist /dev/stdout) \
         <(test/recordDocumentPyCalls.js samples/d4W6NrzCMNVSVD6nWgNrGC.grist /dev/stdout)

Output:

    <                 1623407499.58132,
    ---
    >                 1623407499.60376,
    1195c1195
    <               "B": "bd2487f6-63c9-4f02-bbbc-5c0d674a2dc6"
    ---
    >               "B": "22e1a4fd-297f-4b86-91a2-bc42cc6da4b2"

`export DETERMINISTIC_MODE=1` and repeat. diff is empty!

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2857
2021-06-15 20:58:05 +02:00
George Gevoian
f613b68a9e (core) Fix widget loading spinner title
Summary:
The loading spinner would always display
'Building Table widget' when creating or changing
a widget. This fixes the title to reflect the selected
widget type.

Test Plan:
Updated existing browser tests to verify the loading spinner
title includes the correct widget type.

Reviewers: dsagal, paulfitz

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2863
2021-06-15 08:56:24 -07:00
George Gevoian
3af6dd46ea (core) Allow docs to be permanently deleted in icon view
Summary:
Previously, soft-deleted docs in icon view were still accessible from
the Trash and couldn't be permanently deleted.

Test Plan:
Improved the nbrowser test for deleting docs to verify that it can
be done in both view modes.

Reviewers: dsagal, paulfitz

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2862
2021-06-15 08:33:47 -07:00
Cyprien P
b740028e8f (core) Increases readability of column filter menu code
Summary: The `select-all` originally designed the `All` button but it unfortunately evolves into designating both `All` and `None`. Which is confusing. Replaces with the more general `bulk-action`.

Test Plan: Should not break anything.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2860
2021-06-14 09:10:55 +02:00
George Gevoian
b94eb107d4 (core) Use individual choices for filtering choice lists
Test Plan: Wrote unit and browser tests that verify new behavior.

Reviewers: paulfitz, dsagal

Reviewed By: dsagal

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D2855
2021-06-11 09:34:06 -07:00
Cyprien P
5d3a4b5b5b (core) Introduce tri-state checkboxes
Summary:
 - Takes advantage of native indeterminate state of html checkboxes
 - When an indeterminate checkbox is clicked it turns it into being not checked.

Test Plan: - Added test to projects/UI2018

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2846
2021-06-11 14:53:42 +02:00
George Gevoian
982711dbba (core) Update active view id when default view id changes
Summary:
This fixes a bug where deleting a page with the page id
missing from the URL would cause JS errors to be thrown.

Test Plan:
Verified manually in dev environment. Browser test added
that should hopefully replicate the repro steps and catch
any regressions.

Reviewers: dsagal, paulfitz

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2856
2021-06-10 17:49:39 -07:00