Commit Graph

353 Commits

Author SHA1 Message Date
George Gevoian
e52e15591d (core) Add behavioral and coaching call popups
Summary:
Adds a new category of popups that are shown dynamically when
certain parts of the UI are first rendered, and a free coaching
call popup that's shown to users on their site home page.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3706
2022-12-20 09:49:41 -05:00
Jarosław Sadziński
fa75c93d67 (core) Only owners should be able to rename a document.
Summary:
Checking SCHEMA_EDIT permission when user wants to
update document's name.

Test Plan: New test

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3733
2022-12-20 10:09:49 +01:00
Cyprien P
620e86a9f1 (core) New date filter with a calendar view
Summary:
Implements the new date filtering panel. Design results from long
discussion between: Alex, Anais, Cyprien and Dmitry.

Test environment: https://grist-new-date-range-filter.fly.dev/

Test Plan: Include various new tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3720
2022-12-20 09:58:42 +01:00
Jarosław Sadziński
0b6b8feb2b Removing dependency on REDIS in webhook tests 2022-12-19 18:35:13 +01:00
Paul Fitzpatrick
7dc49f3c85 (core) updates from grist-core 2022-12-19 09:25:09 -05:00
Jarosław Sadziński
629fcccd5a (core) Adding /webhooks endpoint
Summary:
- New /webhooks event that lists all webhooks in a document (available for owners),
- Monitoring webhooks usage and saving it in memory or Redis,
- Loosening _usubscribe API endpoint, so that the information returned from the /webhook endpoint is enough to unsubscribe,
- Owners can remove webhook without the unsubscribe key.

The endpoint lists all webhooks that are registered in a document, not just webhooks from a single table.
There are two status fields. First for the webhook, second for the last request attempt.
Webhook can have 5 statuses: 'idle', 'sending', 'retrying', 'postponed', 'error', which roughly describes what the
sendLoop is currently doing. The 'error' status describes a situation when all request attempts failed and the queue needs
to be drained, so some requests were dropped.

The last request status can only be: 'success', 'failure' or 'rejected'. Rejected means that the last batch was dropped because the
queue was too long.

Test Plan: New and updated tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3727
2022-12-13 22:46:40 +01:00
George Gevoian
e146f95c1c (core) Add new UI for writing memos
Summary:
Adds a new UI for writing access rule memos.

Migrates old memos (written as Python comments) to the new UI.

Test Plan: Browser and migration tests.

Reviewers: jarek, dsagal

Reviewed By: jarek

Subscribers: dsagal, paulfitz

Differential Revision: https://phab.getgrist.com/D3726
2022-12-12 17:52:01 -05:00
Jarosław Sadziński
07f5f86620 Fixing flaky tests. 2022-12-09 20:15:08 +01:00
Jarosław Sadziński
349c8acfdc Ignoring reserved characters in a resource key in a scoped translation helper 2022-12-09 20:14:59 +01:00
Jarosław Sadziński
92d4fca855 (core) Adding DELETE /api/docs/webhooks/queue endpoint to clear the queue
Summary:
Creating an API endpoint to cancel any queued webhook messages from
a document.

Test Plan: Updated

Reviewers: paulfitz, georgegevoian

Reviewed By: paulfitz, georgegevoian

Differential Revision: https://phab.getgrist.com/D3713
2022-12-01 12:23:19 +01:00
Dmitry S
29a7eadb85 (core) Fix problem with localStorage in some cross-origin embed situations
Summary:
- Handle the possibility that any access to localStorage causes error.
- Move getStorage() and getSessionStorage() safe functions to a separate file.
- Use these safe functions in more places.

Test Plan:
Added a test case, using a webdriver instance that blocks third-party cookies,
to enforce third-party restrictions. Added to gristUtil a way to override the
webdriver instance.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3719
2022-11-30 23:52:50 -05:00
Jarosław Sadziński
59942a23b6 (core) Limiting doc remove permission to owners.
Summary:
Guest editors added to a document were able to remove it. This limits this permission
by allowing only owners of a doc to delete it.

Test Plan: Updated

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal, anaisconce

Differential Revision: https://phab.getgrist.com/D3708
2022-11-30 23:37:18 +01:00
George Gevoian
1a6d427339 (core) Update sort and filter UI
Summary:
The sort and filter UI now has a more unified UI, with similar
capabilities that are accessible from different parts of Grist.
It's now also possible to pin individual filters to the filter bar,
which replaces the old toggle for showing all filters in the
filter bar.

Test Plan: Various tests (browser, migration, project).

Reviewers: jarek, dsagal

Reviewed By: jarek, dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3669
2022-11-17 15:33:45 -05:00
Jarosław Sadziński
af462fc938 (core) Fixing the ViewAs feature when the example user exists
Summary:
View as feature uses example.com emails for simulated users. This can break
when such a user already exists in the home db. Here we pretend that these users
don't exist during ACL checks.

Test Plan: Updated and existing

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3705
2022-11-17 16:50:45 +01:00
Jarosław Sadziński
955fdf4ae7 (core) Fixing multicolumn bug on a card view
Summary:
Fixing bug on a card view. Type selector was always showing
'mixed type' value.

Test Plan: Updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3704
2022-11-14 21:05:39 +01:00
Dmitry S
5c67e12aa5 (core) When a checkbox is clicked on a new record, set default values determined by linking
Summary: Fixes a bug (reported in https://community.getgrist.com/t/bug-toggle-column-in-linking-widget-not-triggering-default-value/1657)

Test Plan: Added a test case that fails without this fix.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3699
2022-11-10 14:34:40 -05:00
Jarosław Sadziński
46148aa125 (core) Fix for layout editor when fields are removed
Summary:
Fix for layout editor when fields are removed using the creator panel. Layout editor
wasn't updated properly when "layoutSpecs" were changed.

Test Plan: Updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3700
2022-11-10 10:36:43 +01:00
Jarosław Sadziński
2248053b09 (core) Fixing scrolling for linked sections
Summary:
When a grid is scrolled, and then data is changed (due to click in a linked section), some
records are not rendered, or position of the scroll container is corrupted

Test Plan: Added

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3661
2022-11-09 20:39:26 +01:00
Paul Fitzpatrick
42c3568835 (core) be stricter when replacing documents in the presence of granular access rules
Summary:
The /replace endpoint was built with home-level access control in mind. Updates needed:
  * Only an owner can now replace a document. Only owners are permitted to change granular access rules, and a document replacement could change granular access rules.
  * For the document being substituted in: the user must have complete access to view all material within it.

Test Plan: extended test

Reviewers: georgegevoian, dsagal

Reviewed By: georgegevoian, dsagal

Differential Revision: https://phab.getgrist.com/D3694
2022-11-09 14:14:09 -05:00
Paul Fitzpatrick
b29ce996b6 (core) updates from grist-core 2022-11-07 10:26:26 -05:00
Alex Hall
e590e65a3f (core) Allow requests from untrusted origins but without credentials
Summary:
Allow requests from untrusted origins instead of returning an error, but don't allow credentials (Cookie header) or API keys (Authorization header).

Allow setting the header `Content-type: application/json` as an alternative to `X-Requested-With: XMLHttpRequest` to make it easier for clients to make POST/PUT/PATCH/DELETE requests without authentication.

Discussion: https://grist.slack.com/archives/C0234CPPXPA/p1666355281535479

Test Plan: Added and updated DocApi tests. Tested manually how this affects requests made from a browser.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz, dsagal

Differential Revision: https://phab.getgrist.com/D3678
2022-11-03 13:33:23 +02:00
Paul Fitzpatrick
73ea6286f0
improve how server.restart functions in tests (#338)
The server.restart functionality in tests uses `TEST_CLEAN_DATABASE`. Make sure the way it wants to set this variable takes precedence over the environment.

Add some more `waitForServer`s near clicks.

Co-authored-by: George Gevoian <george@gevoian.com>
2022-11-01 20:01:15 -04:00
Jarosław Sadziński
f781709175 Ignoring expected server failures 2022-10-31 15:46:02 +01:00
Jarosław Sadziński
7c9cb9843e (core) Revealing hidden pages with visible children.
Summary:
When a page is hidden, all its nested pages are shown as children of
a different page that happens to be before (as in pagePos) that page.

This diff shows those pages as CENSORED.

Test Plan: Updated

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3670
2022-10-31 14:02:38 +01:00
Dmitry S
b263d83122 (core) Change 'Clear ... column' menu options to 'Reset', and make resetting columns reset type too
Summary:
The renaming is to clarify that the operation is more of a schema change, than
a data update. In particular, this is to reduce confusion why it is allowed to
anyone having Structure permission in Access Rules.

The resetting of type is a separate but related cleanup. Changing type to Any
returns the column to initial state, letting it guess type from new data, and
making it easy to enter a formula. It applies also to the "Clear and reset"
option in the Creator Panel.

Test Plan: Updated tests, added a check for type changing to Any.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3685
2022-10-31 08:53:42 -04:00
Arnaud Peich
79deeca640
Make a good part of the app localizable and add French translations (#325)
Co-authored-by: Yohan Boniface <yohanboniface@free.fr>
2022-10-28 09:11:08 -07:00
Paul Fitzpatrick
ec20e7fb68 (core) updates from grist-core 2022-10-28 11:49:49 -04:00
Jarosław Sadziński
d81bba625a (core) Fixing background color in frozen columns for zebra stripes
Summary: Background for frozen columns was set to transparent in recent PR, this diff is reverting it.

Test Plan: Updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3680
2022-10-28 09:43:34 +02:00
Louis Delbosc
eea2ef5cfb
Use url.hostname instead of url.host to allow host from environment variable (#326)
Co-authored-by <yohan.boniface@free.fr>
2022-10-25 14:59:17 -04:00
Jarosław Sadziński
64710b60f3 (core) Moving nbrowser tests to grist-core
Summary:
Moving bulk of nbrowser tests to core. Some tests were split and only part of them were moved.
Tests that are left are either: not suitable for grist-core (like billing) or are failing during browser tests (are not reliable).
Four fixtures directory (uploads, docs, exports-csv/excel) where completely moved to grist-core and are linked as folders.
Those changes allows to add an nbrowser test in grist-core or in the main test folder without any need to link it or link a fixture document.

Other changes:
- testrun.sh has been modified, now it runs tests from both folders (test and core/test),
- TestServer used in grist-core is now adding sample orgs and users (kiwi and others),

Test modified
- SelectionSummary: now it is run on a bigScreen, it was failing randomly
- Billing.ts: relative paths were used
- DateEditor: added waitForServer - it was failing in browser mode
- FrozenColumns, ImportFromGDrive, Printing: updated import paths
- UserManager.ts: was split into two parts (it assumed limited products)
- ViewLayoutResize.ts: this test is still in main repo, it is still failing in browser mode tests

Test Plan: Existing

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal, paulfitz

Differential Revision: https://phab.getgrist.com/D3664
2022-10-25 17:22:54 +02:00
George Gevoian
ace698a558
Update saveViewSection test for grist-core (#323) 2022-10-24 16:19:09 -04:00
Paul Fitzpatrick
0c82b746d0 (core) updates from grist-core 2022-10-24 10:53:18 -04:00
Jarosław Sadziński
7c8db90aef (core) Fixing click-away bug for the cell color widget
Summary:
After introducing multi columns operation, color picker
could save a cell style for a wrong column, if the save operation
was triggered by user clicking on one of the cells.

Test Plan: Updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3668
2022-10-24 16:21:14 +02:00
Jarosław Sadziński
6460c22a89 (core) Changing shortcuts for adding and removing rows
Summary:
New shortcuts for removing and adding rows.
For adding a row we now have Mod+(Shift)+Enter
For removing rows we now have Mod+Delete/Mod+Backspace

Before removing rows, the user is prompted to confirm, this prompt
can be dismissed and this setting can be remembered. User needs
to confirm only when using shortcut.

Old shortcuts are still active and shows information about this change.
This information is shown only once, after this shortcuts have default
behavior (zooming).
New users don't see this explanation.

Test Plan: Updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3655
2022-10-21 18:45:25 +02:00
Alex Hall
62792329c3 (core) DocApi meta endpoints: GET /tables and POST/PATCH /tables and /columns
Summary:
Adds new API endpoints to list tables in a document and create or modify tables and columns. The request and response formats are designed to mirror the style of the existing `GET /columns` and `GET/POST/PATCH /records` endpoints.

Discussion: https://grist.slack.com/archives/C0234CPPXPA/p1665139807125649?thread_ts=1628957179.010500&cid=C0234CPPXPA

Test Plan: DocApi test

Reviewers: jarek

Reviewed By: jarek

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3667
2022-10-21 10:15:46 +02:00
Jarosław Sadziński
18ba7994ed Adding type inference and makeT helper function 2022-10-20 10:34:38 +02:00
Jarosław Sadziński
2f29df1b17 Changing domT to a default translation function for browser 2022-10-19 20:44:56 +02:00
Jarosław Sadziński
2586b595a5 Adding domT method for component interpolation 2022-10-19 19:22:37 +02:00
jarek
4bb1d8c011
Merge pull request #312 from incubateur-territoires/arnaudpeich/Split_client_and_server_translations_organize_by_filename
Split client and server translations, organize by filename
2022-10-19 10:38:39 +02:00
George Gevoian
acc218398d (core) Fix hidden columns bug when editing data selection
Summary:
Editing data selection would sometimes cause columns to be hidden in the updated view. A
missing conditional was the culprit: generally, field visibility shouldn't be modified after the view is
updated, but we make an exception for charts to keep certain fields visible or hidden between
updates, so that chart configuration doesn't change too significantly and cause unexpected
data to be displayed. This special behavior for charts was erroneously being applied to non-charts
as well.

Also, when no columns were visible in a view, opening the row menu would cause an error to be
thrown. A loop was inadvertently using null control variables - an explicit check for non-null loop
variables was added, which skips the loop when no columns are visible.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3650
2022-10-18 08:36:11 -07:00
Paul Fitzpatrick
bf24c29de4 (core) updates from grist-core 2022-10-17 10:53:21 -04:00
George Gevoian
7682b3c320 (core) Fix bug preventing changes to ref list table
Summary:
Fixes bug that prevented the table of a reference list from being
changed in the column transform UI.

Test Plan: Browser test.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3666
2022-10-17 07:49:54 -07:00
Jarosław Sadziński
bfd7243fe2 (core) Comments
Summary:
First iteration for comments system for Grist.
- Comments are stored in a generic metatable `_grist_Cells`
- Each comment is connected to a particular cell (hence the generic name of the table)
- Access level works naturally for records stored in this table
-- User can add/read comments for cells he can see
-- User can't update/remove comments that he doesn't own, but he can delete them by removing cells (rows/columns)
-- Anonymous users can't see comments at all.
- Each comment can have replies (but replies can't have more replies)

Comments are hidden by default, they can be enabled by COMMENTS=true env variable.
Some things for follow-up
- Avatars, currently the user's profile image is not shown or retrieved from the server
- Virtual rendering for comments list in creator panel. Currently, there is a limit of 200 comments.

Test Plan: New and existing tests

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3509
2022-10-17 13:38:21 +02:00
Jarosław Sadziński
8be920dd25 (core) Multi-column configuration
Summary:
Creator panel allows now to edit multiple columns at once
for some options that are common for them. Options that
are not common are disabled.

List of options that can be edited for multiple columns:
- Column behavior (but limited to empty/formula columns)
- Alignment and wrapping
- Default style
- Number options (for numeric columns)
- Column types (but only for empty/formula columns)

If multiple columns of the same type are selected, most of
the options are available to change, except formula, trigger formula
and conditional styles.

Editing column label or column id is disabled by default for multiple
selection.

Not related: some tests were fixed due to the change in the column label
and id widget in grist-core (disabled attribute was replaced by readonly).

Test Plan: Updated and new tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3598
2022-10-17 09:51:19 +02:00
Arnaud Peich
5f66a8f298
Return 403 error when origin is not trusted (#310) 2022-10-13 09:13:01 -04:00
Arnaud Peich
390d8406a6 Fix tests 2022-10-13 12:05:19 +02:00
Paul Fitzpatrick
303d071de1 (core) updates from grist-core 2022-10-04 13:22:17 -04:00
Jarosław Sadziński
356090abae (core) Fix for tests failures
Summary:
- DocApi test for Allowed Origin was using a home server endpoint
- Fixing waitForServer, as gristApp can be unavailable for a moment when browser is refreshed
- Fixing MergedOrgs tests typing issue

Test Plan: Updated

Reviewers: cyprien, paulfitz

Reviewed By: cyprien, paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3648
2022-10-03 15:11:59 +02:00
George Gevoian
364610c69d (core) Add cell selection summary
Summary:
Adds a cell selection summary to grid view that shows either a count or
sum of all the selected values. Implementation was done by Dmitry.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: paulfitz, dsagal, jarek

Differential Revision: https://phab.getgrist.com/D3630
2022-09-30 09:11:46 -07:00
Jarosław Sadziński
a8ead09fc5 Adding new lines at the end 2022-09-30 17:00:38 +02:00
Jarosław Sadziński
51d839f082 Adding dedicated folder for external tests 2022-09-30 16:45:35 +02:00
Paul Fitzpatrick
433e1ecfc2 (core) updates from grist-core 2022-09-29 13:14:04 -04:00
Jarosław Sadziński
5219932a1f (core) i18
Summary:
Adding initial work for localization support.

Summary in https://grist.quip.com/OtZKA6RHdQ6T/Internationalization-and-Localization

Test Plan: Not yet

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3633
2022-09-29 18:02:09 +02:00
Louis Delbosc
49b1749e98
Add function to allow hosts from environment variables (#287)
* Add allowed host option to handle CORS requests
* Update readme with new GRIST_ALLOWED_HOSTS environment variable
2022-09-28 12:33:53 -04:00
Alex Hall
1864b7ba5d (core) Add BulkAddOrUpdateRecord action for efficiency
Summary:
This diff adds a new `BulkAddOrUpdateRecord` user action which is what is sounds like:

- A bulk version of the existing `AddOrUpdateRecord` action.
- Much more efficient for operating on many records than applying many individual actions.
- Column values are specified as maps from `colId` to arrays of values as usual.
- Produces bulk versions of `AddRecord` and `UpdateRecord` actions instead of many individual actions.

Examples of users wanting to use something like `AddOrUpdateRecord` with large numbers of records:

- https://grist.slack.com/archives/C0234CPPXPA/p1651789710290879
- https://grist.slack.com/archives/C0234CPPXPA/p1660743493480119
- https://grist.slack.com/archives/C0234CPPXPA/p1660333148491559
- https://grist.slack.com/archives/C0234CPPXPA/p1663069291726159

I tested what made many `AddOrUpdateRecord` actions slow in the first place. It was almost entirely due to producing many individual `AddRecord` user actions. About half of that time was for processing the resulting `AddRecord` doc actions. Lookups and updates were not a problem. With these changes, the slowness is gone.

The Python user action implementation is more complex but there are no surprises. The JS API now groups `records` based on the keys of `require` and `fields` so that `BulkAddOrUpdateRecord` can be applied to each group.

Test Plan: Update and extend Python and DocApi tests.

Reviewers: jarek, paulfitz

Reviewed By: jarek, paulfitz

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3642
2022-09-28 17:58:33 +02:00
Paul Fitzpatrick
7a37797f3b (core) tweak duplicate document test to be more reliable
Summary:
When running as a deployment test, there are some timing issues
in the DuplicateDocument test. These tweaks appear to resolve
them.

Test Plan: checked with staging_deployment test target

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3638
2022-09-22 14:26:56 -04:00
Paul Fitzpatrick
d55b5110ac (core) remove deprecated /download endpoint in favor of newer /api/docs/NNNN/download
Summary:
This endpoint has started to fail when called between a pair
of doc workers. The simplest fix is to simply remove it, it serves no
purpose.

Test Plan: added basic deployment test

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3636
2022-09-20 15:26:04 -04:00
Paul Fitzpatrick
b1921209df (core) updates from grist-core 2022-09-19 12:47:15 -04:00
Jarosław Sadziński
473dcf7c15 (core) Fix for a FreeTeam test
Summary:
FreeTeam test was overlapping with DuplicateDocument test.
It was leaving some documents in personal workspace.

Test Plan: Updated tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3632
2022-09-16 16:48:27 +02:00
Louis Delbosc
494a683332
Export xlsx #256 (#270)
XLSX export of active view / table

Co-authored-by: Louis Delbosc <louis.delbosc.prestataire@anct.gouv.fr>
Co-authored-by: Vincent Viers <vincent.viers@beta.gouv.fr>
2022-09-14 14:55:44 -04:00
Jarosław Sadziński
81c623109f (core) Adding empty workspace intro
Summary: Adding title and buttons for empty workspace screen.

Test Plan: Updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3622
2022-09-12 16:19:31 +02:00
Paul Fitzpatrick
b9441cf8fd (core) updates from grist-core 2022-09-12 09:14:22 -04:00
Jarosław Sadziński
2a3328abfc (core) Clearing selection after rows or cols were removed
Summary:
The selection was not cleared after removing rows, and it still
kept deleted rows ids. Removing selected rows once again
produced an assertion error.
Also, the selection is now cleared when columns are removed.

Test Plan: Updated tests

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3623
2022-09-12 08:40:07 +02:00
Paul Fitzpatrick
bd9ede9d13
give parseDate test more time (#278)
The time it takes parseDate to run depends on how many timezone
names moment knows about.
2022-09-09 17:05:29 -04:00
Jarosław Sadziński
2438a63255 (core) Moving widget tests to core
Summary:
- Custom widget tests are now in grist-core
- Adding buildtools for grist-plugin-api.js

Test Plan: Existing tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3617
2022-09-06 17:17:14 +02:00
Paul Fitzpatrick
d7b3fb972c (core) upgrade typeorm so we can support newer postgres
Summary:
upgrade typeorm version, so Grist can run against newer versions of postgres.

Dusted off some old benchmarking code to verify that important queries don't get slower. They don't appear to, unlike for some intermediate versions of typeorm I tried in the past.

Most of the changes are because `findOne` changed how it interprets its arguments, and the value it returns when nothing is found. For the return value, I stuck with limiting its impact by emulating old behavior (returning undefined rather than null) rather than propagating the change out to parts of the code unrelated to the database.

Test Plan: existing tests pass; manual testing with postgres 10 and 14

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3613
2022-09-02 15:34:21 -04:00
Jarosław Sadziński
b6f5718ad0 (core) Fixing ACIndex highlightMatches functions
Summary:
Highlighting wasn't working correctly for the new
normalized search for autocomplate widgets.

Test Plan: Existing tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3602
2022-08-26 08:34:16 +02:00
George Gevoian
2cb783ea7b (core) Fix bugs with intervals
Summary:
Fixes some bugs involving intervals, and updates RandomizedTimer to support both fixed and
randomized delays, and to better handle async callbacks.

 * Fixed a bug where Throttle would queue up many pidusage calls due to the use of
    setInterval, and the async nature of the calls.

 * Fixed a but where RandomizedTimer (now just Interval) would not be disabled in
    ActiveDoc on doc shutdown if initialization had not yet settled.

Test Plan: Tested manually.

Reviewers: jarek, dsagal

Reviewed By: jarek, dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3604
2022-08-25 12:38:36 -07:00
Jarosław Sadziński
2997434815 (core) Showing a raw data section on a popup
Summary:
Show raw data will now open a popup with
raw section instead of redirecting to raw data page.

Adding new anchor link type "a2" that is able to open
any section in a popup on a current view.

Not related:
Fixing highlightMatches function, after merging core PR.

Test Plan: Updated tests

Reviewers: alexmojaki, georgegevoian

Reviewed By: alexmojaki, georgegevoian

Subscribers: georgegevoian, alexmojaki

Differential Revision: https://phab.getgrist.com/D3592
2022-08-25 09:30:07 +02:00
Jarosław Sadziński
a52d56f613 (core) Moving client and common tests to core
Summary:
- Moved /test/client and /test/common to core.
- Moved two files (CircularArray and RecentItems) from app/common to core/app/common.
- Moved resetOrg test to gen-server.
- `testrun.sh` is now invoking common and client test from core.
- Added missing packages to core's package.json (and revealed underscore as it is used in the main app).
- Removed Coord.js as it is not used anywhere.

Test Plan: Existing tests

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3590
2022-08-23 19:20:10 +02:00
Alex Hall
e06f0bc1d8 (core) Retry flaky daily API usage test
Summary: This particular test fails often enough to be annoying but not often enough to be worrying. It's not clear why, but it seems like a race condition involving redis. Fixing the test 'properly' seems hard and not worth the effort. Looking at the past 20 Jenkins builds, I found this test failing once. If we assume that the probability of failing is 1/20 (it's probably less since the test actually runs 3 times with different server configurations) then the probability of failing 3 times independently is (1/20)^3 = 1/8000, so `this.retry(3)` seems like a good enough solution. See also: https://grist.slack.com/archives/C0234CPPXPA/p1652909955773049

Test Plan: Manually made the test fail randomly 90% of the time, in which case `this.retries(3)` was not usually enough to prevent failures, but `this.retries(300)` was.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3595
2022-08-23 15:22:08 +02:00
Paul Fitzpatrick
028146f88a (core) updates from grist-core 2022-08-22 10:23:26 -04:00
Jarosław Sadziński
44b4ec7edf (core) Removing duplicated tests
Summary: Removing duplicated tests that were moved to grist-core.

Test Plan: Existing tests

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3588
2022-08-19 10:29:53 +02:00
Ronan Amicel
0249b3d31b Add tests for autocomplete without accents 2022-08-17 19:11:52 +02:00
Paul Fitzpatrick
e95b215405 (core) updates from grist-core 2022-08-15 10:00:08 -04:00
Jarosław Sadziński
22ed7d3024 Skipping tests that can't be run on external server. 2022-08-12 17:23:22 +02:00
Jarosław Sadziński
a44a59e591 Tweaking logging in github action and test_under_docker script 2022-08-10 14:16:27 +02:00
Jarosław Sadziński
c7a6784873 Removing grist logs from tests 2022-08-10 13:33:08 +02:00
Jarosław Sadziński
9da6fec7ad Moving 3 ACinput tests to grist core 2022-08-10 13:10:50 +02:00
Jarosław Sadziński
9e4d802405 (core) Implementing row conditional formatting
Summary:
Conditional formatting can now be used for whole rows.
Related fix:
- Font styles weren't applicable for summary columns.
- Checkbox and slider weren't using colors properly

Test Plan: Existing and new tests

Reviewers: paulfitz, georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3547
2022-08-09 20:11:36 +02:00
Paul Fitzpatrick
c72ebf61a7 (core) updates from grist-core 2022-08-08 09:31:07 -04:00
Louis Delbosc
a82d02ebcc feat: add tests for feature 2022-08-08 13:30:12 +02:00
Jarosław Sadziński
7210ef3117 (core) Removing viewer tests from HomeIntro deployment tests
Summary:
Removing welcome tests for viewers from HomeIntro tests, as
they are run against external servers.

Test Plan: moved tests to the FreeTeam test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3563
2022-08-03 19:47:49 +02:00
Jarosław Sadziński
fe220d576f Adding tests for left panel 2022-08-03 16:15:45 +02:00
Jarosław Sadziński
c359547f6b (core) Welcome intro for viewers on a team site.
Summary:
Adding intro for a viewer on a teamsite.
Showing upgrade button for owners only.

Test Plan: new test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3557
2022-08-03 12:50:14 +02:00
Cyprien P
5f51dd7a00 (core) Fix nbrowser/CopyPaste test on Mac
Summary: `nbrowser/CopyPaste` was failing on Mac. Diff fixes that issue.

Test Plan: Update `test/nbrowser/CopyPaste` and `core/test/server/customUtil.ts`.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3552
2022-07-29 14:08:29 +02:00
George Gevoian
aeba738f7c (core) Add product for new personal plan
Summary:
Adds the new personal plan as a product that will be available
in the future. Can be enabled along with other plan-related via
an environment variable.

Test Plan: Browser tests and existing tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3533
2022-07-26 11:33:23 -07:00
Jarosław Sadziński
5c8211c61d (core) Fixing selectby error on the creator panel.
Summary: [Select By] in the creator panel was bugged. It wasn't refreshed in some cases as the observable array that needed to be created seemed too complicated. This Diff recomputes this array when the user wants to change the selection.

Test Plan: added tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3541
2022-07-26 15:46:11 +02:00
Alex Hall
460f22b701 (core) Don't show hidden tables in table data options when adding widgets
Summary:
As reported in https://grist.slack.com/archives/C069RUP71/p1655316194602829, when a table is hidden by ACL, it was still showing as a blank 'ghost' option to select data when adding a widget:

{F55498}

The fix is simply to return `true` from `isHiddenTable` for empty table IDs, which indicate a table hidden by ACL. `TableRec.isHidden` is supposed to match this so I updated it too, and I cleaned up a tiny bit of other related code.

Test Plan: Extended `nbrowser/AccessRules1.ts` to test the data options when adding widgets.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3530
2022-07-21 16:28:28 +02:00
George Gevoian
5c0a250309 (core) Prevent raw table names from overflowing
Summary:
Modifies CSS so that text overflow is handled by displaying an
ellipsis when raw table names exceed the width of their container.

Test Plan: Tested manually.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3524
2022-07-19 08:25:31 -07:00
Alex Hall
4b258ae0fa (core) Fix 'select by' when adding summary table widget to page
Summary:
When adding a summary table widget to a page and using 'select by' in the add widget config (as opposed to in the right panel for an existing widget):

1. If an equivalent summary table already exists, use its referencing columns (if any) to construct link nodes. Previously the source table columns were being used instead, which could include referencing columns that don't have any equivalent in the summary table, and exclude referencing columns in the summary table.
2. If no such summary table exists yet, then keep using the source table columns, but only the selected groupby columns, and
3. After the summary table is created, correct the `linkTargetColRef` (which points to a source table column) to the corresponding column from the new summary table instead.

This fixes bugs which only appeared recently since 'select by' for a summary table previously involved no target columns.

Test Plan: Added two new tests to `nbrowser/SelectBySummaryRef`, and confirmed that they fail without the fixes to all three points above.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3527
2022-07-19 16:28:38 +02:00
George Gevoian
c70b427483 (core) Fix error when filtering empty choice and choice list columns
Summary:
A client error should no longer be thrown when filtering an empty Choice
or Choice List column.

Test Plan: Browser tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3528
2022-07-17 23:35:27 -07:00
Alex Hall
1a6e1872de (core) Miscellaneous little logging improvements
Summary:
1. Log errors in `ActiveDoc.loadDoc` as errors, not just warnings, except for a common 'Cannot create fork' error caused by deployment tests.

2. Log the method name that had an error in `server/lib/Client.ts`.

Discussion: https://grist.slack.com/archives/CR8HZ4P9V/p1652364998893169

Following up on https://phab.getgrist.com/D3522

Test Plan: tested manually, particularly by running the nbrowser/Fork test that led to the initial noisy errors in Slack.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3525
2022-07-15 00:21:44 +02:00
Paul Fitzpatrick
ec8ab598cb (core) add a yarn run cli tool, and add a sqlite gristify option
Summary:
This adds rudimentary support for opening certain SQLite files in Grist.

If you have a file such as `landing.db` in Grist, you can convert it to Grist format by doing (either in monorepo or grist-core):
```
yarn run cli -h
yarn run cli sqlite -h
yarn run cli sqlite gristify landing.db
```

The file is now openable by Grist. To actually do so with the regular Grist server, you'll need to either import it, or convert some doc you don't care about in the `samples/` directory to be a soft link to it (and then force a reload).

This implementation is a rudimentary experiment. Here are some awkwardnesses:
 * Only tables that happen to have a column called `id`, and where the column happens to be an integer, can be opened directly with Grist as it is today. That could be generalized, but it looked more than a Gristathon's worth of work, so I instead used SQLite views.
 * Grist will handle tables that start with an uncapitalized letter a bit erratically. You can successfully add columns, for example, but removing them will cause sadness - Grist will rename the table in a confused way.
 * I didn't attempt to deal with column names with spaces etc (though views could deal with those).
 * I haven't tried to do any fancy type mapping.
 * Columns with constraints can make adding new rows impossible in Grist, since Grist requires that a row can be added with just a single cell set.

Test Plan: added small test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3502
2022-07-14 12:00:30 -04:00
Alex Hall
b8486dcdba (core) Nice summary table IDs
Summary:
Changes auto-generated summary table IDs from e.g. `GristSummary_6_Table1` to `Table1_summary_A_B` (meaning `Table1` grouped by `A` and `B`). This makes it easier to write formulas involving summary tables, make API requests, understand logs, etc.

Because these don't encode the source table ID as reliably as before, `decode_summary_table_name` now uses the summary table schema info, not just the summary table ID. Specifically, it looks at the type of the `group` column, which is `RefList:<source table id>`.

Renaming a source table renames the summary table as before, and now renaming a groupby column renames the summary table as well.

Conflicting table names are resolved in the usual way by adding a number at the end, e.g. `Table1_summary_A_B2`. These summary tables are not automatically renamed when the disambiguation is no longer needed.

A new migration renames all summary tables to the new scheme, and updates formulas using summary tables with a simple regex.

Test Plan:
Updated many tests to use the new style of name.

Added new Python tests to for resolving conflicts when renaming source tables and groupby columns.

Added a test for the migration, including renames in formulas.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3508
2022-07-14 12:09:56 +02:00
Paul Fitzpatrick
f91f45b26d (core) support granular read access for attachments
Summary:
When a user requests to read the contents of an attachment, only allow the request if there exists a cell in an attachment column that contains the attachment and which they have read access to.

This does not cover:
 * Granular write access for attachments. In particular, a user who can write to any attachment column should be considered to have full read access to all attachment columns, currently.
 * Access control of attachment metadata such as name and format.

The implementation uses a sql query that requires a scan, and some notes on how this could be optimized in future. The web client was updated to specify the cell to check for access, and performance seemed fine in casual testing on a doc with 1000s of attachments. I'm not sure how performance would hold up as the set of access rules grows as well.

Test Plan: added tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3490
2022-07-07 07:22:02 -04:00
George Gevoian
a051830aeb (core) Show summary tables on Raw Data page
Summary:
Summary tables now have their own raw viewsection, and are shown
under Raw Data Tables on the Raw Data page.

Test Plan: Browser and Python tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3495
2022-07-06 09:41:48 -07:00
Dmitry S
51ff72c15e (core) Faster builds all around.
Summary:
Building:
- Builds no longer wait for tsc for either client, server, or test targets. All use esbuild which is very fast.
- Build still runs tsc, but only to report errors. This may be turned off with `SKIP_TSC=1` env var.
- Grist-core continues to build using tsc.
- Esbuild requires ES6 module semantics. Typescript's esModuleInterop is turned
  on, so that tsc accepts and enforces correct usage.
- Client-side code is watched and bundled by webpack as before (using esbuild-loader)

Code changes:
- Imports must now follow ES6 semantics: `import * as X from ...` produces a
  module object; to import functions or class instances, use `import X from ...`.
- Everything is now built with isolatedModules flag. Some exports were updated for it.

Packages:
- Upgraded browserify dependency, and related packages (used for the distribution-building step).
- Building the distribution now uses esbuild's minification. babel-minify is no longer used.

Test Plan: Should have no behavior changes, existing tests should pass, and docker image should build too.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3506
2022-07-04 10:42:40 -04:00
Dmitry S
d5ebd49eb7 (core) Parallelize jenkins by running on multiple machines.
Summary:
- Get Jenkins to run on 4 agents in parallel, each executing 4 parallel test runs.
- Add a scheme for automatically selecting non-conflicting ports and Redis DB numbers.
- Add a scheme for automatically deciding how to group tests in large suites (nbrowser, server) to keep groups roughly equal.
- Add a recording of test timings, that's used for the auto-grouping.
- Fix tests that were sensitive to the order in which they were running.

Test Plan: All 5020 tests passed in 9 minutes (as opposed to the previous passing run which took 30).

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3500
2022-07-01 16:18:36 -04:00
Jarosław Sadziński
aefe451bab (core) Polishing upgrade plan UI
Summary:
- Update nudge boxes content and collapsing on personal and free team site
- New confirmation after upgrading from a free team site
- Refactoring ProductUpgrade code, splitting plans / modals and nudges

Test Plan: Manual and updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3481
2022-06-29 18:28:37 +02:00
Dmitry S
dd2eadc86e (core) Speed up and upgrade build.
Summary:
- Upgrades to build-related packages:
  - Upgrade typescript, related libraries and typings.
  - Upgrade webpack, eslint; add tsc-watch, node-dev, eslint_d.

- Build organization changes:
  - Build webpack from original typescript, transpiling only; with errors still
    reported by a background tsc watching process.

- Typescript-related changes:
  - Reduce imports of AWS dependencies (very noticeable speedup)
  - Avoid auto-loading global @types
  - Client code is now built with isolatedModules flag (for safe transpilation)
  - Use allowJs to avoid copying JS files manually.

- Linting changes
  - Enhance Arcanist ESLintLinter to run before/after commands, and set up to use eslint_d
  - Update eslint config, and include .eslintignore to avoid linting generated files.
  - Include a bunch of eslint-prompted and eslint-generated fixes
  - Add no-unused-expression rule to eslint, and fix a few warnings about it

- Other items:
  - Refactor cssInput to avoid circular dependency
  - Remove a bit of unused code, libraries, dependencies

Test Plan: No behavior changes, all existing tests pass. There are 30 tests fewer reported because `test_gpath.py` was removed (it's been unused for years)

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3498
2022-06-27 16:10:10 -04:00
Dmitry S
a91d493ffc (core) Fix issue with 'UNEXPECTED ORDER OF CALLBACKS' in Client.ts.
Summary:
- Substantial refactoring of the logic when the server fails to send some
  messages to a client.
- Add seqId numbers to server messages to ensure reliable order.
- Add a needReload flag in clientConnect for a clear indication whent the
  browser client needs to reload the app.
- Reproduce some potential failure scenarios in a test case (some of which
  previously could have led to incorrectly ordered messages).
- Convert other Comm tests to typescript.
- Tweak logging of Comm and Client to be slightly more concise (in particular,
  avoid logging sessionId)

Note that despite the big refactoring, this only addresses a fairly rare
situation, with websocket failures while server is trying to send to the
client. It includes no improvements for failures while the client is sending to
the server.

(I looked for an existing library that would take care of these issues. A relevant article I found is https://docs.microsoft.com/en-us/azure/azure-web-pubsub/howto-develop-reliable-clients, but it doesn't include a library for both ends, and is still in review. Other libraries with similar purposes did not inspire enough confidence.)

Test Plan: New test cases, which reproduce some previously problematic scenarios.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3470
2022-06-16 23:51:14 -04:00
Paul Fitzpatrick
561d9696aa (core) clean up interaction of forward auth with session
Summary:
For self-hosted Grist, forward auth has proven useful, where
some proxy wrapped around Grist manages authentication, and
passes on user information to Grist in a trusted header.
The current implementation is adequate when Grist is the
only place where the user logs in or out, but is confusing
otherwise (see https://github.com/gristlabs/grist-core/issues/207).
Here we take some steps to broaden the scenarios Grist's
forward auth support can be used with:

  * When a trusted header is present and is blank, treat
    that as the user not being logged in, and don't look
    any further for identity information. Specifically,
    don't look in Grist's session information.
  * Add a `GRIST_IGNORE_SESSION` flag to entirely prevent
    Grist from picking up identity information from a cookie,
    in order to avoid confusion between multiple login methods.
  * Add tests for common scenarios.

Test Plan: added tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3482
2022-06-15 13:06:12 -04:00
Dmitry S
b57a211741 (core) Fix issue with sandboxUtil where %s in message would get interpreted wrongly
Summary:
- Also converted sandboxUtil to typescript.
- The issue with %s manifested when a Python traceback contained "%s" in the
  string; in that case the object with log metadata (e.g. docId) would
  confusingly replace %s as if it were part of the message from Python.

Test Plan: Added a test case for the fix.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3486
2022-06-14 10:34:00 -04:00
Jarosław Sadziński
d92a761f6e (core) Product update popups and hosted stripe integration
Summary:
- Showing nudge to individual users to sign up for free team plan.
- Implementing billing page to upgrade from free team to pro.
- New modal with upgrade options and free team site signup.
- Integrating Stripe-hosted UI for checkout and plan management.

Test Plan: updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3456
2022-06-08 21:10:49 +02:00
Dmitry S
4f1cb53b29 (core) Converting server-side Comm.js to typescript
Summary:
- Add app/common/CommTypes.ts to define types shared by client and server.
- Include @types/ws npm package

Test Plan: Intended to have no changes in behavior

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3467
2022-06-07 15:47:17 -04:00
George Gevoian
090d9af21d (core) Broadcast doc usage updates to clients
Summary:
Introduces a new message type, docUsage, that's broadcast to all connected
clients whenever document usage is updated in ActiveDoc.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3451
2022-06-06 09:55:34 -07:00
Dmitry S
acddd25cfd (core) Update design of empty docs home page, and add a "Manage Team" button.
Summary:
- Remove the empty-folder icon
- Add an "Invite team members" button for owners on empty team sites
- Add a "Browse Templates" button for all other cases on empty sites
- Update intro text for team, including a link to Sprouts
- Update intro text for personal/anon.
- Include a Free/Pro tag for team sites (for now, only "Free")
- Add a "Manage Team" button for owners on home page of all team sites.
- Polished the UI of UserManager: add a transition for the background, and
  delay the appearance of the spinner for fast loads.

Test Plan: Fixed up the HomeIntro tests; added test case for Manage Team button

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3459
2022-06-03 12:58:57 -04:00
Paul Fitzpatrick
b9a4b2b58f (core) add missing tsconfig file that affects IDEs
Summary:
I missed committing a file that is important for editing files comfortably in the ext directory in an IDE. This diff:

 * Adds tsconfig-base-ext.json - that was the only intended change
 * Unrelated: Forces all creation of connections to the home db through a new `getOrCreateConnection` method which changes the `busy_timeout` if using Sqlite. This was an attempt to fix random "database is locked" test failures. I believe multiple connections to the home db as an sqlite file do not happen in self-hosted Grist (where there is a single node process) or in our SaaS (where the database is in postgres). It does affect Grist started using `devServerMain.ts` (where multiple processes accessing same database are started) or various test configurations when extra database connections are opened.
 * Unrelated: I added a `busy_timeout` for session storage, when it uses Sqlite. Again, I don't believe this affects self-hosted Grist or our SaaS.
 * Tweaked a `BillingDiscount` test that looked perhaps vulnerable to a stripe request stalling.

I can't be sure my tweaks actually help, since I didn't succeed in replicating the failures. Update: looks like the "locked" error can still happen :(

Test Plan: manual

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3450
2022-05-27 13:48:58 -04:00
Alex Hall
fcbad1c887 (core) Add GET /attachments endpoint for listing attachment metadata
Summary: Combines the code and behaviour of the existing endpoints `GET /records` (for the general shape of the result and the parameters for sort/filter/limit etc) and retrieving a specific attachment with `GET /attachments/:id` for handling fields specific to attachments.

Test Plan: Added a DocApi test. Also updated one test to use the new endpoint instead of raw `GET /tables/_grist_Attachments/records`.

Reviewers: cyprien

Reviewed By: cyprien

Subscribers: cyprien

Differential Revision: https://phab.getgrist.com/D3443
2022-05-20 20:30:14 +02:00
Jarosław Sadziński
0ab9e4a6a0 (core) Adding GristConnect login system
Summary:
New login system to allow simple SSO flow that is based on Discourse description that is available at:
https://meta.discourse.org/t/discourseconnect-official-single-sign-on-for-discourse-sso/13045

Test Plan: New core test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3418
2022-05-18 20:28:25 +02:00
George Gevoian
2fd8a34ff8 (core) Move Notifier to /ext
Summary:
This makes it possible to configure a SendGrid-based Notifier
instance via a JSON configuration file.

Test Plan: Tested manually.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3432
2022-05-18 08:02:32 -07:00
Paul Fitzpatrick
0a61d740ba (core) clear redis state for docapi tests
Summary:
DocApi tests occasionally fail in a confusing way due to redis
state. This resets the redis state at the beginning of the test.

Test Plan: existing tests pass

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3424
2022-05-11 17:06:42 -04:00
George Gevoian
52eb5325c2 (core) Use different user in ActionLog tests
Summary:
Nearby deployment tests used the same user, which was
causing the full suite to fail. Repeated logins with a second
verification step (TOTP) fail in Cognito if the same verification code
is used twice. This happened to be the case since the gap
between tests wasn't long enough for a new code to be generated.

Test Plan: Manually verified deployment tests now pass.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3414
2022-05-05 14:04:03 -07:00
George Gevoian
1e42871cc9 (core) Add attachment and data size usage
Summary:
Adds attachment and data size to the usage section of
the raw data page. Also makes in-document usage banners
update as user actions are applied, causing them to be
hidden/shown or updated based on the current state of
the document.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3395
2022-05-04 13:46:55 -07:00
Jarosław Sadziński
f194d6861b (core) Updating RawData views
Summary:
- Better focus on the widget title
- Adding columns only to the current view section
- New popup with options when user wants to delete a page
- New dialog to enter table name
- New table as a widget doesn't create a separate page
- Removing a table doesn't remove the primary view

Test Plan: Updated and new tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3410
2022-05-04 21:41:42 +02:00
Alex Hall
0beb2898cb (core) Add flexibility to daily API usage limit
Summary: Allow exceeding the daily API usage limit for a doc based on additional allocations for the current hour and minute. See the doc comment on getDocApiUsageKeysToIncr for details. This means that up to 5 redis keys may be relevant at a time for a single document.

Test Plan: Updated and expanded 'Daily API Limit' tests.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3368
2022-04-28 16:22:18 +02:00
Jarosław Sadziński
6f00106d7c (core) Raw renames
Summary:
A new way for renaming tables.

  - There is a new popup to rename section (where you can also rename the table)
  - Renaming/Deleting page doesn't modify/delete the table.
  - Renaming table can rename a page if the names match (and the page contains a section with that table).
  - User can rename table in Raw Data UI in two ways - either on the listing or by using the section name popup
  - As before, there is no way to change tableId - it is derived from a table name.
  - When the section name is empty the table name is shown instead.
  - White space for section name is allowed (to discuss) - so the user can just paste '   '.
  - Empty name for a page is not allowed (but white space is).
  - Some bugs related to deleting tables with attached summary tables (and with undoing this operation) were fixed (but not all of them yet).

Test Plan: Updated tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3360
2022-04-27 22:21:55 +02:00
Jarosław Sadziński
995bf9b63a (core) Distinct style rules for summary columns
Summary:
Summary columns now have their own conditional rules,
which are not shared with sister columns.

Test Plan: New test

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3388
2022-04-27 20:51:23 +02:00
Alex Hall
a701b4bf13 (core) Remove expired attachments every hour and on shutdown
Summary:
Call ActiveDoc.removeUnusedAttachments every hour using setInterval, and in ActiveDoc.shutdown (which also clears said interval).

Unrelated: small fix to my webhooks code which was creating a redis client on shutdown just to quit it.

Test Plan:
Tweaked DocApi test to remove expired attachments by force-reloading the doc, so that it removes them during shutdown. Extracted a new testing endpoint /verifyFiles to support this test (previously running that code only happened with `/removeUnused?verifyfiles=1`).

Tested the setInterval part manually.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3387
2022-04-22 20:43:59 +02:00
Jarosław Sadziński
77ef9df27d (core) Adding new command Duplicate rows
Summary:
New command "Duplicate rows" is available in the Row/Card Context Menu and as a keyboard shortcut Ctrl+Alt+C.
- All selected rows are duplicated (even if only a single column is selected)
- Rows are inserted immediately after the last selected row (using manualSort value).
- Formulas and CENSORED fields are not copied.
Implemented on the UI level (no new action).

Test Plan: new test

Reviewers: cyprien

Reviewed By: cyprien

Differential Revision: https://phab.getgrist.com/D3371
2022-04-20 17:29:48 +02:00
Alex Hall
64a5c79dbc (core) Limit total attachment file size per document
Summary:
- Add a new parameter `Features.baseMaxAttachmentsBytesPerDocument` and set it to 1GB for the free team product.
- Add a method to DocStorage to calculate the total size of existing and used attachments.
- Add a migration to DocStorage adding an index to make the query in the above method fast.
- Check in ActiveDoc if uploading attachment(s) would exceed the product limit on that document.

Test Plan: Added test in `limits.ts` testing enforcement of the attachment limit.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3374
2022-04-14 16:33:09 +02:00
Jarosław Sadziński
dea1a8ba1b (core) Implementing search on raw data view
Summary:
Search now works on Raw Data Page.
- Search bar option 'Search on all pages' will change to 'Search on all tables' when on the Raw data page, and will allow searching through all tables.
- Little CSS adjustment for an overlay on Raw page (removes z-index as it is not needed, and conflicts with searchbar).
- Search bar option ('search on all') gets white background, little padding, and is moved 2 pixels up, this is needed for Raw page.

Test Plan: new and updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3376
2022-04-13 20:44:37 +02:00
Paul Fitzpatrick
20dd2fc70d (core) allow non-owners to remove themselves from sites/workspaces/docs
Summary:
For users who cannot otherwise change access to a resource, let
them remove themselves. Implemented via the standard endpoints
as a special exception that will process a request from a user
that would otherwise be denied, if the only contents of that
request are a removal of themselves.

Users who can change access are still not permitted to change their
own permissions or to remove themselves, as a precaution against
orphaning resources.

Test Plan: extended and updated tests

Reviewers: cyprien

Reviewed By: cyprien

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3367
2022-04-13 10:04:32 -04:00
Alex Hall
09da815c0c (core) Add /attachments/removeUnused DocApi endpoint to hard delete all unused attachments in document
Summary: Adds methods to delete metadata rows based on timeDeleted. The flag expiredOnly determines if it only deletes attachments that were soft-deleted 7 days ago, or just all soft-deleted rows. Then any actual file data that doesn't have matching metadata is deleted.

Test Plan: DocApi test

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3364
2022-04-12 17:11:11 +02:00
George Gevoian
859c593448 (core) Add authSubject and authProvider to sessions
Summary:
This also updates Authorizer to link the authSubject
to Grist users if not previously linked. Linked subjects
are now used as the username for password-based logins,
instead of emails, which remain as a fallback.

Test Plan: Existing tests, and tested login flows manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3356
2022-04-11 11:42:02 -07:00
Cyprien P
bf8769bc42 (core) Fix linking after a summary update
Summary:
When linking table to a summary, the linking ended up broken after updating the summary group by columns.
This diff fixes that issue.

There were two issues:
 1) some subscriptions were missing due to some .peek() calls instead of directly calling the observable.
 2) the LinkingState instance was not being disposed.
 3) the filterColValues was not updating after source data table has been loaded

Test Plan: Include new test file.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3358
2022-04-08 12:28:42 +02:00
Jarosław Sadziński
34708cd348 (core) Adding font options to the style picker
Summary:
Redesigning color picker:
- Single color palette (no light/dark switch)
- Ability to remove color (new empty button)

New font options in the color picker.
Font options are available on:
- Default cell style
- Conditional rules styles
- Choice/ChoiceList editor and token field
- Filters for Choice/ChoiceList columns

Design document:
https://www.figma.com/file/bRTsb47VIOVBfJPj0qF3C9/Grist-Updates?node-id=415%3A8135

Test Plan: new and updated tests

Reviewers: georgegevoian, alexmojaki

Reviewed By: georgegevoian, alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3335
2022-04-07 20:35:03 +02:00
Cyprien P
98ac2f7e5b (core) add 'Aggregate Values' option to chart config
Summary:
  - adds the `aggregate values` option to chart config
    - aggregation is performed by turning table into summary table
    - change columns options of xaxis and split series selectors to be the source table columns when `aggregate values` is on
    - change xAxis and split series computed to hold colId instead of column id
    - change GristDoc saveViewSection routine to preserve old sections viewFields and options
  - Rename `Group data` into `split series`

quip doc: https://grist.quip.com/tAsCAuv8RiMa/Charts-data-aggregation#temp:C:QcK0ce13e1e8ae64048988f44f9c

Test Plan: Adds ChartAggregate.ts nbrowser test.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3336
2022-04-07 15:34:54 +02:00
Alex Hall
64369df4c3 (core) Add /attachments/updateUsed DocApi endpoint to soft delete all unused attachments in document
Summary:
Builds on https://phab.getgrist.com/D3352

Add DocStorage.scanAttachmentsForUsageChanges to do fancy JSON query to find all attachment metadata rows whose soft deletion status needs updating.

Add ActiveDoc.updateUsedAttachments which uses the above and then applies the appropriate user action if needed to soft delete/undelete metadata rows.

Add endpoint in DocApi calling ActiveDoc method.

Test Plan: Added DocApi test

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3357
2022-04-07 15:08:22 +02:00
Alex Hall
251d79704b (core) Migrate Attachments columns from marshalled blobs to JSON
Summary: Adds a migration in preparation for future work on tracking and deleting attachments. This includes a `_grist_Attachments.timeDeleted` column which isn't used yet, and changing the storage format of user columns of type `Attachments`. DocStorage now treats Attachments like RefList in general (since they use JSON), which also prompted a tiny bit of refactoring.

Test Plan: Added a migration test case showing the change in format.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3352
2022-04-06 13:28:47 +02:00
Paul Fitzpatrick
c6d66e15bf (core) configure typedoc for generating plugin api documentation
Summary:
This annotates the plugin api sufficiently to generate some documentation
for it. See https://github.com/gristlabs/grist-help/pull/139

Contains some small code tweaks for things that caused typedoc some
trouble.

Test Plan: manual inspection of output

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3342
2022-04-04 18:11:33 -04:00
George Gevoian
6305811ca6 (core) Add new Grist login page
Summary:
Adds a new Grist login page to the login app, and replaces the
server-side Cognito Google Sign-In flow with Google's own OAuth flow.

Test Plan: Browser and server tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3332
2022-04-01 15:24:19 -07:00
Paul Fitzpatrick
24522e61ff
remove stray redis dependency, and upgrade node in tests (#173)
* remove stray redis dependency in test
* tweak handling of database connection between tests
* upgrade node versions in tests, type guessing in node 10 has problems
2022-03-28 15:43:47 -04:00
Paul Fitzpatrick
de703343d0 (core) disentangle some server tests, release to core, add GRIST_PROXY_AUTH_HEADER test
Summary:
This shuffles some server tests to make them available in grist-core,
and adds a test for the `GRIST_PROXY_AUTH_HEADER` feature added in
https://github.com/gristlabs/grist-core/pull/165

It includes a fix for a header normalization issue for websocket connections.

Test Plan: added test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3326
2022-03-24 15:11:32 -04:00
Jarosław Sadziński
b1c3943bf4 (core) Conditional formatting rules
Summary:
Adding conditional formatting rules feature.

Each column can have multiple styling rules which are applied in order
when evaluated to a truthy value.

- The creator panel has a new section: Cell Style
- New user action AddEmptyRule for adding an empty rule
- New columns in _grist_Table_columns and fields

A new color picker will be introduced in a follow-up diff (as it is also
used in choice/choice list/filters).

Design document:
https://grist.quip.com/FVzfAgoO5xOF/Conditional-Formatting-Implementation-Design

Test Plan: new tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3282
2022-03-23 13:15:02 +01:00
Jarosław Sadziński
96a34122a5 (core) Restoring cursor position on raw data views
Summary:
This diff introduces cursor features for raw data views:
- Restoring cursor position when the browser window is reloaded
- Restoring the last edit position when the browser window is reloaded

Test Plan: Added tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3314
2022-03-23 12:24:18 +01:00
Jarosław Sadziński
d2b82b84c7 (core) Fixing bug with resuming search on a hidden column.
Summary: Fix for error that happens when a search is resumed after one of the columns was hidden.

Test Plan: Added test that shows the error.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3309
2022-03-09 10:34:17 +01:00
George Gevoian
7ead97b913 (core) Fill name on test login page when possible
Summary:
Updates simulateLogin to fill in the name field of
the test login page. Docker tests were failing because
users created via the test login page were falling back
to their email for their name.

Test Plan: N/A (fixing Docker tests)

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3315
2022-03-08 20:29:39 -08:00
Paul Fitzpatrick
c4d3d7d3bb (core) be careful when reassigning a doc to a worker it was on before
Summary:
Importing a .grist document is implemented in a somewhat clunky way, in a multi-worker setup.

 * First a random worker receives the upload, and updates Grist's various stores appropriately (database, redis, s3).
 * Then a random worker is assigned to serve the document.

If the worker serving the document fails, there is a chance the it will end up assigned to the worker that handled its upload. Currently the worker will misbehave in this case. This diff:

 * Ports a multi-worker test from test/home to run in test/s3, and adds a test simulating a bad scenario seen in the wild.
 * Fixes persistence of any existing document checksum in redis when a worker is assigned.
 * Adds a check when assigned a document to serve, and finding that document already cached locally. It isn't safe to rely only on the document checksum in redis, since that may have expired.
 * Explicitly claims the document on the uploading worker, so this situation becomes even less likely to arise.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3305
2022-03-08 17:20:01 -05:00
George Gevoian
cc1af85594 (core) Add missing .isPresent() to isOnTestLoginPage
Summary:
A staging test was failing because the util was throwing instead
of returning true or false.

Test Plan: N/A (fixing test)

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3301
2022-03-02 10:10:26 -08:00
George Gevoian
10097c5423 (core) Add check for test login page in homeUtil.ts
Summary:
The old regex for login pages, /gristlogin/, used to handle
test login pages too, matching on a query param set in the
URL. A recent change to make the regex more specific broke this.

This fixes that regression by adding an additional check to
isOnLoginPage for the presence of the test login page.

Test Plan: N/A (fixing test)

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3300
2022-03-01 15:44:58 -08:00
George Gevoian
fa68b790bb (core) Remove code for unused welcome flows
Summary: Removes code that was marked for removal.

Test Plan: Existing tests still pass.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3289
2022-02-28 13:21:28 -08:00
George Gevoian
83ba2957bf (core) Update failing HomeIntro core test
Summary:
Core doesn't redirect to Cognito or our own sign-up page
when clicking 'sign up' on the welcome screen. Instead, it
redirects to the test login page.

Test Plan: N/A (fixing test)

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3298
2022-02-28 12:23:28 -08:00
Paul Fitzpatrick
c91593e312 (core) make cognito logins for tests work on slightly smaller screens
Summary:
A selfish diff to tweak deployment tests so they pass on a
slightly smaller laptop screen that for whatever reason trips
a css breakpoint on the cognito login page.

Test Plan: manual

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3292
2022-02-25 15:04:45 -05:00
George Gevoian
a294eefdff (core) Update HomeIntro.ts to check for either login page
Summary:
The "Sign In" button can either redirect you to the Cognito login
page or the Grist sign-up page, depending on whether you've logged
in before. Since the HomeIntro tests can either be run as part of a full
deployment test suite, or individually, the test could redirect you to
both types of login pages.

Test Plan: N/A (fixing test)

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3291
2022-02-24 15:28:16 -08:00
George Gevoian
8f1889d2d7 (core) Fix staging test that would timeout
Summary:
Removes code from simulateLogin that's no longer needed, and increases
the timeout on the util function that checks if we're on the Grist sign-up
page.

Test Plan: N/A (fixing test)

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3287
2022-02-23 19:03:41 -08:00
George Gevoian
ac910389bc (core) Make login URL regexes more specific
Summary:
This should resolve some staging test failures. The previous regexes
weren't quite specific enough, so random parts of a URL that had
traces of the real login URLs were causing isOnLoginPage() and
isOnGristLoginPage() to return true instead of false.

Test Plan: N/A (fixing tests)

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3283
2022-02-23 08:51:59 -08:00
Alex Hall
5b352211c4 (core) Guess date format during type conversion
Summary:
- Adds a dependency moment-guess (https://github.com/apoorv-mishra/moment-guess) to guess date formats from strings. However the npm package is missing source maps which leads to an ugly warning, so currently using a fork until https://github.com/apoorv-mishra/moment-guess/pull/22 is resolved.
- Adds guessDateFormat using moment-guess to determine the best candidate date format. The logic may be refined for e.g. lossless imports where the stakes are higher, but for now we're just trying to make type conversions smoother.
- Uses guessDateFormat to guess widget options when changing column type to date or datetime.
- Uses the date format of the original column when possible instead of guessing.
- Fixes a bug where choices were guessed based on the display column instead of the visible column, which made the guessed choices influenced by which values were referenced as well as completely broken when converting from reflist.
- @dsagal @georgegevoian This builds on https://phab.getgrist.com/D3265, currently unmerged. That diff was created first to alert to the change. Without it there would still be similar test failures/changes here as the date format would often be concretely guessed and saved as YYYY-MM-DD instead of being left as the default `undefined` which is shows as YYYY-MM-DD in the dropdown.

Test Plan: Added a unit test to `parseDate.ts`. Updated several browser tests which show the guessing in action during type conversion quite nicely.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal, georgegevoian

Differential Revision: https://phab.getgrist.com/D3264
2022-02-21 22:39:47 +02:00
George Gevoian
481122b4d3 (core) Fix failing login tests
Summary:
The rollout of the new sign-up page on prod caused a few tests
that previously expected to be on the Cognito sign-up page to fail.

Test Plan: N/A (fixing failing tests).

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3279
2022-02-21 11:50:30 -08:00
Edward Betts
d6e0e1fee3 Correct spelling mistakes 2022-02-19 09:46:49 +00:00
George Gevoian
99f3422217 (core) Add new Grist sign-up page
Summary:
Available at login.getgrist.com/signup, the new sign-up page
includes similar options available on the hosted Cognito sign-up
page, such as support for registering with Google. All previous
redirects to Cognito for sign-up should now redirect to the new
Grist sign-up page.

Login is still handled with the hosted Cognito login page, and there
is a link to go there from the new sign-up page.

Test Plan: Browser, project and server tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3249
2022-02-14 10:32:47 -08:00
Cyprien P
196ab6c473 (core) Adds cell context menu
Summary:
 - Brings in a new utility `contextMenu` to open context menu next to the mouse position
 - Use this utility to show a CellContextMenu, that sort of merge cell context menu and column context menu together.
 - Show cell context menu on context click on any grid's cell.
 - Also takes care of showing the row context menu for detail view on a context click that occurs on cells and not only on the row num header as it was the case prior to this diff.
 - task: https://gristlabs.getgrist.com/doc/check-ins/p/5#a1.s9.r1529.c31
 - discussion: https://grist.quip.com/ETGkAroLnc0Y/Cell-Context-Menu

{F40092}

Test Plan: - Adds project test and nbrowser for cell context menu and new cases for the detail row context menu.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3237
2022-02-07 15:46:17 +01:00
Alex Hall
5d671bf0b3 (core) New type conversion in the backend
Summary: This is https://phab.getgrist.com/D3205 plus some changes (https://github.com/dsagal/grist/compare/type-convert...type-convert-server?expand=1) that move the conversion process to the backend. A new user action ConvertFromColumn uses `call_external` so that the data engine can delegate back to ActiveDoc. Code for creating formatters and parsers is significantly refactored so that most of the logic is in `common` and can be used in different ways.

Test Plan: The original diff adds plenty of tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3240
2022-02-04 20:28:13 +02:00
Alex Hall
fa9e6eee88 (core) Create an extra raw data widget when creating a table
Summary: This is the first step towards raw data views, merely adding metadata without any UI. Every 'normal' table now has a widget referenced by `rawViewSectionRef`. It has no parent view/page and cannot actually be viewed for now. The widget is created during the AddTable user action, and the migration creates a widget for existing tables.

Test Plan: Many tests had to be updated, especially tests that listed all view sections and/or fields.

Reviewers: jarek, dsagal

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3232
2022-02-01 21:19:30 +02:00
Jarosław Sadziński
a685707d50 (core) Renaming filters for choice columns
Summary:
Updating filters when user renames labels in a choice/choice list column.
When there are unsaved filters they are reverted to orginal values (only
for the affected column).

Test Plan: new tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3230
2022-01-25 14:22:56 +01:00
Jarosław Sadziński
d2077bc486 (core) Improving experience when editing group-by column.
Summary:
Improving experience when editing group-by column:
- Disable column rename
- Allow changing most widget properties:
 - Color/Background
 - Number format
 - Date/DateTime format (but not the timezone)
 - All toggle options (for toggle column)
- Remove Edit button on Choice Edit
- Changing the underlying column should reset all those options back to the original column.

Test Plan: nbrowser

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3216
2022-01-18 14:31:31 +01:00
Dmitry S
9c57b565b2 (core) Fix deployment tests.
Summary:
- Skip welcome questions in HomeIntro
- Make Funnel test less flaky, and increase its timeout

Test Plan: Both tests pass staging_deployment when run locally

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3218
2022-01-14 20:23:23 -05:00
George Gevoian
ba6ecc5e9e (core) Move user profile to new page and begin MFA work
Summary:
The user profile dialog is now a separate page, in preparation
for upcoming work to enable MFA. This commit also contains
some MFA changes, but the UI is currently disabled and the
implementation is limited to software tokens (TOTP) only.

Test Plan:
Updated browser tests for new profile page. Tests for MFAConfig
and CognitoClient will be added in a later diff, once the UI is enabled.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3199
2022-01-13 21:21:49 -08:00
Jarosław Sadziński
85ef873ce5 (core) Widget options api
Summary:
Adding configuration options for CustomWidgets.

Custom widgets can now store options (in JSON) in viewSection metadata.

Changes in grist-plugin-api:
- Adding onOptions handler, that will be invoked when the widget is ready and when the configuration is changed
- Adding WidgetAPI - new API to read and save a configuration for widget.

Changes in Grist:
- Rewriting CustomView code, and extracting code that is responsible for showing the iframe and registering Rpc.
- Adding Open Configuration button to Widget section in the Creator panel and in the section menu.
- Custom Widgets can implement "configure" method, to show configuration screen when requested.

Test Plan: Browser tests.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3185
2022-01-13 11:10:17 +01:00
Jarosław Sadziński
08881d9663 (core) Scrolling to the active record when editor is activated
Summary: When an editor is activated by typing, the active view should be scrolled to the active record.

Test Plan: new tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3196
2022-01-10 23:35:08 +01:00
Dmitry S
f024aaaf5d (core) Fix some bugs with ChoiceList in summary tables, and evaluation of lookups.
Summary:
Addresses several issues:
- Error 'Cannot modify summary group-by column' when changing Text ->
  ChoiceList in the presence of summary tables.
- Error 'ModifyColumn in unexpected position' when changing ChoiceList -> Text
  in the presence of summary tables.
- Double-evaluation of trigger formulas in some cases.

Fixes include:
- Fixed verification that summary group-by columns match the underlying ones,
  and added comments to explain.
- Avoid updating non-metadata lookups after each doc-action (early lookups
  generated extra actions to populate summary tables, causing the 'ModifyColumn
  in unexpected position' bug)
- When updating formulas, do update lookups first.
- Made a client-side tweak to avoid a JS error in case of some undos.

Solution to reduce lookups is based on https://phab.getgrist.com/D3069?vs=on&id=12445,
and tests for double-evaluation of trigger formulas are taken from there.

Add a new test case to protect against bugs caused by incorrect order of
evaluating #lookup columns.

Enhanced ChoiceList browser test to check a conversion scenario in the presence
of summary tables, previously triggering bugs.

Test Plan: Various tests added or enhanced.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3184
2021-12-15 09:51:18 -05:00
Paul Fitzpatrick
65ac8aaa85 (core) run test:docker target prior to pushing grist-core image
Summary:
This activates the new tests that can run on a grist-core image,
and will not push a new version to `latest` tag unless those tests
pass.

This also runs a smoke test in the regular CI tests, to confirm the
code continues to work without GRIST_TEST_LOGIN enabled.

Test Plan: tested on a fork of grist-core

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3186
2021-12-14 14:22:02 -05:00
Paul Fitzpatrick
d99db8d016 (core) move more tests to grist-core
Summary:
 * Tie build and run-time docker base images to a consistent version (buster)
 * Extend the test login system activated by GRIST_TEST_LOGIN to ease porting tests that currently rely on cognito (many)
 * Make org resets work in absence of billing endpoints
 * When in-memory session caches are used, add missing invalidation steps
 * Pass org information through sign-ups/sign-ins more carefully
 * For CORS, explicitly trust GRIST_HOST origin when set
 * Move some fixtures and tests to core, focussing on tests that cover existing failures or are in the set of tests run on deployments
 * Retain regular `test` target to run the test suite directly, without docker
 * Add a `test:smoke` target to run a single simple test without `GRIST_TEST_LOGIN` activated
 * Add a `test:docker` target to run the tests against a grist-core docker image - since tests rely on certain fixture teams/docs, added `TEST_SUPPORT_API_KEY` and `TEST_ADD_SAMPLES` flags to ease porting

The tests ported were `nbrowser` tests: `ActionLog.ts` (the first test I tend to port to anything, out of habit), `Fork.ts` (exercises a lot of doc creation paths), `HomeIntro.ts` (a lot of DocMenu exercise), and `DuplicateDocument.ts` (covers a feature known to be failing prior to this diff, the CORS tweak resolves it).

Test Plan: Manually tested via `buildtools/build_core.sh`. In follow up, I want to add running the `test:docker` target in grist-core's workflows. In jenkins, only the smoke test is run. There'd be an argument for running all tests, but they include particularly slow tests, and are duplicates of tests already run (in different configuration admittedly), so I'd like to try first just using them in grist-core to gate updates to any packaged version of Grist (the docker image currently).

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3176
2021-12-10 18:33:07 -05:00
Dmitry S
7a6d726daa (core) Change datepicker in DateEditor to use moment format, show AltText in DateEditor
Summary:
- Rather than translate from moment format to that of bootstrap-datepicker, use
  the customization methods to format datepicker dates using moment directly.
- Fix issue with parseDate() when format includes tokens like Mo or Do
- Fix issue in parseDateTime() that could produce an off-by-one error in date
  depending on local timezone.
- When opening DateEditor, show AltText value if present.

- Add crossorigin=anonymous to scripts that were missing it (including
  bootstrap-datepicker), to ensure that errors from them are reported properly
  rather than as 'Script error.'

Test Plan:
Added test cases to parseDate() test for low-level fixes; added a
browser test for the fixed DateEditor behavior.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3169
2021-12-07 11:33:49 -05:00
Alex Hall
064455b2f7 (core) Parsing pasted datetimes
Summary:
Add function parseDateTime which parses a string containing both date and time componenents, intended for parsing pasted strings.

Add DateTimeParser subclass of ValueParser.

Test Plan: Extended parseDate.ts and CopyPaste.ts tests.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3152
2021-11-25 00:49:01 +02:00
Jarosław Sadziński
fc50079e03 (core) Storing last position for doc and user
Summary: Last position should be stored for document and user.

Test Plan: Updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3143
2021-11-19 12:01:49 +01:00
Paul Fitzpatrick
822372ed7c (core) copy settings related to gvisor from staging to prod
Summary:
This enables ptrace and modifies a nofile limit, to be consistent with staging. The nofile change isn't really needed, it was for gvisor's vfs2 which we are no longer using, it is just for consistency.

This diff also documents the PYTHON_VERSION_ON_CREATION=3 flag active in staging and ~~soon to be active in~~ production.

Test Plan: settings work in staging

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3125
2021-11-08 16:40:03 -05:00
George Gevoian
59699bf446 (core) Add additional org items to AppHeader
Summary:
Adds links to manage team and go to billing account in
the org menu (opened by clicking the dropdown in the
top-left corner of Grist). Tweaks some wording of items
in both AppHeader and AccountWidget, and adds a link
to create a new team site to the Site Switcher in both
menus.

Also tweaks the UI of UserManager by adding
an animation when the manager is opened from the
doc access dialog.

Test Plan: Browser tests.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3121
2021-11-05 08:19:51 -07:00
Jarosław Sadziński
3c72639e25 (core) Adding sort options for columns.
Summary:
Adding sort options for columns.
- Sort menu has a new option "More sort options" that opens up Sort left menu
- Each sort entry has an additional menu with 3 options
-- Order by choice index (for the Choice column, orders by choice position)
-- Empty last (puts empty values last in ascending order, first in descending order)
-- Natural sort (for Text column, compares strings with numbers as numbers)
Updated also CSV/Excel export and api sorting.
Most of the changes in this diff is a sort expression refactoring. Pulling out all the methods
that works on sortExpression array into a single namespace.

Test Plan: Browser tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: dsagal, alexmojaki

Differential Revision: https://phab.getgrist.com/D3077
2021-11-03 15:31:39 +01:00
Alex Hall
d63da496a8 (core) Value parsing for refs, parsing data entry for numbers
Summary:
Handle reference columns in ViewFieldRec.valueParser.

Extracted code for reuse from ReferenceEditor to look up values in the visible column. While I was at it, also extracted a bit of common code from ReferenceEditor and ReferenceListEditor into a new class ReferenceUtils. More refactoring could be done in this area but it's out of scope.

Changed NTextEditor to use field.valueParser, which affects numeric and reference fields. In particular this means numbers are parsed on data entry, it doesn't change anything for references.

Test Plan:
Added more CopyPaste testing to test references.

Tested entering slightly formatted numbers in NumberFormatting.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3094
2021-11-01 19:31:52 +02:00
Alex Hall
e58df5df5b (core) ValueParser for Date columns
Summary: Adds parseDateStrict function based on parseDate, uses it in DateParser subclass of ValueParser.

Test Plan:
Tweaked parseDate test to check parseDateStrict.

Extended test in CopyPaste to test parsing dates as well as numbers.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3088
2021-10-26 13:03:19 +02:00
George Gevoian
62db263d1f (core) Add diff preview to Importer
Summary:
Updates the preview table in Importer to show a diff of changes
when importing into an existing table and updating existing records.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3060
2021-10-08 14:15:07 -07:00
Dmitry S
1517dca644 (core) Implement DiscourseConnect to enable easy sign-in to community forum
Summary:
- Update cookie module, to support modern sameSite settings
- Add a new cookie, grist_sid_status with less-sensitive value, to let less-trusted subdomains know if user is signed in
- The new cookie is kept in-sync with the session cookie.
- For a user signed in once, allow auto-signin is appropriate.
- For a user signed in with multiple accounts, show a page to select which account to use.
- Move css stylings for rendering users to a separate module.

Test Plan: Added a test case with a simulated Discourse page to test redirects and account-selection page.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3047
2021-10-01 11:24:22 -04:00
Jarosław Sadziński
42910cb8f7 (core) Extending Google Drive integration scope
Summary:
New environmental variable GOOGLE_DRIVE_SCOPE that modifies the scope
requested for Google Drive integration.
For prod it has value https://www.googleapis.com/auth/drive.file which leaves
current behavior (Grist is allowed only to access public files and for private
files - it fallbacks to Picker).
For staging it has value https://www.googleapis.com/auth/drive.readonly which
allows Grist to access all private files, and fallbacks to Picker only when the file is
neither public nor private).
Default value is https://www.googleapis.com/auth/drive.file

Test Plan: manual and existing tests

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3038
2021-10-01 10:47:12 +02:00
Paul Fitzpatrick
c4fb40e0bd (core) add gvisor-based sandbox to jenkins test worker image
Summary:
This is preparatory work for running tests with the new sandbox in jenkins.

 * Makes a base image that is now shared by grist servers and jenkins workers.
 * Needed to allow jenkins to run `sudo runsc`.
 * Converged on port 2020 for ssh to workers and servers.

Test Plan: added one runsc-based test and confirmed it was run

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3029
2021-09-20 16:34:40 -04:00
George Gevoian
a825115c04 (core) Limit number of errors shown on narrow screens
Summary: We still show up to 5 on regular-width screens.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3008
2021-08-31 12:15:31 -07:00
Paul Fitzpatrick
08d0ee65d1 (core) tweak how search bar places cursor on sections
Summary: It is important for linking to be maintained, or user could be gravely misled about material in other sections.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3003
2021-08-30 09:26:28 -04:00
George Gevoian
a6e08883e0 (core) Simple localization support and currency selector.
Summary:
- Grist document has a associated "locale" setting that affects how currency is formatted.
- Currency selector for number format.

Test Plan: not done

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D2977
2021-08-26 13:36:49 -07:00
Dmitry S
faa0d9988e (core) When changing a table for a page widget, unset widget-linking to avoid invalid values.
Summary:
Previously, using "Change Widget" allowed one to change the underlying table,
but would keep the linking settings. This could allow invalid settings which
would sometimes lead to JS errors. These manifested in production as
"UserError: Query error: n is not a function".

- Unset linking settings in this case, to avoid invalid values.
- In case invalid values are encountered (e.g. saved previously), treat them as
  unset, to avoid JS errors.
- If an error does occur, report it with a stack trace.

Also, for testing, added 'selectBy' option to gristUtils helpers for using page-widget-picker.

Test Plan: Added test cases for resetting linking, and for ignoring invalid link settings.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D2993
2021-08-25 12:53:46 -04:00
Paul Fitzpatrick
f9630b3aa4 (core) clean up a collection of small problems affecting grist-core
Summary:
 * Remove adjustSession hack, interfering with loading docs under saml.
 * Allow the anonymous user to receive an empty list of workspaces for
   the merged org.
 * Behave better on first page load when org is in path - this used to
   fail because of lack of cookie.  This is very visible in grist-core,
   as a failure to load localhost:8484 on first visit.
 * Mark cookie explicitly as SameSite=Lax to remove a warning in firefox.
 * Make errorPages available in grist-core.

This changes the default behavior of grist-core to now start off in
anonymous mode, with an explicit sign-in step available.  If SAML is not configured,
the sign-in operation will unconditionally sign the user in as a default
user, without any password check or other security.  The user email is
taken from GRIST_DEFAULT_EMAIL if set.  This is a significant change, but
makes anonymous mode available in grist-core (which is convenient
for testing) and makes behavior with and without SAML much more consistent.

Test Plan: updated test; manual (time to start adding grist-core tests though!)

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2980
2021-08-17 21:44:50 -04:00
Jarosław Sadziński
bb55422d9c (core) Fixing UserProfile test that fails randomly
Summary: UserProfile test fails randomly which is caused by the AccountWidget that loads its data asonchrynously.

Test Plan: Existing tests

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D2964
2021-08-05 19:29:19 +02:00
Dmitry S
580cd0e22e (core) Add a waitForServer to fix HomeIntro test that fails for deployments
Test Plan: Tested by running HomeIntro in staging_deployment suite.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D2956
2021-08-01 23:56:20 -04:00
Dmitry S
1605e18f66 (core) Enable auto triggering of Welcome Tour, and various improvements.
Summary:
- Add showGristTour preference, and trigger tour automatically.
- Tour is only triggered for new and anonymous users on a personal org, with
  edit permission.

- Automatically open the right panel at tour start.
- Don't show tours on mobile, since that's not ready (popups are cut off
  and can't be dismissed)
- Cancel previous tour if a new one is somehow started.
- Remove #repeat- trigger hash tags from the URL when the tour starts.
- Ensure Help Center popup is positioned even when left panel is collapsed.
- Polish up the content of the last two cards in the tour.

Test Plan: Added test case for triggering and opening right panel.

Reviewers: alexmojaki, paulfitz

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D2955
2021-07-30 14:17:54 -04:00
George Gevoian
37eed2d3c2 (core) Fix staging test and tweak template line height
Summary:
Staging tests were using an organization named Test Grist,
while local was using one named test-grist. Both are now
named Test Grist, which should help keep things more consistent
between local and deployment test runs.

The inherited line height of template docs in icon view was
different on Windows, cutting off part of the last line of the
description. The description line height should now be fixed
to a reasonable value.

Test Plan: Manually tested CSS fix on a Windows machine.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D2948
2021-07-29 08:16:19 -07:00
George Gevoian
24fc3a2d00 (core) Redesign examples and templates UI
Summary:
The old Examples and Templates workspace is now
a page that pulls templates from a new public Grist Templates org.
The All Documents view will pull featured templates from that org, where
featured templates are simply pinned documents in Grist Templates. The
Examples and Templates page will also show the featured templates, as
well as the rest of the available templates organized by category. The
categories are equivalent to workspaces in Grist Templates, and are
generated dynamically.

Test Plan: Browser tests.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz, dsagal

Subscribers: dsagal, paulfitz, jarek

Differential Revision: https://phab.getgrist.com/D2930
2021-07-28 12:29:03 -07:00
Paul Fitzpatrick
6e15d44cf6 (core) start applying defenses for untrusted document uploads
Summary:
This applies some mitigations suggested by SQLite authors when
opening untrusted SQLite databases, as we do when Grist docs
are uploaded by the user.  See:
  https://www.sqlite.org/security.html#untrusted_sqlite_database_files

Steps implemented in this diff are:
  * Setting `trusted_schema` to off
  * Running a SQLite-level integrity check on uploads

Other steps will require updates to our node-sqlite3 fork, since they
are not available via the node-sqlite3 api (one more reason to migrate
to better-sqlite3).

I haven't yet managed to create a file that triggers an integrity
check failure without also being detected as corruption by sqlite
at a more basic level, so that is a TODO for testing.

Test Plan:
existing tests pass; need to come up with exploits to
actually test the defences and have not yet

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2909
2021-07-14 18:34:27 -04:00
Jarosław Sadziński
625fce5f65 (core) Refactoring google drive plugin
Summary:
Finishing implementation for google drive plugin.
- Refactoring plugin code to make it more robust and to follow grist ux
- Changing the way server hosts untrusted user content, from different domain to different port

Test Plan: Browser tests

Reviewers: dsagal, paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2881
2021-07-14 09:52:04 +02:00
Jarosław Sadziński
bdd4d3c46e (core) Freezing columns on a GridView
Summary:
User can freeze any number of columns, which will not move when a user scrolls grid horizontally.
Main use cases:
- Frozen columns don't move when a user scrolls horizontally
- The number of frozen columns is automatically persisted
- Readonly viewers see frozen columns and can modify them - but the change is not persisted
- On a small screen - frozen columns still moves to the left when scrolled, to reveal at least one column
- There is a single menu option - Toggle freeze - which offers the best action considering selected columns
- When a user clicks a single column - action to freeze/unfreeze is always there
- When a user clicks multiple columns - action is offered only where it makes sens (columns are near the frozen border)

Test Plan: Browser tests

Reviewers: dsagal, paulfitz

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2852
2021-06-18 12:22:13 +02:00
Jarosław Sadziński
698c9d4e40 (core) Readonly editors
Summary:
Grist should not prevent read-only viewers from opening cell editors since they usually provide much more information than is visible in a cell.

Every editor was enhanced with a read-only mode that provides the same information available for an editor but doesn't allow to change the underlying data.

Test Plan: Browser tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2842
2021-06-17 19:12:16 +02:00
Alex Hall
8a940676e9 (core) Generic tools for recording pycalls, deterministic mode.
Summary:
Replaces https://phab.getgrist.com/D2854

Refactoring of NSandbox:
- Simplify arguments to NSandbox.spawn. Only half the arguments were used depending on the flavour, adding a layer of confusion.
- Ensure the same environment variables are passed to both flavours of sandbox
- Simplify passing down environment variables.

Implement deterministic mode with libfaketime and a seeded random instance.
- Include static prebuilt libfaketime.so.1, may need another solution in future for other platforms.

Recording pycalls:
- Add script recordDocumentPyCalls.js to open a single document outside of tests.
- Refactor out recordPyCalls.ts to support various uses.
- Add afterEach hook to save all pycalls from server tests under $PYCALLS_DIR
- Make docTools usable without mocha.
- Add useLocalDoc and loadLocalDoc for loading non-fixture documents

Test Plan:
Made a document with formulas NOW() and UUID()
Compare two document openings in normal mode:

    diff <(test/recordDocumentPyCalls.js samples/d4W6NrzCMNVSVD6nWgNrGC.grist /dev/stdout) \
         <(test/recordDocumentPyCalls.js samples/d4W6NrzCMNVSVD6nWgNrGC.grist /dev/stdout)

Output:

    <                 1623407499.58132,
    ---
    >                 1623407499.60376,
    1195c1195
    <               "B": "bd2487f6-63c9-4f02-bbbc-5c0d674a2dc6"
    ---
    >               "B": "22e1a4fd-297f-4b86-91a2-bc42cc6da4b2"

`export DETERMINISTIC_MODE=1` and repeat. diff is empty!

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2857
2021-06-15 20:58:05 +02:00
Jarosław Sadziński
30bbb6fa94 (core) Summaries test doesn't work in a browser mode
Summary: Summaries test works only in a headless mode. In browser mode it failes as section is to narrow.

Test Plan: n/a

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2859
2021-06-14 16:10:00 +02:00
Dmitry S
d1c1416d78 (core) Add rules to eslint to better match our coding conventions.
Summary:
We used tslint earlier, and on switching to eslint, some rules were not
transfered. This moves more rules over, for consistent conventions or helpful
warnings.

- Name private members with a leading underscore.
- Prefer interface over a type alias.
- Use consistent spacing around ':' in type annotations.
- Use consistent spacing around braces of code blocks.
- Use semicolons consistently at the ends of statements.
- Use braces around even one-liner blocks, like conditionals and loops.
- Warn about shadowed variables.

Test Plan: Fixed all new warnings. Should be no behavior changes in code.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2831
2021-05-24 12:56:18 -04:00
Jarosław Sadziński
5f182841b9 (core) Document keeps track of latest cursor position and latest editor value and is able to restore them when it is reloaded.
Summary: Grist document, when reloaded, is able to restore the latest cursor position and the editor state.

Test Plan: Browser test were created.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D2808
2021-05-18 10:36:59 +02:00
Jarosław Sadziński
2c93eafa44 (core) Null or undefined value on the Integer / Numeric cell
Summary: Text editor for Integer and Numeric column was showing null or undefined when the underlying value was null.

Test Plan: Browser test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2817
2021-05-14 21:41:37 +02:00
Paul Fitzpatrick
e55fba24e7 (core) fix up newRec when column names change; autocomplete after newRec
Summary: This treats newRec in the same way as rec in access formulas.

Test Plan: updated test for column renames; autocomplete checked manually.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2810
2021-05-12 08:29:39 -04:00
Dmitry S
2f26f140c8 (core) Add our very own implementation of tokenfield (aka pillbox, tag list)
Summary:
- Supports auto-completion
- Supports various keyboard interactions to delete/select/copy-paste tokens
- Supports various mouse interactions to select and move tokens.
- Supports undo/redo while editing tokens.

Test Plan: stand-alone fixture and browser test verify the supported behaviors

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2789
2021-05-03 17:16:24 -04:00
Cyprien P
5baae7437a (core) split sort and filter menu into its own button
Summary:
  - New sort and filter button has several states
     - Empty / unsaved / saved
     - offers small save/revert button when unsaved

  - Fix little issue with hanging tooltip when the refElem is disposed.
    - The problem was that if you hover the save (or revert) button
      and then click the button, it causes the button to disappear,
      but the tooltip was staying.

Test Plan: Updated all tests to match the new UI.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal, paulfitz

Differential Revision: https://phab.getgrist.com/D2795
2021-05-03 09:13:28 +02:00
Jarosław Sadziński
8f008d8de2 (core) Select all functionality by clicking the upper left margin corner
Summary: Select all functionality by clicking the upper left margin corner

Test Plan: Browser test created

Reviewers: dsagal, paulfitz

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2793
2021-04-29 20:56:25 +02:00
Cyprien P
2823727da1 (core) add + button to the filter section of the section menu
Test Plan: adds new browser tests

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D2781
2021-04-28 21:51:03 +02:00