Commit Graph

962 Commits

Author SHA1 Message Date
George Gevoian
2cb783ea7b (core) Fix bugs with intervals
Summary:
Fixes some bugs involving intervals, and updates RandomizedTimer to support both fixed and
randomized delays, and to better handle async callbacks.

 * Fixed a bug where Throttle would queue up many pidusage calls due to the use of
    setInterval, and the async nature of the calls.

 * Fixed a but where RandomizedTimer (now just Interval) would not be disabled in
    ActiveDoc on doc shutdown if initialization had not yet settled.

Test Plan: Tested manually.

Reviewers: jarek, dsagal

Reviewed By: jarek, dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3604
2022-08-25 12:38:36 -07:00
Dmitry S
af77824618 (core) Add caching for measuring data size in DocStorage, when data isn't changing
Summary:
getDataSize() call can be expensive and involve lots of disk reading. We can
avoid doing it repeatedly when the document isn't actually changing.

Test Plan: Should have no change in behavior except for timings.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3605
2022-08-25 09:50:23 -04:00
Jarosław Sadziński
2997434815 (core) Showing a raw data section on a popup
Summary:
Show raw data will now open a popup with
raw section instead of redirecting to raw data page.

Adding new anchor link type "a2" that is able to open
any section in a popup on a current view.

Not related:
Fixing highlightMatches function, after merging core PR.

Test Plan: Updated tests

Reviewers: alexmojaki, georgegevoian

Reviewed By: alexmojaki, georgegevoian

Subscribers: georgegevoian, alexmojaki

Differential Revision: https://phab.getgrist.com/D3592
2022-08-25 09:30:07 +02:00
George Gevoian
1c722014fa
Merge pull request #254 from vviers/main
fix(ui): pointer cursor when creating a new widget
2022-08-24 14:40:48 -07:00
Yohan Boniface
50a57c673f
Add GRIST_DEFAULT_LOCALE env var (#257) 2022-08-24 15:24:50 -04:00
Paul Fitzpatrick
177b9d83d9 (core) add a log message on proxying failures
Summary:
When a home server fails to fetch from a doc worker, errors are
passed along to clients but we don't get to see them. This fixes
that omission.

Test Plan:
tested manually, by inserting some code to delay
serving particular test documents.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3601
2022-08-24 09:16:19 -04:00
Vincent Viers
2f48be26b1 fix(ui): add default cursor over disabled items 2022-08-24 09:47:47 +02:00
George Gevoian
56e8e1f4b3 (core) Randomize ActiveDoc interval delays
Summary:
When large numbers of documents were restarted simultaneously, they had
a tendency to schedule intervals to begin at roughly the same moment in
time, causing periodic spikes in load. This randomizes the delay of each
interval to help avoid such spikes.

Test Plan: Tested manually.

Reviewers: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3600
2022-08-23 23:08:07 -07:00
George Gevoian
ed37401b2c (core) Add basic activation page to grist-ee
Summary:
Adds an activation page to grist-ee that currently shows activation status.

Follow-up diffs will introduce additional functionality, such as the ability to
enter activation keys directly from the activation page.

Test Plan: No grist-ee tests (yet).

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal, paulfitz

Differential Revision: https://phab.getgrist.com/D3582
2022-08-23 10:30:52 -07:00
Jarosław Sadziński
a52d56f613 (core) Moving client and common tests to core
Summary:
- Moved /test/client and /test/common to core.
- Moved two files (CircularArray and RecentItems) from app/common to core/app/common.
- Moved resetOrg test to gen-server.
- `testrun.sh` is now invoking common and client test from core.
- Added missing packages to core's package.json (and revealed underscore as it is used in the main app).
- Removed Coord.js as it is not used anywhere.

Test Plan: Existing tests

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3590
2022-08-23 19:20:10 +02:00
Vincent Viers
5371ebcf46 fix(ui): pointer cursor when creating a new widget 2022-08-23 14:30:52 +02:00
Paul Fitzpatrick
028146f88a (core) updates from grist-core 2022-08-22 10:23:26 -04:00
Jarosław Sadziński
a43a5a25a6 (core) Removing old billing landing page.
Summary: Old landing page /docs/billing/signup is not used anymore.

Test Plan: Updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3585
2022-08-22 07:37:13 +02:00
Alex Hall
42060df29a (core) Formula autocomplete improvements for references and lookups
Summary:
Makes the following improvements to formula autocomplete:

- When a user types `$RefCol` (or part of it), also show `$RefCol.VisibleCol` (replace actual column names) in the autocomplete even before the `.` is typed, to help users understand the difference between a raw reference/record and its visible column.
- When a user types a table name, show `.lookupOne` and `.lookupRecords` in the autocomplete, again even before the `.` is typed.
- For `.lookupRecords(` and `.lookupOne(`, once the `(` is entered, suggest each column name as a keyword argument.
- Also suggest lookup arguments involving compatible reference columns, especially 'reverse reference' lookups like `refcol=$id` which are very common and difficult for users.
- To support these features, the Ace editor autocomplete needs some patching to fetch fresh autocomplete options after typing `.` or `(`. This also improves unrelated behaviour that wasn't great before when one column name is contained in another. See the first added browser test.

Discussions:

- https://grist.slack.com/archives/CDHABLZJT/p1659707068383179
- https://grist.quip.com/HoSmAlvFax0j#MbTADAH5kgG
- https://grist.quip.com/HoSmAlvFax0j/Formula-Improvements#temp:C:MbT3649fe964a184e8dada9bbebb

Test Plan: Added Python and nbrowser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3580
2022-08-20 19:11:41 +02:00
Jarosław Sadziński
758c87f002 (core) Adding hide for multiple columns
Summary: New column menu option to hide multiple columns.

Test Plan: new test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3581
2022-08-17 22:31:16 +02:00
Ronan Amicel
cd16650e0e Rename text cleaning function to "normalizeText" 2022-08-17 19:11:52 +02:00
Ronan Amicel
4003d2d7c6 Use lodash to remove accents and ther diacritics
https://lodash.com/docs/4.17.15#deburr
2022-08-17 19:11:52 +02:00
George Gevoian
360d838578 (core) Add Open Graph image tags
Summary: Adds a few missing Open Graph tags to Grist pages.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3586
2022-08-16 10:54:32 -07:00
Ronan Amicel
2a05d04e35 Ignore diacritics in autocomplete
Works for:
- Choice
- Choice List
- Reference
- Reference List

Co-Authored-By: Louis Delbosc <louis.delbosc.prestataire@anct.gouv.fr>
2022-08-16 09:20:08 +02:00
George Gevoian
0c5441b176 (core) Add unquarantine command to admin CLI
Summary:
Adds a CLI command to un-quarantine an active document. Also tweaks the
name of related environment variable to avoid a naming conflict.

Test Plan: Server test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3583
2022-08-15 13:04:55 -07:00
Paul Fitzpatrick
e95b215405 (core) updates from grist-core 2022-08-15 10:00:08 -04:00
Alex Hall
49cb51bac5 (core) Error explanations from friendly-traceback
Summary: Extend formula error messages with explanations from https://github.com/friendly-traceback/friendly-traceback. Only for Python 3.

Test Plan: Updated several Python tests. In general, these require separate branches for Python 2 and 3.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3542
2022-08-12 19:45:00 +02:00
George Gevoian
ebcfd2074f Fix bug that skips empty columns during imports
A faulty conditional in _makeDefaultTransformRule was the cause of the
bug. The conditional isn't necessary, as it's unreachable from the
import flows, so it was removed.
2022-08-11 11:05:30 -07:00
George Gevoian
3ad78590c2 (core) Polish display of table row counts
Summary:
Adds a new dots loader, which is used on the Raw Data page when
certain values are still being calculated (e.g. row counts). Now metrics
whose values aren't known yet will still appear under Usage, but with a
"Loading" message and the dots loader shown. For per-table row counts,
we only show the loader.

Test Plan: Existing tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3566
2022-08-11 09:45:33 -07:00
Jarosław Sadziński
f215ec8b24 (core) Editing text for viewers on empty site
Summary: Small tweaks for text that viewers see on empty site.

Test Plan: Updated tests.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3577
2022-08-09 22:24:54 +02:00
Jarosław Sadziński
9e4d802405 (core) Implementing row conditional formatting
Summary:
Conditional formatting can now be used for whole rows.
Related fix:
- Font styles weren't applicable for summary columns.
- Checkbox and slider weren't using colors properly

Test Plan: Existing and new tests

Reviewers: paulfitz, georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3547
2022-08-09 20:11:36 +02:00
George Gevoian
fbba6b8f52 (core) Add methods for quarantining documents
Summary:
Adds a new CLI command, doc, with a subcommand that quarantines
an active document. Adds a group query param to a housekeeping
endpoint for updating the document group prior to checking if a doc
needs to be reassigned. Both methods require support user credentials.

Test Plan: Server tests. (Additional testing will be done manually on staging.)

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3570
2022-08-09 09:26:48 -07:00
Jarosław Sadziński
ee109e9186 (core) Removing NEW_DEAL flag
Summary:
Removing NEW_DEAL flag checks and exposing all new deal features as default.
Also removing Pages.ts as it was moved to grist-core.

Test Plan: Existing and updated tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3572
2022-08-09 17:29:28 +02:00
Dmitry S
787b70f0f6 (core) Upgrade to latest plotly, and set automargin on Y-Axis
Test Plan: Only tested manually

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3543
2022-08-08 14:47:19 -04:00
Paul Fitzpatrick
c72ebf61a7 (core) updates from grist-core 2022-08-08 09:31:07 -04:00
Louis Delbosc
a82d02ebcc feat: add tests for feature 2022-08-08 13:30:12 +02:00
Jarosław Sadziński
6206644686 (core) Fixing redirect url for landing page
Summary: Redirect URL for landing page wasn't redirecting properly.

Test Plan: Existing tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3564
2022-08-04 18:53:57 +02:00
Louis Delbosc
531ea1a4b5 feat: add rename when clicking on selected label page 2022-08-04 14:18:29 +02:00
George Gevoian
1c6ab775dd (core) Polish upgrade button
Summary:
On mobile, clicking the upgrade button will now immediately display
the plans modal. The button margins have also been adjusted to be
smaller on mobile. Finally, some disabled options related to workspace
sharing in the left panel (on personal sites) are now hidden instead.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3555
2022-08-03 08:29:36 -07:00
George Gevoian
771e1edd54 (core) Keep track of row counts per table
Summary: Displays a live row count of each table on the Raw Data page.

Test Plan: Browser tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3540
2022-08-03 08:13:33 -07:00
Jarosław Sadziński
40c9b8b7e8 (core) New URL that opens Create site popup.
Summary: Adding new url parameter for team site creation

Test Plan: Updated tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3554
2022-08-03 13:09:18 +02:00
Jarosław Sadziński
c359547f6b (core) Welcome intro for viewers on a team site.
Summary:
Adding intro for a viewer on a teamsite.
Showing upgrade button for owners only.

Test Plan: new test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3557
2022-08-03 12:50:14 +02:00
Alex Hall
b7686fa664 (core) Add 'Show raw data' widget menu option
Summary:
Adds a new option to the top of the three dot widget menu which navigates to the raw data widget at the current cell, equivalent to clicking an anchor link.

Clicking the back button returns to the previous view. Clicking the close button shows the list of raw data tables instead, which isn't great, but the same is true when clicking an anchor link.

Test Plan: Added a test to `nbrowser/ChartView1.ts` because charts are the main thing we want this for.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3539
2022-08-02 20:01:44 +02:00
Paul Fitzpatrick
b6890bed4b (core) serialize document uploads and DocSnapshots.versions() to reduce surprises
Summary:
Occasionally, while the versions of a document are being enumerated,
a new version of the document will be created. This is detected and
triggers re-enumeration and a "surprise" log message. This diff
tweaks uploads to be run in series with DocSnapshots operations.
This means that listing versions would be blocked on an upload, or
vice versa, rather than overlapping. This is simpler and more deterministic.
I'm not sure how the user experience will feel if the operations
are slow.

Test Plan: existing tests pass; will see if surprises are reduced

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3551
2022-08-01 15:42:39 -04:00
George Gevoian
d12b05abcb (core) Keep focus when canceling choice rename
Summary:
Small bug fix that ensures focus isn't lost when Escape is pressed
while renaming a choice. This now makes it possible to close the
editor with a subsequent press of Escape.

Test Plan: Browser test.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3556
2022-08-01 12:08:36 -07:00
Paul Fitzpatrick
4c678f12cb (core) dust off electron build a little bit
Summary:
The changes in this diff are sufficient to make this sequence work again:

```
./build electron-dev
bin/electron app/electron/runPrebuild.js
```

This brings up the local server within an electron window.

This is an unambitious diff, aimed at checking how rusty electron support had become. It does not revive Grist as a packaged electron app. The first substantial work needed would be to make the app aware of the local file system again, and think through how local files should be visualized and accessed now. In the past, there was a simple list of grist docs in a directory.

Test Plan: manual

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3534
2022-07-29 11:19:26 -04:00
Paul Fitzpatrick
aeb7a4b849 (core) flesh out getAccessToken API documentation
Summary:
This extends the getAccessToken documentation so it can be picked
up by typedoc and published, and makes a few other tweaks along
the way prompted by a typescript/typedoc version change.

Test Plan: made in concert with a grist-help update

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3548
2022-07-28 08:50:12 -04:00
Cyprien P
80f31bffc2 (core) Allow left pane to auto-expand on mouse over
Summary:
Tweak PagePanels to let the left pane automatically expand on mouse
over. This is to make pages more accessible when the panel is
collapsed.

In this context, when expanding, the left panel overlap the main
content, reducing visual clutter.

Test Plan: updated

Reviewers: jarek

Reviewed By: jarek

Subscribers: anaisconce, jarek

Differential Revision: https://phab.getgrist.com/D3516
2022-07-28 11:41:42 +02:00
George Gevoian
c54dde3dba (core) Populate doc title, description and thumbnail in app.html
Summary:
Fills in the title and description/thumbnail (for templates) in app.html if the
page being requested is for a document.

Test Plan: Tested manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3544
2022-07-27 13:57:59 -07:00
Paul Fitzpatrick
7078922a65 (core) ensure randomness works when sandbox is cloned from a checkpoint
Summary:
This calls a new `initialize` method on the sandbox before we start
doing calculations with it, to make sure that `random.seed()` has
been called. Otherwise, if the sandbox is cloned from a checkpoint,
the seed will have been reset.

The `initialize` method includes the functionality previously done
by `set_doc_url` since it is also initialization/personalization and
this way we avoid introducing another round trip to the sandbox.

Test Plan: tested with grist-core configured to use gvisor

Reviewers: georgegevoian, dsagal

Reviewed By: georgegevoian, dsagal

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3549
2022-07-27 14:59:27 -04:00
Jarosław Sadziński
364715eba3 (core) FormulaEditor was updating formula when nothing has changed
Summary:
FormulaEditor on sidepanel was sending update action when
formula wasn't updated by the user.

Test Plan: Existing tests

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3545
2022-07-27 11:23:18 +02:00
George Gevoian
aeba738f7c (core) Add product for new personal plan
Summary:
Adds the new personal plan as a product that will be available
in the future. Can be enabled along with other plan-related via
an environment variable.

Test Plan: Browser tests and existing tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3533
2022-07-26 11:33:23 -07:00
Jarosław Sadziński
5c8211c61d (core) Fixing selectby error on the creator panel.
Summary: [Select By] in the creator panel was bugged. It wasn't refreshed in some cases as the observable array that needed to be created seemed too complicated. This Diff recomputes this array when the user wants to change the selection.

Test Plan: added tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3541
2022-07-26 15:46:11 +02:00
Alex Hall
460f22b701 (core) Don't show hidden tables in table data options when adding widgets
Summary:
As reported in https://grist.slack.com/archives/C069RUP71/p1655316194602829, when a table is hidden by ACL, it was still showing as a blank 'ghost' option to select data when adding a widget:

{F55498}

The fix is simply to return `true` from `isHiddenTable` for empty table IDs, which indicate a table hidden by ACL. `TableRec.isHidden` is supposed to match this so I updated it too, and I cleaned up a tiny bit of other related code.

Test Plan: Extended `nbrowser/AccessRules1.ts` to test the data options when adding widgets.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3530
2022-07-21 16:28:28 +02:00
Dmitry S
4e805a4d9c (core) Fix sizing of tracebacks in formula errors, to make it scrollable
Summary: When traceback is present, give it 64px, or more if available, or less if less is needed. If less space is available than needed, the traceback will scroll within its allocated area.

Test Plan: The test FieldEditorSizing which tests basic sizing still passes; details with different size of formula and traceback were tested manually.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3531
2022-07-21 09:56:52 -04:00
Paul Fitzpatrick
dd8d2e18f5 (core) add an access token mechanism to help with attachments in custom widgets
Summary:
With this, a custom widget can render an attachment by doing:
```
const tokenInfo = await grist.docApi.getAccessToken({readOnly: true});
const img = document.getElementById('the_image');
const id = record.C[0];  // get an id of an attachment
const src = `${tokenInfo.baseUrl}/attachments/${id}/download?auth=${tokenInfo.token}`;
img.setAttribute('src', src)
```

The access token expires after a few mins, so if a user right-clicks on an image
to save it, they may get access denied unless they refresh the page. A little awkward,
but s3 pre-authorized links behave similarly and it generally isn't a deal-breaker.

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3488
2022-07-19 11:55:18 -04:00
George Gevoian
5c0a250309 (core) Prevent raw table names from overflowing
Summary:
Modifies CSS so that text overflow is handled by displaying an
ellipsis when raw table names exceed the width of their container.

Test Plan: Tested manually.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3524
2022-07-19 08:25:31 -07:00
George Gevoian
3e49fe9a50 (core) Polish ChoiceListEntry drag and drop
Summary:
A green line indicating the insertion point is now shown in the
ChoiceListEntry component when dragging and dropping choices, similar
to the one shown in the choice list cell editor.

Test Plan: Tested manually.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3529
2022-07-19 08:14:04 -07:00
Alex Hall
4b258ae0fa (core) Fix 'select by' when adding summary table widget to page
Summary:
When adding a summary table widget to a page and using 'select by' in the add widget config (as opposed to in the right panel for an existing widget):

1. If an equivalent summary table already exists, use its referencing columns (if any) to construct link nodes. Previously the source table columns were being used instead, which could include referencing columns that don't have any equivalent in the summary table, and exclude referencing columns in the summary table.
2. If no such summary table exists yet, then keep using the source table columns, but only the selected groupby columns, and
3. After the summary table is created, correct the `linkTargetColRef` (which points to a source table column) to the corresponding column from the new summary table instead.

This fixes bugs which only appeared recently since 'select by' for a summary table previously involved no target columns.

Test Plan: Added two new tests to `nbrowser/SelectBySummaryRef`, and confirmed that they fail without the fixes to all three points above.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3527
2022-07-19 16:28:38 +02:00
Alex Hall
f39b496563 (core) Use table title instead of ID in ACL UI
Summary:
Use table titles (i.e. the raw data widget titles) in dropdowns and other parts of the Acess Rules page, instead of the table ID. This is particularly meant for summary tables which have/had an ID of the form `GristSummary_SourceTable_N`, but https://phab.getgrist.com/D3508 is changing that anyway.

The server method `getAclResources` now returns more metadata about each table so that the UI can display titles.

Test Plan: Extended and updated `nbrowser/AccessRules2.ts`. Added a small unit test for constructing table titles from the new description returned by `getAclResources`.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3494
2022-07-19 16:27:17 +02:00
Dmitry S
a0f405e45f (core) Include altSessionId in morgan request logging
Summary:
Most logging now includes altSessionId, but not the message logged at the end
of every request by the 'morgan' logger. This includes altSessionId in those
messages.

Test Plan: Verified that with GRIST_HOSTED_VERSION env var set, altSessionId is included in morgan-produced JSON messages.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3523
2022-07-18 16:09:41 -04:00
Jarosław Sadziński
257dafe423 (core) Improving billing page user experience
Summary:
Improving billing page user experience.
- Updated labels for canceled plan
- Adding option to downgrade from team plan to free team plan
- Updating default name for teamFree plan when it is not available in Stripe
- Minor bug fixes

Test Plan: updated tests

Reviewers: cyprien

Reviewed By: cyprien

Subscribers: cyprien

Differential Revision: https://phab.getgrist.com/D3515
2022-07-18 14:43:14 +02:00
George Gevoian
c70b427483 (core) Fix error when filtering empty choice and choice list columns
Summary:
A client error should no longer be thrown when filtering an empty Choice
or Choice List column.

Test Plan: Browser tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3528
2022-07-17 23:35:27 -07:00
Alex Hall
1a6e1872de (core) Miscellaneous little logging improvements
Summary:
1. Log errors in `ActiveDoc.loadDoc` as errors, not just warnings, except for a common 'Cannot create fork' error caused by deployment tests.

2. Log the method name that had an error in `server/lib/Client.ts`.

Discussion: https://grist.slack.com/archives/CR8HZ4P9V/p1652364998893169

Following up on https://phab.getgrist.com/D3522

Test Plan: tested manually, particularly by running the nbrowser/Fork test that led to the initial noisy errors in Slack.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3525
2022-07-15 00:21:44 +02:00
Alex Hall
333ed863f4 (core) Only allow getFormulaError for clients with access to read the cell
Summary: The previous access check in `getFormulaError` was not strict enough, allowing users to read the values of individual formula cells that they shouldn't be able to. Now `getCellValue` is used to check the access for the specific cell first.

Test Plan: Extended GranularAccess server test.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3526
2022-07-14 22:50:57 +02:00
Paul Fitzpatrick
ec8ab598cb (core) add a yarn run cli tool, and add a sqlite gristify option
Summary:
This adds rudimentary support for opening certain SQLite files in Grist.

If you have a file such as `landing.db` in Grist, you can convert it to Grist format by doing (either in monorepo or grist-core):
```
yarn run cli -h
yarn run cli sqlite -h
yarn run cli sqlite gristify landing.db
```

The file is now openable by Grist. To actually do so with the regular Grist server, you'll need to either import it, or convert some doc you don't care about in the `samples/` directory to be a soft link to it (and then force a reload).

This implementation is a rudimentary experiment. Here are some awkwardnesses:
 * Only tables that happen to have a column called `id`, and where the column happens to be an integer, can be opened directly with Grist as it is today. That could be generalized, but it looked more than a Gristathon's worth of work, so I instead used SQLite views.
 * Grist will handle tables that start with an uncapitalized letter a bit erratically. You can successfully add columns, for example, but removing them will cause sadness - Grist will rename the table in a confused way.
 * I didn't attempt to deal with column names with spaces etc (though views could deal with those).
 * I haven't tried to do any fancy type mapping.
 * Columns with constraints can make adding new rows impossible in Grist, since Grist requires that a row can be added with just a single cell set.

Test Plan: added small test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3502
2022-07-14 12:00:30 -04:00
Alex Hall
b8486dcdba (core) Nice summary table IDs
Summary:
Changes auto-generated summary table IDs from e.g. `GristSummary_6_Table1` to `Table1_summary_A_B` (meaning `Table1` grouped by `A` and `B`). This makes it easier to write formulas involving summary tables, make API requests, understand logs, etc.

Because these don't encode the source table ID as reliably as before, `decode_summary_table_name` now uses the summary table schema info, not just the summary table ID. Specifically, it looks at the type of the `group` column, which is `RefList:<source table id>`.

Renaming a source table renames the summary table as before, and now renaming a groupby column renames the summary table as well.

Conflicting table names are resolved in the usual way by adding a number at the end, e.g. `Table1_summary_A_B2`. These summary tables are not automatically renamed when the disambiguation is no longer needed.

A new migration renames all summary tables to the new scheme, and updates formulas using summary tables with a simple regex.

Test Plan:
Updated many tests to use the new style of name.

Added new Python tests to for resolving conflicts when renaming source tables and groupby columns.

Added a test for the migration, including renames in formulas.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3508
2022-07-14 12:09:56 +02:00
Alex Hall
f1df6c0a46 (core) Prevent logging pointless errors about attachments and data size on shutdown
Summary: As suggested in https://grist.slack.com/archives/CR8HZ4P9V/p1652365399661569?thread_ts=1652364998.893169&cid=CR8HZ4P9V, check if DocStorage is initialized before trying to use it when shutting down, to avoid noisy logging of errors about removing attachments and updating data size.

Test Plan: Tested manually that errors early in loadDoc caused logging of errors about attachments/data size before the changed, but not after.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3522
2022-07-14 12:09:19 +02:00
George Gevoian
c0852761b9 (core) Fix cell selection bugs
Summary:
Cell selection would sometimes get out of sync, causing
unexpected results when pasting. The UI would also incorrectly
indicate that rows/columns were still selected if you clicked the
selected cell (outlined in green) after doing a drag selection of
multiple rows/columns. Finally, canceling a copy operation would
fail to remove the "scissors" outline  around the copied cells if the
cursor was not on the copied selection.

This resolves all of these bugs.

Test Plan: Browser tests.

Reviewers: cyprien

Reviewed By: cyprien

Subscribers: cyprien

Differential Revision: https://phab.getgrist.com/D3517
2022-07-12 08:50:05 -07:00
Alex Hall
77775401fc (core) Don't clear widget options when changing column type
Summary: Previously, changing the type of a column would clear its widget options and conditional style rules by default, with a few exceptions to explicitly keep them. This diff reverses that behaviour, keeping the options by default.

Test Plan: Updated several existing tests, plus lots of manual testing.

Reviewers: cyprien

Reviewed By: cyprien

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3491
2022-07-11 14:27:44 +02:00
Jarosław Sadziński
8469b7ded0 (core) Fixing bug in token field editor with moving choices.
Summary:
Sometimes when rearranging items in choice editor, user can
put the new item inside last entry element, which is not recognized as
a choice entry.

Test Plan: manual tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3519
2022-07-08 21:03:53 +02:00
Jarosław Sadziński
ddb80f111e (core) Reference columns weren't added to Raw Data Views
Summary:
- Adding a column through 'Add Reference Column' adds it to Raw Data
- Migrating RefSelect.js to typescript
- Extending one of the tests

Test Plan: updated

Reviewers: cyprien

Reviewed By: cyprien

Subscribers: cyprien

Differential Revision: https://phab.getgrist.com/D3513
2022-07-07 17:07:13 +02:00
Paul Fitzpatrick
f91f45b26d (core) support granular read access for attachments
Summary:
When a user requests to read the contents of an attachment, only allow the request if there exists a cell in an attachment column that contains the attachment and which they have read access to.

This does not cover:
 * Granular write access for attachments. In particular, a user who can write to any attachment column should be considered to have full read access to all attachment columns, currently.
 * Access control of attachment metadata such as name and format.

The implementation uses a sql query that requires a scan, and some notes on how this could be optimized in future. The web client was updated to specify the cell to check for access, and performance seemed fine in casual testing on a doc with 1000s of attachments. I'm not sure how performance would hold up as the set of access rules grows as well.

Test Plan: added tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3490
2022-07-07 07:22:02 -04:00
George Gevoian
a051830aeb (core) Show summary tables on Raw Data page
Summary:
Summary tables now have their own raw viewsection, and are shown
under Raw Data Tables on the Raw Data page.

Test Plan: Browser and Python tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3495
2022-07-06 09:41:48 -07:00
Cyprien P
808aacdc52 (core) Fix CustomWidget nbrowser test
Summary:
 - test/nbrowser/CustomFilter keeps randomly fail on my local dev
 - it failed with `Cannot read property of null (reading 'postMessage`)` at line below:
 `this._rpc.setSendMessage(msg => this._iframe?.contentWindow!.postMessage(msg, '*'));`
 - I understand it was trying to send message before even the iframe was properly mounted
 - telling rpc to wait for the other end to send ready() successfully differ send message until everything's mounted.

Test Plan:  - should not break anything and test/nbrowser/CustomFilter should stop failing

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3510
2022-07-05 20:27:39 +02:00
Dmitry S
51ff72c15e (core) Faster builds all around.
Summary:
Building:
- Builds no longer wait for tsc for either client, server, or test targets. All use esbuild which is very fast.
- Build still runs tsc, but only to report errors. This may be turned off with `SKIP_TSC=1` env var.
- Grist-core continues to build using tsc.
- Esbuild requires ES6 module semantics. Typescript's esModuleInterop is turned
  on, so that tsc accepts and enforces correct usage.
- Client-side code is watched and bundled by webpack as before (using esbuild-loader)

Code changes:
- Imports must now follow ES6 semantics: `import * as X from ...` produces a
  module object; to import functions or class instances, use `import X from ...`.
- Everything is now built with isolatedModules flag. Some exports were updated for it.

Packages:
- Upgraded browserify dependency, and related packages (used for the distribution-building step).
- Building the distribution now uses esbuild's minification. babel-minify is no longer used.

Test Plan: Should have no behavior changes, existing tests should pass, and docker image should build too.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3506
2022-07-04 10:42:40 -04:00
Jarosław Sadziński
637caf8105 (core) Fixing bug with first click on windows
Summary:
On Windows, last row is selected when a user clicks the 'new row'
while browser doesn't have focus.

Test Plan: manual tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3503
2022-06-30 17:08:57 +02:00
Jarosław Sadziński
aefe451bab (core) Polishing upgrade plan UI
Summary:
- Update nudge boxes content and collapsing on personal and free team site
- New confirmation after upgrading from a free team site
- Refactoring ProductUpgrade code, splitting plans / modals and nudges

Test Plan: Manual and updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3481
2022-06-29 18:28:37 +02:00
Dmitry S
dd2eadc86e (core) Speed up and upgrade build.
Summary:
- Upgrades to build-related packages:
  - Upgrade typescript, related libraries and typings.
  - Upgrade webpack, eslint; add tsc-watch, node-dev, eslint_d.

- Build organization changes:
  - Build webpack from original typescript, transpiling only; with errors still
    reported by a background tsc watching process.

- Typescript-related changes:
  - Reduce imports of AWS dependencies (very noticeable speedup)
  - Avoid auto-loading global @types
  - Client code is now built with isolatedModules flag (for safe transpilation)
  - Use allowJs to avoid copying JS files manually.

- Linting changes
  - Enhance Arcanist ESLintLinter to run before/after commands, and set up to use eslint_d
  - Update eslint config, and include .eslintignore to avoid linting generated files.
  - Include a bunch of eslint-prompted and eslint-generated fixes
  - Add no-unused-expression rule to eslint, and fix a few warnings about it

- Other items:
  - Refactor cssInput to avoid circular dependency
  - Remove a bit of unused code, libraries, dependencies

Test Plan: No behavior changes, all existing tests pass. There are 30 tests fewer reported because `test_gpath.py` was removed (it's been unused for years)

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3498
2022-06-27 16:10:10 -04:00
Cyprien P
64ff9ccd0a (core) Allows range filter for Date, DateTime columns
Summary:
This diff is first of a series of 3 commits to enable range filering
for Date and DateTime columns. Diff only enable setting date's min/max
throw typing dates, Date picker and relative ranges are left for
follow-up commits.

 - Exposes columns value formatter to the range input
 - Fixes column filter func to work with dates

Test Plan:
Adds Date to projects range filter test
Adds Date/DateTime to nbrowser ColumnFilterMenu tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3455
2022-06-23 13:02:37 +02:00
Alex Hall
9fffb491f9 (core) External requests
Summary:
Adds a Python function `REQUEST` which makes an HTTP GET request. Behind the scenes it:

- Raises a special exception to stop trying to evaluate the current cell and just keep the existing value.
- Notes the request arguments which will be returned by `apply_user_actions`.
- Makes the actual request in NodeJS, which sends back the raw response data in a new action `RespondToRequests` which reevaluates the cell(s) that made the request.
- Wraps the response data in a class which mimics the `Response` class of the `requests` library.

In certain cases, this asynchronous flow doesn't work and the sandbox will instead synchronously call an exported JS method:

- When reevaluating a single cell to get a formula error, the request is made synchronously.
- When a formula makes multiple requests, the earlier responses are retrieved synchronously from files which store responses as long as needed to complete evaluating formulas. See https://grist.slack.com/archives/CL1LQ8AT0/p1653399747810139

Test Plan: Added Python and nbrowser tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: paulfitz, dsagal

Differential Revision: https://phab.getgrist.com/D3429
2022-06-17 21:53:20 +02:00
George Gevoian
abebe812db (core) Show Grist video tour after welcome questions
Summary:
After the welcome questions are dismissed, a video tour modal will
now be displayed. The video tour is also accessible via a tool button
in the left panel of the home page, as well as a text button next to
the Examples & Templates header.

Test Plan: Browser tests.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3477
2022-06-16 21:54:17 -07:00
Dmitry S
a91d493ffc (core) Fix issue with 'UNEXPECTED ORDER OF CALLBACKS' in Client.ts.
Summary:
- Substantial refactoring of the logic when the server fails to send some
  messages to a client.
- Add seqId numbers to server messages to ensure reliable order.
- Add a needReload flag in clientConnect for a clear indication whent the
  browser client needs to reload the app.
- Reproduce some potential failure scenarios in a test case (some of which
  previously could have led to incorrectly ordered messages).
- Convert other Comm tests to typescript.
- Tweak logging of Comm and Client to be slightly more concise (in particular,
  avoid logging sessionId)

Note that despite the big refactoring, this only addresses a fairly rare
situation, with websocket failures while server is trying to send to the
client. It includes no improvements for failures while the client is sending to
the server.

(I looked for an existing library that would take care of these issues. A relevant article I found is https://docs.microsoft.com/en-us/azure/azure-web-pubsub/howto-develop-reliable-clients, but it doesn't include a library for both ends, and is still in review. Other libraries with similar purposes did not inspire enough confidence.)

Test Plan: New test cases, which reproduce some previously problematic scenarios.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3470
2022-06-16 23:51:14 -04:00
Paul Fitzpatrick
561d9696aa (core) clean up interaction of forward auth with session
Summary:
For self-hosted Grist, forward auth has proven useful, where
some proxy wrapped around Grist manages authentication, and
passes on user information to Grist in a trusted header.
The current implementation is adequate when Grist is the
only place where the user logs in or out, but is confusing
otherwise (see https://github.com/gristlabs/grist-core/issues/207).
Here we take some steps to broaden the scenarios Grist's
forward auth support can be used with:

  * When a trusted header is present and is blank, treat
    that as the user not being logged in, and don't look
    any further for identity information. Specifically,
    don't look in Grist's session information.
  * Add a `GRIST_IGNORE_SESSION` flag to entirely prevent
    Grist from picking up identity information from a cookie,
    in order to avoid confusion between multiple login methods.
  * Add tests for common scenarios.

Test Plan: added tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3482
2022-06-15 13:06:12 -04:00
Alex Hall
0005ad013e (core) Notify open ActiveDocs when the product is upgraded
Summary:
When an account is upgraded to a new product in Billing, send a message to the redis channel `billingAccount-${accountId}-product-changed`.

ActiveDocs subscribe to this channel. When a message is received, they refresh their product from the database and use it to recalculate doc usage based on new limits. The new usage is broadcast to clients so they see the result of the upgrade live.

Test Plan: Extended nbrowser Billing test to test that a document open in a separate tab has its limit banner cleared immediately on upgrade.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3480
2022-06-14 17:25:45 +02:00
Dmitry S
b57a211741 (core) Fix issue with sandboxUtil where %s in message would get interpreted wrongly
Summary:
- Also converted sandboxUtil to typescript.
- The issue with %s manifested when a Python traceback contained "%s" in the
  string; in that case the object with log metadata (e.g. docId) would
  confusingly replace %s as if it were part of the message from Python.

Test Plan: Added a test case for the fix.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3486
2022-06-14 10:34:00 -04:00
George Gevoian
983638a5c5 (core) Disable overscroll in gridview
Summary:
This blocks browser back/forward trackpad gestures from unintentionally
triggering when the cursor is over an overflown gridview. The gestures
are still allowed elsewhere in the UI, as well as in gridviews that have
not overflown.

Test Plan: Tested manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3474
2022-06-13 11:28:33 -07:00
Paul Fitzpatrick
e5e361808b (core) repair grist-ee build and add smoke test
Summary:
This fixes the grist-ee build after recent changes, by
giving it a stub for ProductUpgrades.ts. Extends the
`core` test to also check if `ext` variant builds, to
catch the most common form of breakage on grist-ee so
far (file organization).

Test Plan: extends test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3484
2022-06-13 13:54:14 -04:00
George Gevoian
7176b7efb6 (core) Use new Banner component for activation messages
Summary: Use new Banner component for activation messages.

Test Plan: Existing tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3483
2022-06-13 10:20:31 -07:00
George Gevoian
02dd96daf2 (core) Add page titles to login and error pages
Summary:
Login and error pages now have their own unique page titles. This also fixes
the bug with the signed-out page having a page title of "Error".

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3479
2022-06-12 20:31:09 -07:00
Alex Hall
0350e2df58 (core) Fix filtering of empty reflists
Summary:
A formula returning an empty RecordSet in a RefList columns results in storing [] instead of null.
This caused a bug where the empty list was 'flattened' and the cell not appearing in filters at all.
This diff fixes the bug by filtering for the default value `null` instead for RefLists and the empty string for ChoiceLists.
I didn't manage to actually reproduce the bug for ChoiceLists, but this seemed the most sensible thing to do.

Test Plan: New nbrowser test.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3478
2022-06-11 00:16:57 +02:00
Jarosław Sadziński
64d9ecacdb (core) Renaming table linked by summary table resulted in error
Summary:
When a table was selected by a summary table, renaming that table using
section widget resulted in a javascript error.

Test Plan: new test

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3475
2022-06-10 09:39:34 +02:00
Alex Hall
1c89d08ea3 (core) Add a row to summary tables grouped by list column(s) corresponding to empty lists
Summary:
Adds some special handling to summary table and lookup logic:

- Source rows with empty choicelists/reflists get a corresponding summary row with an empty string/reference when grouping by that column, instead of excluding them from any group
- Adds a new `QueryOperation` 'empty' in the client which is used in `LinkingState`, `QuerySet`, and `recursiveMoveToCursorPos` to match empty lists in source tables against falsy values in linked summary tables.
- Adds a new parameter `match_empty` to the Python `CONTAINS` function so that regular formulas can implement the same behaviour as summary tables. See https://grist.slack.com/archives/C0234CPPXPA/p1654030490932119
- Uses the new `match_empty` argument in the formula generated for the `group` column when detaching a summary table.

Test Plan: Updated and extended Python and nbrowser tests of summary tables grouped by choicelists to test for new behaviour with empty lists.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3471
2022-06-09 23:38:14 +02:00
Jarosław Sadziński
3b30c052bc (core) Enabling clipboard events on Choice entry field
Summary: Copy/paste/cut events weren't available on choice entry field.

Test Plan: Updated tests

Reviewers: alexmojaki, paulfitz

Reviewed By: alexmojaki, paulfitz

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3476
2022-06-09 16:12:35 +02:00
Jarosław Sadziński
d92a761f6e (core) Product update popups and hosted stripe integration
Summary:
- Showing nudge to individual users to sign up for free team plan.
- Implementing billing page to upgrade from free team to pro.
- New modal with upgrade options and free team site signup.
- Integrating Stripe-hosted UI for checkout and plan management.

Test Plan: updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3456
2022-06-08 21:10:49 +02:00
Dmitry S
3b4d936013 (core) Hide 'helpCenter' and 'templates' elements in HomeIntro when requested
Summary:
Removes the relevant links in the intro text and buttons.

Note that the presence of actual "Examples & Templates" section of the page is controlled by whether any templates are available (returned by `getTemplates` api call)

Test Plan: Tested manually

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3473
2022-06-07 19:00:25 -04:00
Dmitry S
007c0f2af0 (core) Fix some bugs with repositioning rows.
Summary:
- Fixed an issue with manualSort values being very close floats. It is already handled by the data engine, but the client was being unnecessarily proactive and introduced a bug.
- The fix also helps with rearranging rows in filtered situations: they will now stay next to the row before which they were inserted.
- The fix accidentally improves (though doesn't fully fix) the issue where new columns show up in unexpected places in the raw-data column list.
- Fixed another rare bug with row order not getting updated correctly when positions update.

Test Plan: Added test cases for the improved behavior; fixed affected tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3462
2022-06-07 16:55:45 -04:00
Dmitry S
4f1cb53b29 (core) Converting server-side Comm.js to typescript
Summary:
- Add app/common/CommTypes.ts to define types shared by client and server.
- Include @types/ws npm package

Test Plan: Intended to have no changes in behavior

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3467
2022-06-07 15:47:17 -04:00
Alex Hall
519f1be93a (core) Disambiguate label for link between summary table and source table with self reference
Summary:
Following up on a small bug introduced in https://phab.getgrist.com/D3464. When a table has a column referencing the same table, then there can be two 'select by' options with the same label which is just the name of a summary table on the same page. The first option is simply filtering based on the summary table. The second option is linking the ref column in the source against the group column in the summary, but the name of the group column is hidden which leads to the ambiguity.

The solution in this diff is to always show the target node (source table) column name if the source node (summary table) column was the hidden group column. This also changes the label in the case where the reference to the source table isn't in the source table - see the updated test. This isn't strictly necessary in this case so I'm not 100% about the desired behaviour, but I don't think it hurts.

Test Plan: Tested disambiguation manually. Updated existing test.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3472
2022-06-07 17:07:45 +02:00
Cyprien P
4b54c7d99b (core) Fix column and view fields mismatch in filters
Summary:
The problem manifested while filtering by a column, closing the filter
would change the set of rows visible on the table. This would happen
only for rare table.

What caused that problem was that the filter being edited was wrongly
applyed also to another column, so depending on the content of the
column, it would entail unexpected behaviour.

The cause of that wrong association, was a mistakingly comparing the
id of two different type of thing: column and view field. The problem
would manifest if in the same section there were a view fields with
same row id as the column being filtered.

What made that confusion possible is the `.fieldOrColumn:
ViewFieldRec|ColumnRec` property of the FileInfo object, which could
hold either a view fields or a column record and was initialized with
view fields or columns if view fields was not found (ie: hidden
column).

Solution was to make sure FieldInfo is initialized with ColumnRec
alwasy (even for hidden column).

I'm not sure what is the reason why FilterInfo needed to support both
column record and view field record in the past, but it looks like
this is not needed anymore.

As a followup commit I think it would be worth the effort to refactor
FileInfo to accept only ColumnRec.

Test Plan: Includes new regression test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3463
2022-06-07 10:26:15 +02:00
Paul Fitzpatrick
05d1cdf140 (core) limit retries of uploads to external store in tests
Summary:
If an external store fails completely, Grist will continue to
retry uploading to it. This diff updated the HostedStorageManager
test to limit the extent of these retries to the test itself -
otherwise they continue for all other tests in the same process,
potentially disrupting those that read logs. There are other tests
that use s3, but they aren't run in the same process with delicate
log-reading tests, and it isn't quite as clear what improvement
to make there.

Test Plan:
artificially made external store fail, and checked that
test contamination seen previously no longer occurs.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3469
2022-06-06 16:19:41 -04:00
George Gevoian
6dcdd22792 (core) Redirect less often in welcomeNewUser
Summary:
Instead of always redirecting new users to the home page or the (teams) welcome page,
only redirect when the user signed in for the first time on a personal site, has access to
other sites, and isn't already being redirected to a specific page on their personal site.

Also tweaks how invalid Choice column values are displayed to match Choice List
columns, and fixes a small CSS issue with select by in the page widget picker when
there are options with long labels.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3461
2022-06-06 11:26:49 -07:00
George Gevoian
090d9af21d (core) Broadcast doc usage updates to clients
Summary:
Introduces a new message type, docUsage, that's broadcast to all connected
clients whenever document usage is updated in ActiveDoc.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3451
2022-06-06 09:55:34 -07:00
Paul Fitzpatrick
ff77ecc6c6 (core) deal with SQLITE_BUSY: database is locked during tests
Summary:
During testing, and in some dev configurations, we make connections to
the Grist home database from multiple processes, and store it in Sqlite.
This is a situation prone to file locks in Sqlite. We set a busy_timeout
parameter to help, but transactions may still fail immediately. This
diff adds retries to our patch of TypeORM for Sqlite transactions
(which without the patch are already flakey for other reasons
https://github.com/typeorm/typeorm/issues/1884#issuecomment-380767213).

Without this change, I can tickly SQLITE_BUSY problems fairly
reliably with the DocApi tests. With this change, I can't so far.

This change should not affect deployments of our SaaS (which use
Postgres for home db) or grist-core/grist-ee (which access home
db from a single process in default configuration).

Test Plan: repeated running of DocApi tests

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3466
2022-06-03 15:51:59 -04:00
Paul Fitzpatrick
1c6f80f956 (core) make it easier to enable Azure storage without setting GRIST_DOCS_S3_BUCKET
Summary:
Previously, absence of `GRIST_DOCS_S3_BUCKET` was equated with absence
of external storage, but that is no longer true now that Azure is
available. Azure could be used by setting `GRIST_DOCS_S3_BUCKET`
but the alternative `GRIST_AZURE_CONTAINER` flag is friendlier.

Test Plan:
confirmed manually that Azure can be configured and
used now without `GRIST_DOCS_S3_BUCKET`

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3448
2022-06-03 14:50:31 -04:00
Dmitry S
acddd25cfd (core) Update design of empty docs home page, and add a "Manage Team" button.
Summary:
- Remove the empty-folder icon
- Add an "Invite team members" button for owners on empty team sites
- Add a "Browse Templates" button for all other cases on empty sites
- Update intro text for team, including a link to Sprouts
- Update intro text for personal/anon.
- Include a Free/Pro tag for team sites (for now, only "Free")
- Add a "Manage Team" button for owners on home page of all team sites.
- Polished the UI of UserManager: add a transition for the background, and
  delay the appearance of the spinner for fast loads.

Test Plan: Fixed up the HomeIntro tests; added test case for Manage Team button

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3459
2022-06-03 12:58:57 -04:00
Alex Hall
af4738b94a (core) Allow linking to summary table via reference to source table
Summary:
Fixes a bug introduced in https://phab.getgrist.com/D3416 which exposed a new type of linking option that didn't actually work. Specifically it allowed selecting by a summary table when the target widget has a reference column to the source table of the summary. This diff correctly implements this linking by linking the reference column against the 'group' column of the summary table, so the source table data (which the client may not have access to) isn't involved. But the 'group' column name is hidden from the 'select by' option label to avoid confusion for users, so it just looks like another kind of summary table linking, and indeed it can be thought of purely in terms of matching groupby columns etc.

Discussion here: https://grist.slack.com/archives/C02EGJ1FUCV/p1654039063055499

Also fixes a related old bug that offered similar meaningless options involving summary tables, like selecting the summary table by the source table or linking summary tables with disjoint groupby columns.

Test Plan: Updated SelectBySummaryRef test and fixture doc for the new behaviour. Also updated a couple of older tests which were incorrectly asserting the buggy option to select a summary table by the source table.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3464
2022-06-03 17:04:41 +02:00
Cyprien P
815c9e1462 (core) Adds new range filter for numeric columns
Summary:
Shows the range filter next to the filter by values on filter menu. When users
set min and/or max, it takes precendence over the filter by values.

If users set:
 - `[] < [max]` behaves as `less than max`.
 - `[min] < []` behaves as `more than min`.
 - `[min] < [max]` behaves as `between min and max`
 - bounds are always inclusives.
 - when users change min or max the values of the by values filter
   gets checked/unchecked depending on whether they are included by
   the range filter.
 - when users clicks any btn/checkbox of the by values filter both min
   and max input gets cleared, and the filter convert to a filter by
   values.

Test Plan: Adds both projets and nbrowser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3435
2022-05-30 10:30:16 +02:00
Paul Fitzpatrick
dcaa2b4f29 (core) move DocUsageBanner to ext
Summary:
grist-ee build was failing since it didn't have a
DocUsageBanner implementation available. Made the implementation
added to monorepo available, since it will be useful to improve
the activation banner.

Test Plan: manaul

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3452
2022-05-27 22:19:17 -04:00
Paul Fitzpatrick
b9a4b2b58f (core) add missing tsconfig file that affects IDEs
Summary:
I missed committing a file that is important for editing files comfortably in the ext directory in an IDE. This diff:

 * Adds tsconfig-base-ext.json - that was the only intended change
 * Unrelated: Forces all creation of connections to the home db through a new `getOrCreateConnection` method which changes the `busy_timeout` if using Sqlite. This was an attempt to fix random "database is locked" test failures. I believe multiple connections to the home db as an sqlite file do not happen in self-hosted Grist (where there is a single node process) or in our SaaS (where the database is in postgres). It does affect Grist started using `devServerMain.ts` (where multiple processes accessing same database are started) or various test configurations when extra database connections are opened.
 * Unrelated: I added a `busy_timeout` for session storage, when it uses Sqlite. Again, I don't believe this affects self-hosted Grist or our SaaS.
 * Tweaked a `BillingDiscount` test that looked perhaps vulnerable to a stripe request stalling.

I can't be sure my tweaks actually help, since I didn't succeed in replicating the failures. Update: looks like the "locked" error can still happen :(

Test Plan: manual

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3450
2022-05-27 13:48:58 -04:00
Alex Hall
6b372fa6cd (core) Allow configuring (mostly hiding) various little bits of UI
Summary:
Adds two new env vars GRIST_HIDE_UI_ELEMENTS and GRIST_PAGE_TITLE_SUFFIX which translate to values in GristLoadConfig that the server sends the client when loading.

For checkin task https://gristlabs.getgrist.com/doc/check-ins/p/5#a1.s9.r1882.c19

Test Plan: Tested manually

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3449
2022-05-27 14:32:05 +02:00
George Gevoian
74ec9358da (core) Show usage banners in doc menu of free team sites
Summary:
Also fixes a minor CSS regression in UserManager where the
link to add a team member wasn't shown on a separate row.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3444
2022-05-26 15:01:35 -07:00
George Gevoian
3d3a5e334a (core) Update Plugin API documentation
Summary: Updates to Plugin API documentation.

Test Plan: Tested manually in grist-help.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3447
2022-05-24 17:27:34 -07:00
Cyprien P
6793377579 (core) Fix values ordering in column filter menu
Summary:
Column filter menu use to mess up the ordering of the items for
numeric and dates values, and also for ref/reflist columns when the
visible column is a numeric a date column.

Solution was to:
 - use the actual value of the visible column for comparison.
 - use native comparison.
 - tweak the native comparison to make blanks appears before valid value. Indeed, it came up several time that it's convenient to have invalid values show up first in the filter panel, it makes for a convenient way to detect them.

Test Plan: Adds new nbrowser test

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3441
2022-05-24 16:30:28 +02:00
Alex Hall
fcbad1c887 (core) Add GET /attachments endpoint for listing attachment metadata
Summary: Combines the code and behaviour of the existing endpoints `GET /records` (for the general shape of the result and the parameters for sort/filter/limit etc) and retrieving a specific attachment with `GET /attachments/:id` for handling fields specific to attachments.

Test Plan: Added a DocApi test. Also updated one test to use the new endpoint instead of raw `GET /tables/_grist_Attachments/records`.

Reviewers: cyprien

Reviewed By: cyprien

Subscribers: cyprien

Differential Revision: https://phab.getgrist.com/D3443
2022-05-20 20:30:14 +02:00
Dmitry S
9bc04a6e66 (core) Replace nulls in X-axis of charts with '-' to avoid confusion with missing points
Summary:
Seem in a customer doc where Y-scale was wrong for unclear reasons. The cause
is that null-valued X labels cause the corresponding bar (or point) to be
omitted, but still affect the Y-scale. In this diff, such labels are replaced
with "-", so as to show up normally.

Trivial example that reproduces the problem here:
https://public.getgrist.com/iLPpx9C5i8nk/Null-in-X-Axis

Test Plan: Tested manually.

Reviewers: cyprien

Reviewed By: cyprien

Subscribers: cyprien

Differential Revision: https://phab.getgrist.com/D3442
2022-05-20 11:54:20 -04:00
Cyprien P
8f4f21e94a (core) Filter menu show all options for Bool/Choice/Choice List columns
Summary:
> Toggle, Choice, and Choice List need all possible values available in filter, not just values present in current records
https://grist.quip.com/cjw4A8AHx1vh/Filtering-Improvements#temp:C:PZHc42e8be8cd8547bb8ce93fdb0

Test Plan: Adds new nbrowser test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3436
2022-05-19 20:43:29 +02:00
Dmitry S
309ddb0fe7 (core) Move guessing logic for column types to run in node once for all columns.
Summary:
Previously, columns of type Any were created and modified one by one by reusing
the "empty column" logic from the data engine. This copies that logic to Node,
and sets the type of all columns together, to create them with the correct type
in the AddTable call.

This makes imports about twice faster (when slowness is due to many columns),
but doesn't address all cases where individual handling of columns causes slowness.

Test Plan: Added a test case for the new helper function.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3427
2022-05-19 12:49:51 -04:00
George Gevoian
a6063f570a (core) Polish Access Details
Summary:
Instead of showing a blank dialog for users whose access
is limited (e.g. public members), we now show the user's
role and a mention of whether their access is public.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3431
2022-05-19 09:45:26 -07:00
George Gevoian
bad4c68569 (core) Reduce a few log levels to warning
Summary: Reduces the log level in a few places from error to warning.

Test Plan: N/A

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3437
2022-05-18 14:57:43 -07:00
Jarosław Sadziński
0ab9e4a6a0 (core) Adding GristConnect login system
Summary:
New login system to allow simple SSO flow that is based on Discourse description that is available at:
https://meta.discourse.org/t/discourseconnect-official-single-sign-on-for-discourse-sso/13045

Test Plan: New core test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3418
2022-05-18 20:28:25 +02:00
Paul Fitzpatrick
cf23a2d1ee (core) add GVISOR_LIMIT_MEMORY to cap memory available in sandbox
Summary:
This allows limiting the memory available to documents in the sandbox when gvisor is used. If memory limit is exceeded, we offer to open doc in recovery mode. Recovery mode is tweaked to open docs with tables in "ondemand" mode, which will generally take less memory and allow for deleting rows.

The limit is on the size of the virtual address space available to the sandbox (`RLIMIT_AS`), which in practice appears to function as one would want, and is the only practical option. There is a documented `RLIMIT_RSS` limit to `specifies the limit (in bytes) of the process's resident set (the number of virtual pages resident in RAM)` but this is no longer enforced by the kernel (neither the host nor gvisor).

When the sandbox runs out of memory, there are many ways it can fail. This diff catches all the ones I saw, but there could be more.

Test Plan: added tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3398
2022-05-18 14:26:27 -04:00
George Gevoian
2fd8a34ff8 (core) Move Notifier to /ext
Summary:
This makes it possible to configure a SendGrid-based Notifier
instance via a JSON configuration file.

Test Plan: Tested manually.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3432
2022-05-18 08:02:32 -07:00
Cyprien P
365f3c7ae2 (core) Auto-check aggregate values option as soon as table is a summary
Summary:
Auto-check makes sure that the “Aggregate Values” option is checked as
soon as the chart table is summarized. Before, user had to check it
explicitly.

More on this:
https://grist.slack.com/archives/C04AYS9JF/p1649400119930389?thread_ts=1649338496.915759&cid=C04AYS9JF

Test Plan: Updated tests

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3422
2022-05-18 10:10:28 +02:00
Cyprien P
f17e31c023 (core) Ensure chart summary is more detailed then linked summary
Summary:
When a summary table is linked, the column used for linking needs
to be included as a group-by column (or the linking can’t work). A
good example is here:
https://templates-s.getgrist.com/doc/investment-research/p/4

This was request by Dmitry here: https://grist.slack.com/archives/C04AYS9JF/p1649400119930389?thread_ts=1649338496.915759&cid=C04AYS9JF (2nd point)

Test Plan: Added new nbrowser test case.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3408
2022-05-17 08:57:05 +02:00
Paul Fitzpatrick
e4d47a2f3c (core) add minimal support for activation keys
Summary: For grist-ee, expect an activation key in environment variable `GRIST_ACTIVATION` or in a file pointed to by `GRIST_ACTIVATION_FILE`. In absence of key, start a 30-day trial, during which a banner is shown. Once trial expires, installation goes into document-read-only mode.

Test Plan: added a test

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3426
2022-05-16 22:46:23 -04:00
George Gevoian
f48d579f64 (core) Add API endpoint to get site usage summary
Summary:
The summary includes a count of documents that are approaching
limits, in grace period, or delete-only. The endpoint is only accessible
to site owners, and is currently unused. A follow-up diff will add usage
banners to the site home page, which will use the response from the
endpoint to communicate usage information to owners.

Test Plan: Browser and server tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3420
2022-05-16 11:16:19 -07:00
Alex Hall
cbdbe3f605 (core) Switch webhook secret cache from LRU to TTL so that unsubscribing can drain the queue
Summary:
Helps with cases such as https://grist.slack.com/archives/C02EGJ1FUCV/p1652196111066649?thread_ts=1651656433.171889&cid=C02EGJ1FUCV

When a user unsubscribes from a webhook, the secret URL is deleted from the database, but as long as the doc was open it would continue retrying pending requests still in the queue for a long time, using the locally cached value without noticing the effect of unsubscribing. This change allows unsubscribing to have an effect more quickly so that problematic events can be removed from the queue.

Test Plan: existing tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3430
2022-05-16 18:02:19 +02:00
Jarosław Sadziński
f93b4fd3bc (core) Selection for styled columns
Summary:
Grid UI improvements.
- Selecting a column with a custom background, didn't produce a visual difference.
- Hiding num-row right border when it is not needed (it should be used only for frozen columns)

Test Plan: manual tests

Reviewers: cyprien

Reviewed By: cyprien

Differential Revision: https://phab.getgrist.com/D3425
2022-05-13 09:45:17 +02:00
George Gevoian
524dbf34e1 (core) Add config to include custom CSS
Summary:
Adds a new environment variable that allows for custom
CSS to be included in all core static pages.

Test Plan: Tested manually in grist-core.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3419
2022-05-12 11:13:52 -07:00
Paul Fitzpatrick
e6983e9209 (core) add machinery for self-managed flavor of Grist
Summary:
Currently, we have two ways that we deliver Grist. One is grist-core,
which has simple defaults and is relatively easy for third parties to
deploy. The second is our internal build for our SaaS, which is the
opposite. For self-managed Grist, a planned paid on-premise version
of Grist, I adopt the following approach:

 * Use the `grist-core` build mechanism, extending it to accept an
   overlay of extra code if present.
 * Extra code is supplied in a self-contained `ext` directory, with
   an `ext/app` directory that is of same structure as core `app`
   and `stubs/app`.
 * The `ext` directory also contains information about extra
   node dependencies needed beyond that of `grist-core`.
 * The `ext` directory is contained within our monorepo rather than
   `grist-core` since it may contain material not under the Apache
   license.

Docker builds are achieved in our monorepo by using the `--build-context`
functionality to add in `ext` during the regular `grist-core` build:

```
docker buildx build --load -t gristlabs/grist-ee --build-context=ext=../ext .
```

Incremental builds in our monorepo are achieved with the `build_core.sh` helper,
like:

```
buildtools/build_core.sh /tmp/self-managed
cd /tmp/self-managed
yarn start
```

The initial `ext` directory contains material for snapshotting to S3.
If you build the docker image as above, and have S3 access, you can
do something like:

```
docker run -p 8484:8484 --env GRIST_SESSION_SECRET=a-secret \
  --env GRIST_DOCS_S3_BUCKET=grist-docs-test \
  --env GRIST_DOCS_S3_PREFIX=self-managed \
  -v $HOME/.aws:/root/.aws -it gristlabs/grist-ee
```

This will start a version of Grist that is like `grist-core` but with
S3 snapshots enabled. To release this code to `grist-core`, it would
just need to move from `ext/app` to `app` within core.

I tried a lot of ways of organizing self-managed Grist, and this was
what made me happiest. There are a lot of trade-offs, but here is what
I was looking for:

 * Only OSS-code in grist-core. Adding mixed-license material there
   feels unfair to people already working with the repo. That said,
   a possible future is to move away from our private monorepo to
   a public mixed-licence repo, which could have the same relationship
   with grist-core as the monorepo has.
 * Minimal differences between self-managed builds and one of our
   existing builds, ideally hewing as close to grist-core as possible
   for ease of documentation, debugging, and maintenance.
 * Ideally, docker builds without copying files around (the new
   `--build-context` functionality made that possible).
 * Compatibility with monorepo build.

Expressing dependencies of the extra code in `ext` proved tricky to
do in a clean way. Yarn/npm fought me every step of the way - everything
related to optional dependencies was unsatisfactory in some respect.
Yarn2 is flexible but smells like it might be overreach. In the end,
organizing to install non-core dependencies one directory up from the
main build was a good simple trick that saved my bacon.

This diff gets us to the point of building `grist-ee` images conveniently,
but there isn't a public repo people can go look at to see its source. This
could be generated by taking `grist-core`, adding the `ext` directory
to it, and pushing to a distinct repository. I'm not in a hurry to do that,
since a PR to that repo would be hard to sync with our monorepo and
`grist-core`. Also, we don't have any licensing text ready for the `ext`
directory. So leaving that for future work.

Test Plan: manual

Reviewers: georgegevoian, alexmojaki

Reviewed By: georgegevoian, alexmojaki

Differential Revision: https://phab.getgrist.com/D3415
2022-05-12 12:39:52 -04:00
Alex Hall
b878395c21 (core) Allow linking summary tables based on ref/reflist columns (except group)
Summary:
Relax the restriction in `selectBy.isValidLink` so that summary tables can be linked by a column like other tables, except the `group` column. See the discussion on https://grist.slack.com/archives/C0234CPPXPA/p1651773623256959 (the replies are on the following message) for more info on this decision.

Tweaked `LinkingState.ts` since linking with summary tables can now involve a column.

Test Plan: Added a new nbrowser test and fixture checking the options to select by given a summary table with a few ref/reflist columns. Manually tested the behaviour of each option.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3416
2022-05-12 15:59:12 +02:00
Alex Hall
6c90de4d62 (core) Switch excel import parsing from messytables+xlrd to openpyxl, and ignore empty rows
Summary:
Use openpyxl instead of messytables (which used xlrd internally) in import_xls.py.

Skip empty rows since excel files can easily contain huge numbers of them.

Drop support for xls files (which openpyxl doesn't support) in favour of the newer xlsx format.

Fix some details relating to python virtualenvs and dependencies, as Jenkins was failing to find new Python dependencies.

Test Plan: Mostly relying on existing tests. Updated various tests which referred to xls files instead of xlsx. Added a Python test for skipping empty rows.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3406
2022-05-12 14:43:21 +02:00
Cyprien P
87cc7d755d (core) Fix chart mixing up axis
Summary:
Diff fixes couple edge cases:
 - When changing chart's groupby columns, the data-engine changes the view fields, which is not okay with charts, hence makes sure view fields stay the same as much as possible using `this.setSectionViewFieldsFromArray()`
 - Also there's a logic in `this.setSectionViewFieldsFromArray()` that handle what to do when some columns goes missing during updates. Diff fixes this logic two. This is really corner case.

Test Plan: Both cases are not added to ChartAggregate.ts

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3405
2022-05-11 09:18:31 +02:00
Alex Hall
4408315f2e (core) Add AzureExternalStorage
Summary:
Adds a new implementation of the interface ExternalStorage that works for Azure Blob Storage as an alternative to S3, for a specific self-hosting case.

Tweaks HostedStorageManager and ICreate to allow configuring different core implementations of ExternalStorage.

Followup tasks:

- Make this code available to self hosters, possibly by making it open source.
- Add an env var or other config option to specify the preferred type of storage. Currently using the var `AZURE_STORAGE_CONNECTION_STRING` to know how to connect to Azure when requested, but that choice still only lives in test code.

Test Plan: Generalized HostedStorageManager and ExternalStorage tests to test the new AzureExternalStorage alongside S3ExternalStorage. The HostedStorageManager tests also now test the 'cached' in-memory test storage in a way that's closer to the real storage methods.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3413
2022-05-09 21:44:57 +02:00
Jarosław Sadziński
db57815d2b (core) Improving custom widget API. Changing destroy function signature.
Summary:
Destroy function in TableOperations was throwing error when invoked with a single
record id instead of an array. Now it returns a void type.

Also changing mapColumns function signature as it doesn't require options for a default
behavior.

Test Plan: Updated tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3404
2022-05-05 16:34:26 +02:00
George Gevoian
1e42871cc9 (core) Add attachment and data size usage
Summary:
Adds attachment and data size to the usage section of
the raw data page. Also makes in-document usage banners
update as user actions are applied, causing them to be
hidden/shown or updated based on the current state of
the document.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3395
2022-05-04 13:46:55 -07:00
Jarosław Sadziński
f194d6861b (core) Updating RawData views
Summary:
- Better focus on the widget title
- Adding columns only to the current view section
- New popup with options when user wants to delete a page
- New dialog to enter table name
- New table as a widget doesn't create a separate page
- Removing a table doesn't remove the primary view

Test Plan: Updated and new tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3410
2022-05-04 21:41:42 +02:00
Dmitry S
97f3a8805c (core) List 'Other Sites' all one line; fix style ordering issue for sort button.
Test Plan: Tested manually by eyeballing

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3407
2022-04-28 23:00:40 -04:00
George Gevoian
ad04744b4a (core) Fix import bug when skipping non-text columns
Summary:
Skipping columns during incremental imports wasn't working for certain
column types, such as numeric columns. The column's default value was
being used instead (e.g. 0), overwriting values in the destination
table.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3402
2022-04-28 12:46:44 -07:00
Alex Hall
dc9e53edc8 (core) Update the current time in formulas automatically every hour
Summary: Adds a special user action `UpdateCurrentTime` which invalidates an internal engine dependency node that doesn't belong to any table but is 'used' by the `NOW()` function. Applies the action automatically every hour.

Test Plan: Added a Python test for the user action. Tested the interval periodically applying the action manually: {F43312}

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3389
2022-04-28 21:07:40 +02:00
Alex Hall
0beb2898cb (core) Add flexibility to daily API usage limit
Summary: Allow exceeding the daily API usage limit for a doc based on additional allocations for the current hour and minute. See the doc comment on getDocApiUsageKeysToIncr for details. This means that up to 5 redis keys may be relevant at a time for a single document.

Test Plan: Updated and expanded 'Daily API Limit' tests.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3368
2022-04-28 16:22:18 +02:00
Paul Fitzpatrick
4de5928396 (core) when redirecting, use protocol in APP_HOME_URL if available
Summary:
Currently, Grist behind a reverse proxy will generate many
needless redirects via `http`, and can't be used with only
port 443. This diff centralizes generation of these redirects
and uses the protocol in APP_HOME_URL if it is set.

Test Plan:
manually tested by rebuilding grist-core and
doing a reverse proxy deployment that had no support for
port 80. Prior to this change, there are lots of problems;
after, the site works as expected.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3400
2022-04-28 09:13:27 -04:00
Jarosław Sadziński
6f00106d7c (core) Raw renames
Summary:
A new way for renaming tables.

  - There is a new popup to rename section (where you can also rename the table)
  - Renaming/Deleting page doesn't modify/delete the table.
  - Renaming table can rename a page if the names match (and the page contains a section with that table).
  - User can rename table in Raw Data UI in two ways - either on the listing or by using the section name popup
  - As before, there is no way to change tableId - it is derived from a table name.
  - When the section name is empty the table name is shown instead.
  - White space for section name is allowed (to discuss) - so the user can just paste '   '.
  - Empty name for a page is not allowed (but white space is).
  - Some bugs related to deleting tables with attached summary tables (and with undoing this operation) were fixed (but not all of them yet).

Test Plan: Updated tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3360
2022-04-27 22:21:55 +02:00
Jarosław Sadziński
8a1cca629b (core) Changing shortcut for Duplicate row command
Summary:
Changing keyboard shortcut for Duplicate row from
Ctrl+Shift+D to Mod+Shift+D

Test Plan: existing test

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3401
2022-04-27 22:02:27 +02:00
Jarosław Sadziński
995bf9b63a (core) Distinct style rules for summary columns
Summary:
Summary columns now have their own conditional rules,
which are not shared with sister columns.

Test Plan: New test

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3388
2022-04-27 20:51:23 +02:00
Dmitry S
e59dcc142d (core) Show proper message on empty Excel import, rather than a code error
Summary:
- Previously showed "UnboundLocalError". Now will show:
    Import failed: Failed to parse Excel file.
    Error: No tables found (1 empty tables skipped)
- Also fix logging for import code

Test Plan: Added a test case

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3396
2022-04-27 00:49:28 -04:00
Jarosław Sadziński
dcafa96b5b (core) Restoring context menu when comparing documents
Summary:
Fixing bug when comparing documents. User wasn't able
to open cell/column context menu due to javascript error.

Test Plan: Updated tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3397
2022-04-26 21:31:36 +02:00
Alex Hall
040fa85a8b (core) Simplify InitNewDoc since the timezone and locale is never actually used
Summary: InitNewDoc is essentially only used to generate initialDocSql, so it doesn't make sense to set the timezone and locale. They are always set when actually creating a new doc anyway. Discussed in https://grist.slack.com/archives/C0234CPPXPA/p1650312714217089.

Test Plan: this

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3394
2022-04-26 00:08:03 +02:00
George Gevoian
af5b3c9004 (core) Add document usage banners
Summary:
This also enables the new Usage section for all sites. Currently,
it shows metrics for document row count, but only if the user
has full document read access. Otherwise, a message about
insufficient access is shown.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3377
2022-04-25 08:14:52 -07:00
Alex Hall
a701b4bf13 (core) Remove expired attachments every hour and on shutdown
Summary:
Call ActiveDoc.removeUnusedAttachments every hour using setInterval, and in ActiveDoc.shutdown (which also clears said interval).

Unrelated: small fix to my webhooks code which was creating a redis client on shutdown just to quit it.

Test Plan:
Tweaked DocApi test to remove expired attachments by force-reloading the doc, so that it removes them during shutdown. Extracted a new testing endpoint /verifyFiles to support this test (previously running that code only happened with `/removeUnused?verifyfiles=1`).

Tested the setInterval part manually.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3387
2022-04-22 20:43:59 +02:00
Alex Hall
890c550fc3 (core) Don't include adding attachment metadata in undo stack
Summary: Mark actions adding attachment metadata as 'internal' (not part of undo stack) which previously was only for the Calculate action.

Test Plan: Extended nbrowser attachments test

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3380
2022-04-22 18:39:54 +02:00
Jarosław Sadziński
bedb19f9c7 (core) Removing conditional styles for blank rows
Summary:
When grid view was not fully loaded the initial blank row
was marked as invalid by conditional styles. This removes the styling
when row is not fully loaded.

Test Plan: Tested manually

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3392
2022-04-21 11:45:17 +02:00
Jarosław Sadziński
77ef9df27d (core) Adding new command Duplicate rows
Summary:
New command "Duplicate rows" is available in the Row/Card Context Menu and as a keyboard shortcut Ctrl+Alt+C.
- All selected rows are duplicated (even if only a single column is selected)
- Rows are inserted immediately after the last selected row (using manualSort value).
- Formulas and CENSORED fields are not copied.
Implemented on the UI level (no new action).

Test Plan: new test

Reviewers: cyprien

Reviewed By: cyprien

Differential Revision: https://phab.getgrist.com/D3371
2022-04-20 17:29:48 +02:00
Alex Hall
d7514e9cfc (core) Create _grist_Attachments_fileIdent index in new docs
Summary: Patching up the mistake in https://phab.getgrist.com/D3374#inline-38023.

Test Plan: this

Reviewers: dsagal, paulfitz

Reviewed By: dsagal, paulfitz

Differential Revision: https://phab.getgrist.com/D3382
2022-04-19 21:21:52 +02:00
Alex Hall
47b77c8c24 (core) Allow filtering by selected cell value in cell context menu
Summary: Adds a command and `BaseView` method `filterByThisCellValue`.

Test Plan: Added two tests to `nbrowser/CellContextMenu.ts`

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3383
2022-04-19 11:51:47 +02:00
Paul Fitzpatrick
ce7eb05ed4 (core) get user.Name through same mechanism as user.id for websocket Client
Summary:
This avoids an extra database query to look up the user's current
name, by capturing it at the moment their user id is queried.

Test Plan: existing test for user.Name changes continues to pass

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3381
2022-04-14 12:49:35 -04:00
Alex Hall
64a5c79dbc (core) Limit total attachment file size per document
Summary:
- Add a new parameter `Features.baseMaxAttachmentsBytesPerDocument` and set it to 1GB for the free team product.
- Add a method to DocStorage to calculate the total size of existing and used attachments.
- Add a migration to DocStorage adding an index to make the query in the above method fast.
- Check in ActiveDoc if uploading attachment(s) would exceed the product limit on that document.

Test Plan: Added test in `limits.ts` testing enforcement of the attachment limit.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3374
2022-04-14 16:33:09 +02:00
Paul Fitzpatrick
c1af5a9803 (core) have user.Name come from database for websocket users
Summary: The name of a user for actions made using a websocket until now could be inconsistent with that seen by other means. This draws the name from the database, rather than from session information that may have been cached from an identity provider.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3379
2022-04-13 17:46:46 -04:00
Jarosław Sadziński
dea1a8ba1b (core) Implementing search on raw data view
Summary:
Search now works on Raw Data Page.
- Search bar option 'Search on all pages' will change to 'Search on all tables' when on the Raw data page, and will allow searching through all tables.
- Little CSS adjustment for an overlay on Raw page (removes z-index as it is not needed, and conflicts with searchbar).
- Search bar option ('search on all') gets white background, little padding, and is moved 2 pixels up, this is needed for Raw page.

Test Plan: new and updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3376
2022-04-13 20:44:37 +02:00
George Gevoian
007a862333 (core) Include hash/fragment in post-login redirect URLs
Summary: Also cleans up login URLs by excluding params and hashes.

Test Plan: Client and server tests.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3378
2022-04-13 09:16:29 -07:00
Paul Fitzpatrick
20dd2fc70d (core) allow non-owners to remove themselves from sites/workspaces/docs
Summary:
For users who cannot otherwise change access to a resource, let
them remove themselves. Implemented via the standard endpoints
as a special exception that will process a request from a user
that would otherwise be denied, if the only contents of that
request are a removal of themselves.

Users who can change access are still not permitted to change their
own permissions or to remove themselves, as a precaution against
orphaning resources.

Test Plan: extended and updated tests

Reviewers: cyprien

Reviewed By: cyprien

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3367
2022-04-13 10:04:32 -04:00
Cyprien P
25e40bfa9b (core) Fix setting xaxis when both chart aggregation and split series
Summary:
 - Symptoms where that Split Series could end up being turned off for
   no good reason. Also both x axis and split series could be mixed
   up.

 - Problems was caused by call to `setGroupByColumns` which modifies
   the sections viewFields. Diff fixes it by adjustin slightly the
   ordering of function call in `_setXAxis()`.

 - Problem of mixing up x axis and split series was fixed by being
   careful on the order of columns passed to the `setGroupByColumns`
   which then determine the ordering of the view fields.

Test Plan: Includes new test cases

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3365
2022-04-13 11:25:24 +02:00
Alex Hall
09da815c0c (core) Add /attachments/removeUnused DocApi endpoint to hard delete all unused attachments in document
Summary: Adds methods to delete metadata rows based on timeDeleted. The flag expiredOnly determines if it only deletes attachments that were soft-deleted 7 days ago, or just all soft-deleted rows. Then any actual file data that doesn't have matching metadata is deleted.

Test Plan: DocApi test

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3364
2022-04-12 17:11:11 +02:00
Jarosław Sadziński
4401ec4d79 (core) Adding onNewRecord event for custom widgets
Summary:
New onNewRecord event in custom widgets that is triggered
when the user selects a blank row.

Together with this diff there is PR for grist-help https://github.com/gristlabs/grist-help/pull/150

Test Plan: added test case

Reviewers: cyprien

Reviewed By: cyprien

Differential Revision: https://phab.getgrist.com/D3375
2022-04-12 16:51:16 +02:00
Dmitry S
cedcdc6bff (core) Improve debug logging related for client-side errors and sandbox crashes.
Summary:
- Include docId when available for client-side error reporting
- Distinguish sandbox crashes from forced exits

Test Plan: Tested manually

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3373
2022-04-11 17:54:40 -04:00
George Gevoian
859c593448 (core) Add authSubject and authProvider to sessions
Summary:
This also updates Authorizer to link the authSubject
to Grist users if not previously linked. Linked subjects
are now used as the username for password-based logins,
instead of emails, which remain as a fallback.

Test Plan: Existing tests, and tested login flows manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3356
2022-04-11 11:42:02 -07:00
Paul Fitzpatrick
14f7e30e6f (core) add users.options.isConsultant flag, and omit such users from billing
Summary:
This adds an optional `isConsultant` flag to `users.options`, and an endpoint that allows the support user to turn it on or off. Users marked as consultants are not counted as billable members. Follows the example of existing `allowGoogleLogin` option.

Billable members are counted when members are added or removed from a site. Changing the `isConsultant` flag has no immediate or retroactive effect on billing. The number of users in stripe is now set unconditionally, rather than only when it has changed.

Notifications to billing managers are not aware of this billing nuance, but continue to report user counts that include consultants. The notifications link users to the billing page.

Test Plan: extended test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: anaisconce, jarek

Differential Revision: https://phab.getgrist.com/D3362
2022-04-11 10:26:31 -04:00
George Gevoian
4c5de16e2d (core) Include altSessionId in logs
Summary: Adds altSessionId to log output.

Test Plan: Tested manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3355
2022-04-08 16:40:34 -07:00
Cyprien P
bf8769bc42 (core) Fix linking after a summary update
Summary:
When linking table to a summary, the linking ended up broken after updating the summary group by columns.
This diff fixes that issue.

There were two issues:
 1) some subscriptions were missing due to some .peek() calls instead of directly calling the observable.
 2) the LinkingState instance was not being disposed.
 3) the filterColValues was not updating after source data table has been loaded

Test Plan: Include new test file.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3358
2022-04-08 12:28:42 +02:00
Jarosław Sadziński
34708cd348 (core) Adding font options to the style picker
Summary:
Redesigning color picker:
- Single color palette (no light/dark switch)
- Ability to remove color (new empty button)

New font options in the color picker.
Font options are available on:
- Default cell style
- Conditional rules styles
- Choice/ChoiceList editor and token field
- Filters for Choice/ChoiceList columns

Design document:
https://www.figma.com/file/bRTsb47VIOVBfJPj0qF3C9/Grist-Updates?node-id=415%3A8135

Test Plan: new and updated tests

Reviewers: georgegevoian, alexmojaki

Reviewed By: georgegevoian, alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3335
2022-04-07 20:35:03 +02:00
Cyprien P
98ac2f7e5b (core) add 'Aggregate Values' option to chart config
Summary:
  - adds the `aggregate values` option to chart config
    - aggregation is performed by turning table into summary table
    - change columns options of xaxis and split series selectors to be the source table columns when `aggregate values` is on
    - change xAxis and split series computed to hold colId instead of column id
    - change GristDoc saveViewSection routine to preserve old sections viewFields and options
  - Rename `Group data` into `split series`

quip doc: https://grist.quip.com/tAsCAuv8RiMa/Charts-data-aggregation#temp:C:QcK0ce13e1e8ae64048988f44f9c

Test Plan: Adds ChartAggregate.ts nbrowser test.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3336
2022-04-07 15:34:54 +02:00
Alex Hall
64369df4c3 (core) Add /attachments/updateUsed DocApi endpoint to soft delete all unused attachments in document
Summary:
Builds on https://phab.getgrist.com/D3352

Add DocStorage.scanAttachmentsForUsageChanges to do fancy JSON query to find all attachment metadata rows whose soft deletion status needs updating.

Add ActiveDoc.updateUsedAttachments which uses the above and then applies the appropriate user action if needed to soft delete/undelete metadata rows.

Add endpoint in DocApi calling ActiveDoc method.

Test Plan: Added DocApi test

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3357
2022-04-07 15:08:22 +02:00
Alex Hall
251d79704b (core) Migrate Attachments columns from marshalled blobs to JSON
Summary: Adds a migration in preparation for future work on tracking and deleting attachments. This includes a `_grist_Attachments.timeDeleted` column which isn't used yet, and changing the storage format of user columns of type `Attachments`. DocStorage now treats Attachments like RefList in general (since they use JSON), which also prompted a tiny bit of refactoring.

Test Plan: Added a migration test case showing the change in format.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3352
2022-04-06 13:28:47 +02:00
Paul Fitzpatrick
6c6bfee00e (core) fix redirects for multi-team Grist on a single domain
Summary:
The logic for calculating redirects wasn't quite right for Grist
configured to use a single domain, with teams encoded in the path.
This fixes it.

Test Plan: tested manually with docker compose and /etc/hosts

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3359
2022-04-05 17:27:37 -04:00
Alex Hall
bb5f3fc378 (core) Store monthly snapshots for 8 years to give Enterprise plans a more significant advantage
Summary:
Based on a discussion in https://grist.quip.com/ZvttAyjLCI7H#eLVADAbyipu

Without this change, the only difference between Enterprise and Pro plans regarding snapshots is 5 extra snapshots, one per year.

Test Plan: none

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3349
2022-04-05 18:11:13 +02:00
Paul Fitzpatrick
fea8f906d7 (core) add a login method based on headers
Summary:
This fleshes out header-based authentication a little more to
work with traefik-forward-auth.

Test Plan: manually tested

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3348
2022-04-04 18:36:09 -04:00
Paul Fitzpatrick
c6d66e15bf (core) configure typedoc for generating plugin api documentation
Summary:
This annotates the plugin api sufficiently to generate some documentation
for it. See https://github.com/gristlabs/grist-help/pull/139

Contains some small code tweaks for things that caused typedoc some
trouble.

Test Plan: manual inspection of output

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3342
2022-04-04 18:11:33 -04:00
George Gevoian
d8af25de9d (core) Add usage to data tables page
Summary:
Currently, usage is only shown for free team sites, and only
for total number of rows used in a document. Future diffs will
include other usage metrics and browser tests.

Test Plan: Planned for future diffs; UI is still under development.

Reviewers: jarek

Reviewed By: jarek

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3343
2022-04-04 15:08:19 -07:00
Alex Hall
bf271c822b (core) Copy column type and options when pasting into an empty column
Summary:
Adds a `data-grist-col-ref` attribute to the copied HTML, then uses that when pasting to look up the source column and retrieve info about it. Copies the info into the target column if:

- The document is the same (the docId hash matches)
- The source column still exists and has the same type as when copied
- The source type isn't Text, because in that case it's nice if type guessing still happens
- The target column is empty, meaning it has type Any (we check earlier that it's not a formula column)

The info copied is the type, widgetOptions, and reference column settings (visible and display columns) but not conditional formatting.

The changes are mostly in a function `parsePasteForView` which is based on `BaseView._parsePasteForView` but ported to TypeScript in a new file `BaseView2.ts`.

Added a useraction `MaybeCopyDisplayFormula` exposing an existing Python function `maybe_copy_display_formula` because the target column needs a slightly different display formula.

Test Plan: Added a new nbrowser test file and fixture doc.

Reviewers: cyprien

Reviewed By: cyprien

Subscribers: jarek, dsagal

Differential Revision: https://phab.getgrist.com/D3344
2022-04-04 14:53:16 +02:00
George Gevoian
6305811ca6 (core) Add new Grist login page
Summary:
Adds a new Grist login page to the login app, and replaces the
server-side Cognito Google Sign-In flow with Google's own OAuth flow.

Test Plan: Browser and server tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3332
2022-04-01 15:24:19 -07:00
Alex Hall
8fdfb02646 (core) Don't treat Attachments as RefList in creator panel
Summary: Fixing bug introduced in https://phab.getgrist.com/D3338: column config now shows "DATA FROM TABLE" under the Attachments column type as if it were a normal ref/reflist column.

Test Plan: manual

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3350
2022-04-01 22:10:22 +02:00
Dmitry S
d55bdbcdf3 (core) Reopen HelpScout beacon to the last-opened article
Summary:
- When opening HelpScout beacon to an article ("answers"), avoid a 'navigate'
  call to let the beacon show the previously open article.
- Work around a bug with reloading a page with a beacon article open: HelpScout
  renders the last state without triggering usual events.
- Report errors to server when beacon fails to load.
- reportWarning() method now reports the message to the server.

Test Plan: Added a test case

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3345
2022-03-31 14:54:55 -04:00
Jarosław Sadziński
499e24b744 (core) Adding conditional styles to old style widgets
Summary:
Widgets that were using old base Widget class didn't not
create conditional style rules ui.

Additional fixed a little bug - when adding conditional rule
the formula field was marked as having error for a split second.

Test Plan: new test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3346
2022-03-31 18:29:41 +02:00
Alex Hall
21b0ac3eff (core) Enforcing data size limit
Summary:
Track 'data size' in ActiveDoc alongside row count. Measure it at most once every 5 minutes after each change as before, or after every change when it becomes high enough to matter.

A document is now considered to be approaching/exceeding 'the data limit' if either the data size or the row count is approaching/exceeding its own limit.

Unrelated: tweaked teamFreeFeatures.snapshotWindow based on Quip comments

Test Plan: Tested manually that data size is now logged after every change once it gets high enough, but only if the row limit isn't also too high. Still too early for automated tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3341
2022-03-30 17:56:05 +02:00
Alex Hall
06956f84a5 (core) Make Attachments columns get treated like RefLists more
Summary:
Treat the column type 'Attachments' as equivalent to 'RefList:_grist_Attachments' in a few places, because that's essentially what it is. The main goal was to fix parsing strings representing attachments (reflists).

Also removed an unused function.

Test Plan: Tested manually that pasting a CSV/JSON string representation of an attachments reflists works now.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3338
2022-03-28 23:14:29 +02:00
Alex Hall
59436d2bca (core) Grace period and delete-only mode when exceeding row limit
Summary:
Builds upon https://phab.getgrist.com/D3328

- Add HomeDB column `Document.gracePeriodStart`
- When the row count moves above the limit, set it to the current date. When it moves below, set it to null.
- Add DataLimitStatus type indicating if the document is approaching the limit, is in a grace period, or is in delete only mode if the grace period started at least 14 days ago. Compute it in ActiveDoc and send it to client when opening.
- Only allow certain user actions when in delete-only mode.

Follow-up tasks related to this diff:

- When DataLimitStatus in the client is non-empty, show a banner to the appropriate users.
- Only send DataLimitStatus to users with the appropriate access. There's no risk landing this now since real users will only see null until free team sites are released.
- Update DataLimitStatus immediately in the client when it changes, e.g. when user actions are applied or the product is changed. Right now it's only sent when the document loads.
- Update row limit, grace period start, and data limit status in ActiveDoc when the product changes, i.e. the user upgrades/downgrades.
- Account for data size when computing data limit status, not just row counts.

See also the tasks mentioned in https://phab.getgrist.com/D3331

Test Plan: Extended FreeTeam nbrowser test, testing the 4 statuses.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3331
2022-03-25 13:41:33 +02:00
Paul Fitzpatrick
134ae99e9a (core) add gvisor-based sandboxing to core
Summary:
This adds support for gvisor sandboxing in core. When Grist is run outside of a container, regular gvisor can be used (if on linux), and will run in rootless mode. When Grist is run inside a container, docker's default policy is insufficient for running gvisor, so a fork of gvisor is used that has less defence-in-depth but can run without privileges.

Sandboxing is automatically turned on in the Grist core container. It is not turned on automatically when built from source, since it is operating-system dependent.

This diff may break a complex method of testing Grist with gvisor on macs that I may have been the only person using. If anyone complains I'll find time on a mac to fix it :)

This diff includes a small "easter egg" to force document loads, primarily intended for developer use.

Test Plan: existing tests pass; checked that core and saas docker builds function

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3333
2022-03-24 17:04:49 -04:00
Paul Fitzpatrick
de703343d0 (core) disentangle some server tests, release to core, add GRIST_PROXY_AUTH_HEADER test
Summary:
This shuffles some server tests to make them available in grist-core,
and adds a test for the `GRIST_PROXY_AUTH_HEADER` feature added in
https://github.com/gristlabs/grist-core/pull/165

It includes a fix for a header normalization issue for websocket connections.

Test Plan: added test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3326
2022-03-24 15:11:32 -04:00
Jarosław Sadziński
64c9717ac1 (core) Undo bug with summary table and raw data view
Summary:
Clicking undo/redo after converting a table to a summary table navigated
to the raw data view.

Test Plan: new test

Reviewers: georgegevoian, alexmojaki

Reviewed By: georgegevoian, alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3337
2022-03-24 20:03:33 +01:00
Alex Hall
546096fcc9 (core) Clean up and refactor uses of HomeDBManager.getDoc
Summary:
Firstly I just wanted some more consistency and less repetition in places where Documents are retrieved from the DB, so it's more obvious when code differs from the norm. Main changes for that part:

- Let HomeDBManager accept a `Request` directly and convert it to a `Scope`, and use this in a few places.
- `getScope` tries `req.docAuth.docId` if `req.params` doesn't have a docId.

I also refactored how `_createActiveDoc` gets the document URL, separating out getting the document from getting a URL for it. This is because I want to use that document object in a future diff, but I also just find it cleaner. Notable changes for that:

- Extracted a new method `HomeDBManager.getRawDocById` as an alternative to `getDoc` that's explicitly for when you only have a document ID.
- Removed the interface method `GristServer.getDocUrl` and its two implementations because it wasn't used elsewhere and it didn't really add anything on top of getting a doc (now done by `getRawDocById`) and `getResourceUrl`.
- Between `cachedDoc` and `getRawDocById` (which represent previously existing code paths) also try `getDoc(getScope(docSession.req))`, which is new, because it seems better to only `getRawDocById` as a last resort.

Test Plan: Existing tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3328
2022-03-24 13:42:36 +02:00
Jarosław Sadziński
b1c3943bf4 (core) Conditional formatting rules
Summary:
Adding conditional formatting rules feature.

Each column can have multiple styling rules which are applied in order
when evaluated to a truthy value.

- The creator panel has a new section: Cell Style
- New user action AddEmptyRule for adding an empty rule
- New columns in _grist_Table_columns and fields

A new color picker will be introduced in a follow-up diff (as it is also
used in choice/choice list/filters).

Design document:
https://grist.quip.com/FVzfAgoO5xOF/Conditional-Formatting-Implementation-Design

Test Plan: new tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3282
2022-03-23 13:15:02 +01:00
Jarosław Sadziński
96a34122a5 (core) Restoring cursor position on raw data views
Summary:
This diff introduces cursor features for raw data views:
- Restoring cursor position when the browser window is reloaded
- Restoring the last edit position when the browser window is reloaded

Test Plan: Added tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3314
2022-03-23 12:24:18 +01:00
Dmitry S
3b76b33423 (core) Fix bugs when both welcomeTour and docTour are available
Summary:
- Unify where in the code tours get initiated.
- Avoid start a new tour while one is being started or is in progress.
- Ignore welcome tour when on a doc that has a doc tour.
- Fix tours when starting with a special page like Access Rules.
- Remove mention of the no-longer-present "Give Feedback" button in the last
  message of the welcome tour.

Test Plan:
Add a browser test case that docTour preempts the welcome tour and shows no errors
(this test case fails in multiple ways without the changes).

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3330
2022-03-22 16:51:05 -04:00
Alex Hall
1452b6efc3 (core) Improve stacktraces from pyCall
Summary: Capture the stacktrace (via SandboxError) in `_pyCallWait` instead of `_onSandboxMsg` where it's always the same.

Test Plan:
Tested manually, found for example that the stacktrace in the logs changed from being rather useless:

```
at NSandbox._onSandboxMsg (/home/alex/work/grist/_build/core/app/server/lib/NSandbox.js:229:36)
at /home/alex/work/grist/_build/core/app/server/lib/NSandbox.js:179:18
at Unmarshaller.parse (/home/alex/work/grist/_build/core/app/common/marshal.js:289:21)
at NSandbox._onSandboxData (/home/alex/work/grist/_build/core/app/server/lib/NSandbox.js:174:28)
at Socket.<anonymous> (/home/alex/work/grist/_build/core/app/server/lib/NSandbox.js:63:59)
at Socket.emit (events.js:315:20)
at Socket.EventEmitter.emit (domain.js:467:12)
at addChunk (internal/streams/readable.js:309:12)
at readableAddChunk (internal/streams/readable.js:284:9)
at Socket.Readable.push (internal/streams/readable.js:223:10)
at Pipe.onStreamRead (internal/stream_base_commons.js:188:23)
```

to being somewhat more helpful:

```
at NSandbox._pyCallWait (/home/alex/work/grist/_build/core/app/server/lib/NSandbox.js:134:19)
at processTicksAndRejections (internal/process/task_queues.js:93:5)
at async ActiveDoc.applyActionsToDataEngine (/home/alex/work/grist/_build/core/app/server/lib/ActiveDoc.js:1080:39)
at async Sharing._applyActionsToDataEngine (/home/alex/work/grist/_build/core/app/server/lib/Sharing.js:325:37)
```

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3329
2022-03-22 17:00:02 +02:00
Alex Hall
2c9ae6dc94 (core) Enforce daily limit on API usage
Summary:
Keep track of the number of API requests made for this document today in redis. Uses local caches of the count and the document so that usually requests can proceed without waiting for redis or the database.

Moved the free standing function apiThrottle to become a method to avoid adding another layer of request handler callbacks.

Test Plan: Added a DocApi test

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3327
2022-03-22 00:22:45 +02:00
Cyprien P
b6f146d755 (core) Add options to switch chart orientation
Test Plan: Adds nbrowser tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3323
2022-03-21 11:28:44 +01:00
Alex Hall
ec8460b772 (core) Prune snapshots outside the window in product features
Summary:
- Add a method `getSnapshotWindow` to `IInventory` and `DocSnapshotInventory`. It returns a `SnapshotWindow`, which represents a duration of time for which we keep backups for a particular document.
- `DocSnapshotPruner` calls this method and passes the window to `shouldKeepSnapshots` to determine which document versions have fallen outside the window and should be pruned.
- The implementation passed to `DocSnapshotInventory` uses a new method `getDocProduct` in `HomeDBManager` which directly returns the `Product` associated with a document, given only the document ID. Other methods in `HomeDBManager` require passing more information, especially about a user, but `DocSnapshotPruner` only knows about document IDs.

Test Plan: Added a test for `getDocProduct` and a test for `DocSnapshotPruner` where `getSnapshotWindow` is specified.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3322
2022-03-18 18:48:14 +02:00
Cyprien P
21f1dfa56c (core) Add 'stacked' option to charts
Summary:
Adds nbrowser test

 - Also makes sort spec taken into account by Group Data options
 - This is a continuation of https://phab.getgrist.com/D3271
 - We still need to decide whether to add stack chart to area chart type

Test Plan: TBD

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3274
2022-03-18 10:59:12 +01:00
Paul Fitzpatrick
a5f5ecce19 (core) add grist.getTable(tableId) and a getTableId() method in plugin api
Summary:
Makes the new TableOperations API available for all tables
in the document. Adds methods for discovering the tableId of the
selected table. I was very tempted to implement the select() TODO
in the TableOperations API, but it requires a significant refactor
of the backend.

Test Plan: added test

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3325
2022-03-17 16:31:19 -04:00
Dmitry S
fa75f60bfd (core) Fix selection of rows after rows are dragged
Summary: After dragging rows up, selection was set incorrectly.

Test Plan: Expanded a browser test for dragging rows to check selection, which fails without this fix

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3324
2022-03-17 11:49:52 -04:00
George Gevoian
0f4f0d3dad (core) Migrate to SRP and add change password dialog
Summary:
Moves some auth-related UI components, like MFAConfig, out
of core, and adds a new ChangePasswordDialog component for
allowing direct password changes, replacing the old reset password
link to hosted Cognito.

Updates all MFA endpoints to use SRP for authentication.

Also refactors MFAConfig into smaller files, and polishes up some parts
of the UI to be more consistent with the login pages.

Test Plan: New server and deployment tests. Updated existing tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3311
2022-03-16 21:35:06 -07:00
Paul Fitzpatrick
7ba4dff18f (core) updates from grist-core 2022-03-15 13:40:22 -04:00
Paul Fitzpatrick
a641517bb1
Merge pull request #165 from MHOOO/reverse-proxy-auth-support
Reverse proxy auth support
2022-03-15 13:39:29 -04:00
Paul Fitzpatrick
98f64a8461 (core) add grist.selectedTable.create/update/destroy/upsert to custom widget api
Summary: This makes an equivalent of the /records REST endpoint available within custom widgets. For simple operations, it is compatible with https://github.com/airtable/airtable.js/. About half of the diff is refactoring code from DocApi that implements /records using applyUserActions, to make that code available in the plugin api.

Test Plan: added tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3320
2022-03-15 11:11:58 -04:00
Alex Hall
02e69fb685 (core) Crudely show row count and limit in UI
Summary:
Add rowCount returned from sandbox when applying user actions to ActionGroup which is broadcast to clients.

Add rowCount to ActiveDoc and update it after applying user actions.

Add rowCount to OpenLocalDocResult using ActiveDoc value, to show when a client opens a doc before any user actions happen.

Add rowCount observable to DocPageModel which is set when the doc is opened and when action groups are received.

Add crude UI (commented out) in Tool.ts showing the row count and the limit in AppModel.currentFeatures. The actual UI doesn't have a place to go yet.

Followup tasks:

- Real, pretty UI
- Counts per table
- Keep count(s) secret from users with limited access?
- Data size indicator?
- Banner when close to or above limit
- Measure row counts outside of sandbox to avoid spoofing with formula
- Handle changes to the limit when the plan is changed or extra rows are purchased

Test Plan: Tested UI manually, including with free team site, opening a fresh doc, opening an initialised doc, adding rows, undoing, and changes from another tab. Automated tests seem like they should wait for a proper UI.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3318
2022-03-14 21:49:32 +02:00
Thomas Karolski
ccdd551b4d style fixes 2022-03-14 17:51:10 +01:00
Paul Fitzpatrick
b2715ae9ef (core) forbid use of sqlite ATTACH except during VACUUM
Summary:
This calls sqlite3_limit(SQLITE_LIMIT_ATTACHED, 0) so that
if ever an `ATTACH` were snuck into an SQL query, it would be denied.
The limit needs to be waived when calling VACUUM since the implementation
of VACUUM uses ATTACH.

Test Plan: added test; existing tests should pass

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3316
2022-03-14 09:34:44 -04:00
George Gevoian
ad1b4f3cff (core) Record new user sign-ups
Summary:
Adds Google Tag Manager snippet to all login pages, and a new user
preference, recordSignUpEvent, that's set to true on first sign-in. The
client now checks for this preference, and if true, dynamically loads
Google Tag Manager to record a sign-up event. Afterwards, it removes
the preference.

Test Plan: Tested manually.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3319
2022-03-12 14:34:46 -08:00
Thomas Karolski
e5dc2d198f [comm] Use getRequestProfile from Authorizer 2022-03-12 21:00:52 +01:00
Thomas Karolski
9f3ed989c4 [authorizer] Determine auth header to use via an environment variable 2022-03-12 21:00:44 +01:00
Thomas Karolski
c459037b04 [authorizer] Move code for extracting auth header into a function 2022-03-12 21:00:36 +01:00
Jarosław Sadziński
eff78ae2e1 (core) New UI for raw data views
Summary:
Creating new UI for raw data views based on design.
- Renaming left for follow up diff
- Link in the menu is hidden for now
To access raw UI, use /p/data URL.

Test Plan: new tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3306
2022-03-12 13:51:48 +01:00
George Gevoian
f02174eb7e (core) Fix error when canceling import
Summary:
If cancel was clicked while a transform section was still being
generated in the Importer, an error was thrown. This refactors
the cancelImportFiles API action to take in the file upload id
in place of the entire DataSourceTransformed parameter, which
contains other values that are irrelevant to canceling. One of those
values, the transform section id, was causing the error to be thrown
since it was momentarily null.

Test Plan: Tested manually.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3317
2022-03-10 16:24:49 -08:00
Alex Hall
77a5d31afe (core) More accurate data size measurement
Summary: As suggested by @dsagal in https://phab.getgrist.com/D3277#inline-36801, change to query `SUM(pgsize - unused)` instead of `SUM(pgsize)` to measure actual data size more accurately. Technically this doesn't reflect the database file size as accurately, but it should reflect sandbox memory usage better, and more importantly it should allow users to see data size decreasing when they delete stuff.

Test Plan: Tested manually by adding rows to a doc and looking at the logs. The data size is smaller and changes more granularly.

Reviewers: dsagal, paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3313
2022-03-09 12:04:16 +02:00
Thomas Karolski
a584bc3a19 [Comm.js] Return a session profile based on the x-remote-user header if set 2022-03-09 10:00:03 +00:00
Jarosław Sadziński
351d70d4fb (core) Serving widget info page from home url
Summary:
Custom widget into page is served from a homeUrl instead
of untrusted URL, which might be not used in grist-core.

Test Plan: manual test

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3307
2022-03-09 10:34:50 +01:00
Jarosław Sadziński
d2b82b84c7 (core) Fixing bug with resuming search on a hidden column.
Summary: Fix for error that happens when a search is resumed after one of the columns was hidden.

Test Plan: Added test that shows the error.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3309
2022-03-09 10:34:17 +01:00
Paul Fitzpatrick
c4d3d7d3bb (core) be careful when reassigning a doc to a worker it was on before
Summary:
Importing a .grist document is implemented in a somewhat clunky way, in a multi-worker setup.

 * First a random worker receives the upload, and updates Grist's various stores appropriately (database, redis, s3).
 * Then a random worker is assigned to serve the document.

If the worker serving the document fails, there is a chance the it will end up assigned to the worker that handled its upload. Currently the worker will misbehave in this case. This diff:

 * Ports a multi-worker test from test/home to run in test/s3, and adds a test simulating a bad scenario seen in the wild.
 * Fixes persistence of any existing document checksum in redis when a worker is assigned.
 * Adds a check when assigned a document to serve, and finding that document already cached locally. It isn't safe to rely only on the document checksum in redis, since that may have expired.
 * Explicitly claims the document on the uploading worker, so this situation becomes even less likely to arise.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3305
2022-03-08 17:20:01 -05:00
Thomas Karolski
116295e42f Minor refactor & comments 2022-03-08 19:40:25 +00:00
Thomas Karolski
82a7f0a796 Implement support for webserver header based auth 2022-03-08 19:24:11 +00:00
Alex Hall
321019217d (core) Lossless imports
Summary:
- Removed string parsing and some type guessing code from parse_data.py. That logic is now implicitly done by ValueGuesser by leaving the initial column type as Any. parse_data.py mostly comes into play when importing files (e.g. Excel) containing values that already have types, i.e. numbers and dates.
- 0s and 1s are treated as numbers instead of booleans to keep imports lossless.
- Removed dateguess.py and test_dateguess.py.
- Changed what `guessDateFormat` does when multiple date formats work equally well for the given data, in order to be consistent with the old dateguess.py.
- Columns containing numbers are now always imported as Numeric, never Int.
- Removed `NullIfEmptyParser` because it was interfering with the new system. Its purpose was to avoid pointlessly changing a column from Any to Text when no actual data was inserted. A different solution to that problem was already added to `_ensure_column_accepts_data` in the data engine in a recent related diff.

Test Plan:
- Added 2 `nbrowser/Importer2` tests.
- Updated various existing tests.
- Extended testing of `guessDateFormat`. Added `guessDateFormats` to show how ambiguous dates are handled internally.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3302
2022-03-08 12:14:39 +02:00
George Gevoian
9522438967 (core) Add Grist forgot password page
Summary:
The page isn't yet linked to from anywhere in the UI, but
will be soon, once the new login page is ready. The page
can still be accessed at login-[s].getgrist.com/forgot-password,
and the flow is similar to the one used by Cognito's hosted UI.

Also refactors much of the existing login app code into smaller
files with less duplication, tweaks password validation to be closer
to Cognito's requirements, and polishes various parts of the UI,
like the verified page CSS, and the form inputs.

Test Plan: Browser, server and project tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3296
2022-03-07 09:11:28 -08:00
Cyprien P
76481d21e0 (core) Enable selecting x axis for the group data columns
Summary:
Until now, users could not pick the column that's currently serving as
the x axis.

To do that, chart needs to support having the x axis undefined.

This diff do just that:
  - allow x axis to be undefined
  - allow setting x axis from group data

Given that charts axis are stored as indexes of the section view
fields array, implementation required introduction of an extra chart
options: `isAxisUndefined`.

Test Plan: Updates existing test and adds new one.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3304
2022-03-07 10:49:18 +01:00
Paul Fitzpatrick
2563fb745a (core) make Grist easier to run with a single server
Summary:
This makes many small changes so that Grist is less fussy to run as a single instance behind a reverse proxy. Some users had difficulty with the self-connections Grist would make, due to internal network setup, and since these are unnecessary in any case in this scenario, they are now optimized away. Likewise some users had difficulties related to doc worker urls, which are now also optimized away. With these changes, users should be able to get a lot further on first try, at least far enough to open and edit documents.

The `GRIST_SINGLE_ORG` setting was proving a bit confusing, since it appeared to only work when set to `docs`. This diff
adds a check for whether the specified org exists, and if not, it creates it. This still depends on having a user email to make as the owner of the team, so there could be remaining difficulties there.

Test Plan: tested manually with nginx

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3299
2022-03-05 13:30:45 -05:00
Cyprien P
0da397ab90 (core) Fix js error when changing X Axis with chart with no series
Summary:
This error only occured for certain charts type. Some chart handled it correctly, others did not.

This brings a more general handling of this issue which should work for all charts.

Test Plan: New nbrowser test case.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3303
2022-03-04 14:06:19 +01:00
Alex Hall
5213972d24 (core) Guess numeric formatting options
Summary:
Change NumberParse.parse to return not just the parsed number but also information it gathered along the way about how the input string was formatted.

Use this in the new NumberParse.guessOptions to guess the actual widget options based on an array of strings.

Use NumberParse.guessOptions in TypeConversion (for when a user explicitly chooses to change type) and in ValueGuesser (for guesses about strings entered into empty columns).

Test Plan: Adds unit tests for NumberParse and ValueGuesser and updates the TypeChange2 nbrowser test.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3294
2022-03-03 21:32:03 +02:00
Jarosław Sadziński
70373550cf (core) Shortcuts help page can be scrolled using a keyboard
Summary: Restoring keyboard navigation on the shortcuts popup.

Test Plan: manual tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3295
2022-03-02 11:43:05 +01:00
George Gevoian
8f5ce9d610 (core) Wait for transform column before finalizing
Summary:
This adds a new private field to ColumnTransform that settles when
the transform column is done preparing. Finalize then waits for this
promise before proceeding.

A bug was occurring when the prepare promise was still pending,
but finalize was triggered indirectly, causing finalize to throw an error
due to the transform column not yet existing.

Test Plan: Tested manually.

Reviewers: jarek, dsagal

Reviewed By: jarek, dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3288
2022-03-01 13:59:14 -08:00
Alex Hall
599545fb11 (core) Fuller guessing of type and options when adding first data to blank columns
Summary:
Adds `common/ValueGuesser.ts` with logic for guessing column type and widget options (only for dates/datetimes) from an array of strings, and converting the strings to the guessed type in a lossless manner, so that converting back to Text gives the original values.

Changes `_ensure_column_accepts_data` in Python to call an exported JS method using the new logic where possible.

Test Plan: Added `test/common/ValueGuesser.ts` to unit test the core guessing logic and a DocApi end-to-end test for what happens to new columns.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3290
2022-03-01 22:00:45 +02:00
Jarosław Sadziński
ae6c857ac5 (core) Frozen columns reordering
Summary: Bug fix. Moving columns over the frozen set didn't work properly.

Test Plan: new tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3297
2022-02-28 22:52:37 +01:00
George Gevoian
fa68b790bb (core) Remove code for unused welcome flows
Summary: Removes code that was marked for removal.

Test Plan: Existing tests still pass.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3289
2022-02-28 13:21:28 -08:00
George Gevoian
83ba2957bf (core) Update failing HomeIntro core test
Summary:
Core doesn't redirect to Cognito or our own sign-up page
when clicking 'sign up' on the welcome screen. Instead, it
redirects to the test login page.

Test Plan: N/A (fixing test)

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3298
2022-02-28 12:23:28 -08:00
Cyprien P
c2070877b6 (core) Fix out of sync GROUP DATA and X Axis options when table change
Summary:
Fix this issue:
  When using “Change Widget” for a chart of a summary table, if I change group-by columns, the X-Axis dropdown becomes empty, and lists values from previous summary (not the actual summary table that’s now shown in the chart). Need to close and reopen creator panel to fix it.

Test Plan: Include nbrowser test

Reviewers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3284
2022-02-25 14:48:27 +01:00
Alex Hall
3445ecb64c (core) Search backwards with Shift+Enter
Summary: As simple as it sounds. Check ev.shiftKey in Enter key handler in search.ts.

Test Plan: Extended test/nbrowser/Search.ts

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3293
2022-02-25 14:07:15 +02:00
George Gevoian
ff4e5d2769 (core) Send emails when 2FA settings are updated
Summary: When user 2FA status is changed, we now send out emails via SendGrid.

Test Plan: Server tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3280
2022-02-24 12:36:50 -08:00
Paul Fitzpatrick
accd640078 (core) add a user.SessionID value for trigger formulas and granular access rules
Summary:
This makes a `user.SessionID` value available in information about the user, for use with trigger formulas and granular access rules. The ID should be constant within a browser session for anonymous user. For logged in users it simply reflects their user id.

This ID makes it possible to write access rules and trigger formulas that allow different anonymous users to create, view, and edit their own records in a document.

For example, you could have a brain-storming document for puns, and allow anyone to add to it (without logging in), letting people edit their own records, but not showing the records to others until they are approved by a moderator. Without something like this, we could only let anonymous people add one field of a record, and not have a secure way to let them edit that field or others in the same record.

Also adds a `user.IsLoggedIn` flag in passing.

Test Plan: Added a test, updated tests. The test added is a mini-moderation doc, don't use it for real because it allows users to edit their entries after a moderator has approved them.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3273
2022-02-22 12:50:43 -05:00
George Gevoian
95592b81bd (core) Skip /welcome/user page for new users
Summary:
Since the new Grist sign-up page has a required field for
name, we can now skip the welcome page asking for the
same thing. Code and tests that can be removed later are
marked with TODOs.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3266
2022-02-22 08:38:22 -08:00
George Gevoian
36843e632b (core) Hide API key when it's not selected
Summary:
The API key is now hidden by default. Clicking the input
will select and reveal the key. As soon as the key loses
selection, it is hidden again.

Test Plan: Project test.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3270
2022-02-21 17:26:45 -08:00
Alex Hall
437d30bd9f (core) Log number of rows in user tables in data engine
Summary:
Adds a method Table._num_rows using an empty lookup map column.

Adds a method Engine.count_rows which adds them all up.

Returns the count after applying user actions to be logged by ActiveDoc.

Test Plan: Added a unit test in Python. Tested log message manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3275
2022-02-22 00:59:56 +02:00
Alex Hall
f1002c0e67 (core) Regularly log data size in DocStorage.applyStoredActions using sqlite dbstat
Summary:
- Small cleanup: Make DocStorage implement OnDemandStorage, and remove unused execWithBackup
- Upgrade to new versions (.3) of @gristlabs/sqlite3 and connect-sqlite3 to use dbstat
- Add _logDataSize method which queries dbstat, adding up pgsize for tables loaded into the data engine
- Only complete _logDataSize every 5 minutes using new field _lastLoggedDataSize

Test Plan: Tested manually

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3277
2022-02-22 00:59:04 +02:00
Alex Hall
5b352211c4 (core) Guess date format during type conversion
Summary:
- Adds a dependency moment-guess (https://github.com/apoorv-mishra/moment-guess) to guess date formats from strings. However the npm package is missing source maps which leads to an ugly warning, so currently using a fork until https://github.com/apoorv-mishra/moment-guess/pull/22 is resolved.
- Adds guessDateFormat using moment-guess to determine the best candidate date format. The logic may be refined for e.g. lossless imports where the stakes are higher, but for now we're just trying to make type conversions smoother.
- Uses guessDateFormat to guess widget options when changing column type to date or datetime.
- Uses the date format of the original column when possible instead of guessing.
- Fixes a bug where choices were guessed based on the display column instead of the visible column, which made the guessed choices influenced by which values were referenced as well as completely broken when converting from reflist.
- @dsagal @georgegevoian This builds on https://phab.getgrist.com/D3265, currently unmerged. That diff was created first to alert to the change. Without it there would still be similar test failures/changes here as the date format would often be concretely guessed and saved as YYYY-MM-DD instead of being left as the default `undefined` which is shows as YYYY-MM-DD in the dropdown.

Test Plan: Added a unit test to `parseDate.ts`. Updated several browser tests which show the guessing in action during type conversion quite nicely.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal, georgegevoian

Differential Revision: https://phab.getgrist.com/D3264
2022-02-21 22:39:47 +02:00
Paul Fitzpatrick
be8c053922 (core) updates from grist-core 2022-02-21 13:44:13 -05:00
Lee Burton
9c47b9cdee
Fix typo in email fallback for SAML
It looks like nameId doesn't exist as a property but name_id does (as is used elsewhere in the function)
2022-02-20 02:39:30 -08:00
Edward Betts
d6e0e1fee3 Correct spelling mistakes 2022-02-19 09:46:49 +00:00
Alex Hall
8b5e837d9c (core) Use YYYY-MM-DD as default date format for parsing
Summary: By default, new date columns show "YYYY-MM-DD" as the date format in the UI, but under the hood it's stored as null/undefined, which affects parsing during pasting and type conversion. This changes the underlying parsing to always default to YYYY-MM-DD, as if that format was explicitly selected, making things more consistent.

Test Plan: Updated some tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3265
2022-02-18 12:16:42 +02:00
Cyprien P
afa90cc365 (core) Show default context menu on link
Summary:
also:
  - closes opened menu if any when click on a custom widget
  - closes opened menu if any when F2

Test Plan: Include test case

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3269
2022-02-18 10:09:36 +01:00
Dmitry S
9855a145d1 (core) Remove UI-imposed restriction against combining public edit access and access rules
Test Plan: TBD

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3272
2022-02-17 09:57:23 -05:00
Jarosław Sadziński
8d34b70899 (core) Exposing WidgetApi methods in a module scope
Summary:
Exposing WidgetApi methods (getOption, setOption, ...) in a module
scope to simplify custom widget code.

Test Plan: manual tests for now, as plugin is not yet on production.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3268
2022-02-16 16:49:29 +01:00
Cyprien P
2f6eafff35 (core) Adds setSelectedRows to the grist api for custom view
Summary:
This is needed to let custom widget driver filtering of other widget in the same page.

Descripion here:
 - https://grist.quip.com/ctytAQJoFMsM/Hopefully-Small-Projects#temp:C:NNCfe2030b27647439886ca83595

Test Plan: New api tested in a new nbrowser test

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3253
2022-02-15 15:24:15 +01:00
George Gevoian
e264094412 (core) Add account page option to allow Google login
Summary:
Enabled by default, the new checkbox is only visible to
users logged in with email/password, and controls whether it is possible
to log in to the same account via a Google account
(with matching email). When disabled, CognitoClient will refuse logins
from Google if a Grist account with the same email exists.

Test Plan:
Server and browser tests for setting flag. Manual tests to verify
Cognito doesn't allow signing in with Google when flag is disabled.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3257
2022-02-14 16:56:23 -08:00
George Gevoian
99f3422217 (core) Add new Grist sign-up page
Summary:
Available at login.getgrist.com/signup, the new sign-up page
includes similar options available on the hosted Cognito sign-up
page, such as support for registering with Google. All previous
redirects to Cognito for sign-up should now redirect to the new
Grist sign-up page.

Login is still handled with the hosted Cognito login page, and there
is a link to go there from the new sign-up page.

Test Plan: Browser, project and server tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3249
2022-02-14 10:32:47 -08:00
George Gevoian
6abe7d5827 (core) Use original column headers during imports
Summary:
When possible, the original column headers from imported
files will now be used as the labels for Grist columns. This includes
values that were previously invalid Grist column identifiers, such
as those containing Unicode.

Test Plan: Updated server and browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3261
2022-02-13 16:50:19 -08:00
Alex Hall
f877f3859d (core) Use visible column formatting when converting RefList to Text
Summary: Tweaked ReferenceListFormatter and ValueConverter to sensibly convert to Text. Fixes an embarrassing bug exposed during checkin 😱

Test Plan: Add test for converting from RefList displaying formatted dates to Text

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3254
2022-02-12 09:45:57 +02:00
Alex Hall
0de0cb0f4a (core) Add PUT /records DocApi endpoint to AddOrUpdate records
Summary:
As designed in https://grist.quip.com/fZSrAnJKgO5j/Add-or-Update-Records-API

Current `POST /records` adds records, and `PATCH /records` updates them by row ID. This adds `PUT /records` to 'upsert' records, applying the AddOrUpdate user action. PUT was chosen because it's idempotent. Using a separate method (instead of inferring based on the request body) also cleanly separates validation, documentation, etc.

The name `require` for the new property was suggested by Paul because `where` isn't very clear when adding records.

Test Plan: New DocApi tests

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3251
2022-02-12 09:44:34 +02:00