Summary:
Adds a CLI command to un-quarantine an active document. Also tweaks the
name of related environment variable to avoid a naming conflict.
Test Plan: Server test.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3583
Summary: Extend formula error messages with explanations from https://github.com/friendly-traceback/friendly-traceback. Only for Python 3.
Test Plan: Updated several Python tests. In general, these require separate branches for Python 2 and 3.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3542
A faulty conditional in _makeDefaultTransformRule was the cause of the
bug. The conditional isn't necessary, as it's unreachable from the
import flows, so it was removed.
Summary:
Adds a new dots loader, which is used on the Raw Data page when
certain values are still being calculated (e.g. row counts). Now metrics
whose values aren't known yet will still appear under Usage, but with a
"Loading" message and the dots loader shown. For per-table row counts,
we only show the loader.
Test Plan: Existing tests.
Reviewers: jarek
Reviewed By: jarek
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3566
Summary: Small tweaks for text that viewers see on empty site.
Test Plan: Updated tests.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3577
Summary:
Conditional formatting can now be used for whole rows.
Related fix:
- Font styles weren't applicable for summary columns.
- Checkbox and slider weren't using colors properly
Test Plan: Existing and new tests
Reviewers: paulfitz, georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3547
Summary:
Adds a new CLI command, doc, with a subcommand that quarantines
an active document. Adds a group query param to a housekeeping
endpoint for updating the document group prior to checking if a doc
needs to be reassigned. Both methods require support user credentials.
Test Plan: Server tests. (Additional testing will be done manually on staging.)
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3570
Summary:
Removing NEW_DEAL flag checks and exposing all new deal features as default.
Also removing Pages.ts as it was moved to grist-core.
Test Plan: Existing and updated tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3572
Summary:
On mobile, clicking the upgrade button will now immediately display
the plans modal. The button margins have also been adjusted to be
smaller on mobile. Finally, some disabled options related to workspace
sharing in the left panel (on personal sites) are now hidden instead.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3555
Summary: Displays a live row count of each table on the Raw Data page.
Test Plan: Browser tests.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3540
Summary: Adding new url parameter for team site creation
Test Plan: Updated tests.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3554
Summary:
Adding intro for a viewer on a teamsite.
Showing upgrade button for owners only.
Test Plan: new test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3557
Summary:
Adds a new option to the top of the three dot widget menu which navigates to the raw data widget at the current cell, equivalent to clicking an anchor link.
Clicking the back button returns to the previous view. Clicking the close button shows the list of raw data tables instead, which isn't great, but the same is true when clicking an anchor link.
Test Plan: Added a test to `nbrowser/ChartView1.ts` because charts are the main thing we want this for.
Reviewers: jarek
Reviewed By: jarek
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3539
Summary:
Occasionally, while the versions of a document are being enumerated,
a new version of the document will be created. This is detected and
triggers re-enumeration and a "surprise" log message. This diff
tweaks uploads to be run in series with DocSnapshots operations.
This means that listing versions would be blocked on an upload, or
vice versa, rather than overlapping. This is simpler and more deterministic.
I'm not sure how the user experience will feel if the operations
are slow.
Test Plan: existing tests pass; will see if surprises are reduced
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3551
Summary:
Small bug fix that ensures focus isn't lost when Escape is pressed
while renaming a choice. This now makes it possible to close the
editor with a subsequent press of Escape.
Test Plan: Browser test.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3556
Summary:
The changes in this diff are sufficient to make this sequence work again:
```
./build electron-dev
bin/electron app/electron/runPrebuild.js
```
This brings up the local server within an electron window.
This is an unambitious diff, aimed at checking how rusty electron support had become. It does not revive Grist as a packaged electron app. The first substantial work needed would be to make the app aware of the local file system again, and think through how local files should be visualized and accessed now. In the past, there was a simple list of grist docs in a directory.
Test Plan: manual
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3534
Summary:
This extends the getAccessToken documentation so it can be picked
up by typedoc and published, and makes a few other tweaks along
the way prompted by a typescript/typedoc version change.
Test Plan: made in concert with a grist-help update
Reviewers: jarek
Reviewed By: jarek
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3548
Summary:
Tweak PagePanels to let the left pane automatically expand on mouse
over. This is to make pages more accessible when the panel is
collapsed.
In this context, when expanding, the left panel overlap the main
content, reducing visual clutter.
Test Plan: updated
Reviewers: jarek
Reviewed By: jarek
Subscribers: anaisconce, jarek
Differential Revision: https://phab.getgrist.com/D3516
Summary:
Fills in the title and description/thumbnail (for templates) in app.html if the
page being requested is for a document.
Test Plan: Tested manually.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3544
Summary:
This calls a new `initialize` method on the sandbox before we start
doing calculations with it, to make sure that `random.seed()` has
been called. Otherwise, if the sandbox is cloned from a checkpoint,
the seed will have been reset.
The `initialize` method includes the functionality previously done
by `set_doc_url` since it is also initialization/personalization and
this way we avoid introducing another round trip to the sandbox.
Test Plan: tested with grist-core configured to use gvisor
Reviewers: georgegevoian, dsagal
Reviewed By: georgegevoian, dsagal
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3549
Summary:
FormulaEditor on sidepanel was sending update action when
formula wasn't updated by the user.
Test Plan: Existing tests
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3545
Summary:
Adds the new personal plan as a product that will be available
in the future. Can be enabled along with other plan-related via
an environment variable.
Test Plan: Browser tests and existing tests.
Reviewers: jarek
Reviewed By: jarek
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3533
Summary: [Select By] in the creator panel was bugged. It wasn't refreshed in some cases as the observable array that needed to be created seemed too complicated. This Diff recomputes this array when the user wants to change the selection.
Test Plan: added tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3541
Summary:
As reported in https://grist.slack.com/archives/C069RUP71/p1655316194602829, when a table is hidden by ACL, it was still showing as a blank 'ghost' option to select data when adding a widget:
{F55498}
The fix is simply to return `true` from `isHiddenTable` for empty table IDs, which indicate a table hidden by ACL. `TableRec.isHidden` is supposed to match this so I updated it too, and I cleaned up a tiny bit of other related code.
Test Plan: Extended `nbrowser/AccessRules1.ts` to test the data options when adding widgets.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3530
Summary: When traceback is present, give it 64px, or more if available, or less if less is needed. If less space is available than needed, the traceback will scroll within its allocated area.
Test Plan: The test FieldEditorSizing which tests basic sizing still passes; details with different size of formula and traceback were tested manually.
Reviewers: jarek
Reviewed By: jarek
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3531
Summary:
With this, a custom widget can render an attachment by doing:
```
const tokenInfo = await grist.docApi.getAccessToken({readOnly: true});
const img = document.getElementById('the_image');
const id = record.C[0]; // get an id of an attachment
const src = `${tokenInfo.baseUrl}/attachments/${id}/download?auth=${tokenInfo.token}`;
img.setAttribute('src', src)
```
The access token expires after a few mins, so if a user right-clicks on an image
to save it, they may get access denied unless they refresh the page. A little awkward,
but s3 pre-authorized links behave similarly and it generally isn't a deal-breaker.
Test Plan: added tests
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3488
Summary:
Modifies CSS so that text overflow is handled by displaying an
ellipsis when raw table names exceed the width of their container.
Test Plan: Tested manually.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3524
Summary:
A green line indicating the insertion point is now shown in the
ChoiceListEntry component when dragging and dropping choices, similar
to the one shown in the choice list cell editor.
Test Plan: Tested manually.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3529
Summary:
When adding a summary table widget to a page and using 'select by' in the add widget config (as opposed to in the right panel for an existing widget):
1. If an equivalent summary table already exists, use its referencing columns (if any) to construct link nodes. Previously the source table columns were being used instead, which could include referencing columns that don't have any equivalent in the summary table, and exclude referencing columns in the summary table.
2. If no such summary table exists yet, then keep using the source table columns, but only the selected groupby columns, and
3. After the summary table is created, correct the `linkTargetColRef` (which points to a source table column) to the corresponding column from the new summary table instead.
This fixes bugs which only appeared recently since 'select by' for a summary table previously involved no target columns.
Test Plan: Added two new tests to `nbrowser/SelectBySummaryRef`, and confirmed that they fail without the fixes to all three points above.
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3527
Summary:
Use table titles (i.e. the raw data widget titles) in dropdowns and other parts of the Acess Rules page, instead of the table ID. This is particularly meant for summary tables which have/had an ID of the form `GristSummary_SourceTable_N`, but https://phab.getgrist.com/D3508 is changing that anyway.
The server method `getAclResources` now returns more metadata about each table so that the UI can display titles.
Test Plan: Extended and updated `nbrowser/AccessRules2.ts`. Added a small unit test for constructing table titles from the new description returned by `getAclResources`.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3494
Summary:
Most logging now includes altSessionId, but not the message logged at the end
of every request by the 'morgan' logger. This includes altSessionId in those
messages.
Test Plan: Verified that with GRIST_HOSTED_VERSION env var set, altSessionId is included in morgan-produced JSON messages.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: georgegevoian
Differential Revision: https://phab.getgrist.com/D3523
Summary:
Improving billing page user experience.
- Updated labels for canceled plan
- Adding option to downgrade from team plan to free team plan
- Updating default name for teamFree plan when it is not available in Stripe
- Minor bug fixes
Test Plan: updated tests
Reviewers: cyprien
Reviewed By: cyprien
Subscribers: cyprien
Differential Revision: https://phab.getgrist.com/D3515
Summary:
A client error should no longer be thrown when filtering an empty Choice
or Choice List column.
Test Plan: Browser tests.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3528
Summary:
1. Log errors in `ActiveDoc.loadDoc` as errors, not just warnings, except for a common 'Cannot create fork' error caused by deployment tests.
2. Log the method name that had an error in `server/lib/Client.ts`.
Discussion: https://grist.slack.com/archives/CR8HZ4P9V/p1652364998893169
Following up on https://phab.getgrist.com/D3522
Test Plan: tested manually, particularly by running the nbrowser/Fork test that led to the initial noisy errors in Slack.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3525
Summary: The previous access check in `getFormulaError` was not strict enough, allowing users to read the values of individual formula cells that they shouldn't be able to. Now `getCellValue` is used to check the access for the specific cell first.
Test Plan: Extended GranularAccess server test.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3526
Summary:
This adds rudimentary support for opening certain SQLite files in Grist.
If you have a file such as `landing.db` in Grist, you can convert it to Grist format by doing (either in monorepo or grist-core):
```
yarn run cli -h
yarn run cli sqlite -h
yarn run cli sqlite gristify landing.db
```
The file is now openable by Grist. To actually do so with the regular Grist server, you'll need to either import it, or convert some doc you don't care about in the `samples/` directory to be a soft link to it (and then force a reload).
This implementation is a rudimentary experiment. Here are some awkwardnesses:
* Only tables that happen to have a column called `id`, and where the column happens to be an integer, can be opened directly with Grist as it is today. That could be generalized, but it looked more than a Gristathon's worth of work, so I instead used SQLite views.
* Grist will handle tables that start with an uncapitalized letter a bit erratically. You can successfully add columns, for example, but removing them will cause sadness - Grist will rename the table in a confused way.
* I didn't attempt to deal with column names with spaces etc (though views could deal with those).
* I haven't tried to do any fancy type mapping.
* Columns with constraints can make adding new rows impossible in Grist, since Grist requires that a row can be added with just a single cell set.
Test Plan: added small test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3502
Summary:
Changes auto-generated summary table IDs from e.g. `GristSummary_6_Table1` to `Table1_summary_A_B` (meaning `Table1` grouped by `A` and `B`). This makes it easier to write formulas involving summary tables, make API requests, understand logs, etc.
Because these don't encode the source table ID as reliably as before, `decode_summary_table_name` now uses the summary table schema info, not just the summary table ID. Specifically, it looks at the type of the `group` column, which is `RefList:<source table id>`.
Renaming a source table renames the summary table as before, and now renaming a groupby column renames the summary table as well.
Conflicting table names are resolved in the usual way by adding a number at the end, e.g. `Table1_summary_A_B2`. These summary tables are not automatically renamed when the disambiguation is no longer needed.
A new migration renames all summary tables to the new scheme, and updates formulas using summary tables with a simple regex.
Test Plan:
Updated many tests to use the new style of name.
Added new Python tests to for resolving conflicts when renaming source tables and groupby columns.
Added a test for the migration, including renames in formulas.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3508
Summary:
Cell selection would sometimes get out of sync, causing
unexpected results when pasting. The UI would also incorrectly
indicate that rows/columns were still selected if you clicked the
selected cell (outlined in green) after doing a drag selection of
multiple rows/columns. Finally, canceling a copy operation would
fail to remove the "scissors" outline around the copied cells if the
cursor was not on the copied selection.
This resolves all of these bugs.
Test Plan: Browser tests.
Reviewers: cyprien
Reviewed By: cyprien
Subscribers: cyprien
Differential Revision: https://phab.getgrist.com/D3517
Summary: Previously, changing the type of a column would clear its widget options and conditional style rules by default, with a few exceptions to explicitly keep them. This diff reverses that behaviour, keeping the options by default.
Test Plan: Updated several existing tests, plus lots of manual testing.
Reviewers: cyprien
Reviewed By: cyprien
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3491
Summary:
Sometimes when rearranging items in choice editor, user can
put the new item inside last entry element, which is not recognized as
a choice entry.
Test Plan: manual tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3519
Summary:
- Adding a column through 'Add Reference Column' adds it to Raw Data
- Migrating RefSelect.js to typescript
- Extending one of the tests
Test Plan: updated
Reviewers: cyprien
Reviewed By: cyprien
Subscribers: cyprien
Differential Revision: https://phab.getgrist.com/D3513
Summary:
When a user requests to read the contents of an attachment, only allow the request if there exists a cell in an attachment column that contains the attachment and which they have read access to.
This does not cover:
* Granular write access for attachments. In particular, a user who can write to any attachment column should be considered to have full read access to all attachment columns, currently.
* Access control of attachment metadata such as name and format.
The implementation uses a sql query that requires a scan, and some notes on how this could be optimized in future. The web client was updated to specify the cell to check for access, and performance seemed fine in casual testing on a doc with 1000s of attachments. I'm not sure how performance would hold up as the set of access rules grows as well.
Test Plan: added tests
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3490
Summary:
Summary tables now have their own raw viewsection, and are shown
under Raw Data Tables on the Raw Data page.
Test Plan: Browser and Python tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3495
Summary:
- test/nbrowser/CustomFilter keeps randomly fail on my local dev
- it failed with `Cannot read property of null (reading 'postMessage`)` at line below:
`this._rpc.setSendMessage(msg => this._iframe?.contentWindow!.postMessage(msg, '*'));`
- I understand it was trying to send message before even the iframe was properly mounted
- telling rpc to wait for the other end to send ready() successfully differ send message until everything's mounted.
Test Plan: - should not break anything and test/nbrowser/CustomFilter should stop failing
Reviewers: jarek
Reviewed By: jarek
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3510
Summary:
Building:
- Builds no longer wait for tsc for either client, server, or test targets. All use esbuild which is very fast.
- Build still runs tsc, but only to report errors. This may be turned off with `SKIP_TSC=1` env var.
- Grist-core continues to build using tsc.
- Esbuild requires ES6 module semantics. Typescript's esModuleInterop is turned
on, so that tsc accepts and enforces correct usage.
- Client-side code is watched and bundled by webpack as before (using esbuild-loader)
Code changes:
- Imports must now follow ES6 semantics: `import * as X from ...` produces a
module object; to import functions or class instances, use `import X from ...`.
- Everything is now built with isolatedModules flag. Some exports were updated for it.
Packages:
- Upgraded browserify dependency, and related packages (used for the distribution-building step).
- Building the distribution now uses esbuild's minification. babel-minify is no longer used.
Test Plan: Should have no behavior changes, existing tests should pass, and docker image should build too.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3506
Summary:
On Windows, last row is selected when a user clicks the 'new row'
while browser doesn't have focus.
Test Plan: manual tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: georgegevoian
Differential Revision: https://phab.getgrist.com/D3503
Summary:
- Update nudge boxes content and collapsing on personal and free team site
- New confirmation after upgrading from a free team site
- Refactoring ProductUpgrade code, splitting plans / modals and nudges
Test Plan: Manual and updated tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3481
Summary:
- Upgrades to build-related packages:
- Upgrade typescript, related libraries and typings.
- Upgrade webpack, eslint; add tsc-watch, node-dev, eslint_d.
- Build organization changes:
- Build webpack from original typescript, transpiling only; with errors still
reported by a background tsc watching process.
- Typescript-related changes:
- Reduce imports of AWS dependencies (very noticeable speedup)
- Avoid auto-loading global @types
- Client code is now built with isolatedModules flag (for safe transpilation)
- Use allowJs to avoid copying JS files manually.
- Linting changes
- Enhance Arcanist ESLintLinter to run before/after commands, and set up to use eslint_d
- Update eslint config, and include .eslintignore to avoid linting generated files.
- Include a bunch of eslint-prompted and eslint-generated fixes
- Add no-unused-expression rule to eslint, and fix a few warnings about it
- Other items:
- Refactor cssInput to avoid circular dependency
- Remove a bit of unused code, libraries, dependencies
Test Plan: No behavior changes, all existing tests pass. There are 30 tests fewer reported because `test_gpath.py` was removed (it's been unused for years)
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3498
Summary:
This diff is first of a series of 3 commits to enable range filering
for Date and DateTime columns. Diff only enable setting date's min/max
throw typing dates, Date picker and relative ranges are left for
follow-up commits.
- Exposes columns value formatter to the range input
- Fixes column filter func to work with dates
Test Plan:
Adds Date to projects range filter test
Adds Date/DateTime to nbrowser ColumnFilterMenu tests
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3455
Summary:
Adds a Python function `REQUEST` which makes an HTTP GET request. Behind the scenes it:
- Raises a special exception to stop trying to evaluate the current cell and just keep the existing value.
- Notes the request arguments which will be returned by `apply_user_actions`.
- Makes the actual request in NodeJS, which sends back the raw response data in a new action `RespondToRequests` which reevaluates the cell(s) that made the request.
- Wraps the response data in a class which mimics the `Response` class of the `requests` library.
In certain cases, this asynchronous flow doesn't work and the sandbox will instead synchronously call an exported JS method:
- When reevaluating a single cell to get a formula error, the request is made synchronously.
- When a formula makes multiple requests, the earlier responses are retrieved synchronously from files which store responses as long as needed to complete evaluating formulas. See https://grist.slack.com/archives/CL1LQ8AT0/p1653399747810139
Test Plan: Added Python and nbrowser tests.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: paulfitz, dsagal
Differential Revision: https://phab.getgrist.com/D3429
Summary:
After the welcome questions are dismissed, a video tour modal will
now be displayed. The video tour is also accessible via a tool button
in the left panel of the home page, as well as a text button next to
the Examples & Templates header.
Test Plan: Browser tests.
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3477
Summary:
- Substantial refactoring of the logic when the server fails to send some
messages to a client.
- Add seqId numbers to server messages to ensure reliable order.
- Add a needReload flag in clientConnect for a clear indication whent the
browser client needs to reload the app.
- Reproduce some potential failure scenarios in a test case (some of which
previously could have led to incorrectly ordered messages).
- Convert other Comm tests to typescript.
- Tweak logging of Comm and Client to be slightly more concise (in particular,
avoid logging sessionId)
Note that despite the big refactoring, this only addresses a fairly rare
situation, with websocket failures while server is trying to send to the
client. It includes no improvements for failures while the client is sending to
the server.
(I looked for an existing library that would take care of these issues. A relevant article I found is https://docs.microsoft.com/en-us/azure/azure-web-pubsub/howto-develop-reliable-clients, but it doesn't include a library for both ends, and is still in review. Other libraries with similar purposes did not inspire enough confidence.)
Test Plan: New test cases, which reproduce some previously problematic scenarios.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3470
Summary:
For self-hosted Grist, forward auth has proven useful, where
some proxy wrapped around Grist manages authentication, and
passes on user information to Grist in a trusted header.
The current implementation is adequate when Grist is the
only place where the user logs in or out, but is confusing
otherwise (see https://github.com/gristlabs/grist-core/issues/207).
Here we take some steps to broaden the scenarios Grist's
forward auth support can be used with:
* When a trusted header is present and is blank, treat
that as the user not being logged in, and don't look
any further for identity information. Specifically,
don't look in Grist's session information.
* Add a `GRIST_IGNORE_SESSION` flag to entirely prevent
Grist from picking up identity information from a cookie,
in order to avoid confusion between multiple login methods.
* Add tests for common scenarios.
Test Plan: added tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3482
Summary:
When an account is upgraded to a new product in Billing, send a message to the redis channel `billingAccount-${accountId}-product-changed`.
ActiveDocs subscribe to this channel. When a message is received, they refresh their product from the database and use it to recalculate doc usage based on new limits. The new usage is broadcast to clients so they see the result of the upgrade live.
Test Plan: Extended nbrowser Billing test to test that a document open in a separate tab has its limit banner cleared immediately on upgrade.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3480
Summary:
- Also converted sandboxUtil to typescript.
- The issue with %s manifested when a Python traceback contained "%s" in the
string; in that case the object with log metadata (e.g. docId) would
confusingly replace %s as if it were part of the message from Python.
Test Plan: Added a test case for the fix.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3486
Summary:
This blocks browser back/forward trackpad gestures from unintentionally
triggering when the cursor is over an overflown gridview. The gestures
are still allowed elsewhere in the UI, as well as in gridviews that have
not overflown.
Test Plan: Tested manually.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3474
Summary:
This fixes the grist-ee build after recent changes, by
giving it a stub for ProductUpgrades.ts. Extends the
`core` test to also check if `ext` variant builds, to
catch the most common form of breakage on grist-ee so
far (file organization).
Test Plan: extends test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3484
Summary: Use new Banner component for activation messages.
Test Plan: Existing tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3483
Summary:
Login and error pages now have their own unique page titles. This also fixes
the bug with the signed-out page having a page title of "Error".
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3479
Summary:
A formula returning an empty RecordSet in a RefList columns results in storing [] instead of null.
This caused a bug where the empty list was 'flattened' and the cell not appearing in filters at all.
This diff fixes the bug by filtering for the default value `null` instead for RefLists and the empty string for ChoiceLists.
I didn't manage to actually reproduce the bug for ChoiceLists, but this seemed the most sensible thing to do.
Test Plan: New nbrowser test.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3478
Summary:
When a table was selected by a summary table, renaming that table using
section widget resulted in a javascript error.
Test Plan: new test
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3475
Summary:
Adds some special handling to summary table and lookup logic:
- Source rows with empty choicelists/reflists get a corresponding summary row with an empty string/reference when grouping by that column, instead of excluding them from any group
- Adds a new `QueryOperation` 'empty' in the client which is used in `LinkingState`, `QuerySet`, and `recursiveMoveToCursorPos` to match empty lists in source tables against falsy values in linked summary tables.
- Adds a new parameter `match_empty` to the Python `CONTAINS` function so that regular formulas can implement the same behaviour as summary tables. See https://grist.slack.com/archives/C0234CPPXPA/p1654030490932119
- Uses the new `match_empty` argument in the formula generated for the `group` column when detaching a summary table.
Test Plan: Updated and extended Python and nbrowser tests of summary tables grouped by choicelists to test for new behaviour with empty lists.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3471
Summary:
- Showing nudge to individual users to sign up for free team plan.
- Implementing billing page to upgrade from free team to pro.
- New modal with upgrade options and free team site signup.
- Integrating Stripe-hosted UI for checkout and plan management.
Test Plan: updated tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3456
Summary:
Removes the relevant links in the intro text and buttons.
Note that the presence of actual "Examples & Templates" section of the page is controlled by whether any templates are available (returned by `getTemplates` api call)
Test Plan: Tested manually
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3473
Summary:
- Fixed an issue with manualSort values being very close floats. It is already handled by the data engine, but the client was being unnecessarily proactive and introduced a bug.
- The fix also helps with rearranging rows in filtered situations: they will now stay next to the row before which they were inserted.
- The fix accidentally improves (though doesn't fully fix) the issue where new columns show up in unexpected places in the raw-data column list.
- Fixed another rare bug with row order not getting updated correctly when positions update.
Test Plan: Added test cases for the improved behavior; fixed affected tests.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3462
Summary:
- Add app/common/CommTypes.ts to define types shared by client and server.
- Include @types/ws npm package
Test Plan: Intended to have no changes in behavior
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3467
Summary:
Following up on a small bug introduced in https://phab.getgrist.com/D3464. When a table has a column referencing the same table, then there can be two 'select by' options with the same label which is just the name of a summary table on the same page. The first option is simply filtering based on the summary table. The second option is linking the ref column in the source against the group column in the summary, but the name of the group column is hidden which leads to the ambiguity.
The solution in this diff is to always show the target node (source table) column name if the source node (summary table) column was the hidden group column. This also changes the label in the case where the reference to the source table isn't in the source table - see the updated test. This isn't strictly necessary in this case so I'm not 100% about the desired behaviour, but I don't think it hurts.
Test Plan: Tested disambiguation manually. Updated existing test.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3472
Summary:
The problem manifested while filtering by a column, closing the filter
would change the set of rows visible on the table. This would happen
only for rare table.
What caused that problem was that the filter being edited was wrongly
applyed also to another column, so depending on the content of the
column, it would entail unexpected behaviour.
The cause of that wrong association, was a mistakingly comparing the
id of two different type of thing: column and view field. The problem
would manifest if in the same section there were a view fields with
same row id as the column being filtered.
What made that confusion possible is the `.fieldOrColumn:
ViewFieldRec|ColumnRec` property of the FileInfo object, which could
hold either a view fields or a column record and was initialized with
view fields or columns if view fields was not found (ie: hidden
column).
Solution was to make sure FieldInfo is initialized with ColumnRec
alwasy (even for hidden column).
I'm not sure what is the reason why FilterInfo needed to support both
column record and view field record in the past, but it looks like
this is not needed anymore.
As a followup commit I think it would be worth the effort to refactor
FileInfo to accept only ColumnRec.
Test Plan: Includes new regression test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3463
Summary:
If an external store fails completely, Grist will continue to
retry uploading to it. This diff updated the HostedStorageManager
test to limit the extent of these retries to the test itself -
otherwise they continue for all other tests in the same process,
potentially disrupting those that read logs. There are other tests
that use s3, but they aren't run in the same process with delicate
log-reading tests, and it isn't quite as clear what improvement
to make there.
Test Plan:
artificially made external store fail, and checked that
test contamination seen previously no longer occurs.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3469
Summary:
Instead of always redirecting new users to the home page or the (teams) welcome page,
only redirect when the user signed in for the first time on a personal site, has access to
other sites, and isn't already being redirected to a specific page on their personal site.
Also tweaks how invalid Choice column values are displayed to match Choice List
columns, and fixes a small CSS issue with select by in the page widget picker when
there are options with long labels.
Test Plan: Browser tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3461
Summary:
Introduces a new message type, docUsage, that's broadcast to all connected
clients whenever document usage is updated in ActiveDoc.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3451
Summary:
During testing, and in some dev configurations, we make connections to
the Grist home database from multiple processes, and store it in Sqlite.
This is a situation prone to file locks in Sqlite. We set a busy_timeout
parameter to help, but transactions may still fail immediately. This
diff adds retries to our patch of TypeORM for Sqlite transactions
(which without the patch are already flakey for other reasons
https://github.com/typeorm/typeorm/issues/1884#issuecomment-380767213).
Without this change, I can tickly SQLITE_BUSY problems fairly
reliably with the DocApi tests. With this change, I can't so far.
This change should not affect deployments of our SaaS (which use
Postgres for home db) or grist-core/grist-ee (which access home
db from a single process in default configuration).
Test Plan: repeated running of DocApi tests
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3466
Summary:
Previously, absence of `GRIST_DOCS_S3_BUCKET` was equated with absence
of external storage, but that is no longer true now that Azure is
available. Azure could be used by setting `GRIST_DOCS_S3_BUCKET`
but the alternative `GRIST_AZURE_CONTAINER` flag is friendlier.
Test Plan:
confirmed manually that Azure can be configured and
used now without `GRIST_DOCS_S3_BUCKET`
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3448
Summary:
- Remove the empty-folder icon
- Add an "Invite team members" button for owners on empty team sites
- Add a "Browse Templates" button for all other cases on empty sites
- Update intro text for team, including a link to Sprouts
- Update intro text for personal/anon.
- Include a Free/Pro tag for team sites (for now, only "Free")
- Add a "Manage Team" button for owners on home page of all team sites.
- Polished the UI of UserManager: add a transition for the background, and
delay the appearance of the spinner for fast loads.
Test Plan: Fixed up the HomeIntro tests; added test case for Manage Team button
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3459
Summary:
Fixes a bug introduced in https://phab.getgrist.com/D3416 which exposed a new type of linking option that didn't actually work. Specifically it allowed selecting by a summary table when the target widget has a reference column to the source table of the summary. This diff correctly implements this linking by linking the reference column against the 'group' column of the summary table, so the source table data (which the client may not have access to) isn't involved. But the 'group' column name is hidden from the 'select by' option label to avoid confusion for users, so it just looks like another kind of summary table linking, and indeed it can be thought of purely in terms of matching groupby columns etc.
Discussion here: https://grist.slack.com/archives/C02EGJ1FUCV/p1654039063055499
Also fixes a related old bug that offered similar meaningless options involving summary tables, like selecting the summary table by the source table or linking summary tables with disjoint groupby columns.
Test Plan: Updated SelectBySummaryRef test and fixture doc for the new behaviour. Also updated a couple of older tests which were incorrectly asserting the buggy option to select a summary table by the source table.
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3464
Summary:
Shows the range filter next to the filter by values on filter menu. When users
set min and/or max, it takes precendence over the filter by values.
If users set:
- `[] < [max]` behaves as `less than max`.
- `[min] < []` behaves as `more than min`.
- `[min] < [max]` behaves as `between min and max`
- bounds are always inclusives.
- when users change min or max the values of the by values filter
gets checked/unchecked depending on whether they are included by
the range filter.
- when users clicks any btn/checkbox of the by values filter both min
and max input gets cleared, and the filter convert to a filter by
values.
Test Plan: Adds both projets and nbrowser tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3435
Summary:
grist-ee build was failing since it didn't have a
DocUsageBanner implementation available. Made the implementation
added to monorepo available, since it will be useful to improve
the activation banner.
Test Plan: manaul
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: georgegevoian
Differential Revision: https://phab.getgrist.com/D3452
Summary:
I missed committing a file that is important for editing files comfortably in the ext directory in an IDE. This diff:
* Adds tsconfig-base-ext.json - that was the only intended change
* Unrelated: Forces all creation of connections to the home db through a new `getOrCreateConnection` method which changes the `busy_timeout` if using Sqlite. This was an attempt to fix random "database is locked" test failures. I believe multiple connections to the home db as an sqlite file do not happen in self-hosted Grist (where there is a single node process) or in our SaaS (where the database is in postgres). It does affect Grist started using `devServerMain.ts` (where multiple processes accessing same database are started) or various test configurations when extra database connections are opened.
* Unrelated: I added a `busy_timeout` for session storage, when it uses Sqlite. Again, I don't believe this affects self-hosted Grist or our SaaS.
* Tweaked a `BillingDiscount` test that looked perhaps vulnerable to a stripe request stalling.
I can't be sure my tweaks actually help, since I didn't succeed in replicating the failures. Update: looks like the "locked" error can still happen :(
Test Plan: manual
Reviewers: jarek
Reviewed By: jarek
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3450
Summary:
Also fixes a minor CSS regression in UserManager where the
link to add a team member wasn't shown on a separate row.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3444
Summary:
Column filter menu use to mess up the ordering of the items for
numeric and dates values, and also for ref/reflist columns when the
visible column is a numeric a date column.
Solution was to:
- use the actual value of the visible column for comparison.
- use native comparison.
- tweak the native comparison to make blanks appears before valid value. Indeed, it came up several time that it's convenient to have invalid values show up first in the filter panel, it makes for a convenient way to detect them.
Test Plan: Adds new nbrowser test
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3441
Summary: Combines the code and behaviour of the existing endpoints `GET /records` (for the general shape of the result and the parameters for sort/filter/limit etc) and retrieving a specific attachment with `GET /attachments/:id` for handling fields specific to attachments.
Test Plan: Added a DocApi test. Also updated one test to use the new endpoint instead of raw `GET /tables/_grist_Attachments/records`.
Reviewers: cyprien
Reviewed By: cyprien
Subscribers: cyprien
Differential Revision: https://phab.getgrist.com/D3443
Summary:
Seem in a customer doc where Y-scale was wrong for unclear reasons. The cause
is that null-valued X labels cause the corresponding bar (or point) to be
omitted, but still affect the Y-scale. In this diff, such labels are replaced
with "-", so as to show up normally.
Trivial example that reproduces the problem here:
https://public.getgrist.com/iLPpx9C5i8nk/Null-in-X-Axis
Test Plan: Tested manually.
Reviewers: cyprien
Reviewed By: cyprien
Subscribers: cyprien
Differential Revision: https://phab.getgrist.com/D3442
Summary:
Previously, columns of type Any were created and modified one by one by reusing
the "empty column" logic from the data engine. This copies that logic to Node,
and sets the type of all columns together, to create them with the correct type
in the AddTable call.
This makes imports about twice faster (when slowness is due to many columns),
but doesn't address all cases where individual handling of columns causes slowness.
Test Plan: Added a test case for the new helper function.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3427
Summary:
Instead of showing a blank dialog for users whose access
is limited (e.g. public members), we now show the user's
role and a mention of whether their access is public.
Test Plan: Browser tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3431
Summary: Reduces the log level in a few places from error to warning.
Test Plan: N/A
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3437
Summary:
This allows limiting the memory available to documents in the sandbox when gvisor is used. If memory limit is exceeded, we offer to open doc in recovery mode. Recovery mode is tweaked to open docs with tables in "ondemand" mode, which will generally take less memory and allow for deleting rows.
The limit is on the size of the virtual address space available to the sandbox (`RLIMIT_AS`), which in practice appears to function as one would want, and is the only practical option. There is a documented `RLIMIT_RSS` limit to `specifies the limit (in bytes) of the process's resident set (the number of virtual pages resident in RAM)` but this is no longer enforced by the kernel (neither the host nor gvisor).
When the sandbox runs out of memory, there are many ways it can fail. This diff catches all the ones I saw, but there could be more.
Test Plan: added tests
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3398
Summary:
This makes it possible to configure a SendGrid-based Notifier
instance via a JSON configuration file.
Test Plan: Tested manually.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3432
Summary: For grist-ee, expect an activation key in environment variable `GRIST_ACTIVATION` or in a file pointed to by `GRIST_ACTIVATION_FILE`. In absence of key, start a 30-day trial, during which a banner is shown. Once trial expires, installation goes into document-read-only mode.
Test Plan: added a test
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3426
Summary:
The summary includes a count of documents that are approaching
limits, in grace period, or delete-only. The endpoint is only accessible
to site owners, and is currently unused. A follow-up diff will add usage
banners to the site home page, which will use the response from the
endpoint to communicate usage information to owners.
Test Plan: Browser and server tests.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3420
Summary:
Helps with cases such as https://grist.slack.com/archives/C02EGJ1FUCV/p1652196111066649?thread_ts=1651656433.171889&cid=C02EGJ1FUCV
When a user unsubscribes from a webhook, the secret URL is deleted from the database, but as long as the doc was open it would continue retrying pending requests still in the queue for a long time, using the locally cached value without noticing the effect of unsubscribing. This change allows unsubscribing to have an effect more quickly so that problematic events can be removed from the queue.
Test Plan: existing tests
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3430
Summary:
Grid UI improvements.
- Selecting a column with a custom background, didn't produce a visual difference.
- Hiding num-row right border when it is not needed (it should be used only for frozen columns)
Test Plan: manual tests
Reviewers: cyprien
Reviewed By: cyprien
Differential Revision: https://phab.getgrist.com/D3425
Summary:
Adds a new environment variable that allows for custom
CSS to be included in all core static pages.
Test Plan: Tested manually in grist-core.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3419
Summary:
Currently, we have two ways that we deliver Grist. One is grist-core,
which has simple defaults and is relatively easy for third parties to
deploy. The second is our internal build for our SaaS, which is the
opposite. For self-managed Grist, a planned paid on-premise version
of Grist, I adopt the following approach:
* Use the `grist-core` build mechanism, extending it to accept an
overlay of extra code if present.
* Extra code is supplied in a self-contained `ext` directory, with
an `ext/app` directory that is of same structure as core `app`
and `stubs/app`.
* The `ext` directory also contains information about extra
node dependencies needed beyond that of `grist-core`.
* The `ext` directory is contained within our monorepo rather than
`grist-core` since it may contain material not under the Apache
license.
Docker builds are achieved in our monorepo by using the `--build-context`
functionality to add in `ext` during the regular `grist-core` build:
```
docker buildx build --load -t gristlabs/grist-ee --build-context=ext=../ext .
```
Incremental builds in our monorepo are achieved with the `build_core.sh` helper,
like:
```
buildtools/build_core.sh /tmp/self-managed
cd /tmp/self-managed
yarn start
```
The initial `ext` directory contains material for snapshotting to S3.
If you build the docker image as above, and have S3 access, you can
do something like:
```
docker run -p 8484:8484 --env GRIST_SESSION_SECRET=a-secret \
--env GRIST_DOCS_S3_BUCKET=grist-docs-test \
--env GRIST_DOCS_S3_PREFIX=self-managed \
-v $HOME/.aws:/root/.aws -it gristlabs/grist-ee
```
This will start a version of Grist that is like `grist-core` but with
S3 snapshots enabled. To release this code to `grist-core`, it would
just need to move from `ext/app` to `app` within core.
I tried a lot of ways of organizing self-managed Grist, and this was
what made me happiest. There are a lot of trade-offs, but here is what
I was looking for:
* Only OSS-code in grist-core. Adding mixed-license material there
feels unfair to people already working with the repo. That said,
a possible future is to move away from our private monorepo to
a public mixed-licence repo, which could have the same relationship
with grist-core as the monorepo has.
* Minimal differences between self-managed builds and one of our
existing builds, ideally hewing as close to grist-core as possible
for ease of documentation, debugging, and maintenance.
* Ideally, docker builds without copying files around (the new
`--build-context` functionality made that possible).
* Compatibility with monorepo build.
Expressing dependencies of the extra code in `ext` proved tricky to
do in a clean way. Yarn/npm fought me every step of the way - everything
related to optional dependencies was unsatisfactory in some respect.
Yarn2 is flexible but smells like it might be overreach. In the end,
organizing to install non-core dependencies one directory up from the
main build was a good simple trick that saved my bacon.
This diff gets us to the point of building `grist-ee` images conveniently,
but there isn't a public repo people can go look at to see its source. This
could be generated by taking `grist-core`, adding the `ext` directory
to it, and pushing to a distinct repository. I'm not in a hurry to do that,
since a PR to that repo would be hard to sync with our monorepo and
`grist-core`. Also, we don't have any licensing text ready for the `ext`
directory. So leaving that for future work.
Test Plan: manual
Reviewers: georgegevoian, alexmojaki
Reviewed By: georgegevoian, alexmojaki
Differential Revision: https://phab.getgrist.com/D3415
Summary:
Relax the restriction in `selectBy.isValidLink` so that summary tables can be linked by a column like other tables, except the `group` column. See the discussion on https://grist.slack.com/archives/C0234CPPXPA/p1651773623256959 (the replies are on the following message) for more info on this decision.
Tweaked `LinkingState.ts` since linking with summary tables can now involve a column.
Test Plan: Added a new nbrowser test and fixture checking the options to select by given a summary table with a few ref/reflist columns. Manually tested the behaviour of each option.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3416
Summary:
Use openpyxl instead of messytables (which used xlrd internally) in import_xls.py.
Skip empty rows since excel files can easily contain huge numbers of them.
Drop support for xls files (which openpyxl doesn't support) in favour of the newer xlsx format.
Fix some details relating to python virtualenvs and dependencies, as Jenkins was failing to find new Python dependencies.
Test Plan: Mostly relying on existing tests. Updated various tests which referred to xls files instead of xlsx. Added a Python test for skipping empty rows.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3406
Summary:
Diff fixes couple edge cases:
- When changing chart's groupby columns, the data-engine changes the view fields, which is not okay with charts, hence makes sure view fields stay the same as much as possible using `this.setSectionViewFieldsFromArray()`
- Also there's a logic in `this.setSectionViewFieldsFromArray()` that handle what to do when some columns goes missing during updates. Diff fixes this logic two. This is really corner case.
Test Plan: Both cases are not added to ChartAggregate.ts
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3405
Summary:
Adds a new implementation of the interface ExternalStorage that works for Azure Blob Storage as an alternative to S3, for a specific self-hosting case.
Tweaks HostedStorageManager and ICreate to allow configuring different core implementations of ExternalStorage.
Followup tasks:
- Make this code available to self hosters, possibly by making it open source.
- Add an env var or other config option to specify the preferred type of storage. Currently using the var `AZURE_STORAGE_CONNECTION_STRING` to know how to connect to Azure when requested, but that choice still only lives in test code.
Test Plan: Generalized HostedStorageManager and ExternalStorage tests to test the new AzureExternalStorage alongside S3ExternalStorage. The HostedStorageManager tests also now test the 'cached' in-memory test storage in a way that's closer to the real storage methods.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3413
Summary:
Destroy function in TableOperations was throwing error when invoked with a single
record id instead of an array. Now it returns a void type.
Also changing mapColumns function signature as it doesn't require options for a default
behavior.
Test Plan: Updated tests.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3404
Summary:
Adds attachment and data size to the usage section of
the raw data page. Also makes in-document usage banners
update as user actions are applied, causing them to be
hidden/shown or updated based on the current state of
the document.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3395
Summary:
- Better focus on the widget title
- Adding columns only to the current view section
- New popup with options when user wants to delete a page
- New dialog to enter table name
- New table as a widget doesn't create a separate page
- Removing a table doesn't remove the primary view
Test Plan: Updated and new tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3410
Summary:
Skipping columns during incremental imports wasn't working for certain
column types, such as numeric columns. The column's default value was
being used instead (e.g. 0), overwriting values in the destination
table.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3402
Summary: Adds a special user action `UpdateCurrentTime` which invalidates an internal engine dependency node that doesn't belong to any table but is 'used' by the `NOW()` function. Applies the action automatically every hour.
Test Plan: Added a Python test for the user action. Tested the interval periodically applying the action manually: {F43312}
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3389
Summary: Allow exceeding the daily API usage limit for a doc based on additional allocations for the current hour and minute. See the doc comment on getDocApiUsageKeysToIncr for details. This means that up to 5 redis keys may be relevant at a time for a single document.
Test Plan: Updated and expanded 'Daily API Limit' tests.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3368
Summary:
Currently, Grist behind a reverse proxy will generate many
needless redirects via `http`, and can't be used with only
port 443. This diff centralizes generation of these redirects
and uses the protocol in APP_HOME_URL if it is set.
Test Plan:
manually tested by rebuilding grist-core and
doing a reverse proxy deployment that had no support for
port 80. Prior to this change, there are lots of problems;
after, the site works as expected.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3400
Summary:
A new way for renaming tables.
- There is a new popup to rename section (where you can also rename the table)
- Renaming/Deleting page doesn't modify/delete the table.
- Renaming table can rename a page if the names match (and the page contains a section with that table).
- User can rename table in Raw Data UI in two ways - either on the listing or by using the section name popup
- As before, there is no way to change tableId - it is derived from a table name.
- When the section name is empty the table name is shown instead.
- White space for section name is allowed (to discuss) - so the user can just paste ' '.
- Empty name for a page is not allowed (but white space is).
- Some bugs related to deleting tables with attached summary tables (and with undoing this operation) were fixed (but not all of them yet).
Test Plan: Updated tests.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: georgegevoian
Differential Revision: https://phab.getgrist.com/D3360
Summary:
Changing keyboard shortcut for Duplicate row from
Ctrl+Shift+D to Mod+Shift+D
Test Plan: existing test
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3401
Summary:
Summary columns now have their own conditional rules,
which are not shared with sister columns.
Test Plan: New test
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3388
Summary:
- Previously showed "UnboundLocalError". Now will show:
Import failed: Failed to parse Excel file.
Error: No tables found (1 empty tables skipped)
- Also fix logging for import code
Test Plan: Added a test case
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3396
Summary:
Fixing bug when comparing documents. User wasn't able
to open cell/column context menu due to javascript error.
Test Plan: Updated tests
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3397
Summary: InitNewDoc is essentially only used to generate initialDocSql, so it doesn't make sense to set the timezone and locale. They are always set when actually creating a new doc anyway. Discussed in https://grist.slack.com/archives/C0234CPPXPA/p1650312714217089.
Test Plan: this
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3394
Summary:
This also enables the new Usage section for all sites. Currently,
it shows metrics for document row count, but only if the user
has full document read access. Otherwise, a message about
insufficient access is shown.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3377
Summary:
Call ActiveDoc.removeUnusedAttachments every hour using setInterval, and in ActiveDoc.shutdown (which also clears said interval).
Unrelated: small fix to my webhooks code which was creating a redis client on shutdown just to quit it.
Test Plan:
Tweaked DocApi test to remove expired attachments by force-reloading the doc, so that it removes them during shutdown. Extracted a new testing endpoint /verifyFiles to support this test (previously running that code only happened with `/removeUnused?verifyfiles=1`).
Tested the setInterval part manually.
Reviewers: paulfitz, dsagal
Reviewed By: paulfitz
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3387
Summary: Mark actions adding attachment metadata as 'internal' (not part of undo stack) which previously was only for the Calculate action.
Test Plan: Extended nbrowser attachments test
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3380
Summary:
When grid view was not fully loaded the initial blank row
was marked as invalid by conditional styles. This removes the styling
when row is not fully loaded.
Test Plan: Tested manually
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3392
Summary:
New command "Duplicate rows" is available in the Row/Card Context Menu and as a keyboard shortcut Ctrl+Alt+C.
- All selected rows are duplicated (even if only a single column is selected)
- Rows are inserted immediately after the last selected row (using manualSort value).
- Formulas and CENSORED fields are not copied.
Implemented on the UI level (no new action).
Test Plan: new test
Reviewers: cyprien
Reviewed By: cyprien
Differential Revision: https://phab.getgrist.com/D3371
Summary: Adds a command and `BaseView` method `filterByThisCellValue`.
Test Plan: Added two tests to `nbrowser/CellContextMenu.ts`
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3383
Summary:
This avoids an extra database query to look up the user's current
name, by capturing it at the moment their user id is queried.
Test Plan: existing test for user.Name changes continues to pass
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3381
Summary:
- Add a new parameter `Features.baseMaxAttachmentsBytesPerDocument` and set it to 1GB for the free team product.
- Add a method to DocStorage to calculate the total size of existing and used attachments.
- Add a migration to DocStorage adding an index to make the query in the above method fast.
- Check in ActiveDoc if uploading attachment(s) would exceed the product limit on that document.
Test Plan: Added test in `limits.ts` testing enforcement of the attachment limit.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3374
Summary: The name of a user for actions made using a websocket until now could be inconsistent with that seen by other means. This draws the name from the database, rather than from session information that may have been cached from an identity provider.
Test Plan: added test
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3379
Summary:
Search now works on Raw Data Page.
- Search bar option 'Search on all pages' will change to 'Search on all tables' when on the Raw data page, and will allow searching through all tables.
- Little CSS adjustment for an overlay on Raw page (removes z-index as it is not needed, and conflicts with searchbar).
- Search bar option ('search on all') gets white background, little padding, and is moved 2 pixels up, this is needed for Raw page.
Test Plan: new and updated tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3376
Summary: Also cleans up login URLs by excluding params and hashes.
Test Plan: Client and server tests.
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3378
Summary:
For users who cannot otherwise change access to a resource, let
them remove themselves. Implemented via the standard endpoints
as a special exception that will process a request from a user
that would otherwise be denied, if the only contents of that
request are a removal of themselves.
Users who can change access are still not permitted to change their
own permissions or to remove themselves, as a precaution against
orphaning resources.
Test Plan: extended and updated tests
Reviewers: cyprien
Reviewed By: cyprien
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3367
Summary:
- Symptoms where that Split Series could end up being turned off for
no good reason. Also both x axis and split series could be mixed
up.
- Problems was caused by call to `setGroupByColumns` which modifies
the sections viewFields. Diff fixes it by adjustin slightly the
ordering of function call in `_setXAxis()`.
- Problem of mixing up x axis and split series was fixed by being
careful on the order of columns passed to the `setGroupByColumns`
which then determine the ordering of the view fields.
Test Plan: Includes new test cases
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3365
Summary: Adds methods to delete metadata rows based on timeDeleted. The flag expiredOnly determines if it only deletes attachments that were soft-deleted 7 days ago, or just all soft-deleted rows. Then any actual file data that doesn't have matching metadata is deleted.
Test Plan: DocApi test
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3364
Summary:
New onNewRecord event in custom widgets that is triggered
when the user selects a blank row.
Together with this diff there is PR for grist-help https://github.com/gristlabs/grist-help/pull/150
Test Plan: added test case
Reviewers: cyprien
Reviewed By: cyprien
Differential Revision: https://phab.getgrist.com/D3375
Summary:
- Include docId when available for client-side error reporting
- Distinguish sandbox crashes from forced exits
Test Plan: Tested manually
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3373
Summary:
This also updates Authorizer to link the authSubject
to Grist users if not previously linked. Linked subjects
are now used as the username for password-based logins,
instead of emails, which remain as a fallback.
Test Plan: Existing tests, and tested login flows manually.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3356
Summary:
This adds an optional `isConsultant` flag to `users.options`, and an endpoint that allows the support user to turn it on or off. Users marked as consultants are not counted as billable members. Follows the example of existing `allowGoogleLogin` option.
Billable members are counted when members are added or removed from a site. Changing the `isConsultant` flag has no immediate or retroactive effect on billing. The number of users in stripe is now set unconditionally, rather than only when it has changed.
Notifications to billing managers are not aware of this billing nuance, but continue to report user counts that include consultants. The notifications link users to the billing page.
Test Plan: extended test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: anaisconce, jarek
Differential Revision: https://phab.getgrist.com/D3362
Summary:
When linking table to a summary, the linking ended up broken after updating the summary group by columns.
This diff fixes that issue.
There were two issues:
1) some subscriptions were missing due to some .peek() calls instead of directly calling the observable.
2) the LinkingState instance was not being disposed.
3) the filterColValues was not updating after source data table has been loaded
Test Plan: Include new test file.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3358
Summary:
Redesigning color picker:
- Single color palette (no light/dark switch)
- Ability to remove color (new empty button)
New font options in the color picker.
Font options are available on:
- Default cell style
- Conditional rules styles
- Choice/ChoiceList editor and token field
- Filters for Choice/ChoiceList columns
Design document:
https://www.figma.com/file/bRTsb47VIOVBfJPj0qF3C9/Grist-Updates?node-id=415%3A8135
Test Plan: new and updated tests
Reviewers: georgegevoian, alexmojaki
Reviewed By: georgegevoian, alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3335
Summary:
- adds the `aggregate values` option to chart config
- aggregation is performed by turning table into summary table
- change columns options of xaxis and split series selectors to be the source table columns when `aggregate values` is on
- change xAxis and split series computed to hold colId instead of column id
- change GristDoc saveViewSection routine to preserve old sections viewFields and options
- Rename `Group data` into `split series`
quip doc: https://grist.quip.com/tAsCAuv8RiMa/Charts-data-aggregation#temp:C:QcK0ce13e1e8ae64048988f44f9c
Test Plan: Adds ChartAggregate.ts nbrowser test.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3336
Summary:
Builds on https://phab.getgrist.com/D3352
Add DocStorage.scanAttachmentsForUsageChanges to do fancy JSON query to find all attachment metadata rows whose soft deletion status needs updating.
Add ActiveDoc.updateUsedAttachments which uses the above and then applies the appropriate user action if needed to soft delete/undelete metadata rows.
Add endpoint in DocApi calling ActiveDoc method.
Test Plan: Added DocApi test
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3357
Summary: Adds a migration in preparation for future work on tracking and deleting attachments. This includes a `_grist_Attachments.timeDeleted` column which isn't used yet, and changing the storage format of user columns of type `Attachments`. DocStorage now treats Attachments like RefList in general (since they use JSON), which also prompted a tiny bit of refactoring.
Test Plan: Added a migration test case showing the change in format.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3352
Summary:
The logic for calculating redirects wasn't quite right for Grist
configured to use a single domain, with teams encoded in the path.
This fixes it.
Test Plan: tested manually with docker compose and /etc/hosts
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3359
Summary:
Based on a discussion in https://grist.quip.com/ZvttAyjLCI7H#eLVADAbyipu
Without this change, the only difference between Enterprise and Pro plans regarding snapshots is 5 extra snapshots, one per year.
Test Plan: none
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3349
Summary:
This fleshes out header-based authentication a little more to
work with traefik-forward-auth.
Test Plan: manually tested
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3348
Summary:
This annotates the plugin api sufficiently to generate some documentation
for it. See https://github.com/gristlabs/grist-help/pull/139
Contains some small code tweaks for things that caused typedoc some
trouble.
Test Plan: manual inspection of output
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3342
Summary:
Currently, usage is only shown for free team sites, and only
for total number of rows used in a document. Future diffs will
include other usage metrics and browser tests.
Test Plan: Planned for future diffs; UI is still under development.
Reviewers: jarek
Reviewed By: jarek
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3343
Summary:
Adds a `data-grist-col-ref` attribute to the copied HTML, then uses that when pasting to look up the source column and retrieve info about it. Copies the info into the target column if:
- The document is the same (the docId hash matches)
- The source column still exists and has the same type as when copied
- The source type isn't Text, because in that case it's nice if type guessing still happens
- The target column is empty, meaning it has type Any (we check earlier that it's not a formula column)
The info copied is the type, widgetOptions, and reference column settings (visible and display columns) but not conditional formatting.
The changes are mostly in a function `parsePasteForView` which is based on `BaseView._parsePasteForView` but ported to TypeScript in a new file `BaseView2.ts`.
Added a useraction `MaybeCopyDisplayFormula` exposing an existing Python function `maybe_copy_display_formula` because the target column needs a slightly different display formula.
Test Plan: Added a new nbrowser test file and fixture doc.
Reviewers: cyprien
Reviewed By: cyprien
Subscribers: jarek, dsagal
Differential Revision: https://phab.getgrist.com/D3344
Summary:
Adds a new Grist login page to the login app, and replaces the
server-side Cognito Google Sign-In flow with Google's own OAuth flow.
Test Plan: Browser and server tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3332
Summary: Fixing bug introduced in https://phab.getgrist.com/D3338: column config now shows "DATA FROM TABLE" under the Attachments column type as if it were a normal ref/reflist column.
Test Plan: manual
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3350
Summary:
- When opening HelpScout beacon to an article ("answers"), avoid a 'navigate'
call to let the beacon show the previously open article.
- Work around a bug with reloading a page with a beacon article open: HelpScout
renders the last state without triggering usual events.
- Report errors to server when beacon fails to load.
- reportWarning() method now reports the message to the server.
Test Plan: Added a test case
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3345
Summary:
Widgets that were using old base Widget class didn't not
create conditional style rules ui.
Additional fixed a little bug - when adding conditional rule
the formula field was marked as having error for a split second.
Test Plan: new test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3346
Summary:
Track 'data size' in ActiveDoc alongside row count. Measure it at most once every 5 minutes after each change as before, or after every change when it becomes high enough to matter.
A document is now considered to be approaching/exceeding 'the data limit' if either the data size or the row count is approaching/exceeding its own limit.
Unrelated: tweaked teamFreeFeatures.snapshotWindow based on Quip comments
Test Plan: Tested manually that data size is now logged after every change once it gets high enough, but only if the row limit isn't also too high. Still too early for automated tests.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3341
Summary:
Treat the column type 'Attachments' as equivalent to 'RefList:_grist_Attachments' in a few places, because that's essentially what it is. The main goal was to fix parsing strings representing attachments (reflists).
Also removed an unused function.
Test Plan: Tested manually that pasting a CSV/JSON string representation of an attachments reflists works now.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3338
Summary:
Builds upon https://phab.getgrist.com/D3328
- Add HomeDB column `Document.gracePeriodStart`
- When the row count moves above the limit, set it to the current date. When it moves below, set it to null.
- Add DataLimitStatus type indicating if the document is approaching the limit, is in a grace period, or is in delete only mode if the grace period started at least 14 days ago. Compute it in ActiveDoc and send it to client when opening.
- Only allow certain user actions when in delete-only mode.
Follow-up tasks related to this diff:
- When DataLimitStatus in the client is non-empty, show a banner to the appropriate users.
- Only send DataLimitStatus to users with the appropriate access. There's no risk landing this now since real users will only see null until free team sites are released.
- Update DataLimitStatus immediately in the client when it changes, e.g. when user actions are applied or the product is changed. Right now it's only sent when the document loads.
- Update row limit, grace period start, and data limit status in ActiveDoc when the product changes, i.e. the user upgrades/downgrades.
- Account for data size when computing data limit status, not just row counts.
See also the tasks mentioned in https://phab.getgrist.com/D3331
Test Plan: Extended FreeTeam nbrowser test, testing the 4 statuses.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3331
Summary:
This adds support for gvisor sandboxing in core. When Grist is run outside of a container, regular gvisor can be used (if on linux), and will run in rootless mode. When Grist is run inside a container, docker's default policy is insufficient for running gvisor, so a fork of gvisor is used that has less defence-in-depth but can run without privileges.
Sandboxing is automatically turned on in the Grist core container. It is not turned on automatically when built from source, since it is operating-system dependent.
This diff may break a complex method of testing Grist with gvisor on macs that I may have been the only person using. If anyone complains I'll find time on a mac to fix it :)
This diff includes a small "easter egg" to force document loads, primarily intended for developer use.
Test Plan: existing tests pass; checked that core and saas docker builds function
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3333
Summary:
This shuffles some server tests to make them available in grist-core,
and adds a test for the `GRIST_PROXY_AUTH_HEADER` feature added in
https://github.com/gristlabs/grist-core/pull/165
It includes a fix for a header normalization issue for websocket connections.
Test Plan: added test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3326
Summary:
Clicking undo/redo after converting a table to a summary table navigated
to the raw data view.
Test Plan: new test
Reviewers: georgegevoian, alexmojaki
Reviewed By: georgegevoian, alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3337
Summary:
Firstly I just wanted some more consistency and less repetition in places where Documents are retrieved from the DB, so it's more obvious when code differs from the norm. Main changes for that part:
- Let HomeDBManager accept a `Request` directly and convert it to a `Scope`, and use this in a few places.
- `getScope` tries `req.docAuth.docId` if `req.params` doesn't have a docId.
I also refactored how `_createActiveDoc` gets the document URL, separating out getting the document from getting a URL for it. This is because I want to use that document object in a future diff, but I also just find it cleaner. Notable changes for that:
- Extracted a new method `HomeDBManager.getRawDocById` as an alternative to `getDoc` that's explicitly for when you only have a document ID.
- Removed the interface method `GristServer.getDocUrl` and its two implementations because it wasn't used elsewhere and it didn't really add anything on top of getting a doc (now done by `getRawDocById`) and `getResourceUrl`.
- Between `cachedDoc` and `getRawDocById` (which represent previously existing code paths) also try `getDoc(getScope(docSession.req))`, which is new, because it seems better to only `getRawDocById` as a last resort.
Test Plan: Existing tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3328
Summary:
Adding conditional formatting rules feature.
Each column can have multiple styling rules which are applied in order
when evaluated to a truthy value.
- The creator panel has a new section: Cell Style
- New user action AddEmptyRule for adding an empty rule
- New columns in _grist_Table_columns and fields
A new color picker will be introduced in a follow-up diff (as it is also
used in choice/choice list/filters).
Design document:
https://grist.quip.com/FVzfAgoO5xOF/Conditional-Formatting-Implementation-Design
Test Plan: new tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3282
Summary:
This diff introduces cursor features for raw data views:
- Restoring cursor position when the browser window is reloaded
- Restoring the last edit position when the browser window is reloaded
Test Plan: Added tests
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3314
Summary:
- Unify where in the code tours get initiated.
- Avoid start a new tour while one is being started or is in progress.
- Ignore welcome tour when on a doc that has a doc tour.
- Fix tours when starting with a special page like Access Rules.
- Remove mention of the no-longer-present "Give Feedback" button in the last
message of the welcome tour.
Test Plan:
Add a browser test case that docTour preempts the welcome tour and shows no errors
(this test case fails in multiple ways without the changes).
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3330
Summary: Capture the stacktrace (via SandboxError) in `_pyCallWait` instead of `_onSandboxMsg` where it's always the same.
Test Plan:
Tested manually, found for example that the stacktrace in the logs changed from being rather useless:
```
at NSandbox._onSandboxMsg (/home/alex/work/grist/_build/core/app/server/lib/NSandbox.js:229:36)
at /home/alex/work/grist/_build/core/app/server/lib/NSandbox.js:179:18
at Unmarshaller.parse (/home/alex/work/grist/_build/core/app/common/marshal.js:289:21)
at NSandbox._onSandboxData (/home/alex/work/grist/_build/core/app/server/lib/NSandbox.js:174:28)
at Socket.<anonymous> (/home/alex/work/grist/_build/core/app/server/lib/NSandbox.js:63:59)
at Socket.emit (events.js:315:20)
at Socket.EventEmitter.emit (domain.js:467:12)
at addChunk (internal/streams/readable.js:309:12)
at readableAddChunk (internal/streams/readable.js:284:9)
at Socket.Readable.push (internal/streams/readable.js:223:10)
at Pipe.onStreamRead (internal/stream_base_commons.js:188:23)
```
to being somewhat more helpful:
```
at NSandbox._pyCallWait (/home/alex/work/grist/_build/core/app/server/lib/NSandbox.js:134:19)
at processTicksAndRejections (internal/process/task_queues.js:93:5)
at async ActiveDoc.applyActionsToDataEngine (/home/alex/work/grist/_build/core/app/server/lib/ActiveDoc.js:1080:39)
at async Sharing._applyActionsToDataEngine (/home/alex/work/grist/_build/core/app/server/lib/Sharing.js:325:37)
```
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3329
Summary:
Keep track of the number of API requests made for this document today in redis. Uses local caches of the count and the document so that usually requests can proceed without waiting for redis or the database.
Moved the free standing function apiThrottle to become a method to avoid adding another layer of request handler callbacks.
Test Plan: Added a DocApi test
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3327
Summary:
- Add a method `getSnapshotWindow` to `IInventory` and `DocSnapshotInventory`. It returns a `SnapshotWindow`, which represents a duration of time for which we keep backups for a particular document.
- `DocSnapshotPruner` calls this method and passes the window to `shouldKeepSnapshots` to determine which document versions have fallen outside the window and should be pruned.
- The implementation passed to `DocSnapshotInventory` uses a new method `getDocProduct` in `HomeDBManager` which directly returns the `Product` associated with a document, given only the document ID. Other methods in `HomeDBManager` require passing more information, especially about a user, but `DocSnapshotPruner` only knows about document IDs.
Test Plan: Added a test for `getDocProduct` and a test for `DocSnapshotPruner` where `getSnapshotWindow` is specified.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3322
Summary:
Adds nbrowser test
- Also makes sort spec taken into account by Group Data options
- This is a continuation of https://phab.getgrist.com/D3271
- We still need to decide whether to add stack chart to area chart type
Test Plan: TBD
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3274
Summary:
Makes the new TableOperations API available for all tables
in the document. Adds methods for discovering the tableId of the
selected table. I was very tempted to implement the select() TODO
in the TableOperations API, but it requires a significant refactor
of the backend.
Test Plan: added test
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3325
Summary: After dragging rows up, selection was set incorrectly.
Test Plan: Expanded a browser test for dragging rows to check selection, which fails without this fix
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3324
Summary:
Moves some auth-related UI components, like MFAConfig, out
of core, and adds a new ChangePasswordDialog component for
allowing direct password changes, replacing the old reset password
link to hosted Cognito.
Updates all MFA endpoints to use SRP for authentication.
Also refactors MFAConfig into smaller files, and polishes up some parts
of the UI to be more consistent with the login pages.
Test Plan: New server and deployment tests. Updated existing tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3311
Summary: This makes an equivalent of the /records REST endpoint available within custom widgets. For simple operations, it is compatible with https://github.com/airtable/airtable.js/. About half of the diff is refactoring code from DocApi that implements /records using applyUserActions, to make that code available in the plugin api.
Test Plan: added tests
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3320
Summary:
Add rowCount returned from sandbox when applying user actions to ActionGroup which is broadcast to clients.
Add rowCount to ActiveDoc and update it after applying user actions.
Add rowCount to OpenLocalDocResult using ActiveDoc value, to show when a client opens a doc before any user actions happen.
Add rowCount observable to DocPageModel which is set when the doc is opened and when action groups are received.
Add crude UI (commented out) in Tool.ts showing the row count and the limit in AppModel.currentFeatures. The actual UI doesn't have a place to go yet.
Followup tasks:
- Real, pretty UI
- Counts per table
- Keep count(s) secret from users with limited access?
- Data size indicator?
- Banner when close to or above limit
- Measure row counts outside of sandbox to avoid spoofing with formula
- Handle changes to the limit when the plan is changed or extra rows are purchased
Test Plan: Tested UI manually, including with free team site, opening a fresh doc, opening an initialised doc, adding rows, undoing, and changes from another tab. Automated tests seem like they should wait for a proper UI.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3318
Summary:
This calls sqlite3_limit(SQLITE_LIMIT_ATTACHED, 0) so that
if ever an `ATTACH` were snuck into an SQL query, it would be denied.
The limit needs to be waived when calling VACUUM since the implementation
of VACUUM uses ATTACH.
Test Plan: added test; existing tests should pass
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3316
Summary:
Adds Google Tag Manager snippet to all login pages, and a new user
preference, recordSignUpEvent, that's set to true on first sign-in. The
client now checks for this preference, and if true, dynamically loads
Google Tag Manager to record a sign-up event. Afterwards, it removes
the preference.
Test Plan: Tested manually.
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3319
Summary:
Creating new UI for raw data views based on design.
- Renaming left for follow up diff
- Link in the menu is hidden for now
To access raw UI, use /p/data URL.
Test Plan: new tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3306
Summary:
If cancel was clicked while a transform section was still being
generated in the Importer, an error was thrown. This refactors
the cancelImportFiles API action to take in the file upload id
in place of the entire DataSourceTransformed parameter, which
contains other values that are irrelevant to canceling. One of those
values, the transform section id, was causing the error to be thrown
since it was momentarily null.
Test Plan: Tested manually.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3317
Summary: As suggested by @dsagal in https://phab.getgrist.com/D3277#inline-36801, change to query `SUM(pgsize - unused)` instead of `SUM(pgsize)` to measure actual data size more accurately. Technically this doesn't reflect the database file size as accurately, but it should reflect sandbox memory usage better, and more importantly it should allow users to see data size decreasing when they delete stuff.
Test Plan: Tested manually by adding rows to a doc and looking at the logs. The data size is smaller and changes more granularly.
Reviewers: dsagal, paulfitz
Reviewed By: paulfitz
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3313
Summary:
Custom widget into page is served from a homeUrl instead
of untrusted URL, which might be not used in grist-core.
Test Plan: manual test
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3307
Summary: Fix for error that happens when a search is resumed after one of the columns was hidden.
Test Plan: Added test that shows the error.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3309
Summary:
Importing a .grist document is implemented in a somewhat clunky way, in a multi-worker setup.
* First a random worker receives the upload, and updates Grist's various stores appropriately (database, redis, s3).
* Then a random worker is assigned to serve the document.
If the worker serving the document fails, there is a chance the it will end up assigned to the worker that handled its upload. Currently the worker will misbehave in this case. This diff:
* Ports a multi-worker test from test/home to run in test/s3, and adds a test simulating a bad scenario seen in the wild.
* Fixes persistence of any existing document checksum in redis when a worker is assigned.
* Adds a check when assigned a document to serve, and finding that document already cached locally. It isn't safe to rely only on the document checksum in redis, since that may have expired.
* Explicitly claims the document on the uploading worker, so this situation becomes even less likely to arise.
Test Plan: added test
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3305
Summary:
- Removed string parsing and some type guessing code from parse_data.py. That logic is now implicitly done by ValueGuesser by leaving the initial column type as Any. parse_data.py mostly comes into play when importing files (e.g. Excel) containing values that already have types, i.e. numbers and dates.
- 0s and 1s are treated as numbers instead of booleans to keep imports lossless.
- Removed dateguess.py and test_dateguess.py.
- Changed what `guessDateFormat` does when multiple date formats work equally well for the given data, in order to be consistent with the old dateguess.py.
- Columns containing numbers are now always imported as Numeric, never Int.
- Removed `NullIfEmptyParser` because it was interfering with the new system. Its purpose was to avoid pointlessly changing a column from Any to Text when no actual data was inserted. A different solution to that problem was already added to `_ensure_column_accepts_data` in the data engine in a recent related diff.
Test Plan:
- Added 2 `nbrowser/Importer2` tests.
- Updated various existing tests.
- Extended testing of `guessDateFormat`. Added `guessDateFormats` to show how ambiguous dates are handled internally.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3302
Summary:
The page isn't yet linked to from anywhere in the UI, but
will be soon, once the new login page is ready. The page
can still be accessed at login-[s].getgrist.com/forgot-password,
and the flow is similar to the one used by Cognito's hosted UI.
Also refactors much of the existing login app code into smaller
files with less duplication, tweaks password validation to be closer
to Cognito's requirements, and polishes various parts of the UI,
like the verified page CSS, and the form inputs.
Test Plan: Browser, server and project tests.
Reviewers: jarek
Reviewed By: jarek
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3296
Summary:
Until now, users could not pick the column that's currently serving as
the x axis.
To do that, chart needs to support having the x axis undefined.
This diff do just that:
- allow x axis to be undefined
- allow setting x axis from group data
Given that charts axis are stored as indexes of the section view
fields array, implementation required introduction of an extra chart
options: `isAxisUndefined`.
Test Plan: Updates existing test and adds new one.
Reviewers: jarek
Reviewed By: jarek
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3304
Summary:
This makes many small changes so that Grist is less fussy to run as a single instance behind a reverse proxy. Some users had difficulty with the self-connections Grist would make, due to internal network setup, and since these are unnecessary in any case in this scenario, they are now optimized away. Likewise some users had difficulties related to doc worker urls, which are now also optimized away. With these changes, users should be able to get a lot further on first try, at least far enough to open and edit documents.
The `GRIST_SINGLE_ORG` setting was proving a bit confusing, since it appeared to only work when set to `docs`. This diff
adds a check for whether the specified org exists, and if not, it creates it. This still depends on having a user email to make as the owner of the team, so there could be remaining difficulties there.
Test Plan: tested manually with nginx
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3299
Summary:
This error only occured for certain charts type. Some chart handled it correctly, others did not.
This brings a more general handling of this issue which should work for all charts.
Test Plan: New nbrowser test case.
Reviewers: jarek
Reviewed By: jarek
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3303
Summary:
Change NumberParse.parse to return not just the parsed number but also information it gathered along the way about how the input string was formatted.
Use this in the new NumberParse.guessOptions to guess the actual widget options based on an array of strings.
Use NumberParse.guessOptions in TypeConversion (for when a user explicitly chooses to change type) and in ValueGuesser (for guesses about strings entered into empty columns).
Test Plan: Adds unit tests for NumberParse and ValueGuesser and updates the TypeChange2 nbrowser test.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3294
Summary:
This adds a new private field to ColumnTransform that settles when
the transform column is done preparing. Finalize then waits for this
promise before proceeding.
A bug was occurring when the prepare promise was still pending,
but finalize was triggered indirectly, causing finalize to throw an error
due to the transform column not yet existing.
Test Plan: Tested manually.
Reviewers: jarek, dsagal
Reviewed By: jarek, dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3288
Summary:
Adds `common/ValueGuesser.ts` with logic for guessing column type and widget options (only for dates/datetimes) from an array of strings, and converting the strings to the guessed type in a lossless manner, so that converting back to Text gives the original values.
Changes `_ensure_column_accepts_data` in Python to call an exported JS method using the new logic where possible.
Test Plan: Added `test/common/ValueGuesser.ts` to unit test the core guessing logic and a DocApi end-to-end test for what happens to new columns.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3290
Summary: Bug fix. Moving columns over the frozen set didn't work properly.
Test Plan: new tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3297
Summary: Removes code that was marked for removal.
Test Plan: Existing tests still pass.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3289
Summary:
Core doesn't redirect to Cognito or our own sign-up page
when clicking 'sign up' on the welcome screen. Instead, it
redirects to the test login page.
Test Plan: N/A (fixing test)
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3298
Summary:
Fix this issue:
When using “Change Widget” for a chart of a summary table, if I change group-by columns, the X-Axis dropdown becomes empty, and lists values from previous summary (not the actual summary table that’s now shown in the chart). Need to close and reopen creator panel to fix it.
Test Plan: Include nbrowser test
Reviewers: georgegevoian
Differential Revision: https://phab.getgrist.com/D3284
Summary: As simple as it sounds. Check ev.shiftKey in Enter key handler in search.ts.
Test Plan: Extended test/nbrowser/Search.ts
Reviewers: jarek
Reviewed By: jarek
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3293
Summary: When user 2FA status is changed, we now send out emails via SendGrid.
Test Plan: Server tests.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3280
Summary:
This makes a `user.SessionID` value available in information about the user, for use with trigger formulas and granular access rules. The ID should be constant within a browser session for anonymous user. For logged in users it simply reflects their user id.
This ID makes it possible to write access rules and trigger formulas that allow different anonymous users to create, view, and edit their own records in a document.
For example, you could have a brain-storming document for puns, and allow anyone to add to it (without logging in), letting people edit their own records, but not showing the records to others until they are approved by a moderator. Without something like this, we could only let anonymous people add one field of a record, and not have a secure way to let them edit that field or others in the same record.
Also adds a `user.IsLoggedIn` flag in passing.
Test Plan: Added a test, updated tests. The test added is a mini-moderation doc, don't use it for real because it allows users to edit their entries after a moderator has approved them.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3273
Summary:
Since the new Grist sign-up page has a required field for
name, we can now skip the welcome page asking for the
same thing. Code and tests that can be removed later are
marked with TODOs.
Test Plan: Browser tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3266
Summary:
The API key is now hidden by default. Clicking the input
will select and reveal the key. As soon as the key loses
selection, it is hidden again.
Test Plan: Project test.
Reviewers: jarek
Reviewed By: jarek
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3270
Summary:
Adds a method Table._num_rows using an empty lookup map column.
Adds a method Engine.count_rows which adds them all up.
Returns the count after applying user actions to be logged by ActiveDoc.
Test Plan: Added a unit test in Python. Tested log message manually.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3275
Summary:
- Small cleanup: Make DocStorage implement OnDemandStorage, and remove unused execWithBackup
- Upgrade to new versions (.3) of @gristlabs/sqlite3 and connect-sqlite3 to use dbstat
- Add _logDataSize method which queries dbstat, adding up pgsize for tables loaded into the data engine
- Only complete _logDataSize every 5 minutes using new field _lastLoggedDataSize
Test Plan: Tested manually
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3277
Summary:
- Adds a dependency moment-guess (https://github.com/apoorv-mishra/moment-guess) to guess date formats from strings. However the npm package is missing source maps which leads to an ugly warning, so currently using a fork until https://github.com/apoorv-mishra/moment-guess/pull/22 is resolved.
- Adds guessDateFormat using moment-guess to determine the best candidate date format. The logic may be refined for e.g. lossless imports where the stakes are higher, but for now we're just trying to make type conversions smoother.
- Uses guessDateFormat to guess widget options when changing column type to date or datetime.
- Uses the date format of the original column when possible instead of guessing.
- Fixes a bug where choices were guessed based on the display column instead of the visible column, which made the guessed choices influenced by which values were referenced as well as completely broken when converting from reflist.
- @dsagal @georgegevoian This builds on https://phab.getgrist.com/D3265, currently unmerged. That diff was created first to alert to the change. Without it there would still be similar test failures/changes here as the date format would often be concretely guessed and saved as YYYY-MM-DD instead of being left as the default `undefined` which is shows as YYYY-MM-DD in the dropdown.
Test Plan: Added a unit test to `parseDate.ts`. Updated several browser tests which show the guessing in action during type conversion quite nicely.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: dsagal, georgegevoian
Differential Revision: https://phab.getgrist.com/D3264
Summary: By default, new date columns show "YYYY-MM-DD" as the date format in the UI, but under the hood it's stored as null/undefined, which affects parsing during pasting and type conversion. This changes the underlying parsing to always default to YYYY-MM-DD, as if that format was explicitly selected, making things more consistent.
Test Plan: Updated some tests.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3265
Summary:
also:
- closes opened menu if any when click on a custom widget
- closes opened menu if any when F2
Test Plan: Include test case
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3269
Summary:
Exposing WidgetApi methods (getOption, setOption, ...) in a module
scope to simplify custom widget code.
Test Plan: manual tests for now, as plugin is not yet on production.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3268
Summary:
Enabled by default, the new checkbox is only visible to
users logged in with email/password, and controls whether it is possible
to log in to the same account via a Google account
(with matching email). When disabled, CognitoClient will refuse logins
from Google if a Grist account with the same email exists.
Test Plan:
Server and browser tests for setting flag. Manual tests to verify
Cognito doesn't allow signing in with Google when flag is disabled.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3257
Summary:
Available at login.getgrist.com/signup, the new sign-up page
includes similar options available on the hosted Cognito sign-up
page, such as support for registering with Google. All previous
redirects to Cognito for sign-up should now redirect to the new
Grist sign-up page.
Login is still handled with the hosted Cognito login page, and there
is a link to go there from the new sign-up page.
Test Plan: Browser, project and server tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3249
Summary:
When possible, the original column headers from imported
files will now be used as the labels for Grist columns. This includes
values that were previously invalid Grist column identifiers, such
as those containing Unicode.
Test Plan: Updated server and browser tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3261
Summary: Tweaked ReferenceListFormatter and ValueConverter to sensibly convert to Text. Fixes an embarrassing bug exposed during checkin 😱
Test Plan: Add test for converting from RefList displaying formatted dates to Text
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3254
Summary:
As designed in https://grist.quip.com/fZSrAnJKgO5j/Add-or-Update-Records-API
Current `POST /records` adds records, and `PATCH /records` updates them by row ID. This adds `PUT /records` to 'upsert' records, applying the AddOrUpdate user action. PUT was chosen because it's idempotent. Using a separate method (instead of inferring based on the request body) also cleanly separates validation, documentation, etc.
The name `require` for the new property was suggested by Paul because `where` isn't very clear when adding records.
Test Plan: New DocApi tests
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3251
Summary: Bug with 'records' endpoint that was treating 0 as null.
Test Plan: Modified tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3262
Summary:
Also fix few clode glitches
- Attach events handler to the scrollPane using `onMatch` instead of
the cell itself. This streamline cell dom creation a bit while
scrolling.
- Fix memory leaks on contextMenu.
- Also fix Importer and ColumnOps nbrowser test
see:
https://phab.getgrist.com/D3237#inline-36376https://phab.getgrist.com/D3237#inline-36375
Restore default context menu for where there's no custom one
It appears that default context menu adds some value, in particular
for links and for editing text. This diff restores it.
Test Plan: Should not break anything.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3256
Summary:
Custom widgets will receive all visible columns when they are not configured.
This will make the existing configuration work without the need to pick columns.
Test Plan: updated tests
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3259
Summary: Check that view exists first, otherwise fall back to default
Test Plan: Manual
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3260
Summary:
- Added a new special page for viewing raw data widgets:
- Implemented in DataTables.ts
- Accessible only via the special URL path `/p/data`
- Future diffs should make this page prettier and easily accessible
- Shows a list of user tables
- Clicking on a table name shows its `rawViewSection` by setting `GristDoc.viewModel.activeSectionId`. Note that in this case `GristDoc.viewModel` is an empty record, so this is a bit of a hack, but it works well and causes no known issues.
- Added `ViewSectionRec.isRaw` to know if the record represents a raw data widget.
- Added various restrictions in the UI for raw data widgets:
- 'Delete widget' is disabled in the 3-dot widget menu.
- Prevent hiding columns:
- "Hide column" in the column context menu is disabled
- The "VISIBLE/HIDDEN COLUMNS" section of the right panel > Table > Widget is hidden
- The toggle bar isn't configurable to ensure that users know when raw data is filtered:
- The filter bar always shows if and only if some filters are present
- "Toggle Filter Bar" is hidden in:
- Right panel > Table > Sort & Filter
- The sort/filter menu next to the three-dot menu for widgets.
- Other restrictions in the right panel:
- In the Column tab:
- 'Use separate settings' is disabled
- In the Table tab:
- In the Widget subtab:
- 'Change Widget' is hidden
- In the Data subtab:
- 'Edit Data Selection' is hidden
- 'SELECT BY' is hidden
Test Plan: Tested manually. The behaviour of raw data widgets may still change and they aren't easily visible to users yet.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3248
Summary:
Limits crafted for our SaaS product were getting applied to grist-core
users. This diff removes them. There will be limits on a future
self-managed product.
Test Plan: checked manually
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3255
Summary:
Exposing new API in CustomSectionAPI for column mapping.
The custom widget can call configure method (or use a ready method) with additional parameter "columns".
This parameter is a list of column names that should be mapped by the user.
Mapping configuration is exposed through an additional method in the CustomSectionAPI "mappings". It is also available
through the onRecord(s) event.
This DIFF is connected with PR for grist-widgets repository https://github.com/gristlabs/grist-widget/pull/15
Design document and discussion: https://grist.quip.com/Y2waA8h8Zuzu/Custom-Widget-field-mapping
Test Plan: browser tests
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3241
Summary:
- Brings in a new utility `contextMenu` to open context menu next to the mouse position
- Use this utility to show a CellContextMenu, that sort of merge cell context menu and column context menu together.
- Show cell context menu on context click on any grid's cell.
- Also takes care of showing the row context menu for detail view on a context click that occurs on cells and not only on the row num header as it was the case prior to this diff.
- task: https://gristlabs.getgrist.com/doc/check-ins/p/5#a1.s9.r1529.c31
- discussion: https://grist.quip.com/ETGkAroLnc0Y/Cell-Context-Menu
{F40092}
Test Plan: - Adds project test and nbrowser for cell context menu and new cases for the detail row context menu.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3237
Summary: Change code that conditionally depended on #newui in the URL to assume that everything is always in the new UI now.
Test Plan: this
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3246
Summary: This is https://phab.getgrist.com/D3205 plus some changes (https://github.com/dsagal/grist/compare/type-convert...type-convert-server?expand=1) that move the conversion process to the backend. A new user action ConvertFromColumn uses `call_external` so that the data engine can delegate back to ActiveDoc. Code for creating formatters and parsers is significantly refactored so that most of the logic is in `common` and can be used in different ways.
Test Plan: The original diff adds plenty of tests.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3240
Summary:
This adds a Feature object that is an approximation of what we
plan for free team sites. It includes restrictions that are not
yet implemented, and an endpoint for testing.
Test Plan: added a test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3243
Summary: Remove several unused methods in ViewConfigTab.js, and all of SummaryConfig.js.
Test Plan: this
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3245
Summary:
New user action as described in https://grist.quip.com/fZSrAnJKgO5j/Add-or-Update-Records-API, with options to allow most of the mentioned possible behaviours.
The Python code is due to Alex (as should be obvious from the u in behaviours).
Test Plan: Added a unit test.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3239
Summary:
When two widgets are linked by same-record linking, and the source of that link is also filter-linked, then it will pick up default values from its own filter-link source, but the same-record-link target didn't. This fixes that so that default values are filled in intuitively.
Moved the logic of linkingState, linkingFilter, and getDefaultColValues from BaseView.js to LinkingState.ts and ViewSectionRec.ts. In particular getDefaultColValues is now a property of LinkingState which may be copied from the source view section for a same-record link.
Note that `ViewSectionRec.linkingFilter` no longer uses `computerBuilder` and thus doesn't ignore dependencies inside LinkingState any more. I couldn't figure out how to make `linkingFilter` a `pureComputed` (otherwise I get recursion errors) that ignores dependencies. In any case, it's now important to have a dependency on `srcSection.linkingState()` for `getDefaultColValues` to work correctly, so I think this is for the best.
Test Plan: Added a new nbrowser test and fixture.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3238
Summary: The value seems unused
Test Plan: This is me testing if it's actually unused
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3244
Summary: This is the first step towards raw data views, merely adding metadata without any UI. Every 'normal' table now has a widget referenced by `rawViewSectionRef`. It has no parent view/page and cannot actually be viewed for now. The widget is created during the AddTable user action, and the migration creates a widget for existing tables.
Test Plan: Many tests had to be updated, especially tests that listed all view sections and/or fields.
Reviewers: jarek, dsagal
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3232
Summary:
Tweaks CSS of account page, ApiKey and MFAConfig to work better
on narrow-screen devices.
Test Plan: Tested manually.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3234
Summary:
Improving user experience on summary columns.
- Showing 'not-allowed' cursor on sections/menus that can't be changed
- Disabling menu options and buttons in the column behavior section that converts a formula column to a data column
- Showing nicer error message about converting formula to a data column.
Test Plan: manual tests, no behavior change
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3222
Summary:
Grist sessions created pre-MFA do not store access or
refresh tokens, which means that MFA status from Cognito
can't be loaded without requiring re-authentication. MFAConfig
handles this by requiring security verification as usual, and
checking if it needs to reload MFA status on success. If it does, it'll
close the 2FA setup dialog and reload, which should show the correct
2FA configuration status and buttons.
Test Plan: Updated existing tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3231
Summary:
Bumps aws-sdk to the latest version of v2.
Updates CognitoClient to use fake AWS credentials with service provider. While none
of our calls currently require real credentials, we still need to send fake credentials
with certain calls, such as AssociateSoftwareToken, to avoid getting errors back from
AWS about missing credentials.
Fixes some CSS and alignment issues with the authentication method tile buttons in
MFAConfig.
Test Plan: Existing Cognito tests. Manually tested login and 2FA setup still work.
Reviewers: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3228
Summary:
This fixes the cell color only applying to the content height of a
choice or reference list cell, and not the entire cell. Could
be reproduced by causing a row other than the choice/reference
list to grow in height.
Test Plan: Tested manually.
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3226
Summary:
stop providing a default document id DOC_ID_NEW_USER_INFO for
surveying, and don't show survey if a document id is not available.
Test Plan: existing tests pass; grist-core checked
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3225
Summary:
While working on type conversion, I ran into a bug when multiple action groups belonging to a bundle are quickly sent. `DocData._sendActionsImpl` can set the same `linkId: this._lastActionNum` on multiple action groups before `_lastActionNum` is updated with a result from the server. Only one of these groups gets saved in `UndoStack._linkMap` so undoing a bundle misses out on some action groups.
This diff associates each `linkId` key with an array of action groups instead of just one, then combines them all together when undoing.
Test Plan: I've confirmed that this fixes my problem within my type conversion diff, but I haven't found a way to reproduce the general problem in master. I think the existing tests are probably fine since undo is tested extensively everywhere, but I'd also like to see if there are existing bugs which this fixes.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3223
Summary:
Adds a mention of phone message (SMS) verification currently only
supporting U.S. phone numbers, and updates CognitoClient to return
an error when a non-U.S. number is submitted to register_sms/.
Test Plan: Server test.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3227
Summary:
Enables configuration of multi-factor authentication from the
account page (for users who sign in with email/password), and adds
SMS as an authentication method.
Test Plan: Project, browser and server tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3215
Summary:
The routine that makes sure that new charts are created with at least
one non-numeric series did not handle correctly when the table has one
single column.
This diff fixes it.
Test Plan: Adds test case to ChartView3.ts
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki, jarek
Differential Revision: https://phab.getgrist.com/D3224
Summary:
We do not support to show non numeric column as chart series.
However we didn't prevent the user from doing it and it could cause unexpected behaviour such as a missing chart.
This diff addresses that issue by doing two following thing:
1) it prevents user from adding non numeric column as series and
2) it makes sure that if there is a non numeric series it does not mess up the chart (it still can happen that a non numeric series ends up in charts even with 1) for instance if users convert a series' column to a non numeric column for instance).
Links to UI discussion:
- https://grist.quip.com/wb4gAgrQM2aP#TZEADAKPs8n
- https://grist.quip.com/wb4gAgrQM2aP#TZEADAP8S8N
Test Plan:
- new behaviour covered in nbrowser/ChartView3.ts
- Some test were using non-numeric column as series, diff fixes that to.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3206
Summary:
Improving experience when editing group-by column:
- Disable column rename
- Allow changing most widget properties:
- Color/Background
- Number format
- Date/DateTime format (but not the timezone)
- All toggle options (for toggle column)
- Remove Edit button on Choice Edit
- Changing the underlying column should reset all those options back to the original column.
Test Plan: nbrowser
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3216
Summary:
- WelcomeQuestions implements the new popup.
- Popup shows up on any doc-list page, the first time the user visits one after
signing up and setting their name.
- Submits responses to the same "New User Questions" doc, which has been
changed to accept two new columns (ChoiceList of use_cases, and Text for
use_other).
- Improve modals on mobile along the way.
Test Plan: Added browser tests and tested manually
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3213
Summary:
The user profile dialog is now a separate page, in preparation
for upcoming work to enable MFA. This commit also contains
some MFA changes, but the UI is currently disabled and the
implementation is limited to software tokens (TOTP) only.
Test Plan:
Updated browser tests for new profile page. Tests for MFAConfig
and CognitoClient will be added in a later diff, once the UI is enabled.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3199
Summary:
Previously, ref/reflist columns were formatted entirely based on their visible column, since they received values from the visible or display columns rather than the actual row IDs. This creates `ReferenceFormatter` and `ReferenceListFormatter` which still delegate most of the formatting work to a visible column formatter but fix a few issues:
- ReferenceList columns now actually use the options (e.g. date format) of the visible column to format their elements. Previously they were formatted generically because the visible column formatter wasn't expecting a list.
- Invalid references aren't formatted with an `#Invalid Ref` prefix.
- When the ref column displays the Row ID, it doesn't have a visible or display column. Previously this led to the references being formatted as just numbers in most cases, with special code in the widget to display them like `Table1[2]`. Now they are consistently formatted in that style throughout.
Test Plan: Updated existing tests.
Reviewers: jarek
Reviewed By: jarek
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3212
Summary:
Adding configuration options for CustomWidgets.
Custom widgets can now store options (in JSON) in viewSection metadata.
Changes in grist-plugin-api:
- Adding onOptions handler, that will be invoked when the widget is ready and when the configuration is changed
- Adding WidgetAPI - new API to read and save a configuration for widget.
Changes in Grist:
- Rewriting CustomView code, and extracting code that is responsible for showing the iframe and registering Rpc.
- Adding Open Configuration button to Widget section in the Creator panel and in the section menu.
- Custom Widgets can implement "configure" method, to show configuration screen when requested.
Test Plan: Browser tests.
Reviewers: paulfitz, dsagal
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3185
Summary: When editor is opened on any column and closed without entering any value, the column is converted to a text column.
Test Plan: browser tests
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3211
Summary:
Formats lists as CSVs at the top level, so the list `["a", "b"]` gets formatted as `a,b`. Further nesting looks like JSON, with quotes around strings, which get doubled to escape them in the CSV. So the common case looks significantly nicer, but the rare case of nested arrays looks very weird and confusing.
There's also some smaller details about quotes and spaces to discuss if we're happy with the overall idea.
This is part of revamping type conversion and was discussed here: https://grist.quip.com/csqCAfx6KHt2#HOaADA7Q6NM
Test Plan: Updated several tests, need to confirm if we want this behaviour before continuing.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3208
Summary:
When using the grouped data option with a column (A) that has some
blank values, all rows with blank values for A are grouped into one
series.
The issue is that the name that showed on the legend for that series
used to be the name of the yseries, and not the name of the value.
This diff fixes it by showing `[Blank]` instead.
Test Plan: Includes new test case.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3210
Summary: When an editor is activated by typing, the active view should be scrolled to the active record.
Test Plan: new tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3196
Summary:
This updates the grist-core README to list specific features of Grist,
to make it easier for a casual visitor to get a sense of its scope. Adds links
to some new resources (reviews, templates, grist v airtable post) that could
also help. Adds python3 to docker image so that templates work without fuss.
Test Plan: existing tests should pass
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: dsagal, anaisconce
Differential Revision: https://phab.getgrist.com/D3204
Summary:
A check for old browsers and a Grist favicon were not available in
grist-core, leaving harmless but distracting errors in logs. This
adds them.
Test Plan: checked manually
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3207
Summary:
Document owners can now remove doc tours by pressing the button located
to the right of 'Tour of this Document' in the left panel.
Test Plan: Browser test.
Reviewers: jarek
Reviewed By: jarek
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3202
Summary:
The culprit was `series = uniqXValues(series);` because it creates new
series objects when they are used as keys to access error bars info
(`errorBars.get(line)`).
Fixed by making uniqXValues mutating series instead.
Test Plan: Adds a case to test error bars with bar charts.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3198
Summary: Fix for a bug. When a chart had a "Group Data" checked, deleting it produced a JS error.
Test Plan: browser test
Reviewers: cyprien
Reviewed By: cyprien
Differential Revision: https://phab.getgrist.com/D3200
Summary:
When disposing a GridView (i.e. switching pages) there was JS error. It was caused, by
autodisposing knockout observable.
Test Plan: manual tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3201
Summary:
Improving UX for the formula editor
- Formula editor will go into full edit mode only on formula change (not on a mouse click)
- Adding column highlight and a tooltip when in full edit mode
Test Plan: nbrowser tests
Reviewers: cyprien
Reviewed By: cyprien
Differential Revision: https://phab.getgrist.com/D3194
Summary:
By default, plotly's pie chart sort lables by values.
This is iconsistent with how bar charts work and especially annoying
in case of a linked chart because values can change when user navigate
the linked table which causes colors (and display order) of each label
to change. Making it hard to keep track values.
[[ https://grist.quip.com/wb4gAgrQM2aP/Chart-Improvements-November-2021#temp:C:TZE88067825d66c415da9e839488 | Link to video with more details about the issue ]]
Test Plan: Adds a new test case.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3193
Summary:
This turns out necessary because ReferenceList columns are formatted
using the formatter of the associated visibleCol. This works correctly
in dedicated widgets, but in generic code (like SearchModel here), this
formatter needs to handle unexpected values (of type ReferenceList).
Without the fix, it produces JS errors when search reaches a
RefList:<Date> column.
A better fix would allow a formatter to know that it expects a ReferenceList,
AND to know how to format each value of it, but that's a bigger question
that's outside the scope of this fix.
Test Plan: Includes a browser test which reproduces the bug.
Reviewers: cyprien
Reviewed By: cyprien
Differential Revision: https://phab.getgrist.com/D3195
Summary:
Two bugs fixed:
1. On search, when the first result is in the active record, GridView wasn't scrolling to the active record.
2. When an active record was not visible, GridView wasn't scrolling to the active record when the column index was changed.
The problem was that the scrolling behavior was based only on rowIndex which isn't changed (and doesn't notify subscribers) when a column index changes or when the search highlights a cell.
This diff makes the computed depend also on the fieldIndex, and is introducing a new method that can scroll to the active record on demand (which is used by the search).
Test Plan: Updated tests.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3191
Summary:
Fixing a bug: When removing a page with linked sections and then undoing, there are two JS errors raised:
- flexSize is not a function
- getter is not a function
Test Plan: nbrowser tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3192
Summary:
- Adds a function `parseUserAction` for parsing strings in UserActions to `ValueParser.ts`
- Adds a boolean option `parseStrings` to use `parseUserAction` in `ActiveDoc.applyUserActions`, off by default.
- Uses `parseStrings` by default in DocApi (set `?noparse=true` in a request to disable) when adding/updating records through the `/data` or `/records` endpoints or in general with the `/apply` endpoint.
- Uses `parseStrings` for various actions in `ActiveDocImport`. Since most types are parsed in Python before these actions are constructed, this only affects references, which still look like errors in the import preview. Importing references can also easily still run into more complicated problems discussed in https://grist.slack.com/archives/C0234CPPXPA/p1639514844028200
Test Plan:
- Added tests to DocApi to compare behaviour with and without string parsing.
- Added a new browser test, fixture doc, and fixture CSV to test importing a file containing references.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3183
Summary:
Reference/referencelist columns displaying date/datetime columns didn't show the formatting of that column, formatting them as ISO instead. One weird effect of this was that opening the editor suddenly changed the format because the editor formatted the dates correctly. You can see this in the checkin doc as an example.
This was discussed in https://grist.slack.com/archives/C0234CPPXPA/p1636482208111800. Here's the main point:
> both use the visible column formatter's formatAny. the editor uses the value from the visible column, which for a date column is a raw timestamp number. the cell display uses the value from the display column which is of type Any so the value is wrapped in a list starting with 'd'. the former gets formatted according to the formatting options, but the latter just gets formatted as ISO.
Probably a good solution to the broader problem is to ensure that the display column has the same type and widget options as the visible column. That seems potentially messy, so I did something easier: fix `DateFormatter` to accept encoded date/datetime objects. It still receives the correct widget options from the visible column as before but can handle the values from the display column. This might also have other uses in the future.
Test Plan:
- Fixed several tests which previously expected the buggy behaviour.
- Converted ValueFormatter.js tests to typescript and cleaned up the existing code slightly.
- Added tests for DateFormatter and DateTimeFormatter to the ValueFormatter test suite, which only tested numbers before.
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3190
Summary:
Some things (like rendering cells) use the `visibleCol` for `createFormatter`, while other things (like `CopySelection`) used the `displayCol`. For references, the display column has type Any and doesn't know about the original formatting. This resulted in formatting being lost when copying from reference columns even though formatting was preserved when copying from the original (visible) column which looked identical. This diff fixes this and ensures that `createFormatter` is always used with the `visibleCol`. This was agreed on in https://grist.slack.com/archives/C0234CPPXPA/p1639571321043000
Additionally:
- Replaces the functions `createVisibleColFormatter` computed properties `visibleColFormatter` as suggested by a `TODO`.
- Extracts common code from `createVisibleColFormatter` in `ColumnRec` and `ViewFieldRec`
Test Plan: Fixed a test in CopyPaste which displayed the previous inconsistent behaviour.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3189
Summary: For GristDocTours on mobile, ignore the Placement column and always use automatic placement for popups
Test Plan: Tested manually with chrome devtools
Reviewers: jarek
Reviewed By: jarek
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3188
Summary:
Addresses several issues:
- Error 'Cannot modify summary group-by column' when changing Text ->
ChoiceList in the presence of summary tables.
- Error 'ModifyColumn in unexpected position' when changing ChoiceList -> Text
in the presence of summary tables.
- Double-evaluation of trigger formulas in some cases.
Fixes include:
- Fixed verification that summary group-by columns match the underlying ones,
and added comments to explain.
- Avoid updating non-metadata lookups after each doc-action (early lookups
generated extra actions to populate summary tables, causing the 'ModifyColumn
in unexpected position' bug)
- When updating formulas, do update lookups first.
- Made a client-side tweak to avoid a JS error in case of some undos.
Solution to reduce lookups is based on https://phab.getgrist.com/D3069?vs=on&id=12445,
and tests for double-evaluation of trigger formulas are taken from there.
Add a new test case to protect against bugs caused by incorrect order of
evaluating #lookup columns.
Enhanced ChoiceList browser test to check a conversion scenario in the presence
of summary tables, previously triggering bugs.
Test Plan: Various tests added or enhanced.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3184
Summary:
Adding new destination "Skip" for multiple table imports.
Selecting this destination skips the import and makes the preview grayed out.
Test Plan: New Tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3181
Summary:
* Tie build and run-time docker base images to a consistent version (buster)
* Extend the test login system activated by GRIST_TEST_LOGIN to ease porting tests that currently rely on cognito (many)
* Make org resets work in absence of billing endpoints
* When in-memory session caches are used, add missing invalidation steps
* Pass org information through sign-ups/sign-ins more carefully
* For CORS, explicitly trust GRIST_HOST origin when set
* Move some fixtures and tests to core, focussing on tests that cover existing failures or are in the set of tests run on deployments
* Retain regular `test` target to run the test suite directly, without docker
* Add a `test:smoke` target to run a single simple test without `GRIST_TEST_LOGIN` activated
* Add a `test:docker` target to run the tests against a grist-core docker image - since tests rely on certain fixture teams/docs, added `TEST_SUPPORT_API_KEY` and `TEST_ADD_SAMPLES` flags to ease porting
The tests ported were `nbrowser` tests: `ActionLog.ts` (the first test I tend to port to anything, out of habit), `Fork.ts` (exercises a lot of doc creation paths), `HomeIntro.ts` (a lot of DocMenu exercise), and `DuplicateDocument.ts` (covers a feature known to be failing prior to this diff, the CORS tweak resolves it).
Test Plan: Manually tested via `buildtools/build_core.sh`. In follow up, I want to add running the `test:docker` target in grist-core's workflows. In jenkins, only the smoke test is run. There'd be an argument for running all tests, but they include particularly slow tests, and are duplicates of tests already run (in different configuration admittedly), so I'd like to try first just using them in grist-core to gate updates to any packaged version of Grist (the docker image currently).
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3176
Summary:
Backstory: to make examples easier to play with, we:
* Add a special FullCopies permission to let anyone fork/copy them regardless of other access rules
* Open the examples in "prefork" mode by default
That means a random person can open an example and already feel like an owner of it. Getting to this point requires some gymnastics on the back end. As soon as the person makes any change to the document they become truly the owner (of their fork), and life is simple for the back end.
But, if that person does "View As" to look at the preforked document, that is a step too far for the back end - a user, with a special somewhat complicated exception allowing them to act as an owner for some purposes, now wants to pretend to be another user. The logic for this on the back end was doable, but looked hard to review and be confident of, with now three identities with subtle nuances in their interrelationship.
So with this diff, if a non-owner attempts to "View As" another user on a prefork, the client will just fork the document first. This is in principle not necessary, but is much simpler from a security perspective.
Test Plan: extended test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3179
Summary:
The ./test/grist-admin doc-info command line tool was out of date
and not showing user access correctly anymore. This freshens the tool and
adds a small test for it.
Test Plan: Added test.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3182
Summary:
- Fix base href in HelpScout beacon when showing articles (in particular for Firefox)
- Show the 'Answers' tab normally except when reporting an error.
- Combine the "Give Feedback" and "Help Center" buttons into one that normally
opens the beacon (with a link to Help Center and to Community Forum), and a
smaller one that opens the Help Center site in a new tab.
- Update HELP_SCOUT_* env vars to use _V2 suffix, to allow them to coexist with
code using the previous beacon.
Test Plan: Updated the browser test to check the new behavior.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3170
Summary:
Following discussion in https://phab.getgrist.com/D3164:
- Change createParser to accept docData and one or two metadata row IDs and let it extract the metadata, so it's more easily usable in the server.
- Change ViewFieldRec.valueParser observable to a function createValueParser.
Test Plan: Existing tests.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3172
Summary:
This corrects the annotations of users who are not team members but
are guests of other documents. Their annotation was previously
blank, rather than showing as collaborators.
Test Plan: added test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3178
Summary:
The table name now comes first, making it easier to distinguish
tables coming from an Excel file with multiple sheets.
Test Plan: Tested manually and updated browser test.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3173
Summary:
This makes it so a notification about insufficient
write access is no longer shown every time a user
in a read-only document resizes a column.
Test Plan: Browser test.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3171
Summary: Also clean up dom-ownership in Charts using the new grainjs maybeOwned() method.
Test Plan: Should be no behaviour changes; existing tests should pass
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3166
Summary:
- Rather than translate from moment format to that of bootstrap-datepicker, use
the customization methods to format datepicker dates using moment directly.
- Fix issue with parseDate() when format includes tokens like Mo or Do
- Fix issue in parseDateTime() that could produce an off-by-one error in date
depending on local timezone.
- When opening DateEditor, show AltText value if present.
- Add crossorigin=anonymous to scripts that were missing it (including
bootstrap-datepicker), to ensure that errors from them are reported properly
rather than as 'Script error.'
Test Plan:
Added test cases to parseDate() test for low-level fixes; added a
browser test for the fixed DateEditor behavior.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3169
Summary:
Add more method overrides to MetaTableData for extra type safety.
Use MetaTableData, MetaRowRecord, and getMetaTable in more places.
Test Plan: Mostly it just has to compile. Tested manually that types are being checked more strictly now, e.g. by adding a typo to property names. Some type casting has also been removed.
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3168
Summary: Refactoring in preparation for parsing strings from the API. The plan is that the API code will only need to do a server-side version of the code in ViewFieldRec.valueParser (minus ReferenceUtils) which is quite minimal.
Test Plan: Nothing extra here, I don't think it's needed. This stuff will get tested more in a future diff which changes the API.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3164
Summary:
This change causes a notification about missing
write access to no longer be shown when resizing layouts
as a viewer.
Test Plan: Browser test.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3167
Summary:
The widget showing column label, ID, and a button for keeping them linked, used
to get truncated depending on the default size of inputs. This makes it
resize dynamically based on the width of the creator panel.
Test Plan: CSS-only change, tested manually on FF and Chrome.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3165
Summary:
This fixes a problem where a fork could be created, have no changes
made, and then (e.g. if worker rolled over) fail to open with a
`cannot create fork` error. Adds a test that fails priot to this diff.
Test Plan: added test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3162
Summary:
Exposing custom widgets as a dropdown menu in custom section configuration panel.
Adding new environmental variable GRIST_WIDGET_LIST_URL that points to a
json file with an array of available widgets. When not present, custom widget menu is
hidden, exposing only Custom URL option.
Available widget list can be fetched from:
https://github.com/gristlabs/grist-widget/releases/download/latest/manifest.json
Test Plan: New tests, and updated old ones.
Reviewers: paulfitz, dsagal
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3127
Summary:
Removed some TS and python code interacting with this meta table. Not touching schema or migrations.
This is not really necessary, just checking my understanding and cleaning up in preparation for raw data views. I can also remove _grist_TabItems code while I'm at it.
Test Plan: this
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3161
Summary:
Exports used to show generic message on error.
Adding error description to the message.
Test Plan: Updated tests
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3157
Summary: This reduces the size of the bundle 'vendors~GristDoc' from 587036 to 46526. Woops.
Test Plan: this
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3160
Summary:
Fixing js error that happens when closing creator panel with active formula editor.
Styling behavior menu with common styles.
Test Plan: Browser tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3150
Summary: Add doc-id attribute to copied HTML columns next to column type. Only use the raw value (rather than the display value) when the parsed doc-id from pasted HTML matches the current document ID, similar to ensuring that the type matches. This only applies to references and reflists.
Test Plan: Extended CopyPaste.ts
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3154
Summary:
Add function parseDateTime which parses a string containing both date and time componenents, intended for parsing pasted strings.
Add DateTimeParser subclass of ValueParser.
Test Plan: Extended parseDate.ts and CopyPaste.ts tests.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3152
Summary:
Grist would get stuck in a broken state in certain browsers
that enabled the bfcache when the browser back/forward
buttons were clicked. Firefox automatically disabled the cache
since we listen on the 'beforeunload' event, but Chrome and Safari
did not. This change forces a page refresh on pageshow if we
detect that Grist was loaded from the bfcache.
Test Plan: Tested manually in various browsers.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz, jarek
Differential Revision: https://phab.getgrist.com/D3151
Summary:
The express-session middleware, in its regular configuration, will
only set a cookie response header at the beginninng of a session or
when the session contents have changed. It won't set the header if
only the expiration time is changed. This diff uses a dummy `alive`
field to nudge the middleware into setting the header consistently.
Test Plan: tested manually
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3153
Summary:
- Donut charts is same as pie chart with few extra options to control size of the hole and to show/hide a big total in it.
- Add a new option type to tune a numeric options using a slider/spinner/keyboard.
- Add a new option type to tune a numeric options using a slider/keyboard
- Add a new .propWithDefault method to ObjObservable to allows to set a default value when options is undefined.
- mocha-webdriver's findContent does not work to find content in svg elements. So had to tweak original function into a sister function using .textContent instead.
Test Plan: Adds new tests
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: anaisconce, dsagal
Differential Revision: https://phab.getgrist.com/D3107
Summary:
Changes include:
* Hide the colum matching section for new destinations (for now).
* Make the preview table read-only.
* Don't show helper column IDs when the formula editor is open.
* Fix the formula editor autocomplete to show suggestions
from the active transform section.
* Hide the formula icons in the preview table, and other unnecessary
UI elements such as row dropdown menus.
* Keep preview loading spinner shown if scheduled (i.e. debounced) diff updates exist.
Test Plan: Browser tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3148
Summary:
Existing filters are now moved out of fields
and into a new metadata table for filters, and the
client is updated to retrieve/update/save filters from
the new table. This enables storing of filters for
columns that don't have fields (notably, hidden columns).
Test Plan: Browser and server tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3138
Summary:
Added ChoiceListParser capable of parsing JSON, CSVs, and other strings containing user-configured choices (e.g. separated by spaces)
I got a little carried away here. It works, and I can't think of any bugs, but it's complicated enough that there could be hidden edge cases or difficulties maintaining it in the future. The advantage of the current method is that it should work well for ambiguous or poorly formatted inputs, e.g. choices separated only by spaces or choices containing commas which are not escaped/quoted properly. The code can be vastly simplified if we don't try to support that and require that users paste proper JSON or CSVs.
Test Plan: Added a new file test/common/ChoiceListParser.ts with pure unit tests. Waiting for approval of the overall approach before adding to the nbrowser CopyPaste test.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3141
Summary:
Before:
{F37978}
After:
{F37977}
Test Plan: Add a test case for the Card view that the resulting heights are consistent.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3149
Summary: Last position should be stored for document and user.
Test Plan: Updated tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3143
Summary:
BulkAddRecord when finishing imports of nested JSON was throwing
an error due to unchecked access of referencing tables. This adds
a guard to prepare_new_values to handle such cases.
Imports happened to cause this to occur because the order that
imported tables are created/populated isn't aware of references
between tables, so it's possible for a reference column to
exist (momentarily) without a valid reference to another table.
These references are currently fixed after all imported tables are
created/populated.
Test Plan: Browser test.
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3144
Summary: Expanding search input field to full available height, to make the clickable area bigger.
Test Plan: Manual tests on browserstack
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3145
Summary:
The 'Open Access Rules' link is now hidden unless the
UserManager is opened inside a document, and the resource
that users are being managed for is a document.
Test Plan: Browser tests.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3142
Summary:
Currently when reloading a document, we may have two sqlite connections
to the document for a small period of time. This diff removes that
overlap.
Test Plan: added test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3140
Summary: Added a helper to include lots of metadata in every logging call, added and converted many logging calls.
Test Plan: Existing tests pass
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3136
Summary:
Uses the new alwaysPreserveColIds option for action summaries in Triggers.ts.
Triggers.ts is now responsible for generating the summary to make it easy to pass this option. The value of the option is just all colIds mentioned in triggers configured in this document.
Test Plan: Tested adding 200 rows to a subscribed table to ensure the events are not truncated. Also tests batching nicely.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3135
Summary:
Recent python3 changes perturbed timing again, and a few more tests started failing.
Contains an unrelated correction for gvisor running under docker (a useful configuration on macs for debugging gvisor problems, but not supported by throttling code).
Test Plan: updated tests
Reviewers: dsagal, alexmojaki
Reviewed By: dsagal, alexmojaki
Differential Revision: https://phab.getgrist.com/D3129
Summary:
The Importer dialog is now maximized, showing additional column
matching options and information on the left, with the preview
table shown on the right. Columns can be mapped via a select menu
listing all source columns, or by clicking a formula field next to
the menu and directly editing the transform formula.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3096
Summary:
Added a new object type code `l` (for lookup) which can be used in user actions as a temporary cell value in ref[list] columns and is immediately converted to a row ID in the data engine. The value contains the original raw string (to be used as alt text), the column ID to lookup (typically the visible column) and one or more values to lookup.
For reflists, valueParser now tries parsing the string first as JSON, then as a CSV row, and applies the visible column parsed to each item.
Both ref and reflists columns no longer format the parsed value when there's no matching reference, the original unparsed string is used as alttext instead.
Test Plan: Added another table "Multi-References" to CopyPaste test. Made that table and the References table test with and without table data loaded in the browser.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3118
Summary:
For Windows and Ubuntu (on chrome) texteditor was showing scrollbars for
very long words. Adding more space to fix this issue.
Test Plan: manual
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3122
Summary: title
Test Plan: Tested manually, I don't think this needs an automated test. Made a text column with a value `[1, 2]` and converted the column to choice list. Previously this threw a JS error that `tag.trim` wasn't a function. Works now, suggests `1` and `2` as choices in the configuration.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3128
Summary:
Adds links to manage team and go to billing account in
the org menu (opened by clicking the dropdown in the
top-left corner of Grist). Tweaks some wording of items
in both AppHeader and AccountWidget, and adds a link
to create a new team site to the Site Switcher in both
menus.
Also tweaks the UI of UserManager by adding
an animation when the manager is opened from the
doc access dialog.
Test Plan: Browser tests.
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3121
Summary:
If PYTHON_VERSION_ON_CREATION is set in the environment, new documents will be created with a specific desired python version (2 or 3).
This diff commits to offering a choice of engine, so the engine for a document no longer starts to initialize until the document has been fetched and read. Staging (and dev, and testing) has been like this for a while.
Test Plan: added test; manual testing of forks/copies etc
Reviewers: dsagal, alexmojaki
Reviewed By: dsagal, alexmojaki
Differential Revision: https://phab.getgrist.com/D3119
Summary:
Redesigning column type section to make it more user-friendly. Introducing column behavior concept.
Column can be either:
- Empty Formula Column: initial state (user can convert to Formula/Data Column)
- Data Column: non formula column with or without trigger (with option to add trigger, or convert to formula)
- Formula Column: pure formula column, with an option to convert to data column with a trigger.
Test Plan: Existing tests.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3092
Summary:
This marks the engine setting in document settings as experimental,
with a skull and cross-bones.
It also makes sure the setting is shown if PYTHON_VERSION_ON_CREATION
is set (this relates to a separate change to set the default python
version to 3).
Test Plan: manual
Reviewers: alexmojaki, dsagal
Reviewed By: alexmojaki, dsagal
Subscribers: anaisconce
Differential Revision: https://phab.getgrist.com/D3120
Summary:
Grist has, up to now, used a throttling mechanism that allows a sandbox free rein until it starts using above some threshold percentage of a cpu for some time - at that point, we start sending STOP and CONT signals on a duty cycle, with longer and longer STOPped periods until cpu usage is at a threshold. The general idea is to do short jobs quickly, while throttling long jobs (thus unfortunately making them even longer) in order to continue doing other short jobs quickly.
The runsc sandbox is not a single process, there are in fact 5 per sandbox in our setup. Runsc can work with kvm or ptrace. Kvm is not available to us, so we use ptrace. With ptrace, there is one process that is the appropriate one to duty cycle, and another that needs to receive a signal in order to yield. This diff adds the necessary machinery.
This is a conservative change, where I stick with our existing throttling mechanism and adapt it to the new sandbox. It would be reasonable to consider switching throttling. There's a lot the OS allows. We can set a quota for how much cpu a process can use within a given period, for example. However the overall behavior with that would be quite different to what we have, so feels like this would need more discussion.
The implementation contains use of a linux utility `pgrep` since portability is not important (runsc is only available on linux) and there's no node api for enumerating children of a process.
The diff contains some tweaks to `buildtools/contain.sh` to streamline experimenting with Grist and runsc on a mac. It is important for throttling that node and the sandbox processes are in the same process name space, if docker is in between them then some extra machinery is needed (a proxy throttler and a way to communicate with it) which I chose not to implement.
Test Plan: added test; a lot of manual testing
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3113
Summary:
The /assign endpoint checks if a document is on the desired worker
and moves it if not. This is never done under regular operation, but
is useful when quarantining a misbehaving document.
The endpoint was failing to operate correctly if the requester did
not have access to the document. This diff makes the endpoint
accessible through a /housekeeping route, using the same pattern as
the /force-reload endpoint.
Test Plan: added test
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3109
Summary:
- Change "Continue" button to "Review" (we don't charge immediately,
first show a review screen)
- Show more informative messages for certain failures with discount
coupons.
- Focus form elements with error, or at least the part of the form
containing an error.
- Auto-focus discount input box when it gets toggled on.
- Show warning about URL changes only when subdomain is changed.
Test Plan: Updated tests; tested focus and changed error messages manually.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3115
Summary:
Pasting data like `A\nB\tC` was failing because the first row, used for
checking column type (to handle rich data), was failing on an undefined value.
Discovered while trying out the fix in https://phab.getgrist.com/D3110.
Test Plan: Tested manually. The case mentioned now works as expected.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3111
Summary: title
Test Plan:
Tested manually. Blank rows at the end are no longer pasted. Pasting multiple columns separated by tabs can still have blank cells in some cells of the final rows.
I don't think this needs an automated test.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3110
Summary:
Push webhook events to redis queue with key based on docId.
Remove events from redis after sending using LTRIM.
Put failed events back on the end of the queue under normal circumstances.
When the event queue gets too long:
- Wait until it gets consumed before continuing.
- Drop failed events (i.e. don't put them back on the end of the queue)
- Limit webhook retries to 5
Test Plan: Tested that interactions with redis are as expected using redis MONITOR command.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3100
Summary: Allows any timezone abbreviation associated with the given timezone, and simply ignores it. Previously only certain abbreviations worked and they were not unique so using them outside the US was broken.
Test Plan: Added parseDate tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3106
Summary:
Adding sort options for columns.
- Sort menu has a new option "More sort options" that opens up Sort left menu
- Each sort entry has an additional menu with 3 options
-- Order by choice index (for the Choice column, orders by choice position)
-- Empty last (puts empty values last in ascending order, first in descending order)
-- Natural sort (for Text column, compares strings with numbers as numbers)
Updated also CSV/Excel export and api sorting.
Most of the changes in this diff is a sort expression refactoring. Pulling out all the methods
that works on sortExpression array into a single namespace.
Test Plan: Browser tests
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: dsagal, alexmojaki
Differential Revision: https://phab.getgrist.com/D3077
Summary: Handle reflist columns in ViewFieldRec.parseValue
Test Plan: Reused section of test of reference columns
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3105
Summary:
- Grouping series may result in series with inconsistent number of values. This can result in inconsistent ordering of the bars displayed by plotly.
- This diff fixes it by consolidating grouped series by adding unll values for each missing xvalues in the series.
Here a is a minimal example of that bug:
{F36639}
Test Plan: Includes new nbrowser test.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3085
Summary:
Handle reference columns in ViewFieldRec.valueParser.
Extracted code for reuse from ReferenceEditor to look up values in the visible column. While I was at it, also extracted a bit of common code from ReferenceEditor and ReferenceListEditor into a new class ReferenceUtils. More refactoring could be done in this area but it's out of scope.
Changed NTextEditor to use field.valueParser, which affects numeric and reference fields. In particular this means numbers are parsed on data entry, it doesn't change anything for references.
Test Plan:
Added more CopyPaste testing to test references.
Tested entering slightly formatted numbers in NumberFormatting.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3094
Summary:
Searching with the keyboard shortcut selected the previous search text in the
search box, but using the Search icon did not. A user reported it as an
inconvenience: having to manually delete the value before searching for a new
one.
Test Plan: Verified manually
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: georgegevoian
Differential Revision: https://phab.getgrist.com/D3102
Summary:
The document collecting new user info
(https://docs.getgrist.com/doc/GristNewUserInfo) got very slow, taking 40+
seconds for cold open. Sign-up submissions had to wait this time to proceed to
next step, because they waited for the write to this doc, which was blocked on
the Calculate action to complete.
Two changes were made: one to remove all expensive columns and summaries in the
actual doc, so the doc is back to opening in single seconds, and times should
be acceptable now.
The second change is this diff: to avoid waiting for the write step, so that it
doesn't affect users even if it gets slow again.
Test Plan: Existing test continues to work with a minor reliability tweak.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3103
Summary:
Action summaries by default will drop rows in bulk changes, keeping only a few of them as examples. This diff allows overriding that, or selectively preserving some columns in their entirety.
This is intended for use with webhooks.
Test Plan: added test
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3098
Summary:
Shares of the same role (e.g. viewer) at different levels could interact for a resource (e.g. a doc) shared with everyone@, potentially blocking the listing of that resource. This diff removes the interaction.
The permission of a user on a resource is calculated by finding all acl rules that link that resource to a group to which the user belongs, or to a group that has a subgroup to which the user belongs, etc, and then bitwise-or-ing the permissions on the acl rules. A later wrinkle was to allow public sharing via special users. A still later wrinkle was to avoid listing resources if they were only shared with the special everyone@ user, while allowing access to them if user has their full link. That wrinkle had a bug, where if e.g. a doc were shared with everyone@ as a viewer, and the org the doc was in was shared with someone@ as a viewer, and the doc inherited the org permissions via a workspace, then that doc would end up not being listed.
The fix is straightforward enough, but needs different code for postgres and sqlite, and is a bit verbose because we unwrap subgroups to a few levels rather than doing recursion (which looks cleaner but was slower in benchmarks).
Test Plan: added test that fails without this fix
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3095
Summary: Adds parseDateStrict function based on parseDate, uses it in DateParser subclass of ValueParser.
Test Plan:
Tweaked parseDate test to check parseDateStrict.
Extended test in CopyPaste to test parsing dates as well as numbers.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3088
Summary:
- This should make these easier to work with and make changes to.
- Removes one unused method.
Test Plan: No changes of behavior, existing tests should pass.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3091
Summary:
This gives more guidance to users when editing document shares in the UserManager dialog.
* For a document on a team site, any shares with team members are marked `Team member`.
* Shares that count as external collaborators are marked for documents on a team or personal site as `collaborator` (personal site) or `outside collaborator` (team site).
* Collaborators are marked `1 of 2`, `2 of 2`, and then `limit exceeded`.
* On a team site, links are offered for each collaborator to add them to the team. The links lead to a prefilled dialog for managing team membership which can be confirmed immediately, allowing the user to continue without interruption.
* On a personal site, for the last collaborator and beyond, a link is added for creating a team. This isn't seamless since creating a team involves billing etc.
There's a small unrelated tweak in tests to remove a confusing import from `test/browser` in `test/server`.
One thing I didn't get to is checking if owner of doc is owner of site. If they aren't, they may try to add a member and be denied at that point - it would be more polite to change messaging earlier for them.
Test Plan: added and updated tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3083
Summary:
- Sharing, Client, DocClients, HostingStorageManager all include available info.
- In HostingStorageManager, log numSteps and maxStepTimeMs, in case that helps
debug SQLITE_BUSY problem.
- Replace some action-bundle logging with a JSON version aggregating some info.
- Skip logging detailed list of actions in production.
Test Plan: Tested manually by eyeballing log output in dev environment.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3086
Summary:
Add ValueParser file, base class, and subclasses for column types. Only NumericParser is used for now.
Add valueParser field to ViewFieldRec.
Use valueParser when parsing pasted text data in Grid and Detail views.
Test Plan: Add test to nbrowser CopyPaste suite, copying into a numeric column with different currency and locale settings.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3082
Summary:
Record numbers of rows, columns, cells, and bytes of marshalled data for most calls to table_data_from_db
Export new function get_table_stats in the sandbox, which gives the raw numbers and totals.
Get and log these stats in ActiveDoc right after loading tables, before Calculate, so they are logged even in case of errors.
Tweak logging about number of tables, especially number of on-demand tables, to not only show in debug logging.
Test Plan: Updated doc regression tests, that shows what the data looks like nicely.
Reviewers: dsagal, paulfitz
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3081
Summary:
New plan signups now include a discount code field in
the signup form. If a valid discount code is entered, a
discount will be applied on the confirmation page.
Test Plan: Browser and server tests.
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3076
Summary:
Added NumberParse.ts, counterpart of NumberFormat.ts.
Contains generic functionality for parsing numbers formatted by Intl.NumberFormat, not tied to documents or anything.
This doesn't change any actual behaviour, applying this parsing when pasting/typing in numeric columns will be a separate diff.
Test Plan: New file with extensive unit tests.
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3078
Summary:
This verifies that all existing tests are capable of running under python3/gvisor, and fixes the small issues that came up. It does not yet activate python3 tests on all diffs, only diffs that specifically request them.
* Adds a suffix in test names and output directories for tests run with PYTHON_VERSION=3, so that results of the same test run with and without the flag can be aggregated cleanly.
* Adds support for checkpointing to the gvisor sandbox adapter.
* Prepares a checkpoint made after grist python code has loaded in the gvisor sandbox.
* Changes how `DOC_URL` is passed to the sandbox, since it can no longer be passed in as an environment variable when using checkpoints.
* Uses the checkpoint to speed up tests using the gvisor sandbox, otherwise a lot of tests need more time (especially on mac under docker).
* Directs jenkins to run all tests with python2 and python3 when a new file `buildtools/changelogs/python.txt` is touched (this diff counts as touching that file).
* Tweaks miscellaneous tests
- some needed fixes exposed by slightly different timing
- a small number actually give different results in py3 (removal of `u` prefixes).
- some needed a little more time
The DOC_URL change is not the ultimate solution we want for DOC_URL. Eventually it should be a variable that gets updated, like the date perhaps. This is just a small pragmatic change to preserve existing behavior.
Tests are run mindlessly as py3, and for some tests it won't change anything (e.g. if they do not use NSandbox). Tests are not run in parallel, doubling overall test time.
Checkpoints could be useful in deployment, though this diff doesn't use them there.
The application of checkpoints doesn't check for other configuration like 3-versus-5-pipe that we don't actually use.
Python2 tests run using pynbox as always for now.
The diff got sufficiently bulky that I didn't tackle running py3 on "regular" diffs in it. My preference, given that most tests don't appear to stress the python side of things, would be to make a selection of the tests that do and a few wild cards, and run those tests on both pythons rather then all of them. For diffs making a significant python change, I'd propose touching buildtools/changelogs/python.txt for full tests. But this is a conversation in progress.
A total of 6886 tests ran on this diff.
Test Plan: this is a step in preparing tests for py3 transition
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3066
Summary:
Adding validation for api /records endpoint, that checks if the json payload is valid.
Modifying POST /records endpoint to allow creating blank or partial records.
Test Plan: Updated tests
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3061
Summary:
- Puts events on a queue in memory and ensures they are sent in the order they were generated.
- Makes the caller (Sharing.ts) wait until changed records have been fetched from the DB, but allows it to continue after while remaining work happens asynchronously.
- Gathers all new webhook events into an array so they can be backed up to the queue on redis in a single command (in a future diff).
- Uses changes in isReady to determine event type, no more 'existed before'
The structure of the code has changed a lot, so I think the scope of the diff needs to stop here. Lots of work is still deferred in TODOs.
Test Plan: Updated existing test. Actually dropped testing of retry on failures and slowness because it no longer made sense to keep that as part of the current test, so a new test for that will be added in a future diff.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3074
Summary:
Plotly sorts pie charts sectors by default and that is overiding the
section ordering. This diff fixes that by passing setting .sort to
false (thus disabling reordering) when there is a sort spec going on.
Issue was reported by user: https://gristlabs.getgrist.com/k1f3bMzUvitZ/User-Feedback#a1.s3.r333.c19
Test Plan: Added nbrowser test
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3075
Test Plan: Added a test case to verify that nav buttons are now shown and work.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3073
Summary:
When there is a link in a text cell (and formula cells), it will be
rendered with a little clickable icon wrapped in the anchor tag
with a proper link. Only links that starts with https? will be
rendered as links.
Links are shown in a Text and Formula fields, inside a GridView,
CardView and in the Import preview dialog.
Test Plan: Browser tests
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: dsagal, alexmojaki
Differential Revision: https://phab.getgrist.com/D3070
Summary: - Added functionality to modal.ts to allow pending work to delay the closing of the dialog.
Test Plan: Added a test case that tickled a failure previously.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3071
Summary: Add sanitizeWorksheetName function, pass result to library function addWorksheet where error was raised.
Test Plan: Added unit test for sanitizeWorksheetName function, updated a fixture document to use a messy table name.
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3072
Summary: Makes type checking a bit stronger
Test Plan: it just has to compile
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3065
Summary:
Updates the preview table in Importer to show a diff of changes
when importing into an existing table and updating existing records.
Test Plan: Browser tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3060
Summary:
The users shown by the "View As" button are now drawn from more sources:
* There are users the document is shared with. This has been rationalized, the behavior was somewhat erratic. If the user is not an owner of the document, the only user of this kind that will be listed is themselves.
* There are users mentioned in any user attribute table keyed by Email. If name and access columns are present, those are respected, otherwise name is taken from email and access is set to "editors".
* There are example users provided if there are not many other users available.
Test Plan: added and extended tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3045
Summary:
When filtering document updates to send to clients after a change,
censorship of individual cells was being applied to state shared
across the clients. This diff eliminates that shared state, and
extends testing of broadcasts to check different orderings.
Test Plan:
extends a test to tickle a reported bug, and gives
DocClients a knob to control message order needed to tickle
the bug reliably.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3064
Summary:
docker is slow on macs, so use native sandbox-exec by
default for tests involving python3 on macs.
Test Plan: updated test
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3068
Summary:
Removing error styles from user messages. Only unexpected errors are styled with red icon and border.
Removing reportSuccess message - leaving it for another diff.
Test Plan: manual tests
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3063
Summary:
Allowing a user to change labels' in Choice/ChoiceList entry editor. For updated
entries, renaming those values in all cells in the column.
Test Plan: Updated tests
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3057
Summary:
After an incomplete import, any GristHidden_* tables will show up in the page
list, but may not be removable if there is only one non-hidden table remaining.
Such tables should still be removable in this case.
Test Plan: Added a test case
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3058
Summary:
Styling toast notification. Adding colors and icons.
In Grist, changed the default style for errors (will be shown in red), and a style for
Linked copied to clipboard (will be shown in Green).
All other colors are not used currently, left for another diff.
Test Plan: manual
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3053
Summary:
Searching for matching choices was using nativeCompare for a binary search,
while the list was sorted according to localeCompare. This was causing the
search to fail for some strings when the two orderings differ.
Test Plan: Tested manually
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3059
Summary:
Finishing imports now occurs in Node instead of the
data engine, which makes it possible to import into
on-demand tables. Merging code was also refactored
and now uses a SQL query to diff source and destination
tables in order to determine what to update or add.
Also fixes a bug where incremental imports involving
Excel files with multiple sheets would fail due to the UI
not serializing merge options correctly.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3046
Test Plan: Only tested manually that path is included.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3056
Summary:
This makes the `core` test operate on a directory outside the
jenkins workspace, so that packages in the workspace don't
interfere with the test and obscure errors.
It also includes a small type fix for the `core` build.
Test Plan: updating a test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3054
Summary:
- Update cookie module, to support modern sameSite settings
- Add a new cookie, grist_sid_status with less-sensitive value, to let less-trusted subdomains know if user is signed in
- The new cookie is kept in-sync with the session cookie.
- For a user signed in once, allow auto-signin is appropriate.
- For a user signed in with multiple accounts, show a page to select which account to use.
- Move css stylings for rendering users to a separate module.
Test Plan: Added a test case with a simulated Discourse page to test redirects and account-selection page.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3047
Summary:
With this diff, when a user opens a Grist document in a browser, they will be able to view its contents without waiting for the data engine to start up. Once the data engine starts, it will run a calculation and send any updates made. Changes to the document will be blocked until the engine is started and the initial calculation is complete.
The increase in responsiveness is useful in its own right, and also reduces the impact of an extra startup time in a candidate next-generation sandbox.
A small unrelated fix is included for `core/package.json`, to catch up with a recent change to `package.json`.
A small `./build schema` convenience is added to just rebuild the typescript schema file.
Test Plan: added test; existing tests pass - small fixes needed in some cases because of new timing
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3036
Summary:
New environmental variable GOOGLE_DRIVE_SCOPE that modifies the scope
requested for Google Drive integration.
For prod it has value https://www.googleapis.com/auth/drive.file which leaves
current behavior (Grist is allowed only to access public files and for private
files - it fallbacks to Picker).
For staging it has value https://www.googleapis.com/auth/drive.readonly which
allows Grist to access all private files, and fallbacks to Picker only when the file is
neither public nor private).
Default value is https://www.googleapis.com/auth/drive.file
Test Plan: manual and existing tests
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3038
Summary:
Bug reported by user: https://gristlabs.getgrist.com/doc/check-ins/p/3#a1.s7.r1183.c19p
Setting x axis to a column of type ChoiceList was breaking chart.
This diff fixes that by splitting the record into several records: one for each choice.
`test/nbrowser/ChartView1.ts` was becoming too big and long to run, so this diff introduces `test/nbrowser/ChartView2.ts` to add more test and `test/nbrowser/chartViewTestUtils.ts` to put all utilities or testing charts.
Test Plan: Adds new test.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3041
Summary:
This adds a `user:delete` target to the `cli.sh` tool. The desired user will be deleted from our database, from sendgrid, and from cognito.
There is code for scrubbing the user from team sites, but it isn't yet activated, I'm leaving finalizing and writing tests for it for follow-up.
Test Plan: tested manually
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3043
Summary:
This removes the need for any information drawn from the content of recent actions when loading a document.
The undo/redo system does need some facts about recent actions up front. But that system has an important restriction: only actions a particular client is known to have generated can be undone by that client.
So in this diff, as we store which client has performed an action, we also store the few pieces of metadata about that action that the undo/redo system needs: `linkId`, `otherId`, `rowIdHint`, `isUndo` fields. These are all small integers (or in one case a boolean).
An existing limitation is that information about which client has performed which action is stored in memory in the worker, and not persisted anywhere. This diff does not change that limitation, meaning that undos continue to not survive a worker transition. A reasonable way to deal with that would be to back the store with redis.
Test Plan: existing tests pass
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3044
Summary: There was a bad regex processing the document url passed to the sandbox.
Test Plan: added test
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3048
Summary:
Tests DocApi endpoints _subscribe and _unsubscribe, including various bad inputs.
Tests that webhooks are sent to a test express server, with retrying on failure, filtered by event type, and waiting for isReadyColumn.
Test Plan: this
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3042
Summary:
Traceback is available on the Creator Panel in the formula editor. It is evaluated the same way as for normal formulas.
In case when the traceback is not available, only the error name is displayed with information that traceback is not available.
Cell with an error, when edited, shows the previous valid value that was used before the error happened (or None for new rows).
Value is stored inside the RaisedException object that is stored in a cell.
Test Plan: Created tests
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki, dsagal
Differential Revision: https://phab.getgrist.com/D3033
Summary: Adding more locale codes to support more countries in document settings
Test Plan: existing tests
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3018
Summary: Update _create_syntax_error_code to raise an error with similar arguments to the real arguments it already has, with our modifications.
Test Plan: Updated python unit tests
Reviewers: jarek, dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3040
Summary:
- Anchor links with row of 'new' could be created but weren't parsed or used
correctly. This fixes it.
- Also adds UIRowId type for row IDs which includes the special 'new' row. It's
already been used in places as `number|'new'`, this diff gives it a name usable in app/common
(it doesn't touch another name, RowId, that's been available in app/client).
Test Plan: Added a test assert for anchor links to new row
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3039
Summary:
See https://grist.quip.com/VKd3ASF99ezD/Outgoing-Webhooks
- 2 new DocApi endpoints: _subscribe and _unsubscribe, not meant to be user friendly or publicly documented. _unsubscribe should be given the response from _subscribe in the body, e.g:
```
$ curl -X POST -H "Authorization: Bearer 8fd4dc59ecb05ab29ae5a183c03101319b8e6ca9" "http://localhost:8080/api/docs/6WYa23FqWxGNe3AR6DLjCJ/tables/Table2/_subscribe" -H "Content-type: application/json" -d '{"url": "https://webhook.site/a916b526-8afc-46e6-aa8f-a625d0d83ec3", "eventTypes": ["add"], "isReadyColumn": "C"}'
{"unsubscribeKey":"3246f158-55b5-4fc7-baa5-093b75ffa86c","triggerId":2,"webhookId":"853b4bfa-9d39-4639-aa33-7d45354903c0"}
$ curl -X POST -H "Authorization: Bearer 8fd4dc59ecb05ab29ae5a183c03101319b8e6ca9" "http://localhost:8080/api/docs/6WYa23FqWxGNe3AR6DLjCJ/tables/Table2/_unsubscribe" -H "Content-type: application/json" -d '{"unsubscribeKey":"3246f158-55b5-4fc7-baa5-093b75ffa86c","triggerId":2,"webhookId":"853b4bfa-9d39-4639-aa33-7d45354903c0"}'
{"success":true}
```
- New DB entity Secret to hold the webhook URL and unsubscribe key
- New document metatable _grist_Triggers subscribes to table changes and points to a secret to use for a webhook
- New file Triggers.ts processes action summaries and uses the two new tables to send webhooks.
- Also went on a bit of a diversion and made a typesafe subclass of TableData for metatables.
I think this is essentially good enough for a first diff, to keep the diffs manageable and to talk about the overall structure. Future diffs can add tests and more robustness using redis etc. After this diff I can also start building the Zapier integration privately.
Test Plan: Tested manually: see curl commands in summary for an example. Payloads can be seen in https://webhook.site/#!/a916b526-8afc-46e6-aa8f-a625d0d83ec3/0b9fe335-33f7-49fe-b90b-2db5ba53382d/1 . Great site for testing webhooks btw.
Reviewers: dsagal, paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3019
Summary: Update LinkingState._makeSrcCellGetter to account for 'new'
Test Plan: Extended test in RightPanelSelectBy.ts
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3037
Summary:
Used to throw js errors: `Resize must be passed a displayed plot div element.`
Summarizing (or unsummarizing) causes the ChartView view instance to
be replace by a new one in the view layout.
However, the problem is that the old view instance get disposed only
after the new view instance is added to the view layout.
This causes the old view layout to try to resize chart while chart dom
has been removed from the dom (which Plotly does not support).
This diff fixes it by checking the the chart dom elemnt is still in
the dom before making the plotly call to resize the chart.
TODO: It feels weird that the old view instance gets disposed after
the new one get added. Maybe we should check that also.
Test Plan: New test added.
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3035
Test Plan: Added a test with lots of time-parsing cases.
Reviewers: jarek
Reviewed By: jarek
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3034
Summary: Constructs a ClientQuery in a similar way to LinkingState to handle lists.
Test Plan: Extended SelectBySummary and SelectByRefList tests.
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3030
Summary:
While switching the destination site in the Duplicate Document dialog, there
were times when it was saveable even though destination workspaces were still
being fetched. This sometimes causes a test failure, with the document getting
saved to a workspace from the previously-selected org.
Test Plan:
Tested manually; reproduced by adding a conditional artificial delay
in _updateWorkspaces helper.
Reviewers: paulfitz, georgegevoian
Reviewed By: paulfitz, georgegevoian
Differential Revision: https://phab.getgrist.com/D3032
Summary:
Also fixes issue with group data options when switching to pie chart.
Issue was that if the group data picker was on, switching to the pie
chart was not hiding it.
Test Plan: Adds more tests.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3028
Summary:
This unsets the `direct` flag for actions emitted when summary tables are updated. That means those actions will be ignored for access control purposes. So if a user has the right to change a source table, the resulting changes to the summary won't result in the overall action bundle being forbidden.
I don't think I've actually seen the use case that inspired this issue being filed. I could imagine perhaps a user forbidden from creating rows globally making permitted updates that could add rows in a summary (and it being desirable to allow that).
Test Plan: added tests
Reviewers: jarek
Reviewed By: jarek
Subscribers: dsagal, alexmojaki, jarek
Differential Revision: https://phab.getgrist.com/D3022
