Commit Graph

756 Commits

Author SHA1 Message Date
Vincent Viers
2f48be26b1 fix(ui): add default cursor over disabled items 2022-08-24 09:47:47 +02:00
George Gevoian
ed37401b2c (core) Add basic activation page to grist-ee
Summary:
Adds an activation page to grist-ee that currently shows activation status.

Follow-up diffs will introduce additional functionality, such as the ability to
enter activation keys directly from the activation page.

Test Plan: No grist-ee tests (yet).

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal, paulfitz

Differential Revision: https://phab.getgrist.com/D3582
2022-08-23 10:30:52 -07:00
Vincent Viers
5371ebcf46 fix(ui): pointer cursor when creating a new widget 2022-08-23 14:30:52 +02:00
Paul Fitzpatrick
028146f88a (core) updates from grist-core 2022-08-22 10:23:26 -04:00
Jarosław Sadziński
a43a5a25a6 (core) Removing old billing landing page.
Summary: Old landing page /docs/billing/signup is not used anymore.

Test Plan: Updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3585
2022-08-22 07:37:13 +02:00
Alex Hall
42060df29a (core) Formula autocomplete improvements for references and lookups
Summary:
Makes the following improvements to formula autocomplete:

- When a user types `$RefCol` (or part of it), also show `$RefCol.VisibleCol` (replace actual column names) in the autocomplete even before the `.` is typed, to help users understand the difference between a raw reference/record and its visible column.
- When a user types a table name, show `.lookupOne` and `.lookupRecords` in the autocomplete, again even before the `.` is typed.
- For `.lookupRecords(` and `.lookupOne(`, once the `(` is entered, suggest each column name as a keyword argument.
- Also suggest lookup arguments involving compatible reference columns, especially 'reverse reference' lookups like `refcol=$id` which are very common and difficult for users.
- To support these features, the Ace editor autocomplete needs some patching to fetch fresh autocomplete options after typing `.` or `(`. This also improves unrelated behaviour that wasn't great before when one column name is contained in another. See the first added browser test.

Discussions:

- https://grist.slack.com/archives/CDHABLZJT/p1659707068383179
- https://grist.quip.com/HoSmAlvFax0j#MbTADAH5kgG
- https://grist.quip.com/HoSmAlvFax0j/Formula-Improvements#temp:C:MbT3649fe964a184e8dada9bbebb

Test Plan: Added Python and nbrowser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3580
2022-08-20 19:11:41 +02:00
Jarosław Sadziński
758c87f002 (core) Adding hide for multiple columns
Summary: New column menu option to hide multiple columns.

Test Plan: new test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3581
2022-08-17 22:31:16 +02:00
Ronan Amicel
cd16650e0e Rename text cleaning function to "normalizeText" 2022-08-17 19:11:52 +02:00
Ronan Amicel
4003d2d7c6 Use lodash to remove accents and ther diacritics
https://lodash.com/docs/4.17.15#deburr
2022-08-17 19:11:52 +02:00
Ronan Amicel
2a05d04e35 Ignore diacritics in autocomplete
Works for:
- Choice
- Choice List
- Reference
- Reference List

Co-Authored-By: Louis Delbosc <louis.delbosc.prestataire@anct.gouv.fr>
2022-08-16 09:20:08 +02:00
George Gevoian
3ad78590c2 (core) Polish display of table row counts
Summary:
Adds a new dots loader, which is used on the Raw Data page when
certain values are still being calculated (e.g. row counts). Now metrics
whose values aren't known yet will still appear under Usage, but with a
"Loading" message and the dots loader shown. For per-table row counts,
we only show the loader.

Test Plan: Existing tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3566
2022-08-11 09:45:33 -07:00
Jarosław Sadziński
f215ec8b24 (core) Editing text for viewers on empty site
Summary: Small tweaks for text that viewers see on empty site.

Test Plan: Updated tests.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3577
2022-08-09 22:24:54 +02:00
Jarosław Sadziński
9e4d802405 (core) Implementing row conditional formatting
Summary:
Conditional formatting can now be used for whole rows.
Related fix:
- Font styles weren't applicable for summary columns.
- Checkbox and slider weren't using colors properly

Test Plan: Existing and new tests

Reviewers: paulfitz, georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3547
2022-08-09 20:11:36 +02:00
Jarosław Sadziński
ee109e9186 (core) Removing NEW_DEAL flag
Summary:
Removing NEW_DEAL flag checks and exposing all new deal features as default.
Also removing Pages.ts as it was moved to grist-core.

Test Plan: Existing and updated tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3572
2022-08-09 17:29:28 +02:00
Dmitry S
787b70f0f6 (core) Upgrade to latest plotly, and set automargin on Y-Axis
Test Plan: Only tested manually

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3543
2022-08-08 14:47:19 -04:00
Paul Fitzpatrick
c72ebf61a7 (core) updates from grist-core 2022-08-08 09:31:07 -04:00
Louis Delbosc
a82d02ebcc feat: add tests for feature 2022-08-08 13:30:12 +02:00
Louis Delbosc
531ea1a4b5 feat: add rename when clicking on selected label page 2022-08-04 14:18:29 +02:00
George Gevoian
1c6ab775dd (core) Polish upgrade button
Summary:
On mobile, clicking the upgrade button will now immediately display
the plans modal. The button margins have also been adjusted to be
smaller on mobile. Finally, some disabled options related to workspace
sharing in the left panel (on personal sites) are now hidden instead.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3555
2022-08-03 08:29:36 -07:00
George Gevoian
771e1edd54 (core) Keep track of row counts per table
Summary: Displays a live row count of each table on the Raw Data page.

Test Plan: Browser tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3540
2022-08-03 08:13:33 -07:00
Jarosław Sadziński
40c9b8b7e8 (core) New URL that opens Create site popup.
Summary: Adding new url parameter for team site creation

Test Plan: Updated tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3554
2022-08-03 13:09:18 +02:00
Jarosław Sadziński
c359547f6b (core) Welcome intro for viewers on a team site.
Summary:
Adding intro for a viewer on a teamsite.
Showing upgrade button for owners only.

Test Plan: new test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3557
2022-08-03 12:50:14 +02:00
Alex Hall
b7686fa664 (core) Add 'Show raw data' widget menu option
Summary:
Adds a new option to the top of the three dot widget menu which navigates to the raw data widget at the current cell, equivalent to clicking an anchor link.

Clicking the back button returns to the previous view. Clicking the close button shows the list of raw data tables instead, which isn't great, but the same is true when clicking an anchor link.

Test Plan: Added a test to `nbrowser/ChartView1.ts` because charts are the main thing we want this for.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3539
2022-08-02 20:01:44 +02:00
George Gevoian
d12b05abcb (core) Keep focus when canceling choice rename
Summary:
Small bug fix that ensures focus isn't lost when Escape is pressed
while renaming a choice. This now makes it possible to close the
editor with a subsequent press of Escape.

Test Plan: Browser test.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3556
2022-08-01 12:08:36 -07:00
Paul Fitzpatrick
4c678f12cb (core) dust off electron build a little bit
Summary:
The changes in this diff are sufficient to make this sequence work again:

```
./build electron-dev
bin/electron app/electron/runPrebuild.js
```

This brings up the local server within an electron window.

This is an unambitious diff, aimed at checking how rusty electron support had become. It does not revive Grist as a packaged electron app. The first substantial work needed would be to make the app aware of the local file system again, and think through how local files should be visualized and accessed now. In the past, there was a simple list of grist docs in a directory.

Test Plan: manual

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3534
2022-07-29 11:19:26 -04:00
Cyprien P
80f31bffc2 (core) Allow left pane to auto-expand on mouse over
Summary:
Tweak PagePanels to let the left pane automatically expand on mouse
over. This is to make pages more accessible when the panel is
collapsed.

In this context, when expanding, the left panel overlap the main
content, reducing visual clutter.

Test Plan: updated

Reviewers: jarek

Reviewed By: jarek

Subscribers: anaisconce, jarek

Differential Revision: https://phab.getgrist.com/D3516
2022-07-28 11:41:42 +02:00
George Gevoian
c54dde3dba (core) Populate doc title, description and thumbnail in app.html
Summary:
Fills in the title and description/thumbnail (for templates) in app.html if the
page being requested is for a document.

Test Plan: Tested manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3544
2022-07-27 13:57:59 -07:00
Jarosław Sadziński
364715eba3 (core) FormulaEditor was updating formula when nothing has changed
Summary:
FormulaEditor on sidepanel was sending update action when
formula wasn't updated by the user.

Test Plan: Existing tests

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3545
2022-07-27 11:23:18 +02:00
George Gevoian
aeba738f7c (core) Add product for new personal plan
Summary:
Adds the new personal plan as a product that will be available
in the future. Can be enabled along with other plan-related via
an environment variable.

Test Plan: Browser tests and existing tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3533
2022-07-26 11:33:23 -07:00
Jarosław Sadziński
5c8211c61d (core) Fixing selectby error on the creator panel.
Summary: [Select By] in the creator panel was bugged. It wasn't refreshed in some cases as the observable array that needed to be created seemed too complicated. This Diff recomputes this array when the user wants to change the selection.

Test Plan: added tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3541
2022-07-26 15:46:11 +02:00
Alex Hall
460f22b701 (core) Don't show hidden tables in table data options when adding widgets
Summary:
As reported in https://grist.slack.com/archives/C069RUP71/p1655316194602829, when a table is hidden by ACL, it was still showing as a blank 'ghost' option to select data when adding a widget:

{F55498}

The fix is simply to return `true` from `isHiddenTable` for empty table IDs, which indicate a table hidden by ACL. `TableRec.isHidden` is supposed to match this so I updated it too, and I cleaned up a tiny bit of other related code.

Test Plan: Extended `nbrowser/AccessRules1.ts` to test the data options when adding widgets.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3530
2022-07-21 16:28:28 +02:00
Dmitry S
4e805a4d9c (core) Fix sizing of tracebacks in formula errors, to make it scrollable
Summary: When traceback is present, give it 64px, or more if available, or less if less is needed. If less space is available than needed, the traceback will scroll within its allocated area.

Test Plan: The test FieldEditorSizing which tests basic sizing still passes; details with different size of formula and traceback were tested manually.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3531
2022-07-21 09:56:52 -04:00
Paul Fitzpatrick
dd8d2e18f5 (core) add an access token mechanism to help with attachments in custom widgets
Summary:
With this, a custom widget can render an attachment by doing:
```
const tokenInfo = await grist.docApi.getAccessToken({readOnly: true});
const img = document.getElementById('the_image');
const id = record.C[0];  // get an id of an attachment
const src = `${tokenInfo.baseUrl}/attachments/${id}/download?auth=${tokenInfo.token}`;
img.setAttribute('src', src)
```

The access token expires after a few mins, so if a user right-clicks on an image
to save it, they may get access denied unless they refresh the page. A little awkward,
but s3 pre-authorized links behave similarly and it generally isn't a deal-breaker.

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3488
2022-07-19 11:55:18 -04:00
George Gevoian
5c0a250309 (core) Prevent raw table names from overflowing
Summary:
Modifies CSS so that text overflow is handled by displaying an
ellipsis when raw table names exceed the width of their container.

Test Plan: Tested manually.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3524
2022-07-19 08:25:31 -07:00
George Gevoian
3e49fe9a50 (core) Polish ChoiceListEntry drag and drop
Summary:
A green line indicating the insertion point is now shown in the
ChoiceListEntry component when dragging and dropping choices, similar
to the one shown in the choice list cell editor.

Test Plan: Tested manually.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3529
2022-07-19 08:14:04 -07:00
Alex Hall
4b258ae0fa (core) Fix 'select by' when adding summary table widget to page
Summary:
When adding a summary table widget to a page and using 'select by' in the add widget config (as opposed to in the right panel for an existing widget):

1. If an equivalent summary table already exists, use its referencing columns (if any) to construct link nodes. Previously the source table columns were being used instead, which could include referencing columns that don't have any equivalent in the summary table, and exclude referencing columns in the summary table.
2. If no such summary table exists yet, then keep using the source table columns, but only the selected groupby columns, and
3. After the summary table is created, correct the `linkTargetColRef` (which points to a source table column) to the corresponding column from the new summary table instead.

This fixes bugs which only appeared recently since 'select by' for a summary table previously involved no target columns.

Test Plan: Added two new tests to `nbrowser/SelectBySummaryRef`, and confirmed that they fail without the fixes to all three points above.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3527
2022-07-19 16:28:38 +02:00
Alex Hall
f39b496563 (core) Use table title instead of ID in ACL UI
Summary:
Use table titles (i.e. the raw data widget titles) in dropdowns and other parts of the Acess Rules page, instead of the table ID. This is particularly meant for summary tables which have/had an ID of the form `GristSummary_SourceTable_N`, but https://phab.getgrist.com/D3508 is changing that anyway.

The server method `getAclResources` now returns more metadata about each table so that the UI can display titles.

Test Plan: Extended and updated `nbrowser/AccessRules2.ts`. Added a small unit test for constructing table titles from the new description returned by `getAclResources`.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3494
2022-07-19 16:27:17 +02:00
Jarosław Sadziński
257dafe423 (core) Improving billing page user experience
Summary:
Improving billing page user experience.
- Updated labels for canceled plan
- Adding option to downgrade from team plan to free team plan
- Updating default name for teamFree plan when it is not available in Stripe
- Minor bug fixes

Test Plan: updated tests

Reviewers: cyprien

Reviewed By: cyprien

Subscribers: cyprien

Differential Revision: https://phab.getgrist.com/D3515
2022-07-18 14:43:14 +02:00
George Gevoian
c70b427483 (core) Fix error when filtering empty choice and choice list columns
Summary:
A client error should no longer be thrown when filtering an empty Choice
or Choice List column.

Test Plan: Browser tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3528
2022-07-17 23:35:27 -07:00
George Gevoian
c0852761b9 (core) Fix cell selection bugs
Summary:
Cell selection would sometimes get out of sync, causing
unexpected results when pasting. The UI would also incorrectly
indicate that rows/columns were still selected if you clicked the
selected cell (outlined in green) after doing a drag selection of
multiple rows/columns. Finally, canceling a copy operation would
fail to remove the "scissors" outline  around the copied cells if the
cursor was not on the copied selection.

This resolves all of these bugs.

Test Plan: Browser tests.

Reviewers: cyprien

Reviewed By: cyprien

Subscribers: cyprien

Differential Revision: https://phab.getgrist.com/D3517
2022-07-12 08:50:05 -07:00
Alex Hall
77775401fc (core) Don't clear widget options when changing column type
Summary: Previously, changing the type of a column would clear its widget options and conditional style rules by default, with a few exceptions to explicitly keep them. This diff reverses that behaviour, keeping the options by default.

Test Plan: Updated several existing tests, plus lots of manual testing.

Reviewers: cyprien

Reviewed By: cyprien

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3491
2022-07-11 14:27:44 +02:00
Jarosław Sadziński
8469b7ded0 (core) Fixing bug in token field editor with moving choices.
Summary:
Sometimes when rearranging items in choice editor, user can
put the new item inside last entry element, which is not recognized as
a choice entry.

Test Plan: manual tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3519
2022-07-08 21:03:53 +02:00
Jarosław Sadziński
ddb80f111e (core) Reference columns weren't added to Raw Data Views
Summary:
- Adding a column through 'Add Reference Column' adds it to Raw Data
- Migrating RefSelect.js to typescript
- Extending one of the tests

Test Plan: updated

Reviewers: cyprien

Reviewed By: cyprien

Subscribers: cyprien

Differential Revision: https://phab.getgrist.com/D3513
2022-07-07 17:07:13 +02:00
Paul Fitzpatrick
f91f45b26d (core) support granular read access for attachments
Summary:
When a user requests to read the contents of an attachment, only allow the request if there exists a cell in an attachment column that contains the attachment and which they have read access to.

This does not cover:
 * Granular write access for attachments. In particular, a user who can write to any attachment column should be considered to have full read access to all attachment columns, currently.
 * Access control of attachment metadata such as name and format.

The implementation uses a sql query that requires a scan, and some notes on how this could be optimized in future. The web client was updated to specify the cell to check for access, and performance seemed fine in casual testing on a doc with 1000s of attachments. I'm not sure how performance would hold up as the set of access rules grows as well.

Test Plan: added tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3490
2022-07-07 07:22:02 -04:00
George Gevoian
a051830aeb (core) Show summary tables on Raw Data page
Summary:
Summary tables now have their own raw viewsection, and are shown
under Raw Data Tables on the Raw Data page.

Test Plan: Browser and Python tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3495
2022-07-06 09:41:48 -07:00
Cyprien P
808aacdc52 (core) Fix CustomWidget nbrowser test
Summary:
 - test/nbrowser/CustomFilter keeps randomly fail on my local dev
 - it failed with `Cannot read property of null (reading 'postMessage`)` at line below:
 `this._rpc.setSendMessage(msg => this._iframe?.contentWindow!.postMessage(msg, '*'));`
 - I understand it was trying to send message before even the iframe was properly mounted
 - telling rpc to wait for the other end to send ready() successfully differ send message until everything's mounted.

Test Plan:  - should not break anything and test/nbrowser/CustomFilter should stop failing

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3510
2022-07-05 20:27:39 +02:00
Dmitry S
51ff72c15e (core) Faster builds all around.
Summary:
Building:
- Builds no longer wait for tsc for either client, server, or test targets. All use esbuild which is very fast.
- Build still runs tsc, but only to report errors. This may be turned off with `SKIP_TSC=1` env var.
- Grist-core continues to build using tsc.
- Esbuild requires ES6 module semantics. Typescript's esModuleInterop is turned
  on, so that tsc accepts and enforces correct usage.
- Client-side code is watched and bundled by webpack as before (using esbuild-loader)

Code changes:
- Imports must now follow ES6 semantics: `import * as X from ...` produces a
  module object; to import functions or class instances, use `import X from ...`.
- Everything is now built with isolatedModules flag. Some exports were updated for it.

Packages:
- Upgraded browserify dependency, and related packages (used for the distribution-building step).
- Building the distribution now uses esbuild's minification. babel-minify is no longer used.

Test Plan: Should have no behavior changes, existing tests should pass, and docker image should build too.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3506
2022-07-04 10:42:40 -04:00
Jarosław Sadziński
637caf8105 (core) Fixing bug with first click on windows
Summary:
On Windows, last row is selected when a user clicks the 'new row'
while browser doesn't have focus.

Test Plan: manual tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3503
2022-06-30 17:08:57 +02:00
Jarosław Sadziński
aefe451bab (core) Polishing upgrade plan UI
Summary:
- Update nudge boxes content and collapsing on personal and free team site
- New confirmation after upgrading from a free team site
- Refactoring ProductUpgrade code, splitting plans / modals and nudges

Test Plan: Manual and updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3481
2022-06-29 18:28:37 +02:00
Dmitry S
dd2eadc86e (core) Speed up and upgrade build.
Summary:
- Upgrades to build-related packages:
  - Upgrade typescript, related libraries and typings.
  - Upgrade webpack, eslint; add tsc-watch, node-dev, eslint_d.

- Build organization changes:
  - Build webpack from original typescript, transpiling only; with errors still
    reported by a background tsc watching process.

- Typescript-related changes:
  - Reduce imports of AWS dependencies (very noticeable speedup)
  - Avoid auto-loading global @types
  - Client code is now built with isolatedModules flag (for safe transpilation)
  - Use allowJs to avoid copying JS files manually.

- Linting changes
  - Enhance Arcanist ESLintLinter to run before/after commands, and set up to use eslint_d
  - Update eslint config, and include .eslintignore to avoid linting generated files.
  - Include a bunch of eslint-prompted and eslint-generated fixes
  - Add no-unused-expression rule to eslint, and fix a few warnings about it

- Other items:
  - Refactor cssInput to avoid circular dependency
  - Remove a bit of unused code, libraries, dependencies

Test Plan: No behavior changes, all existing tests pass. There are 30 tests fewer reported because `test_gpath.py` was removed (it's been unused for years)

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3498
2022-06-27 16:10:10 -04:00
Cyprien P
64ff9ccd0a (core) Allows range filter for Date, DateTime columns
Summary:
This diff is first of a series of 3 commits to enable range filering
for Date and DateTime columns. Diff only enable setting date's min/max
throw typing dates, Date picker and relative ranges are left for
follow-up commits.

 - Exposes columns value formatter to the range input
 - Fixes column filter func to work with dates

Test Plan:
Adds Date to projects range filter test
Adds Date/DateTime to nbrowser ColumnFilterMenu tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3455
2022-06-23 13:02:37 +02:00
George Gevoian
abebe812db (core) Show Grist video tour after welcome questions
Summary:
After the welcome questions are dismissed, a video tour modal will
now be displayed. The video tour is also accessible via a tool button
in the left panel of the home page, as well as a text button next to
the Examples & Templates header.

Test Plan: Browser tests.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3477
2022-06-16 21:54:17 -07:00
Dmitry S
a91d493ffc (core) Fix issue with 'UNEXPECTED ORDER OF CALLBACKS' in Client.ts.
Summary:
- Substantial refactoring of the logic when the server fails to send some
  messages to a client.
- Add seqId numbers to server messages to ensure reliable order.
- Add a needReload flag in clientConnect for a clear indication whent the
  browser client needs to reload the app.
- Reproduce some potential failure scenarios in a test case (some of which
  previously could have led to incorrectly ordered messages).
- Convert other Comm tests to typescript.
- Tweak logging of Comm and Client to be slightly more concise (in particular,
  avoid logging sessionId)

Note that despite the big refactoring, this only addresses a fairly rare
situation, with websocket failures while server is trying to send to the
client. It includes no improvements for failures while the client is sending to
the server.

(I looked for an existing library that would take care of these issues. A relevant article I found is https://docs.microsoft.com/en-us/azure/azure-web-pubsub/howto-develop-reliable-clients, but it doesn't include a library for both ends, and is still in review. Other libraries with similar purposes did not inspire enough confidence.)

Test Plan: New test cases, which reproduce some previously problematic scenarios.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3470
2022-06-16 23:51:14 -04:00
George Gevoian
983638a5c5 (core) Disable overscroll in gridview
Summary:
This blocks browser back/forward trackpad gestures from unintentionally
triggering when the cursor is over an overflown gridview. The gestures
are still allowed elsewhere in the UI, as well as in gridviews that have
not overflown.

Test Plan: Tested manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3474
2022-06-13 11:28:33 -07:00
Paul Fitzpatrick
e5e361808b (core) repair grist-ee build and add smoke test
Summary:
This fixes the grist-ee build after recent changes, by
giving it a stub for ProductUpgrades.ts. Extends the
`core` test to also check if `ext` variant builds, to
catch the most common form of breakage on grist-ee so
far (file organization).

Test Plan: extends test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3484
2022-06-13 13:54:14 -04:00
George Gevoian
7176b7efb6 (core) Use new Banner component for activation messages
Summary: Use new Banner component for activation messages.

Test Plan: Existing tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3483
2022-06-13 10:20:31 -07:00
George Gevoian
02dd96daf2 (core) Add page titles to login and error pages
Summary:
Login and error pages now have their own unique page titles. This also fixes
the bug with the signed-out page having a page title of "Error".

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3479
2022-06-12 20:31:09 -07:00
Alex Hall
0350e2df58 (core) Fix filtering of empty reflists
Summary:
A formula returning an empty RecordSet in a RefList columns results in storing [] instead of null.
This caused a bug where the empty list was 'flattened' and the cell not appearing in filters at all.
This diff fixes the bug by filtering for the default value `null` instead for RefLists and the empty string for ChoiceLists.
I didn't manage to actually reproduce the bug for ChoiceLists, but this seemed the most sensible thing to do.

Test Plan: New nbrowser test.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3478
2022-06-11 00:16:57 +02:00
Jarosław Sadziński
64d9ecacdb (core) Renaming table linked by summary table resulted in error
Summary:
When a table was selected by a summary table, renaming that table using
section widget resulted in a javascript error.

Test Plan: new test

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3475
2022-06-10 09:39:34 +02:00
Alex Hall
1c89d08ea3 (core) Add a row to summary tables grouped by list column(s) corresponding to empty lists
Summary:
Adds some special handling to summary table and lookup logic:

- Source rows with empty choicelists/reflists get a corresponding summary row with an empty string/reference when grouping by that column, instead of excluding them from any group
- Adds a new `QueryOperation` 'empty' in the client which is used in `LinkingState`, `QuerySet`, and `recursiveMoveToCursorPos` to match empty lists in source tables against falsy values in linked summary tables.
- Adds a new parameter `match_empty` to the Python `CONTAINS` function so that regular formulas can implement the same behaviour as summary tables. See https://grist.slack.com/archives/C0234CPPXPA/p1654030490932119
- Uses the new `match_empty` argument in the formula generated for the `group` column when detaching a summary table.

Test Plan: Updated and extended Python and nbrowser tests of summary tables grouped by choicelists to test for new behaviour with empty lists.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3471
2022-06-09 23:38:14 +02:00
Jarosław Sadziński
3b30c052bc (core) Enabling clipboard events on Choice entry field
Summary: Copy/paste/cut events weren't available on choice entry field.

Test Plan: Updated tests

Reviewers: alexmojaki, paulfitz

Reviewed By: alexmojaki, paulfitz

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3476
2022-06-09 16:12:35 +02:00
Jarosław Sadziński
d92a761f6e (core) Product update popups and hosted stripe integration
Summary:
- Showing nudge to individual users to sign up for free team plan.
- Implementing billing page to upgrade from free team to pro.
- New modal with upgrade options and free team site signup.
- Integrating Stripe-hosted UI for checkout and plan management.

Test Plan: updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3456
2022-06-08 21:10:49 +02:00
Dmitry S
3b4d936013 (core) Hide 'helpCenter' and 'templates' elements in HomeIntro when requested
Summary:
Removes the relevant links in the intro text and buttons.

Note that the presence of actual "Examples & Templates" section of the page is controlled by whether any templates are available (returned by `getTemplates` api call)

Test Plan: Tested manually

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3473
2022-06-07 19:00:25 -04:00
Dmitry S
007c0f2af0 (core) Fix some bugs with repositioning rows.
Summary:
- Fixed an issue with manualSort values being very close floats. It is already handled by the data engine, but the client was being unnecessarily proactive and introduced a bug.
- The fix also helps with rearranging rows in filtered situations: they will now stay next to the row before which they were inserted.
- The fix accidentally improves (though doesn't fully fix) the issue where new columns show up in unexpected places in the raw-data column list.
- Fixed another rare bug with row order not getting updated correctly when positions update.

Test Plan: Added test cases for the improved behavior; fixed affected tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3462
2022-06-07 16:55:45 -04:00
Dmitry S
4f1cb53b29 (core) Converting server-side Comm.js to typescript
Summary:
- Add app/common/CommTypes.ts to define types shared by client and server.
- Include @types/ws npm package

Test Plan: Intended to have no changes in behavior

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3467
2022-06-07 15:47:17 -04:00
Alex Hall
519f1be93a (core) Disambiguate label for link between summary table and source table with self reference
Summary:
Following up on a small bug introduced in https://phab.getgrist.com/D3464. When a table has a column referencing the same table, then there can be two 'select by' options with the same label which is just the name of a summary table on the same page. The first option is simply filtering based on the summary table. The second option is linking the ref column in the source against the group column in the summary, but the name of the group column is hidden which leads to the ambiguity.

The solution in this diff is to always show the target node (source table) column name if the source node (summary table) column was the hidden group column. This also changes the label in the case where the reference to the source table isn't in the source table - see the updated test. This isn't strictly necessary in this case so I'm not 100% about the desired behaviour, but I don't think it hurts.

Test Plan: Tested disambiguation manually. Updated existing test.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3472
2022-06-07 17:07:45 +02:00
Cyprien P
4b54c7d99b (core) Fix column and view fields mismatch in filters
Summary:
The problem manifested while filtering by a column, closing the filter
would change the set of rows visible on the table. This would happen
only for rare table.

What caused that problem was that the filter being edited was wrongly
applyed also to another column, so depending on the content of the
column, it would entail unexpected behaviour.

The cause of that wrong association, was a mistakingly comparing the
id of two different type of thing: column and view field. The problem
would manifest if in the same section there were a view fields with
same row id as the column being filtered.

What made that confusion possible is the `.fieldOrColumn:
ViewFieldRec|ColumnRec` property of the FileInfo object, which could
hold either a view fields or a column record and was initialized with
view fields or columns if view fields was not found (ie: hidden
column).

Solution was to make sure FieldInfo is initialized with ColumnRec
alwasy (even for hidden column).

I'm not sure what is the reason why FilterInfo needed to support both
column record and view field record in the past, but it looks like
this is not needed anymore.

As a followup commit I think it would be worth the effort to refactor
FileInfo to accept only ColumnRec.

Test Plan: Includes new regression test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3463
2022-06-07 10:26:15 +02:00
George Gevoian
6dcdd22792 (core) Redirect less often in welcomeNewUser
Summary:
Instead of always redirecting new users to the home page or the (teams) welcome page,
only redirect when the user signed in for the first time on a personal site, has access to
other sites, and isn't already being redirected to a specific page on their personal site.

Also tweaks how invalid Choice column values are displayed to match Choice List
columns, and fixes a small CSS issue with select by in the page widget picker when
there are options with long labels.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3461
2022-06-06 11:26:49 -07:00
George Gevoian
090d9af21d (core) Broadcast doc usage updates to clients
Summary:
Introduces a new message type, docUsage, that's broadcast to all connected
clients whenever document usage is updated in ActiveDoc.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3451
2022-06-06 09:55:34 -07:00
Dmitry S
acddd25cfd (core) Update design of empty docs home page, and add a "Manage Team" button.
Summary:
- Remove the empty-folder icon
- Add an "Invite team members" button for owners on empty team sites
- Add a "Browse Templates" button for all other cases on empty sites
- Update intro text for team, including a link to Sprouts
- Update intro text for personal/anon.
- Include a Free/Pro tag for team sites (for now, only "Free")
- Add a "Manage Team" button for owners on home page of all team sites.
- Polished the UI of UserManager: add a transition for the background, and
  delay the appearance of the spinner for fast loads.

Test Plan: Fixed up the HomeIntro tests; added test case for Manage Team button

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3459
2022-06-03 12:58:57 -04:00
Alex Hall
af4738b94a (core) Allow linking to summary table via reference to source table
Summary:
Fixes a bug introduced in https://phab.getgrist.com/D3416 which exposed a new type of linking option that didn't actually work. Specifically it allowed selecting by a summary table when the target widget has a reference column to the source table of the summary. This diff correctly implements this linking by linking the reference column against the 'group' column of the summary table, so the source table data (which the client may not have access to) isn't involved. But the 'group' column name is hidden from the 'select by' option label to avoid confusion for users, so it just looks like another kind of summary table linking, and indeed it can be thought of purely in terms of matching groupby columns etc.

Discussion here: https://grist.slack.com/archives/C02EGJ1FUCV/p1654039063055499

Also fixes a related old bug that offered similar meaningless options involving summary tables, like selecting the summary table by the source table or linking summary tables with disjoint groupby columns.

Test Plan: Updated SelectBySummaryRef test and fixture doc for the new behaviour. Also updated a couple of older tests which were incorrectly asserting the buggy option to select a summary table by the source table.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3464
2022-06-03 17:04:41 +02:00
Cyprien P
815c9e1462 (core) Adds new range filter for numeric columns
Summary:
Shows the range filter next to the filter by values on filter menu. When users
set min and/or max, it takes precendence over the filter by values.

If users set:
 - `[] < [max]` behaves as `less than max`.
 - `[min] < []` behaves as `more than min`.
 - `[min] < [max]` behaves as `between min and max`
 - bounds are always inclusives.
 - when users change min or max the values of the by values filter
   gets checked/unchecked depending on whether they are included by
   the range filter.
 - when users clicks any btn/checkbox of the by values filter both min
   and max input gets cleared, and the filter convert to a filter by
   values.

Test Plan: Adds both projets and nbrowser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3435
2022-05-30 10:30:16 +02:00
Alex Hall
6b372fa6cd (core) Allow configuring (mostly hiding) various little bits of UI
Summary:
Adds two new env vars GRIST_HIDE_UI_ELEMENTS and GRIST_PAGE_TITLE_SUFFIX which translate to values in GristLoadConfig that the server sends the client when loading.

For checkin task https://gristlabs.getgrist.com/doc/check-ins/p/5#a1.s9.r1882.c19

Test Plan: Tested manually

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3449
2022-05-27 14:32:05 +02:00
George Gevoian
74ec9358da (core) Show usage banners in doc menu of free team sites
Summary:
Also fixes a minor CSS regression in UserManager where the
link to add a team member wasn't shown on a separate row.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3444
2022-05-26 15:01:35 -07:00
Cyprien P
6793377579 (core) Fix values ordering in column filter menu
Summary:
Column filter menu use to mess up the ordering of the items for
numeric and dates values, and also for ref/reflist columns when the
visible column is a numeric a date column.

Solution was to:
 - use the actual value of the visible column for comparison.
 - use native comparison.
 - tweak the native comparison to make blanks appears before valid value. Indeed, it came up several time that it's convenient to have invalid values show up first in the filter panel, it makes for a convenient way to detect them.

Test Plan: Adds new nbrowser test

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3441
2022-05-24 16:30:28 +02:00
Dmitry S
9bc04a6e66 (core) Replace nulls in X-axis of charts with '-' to avoid confusion with missing points
Summary:
Seem in a customer doc where Y-scale was wrong for unclear reasons. The cause
is that null-valued X labels cause the corresponding bar (or point) to be
omitted, but still affect the Y-scale. In this diff, such labels are replaced
with "-", so as to show up normally.

Trivial example that reproduces the problem here:
https://public.getgrist.com/iLPpx9C5i8nk/Null-in-X-Axis

Test Plan: Tested manually.

Reviewers: cyprien

Reviewed By: cyprien

Subscribers: cyprien

Differential Revision: https://phab.getgrist.com/D3442
2022-05-20 11:54:20 -04:00
Cyprien P
8f4f21e94a (core) Filter menu show all options for Bool/Choice/Choice List columns
Summary:
> Toggle, Choice, and Choice List need all possible values available in filter, not just values present in current records
https://grist.quip.com/cjw4A8AHx1vh/Filtering-Improvements#temp:C:PZHc42e8be8cd8547bb8ce93fdb0

Test Plan: Adds new nbrowser test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3436
2022-05-19 20:43:29 +02:00
George Gevoian
a6063f570a (core) Polish Access Details
Summary:
Instead of showing a blank dialog for users whose access
is limited (e.g. public members), we now show the user's
role and a mention of whether their access is public.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3431
2022-05-19 09:45:26 -07:00
Jarosław Sadziński
0ab9e4a6a0 (core) Adding GristConnect login system
Summary:
New login system to allow simple SSO flow that is based on Discourse description that is available at:
https://meta.discourse.org/t/discourseconnect-official-single-sign-on-for-discourse-sso/13045

Test Plan: New core test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3418
2022-05-18 20:28:25 +02:00
Paul Fitzpatrick
cf23a2d1ee (core) add GVISOR_LIMIT_MEMORY to cap memory available in sandbox
Summary:
This allows limiting the memory available to documents in the sandbox when gvisor is used. If memory limit is exceeded, we offer to open doc in recovery mode. Recovery mode is tweaked to open docs with tables in "ondemand" mode, which will generally take less memory and allow for deleting rows.

The limit is on the size of the virtual address space available to the sandbox (`RLIMIT_AS`), which in practice appears to function as one would want, and is the only practical option. There is a documented `RLIMIT_RSS` limit to `specifies the limit (in bytes) of the process's resident set (the number of virtual pages resident in RAM)` but this is no longer enforced by the kernel (neither the host nor gvisor).

When the sandbox runs out of memory, there are many ways it can fail. This diff catches all the ones I saw, but there could be more.

Test Plan: added tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3398
2022-05-18 14:26:27 -04:00
George Gevoian
2fd8a34ff8 (core) Move Notifier to /ext
Summary:
This makes it possible to configure a SendGrid-based Notifier
instance via a JSON configuration file.

Test Plan: Tested manually.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3432
2022-05-18 08:02:32 -07:00
Cyprien P
365f3c7ae2 (core) Auto-check aggregate values option as soon as table is a summary
Summary:
Auto-check makes sure that the “Aggregate Values” option is checked as
soon as the chart table is summarized. Before, user had to check it
explicitly.

More on this:
https://grist.slack.com/archives/C04AYS9JF/p1649400119930389?thread_ts=1649338496.915759&cid=C04AYS9JF

Test Plan: Updated tests

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3422
2022-05-18 10:10:28 +02:00
Cyprien P
f17e31c023 (core) Ensure chart summary is more detailed then linked summary
Summary:
When a summary table is linked, the column used for linking needs
to be included as a group-by column (or the linking can’t work). A
good example is here:
https://templates-s.getgrist.com/doc/investment-research/p/4

This was request by Dmitry here: https://grist.slack.com/archives/C04AYS9JF/p1649400119930389?thread_ts=1649338496.915759&cid=C04AYS9JF (2nd point)

Test Plan: Added new nbrowser test case.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3408
2022-05-17 08:57:05 +02:00
George Gevoian
f48d579f64 (core) Add API endpoint to get site usage summary
Summary:
The summary includes a count of documents that are approaching
limits, in grace period, or delete-only. The endpoint is only accessible
to site owners, and is currently unused. A follow-up diff will add usage
banners to the site home page, which will use the response from the
endpoint to communicate usage information to owners.

Test Plan: Browser and server tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3420
2022-05-16 11:16:19 -07:00
Jarosław Sadziński
f93b4fd3bc (core) Selection for styled columns
Summary:
Grid UI improvements.
- Selecting a column with a custom background, didn't produce a visual difference.
- Hiding num-row right border when it is not needed (it should be used only for frozen columns)

Test Plan: manual tests

Reviewers: cyprien

Reviewed By: cyprien

Differential Revision: https://phab.getgrist.com/D3425
2022-05-13 09:45:17 +02:00
George Gevoian
524dbf34e1 (core) Add config to include custom CSS
Summary:
Adds a new environment variable that allows for custom
CSS to be included in all core static pages.

Test Plan: Tested manually in grist-core.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3419
2022-05-12 11:13:52 -07:00
Alex Hall
b878395c21 (core) Allow linking summary tables based on ref/reflist columns (except group)
Summary:
Relax the restriction in `selectBy.isValidLink` so that summary tables can be linked by a column like other tables, except the `group` column. See the discussion on https://grist.slack.com/archives/C0234CPPXPA/p1651773623256959 (the replies are on the following message) for more info on this decision.

Tweaked `LinkingState.ts` since linking with summary tables can now involve a column.

Test Plan: Added a new nbrowser test and fixture checking the options to select by given a summary table with a few ref/reflist columns. Manually tested the behaviour of each option.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3416
2022-05-12 15:59:12 +02:00
Alex Hall
6c90de4d62 (core) Switch excel import parsing from messytables+xlrd to openpyxl, and ignore empty rows
Summary:
Use openpyxl instead of messytables (which used xlrd internally) in import_xls.py.

Skip empty rows since excel files can easily contain huge numbers of them.

Drop support for xls files (which openpyxl doesn't support) in favour of the newer xlsx format.

Fix some details relating to python virtualenvs and dependencies, as Jenkins was failing to find new Python dependencies.

Test Plan: Mostly relying on existing tests. Updated various tests which referred to xls files instead of xlsx. Added a Python test for skipping empty rows.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3406
2022-05-12 14:43:21 +02:00
Cyprien P
87cc7d755d (core) Fix chart mixing up axis
Summary:
Diff fixes couple edge cases:
 - When changing chart's groupby columns, the data-engine changes the view fields, which is not okay with charts, hence makes sure view fields stay the same as much as possible using `this.setSectionViewFieldsFromArray()`
 - Also there's a logic in `this.setSectionViewFieldsFromArray()` that handle what to do when some columns goes missing during updates. Diff fixes this logic two. This is really corner case.

Test Plan: Both cases are not added to ChartAggregate.ts

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3405
2022-05-11 09:18:31 +02:00
George Gevoian
1e42871cc9 (core) Add attachment and data size usage
Summary:
Adds attachment and data size to the usage section of
the raw data page. Also makes in-document usage banners
update as user actions are applied, causing them to be
hidden/shown or updated based on the current state of
the document.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3395
2022-05-04 13:46:55 -07:00
Jarosław Sadziński
f194d6861b (core) Updating RawData views
Summary:
- Better focus on the widget title
- Adding columns only to the current view section
- New popup with options when user wants to delete a page
- New dialog to enter table name
- New table as a widget doesn't create a separate page
- Removing a table doesn't remove the primary view

Test Plan: Updated and new tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3410
2022-05-04 21:41:42 +02:00
Dmitry S
97f3a8805c (core) List 'Other Sites' all one line; fix style ordering issue for sort button.
Test Plan: Tested manually by eyeballing

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3407
2022-04-28 23:00:40 -04:00
George Gevoian
ad04744b4a (core) Fix import bug when skipping non-text columns
Summary:
Skipping columns during incremental imports wasn't working for certain
column types, such as numeric columns. The column's default value was
being used instead (e.g. 0), overwriting values in the destination
table.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3402
2022-04-28 12:46:44 -07:00
Jarosław Sadziński
6f00106d7c (core) Raw renames
Summary:
A new way for renaming tables.

  - There is a new popup to rename section (where you can also rename the table)
  - Renaming/Deleting page doesn't modify/delete the table.
  - Renaming table can rename a page if the names match (and the page contains a section with that table).
  - User can rename table in Raw Data UI in two ways - either on the listing or by using the section name popup
  - As before, there is no way to change tableId - it is derived from a table name.
  - When the section name is empty the table name is shown instead.
  - White space for section name is allowed (to discuss) - so the user can just paste '   '.
  - Empty name for a page is not allowed (but white space is).
  - Some bugs related to deleting tables with attached summary tables (and with undoing this operation) were fixed (but not all of them yet).

Test Plan: Updated tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3360
2022-04-27 22:21:55 +02:00
Jarosław Sadziński
8a1cca629b (core) Changing shortcut for Duplicate row command
Summary:
Changing keyboard shortcut for Duplicate row from
Ctrl+Shift+D to Mod+Shift+D

Test Plan: existing test

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3401
2022-04-27 22:02:27 +02:00
Dmitry S
e59dcc142d (core) Show proper message on empty Excel import, rather than a code error
Summary:
- Previously showed "UnboundLocalError". Now will show:
    Import failed: Failed to parse Excel file.
    Error: No tables found (1 empty tables skipped)
- Also fix logging for import code

Test Plan: Added a test case

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3396
2022-04-27 00:49:28 -04:00
Jarosław Sadziński
dcafa96b5b (core) Restoring context menu when comparing documents
Summary:
Fixing bug when comparing documents. User wasn't able
to open cell/column context menu due to javascript error.

Test Plan: Updated tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3397
2022-04-26 21:31:36 +02:00
George Gevoian
af5b3c9004 (core) Add document usage banners
Summary:
This also enables the new Usage section for all sites. Currently,
it shows metrics for document row count, but only if the user
has full document read access. Otherwise, a message about
insufficient access is shown.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3377
2022-04-25 08:14:52 -07:00
Jarosław Sadziński
bedb19f9c7 (core) Removing conditional styles for blank rows
Summary:
When grid view was not fully loaded the initial blank row
was marked as invalid by conditional styles. This removes the styling
when row is not fully loaded.

Test Plan: Tested manually

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3392
2022-04-21 11:45:17 +02:00
Jarosław Sadziński
77ef9df27d (core) Adding new command Duplicate rows
Summary:
New command "Duplicate rows" is available in the Row/Card Context Menu and as a keyboard shortcut Ctrl+Alt+C.
- All selected rows are duplicated (even if only a single column is selected)
- Rows are inserted immediately after the last selected row (using manualSort value).
- Formulas and CENSORED fields are not copied.
Implemented on the UI level (no new action).

Test Plan: new test

Reviewers: cyprien

Reviewed By: cyprien

Differential Revision: https://phab.getgrist.com/D3371
2022-04-20 17:29:48 +02:00
Alex Hall
47b77c8c24 (core) Allow filtering by selected cell value in cell context menu
Summary: Adds a command and `BaseView` method `filterByThisCellValue`.

Test Plan: Added two tests to `nbrowser/CellContextMenu.ts`

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3383
2022-04-19 11:51:47 +02:00
Jarosław Sadziński
dea1a8ba1b (core) Implementing search on raw data view
Summary:
Search now works on Raw Data Page.
- Search bar option 'Search on all pages' will change to 'Search on all tables' when on the Raw data page, and will allow searching through all tables.
- Little CSS adjustment for an overlay on Raw page (removes z-index as it is not needed, and conflicts with searchbar).
- Search bar option ('search on all') gets white background, little padding, and is moved 2 pixels up, this is needed for Raw page.

Test Plan: new and updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3376
2022-04-13 20:44:37 +02:00
George Gevoian
007a862333 (core) Include hash/fragment in post-login redirect URLs
Summary: Also cleans up login URLs by excluding params and hashes.

Test Plan: Client and server tests.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3378
2022-04-13 09:16:29 -07:00
Paul Fitzpatrick
20dd2fc70d (core) allow non-owners to remove themselves from sites/workspaces/docs
Summary:
For users who cannot otherwise change access to a resource, let
them remove themselves. Implemented via the standard endpoints
as a special exception that will process a request from a user
that would otherwise be denied, if the only contents of that
request are a removal of themselves.

Users who can change access are still not permitted to change their
own permissions or to remove themselves, as a precaution against
orphaning resources.

Test Plan: extended and updated tests

Reviewers: cyprien

Reviewed By: cyprien

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3367
2022-04-13 10:04:32 -04:00
Cyprien P
25e40bfa9b (core) Fix setting xaxis when both chart aggregation and split series
Summary:
 - Symptoms where that Split Series could end up being turned off for
   no good reason. Also both x axis and split series could be mixed
   up.

 - Problems was caused by call to `setGroupByColumns` which modifies
   the sections viewFields. Diff fixes it by adjustin slightly the
   ordering of function call in `_setXAxis()`.

 - Problem of mixing up x axis and split series was fixed by being
   careful on the order of columns passed to the `setGroupByColumns`
   which then determine the ordering of the view fields.

Test Plan: Includes new test cases

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3365
2022-04-13 11:25:24 +02:00
Jarosław Sadziński
4401ec4d79 (core) Adding onNewRecord event for custom widgets
Summary:
New onNewRecord event in custom widgets that is triggered
when the user selects a blank row.

Together with this diff there is PR for grist-help https://github.com/gristlabs/grist-help/pull/150

Test Plan: added test case

Reviewers: cyprien

Reviewed By: cyprien

Differential Revision: https://phab.getgrist.com/D3375
2022-04-12 16:51:16 +02:00
Dmitry S
cedcdc6bff (core) Improve debug logging related for client-side errors and sandbox crashes.
Summary:
- Include docId when available for client-side error reporting
- Distinguish sandbox crashes from forced exits

Test Plan: Tested manually

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3373
2022-04-11 17:54:40 -04:00
Paul Fitzpatrick
14f7e30e6f (core) add users.options.isConsultant flag, and omit such users from billing
Summary:
This adds an optional `isConsultant` flag to `users.options`, and an endpoint that allows the support user to turn it on or off. Users marked as consultants are not counted as billable members. Follows the example of existing `allowGoogleLogin` option.

Billable members are counted when members are added or removed from a site. Changing the `isConsultant` flag has no immediate or retroactive effect on billing. The number of users in stripe is now set unconditionally, rather than only when it has changed.

Notifications to billing managers are not aware of this billing nuance, but continue to report user counts that include consultants. The notifications link users to the billing page.

Test Plan: extended test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: anaisconce, jarek

Differential Revision: https://phab.getgrist.com/D3362
2022-04-11 10:26:31 -04:00
Cyprien P
bf8769bc42 (core) Fix linking after a summary update
Summary:
When linking table to a summary, the linking ended up broken after updating the summary group by columns.
This diff fixes that issue.

There were two issues:
 1) some subscriptions were missing due to some .peek() calls instead of directly calling the observable.
 2) the LinkingState instance was not being disposed.
 3) the filterColValues was not updating after source data table has been loaded

Test Plan: Include new test file.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3358
2022-04-08 12:28:42 +02:00
Jarosław Sadziński
34708cd348 (core) Adding font options to the style picker
Summary:
Redesigning color picker:
- Single color palette (no light/dark switch)
- Ability to remove color (new empty button)

New font options in the color picker.
Font options are available on:
- Default cell style
- Conditional rules styles
- Choice/ChoiceList editor and token field
- Filters for Choice/ChoiceList columns

Design document:
https://www.figma.com/file/bRTsb47VIOVBfJPj0qF3C9/Grist-Updates?node-id=415%3A8135

Test Plan: new and updated tests

Reviewers: georgegevoian, alexmojaki

Reviewed By: georgegevoian, alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3335
2022-04-07 20:35:03 +02:00
Cyprien P
98ac2f7e5b (core) add 'Aggregate Values' option to chart config
Summary:
  - adds the `aggregate values` option to chart config
    - aggregation is performed by turning table into summary table
    - change columns options of xaxis and split series selectors to be the source table columns when `aggregate values` is on
    - change xAxis and split series computed to hold colId instead of column id
    - change GristDoc saveViewSection routine to preserve old sections viewFields and options
  - Rename `Group data` into `split series`

quip doc: https://grist.quip.com/tAsCAuv8RiMa/Charts-data-aggregation#temp:C:QcK0ce13e1e8ae64048988f44f9c

Test Plan: Adds ChartAggregate.ts nbrowser test.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3336
2022-04-07 15:34:54 +02:00
George Gevoian
d8af25de9d (core) Add usage to data tables page
Summary:
Currently, usage is only shown for free team sites, and only
for total number of rows used in a document. Future diffs will
include other usage metrics and browser tests.

Test Plan: Planned for future diffs; UI is still under development.

Reviewers: jarek

Reviewed By: jarek

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3343
2022-04-04 15:08:19 -07:00
Alex Hall
bf271c822b (core) Copy column type and options when pasting into an empty column
Summary:
Adds a `data-grist-col-ref` attribute to the copied HTML, then uses that when pasting to look up the source column and retrieve info about it. Copies the info into the target column if:

- The document is the same (the docId hash matches)
- The source column still exists and has the same type as when copied
- The source type isn't Text, because in that case it's nice if type guessing still happens
- The target column is empty, meaning it has type Any (we check earlier that it's not a formula column)

The info copied is the type, widgetOptions, and reference column settings (visible and display columns) but not conditional formatting.

The changes are mostly in a function `parsePasteForView` which is based on `BaseView._parsePasteForView` but ported to TypeScript in a new file `BaseView2.ts`.

Added a useraction `MaybeCopyDisplayFormula` exposing an existing Python function `maybe_copy_display_formula` because the target column needs a slightly different display formula.

Test Plan: Added a new nbrowser test file and fixture doc.

Reviewers: cyprien

Reviewed By: cyprien

Subscribers: jarek, dsagal

Differential Revision: https://phab.getgrist.com/D3344
2022-04-04 14:53:16 +02:00
George Gevoian
6305811ca6 (core) Add new Grist login page
Summary:
Adds a new Grist login page to the login app, and replaces the
server-side Cognito Google Sign-In flow with Google's own OAuth flow.

Test Plan: Browser and server tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3332
2022-04-01 15:24:19 -07:00
Alex Hall
8fdfb02646 (core) Don't treat Attachments as RefList in creator panel
Summary: Fixing bug introduced in https://phab.getgrist.com/D3338: column config now shows "DATA FROM TABLE" under the Attachments column type as if it were a normal ref/reflist column.

Test Plan: manual

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3350
2022-04-01 22:10:22 +02:00
Dmitry S
d55bdbcdf3 (core) Reopen HelpScout beacon to the last-opened article
Summary:
- When opening HelpScout beacon to an article ("answers"), avoid a 'navigate'
  call to let the beacon show the previously open article.
- Work around a bug with reloading a page with a beacon article open: HelpScout
  renders the last state without triggering usual events.
- Report errors to server when beacon fails to load.
- reportWarning() method now reports the message to the server.

Test Plan: Added a test case

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3345
2022-03-31 14:54:55 -04:00
Jarosław Sadziński
499e24b744 (core) Adding conditional styles to old style widgets
Summary:
Widgets that were using old base Widget class didn't not
create conditional style rules ui.

Additional fixed a little bug - when adding conditional rule
the formula field was marked as having error for a split second.

Test Plan: new test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3346
2022-03-31 18:29:41 +02:00
Alex Hall
59436d2bca (core) Grace period and delete-only mode when exceeding row limit
Summary:
Builds upon https://phab.getgrist.com/D3328

- Add HomeDB column `Document.gracePeriodStart`
- When the row count moves above the limit, set it to the current date. When it moves below, set it to null.
- Add DataLimitStatus type indicating if the document is approaching the limit, is in a grace period, or is in delete only mode if the grace period started at least 14 days ago. Compute it in ActiveDoc and send it to client when opening.
- Only allow certain user actions when in delete-only mode.

Follow-up tasks related to this diff:

- When DataLimitStatus in the client is non-empty, show a banner to the appropriate users.
- Only send DataLimitStatus to users with the appropriate access. There's no risk landing this now since real users will only see null until free team sites are released.
- Update DataLimitStatus immediately in the client when it changes, e.g. when user actions are applied or the product is changed. Right now it's only sent when the document loads.
- Update row limit, grace period start, and data limit status in ActiveDoc when the product changes, i.e. the user upgrades/downgrades.
- Account for data size when computing data limit status, not just row counts.

See also the tasks mentioned in https://phab.getgrist.com/D3331

Test Plan: Extended FreeTeam nbrowser test, testing the 4 statuses.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3331
2022-03-25 13:41:33 +02:00
Paul Fitzpatrick
134ae99e9a (core) add gvisor-based sandboxing to core
Summary:
This adds support for gvisor sandboxing in core. When Grist is run outside of a container, regular gvisor can be used (if on linux), and will run in rootless mode. When Grist is run inside a container, docker's default policy is insufficient for running gvisor, so a fork of gvisor is used that has less defence-in-depth but can run without privileges.

Sandboxing is automatically turned on in the Grist core container. It is not turned on automatically when built from source, since it is operating-system dependent.

This diff may break a complex method of testing Grist with gvisor on macs that I may have been the only person using. If anyone complains I'll find time on a mac to fix it :)

This diff includes a small "easter egg" to force document loads, primarily intended for developer use.

Test Plan: existing tests pass; checked that core and saas docker builds function

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3333
2022-03-24 17:04:49 -04:00
Jarosław Sadziński
64c9717ac1 (core) Undo bug with summary table and raw data view
Summary:
Clicking undo/redo after converting a table to a summary table navigated
to the raw data view.

Test Plan: new test

Reviewers: georgegevoian, alexmojaki

Reviewed By: georgegevoian, alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3337
2022-03-24 20:03:33 +01:00
Jarosław Sadziński
b1c3943bf4 (core) Conditional formatting rules
Summary:
Adding conditional formatting rules feature.

Each column can have multiple styling rules which are applied in order
when evaluated to a truthy value.

- The creator panel has a new section: Cell Style
- New user action AddEmptyRule for adding an empty rule
- New columns in _grist_Table_columns and fields

A new color picker will be introduced in a follow-up diff (as it is also
used in choice/choice list/filters).

Design document:
https://grist.quip.com/FVzfAgoO5xOF/Conditional-Formatting-Implementation-Design

Test Plan: new tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3282
2022-03-23 13:15:02 +01:00
Jarosław Sadziński
96a34122a5 (core) Restoring cursor position on raw data views
Summary:
This diff introduces cursor features for raw data views:
- Restoring cursor position when the browser window is reloaded
- Restoring the last edit position when the browser window is reloaded

Test Plan: Added tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3314
2022-03-23 12:24:18 +01:00
Dmitry S
3b76b33423 (core) Fix bugs when both welcomeTour and docTour are available
Summary:
- Unify where in the code tours get initiated.
- Avoid start a new tour while one is being started or is in progress.
- Ignore welcome tour when on a doc that has a doc tour.
- Fix tours when starting with a special page like Access Rules.
- Remove mention of the no-longer-present "Give Feedback" button in the last
  message of the welcome tour.

Test Plan:
Add a browser test case that docTour preempts the welcome tour and shows no errors
(this test case fails in multiple ways without the changes).

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3330
2022-03-22 16:51:05 -04:00
Cyprien P
b6f146d755 (core) Add options to switch chart orientation
Test Plan: Adds nbrowser tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3323
2022-03-21 11:28:44 +01:00
Cyprien P
21f1dfa56c (core) Add 'stacked' option to charts
Summary:
Adds nbrowser test

 - Also makes sort spec taken into account by Group Data options
 - This is a continuation of https://phab.getgrist.com/D3271
 - We still need to decide whether to add stack chart to area chart type

Test Plan: TBD

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3274
2022-03-18 10:59:12 +01:00
Dmitry S
fa75f60bfd (core) Fix selection of rows after rows are dragged
Summary: After dragging rows up, selection was set incorrectly.

Test Plan: Expanded a browser test for dragging rows to check selection, which fails without this fix

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3324
2022-03-17 11:49:52 -04:00
George Gevoian
0f4f0d3dad (core) Migrate to SRP and add change password dialog
Summary:
Moves some auth-related UI components, like MFAConfig, out
of core, and adds a new ChangePasswordDialog component for
allowing direct password changes, replacing the old reset password
link to hosted Cognito.

Updates all MFA endpoints to use SRP for authentication.

Also refactors MFAConfig into smaller files, and polishes up some parts
of the UI to be more consistent with the login pages.

Test Plan: New server and deployment tests. Updated existing tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3311
2022-03-16 21:35:06 -07:00
Paul Fitzpatrick
98f64a8461 (core) add grist.selectedTable.create/update/destroy/upsert to custom widget api
Summary: This makes an equivalent of the /records REST endpoint available within custom widgets. For simple operations, it is compatible with https://github.com/airtable/airtable.js/. About half of the diff is refactoring code from DocApi that implements /records using applyUserActions, to make that code available in the plugin api.

Test Plan: added tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3320
2022-03-15 11:11:58 -04:00
Alex Hall
02e69fb685 (core) Crudely show row count and limit in UI
Summary:
Add rowCount returned from sandbox when applying user actions to ActionGroup which is broadcast to clients.

Add rowCount to ActiveDoc and update it after applying user actions.

Add rowCount to OpenLocalDocResult using ActiveDoc value, to show when a client opens a doc before any user actions happen.

Add rowCount observable to DocPageModel which is set when the doc is opened and when action groups are received.

Add crude UI (commented out) in Tool.ts showing the row count and the limit in AppModel.currentFeatures. The actual UI doesn't have a place to go yet.

Followup tasks:

- Real, pretty UI
- Counts per table
- Keep count(s) secret from users with limited access?
- Data size indicator?
- Banner when close to or above limit
- Measure row counts outside of sandbox to avoid spoofing with formula
- Handle changes to the limit when the plan is changed or extra rows are purchased

Test Plan: Tested UI manually, including with free team site, opening a fresh doc, opening an initialised doc, adding rows, undoing, and changes from another tab. Automated tests seem like they should wait for a proper UI.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3318
2022-03-14 21:49:32 +02:00
George Gevoian
ad1b4f3cff (core) Record new user sign-ups
Summary:
Adds Google Tag Manager snippet to all login pages, and a new user
preference, recordSignUpEvent, that's set to true on first sign-in. The
client now checks for this preference, and if true, dynamically loads
Google Tag Manager to record a sign-up event. Afterwards, it removes
the preference.

Test Plan: Tested manually.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3319
2022-03-12 14:34:46 -08:00
Jarosław Sadziński
eff78ae2e1 (core) New UI for raw data views
Summary:
Creating new UI for raw data views based on design.
- Renaming left for follow up diff
- Link in the menu is hidden for now
To access raw UI, use /p/data URL.

Test Plan: new tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3306
2022-03-12 13:51:48 +01:00
George Gevoian
f02174eb7e (core) Fix error when canceling import
Summary:
If cancel was clicked while a transform section was still being
generated in the Importer, an error was thrown. This refactors
the cancelImportFiles API action to take in the file upload id
in place of the entire DataSourceTransformed parameter, which
contains other values that are irrelevant to canceling. One of those
values, the transform section id, was causing the error to be thrown
since it was momentarily null.

Test Plan: Tested manually.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3317
2022-03-10 16:24:49 -08:00
Jarosław Sadziński
351d70d4fb (core) Serving widget info page from home url
Summary:
Custom widget into page is served from a homeUrl instead
of untrusted URL, which might be not used in grist-core.

Test Plan: manual test

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3307
2022-03-09 10:34:50 +01:00
Jarosław Sadziński
d2b82b84c7 (core) Fixing bug with resuming search on a hidden column.
Summary: Fix for error that happens when a search is resumed after one of the columns was hidden.

Test Plan: Added test that shows the error.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3309
2022-03-09 10:34:17 +01:00
Alex Hall
321019217d (core) Lossless imports
Summary:
- Removed string parsing and some type guessing code from parse_data.py. That logic is now implicitly done by ValueGuesser by leaving the initial column type as Any. parse_data.py mostly comes into play when importing files (e.g. Excel) containing values that already have types, i.e. numbers and dates.
- 0s and 1s are treated as numbers instead of booleans to keep imports lossless.
- Removed dateguess.py and test_dateguess.py.
- Changed what `guessDateFormat` does when multiple date formats work equally well for the given data, in order to be consistent with the old dateguess.py.
- Columns containing numbers are now always imported as Numeric, never Int.
- Removed `NullIfEmptyParser` because it was interfering with the new system. Its purpose was to avoid pointlessly changing a column from Any to Text when no actual data was inserted. A different solution to that problem was already added to `_ensure_column_accepts_data` in the data engine in a recent related diff.

Test Plan:
- Added 2 `nbrowser/Importer2` tests.
- Updated various existing tests.
- Extended testing of `guessDateFormat`. Added `guessDateFormats` to show how ambiguous dates are handled internally.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3302
2022-03-08 12:14:39 +02:00
George Gevoian
9522438967 (core) Add Grist forgot password page
Summary:
The page isn't yet linked to from anywhere in the UI, but
will be soon, once the new login page is ready. The page
can still be accessed at login-[s].getgrist.com/forgot-password,
and the flow is similar to the one used by Cognito's hosted UI.

Also refactors much of the existing login app code into smaller
files with less duplication, tweaks password validation to be closer
to Cognito's requirements, and polishes various parts of the UI,
like the verified page CSS, and the form inputs.

Test Plan: Browser, server and project tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3296
2022-03-07 09:11:28 -08:00
Cyprien P
76481d21e0 (core) Enable selecting x axis for the group data columns
Summary:
Until now, users could not pick the column that's currently serving as
the x axis.

To do that, chart needs to support having the x axis undefined.

This diff do just that:
  - allow x axis to be undefined
  - allow setting x axis from group data

Given that charts axis are stored as indexes of the section view
fields array, implementation required introduction of an extra chart
options: `isAxisUndefined`.

Test Plan: Updates existing test and adds new one.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3304
2022-03-07 10:49:18 +01:00
Paul Fitzpatrick
2563fb745a (core) make Grist easier to run with a single server
Summary:
This makes many small changes so that Grist is less fussy to run as a single instance behind a reverse proxy. Some users had difficulty with the self-connections Grist would make, due to internal network setup, and since these are unnecessary in any case in this scenario, they are now optimized away. Likewise some users had difficulties related to doc worker urls, which are now also optimized away. With these changes, users should be able to get a lot further on first try, at least far enough to open and edit documents.

The `GRIST_SINGLE_ORG` setting was proving a bit confusing, since it appeared to only work when set to `docs`. This diff
adds a check for whether the specified org exists, and if not, it creates it. This still depends on having a user email to make as the owner of the team, so there could be remaining difficulties there.

Test Plan: tested manually with nginx

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3299
2022-03-05 13:30:45 -05:00
Cyprien P
0da397ab90 (core) Fix js error when changing X Axis with chart with no series
Summary:
This error only occured for certain charts type. Some chart handled it correctly, others did not.

This brings a more general handling of this issue which should work for all charts.

Test Plan: New nbrowser test case.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3303
2022-03-04 14:06:19 +01:00
Alex Hall
5213972d24 (core) Guess numeric formatting options
Summary:
Change NumberParse.parse to return not just the parsed number but also information it gathered along the way about how the input string was formatted.

Use this in the new NumberParse.guessOptions to guess the actual widget options based on an array of strings.

Use NumberParse.guessOptions in TypeConversion (for when a user explicitly chooses to change type) and in ValueGuesser (for guesses about strings entered into empty columns).

Test Plan: Adds unit tests for NumberParse and ValueGuesser and updates the TypeChange2 nbrowser test.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3294
2022-03-03 21:32:03 +02:00
Jarosław Sadziński
70373550cf (core) Shortcuts help page can be scrolled using a keyboard
Summary: Restoring keyboard navigation on the shortcuts popup.

Test Plan: manual tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3295
2022-03-02 11:43:05 +01:00
George Gevoian
8f5ce9d610 (core) Wait for transform column before finalizing
Summary:
This adds a new private field to ColumnTransform that settles when
the transform column is done preparing. Finalize then waits for this
promise before proceeding.

A bug was occurring when the prepare promise was still pending,
but finalize was triggered indirectly, causing finalize to throw an error
due to the transform column not yet existing.

Test Plan: Tested manually.

Reviewers: jarek, dsagal

Reviewed By: jarek, dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3288
2022-03-01 13:59:14 -08:00
Alex Hall
599545fb11 (core) Fuller guessing of type and options when adding first data to blank columns
Summary:
Adds `common/ValueGuesser.ts` with logic for guessing column type and widget options (only for dates/datetimes) from an array of strings, and converting the strings to the guessed type in a lossless manner, so that converting back to Text gives the original values.

Changes `_ensure_column_accepts_data` in Python to call an exported JS method using the new logic where possible.

Test Plan: Added `test/common/ValueGuesser.ts` to unit test the core guessing logic and a DocApi end-to-end test for what happens to new columns.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3290
2022-03-01 22:00:45 +02:00
Jarosław Sadziński
ae6c857ac5 (core) Frozen columns reordering
Summary: Bug fix. Moving columns over the frozen set didn't work properly.

Test Plan: new tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3297
2022-02-28 22:52:37 +01:00
George Gevoian
fa68b790bb (core) Remove code for unused welcome flows
Summary: Removes code that was marked for removal.

Test Plan: Existing tests still pass.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3289
2022-02-28 13:21:28 -08:00
Cyprien P
c2070877b6 (core) Fix out of sync GROUP DATA and X Axis options when table change
Summary:
Fix this issue:
  When using “Change Widget” for a chart of a summary table, if I change group-by columns, the X-Axis dropdown becomes empty, and lists values from previous summary (not the actual summary table that’s now shown in the chart). Need to close and reopen creator panel to fix it.

Test Plan: Include nbrowser test

Reviewers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3284
2022-02-25 14:48:27 +01:00
Alex Hall
3445ecb64c (core) Search backwards with Shift+Enter
Summary: As simple as it sounds. Check ev.shiftKey in Enter key handler in search.ts.

Test Plan: Extended test/nbrowser/Search.ts

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3293
2022-02-25 14:07:15 +02:00
Paul Fitzpatrick
accd640078 (core) add a user.SessionID value for trigger formulas and granular access rules
Summary:
This makes a `user.SessionID` value available in information about the user, for use with trigger formulas and granular access rules. The ID should be constant within a browser session for anonymous user. For logged in users it simply reflects their user id.

This ID makes it possible to write access rules and trigger formulas that allow different anonymous users to create, view, and edit their own records in a document.

For example, you could have a brain-storming document for puns, and allow anyone to add to it (without logging in), letting people edit their own records, but not showing the records to others until they are approved by a moderator. Without something like this, we could only let anonymous people add one field of a record, and not have a secure way to let them edit that field or others in the same record.

Also adds a `user.IsLoggedIn` flag in passing.

Test Plan: Added a test, updated tests. The test added is a mini-moderation doc, don't use it for real because it allows users to edit their entries after a moderator has approved them.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3273
2022-02-22 12:50:43 -05:00
George Gevoian
95592b81bd (core) Skip /welcome/user page for new users
Summary:
Since the new Grist sign-up page has a required field for
name, we can now skip the welcome page asking for the
same thing. Code and tests that can be removed later are
marked with TODOs.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3266
2022-02-22 08:38:22 -08:00
George Gevoian
36843e632b (core) Hide API key when it's not selected
Summary:
The API key is now hidden by default. Clicking the input
will select and reveal the key. As soon as the key loses
selection, it is hidden again.

Test Plan: Project test.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3270
2022-02-21 17:26:45 -08:00
Alex Hall
5b352211c4 (core) Guess date format during type conversion
Summary:
- Adds a dependency moment-guess (https://github.com/apoorv-mishra/moment-guess) to guess date formats from strings. However the npm package is missing source maps which leads to an ugly warning, so currently using a fork until https://github.com/apoorv-mishra/moment-guess/pull/22 is resolved.
- Adds guessDateFormat using moment-guess to determine the best candidate date format. The logic may be refined for e.g. lossless imports where the stakes are higher, but for now we're just trying to make type conversions smoother.
- Uses guessDateFormat to guess widget options when changing column type to date or datetime.
- Uses the date format of the original column when possible instead of guessing.
- Fixes a bug where choices were guessed based on the display column instead of the visible column, which made the guessed choices influenced by which values were referenced as well as completely broken when converting from reflist.
- @dsagal @georgegevoian This builds on https://phab.getgrist.com/D3265, currently unmerged. That diff was created first to alert to the change. Without it there would still be similar test failures/changes here as the date format would often be concretely guessed and saved as YYYY-MM-DD instead of being left as the default `undefined` which is shows as YYYY-MM-DD in the dropdown.

Test Plan: Added a unit test to `parseDate.ts`. Updated several browser tests which show the guessing in action during type conversion quite nicely.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal, georgegevoian

Differential Revision: https://phab.getgrist.com/D3264
2022-02-21 22:39:47 +02:00
Paul Fitzpatrick
be8c053922 (core) updates from grist-core 2022-02-21 13:44:13 -05:00
Edward Betts
d6e0e1fee3 Correct spelling mistakes 2022-02-19 09:46:49 +00:00
Cyprien P
afa90cc365 (core) Show default context menu on link
Summary:
also:
  - closes opened menu if any when click on a custom widget
  - closes opened menu if any when F2

Test Plan: Include test case

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3269
2022-02-18 10:09:36 +01:00
Dmitry S
9855a145d1 (core) Remove UI-imposed restriction against combining public edit access and access rules
Test Plan: TBD

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3272
2022-02-17 09:57:23 -05:00
Cyprien P
2f6eafff35 (core) Adds setSelectedRows to the grist api for custom view
Summary:
This is needed to let custom widget driver filtering of other widget in the same page.

Descripion here:
 - https://grist.quip.com/ctytAQJoFMsM/Hopefully-Small-Projects#temp:C:NNCfe2030b27647439886ca83595

Test Plan: New api tested in a new nbrowser test

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3253
2022-02-15 15:24:15 +01:00
George Gevoian
e264094412 (core) Add account page option to allow Google login
Summary:
Enabled by default, the new checkbox is only visible to
users logged in with email/password, and controls whether it is possible
to log in to the same account via a Google account
(with matching email). When disabled, CognitoClient will refuse logins
from Google if a Grist account with the same email exists.

Test Plan:
Server and browser tests for setting flag. Manual tests to verify
Cognito doesn't allow signing in with Google when flag is disabled.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3257
2022-02-14 16:56:23 -08:00
George Gevoian
99f3422217 (core) Add new Grist sign-up page
Summary:
Available at login.getgrist.com/signup, the new sign-up page
includes similar options available on the hosted Cognito sign-up
page, such as support for registering with Google. All previous
redirects to Cognito for sign-up should now redirect to the new
Grist sign-up page.

Login is still handled with the hosted Cognito login page, and there
is a link to go there from the new sign-up page.

Test Plan: Browser, project and server tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3249
2022-02-14 10:32:47 -08:00
Cyprien P
7cc3092e1b (core) Restore default context menu out of std views
Summary:
Also fix few clode glitches
- Attach events handler to the scrollPane using `onMatch` instead of
  the cell itself. This streamline cell dom creation a bit while
  scrolling.
- Fix memory leaks on contextMenu.
- Also fix Importer and ColumnOps nbrowser test

see:
https://phab.getgrist.com/D3237#inline-36376
https://phab.getgrist.com/D3237#inline-36375

Restore default context menu for where there's no custom one

It appears that default context menu adds some value, in particular
for links and for editing text. This diff restores it.

Test Plan: Should not break anything.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3256
2022-02-11 09:05:54 +01:00
Jarosław Sadziński
e99122433a (core) Sending visible columns for not configured widgets
Summary:
Custom widgets will receive all visible columns when they are not configured.
This will make the existing configuration work without the need to pick columns.

Test Plan: updated tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3259
2022-02-10 19:09:26 +01:00
Alex Hall
1c855537d6 (core) Fix non-existent page ID showing all raw data widgets
Summary: Check that view exists first, otherwise fall back to default

Test Plan: Manual

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3260
2022-02-10 20:01:57 +02:00
Alex Hall
592a43ec36 (core) Initial data tables page
Summary:
- Added a new special page for viewing raw data widgets:
  - Implemented in DataTables.ts
  - Accessible only via the special URL path `/p/data`
  - Future diffs should make this page prettier and easily accessible
  - Shows a list of user tables
  - Clicking on a table name shows its `rawViewSection` by setting `GristDoc.viewModel.activeSectionId`. Note that in this case `GristDoc.viewModel` is an empty record, so this is a bit of a hack, but it works well and causes no known issues.
- Added `ViewSectionRec.isRaw` to know if the record represents a raw data widget.
- Added various restrictions in the UI for raw data widgets:
  - 'Delete widget' is disabled in the 3-dot widget menu.
  - Prevent hiding columns:
    - "Hide column" in the column context menu is disabled
    - The "VISIBLE/HIDDEN COLUMNS" section of the right panel > Table > Widget is hidden
  - The toggle bar isn't configurable to ensure that users know when raw data is filtered:
    - The filter bar always shows if and only if some filters are present
    - "Toggle Filter Bar" is hidden in:
      - Right panel > Table > Sort & Filter
      - The sort/filter menu next to the three-dot menu for widgets.
  - Other restrictions in the right panel:
    - In the Column tab:
      - 'Use separate settings' is disabled
    - In the Table tab:
      - In the Widget subtab:
        - 'Change Widget' is hidden
      - In the Data subtab:
        - 'Edit Data Selection' is hidden
        - 'SELECT BY' is hidden

Test Plan: Tested manually. The behaviour of raw data widgets may still change and they aren't easily visible to users yet.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3248
2022-02-10 12:46:19 +02:00
Jarosław Sadziński
b80e56a4e1 (core) Custom Widget column mapping feature.
Summary:
Exposing new API in CustomSectionAPI for column mapping.

The custom widget can call configure method (or use a ready method) with additional parameter "columns".
This parameter is a list of column names that should be mapped by the user.
Mapping configuration is exposed through an additional method in the CustomSectionAPI "mappings". It is also available
through the onRecord(s) event.

This DIFF is connected with PR for grist-widgets repository https://github.com/gristlabs/grist-widget/pull/15

Design document and discussion: https://grist.quip.com/Y2waA8h8Zuzu/Custom-Widget-field-mapping

Test Plan: browser tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3241
2022-02-08 17:41:04 +01:00
Cyprien P
196ab6c473 (core) Adds cell context menu
Summary:
 - Brings in a new utility `contextMenu` to open context menu next to the mouse position
 - Use this utility to show a CellContextMenu, that sort of merge cell context menu and column context menu together.
 - Show cell context menu on context click on any grid's cell.
 - Also takes care of showing the row context menu for detail view on a context click that occurs on cells and not only on the row num header as it was the case prior to this diff.
 - task: https://gristlabs.getgrist.com/doc/check-ins/p/5#a1.s9.r1529.c31
 - discussion: https://grist.quip.com/ETGkAroLnc0Y/Cell-Context-Menu

{F40092}

Test Plan: - Adds project test and nbrowser for cell context menu and new cases for the detail row context menu.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3237
2022-02-07 15:46:17 +01:00
Alex Hall
ec7bc9bef3 (core) Remove code related to newui
Summary: Change code that conditionally depended on #newui in the URL to assume that everything is always in the new UI now.

Test Plan: this

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3246
2022-02-04 20:29:00 +02:00
Alex Hall
5d671bf0b3 (core) New type conversion in the backend
Summary: This is https://phab.getgrist.com/D3205 plus some changes (https://github.com/dsagal/grist/compare/type-convert...type-convert-server?expand=1) that move the conversion process to the backend. A new user action ConvertFromColumn uses `call_external` so that the data engine can delegate back to ActiveDoc. Code for creating formatters and parsers is significantly refactored so that most of the logic is in `common` and can be used in different ways.

Test Plan: The original diff adds plenty of tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3240
2022-02-04 20:28:13 +02:00
Alex Hall
64abfcb0ac (core) Remove some unused code, especially in ViewConfigTab
Summary: Remove several unused methods in ViewConfigTab.js, and all of SummaryConfig.js.

Test Plan: this

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3245
2022-02-04 12:15:00 +02:00
Alex Hall
f110ffdafd (core) Follow chain of same-record links for getDefaultColValues
Summary:
When two widgets are linked by same-record linking, and the source of that link is also filter-linked, then it will pick up default values from its own filter-link source, but the same-record-link target didn't. This fixes that so that default values are filled in intuitively.

Moved the logic of linkingState, linkingFilter, and getDefaultColValues from BaseView.js to LinkingState.ts and ViewSectionRec.ts. In particular getDefaultColValues is now a property of LinkingState which may be copied from the source view section for a same-record link.

Note that `ViewSectionRec.linkingFilter` no longer uses `computerBuilder` and thus doesn't ignore dependencies inside LinkingState any more. I couldn't figure out how to make `linkingFilter` a `pureComputed` (otherwise I get recursion errors) that ignores dependencies. In any case, it's now important to have a dependency on `srcSection.linkingState()` for `getDefaultColValues` to work correctly, so I think this is for the best.

Test Plan: Added a new nbrowser test and fixture.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3238
2022-02-03 18:51:02 +02:00
Alex Hall
22006754af (core) Remove 'new' from IDocPage
Summary: The value seems unused

Test Plan: This is me testing if it's actually unused

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3244
2022-02-03 18:49:26 +02:00
George Gevoian
ddb67ff44e (core) Make new account page mobile-friendly
Summary:
Tweaks CSS of account page, ApiKey and MFAConfig to work better
on narrow-screen devices.

Test Plan: Tested manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3234
2022-01-26 23:38:57 -08:00
Jarosław Sadziński
ae6342810d (core) Summary columns improvemnt.
Summary:
Improving user experience on summary columns.
- Showing 'not-allowed' cursor on sections/menus that can't be changed
- Disabling menu options and buttons in the column behavior section that converts a formula column to a data column
- Showing nicer error message about converting formula to a data column.

Test Plan: manual tests, no behavior change

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3222
2022-01-25 17:51:48 +01:00
George Gevoian
f74002fe32 (core) Handle old Grist sessions in MFAConfig
Summary:
Grist sessions created pre-MFA do not store access or
refresh tokens, which means that MFA status from Cognito
can't be loaded without requiring re-authentication. MFAConfig
handles this by requiring security verification as usual, and
checking if it needs to reload MFA status on success. If it does, it'll
close the 2FA setup dialog and reload, which should show the correct
2FA configuration status and buttons.

Test Plan: Updated existing tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3231
2022-01-24 14:48:07 -08:00
George Gevoian
3289fe330f (core) Update aws-sdk
Summary:
Bumps aws-sdk to the latest version of v2.

Updates CognitoClient to use fake AWS credentials with service provider. While none
of our calls currently require real credentials, we still need to send fake credentials
with certain calls, such as AssociateSoftwareToken, to avoid getting errors back from
AWS about missing credentials.

Fixes some CSS and alignment issues with the authentication method tile buttons in
MFAConfig.

Test Plan: Existing Cognito tests. Manually tested login and 2FA setup still work.

Reviewers: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3228
2022-01-20 18:38:21 -08:00
George Gevoian
cebd7b2d74 (core) Fix choice/ref list cell color not filling cell
Summary:
This fixes the cell color only applying to the content height of a
choice or reference list cell, and not the entire cell. Could
be reproduced by causing a row other than the choice/reference
list to grow in height.

Test Plan: Tested manually.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3226
2022-01-20 14:58:20 -08:00
Paul Fitzpatrick
7440485ebe (core) run survey for new users only when a survey doc is set
Summary:
stop providing a default document id DOC_ID_NEW_USER_INFO for
surveying, and don't show survey if a document id is not available.

Test Plan: existing tests pass; grist-core checked

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3225
2022-01-20 15:50:35 -05:00
Alex Hall
0c80547e0d (core) Change UndoStack._linkMap to store an array of action groups for each linkId
Summary:
While working on type conversion, I ran into a bug when multiple action groups belonging to a bundle are quickly sent. `DocData._sendActionsImpl` can set the same `linkId: this._lastActionNum` on multiple action groups before `_lastActionNum` is updated with a result from the server. Only one of these groups gets saved in `UndoStack._linkMap` so undoing a bundle misses out on some action groups.

This diff associates each `linkId` key with an array of action groups instead of just one, then combines them all together when undoing.

Test Plan: I've confirmed that this fixes my problem within my type conversion diff, but I haven't found a way to reproduce the general problem in master. I think the existing tests are probably fine since undo is tested extensively everywhere, but I'd also like to see if there are existing bugs which this fixes.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3223
2022-01-20 19:00:01 +02:00
George Gevoian
db7d1802ce (core) Clarify phone message only supports U.S. numbers
Summary:
Adds a mention of phone message (SMS) verification currently only
supporting U.S. phone numbers, and updates CognitoClient to return
an error when a non-U.S. number is submitted to register_sms/.

Test Plan: Server test.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3227
2022-01-20 08:27:58 -08:00
George Gevoian
0d005eb78d (core) Enable MFA configuration (and add SMS)
Summary:
Enables configuration of multi-factor authentication from the
account page (for users who sign in with email/password), and adds
SMS as an authentication method.

Test Plan: Project, browser and server tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3215
2022-01-19 13:55:54 -08:00
Cyprien P
1b4580d92e (core) Fix error when creating chart with a single column
Summary:
The routine that makes sure that new charts are created with at least
one non-numeric series did not handle correctly when the table has one
single column.

This diff fixes it.

Test Plan: Adds test case to ChartView3.ts

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki, jarek

Differential Revision: https://phab.getgrist.com/D3224
2022-01-19 16:48:31 +01:00
Cyprien P
c714d09eb8 (core) Disallow using non numeric type in chart's series
Summary:
We do not support to show non numeric column as chart series.
However we didn't prevent the user from doing it and it could cause unexpected behaviour such as a missing chart.
This diff addresses that issue by doing two following thing:
1) it prevents user from adding non numeric column as series and
2) it makes sure that if there is a non numeric series it does not mess up the chart (it still can happen that a non numeric series ends up in charts even with 1) for instance if users convert a series' column to a non numeric column for instance).

Links to UI discussion:
 - https://grist.quip.com/wb4gAgrQM2aP#TZEADAKPs8n
 - https://grist.quip.com/wb4gAgrQM2aP#TZEADAP8S8N

Test Plan:
 - new behaviour covered in nbrowser/ChartView3.ts
 - Some test were using non-numeric column as series, diff fixes that to.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3206
2022-01-18 17:56:53 +01:00
Dmitry S
f9f4245466 (core) Fix adding multiple fields when editing card layout
Summary:
Bug reported by user (https://gristlabs.getgrist.com/k1f3bMzUvitZ/User-Feedback#a1.s3.r408.c8)
Also make the test of editing RecordLayout work better (it was failing on Mac even on master)

Test Plan: Added a test case for adding multiple fields

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3219
2022-01-18 10:37:46 -05:00
Jarosław Sadziński
d2077bc486 (core) Improving experience when editing group-by column.
Summary:
Improving experience when editing group-by column:
- Disable column rename
- Allow changing most widget properties:
 - Color/Background
 - Number format
 - Date/DateTime format (but not the timezone)
 - All toggle options (for toggle column)
- Remove Edit button on Choice Edit
- Changing the underlying column should reset all those options back to the original column.

Test Plan: nbrowser

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3216
2022-01-18 14:31:31 +01:00
Dmitry S
215bb90e68 (core) Replace questionnaire for new users with a popup asking for just their primary use-case.
Summary:
- WelcomeQuestions implements the new popup.
- Popup shows up on any doc-list page, the first time the user visits one after
  signing up and setting their name.
- Submits responses to the same "New User Questions" doc, which has been
  changed to accept two new columns (ChoiceList of use_cases, and Text for
  use_other).
- Improve modals on mobile along the way.

Test Plan: Added browser tests and tested manually

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3213
2022-01-14 10:16:11 -05:00
George Gevoian
ba6ecc5e9e (core) Move user profile to new page and begin MFA work
Summary:
The user profile dialog is now a separate page, in preparation
for upcoming work to enable MFA. This commit also contains
some MFA changes, but the UI is currently disabled and the
implementation is limited to software tokens (TOTP) only.

Test Plan:
Updated browser tests for new profile page. Tests for MFAConfig
and CognitoClient will be added in a later diff, once the UI is enabled.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3199
2022-01-13 21:21:49 -08:00
Alex Hall
8f531ef622 (core) Reference and ReferenceList formatters
Summary:
Previously, ref/reflist columns were formatted entirely based on their visible column, since they received values from the visible or display columns rather than the actual row IDs. This creates `ReferenceFormatter` and `ReferenceListFormatter` which still delegate most of the formatting work to a visible column formatter but fix a few issues:

- ReferenceList columns now actually use the options (e.g. date format) of the visible column to format their elements. Previously they were formatted generically because the visible column formatter wasn't expecting a list.
- Invalid references aren't formatted with an `#Invalid Ref` prefix.
- When the ref column displays the Row ID, it doesn't have a visible or display column. Previously this led to the references being formatted as just numbers in most cases, with special code in the widget to display them like `Table1[2]`. Now they are consistently formatted in that style throughout.

Test Plan: Updated existing tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3212
2022-01-13 18:09:33 +02:00
Jarosław Sadziński
85ef873ce5 (core) Widget options api
Summary:
Adding configuration options for CustomWidgets.

Custom widgets can now store options (in JSON) in viewSection metadata.

Changes in grist-plugin-api:
- Adding onOptions handler, that will be invoked when the widget is ready and when the configuration is changed
- Adding WidgetAPI - new API to read and save a configuration for widget.

Changes in Grist:
- Rewriting CustomView code, and extracting code that is responsible for showing the iframe and registering Rpc.
- Adding Open Configuration button to Widget section in the Creator panel and in the section menu.
- Custom Widgets can implement "configure" method, to show configuration screen when requested.

Test Plan: Browser tests.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3185
2022-01-13 11:10:17 +01:00
Cyprien P
98a331a1e4 (core) Fix the name of the grouped data series when the value is blank
Summary:
When using the grouped data option with a column (A) that has some
blank values, all rows with blank values for A are grouped into one
series.

The issue is that the name that showed on the legend for that series
used to be the name of the yseries, and not the name of the value.

This diff fixes it by showing `[Blank]` instead.

Test Plan: Includes new test case.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3210
2022-01-11 16:36:07 +01:00
Jarosław Sadziński
08881d9663 (core) Scrolling to the active record when editor is activated
Summary: When an editor is activated by typing, the active view should be scrolled to the active record.

Test Plan: new tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3196
2022-01-10 23:35:08 +01:00
Paul Fitzpatrick
975eed8564 (core) fix browser check and favicon in grist-core
Summary:
A check for old browsers and a Grist favicon were not available in
grist-core, leaving harmless but distracting errors in logs. This
adds them.

Test Plan: checked manually

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3207
2022-01-05 18:15:05 -05:00
George Gevoian
62a6190970 (core) Add button for removing doc tours
Summary:
Document owners can now remove doc tours by pressing the button located
to the right of 'Tour of this Document' in the left panel.

Test Plan: Browser test.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3202
2022-01-05 08:19:46 -08:00
Cyprien P
89dc9334c3 (core) Fix error bars for bar chart
Summary:
The culprit was `series = uniqXValues(series);` because it creates new
series objects when they are used as keys to access error bars info
(`errorBars.get(line)`).

Fixed by making uniqXValues mutating series instead.

Test Plan: Adds a case to test error bars with bar charts.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3198
2022-01-04 17:58:52 +01:00
Jarosław Sadziński
50821f655d (core) Duplicate page should copy filters.
Summary: Duplicate page also copies filters for all sections.

Test Plan: nbrowser tests

Reviewers: cyprien, alexmojaki

Reviewed By: cyprien, alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3203
2022-01-04 14:49:33 +01:00
Jarosław Sadziński
52d3f63203 (core) Fixing bug with deleting charts
Summary: Fix for a bug. When a chart had a "Group Data" checked, deleting it produced a JS error.

Test Plan: browser test

Reviewers: cyprien

Reviewed By: cyprien

Differential Revision: https://phab.getgrist.com/D3200
2022-01-03 16:33:12 +01:00
Jarosław Sadziński
24c57e05d7 (core) Fixing dispose bug on GridView
Summary:
When disposing a GridView (i.e. switching pages) there was JS error. It was caused, by
autodisposing knockout observable.

Test Plan: manual tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3201
2021-12-29 22:42:29 +01:00
Jarosław Sadziński
0482c83771 (core) Better UX in full-edit mode for the formula editor
Summary:
Improving UX for the formula editor
- Formula editor will go into full edit mode only on formula change (not on a mouse click)
- Adding column highlight and a tooltip when in full edit mode

Test Plan: nbrowser tests

Reviewers: cyprien

Reviewed By: cyprien

Differential Revision: https://phab.getgrist.com/D3194
2021-12-27 13:56:04 +01:00
Cyprien P
e99cc3ae08 (core) Fix pie chart sectors's ordering issue.
Summary:
By default, plotly's pie chart sort lables by values.

This is iconsistent with how bar charts work and especially annoying
in case of a linked chart because values can change when user navigate
the linked table which causes colors (and display order) of each label
to change. Making it hard to keep track values.

[[ https://grist.quip.com/wb4gAgrQM2aP/Chart-Improvements-November-2021#temp:C:TZE88067825d66c415da9e839488 | Link to video with more details about the issue ]]

Test Plan: Adds a new test case.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3193
2021-12-22 17:34:25 +01:00
Dmitry S
e0fb281eba (core) When searching, use formatter.formatAny() to handle values of invalid type.
Summary:
This turns out necessary because ReferenceList columns are formatted
using the formatter of the associated visibleCol. This works correctly
in dedicated widgets, but in generic code (like SearchModel here), this
formatter needs to handle unexpected values (of type ReferenceList).

Without the fix, it produces JS errors when search reaches a
RefList:<Date> column.

A better fix would allow a formatter to know that it expects a ReferenceList,
AND to know how to format each value of it, but that's a bigger question
that's outside the scope of this fix.

Test Plan: Includes a browser test which reproduces the bug.

Reviewers: cyprien

Reviewed By: cyprien

Differential Revision: https://phab.getgrist.com/D3195
2021-12-21 15:35:40 -05:00
Jarosław Sadziński
c1de16aee7 (core) Scrolling to the active record on search
Summary:
Two bugs fixed:
1. On search, when the first result is in the active record, GridView wasn't scrolling to the active record.
2. When an active record was not visible, GridView wasn't scrolling to the active record when the column index was changed.

The problem was that the scrolling behavior was based only on rowIndex which isn't changed (and doesn't notify subscribers) when a column index changes or when the search highlights a cell.
This diff makes the computed depend also on the fieldIndex, and is introducing a new method that can scroll to the active record on demand (which is used by the search).

Test Plan: Updated tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3191
2021-12-21 09:57:21 +01:00
Jarosław Sadziński
d08fdd772e (core) Fixing bug with undoing page delation with a custom layout
Summary:
Fixing a bug: When removing a page with linked sections and then undoing, there are two JS errors raised:
- flexSize is not a function
- getter is not a function

Test Plan: nbrowser tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3192
2021-12-20 19:32:14 +01:00
Alex Hall
c470c4041b (core) Use visibleCol instead of displayCol with createFormatter
Summary:
Some things (like rendering cells) use the `visibleCol` for `createFormatter`, while other things (like `CopySelection`) used the `displayCol`. For references, the display column has type Any and doesn't know about the original formatting. This resulted in formatting being lost when copying from reference columns even though formatting was preserved when copying from the original (visible) column which looked identical. This diff fixes this and ensures that `createFormatter` is always used with the `visibleCol`. This was agreed on in https://grist.slack.com/archives/C0234CPPXPA/p1639571321043000

Additionally:

- Replaces the functions `createVisibleColFormatter` computed properties `visibleColFormatter` as suggested by a `TODO`.
- Extracts common code from `createVisibleColFormatter` in `ColumnRec` and `ViewFieldRec`

Test Plan: Fixed a test in CopyPaste which displayed the previous inconsistent behaviour.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3189
2021-12-16 22:19:36 +02:00
Alex Hall
30c8ba3019 (core) Use auto popup placement for DocTour on mobile
Summary: For GristDocTours on mobile, ignore the Placement column and always use automatic placement for popups

Test Plan: Tested manually with chrome devtools

Reviewers: jarek

Reviewed By: jarek

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3188
2021-12-16 16:06:15 +02:00
Dmitry S
f024aaaf5d (core) Fix some bugs with ChoiceList in summary tables, and evaluation of lookups.
Summary:
Addresses several issues:
- Error 'Cannot modify summary group-by column' when changing Text ->
  ChoiceList in the presence of summary tables.
- Error 'ModifyColumn in unexpected position' when changing ChoiceList -> Text
  in the presence of summary tables.
- Double-evaluation of trigger formulas in some cases.

Fixes include:
- Fixed verification that summary group-by columns match the underlying ones,
  and added comments to explain.
- Avoid updating non-metadata lookups after each doc-action (early lookups
  generated extra actions to populate summary tables, causing the 'ModifyColumn
  in unexpected position' bug)
- When updating formulas, do update lookups first.
- Made a client-side tweak to avoid a JS error in case of some undos.

Solution to reduce lookups is based on https://phab.getgrist.com/D3069?vs=on&id=12445,
and tests for double-evaluation of trigger formulas are taken from there.

Add a new test case to protect against bugs caused by incorrect order of
evaluating #lookup columns.

Enhanced ChoiceList browser test to check a conversion scenario in the presence
of summary tables, previously triggering bugs.

Test Plan: Various tests added or enhanced.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3184
2021-12-15 09:51:18 -05:00
Jarosław Sadziński
1ae586cf42 (core) Adding Skip options when importing multiple tables.
Summary:
Adding new destination "Skip" for multiple table imports.
Selecting this destination skips the import and makes the preview grayed out.

Test Plan: New Tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3181
2021-12-13 19:07:33 +01:00
Paul Fitzpatrick
d99db8d016 (core) move more tests to grist-core
Summary:
 * Tie build and run-time docker base images to a consistent version (buster)
 * Extend the test login system activated by GRIST_TEST_LOGIN to ease porting tests that currently rely on cognito (many)
 * Make org resets work in absence of billing endpoints
 * When in-memory session caches are used, add missing invalidation steps
 * Pass org information through sign-ups/sign-ins more carefully
 * For CORS, explicitly trust GRIST_HOST origin when set
 * Move some fixtures and tests to core, focussing on tests that cover existing failures or are in the set of tests run on deployments
 * Retain regular `test` target to run the test suite directly, without docker
 * Add a `test:smoke` target to run a single simple test without `GRIST_TEST_LOGIN` activated
 * Add a `test:docker` target to run the tests against a grist-core docker image - since tests rely on certain fixture teams/docs, added `TEST_SUPPORT_API_KEY` and `TEST_ADD_SAMPLES` flags to ease porting

The tests ported were `nbrowser` tests: `ActionLog.ts` (the first test I tend to port to anything, out of habit), `Fork.ts` (exercises a lot of doc creation paths), `HomeIntro.ts` (a lot of DocMenu exercise), and `DuplicateDocument.ts` (covers a feature known to be failing prior to this diff, the CORS tweak resolves it).

Test Plan: Manually tested via `buildtools/build_core.sh`. In follow up, I want to add running the `test:docker` target in grist-core's workflows. In jenkins, only the smoke test is run. There'd be an argument for running all tests, but they include particularly slow tests, and are duplicates of tests already run (in different configuration admittedly), so I'd like to try first just using them in grist-core to gate updates to any packaged version of Grist (the docker image currently).

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3176
2021-12-10 18:33:07 -05:00
Paul Fitzpatrick
307966e84f (core) go ahead and create fork if non-owner wants to do "View As" on prefork
Summary:
Backstory: to make examples easier to play with, we:
 * Add a special FullCopies permission to let anyone fork/copy them regardless of other access rules
 * Open the examples in "prefork" mode by default
That means a random person can open an example and already feel like an owner of it. Getting to this point requires some gymnastics on the back end. As soon as the person makes any change to the document they become truly the owner (of their fork), and life is simple for the back end.

But, if that person does "View As" to look at the preforked document, that is a step too far for the back end - a user, with a special somewhat complicated exception allowing them to act as an owner for some purposes, now wants to pretend to be another user. The logic for this on the back end was doable, but looked hard to review and be confident of, with now three identities with subtle nuances in their interrelationship.

So with this diff, if a non-owner attempts to "View As" another user on a prefork, the client will just fork the document first. This is in principle not necessary, but is much simpler from a security perspective.

Test Plan: extended test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3179
2021-12-10 14:45:36 -05:00
Dmitry S
8100272e9a (core) Update HelpScout beacon to work with embedded documentation articles.
Summary:
- Fix base href in HelpScout beacon when showing articles (in particular for Firefox)
- Show the 'Answers' tab normally except when reporting an error.
- Combine the "Give Feedback" and "Help Center" buttons into one that normally
  opens the beacon (with a link to Help Center and to Community Forum), and a
  smaller one that opens the Help Center site in a new tab.
- Update HELP_SCOUT_* env vars to use _V2 suffix, to allow them to coexist with
  code using the previous beacon.

Test Plan: Updated the browser test to check the new behavior.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3170
2021-12-09 22:22:55 -05:00
Alex Hall
6b448567c9 (core) Refactor more value parsing code into common
Summary:
Following discussion in https://phab.getgrist.com/D3164:

- Change createParser to accept docData and one or two metadata row IDs and let it extract the metadata, so it's more easily usable in the server.
- Change ViewFieldRec.valueParser observable to a function createValueParser.

Test Plan: Existing tests.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3172
2021-12-09 21:05:27 +02:00
George Gevoian
258d3cbb89 (core) Swap table and file names in Importer
Summary:
The table name now comes first, making it easier to distinguish
tables coming from an Excel file with multiple sheets.

Test Plan: Tested manually and updated browser test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3173
2021-12-08 15:49:53 -08:00
George Gevoian
f8e2cc4de3 (core) Skip saving column resizes in read-only mode
Summary:
This makes it so a notification about insufficient
write access is no longer shown every time a user
in a read-only document resizes a column.

Test Plan: Browser test.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3171
2021-12-07 14:37:25 -08:00
Dmitry S
e4314f9def (core) Update grainjs, fix some code affected by stronger types.
Summary: Also clean up dom-ownership in Charts using the new grainjs maybeOwned() method.

Test Plan: Should be no behaviour changes; existing tests should pass

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3166
2021-12-07 17:31:03 -05:00
Dmitry S
7a6d726daa (core) Change datepicker in DateEditor to use moment format, show AltText in DateEditor
Summary:
- Rather than translate from moment format to that of bootstrap-datepicker, use
  the customization methods to format datepicker dates using moment directly.
- Fix issue with parseDate() when format includes tokens like Mo or Do
- Fix issue in parseDateTime() that could produce an off-by-one error in date
  depending on local timezone.
- When opening DateEditor, show AltText value if present.

- Add crossorigin=anonymous to scripts that were missing it (including
  bootstrap-datepicker), to ensure that errors from them are reported properly
  rather than as 'Script error.'

Test Plan:
Added test cases to parseDate() test for low-level fixes; added a
browser test for the fixed DateEditor behavior.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3169
2021-12-07 11:33:49 -05:00
Alex Hall
faec8177ab (core) Use MetaTableData more
Summary:
Add more method overrides to MetaTableData for extra type safety.

Use MetaTableData, MetaRowRecord, and getMetaTable in more places.

Test Plan: Mostly it just has to compile. Tested manually that types are being checked more strictly now, e.g. by adding a typo to property names. Some type casting has also been removed.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3168
2021-12-07 17:09:58 +02:00
Alex Hall
116fb15eda (core) Move most of the reference parsing code into common so that the server can use it
Summary: Refactoring in preparation for parsing strings from the API. The plan is that the API code will only need to do a server-side version of the code in ViewFieldRec.valueParser (minus ReferenceUtils) which is quite minimal.

Test Plan: Nothing extra here, I don't think it's needed. This stuff will get tested more in a future diff which changes the API.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3164
2021-12-07 01:19:18 +02:00
George Gevoian
7f08934cf0 (core) Skip saving layout resizes in read-only mode
Summary:
This change causes a notification about missing
write access to no longer be shown when resizing layouts
as a viewer.

Test Plan: Browser test.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3167
2021-12-06 07:37:45 -08:00
Dmitry S
ef1fc916e6 (core) Fix "Column Label and ID" widget to avoid getting truncated on some browsers
Summary:
The widget showing column label, ID, and a button for keeping them linked, used
to get truncated depending on the default size of inputs. This makes it
resize dynamically based on the width of the creator panel.

Test Plan: CSS-only change, tested manually on FF and Chrome.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3165
2021-12-06 10:21:10 -05:00
Jarosław Sadziński
1425461cd8 (core) Exposing custom widgets on the UI
Summary:
Exposing custom widgets as a dropdown menu in custom section configuration panel.

Adding new environmental variable GRIST_WIDGET_LIST_URL that points to a
json file with an array of available widgets. When not present, custom widget menu is
hidden, exposing only Custom URL option.

Available widget list can be fetched from:
https://github.com/gristlabs/grist-widget/releases/download/latest/manifest.json

Test Plan: New tests, and updated old ones.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3127
2021-12-01 18:21:06 +01:00
Alex Hall
be96db4689 (core) Remove code related to _grist_TableViews
Summary:
Removed some TS and python code interacting with this meta table. Not touching schema or migrations.

This is not really necessary, just checking my understanding and cleaning up in preparation for raw data views. I can also remove _grist_TabItems code while I'm at it.

Test Plan: this

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3161
2021-11-30 23:51:30 +02:00
Alex Hall
90fdb55bfd (core) Fix imports in DocTour.ts, particularly wildcard lodash import causing a big bundle
Summary: This reduces the size of the bundle 'vendors~GristDoc' from 587036 to 46526. Woops.

Test Plan: this

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3160
2021-11-30 16:47:06 +02:00
Jarosław Sadziński
e482427e83 (core) JS error on creator panel and formula editor.
Summary:
Fixing js error that happens when closing creator panel with active formula editor.

Styling behavior menu with common styles.

Test Plan: Browser tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3150
2021-11-30 14:28:22 +01:00
Alex Hall
551ea28fc4 (core) Check document ID when parsing pasted references
Summary: Add doc-id attribute to copied HTML columns next to column type. Only use the raw value (rather than the display value) when the parsed doc-id from pasted HTML matches the current document ID, similar to ensuring that the type matches. This only applies to references and reflists.

Test Plan: Extended CopyPaste.ts

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3154
2021-11-30 13:40:17 +02:00
George Gevoian
e997d091b3 (core) Disable bfcache for all browsers
Summary:
Grist would get stuck in a broken state in certain browsers
that enabled the bfcache when the browser back/forward
buttons were clicked. Firefox automatically disabled the cache
since we listen on the 'beforeunload' event, but Chrome and Safari
did not. This change forces a page refresh on pageshow if we
detect that Grist was loaded from the bfcache.

Test Plan: Tested manually in various browsers.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz, jarek

Differential Revision: https://phab.getgrist.com/D3151
2021-11-24 09:47:00 -08:00
Cyprien P
0b437d1544 (core) Brings in the new donut charts.
Summary:
  - Donut charts is same as pie chart with few extra options to control size of the hole and to show/hide a big total in it.
  - Add a new option type to tune a numeric options using a slider/spinner/keyboard.
  - Add a new option type to tune a numeric options using a slider/keyboard
  - Add a new .propWithDefault method to ObjObservable to allows to set a default value when options is undefined.
  - mocha-webdriver's findContent does not work to find content in svg elements. So had to tweak original function into a sister function using .textContent instead.

Test Plan: Adds new tests

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: anaisconce, dsagal

Differential Revision: https://phab.getgrist.com/D3107
2021-11-23 08:56:38 +01:00
George Gevoian
32bb89235e (core) Polish Importer UI
Summary:
Changes include:
 * Hide the colum matching section for new destinations (for now).
 * Make the preview table read-only.
 * Don't show helper column IDs when the formula editor is open.
 * Fix the formula editor autocomplete to show suggestions
 from the active transform section.
 * Hide the formula icons in the preview table, and other unnecessary
 UI elements such as row dropdown menus.
 * Keep preview loading spinner shown if scheduled (i.e. debounced) diff updates exist.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3148
2021-11-22 11:26:09 -08:00
George Gevoian
7fe4423a6f (core) Allow filtering hidden columns
Summary:
Existing filters are now moved out of fields
and into a new metadata table for filters, and the
client is updated to retrieve/update/save filters from
the new table. This enables storing of filters for
columns that don't have fields (notably, hidden columns).

Test Plan: Browser and server tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3138
2021-11-22 10:26:08 -08:00
Alex Hall
0d460ac2d4 (core) Parsing pasted ChoiceLists
Summary:
Added ChoiceListParser capable of parsing JSON, CSVs, and other strings containing user-configured choices (e.g. separated by spaces)

I got a little carried away here. It works, and I can't think of any bugs, but it's complicated enough that there could be hidden edge cases or difficulties maintaining it in the future. The advantage of the current method is that it should work well for ambiguous or poorly formatted inputs, e.g. choices separated only by spaces or choices containing commas which are not escaped/quoted properly. The code can be vastly simplified if we don't try to support that and require that users paste proper JSON or CSVs.

Test Plan: Added a new file test/common/ChoiceListParser.ts with pure unit tests. Waiting for approval of the overall approach before adding to the nbrowser CopyPaste test.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3141
2021-11-20 19:00:36 +02:00
Dmitry S
7f8f3dc0be (core) Set min-height for cells, including in Card view, to make them look consistent.
Summary:
Before:
{F37978}

After:
{F37977}

Test Plan: Add a test case for the Card view that the resulting heights are consistent.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3149
2021-11-19 17:23:03 -05:00
Jarosław Sadziński
fc50079e03 (core) Storing last position for doc and user
Summary: Last position should be stored for document and user.

Test Plan: Updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3143
2021-11-19 12:01:49 +01:00
Jarosław Sadziński
745ddc21ac (core) Expanding search textbox to full height.
Summary: Expanding search input field to full available height, to make the clickable area bigger.

Test Plan: Manual tests on browserstack

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3145
2021-11-18 23:48:07 +01:00
George Gevoian
05eb7afcb6 (core) Show ACL link in UserManager conditionally
Summary:
The 'Open Access Rules' link is now hidden unless the
UserManager is opened inside a document, and the resource
that users are being managed for is a document.

Test Plan: Browser tests.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3142
2021-11-17 08:43:14 -08:00
George Gevoian
08b1286f4f (core) Add column matching to Importer
Summary:
The Importer dialog is now maximized, showing additional column
matching options and information on the left, with the preview
table shown on the right. Columns can be mapped via a select menu
listing all source columns, or by clicking a formula field next to
the menu and directly editing the transform formula.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3096
2021-11-09 12:30:52 -08:00
Jarosław Sadziński
96fa7ad562 (core) Error message on Duplicate Document
Summary: Fixing error message when user can't copy document.

Test Plan: Updated tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3130
2021-11-09 19:12:57 +01:00
Alex Hall
ecb30eebb8 (core) Parsing multiple values in reflists, parsing refs without table data in client
Summary:
Added a new object type code `l` (for lookup) which can be used in user actions as a temporary cell value in ref[list] columns and is immediately converted to a row ID in the data engine. The value contains the original raw string (to be used as alt text), the column ID to lookup (typically the visible column) and one or more values to lookup.

For reflists, valueParser now tries parsing the string first as JSON, then as a CSV row, and applies the visible column parsed to each item.

Both ref and reflists columns no longer format the parsed value when there's no matching reference, the original unparsed string is used as alttext instead.

Test Plan: Added another table "Multi-References" to CopyPaste test. Made that table and the References table test with and without table data loaded in the browser.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3118
2021-11-09 14:41:04 +02:00
Jarosław Sadziński
b6dd066b7f (core) Adding more space for text editor to avoid scrollbars
Summary:
For Windows and Ubuntu (on chrome) texteditor was showing scrollbars for
very long words. Adding more space to fix this issue.

Test Plan: manual

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3122
2021-11-09 12:36:40 +01:00
Alex Hall
45fc46070d (core) Fix converting column to ChoiceList when cells contain JSON arrays with non-string values
Summary: title

Test Plan: Tested manually, I don't think this needs an automated test. Made a text column with a value `[1, 2]` and converted the column to choice list. Previously this threw a JS error that `tag.trim` wasn't a function. Works now, suggests `1` and `2` as choices in the configuration.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3128
2021-11-09 12:53:20 +02:00
George Gevoian
59699bf446 (core) Add additional org items to AppHeader
Summary:
Adds links to manage team and go to billing account in
the org menu (opened by clicking the dropdown in the
top-left corner of Grist). Tweaks some wording of items
in both AppHeader and AccountWidget, and adds a link
to create a new team site to the Site Switcher in both
menus.

Also tweaks the UI of UserManager by adding
an animation when the manager is opened from the
doc access dialog.

Test Plan: Browser tests.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3121
2021-11-05 08:19:51 -07:00
Jarosław Sadziński
e8e614c584 (core) Formula UI redesign
Summary:
Redesigning column type section to make it more user-friendly. Introducing column behavior concept.
Column can be either:
- Empty Formula Column: initial state (user can convert to Formula/Data Column)
- Data Column: non formula column with or without trigger (with option to add trigger, or convert to formula)
- Formula Column: pure formula column, with an option to convert to data column with a trigger.

Test Plan: Existing tests.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3092
2021-11-05 13:07:30 +01:00
Paul Fitzpatrick
877542225d (core) mark engine setting as experimental
Summary:
This marks the engine setting in document settings as experimental,
with a skull and cross-bones.

It also makes sure the setting is shown if PYTHON_VERSION_ON_CREATION
is set (this relates to a separate change to set the default python
version to 3).

Test Plan: manual

Reviewers: alexmojaki, dsagal

Reviewed By: alexmojaki, dsagal

Subscribers: anaisconce

Differential Revision: https://phab.getgrist.com/D3120
2021-11-04 20:17:24 -04:00
Dmitry S
cce679d928 (core) Add some polish to the billing page, particularly for sign-up.
Summary:
- Change "Continue" button to "Review" (we don't charge immediately,
  first show a review screen)
- Show more informative messages for certain failures with discount
  coupons.
- Focus form elements with error, or at least the part of the form
  containing an error.
- Auto-focus discount input box when it gets toggled on.
- Show warning about URL changes only when subdomain is changed.

Test Plan: Updated tests; tested focus and changed error messages manually.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3115
2021-11-04 14:23:53 -04:00
Dmitry S
0dd4ad34f5 (core) Avoid an error when pasting data with first row shorter than the rest
Summary:
Pasting data like `A\nB\tC` was failing because the first row, used for
checking column type (to handle rich data), was failing on an undefined value.

Discovered while trying out the fix in https://phab.getgrist.com/D3110.

Test Plan: Tested manually. The case mentioned now works as expected.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3111
2021-11-04 10:15:56 -04:00
Alex Hall
dc2aee4be9 (core) Trim trailing whitespace when pasting plain text
Summary: title

Test Plan:
Tested manually. Blank rows at the end are no longer pasted. Pasting multiple columns separated by tabs can still have blank cells in some cells of the final rows.

I don't think this needs an automated test.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3110
2021-11-04 01:26:47 +02:00
Jarosław Sadziński
3c72639e25 (core) Adding sort options for columns.
Summary:
Adding sort options for columns.
- Sort menu has a new option "More sort options" that opens up Sort left menu
- Each sort entry has an additional menu with 3 options
-- Order by choice index (for the Choice column, orders by choice position)
-- Empty last (puts empty values last in ascending order, first in descending order)
-- Natural sort (for Text column, compares strings with numbers as numbers)
Updated also CSV/Excel export and api sorting.
Most of the changes in this diff is a sort expression refactoring. Pulling out all the methods
that works on sortExpression array into a single namespace.

Test Plan: Browser tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: dsagal, alexmojaki

Differential Revision: https://phab.getgrist.com/D3077
2021-11-03 15:31:39 +01:00
Alex Hall
0f946616b6 (core) Parsing pasting in reflist columns
Summary: Handle reflist columns in ViewFieldRec.parseValue

Test Plan: Reused section of test of reference columns

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3105
2021-11-03 14:37:08 +02:00
Cyprien P
ab7af2b2ef (core) Fix bars order in chart bars order when multiseries is on.
Summary:
 - Grouping series may result in series with inconsistent number of values. This can result in inconsistent ordering of the bars displayed by plotly.
 - This diff fixes it by consolidating grouped series by adding unll values for each missing xvalues in the series.

Here a is a minimal example of that bug:
{F36639}

Test Plan: Includes new nbrowser test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3085
2021-11-03 08:58:10 +01:00
Alex Hall
d63da496a8 (core) Value parsing for refs, parsing data entry for numbers
Summary:
Handle reference columns in ViewFieldRec.valueParser.

Extracted code for reuse from ReferenceEditor to look up values in the visible column. While I was at it, also extracted a bit of common code from ReferenceEditor and ReferenceListEditor into a new class ReferenceUtils. More refactoring could be done in this area but it's out of scope.

Changed NTextEditor to use field.valueParser, which affects numeric and reference fields. In particular this means numbers are parsed on data entry, it doesn't change anything for references.

Test Plan:
Added more CopyPaste testing to test references.

Tested entering slightly formatted numbers in NumberFormatting.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3094
2021-11-01 19:31:52 +02:00
Dmitry S
f0da3eb3b2 (core) Select previous search input when clicking the search icon
Summary:
Searching with the keyboard shortcut selected the previous search text in the
search box, but using the Search icon did not. A user reported it as an
inconvenience: having to manually delete the value before searching for a new
one.

Test Plan: Verified manually

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3102
2021-11-01 01:28:08 -04:00
Dmitry S
65e743931b (core) Convert CopySelection and tableUtil to typescript
Summary:
- This should make these easier to work with and make changes to.
- Removes one unused method.

Test Plan: No changes of behavior, existing tests should pass.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3091
2021-10-25 18:55:26 -04:00
Paul Fitzpatrick
f7c9919120 (core) annotate shares listed in UserManager for documents
Summary:
This gives more guidance to users when editing document shares in the UserManager dialog.

  * For a document on a team site, any shares with team members are marked `Team member`.
  * Shares that count as external collaborators are marked for documents on a team or personal site as `collaborator` (personal site) or `outside collaborator` (team site).
  * Collaborators are marked `1 of 2`, `2 of 2`, and then `limit exceeded`.
  * On a team site, links are offered for each collaborator to add them to the team. The links lead to a prefilled dialog for managing team membership which can be confirmed immediately, allowing the user to continue without interruption.
  * On a personal site, for the last collaborator and beyond, a link is added for creating a team. This isn't seamless since creating a team involves billing etc.

There's a small unrelated tweak in tests to remove a confusing import from `test/browser` in `test/server`.

One thing I didn't get to is checking if owner of doc is owner of site. If they aren't, they may try to add a member and be denied at that point - it would be more polite to change messaging earlier for them.

Test Plan: added and updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3083
2021-10-25 14:29:19 -04:00
Jarosław Sadziński
8eeeae7fbf (core) Fixing scrollbars on ace editor for windows users.
Summary: Fixing formula editor scrollbars for windows users.

Test Plan: Manual

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3087
2021-10-25 10:56:43 +02:00
Alex Hall
99878c08ed (core) Add ValueParser, use when pasting
Summary:
Add ValueParser file, base class, and subclasses for column types. Only NumericParser is used for now.

Add valueParser field to ViewFieldRec.

Use valueParser when parsing pasted text data in Grid and Detail views.

Test Plan: Add test to nbrowser CopyPaste suite, copying into a numeric column with different currency and locale settings.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3082
2021-10-21 21:43:19 +02:00
George Gevoian
f2e11a5329 (core) Migrate to Stripe v8 + implement discount codes
Summary:
New plan signups now include a discount code field in
the signup form. If a valid discount code is entered, a
discount will be applied on the confirmation page.

Test Plan: Browser and server tests.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3076
2021-10-20 12:34:03 -07:00
Cyprien P
e3801a5eb9 (core) Fix pie sectors order according to section sort spec
Summary:
Plotly sorts pie charts sectors by default and that is overiding the
section ordering. This diff fixes that by passing setting .sort to
false (thus disabling reordering) when there is a sort spec going on.

Issue was reported by user: https://gristlabs.getgrist.com/k1f3bMzUvitZ/User-Feedback#a1.s3.r333.c19

Test Plan: Added nbrowser test

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3075
2021-10-15 08:38:48 +02:00
Dmitry S
c8da5c7356 (core) Fix Card view linked to summary to show card navigation buttons
Test Plan: Added a test case to verify that nav buttons are now shown and work.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3073
2021-10-14 12:26:54 -04:00
Jarosław Sadziński
67ec52365a (core) Showing links in text cells
Summary:
When there is a link in a text cell (and formula cells), it will be
rendered with a little clickable icon wrapped in the anchor tag
with a proper link. Only links that starts with https? will be
rendered as links.

Links are shown in a Text and Formula fields, inside a GridView,
CardView and in the Import preview dialog.

Test Plan: Browser tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: dsagal, alexmojaki

Differential Revision: https://phab.getgrist.com/D3070
2021-10-13 23:22:03 +02:00
Dmitry S
16eb158673 (core) Make ProfileDialog wait consistently for work to finish before closing.
Summary: - Added functionality to modal.ts to allow pending work to delay the closing of the dialog.

Test Plan: Added a test case that tickled a failure previously.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3071
2021-10-13 16:30:43 -04:00
Alex Hall
a64fb105e3 (core) Use GristObjCode in CellValue
Summary: Makes type checking a bit stronger

Test Plan: it just has to compile

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3065
2021-10-11 14:11:32 +02:00
George Gevoian
62db263d1f (core) Add diff preview to Importer
Summary:
Updates the preview table in Importer to show a diff of changes
when importing into an existing table and updating existing records.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3060
2021-10-08 14:15:07 -07:00
Paul Fitzpatrick
d635c97686 (core) flesh out "View As" feature
Summary:
The users shown by the "View As" button are now drawn from more sources:
 * There are users the document is shared with. This has been rationalized, the behavior was somewhat erratic. If the user is not an owner of the document, the only user of this kind that will be listed is themselves.
 * There are users mentioned in any user attribute table keyed by Email. If name and access columns are present, those are respected, otherwise name is taken from email and access is set to "editors".
 * There are example users provided if there are not many other users available.

Test Plan: added and extended tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3045
2021-10-08 12:00:40 -04:00
Jarosław Sadziński
26766fd4ab (core) Removing error styles from user messages
Summary:
Removing error styles from user messages. Only unexpected errors are styled with red icon and border.
Removing reportSuccess message - leaving it for another diff.

Test Plan: manual tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3063
2021-10-07 17:20:52 +02:00
Jarosław Sadziński
a2e066176c (core) Label editor for Choice/ChoiceList column editor
Summary:
Allowing a user to change labels' in Choice/ChoiceList entry editor. For updated
entries, renaming those values in all cells in the column.

Test Plan: Updated tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3057
2021-10-07 11:55:38 +02:00
Dmitry S
cf7a3153f9 (core) When hidden pages are present in the page list, allow removing them
Summary:
After an incomplete import, any GristHidden_* tables will show up in the page
list, but may not be removable if there is only one non-hidden table remaining.
Such tables should still be removable in this case.

Test Plan: Added a test case

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3058
2021-10-05 15:39:42 -04:00
Jarosław Sadziński
40ddb57dfc (core) Adding colors to toast notification
Summary:
Styling toast notification. Adding colors and icons.
In Grist, changed the default style for errors (will be shown in red), and a style for
Linked copied to clipboard (will be shown in Green).
All other colors are not used currently, left for another diff.

Test Plan: manual

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3053
2021-10-05 10:19:25 +02:00
Dmitry S
43a62e7254 (core) Fix autocomplete selection when options include emoji
Summary:
Searching for matching choices was using nativeCompare for a binary search,
while the list was sorted according to localeCompare. This was causing the
search to fail for some strings when the two orderings differ.

Test Plan: Tested manually

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3059
2021-10-04 22:58:00 -04:00
George Gevoian
e1780e4f58 (core) Migrate import code from data engine to Node
Summary:
Finishing imports now occurs in Node instead of the
data engine, which makes it possible to import into
on-demand tables. Merging code was also refactored
and now uses a SQL query to diff source and destination
tables in order to determine what to update or add.

Also fixes a bug where incremental imports involving
Excel files with multiple sheets would fail due to the UI
not serializing merge options correctly.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3046
2021-10-04 10:27:00 -07:00
Dmitry S
1517dca644 (core) Implement DiscourseConnect to enable easy sign-in to community forum
Summary:
- Update cookie module, to support modern sameSite settings
- Add a new cookie, grist_sid_status with less-sensitive value, to let less-trusted subdomains know if user is signed in
- The new cookie is kept in-sync with the session cookie.
- For a user signed in once, allow auto-signin is appropriate.
- For a user signed in with multiple accounts, show a page to select which account to use.
- Move css stylings for rendering users to a separate module.

Test Plan: Added a test case with a simulated Discourse page to test redirects and account-selection page.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3047
2021-10-01 11:24:22 -04:00
Paul Fitzpatrick
b3b7410ede (core) open documents without blocking on data engine
Summary:
With this diff, when a user opens a Grist document in a browser, they will be able to view its contents without waiting for the data engine to start up. Once the data engine starts, it will run a calculation and send any updates made. Changes to the document will be blocked until the engine is started and the initial calculation is complete.

The increase in responsiveness is useful in its own right, and also reduces the impact of an extra startup time in a candidate next-generation sandbox.

A small unrelated fix is included for `core/package.json`, to catch up with a recent change to `package.json`.

A small `./build schema` convenience is added to just rebuild the typescript schema file.

Test Plan: added test; existing tests pass - small fixes needed in some cases because of new timing

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3036
2021-10-01 10:18:56 -04:00
Jarosław Sadziński
42910cb8f7 (core) Extending Google Drive integration scope
Summary:
New environmental variable GOOGLE_DRIVE_SCOPE that modifies the scope
requested for Google Drive integration.
For prod it has value https://www.googleapis.com/auth/drive.file which leaves
current behavior (Grist is allowed only to access public files and for private
files - it fallbacks to Picker).
For staging it has value https://www.googleapis.com/auth/drive.readonly which
allows Grist to access all private files, and fallbacks to Picker only when the file is
neither public nor private).
Default value is https://www.googleapis.com/auth/drive.file

Test Plan: manual and existing tests

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3038
2021-10-01 10:47:12 +02:00
Cyprien P
a0c53f2b61 (core) Fix chart when x axis is set to a choice list column
Summary:
Bug reported by user: https://gristlabs.getgrist.com/doc/check-ins/p/3#a1.s7.r1183.c19p

Setting x axis to a column of type ChoiceList was breaking chart.

This diff fixes that by splitting the record into several records: one for each choice.

`test/nbrowser/ChartView1.ts` was becoming too big and long to run, so this diff introduces `test/nbrowser/ChartView2.ts` to add more test and `test/nbrowser/chartViewTestUtils.ts` to put all utilities or testing charts.

Test Plan: Adds new test.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3041
2021-10-01 10:03:27 +02:00
Paul Fitzpatrick
876a0298a2 (core) do not look at content of recent actions when loading documents
Summary:
This removes the need for any information drawn from the content of recent actions when loading a document.

The undo/redo system does need some facts about recent actions up front. But that system has an important restriction: only actions a particular client is known to have generated can be undone by that client.

So in this diff, as we store which client has performed an action, we also store the few pieces of metadata about that action that the undo/redo system needs: `linkId`, `otherId`, `rowIdHint`, `isUndo` fields. These are all small integers (or in one case a boolean).

An existing limitation is that information about which client has performed which action is stored in memory in the worker, and not persisted anywhere. This diff does not change that limitation, meaning that undos continue to not survive a worker transition. A reasonable way to deal with that would be to back the store with redis.

Test Plan: existing tests pass

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3044
2021-09-29 11:27:02 -04:00
Jarosław Sadziński
8684c9e930 (core) Adding traceback to trigger formulas
Summary:
Traceback is available on the Creator Panel in the formula editor. It is evaluated the same way as for normal formulas.
In case when the traceback is not available, only the error name is displayed with information that traceback is not available.
Cell with an error, when edited, shows the previous valid value that was used before the error happened (or None for new rows).
Value is stored inside the RaisedException object that is stored in a cell.

Test Plan: Created tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki, dsagal

Differential Revision: https://phab.getgrist.com/D3033
2021-09-27 17:12:39 +02:00
Jarosław Sadziński
048c8ee165 (core) Extending default locale list
Summary: Adding more locale codes to support more countries in document settings

Test Plan: existing tests

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3018
2021-09-24 15:10:13 +02:00
Alex Hall
52fd28815e (core) Raise syntax errors that Python can format nicely to show the location
Summary: Update _create_syntax_error_code to raise an error with similar arguments to the real arguments it already has, with our modifications.

Test Plan: Updated python unit tests

Reviewers: jarek, dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3040
2021-09-24 15:07:09 +02:00
Dmitry S
fb583f303a (core) Support 'new' row in anchor links.
Summary:
- Anchor links with row of 'new' could be created but weren't parsed or used
  correctly. This fixes it.
- Also adds UIRowId type for row IDs which includes the special 'new' row. It's
  already been used in places as `number|'new'`, this diff gives it a name usable in app/common
  (it doesn't touch another name, RowId, that's been available in app/client).

Test Plan: Added a test assert for anchor links to new row

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3039
2021-09-24 09:01:10 -04:00
Alex Hall
de76cc48d1 (core) Move cursor to new record in link target when selecting new record in link source
Summary: Update LinkingState._makeSrcCellGetter to account for 'new'

Test Plan: Extended test in RightPanelSelectBy.ts

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3037
2021-09-22 22:57:05 +02:00
Cyprien P
33f056a187 (core) Fix js errors when (un)summarizing underlying table of charts
Summary:
Used to throw js errors: `Resize must be passed a displayed plot div element.`

Summarizing (or unsummarizing) causes the ChartView view instance to
be replace by a new one in the view layout.
However, the problem is that the old view instance get disposed only
after the new view instance is added to the view layout.
This causes the old view layout to try to resize chart while chart dom
has been removed from the dom (which Plotly does not support).

This diff fixes it by checking the the chart dom elemnt is still in
the dom before making the plotly call to resize the chart.

TODO: It feels weird that the old view instance gets disposed after
the new one get added. Maybe we should check that also.

Test Plan: New test added.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3035
2021-09-22 17:04:52 +02:00
Alex Hall
1a8abdcd96 (core) recursiveMoveToCursorPos with new kinds of linking involving lists
Summary: Constructs a ClientQuery in a similar way to LinkingState to handle lists.

Test Plan: Extended SelectBySummary and SelectByRefList tests.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3030
2021-09-20 23:13:24 +02:00
Dmitry S
cd241a633a (core) In make-copy dialog, ensure it's non-saveable while fetching workspaces after a switch of the destination org
Summary:
While switching the destination site in the Duplicate Document dialog, there
were times when it was saveable even though destination workspaces were still
being fetched. This sometimes causes a test failure, with the document getting
saved to a workspace from the previously-selected org.

Test Plan:
Tested manually; reproduced by adding a conditional artificial delay
in _updateWorkspaces helper.

Reviewers: paulfitz, georgegevoian

Reviewed By: paulfitz, georgegevoian

Differential Revision: https://phab.getgrist.com/D3032
2021-09-20 11:54:06 -04:00
Cyprien P
a1480faa09 (core) Changes X-AXIS to LABEL in the axis config when chart is a pie chart
Summary:
Also fixes issue with group data options when switching to pie chart.

Issue was that if the group data picker was on, switching to the pie
chart was not hiding it.

Test Plan: Adds more tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3028
2021-09-20 08:47:23 +02:00
Alex Hall
80bf4ea2a3 (core) Get first unicode character (e.g. 📝 ) for page label, not 1st byte
Summary:
📝

https://community.getgrist.com/t/feature-request-custom-icon-favicon-for-each-document/210/2

```
>>> json.dumps("📝")
'"\\ud83d\\udcdd"'
>>> json.dumps("")
'"\\u2728"'
```

```
> "📝 abc"[0]
'\uD83D'
> "📝 abc"[1]
'\uDCDD'
> Array.from("📝 abc")
['📝', ' ', 'a', 'b', 'c']
> Array.from("📝 abc")[0]
'📝'
```

Test Plan: no

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3026
2021-09-17 12:59:53 +02:00
George Gevoian
e5ebc4668c (core) Disable mousedown on token delete button
Summary:
This change prevents dragging tokens when the cursor is over the
delete button. Now, trying to drag after mousedown over the delete
button will do nothing if the cursor is released outside the delete
button, and will delete if the cursor is released inside the delete
button.

Test Plan: Tested manually.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3024
2021-09-16 10:45:06 -07:00
Cyprien P
4fcdd2ba07 (core) Fix y-axis blinking in chart view configuration
Summary:
This is a follow up diff for https://phab.getgrist.com/D3021.  Y-axis
draggable list used to blink when user changed either one of the x
axis or groupdata column.

This was due to the fact that all of theses axis are stored into the
same array and changing one of them changes the whole array even
though items relative to the y-axis actually were not changing.

This diff addresses this issue by 1) being carefull at not updating
the array of items when the changes do not impact y axis. And 2) by
adding a freeze observable allowing to freeze the draggable list of
y-axis while actions are being treated on the server.

Test Plan:
Catching such bug is hard, and given that it's only look and fill, maybe not worth the time and effort.

Tested manually though.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3023
2021-09-16 18:18:28 +02:00
George Gevoian
8a7edb6257 (core) Enable incremental imports
Summary:
The import dialog now has an option to 'Update existing records',
which when checked will allow for selection of 1 or more fields
to match source and destination tables on.

If all fields match, then the matched record in the
destination table will be merged with the incoming record
from the source table. This means the incoming values will
replace the destination table values, unless the incoming
values are blank.

Additional merge strategies are implemented in the data
engine, but the import dialog only uses one of the
strategies currently. The others can be exposed in the UI
in the future, and tweak the behavior of how source
and destination values should be merged in different contexts,
such as when blank values exist.

Test Plan: Python and browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3020
2021-09-16 09:15:54 -07:00
Paul Fitzpatrick
a543e5194a (core) add a python3 button
Summary: This adds a dropdown to the document settings model in staging/dev to set the python engine to Python2 or Python3. The setting is saved in `_grist_DocInfo.documentSettings.engine`.

Test Plan: tested manually for now - separate diff needed to add runsc to jenkins setup and make this testable

Reviewers: dsagal, alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3014
2021-09-16 10:06:04 -04:00
Cyprien P
2cf2088373 (core) New chart view axis conf with picker for each of X,Y and group by
Summary:
Chart view used to rely on the same view field configuration as used in any other widget.

This diff allows to explicitely select X-AXIS, Y-AXIS and group by column with column picker.

As charts supports several y-axis, we still use a draggable list to arrange them.

Diff also fix doc to the `insertPositions` function.

Test Plan: Updated the relevant test.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3021
2021-09-16 09:10:54 +02:00
Paul Fitzpatrick
3e5a292cde (core) add tests for site deletion
Summary: This tests site deletion with and without a plan.

Test Plan: adding tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3017
2021-09-14 10:03:18 -04:00
Dmitry S
70160fe01c (core) Rename page option 'Duplicate' to 'Duplicate Page', and shorten warning to be more noticeable
Test Plan: Changes too minor to affect tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3016
2021-09-10 11:25:24 -04:00
Dmitry S
b716a57e31 (core) Avoid loading the Grist frontend app more than once.
Summary:
This can happen thanks to some badly-behaved extensions (e.g. lingvanex), and
results in errors such as "Cannot redefine property: isInput".

Test Plan:
Tested manually that Grist fails to load in Chrome with Lingvanex
extension, and loads successfully with this change.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3013
2021-09-07 16:40:40 -04:00
George Gevoian
0717ee627e (core) Relocate export urls to /download/
Summary:
Moves CSV and XLSX export urls under /download/, and
removes the document title query parameter which is now
retrieved from the backend.

Test Plan: No new tests. Existing tests that verify endpoints still function.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3010
2021-09-02 09:36:33 -07:00
Cyprien P
cecebded1f (core) Fix bar chart weirdness when x axis has redundant values
Summary:
Bar chart was a bit broken when there were redundant values on the X axis: the bars’s height maps to the sum of all the corresponding y values, when the data that shows up on hover is only the last one.

It seems that plotly does not support redundant values in the x axis and in all Plotly examples (implementation relies on plotly) x values only have unique values.

This diff, fixes by making sure x axis has unique values. If user actually wants to plot groups, they'll have to use a summary charts.

Test Plan: tested manually

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3011
2021-09-01 20:58:29 +02:00
George Gevoian
a825115c04 (core) Limit number of errors shown on narrow screens
Summary: We still show up to 5 on regular-width screens.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3008
2021-08-31 12:15:31 -07:00
George Gevoian
ef5da42378 (core) Update export CSV and Excel endpoints
Summary:
The endpoints for exporting CSV and Excel are now under
/api/docs/:docId/ and are forwarded to a doc worker for export.

The Share Menu has been updated to use the new endpoints.

Test Plan: No new tests. Existing tests that verify endpoints work correctly.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3007
2021-08-31 10:47:24 -07:00
Alex Hall
29dd33a45c (core) Add other direction of linking by reflist
Summary:
Allows selecting by a reflist in another table. This generalises cursor-linking with a ref column, but now it's filter linking.

Added another case to LinkingState where the source column is a reflist to the target table, filtering by the id column.

Updated convertQueryFromRefs and related functions to handle this since the id column has no column ref. In this case the string 'id' is used instead of a number.

LinkingState also checks if the source value is a reflist and uses that as the list of filter values instead of a single-element list of the cell value.

Indirect linking also works, where the source and target columns both are both references to the same table. This was the plan for a source reflist and target ref column.
I was surprised to see it also works perfectly when both columns are reflists, and it filters rows where there's an intersection!

Adding rows to the target section using the selected source record for default values is iffy. When filtering by row ID, there's no column for defaults, so the new row disappears.
For a source reflist and target ref, the first value of the reflist is the default, which is okayish. When both are reflists, the full source reflist is the default for the target column.
This seems like a bit much but just using the first value seems a bit arbitrary when there's room for all of them?

While doing all this I noticed an unrelated bug which I fixed as I was refactoring. Previously cursor linking based on a reference column did not update the cursor in the link target
when the value of the selected reference cell changed. Now cursor linking uses a floating row model like most other cases to observe the value correctly.

Test Plan: Extended SelectByRefList test and fixture, added previously failing test to RightPanelSelectBy.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3004
2021-08-30 17:39:15 +02:00
Paul Fitzpatrick
08d0ee65d1 (core) tweak how search bar places cursor on sections
Summary: It is important for linking to be maintained, or user could be gravely misled about material in other sections.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3003
2021-08-30 09:26:28 -04:00
George Gevoian
3d085ff2d7 (core) Fix warning bug in Save Copy dialog
Summary:
The warning about workspace write access would still be shown if
a user picked a workspace they had write access to, and cleared the
Name field in the Save Copy dialog. This fixes the condition for
showing the warning to not show it in this case, and adds a placeholder
to the Name field when it is blank.

Test Plan: Browser test.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3002
2021-08-27 12:02:59 -07:00
Dmitry S
3edb3a6826 (core) Filter out pages for hidden views from the page list.
Summary:
- Filters out hidden pages from docModel.allPagesList (used for knowing default page, and for search iteration).
- Filters out hidden pages from TreeModel (uses raw tableData, so has a different way to filter).

Test Plan: WIP

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D2996
2021-08-27 13:51:57 -04:00
George Gevoian
a6e08883e0 (core) Simple localization support and currency selector.
Summary:
- Grist document has a associated "locale" setting that affects how currency is formatted.
- Currency selector for number format.

Test Plan: not done

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D2977
2021-08-26 13:36:49 -07:00
Alex Hall
7465af8ce8 (core) Port LinkingState.js to TypeScript
Summary: Converted LinkingState from constructor function to class.

Test Plan: no

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2997
2021-08-26 15:10:34 +02:00
Dmitry S
faa0d9988e (core) When changing a table for a page widget, unset widget-linking to avoid invalid values.
Summary:
Previously, using "Change Widget" allowed one to change the underlying table,
but would keep the linking settings. This could allow invalid settings which
would sometimes lead to JS errors. These manifested in production as
"UserError: Query error: n is not a function".

- Unset linking settings in this case, to avoid invalid values.
- In case invalid values are encountered (e.g. saved previously), treat them as
  unset, to avoid JS errors.
- If an error does occur, report it with a stack trace.

Also, for testing, added 'selectBy' option to gristUtils helpers for using page-widget-picker.

Test Plan: Added test cases for resetting linking, and for ignoring invalid link settings.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D2993
2021-08-25 12:53:46 -04:00
Cyprien P
e4633c293c (core) Hide filter bar for inactive widget in mobile mode
Summary:
The filter bar used to show in mobile mode while the widget was inactive as illustrated in this screen shot
{F31970}

This diff fixes it.
{F31971}

Test Plan: Manually tested.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D2958
2021-08-23 10:33:44 +02:00
Alex Hall
9916a2d919 (core) Suggest correct table when converting to RefList
Summary: RecordSets now have new encoding and rendering analogous to Records: `['r', 'Table', [1, 2, 3]]` and `Table[[1, 2, 3]]`.

Test Plan: Added to nbrowser/TypeChange.ts.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2987
2021-08-20 23:04:48 +02:00
Dmitry S
e361a9fd94 (core) Fix a few issues with parsing of dates in DateEditor.
Summary:
- With a format like "DD-MM-YYYY" or "DD MMM YYYY", allow parsing dates
  with two digit year or numeric month (like "16-8-21").
- Interpret two-digit years in the same way for moment parsing and for
  bootstrap-datepicker.
- For partial inputs (like "8/16"), when a format is present, assume that
  provided parts cover the date, then month, then year (even for a format that
  starts with year).

Test Plan: Expanded a unittest

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D2985
2021-08-20 14:54:41 -04:00