Commit Graph

778 Commits

Author SHA1 Message Date
George Gevoian
02dd96daf2 (core) Add page titles to login and error pages
Summary:
Login and error pages now have their own unique page titles. This also fixes
the bug with the signed-out page having a page title of "Error".

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3479
2022-06-12 20:31:09 -07:00
Alex Hall
0350e2df58 (core) Fix filtering of empty reflists
Summary:
A formula returning an empty RecordSet in a RefList columns results in storing [] instead of null.
This caused a bug where the empty list was 'flattened' and the cell not appearing in filters at all.
This diff fixes the bug by filtering for the default value `null` instead for RefLists and the empty string for ChoiceLists.
I didn't manage to actually reproduce the bug for ChoiceLists, but this seemed the most sensible thing to do.

Test Plan: New nbrowser test.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3478
2022-06-11 00:16:57 +02:00
Jarosław Sadziński
64d9ecacdb (core) Renaming table linked by summary table resulted in error
Summary:
When a table was selected by a summary table, renaming that table using
section widget resulted in a javascript error.

Test Plan: new test

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3475
2022-06-10 09:39:34 +02:00
Alex Hall
1c89d08ea3 (core) Add a row to summary tables grouped by list column(s) corresponding to empty lists
Summary:
Adds some special handling to summary table and lookup logic:

- Source rows with empty choicelists/reflists get a corresponding summary row with an empty string/reference when grouping by that column, instead of excluding them from any group
- Adds a new `QueryOperation` 'empty' in the client which is used in `LinkingState`, `QuerySet`, and `recursiveMoveToCursorPos` to match empty lists in source tables against falsy values in linked summary tables.
- Adds a new parameter `match_empty` to the Python `CONTAINS` function so that regular formulas can implement the same behaviour as summary tables. See https://grist.slack.com/archives/C0234CPPXPA/p1654030490932119
- Uses the new `match_empty` argument in the formula generated for the `group` column when detaching a summary table.

Test Plan: Updated and extended Python and nbrowser tests of summary tables grouped by choicelists to test for new behaviour with empty lists.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3471
2022-06-09 23:38:14 +02:00
Jarosław Sadziński
3b30c052bc (core) Enabling clipboard events on Choice entry field
Summary: Copy/paste/cut events weren't available on choice entry field.

Test Plan: Updated tests

Reviewers: alexmojaki, paulfitz

Reviewed By: alexmojaki, paulfitz

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3476
2022-06-09 16:12:35 +02:00
Jarosław Sadziński
d92a761f6e (core) Product update popups and hosted stripe integration
Summary:
- Showing nudge to individual users to sign up for free team plan.
- Implementing billing page to upgrade from free team to pro.
- New modal with upgrade options and free team site signup.
- Integrating Stripe-hosted UI for checkout and plan management.

Test Plan: updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3456
2022-06-08 21:10:49 +02:00
Dmitry S
3b4d936013 (core) Hide 'helpCenter' and 'templates' elements in HomeIntro when requested
Summary:
Removes the relevant links in the intro text and buttons.

Note that the presence of actual "Examples & Templates" section of the page is controlled by whether any templates are available (returned by `getTemplates` api call)

Test Plan: Tested manually

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3473
2022-06-07 19:00:25 -04:00
Dmitry S
007c0f2af0 (core) Fix some bugs with repositioning rows.
Summary:
- Fixed an issue with manualSort values being very close floats. It is already handled by the data engine, but the client was being unnecessarily proactive and introduced a bug.
- The fix also helps with rearranging rows in filtered situations: they will now stay next to the row before which they were inserted.
- The fix accidentally improves (though doesn't fully fix) the issue where new columns show up in unexpected places in the raw-data column list.
- Fixed another rare bug with row order not getting updated correctly when positions update.

Test Plan: Added test cases for the improved behavior; fixed affected tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3462
2022-06-07 16:55:45 -04:00
Dmitry S
4f1cb53b29 (core) Converting server-side Comm.js to typescript
Summary:
- Add app/common/CommTypes.ts to define types shared by client and server.
- Include @types/ws npm package

Test Plan: Intended to have no changes in behavior

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3467
2022-06-07 15:47:17 -04:00
Alex Hall
519f1be93a (core) Disambiguate label for link between summary table and source table with self reference
Summary:
Following up on a small bug introduced in https://phab.getgrist.com/D3464. When a table has a column referencing the same table, then there can be two 'select by' options with the same label which is just the name of a summary table on the same page. The first option is simply filtering based on the summary table. The second option is linking the ref column in the source against the group column in the summary, but the name of the group column is hidden which leads to the ambiguity.

The solution in this diff is to always show the target node (source table) column name if the source node (summary table) column was the hidden group column. This also changes the label in the case where the reference to the source table isn't in the source table - see the updated test. This isn't strictly necessary in this case so I'm not 100% about the desired behaviour, but I don't think it hurts.

Test Plan: Tested disambiguation manually. Updated existing test.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3472
2022-06-07 17:07:45 +02:00
Cyprien P
4b54c7d99b (core) Fix column and view fields mismatch in filters
Summary:
The problem manifested while filtering by a column, closing the filter
would change the set of rows visible on the table. This would happen
only for rare table.

What caused that problem was that the filter being edited was wrongly
applyed also to another column, so depending on the content of the
column, it would entail unexpected behaviour.

The cause of that wrong association, was a mistakingly comparing the
id of two different type of thing: column and view field. The problem
would manifest if in the same section there were a view fields with
same row id as the column being filtered.

What made that confusion possible is the `.fieldOrColumn:
ViewFieldRec|ColumnRec` property of the FileInfo object, which could
hold either a view fields or a column record and was initialized with
view fields or columns if view fields was not found (ie: hidden
column).

Solution was to make sure FieldInfo is initialized with ColumnRec
alwasy (even for hidden column).

I'm not sure what is the reason why FilterInfo needed to support both
column record and view field record in the past, but it looks like
this is not needed anymore.

As a followup commit I think it would be worth the effort to refactor
FileInfo to accept only ColumnRec.

Test Plan: Includes new regression test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3463
2022-06-07 10:26:15 +02:00
Paul Fitzpatrick
05d1cdf140 (core) limit retries of uploads to external store in tests
Summary:
If an external store fails completely, Grist will continue to
retry uploading to it. This diff updated the HostedStorageManager
test to limit the extent of these retries to the test itself -
otherwise they continue for all other tests in the same process,
potentially disrupting those that read logs. There are other tests
that use s3, but they aren't run in the same process with delicate
log-reading tests, and it isn't quite as clear what improvement
to make there.

Test Plan:
artificially made external store fail, and checked that
test contamination seen previously no longer occurs.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3469
2022-06-06 16:19:41 -04:00
George Gevoian
6dcdd22792 (core) Redirect less often in welcomeNewUser
Summary:
Instead of always redirecting new users to the home page or the (teams) welcome page,
only redirect when the user signed in for the first time on a personal site, has access to
other sites, and isn't already being redirected to a specific page on their personal site.

Also tweaks how invalid Choice column values are displayed to match Choice List
columns, and fixes a small CSS issue with select by in the page widget picker when
there are options with long labels.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3461
2022-06-06 11:26:49 -07:00
George Gevoian
090d9af21d (core) Broadcast doc usage updates to clients
Summary:
Introduces a new message type, docUsage, that's broadcast to all connected
clients whenever document usage is updated in ActiveDoc.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3451
2022-06-06 09:55:34 -07:00
Paul Fitzpatrick
ff77ecc6c6 (core) deal with SQLITE_BUSY: database is locked during tests
Summary:
During testing, and in some dev configurations, we make connections to
the Grist home database from multiple processes, and store it in Sqlite.
This is a situation prone to file locks in Sqlite. We set a busy_timeout
parameter to help, but transactions may still fail immediately. This
diff adds retries to our patch of TypeORM for Sqlite transactions
(which without the patch are already flakey for other reasons
https://github.com/typeorm/typeorm/issues/1884#issuecomment-380767213).

Without this change, I can tickly SQLITE_BUSY problems fairly
reliably with the DocApi tests. With this change, I can't so far.

This change should not affect deployments of our SaaS (which use
Postgres for home db) or grist-core/grist-ee (which access home
db from a single process in default configuration).

Test Plan: repeated running of DocApi tests

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3466
2022-06-03 15:51:59 -04:00
Paul Fitzpatrick
1c6f80f956 (core) make it easier to enable Azure storage without setting GRIST_DOCS_S3_BUCKET
Summary:
Previously, absence of `GRIST_DOCS_S3_BUCKET` was equated with absence
of external storage, but that is no longer true now that Azure is
available. Azure could be used by setting `GRIST_DOCS_S3_BUCKET`
but the alternative `GRIST_AZURE_CONTAINER` flag is friendlier.

Test Plan:
confirmed manually that Azure can be configured and
used now without `GRIST_DOCS_S3_BUCKET`

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3448
2022-06-03 14:50:31 -04:00
Dmitry S
acddd25cfd (core) Update design of empty docs home page, and add a "Manage Team" button.
Summary:
- Remove the empty-folder icon
- Add an "Invite team members" button for owners on empty team sites
- Add a "Browse Templates" button for all other cases on empty sites
- Update intro text for team, including a link to Sprouts
- Update intro text for personal/anon.
- Include a Free/Pro tag for team sites (for now, only "Free")
- Add a "Manage Team" button for owners on home page of all team sites.
- Polished the UI of UserManager: add a transition for the background, and
  delay the appearance of the spinner for fast loads.

Test Plan: Fixed up the HomeIntro tests; added test case for Manage Team button

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3459
2022-06-03 12:58:57 -04:00
Alex Hall
af4738b94a (core) Allow linking to summary table via reference to source table
Summary:
Fixes a bug introduced in https://phab.getgrist.com/D3416 which exposed a new type of linking option that didn't actually work. Specifically it allowed selecting by a summary table when the target widget has a reference column to the source table of the summary. This diff correctly implements this linking by linking the reference column against the 'group' column of the summary table, so the source table data (which the client may not have access to) isn't involved. But the 'group' column name is hidden from the 'select by' option label to avoid confusion for users, so it just looks like another kind of summary table linking, and indeed it can be thought of purely in terms of matching groupby columns etc.

Discussion here: https://grist.slack.com/archives/C02EGJ1FUCV/p1654039063055499

Also fixes a related old bug that offered similar meaningless options involving summary tables, like selecting the summary table by the source table or linking summary tables with disjoint groupby columns.

Test Plan: Updated SelectBySummaryRef test and fixture doc for the new behaviour. Also updated a couple of older tests which were incorrectly asserting the buggy option to select a summary table by the source table.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3464
2022-06-03 17:04:41 +02:00
Cyprien P
815c9e1462 (core) Adds new range filter for numeric columns
Summary:
Shows the range filter next to the filter by values on filter menu. When users
set min and/or max, it takes precendence over the filter by values.

If users set:
 - `[] < [max]` behaves as `less than max`.
 - `[min] < []` behaves as `more than min`.
 - `[min] < [max]` behaves as `between min and max`
 - bounds are always inclusives.
 - when users change min or max the values of the by values filter
   gets checked/unchecked depending on whether they are included by
   the range filter.
 - when users clicks any btn/checkbox of the by values filter both min
   and max input gets cleared, and the filter convert to a filter by
   values.

Test Plan: Adds both projets and nbrowser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3435
2022-05-30 10:30:16 +02:00
Paul Fitzpatrick
dcaa2b4f29 (core) move DocUsageBanner to ext
Summary:
grist-ee build was failing since it didn't have a
DocUsageBanner implementation available. Made the implementation
added to monorepo available, since it will be useful to improve
the activation banner.

Test Plan: manaul

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3452
2022-05-27 22:19:17 -04:00
Paul Fitzpatrick
b9a4b2b58f (core) add missing tsconfig file that affects IDEs
Summary:
I missed committing a file that is important for editing files comfortably in the ext directory in an IDE. This diff:

 * Adds tsconfig-base-ext.json - that was the only intended change
 * Unrelated: Forces all creation of connections to the home db through a new `getOrCreateConnection` method which changes the `busy_timeout` if using Sqlite. This was an attempt to fix random "database is locked" test failures. I believe multiple connections to the home db as an sqlite file do not happen in self-hosted Grist (where there is a single node process) or in our SaaS (where the database is in postgres). It does affect Grist started using `devServerMain.ts` (where multiple processes accessing same database are started) or various test configurations when extra database connections are opened.
 * Unrelated: I added a `busy_timeout` for session storage, when it uses Sqlite. Again, I don't believe this affects self-hosted Grist or our SaaS.
 * Tweaked a `BillingDiscount` test that looked perhaps vulnerable to a stripe request stalling.

I can't be sure my tweaks actually help, since I didn't succeed in replicating the failures. Update: looks like the "locked" error can still happen :(

Test Plan: manual

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3450
2022-05-27 13:48:58 -04:00
Alex Hall
6b372fa6cd (core) Allow configuring (mostly hiding) various little bits of UI
Summary:
Adds two new env vars GRIST_HIDE_UI_ELEMENTS and GRIST_PAGE_TITLE_SUFFIX which translate to values in GristLoadConfig that the server sends the client when loading.

For checkin task https://gristlabs.getgrist.com/doc/check-ins/p/5#a1.s9.r1882.c19

Test Plan: Tested manually

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3449
2022-05-27 14:32:05 +02:00
George Gevoian
74ec9358da (core) Show usage banners in doc menu of free team sites
Summary:
Also fixes a minor CSS regression in UserManager where the
link to add a team member wasn't shown on a separate row.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3444
2022-05-26 15:01:35 -07:00
George Gevoian
3d3a5e334a (core) Update Plugin API documentation
Summary: Updates to Plugin API documentation.

Test Plan: Tested manually in grist-help.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3447
2022-05-24 17:27:34 -07:00
Cyprien P
6793377579 (core) Fix values ordering in column filter menu
Summary:
Column filter menu use to mess up the ordering of the items for
numeric and dates values, and also for ref/reflist columns when the
visible column is a numeric a date column.

Solution was to:
 - use the actual value of the visible column for comparison.
 - use native comparison.
 - tweak the native comparison to make blanks appears before valid value. Indeed, it came up several time that it's convenient to have invalid values show up first in the filter panel, it makes for a convenient way to detect them.

Test Plan: Adds new nbrowser test

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3441
2022-05-24 16:30:28 +02:00
Alex Hall
fcbad1c887 (core) Add GET /attachments endpoint for listing attachment metadata
Summary: Combines the code and behaviour of the existing endpoints `GET /records` (for the general shape of the result and the parameters for sort/filter/limit etc) and retrieving a specific attachment with `GET /attachments/:id` for handling fields specific to attachments.

Test Plan: Added a DocApi test. Also updated one test to use the new endpoint instead of raw `GET /tables/_grist_Attachments/records`.

Reviewers: cyprien

Reviewed By: cyprien

Subscribers: cyprien

Differential Revision: https://phab.getgrist.com/D3443
2022-05-20 20:30:14 +02:00
Dmitry S
9bc04a6e66 (core) Replace nulls in X-axis of charts with '-' to avoid confusion with missing points
Summary:
Seem in a customer doc where Y-scale was wrong for unclear reasons. The cause
is that null-valued X labels cause the corresponding bar (or point) to be
omitted, but still affect the Y-scale. In this diff, such labels are replaced
with "-", so as to show up normally.

Trivial example that reproduces the problem here:
https://public.getgrist.com/iLPpx9C5i8nk/Null-in-X-Axis

Test Plan: Tested manually.

Reviewers: cyprien

Reviewed By: cyprien

Subscribers: cyprien

Differential Revision: https://phab.getgrist.com/D3442
2022-05-20 11:54:20 -04:00
Cyprien P
8f4f21e94a (core) Filter menu show all options for Bool/Choice/Choice List columns
Summary:
> Toggle, Choice, and Choice List need all possible values available in filter, not just values present in current records
https://grist.quip.com/cjw4A8AHx1vh/Filtering-Improvements#temp:C:PZHc42e8be8cd8547bb8ce93fdb0

Test Plan: Adds new nbrowser test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3436
2022-05-19 20:43:29 +02:00
Dmitry S
309ddb0fe7 (core) Move guessing logic for column types to run in node once for all columns.
Summary:
Previously, columns of type Any were created and modified one by one by reusing
the "empty column" logic from the data engine. This copies that logic to Node,
and sets the type of all columns together, to create them with the correct type
in the AddTable call.

This makes imports about twice faster (when slowness is due to many columns),
but doesn't address all cases where individual handling of columns causes slowness.

Test Plan: Added a test case for the new helper function.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3427
2022-05-19 12:49:51 -04:00
George Gevoian
a6063f570a (core) Polish Access Details
Summary:
Instead of showing a blank dialog for users whose access
is limited (e.g. public members), we now show the user's
role and a mention of whether their access is public.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3431
2022-05-19 09:45:26 -07:00
George Gevoian
bad4c68569 (core) Reduce a few log levels to warning
Summary: Reduces the log level in a few places from error to warning.

Test Plan: N/A

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3437
2022-05-18 14:57:43 -07:00
Jarosław Sadziński
0ab9e4a6a0 (core) Adding GristConnect login system
Summary:
New login system to allow simple SSO flow that is based on Discourse description that is available at:
https://meta.discourse.org/t/discourseconnect-official-single-sign-on-for-discourse-sso/13045

Test Plan: New core test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3418
2022-05-18 20:28:25 +02:00
Paul Fitzpatrick
cf23a2d1ee (core) add GVISOR_LIMIT_MEMORY to cap memory available in sandbox
Summary:
This allows limiting the memory available to documents in the sandbox when gvisor is used. If memory limit is exceeded, we offer to open doc in recovery mode. Recovery mode is tweaked to open docs with tables in "ondemand" mode, which will generally take less memory and allow for deleting rows.

The limit is on the size of the virtual address space available to the sandbox (`RLIMIT_AS`), which in practice appears to function as one would want, and is the only practical option. There is a documented `RLIMIT_RSS` limit to `specifies the limit (in bytes) of the process's resident set (the number of virtual pages resident in RAM)` but this is no longer enforced by the kernel (neither the host nor gvisor).

When the sandbox runs out of memory, there are many ways it can fail. This diff catches all the ones I saw, but there could be more.

Test Plan: added tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3398
2022-05-18 14:26:27 -04:00
George Gevoian
2fd8a34ff8 (core) Move Notifier to /ext
Summary:
This makes it possible to configure a SendGrid-based Notifier
instance via a JSON configuration file.

Test Plan: Tested manually.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3432
2022-05-18 08:02:32 -07:00
Cyprien P
365f3c7ae2 (core) Auto-check aggregate values option as soon as table is a summary
Summary:
Auto-check makes sure that the “Aggregate Values” option is checked as
soon as the chart table is summarized. Before, user had to check it
explicitly.

More on this:
https://grist.slack.com/archives/C04AYS9JF/p1649400119930389?thread_ts=1649338496.915759&cid=C04AYS9JF

Test Plan: Updated tests

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3422
2022-05-18 10:10:28 +02:00
Cyprien P
f17e31c023 (core) Ensure chart summary is more detailed then linked summary
Summary:
When a summary table is linked, the column used for linking needs
to be included as a group-by column (or the linking can’t work). A
good example is here:
https://templates-s.getgrist.com/doc/investment-research/p/4

This was request by Dmitry here: https://grist.slack.com/archives/C04AYS9JF/p1649400119930389?thread_ts=1649338496.915759&cid=C04AYS9JF (2nd point)

Test Plan: Added new nbrowser test case.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3408
2022-05-17 08:57:05 +02:00
Paul Fitzpatrick
e4d47a2f3c (core) add minimal support for activation keys
Summary: For grist-ee, expect an activation key in environment variable `GRIST_ACTIVATION` or in a file pointed to by `GRIST_ACTIVATION_FILE`. In absence of key, start a 30-day trial, during which a banner is shown. Once trial expires, installation goes into document-read-only mode.

Test Plan: added a test

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3426
2022-05-16 22:46:23 -04:00
George Gevoian
f48d579f64 (core) Add API endpoint to get site usage summary
Summary:
The summary includes a count of documents that are approaching
limits, in grace period, or delete-only. The endpoint is only accessible
to site owners, and is currently unused. A follow-up diff will add usage
banners to the site home page, which will use the response from the
endpoint to communicate usage information to owners.

Test Plan: Browser and server tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3420
2022-05-16 11:16:19 -07:00
Alex Hall
cbdbe3f605 (core) Switch webhook secret cache from LRU to TTL so that unsubscribing can drain the queue
Summary:
Helps with cases such as https://grist.slack.com/archives/C02EGJ1FUCV/p1652196111066649?thread_ts=1651656433.171889&cid=C02EGJ1FUCV

When a user unsubscribes from a webhook, the secret URL is deleted from the database, but as long as the doc was open it would continue retrying pending requests still in the queue for a long time, using the locally cached value without noticing the effect of unsubscribing. This change allows unsubscribing to have an effect more quickly so that problematic events can be removed from the queue.

Test Plan: existing tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3430
2022-05-16 18:02:19 +02:00
Jarosław Sadziński
f93b4fd3bc (core) Selection for styled columns
Summary:
Grid UI improvements.
- Selecting a column with a custom background, didn't produce a visual difference.
- Hiding num-row right border when it is not needed (it should be used only for frozen columns)

Test Plan: manual tests

Reviewers: cyprien

Reviewed By: cyprien

Differential Revision: https://phab.getgrist.com/D3425
2022-05-13 09:45:17 +02:00
George Gevoian
524dbf34e1 (core) Add config to include custom CSS
Summary:
Adds a new environment variable that allows for custom
CSS to be included in all core static pages.

Test Plan: Tested manually in grist-core.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3419
2022-05-12 11:13:52 -07:00
Paul Fitzpatrick
e6983e9209 (core) add machinery for self-managed flavor of Grist
Summary:
Currently, we have two ways that we deliver Grist. One is grist-core,
which has simple defaults and is relatively easy for third parties to
deploy. The second is our internal build for our SaaS, which is the
opposite. For self-managed Grist, a planned paid on-premise version
of Grist, I adopt the following approach:

 * Use the `grist-core` build mechanism, extending it to accept an
   overlay of extra code if present.
 * Extra code is supplied in a self-contained `ext` directory, with
   an `ext/app` directory that is of same structure as core `app`
   and `stubs/app`.
 * The `ext` directory also contains information about extra
   node dependencies needed beyond that of `grist-core`.
 * The `ext` directory is contained within our monorepo rather than
   `grist-core` since it may contain material not under the Apache
   license.

Docker builds are achieved in our monorepo by using the `--build-context`
functionality to add in `ext` during the regular `grist-core` build:

```
docker buildx build --load -t gristlabs/grist-ee --build-context=ext=../ext .
```

Incremental builds in our monorepo are achieved with the `build_core.sh` helper,
like:

```
buildtools/build_core.sh /tmp/self-managed
cd /tmp/self-managed
yarn start
```

The initial `ext` directory contains material for snapshotting to S3.
If you build the docker image as above, and have S3 access, you can
do something like:

```
docker run -p 8484:8484 --env GRIST_SESSION_SECRET=a-secret \
  --env GRIST_DOCS_S3_BUCKET=grist-docs-test \
  --env GRIST_DOCS_S3_PREFIX=self-managed \
  -v $HOME/.aws:/root/.aws -it gristlabs/grist-ee
```

This will start a version of Grist that is like `grist-core` but with
S3 snapshots enabled. To release this code to `grist-core`, it would
just need to move from `ext/app` to `app` within core.

I tried a lot of ways of organizing self-managed Grist, and this was
what made me happiest. There are a lot of trade-offs, but here is what
I was looking for:

 * Only OSS-code in grist-core. Adding mixed-license material there
   feels unfair to people already working with the repo. That said,
   a possible future is to move away from our private monorepo to
   a public mixed-licence repo, which could have the same relationship
   with grist-core as the monorepo has.
 * Minimal differences between self-managed builds and one of our
   existing builds, ideally hewing as close to grist-core as possible
   for ease of documentation, debugging, and maintenance.
 * Ideally, docker builds without copying files around (the new
   `--build-context` functionality made that possible).
 * Compatibility with monorepo build.

Expressing dependencies of the extra code in `ext` proved tricky to
do in a clean way. Yarn/npm fought me every step of the way - everything
related to optional dependencies was unsatisfactory in some respect.
Yarn2 is flexible but smells like it might be overreach. In the end,
organizing to install non-core dependencies one directory up from the
main build was a good simple trick that saved my bacon.

This diff gets us to the point of building `grist-ee` images conveniently,
but there isn't a public repo people can go look at to see its source. This
could be generated by taking `grist-core`, adding the `ext` directory
to it, and pushing to a distinct repository. I'm not in a hurry to do that,
since a PR to that repo would be hard to sync with our monorepo and
`grist-core`. Also, we don't have any licensing text ready for the `ext`
directory. So leaving that for future work.

Test Plan: manual

Reviewers: georgegevoian, alexmojaki

Reviewed By: georgegevoian, alexmojaki

Differential Revision: https://phab.getgrist.com/D3415
2022-05-12 12:39:52 -04:00
Alex Hall
b878395c21 (core) Allow linking summary tables based on ref/reflist columns (except group)
Summary:
Relax the restriction in `selectBy.isValidLink` so that summary tables can be linked by a column like other tables, except the `group` column. See the discussion on https://grist.slack.com/archives/C0234CPPXPA/p1651773623256959 (the replies are on the following message) for more info on this decision.

Tweaked `LinkingState.ts` since linking with summary tables can now involve a column.

Test Plan: Added a new nbrowser test and fixture checking the options to select by given a summary table with a few ref/reflist columns. Manually tested the behaviour of each option.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3416
2022-05-12 15:59:12 +02:00
Alex Hall
6c90de4d62 (core) Switch excel import parsing from messytables+xlrd to openpyxl, and ignore empty rows
Summary:
Use openpyxl instead of messytables (which used xlrd internally) in import_xls.py.

Skip empty rows since excel files can easily contain huge numbers of them.

Drop support for xls files (which openpyxl doesn't support) in favour of the newer xlsx format.

Fix some details relating to python virtualenvs and dependencies, as Jenkins was failing to find new Python dependencies.

Test Plan: Mostly relying on existing tests. Updated various tests which referred to xls files instead of xlsx. Added a Python test for skipping empty rows.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3406
2022-05-12 14:43:21 +02:00
Cyprien P
87cc7d755d (core) Fix chart mixing up axis
Summary:
Diff fixes couple edge cases:
 - When changing chart's groupby columns, the data-engine changes the view fields, which is not okay with charts, hence makes sure view fields stay the same as much as possible using `this.setSectionViewFieldsFromArray()`
 - Also there's a logic in `this.setSectionViewFieldsFromArray()` that handle what to do when some columns goes missing during updates. Diff fixes this logic two. This is really corner case.

Test Plan: Both cases are not added to ChartAggregate.ts

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3405
2022-05-11 09:18:31 +02:00
Alex Hall
4408315f2e (core) Add AzureExternalStorage
Summary:
Adds a new implementation of the interface ExternalStorage that works for Azure Blob Storage as an alternative to S3, for a specific self-hosting case.

Tweaks HostedStorageManager and ICreate to allow configuring different core implementations of ExternalStorage.

Followup tasks:

- Make this code available to self hosters, possibly by making it open source.
- Add an env var or other config option to specify the preferred type of storage. Currently using the var `AZURE_STORAGE_CONNECTION_STRING` to know how to connect to Azure when requested, but that choice still only lives in test code.

Test Plan: Generalized HostedStorageManager and ExternalStorage tests to test the new AzureExternalStorage alongside S3ExternalStorage. The HostedStorageManager tests also now test the 'cached' in-memory test storage in a way that's closer to the real storage methods.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3413
2022-05-09 21:44:57 +02:00
Jarosław Sadziński
db57815d2b (core) Improving custom widget API. Changing destroy function signature.
Summary:
Destroy function in TableOperations was throwing error when invoked with a single
record id instead of an array. Now it returns a void type.

Also changing mapColumns function signature as it doesn't require options for a default
behavior.

Test Plan: Updated tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3404
2022-05-05 16:34:26 +02:00
George Gevoian
1e42871cc9 (core) Add attachment and data size usage
Summary:
Adds attachment and data size to the usage section of
the raw data page. Also makes in-document usage banners
update as user actions are applied, causing them to be
hidden/shown or updated based on the current state of
the document.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3395
2022-05-04 13:46:55 -07:00
Jarosław Sadziński
f194d6861b (core) Updating RawData views
Summary:
- Better focus on the widget title
- Adding columns only to the current view section
- New popup with options when user wants to delete a page
- New dialog to enter table name
- New table as a widget doesn't create a separate page
- Removing a table doesn't remove the primary view

Test Plan: Updated and new tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3410
2022-05-04 21:41:42 +02:00
Dmitry S
97f3a8805c (core) List 'Other Sites' all one line; fix style ordering issue for sort button.
Test Plan: Tested manually by eyeballing

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3407
2022-04-28 23:00:40 -04:00
George Gevoian
ad04744b4a (core) Fix import bug when skipping non-text columns
Summary:
Skipping columns during incremental imports wasn't working for certain
column types, such as numeric columns. The column's default value was
being used instead (e.g. 0), overwriting values in the destination
table.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3402
2022-04-28 12:46:44 -07:00
Alex Hall
dc9e53edc8 (core) Update the current time in formulas automatically every hour
Summary: Adds a special user action `UpdateCurrentTime` which invalidates an internal engine dependency node that doesn't belong to any table but is 'used' by the `NOW()` function. Applies the action automatically every hour.

Test Plan: Added a Python test for the user action. Tested the interval periodically applying the action manually: {F43312}

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3389
2022-04-28 21:07:40 +02:00
Alex Hall
0beb2898cb (core) Add flexibility to daily API usage limit
Summary: Allow exceeding the daily API usage limit for a doc based on additional allocations for the current hour and minute. See the doc comment on getDocApiUsageKeysToIncr for details. This means that up to 5 redis keys may be relevant at a time for a single document.

Test Plan: Updated and expanded 'Daily API Limit' tests.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3368
2022-04-28 16:22:18 +02:00
Paul Fitzpatrick
4de5928396 (core) when redirecting, use protocol in APP_HOME_URL if available
Summary:
Currently, Grist behind a reverse proxy will generate many
needless redirects via `http`, and can't be used with only
port 443. This diff centralizes generation of these redirects
and uses the protocol in APP_HOME_URL if it is set.

Test Plan:
manually tested by rebuilding grist-core and
doing a reverse proxy deployment that had no support for
port 80. Prior to this change, there are lots of problems;
after, the site works as expected.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3400
2022-04-28 09:13:27 -04:00
Jarosław Sadziński
6f00106d7c (core) Raw renames
Summary:
A new way for renaming tables.

  - There is a new popup to rename section (where you can also rename the table)
  - Renaming/Deleting page doesn't modify/delete the table.
  - Renaming table can rename a page if the names match (and the page contains a section with that table).
  - User can rename table in Raw Data UI in two ways - either on the listing or by using the section name popup
  - As before, there is no way to change tableId - it is derived from a table name.
  - When the section name is empty the table name is shown instead.
  - White space for section name is allowed (to discuss) - so the user can just paste '   '.
  - Empty name for a page is not allowed (but white space is).
  - Some bugs related to deleting tables with attached summary tables (and with undoing this operation) were fixed (but not all of them yet).

Test Plan: Updated tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3360
2022-04-27 22:21:55 +02:00
Jarosław Sadziński
8a1cca629b (core) Changing shortcut for Duplicate row command
Summary:
Changing keyboard shortcut for Duplicate row from
Ctrl+Shift+D to Mod+Shift+D

Test Plan: existing test

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3401
2022-04-27 22:02:27 +02:00
Jarosław Sadziński
995bf9b63a (core) Distinct style rules for summary columns
Summary:
Summary columns now have their own conditional rules,
which are not shared with sister columns.

Test Plan: New test

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3388
2022-04-27 20:51:23 +02:00
Dmitry S
e59dcc142d (core) Show proper message on empty Excel import, rather than a code error
Summary:
- Previously showed "UnboundLocalError". Now will show:
    Import failed: Failed to parse Excel file.
    Error: No tables found (1 empty tables skipped)
- Also fix logging for import code

Test Plan: Added a test case

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3396
2022-04-27 00:49:28 -04:00
Jarosław Sadziński
dcafa96b5b (core) Restoring context menu when comparing documents
Summary:
Fixing bug when comparing documents. User wasn't able
to open cell/column context menu due to javascript error.

Test Plan: Updated tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3397
2022-04-26 21:31:36 +02:00
Alex Hall
040fa85a8b (core) Simplify InitNewDoc since the timezone and locale is never actually used
Summary: InitNewDoc is essentially only used to generate initialDocSql, so it doesn't make sense to set the timezone and locale. They are always set when actually creating a new doc anyway. Discussed in https://grist.slack.com/archives/C0234CPPXPA/p1650312714217089.

Test Plan: this

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3394
2022-04-26 00:08:03 +02:00
George Gevoian
af5b3c9004 (core) Add document usage banners
Summary:
This also enables the new Usage section for all sites. Currently,
it shows metrics for document row count, but only if the user
has full document read access. Otherwise, a message about
insufficient access is shown.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3377
2022-04-25 08:14:52 -07:00
Alex Hall
a701b4bf13 (core) Remove expired attachments every hour and on shutdown
Summary:
Call ActiveDoc.removeUnusedAttachments every hour using setInterval, and in ActiveDoc.shutdown (which also clears said interval).

Unrelated: small fix to my webhooks code which was creating a redis client on shutdown just to quit it.

Test Plan:
Tweaked DocApi test to remove expired attachments by force-reloading the doc, so that it removes them during shutdown. Extracted a new testing endpoint /verifyFiles to support this test (previously running that code only happened with `/removeUnused?verifyfiles=1`).

Tested the setInterval part manually.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3387
2022-04-22 20:43:59 +02:00
Alex Hall
890c550fc3 (core) Don't include adding attachment metadata in undo stack
Summary: Mark actions adding attachment metadata as 'internal' (not part of undo stack) which previously was only for the Calculate action.

Test Plan: Extended nbrowser attachments test

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3380
2022-04-22 18:39:54 +02:00
Jarosław Sadziński
bedb19f9c7 (core) Removing conditional styles for blank rows
Summary:
When grid view was not fully loaded the initial blank row
was marked as invalid by conditional styles. This removes the styling
when row is not fully loaded.

Test Plan: Tested manually

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3392
2022-04-21 11:45:17 +02:00
Jarosław Sadziński
77ef9df27d (core) Adding new command Duplicate rows
Summary:
New command "Duplicate rows" is available in the Row/Card Context Menu and as a keyboard shortcut Ctrl+Alt+C.
- All selected rows are duplicated (even if only a single column is selected)
- Rows are inserted immediately after the last selected row (using manualSort value).
- Formulas and CENSORED fields are not copied.
Implemented on the UI level (no new action).

Test Plan: new test

Reviewers: cyprien

Reviewed By: cyprien

Differential Revision: https://phab.getgrist.com/D3371
2022-04-20 17:29:48 +02:00
Alex Hall
d7514e9cfc (core) Create _grist_Attachments_fileIdent index in new docs
Summary: Patching up the mistake in https://phab.getgrist.com/D3374#inline-38023.

Test Plan: this

Reviewers: dsagal, paulfitz

Reviewed By: dsagal, paulfitz

Differential Revision: https://phab.getgrist.com/D3382
2022-04-19 21:21:52 +02:00
Alex Hall
47b77c8c24 (core) Allow filtering by selected cell value in cell context menu
Summary: Adds a command and `BaseView` method `filterByThisCellValue`.

Test Plan: Added two tests to `nbrowser/CellContextMenu.ts`

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3383
2022-04-19 11:51:47 +02:00
Paul Fitzpatrick
ce7eb05ed4 (core) get user.Name through same mechanism as user.id for websocket Client
Summary:
This avoids an extra database query to look up the user's current
name, by capturing it at the moment their user id is queried.

Test Plan: existing test for user.Name changes continues to pass

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3381
2022-04-14 12:49:35 -04:00
Alex Hall
64a5c79dbc (core) Limit total attachment file size per document
Summary:
- Add a new parameter `Features.baseMaxAttachmentsBytesPerDocument` and set it to 1GB for the free team product.
- Add a method to DocStorage to calculate the total size of existing and used attachments.
- Add a migration to DocStorage adding an index to make the query in the above method fast.
- Check in ActiveDoc if uploading attachment(s) would exceed the product limit on that document.

Test Plan: Added test in `limits.ts` testing enforcement of the attachment limit.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3374
2022-04-14 16:33:09 +02:00
Paul Fitzpatrick
c1af5a9803 (core) have user.Name come from database for websocket users
Summary: The name of a user for actions made using a websocket until now could be inconsistent with that seen by other means. This draws the name from the database, rather than from session information that may have been cached from an identity provider.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3379
2022-04-13 17:46:46 -04:00
Jarosław Sadziński
dea1a8ba1b (core) Implementing search on raw data view
Summary:
Search now works on Raw Data Page.
- Search bar option 'Search on all pages' will change to 'Search on all tables' when on the Raw data page, and will allow searching through all tables.
- Little CSS adjustment for an overlay on Raw page (removes z-index as it is not needed, and conflicts with searchbar).
- Search bar option ('search on all') gets white background, little padding, and is moved 2 pixels up, this is needed for Raw page.

Test Plan: new and updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3376
2022-04-13 20:44:37 +02:00
George Gevoian
007a862333 (core) Include hash/fragment in post-login redirect URLs
Summary: Also cleans up login URLs by excluding params and hashes.

Test Plan: Client and server tests.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3378
2022-04-13 09:16:29 -07:00
Paul Fitzpatrick
20dd2fc70d (core) allow non-owners to remove themselves from sites/workspaces/docs
Summary:
For users who cannot otherwise change access to a resource, let
them remove themselves. Implemented via the standard endpoints
as a special exception that will process a request from a user
that would otherwise be denied, if the only contents of that
request are a removal of themselves.

Users who can change access are still not permitted to change their
own permissions or to remove themselves, as a precaution against
orphaning resources.

Test Plan: extended and updated tests

Reviewers: cyprien

Reviewed By: cyprien

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3367
2022-04-13 10:04:32 -04:00
Cyprien P
25e40bfa9b (core) Fix setting xaxis when both chart aggregation and split series
Summary:
 - Symptoms where that Split Series could end up being turned off for
   no good reason. Also both x axis and split series could be mixed
   up.

 - Problems was caused by call to `setGroupByColumns` which modifies
   the sections viewFields. Diff fixes it by adjustin slightly the
   ordering of function call in `_setXAxis()`.

 - Problem of mixing up x axis and split series was fixed by being
   careful on the order of columns passed to the `setGroupByColumns`
   which then determine the ordering of the view fields.

Test Plan: Includes new test cases

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3365
2022-04-13 11:25:24 +02:00
Alex Hall
09da815c0c (core) Add /attachments/removeUnused DocApi endpoint to hard delete all unused attachments in document
Summary: Adds methods to delete metadata rows based on timeDeleted. The flag expiredOnly determines if it only deletes attachments that were soft-deleted 7 days ago, or just all soft-deleted rows. Then any actual file data that doesn't have matching metadata is deleted.

Test Plan: DocApi test

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3364
2022-04-12 17:11:11 +02:00
Jarosław Sadziński
4401ec4d79 (core) Adding onNewRecord event for custom widgets
Summary:
New onNewRecord event in custom widgets that is triggered
when the user selects a blank row.

Together with this diff there is PR for grist-help https://github.com/gristlabs/grist-help/pull/150

Test Plan: added test case

Reviewers: cyprien

Reviewed By: cyprien

Differential Revision: https://phab.getgrist.com/D3375
2022-04-12 16:51:16 +02:00
Dmitry S
cedcdc6bff (core) Improve debug logging related for client-side errors and sandbox crashes.
Summary:
- Include docId when available for client-side error reporting
- Distinguish sandbox crashes from forced exits

Test Plan: Tested manually

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3373
2022-04-11 17:54:40 -04:00
George Gevoian
859c593448 (core) Add authSubject and authProvider to sessions
Summary:
This also updates Authorizer to link the authSubject
to Grist users if not previously linked. Linked subjects
are now used as the username for password-based logins,
instead of emails, which remain as a fallback.

Test Plan: Existing tests, and tested login flows manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3356
2022-04-11 11:42:02 -07:00
Paul Fitzpatrick
14f7e30e6f (core) add users.options.isConsultant flag, and omit such users from billing
Summary:
This adds an optional `isConsultant` flag to `users.options`, and an endpoint that allows the support user to turn it on or off. Users marked as consultants are not counted as billable members. Follows the example of existing `allowGoogleLogin` option.

Billable members are counted when members are added or removed from a site. Changing the `isConsultant` flag has no immediate or retroactive effect on billing. The number of users in stripe is now set unconditionally, rather than only when it has changed.

Notifications to billing managers are not aware of this billing nuance, but continue to report user counts that include consultants. The notifications link users to the billing page.

Test Plan: extended test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: anaisconce, jarek

Differential Revision: https://phab.getgrist.com/D3362
2022-04-11 10:26:31 -04:00
George Gevoian
4c5de16e2d (core) Include altSessionId in logs
Summary: Adds altSessionId to log output.

Test Plan: Tested manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3355
2022-04-08 16:40:34 -07:00
Cyprien P
bf8769bc42 (core) Fix linking after a summary update
Summary:
When linking table to a summary, the linking ended up broken after updating the summary group by columns.
This diff fixes that issue.

There were two issues:
 1) some subscriptions were missing due to some .peek() calls instead of directly calling the observable.
 2) the LinkingState instance was not being disposed.
 3) the filterColValues was not updating after source data table has been loaded

Test Plan: Include new test file.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3358
2022-04-08 12:28:42 +02:00
Jarosław Sadziński
34708cd348 (core) Adding font options to the style picker
Summary:
Redesigning color picker:
- Single color palette (no light/dark switch)
- Ability to remove color (new empty button)

New font options in the color picker.
Font options are available on:
- Default cell style
- Conditional rules styles
- Choice/ChoiceList editor and token field
- Filters for Choice/ChoiceList columns

Design document:
https://www.figma.com/file/bRTsb47VIOVBfJPj0qF3C9/Grist-Updates?node-id=415%3A8135

Test Plan: new and updated tests

Reviewers: georgegevoian, alexmojaki

Reviewed By: georgegevoian, alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3335
2022-04-07 20:35:03 +02:00
Cyprien P
98ac2f7e5b (core) add 'Aggregate Values' option to chart config
Summary:
  - adds the `aggregate values` option to chart config
    - aggregation is performed by turning table into summary table
    - change columns options of xaxis and split series selectors to be the source table columns when `aggregate values` is on
    - change xAxis and split series computed to hold colId instead of column id
    - change GristDoc saveViewSection routine to preserve old sections viewFields and options
  - Rename `Group data` into `split series`

quip doc: https://grist.quip.com/tAsCAuv8RiMa/Charts-data-aggregation#temp:C:QcK0ce13e1e8ae64048988f44f9c

Test Plan: Adds ChartAggregate.ts nbrowser test.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3336
2022-04-07 15:34:54 +02:00
Alex Hall
64369df4c3 (core) Add /attachments/updateUsed DocApi endpoint to soft delete all unused attachments in document
Summary:
Builds on https://phab.getgrist.com/D3352

Add DocStorage.scanAttachmentsForUsageChanges to do fancy JSON query to find all attachment metadata rows whose soft deletion status needs updating.

Add ActiveDoc.updateUsedAttachments which uses the above and then applies the appropriate user action if needed to soft delete/undelete metadata rows.

Add endpoint in DocApi calling ActiveDoc method.

Test Plan: Added DocApi test

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3357
2022-04-07 15:08:22 +02:00
Alex Hall
251d79704b (core) Migrate Attachments columns from marshalled blobs to JSON
Summary: Adds a migration in preparation for future work on tracking and deleting attachments. This includes a `_grist_Attachments.timeDeleted` column which isn't used yet, and changing the storage format of user columns of type `Attachments`. DocStorage now treats Attachments like RefList in general (since they use JSON), which also prompted a tiny bit of refactoring.

Test Plan: Added a migration test case showing the change in format.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3352
2022-04-06 13:28:47 +02:00
Paul Fitzpatrick
6c6bfee00e (core) fix redirects for multi-team Grist on a single domain
Summary:
The logic for calculating redirects wasn't quite right for Grist
configured to use a single domain, with teams encoded in the path.
This fixes it.

Test Plan: tested manually with docker compose and /etc/hosts

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3359
2022-04-05 17:27:37 -04:00
Alex Hall
bb5f3fc378 (core) Store monthly snapshots for 8 years to give Enterprise plans a more significant advantage
Summary:
Based on a discussion in https://grist.quip.com/ZvttAyjLCI7H#eLVADAbyipu

Without this change, the only difference between Enterprise and Pro plans regarding snapshots is 5 extra snapshots, one per year.

Test Plan: none

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3349
2022-04-05 18:11:13 +02:00
Paul Fitzpatrick
fea8f906d7 (core) add a login method based on headers
Summary:
This fleshes out header-based authentication a little more to
work with traefik-forward-auth.

Test Plan: manually tested

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3348
2022-04-04 18:36:09 -04:00
Paul Fitzpatrick
c6d66e15bf (core) configure typedoc for generating plugin api documentation
Summary:
This annotates the plugin api sufficiently to generate some documentation
for it. See https://github.com/gristlabs/grist-help/pull/139

Contains some small code tweaks for things that caused typedoc some
trouble.

Test Plan: manual inspection of output

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3342
2022-04-04 18:11:33 -04:00
George Gevoian
d8af25de9d (core) Add usage to data tables page
Summary:
Currently, usage is only shown for free team sites, and only
for total number of rows used in a document. Future diffs will
include other usage metrics and browser tests.

Test Plan: Planned for future diffs; UI is still under development.

Reviewers: jarek

Reviewed By: jarek

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3343
2022-04-04 15:08:19 -07:00
Alex Hall
bf271c822b (core) Copy column type and options when pasting into an empty column
Summary:
Adds a `data-grist-col-ref` attribute to the copied HTML, then uses that when pasting to look up the source column and retrieve info about it. Copies the info into the target column if:

- The document is the same (the docId hash matches)
- The source column still exists and has the same type as when copied
- The source type isn't Text, because in that case it's nice if type guessing still happens
- The target column is empty, meaning it has type Any (we check earlier that it's not a formula column)

The info copied is the type, widgetOptions, and reference column settings (visible and display columns) but not conditional formatting.

The changes are mostly in a function `parsePasteForView` which is based on `BaseView._parsePasteForView` but ported to TypeScript in a new file `BaseView2.ts`.

Added a useraction `MaybeCopyDisplayFormula` exposing an existing Python function `maybe_copy_display_formula` because the target column needs a slightly different display formula.

Test Plan: Added a new nbrowser test file and fixture doc.

Reviewers: cyprien

Reviewed By: cyprien

Subscribers: jarek, dsagal

Differential Revision: https://phab.getgrist.com/D3344
2022-04-04 14:53:16 +02:00
George Gevoian
6305811ca6 (core) Add new Grist login page
Summary:
Adds a new Grist login page to the login app, and replaces the
server-side Cognito Google Sign-In flow with Google's own OAuth flow.

Test Plan: Browser and server tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3332
2022-04-01 15:24:19 -07:00
Alex Hall
8fdfb02646 (core) Don't treat Attachments as RefList in creator panel
Summary: Fixing bug introduced in https://phab.getgrist.com/D3338: column config now shows "DATA FROM TABLE" under the Attachments column type as if it were a normal ref/reflist column.

Test Plan: manual

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3350
2022-04-01 22:10:22 +02:00
Dmitry S
d55bdbcdf3 (core) Reopen HelpScout beacon to the last-opened article
Summary:
- When opening HelpScout beacon to an article ("answers"), avoid a 'navigate'
  call to let the beacon show the previously open article.
- Work around a bug with reloading a page with a beacon article open: HelpScout
  renders the last state without triggering usual events.
- Report errors to server when beacon fails to load.
- reportWarning() method now reports the message to the server.

Test Plan: Added a test case

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3345
2022-03-31 14:54:55 -04:00
Jarosław Sadziński
499e24b744 (core) Adding conditional styles to old style widgets
Summary:
Widgets that were using old base Widget class didn't not
create conditional style rules ui.

Additional fixed a little bug - when adding conditional rule
the formula field was marked as having error for a split second.

Test Plan: new test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3346
2022-03-31 18:29:41 +02:00
Alex Hall
21b0ac3eff (core) Enforcing data size limit
Summary:
Track 'data size' in ActiveDoc alongside row count. Measure it at most once every 5 minutes after each change as before, or after every change when it becomes high enough to matter.

A document is now considered to be approaching/exceeding 'the data limit' if either the data size or the row count is approaching/exceeding its own limit.

Unrelated: tweaked teamFreeFeatures.snapshotWindow based on Quip comments

Test Plan: Tested manually that data size is now logged after every change once it gets high enough, but only if the row limit isn't also too high. Still too early for automated tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3341
2022-03-30 17:56:05 +02:00
Alex Hall
06956f84a5 (core) Make Attachments columns get treated like RefLists more
Summary:
Treat the column type 'Attachments' as equivalent to 'RefList:_grist_Attachments' in a few places, because that's essentially what it is. The main goal was to fix parsing strings representing attachments (reflists).

Also removed an unused function.

Test Plan: Tested manually that pasting a CSV/JSON string representation of an attachments reflists works now.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3338
2022-03-28 23:14:29 +02:00
Alex Hall
59436d2bca (core) Grace period and delete-only mode when exceeding row limit
Summary:
Builds upon https://phab.getgrist.com/D3328

- Add HomeDB column `Document.gracePeriodStart`
- When the row count moves above the limit, set it to the current date. When it moves below, set it to null.
- Add DataLimitStatus type indicating if the document is approaching the limit, is in a grace period, or is in delete only mode if the grace period started at least 14 days ago. Compute it in ActiveDoc and send it to client when opening.
- Only allow certain user actions when in delete-only mode.

Follow-up tasks related to this diff:

- When DataLimitStatus in the client is non-empty, show a banner to the appropriate users.
- Only send DataLimitStatus to users with the appropriate access. There's no risk landing this now since real users will only see null until free team sites are released.
- Update DataLimitStatus immediately in the client when it changes, e.g. when user actions are applied or the product is changed. Right now it's only sent when the document loads.
- Update row limit, grace period start, and data limit status in ActiveDoc when the product changes, i.e. the user upgrades/downgrades.
- Account for data size when computing data limit status, not just row counts.

See also the tasks mentioned in https://phab.getgrist.com/D3331

Test Plan: Extended FreeTeam nbrowser test, testing the 4 statuses.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3331
2022-03-25 13:41:33 +02:00
Paul Fitzpatrick
134ae99e9a (core) add gvisor-based sandboxing to core
Summary:
This adds support for gvisor sandboxing in core. When Grist is run outside of a container, regular gvisor can be used (if on linux), and will run in rootless mode. When Grist is run inside a container, docker's default policy is insufficient for running gvisor, so a fork of gvisor is used that has less defence-in-depth but can run without privileges.

Sandboxing is automatically turned on in the Grist core container. It is not turned on automatically when built from source, since it is operating-system dependent.

This diff may break a complex method of testing Grist with gvisor on macs that I may have been the only person using. If anyone complains I'll find time on a mac to fix it :)

This diff includes a small "easter egg" to force document loads, primarily intended for developer use.

Test Plan: existing tests pass; checked that core and saas docker builds function

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3333
2022-03-24 17:04:49 -04:00
Paul Fitzpatrick
de703343d0 (core) disentangle some server tests, release to core, add GRIST_PROXY_AUTH_HEADER test
Summary:
This shuffles some server tests to make them available in grist-core,
and adds a test for the `GRIST_PROXY_AUTH_HEADER` feature added in
https://github.com/gristlabs/grist-core/pull/165

It includes a fix for a header normalization issue for websocket connections.

Test Plan: added test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3326
2022-03-24 15:11:32 -04:00
Jarosław Sadziński
64c9717ac1 (core) Undo bug with summary table and raw data view
Summary:
Clicking undo/redo after converting a table to a summary table navigated
to the raw data view.

Test Plan: new test

Reviewers: georgegevoian, alexmojaki

Reviewed By: georgegevoian, alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3337
2022-03-24 20:03:33 +01:00
Alex Hall
546096fcc9 (core) Clean up and refactor uses of HomeDBManager.getDoc
Summary:
Firstly I just wanted some more consistency and less repetition in places where Documents are retrieved from the DB, so it's more obvious when code differs from the norm. Main changes for that part:

- Let HomeDBManager accept a `Request` directly and convert it to a `Scope`, and use this in a few places.
- `getScope` tries `req.docAuth.docId` if `req.params` doesn't have a docId.

I also refactored how `_createActiveDoc` gets the document URL, separating out getting the document from getting a URL for it. This is because I want to use that document object in a future diff, but I also just find it cleaner. Notable changes for that:

- Extracted a new method `HomeDBManager.getRawDocById` as an alternative to `getDoc` that's explicitly for when you only have a document ID.
- Removed the interface method `GristServer.getDocUrl` and its two implementations because it wasn't used elsewhere and it didn't really add anything on top of getting a doc (now done by `getRawDocById`) and `getResourceUrl`.
- Between `cachedDoc` and `getRawDocById` (which represent previously existing code paths) also try `getDoc(getScope(docSession.req))`, which is new, because it seems better to only `getRawDocById` as a last resort.

Test Plan: Existing tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3328
2022-03-24 13:42:36 +02:00
Jarosław Sadziński
b1c3943bf4 (core) Conditional formatting rules
Summary:
Adding conditional formatting rules feature.

Each column can have multiple styling rules which are applied in order
when evaluated to a truthy value.

- The creator panel has a new section: Cell Style
- New user action AddEmptyRule for adding an empty rule
- New columns in _grist_Table_columns and fields

A new color picker will be introduced in a follow-up diff (as it is also
used in choice/choice list/filters).

Design document:
https://grist.quip.com/FVzfAgoO5xOF/Conditional-Formatting-Implementation-Design

Test Plan: new tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3282
2022-03-23 13:15:02 +01:00
Jarosław Sadziński
96a34122a5 (core) Restoring cursor position on raw data views
Summary:
This diff introduces cursor features for raw data views:
- Restoring cursor position when the browser window is reloaded
- Restoring the last edit position when the browser window is reloaded

Test Plan: Added tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3314
2022-03-23 12:24:18 +01:00
Dmitry S
3b76b33423 (core) Fix bugs when both welcomeTour and docTour are available
Summary:
- Unify where in the code tours get initiated.
- Avoid start a new tour while one is being started or is in progress.
- Ignore welcome tour when on a doc that has a doc tour.
- Fix tours when starting with a special page like Access Rules.
- Remove mention of the no-longer-present "Give Feedback" button in the last
  message of the welcome tour.

Test Plan:
Add a browser test case that docTour preempts the welcome tour and shows no errors
(this test case fails in multiple ways without the changes).

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3330
2022-03-22 16:51:05 -04:00
Alex Hall
1452b6efc3 (core) Improve stacktraces from pyCall
Summary: Capture the stacktrace (via SandboxError) in `_pyCallWait` instead of `_onSandboxMsg` where it's always the same.

Test Plan:
Tested manually, found for example that the stacktrace in the logs changed from being rather useless:

```
at NSandbox._onSandboxMsg (/home/alex/work/grist/_build/core/app/server/lib/NSandbox.js:229:36)
at /home/alex/work/grist/_build/core/app/server/lib/NSandbox.js:179:18
at Unmarshaller.parse (/home/alex/work/grist/_build/core/app/common/marshal.js:289:21)
at NSandbox._onSandboxData (/home/alex/work/grist/_build/core/app/server/lib/NSandbox.js:174:28)
at Socket.<anonymous> (/home/alex/work/grist/_build/core/app/server/lib/NSandbox.js:63:59)
at Socket.emit (events.js:315:20)
at Socket.EventEmitter.emit (domain.js:467:12)
at addChunk (internal/streams/readable.js:309:12)
at readableAddChunk (internal/streams/readable.js:284:9)
at Socket.Readable.push (internal/streams/readable.js:223:10)
at Pipe.onStreamRead (internal/stream_base_commons.js:188:23)
```

to being somewhat more helpful:

```
at NSandbox._pyCallWait (/home/alex/work/grist/_build/core/app/server/lib/NSandbox.js:134:19)
at processTicksAndRejections (internal/process/task_queues.js:93:5)
at async ActiveDoc.applyActionsToDataEngine (/home/alex/work/grist/_build/core/app/server/lib/ActiveDoc.js:1080:39)
at async Sharing._applyActionsToDataEngine (/home/alex/work/grist/_build/core/app/server/lib/Sharing.js:325:37)
```

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3329
2022-03-22 17:00:02 +02:00
Alex Hall
2c9ae6dc94 (core) Enforce daily limit on API usage
Summary:
Keep track of the number of API requests made for this document today in redis. Uses local caches of the count and the document so that usually requests can proceed without waiting for redis or the database.

Moved the free standing function apiThrottle to become a method to avoid adding another layer of request handler callbacks.

Test Plan: Added a DocApi test

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3327
2022-03-22 00:22:45 +02:00
Cyprien P
b6f146d755 (core) Add options to switch chart orientation
Test Plan: Adds nbrowser tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3323
2022-03-21 11:28:44 +01:00
Alex Hall
ec8460b772 (core) Prune snapshots outside the window in product features
Summary:
- Add a method `getSnapshotWindow` to `IInventory` and `DocSnapshotInventory`. It returns a `SnapshotWindow`, which represents a duration of time for which we keep backups for a particular document.
- `DocSnapshotPruner` calls this method and passes the window to `shouldKeepSnapshots` to determine which document versions have fallen outside the window and should be pruned.
- The implementation passed to `DocSnapshotInventory` uses a new method `getDocProduct` in `HomeDBManager` which directly returns the `Product` associated with a document, given only the document ID. Other methods in `HomeDBManager` require passing more information, especially about a user, but `DocSnapshotPruner` only knows about document IDs.

Test Plan: Added a test for `getDocProduct` and a test for `DocSnapshotPruner` where `getSnapshotWindow` is specified.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3322
2022-03-18 18:48:14 +02:00
Cyprien P
21f1dfa56c (core) Add 'stacked' option to charts
Summary:
Adds nbrowser test

 - Also makes sort spec taken into account by Group Data options
 - This is a continuation of https://phab.getgrist.com/D3271
 - We still need to decide whether to add stack chart to area chart type

Test Plan: TBD

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3274
2022-03-18 10:59:12 +01:00
Paul Fitzpatrick
a5f5ecce19 (core) add grist.getTable(tableId) and a getTableId() method in plugin api
Summary:
Makes the new TableOperations API available for all tables
in the document. Adds methods for discovering the tableId of the
selected table. I was very tempted to implement the select() TODO
in the TableOperations API, but it requires a significant refactor
of the backend.

Test Plan: added test

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3325
2022-03-17 16:31:19 -04:00
Dmitry S
fa75f60bfd (core) Fix selection of rows after rows are dragged
Summary: After dragging rows up, selection was set incorrectly.

Test Plan: Expanded a browser test for dragging rows to check selection, which fails without this fix

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3324
2022-03-17 11:49:52 -04:00
George Gevoian
0f4f0d3dad (core) Migrate to SRP and add change password dialog
Summary:
Moves some auth-related UI components, like MFAConfig, out
of core, and adds a new ChangePasswordDialog component for
allowing direct password changes, replacing the old reset password
link to hosted Cognito.

Updates all MFA endpoints to use SRP for authentication.

Also refactors MFAConfig into smaller files, and polishes up some parts
of the UI to be more consistent with the login pages.

Test Plan: New server and deployment tests. Updated existing tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3311
2022-03-16 21:35:06 -07:00
Paul Fitzpatrick
7ba4dff18f (core) updates from grist-core 2022-03-15 13:40:22 -04:00
Paul Fitzpatrick
a641517bb1
Merge pull request #165 from MHOOO/reverse-proxy-auth-support
Reverse proxy auth support
2022-03-15 13:39:29 -04:00
Paul Fitzpatrick
98f64a8461 (core) add grist.selectedTable.create/update/destroy/upsert to custom widget api
Summary: This makes an equivalent of the /records REST endpoint available within custom widgets. For simple operations, it is compatible with https://github.com/airtable/airtable.js/. About half of the diff is refactoring code from DocApi that implements /records using applyUserActions, to make that code available in the plugin api.

Test Plan: added tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3320
2022-03-15 11:11:58 -04:00
Alex Hall
02e69fb685 (core) Crudely show row count and limit in UI
Summary:
Add rowCount returned from sandbox when applying user actions to ActionGroup which is broadcast to clients.

Add rowCount to ActiveDoc and update it after applying user actions.

Add rowCount to OpenLocalDocResult using ActiveDoc value, to show when a client opens a doc before any user actions happen.

Add rowCount observable to DocPageModel which is set when the doc is opened and when action groups are received.

Add crude UI (commented out) in Tool.ts showing the row count and the limit in AppModel.currentFeatures. The actual UI doesn't have a place to go yet.

Followup tasks:

- Real, pretty UI
- Counts per table
- Keep count(s) secret from users with limited access?
- Data size indicator?
- Banner when close to or above limit
- Measure row counts outside of sandbox to avoid spoofing with formula
- Handle changes to the limit when the plan is changed or extra rows are purchased

Test Plan: Tested UI manually, including with free team site, opening a fresh doc, opening an initialised doc, adding rows, undoing, and changes from another tab. Automated tests seem like they should wait for a proper UI.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3318
2022-03-14 21:49:32 +02:00
Thomas Karolski
ccdd551b4d style fixes 2022-03-14 17:51:10 +01:00
Paul Fitzpatrick
b2715ae9ef (core) forbid use of sqlite ATTACH except during VACUUM
Summary:
This calls sqlite3_limit(SQLITE_LIMIT_ATTACHED, 0) so that
if ever an `ATTACH` were snuck into an SQL query, it would be denied.
The limit needs to be waived when calling VACUUM since the implementation
of VACUUM uses ATTACH.

Test Plan: added test; existing tests should pass

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3316
2022-03-14 09:34:44 -04:00
George Gevoian
ad1b4f3cff (core) Record new user sign-ups
Summary:
Adds Google Tag Manager snippet to all login pages, and a new user
preference, recordSignUpEvent, that's set to true on first sign-in. The
client now checks for this preference, and if true, dynamically loads
Google Tag Manager to record a sign-up event. Afterwards, it removes
the preference.

Test Plan: Tested manually.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3319
2022-03-12 14:34:46 -08:00
Thomas Karolski
e5dc2d198f [comm] Use getRequestProfile from Authorizer 2022-03-12 21:00:52 +01:00
Thomas Karolski
9f3ed989c4 [authorizer] Determine auth header to use via an environment variable 2022-03-12 21:00:44 +01:00
Thomas Karolski
c459037b04 [authorizer] Move code for extracting auth header into a function 2022-03-12 21:00:36 +01:00
Jarosław Sadziński
eff78ae2e1 (core) New UI for raw data views
Summary:
Creating new UI for raw data views based on design.
- Renaming left for follow up diff
- Link in the menu is hidden for now
To access raw UI, use /p/data URL.

Test Plan: new tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3306
2022-03-12 13:51:48 +01:00
George Gevoian
f02174eb7e (core) Fix error when canceling import
Summary:
If cancel was clicked while a transform section was still being
generated in the Importer, an error was thrown. This refactors
the cancelImportFiles API action to take in the file upload id
in place of the entire DataSourceTransformed parameter, which
contains other values that are irrelevant to canceling. One of those
values, the transform section id, was causing the error to be thrown
since it was momentarily null.

Test Plan: Tested manually.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3317
2022-03-10 16:24:49 -08:00
Alex Hall
77a5d31afe (core) More accurate data size measurement
Summary: As suggested by @dsagal in https://phab.getgrist.com/D3277#inline-36801, change to query `SUM(pgsize - unused)` instead of `SUM(pgsize)` to measure actual data size more accurately. Technically this doesn't reflect the database file size as accurately, but it should reflect sandbox memory usage better, and more importantly it should allow users to see data size decreasing when they delete stuff.

Test Plan: Tested manually by adding rows to a doc and looking at the logs. The data size is smaller and changes more granularly.

Reviewers: dsagal, paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3313
2022-03-09 12:04:16 +02:00
Thomas Karolski
a584bc3a19 [Comm.js] Return a session profile based on the x-remote-user header if set 2022-03-09 10:00:03 +00:00
Jarosław Sadziński
351d70d4fb (core) Serving widget info page from home url
Summary:
Custom widget into page is served from a homeUrl instead
of untrusted URL, which might be not used in grist-core.

Test Plan: manual test

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3307
2022-03-09 10:34:50 +01:00
Jarosław Sadziński
d2b82b84c7 (core) Fixing bug with resuming search on a hidden column.
Summary: Fix for error that happens when a search is resumed after one of the columns was hidden.

Test Plan: Added test that shows the error.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3309
2022-03-09 10:34:17 +01:00
Paul Fitzpatrick
c4d3d7d3bb (core) be careful when reassigning a doc to a worker it was on before
Summary:
Importing a .grist document is implemented in a somewhat clunky way, in a multi-worker setup.

 * First a random worker receives the upload, and updates Grist's various stores appropriately (database, redis, s3).
 * Then a random worker is assigned to serve the document.

If the worker serving the document fails, there is a chance the it will end up assigned to the worker that handled its upload. Currently the worker will misbehave in this case. This diff:

 * Ports a multi-worker test from test/home to run in test/s3, and adds a test simulating a bad scenario seen in the wild.
 * Fixes persistence of any existing document checksum in redis when a worker is assigned.
 * Adds a check when assigned a document to serve, and finding that document already cached locally. It isn't safe to rely only on the document checksum in redis, since that may have expired.
 * Explicitly claims the document on the uploading worker, so this situation becomes even less likely to arise.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3305
2022-03-08 17:20:01 -05:00
Thomas Karolski
116295e42f Minor refactor & comments 2022-03-08 19:40:25 +00:00
Thomas Karolski
82a7f0a796 Implement support for webserver header based auth 2022-03-08 19:24:11 +00:00
Alex Hall
321019217d (core) Lossless imports
Summary:
- Removed string parsing and some type guessing code from parse_data.py. That logic is now implicitly done by ValueGuesser by leaving the initial column type as Any. parse_data.py mostly comes into play when importing files (e.g. Excel) containing values that already have types, i.e. numbers and dates.
- 0s and 1s are treated as numbers instead of booleans to keep imports lossless.
- Removed dateguess.py and test_dateguess.py.
- Changed what `guessDateFormat` does when multiple date formats work equally well for the given data, in order to be consistent with the old dateguess.py.
- Columns containing numbers are now always imported as Numeric, never Int.
- Removed `NullIfEmptyParser` because it was interfering with the new system. Its purpose was to avoid pointlessly changing a column from Any to Text when no actual data was inserted. A different solution to that problem was already added to `_ensure_column_accepts_data` in the data engine in a recent related diff.

Test Plan:
- Added 2 `nbrowser/Importer2` tests.
- Updated various existing tests.
- Extended testing of `guessDateFormat`. Added `guessDateFormats` to show how ambiguous dates are handled internally.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3302
2022-03-08 12:14:39 +02:00
George Gevoian
9522438967 (core) Add Grist forgot password page
Summary:
The page isn't yet linked to from anywhere in the UI, but
will be soon, once the new login page is ready. The page
can still be accessed at login-[s].getgrist.com/forgot-password,
and the flow is similar to the one used by Cognito's hosted UI.

Also refactors much of the existing login app code into smaller
files with less duplication, tweaks password validation to be closer
to Cognito's requirements, and polishes various parts of the UI,
like the verified page CSS, and the form inputs.

Test Plan: Browser, server and project tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3296
2022-03-07 09:11:28 -08:00
Cyprien P
76481d21e0 (core) Enable selecting x axis for the group data columns
Summary:
Until now, users could not pick the column that's currently serving as
the x axis.

To do that, chart needs to support having the x axis undefined.

This diff do just that:
  - allow x axis to be undefined
  - allow setting x axis from group data

Given that charts axis are stored as indexes of the section view
fields array, implementation required introduction of an extra chart
options: `isAxisUndefined`.

Test Plan: Updates existing test and adds new one.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3304
2022-03-07 10:49:18 +01:00
Paul Fitzpatrick
2563fb745a (core) make Grist easier to run with a single server
Summary:
This makes many small changes so that Grist is less fussy to run as a single instance behind a reverse proxy. Some users had difficulty with the self-connections Grist would make, due to internal network setup, and since these are unnecessary in any case in this scenario, they are now optimized away. Likewise some users had difficulties related to doc worker urls, which are now also optimized away. With these changes, users should be able to get a lot further on first try, at least far enough to open and edit documents.

The `GRIST_SINGLE_ORG` setting was proving a bit confusing, since it appeared to only work when set to `docs`. This diff
adds a check for whether the specified org exists, and if not, it creates it. This still depends on having a user email to make as the owner of the team, so there could be remaining difficulties there.

Test Plan: tested manually with nginx

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3299
2022-03-05 13:30:45 -05:00
Cyprien P
0da397ab90 (core) Fix js error when changing X Axis with chart with no series
Summary:
This error only occured for certain charts type. Some chart handled it correctly, others did not.

This brings a more general handling of this issue which should work for all charts.

Test Plan: New nbrowser test case.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3303
2022-03-04 14:06:19 +01:00
Alex Hall
5213972d24 (core) Guess numeric formatting options
Summary:
Change NumberParse.parse to return not just the parsed number but also information it gathered along the way about how the input string was formatted.

Use this in the new NumberParse.guessOptions to guess the actual widget options based on an array of strings.

Use NumberParse.guessOptions in TypeConversion (for when a user explicitly chooses to change type) and in ValueGuesser (for guesses about strings entered into empty columns).

Test Plan: Adds unit tests for NumberParse and ValueGuesser and updates the TypeChange2 nbrowser test.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3294
2022-03-03 21:32:03 +02:00
Jarosław Sadziński
70373550cf (core) Shortcuts help page can be scrolled using a keyboard
Summary: Restoring keyboard navigation on the shortcuts popup.

Test Plan: manual tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3295
2022-03-02 11:43:05 +01:00
George Gevoian
8f5ce9d610 (core) Wait for transform column before finalizing
Summary:
This adds a new private field to ColumnTransform that settles when
the transform column is done preparing. Finalize then waits for this
promise before proceeding.

A bug was occurring when the prepare promise was still pending,
but finalize was triggered indirectly, causing finalize to throw an error
due to the transform column not yet existing.

Test Plan: Tested manually.

Reviewers: jarek, dsagal

Reviewed By: jarek, dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3288
2022-03-01 13:59:14 -08:00
Alex Hall
599545fb11 (core) Fuller guessing of type and options when adding first data to blank columns
Summary:
Adds `common/ValueGuesser.ts` with logic for guessing column type and widget options (only for dates/datetimes) from an array of strings, and converting the strings to the guessed type in a lossless manner, so that converting back to Text gives the original values.

Changes `_ensure_column_accepts_data` in Python to call an exported JS method using the new logic where possible.

Test Plan: Added `test/common/ValueGuesser.ts` to unit test the core guessing logic and a DocApi end-to-end test for what happens to new columns.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3290
2022-03-01 22:00:45 +02:00
Jarosław Sadziński
ae6c857ac5 (core) Frozen columns reordering
Summary: Bug fix. Moving columns over the frozen set didn't work properly.

Test Plan: new tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3297
2022-02-28 22:52:37 +01:00
George Gevoian
fa68b790bb (core) Remove code for unused welcome flows
Summary: Removes code that was marked for removal.

Test Plan: Existing tests still pass.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3289
2022-02-28 13:21:28 -08:00
George Gevoian
83ba2957bf (core) Update failing HomeIntro core test
Summary:
Core doesn't redirect to Cognito or our own sign-up page
when clicking 'sign up' on the welcome screen. Instead, it
redirects to the test login page.

Test Plan: N/A (fixing test)

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3298
2022-02-28 12:23:28 -08:00
Cyprien P
c2070877b6 (core) Fix out of sync GROUP DATA and X Axis options when table change
Summary:
Fix this issue:
  When using “Change Widget” for a chart of a summary table, if I change group-by columns, the X-Axis dropdown becomes empty, and lists values from previous summary (not the actual summary table that’s now shown in the chart). Need to close and reopen creator panel to fix it.

Test Plan: Include nbrowser test

Reviewers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3284
2022-02-25 14:48:27 +01:00
Alex Hall
3445ecb64c (core) Search backwards with Shift+Enter
Summary: As simple as it sounds. Check ev.shiftKey in Enter key handler in search.ts.

Test Plan: Extended test/nbrowser/Search.ts

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3293
2022-02-25 14:07:15 +02:00
George Gevoian
ff4e5d2769 (core) Send emails when 2FA settings are updated
Summary: When user 2FA status is changed, we now send out emails via SendGrid.

Test Plan: Server tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3280
2022-02-24 12:36:50 -08:00
Paul Fitzpatrick
accd640078 (core) add a user.SessionID value for trigger formulas and granular access rules
Summary:
This makes a `user.SessionID` value available in information about the user, for use with trigger formulas and granular access rules. The ID should be constant within a browser session for anonymous user. For logged in users it simply reflects their user id.

This ID makes it possible to write access rules and trigger formulas that allow different anonymous users to create, view, and edit their own records in a document.

For example, you could have a brain-storming document for puns, and allow anyone to add to it (without logging in), letting people edit their own records, but not showing the records to others until they are approved by a moderator. Without something like this, we could only let anonymous people add one field of a record, and not have a secure way to let them edit that field or others in the same record.

Also adds a `user.IsLoggedIn` flag in passing.

Test Plan: Added a test, updated tests. The test added is a mini-moderation doc, don't use it for real because it allows users to edit their entries after a moderator has approved them.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3273
2022-02-22 12:50:43 -05:00
George Gevoian
95592b81bd (core) Skip /welcome/user page for new users
Summary:
Since the new Grist sign-up page has a required field for
name, we can now skip the welcome page asking for the
same thing. Code and tests that can be removed later are
marked with TODOs.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3266
2022-02-22 08:38:22 -08:00
George Gevoian
36843e632b (core) Hide API key when it's not selected
Summary:
The API key is now hidden by default. Clicking the input
will select and reveal the key. As soon as the key loses
selection, it is hidden again.

Test Plan: Project test.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3270
2022-02-21 17:26:45 -08:00