Commit Graph

1459 Commits

Author SHA1 Message Date
Florent
7bc862fb02
Add header=colId option for the table-schema API #719 (#749) 2023-11-17 17:45:15 +02:00
Dmitry S
2eec48b685 (core) Support adjusting OOM score for child sandbox processes.
Summary:
Also update error handling in NSandbox initialization to avoid node
exiting when sandbox can't be created.

Info on oom_score and choom: https://man7.org/linux/man-pages/man1/choom.1.html

Empirically, in docker and under fargate, oom_score (visible in
/proc/PID/oom_score) seems to be calculated approximately as

    ((ProcessRSS / TotalMemory * 1000 + 999 + oom_score_adj) * 2/3)

though this doesn't correspond to any documentation I could find.

In addition, under docker / fargate it does not work to set oom_score_adj (with choom or via /proc/PID/oom_score_adj) to a negative value to give priority to a node process. That's why this diff adjust the score up for sandbox processes instead.

Test Plan:
Checked that grist-omnibus built with this change respects the
variable, and sandbox processes end up with higher oom_score values.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4112
2023-11-16 17:43:06 -05:00
George Gevoian
c9bba5207e (core) Add more telemetry events
Summary: Adds new telemetry events and a flag for whether an event originated from a team site.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: jarek, dsagal

Differential Revision: https://phab.getgrist.com/D4105
2023-11-16 12:38:08 -05:00
Florent
e8789e6531
Issue 740 OIDC login redirect (#742)
* Fix OIDC redirects from team site to personal page after login #740

Also:
 - compare state in session and state passed through parameters
 (otherwise the state won't have any effect regarding the security).
 - delete the session even after an authentication failure

* More logs for OIDC #740

---------

Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-11-15 09:23:32 -05:00
Paul Fitzpatrick
a14543008d (core) updates from grist-core 2023-11-14 16:00:06 -05:00
Paul Fitzpatrick
12317097f7 (core) move some untagged assets served on plugin port
Summary: Some untagged assets on the plugin port could be a problem if that port is merged with the regular Grist app port, so we nest them within a non-conflicting path (/plugins/assets).

Test Plan: see if a test fails anywhere

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4116
2023-11-14 15:58:00 -05:00
Alex Hall
5197891427 (core) Remove transform columns on shutdown
Summary: Call a new user action `RemoveTransformColumns` in ActiveDoc shutdown.

Test Plan: Added nbrowser test

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4107
2023-11-14 22:31:34 +02:00
Alex Hall
3dfe4be5f3 (core) Remove DB transaction from webhook update, add mutex to all webhook endpoints
Summary:
This removes problematic code that was holding a HomeDB transaction while applying user actions which could hang indefinitely, especially if the webhook queue is full as in https://grist.slack.com/archives/C05DBJ6LA1F/p1698159750945949.

The discussion about adding this code is here: https://phab.getgrist.com/D3821#inline-45054

The initial motivation was to roll back HomeDB changes if something went wrong while applying user actions, to avoid saving only part of the changes the user requested. I think it's actually fine to just allow such a partial save to happen - I don't see anything particularly undesirable about keeping an update to the webhook URL if other updates requested by the user didn't also get applied, as the fields don't affect each other.

The comment approving the transaction approach said "so we shouldn't end up leave the transaction hanging around too long" which has been falsified.

It looks like there was also some desire to prevent a mess caused by multiple simultaneous calls to this endpoint, which the transaction may have helped with a little, but didn't really seem like a solution. Comments in `Triggers.ts` also mention fears of race conditions when clearing (some of) the queue and the need for some locking. So I wrapped all webhook-related endpoints in a simple `Mutex` held by the `ActiveDoc` to prevent simultaneous changes. I *think* this is a good thing. These endpoints shouldn't be called frequently enough to create a performance issue, and this shouldn't affect actually sending webhook events when records are added/updated. And it does seem like interleaving calls to these endpoints could cause very weird problems.

Test Plan: Nothing yet, I'd like to hear if others think this is sensible.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D4111
2023-11-14 16:51:21 +02:00
Jarosław Sadziński
67dc41776d (core) Treating null in toggle column as a valid value
Summary:
When action is pruned by ACL rules, the rule engine sometimes
rewrites actions, setting NULLs for boolean columns as a default value.
Null was not considered a valid value and it resulted in a bug in the UI
where the editor for the Toggle column was not rendered.

Test Plan: Added new

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D4110
2023-11-13 20:38:30 +01:00
Paul Fitzpatrick
2f96a56150
fix shadowed variable in CustomView (#743)
This corrects a minor linting problem that recently crept in, by
renaming a variable.
2023-11-13 10:56:01 -05:00
Paul Fitzpatrick
68801474b1 (core) updates from grist-core 2023-11-13 07:58:49 -05:00
Florent
a4998b4b21
Add native OIDC support in Grist #707 (#717) 2023-11-09 14:56:39 -05:00
wunter8
73ab1de6e4
Remove attachments and system files from document download as template (#729) 2023-11-08 15:09:01 -05:00
Jarosław Sadziński
3c219e05f6 (core) Removing the new menu flag
Summary: Enabling the `GRIST_NEW_COLUMN_MENU` flag by default and removing it.

Test Plan: Existing

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4098
2023-11-08 07:37:54 +01:00
Jarosław Sadziński
9262e1f1ef (core) Fixing bug with collapsed custom widget.
Summary:
Fix for a bug. Custom widget when collapsed and expanded was disconnecting from
Grist, as WidgetFrame was disposed to early.

Test Plan: Added new

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4109
2023-11-08 07:36:21 +01:00
Dmitry S
3210eee24f (core) Revamp ForwardAuthLogin and unify with GRIST_PROXY_AUTH_HEADER
Summary:
By default, only respect GRIST_FORWARD_AUTH_HEADER on login endpoints; sessions are used elsewhere.

With GRIST_IGNORE_SESSION, do not use sessions, and respect GRIST_FORWARD_AUTH_HEADER on all endpoints.

GRIST_PROXY_AUTH_HEADER is now a synonym to GRIST_FORWARD_AUTH_HEADER.

Test Plan: Fixed tests. Tested first approach (no GRIST_IGNORE_SESSION) with grist-omnibus manually. Tested the second approach (with GRIST_IGNORE_SESSION) with a Apache-based setup enforcing http basic auth on all endpoints.

Reviewers: paulfitz, georgegevoian

Reviewed By: paulfitz, georgegevoian

Differential Revision: https://phab.getgrist.com/D4104
2023-11-07 16:30:49 -05:00
Alex Hall
b7e9d2705e (core) When a webhook is disabled, clear its queue
Summary: Also fixes a few bugs found along the way, particularly that webhook payloads could contain stale data.

Test Plan: Added an nbrowser test, made existing test a bit more detailed.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4102
2023-11-07 15:48:35 +02:00
George Gevoian
95cbbb8910 (core) Improve dark mode in tutorials
Summary:
Headings 4, 5, and 6 are now properly visible in dark mode. Additionally,
pre-formatted text and code blocks have improved styling in dark mode.

Test Plan: Manual.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4108
2023-11-06 13:06:13 -05:00
Paul Fitzpatrick
8053c81d02 (core) updates from grist-core 2023-11-06 08:20:57 -05:00
Florent
10822d3b86
getHostType: consider APP_DOC_INTERNAL_URL as native (#715)
The getHostType() now returns "native" when the host corresponds to the value of APP_DOC_INTERNAL_URL. T
While trying to scale, with a different internal and public URL for doc workers, and having configured the org to be specified in the path (GRIST_ORG_IN_PATH=true), the APP_DOC_INTERNAL_URL parameter was not treated as internal which made the connection between home server and doc workers impossible.

---------
https://github.com/gristlabs/grist-core/pull/715
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-11-06 09:24:59 +01:00
George Gevoian
7a85aaa7a1 (core) Add new telemetry events
Summary: Adds a handful of new telemetry events, and makes a few tweaks to allow for better organization of telemetry.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D4100
2023-11-01 10:49:33 -04:00
George Gevoian
51f7402297 (core) Show tooltips in other Grist flavors
Summary:
This enables tooltips in other Grist deployment types (e.g. grist-core). Previously,
most of these tooltips were only enabled in the SaaS offering of Grist.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D4097
2023-10-31 23:56:27 -04:00
Paul Fitzpatrick
4fb1df567b (core) change handling of server access prior to full configuration
Summary:
Recently, the server became more strict about not responding to
requests before being fully configured. This is a problem when
a doc worker is trying to check whether it has become available
from a load balancer.

This change gives access to health checks prior to configuration
being complete. Otherwise, app endpoints accessed before full
configuration return a 503.

A flag is added to /status to allow checking explicitly for
the server being ready and configured.

Test Plan: manual

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D4103
2023-10-31 17:42:48 -04:00
Paul Fitzpatrick
07bb90b5a6
allow bundled widgets to be hidden from dropdown, and nested (#714)
This makes a few refinements to bundling widgets:

  * A widget with `published: false` is not shown in the
    custom widget dropdown in the UI. This is so widgets
    can be bundled with the app for "native" use (like the
    calendar widget) without immediately resulting in an
    extra listing in the UI. (There are improvements we'd
    like to make to the UI to better communicate widget
    provenance and quality eventually, which would be a
    helpful alternative to just a binary flag.)

  * A relative path to the custom widget manifest is
    respected. This will make the bundling process marginally
    neater.
2023-10-30 21:13:21 -04:00
Paul Fitzpatrick
6059bdcf66
rename variable to avoid shadowing another (#712)
This fixes a small lint issue in some widget bundling code.
2023-10-30 09:52:42 -04:00
Jarosław Sadziński
c7ba31eb7d (core) Guessing column widget options when transforming from
Summary:
When converting changing the type of Any column, try to guess
the widgetOptions. Especially important for choice and choiceList types.

Test Plan: Existing

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D4088
2023-10-30 13:36:39 +01:00
Paul Fitzpatrick
cc9a9ae8c5 (core) support for bundling custom widgets with the Grist app
Summary:
This adds support for bundling custom widgets with the Grist app, as follows:

 * Adds a new `widgets` component to plugins mechanism.
 * When a set of widgets is provided in a plugin, the html/js/css assets for those widgets are served on the existing untrusted user content port.
 * Any bundled `grist-plugin-api.js` will be served with the Grist app's own version of that file. It is important that bundled widgets not refer to https://docs.getgrist.com for the plugin js, since they must be capable of working offline.
 * The logic for configuring that port is updated a bit.
 * I removed the CustomAttachedView class in favor of applying settings of bundled custom widgets more directly, without modification on view.

Any Grist installation via docker will need an extra step now, since there is an extra port that needs exposing for full functionality. I did add a `GRIST_TRUST_PLUGINS` option for anyone who really doesn't want to do this, and would prefer to trust the plugins and have them served on the same port.

Actually making use of bundling will be another step. It'll be important to mesh it with our SaaS's use of APP_STATIC_URL for serving most static assets.

Design sketch: https://grist.quip.com/bJlWACWzr2R9/Bundled-custom-widgets

Test Plan: added a test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4069
2023-10-27 17:00:10 -04:00
George Gevoian
cb0ce9b20f (core) Avoid reporting unhelpful ResizeObserver error
Summary: The error appears to be benign and not caused by any of our code.

Test Plan:
Reproducing was tricky.

On the Access Rules page, changing the browser's zoom level caused the error to surface. Weirdly enough, it only happened when connected to a larger, external monitor, and not on my laptop's internal monitor.

In any case, manual testing was done to confirm the error is no longer reported to the user after this change.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D4095
2023-10-27 15:26:20 -04:00
George Gevoian
ce23887be0 (core) Avoid editing fields on toggle dbclick
Summary:
This prevents a quirky UI behavior where double-clicking a toggle would cause
the field to start being edited.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4093
2023-10-27 15:08:24 -04:00
Alex Hall
4e67c679b2 (core) Options for plugin API functions which fetch data from the selected table or record
Summary:
Adds a new interface `FetchSelectedOptions` with three keys (including the preexisting `keepEncoded`) and adds/updates an optional `options: FetchSelectedOptions` to six related functions which fetch data from the selected table or record. The `keepEncoded` and `format` options have different default values for different methods for backwards compatibility, but otherwise the different methods now have much more similar behaviour. The new `includeColumns` option allows fetching all columns which was previously only possible using `docApi.fetchTable` (which wasn't always a great alternative) but this requires full access to avoid exposing more data than before and violating user expectations.

Eventually, similar options should be added to `docApi.fetchTable` to make the API even more consistent.

Discussion: https://grist.slack.com/archives/C0234CPPXPA/p1696510548994899

Test Plan: Added a new nbrowser test with a corresponding fixture site and document, showing how the functions have different default option values but are all configurable now.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4077
2023-10-26 23:46:00 +02:00
George Gevoian
1a04c2cffe (core) Improve context menu placement on narrow screens
Summary: On narrow screens, the menu is now less likely to overflow the viewport.

Test Plan: Manual.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D4094
2023-10-25 10:35:49 -04:00
George Gevoian
fed697e676 (core) Tweak Add Column menu
Summary:
Tweaking behavior of the unreleased Add Column menu per feedback from
Anais and Dmitry.

Test Plan: WIP

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4089
2023-10-24 15:35:42 -04:00
George Gevoian
de33c5a3c6 (core) Add tests and tooltips to new Add Column menu
Summary: Adds tooltips to the menu and tests for recently-added functionality.

Test Plan: Browser tests.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Subscribers: JakubSerafin

Differential Revision: https://phab.getgrist.com/D4087
2023-10-24 11:53:26 -04:00
Jarosław Sadziński
69d5ee53a8 (core) Treating API urls as external in cells
Summary:
Links for the API endpoints in a cell didn't work as they were interpreted as
internal routes. Now they are properly detected as external.

Test Plan: Added new test

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D4078
2023-10-24 08:55:08 +02:00
Jakub Serafin
77726849ad (core) setting timezone on create column
Summary: Before this diff, "Create at" and "Modify at" had empty "Timezone" field in column editor panel. This diff is setting document timezone to DateTime column created by this shortcuts

Test Plan: Manual so far

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D4086
2023-10-24 08:26:26 +02:00
Jakub Serafin
d0c1ca2174 (core) Summary:
Cleaning code that was wrongly merged during D4083

Test Plan: Manual smoke tests - create columns and references are working

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4085
2023-10-20 15:43:32 +02:00
Jakub Serafin
91f7606ae6 (core) Aggregate and reverse lookups
Summary:
Reverse and Aggregation lookup.
Aggregation lookup works when table have a reference list column. It allow to list value of any fields of a referenced values, or to make some basic operation on them (sum, average, count)
Reverse lookup works as reverse one, but it allow do to the same operations on all rows that have reference to given row

Test Plan: Manual so far.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4083
2023-10-20 13:26:17 +02:00
George Gevoian
74485f412d (core) Fix delete user button for Google-only accounts
Summary:
An unhandled error was being thrown by CognitoClient when a user was unable
to be found during account deletion. Google-only accounts are no longer
associated with a user in Cognito, so the error was actually benign. A warning is
now logged instead.

Test Plan: Manual.

Reviewers: paulfitz, jarek

Reviewed By: paulfitz, jarek

Differential Revision: https://phab.getgrist.com/D4073
2023-10-18 10:52:58 -04:00
George Gevoian
f6256646ef (core) Remove deprecated code
Summary:
The removed code is no longer reachable now that a new version of the plugin
API is being served by Grist on production.

Test Plan: Existing tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D4081
2023-10-18 09:39:12 -04:00
George Gevoian
f1cf92aca1 (core) Polish new Add Column menu
Summary: Fixes and features for the unreleased Add Column menu.

Test Plan: Manual.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D4076
2023-10-17 15:39:53 -04:00
Paul Fitzpatrick
f32563e8fb (core) updates from grist-core 2023-10-17 06:47:46 +02:00
Florent
eb55afcbc4
Option to export colId as header in CSV / XSLX instead of label (#688) (#692) 2023-10-15 20:17:43 -04:00
Jakub Serafin
2521db4c55 (core) New Columns Menu
Summary:
A menu to be shown when new colum button is added. It's give access to various diffrent shortcuts, like adding new column, unhiding existing ones, fast adding lookup columns or trigger one (authoriship or timestamp). Design document can be found here: https://grist.quip.com/CTgxAQv9Ghjt/Add-Columns-more-easily
To turn on this menu flag GRIST_NEW_COLUMN_MENU to 1

Test Plan: UI tests suite under nbrowser/GridViewNewColumnMenu.ts

Reviewers: jarek, georgegevoian

Reviewed By: georgegevoian

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4074
2023-10-13 22:35:36 +02:00
CamilleLegeron
f66ecbd6df
feat: allow using the existing numeric table IDs in the API (#690) 2023-10-12 19:32:22 +02:00
George Gevoian
0cadb93d25 (core) Update dependencies
Summary:
Changes the minimum version of Node to 18, and updates the Docker images and GitHub workflows to build Grist with Node 18.

Also updates various dependencies and scripts to support building running tests with arm64 builds of Node.

Test Plan: Existing tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3968
2023-10-11 17:36:58 -04:00
Dmitry S
519f2f4fb6 (core) Fix the switch of a new column from Common to Separate settings
Summary:
The switching between Common and Separate settings for a field was broken when
the field had never modified settings.

Test Plan: Added a test case that tickles the issue and tests the fix.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4072
2023-10-11 09:23:21 -04:00
George Gevoian
1be1e5f647 (core) Move theme from ConfigNotifier to ThemeNotifier
Summary:
This reverts the behavior of onOptions, which had unintentionally
changed recently and no longer matched the API documentation.

Test Plan: Existing tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4064
2023-10-10 21:45:10 -04:00
Jarosław Sadziński
a101337213 (core) Disabling unsaved changes when logging in
Summary:
Disabling unsaved changes when user tries to login or singup as those endpoints
tries to redirect back to the unsaved fork.

Test Plan: Added new test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D4071
2023-10-10 16:53:50 +02:00
George Gevoian
988ab47376 (core) Scroll shift selection into view
Summary: Using the selection shortcuts will now scroll the selection into view.

Test Plan: Manual.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4055
2023-10-10 10:12:56 -04:00
Jarosław Sadziński
083a20417e (core) Tests and bug fixes for bidirectional linking
Summary:
- Adding tests for bidirectional linking
- Fixing loop bug for bidirectional linking in custom widgets which use row filtering

Test Plan: New tests

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Differential Revision: https://phab.getgrist.com/D4070
2023-10-10 15:31:48 +02:00
Jarosław Sadziński
a8e0f96813 (core) Extending widget
Summary:
- Adding new option 'strictType' to widget mapping
- Refreshing mappings when widget options are changed

Test Plan: Updated

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Differential Revision: https://phab.getgrist.com/D4061
2023-10-09 14:53:20 +02:00
Jarosław Sadziński
572279916e (core) Mappings improvements
Summary:
- Adding new icon for calendar view (the old one by just bigger)
- When there are no columns to map the select box is grayed out
- Optional mappings can be cleared now

Test Plan: Added

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Subscribers: JakubSerafin

Differential Revision: https://phab.getgrist.com/D4066
2023-10-09 10:58:54 +02:00
Jarosław Sadziński
34f366585d (core) Removing org check in the closeOrg endpoint.
Summary:
Fixing a bug: account couldn't be closed when there was
a team site shared with everyone. Endpoint was checking
if there are any other team sites available, but this is already
tested in "Doom" in a better way, so this check was removed.

Test Plan: New test

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4067
2023-10-06 15:35:42 +02:00
Jarosław Sadziński
ad299f338a (core) Converting big number (9 digits or more) to date directly
Summary:
Interpret huge numbers (>8 digits) as timestamps when converting numeric column to date.
Convert date/date time columns to timestamp when converted from numeric/int column.

Test Plan: Updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal, alexmojaki

Differential Revision: https://phab.getgrist.com/D4030
2023-10-05 16:58:25 +02:00
Jakub Serafin
498ad07d38 (core) passing language as a query parameter to custom widgets
Summary: to allow custom widget having optional translations, lagunage seeted in user profile is passed as query parameter to custom widget

Test Plan: test added to check if query parameter is existing in url when settings is changed in profile

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: jarek, paulfitz

Differential Revision: https://phab.getgrist.com/D4045
2023-10-03 14:14:32 +02:00
Dmitry S
fbae81648c (core) Add options to /status health-check endpoints to check DB and Redis liveness.
Summary:
- /status accepts new optional query parameters: db=1, redis=1, and timeout=<ms> (defaults to 10_000).
- These verify that the server can make trivial calls to DB/Redis, and that they return within the timeout.
- New HealthCheck tests simulates DB and Redis problems.
- Added resilience to Redis reconnects (helped by a test case that simulates disconnects)
- When closing Redis-based session store, disconnect from Redis (to avoid hanging tests)

Some associated test reorg:
- Move stripeTools out of test/nbrowser, and remove an unnecessary dependency,
  to avoid starting up browser for gen-server tests.
- Move TcpForwarder to its own file, to use in the new test.

Test Plan: Added a new HealthCheck test that simulates DB and Redis problems.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4054
2023-10-02 14:41:04 -04:00
Jarosław Sadziński
61023ecaa9 (core) Preventing updates for widgets options when nothing has changed
Summary:
Custom widget was reseving its options, even though they haven't been changed. This
resulted with an ACL error and a popup message in readonly mode.

Test Plan:
Existing and manual.
To recreate:
1. Create a page with a calendar widget (don't change anything)
2. View this page as a Viewer (using ACL dropdown in tools)
3. The error should be shown.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D4060
2023-09-27 23:19:25 +02:00
George Gevoian
f38df564a9 (core) Add Command API to Grist Plugin API
Summary:
The new Command API provides limited access to Grist Commands from within cusotm
widgets. This includes the ability to perform undo and redo, which is bound to
the same keyboard shortcut as Grist by default.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: paulfitz, jarek

Differential Revision: https://phab.getgrist.com/D4050
2023-09-27 13:25:18 -04:00
Alex Hall
9b36fb4dab (core) Fix error in sandbox when removing multiple summary source columns
Summary:
Fixes a very specific bug reported here: https://grist.slack.com/archives/C069RUP71/p1694630242765769

The error occurred when:

1. Removing multiple columns simultaneously
2. Those columns were sources of groupby columns for a summary table (so removing them meant recreating the summary table and thus deleting its columns)
3. There was a display column for one of the columns that got deleted (either directly or indirectly) which was set to be automatically removed since it was no longer needed, but this failed because the column was already deleted as part of earlier table removal.

I fixed this by making `apply_auto_removes` remove table records last, so removing the display column wouldn't be a problem.

That fixed the original error, but then I noticed that trying to undo the removal could lead to another error (under even more specific circumstances). It's hard to see exactly why, but I can see that just 3 `RemoveColumn` user actions generated over 100 doc actions and corresponding undo actions, hence the difficulty in narrowing the problem down. This is partly because removing a single column would recreate a summary table, only for that table to be immediately replaced again when another column was removed. Making the frontend send a single `[BulkRemoveRecord, _grist_Tables_column, ...] ` leads to a more efficient and sensible process with about half as many doc actions and no undo error. I think this alone would also solve the original error, but the data engine change seems more generally helpful and worth keeping.

Test Plan: Added a Python test and an nbrowser test

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4052
2023-09-27 16:23:49 +02:00
Paul Fitzpatrick
bcc59c0f46 (core) updates from grist-core 2023-09-27 09:18:41 -04:00
Jarosław Sadziński
cce185956c (core) Delete my account button
Summary:
Adding new "Delete my account" button to the profile page that allows users to remove completely
their accounts as long as they don't own any team site.

Test Plan: Added

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian, paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4037
2023-09-27 14:49:23 +02:00
George Gevoian
e033889b6a (core) Fix calendar and card tip bug on mobile
Summary:
On mobile, tips for calendar and card list aren't currently
shown, but the creator panel was still automatically being
opened in preparation for showing the tip.

Test Plan: Manual and existing tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D4053
2023-09-26 21:05:30 -04:00
Dmitry S
82c95ec074 (core) Allow scrolling the grid horizontally while column-shadow is under the cursor
Test Plan: Tested manually in Firefox.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4056
2023-09-26 20:42:10 -04:00
Dmitry S
2705d41c34 (core) Add timeouts to prevent ActiveDoc bad state during shutdown.
Summary:
Add two shutdown-related timeouts.

1. One is to limit the duration of any work that happens once shutdown
   begins. In particular, waiting for an update to current time could block
   indefinitely if the data engine is unresponsive. Such awaits are now
   limited to 5 seconds.

2. The other is to allow documents to get shutdown for inactivity even when
   some work takes forever. Certain work (e.g. applying user actions)
   generally prevents a document from shutting down while it's pending. This
   prevention is now limited to 5 minutes.

   Shutting down a doc while something is pending may break some
   assumptions, and lead to errors. The timeout is long to let us assume
   that the work is stuck, and that errors are better than waiting forever.

Other changes:
- Periodic ActiveDoc work (intervals) is now started when a doc finishes
  loading rather than in the constructor. The difference only showed up in
  tests which makes the intervals much shorter.

- Move timeoutReached() utility function to gutil, and use it for
  isLongerThan(), since they are basically identical. Also makes sure that the
  timer in these is cleared in all cases.

- Remove duplicate waitForIt implementation (previously had a copy in both
  test/server and core/test/server).

- Change testUtil.captureLog to pass messages to its callback, to allow asserts
  on messages within the callback.

Test Plan:
Added new unittests for the new shutdowns, including a replication
of a bad state that was possible during shutdown.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4040
2023-09-26 15:32:49 -04:00
Janet Vorobyeva
29f07a8a4f
Bidirectional Linking (#622)
Allows bidirectional / cyclic linking for same-record cursor links.
This should allow multiple sections to all synchronize their cursors,
such that clicking in any one of them will move all the others.

Works even if some sections in the cycle have rows filtered out (the 
filtered-out sections might desync their cursors, but the correct cursor
position will still propagate downstream, and they'll re-sync if clicking on
a row that is present in them)

Under the hood, each cursor has a _lastEditedAt counter, updated when
a user's action changes the cursor in a section, such that we can always
tell which section was touched most recently. This is used to resolve
conflicts stably when dealing with cycles or chains of cursor-links.

Updated selectBy and recursiveMoveToCursorPos to handle cycles

Updated tests for selectBy behavior

However, main bidirectional-linking tests are not in this commit, they'll come in a subsequent PR
2023-09-25 18:48:18 -04:00
George Gevoian
e01ebe1ae8 (core) Fix blurry tooltips
Summary: GPU acceleration was causing tooltip text to appear blurry.

Test Plan: Manual.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4046
2023-09-22 03:51:58 -04:00
George Gevoian
273b976cab (core) Polish dark mode and remove beta tag
Summary:
Polishes support for dark mode and enables syncing with the OS theme
by default.

Test Plan: Manual.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Subscribers: JakubSerafin

Differential Revision: https://phab.getgrist.com/D4041
2023-09-21 13:14:48 -04:00
George Gevoian
d1826987bb (core) Revert part of the recent tweaks to AppHeader
Summary:
In non-SaaS Grist, the AppHeader will no longer link to templates
page. Also, the last visited site is now reset when switching accounts.

Test Plan: Existing tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4048
2023-09-20 11:27:46 -04:00
George Gevoian
927e92e3e8 (core) Add tip for calendar widget configuration
Summary:
The tip is shown in the creator panel, in the subtab that lists the column
mapping configuration for the calendar widget. The panel now  automatically
opens the first time a calendar widget is added to a page (via the Add New menu).

Test Plan: Manual.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Subscribers: JakubSerafin, jarek

Differential Revision: https://phab.getgrist.com/D4047
2023-09-20 11:10:58 -04:00
George Gevoian
581a62306e (core) Update video tour
Summary: The embed id of the video tour now points to the new introduction video.

Test Plan: Manual.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4049
2023-09-19 16:28:36 -04:00
George Gevoian
4c25aa7d3d (core) Support dark mode in custom widgets
Test Plan: Manual.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4036
2023-09-19 16:08:21 -04:00
Paul Fitzpatrick
ffbf93b85f (core) updates from grist-core 2023-09-18 15:04:08 -04:00
Jarosław Sadziński
d13a75a453 (core) Avoiding the view layout's rebuild, when nothing has changed.
Summary:
Layout is rebuild when it is updated from outside, for example after saving. But actually we don't need to rebuild it, because most of the time nothing has changed.
This is important for custom widgets, which will reload the iframe, even though the dom is not changed, but just moved from previous layout to the new one.

Test Plan: Manual and existing.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4039
2023-09-18 09:47:50 +02:00
Janet Vorobyeva
8fd6bd1afb
Catches isDisposed bug with rightpanel linkInfo (#676)
* Catches isDisposed bug with rightpanel linkInfo

Would happen when deleting srcSection of a link with rightpanel
open to the linking tab
2023-09-15 14:05:26 -04:00
Jarosław Sadziński
be995d4857 (core) Clicking on a link in a cell won't open the editor.
Summary:
Simple click (a click on a already focused cell) was opening the editor
even though user clicked an anchor in the cell.

Test Plan: Added test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D4044
2023-09-14 17:28:50 +02:00
Paul Fitzpatrick
bc6f550471
normalize logging of dates in siteUsage across databases (#675)
Date handling for SQLite and PostgreSQL is inconsistent. This makes
sure that dates in `siteUsage` logs are in a consistent format that
includes time zone information.
2023-09-13 14:44:04 -04:00
Jarosław Sadziński
2b11000457 (core) Multiple types in columns mapping
Summary: Mapping for a single column in custom widgets accepts now multiple types as comma separeted list.

Test Plan: Added new

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Differential Revision: https://phab.getgrist.com/D4042
2023-09-13 14:41:31 +02:00
George Gevoian
40c5f7b738 (core) Add documentCreated telemetry event
Summary:
The event is triggered whenever a document is created, imported, or
duplicated.

Test Plan: Tested manually.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4035
2023-09-13 01:13:00 -04:00
George Gevoian
76e822eb23 (core) Add welcomeQuestionsSubmitted telemetry event
Summary:
The new event captures responses to the welcome questionnaire.

Responses are also still sent to the special Grist document configured with
the DOC_ID_NEW_USER_INFO variable.

Test Plan: Tested manually.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4034
2023-09-13 00:31:04 -04:00
Paul Fitzpatrick
f659f3655d (core) updates from grist-core 2023-09-12 14:16:02 -04:00
Jarosław Sadziński
75d6b7ab30 (core) Fixing code after core merge
Summary: Fixing code after last core-merge

Test Plan: Existing

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Subscribers: JakubSerafin

Differential Revision: https://phab.getgrist.com/D4038
2023-09-12 11:59:36 +02:00
Janet Vorobyeva
5a4433c60c
Fixes linking "srcSection disposed" bug (#670)
Test "RawData should remove all tables except one (including referenced
summary table)"

was failing with error: "this._srcSection.activeRowId is not a function"
added an extra isDisposed check to catch it
2023-09-12 00:15:10 -04:00
Paul Fitzpatrick
18f7e255df (core) updates from grist-core 2023-09-11 10:00:39 -04:00
Paul Fitzpatrick
585cf02f6c
add hooks for tweaking how downloads happen (for grist-static) (#665)
This makes three main changes:
  * Adds a hook to transform download links.
  * Adds a hook to add an externally created ActiveDoc to a DocManager.
  * Rejiggers XLSX export code so it can be used without streaming,
    which is currently tricky in a browser. Regular usage with node
    continues to use streaming.

With these changes, I have a POC in hand that updates grist-static
to support downloading CSVs, XLSXs, and .grist files.
2023-09-09 14:50:32 -04:00
Janet Vorobyeva
f8c1bd612c
Linkstate refactor (#609)
* Linkingstate Refactor, and displaying link info in rightpanel

Big refactor to LinkingState
    Collects descriptive/user-facing labels into FilterState
    Unifies/cleans up some logic
    Adds LinkTypeDescription, a string enum which can be used
    to easily switch/case between various cases of linking, and
    codifies the logic in one place (currently only used for linkInfo)

Adds Link info to creator panel, near SelectBy dropdown

Bugfix: Disables linking from Attachment columns
Bugfix/Behavior change: changed linking with empty RefLists to better
match behavior of refs.
    for context: Linking by a blank Ref filters to show records with a
    blank value for that Ref. Previously this didn't work with RefLists. 
    Linking from a blank refList would show no records 
    (except in some cases involving summary tables)
    Fixed this so that linking by a blank val consistently means "show
    all records where the corresponding col is blank"
2023-09-08 14:21:32 -04:00
Florent
5ff79703b4
Introduce GRIST_ANON_PLAYGROUND variable #642 (#651)
* `GRIST_ANON_PLAYGROUND`: When set to 'false' deny anonymous users access to the home page
 * `GRIST_FORCE_LOGIN`: Much like `GRIST_ANON_PLAYGROUND` but don't support anonymous access at all (features like sharing docs publicly requires authentication)

---------

Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-09-08 09:05:52 -04:00
Jarosław Sadziński
a0fc11c8d1 (core) Fixing memos in record dependent rules.
Summary:
Memos in row dependent rules were ignored if the action was denied unconditionally. It prevented creating different memos for different users.
Now ACL is forced to check each row, to find the correct memo to show when create/update/delete action is denied.

Test Plan: Existing and new.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz, dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D4024
2023-09-08 10:16:09 +02:00
Jarosław Sadziński
4ead16176f (core) Showing workspace selector on duplicate document popup
Summary:
Duplicate document popup wasn't offering workspace selection for personal orgs.
Also, when workspace was removed, the URL wasn't updated which left user with an error page `workspace not found`.

Test Plan: Updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4031
2023-09-07 16:16:33 +02:00
George Gevoian
3dadf93c98 (core) Add support for auto-copying docs on signup
Summary:
The new "copyDoc" query parameter on the login page sets a short-lived cookie, which is
then read when welcoming a new user to copy that document to their Home workspace, and
redirect to it. Currently, only templates and bare forks set this parameter.

A new API endpoint for copying a document to a workspace was also added.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3992
2023-09-06 15:12:08 -04:00
George Gevoian
90fb4434cc
Add ws id and doc name params to POST /docs (#655) 2023-09-05 14:27:35 -04:00
Florent
1ac15912b1
Make server take into account GRIST_SERVERS (#659)
Also ensure that only a home server will setup the database.

resolves #654

Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-09-05 12:05:29 -04:00
Paul Fitzpatrick
6dab12f301 (core) updates from grist-core 2023-09-05 11:12:15 -04:00
Paul Fitzpatrick
bfd0fa8c7f
add an endpoint for doing SQL selects (#641)
* add an endpoint for doing SQL selects

This adds an endpoint for doing SQL selects directly on a Grist document. Other kinds of statements are not supported. There is a default timeout of a second on queries.

This follows loosely an API design by Alex Hall.

Co-authored-by: jarek <jaroslaw.sadzinski@gmail.com>
2023-09-04 09:21:18 -04:00
Andrew Langford
b465e07bb4
Fixed typo in a comment (#656) 2023-09-01 09:59:33 -04:00
George Gevoian
70feb336d9 (core) Add AI assistant usage banners
Summary:
Banners are now shown when there are low or no AI assistant
credits remaining.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4018
2023-08-30 16:00:04 -04:00
Jarosław Sadziński
98068cb86c (core) Hiding expand button if there is single widget on a page
Summary:
Expand button on a page with a single widget seems pointless and presents a bad UX when
embedded.

Test Plan: new tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D4020
2023-08-30 21:28:41 +02:00
Jarosław Sadziński
58323f5313 (core) Adding testId to the widget iframe once it receives the ready message
Summary:
Iframe with custom widget is marked with a test class `test-custom-widget-ready` when
it receives the `ready` message from the rendered widget.

Test Plan: Added and updated. Existing test should pass.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D4023
2023-08-30 21:28:06 +02:00
Jakub Serafin
daf7614fd7 (core) fix for duplicated "Learn more about custom widgets" section
Test Plan: I checked if section is showed only once after this change. it is.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4022
2023-08-30 16:43:49 +02:00
George Gevoian
ac6933e39f (core) Fix unreadable text in dark mode
Summary:
The message shown in a linked widget when no row is
selected was unreadable in dark mode.

Test Plan: Manual.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4019
2023-08-30 10:25:54 -04:00
Jakub Serafin
1bc97773c8 (core) fix for master not building - in gristUrls.ts imports was duplicated
Test Plan: none

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4021
2023-08-30 11:06:31 +02:00
Jakub Serafin
942fc96225 (core) custom widget appear as build-in widget
Summary: Added boilerplate code needed to create new wigets in "Add new" menu, that are wrapped around existing custom widgets. More details can be found here: https://grist.quip.com/larhAGRKyl6Z/Custom-widgets-in-Add-Widget-menu

Test Plan: nbowser tests added to verify if item in menu exits, if widget is rendered, and right side menu has widget selection and read access selection hided.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3994
2023-08-30 09:44:25 +02:00
Paul Fitzpatrick
8d687a7651
remove a log message about fetching URLs (#643)
Every fetch made from the client is logged to the console.
But this isn't really necessary, and is particularly confusing
in grist-static, where those fetches are virtualized.

Tests in grist-saas may need adjusting to remove the logger.
2023-08-29 08:49:25 -04:00
Jarosław Sadziński
b6a431dd58 (core) Cursor in custom widgets
Summary:
Adding a new method `setCursorPos` in the widget API, and a new configuration option for the ready message `allowSelectBy` that exposes custom widgets in the `Select by` dropdown.
With this, a custom widget can control the position of the linked widgets and is able to change the column in the creator panel.

Test Plan: Added new test. Existing tests should pass.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Subscribers: JakubSerafin

Differential Revision: https://phab.getgrist.com/D3993
2023-08-29 09:19:52 +02:00
Paul Fitzpatrick
c02acff361 (core) updates from grist-core 2023-08-28 08:24:41 -04:00
Paul Fitzpatrick
7aebdd15f6
tweak XLSX export worker to make Piscina happy under Electon (#646)
Adds a dummy default export to the worker exporter script used
for producing XLSX. This method exists only to make Piscina
happier. With it, Piscina will load this file using a regular
require(), which under Electron will deal fine with Electron's
ASAR app bundle. Without it, Piscina will try fancier methods
that aren't at the time of writing correctly patched to
deal with an ASAR app bundle, and so report that this
file doesn't exist instead of exporting an XLSX file.

I tried various other solutions such as upgrading Electron,
unpacking various files, patching Piscina, and this was
overall the simplest.

See https://github.com/gristlabs/grist-electron/issues/9
2023-08-28 08:11:01 -04:00
Jakub Serafin
2073e6224e (core) Inactive card list widget now have idicator that show cursor position of linked widget
Test Plan: nbrowser test added

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4009
2023-08-25 16:27:20 +02:00
Dmitry S
d5a4605d2a (core) Improve encoding detection for csv imports, and make encoding an editable option.
Summary:
- Using a sample of data was causing poor detection if the sample were
  cut mid-character. Switch to using line-based detection.
- Add a simple option for changing encoding. No convenient UI is offered
  since config UI is auto-generated, but this at least makes it possible to
  recover from bad guesses.
- Upgrades chardet library for good measure.

- Also fixes python3-building step, to more reliably rebuild Python
  dependencies when requirements3.* files change.

Test Plan:
Added a python-side test case, and a browser test that encodings can
be switched, errors are displayed, and wrong encodings fail recoverably.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3979
2023-08-24 09:50:52 -04:00
Florent
ee31764b83
DocApi: Implement DELETE on columns (#601) (#640)
* Factorize generateDocAndUrl
* Add describe for regrouping /records
2023-08-24 14:33:53 +02:00
Florent
9dfebefc9b
Dump the rule for ACL formula warnings (#639)
When an ACL formula fails to be run, a warning is printed. However, it is painful to know which formula is concerned by the warning.

ACL: if RHS is null, return false for "in" and "not in"
2023-08-23 09:23:29 -04:00
Janet Vorobyeva
b41fa31dd9
Fixes bug 'this.valueFormatter is not a function' (#632)
https://grist.slack.com/archives/C069RUP71/p1692034396980779
Bug showed up when deleting a page with the right kinds of
widgets/fields, due to a missing isDisposed check

Bug was found in DateTextBox, but I added the fix to NTextBox which
had an identical bit of code
2023-08-22 03:17:01 -04:00
Paul Fitzpatrick
b3a71374d1 (core) updates from grist-core 2023-08-21 09:15:53 -04:00
Paul Fitzpatrick
0be858c19d
allow AI Assistance to run against any chat-completion-style endpoint (#630)
This adds an ASSISTANT_CHAT_COMPLETION_ENDPOINT which can be used
to enable AI Assistance instead of an OpenAI API key. The assistant
then works against compatible endpoints, in the mechanical sense.
Quality of course will depend on the model. I found some tweaks
to the prompt that work well both for Llama-2 and for OpenAI's models,
but I'm not including them here because they would conflict with some
prompt changes that are already in the works.

Co-authored-by: Alex Hall <alex.mojaki@gmail.com>
2023-08-18 16:14:42 -04:00
Alex Hall
166312be3a (core) Add dialog with options to allow downloading without history or data
Summary:
{F74398}

Refactored the 'radio checkboxes' in the modal for deleting a page and reused them here.

The option to download as a template already existed in the server code but wasn't being exercised by the frontend. Also added an option to remove just the history, which is the main motivation for this diff.

Test Plan: Expanded the existing nbrowser test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3999
2023-08-18 15:38:24 +02:00
Alex Hall
a1d31e41ad (core) Prevent the AI assistant response from including class definitions
Summary:
Sometimes the model repeats the classes given in the prompt which would mess up extracting the actual formula. This diff solves this by:

1. Changes the generated Python schema so that (a) the thing that needs completing is a plain top level function instead of a property/method inside the class and (2) the classes are fully valid syntax, which makes it easier to
2. Remove classes from the parsed Python code when converting the completion to a formula.
3. Tweak the prompt wording to discourage including classes in general, especially because sometimes the model tries to solve the problem by defining extra methods/attributes/classes.

While I was at it, I changed type hints to use builtins (e.g. `list` instead of `List`) to prevent `from typing import List` which was happening sometimes and would look weird in a formula. Similarly I removed `@dataclass` since that also implies an import, and this also fits with the tweaked wording that the classes are fake.

Test Plan:
Added a new test case to the formula dataset which triggers the unwanted behaviour. The factors that seem to trigger the problem are (1) a small schema so the classes are easier to repeat and (2) the need to import modules, which the model wants to place before all other code. The case failed before this diff and succeeded after. The tweaked wording reduces the chances of repeating the classes but didn't eliminate it, so forcibly removing the classes in Python was needed.

There were also a couple of other existing cases where repeating the classes was observed before but not after.

Overall the score increased from 49 to 51 out of 69 (including the new case). At one point the score was 53, but changes in whitespace were enough to make it drop again.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4000
2023-08-18 12:50:09 +02:00
Alex Hall
999d723d14 (core) Cleanup old frontend code for handling empty summary groups
Summary: The removed TODO was resolved in https://phab.getgrist.com/D3489

Test Plan: Existing tests

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4006
2023-08-18 12:46:15 +02:00
Paul Fitzpatrick
fe846d3068 (core) allow copying errors from regular cell editor
Summary:
There was already some support for copying errors from the detached/expanded formula editor. This adds support for copying errors in the regular cell editor.

Getting error details is now done only by clicking on the expand icon - previously you could click on the error text itself.

A few unrelated test changes are made for exact-pixel checks that were often out by approx 1 pixel on my machine.

Test Plan: Updated tests. Tested copying/pasting manually.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4002
2023-08-17 09:35:22 -04:00
Jarosław Sadziński
fad421b7c0 (core) Removing temporary pro site
Summary: Creating a pro team site after Stripe checkout. Previously a stub site was always created and never removed, even if a user cancels the checkout process, which resulted in multiple 'ghost' sites that can't be removed.

Test Plan: Updated and added

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3985
2023-08-16 08:11:26 +02:00
Philip Standt
9df62e3d81
Add appearance parameter to override theme preferences (#620) 2023-08-15 10:29:29 -07:00
Paul Fitzpatrick
f1a0b61e15 (core) updates from grist-core 2023-08-15 12:15:16 -04:00
Philip Standt
cbdffdfff8
Support grid selection with Ctrl+Shift+Arrow (#615) 2023-08-14 09:28:41 -07:00
Florent
fde56ffa33
DocApi: Introduce hidden option for GET on records (#623)
Also test that PUT /columns?replaceall=1 does not remove hidden columns
2023-08-14 16:17:46 +02:00
Florent
e93ad5a0c5
Issue #601: implement PUT on columns (#605)
* Also Add includeHidden param for GET on columns
2023-08-11 15:12:43 +02:00
George Gevoian
0f2b1f6a89 (core) Polish "Click to insert $id" tooltip
Summary:
The tooltip also broke recently on Firefox after upgrading Ace. This
includes a fix for the regression.

Test Plan: Updated test.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek, dsagal

Differential Revision: https://phab.getgrist.com/D3982
2023-08-09 07:38:34 -07:00
Jarosław Sadziński
732611c356 (core) Removing GRIST_FORMULA_ASSISTANT flag
Summary:
A floating formula editor is available by default and in the basic setup allows just formula modification.
AI assistant is now an optional component of the floating editor and it is controlled by OPENAPI_KEY presence.
Env variable GRIST_FORMULA_ASSISTANT was removed, new feature flag HAS_FORMULA_ASSISTANT is derived from the presence of OPENAPI_KEY.

Also updated anonymous signup nudge. By default it displays only info that this feature is only for logged in users.

Test Plan: updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3987
2023-08-09 10:08:18 +02:00
George Gevoian
24bbf375f9 (core) Add earliestDocCreatedAt to telemetry
Summary: This tracks the earliest document creation time, if any, for each site.

Test Plan: Server tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3991
2023-08-08 00:48:18 -07:00
Dmitry S
efb5fc63d7 (core) Change the label/nickname of the paid plan to Pro
Summary:
This is to match the pricing page, and to reduce confusion. The same
change was made to the description of the corresponding "pricing" object
on Stripe. The Stripe value is what's used in practice; this value is
actually only used as a fallback.

Test Plan: The code change does not affect tests, but several tests fixed after grist-core merge.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3990
2023-08-07 23:45:18 -04:00
Paul Fitzpatrick
031076cd07 (core) updates from grist-core 2023-08-07 14:29:30 -04:00
CamilleLegeron
02841bd15c
Header colored (#581) 2023-08-07 11:01:35 -07:00
Dmitry S
526a5df157 (core) Manage memory used for websocket responses to reduce the risk of server crashes.
Summary:
- Implements MemoryPool for waiting on memory reservations.
- Uses MemoryPool to control memory used for stringifying JSON responses in Client.ts
- Limits total size of _missedMessages that may be queued for a particular client.
- Upgrades ws library, which may reduce memory usage, and allows pausing the websocket for testing.
  - The upgrade changed subtle behavior corners, requiring various fixes to code and tests.

- dos.ts:
  - Includes Paul's fixes and updates to the dos.ts script for manual stress-testing.
  - Logging tweaks, to avoid excessive dumps on uncaughtError, and include timestamps.

Test Plan:
- Includes a test that measures heap size, and fails without memory management.
- Includes a unittest for MemoryPool
- Some cleanup and additions to TestServer helper; in particular adds makeUserApi() helper used in multiple tests.
- Some fixes related to ws upgrade.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3974
2023-08-07 11:28:31 -04:00
Paul Fitzpatrick
016ab66ea0
fix type issue if python is missing when running unsandboxed (#610)
Clean up a small recently introduced type issue.
2023-08-05 11:24:01 -04:00
Jarosław Sadziński
00b88fd683 (core) Adding links to description tooltips
Summary: Column and widget descriptions now support links in text.

Test Plan: Updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3981
2023-08-04 18:34:04 +02:00
Alex Hall
5dfa9a542c (core) Upgrade to Python 3.11
Summary: Replaced mentions of Python 3.9 with 3.11

Test Plan: this

Reviewers: paulfitz, georgegevoian

Reviewed By: paulfitz, georgegevoian

Subscribers: dsagal, georgegevoian, paulfitz

Differential Revision: https://phab.getgrist.com/D3980
2023-08-04 18:19:40 +02:00
Jarosław Sadziński
6416994c22 (core) Import redesign
Summary: New UI design for incremental imports.

Test Plan: Updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3945
2023-08-04 14:59:55 +02:00
George Gevoian
05c15e4ec3 (core) Show tweaked formula in AI responses
Summary:
The formula that's used when the Apply button is clicked, and the formula that's
shown in responses from the Formula Assistant should now be the same. Previously, they
would differ slightly.

Test Plan: Server tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3977
2023-08-04 01:31:31 -07:00
Jarosław Sadziński
4cfa033078 (core) TypeTransform race condition fix
Summary:
TypeTransformation was flaky. Probably after upgrading AceEditor we introduced a race condition between updating the revised formula and doing the transformation. Now we explicitly make sure that the formula is updated.

I also fixed some other flaky tests.

Test Plan: Updated

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3984
2023-08-02 17:11:34 +02:00
Paul Fitzpatrick
8110a26873 (core) updates from grist-core 2023-08-01 10:52:34 -04:00
Paul Fitzpatrick
61f954ff05
move getTemplateOrg method; enable template org in docker tests (#602)
* move getTemplateOrg method; enable template org in docker tests

This moves the `getTemplateOrg` method to a neutral venue for the
convenience of `grist-static`, otherwise a lot of awkward dependencies
get pulled in needlessly in new parts of the app.

This also fixes docker tests using the template org.
2023-07-31 16:10:59 -04:00
George Gevoian
9484fa14db (core) Fix mislabeled telemetry metadata
Summary:
The oldFormula and newFormula metadata fields had their names switched by
mistake.

Test Plan: N/A

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Differential Revision: https://phab.getgrist.com/D3976
2023-07-31 00:55:28 -07:00
John Cant
e1df6039c2
REQUEST now supports POST (#588)
* REQUEST now supports POST
* Add extra flag for enabling REQUEST, also update README and comments

Co-authored-by: John Cant <a.jonncant@gmail.com>
Co-authored-by: Alex Hall <alex.mojaki@gmail.com>
2023-07-30 15:13:43 -04:00
Alex Hall
9b87a6f06a (core) updates from grist-core 2023-07-28 17:40:31 +02:00
Jakub Serafin
d4c5082799
reverting whitespace changes in DocPageModel (#594) 2023-07-27 14:36:43 +02:00
Paul Fitzpatrick
f0b9e1f7e9
allow Grist front-end to function when location history is unavailable (#596)
* allow Grist front-end to function when location history is unavailable

When the Grist front-end is embedded in an iframe, using a srcdoc
attribute, history.pushState and similar methods are unavailable.
Currently, that makes it impossible to navigate between Grist pages,
since an access error is thrown (behavior may be browser dependent).
With this change, navigation succeeds.

* give unrelated possibly slow test a little more time
2023-07-27 07:02:17 -04:00
George Gevoian
a77170c4bd (core) Tweak navbar, breadcrumbs, and sign-in buttons
Summary:
The changes are intended to smooth over some sharp edges when a signed-out user
is using Grist (particularly while on the templates site).

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3957
2023-07-26 22:26:55 -07:00
Alex Hall
bc54a6646e (core) Filter rows based on linked widgets when exporting view
Summary:
Fixes a problem reported here: https://community.getgrist.com/t/exporting-the-records-in-a-linked-view/2556/4

The download CSV/Excel link now contains an additional `linkingFilter` URL parameter containing JSON-encoded `filters` and `operations`. This object is originally created in the frontend in `LinkingState`, and previously it was only used internally in the frontend. It would make its way via `QuerySetManager` to `QuerySet.getFilterFunc` where the actual filtering logic happened. Now most of that logic has been moved to a similar function in `common`. The new function works with a new interface `ColumnGettersByColId` which abstract over the different ways data is accessed in the client and server in this context. There's no significant new logic in the diff, just refactoring and wiring.

Test Plan: Expanded two `nbrowser/SelectBy*.ts` test suites to also check the contents of a downloaded CSV in different linking scenarios.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3961
2023-07-26 21:49:52 +02:00
Vincent Viers
ab6807c342
fix: don't count falses in summary (#589) 2023-07-26 12:08:04 -07:00
Alex Hall
3f71c9c488 (core) Cleanup: Remove unused AssistanceRequest.regenerate
Summary: Finish what was started in https://phab.getgrist.com/D3970

Test Plan: existing tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3972
2023-07-26 15:51:46 +02:00
Jakub Serafin
f7fdfab6bf (core) GET endpoint for webhooks returns now data in format {webhooks:[...]}
Summary:
Rework of endpoint GET  for webhooks to make it coherent with other endpoints. Now data should be return in {webhooks:[{id:"...",fields:{"..."}]} format

```
{
    "webhooks": [
        {
            "id": ...
            "fields": {
                "url": ...
                "unsubscribeKey": ...
                "eventTypes": [
                    "add",
                    "update"
                ],
                "isReadyColumn": null,
                "tableId": "...",
                "enabled": false,
                "name": "...",
                "memo": "..."
            },
            "usage": {
                "status": "idle",
                "numWaiting": 0,
                "lastEventBatch": null
            }
        },
        {
            "id": "...",
            "fields": {
                "url": "...",
                "unsubscribeKey": "...",
                "eventTypes": [
                    "add",
                    "update"
                ],
                "isReadyColumn": null,
                "tableId": "...",
                "enabled": true,
                "name": "...",
                "memo": "..."
            },
            "usage": {
                "status": "error",
                "numWaiting": 0,
                "updatedTime": 1689076978098,
                "lastEventBatch": {
                    "status": "rejected",
                    "httpStatus": 404,
                    "errorMessage": "{\"success\":false,\"error\":{\"message\":\"Alias 5a9bf6a8-4865-403a-bec6-b4ko not found\",\"id\":null}}",
                    "size": 49,
                    "attempts": 5
                },
                "lastSuccessTime": null,
                "lastFailureTime": 1689076978097,
                "lastErrorMessage": "{\"success\":false,\"error\":{\"message\":\"Alias 5a9bf6a8-4865-403a-bec6-b4ko not found\",\"id\":null}}",
                "lastHttpStatus": 404
            }
        }
    ]
}
```

Test Plan: new test added to check if GET data fromat is correct. Other tests fixed to handle changed endpoint.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3966
2023-07-26 11:36:24 +02:00
Paul Fitzpatrick
cd8eac36b8 (core) updates from grist-core 2023-07-25 17:14:17 -04:00
Janet Vorobyeva
b2e98230b4
Linux-specific fonts: Liberation Sans (#572)
* Added linux-specific fonts

Currently, ubuntu has a lot of issues with fonts.
Both the regular font-family and the data-font-family end up
evaluating to NimbusSans on my system
  (ubuntu's fallback font from helvetica)

NimbusSans unfortunately is noticably too high, cause text in buttons,
emoji on the left pane, and other vertical text alignment to be too high

This diff explicitly says to use Liberation Sans which should
have no effect on windows/mac systems, but should significantly improve
appearance on ubuntu (and hopefully other linuxes)

Both of these fonts are some of the more widely supported linux fonts,
see:
https://www.webfx.com/blog/web-design/a-web-designers-guide-to-linux-fonts/
2023-07-25 16:39:47 -04:00
George Gevoian
5e26401a24 (core) Re-render assistant code blocks on theme change
Summary:
When the browser theme transitioned while the assistant was open, messages
containing code blocks were not being re-rendered with the new theme.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3965
2023-07-24 19:09:01 -04:00
Alex Hall
391c8ee087 (core) Allow assistant to evaluate current formula
Summary:
Replaces https://phab.getgrist.com/D3940, particularly to avoid doing potentially unwanted things automatically.

Adds optional fields `evaluateCurrentFormula?: boolean; rowId?: number` to `FormulaAssistanceContext` (part of `AssistanceRequest`). When `evaluateCurrentFormula` is `true`, calls a new function `evaluate_formula` in the sandbox which computes the existing formula in the column (regardless of anything the AI may have suggested) and uses that to generate an additional system message which is added before the user's message. In theory this could be used in an interface where users ask why a formula doesn't work, including possibly a formula suggested by the AI. For now, it's only used in `runCompletion_impl.ts` for experimenting.

Also cleaned up a bit, removing `_chatMode` which is always `true` now, and uses of `regenerate` which is always `false`.

Test Plan: Updated `runCompletion_impl` to optionally use the new feature, in which case it now scores 51/68 instead of 49/68.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3970
2023-07-24 21:59:00 +02:00
Paul Fitzpatrick
8ecf0d7d0a
change order of new private method to fix lint after saas/core sync (#587) 2023-07-23 14:51:19 -04:00
Paul Fitzpatrick
7256e0c245 (core) updates from grist-core 2023-07-23 11:53:20 -04:00
George Gevoian
90cf018133 (core) Update assistant link
Summary: Updates the "Learn more" link now that the page is published.

Test Plan: N/A

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3969
2023-07-21 13:43:31 -04:00
George Gevoian
beffd02c41 (core) Improve highlighting of previewed formula
Summary:
Also improves highlighting of columns when the "Click to insert"
tooltip is shown, and improves highlighting of transforming columns.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3962
2023-07-20 14:34:22 -04:00
Alex Hall
5a703a1972 (core) Send hash of user ID in OpenAI API requests
Summary: Following recommendation in https://platform.openai.com/docs/guides/safety-best-practices/end-user-ids

Test Plan: Checked that running server test shows log of hash of 'user id' (which is null because it's a fake session)

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz, georgegevoian

Differential Revision: https://phab.getgrist.com/D3958
2023-07-20 19:50:41 +02:00
George Gevoian
0a34292536 (core) Add telemetry for AI Assistant
Summary: Also fixes a few bugs with some telemetry events not being recorded.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3960
2023-07-20 12:50:26 -04:00
Florent
5e33b68753
Issue 359 support scaleway (#577)
* Fix support of Scaleway S3 bucket #359

While MinIO and AWS return versionId as strings, other S3 API
implementations return versionId as integers.

We must carefully convert the versionId as string in order to cover
these various behaviors.

Also ensure that docStorage is initialized before attempting to
calculate the data size in order to avoid an exception.

* Add unit tests for MinIOExternalStorage#versions() #359

Introduced some unit tests to :
 - ensure listObjects is called with the right arguments;
 - cover the case when a S3 bucket implementation does not return the
   versionId as a string but rather as an integer (like Scaleway):
   in such a case, ensure that the returned snapshotId is a string;
 - cover the case when the listObjects function emits an error, ensure the
   versions() call rejets with the error emitted;
 - that the deleteMarkers are only returned when the
   includeDeleteMarkers is passed;

---------

Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-07-20 06:16:58 -04:00
George Gevoian
0040716006 (core) Disable undo/redo in detached formula editor
Summary:
Undo and redo were causing errors to be thrown while the editor was detached. In the
interest of time, we'll disable undo/redo until we have a chance to look at whether
we can support it in the editor.

Test Plan: Manual.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Differential Revision: https://phab.getgrist.com/D3959
2023-07-19 11:39:14 -04:00
Dmitry S
534615dd50 (core) Update logging in sandbox code, and log tracebacks as single log messages.
Summary:
- Replace logger module by the standard module 'logging'.
- When a log message from the sandbox includes newlines (e.g. for tracebacks),
  keep those lines together in the Node log message.

  Previously each line was a different message, making it difficult to view
  tracebacks, particularly in prod where each line becomes a separate message
  object.

- Fix assorted lint errors.

Test Plan: Added a test for the log-line splitting and escaping logic.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3956
2023-07-18 11:21:25 -04:00
Alex Hall
7fd48364df (core) Improved error messages, retries, and handling of token limits in AI assistant
Summary:
In a nutshell:

- More specific and helpful error messages are shown to the user
- API requests are only retried when needed
- The system deals with reaching the maximum token limit better, especially by switching to a model with a bigger limit

In more detail:

- `COMPLETION_MODEL` configuration has been removed. By default `gpt-3.5-turbo-0613` is used which accepts 4k tokens. If that's not enough, `gpt-3.5-turbo-16k-0613` is used instead.
- Switching to the bigger model happens when either the prompt is too long by itself (the API immediately returns an error code) or the model reaches the 4k limit itself in the process of generating a response and thus returns an incomplete response. The latter case is made possible by removing the `max_tokens: 1500` in the request, which was very generous and would have lead to switching to the more expensive model more often than needed. The downside is that the user has to wait a bit longer for the response.
- If the bigger 16k token limit is also exceeded, the assistant immediately responds (instead of retrying as before) with an error message including suggestions. The suggestions include restarting the conversation if and only if the user has sent multiple messages.
- If a request fails because Grist has reached its OpenAI monthly billing quota, the assistant immediately responds (instead of retrying as before) with an error message suggesting that the user try again tomorrow.
- If a request fails for some other reason, the assistant retries, and if all attempts fail then the user is told to try again in a few minutes and is shown the exact error message, including the API response if there is one.
- Retrying only happens when an API request fails, whereas previously the system also retried errors from a much bigger scope which included calls to the sandbox. The downside is that the hugging face assistant no longer retries, although that code is currently disabled anyway.
- The assistant no longer waits an additional second after the final retry attempt fails.

Test Plan: Added a new server test file with several unit tests using faked OpenAI responses, including the happy path which wasn't really tested before.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3955
2023-07-18 16:01:37 +02:00
Jakub Serafin
d894b60fd4 (core) deleting queue from single webhook
Summary: Using standard tost notification, message about webhook queue being overflown was added. message is permanent as long as queue is full. Message contains linkt to the webhook setings

Test Plan: two nbrowser test was added - one to check if message is show when queue is full, and second to check if message is dismiss when queue was cleaned.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3929
2023-07-18 11:46:10 +02:00
Paul Fitzpatrick
450472f74c (core) updates from grist-core 2023-07-17 01:22:18 -04:00
CamilleLegeron
61bd064f73
i18n: userManager translation + some forgotten translations (#557)
* translation: add userManager translation + some forgotten translations
* use '\' caracter for multiple-line strings
2023-07-16 12:52:13 -04:00
Paul Fitzpatrick
a26eef05b0 (core) remove superfluous import
Summary:
Remove a duplicate import, perhaps introduced during merge of
a relatively long-lived branch.

Test Plan: existing tests should pass

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3953
2023-07-14 12:22:56 -04:00
Jakub Serafin
a9f4cfde90 (core) API reworked to use POST to create webhook and DELET to remove it
Summary:
introduces POST /api/docs/{docId}/webhooks and DELETE /api/docs/{docId}/webhooks/{webhookId} on place of old _subscribe and _unsubscribe endpoints.
Remove checking for unsubscribeKey while deleting webhook - only owner can delete webhook using DELETE endpoint. subscription key is still needed for _unsubscribe endpoint.
old _unsubscribe and _subscribe endpoints are still active and work as before - no changes there.

Posting schema:

```
POST /api/docs/[docId]/webhooks
```

Request Body:

```
{
    "webhooks": [
        {
            "fields": {
                "url": "https://webhook.site/3bd02246-f122-445e-ba7f-bf5ea5bb6eb1",
                "eventTypes": [
                    "add",
                    "update"
                ],
                "enabled": true,
                "name": "WebhookName",
                "memo": "just a text",
                "tableId": "Table1"
            }
        },
        {
            "fields": {
                "url": "https://webhook.site/3bd02246-f122-445e-ba7f-bf5ea5bb6eb2",
                "eventTypes": [
                    "add",
                ],
                "enabled": true,
                "name": "OtherWebhookName",
                "memo": "just a text",
                "tableId": "Table1"
            }
        }
    ]
}
```

Expected response: WebhookId for each webhook posted:

```
{
    "webhooks": [
        {
            "id": "85c77108-f1e1-4217-a50d-acd1c5996da2"
        },
        {
            "id": "d87a6402-cfd7-4822-878c-657308fcc8c3"
        }
    ]
}
```

Deleting webhooks:

```
DELETE api/docs/[docId]/webhooks/[webhookId]
```

there is no payload in DELETE request. Therefore only one webhook can be deleted at once

Response:

```
{
    "success": true
}
```

Test Plan: Old unit test improved to handle new endpoints, and one more added to check if endpoints are in fact created/removed

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: paulfitz, alexmojaki

Differential Revision: https://phab.getgrist.com/D3916
2023-07-14 15:01:46 +02:00
George Gevoian
ea8a59c5e9 (core) Implement AI Assistant UI V2
Summary:
Implements the latest design of the Formula AI Assistant.

Also switches out brace to the latest build of ace.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3949
2023-07-13 10:30:35 -04:00
Florent
7694588a42
External storage: split checkBackend and configure (follow-up #545) (#567)
This fixes an issue with external storage in saas environment.

See https://github.com/gristlabs/grist-core/pull/546/files#r1259340397

Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-07-13 02:44:46 -04:00
Dmitry S
8581492912 (core) When a page starts with a number, don't treat it as an emoji
Test Plan: Added a check to the emoji test.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3951
2023-07-12 15:13:25 -04:00
Alex Hall
152dc832f1
Split out new importFileAsNewTable method for grist-static (#564)
Also add column types to Limit entity to fix errors.
2023-07-12 15:57:02 +02:00
Paul Fitzpatrick
958ea096f3
fix a node-sqlite3-ism that breaks record removal in grist-static (#566)
Grist by default uses node-sqlite3 to manipulate data in an
SQLite database. If a single parameter is passed to `run`
and it is a list, the list is unpacked and its contents treated
as the actual parameters. In grist-static, we use other SQLite
interfaces that don't have that automatic unpacking. Most
calls like this have been removed from Grist, but at least one
was missed, and was causing symptoms such as
https://github.com/gristlabs/grist-static/issues/5

This change should make no difference to regular Grist, but
resolves the grist-static problems.
2023-07-11 05:52:06 -04:00
Paul Fitzpatrick
733b6b3d29 (core) add back config call for external storage
Summary:
add back config call for external storage
in the right order.

Test Plan: existing saas tests catch this

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3946
2023-07-10 14:16:48 -04:00
Paul Fitzpatrick
2c7ad727d1 (core) resolved some divergence in mergedServerMain 2023-07-10 07:50:52 -04:00
Jarosław Sadziński
d13b9b9019 (core) Billing for formula assistant
Summary:
Adding limits for AI calls and connecting those limits with a Stripe Account.

- New table in homedb called `limits`
- All calls to the AI are not routed through DocApi and measured.
- All products now contain a special key `assistantLimit`, with a default value 0
- Limit is reset every time the subscription has changed its period
- The billing page is updated with two new options that describe the AI plan
- There is a new popup that allows the user to upgrade to a higher plan
- Tiers are read directly from the Stripe product with a volume pricing model

Test Plan: Updated and added

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3907
2023-07-10 13:24:08 +02:00
Florent
b6b2d05be0
Abort when MinIO bucket does not have versioning enabled #545 (#546)
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-07-10 06:24:55 -04:00
Jarosław Sadziński
75d979abdb (core) Fixing cursor position for filtered linked section.
Summary:
In a selector table, when a selected row is filtered out of view, linked widgets should update based on the newly selected row.
There were a few bugs that contributed to this wrong behavior:
- Gridview wasn't subscribing to the current row id, and the row with id 'new' was being converted to the first row
- Cursor was keeping track of the currently selected row id, it was hiding a problem behind the proper rowIndex
- Undo/redo somehow leveraged the wrong rowId from the cursor during the position restore.

The `No data` text was also changed to be more meaningful.

Test Plan: Added and updated.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3937
2023-07-07 19:04:30 +02:00
George Gevoian
9a80f4bdf5 (core) Exclude additional pages from needsOrg
Summary:
A recent change broke the ability to open the account page if the current
org was inaccessible or invalid. Now the account page, and a few additional
pages, are excluded from needing an org to load.

Test Plan: Browser test.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3944
2023-07-06 11:46:30 -04:00
George Gevoian
271a5735c3 (core) Fix focus bug in filter menu
Summary:
It was impossible to focus the search input in the filter menu if either
of the range filter inputs were focused.

Test Plan: Browser test.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3942
2023-07-05 11:08:48 -04:00
Jarosław Sadziński
d5fe26f63c (core) Allowing selecting and copying texts from action log
Summary: Text in Document History > Activity tab is now selectable and one can copy it.

Test Plan: Manual

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3939
2023-07-05 17:00:16 +02:00
George Gevoian
35237a5835 (core) Add Support Grist page and nudge
Summary:
Adds a new Support Grist page (accessible only in grist-core), containing
options to opt in to telemetry and sponsor Grist Labs on GitHub.

A nudge is also shown in the doc menu, which can be collapsed or permanently
dismissed.

Test Plan: Browser and server tests.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz

Subscribers: jarek, dsagal

Differential Revision: https://phab.getgrist.com/D3926
2023-07-04 17:36:59 -04:00
Dmitry S
2b581ab7dc (core) Fix issue with lodash's map interpreting objects with length as array-like
Summary:
Here's a series of badness that easily leads to a crash, in reverse order:
- Lodash's map() function interprets an object with a .length property as an array.
- Some very old code generated human-friendly descriptions of user actions,
  applying map() to parts of them. It so happens that this generated description
  isn't even used.
- If a user action is encountered with a sufficiently large length propery,
  map() would exhaust the server memory.

Fixed by removing old unneeded code, and replacing some other occurrences of
lodash's map() with native equivalents.

Test Plan: Tested manually on a local reproduction of the issue.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3938
2023-07-01 21:07:39 -04:00
Dmitry S
b0aa17c932 (core) Detect when a page initial is an emoji, avoid repeating it, and style it better
Summary:
- Detecting emoji is surprisingly tricky; we use a fancy regex as a decent heuristic.
- Icons are a little larger than before.
- Styling tweaked for light and dark modes
- In case the OS doesn't render the emoji as one character, truncate what's
  shown in the icon box.

Test Plan: Added a test case.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3904
2023-06-30 16:57:35 -04:00
Dmitry S
4ea748b1a3 (core) Adjust the style of the 'attic' to have more of a toolbar feel, flush to the top, and taking up less space
Test Plan: Checked new looks manually, behavior should not be affected.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3934
2023-06-30 08:22:26 -04:00
Alex Hall
bb7cf6ba20 (core) Modify prompt so that model may say it cannot help with certain requests.
Summary:
This tweaks the prompting so that the user's message is given on its own instead of as a docstring within Python. This is so that the prompt makes sense when:

- the user asks a question such as "Can you write me a formula which does ...?" rather than describing their formula as a docstring would, or
- the user sends a message that doesn't ask for a formula at all (https://grist.slack.com/archives/C0234CPPXPA/p1687699944315069?thread_ts=1687698078.832209&cid=C0234CPPXPA)

Also added wording for the model to refuse when the user asks for something that the model cannot do.

Because the code (and maybe in some cases the model) for non-ChatGPT models relies on the prompt consisting entirely of Python code produced by the data engine (which no longer contains the user's message) those code paths have been disabled for now. Updating them now seems like undesirable drag, I think it'd be better to revisit this when iteration/experimentation has slowed down and stabilised.

Test Plan:
Added entries to the formula dataset where the response shouldn't contain a formula, indicated by the value `1` for the new column `no_formula`.

This is somewhat successful, as the model does refuse to help in some of the new test cases, but not all. Performance on existing entries also seems a bit worse, but it's hard to distinguish this from random noise. Hopefully this can be remedied in the future with more work, e.g. automatic followup messages containing example inputs and outputs.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3936
2023-06-27 15:57:56 +02:00
Paul Fitzpatrick
fc16b4c8f6 (core) updates from grist-core 2023-06-27 03:04:42 -04:00
Paul Fitzpatrick
7d3b4b49d5 (core) forward more kinds of edits to a virtual table
Summary:
Some edits to virtual tables (such as webhook lists) happen
via a route that was not yet handled. Actually Cyprien (the
original author) had handled this case but it got removed
because I didn't know what it was for :-). This brings back
support for edits by this route.

Test Plan: added a test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3924
2023-06-26 12:05:34 -04:00
George Gevoian
25b71c4e57 (core) Polish doc tutorials
Summary:
The GristDocTutorial table is now always visible to users with edit
access to the trunk, and the Share menu is now available within
tutorial forks, making it easier for editors to replace the original
tutorial trunk with changes made in the fork, and for viewers to export
their copy of the tutorial.

Also, changes to the GristDocTutorial table are now immediately reflected
in the tutorial popup.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3930
2023-06-23 23:56:20 -04:00
Dmitry S
3fa5125cf7 (core) Highlight rows used as a selector in linking, but do not show 'inactive' cursors.
Summary:
1. Introduces another highlight for link-selector rows, with the same color as
   regular selection, and allowing to overlap with regular selection.
2. Don't show "secondary" cursors (those in inactive sections), to keep a single
   cursor on the screen, since having multiple (which different in color) could
   cause confusion.
3. An unrelated improvement (prompted by a new fixture doc) is to default the
   active section to the top-left one (rather than the one with smallest rowId).
4. Another unrelated improvement (prompted by a test affected by the previous unrelated improvement) is to skip chart widgets when searching (previously search would step through those with an invisible "cursor").

Includes also tweaks for better testing on Arm-based Macs:
- Add support for TEST_CHROME_BINARY_PATH environment variable (helpful for a Mac arm64 architecture workaround)
- Remove unsetting of SELENIUM_REMOTE_URL when running headless (unlikely to affect anyone, and can be done outside the script, but interferes with the Mac workaround)

Test Plan: Added a new test case that cursor and linking-selector CSS classes are present or absent appropriately. Fixed test affected by the fix to default active section.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3891
2023-06-21 12:21:19 -04:00
Paul Fitzpatrick
7e50467396 (core) tweak handler for aborted connections to work on modern node
Summary:
It became hard to detect aborted connections in node 16.
In node 14, req.on('close', ...) did the job. Thid diff adds a
work-around, until a better way is discovered or added.
Aborting a req will typically lead to 'close' being called
on the response, without writableFinished being set.

 - https://github.com/nodejs/node/issues/38924
 - https://github.com/nodejs/node/issues/40775

Test Plan:
existing DocApiForwarder test passes; manually
checking on various node versions.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Differential Revision: https://phab.getgrist.com/D3923
2023-06-16 10:20:53 -04:00
Alex Hall
52469c5a7e (core) Improve parsing formula from completion
Summary:
The previous code for extracting a Python formula from the LLM completion involved some shaky string manipulation which this improves on.
Overall the 'test results' from `runCompletion` went from 37/47 to 45/47 for `gpt-3.5-turbo-0613`.

The biggest problem that motivated these changes was that it assumed that code was always inside a markdown code block
(i.e. triple backticks) and so if there was no block there was no code. But the completion often consists of *only* code
with no accompanying explanation or markdown. By parsing the completion in Python instead of JS,
we can easily check if the entire completion is valid Python syntax and accept it if it is.

I also noticed one failure resulting from the completion containing the full function (instead of just the body)
and necessary imports before that function instead of inside. The new parsing moves import inside.

Test Plan: Added a Python unit test

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3922
2023-06-16 13:38:20 +02:00
George Gevoian
0b64e408b0 (core) Fix scrolling and column title bugs
Summary:
On Firefox and Safari, setting scrollLeft to a max safe integer was
causing it to be treated as 0. It's not clear why - for now, the
scrollWidth is used instead.

Also fixes a bug where the column title popup wouldn't appear for a
new column if tab was previously used to close the same popup for
the last column.

Test Plan: Browser test.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3911
2023-06-15 14:51:10 -04:00
Dmitry S
41280a31f2 (core) Fix recording of signups, and record as telemetry too.
Summary:
Previously we failed to log signup info for users who signed up via
Google. This fixes that issue by recording it on first post-signup
visit. It also includes signup as a new telemetry event, recorded at the
same point.

Test Plan: Tested locally to see that a signup produces an appropriate log message and telemetry event.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3921
2023-06-15 00:15:18 -04:00
Dmitry S
2740884e3c (core) Improve the look and behavior of /welcome/teams page (also shown for /welcome/start)
Summary:
- Move css module for the login page css to core/, to be reusable in core/ pages.
- Move /welcome/teams implementation to WelcomeSitePicker.ts
- List users for personal sites, as well as team sites.
- Add org param to setSessionActive() API method and end endpoint, to allow
  switching the specified org to another user.
- Add a little safety to getOrgUrl() function.

Test Plan: Added a test case for the new behaviors of the /welcome/teams page.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3914
2023-06-13 20:40:59 -04:00
Florent
3d808f67f1
Visually identify that a field is a drop-down list (single choice) #491 (#525)
* Visually identify that a field is a drop-down list (single choice) #491

---------

Co-authored-by: Florent <florent.fayolle@beta.gouv.fr>
2023-06-13 13:14:01 -04:00
Dmitry S
812cded291 (core) Fix flaky FormulaEditor test by adjusting focus behavior of FormulaEditor.
Summary:
The issue is in the app due to the possibility of subtle differences in
order of events. It's hard to trigger a wrong order, but the fix is
intended to make it impossible.

Test Plan: Running test with a bunch of iterations, to see how reliable it is

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3918
2023-06-13 09:59:59 -04:00
Paul Fitzpatrick
c5e750abc6 (core) add a cli command to view telemetry settings
Summary:
This adds a `yarn cli settings telemetry [--json] [--all]` command
that allows telemetry settings to be inspected. It is useful for
keeping documentation about telemetry up to date.

Test Plan:
manual (a bit cheeky; justified on basis of breakage
not being very important yet, this is essentially an internal
feature)

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3917
2023-06-12 09:58:38 -04:00
George Gevoian
a460563daf (core) Polish telemetry code
Summary: Also fixes a few small bugs with telemetry collection.

Test Plan: Server and manual tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3915
2023-06-09 13:03:10 -04:00
Alex Hall
6ac0bc3bbb (core) Implement exported functions without relying on ActiveDoc.docData
Summary:
For grist-static, we want to the data engine to be able to call external/exported JS functions directly,
rather than via the node 'server' living in another thread which requires synchronous communication hackery.

As a step in that direction, this diff changes the exported functions that we care about (guessColInfo and convertFromColumn)
to just using the top-level functions instead of relying on fields in ActiveDoc, namely docData.

For guessColInfo, this is done by directly passing the small amount of metadata that was previously retrieved from the DocData.

For convertFromColumn, disentangling DocData is a lot more complicated, so instead we construct a fresh DocData object using
the required metadata tables which are now passed in by the data engine.

Test Plan: Existing tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3913
2023-06-07 22:30:01 +02:00
George Gevoian
10f5f0cb37 (core) Add optional telemetry to grist-core
Summary:
Adds support for optional telemetry to grist-core.

A new environment variable, GRIST_TELEMETRY_LEVEL, controls the level of telemetry collected.

Test Plan: Server and unit tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal, anaisconce

Differential Revision: https://phab.getgrist.com/D3880
2023-06-07 12:00:51 -04:00
George Gevoian
0d082c9cfc (core) Wrap buttons in filter bar
Summary:
Filter bar buttons used to overflow, which required horizontal scrolling. Buttons
now wrap instead.

Test Plan: Tested manually.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3901
2023-06-06 11:34:49 -04:00