Commit Graph

544 Commits

Author SHA1 Message Date
Dmitry S
2eec48b685 (core) Support adjusting OOM score for child sandbox processes.
Summary:
Also update error handling in NSandbox initialization to avoid node
exiting when sandbox can't be created.

Info on oom_score and choom: https://man7.org/linux/man-pages/man1/choom.1.html

Empirically, in docker and under fargate, oom_score (visible in
/proc/PID/oom_score) seems to be calculated approximately as

    ((ProcessRSS / TotalMemory * 1000 + 999 + oom_score_adj) * 2/3)

though this doesn't correspond to any documentation I could find.

In addition, under docker / fargate it does not work to set oom_score_adj (with choom or via /proc/PID/oom_score_adj) to a negative value to give priority to a node process. That's why this diff adjust the score up for sandbox processes instead.

Test Plan:
Checked that grist-omnibus built with this change respects the
variable, and sandbox processes end up with higher oom_score values.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4112
2023-11-16 17:43:06 -05:00
George Gevoian
c9bba5207e (core) Add more telemetry events
Summary: Adds new telemetry events and a flag for whether an event originated from a team site.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: jarek, dsagal

Differential Revision: https://phab.getgrist.com/D4105
2023-11-16 12:38:08 -05:00
Paul Fitzpatrick
12317097f7 (core) move some untagged assets served on plugin port
Summary: Some untagged assets on the plugin port could be a problem if that port is merged with the regular Grist app port, so we nest them within a non-conflicting path (/plugins/assets).

Test Plan: see if a test fails anywhere

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4116
2023-11-14 15:58:00 -05:00
Alex Hall
5197891427 (core) Remove transform columns on shutdown
Summary: Call a new user action `RemoveTransformColumns` in ActiveDoc shutdown.

Test Plan: Added nbrowser test

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4107
2023-11-14 22:31:34 +02:00
Alex Hall
3dfe4be5f3 (core) Remove DB transaction from webhook update, add mutex to all webhook endpoints
Summary:
This removes problematic code that was holding a HomeDB transaction while applying user actions which could hang indefinitely, especially if the webhook queue is full as in https://grist.slack.com/archives/C05DBJ6LA1F/p1698159750945949.

The discussion about adding this code is here: https://phab.getgrist.com/D3821#inline-45054

The initial motivation was to roll back HomeDB changes if something went wrong while applying user actions, to avoid saving only part of the changes the user requested. I think it's actually fine to just allow such a partial save to happen - I don't see anything particularly undesirable about keeping an update to the webhook URL if other updates requested by the user didn't also get applied, as the fields don't affect each other.

The comment approving the transaction approach said "so we shouldn't end up leave the transaction hanging around too long" which has been falsified.

It looks like there was also some desire to prevent a mess caused by multiple simultaneous calls to this endpoint, which the transaction may have helped with a little, but didn't really seem like a solution. Comments in `Triggers.ts` also mention fears of race conditions when clearing (some of) the queue and the need for some locking. So I wrapped all webhook-related endpoints in a simple `Mutex` held by the `ActiveDoc` to prevent simultaneous changes. I *think* this is a good thing. These endpoints shouldn't be called frequently enough to create a performance issue, and this shouldn't affect actually sending webhook events when records are added/updated. And it does seem like interleaving calls to these endpoints could cause very weird problems.

Test Plan: Nothing yet, I'd like to hear if others think this is sensible.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D4111
2023-11-14 16:51:21 +02:00
Paul Fitzpatrick
68801474b1 (core) updates from grist-core 2023-11-13 07:58:49 -05:00
Florent
a4998b4b21
Add native OIDC support in Grist #707 (#717) 2023-11-09 14:56:39 -05:00
wunter8
73ab1de6e4
Remove attachments and system files from document download as template (#729) 2023-11-08 15:09:01 -05:00
Jarosław Sadziński
3c219e05f6 (core) Removing the new menu flag
Summary: Enabling the `GRIST_NEW_COLUMN_MENU` flag by default and removing it.

Test Plan: Existing

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4098
2023-11-08 07:37:54 +01:00
Dmitry S
3210eee24f (core) Revamp ForwardAuthLogin and unify with GRIST_PROXY_AUTH_HEADER
Summary:
By default, only respect GRIST_FORWARD_AUTH_HEADER on login endpoints; sessions are used elsewhere.

With GRIST_IGNORE_SESSION, do not use sessions, and respect GRIST_FORWARD_AUTH_HEADER on all endpoints.

GRIST_PROXY_AUTH_HEADER is now a synonym to GRIST_FORWARD_AUTH_HEADER.

Test Plan: Fixed tests. Tested first approach (no GRIST_IGNORE_SESSION) with grist-omnibus manually. Tested the second approach (with GRIST_IGNORE_SESSION) with a Apache-based setup enforcing http basic auth on all endpoints.

Reviewers: paulfitz, georgegevoian

Reviewed By: paulfitz, georgegevoian

Differential Revision: https://phab.getgrist.com/D4104
2023-11-07 16:30:49 -05:00
Alex Hall
b7e9d2705e (core) When a webhook is disabled, clear its queue
Summary: Also fixes a few bugs found along the way, particularly that webhook payloads could contain stale data.

Test Plan: Added an nbrowser test, made existing test a bit more detailed.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4102
2023-11-07 15:48:35 +02:00
Paul Fitzpatrick
8053c81d02 (core) updates from grist-core 2023-11-06 08:20:57 -05:00
George Gevoian
7a85aaa7a1 (core) Add new telemetry events
Summary: Adds a handful of new telemetry events, and makes a few tweaks to allow for better organization of telemetry.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D4100
2023-11-01 10:49:33 -04:00
Paul Fitzpatrick
4fb1df567b (core) change handling of server access prior to full configuration
Summary:
Recently, the server became more strict about not responding to
requests before being fully configured. This is a problem when
a doc worker is trying to check whether it has become available
from a load balancer.

This change gives access to health checks prior to configuration
being complete. Otherwise, app endpoints accessed before full
configuration return a 503.

A flag is added to /status to allow checking explicitly for
the server being ready and configured.

Test Plan: manual

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D4103
2023-10-31 17:42:48 -04:00
Paul Fitzpatrick
07bb90b5a6
allow bundled widgets to be hidden from dropdown, and nested (#714)
This makes a few refinements to bundling widgets:

  * A widget with `published: false` is not shown in the
    custom widget dropdown in the UI. This is so widgets
    can be bundled with the app for "native" use (like the
    calendar widget) without immediately resulting in an
    extra listing in the UI. (There are improvements we'd
    like to make to the UI to better communicate widget
    provenance and quality eventually, which would be a
    helpful alternative to just a binary flag.)

  * A relative path to the custom widget manifest is
    respected. This will make the bundling process marginally
    neater.
2023-10-30 21:13:21 -04:00
Paul Fitzpatrick
6059bdcf66
rename variable to avoid shadowing another (#712)
This fixes a small lint issue in some widget bundling code.
2023-10-30 09:52:42 -04:00
Paul Fitzpatrick
cc9a9ae8c5 (core) support for bundling custom widgets with the Grist app
Summary:
This adds support for bundling custom widgets with the Grist app, as follows:

 * Adds a new `widgets` component to plugins mechanism.
 * When a set of widgets is provided in a plugin, the html/js/css assets for those widgets are served on the existing untrusted user content port.
 * Any bundled `grist-plugin-api.js` will be served with the Grist app's own version of that file. It is important that bundled widgets not refer to https://docs.getgrist.com for the plugin js, since they must be capable of working offline.
 * The logic for configuring that port is updated a bit.
 * I removed the CustomAttachedView class in favor of applying settings of bundled custom widgets more directly, without modification on view.

Any Grist installation via docker will need an extra step now, since there is an extra port that needs exposing for full functionality. I did add a `GRIST_TRUST_PLUGINS` option for anyone who really doesn't want to do this, and would prefer to trust the plugins and have them served on the same port.

Actually making use of bundling will be another step. It'll be important to mesh it with our SaaS's use of APP_STATIC_URL for serving most static assets.

Design sketch: https://grist.quip.com/bJlWACWzr2R9/Bundled-custom-widgets

Test Plan: added a test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4069
2023-10-27 17:00:10 -04:00
Jarosław Sadziński
69d5ee53a8 (core) Treating API urls as external in cells
Summary:
Links for the API endpoints in a cell didn't work as they were interpreted as
internal routes. Now they are properly detected as external.

Test Plan: Added new test

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D4078
2023-10-24 08:55:08 +02:00
George Gevoian
74485f412d (core) Fix delete user button for Google-only accounts
Summary:
An unhandled error was being thrown by CognitoClient when a user was unable
to be found during account deletion. Google-only accounts are no longer
associated with a user in Cognito, so the error was actually benign. A warning is
now logged instead.

Test Plan: Manual.

Reviewers: paulfitz, jarek

Reviewed By: paulfitz, jarek

Differential Revision: https://phab.getgrist.com/D4073
2023-10-18 10:52:58 -04:00
Paul Fitzpatrick
f32563e8fb (core) updates from grist-core 2023-10-17 06:47:46 +02:00
Florent
eb55afcbc4
Option to export colId as header in CSV / XSLX instead of label (#688) (#692) 2023-10-15 20:17:43 -04:00
Jakub Serafin
2521db4c55 (core) New Columns Menu
Summary:
A menu to be shown when new colum button is added. It's give access to various diffrent shortcuts, like adding new column, unhiding existing ones, fast adding lookup columns or trigger one (authoriship or timestamp). Design document can be found here: https://grist.quip.com/CTgxAQv9Ghjt/Add-Columns-more-easily
To turn on this menu flag GRIST_NEW_COLUMN_MENU to 1

Test Plan: UI tests suite under nbrowser/GridViewNewColumnMenu.ts

Reviewers: jarek, georgegevoian

Reviewed By: georgegevoian

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4074
2023-10-13 22:35:36 +02:00
CamilleLegeron
f66ecbd6df
feat: allow using the existing numeric table IDs in the API (#690) 2023-10-12 19:32:22 +02:00
George Gevoian
0cadb93d25 (core) Update dependencies
Summary:
Changes the minimum version of Node to 18, and updates the Docker images and GitHub workflows to build Grist with Node 18.

Also updates various dependencies and scripts to support building running tests with arm64 builds of Node.

Test Plan: Existing tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3968
2023-10-11 17:36:58 -04:00
Jarosław Sadziński
34f366585d (core) Removing org check in the closeOrg endpoint.
Summary:
Fixing a bug: account couldn't be closed when there was
a team site shared with everyone. Endpoint was checking
if there are any other team sites available, but this is already
tested in "Doom" in a better way, so this check was removed.

Test Plan: New test

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4067
2023-10-06 15:35:42 +02:00
Dmitry S
fbae81648c (core) Add options to /status health-check endpoints to check DB and Redis liveness.
Summary:
- /status accepts new optional query parameters: db=1, redis=1, and timeout=<ms> (defaults to 10_000).
- These verify that the server can make trivial calls to DB/Redis, and that they return within the timeout.
- New HealthCheck tests simulates DB and Redis problems.
- Added resilience to Redis reconnects (helped by a test case that simulates disconnects)
- When closing Redis-based session store, disconnect from Redis (to avoid hanging tests)

Some associated test reorg:
- Move stripeTools out of test/nbrowser, and remove an unnecessary dependency,
  to avoid starting up browser for gen-server tests.
- Move TcpForwarder to its own file, to use in the new test.

Test Plan: Added a new HealthCheck test that simulates DB and Redis problems.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4054
2023-10-02 14:41:04 -04:00
Jarosław Sadziński
cce185956c (core) Delete my account button
Summary:
Adding new "Delete my account" button to the profile page that allows users to remove completely
their accounts as long as they don't own any team site.

Test Plan: Added

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian, paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4037
2023-09-27 14:49:23 +02:00
Dmitry S
2705d41c34 (core) Add timeouts to prevent ActiveDoc bad state during shutdown.
Summary:
Add two shutdown-related timeouts.

1. One is to limit the duration of any work that happens once shutdown
   begins. In particular, waiting for an update to current time could block
   indefinitely if the data engine is unresponsive. Such awaits are now
   limited to 5 seconds.

2. The other is to allow documents to get shutdown for inactivity even when
   some work takes forever. Certain work (e.g. applying user actions)
   generally prevents a document from shutting down while it's pending. This
   prevention is now limited to 5 minutes.

   Shutting down a doc while something is pending may break some
   assumptions, and lead to errors. The timeout is long to let us assume
   that the work is stuck, and that errors are better than waiting forever.

Other changes:
- Periodic ActiveDoc work (intervals) is now started when a doc finishes
  loading rather than in the constructor. The difference only showed up in
  tests which makes the intervals much shorter.

- Move timeoutReached() utility function to gutil, and use it for
  isLongerThan(), since they are basically identical. Also makes sure that the
  timer in these is cleared in all cases.

- Remove duplicate waitForIt implementation (previously had a copy in both
  test/server and core/test/server).

- Change testUtil.captureLog to pass messages to its callback, to allow asserts
  on messages within the callback.

Test Plan:
Added new unittests for the new shutdowns, including a replication
of a bad state that was possible during shutdown.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4040
2023-09-26 15:32:49 -04:00
George Gevoian
40c5f7b738 (core) Add documentCreated telemetry event
Summary:
The event is triggered whenever a document is created, imported, or
duplicated.

Test Plan: Tested manually.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4035
2023-09-13 01:13:00 -04:00
George Gevoian
76e822eb23 (core) Add welcomeQuestionsSubmitted telemetry event
Summary:
The new event captures responses to the welcome questionnaire.

Responses are also still sent to the special Grist document configured with
the DOC_ID_NEW_USER_INFO variable.

Test Plan: Tested manually.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4034
2023-09-13 00:31:04 -04:00
Paul Fitzpatrick
18f7e255df (core) updates from grist-core 2023-09-11 10:00:39 -04:00
Paul Fitzpatrick
585cf02f6c
add hooks for tweaking how downloads happen (for grist-static) (#665)
This makes three main changes:
  * Adds a hook to transform download links.
  * Adds a hook to add an externally created ActiveDoc to a DocManager.
  * Rejiggers XLSX export code so it can be used without streaming,
    which is currently tricky in a browser. Regular usage with node
    continues to use streaming.

With these changes, I have a POC in hand that updates grist-static
to support downloading CSVs, XLSXs, and .grist files.
2023-09-09 14:50:32 -04:00
Florent
5ff79703b4
Introduce GRIST_ANON_PLAYGROUND variable #642 (#651)
* `GRIST_ANON_PLAYGROUND`: When set to 'false' deny anonymous users access to the home page
 * `GRIST_FORCE_LOGIN`: Much like `GRIST_ANON_PLAYGROUND` but don't support anonymous access at all (features like sharing docs publicly requires authentication)

---------

Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-09-08 09:05:52 -04:00
Jarosław Sadziński
a0fc11c8d1 (core) Fixing memos in record dependent rules.
Summary:
Memos in row dependent rules were ignored if the action was denied unconditionally. It prevented creating different memos for different users.
Now ACL is forced to check each row, to find the correct memo to show when create/update/delete action is denied.

Test Plan: Existing and new.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz, dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D4024
2023-09-08 10:16:09 +02:00
George Gevoian
3dadf93c98 (core) Add support for auto-copying docs on signup
Summary:
The new "copyDoc" query parameter on the login page sets a short-lived cookie, which is
then read when welcoming a new user to copy that document to their Home workspace, and
redirect to it. Currently, only templates and bare forks set this parameter.

A new API endpoint for copying a document to a workspace was also added.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3992
2023-09-06 15:12:08 -04:00
George Gevoian
90fb4434cc
Add ws id and doc name params to POST /docs (#655) 2023-09-05 14:27:35 -04:00
Paul Fitzpatrick
6dab12f301 (core) updates from grist-core 2023-09-05 11:12:15 -04:00
Paul Fitzpatrick
bfd0fa8c7f
add an endpoint for doing SQL selects (#641)
* add an endpoint for doing SQL selects

This adds an endpoint for doing SQL selects directly on a Grist document. Other kinds of statements are not supported. There is a default timeout of a second on queries.

This follows loosely an API design by Alex Hall.

Co-authored-by: jarek <jaroslaw.sadzinski@gmail.com>
2023-09-04 09:21:18 -04:00
George Gevoian
70feb336d9 (core) Add AI assistant usage banners
Summary:
Banners are now shown when there are low or no AI assistant
credits remaining.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4018
2023-08-30 16:00:04 -04:00
Jakub Serafin
942fc96225 (core) custom widget appear as build-in widget
Summary: Added boilerplate code needed to create new wigets in "Add new" menu, that are wrapped around existing custom widgets. More details can be found here: https://grist.quip.com/larhAGRKyl6Z/Custom-widgets-in-Add-Widget-menu

Test Plan: nbowser tests added to verify if item in menu exits, if widget is rendered, and right side menu has widget selection and read access selection hided.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3994
2023-08-30 09:44:25 +02:00
Jarosław Sadziński
b6a431dd58 (core) Cursor in custom widgets
Summary:
Adding a new method `setCursorPos` in the widget API, and a new configuration option for the ready message `allowSelectBy` that exposes custom widgets in the `Select by` dropdown.
With this, a custom widget can control the position of the linked widgets and is able to change the column in the creator panel.

Test Plan: Added new test. Existing tests should pass.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Subscribers: JakubSerafin

Differential Revision: https://phab.getgrist.com/D3993
2023-08-29 09:19:52 +02:00
Paul Fitzpatrick
7aebdd15f6
tweak XLSX export worker to make Piscina happy under Electon (#646)
Adds a dummy default export to the worker exporter script used
for producing XLSX. This method exists only to make Piscina
happier. With it, Piscina will load this file using a regular
require(), which under Electron will deal fine with Electron's
ASAR app bundle. Without it, Piscina will try fancier methods
that aren't at the time of writing correctly patched to
deal with an ASAR app bundle, and so report that this
file doesn't exist instead of exporting an XLSX file.

I tried various other solutions such as upgrading Electron,
unpacking various files, patching Piscina, and this was
overall the simplest.

See https://github.com/gristlabs/grist-electron/issues/9
2023-08-28 08:11:01 -04:00
Florent
ee31764b83
DocApi: Implement DELETE on columns (#601) (#640)
* Factorize generateDocAndUrl
* Add describe for regrouping /records
2023-08-24 14:33:53 +02:00
Florent
9dfebefc9b
Dump the rule for ACL formula warnings (#639)
When an ACL formula fails to be run, a warning is printed. However, it is painful to know which formula is concerned by the warning.

ACL: if RHS is null, return false for "in" and "not in"
2023-08-23 09:23:29 -04:00
Paul Fitzpatrick
b3a71374d1 (core) updates from grist-core 2023-08-21 09:15:53 -04:00
Paul Fitzpatrick
0be858c19d
allow AI Assistance to run against any chat-completion-style endpoint (#630)
This adds an ASSISTANT_CHAT_COMPLETION_ENDPOINT which can be used
to enable AI Assistance instead of an OpenAI API key. The assistant
then works against compatible endpoints, in the mechanical sense.
Quality of course will depend on the model. I found some tweaks
to the prompt that work well both for Llama-2 and for OpenAI's models,
but I'm not including them here because they would conflict with some
prompt changes that are already in the works.

Co-authored-by: Alex Hall <alex.mojaki@gmail.com>
2023-08-18 16:14:42 -04:00
Alex Hall
166312be3a (core) Add dialog with options to allow downloading without history or data
Summary:
{F74398}

Refactored the 'radio checkboxes' in the modal for deleting a page and reused them here.

The option to download as a template already existed in the server code but wasn't being exercised by the frontend. Also added an option to remove just the history, which is the main motivation for this diff.

Test Plan: Expanded the existing nbrowser test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3999
2023-08-18 15:38:24 +02:00
Alex Hall
a1d31e41ad (core) Prevent the AI assistant response from including class definitions
Summary:
Sometimes the model repeats the classes given in the prompt which would mess up extracting the actual formula. This diff solves this by:

1. Changes the generated Python schema so that (a) the thing that needs completing is a plain top level function instead of a property/method inside the class and (2) the classes are fully valid syntax, which makes it easier to
2. Remove classes from the parsed Python code when converting the completion to a formula.
3. Tweak the prompt wording to discourage including classes in general, especially because sometimes the model tries to solve the problem by defining extra methods/attributes/classes.

While I was at it, I changed type hints to use builtins (e.g. `list` instead of `List`) to prevent `from typing import List` which was happening sometimes and would look weird in a formula. Similarly I removed `@dataclass` since that also implies an import, and this also fits with the tweaked wording that the classes are fake.

Test Plan:
Added a new test case to the formula dataset which triggers the unwanted behaviour. The factors that seem to trigger the problem are (1) a small schema so the classes are easier to repeat and (2) the need to import modules, which the model wants to place before all other code. The case failed before this diff and succeeded after. The tweaked wording reduces the chances of repeating the classes but didn't eliminate it, so forcibly removing the classes in Python was needed.

There were also a couple of other existing cases where repeating the classes was observed before but not after.

Overall the score increased from 49 to 51 out of 69 (including the new case). At one point the score was 53, but changes in whitespace were enough to make it drop again.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4000
2023-08-18 12:50:09 +02:00
Florent
fde56ffa33
DocApi: Introduce hidden option for GET on records (#623)
Also test that PUT /columns?replaceall=1 does not remove hidden columns
2023-08-14 16:17:46 +02:00
Florent
e93ad5a0c5
Issue #601: implement PUT on columns (#605)
* Also Add includeHidden param for GET on columns
2023-08-11 15:12:43 +02:00