Commit Graph

154 Commits

Author SHA1 Message Date
Florent
fde6c8142d
Support nonce and acr with OIDC + other improvements and tests (#883)
* Introduces new configuration variables for OIDC:
  - GRIST_OIDC_IDP_ENABLED_PROTECTIONS
  - GRIST_OIDC_IDP_ACR_VALUES
  - GRIST_OIDC_IDP_EXTRA_CLIENT_METADATA
* Implements all supported protections in oidc/Protections.ts
* Includes a better error page for failed OIDC logins
* Includes some other improvements, e.g. to logging, to OIDC
* Adds a large unit test for OIDCConfig
* Adds support for SERVER_NODE_OPTIONS for running tests
* Adds to documentation/develop.md info about GREP_TESTS, VERBOSE, and SERVER_NODE_OPTIONS.
2024-08-08 15:35:37 -04:00
Paul Fitzpatrick
fc3a7f580c
make access control for ConvertFromColumn action less brutal (#1111)
Access control for ConvertFromColumn in the presence of access rules had previously been left as a TODO. This change allows the action when the user has schema rights. Because schema rights let you create formulas, they let you read anything, so there is currently no value in nuance here.
2024-07-24 11:41:50 -04:00
Jordi Gutiérrez Hermoso
e30a090a4e config: remove all async/await around config read functions
Now that reading is synchronous, there's no need to have any more
async/await in regards to the those config functions.
2024-07-18 14:32:46 -04:00
Paul Fitzpatrick
db52bb0082 (core) move test/server/lib/GranularAccess.ts to core
Summary:
Move an important set of tests that were in our SaaS
repo for no good reason.

Test Plan: moving tests

Reviewers: jordigh

Reviewed By: jordigh

Differential Revision: https://phab.getgrist.com/D4300
2024-07-17 12:16:42 -04:00
Florent
39eb042ff1
Remove GRIST_SKIP_REDIS_CHECKSUM_MISMATCH (#1098)
Skipping the redis checksum mismatch is now generalized. A warning is
logged when we see a mismatch.
2024-07-10 14:28:20 -04:00
Spoffy
6908807236
Extracts config.json into its own module (#1061)
This adds a config file that's loaded very early on during startup. 

It enables us to save/load settings from within Grist's admin panel, that affect the startup of the FlexServer.

The config file loading:
- Is type-safe, 
- Validates the config file on startup
- Provides a path to upgrade to future versions.

It should be extensible from other versions of Grist (such as desktop), by overriding `getGlobalConfig` in stubs.

----

Some minor refactors needed to occur to make this possible. This includes:
- Extracting config loading into its own module (out of FlexServer).
- Cleaning up the `loadConfig` function in FlexServer into `loadLoginSystem` (which is what its main purpose was before).
2024-07-08 15:40:45 +01:00
Florent
786ba6b31e
Move HomeDBManager to gen-server/lib/homedb (#1076) 2024-07-05 16:02:39 +02:00
CamilleLegeron
0bfdaa9c02
Add authorization header in webhooks stored in secrets table (#941)
Summary:
Adding authorization header support for webhooks.

Issue:  https://github.com/gristlabs/grist-core/issues/827

---------

Co-authored-by: Florent <florent.git@zeteo.me>
2024-07-04 14:17:10 +02:00
Jarosław Sadziński
184be9387f (core) Enabling telemetry on /api/version endpoint
Summary:
Version API endpoint wasn't logging telemetry from POST requests. The issue was in registration
order, this endpoint was registered before `expressJson` and it couldn't read json body in the handler.

Test Plan: Added new test

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4277
2024-06-27 12:05:15 +02:00
Jarosław Sadziński
05214d8f9a (core) Port allocation fix in TestServer
Summary:
- Fixing port allocation in TestServer
- Extending logging in the Billing test
- Fixing negative rowIds support for add/remove actions
- Making FormulaEditor and CardView tests less flacky

Test Plan: Existing

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz, dsagal

Differential Revision: https://phab.getgrist.com/D4280
2024-06-24 22:10:58 +02:00
Jarosław Sadziński
76d94483ad (core) Adding fixSiteProducts that changes orgs from teamFree to Free product if it was set be default
Summary:
After release on 2024-06-12 (1.1.15) the GRIST_DEFAULT_PRODUCT env variable wasn't respected by the
method that started the server in single org mode. In all deployments (apart from saas), the default product
used for new sites is set to `Free`, but the code that starts the server enforced `teamFree` product.

This change adds a fix routine that fixes this issue by rewriting team sites from `teamFree` product to `Free`
product only if:
- The default product is set to `Free`
- The deployment type is something other then 'saas'.

Additionally there is a test that will fail after 2024.10.01, as this fix should be removed before this date.

Test Plan: Added test

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4272
2024-06-18 15:05:57 +02:00
George Gevoian
72066bf0e4 (core) Support user variable in dropdown conditions
Summary:
Dropdown conditions can now reference a `user` variable, similar to the
one available in Access Rules.

Test Plan: Browser test.

Reviewers: jarek, paulfitz

Reviewed By: jarek, paulfitz

Differential Revision: https://phab.getgrist.com/D4255
2024-06-04 06:56:55 -07:00
Paul Fitzpatrick
76a43129f1 (core) updates from grist-core 2024-05-23 13:27:59 -04:00
Jarosław Sadziński
a6ffa6096a (core) Adding UI for timing API
Summary:
Adding new buttons to control the `timing` API and a way to view the results
using virtual table features.

Test Plan: Added new

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4252
2024-05-22 14:56:53 +02:00
Jarosław Sadziński
60423edc17 (core) Customizable stripe plans.
Summary:
- Reading plans from Stripe, and allowing Stripe to define custom plans.
- Storing product features (aka limits) in Stripe, that override those in db.
- Adding hierarchical data in Stripe. All features are defined at Product level but can be overwritten on Price levels.
- New options for Support user to
-- Override product for team site (if he is added as a billing manager)
-- Override subscription and customer id for a team site
-- Attach an "offer", an custom plan configured in stripe that a team site can use
-- Enabling wire transfer for subscription by allowing subscription to be created without a payment method (which is customizable)

Test Plan: Updated and new.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4201
2024-05-19 09:09:19 +02:00
Florent
5e3cd94177
Introduce APP_HOME_INTERNAL_URL and fix duplicate docs (#915)
Context:

On self-hosted instances, some places in the code rely on the fact that we resolves public domains while being behind reverse proxies. This leads to cases where features are not available, such as the "Duplicate document" one.

Bugs that are solved - n self-hosted instances:

Impossible to open templates and tutorials right after having converted them;
Impossible to submit forms since version 1.1.13;
Impossible to restore a previous version of a document (snapshot);
Impossible to copy a document;

Solution:

Introduce the APP_HOME_INTERNAL_URL env variable, which is quite the same as APP_DOC_INTERNAL_URL except that it may point to any home worker;
Make /api/worker/:assignmentId([^/]+)/?* return not only the doc worker public url but also the internal one, and adapt the call points like fetchDocs;
Ensure that the home and doc worker internal urls are trusted by trustOrigin;

---------

Co-authored-by: jordigh <jordigh@octave.org>
2024-05-14 12:58:41 -04:00
Jarosław Sadziński
ecf242c6c6 (core) Adding latest version section to the admin panel.
Summary:
Update for the admin page to show the latest available version information.
- Latest version is read from docs.getgrist.com by default
- It sends basic information (installationId, deployment type, and version)
- Checks are done only on the page itself
- The actual request is routed through the API (to avoid CORS)

Test Plan: Added new test

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4238
2024-04-29 21:59:36 +02:00
Paul Fitzpatrick
a3442aee77 (core) updates from grist-core 2024-04-29 15:01:40 -04:00
fflorent
920eb09b3b Fix ManyFetches test so it checks TEST_REDIS_URL is set 2024-04-29 14:54:36 -04:00
fflorent
f405ae715b Bump dependencies versions 2024-04-29 14:54:36 -04:00
George Gevoian
3112433a58 (core) Add dropdown conditions
Summary:
Dropdown conditions let you specify a predicate formula that's used to filter
choices and references in their respective autocomplete dropdown menus.

Test Plan: Python and browser tests (WIP).

Reviewers: jarek, paulfitz

Reviewed By: jarek

Subscribers: dsagal, paulfitz

Differential Revision: https://phab.getgrist.com/D4235
2024-04-26 16:57:55 -04:00
Jarosław Sadziński
bd07e9c026 (core) New API to collect timing information from formula evaluation.
Summary:
- /timing/start endpoint to start collecting information
- /timing/stop endpoint to stop collecting
- /timing to retrive data gatherd so far

Timings are collected for all columns (including hidden/helpers/system)

Test Plan: Added new

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D4230
2024-04-24 11:07:11 +02:00
Paul Fitzpatrick
4567fad947 (core) updates from grist-core 2024-04-15 09:24:09 -04:00
George Gevoian
0130409447 (core) Fix null references in form fields bug
Summary:
Shares and documents would both produce a rule set for the same column
if the document rule set was for multiple columns. In this case, it was causing
one of the rules to be overwritten by the other (specifically, the rule granting
access to form references was not being applied in shares). The symptom was
`null` values in place of the referenced table's values.

We address this by splitting any rule sets for multiple columns that are also
affected by shares, so that they can be overridden by shares without causing a
conflicting rule set to be created (i.e. 2 column rule sets containing the same column).

Test Plan: Server tests.

Reviewers: dsagal, paulfitz

Reviewed By: dsagal, paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D4208
2024-04-12 15:46:21 -07:00
CamilleLegeron
76ef4d54f8
Webhook trigger update by column (#832)
Add functionality to filter webhooks based on a column or columns.
2024-04-12 16:04:37 -04:00
Jarosław Sadziński
bddbcddbef (core) Endpoint to report on the latest version of stable grist-core image
Summary:
New endpoint `/api/version` that returns latest version of stable docker image in format:
```
{"latestVersion":"1.1.12","
  updatedAt":"2024-03-06T06:28:25.752337Z","
  isCritical":false,
 "updateURL":"https://hub.docker.com/r/gristlabs/grist"
}
```

It connects to docker hub API and reads the version from the tag lists endpoint.
Stores telemetry passed from the client such us: current version, deployment type, installationId and others.

Test Plan: Added new test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4220
2024-04-09 17:03:49 +02:00
Florent
4a9b6fea9d
Shutdown Doc worker when it is not considered as available in Redis #831 (#856)
* Shutdown Doc worker when it is not considered as available in Redis
* Use isAffirmative for GRIST_MANAGED_WORKERS
* Upgrade Sinon for the tests
* Run Smoke test with pages in English
* Add logic in /status endpoint
2024-04-04 10:25:42 -04:00
Paul Fitzpatrick
4736ca490c (core) updates from grist-core 2024-04-01 09:22:25 -04:00
Jonathan Perret
96b652fb52
Support HTTP long polling as an alternative to WebSockets (#859)
The motivation for supporting an alternative to WebSockets is that while all browsers supported by Grist offer native WebSocket support, some networking environments do not allow WebSocket traffic.

Engine.IO is used as the underlying implementation of HTTP long polling. The Grist client will first attempt a regular WebSocket connection, using the same protocol and endpoints as before, but fall back to long polling using Engine.IO if the WebSocket connection fails.

Include these changes:
- CORS websocket requests are now rejected as a stronger security measure. This shouldn’t affect anything in practice; but previously it could be possible to make unauthenticated websocket requests from another origin.
- GRIST_HOST variable no longer affects CORS responses (also should not affect anything in practice, as it wasn't serving a useful purpose)
2024-03-28 13:22:20 -04:00
Dmitry S
e380fcfa90 (core) Admin Panel and InstallAdmin class to identify installation admins.
Summary:
- Add InstallAdmin class to identify users who can manage Grist installation.

  This is overridable by different Grist flavors (e.g. different in SaaS).
  It generalizes previous logic used to decide who can control Activation
  settings (e.g. enable telemetry).

- Implement a basic Admin Panel at /admin, and move items previously in the
  "Support Grist" page into the "Support Grist" section of the Admin Panel.

- Replace "Support Grist" menu items with "Admin Panel" and show only to admins.

- Add "Support Grist" links to Github sponsorship to user-account menu.

- Add "Support Grist" button to top-bar, which
  - for admins, replaces the previous "Contribute" button and reopens the "Support Grist / opt-in to telemetry" nudge (unchanged)
  - for everyone else, links to Github sponsorship
  - in either case, user can dismiss it.

Test Plan: Shuffled some test cases between Support Grist and the new Admin Panel, and added some new cases.

Reviewers: jarek, paulfitz

Reviewed By: jarek, paulfitz

Differential Revision: https://phab.getgrist.com/D4194
2024-03-25 12:18:38 -04:00
Jonathan Perret
b054810fd1
Remove the GRIST_ALLOWED_HOSTS environment variable (#899)
This mostly reverts commit 49b1749e98.
2024-03-15 09:40:56 -04:00
Florent
1e3e076820
Add option to skip Checksum verification between doc storage and Redis #751 (#767)
This check should be unnecessary for stores with strong consistency guarantees (virtually everywhere now).

---------

Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2024-03-07 14:11:17 -05:00
George Gevoian
94eec5e906 (core) Add AI Assistant retry with shorter prompt
Summary:
If the longer OpenAI model exceeds the OpenAPI context length, we now perform another retry with a
shorter variant of the formula prompt. The shorter prompt excludes non-referenced tables and lookup
method definitions, which should help reduce token usage in documents with larger schemas.

Test Plan: Server test.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Subscribers: JakubSerafin

Differential Revision: https://phab.getgrist.com/D4184
2024-02-12 11:06:52 -05:00
Florent
5533b9b7ee
Fix server crash when client passes malformed JSON (#826)
* Fix server crash when client passes malformed JSON

* Take remarks into account

---------

Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2024-01-23 12:07:39 -05:00
Paul Fitzpatrick
a311b8b3e5 (core) updates from grist-core 2024-01-08 09:19:50 -05:00
Paul Fitzpatrick
2a206dfcf8 (core) add initial support for special shares
Summary:
This gives a mechanism for controlling access control within a document that is distinct from (though implemented with the same machinery as) granular access rules.

It was hard to find a good way to insert this that didn't dissolve in a soup of complications, so here's what I went with:
 * When reading rules, if there are shares, extra rules are added.
 * If there are shares, all rules are made conditional on a "ShareRef" user property.
 * "ShareRef" is null when a doc is accessed in normal way, and the row id of a share when accessed via a share.

There's no UI for controlling shares (George is working on it for forms), but you can do it by editing a `_grist_Shares` table in a document. Suppose you make a fresh document with a single page/table/widget, then to create an empty share you can do:

```
gristDocPageModel.gristDoc.get().docData.sendAction(['AddRecord', '_grist_Shares', null, {linkId: 'xyz', options: '{"publish": true}'}])
```

If you look at the home db now there should be something in the `shares` table:

```
$ sqlite3 -table landing.db "select * from shares"
+----+------------------------+------------------------+--------------+---------+
| id |          key           |         doc_id         |   link_id    | options |
+----+------------------------+------------------------+--------------+---------+
| 1  | gSL4g38PsyautLHnjmXh2K | 4qYuace1xP2CTcPunFdtan | xyz | ...      |
+----+------------------------+------------------------+--------------+---------+
```

If you take the key from that (gSL4g38PsyautLHnjmXh2K in this case) and replace the document's urlId in its URL with `s.<key>` (in this case `s.gSL4g38PsyautLHnjmXh2K` then you can use the regular document landing page (it will be quite blank initially) or API endpoint via the share.

E.g. for me `http://localhost:8080/o/docs/s0gSL4g38PsyautLHnjmXh2K/share-inter-3` accesses the doc.

To actually share some material - useful commands:

```
gristDocPageModel.gristDoc.get().docData.getMetaTable('_grist_Views_section').getRecords()
gristDocPageModel.gristDoc.get().docData.sendAction(['UpdateRecord', '_grist_Views_section', 1, {shareOptions: '{"publish": true, "form": true}'}])
gristDocPageModel.gristDoc.get().docData.getMetaTable('_grist_Pages').getRecords()
gristDocPageModel.gristDoc.get().docData.sendAction(['UpdateRecord', '_grist_Pages', 1, {shareRef: 1}])
```

For a share to be effective, at least one page needs to have its shareRef set to the rowId of the share, and at least one widget on one of those pages needs to have its shareOptions set to {"publish": "true", "form": "true"} (meaning turn on sharing, and include form sharing), and the share itself needs {"publish": true} on its options.

I think special shares are kind of incompatible with public sharing, since by their nature (allowing access to all endpoints) they easily expose the docId, and changing that would be hard.

Test Plan: tests added

Reviewers: dsagal, georgegevoian

Reviewed By: dsagal, georgegevoian

Subscribers: jarek, dsagal

Differential Revision: https://phab.getgrist.com/D4144
2024-01-04 05:57:38 -05:00
Florent
837597cd55
Fix deadlock with webhooks on document load #799 (#812) 2024-01-03 21:47:53 +02:00
Florent
6722512d96
Completely ignored disabled webhooks (#800) 2024-01-03 20:06:38 +02:00
Paul Fitzpatrick
e0d44eff1f
set TEST_CLEAN_DATABASE when running server tests (#806)
After adding a batch of new server tests, some interactions between
tests have shown up via a shared database. This sets an existing flag
for dealing with this problem, that is used during browser tests but
hadn't been needed before for server tests.
2023-12-27 09:56:59 -05:00
Alex Hall
887717bb15 (core) Decode cell values to prevent working around rule using 'in' on lists
Summary:
Fixes bug described in https://grist.slack.com/archives/C069RUP71/p1699643458649019

Decodes cell values obtained from `InfoView.get` when evaluating user-defined ACL formulas, i.e. the result of `rec.foo` in such a formula. In particular this is so that `rec.some_list` loses the leading `L` type code and behaves sensibly in an expression like `thing in rec.some_list`.

`InfoView.get` is called in many places, but for every usage I found other than here, leaving the cell values encoded was best.

Test Plan: Added two unit server tests. The first is for the main bug involving lists. The second checks the only other plausible way I could think of that this change affects behaviour, and it seems to be for the better since both tests failed before. Most operations involving non-primitive cell values don't do anything sensible with or without decoding, so behaviour shouldn't change meaningfully in those cases.

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian, paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4123
2023-12-04 23:34:08 +02:00
Dmitry S
4d9bbf6263 (core) Exit more cleanly on unhandled errors, and handle errors writing to Clients.
Summary:
- Node has a strong recommendation to assume bad state and exit promptly on
  unhandled exceptions and rejections. We follow it, and only make an effort to
  clean up before exiting, and to log the error in a more standard way.

- The only case seen in recent month of an unhandled rejection was for
  attempting to write overly large JSON to a Client websocket. Ensure that's
  handled, and add a test case that artificially reproduces this scenario.

Test Plan:
Added a test case for failing write to Client, and a test case that unhandled
errors indeed kill the server but with an attempt at cleanup.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4124
2023-12-01 09:42:00 -05:00
Florent
cf0cbb404e
Allow URLs with only a docID #768 (#771)
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-11-29 15:13:29 -05:00
Paul Fitzpatrick
cea0404a22 (core) updates from grist-core 2023-11-20 11:28:50 -05:00
George Gevoian
caf830db08 (core) Record Cards
Summary:
Adds a new Record Card view section to each non-summary table, which can be from opened from various parts of the Grist UI to view and edit records in a popup card view.

Work is still ongoing, so the feature is locked away behind a flag; follow-up work is planned to finish up the implementation and add end-to-end tests.

Test Plan: Python and server tests. Browser tests will be included in a follow-up.

Reviewers: jarek, paulfitz

Reviewed By: jarek

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4114
2023-11-19 20:12:37 -05:00
Florent
7bc862fb02
Add header=colId option for the table-schema API #719 (#749) 2023-11-17 17:45:15 +02:00
Alex Hall
5197891427 (core) Remove transform columns on shutdown
Summary: Call a new user action `RemoveTransformColumns` in ActiveDoc shutdown.

Test Plan: Added nbrowser test

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4107
2023-11-14 22:31:34 +02:00
Dmitry S
3210eee24f (core) Revamp ForwardAuthLogin and unify with GRIST_PROXY_AUTH_HEADER
Summary:
By default, only respect GRIST_FORWARD_AUTH_HEADER on login endpoints; sessions are used elsewhere.

With GRIST_IGNORE_SESSION, do not use sessions, and respect GRIST_FORWARD_AUTH_HEADER on all endpoints.

GRIST_PROXY_AUTH_HEADER is now a synonym to GRIST_FORWARD_AUTH_HEADER.

Test Plan: Fixed tests. Tested first approach (no GRIST_IGNORE_SESSION) with grist-omnibus manually. Tested the second approach (with GRIST_IGNORE_SESSION) with a Apache-based setup enforcing http basic auth on all endpoints.

Reviewers: paulfitz, georgegevoian

Reviewed By: paulfitz, georgegevoian

Differential Revision: https://phab.getgrist.com/D4104
2023-11-07 16:30:49 -05:00
Paul Fitzpatrick
8053c81d02 (core) updates from grist-core 2023-11-06 08:20:57 -05:00
Florent
10822d3b86
getHostType: consider APP_DOC_INTERNAL_URL as native (#715)
The getHostType() now returns "native" when the host corresponds to the value of APP_DOC_INTERNAL_URL. T
While trying to scale, with a different internal and public URL for doc workers, and having configured the org to be specified in the path (GRIST_ORG_IN_PATH=true), the APP_DOC_INTERNAL_URL parameter was not treated as internal which made the connection between home server and doc workers impossible.

---------
https://github.com/gristlabs/grist-core/pull/715
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-11-06 09:24:59 +01:00
George Gevoian
7a85aaa7a1 (core) Add new telemetry events
Summary: Adds a handful of new telemetry events, and makes a few tweaks to allow for better organization of telemetry.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D4100
2023-11-01 10:49:33 -04:00