Commit Graph

595 Commits

Author SHA1 Message Date
Paul Fitzpatrick
d008a32eb3 (core) updates from grist-core 2024-02-12 08:15:38 -05:00
Dmitry S
7de33078f3 (core) Support GRIST_INCLUDE_CUSTOM_SCRIPT_URL for custom scripts, specifically for rapid prototyping support
Test Plan: Tested manually on localhost

Reviewers: paulfitz, georgegevoian

Reviewed By: paulfitz, georgegevoian

Subscribers: paulfitz, georgegevoian

Differential Revision: https://phab.getgrist.com/D4186
2024-02-08 08:56:17 -05:00
CamilleLegeron
a0a968a2d8
Make free coaching call url configurable and popup translatable (#823) 2024-02-07 11:53:48 -05:00
Paul Fitzpatrick
23456ca43d
adapt to slight differences in linting (#842)
grist-saas and grist-core builds have slight differences in the depth of type information inferred.
2024-02-05 09:42:57 -05:00
George Gevoian
cb298e63d4 (core) Fix bug forcing login on some form URLs
Summary:
Login (and other) middleware was included in the public form URL by mistake,
forcing logins on forms hosted on non-personal sites.

Test Plan: Browser test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D4181
2024-02-01 12:10:52 -05:00
Dmitry S
11afc08f65 (core) Improve API Console and link from Document Settings.
Summary:
Changes to building and serving:
- Remove unpkg dependencies, add npm module for swagger-ui-dist instead.
- Move apiconsole JS logic into core/app/client/apiconsole.ts, and use TypeScript.
- Add symlinks to swagger in static/ and core/static/.
- Refactor loadScript, and add loadCssFile; use these to load swagger-ui resources.

Changes to console itself:
- Support docId, workspaceId, orgId URL parameters. When present, the matching
  value in dropdowns is moved to the front and marked as "(Current)".
- Fix the ordering of example values, particularly for workspaces.
- Remove unwanted example values.
- Hide confusing "Authorize" button.
- Hide API keys, and rely consistently on cookies for executing API calls.

Integration into Grist:
- Added a button to Document Settings, just under document ID in "API".
- The button opens a separate page, passing in org, workspace, and doc info for the current doc.

Test Plan: Only tested manually, no automated tests yet.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4173
2024-01-29 10:08:19 -05:00
Paul Fitzpatrick
fb276bade7 (core) updates from grist-core 2024-01-29 09:16:55 -05:00
Paul Fitzpatrick
457b9714d1 (core) change ActiveDoc load and Sandbox call log messages to be aggregation-friendly
Summary:
Takes some numbers embedded in strings and moves them to the context, so
they can be aggregated and trends inspected more easily.

Test Plan: tested manually by looking at logs

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D4175
2024-01-24 14:06:38 -05:00
Jarosław Sadziński
372d86618f (core) Required fields in forms and bug fixes
Summary:
- Adding little green asterisk at the end of field title.
- Fixing bug on columns component. Adding paragraph as a column and then selecting it was throwing error in the RightPanel
- Fixing boolean column bug in the editor
- Adding (--Choose--) placeholder for dropdowns
- Fixing columns logic: Dragging and dropping columns can unexpectedly add more columns.
- Added favicon and default page title
- Added svg to sync file for electron.

Test Plan: Updated

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D4172
2024-01-24 18:16:48 +01:00
George Gevoian
6cb8614017 (core) Polish forms
Summary:
  - Updates styling of form submitted page.
  - Tweaks styling of checkboxes, labels, and questions on form page.
  - Adds new form 404 page.
  - Adds checkbox to not show warning again when publishing or un-publishing a form.
  - Excludes formula, hidden, and attachment columns in submitted form data.
  - Adds placeholder text to form configuration inputs.
  - Improves dark mode styling in Form widget.
  - Updates default title and description of new forms.
  - Updates styling of Form widget buttons.
  - Fixes form success text input handling.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4170
2024-01-24 02:18:03 -08:00
Jarosław Sadziński
95c0441d84 (core) Form kanban tasks
Summary:
- Open all links in a new tab
- Excluding not filled columns (to fix trigger formulas)
- Fixed Ref/RefList submission
- Removing redundant type definitions for Box
- Adding header menu item
- Default empty values in select control

Test Plan: Updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4166
2024-01-23 22:11:48 +01:00
Florent
5533b9b7ee
Fix server crash when client passes malformed JSON (#826)
* Fix server crash when client passes malformed JSON

* Take remarks into account

---------

Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2024-01-23 12:07:39 -05:00
Paul Fitzpatrick
007c4492dc (core) notify home db of shares when copying/forking/uploading docs
Summary:
The first time a worker opens a document, it will now check if it has any shares the home db needs to be aware of. If so, they will be added. This is important for documents uploaded/copied/forked/replaced, so that their shares work out of the box.

In future, may want some UI to give user control of whether shares are activated after upload/copy/fork/replace.

It seems tricky currently to know if a document is being opened for the first time. As a proxy, I check whether usage information has been calculated and saved to the db, since I can determine that without adding another db query. It is safe to synchronize shares more than necessary.

This leaves two gaps:
 * If a document is created/uploaded/copied/forked/replaced and no attempt is made to access it prior to using a share, then that share won't actually be available. Not a problem currently I think, since how would a user have determined the share key. But in future it would be good to also do a sync after creation/upload/copy/fork/replacement/...
 * On document replacement, usage info is reset but not absolutely immediately. So in principle shares could fail to be created on first load of the replacement. Usage info reset could be tweaked to give a guarantee here, but also fixing the first point would resolve this second point too.

Test Plan: copy test added

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4165
2024-01-23 08:09:20 -05:00
Jarosław Sadziński
0aad09a4ed (core) Forms improvements
Summary:
Forms improvements and following new design
- New headers
- New UI
- New right panel options

Test Plan: Tests updated

Reviewers: georgegevoian, dsagal

Reviewed By: georgegevoian

Subscribers: dsagal, paulfitz

Differential Revision: https://phab.getgrist.com/D4158
2024-01-19 10:34:03 +01:00
George Gevoian
e12471347b (core) Form Publishing
Summary:
Adds initial implementation of form publishing, built upon WYSIWYS shares.

A simple UI for publishing and unpublishing forms is included.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: paulfitz, jarek

Differential Revision: https://phab.getgrist.com/D4154
2024-01-12 11:58:12 -08:00
Paul Fitzpatrick
a311b8b3e5 (core) updates from grist-core 2024-01-08 09:19:50 -05:00
Paul Fitzpatrick
2a206dfcf8 (core) add initial support for special shares
Summary:
This gives a mechanism for controlling access control within a document that is distinct from (though implemented with the same machinery as) granular access rules.

It was hard to find a good way to insert this that didn't dissolve in a soup of complications, so here's what I went with:
 * When reading rules, if there are shares, extra rules are added.
 * If there are shares, all rules are made conditional on a "ShareRef" user property.
 * "ShareRef" is null when a doc is accessed in normal way, and the row id of a share when accessed via a share.

There's no UI for controlling shares (George is working on it for forms), but you can do it by editing a `_grist_Shares` table in a document. Suppose you make a fresh document with a single page/table/widget, then to create an empty share you can do:

```
gristDocPageModel.gristDoc.get().docData.sendAction(['AddRecord', '_grist_Shares', null, {linkId: 'xyz', options: '{"publish": true}'}])
```

If you look at the home db now there should be something in the `shares` table:

```
$ sqlite3 -table landing.db "select * from shares"
+----+------------------------+------------------------+--------------+---------+
| id |          key           |         doc_id         |   link_id    | options |
+----+------------------------+------------------------+--------------+---------+
| 1  | gSL4g38PsyautLHnjmXh2K | 4qYuace1xP2CTcPunFdtan | xyz | ...      |
+----+------------------------+------------------------+--------------+---------+
```

If you take the key from that (gSL4g38PsyautLHnjmXh2K in this case) and replace the document's urlId in its URL with `s.<key>` (in this case `s.gSL4g38PsyautLHnjmXh2K` then you can use the regular document landing page (it will be quite blank initially) or API endpoint via the share.

E.g. for me `http://localhost:8080/o/docs/s0gSL4g38PsyautLHnjmXh2K/share-inter-3` accesses the doc.

To actually share some material - useful commands:

```
gristDocPageModel.gristDoc.get().docData.getMetaTable('_grist_Views_section').getRecords()
gristDocPageModel.gristDoc.get().docData.sendAction(['UpdateRecord', '_grist_Views_section', 1, {shareOptions: '{"publish": true, "form": true}'}])
gristDocPageModel.gristDoc.get().docData.getMetaTable('_grist_Pages').getRecords()
gristDocPageModel.gristDoc.get().docData.sendAction(['UpdateRecord', '_grist_Pages', 1, {shareRef: 1}])
```

For a share to be effective, at least one page needs to have its shareRef set to the rowId of the share, and at least one widget on one of those pages needs to have its shareOptions set to {"publish": "true", "form": "true"} (meaning turn on sharing, and include form sharing), and the share itself needs {"publish": true} on its options.

I think special shares are kind of incompatible with public sharing, since by their nature (allowing access to all endpoints) they easily expose the docId, and changing that would be hard.

Test Plan: tests added

Reviewers: dsagal, georgegevoian

Reviewed By: dsagal, georgegevoian

Subscribers: jarek, dsagal

Differential Revision: https://phab.getgrist.com/D4144
2024-01-04 05:57:38 -05:00
jyio
ba14a1bea7
OIDC: Support overriding end_session_endpoint using environment variable GRIST_OIDC_IDP_END_SESSION_ENDPOINT (#802)
Support overriding `end_session_endpoint` using environment variable `GRIST_OIDC_IDP_END_SESSION_ENDPOINT`
2024-01-03 15:49:32 -05:00
Florent
837597cd55
Fix deadlock with webhooks on document load #799 (#812) 2024-01-03 21:47:53 +02:00
Florent
6722512d96
Completely ignored disabled webhooks (#800) 2024-01-03 20:06:38 +02:00
Alex Hall
a2bd753649 (core) API console
Summary: Adds a custom interactive Swagger API console at `/apiconsole`. For now, this isn't visibly linked anywhere.

Test Plan: Manual, this is still an experimental and private feature. The idea is to merge this soon so that we have a chance to try it out in production.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4151
2023-12-27 22:02:30 +02:00
Paul Fitzpatrick
145138b7e9 (core) move apiserver tests to core, disentangling notifier+billing parts
Summary:
This moves some more tests to core that would be useful for ANCT,
which had been stuck in grist-saas due to some entanglements with
sendgrid and billing. For sendgrid, I've moved around just enough
material to permit the tests to run mostly unchanged. Ideally
the interface to a notification system would be generalized, but
that's a bigger project.

Test Plan:
checked that tests are likely to run as expected
in core using preview laid out by ./buildtools/build_core.sh

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4149
2023-12-26 08:43:12 -05:00
Jarosław Sadziński
a424450cbe (core) Forms feature
Summary:
A new widget type Forms. For now hidden behind GRIST_EXPERIMENTAL_PLUGINS().
This diff contains all the core moving parts as a serves as a base to extend this functionality
further.

Test Plan: New test added

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4130
2023-12-20 13:23:12 +01:00
Paul Fitzpatrick
2f0dbb7d25 (core) updates from grist-core 2023-12-18 09:28:35 -05:00
jyio
0fc3f80ff0
Rename endSessionEndpoint flag to skipEndSessionEndpoint (#793) 2023-12-17 15:10:10 -05:00
George Gevoian
6b1e98128a (core) Add visitorId to telemetry
Summary: The visitorId is added to telemetry events from Matomo's cookie, if present.

Test Plan: Manual.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D4132
2023-12-11 11:16:23 -05:00
Paul Fitzpatrick
1ccad9171e (core) updates from grist-core 2023-12-11 09:07:35 -05:00
Paul Fitzpatrick
0a69501e58
Change how a builtin custom widget bundle is found (#783)
This change makes builtin custom widget bundles work on grist-electron,
by finding the package in a slightly more flexible way.

It also includes a related change to make a widget manifest fetched
from the network optional if a flag is present, with an error being
logged rather than thrown. This could make it harder to track down
why custom widgets aren't available, but makes it easier to make
grist-electron work (including calendars) when the network is shut off.
Ideally we'd do something fancier when we can.
2023-12-05 11:51:22 -05:00
Alex Hall
887717bb15 (core) Decode cell values to prevent working around rule using 'in' on lists
Summary:
Fixes bug described in https://grist.slack.com/archives/C069RUP71/p1699643458649019

Decodes cell values obtained from `InfoView.get` when evaluating user-defined ACL formulas, i.e. the result of `rec.foo` in such a formula. In particular this is so that `rec.some_list` loses the leading `L` type code and behaves sensibly in an expression like `thing in rec.some_list`.

`InfoView.get` is called in many places, but for every usage I found other than here, leaving the cell values encoded was best.

Test Plan: Added two unit server tests. The first is for the main bug involving lists. The second checks the only other plausible way I could think of that this change affects behaviour, and it seems to be for the better since both tests failed before. Most operations involving non-primitive cell values don't do anything sensible with or without decoding, so behaviour shouldn't change meaningfully in those cases.

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian, paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4123
2023-12-04 23:34:08 +02:00
Paul Fitzpatrick
bcb9740d89 (core) update tests and endpoints after bare /urlId redirect added
Summary:
 * Some tests needed updating because fake document ids in tests were changed to be valid urlIds (the existing ones were too short).
 * urlId capture is tweaked to not allow hyphens, so some long login-related paths don't get confused with documents.

Test Plan: tests should pass again

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D4134
2023-12-04 11:14:10 -05:00
Dmitry S
4d9bbf6263 (core) Exit more cleanly on unhandled errors, and handle errors writing to Clients.
Summary:
- Node has a strong recommendation to assume bad state and exit promptly on
  unhandled exceptions and rejections. We follow it, and only make an effort to
  clean up before exiting, and to log the error in a more standard way.

- The only case seen in recent month of an unhandled rejection was for
  attempting to write overly large JSON to a Client websocket. Ensure that's
  handled, and add a test case that artificially reproduces this scenario.

Test Plan:
Added a test case for failing write to Client, and a test case that unhandled
errors indeed kill the server but with an attempt at cleanup.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4124
2023-12-01 09:42:00 -05:00
Paul Fitzpatrick
d89e008a75 (core) updates from grist-core 2023-12-01 08:14:12 -05:00
Florent
cf0cbb404e
Allow URLs with only a docID #768 (#771)
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-11-29 15:13:29 -05:00
Dmitry S
a2688deb9c (core) For slowly reported telemetry, add breaks to give other processes a chance to run
Summary:
- Also, avoid scanning the database if relevant telemetry is off.
- Also, report time during breaks of reporting telemetry.

Test Plan: Tested manually in dev only that breaks are taken.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4125
2023-11-28 14:34:36 -05:00
Paul Fitzpatrick
de13a2fd7a
include the @gristlabs/grist-widget package, so Calendar is always available (#745)
The Calendar feature was implemented as a custom widget. To make it
available offline, we prepare a package that includes it, and add that
to Grist. The PluginManager is configured to find it.

An optional `GRIST_SKIP_BUNDLED_WIDGETS` flag is added to disable
widgets bundled this way from being used. This may be needed by
the tests in grist-widget to avoid getting an echo :-)
2023-11-28 09:28:15 -05:00
Florent
961b1c1956
Introduce TYPEORM_EXTRA env variable (#770)
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-11-27 10:47:56 -05:00
Florent
1fec674d28
OIDC: ensure that email_veridied is set by default (#765)
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-11-22 14:58:29 -05:00
Florent
f8c6892643
OIDC allow to configure name and email attrs, and to skipp end session endpoint (#746)
* support GRIST_OIDC_SP_PROFILE_NAME_ATTR, defaulting to the concatenation of "given_name" + "family_name" or the "name" attribute.
* support GRIST_OIDC_SP_PROFILE_EMAIL_ATTR, defaulting to "email".
* support GRIST_OIDC_IDP_SKIP_END_SESSION_ENDPOINT: If set to "true", will not attempt to call the IdP's end_session_endpoint. Fail early if the endpoint does not exist, and this variable isn't set.

The last part is because some IdPs like Gitlab do not provide end_session_endpoint. In such cases, GRIST_OIDC_IDP_SKIP_END_SESSION_ENDPOINT=true should be set to have the Grist logout button only log out of Grist, and not out of the IdP.

---------

Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-11-21 15:20:40 -05:00
Paul Fitzpatrick
cea0404a22 (core) updates from grist-core 2023-11-20 11:28:50 -05:00
George Gevoian
caf830db08 (core) Record Cards
Summary:
Adds a new Record Card view section to each non-summary table, which can be from opened from various parts of the Grist UI to view and edit records in a popup card view.

Work is still ongoing, so the feature is locked away behind a flag; follow-up work is planned to finish up the implementation and add end-to-end tests.

Test Plan: Python and server tests. Browser tests will be included in a follow-up.

Reviewers: jarek, paulfitz

Reviewed By: jarek

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4114
2023-11-19 20:12:37 -05:00
Florent
7bc862fb02
Add header=colId option for the table-schema API #719 (#749) 2023-11-17 17:45:15 +02:00
Dmitry S
2eec48b685 (core) Support adjusting OOM score for child sandbox processes.
Summary:
Also update error handling in NSandbox initialization to avoid node
exiting when sandbox can't be created.

Info on oom_score and choom: https://man7.org/linux/man-pages/man1/choom.1.html

Empirically, in docker and under fargate, oom_score (visible in
/proc/PID/oom_score) seems to be calculated approximately as

    ((ProcessRSS / TotalMemory * 1000 + 999 + oom_score_adj) * 2/3)

though this doesn't correspond to any documentation I could find.

In addition, under docker / fargate it does not work to set oom_score_adj (with choom or via /proc/PID/oom_score_adj) to a negative value to give priority to a node process. That's why this diff adjust the score up for sandbox processes instead.

Test Plan:
Checked that grist-omnibus built with this change respects the
variable, and sandbox processes end up with higher oom_score values.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4112
2023-11-16 17:43:06 -05:00
George Gevoian
c9bba5207e (core) Add more telemetry events
Summary: Adds new telemetry events and a flag for whether an event originated from a team site.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: jarek, dsagal

Differential Revision: https://phab.getgrist.com/D4105
2023-11-16 12:38:08 -05:00
Florent
e8789e6531
Issue 740 OIDC login redirect (#742)
* Fix OIDC redirects from team site to personal page after login #740

Also:
 - compare state in session and state passed through parameters
 (otherwise the state won't have any effect regarding the security).
 - delete the session even after an authentication failure

* More logs for OIDC #740

---------

Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-11-15 09:23:32 -05:00
Paul Fitzpatrick
12317097f7 (core) move some untagged assets served on plugin port
Summary: Some untagged assets on the plugin port could be a problem if that port is merged with the regular Grist app port, so we nest them within a non-conflicting path (/plugins/assets).

Test Plan: see if a test fails anywhere

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4116
2023-11-14 15:58:00 -05:00
Alex Hall
5197891427 (core) Remove transform columns on shutdown
Summary: Call a new user action `RemoveTransformColumns` in ActiveDoc shutdown.

Test Plan: Added nbrowser test

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4107
2023-11-14 22:31:34 +02:00
Alex Hall
3dfe4be5f3 (core) Remove DB transaction from webhook update, add mutex to all webhook endpoints
Summary:
This removes problematic code that was holding a HomeDB transaction while applying user actions which could hang indefinitely, especially if the webhook queue is full as in https://grist.slack.com/archives/C05DBJ6LA1F/p1698159750945949.

The discussion about adding this code is here: https://phab.getgrist.com/D3821#inline-45054

The initial motivation was to roll back HomeDB changes if something went wrong while applying user actions, to avoid saving only part of the changes the user requested. I think it's actually fine to just allow such a partial save to happen - I don't see anything particularly undesirable about keeping an update to the webhook URL if other updates requested by the user didn't also get applied, as the fields don't affect each other.

The comment approving the transaction approach said "so we shouldn't end up leave the transaction hanging around too long" which has been falsified.

It looks like there was also some desire to prevent a mess caused by multiple simultaneous calls to this endpoint, which the transaction may have helped with a little, but didn't really seem like a solution. Comments in `Triggers.ts` also mention fears of race conditions when clearing (some of) the queue and the need for some locking. So I wrapped all webhook-related endpoints in a simple `Mutex` held by the `ActiveDoc` to prevent simultaneous changes. I *think* this is a good thing. These endpoints shouldn't be called frequently enough to create a performance issue, and this shouldn't affect actually sending webhook events when records are added/updated. And it does seem like interleaving calls to these endpoints could cause very weird problems.

Test Plan: Nothing yet, I'd like to hear if others think this is sensible.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D4111
2023-11-14 16:51:21 +02:00
Paul Fitzpatrick
68801474b1 (core) updates from grist-core 2023-11-13 07:58:49 -05:00
Florent
a4998b4b21
Add native OIDC support in Grist #707 (#717) 2023-11-09 14:56:39 -05:00
wunter8
73ab1de6e4
Remove attachments and system files from document download as template (#729) 2023-11-08 15:09:01 -05:00