Commit Graph

1345 Commits

Author SHA1 Message Date
Florent
6722512d96
Completely ignored disabled webhooks (#800) 2024-01-03 20:06:38 +02:00
Alex Hall
225a76c9cb (core) Don't throw error in onRecord(s) for insufficient access for includeColumns
Summary:
This removes checking for full access in `onRecord/onRecords` when `includeColumns` is a non-default value. The check had two problems:

1. It relied on the access level being present in the URL query parameters, which doesn't work if the page has redirected. See the discussion in https://grist.slack.com/archives/C0234CPPXPA/p1702576602615509. There seems to be no way to reliably and synchronously check the access level.
2. Calling `onRecords` before `ready` and forgetting to handle an error from the access check meant that `ready` wouldn't be called, so Grist couldn't request the correct access level from the user. I made this mistake and it seems like a nasty footgun.

Ultimately this has no effect on security, as an error will still be raised, but in a place where the widget developer can't catch it. They'll still see an error message in the console, and they can still check the access level reliably using `onOptions`, so I think this is OK.

Test Plan: Updated nbrowser test

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian, paulfitz

Differential Revision: https://phab.getgrist.com/D4145
2023-12-30 10:16:40 +02:00
Alex Hall
a2bd753649 (core) API console
Summary: Adds a custom interactive Swagger API console at `/apiconsole`. For now, this isn't visibly linked anywhere.

Test Plan: Manual, this is still an experimental and private feature. The idea is to merge this soon so that we have a chance to try it out in production.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4151
2023-12-27 22:02:30 +02:00
Paul Fitzpatrick
145138b7e9 (core) move apiserver tests to core, disentangling notifier+billing parts
Summary:
This moves some more tests to core that would be useful for ANCT,
which had been stuck in grist-saas due to some entanglements with
sendgrid and billing. For sendgrid, I've moved around just enough
material to permit the tests to run mostly unchanged. Ideally
the interface to a notification system would be generalized, but
that's a bigger project.

Test Plan:
checked that tests are likely to run as expected
in core using preview laid out by ./buildtools/build_core.sh

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4149
2023-12-26 08:43:12 -05:00
Dmitry S
7e57b8c7a7 (core) Remove empty keyboard shortcut for command that shouldn't have one
Test Plan: Tested manually that strange-looking shortcut line for "Detach active editor" is gone.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4148
2023-12-20 10:51:14 -05:00
Jarosław Sadziński
a424450cbe (core) Forms feature
Summary:
A new widget type Forms. For now hidden behind GRIST_EXPERIMENTAL_PLUGINS().
This diff contains all the core moving parts as a serves as a base to extend this functionality
further.

Test Plan: New test added

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4130
2023-12-20 13:23:12 +01:00
Jarosław Sadziński
337757d0ba (core) Fix for linking issue.
Summary:
If linking state changes multiple times frequently the code that simulates async operation is
wrongly debounced, which causes inverted order of execution. This fix makes sure that only the last
call to filter function is used.

Test Plan: Adding new test

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D4139
2023-12-20 12:57:41 +01:00
George Gevoian
af69a4c8f4 (core) Make Raw Data page more responsive
Summary:
With both panels expanded and a narrow viewport, the tables on the
Raw Data page would visibly overflow. This improves things so that
overflow is handled more gracefully.

Test Plan: Manual.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4147
2023-12-19 19:15:28 -05:00
Alex Hall
bd52665f96 (core) Allow adding rows to widgets filtered by a link using a formula column
Summary:
When a widget `A` is selected by a widget `B` so that `A` is filtered, adding a new row to `A` uses the values in the selected row of `B` and the columns relevant to the linking as default values for the new row. This ensures that the new row matches the current linking filter and remains visible. However this would previously cause a sandbox error when one of the linking columns was a formula column, which doesn't allow setting values. This diff ignores formula columns when picking default values.

Since the value of the formula column in the new row typically won't match the linking filter, extra measures are needed to avoid the new row immediately disappearing. Regular filters already have a mechanism for this, but I didn't manage to extend it to also work for linking. Thanks @dsagal for creating `UnionRowSource` (originally in D4017) which is now used as the solution for temporarily exempting rows from both kinds of filtering.

While testing, I also came across another bug in linking summary tables that caused incorrect filtering, which I fixed with some changes to `DynamicQuerySet`.

Test Plan: Extended an nbrowser test, which both tests for the main change as well as the secondary bugfix.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D4135
2023-12-18 20:28:41 +02:00
Paul Fitzpatrick
2f0dbb7d25 (core) updates from grist-core 2023-12-18 09:28:35 -05:00
jyio
0fc3f80ff0
Rename endSessionEndpoint flag to skipEndSessionEndpoint (#793) 2023-12-17 15:10:10 -05:00
George Gevoian
cd4f5e5adb (core) Trigger email verification flow when needed
Summary:
When an unconfirmed user tries to sign in or reset their password, the email
verification flow is now automatically restarted. Prior to this change, an
unconfirmed user error was shown in the client.

Test Plan: Deployment tests.

Reviewers: paulfitz, dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D4133
2023-12-13 09:18:41 -05:00
George Gevoian
6b1e98128a (core) Add visitorId to telemetry
Summary: The visitorId is added to telemetry events from Matomo's cookie, if present.

Test Plan: Manual.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D4132
2023-12-11 11:16:23 -05:00
Paul Fitzpatrick
1ccad9171e (core) updates from grist-core 2023-12-11 09:07:35 -05:00
George Gevoian
7e05284cf2 (core) Add shortcut for opening Record Card
Summary: Also adds tests for previously untested Record Card behavior.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4136
2023-12-08 11:32:45 -05:00
Jarosław Sadziński
d0318af39b (core) Fixing bug with hiding multiple columns
Summary:
Selection in GridView wasn't updated when fields were removed, and the selected
column index was out of bounds.

Test Plan: New test added

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Differential Revision: https://phab.getgrist.com/D4137
2023-12-08 10:20:43 +01:00
Paul Fitzpatrick
0a69501e58
Change how a builtin custom widget bundle is found (#783)
This change makes builtin custom widget bundles work on grist-electron,
by finding the package in a slightly more flexible way.

It also includes a related change to make a widget manifest fetched
from the network optional if a flag is present, with an error being
logged rather than thrown. This could make it harder to track down
why custom widgets aren't available, but makes it easier to make
grist-electron work (including calendars) when the network is shut off.
Ideally we'd do something fancier when we can.
2023-12-05 11:51:22 -05:00
Jakub Serafin
8038ee881c (core) Capitalization in new colum menu
Summary:
- "Add Column" -> "Add column"
- "Detect Duplicates in" -> "Detect duplicates in"
- "Last Updated At" -> "Last updated at"
- "Created By" (At) -> "Created by" (at)
- "Last Updated By" -> "Last updated by"

Test Plan: I looked at menu and cannot see any more capital letters anywhere other that begining of the menu position.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D4127
2023-12-05 17:01:22 +01:00
Alex Hall
a1c62f32f4 (core) Fix selecting new row in chain of filter links
Summary:
When the 'new' row of a table is selected, another table filter linked to the first shows no data. This diff ensures that a third table filtered by the second also shows no data, i.e. that it behaves the same as if the second table was also on the 'new' row. Video of the bug: https://grist.slack.com/archives/C069RUP71/p1692622810900179

The functional code is copied almost verbatim from https://github.com/gristlabs/grist-core/pull/666 by @jvorob which was working correctly. A comment there mentioned a possible bug where:

> ...you can have the grayed-out "No row selected" text from disableEditing but still have rows showing up in the section. Haven't been able to reproduce...

I noticed this behaviour when I copied only part of the fix, but it disappeared after copying the whole thing, so it seems likely to me that this is why it couldn't be reproduced.

Test Plan: Added a new nbrowser test with a new fixture, which also tests filter link chains and selecting the new row more generally, since I couldn't find other tests of this.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: jvorob

Differential Revision: https://phab.getgrist.com/D4126
2023-12-05 16:56:13 +02:00
Dmitry S
09c84734db (core) Enable the 'none' option in ColorSelect for cell and header text styles
Test Plan: Added a test case

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4120
2023-12-04 19:13:52 -05:00
George Gevoian
cc56e91f5b (core) Allow descriptions for Raw Data tables
Summary: Descriptions can now be set on Raw Data table sections.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4131
2023-12-04 16:52:56 -05:00
Alex Hall
887717bb15 (core) Decode cell values to prevent working around rule using 'in' on lists
Summary:
Fixes bug described in https://grist.slack.com/archives/C069RUP71/p1699643458649019

Decodes cell values obtained from `InfoView.get` when evaluating user-defined ACL formulas, i.e. the result of `rec.foo` in such a formula. In particular this is so that `rec.some_list` loses the leading `L` type code and behaves sensibly in an expression like `thing in rec.some_list`.

`InfoView.get` is called in many places, but for every usage I found other than here, leaving the cell values encoded was best.

Test Plan: Added two unit server tests. The first is for the main bug involving lists. The second checks the only other plausible way I could think of that this change affects behaviour, and it seems to be for the better since both tests failed before. Most operations involving non-primitive cell values don't do anything sensible with or without decoding, so behaviour shouldn't change meaningfully in those cases.

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian, paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4123
2023-12-04 23:34:08 +02:00
Paul Fitzpatrick
bcb9740d89 (core) update tests and endpoints after bare /urlId redirect added
Summary:
 * Some tests needed updating because fake document ids in tests were changed to be valid urlIds (the existing ones were too short).
 * urlId capture is tweaked to not allow hyphens, so some long login-related paths don't get confused with documents.

Test Plan: tests should pass again

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D4134
2023-12-04 11:14:10 -05:00
Dmitry S
4d9bbf6263 (core) Exit more cleanly on unhandled errors, and handle errors writing to Clients.
Summary:
- Node has a strong recommendation to assume bad state and exit promptly on
  unhandled exceptions and rejections. We follow it, and only make an effort to
  clean up before exiting, and to log the error in a more standard way.

- The only case seen in recent month of an unhandled rejection was for
  attempting to write overly large JSON to a Client websocket. Ensure that's
  handled, and add a test case that artificially reproduces this scenario.

Test Plan:
Added a test case for failing write to Client, and a test case that unhandled
errors indeed kill the server but with an attempt at cleanup.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4124
2023-12-01 09:42:00 -05:00
Paul Fitzpatrick
d89e008a75 (core) updates from grist-core 2023-12-01 08:14:12 -05:00
Florent
cf0cbb404e
Allow URLs with only a docID #768 (#771)
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-11-29 15:13:29 -05:00
Dmitry S
a2688deb9c (core) For slowly reported telemetry, add breaks to give other processes a chance to run
Summary:
- Also, avoid scanning the database if relevant telemetry is off.
- Also, report time during breaks of reporting telemetry.

Test Plan: Tested manually in dev only that breaks are taken.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4125
2023-11-28 14:34:36 -05:00
Paul Fitzpatrick
de13a2fd7a
include the @gristlabs/grist-widget package, so Calendar is always available (#745)
The Calendar feature was implemented as a custom widget. To make it
available offline, we prepare a package that includes it, and add that
to Grist. The PluginManager is configured to find it.

An optional `GRIST_SKIP_BUNDLED_WIDGETS` flag is added to disable
widgets bundled this way from being used. This may be needed by
the tests in grist-widget to avoid getting an echo :-)
2023-11-28 09:28:15 -05:00
Florent
961b1c1956
Introduce TYPEORM_EXTRA env variable (#770)
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-11-27 10:47:56 -05:00
Paul Fitzpatrick
65a1863015 (core) updates from grist-core 2023-11-27 04:25:34 -05:00
Jakub Serafin
c04f61738c (core) "add column with type" and "add formula column" options in new column menu
Summary: New Column menu was enhanced by "add column with type" and "add formula column" options. First one allow user to chose the type of newly created column, to save time for selecting this option in creator menu. "Add formula column" opens formula editor in popup state right after creating the column. In this case, renaming column popup is ignored to not overburden user with to many popup at once.

Test Plan: new nbrowser test was added to check validity of menu items, and output of menu action - if columns have given types or if formula editor popup is opened and functionin, accordingly.

Reviewers: georgegevoian

Differential Revision: https://phab.getgrist.com/D4113
2023-11-23 14:31:27 +01:00
Florent
1fec674d28
OIDC: ensure that email_veridied is set by default (#765)
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-11-22 14:58:29 -05:00
George Gevoian
707a8c7b32 (core) Enable Record Cards
Summary:
Adds remaining functionality, fixes, and polish to Record Cards and
removes their feature flag, enabling them by default.

Test Plan: Tests deferred; will be included in a follow-up diff.

Reviewers: jarek, paulfitz

Reviewed By: jarek

Subscribers: paulfitz, jarek

Differential Revision: https://phab.getgrist.com/D4121
2023-11-21 16:49:41 -05:00
Florent
f8c6892643
OIDC allow to configure name and email attrs, and to skipp end session endpoint (#746)
* support GRIST_OIDC_SP_PROFILE_NAME_ATTR, defaulting to the concatenation of "given_name" + "family_name" or the "name" attribute.
* support GRIST_OIDC_SP_PROFILE_EMAIL_ATTR, defaulting to "email".
* support GRIST_OIDC_IDP_SKIP_END_SESSION_ENDPOINT: If set to "true", will not attempt to call the IdP's end_session_endpoint. Fail early if the endpoint does not exist, and this variable isn't set.

The last part is because some IdPs like Gitlab do not provide end_session_endpoint. In such cases, GRIST_OIDC_IDP_SKIP_END_SESSION_ENDPOINT=true should be set to have the Grist logout button only log out of Grist, and not out of the IdP.

---------

Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-11-21 15:20:40 -05:00
Paul Fitzpatrick
cea0404a22 (core) updates from grist-core 2023-11-20 11:28:50 -05:00
George Gevoian
caf830db08 (core) Record Cards
Summary:
Adds a new Record Card view section to each non-summary table, which can be from opened from various parts of the Grist UI to view and edit records in a popup card view.

Work is still ongoing, so the feature is locked away behind a flag; follow-up work is planned to finish up the implementation and add end-to-end tests.

Test Plan: Python and server tests. Browser tests will be included in a follow-up.

Reviewers: jarek, paulfitz

Reviewed By: jarek

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4114
2023-11-19 20:12:37 -05:00
Florent
7bc862fb02
Add header=colId option for the table-schema API #719 (#749) 2023-11-17 17:45:15 +02:00
Dmitry S
2eec48b685 (core) Support adjusting OOM score for child sandbox processes.
Summary:
Also update error handling in NSandbox initialization to avoid node
exiting when sandbox can't be created.

Info on oom_score and choom: https://man7.org/linux/man-pages/man1/choom.1.html

Empirically, in docker and under fargate, oom_score (visible in
/proc/PID/oom_score) seems to be calculated approximately as

    ((ProcessRSS / TotalMemory * 1000 + 999 + oom_score_adj) * 2/3)

though this doesn't correspond to any documentation I could find.

In addition, under docker / fargate it does not work to set oom_score_adj (with choom or via /proc/PID/oom_score_adj) to a negative value to give priority to a node process. That's why this diff adjust the score up for sandbox processes instead.

Test Plan:
Checked that grist-omnibus built with this change respects the
variable, and sandbox processes end up with higher oom_score values.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4112
2023-11-16 17:43:06 -05:00
George Gevoian
c9bba5207e (core) Add more telemetry events
Summary: Adds new telemetry events and a flag for whether an event originated from a team site.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: jarek, dsagal

Differential Revision: https://phab.getgrist.com/D4105
2023-11-16 12:38:08 -05:00
Florent
e8789e6531
Issue 740 OIDC login redirect (#742)
* Fix OIDC redirects from team site to personal page after login #740

Also:
 - compare state in session and state passed through parameters
 (otherwise the state won't have any effect regarding the security).
 - delete the session even after an authentication failure

* More logs for OIDC #740

---------

Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-11-15 09:23:32 -05:00
Paul Fitzpatrick
a14543008d (core) updates from grist-core 2023-11-14 16:00:06 -05:00
Paul Fitzpatrick
12317097f7 (core) move some untagged assets served on plugin port
Summary: Some untagged assets on the plugin port could be a problem if that port is merged with the regular Grist app port, so we nest them within a non-conflicting path (/plugins/assets).

Test Plan: see if a test fails anywhere

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4116
2023-11-14 15:58:00 -05:00
Alex Hall
5197891427 (core) Remove transform columns on shutdown
Summary: Call a new user action `RemoveTransformColumns` in ActiveDoc shutdown.

Test Plan: Added nbrowser test

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4107
2023-11-14 22:31:34 +02:00
Alex Hall
3dfe4be5f3 (core) Remove DB transaction from webhook update, add mutex to all webhook endpoints
Summary:
This removes problematic code that was holding a HomeDB transaction while applying user actions which could hang indefinitely, especially if the webhook queue is full as in https://grist.slack.com/archives/C05DBJ6LA1F/p1698159750945949.

The discussion about adding this code is here: https://phab.getgrist.com/D3821#inline-45054

The initial motivation was to roll back HomeDB changes if something went wrong while applying user actions, to avoid saving only part of the changes the user requested. I think it's actually fine to just allow such a partial save to happen - I don't see anything particularly undesirable about keeping an update to the webhook URL if other updates requested by the user didn't also get applied, as the fields don't affect each other.

The comment approving the transaction approach said "so we shouldn't end up leave the transaction hanging around too long" which has been falsified.

It looks like there was also some desire to prevent a mess caused by multiple simultaneous calls to this endpoint, which the transaction may have helped with a little, but didn't really seem like a solution. Comments in `Triggers.ts` also mention fears of race conditions when clearing (some of) the queue and the need for some locking. So I wrapped all webhook-related endpoints in a simple `Mutex` held by the `ActiveDoc` to prevent simultaneous changes. I *think* this is a good thing. These endpoints shouldn't be called frequently enough to create a performance issue, and this shouldn't affect actually sending webhook events when records are added/updated. And it does seem like interleaving calls to these endpoints could cause very weird problems.

Test Plan: Nothing yet, I'd like to hear if others think this is sensible.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D4111
2023-11-14 16:51:21 +02:00
Jarosław Sadziński
67dc41776d (core) Treating null in toggle column as a valid value
Summary:
When action is pruned by ACL rules, the rule engine sometimes
rewrites actions, setting NULLs for boolean columns as a default value.
Null was not considered a valid value and it resulted in a bug in the UI
where the editor for the Toggle column was not rendered.

Test Plan: Added new

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D4110
2023-11-13 20:38:30 +01:00
Paul Fitzpatrick
2f96a56150
fix shadowed variable in CustomView (#743)
This corrects a minor linting problem that recently crept in, by
renaming a variable.
2023-11-13 10:56:01 -05:00
Paul Fitzpatrick
68801474b1 (core) updates from grist-core 2023-11-13 07:58:49 -05:00
Florent
a4998b4b21
Add native OIDC support in Grist #707 (#717) 2023-11-09 14:56:39 -05:00
wunter8
73ab1de6e4
Remove attachments and system files from document download as template (#729) 2023-11-08 15:09:01 -05:00
Jarosław Sadziński
3c219e05f6 (core) Removing the new menu flag
Summary: Enabling the `GRIST_NEW_COLUMN_MENU` flag by default and removing it.

Test Plan: Existing

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4098
2023-11-08 07:37:54 +01:00