Commit Graph

1321 Commits

Author SHA1 Message Date
Paul Fitzpatrick
d89e008a75 (core) updates from grist-core 2023-12-01 08:14:12 -05:00
Florent
cf0cbb404e
Allow URLs with only a docID #768 (#771)
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-11-29 15:13:29 -05:00
Dmitry S
a2688deb9c (core) For slowly reported telemetry, add breaks to give other processes a chance to run
Summary:
- Also, avoid scanning the database if relevant telemetry is off.
- Also, report time during breaks of reporting telemetry.

Test Plan: Tested manually in dev only that breaks are taken.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4125
2023-11-28 14:34:36 -05:00
Paul Fitzpatrick
de13a2fd7a
include the @gristlabs/grist-widget package, so Calendar is always available (#745)
The Calendar feature was implemented as a custom widget. To make it
available offline, we prepare a package that includes it, and add that
to Grist. The PluginManager is configured to find it.

An optional `GRIST_SKIP_BUNDLED_WIDGETS` flag is added to disable
widgets bundled this way from being used. This may be needed by
the tests in grist-widget to avoid getting an echo :-)
2023-11-28 09:28:15 -05:00
Florent
961b1c1956
Introduce TYPEORM_EXTRA env variable (#770)
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-11-27 10:47:56 -05:00
Paul Fitzpatrick
65a1863015 (core) updates from grist-core 2023-11-27 04:25:34 -05:00
Jakub Serafin
c04f61738c (core) "add column with type" and "add formula column" options in new column menu
Summary: New Column menu was enhanced by "add column with type" and "add formula column" options. First one allow user to chose the type of newly created column, to save time for selecting this option in creator menu. "Add formula column" opens formula editor in popup state right after creating the column. In this case, renaming column popup is ignored to not overburden user with to many popup at once.

Test Plan: new nbrowser test was added to check validity of menu items, and output of menu action - if columns have given types or if formula editor popup is opened and functionin, accordingly.

Reviewers: georgegevoian

Differential Revision: https://phab.getgrist.com/D4113
2023-11-23 14:31:27 +01:00
Florent
1fec674d28
OIDC: ensure that email_veridied is set by default (#765)
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-11-22 14:58:29 -05:00
George Gevoian
707a8c7b32 (core) Enable Record Cards
Summary:
Adds remaining functionality, fixes, and polish to Record Cards and
removes their feature flag, enabling them by default.

Test Plan: Tests deferred; will be included in a follow-up diff.

Reviewers: jarek, paulfitz

Reviewed By: jarek

Subscribers: paulfitz, jarek

Differential Revision: https://phab.getgrist.com/D4121
2023-11-21 16:49:41 -05:00
Florent
f8c6892643
OIDC allow to configure name and email attrs, and to skipp end session endpoint (#746)
* support GRIST_OIDC_SP_PROFILE_NAME_ATTR, defaulting to the concatenation of "given_name" + "family_name" or the "name" attribute.
* support GRIST_OIDC_SP_PROFILE_EMAIL_ATTR, defaulting to "email".
* support GRIST_OIDC_IDP_SKIP_END_SESSION_ENDPOINT: If set to "true", will not attempt to call the IdP's end_session_endpoint. Fail early if the endpoint does not exist, and this variable isn't set.

The last part is because some IdPs like Gitlab do not provide end_session_endpoint. In such cases, GRIST_OIDC_IDP_SKIP_END_SESSION_ENDPOINT=true should be set to have the Grist logout button only log out of Grist, and not out of the IdP.

---------

Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-11-21 15:20:40 -05:00
Paul Fitzpatrick
cea0404a22 (core) updates from grist-core 2023-11-20 11:28:50 -05:00
George Gevoian
caf830db08 (core) Record Cards
Summary:
Adds a new Record Card view section to each non-summary table, which can be from opened from various parts of the Grist UI to view and edit records in a popup card view.

Work is still ongoing, so the feature is locked away behind a flag; follow-up work is planned to finish up the implementation and add end-to-end tests.

Test Plan: Python and server tests. Browser tests will be included in a follow-up.

Reviewers: jarek, paulfitz

Reviewed By: jarek

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4114
2023-11-19 20:12:37 -05:00
Florent
7bc862fb02
Add header=colId option for the table-schema API #719 (#749) 2023-11-17 17:45:15 +02:00
Dmitry S
2eec48b685 (core) Support adjusting OOM score for child sandbox processes.
Summary:
Also update error handling in NSandbox initialization to avoid node
exiting when sandbox can't be created.

Info on oom_score and choom: https://man7.org/linux/man-pages/man1/choom.1.html

Empirically, in docker and under fargate, oom_score (visible in
/proc/PID/oom_score) seems to be calculated approximately as

    ((ProcessRSS / TotalMemory * 1000 + 999 + oom_score_adj) * 2/3)

though this doesn't correspond to any documentation I could find.

In addition, under docker / fargate it does not work to set oom_score_adj (with choom or via /proc/PID/oom_score_adj) to a negative value to give priority to a node process. That's why this diff adjust the score up for sandbox processes instead.

Test Plan:
Checked that grist-omnibus built with this change respects the
variable, and sandbox processes end up with higher oom_score values.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4112
2023-11-16 17:43:06 -05:00
George Gevoian
c9bba5207e (core) Add more telemetry events
Summary: Adds new telemetry events and a flag for whether an event originated from a team site.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: jarek, dsagal

Differential Revision: https://phab.getgrist.com/D4105
2023-11-16 12:38:08 -05:00
Florent
e8789e6531
Issue 740 OIDC login redirect (#742)
* Fix OIDC redirects from team site to personal page after login #740

Also:
 - compare state in session and state passed through parameters
 (otherwise the state won't have any effect regarding the security).
 - delete the session even after an authentication failure

* More logs for OIDC #740

---------

Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-11-15 09:23:32 -05:00
Paul Fitzpatrick
a14543008d (core) updates from grist-core 2023-11-14 16:00:06 -05:00
Paul Fitzpatrick
12317097f7 (core) move some untagged assets served on plugin port
Summary: Some untagged assets on the plugin port could be a problem if that port is merged with the regular Grist app port, so we nest them within a non-conflicting path (/plugins/assets).

Test Plan: see if a test fails anywhere

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4116
2023-11-14 15:58:00 -05:00
Alex Hall
5197891427 (core) Remove transform columns on shutdown
Summary: Call a new user action `RemoveTransformColumns` in ActiveDoc shutdown.

Test Plan: Added nbrowser test

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4107
2023-11-14 22:31:34 +02:00
Alex Hall
3dfe4be5f3 (core) Remove DB transaction from webhook update, add mutex to all webhook endpoints
Summary:
This removes problematic code that was holding a HomeDB transaction while applying user actions which could hang indefinitely, especially if the webhook queue is full as in https://grist.slack.com/archives/C05DBJ6LA1F/p1698159750945949.

The discussion about adding this code is here: https://phab.getgrist.com/D3821#inline-45054

The initial motivation was to roll back HomeDB changes if something went wrong while applying user actions, to avoid saving only part of the changes the user requested. I think it's actually fine to just allow such a partial save to happen - I don't see anything particularly undesirable about keeping an update to the webhook URL if other updates requested by the user didn't also get applied, as the fields don't affect each other.

The comment approving the transaction approach said "so we shouldn't end up leave the transaction hanging around too long" which has been falsified.

It looks like there was also some desire to prevent a mess caused by multiple simultaneous calls to this endpoint, which the transaction may have helped with a little, but didn't really seem like a solution. Comments in `Triggers.ts` also mention fears of race conditions when clearing (some of) the queue and the need for some locking. So I wrapped all webhook-related endpoints in a simple `Mutex` held by the `ActiveDoc` to prevent simultaneous changes. I *think* this is a good thing. These endpoints shouldn't be called frequently enough to create a performance issue, and this shouldn't affect actually sending webhook events when records are added/updated. And it does seem like interleaving calls to these endpoints could cause very weird problems.

Test Plan: Nothing yet, I'd like to hear if others think this is sensible.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D4111
2023-11-14 16:51:21 +02:00
Jarosław Sadziński
67dc41776d (core) Treating null in toggle column as a valid value
Summary:
When action is pruned by ACL rules, the rule engine sometimes
rewrites actions, setting NULLs for boolean columns as a default value.
Null was not considered a valid value and it resulted in a bug in the UI
where the editor for the Toggle column was not rendered.

Test Plan: Added new

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D4110
2023-11-13 20:38:30 +01:00
Paul Fitzpatrick
2f96a56150
fix shadowed variable in CustomView (#743)
This corrects a minor linting problem that recently crept in, by
renaming a variable.
2023-11-13 10:56:01 -05:00
Paul Fitzpatrick
68801474b1 (core) updates from grist-core 2023-11-13 07:58:49 -05:00
Florent
a4998b4b21
Add native OIDC support in Grist #707 (#717) 2023-11-09 14:56:39 -05:00
wunter8
73ab1de6e4
Remove attachments and system files from document download as template (#729) 2023-11-08 15:09:01 -05:00
Jarosław Sadziński
3c219e05f6 (core) Removing the new menu flag
Summary: Enabling the `GRIST_NEW_COLUMN_MENU` flag by default and removing it.

Test Plan: Existing

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4098
2023-11-08 07:37:54 +01:00
Jarosław Sadziński
9262e1f1ef (core) Fixing bug with collapsed custom widget.
Summary:
Fix for a bug. Custom widget when collapsed and expanded was disconnecting from
Grist, as WidgetFrame was disposed to early.

Test Plan: Added new

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4109
2023-11-08 07:36:21 +01:00
Dmitry S
3210eee24f (core) Revamp ForwardAuthLogin and unify with GRIST_PROXY_AUTH_HEADER
Summary:
By default, only respect GRIST_FORWARD_AUTH_HEADER on login endpoints; sessions are used elsewhere.

With GRIST_IGNORE_SESSION, do not use sessions, and respect GRIST_FORWARD_AUTH_HEADER on all endpoints.

GRIST_PROXY_AUTH_HEADER is now a synonym to GRIST_FORWARD_AUTH_HEADER.

Test Plan: Fixed tests. Tested first approach (no GRIST_IGNORE_SESSION) with grist-omnibus manually. Tested the second approach (with GRIST_IGNORE_SESSION) with a Apache-based setup enforcing http basic auth on all endpoints.

Reviewers: paulfitz, georgegevoian

Reviewed By: paulfitz, georgegevoian

Differential Revision: https://phab.getgrist.com/D4104
2023-11-07 16:30:49 -05:00
Alex Hall
b7e9d2705e (core) When a webhook is disabled, clear its queue
Summary: Also fixes a few bugs found along the way, particularly that webhook payloads could contain stale data.

Test Plan: Added an nbrowser test, made existing test a bit more detailed.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D4102
2023-11-07 15:48:35 +02:00
George Gevoian
95cbbb8910 (core) Improve dark mode in tutorials
Summary:
Headings 4, 5, and 6 are now properly visible in dark mode. Additionally,
pre-formatted text and code blocks have improved styling in dark mode.

Test Plan: Manual.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4108
2023-11-06 13:06:13 -05:00
Paul Fitzpatrick
8053c81d02 (core) updates from grist-core 2023-11-06 08:20:57 -05:00
Florent
10822d3b86
getHostType: consider APP_DOC_INTERNAL_URL as native (#715)
The getHostType() now returns "native" when the host corresponds to the value of APP_DOC_INTERNAL_URL. T
While trying to scale, with a different internal and public URL for doc workers, and having configured the org to be specified in the path (GRIST_ORG_IN_PATH=true), the APP_DOC_INTERNAL_URL parameter was not treated as internal which made the connection between home server and doc workers impossible.

---------
https://github.com/gristlabs/grist-core/pull/715
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-11-06 09:24:59 +01:00
George Gevoian
7a85aaa7a1 (core) Add new telemetry events
Summary: Adds a handful of new telemetry events, and makes a few tweaks to allow for better organization of telemetry.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D4100
2023-11-01 10:49:33 -04:00
George Gevoian
51f7402297 (core) Show tooltips in other Grist flavors
Summary:
This enables tooltips in other Grist deployment types (e.g. grist-core). Previously,
most of these tooltips were only enabled in the SaaS offering of Grist.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D4097
2023-10-31 23:56:27 -04:00
Paul Fitzpatrick
4fb1df567b (core) change handling of server access prior to full configuration
Summary:
Recently, the server became more strict about not responding to
requests before being fully configured. This is a problem when
a doc worker is trying to check whether it has become available
from a load balancer.

This change gives access to health checks prior to configuration
being complete. Otherwise, app endpoints accessed before full
configuration return a 503.

A flag is added to /status to allow checking explicitly for
the server being ready and configured.

Test Plan: manual

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D4103
2023-10-31 17:42:48 -04:00
Paul Fitzpatrick
07bb90b5a6
allow bundled widgets to be hidden from dropdown, and nested (#714)
This makes a few refinements to bundling widgets:

  * A widget with `published: false` is not shown in the
    custom widget dropdown in the UI. This is so widgets
    can be bundled with the app for "native" use (like the
    calendar widget) without immediately resulting in an
    extra listing in the UI. (There are improvements we'd
    like to make to the UI to better communicate widget
    provenance and quality eventually, which would be a
    helpful alternative to just a binary flag.)

  * A relative path to the custom widget manifest is
    respected. This will make the bundling process marginally
    neater.
2023-10-30 21:13:21 -04:00
Paul Fitzpatrick
6059bdcf66
rename variable to avoid shadowing another (#712)
This fixes a small lint issue in some widget bundling code.
2023-10-30 09:52:42 -04:00
Jarosław Sadziński
c7ba31eb7d (core) Guessing column widget options when transforming from
Summary:
When converting changing the type of Any column, try to guess
the widgetOptions. Especially important for choice and choiceList types.

Test Plan: Existing

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D4088
2023-10-30 13:36:39 +01:00
Paul Fitzpatrick
cc9a9ae8c5 (core) support for bundling custom widgets with the Grist app
Summary:
This adds support for bundling custom widgets with the Grist app, as follows:

 * Adds a new `widgets` component to plugins mechanism.
 * When a set of widgets is provided in a plugin, the html/js/css assets for those widgets are served on the existing untrusted user content port.
 * Any bundled `grist-plugin-api.js` will be served with the Grist app's own version of that file. It is important that bundled widgets not refer to https://docs.getgrist.com for the plugin js, since they must be capable of working offline.
 * The logic for configuring that port is updated a bit.
 * I removed the CustomAttachedView class in favor of applying settings of bundled custom widgets more directly, without modification on view.

Any Grist installation via docker will need an extra step now, since there is an extra port that needs exposing for full functionality. I did add a `GRIST_TRUST_PLUGINS` option for anyone who really doesn't want to do this, and would prefer to trust the plugins and have them served on the same port.

Actually making use of bundling will be another step. It'll be important to mesh it with our SaaS's use of APP_STATIC_URL for serving most static assets.

Design sketch: https://grist.quip.com/bJlWACWzr2R9/Bundled-custom-widgets

Test Plan: added a test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4069
2023-10-27 17:00:10 -04:00
George Gevoian
cb0ce9b20f (core) Avoid reporting unhelpful ResizeObserver error
Summary: The error appears to be benign and not caused by any of our code.

Test Plan:
Reproducing was tricky.

On the Access Rules page, changing the browser's zoom level caused the error to surface. Weirdly enough, it only happened when connected to a larger, external monitor, and not on my laptop's internal monitor.

In any case, manual testing was done to confirm the error is no longer reported to the user after this change.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D4095
2023-10-27 15:26:20 -04:00
George Gevoian
ce23887be0 (core) Avoid editing fields on toggle dbclick
Summary:
This prevents a quirky UI behavior where double-clicking a toggle would cause
the field to start being edited.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4093
2023-10-27 15:08:24 -04:00
Alex Hall
4e67c679b2 (core) Options for plugin API functions which fetch data from the selected table or record
Summary:
Adds a new interface `FetchSelectedOptions` with three keys (including the preexisting `keepEncoded`) and adds/updates an optional `options: FetchSelectedOptions` to six related functions which fetch data from the selected table or record. The `keepEncoded` and `format` options have different default values for different methods for backwards compatibility, but otherwise the different methods now have much more similar behaviour. The new `includeColumns` option allows fetching all columns which was previously only possible using `docApi.fetchTable` (which wasn't always a great alternative) but this requires full access to avoid exposing more data than before and violating user expectations.

Eventually, similar options should be added to `docApi.fetchTable` to make the API even more consistent.

Discussion: https://grist.slack.com/archives/C0234CPPXPA/p1696510548994899

Test Plan: Added a new nbrowser test with a corresponding fixture site and document, showing how the functions have different default option values but are all configurable now.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4077
2023-10-26 23:46:00 +02:00
George Gevoian
1a04c2cffe (core) Improve context menu placement on narrow screens
Summary: On narrow screens, the menu is now less likely to overflow the viewport.

Test Plan: Manual.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D4094
2023-10-25 10:35:49 -04:00
George Gevoian
fed697e676 (core) Tweak Add Column menu
Summary:
Tweaking behavior of the unreleased Add Column menu per feedback from
Anais and Dmitry.

Test Plan: WIP

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4089
2023-10-24 15:35:42 -04:00
George Gevoian
de33c5a3c6 (core) Add tests and tooltips to new Add Column menu
Summary: Adds tooltips to the menu and tests for recently-added functionality.

Test Plan: Browser tests.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Subscribers: JakubSerafin

Differential Revision: https://phab.getgrist.com/D4087
2023-10-24 11:53:26 -04:00
Jarosław Sadziński
69d5ee53a8 (core) Treating API urls as external in cells
Summary:
Links for the API endpoints in a cell didn't work as they were interpreted as
internal routes. Now they are properly detected as external.

Test Plan: Added new test

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D4078
2023-10-24 08:55:08 +02:00
Jakub Serafin
77726849ad (core) setting timezone on create column
Summary: Before this diff, "Create at" and "Modify at" had empty "Timezone" field in column editor panel. This diff is setting document timezone to DateTime column created by this shortcuts

Test Plan: Manual so far

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D4086
2023-10-24 08:26:26 +02:00
Jakub Serafin
d0c1ca2174 (core) Summary:
Cleaning code that was wrongly merged during D4083

Test Plan: Manual smoke tests - create columns and references are working

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4085
2023-10-20 15:43:32 +02:00
Jakub Serafin
91f7606ae6 (core) Aggregate and reverse lookups
Summary:
Reverse and Aggregation lookup.
Aggregation lookup works when table have a reference list column. It allow to list value of any fields of a referenced values, or to make some basic operation on them (sum, average, count)
Reverse lookup works as reverse one, but it allow do to the same operations on all rows that have reference to given row

Test Plan: Manual so far.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4083
2023-10-20 13:26:17 +02:00
George Gevoian
74485f412d (core) Fix delete user button for Google-only accounts
Summary:
An unhandled error was being thrown by CognitoClient when a user was unable
to be found during account deletion. Google-only accounts are no longer
associated with a user in Cognito, so the error was actually benign. A warning is
now logged instead.

Test Plan: Manual.

Reviewers: paulfitz, jarek

Reviewed By: paulfitz, jarek

Differential Revision: https://phab.getgrist.com/D4073
2023-10-18 10:52:58 -04:00