Summary:
This moves some more tests to core that would be useful for ANCT,
which had been stuck in grist-saas due to some entanglements with
sendgrid and billing. For sendgrid, I've moved around just enough
material to permit the tests to run mostly unchanged. Ideally
the interface to a notification system would be generalized, but
that's a bigger project.
Test Plan:
checked that tests are likely to run as expected
in core using preview laid out by ./buildtools/build_core.sh
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4149
Summary:
A new widget type Forms. For now hidden behind GRIST_EXPERIMENTAL_PLUGINS().
This diff contains all the core moving parts as a serves as a base to extend this functionality
further.
Test Plan: New test added
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4130
Summary: The visitorId is added to telemetry events from Matomo's cookie, if present.
Test Plan: Manual.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D4132
This change makes builtin custom widget bundles work on grist-electron,
by finding the package in a slightly more flexible way.
It also includes a related change to make a widget manifest fetched
from the network optional if a flag is present, with an error being
logged rather than thrown. This could make it harder to track down
why custom widgets aren't available, but makes it easier to make
grist-electron work (including calendars) when the network is shut off.
Ideally we'd do something fancier when we can.
Summary:
Fixes bug described in https://grist.slack.com/archives/C069RUP71/p1699643458649019
Decodes cell values obtained from `InfoView.get` when evaluating user-defined ACL formulas, i.e. the result of `rec.foo` in such a formula. In particular this is so that `rec.some_list` loses the leading `L` type code and behaves sensibly in an expression like `thing in rec.some_list`.
`InfoView.get` is called in many places, but for every usage I found other than here, leaving the cell values encoded was best.
Test Plan: Added two unit server tests. The first is for the main bug involving lists. The second checks the only other plausible way I could think of that this change affects behaviour, and it seems to be for the better since both tests failed before. Most operations involving non-primitive cell values don't do anything sensible with or without decoding, so behaviour shouldn't change meaningfully in those cases.
Reviewers: georgegevoian, paulfitz
Reviewed By: georgegevoian, paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4123
Summary:
* Some tests needed updating because fake document ids in tests were changed to be valid urlIds (the existing ones were too short).
* urlId capture is tweaked to not allow hyphens, so some long login-related paths don't get confused with documents.
Test Plan: tests should pass again
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D4134
Summary:
- Node has a strong recommendation to assume bad state and exit promptly on
unhandled exceptions and rejections. We follow it, and only make an effort to
clean up before exiting, and to log the error in a more standard way.
- The only case seen in recent month of an unhandled rejection was for
attempting to write overly large JSON to a Client websocket. Ensure that's
handled, and add a test case that artificially reproduces this scenario.
Test Plan:
Added a test case for failing write to Client, and a test case that unhandled
errors indeed kill the server but with an attempt at cleanup.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4124
Summary:
- Also, avoid scanning the database if relevant telemetry is off.
- Also, report time during breaks of reporting telemetry.
Test Plan: Tested manually in dev only that breaks are taken.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4125
The Calendar feature was implemented as a custom widget. To make it
available offline, we prepare a package that includes it, and add that
to Grist. The PluginManager is configured to find it.
An optional `GRIST_SKIP_BUNDLED_WIDGETS` flag is added to disable
widgets bundled this way from being used. This may be needed by
the tests in grist-widget to avoid getting an echo :-)
* support GRIST_OIDC_SP_PROFILE_NAME_ATTR, defaulting to the concatenation of "given_name" + "family_name" or the "name" attribute.
* support GRIST_OIDC_SP_PROFILE_EMAIL_ATTR, defaulting to "email".
* support GRIST_OIDC_IDP_SKIP_END_SESSION_ENDPOINT: If set to "true", will not attempt to call the IdP's end_session_endpoint. Fail early if the endpoint does not exist, and this variable isn't set.
The last part is because some IdPs like Gitlab do not provide end_session_endpoint. In such cases, GRIST_OIDC_IDP_SKIP_END_SESSION_ENDPOINT=true should be set to have the Grist logout button only log out of Grist, and not out of the IdP.
---------
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
Summary:
Adds a new Record Card view section to each non-summary table, which can be from opened from various parts of the Grist UI to view and edit records in a popup card view.
Work is still ongoing, so the feature is locked away behind a flag; follow-up work is planned to finish up the implementation and add end-to-end tests.
Test Plan: Python and server tests. Browser tests will be included in a follow-up.
Reviewers: jarek, paulfitz
Reviewed By: jarek
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4114
Summary:
Also update error handling in NSandbox initialization to avoid node
exiting when sandbox can't be created.
Info on oom_score and choom: https://man7.org/linux/man-pages/man1/choom.1.html
Empirically, in docker and under fargate, oom_score (visible in
/proc/PID/oom_score) seems to be calculated approximately as
((ProcessRSS / TotalMemory * 1000 + 999 + oom_score_adj) * 2/3)
though this doesn't correspond to any documentation I could find.
In addition, under docker / fargate it does not work to set oom_score_adj (with choom or via /proc/PID/oom_score_adj) to a negative value to give priority to a node process. That's why this diff adjust the score up for sandbox processes instead.
Test Plan:
Checked that grist-omnibus built with this change respects the
variable, and sandbox processes end up with higher oom_score values.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4112
Summary: Adds new telemetry events and a flag for whether an event originated from a team site.
Test Plan: Manual.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: jarek, dsagal
Differential Revision: https://phab.getgrist.com/D4105
* Fix OIDC redirects from team site to personal page after login #740
Also:
- compare state in session and state passed through parameters
(otherwise the state won't have any effect regarding the security).
- delete the session even after an authentication failure
* More logs for OIDC #740
---------
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
Summary: Some untagged assets on the plugin port could be a problem if that port is merged with the regular Grist app port, so we nest them within a non-conflicting path (/plugins/assets).
Test Plan: see if a test fails anywhere
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4116
Summary: Call a new user action `RemoveTransformColumns` in ActiveDoc shutdown.
Test Plan: Added nbrowser test
Reviewers: georgegevoian, paulfitz
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4107
Summary:
This removes problematic code that was holding a HomeDB transaction while applying user actions which could hang indefinitely, especially if the webhook queue is full as in https://grist.slack.com/archives/C05DBJ6LA1F/p1698159750945949.
The discussion about adding this code is here: https://phab.getgrist.com/D3821#inline-45054
The initial motivation was to roll back HomeDB changes if something went wrong while applying user actions, to avoid saving only part of the changes the user requested. I think it's actually fine to just allow such a partial save to happen - I don't see anything particularly undesirable about keeping an update to the webhook URL if other updates requested by the user didn't also get applied, as the fields don't affect each other.
The comment approving the transaction approach said "so we shouldn't end up leave the transaction hanging around too long" which has been falsified.
It looks like there was also some desire to prevent a mess caused by multiple simultaneous calls to this endpoint, which the transaction may have helped with a little, but didn't really seem like a solution. Comments in `Triggers.ts` also mention fears of race conditions when clearing (some of) the queue and the need for some locking. So I wrapped all webhook-related endpoints in a simple `Mutex` held by the `ActiveDoc` to prevent simultaneous changes. I *think* this is a good thing. These endpoints shouldn't be called frequently enough to create a performance issue, and this shouldn't affect actually sending webhook events when records are added/updated. And it does seem like interleaving calls to these endpoints could cause very weird problems.
Test Plan: Nothing yet, I'd like to hear if others think this is sensible.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D4111
Summary: Enabling the `GRIST_NEW_COLUMN_MENU` flag by default and removing it.
Test Plan: Existing
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4098
Summary:
By default, only respect GRIST_FORWARD_AUTH_HEADER on login endpoints; sessions are used elsewhere.
With GRIST_IGNORE_SESSION, do not use sessions, and respect GRIST_FORWARD_AUTH_HEADER on all endpoints.
GRIST_PROXY_AUTH_HEADER is now a synonym to GRIST_FORWARD_AUTH_HEADER.
Test Plan: Fixed tests. Tested first approach (no GRIST_IGNORE_SESSION) with grist-omnibus manually. Tested the second approach (with GRIST_IGNORE_SESSION) with a Apache-based setup enforcing http basic auth on all endpoints.
Reviewers: paulfitz, georgegevoian
Reviewed By: paulfitz, georgegevoian
Differential Revision: https://phab.getgrist.com/D4104
Summary: Also fixes a few bugs found along the way, particularly that webhook payloads could contain stale data.
Test Plan: Added an nbrowser test, made existing test a bit more detailed.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4102
Summary: Adds a handful of new telemetry events, and makes a few tweaks to allow for better organization of telemetry.
Test Plan: Manual.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D4100
Summary:
Recently, the server became more strict about not responding to
requests before being fully configured. This is a problem when
a doc worker is trying to check whether it has become available
from a load balancer.
This change gives access to health checks prior to configuration
being complete. Otherwise, app endpoints accessed before full
configuration return a 503.
A flag is added to /status to allow checking explicitly for
the server being ready and configured.
Test Plan: manual
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D4103
This makes a few refinements to bundling widgets:
* A widget with `published: false` is not shown in the
custom widget dropdown in the UI. This is so widgets
can be bundled with the app for "native" use (like the
calendar widget) without immediately resulting in an
extra listing in the UI. (There are improvements we'd
like to make to the UI to better communicate widget
provenance and quality eventually, which would be a
helpful alternative to just a binary flag.)
* A relative path to the custom widget manifest is
respected. This will make the bundling process marginally
neater.
Summary:
This adds support for bundling custom widgets with the Grist app, as follows:
* Adds a new `widgets` component to plugins mechanism.
* When a set of widgets is provided in a plugin, the html/js/css assets for those widgets are served on the existing untrusted user content port.
* Any bundled `grist-plugin-api.js` will be served with the Grist app's own version of that file. It is important that bundled widgets not refer to https://docs.getgrist.com for the plugin js, since they must be capable of working offline.
* The logic for configuring that port is updated a bit.
* I removed the CustomAttachedView class in favor of applying settings of bundled custom widgets more directly, without modification on view.
Any Grist installation via docker will need an extra step now, since there is an extra port that needs exposing for full functionality. I did add a `GRIST_TRUST_PLUGINS` option for anyone who really doesn't want to do this, and would prefer to trust the plugins and have them served on the same port.
Actually making use of bundling will be another step. It'll be important to mesh it with our SaaS's use of APP_STATIC_URL for serving most static assets.
Design sketch: https://grist.quip.com/bJlWACWzr2R9/Bundled-custom-widgets
Test Plan: added a test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4069
Summary:
Links for the API endpoints in a cell didn't work as they were interpreted as
internal routes. Now they are properly detected as external.
Test Plan: Added new test
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D4078
Summary:
An unhandled error was being thrown by CognitoClient when a user was unable
to be found during account deletion. Google-only accounts are no longer
associated with a user in Cognito, so the error was actually benign. A warning is
now logged instead.
Test Plan: Manual.
Reviewers: paulfitz, jarek
Reviewed By: paulfitz, jarek
Differential Revision: https://phab.getgrist.com/D4073
Summary:
A menu to be shown when new colum button is added. It's give access to various diffrent shortcuts, like adding new column, unhiding existing ones, fast adding lookup columns or trigger one (authoriship or timestamp). Design document can be found here: https://grist.quip.com/CTgxAQv9Ghjt/Add-Columns-more-easily
To turn on this menu flag GRIST_NEW_COLUMN_MENU to 1
Test Plan: UI tests suite under nbrowser/GridViewNewColumnMenu.ts
Reviewers: jarek, georgegevoian
Reviewed By: georgegevoian
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4074
Summary:
Changes the minimum version of Node to 18, and updates the Docker images and GitHub workflows to build Grist with Node 18.
Also updates various dependencies and scripts to support building running tests with arm64 builds of Node.
Test Plan: Existing tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3968
Summary:
Fixing a bug: account couldn't be closed when there was
a team site shared with everyone. Endpoint was checking
if there are any other team sites available, but this is already
tested in "Doom" in a better way, so this check was removed.
Test Plan: New test
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4067
Summary:
- /status accepts new optional query parameters: db=1, redis=1, and timeout=<ms> (defaults to 10_000).
- These verify that the server can make trivial calls to DB/Redis, and that they return within the timeout.
- New HealthCheck tests simulates DB and Redis problems.
- Added resilience to Redis reconnects (helped by a test case that simulates disconnects)
- When closing Redis-based session store, disconnect from Redis (to avoid hanging tests)
Some associated test reorg:
- Move stripeTools out of test/nbrowser, and remove an unnecessary dependency,
to avoid starting up browser for gen-server tests.
- Move TcpForwarder to its own file, to use in the new test.
Test Plan: Added a new HealthCheck test that simulates DB and Redis problems.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4054
Summary:
Adding new "Delete my account" button to the profile page that allows users to remove completely
their accounts as long as they don't own any team site.
Test Plan: Added
Reviewers: georgegevoian, paulfitz
Reviewed By: georgegevoian, paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4037
Summary:
Add two shutdown-related timeouts.
1. One is to limit the duration of any work that happens once shutdown
begins. In particular, waiting for an update to current time could block
indefinitely if the data engine is unresponsive. Such awaits are now
limited to 5 seconds.
2. The other is to allow documents to get shutdown for inactivity even when
some work takes forever. Certain work (e.g. applying user actions)
generally prevents a document from shutting down while it's pending. This
prevention is now limited to 5 minutes.
Shutting down a doc while something is pending may break some
assumptions, and lead to errors. The timeout is long to let us assume
that the work is stuck, and that errors are better than waiting forever.
Other changes:
- Periodic ActiveDoc work (intervals) is now started when a doc finishes
loading rather than in the constructor. The difference only showed up in
tests which makes the intervals much shorter.
- Move timeoutReached() utility function to gutil, and use it for
isLongerThan(), since they are basically identical. Also makes sure that the
timer in these is cleared in all cases.
- Remove duplicate waitForIt implementation (previously had a copy in both
test/server and core/test/server).
- Change testUtil.captureLog to pass messages to its callback, to allow asserts
on messages within the callback.
Test Plan:
Added new unittests for the new shutdowns, including a replication
of a bad state that was possible during shutdown.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4040
Summary:
The event is triggered whenever a document is created, imported, or
duplicated.
Test Plan: Tested manually.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D4035
