Add body in log requests.
GRIST_LOG_SKIP_HTTP is a badly named environment variable and its
expected values are confusing (to log the requests, you actually have to
set its value to "", and setting to "false" actually is equivalent to
setting to "true").
We deprecate this env variable in favor of GRIST_LOG_HTTP which is more
convenient and understandable:
- by default, its undefined, so nothing is logged;
- to enable the logs, you just have to set GRIST_LOG_HTTP=true
Also this commit removes the default value for GRIST_LOG_SKIP_HTTP,
because we don't have to set it to "true" to actually disable the
requests logging thanks to GRIST_LOG_HTTP. FlexServer now handles
the historical behavior for this deprecated variable.
---------
Co-authored-by: Jonathan Perret <j-github@jonathanperret.net>
* Introduces new configuration variables for OIDC:
- GRIST_OIDC_IDP_ENABLED_PROTECTIONS
- GRIST_OIDC_IDP_ACR_VALUES
- GRIST_OIDC_IDP_EXTRA_CLIENT_METADATA
* Implements all supported protections in oidc/Protections.ts
* Includes a better error page for failed OIDC logins
* Includes some other improvements, e.g. to logging, to OIDC
* Adds a large unit test for OIDCConfig
* Adds support for SERVER_NODE_OPTIONS for running tests
* Adds to documentation/develop.md info about GREP_TESTS, VERBOSE, and SERVER_NODE_OPTIONS.
Summary:
A new onboarding page is now shown to all new users visiting the doc
menu for the first time. Tutorial cards on the doc menu have been
replaced with a new version that tracks completion progress, alongside
a new card that opens the orientation video.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D4296
This adds a config file that's loaded very early on during startup.
It enables us to save/load settings from within Grist's admin panel, that affect the startup of the FlexServer.
The config file loading:
- Is type-safe,
- Validates the config file on startup
- Provides a path to upgrade to future versions.
It should be extensible from other versions of Grist (such as desktop), by overriding `getGlobalConfig` in stubs.
----
Some minor refactors needed to occur to make this possible. This includes:
- Extracting config loading into its own module (out of FlexServer).
- Cleaning up the `loadConfig` function in FlexServer into `loadLoginSystem` (which is what its main purpose was before).
Summary:
Version API endpoint wasn't logging telemetry from POST requests. The issue was in registration
order, this endpoint was registered before `expressJson` and it couldn't read json body in the handler.
Test Plan: Added new test
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4277
This adds an endpoint for the admin user to be able to signal to a
controlling process to restart the server. This is intended for
`docker-runner.mjs`.
Summary:
- Reading plans from Stripe, and allowing Stripe to define custom plans.
- Storing product features (aka limits) in Stripe, that override those in db.
- Adding hierarchical data in Stripe. All features are defined at Product level but can be overwritten on Price levels.
- New options for Support user to
-- Override product for team site (if he is added as a billing manager)
-- Override subscription and customer id for a team site
-- Attach an "offer", an custom plan configured in stripe that a team site can use
-- Enabling wire transfer for subscription by allowing subscription to be created without a payment method (which is customizable)
Test Plan: Updated and new.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4201
* Adds authentication mechanism to admin panel
Adds field to the "Security settings" admin display, showing the
currently configured authentication mechanism.
* Adds 14px margin to admin panel names
Context:
On self-hosted instances, some places in the code rely on the fact that we resolves public domains while being behind reverse proxies. This leads to cases where features are not available, such as the "Duplicate document" one.
Bugs that are solved - n self-hosted instances:
Impossible to open templates and tutorials right after having converted them;
Impossible to submit forms since version 1.1.13;
Impossible to restore a previous version of a document (snapshot);
Impossible to copy a document;
Solution:
Introduce the APP_HOME_INTERNAL_URL env variable, which is quite the same as APP_DOC_INTERNAL_URL except that it may point to any home worker;
Make /api/worker/:assignmentId([^/]+)/?* return not only the doc worker public url but also the internal one, and adapt the call points like fetchDocs;
Ensure that the home and doc worker internal urls are trusted by trustOrigin;
---------
Co-authored-by: jordigh <jordigh@octave.org>
Summary:
This adds a basic sandbox check to the admin panel. It also makes
the "probes" used in the boot page available from the admin panel,
though they are not yet displayed. The sandbox check is built as
a probe.
In the interests of time, a lot of steps had to be deferred:
* Reconcile fully the admin panel and boot page. Specifically, the
admin panel should be equally robust to common configuration problems.
* Add tests for the sandbox check.
* Generalize to multi-server setups. The read-out will not yet be useful
for setups where doc workers and home servers are configured
separately.
Test Plan: Added new test
Reviewers: jarek, georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4241
Summary:
Update for the admin page to show the latest available version information.
- Latest version is read from docs.getgrist.com by default
- It sends basic information (installationId, deployment type, and version)
- Checks are done only on the page itself
- The actual request is routed through the API (to avoid CORS)
Test Plan: Added new test
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4238
Summary:
New endpoint `/api/version` that returns latest version of stable docker image in format:
```
{"latestVersion":"1.1.12","
updatedAt":"2024-03-06T06:28:25.752337Z","
isCritical":false,
"updateURL":"https://hub.docker.com/r/gristlabs/grist"
}
```
It connects to docker hub API and reads the version from the tag lists endpoint.
Stores telemetry passed from the client such us: current version, deployment type, installationId and others.
Test Plan: Added new test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4220
* Shutdown Doc worker when it is not considered as available in Redis
* Use isAffirmative for GRIST_MANAGED_WORKERS
* Upgrade Sinon for the tests
* Run Smoke test with pages in English
* Add logic in /status endpoint
Summary:
Front-end code can now test if emails are enabled
and hide some parts of UI based on it.
Test Plan:
Only secondery text was hidden on add users dialog.
Tested manually.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: georgegevoian
Differential Revision: https://phab.getgrist.com/D4221
Summary:
- Add InstallAdmin class to identify users who can manage Grist installation.
This is overridable by different Grist flavors (e.g. different in SaaS).
It generalizes previous logic used to decide who can control Activation
settings (e.g. enable telemetry).
- Implement a basic Admin Panel at /admin, and move items previously in the
"Support Grist" page into the "Support Grist" section of the Admin Panel.
- Replace "Support Grist" menu items with "Admin Panel" and show only to admins.
- Add "Support Grist" links to Github sponsorship to user-account menu.
- Add "Support Grist" button to top-bar, which
- for admins, replaces the previous "Contribute" button and reopens the "Support Grist / opt-in to telemetry" nudge (unchanged)
- for everyone else, links to Github sponsorship
- in either case, user can dismiss it.
Test Plan: Shuffled some test cases between Support Grist and the new Admin Panel, and added some new cases.
Reviewers: jarek, paulfitz
Reviewed By: jarek, paulfitz
Differential Revision: https://phab.getgrist.com/D4194
This is a start at a page for diagnosing problems while setting up Grist. Starting to add some diagnostics based on feedback in github issues. We should make Grist installation easier! But when there is a problem it should be easier to diagnose than it is now, and this may help. The page is ugly and doesn't have many diagnostics yet, but we can iterate.
Visit `/boot` on a Grist server for tips on how to use this feature.
Summary:
Login (and other) middleware was included in the public form URL by mistake,
forcing logins on forms hosted on non-personal sites.
Test Plan: Browser test.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D4181
Summary:
- Updates styling of form submitted page.
- Tweaks styling of checkboxes, labels, and questions on form page.
- Adds new form 404 page.
- Adds checkbox to not show warning again when publishing or un-publishing a form.
- Excludes formula, hidden, and attachment columns in submitted form data.
- Adds placeholder text to form configuration inputs.
- Improves dark mode styling in Form widget.
- Updates default title and description of new forms.
- Updates styling of Form widget buttons.
- Fixes form success text input handling.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D4170
Summary:
A new widget type Forms. For now hidden behind GRIST_EXPERIMENTAL_PLUGINS().
This diff contains all the core moving parts as a serves as a base to extend this functionality
further.
Test Plan: New test added
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4130
This change makes builtin custom widget bundles work on grist-electron,
by finding the package in a slightly more flexible way.
It also includes a related change to make a widget manifest fetched
from the network optional if a flag is present, with an error being
logged rather than thrown. This could make it harder to track down
why custom widgets aren't available, but makes it easier to make
grist-electron work (including calendars) when the network is shut off.
Ideally we'd do something fancier when we can.
Summary:
- Node has a strong recommendation to assume bad state and exit promptly on
unhandled exceptions and rejections. We follow it, and only make an effort to
clean up before exiting, and to log the error in a more standard way.
- The only case seen in recent month of an unhandled rejection was for
attempting to write overly large JSON to a Client websocket. Ensure that's
handled, and add a test case that artificially reproduces this scenario.
Test Plan:
Added a test case for failing write to Client, and a test case that unhandled
errors indeed kill the server but with an attempt at cleanup.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4124
The Calendar feature was implemented as a custom widget. To make it
available offline, we prepare a package that includes it, and add that
to Grist. The PluginManager is configured to find it.
An optional `GRIST_SKIP_BUNDLED_WIDGETS` flag is added to disable
widgets bundled this way from being used. This may be needed by
the tests in grist-widget to avoid getting an echo :-)
Summary: Some untagged assets on the plugin port could be a problem if that port is merged with the regular Grist app port, so we nest them within a non-conflicting path (/plugins/assets).
Test Plan: see if a test fails anywhere
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4116
Summary:
By default, only respect GRIST_FORWARD_AUTH_HEADER on login endpoints; sessions are used elsewhere.
With GRIST_IGNORE_SESSION, do not use sessions, and respect GRIST_FORWARD_AUTH_HEADER on all endpoints.
GRIST_PROXY_AUTH_HEADER is now a synonym to GRIST_FORWARD_AUTH_HEADER.
Test Plan: Fixed tests. Tested first approach (no GRIST_IGNORE_SESSION) with grist-omnibus manually. Tested the second approach (with GRIST_IGNORE_SESSION) with a Apache-based setup enforcing http basic auth on all endpoints.
Reviewers: paulfitz, georgegevoian
Reviewed By: paulfitz, georgegevoian
Differential Revision: https://phab.getgrist.com/D4104
Summary: Adds a handful of new telemetry events, and makes a few tweaks to allow for better organization of telemetry.
Test Plan: Manual.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D4100
Summary:
Recently, the server became more strict about not responding to
requests before being fully configured. This is a problem when
a doc worker is trying to check whether it has become available
from a load balancer.
This change gives access to health checks prior to configuration
being complete. Otherwise, app endpoints accessed before full
configuration return a 503.
A flag is added to /status to allow checking explicitly for
the server being ready and configured.
Test Plan: manual
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D4103
Summary:
This adds support for bundling custom widgets with the Grist app, as follows:
* Adds a new `widgets` component to plugins mechanism.
* When a set of widgets is provided in a plugin, the html/js/css assets for those widgets are served on the existing untrusted user content port.
* Any bundled `grist-plugin-api.js` will be served with the Grist app's own version of that file. It is important that bundled widgets not refer to https://docs.getgrist.com for the plugin js, since they must be capable of working offline.
* The logic for configuring that port is updated a bit.
* I removed the CustomAttachedView class in favor of applying settings of bundled custom widgets more directly, without modification on view.
Any Grist installation via docker will need an extra step now, since there is an extra port that needs exposing for full functionality. I did add a `GRIST_TRUST_PLUGINS` option for anyone who really doesn't want to do this, and would prefer to trust the plugins and have them served on the same port.
Actually making use of bundling will be another step. It'll be important to mesh it with our SaaS's use of APP_STATIC_URL for serving most static assets.
Design sketch: https://grist.quip.com/bJlWACWzr2R9/Bundled-custom-widgets
Test Plan: added a test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4069
Summary:
Links for the API endpoints in a cell didn't work as they were interpreted as
internal routes. Now they are properly detected as external.
Test Plan: Added new test
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D4078
Summary:
Changes the minimum version of Node to 18, and updates the Docker images and GitHub workflows to build Grist with Node 18.
Also updates various dependencies and scripts to support building running tests with arm64 builds of Node.
Test Plan: Existing tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3968
Summary:
Fixing a bug: account couldn't be closed when there was
a team site shared with everyone. Endpoint was checking
if there are any other team sites available, but this is already
tested in "Doom" in a better way, so this check was removed.
Test Plan: New test
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4067
Summary:
- /status accepts new optional query parameters: db=1, redis=1, and timeout=<ms> (defaults to 10_000).
- These verify that the server can make trivial calls to DB/Redis, and that they return within the timeout.
- New HealthCheck tests simulates DB and Redis problems.
- Added resilience to Redis reconnects (helped by a test case that simulates disconnects)
- When closing Redis-based session store, disconnect from Redis (to avoid hanging tests)
Some associated test reorg:
- Move stripeTools out of test/nbrowser, and remove an unnecessary dependency,
to avoid starting up browser for gen-server tests.
- Move TcpForwarder to its own file, to use in the new test.
Test Plan: Added a new HealthCheck test that simulates DB and Redis problems.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4054
Summary:
Adding new "Delete my account" button to the profile page that allows users to remove completely
their accounts as long as they don't own any team site.
Test Plan: Added
Reviewers: georgegevoian, paulfitz
Reviewed By: georgegevoian, paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4037
Summary:
The event is triggered whenever a document is created, imported, or
duplicated.
Test Plan: Tested manually.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D4035
Summary:
The new event captures responses to the welcome questionnaire.
Responses are also still sent to the special Grist document configured with
the DOC_ID_NEW_USER_INFO variable.
Test Plan: Tested manually.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D4034
* `GRIST_ANON_PLAYGROUND`: When set to 'false' deny anonymous users access to the home page
* `GRIST_FORCE_LOGIN`: Much like `GRIST_ANON_PLAYGROUND` but don't support anonymous access at all (features like sharing docs publicly requires authentication)
---------
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
Summary:
The new "copyDoc" query parameter on the login page sets a short-lived cookie, which is
then read when welcoming a new user to copy that document to their Home workspace, and
redirect to it. Currently, only templates and bare forks set this parameter.
A new API endpoint for copying a document to a workspace was also added.
Test Plan: Browser tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3992
