This is a start at a page for diagnosing problems while setting up Grist. Starting to add some diagnostics based on feedback in github issues. We should make Grist installation easier! But when there is a problem it should be easier to diagnose than it is now, and this may help. The page is ugly and doesn't have many diagnostics yet, but we can iterate.
Visit `/boot` on a Grist server for tips on how to use this feature.
Summary:
There has been inconsistency in using display email vs normalized email, which
ends up creating some duplication in downstream analyses (e.g. the same user
showing up twice with different capitalization).
1. Add UserProfile.loginEmail field with normalized email to prefer, when set, over the inconsistently used UserProfile.email.
2. In one place where it's not available, normalize the display email manually.
3. Clean up some code in Client.ts.
Unrelated tweak to API Console to be clear when a URL parameter wasn't found (rather than show whatever happens to be the first value).
Several test robustness improvements:
- Misplaced parenthesis in gristWebDriverUtils has been causing optTimeout argument to be ignored in tests, and treated always as indefinite.
- Attempt to fix SortMenu test by ignoring (retrying with logging) errors in waitForServer, which include "script timeout" errors that come from a non-configurable selenium or chromedriver timeout.
- Attempt to improve onNewTab() helper, which plays a role in failing Billing tests.
Test Plan: Tested manually the capitalization of logged emails. Counting on existing tests to catch issues.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4188
Summary:
Fixes misc. bugs with forms, updates Grist URLs on static form pages to link
to the new forms marketing page, and adds a forms announcement popup that's
shown next to the Add New button within a document.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D4185
Summary: The new events capture usage of forms, widgets, access rules, and onboarding tours and tips.
Test Plan: Manual.
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D4189
Summary:
If the longer OpenAI model exceeds the OpenAPI context length, we now perform another retry with a
shorter variant of the formula prompt. The shorter prompt excludes non-referenced tables and lookup
method definitions, which should help reduce token usage in documents with larger schemas.
Test Plan: Server test.
Reviewers: JakubSerafin
Reviewed By: JakubSerafin
Subscribers: JakubSerafin
Differential Revision: https://phab.getgrist.com/D4184
Summary:
Login (and other) middleware was included in the public form URL by mistake,
forcing logins on forms hosted on non-personal sites.
Test Plan: Browser test.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D4181
Summary:
Changes to building and serving:
- Remove unpkg dependencies, add npm module for swagger-ui-dist instead.
- Move apiconsole JS logic into core/app/client/apiconsole.ts, and use TypeScript.
- Add symlinks to swagger in static/ and core/static/.
- Refactor loadScript, and add loadCssFile; use these to load swagger-ui resources.
Changes to console itself:
- Support docId, workspaceId, orgId URL parameters. When present, the matching
value in dropdowns is moved to the front and marked as "(Current)".
- Fix the ordering of example values, particularly for workspaces.
- Remove unwanted example values.
- Hide confusing "Authorize" button.
- Hide API keys, and rely consistently on cookies for executing API calls.
Integration into Grist:
- Added a button to Document Settings, just under document ID in "API".
- The button opens a separate page, passing in org, workspace, and doc info for the current doc.
Test Plan: Only tested manually, no automated tests yet.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D4173
Summary:
Takes some numbers embedded in strings and moves them to the context, so
they can be aggregated and trends inspected more easily.
Test Plan: tested manually by looking at logs
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D4175
Summary:
- Adding little green asterisk at the end of field title.
- Fixing bug on columns component. Adding paragraph as a column and then selecting it was throwing error in the RightPanel
- Fixing boolean column bug in the editor
- Adding (--Choose--) placeholder for dropdowns
- Fixing columns logic: Dragging and dropping columns can unexpectedly add more columns.
- Added favicon and default page title
- Added svg to sync file for electron.
Test Plan: Updated
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D4172
Summary:
- Updates styling of form submitted page.
- Tweaks styling of checkboxes, labels, and questions on form page.
- Adds new form 404 page.
- Adds checkbox to not show warning again when publishing or un-publishing a form.
- Excludes formula, hidden, and attachment columns in submitted form data.
- Adds placeholder text to form configuration inputs.
- Improves dark mode styling in Form widget.
- Updates default title and description of new forms.
- Updates styling of Form widget buttons.
- Fixes form success text input handling.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D4170
Summary:
- Open all links in a new tab
- Excluding not filled columns (to fix trigger formulas)
- Fixed Ref/RefList submission
- Removing redundant type definitions for Box
- Adding header menu item
- Default empty values in select control
Test Plan: Updated
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4166
* Fix server crash when client passes malformed JSON
* Take remarks into account
---------
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
Summary:
The first time a worker opens a document, it will now check if it has any shares the home db needs to be aware of. If so, they will be added. This is important for documents uploaded/copied/forked/replaced, so that their shares work out of the box.
In future, may want some UI to give user control of whether shares are activated after upload/copy/fork/replace.
It seems tricky currently to know if a document is being opened for the first time. As a proxy, I check whether usage information has been calculated and saved to the db, since I can determine that without adding another db query. It is safe to synchronize shares more than necessary.
This leaves two gaps:
* If a document is created/uploaded/copied/forked/replaced and no attempt is made to access it prior to using a share, then that share won't actually be available. Not a problem currently I think, since how would a user have determined the share key. But in future it would be good to also do a sync after creation/upload/copy/fork/replacement/...
* On document replacement, usage info is reset but not absolutely immediately. So in principle shares could fail to be created on first load of the replacement. Usage info reset could be tweaked to give a guarantee here, but also fixing the first point would resolve this second point too.
Test Plan: copy test added
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4165
Summary:
Forms improvements and following new design
- New headers
- New UI
- New right panel options
Test Plan: Tests updated
Reviewers: georgegevoian, dsagal
Reviewed By: georgegevoian
Subscribers: dsagal, paulfitz
Differential Revision: https://phab.getgrist.com/D4158
Summary:
Adds initial implementation of form publishing, built upon WYSIWYS shares.
A simple UI for publishing and unpublishing forms is included.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Subscribers: paulfitz, jarek
Differential Revision: https://phab.getgrist.com/D4154
Summary:
This gives a mechanism for controlling access control within a document that is distinct from (though implemented with the same machinery as) granular access rules.
It was hard to find a good way to insert this that didn't dissolve in a soup of complications, so here's what I went with:
* When reading rules, if there are shares, extra rules are added.
* If there are shares, all rules are made conditional on a "ShareRef" user property.
* "ShareRef" is null when a doc is accessed in normal way, and the row id of a share when accessed via a share.
There's no UI for controlling shares (George is working on it for forms), but you can do it by editing a `_grist_Shares` table in a document. Suppose you make a fresh document with a single page/table/widget, then to create an empty share you can do:
```
gristDocPageModel.gristDoc.get().docData.sendAction(['AddRecord', '_grist_Shares', null, {linkId: 'xyz', options: '{"publish": true}'}])
```
If you look at the home db now there should be something in the `shares` table:
```
$ sqlite3 -table landing.db "select * from shares"
+----+------------------------+------------------------+--------------+---------+
| id | key | doc_id | link_id | options |
+----+------------------------+------------------------+--------------+---------+
| 1 | gSL4g38PsyautLHnjmXh2K | 4qYuace1xP2CTcPunFdtan | xyz | ... |
+----+------------------------+------------------------+--------------+---------+
```
If you take the key from that (gSL4g38PsyautLHnjmXh2K in this case) and replace the document's urlId in its URL with `s.<key>` (in this case `s.gSL4g38PsyautLHnjmXh2K` then you can use the regular document landing page (it will be quite blank initially) or API endpoint via the share.
E.g. for me `http://localhost:8080/o/docs/s0gSL4g38PsyautLHnjmXh2K/share-inter-3` accesses the doc.
To actually share some material - useful commands:
```
gristDocPageModel.gristDoc.get().docData.getMetaTable('_grist_Views_section').getRecords()
gristDocPageModel.gristDoc.get().docData.sendAction(['UpdateRecord', '_grist_Views_section', 1, {shareOptions: '{"publish": true, "form": true}'}])
gristDocPageModel.gristDoc.get().docData.getMetaTable('_grist_Pages').getRecords()
gristDocPageModel.gristDoc.get().docData.sendAction(['UpdateRecord', '_grist_Pages', 1, {shareRef: 1}])
```
For a share to be effective, at least one page needs to have its shareRef set to the rowId of the share, and at least one widget on one of those pages needs to have its shareOptions set to {"publish": "true", "form": "true"} (meaning turn on sharing, and include form sharing), and the share itself needs {"publish": true} on its options.
I think special shares are kind of incompatible with public sharing, since by their nature (allowing access to all endpoints) they easily expose the docId, and changing that would be hard.
Test Plan: tests added
Reviewers: dsagal, georgegevoian
Reviewed By: dsagal, georgegevoian
Subscribers: jarek, dsagal
Differential Revision: https://phab.getgrist.com/D4144
Summary: Adds a custom interactive Swagger API console at `/apiconsole`. For now, this isn't visibly linked anywhere.
Test Plan: Manual, this is still an experimental and private feature. The idea is to merge this soon so that we have a chance to try it out in production.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4151
Summary:
This moves some more tests to core that would be useful for ANCT,
which had been stuck in grist-saas due to some entanglements with
sendgrid and billing. For sendgrid, I've moved around just enough
material to permit the tests to run mostly unchanged. Ideally
the interface to a notification system would be generalized, but
that's a bigger project.
Test Plan:
checked that tests are likely to run as expected
in core using preview laid out by ./buildtools/build_core.sh
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4149
Summary:
A new widget type Forms. For now hidden behind GRIST_EXPERIMENTAL_PLUGINS().
This diff contains all the core moving parts as a serves as a base to extend this functionality
further.
Test Plan: New test added
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4130
Summary: The visitorId is added to telemetry events from Matomo's cookie, if present.
Test Plan: Manual.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D4132
This change makes builtin custom widget bundles work on grist-electron,
by finding the package in a slightly more flexible way.
It also includes a related change to make a widget manifest fetched
from the network optional if a flag is present, with an error being
logged rather than thrown. This could make it harder to track down
why custom widgets aren't available, but makes it easier to make
grist-electron work (including calendars) when the network is shut off.
Ideally we'd do something fancier when we can.
Summary:
Fixes bug described in https://grist.slack.com/archives/C069RUP71/p1699643458649019
Decodes cell values obtained from `InfoView.get` when evaluating user-defined ACL formulas, i.e. the result of `rec.foo` in such a formula. In particular this is so that `rec.some_list` loses the leading `L` type code and behaves sensibly in an expression like `thing in rec.some_list`.
`InfoView.get` is called in many places, but for every usage I found other than here, leaving the cell values encoded was best.
Test Plan: Added two unit server tests. The first is for the main bug involving lists. The second checks the only other plausible way I could think of that this change affects behaviour, and it seems to be for the better since both tests failed before. Most operations involving non-primitive cell values don't do anything sensible with or without decoding, so behaviour shouldn't change meaningfully in those cases.
Reviewers: georgegevoian, paulfitz
Reviewed By: georgegevoian, paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4123
Summary:
* Some tests needed updating because fake document ids in tests were changed to be valid urlIds (the existing ones were too short).
* urlId capture is tweaked to not allow hyphens, so some long login-related paths don't get confused with documents.
Test Plan: tests should pass again
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D4134
Summary:
- Node has a strong recommendation to assume bad state and exit promptly on
unhandled exceptions and rejections. We follow it, and only make an effort to
clean up before exiting, and to log the error in a more standard way.
- The only case seen in recent month of an unhandled rejection was for
attempting to write overly large JSON to a Client websocket. Ensure that's
handled, and add a test case that artificially reproduces this scenario.
Test Plan:
Added a test case for failing write to Client, and a test case that unhandled
errors indeed kill the server but with an attempt at cleanup.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4124
Summary:
- Also, avoid scanning the database if relevant telemetry is off.
- Also, report time during breaks of reporting telemetry.
Test Plan: Tested manually in dev only that breaks are taken.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4125
The Calendar feature was implemented as a custom widget. To make it
available offline, we prepare a package that includes it, and add that
to Grist. The PluginManager is configured to find it.
An optional `GRIST_SKIP_BUNDLED_WIDGETS` flag is added to disable
widgets bundled this way from being used. This may be needed by
the tests in grist-widget to avoid getting an echo :-)
* support GRIST_OIDC_SP_PROFILE_NAME_ATTR, defaulting to the concatenation of "given_name" + "family_name" or the "name" attribute.
* support GRIST_OIDC_SP_PROFILE_EMAIL_ATTR, defaulting to "email".
* support GRIST_OIDC_IDP_SKIP_END_SESSION_ENDPOINT: If set to "true", will not attempt to call the IdP's end_session_endpoint. Fail early if the endpoint does not exist, and this variable isn't set.
The last part is because some IdPs like Gitlab do not provide end_session_endpoint. In such cases, GRIST_OIDC_IDP_SKIP_END_SESSION_ENDPOINT=true should be set to have the Grist logout button only log out of Grist, and not out of the IdP.
---------
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
Summary:
Adds a new Record Card view section to each non-summary table, which can be from opened from various parts of the Grist UI to view and edit records in a popup card view.
Work is still ongoing, so the feature is locked away behind a flag; follow-up work is planned to finish up the implementation and add end-to-end tests.
Test Plan: Python and server tests. Browser tests will be included in a follow-up.
Reviewers: jarek, paulfitz
Reviewed By: jarek
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4114
Summary:
Also update error handling in NSandbox initialization to avoid node
exiting when sandbox can't be created.
Info on oom_score and choom: https://man7.org/linux/man-pages/man1/choom.1.html
Empirically, in docker and under fargate, oom_score (visible in
/proc/PID/oom_score) seems to be calculated approximately as
((ProcessRSS / TotalMemory * 1000 + 999 + oom_score_adj) * 2/3)
though this doesn't correspond to any documentation I could find.
In addition, under docker / fargate it does not work to set oom_score_adj (with choom or via /proc/PID/oom_score_adj) to a negative value to give priority to a node process. That's why this diff adjust the score up for sandbox processes instead.
Test Plan:
Checked that grist-omnibus built with this change respects the
variable, and sandbox processes end up with higher oom_score values.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4112
Summary: Adds new telemetry events and a flag for whether an event originated from a team site.
Test Plan: Manual.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: jarek, dsagal
Differential Revision: https://phab.getgrist.com/D4105
* Fix OIDC redirects from team site to personal page after login #740
Also:
- compare state in session and state passed through parameters
(otherwise the state won't have any effect regarding the security).
- delete the session even after an authentication failure
* More logs for OIDC #740
---------
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
Paul Fitzpatrick
George Gevoian
Dmitry S
CamilleLegeron
Jarosław Sadziński
Florent
jyio
Alex Hall