Summary:
Also update error handling in NSandbox initialization to avoid node
exiting when sandbox can't be created.
Info on oom_score and choom: https://man7.org/linux/man-pages/man1/choom.1.html
Empirically, in docker and under fargate, oom_score (visible in
/proc/PID/oom_score) seems to be calculated approximately as
((ProcessRSS / TotalMemory * 1000 + 999 + oom_score_adj) * 2/3)
though this doesn't correspond to any documentation I could find.
In addition, under docker / fargate it does not work to set oom_score_adj (with choom or via /proc/PID/oom_score_adj) to a negative value to give priority to a node process. That's why this diff adjust the score up for sandbox processes instead.
Test Plan:
Checked that grist-omnibus built with this change respects the
variable, and sandbox processes end up with higher oom_score values.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4112
Summary: Adds new telemetry events and a flag for whether an event originated from a team site.
Test Plan: Manual.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: jarek, dsagal
Differential Revision: https://phab.getgrist.com/D4105
* Fix OIDC redirects from team site to personal page after login #740
Also:
- compare state in session and state passed through parameters
(otherwise the state won't have any effect regarding the security).
- delete the session even after an authentication failure
* More logs for OIDC #740
---------
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
Summary: Some untagged assets on the plugin port could be a problem if that port is merged with the regular Grist app port, so we nest them within a non-conflicting path (/plugins/assets).
Test Plan: see if a test fails anywhere
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4116
Summary: Call a new user action `RemoveTransformColumns` in ActiveDoc shutdown.
Test Plan: Added nbrowser test
Reviewers: georgegevoian, paulfitz
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4107
Summary:
This removes problematic code that was holding a HomeDB transaction while applying user actions which could hang indefinitely, especially if the webhook queue is full as in https://grist.slack.com/archives/C05DBJ6LA1F/p1698159750945949.
The discussion about adding this code is here: https://phab.getgrist.com/D3821#inline-45054
The initial motivation was to roll back HomeDB changes if something went wrong while applying user actions, to avoid saving only part of the changes the user requested. I think it's actually fine to just allow such a partial save to happen - I don't see anything particularly undesirable about keeping an update to the webhook URL if other updates requested by the user didn't also get applied, as the fields don't affect each other.
The comment approving the transaction approach said "so we shouldn't end up leave the transaction hanging around too long" which has been falsified.
It looks like there was also some desire to prevent a mess caused by multiple simultaneous calls to this endpoint, which the transaction may have helped with a little, but didn't really seem like a solution. Comments in `Triggers.ts` also mention fears of race conditions when clearing (some of) the queue and the need for some locking. So I wrapped all webhook-related endpoints in a simple `Mutex` held by the `ActiveDoc` to prevent simultaneous changes. I *think* this is a good thing. These endpoints shouldn't be called frequently enough to create a performance issue, and this shouldn't affect actually sending webhook events when records are added/updated. And it does seem like interleaving calls to these endpoints could cause very weird problems.
Test Plan: Nothing yet, I'd like to hear if others think this is sensible.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D4111
Summary:
When action is pruned by ACL rules, the rule engine sometimes
rewrites actions, setting NULLs for boolean columns as a default value.
Null was not considered a valid value and it resulted in a bug in the UI
where the editor for the Toggle column was not rendered.
Test Plan: Added new
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D4110
Pyodide packages needed rebuilding, and the pyodide project has
moved on a bit so a new version number is needed. The new packages
have already been built and pushed to S3.
To verify, go to `sandbox/pyodide` and follow the README there.
Then at top level do `GRIST_SANDBOX_FLAVOR=pyodide`. Try to
create and edit a document using formulas. It should work.
Summary: Enabling the `GRIST_NEW_COLUMN_MENU` flag by default and removing it.
Test Plan: Existing
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4098
Summary:
Fix for a bug. Custom widget when collapsed and expanded was disconnecting from
Grist, as WidgetFrame was disposed to early.
Test Plan: Added new
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4109
Summary:
By default, only respect GRIST_FORWARD_AUTH_HEADER on login endpoints; sessions are used elsewhere.
With GRIST_IGNORE_SESSION, do not use sessions, and respect GRIST_FORWARD_AUTH_HEADER on all endpoints.
GRIST_PROXY_AUTH_HEADER is now a synonym to GRIST_FORWARD_AUTH_HEADER.
Test Plan: Fixed tests. Tested first approach (no GRIST_IGNORE_SESSION) with grist-omnibus manually. Tested the second approach (with GRIST_IGNORE_SESSION) with a Apache-based setup enforcing http basic auth on all endpoints.
Reviewers: paulfitz, georgegevoian
Reviewed By: paulfitz, georgegevoian
Differential Revision: https://phab.getgrist.com/D4104
Summary: Also fixes a few bugs found along the way, particularly that webhook payloads could contain stale data.
Test Plan: Added an nbrowser test, made existing test a bit more detailed.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4102
Summary:
Headings 4, 5, and 6 are now properly visible in dark mode. Additionally,
pre-formatted text and code blocks have improved styling in dark mode.
Test Plan: Manual.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D4108
The getHostType() now returns "native" when the host corresponds to the value of APP_DOC_INTERNAL_URL. T
While trying to scale, with a different internal and public URL for doc workers, and having configured the org to be specified in the path (GRIST_ORG_IN_PATH=true), the APP_DOC_INTERNAL_URL parameter was not treated as internal which made the connection between home server and doc workers impossible.
---------
https://github.com/gristlabs/grist-core/pull/715
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
Summary: While looking at webhooks code, I noticed that it calls `ActiveDoc.fetchQuery` (which I think typically leads to the sandbox method `fetch_table`) with a query on the `id` column, where the values could potentially be all row IDs in the table (e.g. if a new column was added, as in a recent discussion about webhooks gone wrong). This suggests that `fetch_table` should try to avoid using a list for values when possible. In practice this only starts to become an issue at about 10k rows, so I don't know if this has caused any real problems, but it seemed like something worth fixing.
Test Plan:
Extended unit tests for correctness. Tested performance manually. Made a doc with this formula:
```
import time
row_ids = list(Table2.table.row_ids)
query = {"id": row_ids}
start = time.time()
result = table.table._engine.fetch_table('Table2', query=query)
end = time.time()
assert result[1] == row_ids
end - start, len(row_ids)
```
Then put a bunch of rows in `Table2`. This diff made the returned elapsed time much less.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D4092
Summary: Adds a handful of new telemetry events, and makes a few tweaks to allow for better organization of telemetry.
Test Plan: Manual.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D4100
Florent
Dmitry S
Franček Prijatelj
George Gevoian
nbush
Paul Fitzpatrick
Alex Hall
Jarosław Sadziński
Jakub Serafin
Paul Janzen
mohammadA
Matěj Bláha
gallegonovato
wunter8
Peter Dave Hello
Bonislaw
Владимир В
George Gevoian
github-actions[bot]
Ariejan de Vroom
Riccardo Polignieri