* switch default LLM from a model that is going away
If an api key is provided, Grist can use an LLM as an assistant
for writing formulas. The LLM can be self-hosted or an external
service. The default external service is OpenAI. This commit
freshens the default model used, if this feature is enabled,
since the existing one is going away. Benchmarking suggests the
results are generally better, though not dramatically so.
The feature of falling back on a longer context model is no longer
as important, but is retained since it could be useful for self-hosters.
* update long context model tests
Summary:
Adds machinery to support audit logging in the backend.
Logging is currently implemented by streaming events to external HTTP
endpoints. All flavors of Grist support a default "grist" payload format,
and Grist Enterprise additionally supports an HEC-compatible payload format.
Logging of all audit events will be added at a later date.
Test Plan: Server tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D4331
Summary:
Permissions for admin billing endpoints were changed
- Support user can't use admin subscription endpoints
- Installation admin (as support user) can see billing details on any site
- Installation admin (unlike support user) can replace subscription (or attach payment) on any site, regardless permissions
Installation admin is any user that belongs to a special `admin` org. If `admin` org is not defined, it defaults to
support user. In that case, with this diff, the support user receives admin's permissions, and now can replace subscription on
any site (without being billing manager).
Test Plan: Added new test
Reviewers: dsagal, paulfitz
Reviewed By: dsagal, paulfitz
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D4338
Summary:
Adding support for 2-way references in data engine.
- Columns have an `reverseCol` field, which says "this is a reverse of the given column, update me when that one changes".
- At the time of setting `reverseCol`, we ensure that it's symmetrical to make a 2-way reference.
- Elsewhere we just implement syncing in one direction:
- When `reverseCol` is present, user code is generated with a type like `grist.ReferenceList("Tasks", reverse_of="Assignee")`
- On updating a ref column, we use `prepare_new_values()` method to generate corresponding updates to any column that's a reverse of it.
- The `prepare_new_values()` approach is extended to support this.
- We don't add (or remove) any mappings between rows, and rely on existing mappings (in a ref column's `_relation`) to create reverse updates.
NOTE This is polished version of https://phab.getgrist.com/D4307 with tests and 3 bug fixes
- Column transformation didn't work when transforming RefList to Ref, the reverse column became out of sync
- Tables with reverse columns couldn't be removed
- Setting json arrays to RefList didn't work if arrays contained other things besides ints
Those fixes are covered by new tests.
Test Plan: New tests
Reviewers: georgegevoian, paulfitz, dsagal
Reviewed By: georgegevoian, paulfitz
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D4322
Summary:
Removing `createNewConnection` method that was used in tests to create a
"scoped" version of `HomeDbManager`. Currently this won't work as there are
many methods (like `Users.findOne`) that are using the default (global) connection.
Additionally `HomeDBManger` had couple of bugs that were causing locks, which
manifested themselves in postgresql tests (that are not serializing transactions).
Repository methods like `Users.findOne` or `user.save()`, even when wrapped in
transaction were using a separate connection from the pool (and a separate
transaction).
Some tests in `UsersManager` are still skipped or refactored, as sinon's `fakeTimers`
doesn't work well with postgresql driver (which is using `setTimout` a lot).
Date mappings in `User` entity were fixed, they were using `SQLite` configuration only,
which caused problems with postgresql database.
Test Plan: Refactored.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4342
Summary:
Billing managers can now add other billing managers. Before
only managers with owner access could do that.
Test Plan: Added test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4339
Context
HomeDBManager lacks of direct tests, which makes hard to make rework or refactorations.
Proposed solution
Specifically here, I introduce tests which call exposed UsersManager methods directly and check their result.
Also:
I removed updateUserName which seems to me useless (updateUser does the same work)
Taking a look at the getUserByLogin methods, it appears that Typescirpt infers it returns a Promise<User|null> while in no case it may resolve a nullish value, therefore I have forced to return a Promise<User> and have changed the call sites to reflect the change.
Related issues
I make this change for then working on #870
Summary:
Addresses request https://github.com/gristlabs/grist-core/issues/443
The bootstrap-datepicker used for the Date/DateTime dropdown calendar does
support a number of locales. This diff loads them on-demand, if available,
based on the document's locale setting.
Also:
- Improves NewBaseEditor typings, reduces some casts, adds comments.
- Converts DateEditor and DateTimeEditor to typescript.
- Moves DateEditor nbrowser test to core.
Test Plan: Added a test case for locales to DateEditor test.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D4335
Summary:
There is a range of types that -- when returned from formulas -- used to cause
bad errors (that looked like a data engine crash and were reported as "Memory
Error") because they looked like primitive types but were not marshallable. For
example, IntEnum.
We now encode such values as the primitive type they are based on.
Test Plan:
- Added a unittest that encode_object() now handles problematic values.
- Added a browser test case that problematic values are no longer causing errors.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D4336
Summary:
Before this change we would always say there are 14 days remaining,
regardless of how many actually are remaining. Let's pass around a
different `dataLimitsInfo` object that also reports the number of days
remaining.
Test Plan: Ensure the test suite passes.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4332
Add body in log requests.
GRIST_LOG_SKIP_HTTP is a badly named environment variable and its
expected values are confusing (to log the requests, you actually have to
set its value to "", and setting to "false" actually is equivalent to
setting to "true").
We deprecate this env variable in favor of GRIST_LOG_HTTP which is more
convenient and understandable:
- by default, its undefined, so nothing is logged;
- to enable the logs, you just have to set GRIST_LOG_HTTP=true
Also this commit removes the default value for GRIST_LOG_SKIP_HTTP,
because we don't have to set it to "true" to actually disable the
requests logging thanks to GRIST_LOG_HTTP. FlexServer now handles
the historical behavior for this deprecated variable.
---------
Co-authored-by: Jonathan Perret <j-github@jonathanperret.net>
Summary:
Text columns can now display their values as Markdown-formatted text
by changing their cell format to "Markdown". A minimal subset of the
Markdown specification is currently supported.
Test Plan: Browser tests.
Reviewers: Spoffy, dsagal
Reviewed By: Spoffy, dsagal
Subscribers: dsagal, Spoffy
Differential Revision: https://phab.getgrist.com/D4326
Summary:
This includes two fixes: one to ensure that any exception from websocket
upgrade handlers are handled (by destroying the socket). A test case is
added for this.
The other is to ensure verifyClient returns false instead of failing; this
should lead to a better error to the client (Forbidden, rather than just socket
close). This is only tested manually with a curl request.
Test Plan: Added a test case for the more sensitive half of the fix.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: georgegevoian
Differential Revision: https://phab.getgrist.com/D4323
Summary:
These are tests that we just never moved into the public
repo. It's just a small chore to make them public.
Test Plan: Make sure the tests still pass
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D4311
Summary:
Custom widgets are now shown in a gallery.
The gallery is automatically opened when a new custom widget is
added to a page.
Descriptions, authors, and update times are pulled from the widget
manifest.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D4309
Summary:
Page is now hidden when any of its ancestor (or the page itself)
is censored.
Test Plan: Updated
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4319
* Introduces new configuration variables for OIDC:
- GRIST_OIDC_IDP_ENABLED_PROTECTIONS
- GRIST_OIDC_IDP_ACR_VALUES
- GRIST_OIDC_IDP_EXTRA_CLIENT_METADATA
* Implements all supported protections in oidc/Protections.ts
* Includes a better error page for failed OIDC logins
* Includes some other improvements, e.g. to logging, to OIDC
* Adds a large unit test for OIDCConfig
* Adds support for SERVER_NODE_OPTIONS for running tests
* Adds to documentation/develop.md info about GREP_TESTS, VERBOSE, and SERVER_NODE_OPTIONS.
For #1140, I considered trying to use the existing fields in a better
way, but because we already use the activations table to store
preferences, we need to keep all of the existing data and its usage
as-is.
The enterprise code will use this new column to decide how long the
trial period should be.
Summary:
When last widget for a table is removed, user is informed
about that and can decide between removing the widget and removing
both table and widget
Test Plan: Updated
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4295
Access control for ConvertFromColumn in the presence of access rules had previously been left as a TODO. This change allows the action when the user has schema rights. Because schema rights let you create formulas, they let you read anything, so there is currently no value in nuance here.
Summary:
A new onboarding page is now shown to all new users visiting the doc
menu for the first time. Tutorial cards on the doc menu have been
replaced with a new version that tracks completion progress, alongside
a new card that opens the orientation video.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D4296
Summary: Forcing category xaxis type for bar chart when labels are not numerical.
Test Plan: Added new and updated existing
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D4297
Summary:
Move an important set of tests that were in our SaaS
repo for no good reason.
Test Plan: moving tests
Reviewers: jordigh
Reviewed By: jordigh
Differential Revision: https://phab.getgrist.com/D4300
This adds a config file that's loaded very early on during startup.
It enables us to save/load settings from within Grist's admin panel, that affect the startup of the FlexServer.
The config file loading:
- Is type-safe,
- Validates the config file on startup
- Provides a path to upgrade to future versions.
It should be extensible from other versions of Grist (such as desktop), by overriding `getGlobalConfig` in stubs.
----
Some minor refactors needed to occur to make this possible. This includes:
- Extracting config loading into its own module (out of FlexServer).
- Cleaning up the `loadConfig` function in FlexServer into `loadLoginSystem` (which is what its main purpose was before).
Summary:
Some editors do some async work before saving the value (Ref column can add new
records). Those actions were send without bundling, so it wasn't possible to undo those
actions with togheter.
Test Plan: Added new test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4285
Summary:
- Adding confirmation dialog when user doesn't want to cancel site
- Changing `Cancel subscription` to `Cancel plan`
- Removing `Pro` from upgrade header on pricing modal
- Better handling situation when there is no default price
- Removing mentions about sprouts program
- Removing cache for stripe plans
Test Plan: Updated tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4273
Summary:
Version API endpoint wasn't logging telemetry from POST requests. The issue was in registration
order, this endpoint was registered before `expressJson` and it couldn't read json body in the handler.
Test Plan: Added new test
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4277
Summary:
- Fixing port allocation in TestServer
- Extending logging in the Billing test
- Fixing negative rowIds support for add/remove actions
- Making FormulaEditor and CardView tests less flacky
Test Plan: Existing
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz, dsagal
Differential Revision: https://phab.getgrist.com/D4280
Summary:
For non-owners, the timing section of Document Settings is now disabled.
For non-editors, the "Reload" section is disabled.
Test Plan: Added a test case for timing being disabled.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D4275
Summary:
After release on 2024-06-12 (1.1.15) the GRIST_DEFAULT_PRODUCT env variable wasn't respected by the
method that started the server in single org mode. In all deployments (apart from saas), the default product
used for new sites is set to `Free`, but the code that starts the server enforced `teamFree` product.
This change adds a fix routine that fixes this issue by rewriting team sites from `teamFree` product to `Free`
product only if:
- The default product is set to `Free`
- The deployment type is something other then 'saas'.
Additionally there is a test that will fail after 2024.10.01, as this fix should be removed before this date.
Test Plan: Added test
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4272
This fixes the flaky test in "ViewLayoutCollapse.ts": "fix: should not dispose the instance when drag is cancelled".
The 'mouseenter' event wasn't consistently triggering properly on the drop target (LayoutEditor.ts - line 342) when the mouse was moved onto it.
The change simulates a "drag" over the drop target, moving the mouse into multiple positions over it, seemingly fixing the problem.
Paul Fitzpatrick
George Gevoian
Jarosław Sadziński
Florent
Dmitry S
George Gevoian
Jordi Gutiérrez Hermoso
Grégoire Cutzach
Spoffy
CamilleLegeron