Commit Graph

1784 Commits

Author SHA1 Message Date
George Gevoian
5e26401a24 (core) Re-render assistant code blocks on theme change
Summary:
When the browser theme transitioned while the assistant was open, messages
containing code blocks were not being re-rendered with the new theme.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3965
2023-07-24 19:09:01 -04:00
Alex Hall
391c8ee087 (core) Allow assistant to evaluate current formula
Summary:
Replaces https://phab.getgrist.com/D3940, particularly to avoid doing potentially unwanted things automatically.

Adds optional fields `evaluateCurrentFormula?: boolean; rowId?: number` to `FormulaAssistanceContext` (part of `AssistanceRequest`). When `evaluateCurrentFormula` is `true`, calls a new function `evaluate_formula` in the sandbox which computes the existing formula in the column (regardless of anything the AI may have suggested) and uses that to generate an additional system message which is added before the user's message. In theory this could be used in an interface where users ask why a formula doesn't work, including possibly a formula suggested by the AI. For now, it's only used in `runCompletion_impl.ts` for experimenting.

Also cleaned up a bit, removing `_chatMode` which is always `true` now, and uses of `regenerate` which is always `false`.

Test Plan: Updated `runCompletion_impl` to optionally use the new feature, in which case it now scores 51/68 instead of 49/68.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3970
2023-07-24 21:59:00 +02:00
Paul Fitzpatrick
14b14f116e (core) export WindowDimensions from gristUtils after refactor
Summary:
WindowDimensions from gristUtils needs exporting
after it got moved to another file.

Test Plan: existing tests should pass

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3971
2023-07-24 15:10:51 -04:00
Paul Fitzpatrick
8ecf0d7d0a
change order of new private method to fix lint after saas/core sync (#587) 2023-07-23 14:51:19 -04:00
Paul Fitzpatrick
7256e0c245 (core) updates from grist-core 2023-07-23 11:53:20 -04:00
Camille L
ed649e12c5
Translated using Weblate (French)
Currently translated at 88.9% (825 of 927 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/fr/
2023-07-22 16:02:28 +02:00
RaywanS
61dc8bd646
Translated using Weblate (French)
Currently translated at 88.9% (825 of 927 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/fr/
2023-07-22 16:02:28 +02:00
George Gevoian
90cf018133 (core) Update assistant link
Summary: Updates the "Learn more" link now that the page is published.

Test Plan: N/A

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3969
2023-07-21 13:43:31 -04:00
Paul Fitzpatrick
b054594840
factor out low-dependency browser test code useful for grist-widget (#576)
It would be useful to write browser tests that use Grist for some
of our other repositories (e.g. grist-widget, grist-static).
This is a first baby step to factor out some useful code that has
no code dependencies beyond mocha-webdriver, for use in grist-widget.
2023-07-21 10:05:43 -04:00
Владимир В
c7d9975e35
Translated using Weblate (Russian)
Currently translated at 99.5% (903 of 907 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/ru/
2023-07-21 07:07:05 +02:00
Paul Janzen
b821bdcd9a
Translated using Weblate (Portuguese (Brazil))
Currently translated at 100.0% (907 of 907 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/pt_BR/
2023-07-21 07:07:05 +02:00
Camille L
745f4517a9
Translated using Weblate (French)
Currently translated at 90.6% (822 of 907 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/fr/
2023-07-21 07:07:05 +02:00
RaywanS
59c45887ea
Translated using Weblate (French)
Currently translated at 90.6% (822 of 907 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/fr/
2023-07-21 07:07:05 +02:00
Alex Hall
0469a98c08 (core) Document configuring AI assistance
Summary:
Added OPENAI_API_KEY and GRIST_FORMULA_ASSISTANT to the README. GRIST_FORMULA_ASSISTANT may be removed in the long term, but for now the goal is just to get something into the grist-core README quickly for self-hosters.

Removed `documentation/llm.md` because it's outdated and not really providing value.

Test Plan: none

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3963
2023-07-20 21:29:59 +02:00
George Gevoian
788a6d01ce (core) Move CellColor tests to grist-core
Test Plan: N/A

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3964
2023-07-20 15:17:10 -04:00
George Gevoian
beffd02c41 (core) Improve highlighting of previewed formula
Summary:
Also improves highlighting of columns when the "Click to insert"
tooltip is shown, and improves highlighting of transforming columns.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3962
2023-07-20 14:34:22 -04:00
Alex Hall
5a703a1972 (core) Send hash of user ID in OpenAI API requests
Summary: Following recommendation in https://platform.openai.com/docs/guides/safety-best-practices/end-user-ids

Test Plan: Checked that running server test shows log of hash of 'user id' (which is null because it's a fake session)

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz, georgegevoian

Differential Revision: https://phab.getgrist.com/D3958
2023-07-20 19:50:41 +02:00
George Gevoian
0a34292536 (core) Add telemetry for AI Assistant
Summary: Also fixes a few bugs with some telemetry events not being recorded.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3960
2023-07-20 12:50:26 -04:00
Florent
5e33b68753
Issue 359 support scaleway (#577)
* Fix support of Scaleway S3 bucket #359

While MinIO and AWS return versionId as strings, other S3 API
implementations return versionId as integers.

We must carefully convert the versionId as string in order to cover
these various behaviors.

Also ensure that docStorage is initialized before attempting to
calculate the data size in order to avoid an exception.

* Add unit tests for MinIOExternalStorage#versions() #359

Introduced some unit tests to :
 - ensure listObjects is called with the right arguments;
 - cover the case when a S3 bucket implementation does not return the
   versionId as a string but rather as an integer (like Scaleway):
   in such a case, ensure that the returned snapshotId is a string;
 - cover the case when the listObjects function emits an error, ensure the
   versions() call rejets with the error emitted;
 - that the deleteMarkers are only returned when the
   includeDeleteMarkers is passed;

---------

Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-07-20 06:16:58 -04:00
George Gevoian
0040716006 (core) Disable undo/redo in detached formula editor
Summary:
Undo and redo were causing errors to be thrown while the editor was detached. In the
interest of time, we'll disable undo/redo until we have a chance to look at whether
we can support it in the editor.

Test Plan: Manual.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Differential Revision: https://phab.getgrist.com/D3959
2023-07-19 11:39:14 -04:00
github-actions[bot]
f1e8cba57f
automated update to translation keys (#575)
Co-authored-by: Paul's Grist Bot <paul+bot@getgrist.com>
2023-07-19 09:42:20 -04:00
Camille L
3a247c21a2
Translated using Weblate (French)
Currently translated at 95.6% (817 of 854 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/fr/
2023-07-19 12:08:35 +02:00
Dmitry S
c0ed4a8a60 (core) Use a new docker image for tests in Jenkins
Summary:
The image Jenkins used up to now was from 2021. We updated buildtools/jenkins-ec2/Dockerfile since then but couldn't use new image; we can now that an issue with mounts in gvisor has been resolved.

The new image uses newer Chrome (111 vs 95); 111 is the latest version that works with the webdriver version we are using (after that one, copy-pasting tests fail).

In addition, this adds a more precise way to maintain python dependencies: by specifying top-level dependencies in the new core/sandbox/requirements3.in. An updated build step compiles that into requirements3.txt, and syncs venv to that using pip-sync.

This addresses the issue that previously, removing a module from dependencies would not have caused the build to remove it from sandbox_venv3.

Test Plan: Existing tests pass on the new image.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3952
2023-07-19 00:54:14 -04:00
Dmitry S
534615dd50 (core) Update logging in sandbox code, and log tracebacks as single log messages.
Summary:
- Replace logger module by the standard module 'logging'.
- When a log message from the sandbox includes newlines (e.g. for tracebacks),
  keep those lines together in the Node log message.

  Previously each line was a different message, making it difficult to view
  tracebacks, particularly in prod where each line becomes a separate message
  object.

- Fix assorted lint errors.

Test Plan: Added a test for the log-line splitting and escaping logic.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3956
2023-07-18 11:21:25 -04:00
Alex Hall
7fd48364df (core) Improved error messages, retries, and handling of token limits in AI assistant
Summary:
In a nutshell:

- More specific and helpful error messages are shown to the user
- API requests are only retried when needed
- The system deals with reaching the maximum token limit better, especially by switching to a model with a bigger limit

In more detail:

- `COMPLETION_MODEL` configuration has been removed. By default `gpt-3.5-turbo-0613` is used which accepts 4k tokens. If that's not enough, `gpt-3.5-turbo-16k-0613` is used instead.
- Switching to the bigger model happens when either the prompt is too long by itself (the API immediately returns an error code) or the model reaches the 4k limit itself in the process of generating a response and thus returns an incomplete response. The latter case is made possible by removing the `max_tokens: 1500` in the request, which was very generous and would have lead to switching to the more expensive model more often than needed. The downside is that the user has to wait a bit longer for the response.
- If the bigger 16k token limit is also exceeded, the assistant immediately responds (instead of retrying as before) with an error message including suggestions. The suggestions include restarting the conversation if and only if the user has sent multiple messages.
- If a request fails because Grist has reached its OpenAI monthly billing quota, the assistant immediately responds (instead of retrying as before) with an error message suggesting that the user try again tomorrow.
- If a request fails for some other reason, the assistant retries, and if all attempts fail then the user is told to try again in a few minutes and is shown the exact error message, including the API response if there is one.
- Retrying only happens when an API request fails, whereas previously the system also retried errors from a much bigger scope which included calls to the sandbox. The downside is that the hugging face assistant no longer retries, although that code is currently disabled anyway.
- The assistant no longer waits an additional second after the final retry attempt fails.

Test Plan: Added a new server test file with several unit tests using faked OpenAI responses, including the happy path which wasn't really tested before.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3955
2023-07-18 16:01:37 +02:00
Jakub Serafin
d894b60fd4 (core) deleting queue from single webhook
Summary: Using standard tost notification, message about webhook queue being overflown was added. message is permanent as long as queue is full. Message contains linkt to the webhook setings

Test Plan: two nbrowser test was added - one to check if message is show when queue is full, and second to check if message is dismiss when queue was cleaned.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3929
2023-07-18 11:46:10 +02:00
Paul Fitzpatrick
450472f74c (core) updates from grist-core 2023-07-17 01:22:18 -04:00
George Gevoian
e208f827af (core) Add documentation to grist-core
Summary: Adds some documentation about Grist's components and infrastructure.

Test Plan: N/A

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3941
2023-07-16 22:27:12 -04:00
CamilleLegeron
61bd064f73
i18n: userManager translation + some forgotten translations (#557)
* translation: add userManager translation + some forgotten translations
* use '\' caracter for multiple-line strings
2023-07-16 12:52:13 -04:00
Riccardo Polignieri
b4b0c805ff
Translated using Weblate (Italian)
Currently translated at 100.0% (854 of 854 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/it/
2023-07-15 21:51:59 +02:00
Paul Janzen
1117886e15
Translated using Weblate (German)
Currently translated at 100.0% (854 of 854 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/de/
2023-07-15 21:51:58 +02:00
Paul Janzen
fa891bf29c
Translated using Weblate (Spanish)
Currently translated at 100.0% (854 of 854 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/es/
2023-07-15 21:51:58 +02:00
Paul Janzen
5f54082829
Translated using Weblate (Portuguese (Brazil))
Currently translated at 100.0% (854 of 854 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/pt_BR/
2023-07-15 21:51:57 +02:00
Paul Fitzpatrick
a26eef05b0 (core) remove superfluous import
Summary:
Remove a duplicate import, perhaps introduced during merge of
a relatively long-lived branch.

Test Plan: existing tests should pass

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3953
2023-07-14 12:22:56 -04:00
Jakub Serafin
a9f4cfde90 (core) API reworked to use POST to create webhook and DELET to remove it
Summary:
introduces POST /api/docs/{docId}/webhooks and DELETE /api/docs/{docId}/webhooks/{webhookId} on place of old _subscribe and _unsubscribe endpoints.
Remove checking for unsubscribeKey while deleting webhook - only owner can delete webhook using DELETE endpoint. subscription key is still needed for _unsubscribe endpoint.
old _unsubscribe and _subscribe endpoints are still active and work as before - no changes there.

Posting schema:

```
POST /api/docs/[docId]/webhooks
```

Request Body:

```
{
    "webhooks": [
        {
            "fields": {
                "url": "https://webhook.site/3bd02246-f122-445e-ba7f-bf5ea5bb6eb1",
                "eventTypes": [
                    "add",
                    "update"
                ],
                "enabled": true,
                "name": "WebhookName",
                "memo": "just a text",
                "tableId": "Table1"
            }
        },
        {
            "fields": {
                "url": "https://webhook.site/3bd02246-f122-445e-ba7f-bf5ea5bb6eb2",
                "eventTypes": [
                    "add",
                ],
                "enabled": true,
                "name": "OtherWebhookName",
                "memo": "just a text",
                "tableId": "Table1"
            }
        }
    ]
}
```

Expected response: WebhookId for each webhook posted:

```
{
    "webhooks": [
        {
            "id": "85c77108-f1e1-4217-a50d-acd1c5996da2"
        },
        {
            "id": "d87a6402-cfd7-4822-878c-657308fcc8c3"
        }
    ]
}
```

Deleting webhooks:

```
DELETE api/docs/[docId]/webhooks/[webhookId]
```

there is no payload in DELETE request. Therefore only one webhook can be deleted at once

Response:

```
{
    "success": true
}
```

Test Plan: Old unit test improved to handle new endpoints, and one more added to check if endpoints are in fact created/removed

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: paulfitz, alexmojaki

Differential Revision: https://phab.getgrist.com/D3916
2023-07-14 15:01:46 +02:00
Paul Janzen
3bff145d48
Translated using Weblate (Portuguese)
Currently translated at 100.0% (823 of 823 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/pt/
2023-07-13 18:50:04 +02:00
George Gevoian
ea8a59c5e9 (core) Implement AI Assistant UI V2
Summary:
Implements the latest design of the Formula AI Assistant.

Also switches out brace to the latest build of ace.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3949
2023-07-13 10:30:35 -04:00
Florent
7694588a42
External storage: split checkBackend and configure (follow-up #545) (#567)
This fixes an issue with external storage in saas environment.

See https://github.com/gristlabs/grist-core/pull/546/files#r1259340397

Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-07-13 02:44:46 -04:00
Paul Fitzpatrick
b0f76a152f
add more docker label choices (#570) 2023-07-12 22:31:00 +01:00
Dmitry S
8581492912 (core) When a page starts with a number, don't treat it as an emoji
Test Plan: Added a check to the emoji test.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3951
2023-07-12 15:13:25 -04:00
Alex Hall
152dc832f1
Split out new importFileAsNewTable method for grist-static (#564)
Also add column types to Limit entity to fix errors.
2023-07-12 15:57:02 +02:00
Paul Fitzpatrick
958ea096f3
fix a node-sqlite3-ism that breaks record removal in grist-static (#566)
Grist by default uses node-sqlite3 to manipulate data in an
SQLite database. If a single parameter is passed to `run`
and it is a list, the list is unpacked and its contents treated
as the actual parameters. In grist-static, we use other SQLite
interfaces that don't have that automatic unpacking. Most
calls like this have been removed from Grist, but at least one
was missed, and was causing symptoms such as
https://github.com/gristlabs/grist-static/issues/5

This change should make no difference to regular Grist, but
resolves the grist-static problems.
2023-07-11 05:52:06 -04:00
Paul Fitzpatrick
b2743cac2d (core) updates from grist-core 2023-07-11 05:41:02 -04:00
Paul Fitzpatrick
733b6b3d29 (core) add back config call for external storage
Summary:
add back config call for external storage
in the right order.

Test Plan: existing saas tests catch this

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3946
2023-07-10 14:16:48 -04:00
github-actions[bot]
b2d06bdba5
automated update to translation keys (#563)
Co-authored-by: Paul's Grist Bot <paul+bot@getgrist.com>
2023-07-10 08:48:13 -04:00
Paul Fitzpatrick
2c7ad727d1 (core) resolved some divergence in mergedServerMain 2023-07-10 07:50:52 -04:00
Jarosław Sadziński
d13b9b9019 (core) Billing for formula assistant
Summary:
Adding limits for AI calls and connecting those limits with a Stripe Account.

- New table in homedb called `limits`
- All calls to the AI are not routed through DocApi and measured.
- All products now contain a special key `assistantLimit`, with a default value 0
- Limit is reset every time the subscription has changed its period
- The billing page is updated with two new options that describe the AI plan
- There is a new popup that allows the user to upgrade to a higher plan
- Tiers are read directly from the Stripe product with a volume pricing model

Test Plan: Updated and added

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3907
2023-07-10 13:24:08 +02:00
Florent
b6b2d05be0
Abort when MinIO bucket does not have versioning enabled #545 (#546)
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-07-10 06:24:55 -04:00
Florent
a56b0448ff
Fix DEBUG not accepting numbers #559 (#561)
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-07-07 18:17:31 -04:00
Jarosław Sadziński
75d979abdb (core) Fixing cursor position for filtered linked section.
Summary:
In a selector table, when a selected row is filtered out of view, linked widgets should update based on the newly selected row.
There were a few bugs that contributed to this wrong behavior:
- Gridview wasn't subscribing to the current row id, and the row with id 'new' was being converted to the first row
- Cursor was keeping track of the currently selected row id, it was hiding a problem behind the proper rowIndex
- Undo/redo somehow leveraged the wrong rowId from the cursor during the position restore.

The `No data` text was also changed to be more meaningful.

Test Plan: Added and updated.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3937
2023-07-07 19:04:30 +02:00