Commit Graph

1195 Commits

Author SHA1 Message Date
Alex Hall
a1d31e41ad (core) Prevent the AI assistant response from including class definitions
Summary:
Sometimes the model repeats the classes given in the prompt which would mess up extracting the actual formula. This diff solves this by:

1. Changes the generated Python schema so that (a) the thing that needs completing is a plain top level function instead of a property/method inside the class and (2) the classes are fully valid syntax, which makes it easier to
2. Remove classes from the parsed Python code when converting the completion to a formula.
3. Tweak the prompt wording to discourage including classes in general, especially because sometimes the model tries to solve the problem by defining extra methods/attributes/classes.

While I was at it, I changed type hints to use builtins (e.g. `list` instead of `List`) to prevent `from typing import List` which was happening sometimes and would look weird in a formula. Similarly I removed `@dataclass` since that also implies an import, and this also fits with the tweaked wording that the classes are fake.

Test Plan:
Added a new test case to the formula dataset which triggers the unwanted behaviour. The factors that seem to trigger the problem are (1) a small schema so the classes are easier to repeat and (2) the need to import modules, which the model wants to place before all other code. The case failed before this diff and succeeded after. The tweaked wording reduces the chances of repeating the classes but didn't eliminate it, so forcibly removing the classes in Python was needed.

There were also a couple of other existing cases where repeating the classes was observed before but not after.

Overall the score increased from 49 to 51 out of 69 (including the new case). At one point the score was 53, but changes in whitespace were enough to make it drop again.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4000
2023-08-18 12:50:09 +02:00
Alex Hall
999d723d14 (core) Cleanup old frontend code for handling empty summary groups
Summary: The removed TODO was resolved in https://phab.getgrist.com/D3489

Test Plan: Existing tests

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4006
2023-08-18 12:46:15 +02:00
Paul Fitzpatrick
fe846d3068 (core) allow copying errors from regular cell editor
Summary:
There was already some support for copying errors from the detached/expanded formula editor. This adds support for copying errors in the regular cell editor.

Getting error details is now done only by clicking on the expand icon - previously you could click on the error text itself.

A few unrelated test changes are made for exact-pixel checks that were often out by approx 1 pixel on my machine.

Test Plan: Updated tests. Tested copying/pasting manually.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4002
2023-08-17 09:35:22 -04:00
Jarosław Sadziński
fad421b7c0 (core) Removing temporary pro site
Summary: Creating a pro team site after Stripe checkout. Previously a stub site was always created and never removed, even if a user cancels the checkout process, which resulted in multiple 'ghost' sites that can't be removed.

Test Plan: Updated and added

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3985
2023-08-16 08:11:26 +02:00
Paul Fitzpatrick
f1a0b61e15 (core) updates from grist-core 2023-08-15 12:15:16 -04:00
Philip Standt
cbdffdfff8
Support grid selection with Ctrl+Shift+Arrow (#615) 2023-08-14 09:28:41 -07:00
Florent
fde56ffa33
DocApi: Introduce hidden option for GET on records (#623)
Also test that PUT /columns?replaceall=1 does not remove hidden columns
2023-08-14 16:17:46 +02:00
Florent
e93ad5a0c5
Issue #601: implement PUT on columns (#605)
* Also Add includeHidden param for GET on columns
2023-08-11 15:12:43 +02:00
George Gevoian
0f2b1f6a89 (core) Polish "Click to insert $id" tooltip
Summary:
The tooltip also broke recently on Firefox after upgrading Ace. This
includes a fix for the regression.

Test Plan: Updated test.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek, dsagal

Differential Revision: https://phab.getgrist.com/D3982
2023-08-09 07:38:34 -07:00
Jarosław Sadziński
732611c356 (core) Removing GRIST_FORMULA_ASSISTANT flag
Summary:
A floating formula editor is available by default and in the basic setup allows just formula modification.
AI assistant is now an optional component of the floating editor and it is controlled by OPENAPI_KEY presence.
Env variable GRIST_FORMULA_ASSISTANT was removed, new feature flag HAS_FORMULA_ASSISTANT is derived from the presence of OPENAPI_KEY.

Also updated anonymous signup nudge. By default it displays only info that this feature is only for logged in users.

Test Plan: updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3987
2023-08-09 10:08:18 +02:00
George Gevoian
24bbf375f9 (core) Add earliestDocCreatedAt to telemetry
Summary: This tracks the earliest document creation time, if any, for each site.

Test Plan: Server tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3991
2023-08-08 00:48:18 -07:00
Dmitry S
efb5fc63d7 (core) Change the label/nickname of the paid plan to Pro
Summary:
This is to match the pricing page, and to reduce confusion. The same
change was made to the description of the corresponding "pricing" object
on Stripe. The Stripe value is what's used in practice; this value is
actually only used as a fallback.

Test Plan: The code change does not affect tests, but several tests fixed after grist-core merge.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3990
2023-08-07 23:45:18 -04:00
Paul Fitzpatrick
031076cd07 (core) updates from grist-core 2023-08-07 14:29:30 -04:00
CamilleLegeron
02841bd15c
Header colored (#581) 2023-08-07 11:01:35 -07:00
Dmitry S
526a5df157 (core) Manage memory used for websocket responses to reduce the risk of server crashes.
Summary:
- Implements MemoryPool for waiting on memory reservations.
- Uses MemoryPool to control memory used for stringifying JSON responses in Client.ts
- Limits total size of _missedMessages that may be queued for a particular client.
- Upgrades ws library, which may reduce memory usage, and allows pausing the websocket for testing.
  - The upgrade changed subtle behavior corners, requiring various fixes to code and tests.

- dos.ts:
  - Includes Paul's fixes and updates to the dos.ts script for manual stress-testing.
  - Logging tweaks, to avoid excessive dumps on uncaughtError, and include timestamps.

Test Plan:
- Includes a test that measures heap size, and fails without memory management.
- Includes a unittest for MemoryPool
- Some cleanup and additions to TestServer helper; in particular adds makeUserApi() helper used in multiple tests.
- Some fixes related to ws upgrade.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3974
2023-08-07 11:28:31 -04:00
Paul Fitzpatrick
016ab66ea0
fix type issue if python is missing when running unsandboxed (#610)
Clean up a small recently introduced type issue.
2023-08-05 11:24:01 -04:00
Jarosław Sadziński
00b88fd683 (core) Adding links to description tooltips
Summary: Column and widget descriptions now support links in text.

Test Plan: Updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3981
2023-08-04 18:34:04 +02:00
Alex Hall
5dfa9a542c (core) Upgrade to Python 3.11
Summary: Replaced mentions of Python 3.9 with 3.11

Test Plan: this

Reviewers: paulfitz, georgegevoian

Reviewed By: paulfitz, georgegevoian

Subscribers: dsagal, georgegevoian, paulfitz

Differential Revision: https://phab.getgrist.com/D3980
2023-08-04 18:19:40 +02:00
Jarosław Sadziński
6416994c22 (core) Import redesign
Summary: New UI design for incremental imports.

Test Plan: Updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3945
2023-08-04 14:59:55 +02:00
George Gevoian
05c15e4ec3 (core) Show tweaked formula in AI responses
Summary:
The formula that's used when the Apply button is clicked, and the formula that's
shown in responses from the Formula Assistant should now be the same. Previously, they
would differ slightly.

Test Plan: Server tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3977
2023-08-04 01:31:31 -07:00
Jarosław Sadziński
4cfa033078 (core) TypeTransform race condition fix
Summary:
TypeTransformation was flaky. Probably after upgrading AceEditor we introduced a race condition between updating the revised formula and doing the transformation. Now we explicitly make sure that the formula is updated.

I also fixed some other flaky tests.

Test Plan: Updated

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3984
2023-08-02 17:11:34 +02:00
Paul Fitzpatrick
8110a26873 (core) updates from grist-core 2023-08-01 10:52:34 -04:00
Paul Fitzpatrick
61f954ff05
move getTemplateOrg method; enable template org in docker tests (#602)
* move getTemplateOrg method; enable template org in docker tests

This moves the `getTemplateOrg` method to a neutral venue for the
convenience of `grist-static`, otherwise a lot of awkward dependencies
get pulled in needlessly in new parts of the app.

This also fixes docker tests using the template org.
2023-07-31 16:10:59 -04:00
George Gevoian
9484fa14db (core) Fix mislabeled telemetry metadata
Summary:
The oldFormula and newFormula metadata fields had their names switched by
mistake.

Test Plan: N/A

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Differential Revision: https://phab.getgrist.com/D3976
2023-07-31 00:55:28 -07:00
John Cant
e1df6039c2
REQUEST now supports POST (#588)
* REQUEST now supports POST
* Add extra flag for enabling REQUEST, also update README and comments

Co-authored-by: John Cant <a.jonncant@gmail.com>
Co-authored-by: Alex Hall <alex.mojaki@gmail.com>
2023-07-30 15:13:43 -04:00
Alex Hall
9b87a6f06a (core) updates from grist-core 2023-07-28 17:40:31 +02:00
Jakub Serafin
d4c5082799
reverting whitespace changes in DocPageModel (#594) 2023-07-27 14:36:43 +02:00
Paul Fitzpatrick
f0b9e1f7e9
allow Grist front-end to function when location history is unavailable (#596)
* allow Grist front-end to function when location history is unavailable

When the Grist front-end is embedded in an iframe, using a srcdoc
attribute, history.pushState and similar methods are unavailable.
Currently, that makes it impossible to navigate between Grist pages,
since an access error is thrown (behavior may be browser dependent).
With this change, navigation succeeds.

* give unrelated possibly slow test a little more time
2023-07-27 07:02:17 -04:00
George Gevoian
a77170c4bd (core) Tweak navbar, breadcrumbs, and sign-in buttons
Summary:
The changes are intended to smooth over some sharp edges when a signed-out user
is using Grist (particularly while on the templates site).

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3957
2023-07-26 22:26:55 -07:00
Alex Hall
bc54a6646e (core) Filter rows based on linked widgets when exporting view
Summary:
Fixes a problem reported here: https://community.getgrist.com/t/exporting-the-records-in-a-linked-view/2556/4

The download CSV/Excel link now contains an additional `linkingFilter` URL parameter containing JSON-encoded `filters` and `operations`. This object is originally created in the frontend in `LinkingState`, and previously it was only used internally in the frontend. It would make its way via `QuerySetManager` to `QuerySet.getFilterFunc` where the actual filtering logic happened. Now most of that logic has been moved to a similar function in `common`. The new function works with a new interface `ColumnGettersByColId` which abstract over the different ways data is accessed in the client and server in this context. There's no significant new logic in the diff, just refactoring and wiring.

Test Plan: Expanded two `nbrowser/SelectBy*.ts` test suites to also check the contents of a downloaded CSV in different linking scenarios.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3961
2023-07-26 21:49:52 +02:00
Vincent Viers
ab6807c342
fix: don't count falses in summary (#589) 2023-07-26 12:08:04 -07:00
Alex Hall
3f71c9c488 (core) Cleanup: Remove unused AssistanceRequest.regenerate
Summary: Finish what was started in https://phab.getgrist.com/D3970

Test Plan: existing tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3972
2023-07-26 15:51:46 +02:00
Jakub Serafin
f7fdfab6bf (core) GET endpoint for webhooks returns now data in format {webhooks:[...]}
Summary:
Rework of endpoint GET  for webhooks to make it coherent with other endpoints. Now data should be return in {webhooks:[{id:"...",fields:{"..."}]} format

```
{
    "webhooks": [
        {
            "id": ...
            "fields": {
                "url": ...
                "unsubscribeKey": ...
                "eventTypes": [
                    "add",
                    "update"
                ],
                "isReadyColumn": null,
                "tableId": "...",
                "enabled": false,
                "name": "...",
                "memo": "..."
            },
            "usage": {
                "status": "idle",
                "numWaiting": 0,
                "lastEventBatch": null
            }
        },
        {
            "id": "...",
            "fields": {
                "url": "...",
                "unsubscribeKey": "...",
                "eventTypes": [
                    "add",
                    "update"
                ],
                "isReadyColumn": null,
                "tableId": "...",
                "enabled": true,
                "name": "...",
                "memo": "..."
            },
            "usage": {
                "status": "error",
                "numWaiting": 0,
                "updatedTime": 1689076978098,
                "lastEventBatch": {
                    "status": "rejected",
                    "httpStatus": 404,
                    "errorMessage": "{\"success\":false,\"error\":{\"message\":\"Alias 5a9bf6a8-4865-403a-bec6-b4ko not found\",\"id\":null}}",
                    "size": 49,
                    "attempts": 5
                },
                "lastSuccessTime": null,
                "lastFailureTime": 1689076978097,
                "lastErrorMessage": "{\"success\":false,\"error\":{\"message\":\"Alias 5a9bf6a8-4865-403a-bec6-b4ko not found\",\"id\":null}}",
                "lastHttpStatus": 404
            }
        }
    ]
}
```

Test Plan: new test added to check if GET data fromat is correct. Other tests fixed to handle changed endpoint.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3966
2023-07-26 11:36:24 +02:00
Paul Fitzpatrick
cd8eac36b8 (core) updates from grist-core 2023-07-25 17:14:17 -04:00
Janet Vorobyeva
b2e98230b4
Linux-specific fonts: Liberation Sans (#572)
* Added linux-specific fonts

Currently, ubuntu has a lot of issues with fonts.
Both the regular font-family and the data-font-family end up
evaluating to NimbusSans on my system
  (ubuntu's fallback font from helvetica)

NimbusSans unfortunately is noticably too high, cause text in buttons,
emoji on the left pane, and other vertical text alignment to be too high

This diff explicitly says to use Liberation Sans which should
have no effect on windows/mac systems, but should significantly improve
appearance on ubuntu (and hopefully other linuxes)

Both of these fonts are some of the more widely supported linux fonts,
see:
https://www.webfx.com/blog/web-design/a-web-designers-guide-to-linux-fonts/
2023-07-25 16:39:47 -04:00
George Gevoian
5e26401a24 (core) Re-render assistant code blocks on theme change
Summary:
When the browser theme transitioned while the assistant was open, messages
containing code blocks were not being re-rendered with the new theme.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3965
2023-07-24 19:09:01 -04:00
Alex Hall
391c8ee087 (core) Allow assistant to evaluate current formula
Summary:
Replaces https://phab.getgrist.com/D3940, particularly to avoid doing potentially unwanted things automatically.

Adds optional fields `evaluateCurrentFormula?: boolean; rowId?: number` to `FormulaAssistanceContext` (part of `AssistanceRequest`). When `evaluateCurrentFormula` is `true`, calls a new function `evaluate_formula` in the sandbox which computes the existing formula in the column (regardless of anything the AI may have suggested) and uses that to generate an additional system message which is added before the user's message. In theory this could be used in an interface where users ask why a formula doesn't work, including possibly a formula suggested by the AI. For now, it's only used in `runCompletion_impl.ts` for experimenting.

Also cleaned up a bit, removing `_chatMode` which is always `true` now, and uses of `regenerate` which is always `false`.

Test Plan: Updated `runCompletion_impl` to optionally use the new feature, in which case it now scores 51/68 instead of 49/68.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3970
2023-07-24 21:59:00 +02:00
Paul Fitzpatrick
8ecf0d7d0a
change order of new private method to fix lint after saas/core sync (#587) 2023-07-23 14:51:19 -04:00
Paul Fitzpatrick
7256e0c245 (core) updates from grist-core 2023-07-23 11:53:20 -04:00
George Gevoian
90cf018133 (core) Update assistant link
Summary: Updates the "Learn more" link now that the page is published.

Test Plan: N/A

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3969
2023-07-21 13:43:31 -04:00
George Gevoian
beffd02c41 (core) Improve highlighting of previewed formula
Summary:
Also improves highlighting of columns when the "Click to insert"
tooltip is shown, and improves highlighting of transforming columns.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3962
2023-07-20 14:34:22 -04:00
Alex Hall
5a703a1972 (core) Send hash of user ID in OpenAI API requests
Summary: Following recommendation in https://platform.openai.com/docs/guides/safety-best-practices/end-user-ids

Test Plan: Checked that running server test shows log of hash of 'user id' (which is null because it's a fake session)

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz, georgegevoian

Differential Revision: https://phab.getgrist.com/D3958
2023-07-20 19:50:41 +02:00
George Gevoian
0a34292536 (core) Add telemetry for AI Assistant
Summary: Also fixes a few bugs with some telemetry events not being recorded.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3960
2023-07-20 12:50:26 -04:00
Florent
5e33b68753
Issue 359 support scaleway (#577)
* Fix support of Scaleway S3 bucket #359

While MinIO and AWS return versionId as strings, other S3 API
implementations return versionId as integers.

We must carefully convert the versionId as string in order to cover
these various behaviors.

Also ensure that docStorage is initialized before attempting to
calculate the data size in order to avoid an exception.

* Add unit tests for MinIOExternalStorage#versions() #359

Introduced some unit tests to :
 - ensure listObjects is called with the right arguments;
 - cover the case when a S3 bucket implementation does not return the
   versionId as a string but rather as an integer (like Scaleway):
   in such a case, ensure that the returned snapshotId is a string;
 - cover the case when the listObjects function emits an error, ensure the
   versions() call rejets with the error emitted;
 - that the deleteMarkers are only returned when the
   includeDeleteMarkers is passed;

---------

Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-07-20 06:16:58 -04:00
George Gevoian
0040716006 (core) Disable undo/redo in detached formula editor
Summary:
Undo and redo were causing errors to be thrown while the editor was detached. In the
interest of time, we'll disable undo/redo until we have a chance to look at whether
we can support it in the editor.

Test Plan: Manual.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Differential Revision: https://phab.getgrist.com/D3959
2023-07-19 11:39:14 -04:00
Dmitry S
534615dd50 (core) Update logging in sandbox code, and log tracebacks as single log messages.
Summary:
- Replace logger module by the standard module 'logging'.
- When a log message from the sandbox includes newlines (e.g. for tracebacks),
  keep those lines together in the Node log message.

  Previously each line was a different message, making it difficult to view
  tracebacks, particularly in prod where each line becomes a separate message
  object.

- Fix assorted lint errors.

Test Plan: Added a test for the log-line splitting and escaping logic.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3956
2023-07-18 11:21:25 -04:00
Alex Hall
7fd48364df (core) Improved error messages, retries, and handling of token limits in AI assistant
Summary:
In a nutshell:

- More specific and helpful error messages are shown to the user
- API requests are only retried when needed
- The system deals with reaching the maximum token limit better, especially by switching to a model with a bigger limit

In more detail:

- `COMPLETION_MODEL` configuration has been removed. By default `gpt-3.5-turbo-0613` is used which accepts 4k tokens. If that's not enough, `gpt-3.5-turbo-16k-0613` is used instead.
- Switching to the bigger model happens when either the prompt is too long by itself (the API immediately returns an error code) or the model reaches the 4k limit itself in the process of generating a response and thus returns an incomplete response. The latter case is made possible by removing the `max_tokens: 1500` in the request, which was very generous and would have lead to switching to the more expensive model more often than needed. The downside is that the user has to wait a bit longer for the response.
- If the bigger 16k token limit is also exceeded, the assistant immediately responds (instead of retrying as before) with an error message including suggestions. The suggestions include restarting the conversation if and only if the user has sent multiple messages.
- If a request fails because Grist has reached its OpenAI monthly billing quota, the assistant immediately responds (instead of retrying as before) with an error message suggesting that the user try again tomorrow.
- If a request fails for some other reason, the assistant retries, and if all attempts fail then the user is told to try again in a few minutes and is shown the exact error message, including the API response if there is one.
- Retrying only happens when an API request fails, whereas previously the system also retried errors from a much bigger scope which included calls to the sandbox. The downside is that the hugging face assistant no longer retries, although that code is currently disabled anyway.
- The assistant no longer waits an additional second after the final retry attempt fails.

Test Plan: Added a new server test file with several unit tests using faked OpenAI responses, including the happy path which wasn't really tested before.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3955
2023-07-18 16:01:37 +02:00
Jakub Serafin
d894b60fd4 (core) deleting queue from single webhook
Summary: Using standard tost notification, message about webhook queue being overflown was added. message is permanent as long as queue is full. Message contains linkt to the webhook setings

Test Plan: two nbrowser test was added - one to check if message is show when queue is full, and second to check if message is dismiss when queue was cleaned.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3929
2023-07-18 11:46:10 +02:00
Paul Fitzpatrick
450472f74c (core) updates from grist-core 2023-07-17 01:22:18 -04:00
CamilleLegeron
61bd064f73
i18n: userManager translation + some forgotten translations (#557)
* translation: add userManager translation + some forgotten translations
* use '\' caracter for multiple-line strings
2023-07-16 12:52:13 -04:00