Commit Graph

1295 Commits

Author SHA1 Message Date
Janet Vorobyeva
29f07a8a4f
Bidirectional Linking (#622)
Allows bidirectional / cyclic linking for same-record cursor links.
This should allow multiple sections to all synchronize their cursors,
such that clicking in any one of them will move all the others.

Works even if some sections in the cycle have rows filtered out (the 
filtered-out sections might desync their cursors, but the correct cursor
position will still propagate downstream, and they'll re-sync if clicking on
a row that is present in them)

Under the hood, each cursor has a _lastEditedAt counter, updated when
a user's action changes the cursor in a section, such that we can always
tell which section was touched most recently. This is used to resolve
conflicts stably when dealing with cycles or chains of cursor-links.

Updated selectBy and recursiveMoveToCursorPos to handle cycles

Updated tests for selectBy behavior

However, main bidirectional-linking tests are not in this commit, they'll come in a subsequent PR
2023-09-25 18:48:18 -04:00
George Gevoian
e01ebe1ae8 (core) Fix blurry tooltips
Summary: GPU acceleration was causing tooltip text to appear blurry.

Test Plan: Manual.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4046
2023-09-22 03:51:58 -04:00
George Gevoian
273b976cab (core) Polish dark mode and remove beta tag
Summary:
Polishes support for dark mode and enables syncing with the OS theme
by default.

Test Plan: Manual.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Subscribers: JakubSerafin

Differential Revision: https://phab.getgrist.com/D4041
2023-09-21 13:14:48 -04:00
George Gevoian
d1826987bb (core) Revert part of the recent tweaks to AppHeader
Summary:
In non-SaaS Grist, the AppHeader will no longer link to templates
page. Also, the last visited site is now reset when switching accounts.

Test Plan: Existing tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4048
2023-09-20 11:27:46 -04:00
George Gevoian
927e92e3e8 (core) Add tip for calendar widget configuration
Summary:
The tip is shown in the creator panel, in the subtab that lists the column
mapping configuration for the calendar widget. The panel now  automatically
opens the first time a calendar widget is added to a page (via the Add New menu).

Test Plan: Manual.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Subscribers: JakubSerafin, jarek

Differential Revision: https://phab.getgrist.com/D4047
2023-09-20 11:10:58 -04:00
George Gevoian
581a62306e (core) Update video tour
Summary: The embed id of the video tour now points to the new introduction video.

Test Plan: Manual.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4049
2023-09-19 16:28:36 -04:00
George Gevoian
4c25aa7d3d (core) Support dark mode in custom widgets
Test Plan: Manual.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4036
2023-09-19 16:08:21 -04:00
Paul Fitzpatrick
ffbf93b85f (core) updates from grist-core 2023-09-18 15:04:08 -04:00
Jarosław Sadziński
d13a75a453 (core) Avoiding the view layout's rebuild, when nothing has changed.
Summary:
Layout is rebuild when it is updated from outside, for example after saving. But actually we don't need to rebuild it, because most of the time nothing has changed.
This is important for custom widgets, which will reload the iframe, even though the dom is not changed, but just moved from previous layout to the new one.

Test Plan: Manual and existing.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4039
2023-09-18 09:47:50 +02:00
Janet Vorobyeva
8fd6bd1afb
Catches isDisposed bug with rightpanel linkInfo (#676)
* Catches isDisposed bug with rightpanel linkInfo

Would happen when deleting srcSection of a link with rightpanel
open to the linking tab
2023-09-15 14:05:26 -04:00
Jarosław Sadziński
be995d4857 (core) Clicking on a link in a cell won't open the editor.
Summary:
Simple click (a click on a already focused cell) was opening the editor
even though user clicked an anchor in the cell.

Test Plan: Added test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D4044
2023-09-14 17:28:50 +02:00
Paul Fitzpatrick
bc6f550471
normalize logging of dates in siteUsage across databases (#675)
Date handling for SQLite and PostgreSQL is inconsistent. This makes
sure that dates in `siteUsage` logs are in a consistent format that
includes time zone information.
2023-09-13 14:44:04 -04:00
Jarosław Sadziński
2b11000457 (core) Multiple types in columns mapping
Summary: Mapping for a single column in custom widgets accepts now multiple types as comma separeted list.

Test Plan: Added new

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Differential Revision: https://phab.getgrist.com/D4042
2023-09-13 14:41:31 +02:00
George Gevoian
40c5f7b738 (core) Add documentCreated telemetry event
Summary:
The event is triggered whenever a document is created, imported, or
duplicated.

Test Plan: Tested manually.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4035
2023-09-13 01:13:00 -04:00
George Gevoian
76e822eb23 (core) Add welcomeQuestionsSubmitted telemetry event
Summary:
The new event captures responses to the welcome questionnaire.

Responses are also still sent to the special Grist document configured with
the DOC_ID_NEW_USER_INFO variable.

Test Plan: Tested manually.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4034
2023-09-13 00:31:04 -04:00
Paul Fitzpatrick
f659f3655d (core) updates from grist-core 2023-09-12 14:16:02 -04:00
Jarosław Sadziński
75d6b7ab30 (core) Fixing code after core merge
Summary: Fixing code after last core-merge

Test Plan: Existing

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Subscribers: JakubSerafin

Differential Revision: https://phab.getgrist.com/D4038
2023-09-12 11:59:36 +02:00
Janet Vorobyeva
5a4433c60c
Fixes linking "srcSection disposed" bug (#670)
Test "RawData should remove all tables except one (including referenced
summary table)"

was failing with error: "this._srcSection.activeRowId is not a function"
added an extra isDisposed check to catch it
2023-09-12 00:15:10 -04:00
Paul Fitzpatrick
18f7e255df (core) updates from grist-core 2023-09-11 10:00:39 -04:00
Paul Fitzpatrick
585cf02f6c
add hooks for tweaking how downloads happen (for grist-static) (#665)
This makes three main changes:
  * Adds a hook to transform download links.
  * Adds a hook to add an externally created ActiveDoc to a DocManager.
  * Rejiggers XLSX export code so it can be used without streaming,
    which is currently tricky in a browser. Regular usage with node
    continues to use streaming.

With these changes, I have a POC in hand that updates grist-static
to support downloading CSVs, XLSXs, and .grist files.
2023-09-09 14:50:32 -04:00
Janet Vorobyeva
f8c1bd612c
Linkstate refactor (#609)
* Linkingstate Refactor, and displaying link info in rightpanel

Big refactor to LinkingState
    Collects descriptive/user-facing labels into FilterState
    Unifies/cleans up some logic
    Adds LinkTypeDescription, a string enum which can be used
    to easily switch/case between various cases of linking, and
    codifies the logic in one place (currently only used for linkInfo)

Adds Link info to creator panel, near SelectBy dropdown

Bugfix: Disables linking from Attachment columns
Bugfix/Behavior change: changed linking with empty RefLists to better
match behavior of refs.
    for context: Linking by a blank Ref filters to show records with a
    blank value for that Ref. Previously this didn't work with RefLists. 
    Linking from a blank refList would show no records 
    (except in some cases involving summary tables)
    Fixed this so that linking by a blank val consistently means "show
    all records where the corresponding col is blank"
2023-09-08 14:21:32 -04:00
Florent
5ff79703b4
Introduce GRIST_ANON_PLAYGROUND variable #642 (#651)
* `GRIST_ANON_PLAYGROUND`: When set to 'false' deny anonymous users access to the home page
 * `GRIST_FORCE_LOGIN`: Much like `GRIST_ANON_PLAYGROUND` but don't support anonymous access at all (features like sharing docs publicly requires authentication)

---------

Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-09-08 09:05:52 -04:00
Jarosław Sadziński
a0fc11c8d1 (core) Fixing memos in record dependent rules.
Summary:
Memos in row dependent rules were ignored if the action was denied unconditionally. It prevented creating different memos for different users.
Now ACL is forced to check each row, to find the correct memo to show when create/update/delete action is denied.

Test Plan: Existing and new.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz, dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D4024
2023-09-08 10:16:09 +02:00
Jarosław Sadziński
4ead16176f (core) Showing workspace selector on duplicate document popup
Summary:
Duplicate document popup wasn't offering workspace selection for personal orgs.
Also, when workspace was removed, the URL wasn't updated which left user with an error page `workspace not found`.

Test Plan: Updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4031
2023-09-07 16:16:33 +02:00
George Gevoian
3dadf93c98 (core) Add support for auto-copying docs on signup
Summary:
The new "copyDoc" query parameter on the login page sets a short-lived cookie, which is
then read when welcoming a new user to copy that document to their Home workspace, and
redirect to it. Currently, only templates and bare forks set this parameter.

A new API endpoint for copying a document to a workspace was also added.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3992
2023-09-06 15:12:08 -04:00
George Gevoian
90fb4434cc
Add ws id and doc name params to POST /docs (#655) 2023-09-05 14:27:35 -04:00
Florent
1ac15912b1
Make server take into account GRIST_SERVERS (#659)
Also ensure that only a home server will setup the database.

resolves #654

Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-09-05 12:05:29 -04:00
Paul Fitzpatrick
6dab12f301 (core) updates from grist-core 2023-09-05 11:12:15 -04:00
Paul Fitzpatrick
bfd0fa8c7f
add an endpoint for doing SQL selects (#641)
* add an endpoint for doing SQL selects

This adds an endpoint for doing SQL selects directly on a Grist document. Other kinds of statements are not supported. There is a default timeout of a second on queries.

This follows loosely an API design by Alex Hall.

Co-authored-by: jarek <jaroslaw.sadzinski@gmail.com>
2023-09-04 09:21:18 -04:00
Andrew Langford
b465e07bb4
Fixed typo in a comment (#656) 2023-09-01 09:59:33 -04:00
George Gevoian
70feb336d9 (core) Add AI assistant usage banners
Summary:
Banners are now shown when there are low or no AI assistant
credits remaining.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4018
2023-08-30 16:00:04 -04:00
Jarosław Sadziński
98068cb86c (core) Hiding expand button if there is single widget on a page
Summary:
Expand button on a page with a single widget seems pointless and presents a bad UX when
embedded.

Test Plan: new tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D4020
2023-08-30 21:28:41 +02:00
Jarosław Sadziński
58323f5313 (core) Adding testId to the widget iframe once it receives the ready message
Summary:
Iframe with custom widget is marked with a test class `test-custom-widget-ready` when
it receives the `ready` message from the rendered widget.

Test Plan: Added and updated. Existing test should pass.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D4023
2023-08-30 21:28:06 +02:00
Jakub Serafin
daf7614fd7 (core) fix for duplicated "Learn more about custom widgets" section
Test Plan: I checked if section is showed only once after this change. it is.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4022
2023-08-30 16:43:49 +02:00
George Gevoian
ac6933e39f (core) Fix unreadable text in dark mode
Summary:
The message shown in a linked widget when no row is
selected was unreadable in dark mode.

Test Plan: Manual.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4019
2023-08-30 10:25:54 -04:00
Jakub Serafin
1bc97773c8 (core) fix for master not building - in gristUrls.ts imports was duplicated
Test Plan: none

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4021
2023-08-30 11:06:31 +02:00
Jakub Serafin
942fc96225 (core) custom widget appear as build-in widget
Summary: Added boilerplate code needed to create new wigets in "Add new" menu, that are wrapped around existing custom widgets. More details can be found here: https://grist.quip.com/larhAGRKyl6Z/Custom-widgets-in-Add-Widget-menu

Test Plan: nbowser tests added to verify if item in menu exits, if widget is rendered, and right side menu has widget selection and read access selection hided.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3994
2023-08-30 09:44:25 +02:00
Paul Fitzpatrick
8d687a7651
remove a log message about fetching URLs (#643)
Every fetch made from the client is logged to the console.
But this isn't really necessary, and is particularly confusing
in grist-static, where those fetches are virtualized.

Tests in grist-saas may need adjusting to remove the logger.
2023-08-29 08:49:25 -04:00
Jarosław Sadziński
b6a431dd58 (core) Cursor in custom widgets
Summary:
Adding a new method `setCursorPos` in the widget API, and a new configuration option for the ready message `allowSelectBy` that exposes custom widgets in the `Select by` dropdown.
With this, a custom widget can control the position of the linked widgets and is able to change the column in the creator panel.

Test Plan: Added new test. Existing tests should pass.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Subscribers: JakubSerafin

Differential Revision: https://phab.getgrist.com/D3993
2023-08-29 09:19:52 +02:00
Paul Fitzpatrick
c02acff361 (core) updates from grist-core 2023-08-28 08:24:41 -04:00
Paul Fitzpatrick
7aebdd15f6
tweak XLSX export worker to make Piscina happy under Electon (#646)
Adds a dummy default export to the worker exporter script used
for producing XLSX. This method exists only to make Piscina
happier. With it, Piscina will load this file using a regular
require(), which under Electron will deal fine with Electron's
ASAR app bundle. Without it, Piscina will try fancier methods
that aren't at the time of writing correctly patched to
deal with an ASAR app bundle, and so report that this
file doesn't exist instead of exporting an XLSX file.

I tried various other solutions such as upgrading Electron,
unpacking various files, patching Piscina, and this was
overall the simplest.

See https://github.com/gristlabs/grist-electron/issues/9
2023-08-28 08:11:01 -04:00
Jakub Serafin
2073e6224e (core) Inactive card list widget now have idicator that show cursor position of linked widget
Test Plan: nbrowser test added

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4009
2023-08-25 16:27:20 +02:00
Dmitry S
d5a4605d2a (core) Improve encoding detection for csv imports, and make encoding an editable option.
Summary:
- Using a sample of data was causing poor detection if the sample were
  cut mid-character. Switch to using line-based detection.
- Add a simple option for changing encoding. No convenient UI is offered
  since config UI is auto-generated, but this at least makes it possible to
  recover from bad guesses.
- Upgrades chardet library for good measure.

- Also fixes python3-building step, to more reliably rebuild Python
  dependencies when requirements3.* files change.

Test Plan:
Added a python-side test case, and a browser test that encodings can
be switched, errors are displayed, and wrong encodings fail recoverably.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3979
2023-08-24 09:50:52 -04:00
Florent
ee31764b83
DocApi: Implement DELETE on columns (#601) (#640)
* Factorize generateDocAndUrl
* Add describe for regrouping /records
2023-08-24 14:33:53 +02:00
Florent
9dfebefc9b
Dump the rule for ACL formula warnings (#639)
When an ACL formula fails to be run, a warning is printed. However, it is painful to know which formula is concerned by the warning.

ACL: if RHS is null, return false for "in" and "not in"
2023-08-23 09:23:29 -04:00
Janet Vorobyeva
b41fa31dd9
Fixes bug 'this.valueFormatter is not a function' (#632)
https://grist.slack.com/archives/C069RUP71/p1692034396980779
Bug showed up when deleting a page with the right kinds of
widgets/fields, due to a missing isDisposed check

Bug was found in DateTextBox, but I added the fix to NTextBox which
had an identical bit of code
2023-08-22 03:17:01 -04:00
Paul Fitzpatrick
b3a71374d1 (core) updates from grist-core 2023-08-21 09:15:53 -04:00
Paul Fitzpatrick
0be858c19d
allow AI Assistance to run against any chat-completion-style endpoint (#630)
This adds an ASSISTANT_CHAT_COMPLETION_ENDPOINT which can be used
to enable AI Assistance instead of an OpenAI API key. The assistant
then works against compatible endpoints, in the mechanical sense.
Quality of course will depend on the model. I found some tweaks
to the prompt that work well both for Llama-2 and for OpenAI's models,
but I'm not including them here because they would conflict with some
prompt changes that are already in the works.

Co-authored-by: Alex Hall <alex.mojaki@gmail.com>
2023-08-18 16:14:42 -04:00
Alex Hall
166312be3a (core) Add dialog with options to allow downloading without history or data
Summary:
{F74398}

Refactored the 'radio checkboxes' in the modal for deleting a page and reused them here.

The option to download as a template already existed in the server code but wasn't being exercised by the frontend. Also added an option to remove just the history, which is the main motivation for this diff.

Test Plan: Expanded the existing nbrowser test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3999
2023-08-18 15:38:24 +02:00
Alex Hall
a1d31e41ad (core) Prevent the AI assistant response from including class definitions
Summary:
Sometimes the model repeats the classes given in the prompt which would mess up extracting the actual formula. This diff solves this by:

1. Changes the generated Python schema so that (a) the thing that needs completing is a plain top level function instead of a property/method inside the class and (2) the classes are fully valid syntax, which makes it easier to
2. Remove classes from the parsed Python code when converting the completion to a formula.
3. Tweak the prompt wording to discourage including classes in general, especially because sometimes the model tries to solve the problem by defining extra methods/attributes/classes.

While I was at it, I changed type hints to use builtins (e.g. `list` instead of `List`) to prevent `from typing import List` which was happening sometimes and would look weird in a formula. Similarly I removed `@dataclass` since that also implies an import, and this also fits with the tweaked wording that the classes are fake.

Test Plan:
Added a new test case to the formula dataset which triggers the unwanted behaviour. The factors that seem to trigger the problem are (1) a small schema so the classes are easier to repeat and (2) the need to import modules, which the model wants to place before all other code. The case failed before this diff and succeeded after. The tweaked wording reduces the chances of repeating the classes but didn't eliminate it, so forcibly removing the classes in Python was needed.

There were also a couple of other existing cases where repeating the classes was observed before but not after.

Overall the score increased from 49 to 51 out of 69 (including the new case). At one point the score was 53, but changes in whitespace were enough to make it drop again.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4000
2023-08-18 12:50:09 +02:00
Alex Hall
999d723d14 (core) Cleanup old frontend code for handling empty summary groups
Summary: The removed TODO was resolved in https://phab.getgrist.com/D3489

Test Plan: Existing tests

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D4006
2023-08-18 12:46:15 +02:00
Paul Fitzpatrick
fe846d3068 (core) allow copying errors from regular cell editor
Summary:
There was already some support for copying errors from the detached/expanded formula editor. This adds support for copying errors in the regular cell editor.

Getting error details is now done only by clicking on the expand icon - previously you could click on the error text itself.

A few unrelated test changes are made for exact-pixel checks that were often out by approx 1 pixel on my machine.

Test Plan: Updated tests. Tested copying/pasting manually.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4002
2023-08-17 09:35:22 -04:00
Jarosław Sadziński
fad421b7c0 (core) Removing temporary pro site
Summary: Creating a pro team site after Stripe checkout. Previously a stub site was always created and never removed, even if a user cancels the checkout process, which resulted in multiple 'ghost' sites that can't be removed.

Test Plan: Updated and added

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3985
2023-08-16 08:11:26 +02:00
Philip Standt
9df62e3d81
Add appearance parameter to override theme preferences (#620) 2023-08-15 10:29:29 -07:00
Paul Fitzpatrick
f1a0b61e15 (core) updates from grist-core 2023-08-15 12:15:16 -04:00
Philip Standt
cbdffdfff8
Support grid selection with Ctrl+Shift+Arrow (#615) 2023-08-14 09:28:41 -07:00
Florent
fde56ffa33
DocApi: Introduce hidden option for GET on records (#623)
Also test that PUT /columns?replaceall=1 does not remove hidden columns
2023-08-14 16:17:46 +02:00
Florent
e93ad5a0c5
Issue #601: implement PUT on columns (#605)
* Also Add includeHidden param for GET on columns
2023-08-11 15:12:43 +02:00
George Gevoian
0f2b1f6a89 (core) Polish "Click to insert $id" tooltip
Summary:
The tooltip also broke recently on Firefox after upgrading Ace. This
includes a fix for the regression.

Test Plan: Updated test.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek, dsagal

Differential Revision: https://phab.getgrist.com/D3982
2023-08-09 07:38:34 -07:00
Jarosław Sadziński
732611c356 (core) Removing GRIST_FORMULA_ASSISTANT flag
Summary:
A floating formula editor is available by default and in the basic setup allows just formula modification.
AI assistant is now an optional component of the floating editor and it is controlled by OPENAPI_KEY presence.
Env variable GRIST_FORMULA_ASSISTANT was removed, new feature flag HAS_FORMULA_ASSISTANT is derived from the presence of OPENAPI_KEY.

Also updated anonymous signup nudge. By default it displays only info that this feature is only for logged in users.

Test Plan: updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3987
2023-08-09 10:08:18 +02:00
George Gevoian
24bbf375f9 (core) Add earliestDocCreatedAt to telemetry
Summary: This tracks the earliest document creation time, if any, for each site.

Test Plan: Server tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3991
2023-08-08 00:48:18 -07:00
Dmitry S
efb5fc63d7 (core) Change the label/nickname of the paid plan to Pro
Summary:
This is to match the pricing page, and to reduce confusion. The same
change was made to the description of the corresponding "pricing" object
on Stripe. The Stripe value is what's used in practice; this value is
actually only used as a fallback.

Test Plan: The code change does not affect tests, but several tests fixed after grist-core merge.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3990
2023-08-07 23:45:18 -04:00
Paul Fitzpatrick
031076cd07 (core) updates from grist-core 2023-08-07 14:29:30 -04:00
CamilleLegeron
02841bd15c
Header colored (#581) 2023-08-07 11:01:35 -07:00
Dmitry S
526a5df157 (core) Manage memory used for websocket responses to reduce the risk of server crashes.
Summary:
- Implements MemoryPool for waiting on memory reservations.
- Uses MemoryPool to control memory used for stringifying JSON responses in Client.ts
- Limits total size of _missedMessages that may be queued for a particular client.
- Upgrades ws library, which may reduce memory usage, and allows pausing the websocket for testing.
  - The upgrade changed subtle behavior corners, requiring various fixes to code and tests.

- dos.ts:
  - Includes Paul's fixes and updates to the dos.ts script for manual stress-testing.
  - Logging tweaks, to avoid excessive dumps on uncaughtError, and include timestamps.

Test Plan:
- Includes a test that measures heap size, and fails without memory management.
- Includes a unittest for MemoryPool
- Some cleanup and additions to TestServer helper; in particular adds makeUserApi() helper used in multiple tests.
- Some fixes related to ws upgrade.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3974
2023-08-07 11:28:31 -04:00
Paul Fitzpatrick
016ab66ea0
fix type issue if python is missing when running unsandboxed (#610)
Clean up a small recently introduced type issue.
2023-08-05 11:24:01 -04:00
Jarosław Sadziński
00b88fd683 (core) Adding links to description tooltips
Summary: Column and widget descriptions now support links in text.

Test Plan: Updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3981
2023-08-04 18:34:04 +02:00
Alex Hall
5dfa9a542c (core) Upgrade to Python 3.11
Summary: Replaced mentions of Python 3.9 with 3.11

Test Plan: this

Reviewers: paulfitz, georgegevoian

Reviewed By: paulfitz, georgegevoian

Subscribers: dsagal, georgegevoian, paulfitz

Differential Revision: https://phab.getgrist.com/D3980
2023-08-04 18:19:40 +02:00
Jarosław Sadziński
6416994c22 (core) Import redesign
Summary: New UI design for incremental imports.

Test Plan: Updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3945
2023-08-04 14:59:55 +02:00
George Gevoian
05c15e4ec3 (core) Show tweaked formula in AI responses
Summary:
The formula that's used when the Apply button is clicked, and the formula that's
shown in responses from the Formula Assistant should now be the same. Previously, they
would differ slightly.

Test Plan: Server tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3977
2023-08-04 01:31:31 -07:00
Jarosław Sadziński
4cfa033078 (core) TypeTransform race condition fix
Summary:
TypeTransformation was flaky. Probably after upgrading AceEditor we introduced a race condition between updating the revised formula and doing the transformation. Now we explicitly make sure that the formula is updated.

I also fixed some other flaky tests.

Test Plan: Updated

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3984
2023-08-02 17:11:34 +02:00
Paul Fitzpatrick
8110a26873 (core) updates from grist-core 2023-08-01 10:52:34 -04:00
Paul Fitzpatrick
61f954ff05
move getTemplateOrg method; enable template org in docker tests (#602)
* move getTemplateOrg method; enable template org in docker tests

This moves the `getTemplateOrg` method to a neutral venue for the
convenience of `grist-static`, otherwise a lot of awkward dependencies
get pulled in needlessly in new parts of the app.

This also fixes docker tests using the template org.
2023-07-31 16:10:59 -04:00
George Gevoian
9484fa14db (core) Fix mislabeled telemetry metadata
Summary:
The oldFormula and newFormula metadata fields had their names switched by
mistake.

Test Plan: N/A

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Differential Revision: https://phab.getgrist.com/D3976
2023-07-31 00:55:28 -07:00
John Cant
e1df6039c2
REQUEST now supports POST (#588)
* REQUEST now supports POST
* Add extra flag for enabling REQUEST, also update README and comments

Co-authored-by: John Cant <a.jonncant@gmail.com>
Co-authored-by: Alex Hall <alex.mojaki@gmail.com>
2023-07-30 15:13:43 -04:00
Alex Hall
9b87a6f06a (core) updates from grist-core 2023-07-28 17:40:31 +02:00
Jakub Serafin
d4c5082799
reverting whitespace changes in DocPageModel (#594) 2023-07-27 14:36:43 +02:00
Paul Fitzpatrick
f0b9e1f7e9
allow Grist front-end to function when location history is unavailable (#596)
* allow Grist front-end to function when location history is unavailable

When the Grist front-end is embedded in an iframe, using a srcdoc
attribute, history.pushState and similar methods are unavailable.
Currently, that makes it impossible to navigate between Grist pages,
since an access error is thrown (behavior may be browser dependent).
With this change, navigation succeeds.

* give unrelated possibly slow test a little more time
2023-07-27 07:02:17 -04:00
George Gevoian
a77170c4bd (core) Tweak navbar, breadcrumbs, and sign-in buttons
Summary:
The changes are intended to smooth over some sharp edges when a signed-out user
is using Grist (particularly while on the templates site).

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3957
2023-07-26 22:26:55 -07:00
Alex Hall
bc54a6646e (core) Filter rows based on linked widgets when exporting view
Summary:
Fixes a problem reported here: https://community.getgrist.com/t/exporting-the-records-in-a-linked-view/2556/4

The download CSV/Excel link now contains an additional `linkingFilter` URL parameter containing JSON-encoded `filters` and `operations`. This object is originally created in the frontend in `LinkingState`, and previously it was only used internally in the frontend. It would make its way via `QuerySetManager` to `QuerySet.getFilterFunc` where the actual filtering logic happened. Now most of that logic has been moved to a similar function in `common`. The new function works with a new interface `ColumnGettersByColId` which abstract over the different ways data is accessed in the client and server in this context. There's no significant new logic in the diff, just refactoring and wiring.

Test Plan: Expanded two `nbrowser/SelectBy*.ts` test suites to also check the contents of a downloaded CSV in different linking scenarios.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3961
2023-07-26 21:49:52 +02:00
Vincent Viers
ab6807c342
fix: don't count falses in summary (#589) 2023-07-26 12:08:04 -07:00
Alex Hall
3f71c9c488 (core) Cleanup: Remove unused AssistanceRequest.regenerate
Summary: Finish what was started in https://phab.getgrist.com/D3970

Test Plan: existing tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3972
2023-07-26 15:51:46 +02:00
Jakub Serafin
f7fdfab6bf (core) GET endpoint for webhooks returns now data in format {webhooks:[...]}
Summary:
Rework of endpoint GET  for webhooks to make it coherent with other endpoints. Now data should be return in {webhooks:[{id:"...",fields:{"..."}]} format

```
{
    "webhooks": [
        {
            "id": ...
            "fields": {
                "url": ...
                "unsubscribeKey": ...
                "eventTypes": [
                    "add",
                    "update"
                ],
                "isReadyColumn": null,
                "tableId": "...",
                "enabled": false,
                "name": "...",
                "memo": "..."
            },
            "usage": {
                "status": "idle",
                "numWaiting": 0,
                "lastEventBatch": null
            }
        },
        {
            "id": "...",
            "fields": {
                "url": "...",
                "unsubscribeKey": "...",
                "eventTypes": [
                    "add",
                    "update"
                ],
                "isReadyColumn": null,
                "tableId": "...",
                "enabled": true,
                "name": "...",
                "memo": "..."
            },
            "usage": {
                "status": "error",
                "numWaiting": 0,
                "updatedTime": 1689076978098,
                "lastEventBatch": {
                    "status": "rejected",
                    "httpStatus": 404,
                    "errorMessage": "{\"success\":false,\"error\":{\"message\":\"Alias 5a9bf6a8-4865-403a-bec6-b4ko not found\",\"id\":null}}",
                    "size": 49,
                    "attempts": 5
                },
                "lastSuccessTime": null,
                "lastFailureTime": 1689076978097,
                "lastErrorMessage": "{\"success\":false,\"error\":{\"message\":\"Alias 5a9bf6a8-4865-403a-bec6-b4ko not found\",\"id\":null}}",
                "lastHttpStatus": 404
            }
        }
    ]
}
```

Test Plan: new test added to check if GET data fromat is correct. Other tests fixed to handle changed endpoint.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3966
2023-07-26 11:36:24 +02:00
Paul Fitzpatrick
cd8eac36b8 (core) updates from grist-core 2023-07-25 17:14:17 -04:00
Janet Vorobyeva
b2e98230b4
Linux-specific fonts: Liberation Sans (#572)
* Added linux-specific fonts

Currently, ubuntu has a lot of issues with fonts.
Both the regular font-family and the data-font-family end up
evaluating to NimbusSans on my system
  (ubuntu's fallback font from helvetica)

NimbusSans unfortunately is noticably too high, cause text in buttons,
emoji on the left pane, and other vertical text alignment to be too high

This diff explicitly says to use Liberation Sans which should
have no effect on windows/mac systems, but should significantly improve
appearance on ubuntu (and hopefully other linuxes)

Both of these fonts are some of the more widely supported linux fonts,
see:
https://www.webfx.com/blog/web-design/a-web-designers-guide-to-linux-fonts/
2023-07-25 16:39:47 -04:00
George Gevoian
5e26401a24 (core) Re-render assistant code blocks on theme change
Summary:
When the browser theme transitioned while the assistant was open, messages
containing code blocks were not being re-rendered with the new theme.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3965
2023-07-24 19:09:01 -04:00
Alex Hall
391c8ee087 (core) Allow assistant to evaluate current formula
Summary:
Replaces https://phab.getgrist.com/D3940, particularly to avoid doing potentially unwanted things automatically.

Adds optional fields `evaluateCurrentFormula?: boolean; rowId?: number` to `FormulaAssistanceContext` (part of `AssistanceRequest`). When `evaluateCurrentFormula` is `true`, calls a new function `evaluate_formula` in the sandbox which computes the existing formula in the column (regardless of anything the AI may have suggested) and uses that to generate an additional system message which is added before the user's message. In theory this could be used in an interface where users ask why a formula doesn't work, including possibly a formula suggested by the AI. For now, it's only used in `runCompletion_impl.ts` for experimenting.

Also cleaned up a bit, removing `_chatMode` which is always `true` now, and uses of `regenerate` which is always `false`.

Test Plan: Updated `runCompletion_impl` to optionally use the new feature, in which case it now scores 51/68 instead of 49/68.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3970
2023-07-24 21:59:00 +02:00
Paul Fitzpatrick
8ecf0d7d0a
change order of new private method to fix lint after saas/core sync (#587) 2023-07-23 14:51:19 -04:00
Paul Fitzpatrick
7256e0c245 (core) updates from grist-core 2023-07-23 11:53:20 -04:00
George Gevoian
90cf018133 (core) Update assistant link
Summary: Updates the "Learn more" link now that the page is published.

Test Plan: N/A

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3969
2023-07-21 13:43:31 -04:00
George Gevoian
beffd02c41 (core) Improve highlighting of previewed formula
Summary:
Also improves highlighting of columns when the "Click to insert"
tooltip is shown, and improves highlighting of transforming columns.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3962
2023-07-20 14:34:22 -04:00
Alex Hall
5a703a1972 (core) Send hash of user ID in OpenAI API requests
Summary: Following recommendation in https://platform.openai.com/docs/guides/safety-best-practices/end-user-ids

Test Plan: Checked that running server test shows log of hash of 'user id' (which is null because it's a fake session)

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz, georgegevoian

Differential Revision: https://phab.getgrist.com/D3958
2023-07-20 19:50:41 +02:00
George Gevoian
0a34292536 (core) Add telemetry for AI Assistant
Summary: Also fixes a few bugs with some telemetry events not being recorded.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3960
2023-07-20 12:50:26 -04:00
Florent
5e33b68753
Issue 359 support scaleway (#577)
* Fix support of Scaleway S3 bucket #359

While MinIO and AWS return versionId as strings, other S3 API
implementations return versionId as integers.

We must carefully convert the versionId as string in order to cover
these various behaviors.

Also ensure that docStorage is initialized before attempting to
calculate the data size in order to avoid an exception.

* Add unit tests for MinIOExternalStorage#versions() #359

Introduced some unit tests to :
 - ensure listObjects is called with the right arguments;
 - cover the case when a S3 bucket implementation does not return the
   versionId as a string but rather as an integer (like Scaleway):
   in such a case, ensure that the returned snapshotId is a string;
 - cover the case when the listObjects function emits an error, ensure the
   versions() call rejets with the error emitted;
 - that the deleteMarkers are only returned when the
   includeDeleteMarkers is passed;

---------

Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-07-20 06:16:58 -04:00
George Gevoian
0040716006 (core) Disable undo/redo in detached formula editor
Summary:
Undo and redo were causing errors to be thrown while the editor was detached. In the
interest of time, we'll disable undo/redo until we have a chance to look at whether
we can support it in the editor.

Test Plan: Manual.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Differential Revision: https://phab.getgrist.com/D3959
2023-07-19 11:39:14 -04:00
Dmitry S
534615dd50 (core) Update logging in sandbox code, and log tracebacks as single log messages.
Summary:
- Replace logger module by the standard module 'logging'.
- When a log message from the sandbox includes newlines (e.g. for tracebacks),
  keep those lines together in the Node log message.

  Previously each line was a different message, making it difficult to view
  tracebacks, particularly in prod where each line becomes a separate message
  object.

- Fix assorted lint errors.

Test Plan: Added a test for the log-line splitting and escaping logic.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3956
2023-07-18 11:21:25 -04:00
Alex Hall
7fd48364df (core) Improved error messages, retries, and handling of token limits in AI assistant
Summary:
In a nutshell:

- More specific and helpful error messages are shown to the user
- API requests are only retried when needed
- The system deals with reaching the maximum token limit better, especially by switching to a model with a bigger limit

In more detail:

- `COMPLETION_MODEL` configuration has been removed. By default `gpt-3.5-turbo-0613` is used which accepts 4k tokens. If that's not enough, `gpt-3.5-turbo-16k-0613` is used instead.
- Switching to the bigger model happens when either the prompt is too long by itself (the API immediately returns an error code) or the model reaches the 4k limit itself in the process of generating a response and thus returns an incomplete response. The latter case is made possible by removing the `max_tokens: 1500` in the request, which was very generous and would have lead to switching to the more expensive model more often than needed. The downside is that the user has to wait a bit longer for the response.
- If the bigger 16k token limit is also exceeded, the assistant immediately responds (instead of retrying as before) with an error message including suggestions. The suggestions include restarting the conversation if and only if the user has sent multiple messages.
- If a request fails because Grist has reached its OpenAI monthly billing quota, the assistant immediately responds (instead of retrying as before) with an error message suggesting that the user try again tomorrow.
- If a request fails for some other reason, the assistant retries, and if all attempts fail then the user is told to try again in a few minutes and is shown the exact error message, including the API response if there is one.
- Retrying only happens when an API request fails, whereas previously the system also retried errors from a much bigger scope which included calls to the sandbox. The downside is that the hugging face assistant no longer retries, although that code is currently disabled anyway.
- The assistant no longer waits an additional second after the final retry attempt fails.

Test Plan: Added a new server test file with several unit tests using faked OpenAI responses, including the happy path which wasn't really tested before.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3955
2023-07-18 16:01:37 +02:00
Jakub Serafin
d894b60fd4 (core) deleting queue from single webhook
Summary: Using standard tost notification, message about webhook queue being overflown was added. message is permanent as long as queue is full. Message contains linkt to the webhook setings

Test Plan: two nbrowser test was added - one to check if message is show when queue is full, and second to check if message is dismiss when queue was cleaned.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3929
2023-07-18 11:46:10 +02:00
Paul Fitzpatrick
450472f74c (core) updates from grist-core 2023-07-17 01:22:18 -04:00
CamilleLegeron
61bd064f73
i18n: userManager translation + some forgotten translations (#557)
* translation: add userManager translation + some forgotten translations
* use '\' caracter for multiple-line strings
2023-07-16 12:52:13 -04:00