Commit Graph

439 Commits

Author SHA1 Message Date
Paul Fitzpatrick
c02acff361 (core) updates from grist-core 2023-08-28 08:24:41 -04:00
Jakub Serafin
2073e6224e (core) Inactive card list widget now have idicator that show cursor position of linked widget
Test Plan: nbrowser test added

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4009
2023-08-25 16:27:20 +02:00
Dmitry S
d5a4605d2a (core) Improve encoding detection for csv imports, and make encoding an editable option.
Summary:
- Using a sample of data was causing poor detection if the sample were
  cut mid-character. Switch to using line-based detection.
- Add a simple option for changing encoding. No convenient UI is offered
  since config UI is auto-generated, but this at least makes it possible to
  recover from bad guesses.
- Upgrades chardet library for good measure.

- Also fixes python3-building step, to more reliably rebuild Python
  dependencies when requirements3.* files change.

Test Plan:
Added a python-side test case, and a browser test that encodings can
be switched, errors are displayed, and wrong encodings fail recoverably.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3979
2023-08-24 09:50:52 -04:00
Florent
ee31764b83
DocApi: Implement DELETE on columns (#601) (#640)
* Factorize generateDocAndUrl
* Add describe for regrouping /records
2023-08-24 14:33:53 +02:00
Florent
9dfebefc9b
Dump the rule for ACL formula warnings (#639)
When an ACL formula fails to be run, a warning is printed. However, it is painful to know which formula is concerned by the warning.

ACL: if RHS is null, return false for "in" and "not in"
2023-08-23 09:23:29 -04:00
Paul Fitzpatrick
f6e8b9c8a7 (core) move ACLFormula test
Summary:
move a test of access control formulas
to grist-core

Test Plan: moving a test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4013
2023-08-22 14:24:51 -04:00
Paul Fitzpatrick
c31999e604 (core) move assistant test
Summary:
Moves assistant test to core.
Fixes new name for constaint.
Works around a chai-as-promised issue.

Test Plan: this

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4010
2023-08-22 08:14:48 -04:00
Paul Fitzpatrick
b3a71374d1 (core) updates from grist-core 2023-08-21 09:15:53 -04:00
Paul Fitzpatrick
0be858c19d
allow AI Assistance to run against any chat-completion-style endpoint (#630)
This adds an ASSISTANT_CHAT_COMPLETION_ENDPOINT which can be used
to enable AI Assistance instead of an OpenAI API key. The assistant
then works against compatible endpoints, in the mechanical sense.
Quality of course will depend on the model. I found some tweaks
to the prompt that work well both for Llama-2 and for OpenAI's models,
but I'm not including them here because they would conflict with some
prompt changes that are already in the works.

Co-authored-by: Alex Hall <alex.mojaki@gmail.com>
2023-08-18 16:14:42 -04:00
Alex Hall
166312be3a (core) Add dialog with options to allow downloading without history or data
Summary:
{F74398}

Refactored the 'radio checkboxes' in the modal for deleting a page and reused them here.

The option to download as a template already existed in the server code but wasn't being exercised by the frontend. Also added an option to remove just the history, which is the main motivation for this diff.

Test Plan: Expanded the existing nbrowser test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3999
2023-08-18 15:38:24 +02:00
Alex Hall
a1d31e41ad (core) Prevent the AI assistant response from including class definitions
Summary:
Sometimes the model repeats the classes given in the prompt which would mess up extracting the actual formula. This diff solves this by:

1. Changes the generated Python schema so that (a) the thing that needs completing is a plain top level function instead of a property/method inside the class and (2) the classes are fully valid syntax, which makes it easier to
2. Remove classes from the parsed Python code when converting the completion to a formula.
3. Tweak the prompt wording to discourage including classes in general, especially because sometimes the model tries to solve the problem by defining extra methods/attributes/classes.

While I was at it, I changed type hints to use builtins (e.g. `list` instead of `List`) to prevent `from typing import List` which was happening sometimes and would look weird in a formula. Similarly I removed `@dataclass` since that also implies an import, and this also fits with the tweaked wording that the classes are fake.

Test Plan:
Added a new test case to the formula dataset which triggers the unwanted behaviour. The factors that seem to trigger the problem are (1) a small schema so the classes are easier to repeat and (2) the need to import modules, which the model wants to place before all other code. The case failed before this diff and succeeded after. The tweaked wording reduces the chances of repeating the classes but didn't eliminate it, so forcibly removing the classes in Python was needed.

There were also a couple of other existing cases where repeating the classes was observed before but not after.

Overall the score increased from 49 to 51 out of 69 (including the new case). At one point the score was 53, but changes in whitespace were enough to make it drop again.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4000
2023-08-18 12:50:09 +02:00
Philip Standt
9df62e3d81
Add appearance parameter to override theme preferences (#620) 2023-08-15 10:29:29 -07:00
Paul Fitzpatrick
f1a0b61e15 (core) updates from grist-core 2023-08-15 12:15:16 -04:00
Philip Standt
cbdffdfff8
Support grid selection with Ctrl+Shift+Arrow (#615) 2023-08-14 09:28:41 -07:00
Florent
fde56ffa33
DocApi: Introduce hidden option for GET on records (#623)
Also test that PUT /columns?replaceall=1 does not remove hidden columns
2023-08-14 16:17:46 +02:00
Florent
e93ad5a0c5
Issue #601: implement PUT on columns (#605)
* Also Add includeHidden param for GET on columns
2023-08-11 15:12:43 +02:00
George Gevoian
0f2b1f6a89 (core) Polish "Click to insert $id" tooltip
Summary:
The tooltip also broke recently on Firefox after upgrading Ace. This
includes a fix for the regression.

Test Plan: Updated test.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek, dsagal

Differential Revision: https://phab.getgrist.com/D3982
2023-08-09 07:38:34 -07:00
Paul Fitzpatrick
bb6926d2a0 (core) updates from grist-core 2023-08-09 09:58:42 -04:00
Paul Fitzpatrick
d6ab1f45c2 (core) move LinkingError and RawData tests
Summary:
Moves some more relevant tests to grist-core.
Small change to declarations that seemed necessary when
running buildtools/build_core.sh now, but I'm not sure
what changed. CI hasn't flagged anything, so smells like
the version of some package not nailed down.

Test Plan:
tried running tests in new location using
buildtools/build_core.sh

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3996
2023-08-09 09:46:36 -04:00
Jarosław Sadziński
732611c356 (core) Removing GRIST_FORMULA_ASSISTANT flag
Summary:
A floating formula editor is available by default and in the basic setup allows just formula modification.
AI assistant is now an optional component of the floating editor and it is controlled by OPENAPI_KEY presence.
Env variable GRIST_FORMULA_ASSISTANT was removed, new feature flag HAS_FORMULA_ASSISTANT is derived from the presence of OPENAPI_KEY.

Also updated anonymous signup nudge. By default it displays only info that this feature is only for logged in users.

Test Plan: updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3987
2023-08-09 10:08:18 +02:00
George Gevoian
24bbf375f9 (core) Add earliestDocCreatedAt to telemetry
Summary: This tracks the earliest document creation time, if any, for each site.

Test Plan: Server tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3991
2023-08-08 00:48:18 -07:00
Dmitry S
efb5fc63d7 (core) Change the label/nickname of the paid plan to Pro
Summary:
This is to match the pricing page, and to reduce confusion. The same
change was made to the description of the corresponding "pricing" object
on Stripe. The Stripe value is what's used in practice; this value is
actually only used as a fallback.

Test Plan: The code change does not affect tests, but several tests fixed after grist-core merge.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3990
2023-08-07 23:45:18 -04:00
Dmitry
c9489e4a61
Fix ManyFetches test, and include into grist-core a needed upgrade to the 'ws' package. (#614)
This is a follow-up fix to 526a5df157.
2023-08-07 18:39:14 -04:00
Paul Fitzpatrick
031076cd07 (core) updates from grist-core 2023-08-07 14:29:30 -04:00
CamilleLegeron
02841bd15c
Header colored (#581) 2023-08-07 11:01:35 -07:00
Dmitry S
526a5df157 (core) Manage memory used for websocket responses to reduce the risk of server crashes.
Summary:
- Implements MemoryPool for waiting on memory reservations.
- Uses MemoryPool to control memory used for stringifying JSON responses in Client.ts
- Limits total size of _missedMessages that may be queued for a particular client.
- Upgrades ws library, which may reduce memory usage, and allows pausing the websocket for testing.
  - The upgrade changed subtle behavior corners, requiring various fixes to code and tests.

- dos.ts:
  - Includes Paul's fixes and updates to the dos.ts script for manual stress-testing.
  - Logging tweaks, to avoid excessive dumps on uncaughtError, and include timestamps.

Test Plan:
- Includes a test that measures heap size, and fails without memory management.
- Includes a unittest for MemoryPool
- Some cleanup and additions to TestServer helper; in particular adds makeUserApi() helper used in multiple tests.
- Some fixes related to ws upgrade.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3974
2023-08-07 11:28:31 -04:00
jarek
7c114bf600
Restoring default ports for webhook tests (#612) 2023-08-07 16:37:24 +02:00
Jarosław Sadziński
00b88fd683 (core) Adding links to description tooltips
Summary: Column and widget descriptions now support links in text.

Test Plan: Updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3981
2023-08-04 18:34:04 +02:00
Jarosław Sadziński
4cfa033078 (core) TypeTransform race condition fix
Summary:
TypeTransformation was flaky. Probably after upgrading AceEditor we introduced a race condition between updating the revised formula and doing the transformation. Now we explicitly make sure that the formula is updated.

I also fixed some other flaky tests.

Test Plan: Updated

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3984
2023-08-02 17:11:34 +02:00
Paul Fitzpatrick
61f954ff05
move getTemplateOrg method; enable template org in docker tests (#602)
* move getTemplateOrg method; enable template org in docker tests

This moves the `getTemplateOrg` method to a neutral venue for the
convenience of `grist-static`, otherwise a lot of awkward dependencies
get pulled in needlessly in new parts of the app.

This also fixes docker tests using the template org.
2023-07-31 16:10:59 -04:00
Alex Hall
9b87a6f06a (core) updates from grist-core 2023-07-28 17:40:31 +02:00
Paul Fitzpatrick
f0b9e1f7e9
allow Grist front-end to function when location history is unavailable (#596)
* allow Grist front-end to function when location history is unavailable

When the Grist front-end is embedded in an iframe, using a srcdoc
attribute, history.pushState and similar methods are unavailable.
Currently, that makes it impossible to navigate between Grist pages,
since an access error is thrown (behavior may be browser dependent).
With this change, navigation succeeds.

* give unrelated possibly slow test a little more time
2023-07-27 07:02:17 -04:00
George Gevoian
a77170c4bd (core) Tweak navbar, breadcrumbs, and sign-in buttons
Summary:
The changes are intended to smooth over some sharp edges when a signed-out user
is using Grist (particularly while on the templates site).

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3957
2023-07-26 22:26:55 -07:00
Alex Hall
bc54a6646e (core) Filter rows based on linked widgets when exporting view
Summary:
Fixes a problem reported here: https://community.getgrist.com/t/exporting-the-records-in-a-linked-view/2556/4

The download CSV/Excel link now contains an additional `linkingFilter` URL parameter containing JSON-encoded `filters` and `operations`. This object is originally created in the frontend in `LinkingState`, and previously it was only used internally in the frontend. It would make its way via `QuerySetManager` to `QuerySet.getFilterFunc` where the actual filtering logic happened. Now most of that logic has been moved to a similar function in `common`. The new function works with a new interface `ColumnGettersByColId` which abstract over the different ways data is accessed in the client and server in this context. There's no significant new logic in the diff, just refactoring and wiring.

Test Plan: Expanded two `nbrowser/SelectBy*.ts` test suites to also check the contents of a downloaded CSV in different linking scenarios.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3961
2023-07-26 21:49:52 +02:00
Vincent Viers
ab6807c342
fix: don't count falses in summary (#589) 2023-07-26 12:08:04 -07:00
Paul Fitzpatrick
71c6537c49
tweak a date test that can occasionally fail (#592)
A date test was noted to fail, with a formula intended for
a cell ending up in the column header. This may help.
Entering formulas reliably requires waiting for a particular
focus state.
2023-07-26 08:08:55 -04:00
Paul Fitzpatrick
bc599916f5
tweak webhook overflow test (#590)
This test appears to fail if toasts aren't issued fast enough
(perhaps because an empty toast list may count as truthy?).
I might be missing something since I don't understand the
purpose of waitForOverflownMessageToDisappear apart from
lengthening a timeout.
2023-07-26 06:20:20 -04:00
Jakub Serafin
f7fdfab6bf (core) GET endpoint for webhooks returns now data in format {webhooks:[...]}
Summary:
Rework of endpoint GET  for webhooks to make it coherent with other endpoints. Now data should be return in {webhooks:[{id:"...",fields:{"..."}]} format

```
{
    "webhooks": [
        {
            "id": ...
            "fields": {
                "url": ...
                "unsubscribeKey": ...
                "eventTypes": [
                    "add",
                    "update"
                ],
                "isReadyColumn": null,
                "tableId": "...",
                "enabled": false,
                "name": "...",
                "memo": "..."
            },
            "usage": {
                "status": "idle",
                "numWaiting": 0,
                "lastEventBatch": null
            }
        },
        {
            "id": "...",
            "fields": {
                "url": "...",
                "unsubscribeKey": "...",
                "eventTypes": [
                    "add",
                    "update"
                ],
                "isReadyColumn": null,
                "tableId": "...",
                "enabled": true,
                "name": "...",
                "memo": "..."
            },
            "usage": {
                "status": "error",
                "numWaiting": 0,
                "updatedTime": 1689076978098,
                "lastEventBatch": {
                    "status": "rejected",
                    "httpStatus": 404,
                    "errorMessage": "{\"success\":false,\"error\":{\"message\":\"Alias 5a9bf6a8-4865-403a-bec6-b4ko not found\",\"id\":null}}",
                    "size": 49,
                    "attempts": 5
                },
                "lastSuccessTime": null,
                "lastFailureTime": 1689076978097,
                "lastErrorMessage": "{\"success\":false,\"error\":{\"message\":\"Alias 5a9bf6a8-4865-403a-bec6-b4ko not found\",\"id\":null}}",
                "lastHttpStatus": 404
            }
        }
    ]
}
```

Test Plan: new test added to check if GET data fromat is correct. Other tests fixed to handle changed endpoint.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3966
2023-07-26 11:36:24 +02:00
Alex Hall
391c8ee087 (core) Allow assistant to evaluate current formula
Summary:
Replaces https://phab.getgrist.com/D3940, particularly to avoid doing potentially unwanted things automatically.

Adds optional fields `evaluateCurrentFormula?: boolean; rowId?: number` to `FormulaAssistanceContext` (part of `AssistanceRequest`). When `evaluateCurrentFormula` is `true`, calls a new function `evaluate_formula` in the sandbox which computes the existing formula in the column (regardless of anything the AI may have suggested) and uses that to generate an additional system message which is added before the user's message. In theory this could be used in an interface where users ask why a formula doesn't work, including possibly a formula suggested by the AI. For now, it's only used in `runCompletion_impl.ts` for experimenting.

Also cleaned up a bit, removing `_chatMode` which is always `true` now, and uses of `regenerate` which is always `false`.

Test Plan: Updated `runCompletion_impl` to optionally use the new feature, in which case it now scores 51/68 instead of 49/68.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3970
2023-07-24 21:59:00 +02:00
Paul Fitzpatrick
14b14f116e (core) export WindowDimensions from gristUtils after refactor
Summary:
WindowDimensions from gristUtils needs exporting
after it got moved to another file.

Test Plan: existing tests should pass

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3971
2023-07-24 15:10:51 -04:00
Paul Fitzpatrick
7256e0c245 (core) updates from grist-core 2023-07-23 11:53:20 -04:00
Paul Fitzpatrick
b054594840
factor out low-dependency browser test code useful for grist-widget (#576)
It would be useful to write browser tests that use Grist for some
of our other repositories (e.g. grist-widget, grist-static).
This is a first baby step to factor out some useful code that has
no code dependencies beyond mocha-webdriver, for use in grist-widget.
2023-07-21 10:05:43 -04:00
George Gevoian
788a6d01ce (core) Move CellColor tests to grist-core
Test Plan: N/A

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3964
2023-07-20 15:17:10 -04:00
George Gevoian
0a34292536 (core) Add telemetry for AI Assistant
Summary: Also fixes a few bugs with some telemetry events not being recorded.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3960
2023-07-20 12:50:26 -04:00
Florent
5e33b68753
Issue 359 support scaleway (#577)
* Fix support of Scaleway S3 bucket #359

While MinIO and AWS return versionId as strings, other S3 API
implementations return versionId as integers.

We must carefully convert the versionId as string in order to cover
these various behaviors.

Also ensure that docStorage is initialized before attempting to
calculate the data size in order to avoid an exception.

* Add unit tests for MinIOExternalStorage#versions() #359

Introduced some unit tests to :
 - ensure listObjects is called with the right arguments;
 - cover the case when a S3 bucket implementation does not return the
   versionId as a string but rather as an integer (like Scaleway):
   in such a case, ensure that the returned snapshotId is a string;
 - cover the case when the listObjects function emits an error, ensure the
   versions() call rejets with the error emitted;
 - that the deleteMarkers are only returned when the
   includeDeleteMarkers is passed;

---------

Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-07-20 06:16:58 -04:00
Jakub Serafin
d894b60fd4 (core) deleting queue from single webhook
Summary: Using standard tost notification, message about webhook queue being overflown was added. message is permanent as long as queue is full. Message contains linkt to the webhook setings

Test Plan: two nbrowser test was added - one to check if message is show when queue is full, and second to check if message is dismiss when queue was cleaned.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3929
2023-07-18 11:46:10 +02:00
Jakub Serafin
a9f4cfde90 (core) API reworked to use POST to create webhook and DELET to remove it
Summary:
introduces POST /api/docs/{docId}/webhooks and DELETE /api/docs/{docId}/webhooks/{webhookId} on place of old _subscribe and _unsubscribe endpoints.
Remove checking for unsubscribeKey while deleting webhook - only owner can delete webhook using DELETE endpoint. subscription key is still needed for _unsubscribe endpoint.
old _unsubscribe and _subscribe endpoints are still active and work as before - no changes there.

Posting schema:

```
POST /api/docs/[docId]/webhooks
```

Request Body:

```
{
    "webhooks": [
        {
            "fields": {
                "url": "https://webhook.site/3bd02246-f122-445e-ba7f-bf5ea5bb6eb1",
                "eventTypes": [
                    "add",
                    "update"
                ],
                "enabled": true,
                "name": "WebhookName",
                "memo": "just a text",
                "tableId": "Table1"
            }
        },
        {
            "fields": {
                "url": "https://webhook.site/3bd02246-f122-445e-ba7f-bf5ea5bb6eb2",
                "eventTypes": [
                    "add",
                ],
                "enabled": true,
                "name": "OtherWebhookName",
                "memo": "just a text",
                "tableId": "Table1"
            }
        }
    ]
}
```

Expected response: WebhookId for each webhook posted:

```
{
    "webhooks": [
        {
            "id": "85c77108-f1e1-4217-a50d-acd1c5996da2"
        },
        {
            "id": "d87a6402-cfd7-4822-878c-657308fcc8c3"
        }
    ]
}
```

Deleting webhooks:

```
DELETE api/docs/[docId]/webhooks/[webhookId]
```

there is no payload in DELETE request. Therefore only one webhook can be deleted at once

Response:

```
{
    "success": true
}
```

Test Plan: Old unit test improved to handle new endpoints, and one more added to check if endpoints are in fact created/removed

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: paulfitz, alexmojaki

Differential Revision: https://phab.getgrist.com/D3916
2023-07-14 15:01:46 +02:00
Dmitry S
8581492912 (core) When a page starts with a number, don't treat it as an emoji
Test Plan: Added a check to the emoji test.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3951
2023-07-12 15:13:25 -04:00
Jarosław Sadziński
d13b9b9019 (core) Billing for formula assistant
Summary:
Adding limits for AI calls and connecting those limits with a Stripe Account.

- New table in homedb called `limits`
- All calls to the AI are not routed through DocApi and measured.
- All products now contain a special key `assistantLimit`, with a default value 0
- Limit is reset every time the subscription has changed its period
- The billing page is updated with two new options that describe the AI plan
- There is a new popup that allows the user to upgrade to a higher plan
- Tiers are read directly from the Stripe product with a volume pricing model

Test Plan: Updated and added

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3907
2023-07-10 13:24:08 +02:00
Jarosław Sadziński
75d979abdb (core) Fixing cursor position for filtered linked section.
Summary:
In a selector table, when a selected row is filtered out of view, linked widgets should update based on the newly selected row.
There were a few bugs that contributed to this wrong behavior:
- Gridview wasn't subscribing to the current row id, and the row with id 'new' was being converted to the first row
- Cursor was keeping track of the currently selected row id, it was hiding a problem behind the proper rowIndex
- Undo/redo somehow leveraged the wrong rowId from the cursor during the position restore.

The `No data` text was also changed to be more meaningful.

Test Plan: Added and updated.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3937
2023-07-07 19:04:30 +02:00
George Gevoian
35237a5835 (core) Add Support Grist page and nudge
Summary:
Adds a new Support Grist page (accessible only in grist-core), containing
options to opt in to telemetry and sponsor Grist Labs on GitHub.

A nudge is also shown in the doc menu, which can be collapsed or permanently
dismissed.

Test Plan: Browser and server tests.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz

Subscribers: jarek, dsagal

Differential Revision: https://phab.getgrist.com/D3926
2023-07-04 17:36:59 -04:00
Paul Fitzpatrick
051c6d52fe (core) updates from grist-core 2023-07-03 02:21:48 -04:00
Dmitry S
b0aa17c932 (core) Detect when a page initial is an emoji, avoid repeating it, and style it better
Summary:
- Detecting emoji is surprisingly tricky; we use a fancy regex as a decent heuristic.
- Icons are a little larger than before.
- Styling tweaked for light and dark modes
- In case the OS doesn't render the emoji as one character, truncate what's
  shown in the icon box.

Test Plan: Added a test case.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3904
2023-06-30 16:57:35 -04:00
Paul Fitzpatrick
70935a4fa4
skip building test harness in docker image (#551)
A small test harness bundle was recently added that is breaking the docker image build. It could be added to the docker image, but that would introduce a bunch of extraneous test file dependencies. So this tweaks the build to simply skip the test bundle if its primary source file is not found.

Also added some other test fixes along the way:
  * make a custom widget test more reliable
  * update a localization test now that `pt` exists
  * store more log info in artifact on error
2023-06-30 05:50:40 -04:00
Paul Fitzpatrick
230e84f48a
avoid test files importing other test files (#550)
There's a little nest of SelectBy tests that sometimes fail.
They are the only tests with an import of a helper function from
an other file that contains tests. Such imports have caused trouble
with mocha in the past. I'm not sure if that is the case now, but
I'd like to eliminate it as a possibility.
2023-06-29 03:15:14 -04:00
Paul Fitzpatrick
01069a69b0
adapt tests after switch to parallel runs (#547)
Some browser tests are now run in parallel. A few tests have become unreliable, and need a little love. Also, create and save mocha webdriver logs.
2023-06-28 17:17:14 -04:00
Florent
71bff10566
Fix GREP_TESTS unbound error in test_under_docker.sh (#549)
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-06-28 09:15:33 -04:00
Alex Hall
bb7cf6ba20 (core) Modify prompt so that model may say it cannot help with certain requests.
Summary:
This tweaks the prompting so that the user's message is given on its own instead of as a docstring within Python. This is so that the prompt makes sense when:

- the user asks a question such as "Can you write me a formula which does ...?" rather than describing their formula as a docstring would, or
- the user sends a message that doesn't ask for a formula at all (https://grist.slack.com/archives/C0234CPPXPA/p1687699944315069?thread_ts=1687698078.832209&cid=C0234CPPXPA)

Also added wording for the model to refuse when the user asks for something that the model cannot do.

Because the code (and maybe in some cases the model) for non-ChatGPT models relies on the prompt consisting entirely of Python code produced by the data engine (which no longer contains the user's message) those code paths have been disabled for now. Updating them now seems like undesirable drag, I think it'd be better to revisit this when iteration/experimentation has slowed down and stabilised.

Test Plan:
Added entries to the formula dataset where the response shouldn't contain a formula, indicated by the value `1` for the new column `no_formula`.

This is somewhat successful, as the model does refuse to help in some of the new test cases, but not all. Performance on existing entries also seems a bit worse, but it's hard to distinguish this from random noise. Hopefully this can be remedied in the future with more work, e.g. automatic followup messages containing example inputs and outputs.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3936
2023-06-27 15:57:56 +02:00
Paul Fitzpatrick
fc16b4c8f6 (core) updates from grist-core 2023-06-27 03:04:42 -04:00
Paul Fitzpatrick
bcbf57d590 (core) bump mocha version to allow parallel tests; move more tests to core
Summary:
This uses a newer version of mocha in grist-core so that tests can be run in parallel. That allows more tests to be moved without slowing things down overall. Tests moved are venerable browser tests; only the ones that "just work" or worked without too much trouble to are moved, in order to keep the diff from growing too large. Will wrestle with more in follow up.

Parallelism is at the file level, rather than the individual test.

The newer version of mocha isn't needed for grist-saas repo; tests are parallelized in our internal CI by other means. I've chosen to allocate files to workers in a cruder way than our internal CI, based on initial characters rather than an automated process. The automated process would need some reworking to be compatible with mocha running in parallel mode.

Test Plan: this diff was tested first on grist-core, then ported to grist-saas so saas repo history will correctly track history of moved files.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3927
2023-06-27 02:55:34 -04:00
Paul Fitzpatrick
7d3b4b49d5 (core) forward more kinds of edits to a virtual table
Summary:
Some edits to virtual tables (such as webhook lists) happen
via a route that was not yet handled. Actually Cyprien (the
original author) had handled this case but it got removed
because I didn't know what it was for :-). This brings back
support for edits by this route.

Test Plan: added a test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3924
2023-06-26 12:05:34 -04:00
Alex Hall
d88f79bc5e (core) Add cases involving lookups to formula dataset
Summary: I looked through the template documents mentioned in `formula-dataset-index.csv` and selected formulas involving lookups to add to the CSV, particularly nontrivial formulas.

Test Plan: Running the test script on the new dataset gives a score of 47/61 compared to the previous 45/47, i.e. it scores 2/14 on the new entries. Lookups are clearly challenging and we'll need to add more information to the prompt, maybe even consider a more complicated strategy than a single prompt. This diff is purely for expanding the dataset, improving performance will come later.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3931
2023-06-26 13:18:52 +02:00
George Gevoian
25b71c4e57 (core) Polish doc tutorials
Summary:
The GristDocTutorial table is now always visible to users with edit
access to the trunk, and the Share menu is now available within
tutorial forks, making it easier for editors to replace the original
tutorial trunk with changes made in the fork, and for viewers to export
their copy of the tutorial.

Also, changes to the GristDocTutorial table are now immediately reflected
in the tutorial popup.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3930
2023-06-23 23:56:20 -04:00
Jarosław Sadziński
e09e8f7160 (core) Cleaning after app sumo and billing discount tests
Summary:
- Cleaning stripe data after billing tests
- Better stripe webhook test integration, that should fix test interference
- Not importing why-is-node-running when its not needed, which improves dev experience.

Test Plan: Modified

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3932
2023-06-23 16:41:45 +02:00
Dmitry S
3fa5125cf7 (core) Highlight rows used as a selector in linking, but do not show 'inactive' cursors.
Summary:
1. Introduces another highlight for link-selector rows, with the same color as
   regular selection, and allowing to overlap with regular selection.
2. Don't show "secondary" cursors (those in inactive sections), to keep a single
   cursor on the screen, since having multiple (which different in color) could
   cause confusion.
3. An unrelated improvement (prompted by a new fixture doc) is to default the
   active section to the top-left one (rather than the one with smallest rowId).
4. Another unrelated improvement (prompted by a test affected by the previous unrelated improvement) is to skip chart widgets when searching (previously search would step through those with an invisible "cursor").

Includes also tweaks for better testing on Arm-based Macs:
- Add support for TEST_CHROME_BINARY_PATH environment variable (helpful for a Mac arm64 architecture workaround)
- Remove unsetting of SELENIUM_REMOTE_URL when running headless (unlikely to affect anyone, and can be done outside the script, but interferes with the Mac workaround)

Test Plan: Added a new test case that cursor and linking-selector CSS classes are present or absent appropriately. Fixed test affected by the fix to default active section.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3891
2023-06-21 12:21:19 -04:00
Alex Hall
52469c5a7e (core) Improve parsing formula from completion
Summary:
The previous code for extracting a Python formula from the LLM completion involved some shaky string manipulation which this improves on.
Overall the 'test results' from `runCompletion` went from 37/47 to 45/47 for `gpt-3.5-turbo-0613`.

The biggest problem that motivated these changes was that it assumed that code was always inside a markdown code block
(i.e. triple backticks) and so if there was no block there was no code. But the completion often consists of *only* code
with no accompanying explanation or markdown. By parsing the completion in Python instead of JS,
we can easily check if the entire completion is valid Python syntax and accept it if it is.

I also noticed one failure resulting from the completion containing the full function (instead of just the body)
and necessary imports before that function instead of inside. The new parsing moves import inside.

Test Plan: Added a Python unit test

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3922
2023-06-16 13:38:20 +02:00
Florent
fa306eba15
Support spaces in GREP_TESTS values (#535)
Not only GREP_TESTS can be assigned a single word like:
GREP_TESTS=DocApi yarn test

But also can be assigned a whole sentence part:
GREP_TESTS="supports ascending sort" yarn test

That's especially useful to run a single test (and not a whole test
suit)

Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2023-06-15 15:45:32 -04:00
George Gevoian
0b64e408b0 (core) Fix scrolling and column title bugs
Summary:
On Firefox and Safari, setting scrollLeft to a max safe integer was
causing it to be treated as 0. It's not clear why - for now, the
scrollWidth is used instead.

Also fixes a bug where the column title popup wouldn't appear for a
new column if tab was previously used to close the same popup for
the last column.

Test Plan: Browser test.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3911
2023-06-15 14:51:10 -04:00
George Gevoian
a460563daf (core) Polish telemetry code
Summary: Also fixes a few small bugs with telemetry collection.

Test Plan: Server and manual tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3915
2023-06-09 13:03:10 -04:00
George Gevoian
10f5f0cb37 (core) Add optional telemetry to grist-core
Summary:
Adds support for optional telemetry to grist-core.

A new environment variable, GRIST_TELEMETRY_LEVEL, controls the level of telemetry collected.

Test Plan: Server and unit tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal, anaisconce

Differential Revision: https://phab.getgrist.com/D3880
2023-06-07 12:00:51 -04:00
Jakub Serafin
90e902c10f (core) sanitizing redis errors
Summary:
sanitazing errors output in webhooks to protect users data (not show them in logs and other places).
Because redis is returing whole payload when error occur, best approach is to hijack exception as close to redis operation as posible and sanitize the data.
We need to know data structure do do this corretly tho. Currently I decided to just censore everything that has "payload" key.

Test Plan: Because logs that need to be sanitized come from redis, to be valid tested we should force redis to crash. It's hard to do in our integration test setup. In this moment, unit test is all we got.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3905
2023-06-06 10:51:17 +02:00
Jarosław Sadziński
da323fb741 (core) Floating formula editor
Summary:
Adding a way to detach an editor. Initially only implemented for the formula editor, includes redesign for the AI part.
- Initially, the detached editor is tight with the formula assistant and both are behind GRIST_FORMULA_ASSISTANT flag, but this can be relaxed
later on, as the detached editor can be used on its own.

- Detached editor is only supported in regular fields and on the creator panel. It is not supported yet for conditional styles, due to preview limitations.
- Old code for the assistant was removed completely, as it was only a temporary solution, but the AI conversation part was copied to the new one.
- Prompting was not modified in this diff, it will be included in the follow-up with more test cases.

Test Plan: Added only new tests; existing tests should pass.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Differential Revision: https://phab.getgrist.com/D3863
2023-06-02 17:59:22 +02:00
Dmitry S
d191859be7 (core) For exporting XLSX, do it memory-efficiently in a worker thread.
Summary:
- Excel exports were awfully memory-inefficient, causing occasional docWorker
  crashes. The fix is to use the "streaming writer" option of ExcelJS
  https://github.com/exceljs/exceljs#streaming-xlsx-writercontents. (Empirically
  on one example, max memory went down from 3G to 100M)
- It's also CPU intensive and synchronous, and can block node for tens of
  seconds. The fix is to use a worker-thread. This diff uses "piscina" library
  for a pool of threads.
- Additionally, adds ProcessMonitor that logs memory and cpu usage,
  particularly when those change significantly.
- Also introduces request cancellation, so that a long download cancelled by
  the user will cancel the work being done in the worker thread.

Test Plan:
Updated previous export tests; memory and CPU performance tested
manually by watching output of ProcessMonitor.

Difference visible in these log excerpts:

Before (total time to serve request 22 sec):
```
Telemetry processMonitor heapUsedMB=2187, heapTotalMB=2234, cpuAverage=1.13, intervalMs=17911
Telemetry processMonitor heapUsedMB=2188, heapTotalMB=2234, cpuAverage=0.66, intervalMs=5005
Telemetry processMonitor heapUsedMB=2188, heapTotalMB=2234, cpuAverage=0, intervalMs=5005
Telemetry processMonitor heapUsedMB=71, heapTotalMB=75, cpuAverage=0.13, intervalMs=5002
```
After (total time to server request 18 sec):
```
Telemetry processMonitor heapUsedMB=109, heapTotalMB=144, cpuAverage=0.5, intervalMs=5001
Telemetry processMonitor heapUsedMB=109, heapTotalMB=144, cpuAverage=1.39, intervalMs=5002
Telemetry processMonitor heapUsedMB=94, heapTotalMB=131, cpuAverage=1.13, intervalMs=5000
Telemetry processMonitor heapUsedMB=94, heapTotalMB=131, cpuAverage=1.35, intervalMs=5001
```
Note in "Before" that heapTotalMB goes up to 2GB in the first case, and "intervalMs" of 17 seconds indicates that node was unresponsive for that long. In the second case, heapTotalMB stays low, and the main thread remains responsive the whole time.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3906
2023-06-01 12:06:48 -04:00
Jarosław Sadziński
c592691e31 (core) Fixing DELETE and BACKSPACE keys on ChoiceList and RefList editor.
Summary:
Choice/Reference List editor wasn't clearing itself when it received an empty string. It led
to a bug on the Card widget where pressing those keys resulted in the same behavior as
pressing Enter - it just opened the editor.
Grid view has it's own implementation for those keys, so it wasn't affected.

Test Plan: Added new test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3908
2023-06-01 16:53:38 +02:00
Paul Fitzpatrick
dad41b2567 (core) updates from grist-core 2023-05-30 08:32:34 -04:00
Paul Fitzpatrick
33d24e50f6
use gu.toggleSidePanel to open side panel (#523)
This hopefully fixes a test that was occasionally failing.
2023-05-24 16:29:14 -04:00
George Gevoian
ff03d32688 (core) Set DateTime timezone during xlsx import
Summary:
DateTime columns had a blank timezone after xlsx imports because the
timezone was not included in the column type. We now append the
document's timezone to the type of all imported DateTime columns.

Test Plan: Server test.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3896
2023-05-24 11:39:49 -04:00
George Gevoian
d5b8240c07 (core) Fix snapshot migrations
Summary:
Migrations were failing in snapshots due to the sandbox no longer
being started in snapshots. We now start up an instance of the
sandbox whenever there are migrations to run, and immediately shut
it down on completion.

Test Plan: Server test.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3898
2023-05-23 15:50:15 -04:00
Paul Fitzpatrick
7be0ee289d
support other SQLite wrappers, and various hooks needed by grist-static (#516) 2023-05-23 15:17:28 -04:00
George Gevoian
bd474a382f
Fix Features and DocTutorial tests (#520) 2023-05-23 14:34:48 -04:00
Paul Fitzpatrick
3f3a0d3aa1 (core) support a wildcard option for ALLOWED_WEBHOOK_DOMAINS
Summary:
Now that webhook payload delivery can be done using a proxy,
it may be desirable to no longer require a set of `ALLOWED_WEBHOOK_DOMAINS`.
This diff allows this variable to be set to `*`. With this setting,
any domain, and both `http` and `https` protocols will now be accepted.

Another possibility would be to default to unchecked
behavior if `ALLOWED_WEBHOOK_DOMAINS` is not set. But this would
introduce a new kind of vulnerability to unconfigured Grist
installations.

Test Plan: switched a test from naming a domain to using `*`

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3903
2023-05-23 13:40:52 -04:00
Dmitry S
d4bc6246f1 (core) Fix breakage on Firefox iOS
Summary:
Grist recently stopped working on Firefox on iOS. The cause turns out an uncaught error, which is reported as an unhelpful "Script Error", but the act of reporting it causes additional errors, leading to an infinite loop and an unusable browser tab.

Firefox-iOS is to blame, but a workaround is preventing a flood of "Script Error" messages. Specifically, we report only the first of these, and only to the server, suppressing the user-visible toast.

Test Plan: Tested manually on Firefox on iOS. Added a test case, and improve other tests of uncaught errors.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3902
2023-05-22 16:22:41 -04:00
George Gevoian
f18bb3e39d (core) Add GRIST_UI_FEATURES env variable
Summary:
Tutorials are now hidden by default in grist-core and grist-ee, and can
be re-enabled via a new env variable, GRIST_UI_FEATURES, which accepts
a comma-separated list of UI features to enable.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3885
2023-05-22 16:05:51 -04:00
Dmitry S
be5cb9124a (core) Add logging of errors whenever ProxyAgent is used, and a test for it.
Summary:
Also:
- Move ProxyAgent to from app/server/utils to app/server/lib, which is
  the more usual place for such classes.
- Refactor a helper (delayAbort) that node was reporting a leak in.

Test Plan: Added a test case, and tested manually.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Subscribers: JakubSerafin, paulfitz

Differential Revision: https://phab.getgrist.com/D3897
2023-05-17 10:21:53 -04:00
Paul Fitzpatrick
84854b7cfa (core) updates from grist-core 2023-05-15 12:01:19 -04:00
Paul Fitzpatrick
b58929f095 (core) reconcile webhook and widget description migrations
Summary:
Due to a mishap, two distinct migrations with the same migration
number were introduced into Grist. This diff reconciles them as
best we can, by adding another migration to make sure both desired
changes have run (and running them if not).

Test Plan:
updated a test; checked manually that documents
with different 38 migrations are handled as expected.

Reviewers: georgegevoian, jarek

Reviewed By: georgegevoian, jarek

Differential Revision: https://phab.getgrist.com/D3895
2023-05-15 11:56:15 -04:00
CamilleLegeron
c16204f8ad
feature widget description (#483)
Add description to widget title popup and right panel
2023-05-12 09:08:28 -04:00
George Gevoian
18ad39cba3 (core) Add cut, copy, and paste to context menu
Summary:
On supported browsers, the new context menu commands work exactly as they do
via keyboard shortcuts. On unsupported browsers, an unavailable command
modal is shown with a suggestion to use keyboard shortcuts instead.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3867
2023-05-10 00:48:15 -04:00
Paul Fitzpatrick
603238e966 (core) Adds a UI panel for managing webhooks
Summary:
This adds a UI panel for managing webhooks. Work started by Cyprien Pindat. You can find the UI on a document's settings page. Main changes relative to Cyprien's demo:

  * Changed behavior of virtual table to be more consistent with the rest of Grist, by factoring out part of the implementation of on-demand tables.
  * Cell values that would create an error can now be denied and reverted (as for the rest of Grist).
  * Changes made by other users are integrated in a sane way.
  * Basic undo/redo support is added using the regular undo/redo stack.
  * The table list in the drop-down is now updated if schema changes.
  * Added a notification from back-end when webhook status is updated so constant polling isn't needed to support multi-user operation.
  *  Factored out webhook specific logic from general virtual table support.
  * Made a bunch of fixes to various broken behavior.
  * Added tests.

The code remains somewhat unpolished, and behavior in the presence of errors is imperfect in general but may be adequate for this case.

I assume that we'll soon be lifting the restriction on the set of domains that are supported for webhooks - otherwise we'd want to provide some friendly way to discover that list of supported domains rather than just throwing an error.

I don't actually know a lot about how the front-end works - it looks like tables/columns/fields/sections can be safely added if they have string ids that won't collide with bone fide numeric ids from the back end. Sneaky.

Contains a migration, so needs an extra reviewer for that.

Test Plan: added tests

Reviewers: jarek, dsagal

Reviewed By: jarek, dsagal

Differential Revision: https://phab.getgrist.com/D3856
2023-05-08 18:25:27 -04:00
Paul Fitzpatrick
5e9f2e06ea (core) updates from grist-core 2023-05-08 14:17:45 -04:00
George Gevoian
9438f315e9 (core) Save choice config on focus loss
Summary: Changes to choices are now saved whenever focus leaves the editor.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3879
2023-05-08 14:16:10 -04:00
Paul Fitzpatrick
51a195bd94
add support for conversational state to assistance endpoint (#506)
* add support for conversational state to assistance endpoint

This refactors the assistance code somewhat, to allow carrying
along some conversational state. It extends the OpenAI-flavored
assistant to make use of that state to have a conversation.
The front-end is tweaked a little bit to allow for replies that
don't have any code in them (though I didn't get into formatting
such replies nicely).

Currently tested primarily through the runCompletion script,
which has been extended a bit to allow testing simulated
conversations (where an error is pasted in follow-up, or
an expected-vs-actual comparison).

Co-authored-by: George Gevoian <85144792+georgegevoian@users.noreply.github.com>
2023-05-08 14:15:22 -04:00
Jakub Serafin
440d5b935a (core) Proxy Agent moved to the separate file, Triggers are using proxy now to perform fetch
Summary:
- Webhooks form Triggers.ts should now use proxy if it's configured
- Proxy handling code separated to ProxyAgent.ts
- Tests for ProxyAgent
- Integration/API Tests for using Proxy in webhooks
- a bit of refactor - proxy test uses mostly the same codebase as DocApi.ts, but because last one if over 4000 lines long, I've put it into separated file, and extract some common parts (there is some duplicates tho)
- some cleanup in files that I've touched

Test Plan:
Manual test to check if proxy is used on the staging env

Automatic test checking if (fake) proxy was called

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3860
2023-05-08 11:54:09 +02:00
Jakub Serafin
37347a79c0 (core) Currency from grist column is persistent when exporting to excel
Summary:
- when grist table is exported, currency is check and introduced in cell format in the form of "[currency symbol] [value]" (for example: zł 10000, $ 5000) . It's not what some cultures should display currences, but it's close enought
- when no symbol is defined for the currency, currency 3 letters code is used instead
- when currency is unknown, we are falling back to "$"

Test Plan: - nbrowser test scenario added for that purpose, please check Currences.xlsx to see output format exported.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3886
2023-05-08 10:39:01 +02:00
Jarosław Sadziński
ca3cf0cd06 (core) Preserving rules when data is transformed
Summary: Rules where removed when data in column was transformed.

Test Plan: Added new test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3883
2023-05-05 13:44:09 +02:00
Paul Fitzpatrick
8810aa3bd3 (core) port test/home tests
Summary:
This ports the useful parts of the test/home tests to test/nbrowser (a chunk of the DocMenu tests were already covered).

I ripped out a chunk of test/browser code that is now no longer used.

I made a few changes to unrelated tests that happened to fail.

Test Plan: ported tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3888
2023-05-04 14:45:04 -04:00
Jarosław Sadziński
5f9f4868ae (core) Showing close button when column is added
Summary:
When a column is added the rename popup had a disabled save button.
Now we always show either:
Just "Close" if there are no changes.
"Save" and "Cancel" if there are changes.

Also, the description is trimmed when saved through the creator panel.

Test Plan: Added

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3872
2023-05-04 11:35:16 +02:00
George Gevoian
959f8a45c6 (core) Direct users to last visited site when possible
Summary:
When clicking the logo in the top-left corner, or finishing a tutorial, we
now direct users to the site they last visited, if possible. If unknown, a
new redirect endpoint, /welcome/home, is used instead, which directs users
to a sensible location based on the number of sites they have.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3878
2023-05-02 07:48:22 -07:00
Dmitry S
65013331a3 (core) Fix imports into reference columns, and support two ways to import Numeric as a reference.
Summary:
- When importing into a Ref column, use lookupOne() formula for correct previews.
- When selecting columns to import into a Ref column, now a Numeric column like
  'Order' will produce two options: "Order" and "Order (as row ID)".
- Fixes exports to correct the formatting of visible columns. This addresses multiple bugs:
  1. Formatting wasn't used, e.g. a Ref showing a custom-formatted date was still presented as YYYY-MM-DD in CSVs.
  2. Ref showing a Numeric column was formatted as if a row ID (e.g. `Table1[1.5]`), which is very wrong.
- If importing into a table that doesn't have a primary view, don't switch page after import.

Refactorings:
- Generalize GenImporterView to be usable in more cases; removed near-duplicated logic from node side
- Some other refactoring in importing code.
- Fix field/column option selection in ValueParser
- Add NUM() helper to turn integer-valued floats into ints, useful for "as row ID" lookups.

Test Plan: Added test cases for imports into reference columns, updated Exports test fixtures.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3875
2023-05-02 10:28:14 -04:00
Dmitry S
b4cc519616 (core) Ignore leading whitespace in formulas, and strip out leading '=' sign users might add
Summary:
This addresses two issues, differently:
- For a formula with leading whitespace, like " 1+1", it is stored as is, but
  is fixed to work (it should be valid Python, and whitespace is only stripped out
  at parsing time to avoid intentation errors caused by the way it gets parsed)
- For a formula with a leading equals-sign ("="), it is stripped out on the
  client side before the formula is stored. Grist documentation uses leading
  "=" to indicate formulas (because UI shows an "=" icon), and Excel formulas
  actually contain the leading "=", so it is a common mistake to include it.

Test Plan: Added new test cases

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3873
2023-04-25 15:28:40 -04:00