Commit Graph

461 Commits

Author SHA1 Message Date
George Gevoian
0a34292536 (core) Add telemetry for AI Assistant
Summary: Also fixes a few bugs with some telemetry events not being recorded.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3960
2023-07-20 12:50:26 -04:00
Alex Hall
7fd48364df (core) Improved error messages, retries, and handling of token limits in AI assistant
Summary:
In a nutshell:

- More specific and helpful error messages are shown to the user
- API requests are only retried when needed
- The system deals with reaching the maximum token limit better, especially by switching to a model with a bigger limit

In more detail:

- `COMPLETION_MODEL` configuration has been removed. By default `gpt-3.5-turbo-0613` is used which accepts 4k tokens. If that's not enough, `gpt-3.5-turbo-16k-0613` is used instead.
- Switching to the bigger model happens when either the prompt is too long by itself (the API immediately returns an error code) or the model reaches the 4k limit itself in the process of generating a response and thus returns an incomplete response. The latter case is made possible by removing the `max_tokens: 1500` in the request, which was very generous and would have lead to switching to the more expensive model more often than needed. The downside is that the user has to wait a bit longer for the response.
- If the bigger 16k token limit is also exceeded, the assistant immediately responds (instead of retrying as before) with an error message including suggestions. The suggestions include restarting the conversation if and only if the user has sent multiple messages.
- If a request fails because Grist has reached its OpenAI monthly billing quota, the assistant immediately responds (instead of retrying as before) with an error message suggesting that the user try again tomorrow.
- If a request fails for some other reason, the assistant retries, and if all attempts fail then the user is told to try again in a few minutes and is shown the exact error message, including the API response if there is one.
- Retrying only happens when an API request fails, whereas previously the system also retried errors from a much bigger scope which included calls to the sandbox. The downside is that the hugging face assistant no longer retries, although that code is currently disabled anyway.
- The assistant no longer waits an additional second after the final retry attempt fails.

Test Plan: Added a new server test file with several unit tests using faked OpenAI responses, including the happy path which wasn't really tested before.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3955
2023-07-18 16:01:37 +02:00
Jakub Serafin
d894b60fd4 (core) deleting queue from single webhook
Summary: Using standard tost notification, message about webhook queue being overflown was added. message is permanent as long as queue is full. Message contains linkt to the webhook setings

Test Plan: two nbrowser test was added - one to check if message is show when queue is full, and second to check if message is dismiss when queue was cleaned.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3929
2023-07-18 11:46:10 +02:00
Jakub Serafin
a9f4cfde90 (core) API reworked to use POST to create webhook and DELET to remove it
Summary:
introduces POST /api/docs/{docId}/webhooks and DELETE /api/docs/{docId}/webhooks/{webhookId} on place of old _subscribe and _unsubscribe endpoints.
Remove checking for unsubscribeKey while deleting webhook - only owner can delete webhook using DELETE endpoint. subscription key is still needed for _unsubscribe endpoint.
old _unsubscribe and _subscribe endpoints are still active and work as before - no changes there.

Posting schema:

```
POST /api/docs/[docId]/webhooks
```

Request Body:

```
{
    "webhooks": [
        {
            "fields": {
                "url": "https://webhook.site/3bd02246-f122-445e-ba7f-bf5ea5bb6eb1",
                "eventTypes": [
                    "add",
                    "update"
                ],
                "enabled": true,
                "name": "WebhookName",
                "memo": "just a text",
                "tableId": "Table1"
            }
        },
        {
            "fields": {
                "url": "https://webhook.site/3bd02246-f122-445e-ba7f-bf5ea5bb6eb2",
                "eventTypes": [
                    "add",
                ],
                "enabled": true,
                "name": "OtherWebhookName",
                "memo": "just a text",
                "tableId": "Table1"
            }
        }
    ]
}
```

Expected response: WebhookId for each webhook posted:

```
{
    "webhooks": [
        {
            "id": "85c77108-f1e1-4217-a50d-acd1c5996da2"
        },
        {
            "id": "d87a6402-cfd7-4822-878c-657308fcc8c3"
        }
    ]
}
```

Deleting webhooks:

```
DELETE api/docs/[docId]/webhooks/[webhookId]
```

there is no payload in DELETE request. Therefore only one webhook can be deleted at once

Response:

```
{
    "success": true
}
```

Test Plan: Old unit test improved to handle new endpoints, and one more added to check if endpoints are in fact created/removed

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: paulfitz, alexmojaki

Differential Revision: https://phab.getgrist.com/D3916
2023-07-14 15:01:46 +02:00
George Gevoian
ea8a59c5e9 (core) Implement AI Assistant UI V2
Summary:
Implements the latest design of the Formula AI Assistant.

Also switches out brace to the latest build of ace.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3949
2023-07-13 10:30:35 -04:00
Jarosław Sadziński
d13b9b9019 (core) Billing for formula assistant
Summary:
Adding limits for AI calls and connecting those limits with a Stripe Account.

- New table in homedb called `limits`
- All calls to the AI are not routed through DocApi and measured.
- All products now contain a special key `assistantLimit`, with a default value 0
- Limit is reset every time the subscription has changed its period
- The billing page is updated with two new options that describe the AI plan
- There is a new popup that allows the user to upgrade to a higher plan
- Tiers are read directly from the Stripe product with a volume pricing model

Test Plan: Updated and added

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3907
2023-07-10 13:24:08 +02:00
Jarosław Sadziński
75d979abdb (core) Fixing cursor position for filtered linked section.
Summary:
In a selector table, when a selected row is filtered out of view, linked widgets should update based on the newly selected row.
There were a few bugs that contributed to this wrong behavior:
- Gridview wasn't subscribing to the current row id, and the row with id 'new' was being converted to the first row
- Cursor was keeping track of the currently selected row id, it was hiding a problem behind the proper rowIndex
- Undo/redo somehow leveraged the wrong rowId from the cursor during the position restore.

The `No data` text was also changed to be more meaningful.

Test Plan: Added and updated.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3937
2023-07-07 19:04:30 +02:00
George Gevoian
35237a5835 (core) Add Support Grist page and nudge
Summary:
Adds a new Support Grist page (accessible only in grist-core), containing
options to opt in to telemetry and sponsor Grist Labs on GitHub.

A nudge is also shown in the doc menu, which can be collapsed or permanently
dismissed.

Test Plan: Browser and server tests.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz

Subscribers: jarek, dsagal

Differential Revision: https://phab.getgrist.com/D3926
2023-07-04 17:36:59 -04:00
Dmitry S
2b581ab7dc (core) Fix issue with lodash's map interpreting objects with length as array-like
Summary:
Here's a series of badness that easily leads to a crash, in reverse order:
- Lodash's map() function interprets an object with a .length property as an array.
- Some very old code generated human-friendly descriptions of user actions,
  applying map() to parts of them. It so happens that this generated description
  isn't even used.
- If a user action is encountered with a sufficiently large length propery,
  map() would exhaust the server memory.

Fixed by removing old unneeded code, and replacing some other occurrences of
lodash's map() with native equivalents.

Test Plan: Tested manually on a local reproduction of the issue.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3938
2023-07-01 21:07:39 -04:00
Dmitry S
b0aa17c932 (core) Detect when a page initial is an emoji, avoid repeating it, and style it better
Summary:
- Detecting emoji is surprisingly tricky; we use a fancy regex as a decent heuristic.
- Icons are a little larger than before.
- Styling tweaked for light and dark modes
- In case the OS doesn't render the emoji as one character, truncate what's
  shown in the icon box.

Test Plan: Added a test case.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3904
2023-06-30 16:57:35 -04:00
Paul Fitzpatrick
7d3b4b49d5 (core) forward more kinds of edits to a virtual table
Summary:
Some edits to virtual tables (such as webhook lists) happen
via a route that was not yet handled. Actually Cyprien (the
original author) had handled this case but it got removed
because I didn't know what it was for :-). This brings back
support for edits by this route.

Test Plan: added a test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3924
2023-06-26 12:05:34 -04:00
Dmitry S
41280a31f2 (core) Fix recording of signups, and record as telemetry too.
Summary:
Previously we failed to log signup info for users who signed up via
Google. This fixes that issue by recording it on first post-signup
visit. It also includes signup as a new telemetry event, recorded at the
same point.

Test Plan: Tested locally to see that a signup produces an appropriate log message and telemetry event.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3921
2023-06-15 00:15:18 -04:00
Dmitry S
2740884e3c (core) Improve the look and behavior of /welcome/teams page (also shown for /welcome/start)
Summary:
- Move css module for the login page css to core/, to be reusable in core/ pages.
- Move /welcome/teams implementation to WelcomeSitePicker.ts
- List users for personal sites, as well as team sites.
- Add org param to setSessionActive() API method and end endpoint, to allow
  switching the specified org to another user.
- Add a little safety to getOrgUrl() function.

Test Plan: Added a test case for the new behaviors of the /welcome/teams page.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3914
2023-06-13 20:40:59 -04:00
George Gevoian
a460563daf (core) Polish telemetry code
Summary: Also fixes a few small bugs with telemetry collection.

Test Plan: Server and manual tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3915
2023-06-09 13:03:10 -04:00
Alex Hall
6ac0bc3bbb (core) Implement exported functions without relying on ActiveDoc.docData
Summary:
For grist-static, we want to the data engine to be able to call external/exported JS functions directly,
rather than via the node 'server' living in another thread which requires synchronous communication hackery.

As a step in that direction, this diff changes the exported functions that we care about (guessColInfo and convertFromColumn)
to just using the top-level functions instead of relying on fields in ActiveDoc, namely docData.

For guessColInfo, this is done by directly passing the small amount of metadata that was previously retrieved from the DocData.

For convertFromColumn, disentangling DocData is a lot more complicated, so instead we construct a fresh DocData object using
the required metadata tables which are now passed in by the data engine.

Test Plan: Existing tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3913
2023-06-07 22:30:01 +02:00
George Gevoian
10f5f0cb37 (core) Add optional telemetry to grist-core
Summary:
Adds support for optional telemetry to grist-core.

A new environment variable, GRIST_TELEMETRY_LEVEL, controls the level of telemetry collected.

Test Plan: Server and unit tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal, anaisconce

Differential Revision: https://phab.getgrist.com/D3880
2023-06-07 12:00:51 -04:00
Paul Fitzpatrick
0fb5092e33 (core) updates from grist-core 2023-06-05 08:33:28 -04:00
Jarosław Sadziński
da323fb741 (core) Floating formula editor
Summary:
Adding a way to detach an editor. Initially only implemented for the formula editor, includes redesign for the AI part.
- Initially, the detached editor is tight with the formula assistant and both are behind GRIST_FORMULA_ASSISTANT flag, but this can be relaxed
later on, as the detached editor can be used on its own.

- Detached editor is only supported in regular fields and on the creator panel. It is not supported yet for conditional styles, due to preview limitations.
- Old code for the assistant was removed completely, as it was only a temporary solution, but the AI conversation part was copied to the new one.
- Prompting was not modified in this diff, it will be included in the follow-up with more test cases.

Test Plan: Added only new tests; existing tests should pass.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Differential Revision: https://phab.getgrist.com/D3863
2023-06-02 17:59:22 +02:00
jarek
9edbf9f415
Setting main grist-widget repository as a default (#524)
* Setting main grist-widget repository as a default to GRIST_WIDGET_LIST_URL

* Putting widget repository URL in commonURLs
2023-06-02 10:17:09 +02:00
Dmitry S
d191859be7 (core) For exporting XLSX, do it memory-efficiently in a worker thread.
Summary:
- Excel exports were awfully memory-inefficient, causing occasional docWorker
  crashes. The fix is to use the "streaming writer" option of ExcelJS
  https://github.com/exceljs/exceljs#streaming-xlsx-writercontents. (Empirically
  on one example, max memory went down from 3G to 100M)
- It's also CPU intensive and synchronous, and can block node for tens of
  seconds. The fix is to use a worker-thread. This diff uses "piscina" library
  for a pool of threads.
- Additionally, adds ProcessMonitor that logs memory and cpu usage,
  particularly when those change significantly.
- Also introduces request cancellation, so that a long download cancelled by
  the user will cancel the work being done in the worker thread.

Test Plan:
Updated previous export tests; memory and CPU performance tested
manually by watching output of ProcessMonitor.

Difference visible in these log excerpts:

Before (total time to serve request 22 sec):
```
Telemetry processMonitor heapUsedMB=2187, heapTotalMB=2234, cpuAverage=1.13, intervalMs=17911
Telemetry processMonitor heapUsedMB=2188, heapTotalMB=2234, cpuAverage=0.66, intervalMs=5005
Telemetry processMonitor heapUsedMB=2188, heapTotalMB=2234, cpuAverage=0, intervalMs=5005
Telemetry processMonitor heapUsedMB=71, heapTotalMB=75, cpuAverage=0.13, intervalMs=5002
```
After (total time to server request 18 sec):
```
Telemetry processMonitor heapUsedMB=109, heapTotalMB=144, cpuAverage=0.5, intervalMs=5001
Telemetry processMonitor heapUsedMB=109, heapTotalMB=144, cpuAverage=1.39, intervalMs=5002
Telemetry processMonitor heapUsedMB=94, heapTotalMB=131, cpuAverage=1.13, intervalMs=5000
Telemetry processMonitor heapUsedMB=94, heapTotalMB=131, cpuAverage=1.35, intervalMs=5001
```
Note in "Before" that heapTotalMB goes up to 2GB in the first case, and "intervalMs" of 17 seconds indicates that node was unresponsive for that long. In the second case, heapTotalMB stays low, and the main thread remains responsive the whole time.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3906
2023-06-01 12:06:48 -04:00
Paul Fitzpatrick
dad41b2567 (core) updates from grist-core 2023-05-30 08:32:34 -04:00
George Gevoian
ff03d32688 (core) Set DateTime timezone during xlsx import
Summary:
DateTime columns had a blank timezone after xlsx imports because the
timezone was not included in the column type. We now append the
document's timezone to the type of all imported DateTime columns.

Test Plan: Server test.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3896
2023-05-24 11:39:49 -04:00
Paul Fitzpatrick
7be0ee289d
support other SQLite wrappers, and various hooks needed by grist-static (#516) 2023-05-23 15:17:28 -04:00
George Gevoian
f18bb3e39d (core) Add GRIST_UI_FEATURES env variable
Summary:
Tutorials are now hidden by default in grist-core and grist-ee, and can
be re-enabled via a new env variable, GRIST_UI_FEATURES, which accepts
a comma-separated list of UI features to enable.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3885
2023-05-22 16:05:51 -04:00
George Gevoian
1e873b4203 (core) Tweak telemetry
Summary: Adjusts the level of telemetry collected from Grist SaaS.

Test Plan: Tested manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3899
2023-05-19 13:06:15 -04:00
Paul Fitzpatrick
b58929f095 (core) reconcile webhook and widget description migrations
Summary:
Due to a mishap, two distinct migrations with the same migration
number were introduced into Grist. This diff reconciles them as
best we can, by adding another migration to make sure both desired
changes have run (and running them if not).

Test Plan:
updated a test; checked manually that documents
with different 38 migrations are handled as expected.

Reviewers: georgegevoian, jarek

Reviewed By: georgegevoian, jarek

Differential Revision: https://phab.getgrist.com/D3895
2023-05-15 11:56:15 -04:00
Paul Fitzpatrick
603238e966 (core) Adds a UI panel for managing webhooks
Summary:
This adds a UI panel for managing webhooks. Work started by Cyprien Pindat. You can find the UI on a document's settings page. Main changes relative to Cyprien's demo:

  * Changed behavior of virtual table to be more consistent with the rest of Grist, by factoring out part of the implementation of on-demand tables.
  * Cell values that would create an error can now be denied and reverted (as for the rest of Grist).
  * Changes made by other users are integrated in a sane way.
  * Basic undo/redo support is added using the regular undo/redo stack.
  * The table list in the drop-down is now updated if schema changes.
  * Added a notification from back-end when webhook status is updated so constant polling isn't needed to support multi-user operation.
  *  Factored out webhook specific logic from general virtual table support.
  * Made a bunch of fixes to various broken behavior.
  * Added tests.

The code remains somewhat unpolished, and behavior in the presence of errors is imperfect in general but may be adequate for this case.

I assume that we'll soon be lifting the restriction on the set of domains that are supported for webhooks - otherwise we'd want to provide some friendly way to discover that list of supported domains rather than just throwing an error.

I don't actually know a lot about how the front-end works - it looks like tables/columns/fields/sections can be safely added if they have string ids that won't collide with bone fide numeric ids from the back end. Sneaky.

Contains a migration, so needs an extra reviewer for that.

Test Plan: added tests

Reviewers: jarek, dsagal

Reviewed By: jarek, dsagal

Differential Revision: https://phab.getgrist.com/D3856
2023-05-08 18:25:27 -04:00
Paul Fitzpatrick
51a195bd94
add support for conversational state to assistance endpoint (#506)
* add support for conversational state to assistance endpoint

This refactors the assistance code somewhat, to allow carrying
along some conversational state. It extends the OpenAI-flavored
assistant to make use of that state to have a conversation.
The front-end is tweaked a little bit to allow for replies that
don't have any code in them (though I didn't get into formatting
such replies nicely).

Currently tested primarily through the runCompletion script,
which has been extended a bit to allow testing simulated
conversations (where an error is pasted in follow-up, or
an expected-vs-actual comparison).

Co-authored-by: George Gevoian <85144792+georgegevoian@users.noreply.github.com>
2023-05-08 14:15:22 -04:00
Dmitry S
8f34ba5157 (core) Remove a defunct URL constant and a product flavor.
Summary:
The URL /create-team-site on marketing site is unused, and no longer pointing to
anything functional.

The "efcr" product flavor has been defunct for a long time. Remove
references to it.

Test Plan: No tests should be affected

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Subscribers: JakubSerafin

Differential Revision: https://phab.getgrist.com/D3890
2023-05-05 18:28:04 -04:00
Dmitry S
65013331a3 (core) Fix imports into reference columns, and support two ways to import Numeric as a reference.
Summary:
- When importing into a Ref column, use lookupOne() formula for correct previews.
- When selecting columns to import into a Ref column, now a Numeric column like
  'Order' will produce two options: "Order" and "Order (as row ID)".
- Fixes exports to correct the formatting of visible columns. This addresses multiple bugs:
  1. Formatting wasn't used, e.g. a Ref showing a custom-formatted date was still presented as YYYY-MM-DD in CSVs.
  2. Ref showing a Numeric column was formatted as if a row ID (e.g. `Table1[1.5]`), which is very wrong.
- If importing into a table that doesn't have a primary view, don't switch page after import.

Refactorings:
- Generalize GenImporterView to be usable in more cases; removed near-duplicated logic from node side
- Some other refactoring in importing code.
- Fix field/column option selection in ValueParser
- Add NUM() helper to turn integer-valued floats into ints, useful for "as row ID" lookups.

Test Plan: Added test cases for imports into reference columns, updated Exports test fixtures.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3875
2023-05-02 10:28:14 -04:00
Paul Fitzpatrick
f490854324 (core) updates from grist-core 2023-04-17 09:58:04 -04:00
Vinschni
bcb5916434
make sendToDrive hidable (#493)
Make "sendToDrive" button from share menu hidable.
2023-04-15 16:22:56 -04:00
Jarosław Sadziński
40ea6bb2bc (core) Focusing the creator panel on the table wiget for charts and custom views
Summary:
When a chart page/section is added and the creator panel is already open, focus is set to a "table" tab.
When a custom view is added as a page/section, the same thing happens, but there is also a behavioral tooltip shown
for the custom URL.

Test Plan: Updated and added.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3857
2023-04-14 12:45:24 +02:00
George Gevoian
8a0bb4d4fe (core) Improve dark mode
Summary:
Enhances dark mode support for the formula editor, and adds support to
the color select popup. Also fixes some bugs with dark mode.

Test Plan: Tested manually.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3847
2023-04-12 01:58:48 -04:00
George Gevoian
a19ba0813a (core) Add telemetry
Test Plan: Server tests.

Reviewers: jarek

Differential Revision: https://phab.getgrist.com/D3818
2023-04-06 12:34:54 -04:00
Cyprien P
d8a063284a (core) Adds endpoint to update webhook
Summary:
Adds a new endpoint to update webhook.

Perform some refactoring to allow code reuse from endpoint allowing to _subscribe and _unsubscribe webhooks.

One aspect of webhook is that url are stored in the home db while the rest of the fields (tableRef, isReadyColRef, ...) are stored in sqlite. So care must be taken when updating fields, to properly rollback if anything should fail.

Follow up diff will bring UI to edit webhook list

Test Plan: Updated doc api server tests

Reviewers: jarek

Reviewed By: jarek

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3821
2023-03-31 19:26:02 +02:00
Jarosław Sadziński
a317a727c8 (core) Adding tutorial card
Summary:
For now only html stub and docList adjustement for showing a tutorial card.
It will be used in future diffs after tutorial implementation.

Test Plan: Manual

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3750
2023-03-28 19:57:52 +02:00
George Gevoian
01fbe871aa (core) Add April Fools easter egg
Summary: What happens when you type "rr" instead of "r" in an anchor link's row number?

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3829
2023-03-27 14:12:52 -04:00
Jarosław Sadziński
d29770511c (core) Draft version of AI assistant
Summary:
The feature is behind a flag GRIST_FORMULA_ASSISTANT (must be "true"). But can be enabled in the
developer console by invoking GRIST_FORMULA_ASSISTANT.set(true).

Keys can be overriden in the document settings page.

Test Plan: For now just a stub test that checks if this feature is disabled by default.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3815
2023-03-24 10:07:26 +01:00
George Gevoian
be8e13df64 (core) Add initial tutorials implementation
Summary:
Documents can now be flagged as tutorials, which causes them to display
Markdown-formatted slides from a special GristDocTutorial table. Tutorial
documents are forked on open, and remember the last slide a user was on.
They can be restarted too, which prepares a new fork of the tutorial.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3813
2023-03-22 10:09:02 -04:00
Paul Fitzpatrick
572995f19a (core) updates from grist-core 2023-03-20 09:46:37 -04:00
Louis Delbosc
c54e910fd6
Export table schema (#459)
* add endpoint
* Add table-schema transformation data
2023-03-16 17:37:24 -04:00
Cyprien P
1ff93f89c2 (core) Porting the AI evaluation script
Summary:
Porting script that run an evaluation against our formula dataset.

To test you need an openai key (see here: https://platform.openai.com/)
or hugging face (it should work as well), then checkout the branch and run

`OPENAI_API_KEY=<my_openai_api_key> node core/test/formula-dataset/runCompletion.js`

Test Plan:
Needs manually testing: so far there is no plan to make it part of CI.

The current score is somewhere around 34 successful prompts over a total of 47.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3816
2023-03-15 14:54:28 +01:00
Jarosław Sadziński
b3590c8a6f (core) Removing warnings about deprecated shortcuts.
Summary:
Warnings about deprecated shortcuts are no longer needed.
As a side effect it fixes a bug that caused those warnings to not persist its
state on pages with charts.

Test Plan: Removed

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3820
2023-03-15 10:50:14 +01:00
Jarosław Sadziński
59cf654190 (core) Minimazing widgets
Summary:
A feature that allows minimizing widgets on the ViewLayout.
- Code in ViewLayout and Layout hasn't been changed. Only some methods or variables were made public, and some events are now triggered when a section is dragged.
- Widgets can be collapsed or expanded (added back to the main area)
- Collapsed widgets can be expanded and shown as a popup
- Collapsed widgets support drugging, reordering, and transferring between the main and collapsed areas.

Test Plan: New test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3779
2023-03-02 11:22:49 +01:00
George Gevoian
aa3faae25b (core) Fix test util and upgrade fixtures
Summary:
Fixes a test util that broke after a sync with grist-core, and upgrades
fixtures after a migration was added.

Test Plan: N/A

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3810
2023-03-01 10:39:09 -05:00
jarek
cee0cdcd67
Merge pull request #406 from incubateur-territoires/column-description
feat: Add a description to a grist table column
2023-02-23 17:16:17 +01:00
Paul Fitzpatrick
cbf925aa00 (core) updates from grist-core 2023-02-21 10:49:19 -05:00
George Gevoian
1ac4931c22 (core) Persist forks in home db
Summary:
Adds information about forks to the home db. This will be used
later by the UI to list forks of documents.

Test Plan: Browser and server tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3772
2023-02-20 22:46:36 -05:00
Paul Fitzpatrick
3aba7f6208 (core) set up a trialing list and a call scheduled flag for Grist SaaS
Summary:
This adds support for the automation described in
https://grist.quip.com/rt3tAoEH4kuy/Automate-Sprouts-outreach-email

Test Plan: added tests; will need manual testing and iterative integration work

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3804
2023-02-20 15:32:58 -05:00
Paul Fitzpatrick
f7f76fb5e7
A set of tweaks to simplify electron packaging (#421)
* Replace `ormconfig.js` with a newer mechanism of configuring
    TypeORM that can be included in the source code properly.
    The path to `ormconfig.js` has always been awkward to handle,
    and eliminating the file makes building different Grist setups
    a bit simpler.
  * Remove `electron` package. It is barely used, just for some old
    remnants of an older attempt at electron packaging. It was used
    for two types, which I left at `any` for now. More code pruning is
    no doubt possible here, but I'd rather do it when Electron packaging
    has solidified.
  * Add a hook for replacing the login system, and for adding some
    extra middleware the login system may need.
  * Add support for some more possible locations of Python, which
    arise when a standalone version of it is included in the Electron
    package. This isn't very general purpose, just configurations
    that I found useful.
  * Support using grist-core within a yarn workspace - the only tweak
    needed was webpack related.
  * Allow an external ID to be optionally associated with documents.
2023-02-13 15:52:17 -05:00
Jarosław Sadziński
6e3f0f2b35 (core) Porting back AI formula backend
Summary: This is a backend part for the formula AI.

Test Plan: New tests

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: cyprien

Differential Revision: https://phab.getgrist.com/D3786
2023-02-08 17:15:59 +01:00
Paul Fitzpatrick
363fb7f89e (core) updates from grist-core 2023-02-06 08:51:26 -05:00
jarek
5a324f1358
Adding new flags for supported locales (#420)
* Adding new flags for supported locales
* Removing pl-PL locale code from tests
* Linting
2023-02-03 18:56:24 -05:00
George Gevoian
6ccc195330 (core) Add dark theme to date picker
Summary:
Updates the date picker to use a dark theme when dark mode
is enabled.

Test Plan: Manual.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3780
2023-02-03 10:37:12 -05:00
Camille
95f1a41618 Merge branch 'main' into column-description 2023-02-01 09:37:19 +01:00
Cyprien P
d2f98b4036 (core) Fix the empty values last option in multicol sorting
Summary:
The "Empty values last" options was breaking sort when multiple
columns are involved in the sort spec. Problem comes from wrong
handling of equals.

Diff fixes that issue and update test.

Test Plan: Updates nbrowser test

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3774
2023-01-31 12:14:22 +01:00
Camille
51f6cf3070 feat(columnDesc): code styling 2023-01-26 14:54:37 +01:00
Camille
009ebefd96 feat(ColumnDesc): create column description in database and link it to visual behaviors 2023-01-26 14:54:37 +01:00
Jarosław Sadziński
90d3ee037a (core) User language switcher
Summary:
New language selector on the Account page for logged-in users.
New icon for switching language for an anonymous user.

For anonymous users, language is stored in a cookie grist_user_locale.
Language is stored in user settings for authenticated users and takes
precedence over what is stored in the cookie.

Test Plan: New tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3766
2023-01-26 09:47:14 +01:00
Cyprien P
a8827b152e (core) New document settings page
Summary:
This diff implement new page for document settings which replaces the old modal settings.
Diff also adds a new `Settings + API` page item below tools in the left panel that link to that very doc settings page.

Test Plan: Updated existing tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3761
2023-01-19 10:21:23 +01:00
Dmitry S
45c7602f49 (core) Implement checkbox for SchemaEdit permission in Access Rules UI.
Summary:
- Introduces a fictitious "*SPECIAL:SchemaEdit" resource in UI only.
- Hides "S" bit for the default rule section.
- Shows a checkbox UI similar to other checkboxes, with an additional
  dismissable warning.

Test Plan: Added a browser test

Reviewers: paulfitz, georgegevoian

Reviewed By: paulfitz, georgegevoian

Differential Revision: https://phab.getgrist.com/D3765
2023-01-18 10:55:44 -05:00
Cyprien P
18d016c745 (core) Fix date filter for DateTime columns.
Summary:
Date filter was not taking timezone correclty into account, which was
causing to wrong-inclusion and wrong-exclusion of dates near the
bounds.

Diff fixes that, it also bring little refactoring that hopefully clarifies things a little.

Test Plan: Includes brand new test for `app/common/ColumnFilterFunc`.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3763
2023-01-18 10:18:15 +01:00
George Gevoian
db64dfeef0 (core) Add tip for "Add New" button
Summary:
Adds a new tip for the doc menu's Add New button. The tip is
shown only when the current user is an editor or owner, and
the site is non-empty. The presence of welcome videos or
popups will also cause the tip to not be shown; it will instead
be shown the next time the doc menu is visited.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3757
2023-01-16 16:50:42 -08:00
Cyprien P
b7f65ff408 (core) Adds dots menu to access rules page item
Summary: In Access rules page item, adds “…” buttons that shows a menu of users to view-as:

Test Plan: Include new nbrowser test

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3751
2023-01-12 09:40:45 +01:00
George Gevoian
9e009bbab9 (core) Update memo notification styling
Summary: Updates CSS for memo notifications to match latest Figma designs.

Test Plan: N/A

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3737
2023-01-09 14:34:59 -05:00
Paul Fitzpatrick
e6692c2793 (core) add a checkbox for owner "boss mode"
Summary:
Implement a checkbox that grants owners full access to tables by default, when creating new table/column rules.
 * Checkbox appears above default rules.
 * When set, a rule giving owners full access will be inserted in any new rule set started for tables or columns.
 * The checkbox can be expanded to allow customization of the rules.

https://gristlabs.getgrist.com/doc/check-ins/p/3#a1.s7.r2251.c19

Test Plan: added tests

Reviewers: jarek

Reviewed By: jarek

Subscribers: anaisconce

Differential Revision: https://phab.getgrist.com/D3756
2023-01-09 13:20:23 -05:00
Jarosław Sadziński
b59829d57e (core) Updating dependencies for moment package
Summary:
moment.js version in yarn.lock file was outdated and contained
a set of typings that are diffrent from the one visible from grist-core.

Test Plan: Existing tests

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3760
2023-01-09 18:20:01 +01:00
Jarosław Sadziński
30600c5dc2 (core) Making makeT helper use the unprefixed key as a fallback
Summary:
By default the fallback contains the prefix, which doesn't
work well with a key based translations. Now makeT helper will fallback
to the passed key (without a prefix).

Test Plan: Added new client test

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3758
2023-01-05 16:25:19 +01:00
Jarosław Sadziński
fd02a00a0e Fixing all eslint's reported error 2023-01-03 17:22:58 +01:00
George Gevoian
77e85c132c
Fix control-border CSS regression (#388) 2023-01-03 10:16:23 -05:00
Cyprien P
cabac3d9d8 (core) Adds new view as banner
Summary:
Diff removes view-as pill in the document breadcrumbs and add new view-as banner.

Note: Banners are still missing mechanism to handle several banners. As of now both doc-usage and view-as banners could show up at the same time.

Test Plan: Refactored existing test.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3732
2023-01-03 12:33:34 +01:00
Paul Fitzpatrick
472a9a186e (core) control the distribution of attachment metadata
Summary:
for users who don't automatically have deep rights
to the document, provide them with attachment metadata only
for rows they have access to. This is a little tricky to
do efficiently. We provide attachment metadata when an
individual table is fetched, rather than on initial document
load, so we don't block that load on a full document scan.
We provide attachment metadata to a client when we see that
we are shipping rows mentioning particular attachments,
without making any effort to keep track of the metadata they
already have.

Test Plan: updated tests

Reviewers: dsagal, jarek

Reviewed By: dsagal, jarek

Differential Revision: https://phab.getgrist.com/D3722
2022-12-22 09:10:30 -05:00
Jarosław Sadziński
2a86cde474 (core) Hiding helper columns used for column transformation
Summary:
When a column is transformed, it creates two helper columns whose values are always
broadcasted to all clients. Now when there are some ACL rules, we are going to prune
those columns from messages sent to other connected clients.

Test Plan: Added new tests

Reviewers: dsagal, paulfitz

Reviewed By: dsagal, paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3728
2022-12-21 10:07:17 +01:00
George Gevoian
e52e15591d (core) Add behavioral and coaching call popups
Summary:
Adds a new category of popups that are shown dynamically when
certain parts of the UI are first rendered, and a free coaching
call popup that's shown to users on their site home page.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3706
2022-12-20 09:49:41 -05:00
Cyprien P
620e86a9f1 (core) New date filter with a calendar view
Summary:
Implements the new date filtering panel. Design results from long
discussion between: Alex, Anais, Cyprien and Dmitry.

Test environment: https://grist-new-date-range-filter.fly.dev/

Test Plan: Include various new tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3720
2022-12-20 09:58:42 +01:00
George Gevoian
c558800de5 (core) Avoid flagging support user as collaborator
Summary:
When initially added in the User Manager, the support user
(e.g. support@getgrist.com) was misleadingly being annotated as
a free collaborator. This fixes the annotation to be "Grist support"
instead.

Test Plan: Browser test.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3730
2022-12-14 01:23:49 -05:00
George Gevoian
e146f95c1c (core) Add new UI for writing memos
Summary:
Adds a new UI for writing access rule memos.

Migrates old memos (written as Python comments) to the new UI.

Test Plan: Browser and migration tests.

Reviewers: jarek, dsagal

Reviewed By: jarek

Subscribers: dsagal, paulfitz

Differential Revision: https://phab.getgrist.com/D3726
2022-12-12 17:52:01 -05:00
Paul Fitzpatrick
ebaf04dace (core) add buttons to delete bad rules
Summary:
When access rules refer to tables and/or columns that no longer exist, offer convenient buttons to remove these rules.

It could alternatively be useful to generate errors when deleting tables or columns that are mentioned in access rules, and refuse to do so unless the access rules are updated first.

Test Plan: added and updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3718
2022-12-05 11:49:41 -05:00
Louis Delbosc
4116949ea5
Add translation for boolean value in toggle columns for text cell (#364) 2022-12-02 15:49:55 -05:00
Louis Delbosc
a4b1145605
Add GRIST_HELP_CENTER environment variable (#363) 2022-11-28 15:19:31 -05:00
Paul Fitzpatrick
312d2331a8 (core) lock down modification of the _grist_Attachments table
Summary:
Rows in the _grist_Attachments table have a special lifecycle,
being created by a special method, and deleted via a special
process. All other modifications are now rejected, for simplicity.

Test Plan: added test

Reviewers: dsagal, jarek

Reviewed By: dsagal, jarek

Differential Revision: https://phab.getgrist.com/D3712
2022-11-22 11:30:54 -05:00
George Gevoian
1a6d427339 (core) Update sort and filter UI
Summary:
The sort and filter UI now has a more unified UI, with similar
capabilities that are accessible from different parts of Grist.
It's now also possible to pin individual filters to the filter bar,
which replaces the old toggle for showing all filters in the
filter bar.

Test Plan: Various tests (browser, migration, project).

Reviewers: jarek, dsagal

Reviewed By: jarek, dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3669
2022-11-17 15:33:45 -05:00
Paul Fitzpatrick
ea71312d0e (core) deal with write access for attachments
Summary:
Attachments are a special case for granular access control. A user is now allowed to read a given attachment if they have read access to a cell containing its id. So when a user writes to a cell in an attachment column, it is important that they can only write the ids of cells to which they have access. This diff allows a user to add an attachment id in a cell if:

  * The user already has access to that a attachment via some existing cell, or
  * The user recently updated the attachment, or
  * The attachment change is from an undo/redo of a previous action attributed to that user

Test Plan: Updated tests

Reviewers: georgegevoian, dsagal

Reviewed By: georgegevoian, dsagal

Differential Revision: https://phab.getgrist.com/D3681
2022-11-15 09:52:32 -05:00
Jarosław Sadziński
7c9cb9843e (core) Revealing hidden pages with visible children.
Summary:
When a page is hidden, all its nested pages are shown as children of
a different page that happens to be before (as in pagePos) that page.

This diff shows those pages as CENSORED.

Test Plan: Updated

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3670
2022-10-31 14:02:38 +01:00
George Gevoian
3145af36c6 (core) Add dark mode for comments
Summary:
Also fixes the CSS for the page/workspace input so that it's always
readable in dark mode.

Test Plan: Tested manually.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3682
2022-10-25 08:19:06 -07:00
Jarosław Sadziński
6460c22a89 (core) Changing shortcuts for adding and removing rows
Summary:
New shortcuts for removing and adding rows.
For adding a row we now have Mod+(Shift)+Enter
For removing rows we now have Mod+Delete/Mod+Backspace

Before removing rows, the user is prompted to confirm, this prompt
can be dismissed and this setting can be remembered. User needs
to confirm only when using shortcut.

Old shortcuts are still active and shows information about this change.
This information is shown only once, after this shortcuts have default
behavior (zooming).
New users don't see this explanation.

Test Plan: Updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3655
2022-10-21 18:45:25 +02:00
George Gevoian
4c662253a9 (core) Add info and hover tooltips
Summary:
Adds tooltip buttons to various parts of the UI that either open a popup with
information when clicked, or show a label on hover.

Test Plan: Project tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3657
2022-10-20 15:17:44 -07:00
Jarosław Sadziński
bfd7243fe2 (core) Comments
Summary:
First iteration for comments system for Grist.
- Comments are stored in a generic metatable `_grist_Cells`
- Each comment is connected to a particular cell (hence the generic name of the table)
- Access level works naturally for records stored in this table
-- User can add/read comments for cells he can see
-- User can't update/remove comments that he doesn't own, but he can delete them by removing cells (rows/columns)
-- Anonymous users can't see comments at all.
- Each comment can have replies (but replies can't have more replies)

Comments are hidden by default, they can be enabled by COMMENTS=true env variable.
Some things for follow-up
- Avatars, currently the user's profile image is not shown or retrieved from the server
- Virtual rendering for comments list in creator panel. Currently, there is a limit of 200 comments.

Test Plan: New and existing tests

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3509
2022-10-17 13:38:21 +02:00
Jarosław Sadziński
8be920dd25 (core) Multi-column configuration
Summary:
Creator panel allows now to edit multiple columns at once
for some options that are common for them. Options that
are not common are disabled.

List of options that can be edited for multiple columns:
- Column behavior (but limited to empty/formula columns)
- Alignment and wrapping
- Default style
- Number options (for numeric columns)
- Column types (but only for empty/formula columns)

If multiple columns of the same type are selected, most of
the options are available to change, except formula, trigger formula
and conditional styles.

Editing column label or column id is disabled by default for multiple
selection.

Not related: some tests were fixed due to the change in the column label
and id widget in grist-core (disabled attribute was replaced by readonly).

Test Plan: Updated and new tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3598
2022-10-17 09:51:19 +02:00
Paul Fitzpatrick
ab3cdb62ac (core) make the support account configurable, and allow listing public sites
Summary:
This makes two small tweaks based on a user's questions about sharing
sites publicly for a self-managed installation:

 * The support user `support@getgrist.com` is made configurable with
   `GRIST_SUPPORT_EMAIL`. This came up because only the support user
   can share material with the special "everyone" user. This restriction
   was added to avoid spam.
 * Regardless of public sharing settings, for our SaaS we had
   decided not to list public sites to anonymous users. That is
   somewhat a question of taste, so a `GRIST_LIST_PUBLIC_SITES` flag
   is added to override this choice.

Public sharing isn't in a well polished state, and this diff doesn't
advance that, in fact it adds a new wrinkle :-/

Test Plan: existing tests pass; manual testing

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3663
2022-10-14 14:23:59 -04:00
Paul Fitzpatrick
6dd2068218 (core) show package.json version when hovering on Grist icon in grist-core
Summary:
This makes the version shown when hovering on the Grist icon equal
the version set in package.json, for a grist-core build. Previously
the number shown was a hard-coded placeholder.

The Grist SaaS build has some build machinery dealing with the
version number that should be unaffected by this change for now.

Test Plan: tested manually with build_core.sh

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3659
2022-10-12 16:02:01 -04:00
George Gevoian
74c556ea85 (core) Fix CSS errors related to dark mode
Summary: Fixes a few cosmetic CSS bugs since dark mode landed.

Test Plan: Tested manually.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3649
2022-10-05 00:05:57 -07:00
Jarosław Sadziński
9628253fd8 (core) Adding new column in users table "ref" with unique identifier.
Summary:
There is a new column in users table called ref (user reference).
It holds user's unique reference number that can be used for features
that require some kind of ownership logic (like comments).

Test Plan: Updated tests

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian, paulfitz

Differential Revision: https://phab.getgrist.com/D3641
2022-10-04 15:19:28 +02:00
George Gevoian
364610c69d (core) Add cell selection summary
Summary:
Adds a cell selection summary to grid view that shows either a count or
sum of all the selected values. Implementation was done by Dmitry.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: paulfitz, dsagal, jarek

Differential Revision: https://phab.getgrist.com/D3630
2022-09-30 09:11:46 -07:00
Jarosław Sadziński
5219932a1f (core) i18
Summary:
Adding initial work for localization support.

Summary in https://grist.quip.com/OtZKA6RHdQ6T/Internationalization-and-Localization

Test Plan: Not yet

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3633
2022-09-29 18:02:09 +02:00
George Gevoian
cd64237dad (core) Allow duplicating tables from Raw Data page
Summary:
Adds a "Duplicate Table" menu option to the tables listed on
the Raw Data page. Clicking it opens a dialog that allows you to
make a copy of the table (with or without its data).

Test Plan: Python, server, and browser tests.

Reviewers: jarek, paulfitz

Reviewed By: jarek, paulfitz

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3619
2022-09-29 08:59:42 -07:00
Paul Fitzpatrick
0eb1fec3d7 (core) show any error when creating initial team site with GRIST_SINGLE_ORG
Summary: Also clarifies that only lowercase letters are accepted.

Test Plan: manual

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3647
2022-09-29 10:12:32 -04:00
Alex Hall
792565976a (core) Show example values in formula autocomplete
Summary:
This diff adds a preview of the value of certain autocomplete suggestions, especially of the form `$foo.bar` or `user.email`. The main initial motivation was to show the difference between `$Ref` and `$Ref.DisplayCol`, but the feature is more general.

The client now sends the row ID of the row being edited (along with the table and column IDs which were already sent) to the server to fetch autocomplete suggestions. The returned suggestions are now tuples `(suggestion, example_value)` where `example_value` is a string or null. The example value is simply obtained by evaluating (in a controlled way) the suggestion in the context of the given record and the current user. The string representation is similar to the standard `repr` but dates and datetimes are formatted, and the whole thing is truncated for efficiency.

The example values are shown in the autocomplete popup separated from the actual suggestion by a number of spaces calculated to:

1. Clearly separate the suggestion from the values
2. Left-align the example values in most cases
3. Avoid having so much space such that connecting suggestions and values becomes visually difficult.

The tokenization of the row is then tweaked to show the example in light grey to deemphasise it.

Main discussion where the above was decided: https://grist.slack.com/archives/CDHABLZJT/p1661795588100009

The diff also includes various other small improvements and fixes:

- The autocomplete popup is much wider to make room for long suggestions, particularly lookups, as pointed out in https://phab.getgrist.com/D3580#inline-41007. The wide popup is the reason a fancy solution was needed to position the example values. I didn't see a way to dynamically resize the popup based on suggestions, and it didn't seem like a good idea to try.
- The `grist` and `python` labels previously shown on the right are removed. They were not helpful (https://grist.slack.com/archives/CDHABLZJT/p1659697086155179) and would get in the way of the example values.
- Fixed a bug in our custom tokenization that caused function arguments to be weirdly truncated in the middle: https://grist.slack.com/archives/CDHABLZJT/p1661956353699169?thread_ts=1661953258.342739&cid=CDHABLZJT and https://grist.slack.com/archives/C069RUP71/p1659696778991339
- Hide suggestions involving helper columns like `$gristHelper_Display` or `Table.lookupRecords(gristHelper_Display=` (https://grist.slack.com/archives/CDHABLZJT/p1661953258342739). The former has been around for a while and seems to be a mistake. The fix is simply to use `is_visible_column` instead of `is_user_column`. Since the latter is not used anywhere else, and using it in the first place seems like a mistake more than anything else, I've also removed the function to prevent similar mistakes in the future.
- Don't suggest private columns as lookup arguments: https://grist.slack.com/archives/CDHABLZJT/p1662133416652499?thread_ts=1661795588.100009&cid=CDHABLZJT
- Only fetch fresh suggestions specifically after typing `lookupRecords(` or `lookupOne(` rather than just `(`, as this would needlessly hide function suggestions which could still be useful to see the arguments. However this only makes a difference when there are still multiple matching suggestions, otherwise Ace hides them anyway.

Test Plan: Extended and updated several Python and browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3611
2022-09-28 19:42:36 +02:00
Alex Hall
1864b7ba5d (core) Add BulkAddOrUpdateRecord action for efficiency
Summary:
This diff adds a new `BulkAddOrUpdateRecord` user action which is what is sounds like:

- A bulk version of the existing `AddOrUpdateRecord` action.
- Much more efficient for operating on many records than applying many individual actions.
- Column values are specified as maps from `colId` to arrays of values as usual.
- Produces bulk versions of `AddRecord` and `UpdateRecord` actions instead of many individual actions.

Examples of users wanting to use something like `AddOrUpdateRecord` with large numbers of records:

- https://grist.slack.com/archives/C0234CPPXPA/p1651789710290879
- https://grist.slack.com/archives/C0234CPPXPA/p1660743493480119
- https://grist.slack.com/archives/C0234CPPXPA/p1660333148491559
- https://grist.slack.com/archives/C0234CPPXPA/p1663069291726159

I tested what made many `AddOrUpdateRecord` actions slow in the first place. It was almost entirely due to producing many individual `AddRecord` user actions. About half of that time was for processing the resulting `AddRecord` doc actions. Lookups and updates were not a problem. With these changes, the slowness is gone.

The Python user action implementation is more complex but there are no surprises. The JS API now groups `records` based on the keys of `require` and `fields` so that `BulkAddOrUpdateRecord` can be applied to each group.

Test Plan: Update and extend Python and DocApi tests.

Reviewers: jarek, paulfitz

Reviewed By: jarek, paulfitz

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3642
2022-09-28 17:58:33 +02:00
Alex Hall
d140b49ba3 (core) Include helper columns in ACL rules
Summary: Extend the way ACL resources are read in the server so that if a rule applies to a specific column then that rule also applies to helper columns belonging to that column, as well as helper columns belonging to fields which display that column. This is particularly intended for display columns of reference columns, but it also applies to conditional formatting rule columns.

Test Plan: Added a server test

Reviewers: paulfitz, jarek

Reviewed By: paulfitz, jarek

Differential Revision: https://phab.getgrist.com/D3628
2022-09-26 16:08:56 +02:00
Paul Fitzpatrick
d55b5110ac (core) remove deprecated /download endpoint in favor of newer /api/docs/NNNN/download
Summary:
This endpoint has started to fail when called between a pair
of doc workers. The simplest fix is to simply remove it, it serves no
purpose.

Test Plan: added basic deployment test

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3636
2022-09-20 15:26:04 -04:00
Louis Delbosc
494a683332
Export xlsx #256 (#270)
XLSX export of active view / table

Co-authored-by: Louis Delbosc <louis.delbosc.prestataire@anct.gouv.fr>
Co-authored-by: Vincent Viers <vincent.viers@beta.gouv.fr>
2022-09-14 14:55:44 -04:00
George Gevoian
965cf7e076 Fix incorrect CSS fallback values 2022-09-08 14:30:46 -07:00
George Gevoian
ec157dc469 (core) Add dark mode to user preferences
Summary:
Adds initial implementation of dark mode. Preferences for dark mode are
available on the account settings page. Dark mode is currently a beta feature
as there are still some small bugs to squash and a few remaining UI elements
to style.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: paulfitz, jarek

Differential Revision: https://phab.getgrist.com/D3587
2022-09-05 19:17:32 -07:00
Dmitry S
1c24bfc8a6 (core) Fix exports to CSV/XLSX/etc when data is restricted by access rules
Summary:
- The issue manifested as error "Cannot read property '0' of undefined" in some
  cases, and as "Blocked by table read access rules" in others (instead of
  limiting output to what's not blocked)
- Goes deeper: exports weren't respecting metadata censoring.
- The fix changes exports to use censored metadata, which addresses both errors above.
- Includes an improvement to column ordering in XLSX exports.

Test Plan: Add a server test for CSV and XLSX exports with access rules

Reviewers: paulfitz, georgegevoian

Reviewed By: paulfitz, georgegevoian

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3615
2022-09-02 10:59:59 -04:00
George Gevoian
2cb783ea7b (core) Fix bugs with intervals
Summary:
Fixes some bugs involving intervals, and updates RandomizedTimer to support both fixed and
randomized delays, and to better handle async callbacks.

 * Fixed a bug where Throttle would queue up many pidusage calls due to the use of
    setInterval, and the async nature of the calls.

 * Fixed a but where RandomizedTimer (now just Interval) would not be disabled in
    ActiveDoc on doc shutdown if initialization had not yet settled.

Test Plan: Tested manually.

Reviewers: jarek, dsagal

Reviewed By: jarek, dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3604
2022-08-25 12:38:36 -07:00
Jarosław Sadziński
2997434815 (core) Showing a raw data section on a popup
Summary:
Show raw data will now open a popup with
raw section instead of redirecting to raw data page.

Adding new anchor link type "a2" that is able to open
any section in a popup on a current view.

Not related:
Fixing highlightMatches function, after merging core PR.

Test Plan: Updated tests

Reviewers: alexmojaki, georgegevoian

Reviewed By: alexmojaki, georgegevoian

Subscribers: georgegevoian, alexmojaki

Differential Revision: https://phab.getgrist.com/D3592
2022-08-25 09:30:07 +02:00
George Gevoian
56e8e1f4b3 (core) Randomize ActiveDoc interval delays
Summary:
When large numbers of documents were restarted simultaneously, they had
a tendency to schedule intervals to begin at roughly the same moment in
time, causing periodic spikes in load. This randomizes the delay of each
interval to help avoid such spikes.

Test Plan: Tested manually.

Reviewers: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3600
2022-08-23 23:08:07 -07:00
George Gevoian
ed37401b2c (core) Add basic activation page to grist-ee
Summary:
Adds an activation page to grist-ee that currently shows activation status.

Follow-up diffs will introduce additional functionality, such as the ability to
enter activation keys directly from the activation page.

Test Plan: No grist-ee tests (yet).

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal, paulfitz

Differential Revision: https://phab.getgrist.com/D3582
2022-08-23 10:30:52 -07:00
Jarosław Sadziński
a52d56f613 (core) Moving client and common tests to core
Summary:
- Moved /test/client and /test/common to core.
- Moved two files (CircularArray and RecentItems) from app/common to core/app/common.
- Moved resetOrg test to gen-server.
- `testrun.sh` is now invoking common and client test from core.
- Added missing packages to core's package.json (and revealed underscore as it is used in the main app).
- Removed Coord.js as it is not used anywhere.

Test Plan: Existing tests

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3590
2022-08-23 19:20:10 +02:00
Jarosław Sadziński
9e4d802405 (core) Implementing row conditional formatting
Summary:
Conditional formatting can now be used for whole rows.
Related fix:
- Font styles weren't applicable for summary columns.
- Checkbox and slider weren't using colors properly

Test Plan: Existing and new tests

Reviewers: paulfitz, georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3547
2022-08-09 20:11:36 +02:00
Jarosław Sadziński
ee109e9186 (core) Removing NEW_DEAL flag
Summary:
Removing NEW_DEAL flag checks and exposing all new deal features as default.
Also removing Pages.ts as it was moved to grist-core.

Test Plan: Existing and updated tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3572
2022-08-09 17:29:28 +02:00
George Gevoian
771e1edd54 (core) Keep track of row counts per table
Summary: Displays a live row count of each table on the Raw Data page.

Test Plan: Browser tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3540
2022-08-03 08:13:33 -07:00
Jarosław Sadziński
40c9b8b7e8 (core) New URL that opens Create site popup.
Summary: Adding new url parameter for team site creation

Test Plan: Updated tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3554
2022-08-03 13:09:18 +02:00
George Gevoian
aeba738f7c (core) Add product for new personal plan
Summary:
Adds the new personal plan as a product that will be available
in the future. Can be enabled along with other plan-related via
an environment variable.

Test Plan: Browser tests and existing tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3533
2022-07-26 11:33:23 -07:00
Jarosław Sadziński
5c8211c61d (core) Fixing selectby error on the creator panel.
Summary: [Select By] in the creator panel was bugged. It wasn't refreshed in some cases as the observable array that needed to be created seemed too complicated. This Diff recomputes this array when the user wants to change the selection.

Test Plan: added tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3541
2022-07-26 15:46:11 +02:00
Alex Hall
460f22b701 (core) Don't show hidden tables in table data options when adding widgets
Summary:
As reported in https://grist.slack.com/archives/C069RUP71/p1655316194602829, when a table is hidden by ACL, it was still showing as a blank 'ghost' option to select data when adding a widget:

{F55498}

The fix is simply to return `true` from `isHiddenTable` for empty table IDs, which indicate a table hidden by ACL. `TableRec.isHidden` is supposed to match this so I updated it too, and I cleaned up a tiny bit of other related code.

Test Plan: Extended `nbrowser/AccessRules1.ts` to test the data options when adding widgets.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3530
2022-07-21 16:28:28 +02:00
Paul Fitzpatrick
dd8d2e18f5 (core) add an access token mechanism to help with attachments in custom widgets
Summary:
With this, a custom widget can render an attachment by doing:
```
const tokenInfo = await grist.docApi.getAccessToken({readOnly: true});
const img = document.getElementById('the_image');
const id = record.C[0];  // get an id of an attachment
const src = `${tokenInfo.baseUrl}/attachments/${id}/download?auth=${tokenInfo.token}`;
img.setAttribute('src', src)
```

The access token expires after a few mins, so if a user right-clicks on an image
to save it, they may get access denied unless they refresh the page. A little awkward,
but s3 pre-authorized links behave similarly and it generally isn't a deal-breaker.

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3488
2022-07-19 11:55:18 -04:00
Alex Hall
f39b496563 (core) Use table title instead of ID in ACL UI
Summary:
Use table titles (i.e. the raw data widget titles) in dropdowns and other parts of the Acess Rules page, instead of the table ID. This is particularly meant for summary tables which have/had an ID of the form `GristSummary_SourceTable_N`, but https://phab.getgrist.com/D3508 is changing that anyway.

The server method `getAclResources` now returns more metadata about each table so that the UI can display titles.

Test Plan: Extended and updated `nbrowser/AccessRules2.ts`. Added a small unit test for constructing table titles from the new description returned by `getAclResources`.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3494
2022-07-19 16:27:17 +02:00
Jarosław Sadziński
257dafe423 (core) Improving billing page user experience
Summary:
Improving billing page user experience.
- Updated labels for canceled plan
- Adding option to downgrade from team plan to free team plan
- Updating default name for teamFree plan when it is not available in Stripe
- Minor bug fixes

Test Plan: updated tests

Reviewers: cyprien

Reviewed By: cyprien

Subscribers: cyprien

Differential Revision: https://phab.getgrist.com/D3515
2022-07-18 14:43:14 +02:00
Alex Hall
b8486dcdba (core) Nice summary table IDs
Summary:
Changes auto-generated summary table IDs from e.g. `GristSummary_6_Table1` to `Table1_summary_A_B` (meaning `Table1` grouped by `A` and `B`). This makes it easier to write formulas involving summary tables, make API requests, understand logs, etc.

Because these don't encode the source table ID as reliably as before, `decode_summary_table_name` now uses the summary table schema info, not just the summary table ID. Specifically, it looks at the type of the `group` column, which is `RefList:<source table id>`.

Renaming a source table renames the summary table as before, and now renaming a groupby column renames the summary table as well.

Conflicting table names are resolved in the usual way by adding a number at the end, e.g. `Table1_summary_A_B2`. These summary tables are not automatically renamed when the disambiguation is no longer needed.

A new migration renames all summary tables to the new scheme, and updates formulas using summary tables with a simple regex.

Test Plan:
Updated many tests to use the new style of name.

Added new Python tests to for resolving conflicts when renaming source tables and groupby columns.

Added a test for the migration, including renames in formulas.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3508
2022-07-14 12:09:56 +02:00
Paul Fitzpatrick
f91f45b26d (core) support granular read access for attachments
Summary:
When a user requests to read the contents of an attachment, only allow the request if there exists a cell in an attachment column that contains the attachment and which they have read access to.

This does not cover:
 * Granular write access for attachments. In particular, a user who can write to any attachment column should be considered to have full read access to all attachment columns, currently.
 * Access control of attachment metadata such as name and format.

The implementation uses a sql query that requires a scan, and some notes on how this could be optimized in future. The web client was updated to specify the cell to check for access, and performance seemed fine in casual testing on a doc with 1000s of attachments. I'm not sure how performance would hold up as the set of access rules grows as well.

Test Plan: added tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3490
2022-07-07 07:22:02 -04:00
George Gevoian
a051830aeb (core) Show summary tables on Raw Data page
Summary:
Summary tables now have their own raw viewsection, and are shown
under Raw Data Tables on the Raw Data page.

Test Plan: Browser and Python tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3495
2022-07-06 09:41:48 -07:00
Dmitry S
51ff72c15e (core) Faster builds all around.
Summary:
Building:
- Builds no longer wait for tsc for either client, server, or test targets. All use esbuild which is very fast.
- Build still runs tsc, but only to report errors. This may be turned off with `SKIP_TSC=1` env var.
- Grist-core continues to build using tsc.
- Esbuild requires ES6 module semantics. Typescript's esModuleInterop is turned
  on, so that tsc accepts and enforces correct usage.
- Client-side code is watched and bundled by webpack as before (using esbuild-loader)

Code changes:
- Imports must now follow ES6 semantics: `import * as X from ...` produces a
  module object; to import functions or class instances, use `import X from ...`.
- Everything is now built with isolatedModules flag. Some exports were updated for it.

Packages:
- Upgraded browserify dependency, and related packages (used for the distribution-building step).
- Building the distribution now uses esbuild's minification. babel-minify is no longer used.

Test Plan: Should have no behavior changes, existing tests should pass, and docker image should build too.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3506
2022-07-04 10:42:40 -04:00
Jarosław Sadziński
aefe451bab (core) Polishing upgrade plan UI
Summary:
- Update nudge boxes content and collapsing on personal and free team site
- New confirmation after upgrading from a free team site
- Refactoring ProductUpgrade code, splitting plans / modals and nudges

Test Plan: Manual and updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3481
2022-06-29 18:28:37 +02:00
Dmitry S
dd2eadc86e (core) Speed up and upgrade build.
Summary:
- Upgrades to build-related packages:
  - Upgrade typescript, related libraries and typings.
  - Upgrade webpack, eslint; add tsc-watch, node-dev, eslint_d.

- Build organization changes:
  - Build webpack from original typescript, transpiling only; with errors still
    reported by a background tsc watching process.

- Typescript-related changes:
  - Reduce imports of AWS dependencies (very noticeable speedup)
  - Avoid auto-loading global @types
  - Client code is now built with isolatedModules flag (for safe transpilation)
  - Use allowJs to avoid copying JS files manually.

- Linting changes
  - Enhance Arcanist ESLintLinter to run before/after commands, and set up to use eslint_d
  - Update eslint config, and include .eslintignore to avoid linting generated files.
  - Include a bunch of eslint-prompted and eslint-generated fixes
  - Add no-unused-expression rule to eslint, and fix a few warnings about it

- Other items:
  - Refactor cssInput to avoid circular dependency
  - Remove a bit of unused code, libraries, dependencies

Test Plan: No behavior changes, all existing tests pass. There are 30 tests fewer reported because `test_gpath.py` was removed (it's been unused for years)

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3498
2022-06-27 16:10:10 -04:00
Cyprien P
64ff9ccd0a (core) Allows range filter for Date, DateTime columns
Summary:
This diff is first of a series of 3 commits to enable range filering
for Date and DateTime columns. Diff only enable setting date's min/max
throw typing dates, Date picker and relative ranges are left for
follow-up commits.

 - Exposes columns value formatter to the range input
 - Fixes column filter func to work with dates

Test Plan:
Adds Date to projects range filter test
Adds Date/DateTime to nbrowser ColumnFilterMenu tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3455
2022-06-23 13:02:37 +02:00
Alex Hall
9fffb491f9 (core) External requests
Summary:
Adds a Python function `REQUEST` which makes an HTTP GET request. Behind the scenes it:

- Raises a special exception to stop trying to evaluate the current cell and just keep the existing value.
- Notes the request arguments which will be returned by `apply_user_actions`.
- Makes the actual request in NodeJS, which sends back the raw response data in a new action `RespondToRequests` which reevaluates the cell(s) that made the request.
- Wraps the response data in a class which mimics the `Response` class of the `requests` library.

In certain cases, this asynchronous flow doesn't work and the sandbox will instead synchronously call an exported JS method:

- When reevaluating a single cell to get a formula error, the request is made synchronously.
- When a formula makes multiple requests, the earlier responses are retrieved synchronously from files which store responses as long as needed to complete evaluating formulas. See https://grist.slack.com/archives/CL1LQ8AT0/p1653399747810139

Test Plan: Added Python and nbrowser tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: paulfitz, dsagal

Differential Revision: https://phab.getgrist.com/D3429
2022-06-17 21:53:20 +02:00
George Gevoian
abebe812db (core) Show Grist video tour after welcome questions
Summary:
After the welcome questions are dismissed, a video tour modal will
now be displayed. The video tour is also accessible via a tool button
in the left panel of the home page, as well as a text button next to
the Examples & Templates header.

Test Plan: Browser tests.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3477
2022-06-16 21:54:17 -07:00
Dmitry S
a91d493ffc (core) Fix issue with 'UNEXPECTED ORDER OF CALLBACKS' in Client.ts.
Summary:
- Substantial refactoring of the logic when the server fails to send some
  messages to a client.
- Add seqId numbers to server messages to ensure reliable order.
- Add a needReload flag in clientConnect for a clear indication whent the
  browser client needs to reload the app.
- Reproduce some potential failure scenarios in a test case (some of which
  previously could have led to incorrectly ordered messages).
- Convert other Comm tests to typescript.
- Tweak logging of Comm and Client to be slightly more concise (in particular,
  avoid logging sessionId)

Note that despite the big refactoring, this only addresses a fairly rare
situation, with websocket failures while server is trying to send to the
client. It includes no improvements for failures while the client is sending to
the server.

(I looked for an existing library that would take care of these issues. A relevant article I found is https://docs.microsoft.com/en-us/azure/azure-web-pubsub/howto-develop-reliable-clients, but it doesn't include a library for both ends, and is still in review. Other libraries with similar purposes did not inspire enough confidence.)

Test Plan: New test cases, which reproduce some previously problematic scenarios.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3470
2022-06-16 23:51:14 -04:00
Paul Fitzpatrick
561d9696aa (core) clean up interaction of forward auth with session
Summary:
For self-hosted Grist, forward auth has proven useful, where
some proxy wrapped around Grist manages authentication, and
passes on user information to Grist in a trusted header.
The current implementation is adequate when Grist is the
only place where the user logs in or out, but is confusing
otherwise (see https://github.com/gristlabs/grist-core/issues/207).
Here we take some steps to broaden the scenarios Grist's
forward auth support can be used with:

  * When a trusted header is present and is blank, treat
    that as the user not being logged in, and don't look
    any further for identity information. Specifically,
    don't look in Grist's session information.
  * Add a `GRIST_IGNORE_SESSION` flag to entirely prevent
    Grist from picking up identity information from a cookie,
    in order to avoid confusion between multiple login methods.
  * Add tests for common scenarios.

Test Plan: added tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3482
2022-06-15 13:06:12 -04:00
Alex Hall
0350e2df58 (core) Fix filtering of empty reflists
Summary:
A formula returning an empty RecordSet in a RefList columns results in storing [] instead of null.
This caused a bug where the empty list was 'flattened' and the cell not appearing in filters at all.
This diff fixes the bug by filtering for the default value `null` instead for RefLists and the empty string for ChoiceLists.
I didn't manage to actually reproduce the bug for ChoiceLists, but this seemed the most sensible thing to do.

Test Plan: New nbrowser test.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3478
2022-06-11 00:16:57 +02:00
Alex Hall
1c89d08ea3 (core) Add a row to summary tables grouped by list column(s) corresponding to empty lists
Summary:
Adds some special handling to summary table and lookup logic:

- Source rows with empty choicelists/reflists get a corresponding summary row with an empty string/reference when grouping by that column, instead of excluding them from any group
- Adds a new `QueryOperation` 'empty' in the client which is used in `LinkingState`, `QuerySet`, and `recursiveMoveToCursorPos` to match empty lists in source tables against falsy values in linked summary tables.
- Adds a new parameter `match_empty` to the Python `CONTAINS` function so that regular formulas can implement the same behaviour as summary tables. See https://grist.slack.com/archives/C0234CPPXPA/p1654030490932119
- Uses the new `match_empty` argument in the formula generated for the `group` column when detaching a summary table.

Test Plan: Updated and extended Python and nbrowser tests of summary tables grouped by choicelists to test for new behaviour with empty lists.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3471
2022-06-09 23:38:14 +02:00
Jarosław Sadziński
d92a761f6e (core) Product update popups and hosted stripe integration
Summary:
- Showing nudge to individual users to sign up for free team plan.
- Implementing billing page to upgrade from free team to pro.
- New modal with upgrade options and free team site signup.
- Integrating Stripe-hosted UI for checkout and plan management.

Test Plan: updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3456
2022-06-08 21:10:49 +02:00
Dmitry S
4f1cb53b29 (core) Converting server-side Comm.js to typescript
Summary:
- Add app/common/CommTypes.ts to define types shared by client and server.
- Include @types/ws npm package

Test Plan: Intended to have no changes in behavior

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3467
2022-06-07 15:47:17 -04:00
Paul Fitzpatrick
05d1cdf140 (core) limit retries of uploads to external store in tests
Summary:
If an external store fails completely, Grist will continue to
retry uploading to it. This diff updated the HostedStorageManager
test to limit the extent of these retries to the test itself -
otherwise they continue for all other tests in the same process,
potentially disrupting those that read logs. There are other tests
that use s3, but they aren't run in the same process with delicate
log-reading tests, and it isn't quite as clear what improvement
to make there.

Test Plan:
artificially made external store fail, and checked that
test contamination seen previously no longer occurs.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3469
2022-06-06 16:19:41 -04:00
George Gevoian
6dcdd22792 (core) Redirect less often in welcomeNewUser
Summary:
Instead of always redirecting new users to the home page or the (teams) welcome page,
only redirect when the user signed in for the first time on a personal site, has access to
other sites, and isn't already being redirected to a specific page on their personal site.

Also tweaks how invalid Choice column values are displayed to match Choice List
columns, and fixes a small CSS issue with select by in the page widget picker when
there are options with long labels.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3461
2022-06-06 11:26:49 -07:00
George Gevoian
090d9af21d (core) Broadcast doc usage updates to clients
Summary:
Introduces a new message type, docUsage, that's broadcast to all connected
clients whenever document usage is updated in ActiveDoc.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3451
2022-06-06 09:55:34 -07:00
Dmitry S
acddd25cfd (core) Update design of empty docs home page, and add a "Manage Team" button.
Summary:
- Remove the empty-folder icon
- Add an "Invite team members" button for owners on empty team sites
- Add a "Browse Templates" button for all other cases on empty sites
- Update intro text for team, including a link to Sprouts
- Update intro text for personal/anon.
- Include a Free/Pro tag for team sites (for now, only "Free")
- Add a "Manage Team" button for owners on home page of all team sites.
- Polished the UI of UserManager: add a transition for the background, and
  delay the appearance of the spinner for fast loads.

Test Plan: Fixed up the HomeIntro tests; added test case for Manage Team button

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3459
2022-06-03 12:58:57 -04:00
Cyprien P
815c9e1462 (core) Adds new range filter for numeric columns
Summary:
Shows the range filter next to the filter by values on filter menu. When users
set min and/or max, it takes precendence over the filter by values.

If users set:
 - `[] < [max]` behaves as `less than max`.
 - `[min] < []` behaves as `more than min`.
 - `[min] < [max]` behaves as `between min and max`
 - bounds are always inclusives.
 - when users change min or max the values of the by values filter
   gets checked/unchecked depending on whether they are included by
   the range filter.
 - when users clicks any btn/checkbox of the by values filter both min
   and max input gets cleared, and the filter convert to a filter by
   values.

Test Plan: Adds both projets and nbrowser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3435
2022-05-30 10:30:16 +02:00
Alex Hall
6b372fa6cd (core) Allow configuring (mostly hiding) various little bits of UI
Summary:
Adds two new env vars GRIST_HIDE_UI_ELEMENTS and GRIST_PAGE_TITLE_SUFFIX which translate to values in GristLoadConfig that the server sends the client when loading.

For checkin task https://gristlabs.getgrist.com/doc/check-ins/p/5#a1.s9.r1882.c19

Test Plan: Tested manually

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3449
2022-05-27 14:32:05 +02:00
George Gevoian
74ec9358da (core) Show usage banners in doc menu of free team sites
Summary:
Also fixes a minor CSS regression in UserManager where the
link to add a team member wasn't shown on a separate row.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3444
2022-05-26 15:01:35 -07:00
Dmitry S
309ddb0fe7 (core) Move guessing logic for column types to run in node once for all columns.
Summary:
Previously, columns of type Any were created and modified one by one by reusing
the "empty column" logic from the data engine. This copies that logic to Node,
and sets the type of all columns together, to create them with the correct type
in the AddTable call.

This makes imports about twice faster (when slowness is due to many columns),
but doesn't address all cases where individual handling of columns causes slowness.

Test Plan: Added a test case for the new helper function.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3427
2022-05-19 12:49:51 -04:00
George Gevoian
a6063f570a (core) Polish Access Details
Summary:
Instead of showing a blank dialog for users whose access
is limited (e.g. public members), we now show the user's
role and a mention of whether their access is public.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3431
2022-05-19 09:45:26 -07:00
Jarosław Sadziński
0ab9e4a6a0 (core) Adding GristConnect login system
Summary:
New login system to allow simple SSO flow that is based on Discourse description that is available at:
https://meta.discourse.org/t/discourseconnect-official-single-sign-on-for-discourse-sso/13045

Test Plan: New core test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3418
2022-05-18 20:28:25 +02:00
George Gevoian
2fd8a34ff8 (core) Move Notifier to /ext
Summary:
This makes it possible to configure a SendGrid-based Notifier
instance via a JSON configuration file.

Test Plan: Tested manually.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3432
2022-05-18 08:02:32 -07:00
Paul Fitzpatrick
e4d47a2f3c (core) add minimal support for activation keys
Summary: For grist-ee, expect an activation key in environment variable `GRIST_ACTIVATION` or in a file pointed to by `GRIST_ACTIVATION_FILE`. In absence of key, start a 30-day trial, during which a banner is shown. Once trial expires, installation goes into document-read-only mode.

Test Plan: added a test

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3426
2022-05-16 22:46:23 -04:00
George Gevoian
f48d579f64 (core) Add API endpoint to get site usage summary
Summary:
The summary includes a count of documents that are approaching
limits, in grace period, or delete-only. The endpoint is only accessible
to site owners, and is currently unused. A follow-up diff will add usage
banners to the site home page, which will use the response from the
endpoint to communicate usage information to owners.

Test Plan: Browser and server tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3420
2022-05-16 11:16:19 -07:00
Alex Hall
4408315f2e (core) Add AzureExternalStorage
Summary:
Adds a new implementation of the interface ExternalStorage that works for Azure Blob Storage as an alternative to S3, for a specific self-hosting case.

Tweaks HostedStorageManager and ICreate to allow configuring different core implementations of ExternalStorage.

Followup tasks:

- Make this code available to self hosters, possibly by making it open source.
- Add an env var or other config option to specify the preferred type of storage. Currently using the var `AZURE_STORAGE_CONNECTION_STRING` to know how to connect to Azure when requested, but that choice still only lives in test code.

Test Plan: Generalized HostedStorageManager and ExternalStorage tests to test the new AzureExternalStorage alongside S3ExternalStorage. The HostedStorageManager tests also now test the 'cached' in-memory test storage in a way that's closer to the real storage methods.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3413
2022-05-09 21:44:57 +02:00