Summary:
introduces POST /api/docs/{docId}/webhooks and DELETE /api/docs/{docId}/webhooks/{webhookId} on place of old _subscribe and _unsubscribe endpoints.
Remove checking for unsubscribeKey while deleting webhook - only owner can delete webhook using DELETE endpoint. subscription key is still needed for _unsubscribe endpoint.
old _unsubscribe and _subscribe endpoints are still active and work as before - no changes there.
Posting schema:
```
POST /api/docs/[docId]/webhooks
```
Request Body:
```
{
"webhooks": [
{
"fields": {
"url": "https://webhook.site/3bd02246-f122-445e-ba7f-bf5ea5bb6eb1",
"eventTypes": [
"add",
"update"
],
"enabled": true,
"name": "WebhookName",
"memo": "just a text",
"tableId": "Table1"
}
},
{
"fields": {
"url": "https://webhook.site/3bd02246-f122-445e-ba7f-bf5ea5bb6eb2",
"eventTypes": [
"add",
],
"enabled": true,
"name": "OtherWebhookName",
"memo": "just a text",
"tableId": "Table1"
}
}
]
}
```
Expected response: WebhookId for each webhook posted:
```
{
"webhooks": [
{
"id": "85c77108-f1e1-4217-a50d-acd1c5996da2"
},
{
"id": "d87a6402-cfd7-4822-878c-657308fcc8c3"
}
]
}
```
Deleting webhooks:
```
DELETE api/docs/[docId]/webhooks/[webhookId]
```
there is no payload in DELETE request. Therefore only one webhook can be deleted at once
Response:
```
{
"success": true
}
```
Test Plan: Old unit test improved to handle new endpoints, and one more added to check if endpoints are in fact created/removed
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: paulfitz, alexmojaki
Differential Revision: https://phab.getgrist.com/D3916
Summary:
Implements the latest design of the Formula AI Assistant.
Also switches out brace to the latest build of ace.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3949
Summary:
Adding limits for AI calls and connecting those limits with a Stripe Account.
- New table in homedb called `limits`
- All calls to the AI are not routed through DocApi and measured.
- All products now contain a special key `assistantLimit`, with a default value 0
- Limit is reset every time the subscription has changed its period
- The billing page is updated with two new options that describe the AI plan
- There is a new popup that allows the user to upgrade to a higher plan
- Tiers are read directly from the Stripe product with a volume pricing model
Test Plan: Updated and added
Reviewers: georgegevoian, paulfitz
Reviewed By: georgegevoian
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3907
Summary:
In a selector table, when a selected row is filtered out of view, linked widgets should update based on the newly selected row.
There were a few bugs that contributed to this wrong behavior:
- Gridview wasn't subscribing to the current row id, and the row with id 'new' was being converted to the first row
- Cursor was keeping track of the currently selected row id, it was hiding a problem behind the proper rowIndex
- Undo/redo somehow leveraged the wrong rowId from the cursor during the position restore.
The `No data` text was also changed to be more meaningful.
Test Plan: Added and updated.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3937
Summary:
Adds a new Support Grist page (accessible only in grist-core), containing
options to opt in to telemetry and sponsor Grist Labs on GitHub.
A nudge is also shown in the doc menu, which can be collapsed or permanently
dismissed.
Test Plan: Browser and server tests.
Reviewers: paulfitz, dsagal
Reviewed By: paulfitz
Subscribers: jarek, dsagal
Differential Revision: https://phab.getgrist.com/D3926
Summary:
Here's a series of badness that easily leads to a crash, in reverse order:
- Lodash's map() function interprets an object with a .length property as an array.
- Some very old code generated human-friendly descriptions of user actions,
applying map() to parts of them. It so happens that this generated description
isn't even used.
- If a user action is encountered with a sufficiently large length propery,
map() would exhaust the server memory.
Fixed by removing old unneeded code, and replacing some other occurrences of
lodash's map() with native equivalents.
Test Plan: Tested manually on a local reproduction of the issue.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3938
Summary:
- Detecting emoji is surprisingly tricky; we use a fancy regex as a decent heuristic.
- Icons are a little larger than before.
- Styling tweaked for light and dark modes
- In case the OS doesn't render the emoji as one character, truncate what's
shown in the icon box.
Test Plan: Added a test case.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3904
Summary:
Some edits to virtual tables (such as webhook lists) happen
via a route that was not yet handled. Actually Cyprien (the
original author) had handled this case but it got removed
because I didn't know what it was for :-). This brings back
support for edits by this route.
Test Plan: added a test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3924
Summary:
Previously we failed to log signup info for users who signed up via
Google. This fixes that issue by recording it on first post-signup
visit. It also includes signup as a new telemetry event, recorded at the
same point.
Test Plan: Tested locally to see that a signup produces an appropriate log message and telemetry event.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3921
Summary:
- Move css module for the login page css to core/, to be reusable in core/ pages.
- Move /welcome/teams implementation to WelcomeSitePicker.ts
- List users for personal sites, as well as team sites.
- Add org param to setSessionActive() API method and end endpoint, to allow
switching the specified org to another user.
- Add a little safety to getOrgUrl() function.
Test Plan: Added a test case for the new behaviors of the /welcome/teams page.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3914
Summary: Also fixes a few small bugs with telemetry collection.
Test Plan: Server and manual tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3915
Summary:
For grist-static, we want to the data engine to be able to call external/exported JS functions directly,
rather than via the node 'server' living in another thread which requires synchronous communication hackery.
As a step in that direction, this diff changes the exported functions that we care about (guessColInfo and convertFromColumn)
to just using the top-level functions instead of relying on fields in ActiveDoc, namely docData.
For guessColInfo, this is done by directly passing the small amount of metadata that was previously retrieved from the DocData.
For convertFromColumn, disentangling DocData is a lot more complicated, so instead we construct a fresh DocData object using
the required metadata tables which are now passed in by the data engine.
Test Plan: Existing tests
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3913
Summary:
Adds support for optional telemetry to grist-core.
A new environment variable, GRIST_TELEMETRY_LEVEL, controls the level of telemetry collected.
Test Plan: Server and unit tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: dsagal, anaisconce
Differential Revision: https://phab.getgrist.com/D3880
Summary:
Adding a way to detach an editor. Initially only implemented for the formula editor, includes redesign for the AI part.
- Initially, the detached editor is tight with the formula assistant and both are behind GRIST_FORMULA_ASSISTANT flag, but this can be relaxed
later on, as the detached editor can be used on its own.
- Detached editor is only supported in regular fields and on the creator panel. It is not supported yet for conditional styles, due to preview limitations.
- Old code for the assistant was removed completely, as it was only a temporary solution, but the AI conversation part was copied to the new one.
- Prompting was not modified in this diff, it will be included in the follow-up with more test cases.
Test Plan: Added only new tests; existing tests should pass.
Reviewers: JakubSerafin
Reviewed By: JakubSerafin
Differential Revision: https://phab.getgrist.com/D3863
Summary:
- Excel exports were awfully memory-inefficient, causing occasional docWorker
crashes. The fix is to use the "streaming writer" option of ExcelJS
https://github.com/exceljs/exceljs#streaming-xlsx-writercontents. (Empirically
on one example, max memory went down from 3G to 100M)
- It's also CPU intensive and synchronous, and can block node for tens of
seconds. The fix is to use a worker-thread. This diff uses "piscina" library
for a pool of threads.
- Additionally, adds ProcessMonitor that logs memory and cpu usage,
particularly when those change significantly.
- Also introduces request cancellation, so that a long download cancelled by
the user will cancel the work being done in the worker thread.
Test Plan:
Updated previous export tests; memory and CPU performance tested
manually by watching output of ProcessMonitor.
Difference visible in these log excerpts:
Before (total time to serve request 22 sec):
```
Telemetry processMonitor heapUsedMB=2187, heapTotalMB=2234, cpuAverage=1.13, intervalMs=17911
Telemetry processMonitor heapUsedMB=2188, heapTotalMB=2234, cpuAverage=0.66, intervalMs=5005
Telemetry processMonitor heapUsedMB=2188, heapTotalMB=2234, cpuAverage=0, intervalMs=5005
Telemetry processMonitor heapUsedMB=71, heapTotalMB=75, cpuAverage=0.13, intervalMs=5002
```
After (total time to server request 18 sec):
```
Telemetry processMonitor heapUsedMB=109, heapTotalMB=144, cpuAverage=0.5, intervalMs=5001
Telemetry processMonitor heapUsedMB=109, heapTotalMB=144, cpuAverage=1.39, intervalMs=5002
Telemetry processMonitor heapUsedMB=94, heapTotalMB=131, cpuAverage=1.13, intervalMs=5000
Telemetry processMonitor heapUsedMB=94, heapTotalMB=131, cpuAverage=1.35, intervalMs=5001
```
Note in "Before" that heapTotalMB goes up to 2GB in the first case, and "intervalMs" of 17 seconds indicates that node was unresponsive for that long. In the second case, heapTotalMB stays low, and the main thread remains responsive the whole time.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3906
Summary:
DateTime columns had a blank timezone after xlsx imports because the
timezone was not included in the column type. We now append the
document's timezone to the type of all imported DateTime columns.
Test Plan: Server test.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3896
Summary:
Tutorials are now hidden by default in grist-core and grist-ee, and can
be re-enabled via a new env variable, GRIST_UI_FEATURES, which accepts
a comma-separated list of UI features to enable.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3885
Summary:
Due to a mishap, two distinct migrations with the same migration
number were introduced into Grist. This diff reconciles them as
best we can, by adding another migration to make sure both desired
changes have run (and running them if not).
Test Plan:
updated a test; checked manually that documents
with different 38 migrations are handled as expected.
Reviewers: georgegevoian, jarek
Reviewed By: georgegevoian, jarek
Differential Revision: https://phab.getgrist.com/D3895
Summary:
This adds a UI panel for managing webhooks. Work started by Cyprien Pindat. You can find the UI on a document's settings page. Main changes relative to Cyprien's demo:
* Changed behavior of virtual table to be more consistent with the rest of Grist, by factoring out part of the implementation of on-demand tables.
* Cell values that would create an error can now be denied and reverted (as for the rest of Grist).
* Changes made by other users are integrated in a sane way.
* Basic undo/redo support is added using the regular undo/redo stack.
* The table list in the drop-down is now updated if schema changes.
* Added a notification from back-end when webhook status is updated so constant polling isn't needed to support multi-user operation.
* Factored out webhook specific logic from general virtual table support.
* Made a bunch of fixes to various broken behavior.
* Added tests.
The code remains somewhat unpolished, and behavior in the presence of errors is imperfect in general but may be adequate for this case.
I assume that we'll soon be lifting the restriction on the set of domains that are supported for webhooks - otherwise we'd want to provide some friendly way to discover that list of supported domains rather than just throwing an error.
I don't actually know a lot about how the front-end works - it looks like tables/columns/fields/sections can be safely added if they have string ids that won't collide with bone fide numeric ids from the back end. Sneaky.
Contains a migration, so needs an extra reviewer for that.
Test Plan: added tests
Reviewers: jarek, dsagal
Reviewed By: jarek, dsagal
Differential Revision: https://phab.getgrist.com/D3856
* add support for conversational state to assistance endpoint
This refactors the assistance code somewhat, to allow carrying
along some conversational state. It extends the OpenAI-flavored
assistant to make use of that state to have a conversation.
The front-end is tweaked a little bit to allow for replies that
don't have any code in them (though I didn't get into formatting
such replies nicely).
Currently tested primarily through the runCompletion script,
which has been extended a bit to allow testing simulated
conversations (where an error is pasted in follow-up, or
an expected-vs-actual comparison).
Co-authored-by: George Gevoian <85144792+georgegevoian@users.noreply.github.com>
Summary:
The URL /create-team-site on marketing site is unused, and no longer pointing to
anything functional.
The "efcr" product flavor has been defunct for a long time. Remove
references to it.
Test Plan: No tests should be affected
Reviewers: JakubSerafin
Reviewed By: JakubSerafin
Subscribers: JakubSerafin
Differential Revision: https://phab.getgrist.com/D3890
Summary:
- When importing into a Ref column, use lookupOne() formula for correct previews.
- When selecting columns to import into a Ref column, now a Numeric column like
'Order' will produce two options: "Order" and "Order (as row ID)".
- Fixes exports to correct the formatting of visible columns. This addresses multiple bugs:
1. Formatting wasn't used, e.g. a Ref showing a custom-formatted date was still presented as YYYY-MM-DD in CSVs.
2. Ref showing a Numeric column was formatted as if a row ID (e.g. `Table1[1.5]`), which is very wrong.
- If importing into a table that doesn't have a primary view, don't switch page after import.
Refactorings:
- Generalize GenImporterView to be usable in more cases; removed near-duplicated logic from node side
- Some other refactoring in importing code.
- Fix field/column option selection in ValueParser
- Add NUM() helper to turn integer-valued floats into ints, useful for "as row ID" lookups.
Test Plan: Added test cases for imports into reference columns, updated Exports test fixtures.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3875
Summary:
When a chart page/section is added and the creator panel is already open, focus is set to a "table" tab.
When a custom view is added as a page/section, the same thing happens, but there is also a behavioral tooltip shown
for the custom URL.
Test Plan: Updated and added.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3857
Summary:
Enhances dark mode support for the formula editor, and adds support to
the color select popup. Also fixes some bugs with dark mode.
Test Plan: Tested manually.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3847
Summary:
Adds a new endpoint to update webhook.
Perform some refactoring to allow code reuse from endpoint allowing to _subscribe and _unsubscribe webhooks.
One aspect of webhook is that url are stored in the home db while the rest of the fields (tableRef, isReadyColRef, ...) are stored in sqlite. So care must be taken when updating fields, to properly rollback if anything should fail.
Follow up diff will bring UI to edit webhook list
Test Plan: Updated doc api server tests
Reviewers: jarek
Reviewed By: jarek
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3821
Summary:
For now only html stub and docList adjustement for showing a tutorial card.
It will be used in future diffs after tutorial implementation.
Test Plan: Manual
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3750
Summary: What happens when you type "rr" instead of "r" in an anchor link's row number?
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3829
Summary:
The feature is behind a flag GRIST_FORMULA_ASSISTANT (must be "true"). But can be enabled in the
developer console by invoking GRIST_FORMULA_ASSISTANT.set(true).
Keys can be overriden in the document settings page.
Test Plan: For now just a stub test that checks if this feature is disabled by default.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3815
Summary:
Documents can now be flagged as tutorials, which causes them to display
Markdown-formatted slides from a special GristDocTutorial table. Tutorial
documents are forked on open, and remember the last slide a user was on.
They can be restarted too, which prepares a new fork of the tutorial.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3813
Summary:
Porting script that run an evaluation against our formula dataset.
To test you need an openai key (see here: https://platform.openai.com/)
or hugging face (it should work as well), then checkout the branch and run
`OPENAI_API_KEY=<my_openai_api_key> node core/test/formula-dataset/runCompletion.js`
Test Plan:
Needs manually testing: so far there is no plan to make it part of CI.
The current score is somewhere around 34 successful prompts over a total of 47.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3816
Summary:
Warnings about deprecated shortcuts are no longer needed.
As a side effect it fixes a bug that caused those warnings to not persist its
state on pages with charts.
Test Plan: Removed
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3820
Summary:
A feature that allows minimizing widgets on the ViewLayout.
- Code in ViewLayout and Layout hasn't been changed. Only some methods or variables were made public, and some events are now triggered when a section is dragged.
- Widgets can be collapsed or expanded (added back to the main area)
- Collapsed widgets can be expanded and shown as a popup
- Collapsed widgets support drugging, reordering, and transferring between the main and collapsed areas.
Test Plan: New test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3779
Summary:
Fixes a test util that broke after a sync with grist-core, and upgrades
fixtures after a migration was added.
Test Plan: N/A
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3810
Summary:
Adds information about forks to the home db. This will be used
later by the UI to list forks of documents.
Test Plan: Browser and server tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3772
* Replace `ormconfig.js` with a newer mechanism of configuring
TypeORM that can be included in the source code properly.
The path to `ormconfig.js` has always been awkward to handle,
and eliminating the file makes building different Grist setups
a bit simpler.
* Remove `electron` package. It is barely used, just for some old
remnants of an older attempt at electron packaging. It was used
for two types, which I left at `any` for now. More code pruning is
no doubt possible here, but I'd rather do it when Electron packaging
has solidified.
* Add a hook for replacing the login system, and for adding some
extra middleware the login system may need.
* Add support for some more possible locations of Python, which
arise when a standalone version of it is included in the Electron
package. This isn't very general purpose, just configurations
that I found useful.
* Support using grist-core within a yarn workspace - the only tweak
needed was webpack related.
* Allow an external ID to be optionally associated with documents.
Summary: This is a backend part for the formula AI.
Test Plan: New tests
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: cyprien
Differential Revision: https://phab.getgrist.com/D3786
