Commit Graph

1155 Commits

Author SHA1 Message Date
jarek
9edbf9f415
Setting main grist-widget repository as a default (#524)
* Setting main grist-widget repository as a default to GRIST_WIDGET_LIST_URL

* Putting widget repository URL in commonURLs
2023-06-02 10:17:09 +02:00
Dmitry S
d191859be7 (core) For exporting XLSX, do it memory-efficiently in a worker thread.
Summary:
- Excel exports were awfully memory-inefficient, causing occasional docWorker
  crashes. The fix is to use the "streaming writer" option of ExcelJS
  https://github.com/exceljs/exceljs#streaming-xlsx-writercontents. (Empirically
  on one example, max memory went down from 3G to 100M)
- It's also CPU intensive and synchronous, and can block node for tens of
  seconds. The fix is to use a worker-thread. This diff uses "piscina" library
  for a pool of threads.
- Additionally, adds ProcessMonitor that logs memory and cpu usage,
  particularly when those change significantly.
- Also introduces request cancellation, so that a long download cancelled by
  the user will cancel the work being done in the worker thread.

Test Plan:
Updated previous export tests; memory and CPU performance tested
manually by watching output of ProcessMonitor.

Difference visible in these log excerpts:

Before (total time to serve request 22 sec):
```
Telemetry processMonitor heapUsedMB=2187, heapTotalMB=2234, cpuAverage=1.13, intervalMs=17911
Telemetry processMonitor heapUsedMB=2188, heapTotalMB=2234, cpuAverage=0.66, intervalMs=5005
Telemetry processMonitor heapUsedMB=2188, heapTotalMB=2234, cpuAverage=0, intervalMs=5005
Telemetry processMonitor heapUsedMB=71, heapTotalMB=75, cpuAverage=0.13, intervalMs=5002
```
After (total time to server request 18 sec):
```
Telemetry processMonitor heapUsedMB=109, heapTotalMB=144, cpuAverage=0.5, intervalMs=5001
Telemetry processMonitor heapUsedMB=109, heapTotalMB=144, cpuAverage=1.39, intervalMs=5002
Telemetry processMonitor heapUsedMB=94, heapTotalMB=131, cpuAverage=1.13, intervalMs=5000
Telemetry processMonitor heapUsedMB=94, heapTotalMB=131, cpuAverage=1.35, intervalMs=5001
```
Note in "Before" that heapTotalMB goes up to 2GB in the first case, and "intervalMs" of 17 seconds indicates that node was unresponsive for that long. In the second case, heapTotalMB stays low, and the main thread remains responsive the whole time.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3906
2023-06-01 12:06:48 -04:00
Dmitry S
6b338256e0 (core) On mobile, only show a warning for older and unknown browsers rather than for all
Test Plan:
Tested listed versions against docs.getgrist.com using browserstack, and tested
with an iPhone and local server that new settings are noticed on mobile.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3909
2023-06-01 11:15:21 -04:00
Jarosław Sadziński
c592691e31 (core) Fixing DELETE and BACKSPACE keys on ChoiceList and RefList editor.
Summary:
Choice/Reference List editor wasn't clearing itself when it received an empty string. It led
to a bug on the Card widget where pressing those keys resulted in the same behavior as
pressing Enter - it just opened the editor.
Grid view has it's own implementation for those keys, so it wasn't affected.

Test Plan: Added new test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3908
2023-06-01 16:53:38 +02:00
Paul Fitzpatrick
dad41b2567 (core) updates from grist-core 2023-05-30 08:32:34 -04:00
George Gevoian
ff03d32688 (core) Set DateTime timezone during xlsx import
Summary:
DateTime columns had a blank timezone after xlsx imports because the
timezone was not included in the column type. We now append the
document's timezone to the type of all imported DateTime columns.

Test Plan: Server test.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3896
2023-05-24 11:39:49 -04:00
George Gevoian
d5b8240c07 (core) Fix snapshot migrations
Summary:
Migrations were failing in snapshots due to the sandbox no longer
being started in snapshots. We now start up an instance of the
sandbox whenever there are migrations to run, and immediately shut
it down on completion.

Test Plan: Server test.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3898
2023-05-23 15:50:15 -04:00
Paul Fitzpatrick
7be0ee289d
support other SQLite wrappers, and various hooks needed by grist-static (#516) 2023-05-23 15:17:28 -04:00
Paul Fitzpatrick
3f3a0d3aa1 (core) support a wildcard option for ALLOWED_WEBHOOK_DOMAINS
Summary:
Now that webhook payload delivery can be done using a proxy,
it may be desirable to no longer require a set of `ALLOWED_WEBHOOK_DOMAINS`.
This diff allows this variable to be set to `*`. With this setting,
any domain, and both `http` and `https` protocols will now be accepted.

Another possibility would be to default to unchecked
behavior if `ALLOWED_WEBHOOK_DOMAINS` is not set. But this would
introduce a new kind of vulnerability to unconfigured Grist
installations.

Test Plan: switched a test from naming a domain to using `*`

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3903
2023-05-23 13:40:52 -04:00
Paul Fitzpatrick
4cc19ff748 (core) updates from grist-core 2023-05-23 09:10:34 -04:00
Dmitry S
d4bc6246f1 (core) Fix breakage on Firefox iOS
Summary:
Grist recently stopped working on Firefox on iOS. The cause turns out an uncaught error, which is reported as an unhelpful "Script Error", but the act of reporting it causes additional errors, leading to an infinite loop and an unusable browser tab.

Firefox-iOS is to blame, but a workaround is preventing a flood of "Script Error" messages. Specifically, we report only the first of these, and only to the server, suppressing the user-visible toast.

Test Plan: Tested manually on Firefox on iOS. Added a test case, and improve other tests of uncaught errors.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3902
2023-05-22 16:22:41 -04:00
George Gevoian
f18bb3e39d (core) Add GRIST_UI_FEATURES env variable
Summary:
Tutorials are now hidden by default in grist-core and grist-ee, and can
be re-enabled via a new env variable, GRIST_UI_FEATURES, which accepts
a comma-separated list of UI features to enable.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3885
2023-05-22 16:05:51 -04:00
wunter8
e7933d7c59
(core) Remove bool filtered rows immediately on change (#517)
* (core) Remove bool filtered rows immediately on change
2023-05-20 19:55:36 -04:00
George Gevoian
1e873b4203 (core) Tweak telemetry
Summary: Adjusts the level of telemetry collected from Grist SaaS.

Test Plan: Tested manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3899
2023-05-19 13:06:15 -04:00
Dmitry S
be5cb9124a (core) Add logging of errors whenever ProxyAgent is used, and a test for it.
Summary:
Also:
- Move ProxyAgent to from app/server/utils to app/server/lib, which is
  the more usual place for such classes.
- Refactor a helper (delayAbort) that node was reporting a leak in.

Test Plan: Added a test case, and tested manually.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Subscribers: JakubSerafin, paulfitz

Differential Revision: https://phab.getgrist.com/D3897
2023-05-17 10:21:53 -04:00
Paul Fitzpatrick
84854b7cfa (core) updates from grist-core 2023-05-15 12:01:19 -04:00
Paul Fitzpatrick
b58929f095 (core) reconcile webhook and widget description migrations
Summary:
Due to a mishap, two distinct migrations with the same migration
number were introduced into Grist. This diff reconciles them as
best we can, by adding another migration to make sure both desired
changes have run (and running them if not).

Test Plan:
updated a test; checked manually that documents
with different 38 migrations are handled as expected.

Reviewers: georgegevoian, jarek

Reviewed By: georgegevoian, jarek

Differential Revision: https://phab.getgrist.com/D3895
2023-05-15 11:56:15 -04:00
softwareguru90
72730461eb
Track saved version per hour, day, week, month, year, and number of times a version with parameter (#509)
Determining the number of snapshots to be kept with a parameter
2023-05-12 11:38:29 -04:00
CamilleLegeron
c16204f8ad
feature widget description (#483)
Add description to widget title popup and right panel
2023-05-12 09:08:28 -04:00
George Gevoian
18ad39cba3 (core) Add cut, copy, and paste to context menu
Summary:
On supported browsers, the new context menu commands work exactly as they do
via keyboard shortcuts. On unsupported browsers, an unavailable command
modal is shown with a suggestion to use keyboard shortcuts instead.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3867
2023-05-10 00:48:15 -04:00
George Gevoian
c6ec8339f5 (core) Avoid adding extra "(" when autocompleting
Summary:
This tweaks formula autocomplete to avoid adding an extra "(" when we
detect that a function or method is being renamed.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3877
2023-05-09 20:03:37 -04:00
Paul Fitzpatrick
603238e966 (core) Adds a UI panel for managing webhooks
Summary:
This adds a UI panel for managing webhooks. Work started by Cyprien Pindat. You can find the UI on a document's settings page. Main changes relative to Cyprien's demo:

  * Changed behavior of virtual table to be more consistent with the rest of Grist, by factoring out part of the implementation of on-demand tables.
  * Cell values that would create an error can now be denied and reverted (as for the rest of Grist).
  * Changes made by other users are integrated in a sane way.
  * Basic undo/redo support is added using the regular undo/redo stack.
  * The table list in the drop-down is now updated if schema changes.
  * Added a notification from back-end when webhook status is updated so constant polling isn't needed to support multi-user operation.
  *  Factored out webhook specific logic from general virtual table support.
  * Made a bunch of fixes to various broken behavior.
  * Added tests.

The code remains somewhat unpolished, and behavior in the presence of errors is imperfect in general but may be adequate for this case.

I assume that we'll soon be lifting the restriction on the set of domains that are supported for webhooks - otherwise we'd want to provide some friendly way to discover that list of supported domains rather than just throwing an error.

I don't actually know a lot about how the front-end works - it looks like tables/columns/fields/sections can be safely added if they have string ids that won't collide with bone fide numeric ids from the back end. Sneaky.

Contains a migration, so needs an extra reviewer for that.

Test Plan: added tests

Reviewers: jarek, dsagal

Reviewed By: jarek, dsagal

Differential Revision: https://phab.getgrist.com/D3856
2023-05-08 18:25:27 -04:00
Paul Fitzpatrick
5e9f2e06ea (core) updates from grist-core 2023-05-08 14:17:45 -04:00
George Gevoian
9438f315e9 (core) Save choice config on focus loss
Summary: Changes to choices are now saved whenever focus leaves the editor.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3879
2023-05-08 14:16:10 -04:00
Paul Fitzpatrick
51a195bd94
add support for conversational state to assistance endpoint (#506)
* add support for conversational state to assistance endpoint

This refactors the assistance code somewhat, to allow carrying
along some conversational state. It extends the OpenAI-flavored
assistant to make use of that state to have a conversation.
The front-end is tweaked a little bit to allow for replies that
don't have any code in them (though I didn't get into formatting
such replies nicely).

Currently tested primarily through the runCompletion script,
which has been extended a bit to allow testing simulated
conversations (where an error is pasted in follow-up, or
an expected-vs-actual comparison).

Co-authored-by: George Gevoian <85144792+georgegevoian@users.noreply.github.com>
2023-05-08 14:15:22 -04:00
Jakub Serafin
440d5b935a (core) Proxy Agent moved to the separate file, Triggers are using proxy now to perform fetch
Summary:
- Webhooks form Triggers.ts should now use proxy if it's configured
- Proxy handling code separated to ProxyAgent.ts
- Tests for ProxyAgent
- Integration/API Tests for using Proxy in webhooks
- a bit of refactor - proxy test uses mostly the same codebase as DocApi.ts, but because last one if over 4000 lines long, I've put it into separated file, and extract some common parts (there is some duplicates tho)
- some cleanup in files that I've touched

Test Plan:
Manual test to check if proxy is used on the staging env

Automatic test checking if (fake) proxy was called

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3860
2023-05-08 11:54:09 +02:00
Jakub Serafin
37347a79c0 (core) Currency from grist column is persistent when exporting to excel
Summary:
- when grist table is exported, currency is check and introduced in cell format in the form of "[currency symbol] [value]" (for example: zł 10000, $ 5000) . It's not what some cultures should display currences, but it's close enought
- when no symbol is defined for the currency, currency 3 letters code is used instead
- when currency is unknown, we are falling back to "$"

Test Plan: - nbrowser test scenario added for that purpose, please check Currences.xlsx to see output format exported.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3886
2023-05-08 10:39:01 +02:00
Dmitry S
8f34ba5157 (core) Remove a defunct URL constant and a product flavor.
Summary:
The URL /create-team-site on marketing site is unused, and no longer pointing to
anything functional.

The "efcr" product flavor has been defunct for a long time. Remove
references to it.

Test Plan: No tests should be affected

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Subscribers: JakubSerafin

Differential Revision: https://phab.getgrist.com/D3890
2023-05-05 18:28:04 -04:00
Jarosław Sadziński
ca3cf0cd06 (core) Preserving rules when data is transformed
Summary: Rules where removed when data in column was transformed.

Test Plan: Added new test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3883
2023-05-05 13:44:09 +02:00
Jarosław Sadziński
5f9f4868ae (core) Showing close button when column is added
Summary:
When a column is added the rename popup had a disabled save button.
Now we always show either:
Just "Close" if there are no changes.
"Save" and "Cancel" if there are changes.

Also, the description is trimmed when saved through the creator panel.

Test Plan: Added

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3872
2023-05-04 11:35:16 +02:00
George Gevoian
959f8a45c6 (core) Direct users to last visited site when possible
Summary:
When clicking the logo in the top-left corner, or finishing a tutorial, we
now direct users to the site they last visited, if possible. If unknown, a
new redirect endpoint, /welcome/home, is used instead, which directs users
to a sensible location based on the number of sites they have.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3878
2023-05-02 07:48:22 -07:00
Dmitry S
65013331a3 (core) Fix imports into reference columns, and support two ways to import Numeric as a reference.
Summary:
- When importing into a Ref column, use lookupOne() formula for correct previews.
- When selecting columns to import into a Ref column, now a Numeric column like
  'Order' will produce two options: "Order" and "Order (as row ID)".
- Fixes exports to correct the formatting of visible columns. This addresses multiple bugs:
  1. Formatting wasn't used, e.g. a Ref showing a custom-formatted date was still presented as YYYY-MM-DD in CSVs.
  2. Ref showing a Numeric column was formatted as if a row ID (e.g. `Table1[1.5]`), which is very wrong.
- If importing into a table that doesn't have a primary view, don't switch page after import.

Refactorings:
- Generalize GenImporterView to be usable in more cases; removed near-duplicated logic from node side
- Some other refactoring in importing code.
- Fix field/column option selection in ValueParser
- Add NUM() helper to turn integer-valued floats into ints, useful for "as row ID" lookups.

Test Plan: Added test cases for imports into reference columns, updated Exports test fixtures.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3875
2023-05-02 10:28:14 -04:00
Jarosław Sadziński
0bc758e3d0 (core) Styling text button for left alignment
Summary: Adding text alignment for text buttons.

Test Plan: Existing.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Differential Revision: https://phab.getgrist.com/D3884
2023-04-27 13:03:38 +02:00
Dmitry S
3513c1e8d2 (core) In formula autocompletions, fix suggestions after an opening parenthesis.
Summary:
Previously, a change was made to include the opening parenthesis into the
"identifier regex", so that backend received the function call name for cases
like 'T.lookupOne(B' (to autocomplete columns of table T that start with "B").
Unfortunately, that interferes with common cases like 'SUM($B', since instead
of sending '$B', it is now including the function name.

To properly fix the common case, we need to exclude the function name from the
call, but for lookups we need to include it. ACE's methods aren't flexible
enough to override this logic cleanly, so some monkey-patching was needed.

Test Plan: Added a test case

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3874
2023-04-26 15:26:44 -04:00
George Gevoian
8bedaedab6 (core) Fix ACL bug with rec.id in formula
Summary:
A regression was causing rec.id in formulas to be flagged as
invalid.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3882
2023-04-26 00:31:52 -07:00
Dmitry S
b4cc519616 (core) Ignore leading whitespace in formulas, and strip out leading '=' sign users might add
Summary:
This addresses two issues, differently:
- For a formula with leading whitespace, like " 1+1", it is stored as is, but
  is fixed to work (it should be valid Python, and whitespace is only stripped out
  at parsing time to avoid intentation errors caused by the way it gets parsed)
- For a formula with a leading equals-sign ("="), it is stripped out on the
  client side before the formula is stored. Grist documentation uses leading
  "=" to indicate formulas (because UI shows an "=" icon), and Excel formulas
  actually contain the leading "=", so it is a common mistake to include it.

Test Plan: Added new test cases

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3873
2023-04-25 15:28:40 -04:00
Paul Fitzpatrick
1d93923efe (core) updates from grist-core 2023-04-24 08:51:16 -04:00
Jarosław Sadziński
b4c4a62a73 (core) Migrating commands to typescript
Summary: Migrating commands and commandList to typescript

Test Plan: Existing

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3871
2023-04-24 09:48:40 +02:00
Jarosław Sadziński
b13fb1d97e (core) Adding description icon and tooltip in the GridView
Summary: Column description and new renaming popup for the GridView.

Test Plan: Updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3838
2023-04-20 18:21:31 +02:00
George Gevoian
3aac027a13 (core) Polish tutorial popups
Summary:
Includes the following changes:
 *  Adds "Click to expand" hover tooltip to all images
 *  Adds support for minimize/maximize by double clicking tutorial popup header
 *  Add New menu (and all other popups) should now persist when user moves tutorial popup
 *  Preserves scrollbar position when minimizing and maximizing tutorial popup
 *  Formula cell editor (and other elements) should now be stacked under tutorial

Test Plan: Browser and manual tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3864
2023-04-20 12:20:03 -04:00
Jarosław Sadziński
35f7072bea (core) Enabling height adjustment on tutorial popup
Summary:
Support height adjustment for tutorial popup, also change the way we calculate
delta for the movement to make it follow the cursor more smoothly.

Test Plan: Added

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: JakubSerafin

Differential Revision: https://phab.getgrist.com/D3858
2023-04-19 17:03:24 +02:00
George Gevoian
c077f3c304 (core) Fix reporting of blank attachment types
Summary:
The conditions in the map/filter of attachments was faulty, causing
blank attachment types to slip through.

Test Plan: Tested manually.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3865
2023-04-19 09:37:03 -04:00
George Gevoian
b15ae98349 (core) Fix browser history bug with tutorials
Summary:
History wasn't being replaced in some cases, which was
causing a bug where trying to leave a tutorial fork via the
browser's back button would navigate back to the trunk, and
trigger forking again. This effectively made it impossible to
leave a tutorial.

Also adds support for specifying custom CSS classes for
tutorial Markdown images.

Test Plan: Browser test.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Differential Revision: https://phab.getgrist.com/D3866
2023-04-19 00:22:42 -04:00
Jarosław Sadziński
69fea132de (core) Fix for the creator panel on custom section.
Summary: Stop opening the creator panel on chart or custom widgets. Only switch tabs when panel is open

Test Plan: Updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3868
2023-04-18 14:11:27 +02:00
CamilleLegeron
3775317eec
feat: allow simple click on certain cases for editing field in Widget Card (#446)
* feat: allow simple click on certain cases for editing field in cardView

* remove empty line

* test fix : add trick to limit the context of event

* desable the simple click behavior on widget card in mobile

* Delete timings.txt

* Delete xunit.xml

* clean-feat(Simple click on Card): use field type and id

* ignore local testing files

* codeStyle(single click on card): rename var + move util function in DetailView.prototype

* remove unused var

* CIFix(SingleClickOnCard): escape of field editor before close test
2023-04-17 11:14:25 -04:00
CamilleLegeron
3047bde1e4
rename welcomeTour file to WelcomeTour (#489) 2023-04-17 11:12:30 -04:00
Paul Fitzpatrick
f490854324 (core) updates from grist-core 2023-04-17 09:58:04 -04:00
George Gevoian
36f3fd0120 (core) Fix owner view access to snapshots
Summary:
Owners weren't able to access snapshots if access rules
that denied access to non-owners existed. The backend
was lowering snapshot document access to "viewers" as
part of implementing read-only behavior; this is now done
in the client, with document access for snapshots now
accurately reflecting the user's trunk access.

Additionally, sandboxes are no longer created for snapshots,
and background intervals aren't started for snapshots.

Test Plan: Browser test.

Reviewers: jarek, paulfitz

Reviewed By: jarek, paulfitz

Differential Revision: https://phab.getgrist.com/D3849
2023-04-17 00:16:59 -04:00
Vinschni
bcb5916434
make sendToDrive hidable (#493)
Make "sendToDrive" button from share menu hidable.
2023-04-15 16:22:56 -04:00
Jarosław Sadziński
40ea6bb2bc (core) Focusing the creator panel on the table wiget for charts and custom views
Summary:
When a chart page/section is added and the creator panel is already open, focus is set to a "table" tab.
When a custom view is added as a page/section, the same thing happens, but there is also a behavioral tooltip shown
for the custom URL.

Test Plan: Updated and added.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3857
2023-04-14 12:45:24 +02:00
Paul Fitzpatrick
cc0e1154d0 (core) port some more test/browser tests to newer selenium
Summary:
Ports more test/browser tests from *.test.js (run using an old selenium setup) to *.ntest.js (run using newer setup).

Weird test failures happened due to a change in timing. Eventually tracked in down to billing changes in one test suite resulting in reloads in another test suite, since it turns out redis pub/sub channels are not scoped to the redis database specified in REDIS_URL, but are global:
  https://redis.io/docs/manual/pubsub/#database--scoping.

Test Plan: Ported tests should run and pass

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3844
2023-04-12 13:00:53 -04:00
George Gevoian
900859854c (core) Convert site metric strings to ints
Summary:
Counts were being sent as strings to the Elasticsearch instance because the
SQL queries were returning strings for the aggregate columns instead of
integers.

Test Plan: N/A

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3859
2023-04-12 11:26:52 -04:00
George Gevoian
c5029af4e7 (core) Remove duplicate shouldHideUiElement
Summary: Also adds an explanatory comment for a recently-added column.

Test Plan: N/A

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3854
2023-04-12 02:14:04 -04:00
George Gevoian
8a0bb4d4fe (core) Improve dark mode
Summary:
Enhances dark mode support for the formula editor, and adds support to
the color select popup. Also fixes some bugs with dark mode.

Test Plan: Tested manually.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3847
2023-04-12 01:58:48 -04:00
Louis Delbosc
053ef45d64
Keep choices after apply formula (#474) 2023-04-11 09:31:46 +02:00
George Gevoian
a19ba0813a (core) Add telemetry
Test Plan: Server tests.

Reviewers: jarek

Differential Revision: https://phab.getgrist.com/D3818
2023-04-06 12:34:54 -04:00
Paul Fitzpatrick
6a4b7d96e8 (core) updates from grist-core 2023-04-06 10:02:41 -04:00
George Gevoian
4670d60c01 (core) Hide tutorial card with templates
Summary: Hides the tutorial card if GRIST_HIDE_UI_ELEMENTS includes "templates".

Test Plan: Tested manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3850
2023-04-04 18:23:55 -04:00
Paul Fitzpatrick
903c81d348
close db after checkAllegedGristDoc (#482)
This closes a file left open during importing, not by the import itself, but by a SQLite integrity check. This was causing imports to fail on Windows (see https://github.com/gristlabs/grist-electron/issues/3)
2023-04-04 15:59:26 -04:00
Jarosław Sadziński
7aa83858bd (core) Fixing bug with the import preview being not responsive
Summary: Fixing the bug by setting viewInstance on the temp viewsection record

Test Plan: Added

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3848
2023-04-04 18:55:23 +02:00
Paul Fitzpatrick
b2b41a0bf8
fix log.add under electron (#478)
Expose the log.add method, used in old Electron code. Without this change, using the Electron build of Grist fails if is configured for debug log output.
2023-04-03 13:29:35 -04:00
Paul Fitzpatrick
f5eb8f7730
do not count unready translations in localization test (#480)
This updates the expected count of languages in a localization test
to omit translations that have been started but are not yet ready
to be offered to users by default.
2023-04-03 12:55:08 -04:00
Paul Fitzpatrick
5730262c69 (core) updates from grist-core 2023-04-03 09:52:54 -04:00
Cyprien P
d8a063284a (core) Adds endpoint to update webhook
Summary:
Adds a new endpoint to update webhook.

Perform some refactoring to allow code reuse from endpoint allowing to _subscribe and _unsubscribe webhooks.

One aspect of webhook is that url are stored in the home db while the rest of the fields (tableRef, isReadyColRef, ...) are stored in sqlite. So care must be taken when updating fields, to properly rollback if anything should fail.

Follow up diff will bring UI to edit webhook list

Test Plan: Updated doc api server tests

Reviewers: jarek

Reviewed By: jarek

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3821
2023-03-31 19:26:02 +02:00
Jarosław Sadziński
8cb928e83d (core) Removing translation from reported answeres on welcome screen
Summary:
Questions were saved as localized values. Now they are saved in English using
resource keys.

Test Plan: Manual

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3845
2023-03-30 18:33:00 +02:00
George Gevoian
8833958963 (core) Fix small Start Tutorial button
Summary: The button was missing an href.

Test Plan: Tested manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3843
2023-03-29 23:14:15 -04:00
George Gevoian
0006e0604b (core) Disable tips during tutorials
Summary: Behavioral tips shouldn't be shown during tutorials.

Test Plan: Browser test.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3842
2023-03-29 16:10:23 -04:00
George Gevoian
5b2fb62627 (core) Avoid clashes with rick rowing and tours
Summary:
In certain scenarios, it's possible for a Grist tour and a tip that kicks off
an easter egg to be shown on screen at the same time. To deconflict, we
now check for the presence of one or the other and avoid showing both.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3840
2023-03-29 10:42:56 -04:00
George Gevoian
58ef8d59b5 (core) Show coaching popup after Add New tip is dismissed
Summary:
This is to prevent too many popups from being shown on screen at any
given moment.

Test Plan: Tested manually.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3841
2023-03-29 09:27:25 -04:00
Jarosław Sadziński
a317a727c8 (core) Adding tutorial card
Summary:
For now only html stub and docList adjustement for showing a tutorial card.
It will be used in future diffs after tutorial implementation.

Test Plan: Manual

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3750
2023-03-28 19:57:52 +02:00
Jarosław Sadziński
6b5c178953 (core) Fixing bug with type change when a collapsed section is maximized
Summary:
Changing type of a collapsed widget didn't work. It was grabbed
by the collapsed dom to soon.

Test Plan: Updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3839
2023-03-28 17:47:28 +02:00
Paul Fitzpatrick
a941a72904
only offer languages that have been marked as substantially complete (#466)
This checks for languages that have a special key translated.
Any that don't have the key translated, are not offered to the
user (unless GRIST_OFFER_ALL_LANGUAGES is set).

Co-authored-by: jarek <jaroslaw.sadzinski@gmail.com>
2023-03-28 10:59:17 -04:00
George Gevoian
3a1fbbc533 (core) Un-escape names in HomeIntro
Summary:
Characters such as & were mistakenly being escaped in the site name during
translation.

Test Plan: Tested manually.

Reviewers: jarek

Differential Revision: https://phab.getgrist.com/D3836
2023-03-28 07:43:43 -04:00
Jarosław Sadziński
edff50238d (core) Proper way to recreate collapsed section.
Summary: Fix for a bug with a collapsed custom widget, which doesn't work it is detached from a dom.

Test Plan: Added

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3835
2023-03-27 22:11:18 +02:00
George Gevoian
01fbe871aa (core) Add April Fools easter egg
Summary: What happens when you type "rr" instead of "r" in an anchor link's row number?

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3829
2023-03-27 14:12:52 -04:00
Paul Fitzpatrick
503e56db40
rename 'theme' that is shadowing an import (#473) 2023-03-27 12:38:07 -04:00
Paul Fitzpatrick
6485fbd826 (core) updates from grist-core 2023-03-27 09:48:59 -04:00
jarek
f9764f9780
Using static text in Grid menu for inserting columns. (#468) 2023-03-24 10:16:27 +01:00
Jarosław Sadziński
d29770511c (core) Draft version of AI assistant
Summary:
The feature is behind a flag GRIST_FORMULA_ASSISTANT (must be "true"). But can be enabled in the
developer console by invoking GRIST_FORMULA_ASSISTANT.set(true).

Keys can be overriden in the document settings page.

Test Plan: For now just a stub test that checks if this feature is disabled by default.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3815
2023-03-24 10:07:26 +01:00
Dmitry S
33c08057ad (core) Bug fix: when on Raw Data popup, clicks on cells shouldn't navigate the page
Test Plan: Added a test check that fails without the fix

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3831
2023-03-23 16:23:59 -04:00
Dmitry S
22dbab0123 (core) Add tooltip for the opener icon for the Creator Panel
Test Plan: Tested manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3830
2023-03-23 11:55:34 -04:00
Paul Fitzpatrick
12f9567ff4 (core) add a /welcome/start endpoint that forwards sensibly
Summary:
This adds a nuanced redirecting endpoint. For example, on
docs.getgrist.com it does:

 1) If logged in and no team site -> https://docs.getgrist.com/
 2) If logged in and has team sites -> https://docs.getgrist.com/welcome/teams
 3) If logged out but has a cookie -> /login, then 1 or 2
 4) If entirely unknown -> /signup

Test Plan: added a test; tested behavior through logins manually

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3828
2023-03-22 15:33:53 -04:00
Jarosław Sadziński
a9ff6b9a84 (core) Fixing couple of bugs in collapsed section layout
Summary:
The previous implementation for collapsing sections involved disposing of a view instance (Grid or Chart component). This caused numerous bugs with
linking sections as the implementation is located in the BaseView.js. Now the view instance is kept and attached to a dom in a hidden div, so it can respond
and function as a normal rendered section. It is also passed from between collapsed and main layout, when sections are dragged or moved using section's
menu commands (`collapse` and `add to main page`)

It also implies that the ViewLayout must still be rendered when a section is maximized (as it is responsible for the view instance), so the dom, and
some logic for rendering it, had to be changed.

Test Plan: New and updated

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3826
2023-03-22 16:23:09 +01:00
George Gevoian
be8e13df64 (core) Add initial tutorials implementation
Summary:
Documents can now be flagged as tutorials, which causes them to display
Markdown-formatted slides from a special GristDocTutorial table. Tutorial
documents are forked on open, and remember the last slide a user was on.
They can be restarted too, which prepares a new fork of the tutorial.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3813
2023-03-22 10:09:02 -04:00
George Gevoian
210aa92eed (core) Fix dark mode on welcome pages
Summary:
The welcome pages were unreadable because dark mode CSS was not
being applied.

Test Plan: Tested manually.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3827
2023-03-21 03:22:45 -04:00
Paul Fitzpatrick
327c78aa95
move a newly introduced private ActiveDoc method into the expected location (#465)
* place a new private ActiveDoc method in the expected order

This was tickling a lint failure on grist-core.

* reset an English translation that interferes with test currently
2023-03-20 11:25:09 -04:00
Paul Fitzpatrick
572995f19a (core) updates from grist-core 2023-03-20 09:46:37 -04:00
Paul Fitzpatrick
391d39effc (core) restore ActiveDoc shutdown behavior on error
Summary:
A recent change perturbed some error handling when an ActiveDoc
is shutting down. It is important that errors get thrown when
attempting to replace a non-existent document. My bad in review
for not catching.

Test Plan: Snapshot test passes again

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian, cyprien

Differential Revision: https://phab.getgrist.com/D3824
2023-03-17 12:18:09 -04:00
Louis Delbosc
c54e910fd6
Export table schema (#459)
* add endpoint
* Add table-schema transformation data
2023-03-16 17:37:24 -04:00
Cyprien P
1ff93f89c2 (core) Porting the AI evaluation script
Summary:
Porting script that run an evaluation against our formula dataset.

To test you need an openai key (see here: https://platform.openai.com/)
or hugging face (it should work as well), then checkout the branch and run

`OPENAI_API_KEY=<my_openai_api_key> node core/test/formula-dataset/runCompletion.js`

Test Plan:
Needs manually testing: so far there is no plan to make it part of CI.

The current score is somewhere around 34 successful prompts over a total of 47.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3816
2023-03-15 14:54:28 +01:00
Jarosław Sadziński
b3590c8a6f (core) Removing warnings about deprecated shortcuts.
Summary:
Warnings about deprecated shortcuts are no longer needed.
As a side effect it fixes a bug that caused those warnings to not persist its
state on pages with charts.

Test Plan: Removed

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3820
2023-03-15 10:50:14 +01:00
George Gevoian
8a6962d3e6 (core) Hide id column in Access Rules
Summary: The id column was a possible option for rulesets, which was a bug.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3817
2023-03-14 12:34:23 -04:00
Dmitry S
efd92c6c2a (core) Two fixes to tests affected by changes that came from grist-core.
Summary:
- For python2, skip some tests of renaming which produce different results
  because of an un-upgradable astroid version.
- Fix test affected by pyCall() having changed to async; avoid hanging timeout
  callback in case of error.

Test Plan: All test cases should now pass (with 4 getting skipped)

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3819
2023-03-14 11:57:01 -04:00
Dmitry S
86681de595 (core) Add overflowTooltip() tool, and use for long tables in widget picker, and long page names.
Summary:
Usage is simply to call `overflowTooltip()` with no arguments, as an argument
to an element whose text may overflow. On 'mouseenter', it'll check for
overflow and show the element's .textContent in a tooltip.

- Added for long table names in the widget picker (Add Page, Add Widget to Page).
- Added for long page names in the left-panel list of pages.

Test Plan: Added test cases for the new overflow tooltips

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3814
2023-03-13 21:47:35 -04:00
Paul Fitzpatrick
b28cbb378a
add a special key to allow translators to exclude language until ready (#461)
This adds a special key to be translated. While the key has not been
translated, the language should not be offered or used by default.
The key is not yet effective - that will be follow-up once the key
has propagated to weblate and existing translations have had time to
update.
2023-03-13 11:47:22 -04:00
Cyprien P
b312b3b08b (core) Fix css issue with tooltip in filter menu on safari
Summary:
Small glitch on safari: when we show behavioural tooltips the content
of the tooltip is first added to the parent of the target elem, then
we set tooltip's container positioning to absolute which normally causes recompute
of the layout. But in safari it doesn't, hence the button shows as if
the tooltip was still in there, as a sibling.

Diff fixes that issue by forcing positioning to absolute on the tooltip container.

{F68474}

Test Plan: Should not break anything.

Reviewers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3802
2023-03-13 15:39:41 +01:00
Paul Fitzpatrick
186263b4fb (core) updates from grist-core 2023-03-13 09:27:21 -04:00
Dmitry S
1274fe55fb (core) Fix linking of new records when attachment is the first thing added.
Summary:
Fixes a bug when in a linked widget, the automatic reference wasn't being set
for a new record if attachment is the first thing that gets added to the
record.

- Move handling of 'setCursorPos' pseudo-command to GristDoc to support cross-section switching (relevant when moving attachment into a cell of a non-active page widget)
- Modernize code for AttachmentsWidget slightly (better typings, css conventions)
- Change the fix in https://phab.getgrist.com/D3796 from using isolate to using different z-index values, to avoid a change in the look of the cursor on Attachment cells.

Test Plan: Added a test case for what's possible to test with webdriver.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3811
2023-03-09 08:54:36 -05:00
Paul Fitzpatrick
66643a5e6b
add a pyodide-based "sandbox" flavor (#437)
This adds a new `GRIST_SANDBOX_FLAVOR=pyodide` option where the
version of Python used for the data engine is wasm, and so can
be run by node like the rest of the back end. It still runs as
a separate process.

There are a few small version changes made to packages to avoid
various awkwardnesses present in the current versions. All existing
tests pass.

This is very experimental. To use, you'll need something with
a bash shell and make. First do:
```
cd sandbox/pyodide
make setup           # README.md and Makefile have details
cd ..
```

Then running Grist as:
```
GRIST_SANDBOX_FLAVOR=pyodide yarn start
```
should work. Adding a formula with content:
```
import sys; return sys.version
```
should return a different Python version than other sandboxes.

The motivation for this work is to have a form of sandboxing
that will work on Windows for Grist Electron (for Linux we have
gvisor/runsc, for Mac we have sandbox-exec, but I haven't found
anything comparable for Windows).

It also brings a back-end-free version of Grist a bit closer, for
use-cases where that would make sense - such as serving a report
(in the form of a Grist document) on a static site.
2023-03-06 16:56:25 -05:00
Paul Fitzpatrick
055522d374 (core) updates from grist-core 2023-03-06 08:21:09 -05:00