Commit Graph

1936 Commits

Author SHA1 Message Date
Alex Hall
6ac0bc3bbb (core) Implement exported functions without relying on ActiveDoc.docData
Summary:
For grist-static, we want to the data engine to be able to call external/exported JS functions directly,
rather than via the node 'server' living in another thread which requires synchronous communication hackery.

As a step in that direction, this diff changes the exported functions that we care about (guessColInfo and convertFromColumn)
to just using the top-level functions instead of relying on fields in ActiveDoc, namely docData.

For guessColInfo, this is done by directly passing the small amount of metadata that was previously retrieved from the DocData.

For convertFromColumn, disentangling DocData is a lot more complicated, so instead we construct a fresh DocData object using
the required metadata tables which are now passed in by the data engine.

Test Plan: Existing tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3913
2023-06-07 22:30:01 +02:00
George Gevoian
10f5f0cb37 (core) Add optional telemetry to grist-core
Summary:
Adds support for optional telemetry to grist-core.

A new environment variable, GRIST_TELEMETRY_LEVEL, controls the level of telemetry collected.

Test Plan: Server and unit tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal, anaisconce

Differential Revision: https://phab.getgrist.com/D3880
2023-06-07 12:00:51 -04:00
George Gevoian
0d082c9cfc (core) Wrap buttons in filter bar
Summary:
Filter bar buttons used to overflow, which required horizontal scrolling. Buttons
now wrap instead.

Test Plan: Tested manually.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3901
2023-06-06 11:34:49 -04:00
Jakub Serafin
90e902c10f (core) sanitizing redis errors
Summary:
sanitazing errors output in webhooks to protect users data (not show them in logs and other places).
Because redis is returing whole payload when error occur, best approach is to hijack exception as close to redis operation as posible and sanitize the data.
We need to know data structure do do this corretly tho. Currently I decided to just censore everything that has "payload" key.

Test Plan: Because logs that need to be sanitized come from redis, to be valid tested we should force redis to crash. It's hard to do in our integration test setup. In this moment, unit test is all we got.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3905
2023-06-06 10:51:17 +02:00
github-actions[bot]
805485ed39
automated update to translation keys (#526)
Co-authored-by: Paul's Grist Bot <paul+bot@getgrist.com>
2023-06-05 08:53:49 -04:00
Paul Fitzpatrick
0fb5092e33 (core) updates from grist-core 2023-06-05 08:33:28 -04:00
Rémi LEBLOND
b9083d9a6d
Translated using Weblate (French)
Currently translated at 100.0% (803 of 803 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/fr/
2023-06-04 23:49:11 +02:00
Paul Fitzpatrick
cd51eee160
v1.1.1 2023-06-02 14:15:34 -04:00
Jarosław Sadziński
da323fb741 (core) Floating formula editor
Summary:
Adding a way to detach an editor. Initially only implemented for the formula editor, includes redesign for the AI part.
- Initially, the detached editor is tight with the formula assistant and both are behind GRIST_FORMULA_ASSISTANT flag, but this can be relaxed
later on, as the detached editor can be used on its own.

- Detached editor is only supported in regular fields and on the creator panel. It is not supported yet for conditional styles, due to preview limitations.
- Old code for the assistant was removed completely, as it was only a temporary solution, but the AI conversation part was copied to the new one.
- Prompting was not modified in this diff, it will be included in the follow-up with more test cases.

Test Plan: Added only new tests; existing tests should pass.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Differential Revision: https://phab.getgrist.com/D3863
2023-06-02 17:59:22 +02:00
Dmitry S
e10067ff78 (core) Rearrange ExportXLSX code and fix ExportsAccessRules test that became flaky
Summary:
- Move makeXLSX* methods to workerExporter file to avoid the risk of creating a piscina worker pool from a thread.
- Increase request timeout in ExportsAccessRules test that started failing occasionally

Test Plan: Test should succeed more reliably

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3910
2023-06-02 10:23:33 -04:00
jarek
9edbf9f415
Setting main grist-widget repository as a default (#524)
* Setting main grist-widget repository as a default to GRIST_WIDGET_LIST_URL

* Putting widget repository URL in commonURLs
2023-06-02 10:17:09 +02:00
Dmitry S
d191859be7 (core) For exporting XLSX, do it memory-efficiently in a worker thread.
Summary:
- Excel exports were awfully memory-inefficient, causing occasional docWorker
  crashes. The fix is to use the "streaming writer" option of ExcelJS
  https://github.com/exceljs/exceljs#streaming-xlsx-writercontents. (Empirically
  on one example, max memory went down from 3G to 100M)
- It's also CPU intensive and synchronous, and can block node for tens of
  seconds. The fix is to use a worker-thread. This diff uses "piscina" library
  for a pool of threads.
- Additionally, adds ProcessMonitor that logs memory and cpu usage,
  particularly when those change significantly.
- Also introduces request cancellation, so that a long download cancelled by
  the user will cancel the work being done in the worker thread.

Test Plan:
Updated previous export tests; memory and CPU performance tested
manually by watching output of ProcessMonitor.

Difference visible in these log excerpts:

Before (total time to serve request 22 sec):
```
Telemetry processMonitor heapUsedMB=2187, heapTotalMB=2234, cpuAverage=1.13, intervalMs=17911
Telemetry processMonitor heapUsedMB=2188, heapTotalMB=2234, cpuAverage=0.66, intervalMs=5005
Telemetry processMonitor heapUsedMB=2188, heapTotalMB=2234, cpuAverage=0, intervalMs=5005
Telemetry processMonitor heapUsedMB=71, heapTotalMB=75, cpuAverage=0.13, intervalMs=5002
```
After (total time to server request 18 sec):
```
Telemetry processMonitor heapUsedMB=109, heapTotalMB=144, cpuAverage=0.5, intervalMs=5001
Telemetry processMonitor heapUsedMB=109, heapTotalMB=144, cpuAverage=1.39, intervalMs=5002
Telemetry processMonitor heapUsedMB=94, heapTotalMB=131, cpuAverage=1.13, intervalMs=5000
Telemetry processMonitor heapUsedMB=94, heapTotalMB=131, cpuAverage=1.35, intervalMs=5001
```
Note in "Before" that heapTotalMB goes up to 2GB in the first case, and "intervalMs" of 17 seconds indicates that node was unresponsive for that long. In the second case, heapTotalMB stays low, and the main thread remains responsive the whole time.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3906
2023-06-01 12:06:48 -04:00
Dmitry S
6b338256e0 (core) On mobile, only show a warning for older and unknown browsers rather than for all
Test Plan:
Tested listed versions against docs.getgrist.com using browserstack, and tested
with an iPhone and local server that new settings are noticed on mobile.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3909
2023-06-01 11:15:21 -04:00
Jarosław Sadziński
c592691e31 (core) Fixing DELETE and BACKSPACE keys on ChoiceList and RefList editor.
Summary:
Choice/Reference List editor wasn't clearing itself when it received an empty string. It led
to a bug on the Card widget where pressing those keys resulted in the same behavior as
pressing Enter - it just opened the editor.
Grid view has it's own implementation for those keys, so it wasn't affected.

Test Plan: Added new test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3908
2023-06-01 16:53:38 +02:00
Paul Fitzpatrick
dad41b2567 (core) updates from grist-core 2023-05-30 08:32:34 -04:00
Camille L
af210002b8
Translated using Weblate (French)
Currently translated at 98.0% (787 of 803 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/fr/
2023-05-26 10:51:46 +02:00
Paul Fitzpatrick
33d24e50f6
use gu.toggleSidePanel to open side panel (#523)
This hopefully fixes a test that was occasionally failing.
2023-05-24 16:29:14 -04:00
George Gevoian
ff03d32688 (core) Set DateTime timezone during xlsx import
Summary:
DateTime columns had a blank timezone after xlsx imports because the
timezone was not included in the column type. We now append the
document's timezone to the type of all imported DateTime columns.

Test Plan: Server test.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3896
2023-05-24 11:39:49 -04:00
George Gevoian
d5b8240c07 (core) Fix snapshot migrations
Summary:
Migrations were failing in snapshots due to the sandbox no longer
being started in snapshots. We now start up an instance of the
sandbox whenever there are migrations to run, and immediately shut
it down on completion.

Test Plan: Server test.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3898
2023-05-23 15:50:15 -04:00
Paul Fitzpatrick
7be0ee289d
support other SQLite wrappers, and various hooks needed by grist-static (#516) 2023-05-23 15:17:28 -04:00
George Gevoian
bd474a382f
Fix Features and DocTutorial tests (#520) 2023-05-23 14:34:48 -04:00
Paul Fitzpatrick
3f3a0d3aa1 (core) support a wildcard option for ALLOWED_WEBHOOK_DOMAINS
Summary:
Now that webhook payload delivery can be done using a proxy,
it may be desirable to no longer require a set of `ALLOWED_WEBHOOK_DOMAINS`.
This diff allows this variable to be set to `*`. With this setting,
any domain, and both `http` and `https` protocols will now be accepted.

Another possibility would be to default to unchecked
behavior if `ALLOWED_WEBHOOK_DOMAINS` is not set. But this would
introduce a new kind of vulnerability to unconfigured Grist
installations.

Test Plan: switched a test from naming a domain to using `*`

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3903
2023-05-23 13:40:52 -04:00
Paul Fitzpatrick
4cc19ff748 (core) updates from grist-core 2023-05-23 09:10:34 -04:00
Riccardo Polignieri
101b4ad1eb
Translated using Weblate (Italian)
Currently translated at 100.0% (803 of 803 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/it/
2023-05-23 11:50:21 +02:00
Владимир В
f280877413
Translated using Weblate (Russian)
Currently translated at 99.6% (800 of 803 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/ru/
2023-05-23 11:50:21 +02:00
Camille L
c50efe6e30
Translated using Weblate (French)
Currently translated at 98.0% (787 of 803 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/fr/
2023-05-23 11:50:21 +02:00
Dmitry S
d4bc6246f1 (core) Fix breakage on Firefox iOS
Summary:
Grist recently stopped working on Firefox on iOS. The cause turns out an uncaught error, which is reported as an unhelpful "Script Error", but the act of reporting it causes additional errors, leading to an infinite loop and an unusable browser tab.

Firefox-iOS is to blame, but a workaround is preventing a flood of "Script Error" messages. Specifically, we report only the first of these, and only to the server, suppressing the user-visible toast.

Test Plan: Tested manually on Firefox on iOS. Added a test case, and improve other tests of uncaught errors.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3902
2023-05-22 16:22:41 -04:00
George Gevoian
f18bb3e39d (core) Add GRIST_UI_FEATURES env variable
Summary:
Tutorials are now hidden by default in grist-core and grist-ee, and can
be re-enabled via a new env variable, GRIST_UI_FEATURES, which accepts
a comma-separated list of UI features to enable.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3885
2023-05-22 16:05:51 -04:00
wunter8
e7933d7c59
(core) Remove bool filtered rows immediately on change (#517)
* (core) Remove bool filtered rows immediately on change
2023-05-20 19:55:36 -04:00
George Gevoian
1e873b4203 (core) Tweak telemetry
Summary: Adjusts the level of telemetry collected from Grist SaaS.

Test Plan: Tested manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3899
2023-05-19 13:06:15 -04:00
Paul Janzen
62b8f54672
Translated using Weblate (Spanish)
Currently translated at 100.0% (803 of 803 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/es/
2023-05-18 18:49:16 +02:00
Paul Janzen
417e500d3a
Translated using Weblate (German)
Currently translated at 100.0% (803 of 803 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/de/
2023-05-17 18:14:07 +02:00
Paul Janzen
47e6d71f8e
Translated using Weblate (Spanish)
Currently translated at 100.0% (803 of 803 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/es/
2023-05-17 18:14:07 +02:00
gallegonovato
e2ad69828b
Translated using Weblate (Spanish)
Currently translated at 100.0% (803 of 803 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/es/
2023-05-17 18:14:06 +02:00
Paul Janzen
57beb25a5d
Translated using Weblate (Portuguese (Brazil))
Currently translated at 100.0% (803 of 803 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/pt_BR/
2023-05-17 18:14:06 +02:00
Dmitry S
be5cb9124a (core) Add logging of errors whenever ProxyAgent is used, and a test for it.
Summary:
Also:
- Move ProxyAgent to from app/server/utils to app/server/lib, which is
  the more usual place for such classes.
- Refactor a helper (delayAbort) that node was reporting a leak in.

Test Plan: Added a test case, and tested manually.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Subscribers: JakubSerafin, paulfitz

Differential Revision: https://phab.getgrist.com/D3897
2023-05-17 10:21:53 -04:00
Camille L
5545f97b8d
Translated using Weblate (French)
Currently translated at 99.8% (786 of 787 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/fr/
2023-05-17 10:52:01 +02:00
github-actions[bot]
3e0045db45
automated update to translation keys (#518)
Co-authored-by: Paul's Grist Bot <paul+bot@getgrist.com>
2023-05-15 12:27:27 -04:00
Paul Fitzpatrick
84854b7cfa (core) updates from grist-core 2023-05-15 12:01:19 -04:00
Paul Fitzpatrick
b58929f095 (core) reconcile webhook and widget description migrations
Summary:
Due to a mishap, two distinct migrations with the same migration
number were introduced into Grist. This diff reconciles them as
best we can, by adding another migration to make sure both desired
changes have run (and running them if not).

Test Plan:
updated a test; checked manually that documents
with different 38 migrations are handled as expected.

Reviewers: georgegevoian, jarek

Reviewed By: georgegevoian, jarek

Differential Revision: https://phab.getgrist.com/D3895
2023-05-15 11:56:15 -04:00
Riccardo Polignieri
5c36ae6988
Translated using Weblate (Italian)
Currently translated at 100.0% (787 of 787 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/it/
2023-05-14 18:49:32 +02:00
Paul Janzen
cf8558e6b4
Translated using Weblate (German)
Currently translated at 100.0% (787 of 787 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/de/
2023-05-14 18:49:32 +02:00
gallegonovato
a6409776e7
Translated using Weblate (Spanish)
Currently translated at 100.0% (787 of 787 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/es/
2023-05-14 18:49:31 +02:00
Paul Janzen
a1fbb66304
Translated using Weblate (Portuguese (Brazil))
Currently translated at 100.0% (787 of 787 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/pt_BR/
2023-05-14 18:49:31 +02:00
Paul Janzen
01931bfe01
Translated using Weblate (German)
Currently translated at 100.0% (786 of 786 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/de/
2023-05-13 03:51:01 +02:00
Paul Janzen
e0e29eccb1
Translated using Weblate (Spanish)
Currently translated at 100.0% (786 of 786 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/es/
2023-05-13 03:51:01 +02:00
Paul Janzen
b611526fde
Translated using Weblate (Portuguese (Brazil))
Currently translated at 100.0% (786 of 786 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/pt_BR/
2023-05-13 03:51:01 +02:00
softwareguru90
72730461eb
Track saved version per hour, day, week, month, year, and number of times a version with parameter (#509)
Determining the number of snapshots to be kept with a parameter
2023-05-12 11:38:29 -04:00
CamilleLegeron
c16204f8ad
feature widget description (#483)
Add description to widget title popup and right panel
2023-05-12 09:08:28 -04:00
Camille L
a019c406ab
Translated using Weblate (French)
Currently translated at 98.0% (769 of 784 strings)

Translation: Grist/client
Translate-URL: https://hosted.weblate.org/projects/grist/client/fr/
2023-05-11 19:47:52 +02:00