Summary:
- Webhooks form Triggers.ts should now use proxy if it's configured
- Proxy handling code separated to ProxyAgent.ts
- Tests for ProxyAgent
- Integration/API Tests for using Proxy in webhooks
- a bit of refactor - proxy test uses mostly the same codebase as DocApi.ts, but because last one if over 4000 lines long, I've put it into separated file, and extract some common parts (there is some duplicates tho)
- some cleanup in files that I've touched
Test Plan:
Manual test to check if proxy is used on the staging env
Automatic test checking if (fake) proxy was called
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3860
Summary:
- when grist table is exported, currency is check and introduced in cell format in the form of "[currency symbol] [value]" (for example: zł 10000, $ 5000) . It's not what some cultures should display currences, but it's close enought
- when no symbol is defined for the currency, currency 3 letters code is used instead
- when currency is unknown, we are falling back to "$"
Test Plan: - nbrowser test scenario added for that purpose, please check Currences.xlsx to see output format exported.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3886
Summary:
The URL /create-team-site on marketing site is unused, and no longer pointing to
anything functional.
The "efcr" product flavor has been defunct for a long time. Remove
references to it.
Test Plan: No tests should be affected
Reviewers: JakubSerafin
Reviewed By: JakubSerafin
Subscribers: JakubSerafin
Differential Revision: https://phab.getgrist.com/D3890
Summary:
When clicking the logo in the top-left corner, or finishing a tutorial, we
now direct users to the site they last visited, if possible. If unknown, a
new redirect endpoint, /welcome/home, is used instead, which directs users
to a sensible location based on the number of sites they have.
Test Plan: Browser tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3878
Summary:
- When importing into a Ref column, use lookupOne() formula for correct previews.
- When selecting columns to import into a Ref column, now a Numeric column like
'Order' will produce two options: "Order" and "Order (as row ID)".
- Fixes exports to correct the formatting of visible columns. This addresses multiple bugs:
1. Formatting wasn't used, e.g. a Ref showing a custom-formatted date was still presented as YYYY-MM-DD in CSVs.
2. Ref showing a Numeric column was formatted as if a row ID (e.g. `Table1[1.5]`), which is very wrong.
- If importing into a table that doesn't have a primary view, don't switch page after import.
Refactorings:
- Generalize GenImporterView to be usable in more cases; removed near-duplicated logic from node side
- Some other refactoring in importing code.
- Fix field/column option selection in ValueParser
- Add NUM() helper to turn integer-valued floats into ints, useful for "as row ID" lookups.
Test Plan: Added test cases for imports into reference columns, updated Exports test fixtures.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3875
Summary:
The conditions in the map/filter of attachments was faulty, causing
blank attachment types to slip through.
Test Plan: Tested manually.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3865
Summary:
Owners weren't able to access snapshots if access rules
that denied access to non-owners existed. The backend
was lowering snapshot document access to "viewers" as
part of implementing read-only behavior; this is now done
in the client, with document access for snapshots now
accurately reflecting the user's trunk access.
Additionally, sandboxes are no longer created for snapshots,
and background intervals aren't started for snapshots.
Test Plan: Browser test.
Reviewers: jarek, paulfitz
Reviewed By: jarek, paulfitz
Differential Revision: https://phab.getgrist.com/D3849
Summary:
Ports more test/browser tests from *.test.js (run using an old selenium setup) to *.ntest.js (run using newer setup).
Weird test failures happened due to a change in timing. Eventually tracked in down to billing changes in one test suite resulting in reloads in another test suite, since it turns out redis pub/sub channels are not scoped to the redis database specified in REDIS_URL, but are global:
https://redis.io/docs/manual/pubsub/#database--scoping.
Test Plan: Ported tests should run and pass
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3844
This closes a file left open during importing, not by the import itself, but by a SQLite integrity check. This was causing imports to fail on Windows (see https://github.com/gristlabs/grist-electron/issues/3)
Expose the log.add method, used in old Electron code. Without this change, using the Electron build of Grist fails if is configured for debug log output.
Summary:
Adds a new endpoint to update webhook.
Perform some refactoring to allow code reuse from endpoint allowing to _subscribe and _unsubscribe webhooks.
One aspect of webhook is that url are stored in the home db while the rest of the fields (tableRef, isReadyColRef, ...) are stored in sqlite. So care must be taken when updating fields, to properly rollback if anything should fail.
Follow up diff will bring UI to edit webhook list
Test Plan: Updated doc api server tests
Reviewers: jarek
Reviewed By: jarek
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3821
This checks for languages that have a special key translated.
Any that don't have the key translated, are not offered to the
user (unless GRIST_OFFER_ALL_LANGUAGES is set).
Co-authored-by: jarek <jaroslaw.sadzinski@gmail.com>
Summary:
The feature is behind a flag GRIST_FORMULA_ASSISTANT (must be "true"). But can be enabled in the
developer console by invoking GRIST_FORMULA_ASSISTANT.set(true).
Keys can be overriden in the document settings page.
Test Plan: For now just a stub test that checks if this feature is disabled by default.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3815
Summary:
This adds a nuanced redirecting endpoint. For example, on
docs.getgrist.com it does:
1) If logged in and no team site -> https://docs.getgrist.com/
2) If logged in and has team sites -> https://docs.getgrist.com/welcome/teams
3) If logged out but has a cookie -> /login, then 1 or 2
4) If entirely unknown -> /signup
Test Plan: added a test; tested behavior through logins manually
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3828
Summary:
Documents can now be flagged as tutorials, which causes them to display
Markdown-formatted slides from a special GristDocTutorial table. Tutorial
documents are forked on open, and remember the last slide a user was on.
They can be restarted too, which prepares a new fork of the tutorial.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3813
* place a new private ActiveDoc method in the expected order
This was tickling a lint failure on grist-core.
* reset an English translation that interferes with test currently
Summary:
A recent change perturbed some error handling when an ActiveDoc
is shutting down. It is important that errors get thrown when
attempting to replace a non-existent document. My bad in review
for not catching.
Test Plan: Snapshot test passes again
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: georgegevoian, cyprien
Differential Revision: https://phab.getgrist.com/D3824
Summary:
Porting script that run an evaluation against our formula dataset.
To test you need an openai key (see here: https://platform.openai.com/)
or hugging face (it should work as well), then checkout the branch and run
`OPENAI_API_KEY=<my_openai_api_key> node core/test/formula-dataset/runCompletion.js`
Test Plan:
Needs manually testing: so far there is no plan to make it part of CI.
The current score is somewhere around 34 successful prompts over a total of 47.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3816
Summary:
Warnings about deprecated shortcuts are no longer needed.
As a side effect it fixes a bug that caused those warnings to not persist its
state on pages with charts.
Test Plan: Removed
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3820
Summary:
- For python2, skip some tests of renaming which produce different results
because of an un-upgradable astroid version.
- Fix test affected by pyCall() having changed to async; avoid hanging timeout
callback in case of error.
Test Plan: All test cases should now pass (with 4 getting skipped)
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3819
This adds a new `GRIST_SANDBOX_FLAVOR=pyodide` option where the
version of Python used for the data engine is wasm, and so can
be run by node like the rest of the back end. It still runs as
a separate process.
There are a few small version changes made to packages to avoid
various awkwardnesses present in the current versions. All existing
tests pass.
This is very experimental. To use, you'll need something with
a bash shell and make. First do:
```
cd sandbox/pyodide
make setup # README.md and Makefile have details
cd ..
```
Then running Grist as:
```
GRIST_SANDBOX_FLAVOR=pyodide yarn start
```
should work. Adding a formula with content:
```
import sys; return sys.version
```
should return a different Python version than other sandboxes.
The motivation for this work is to have a form of sandboxing
that will work on Windows for Grist Electron (for Linux we have
gvisor/runsc, for Mac we have sandbox-exec, but I haven't found
anything comparable for Windows).
It also brings a back-end-free version of Grist a bit closer, for
use-cases where that would make sense - such as serving a report
(in the form of a Grist document) on a static site.
Summary:
This removes ormconfig.js from the saas build since it is no longer
needed (and has always been a pain point).
This expands some sendgrid logging to help figure out a problem.
Test Plan: existing tests should pass
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3808
Summary:
Adds information about forks to the home db. This will be used
later by the UI to list forks of documents.
Test Plan: Browser and server tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3772
* Replace `ormconfig.js` with a newer mechanism of configuring
TypeORM that can be included in the source code properly.
The path to `ormconfig.js` has always been awkward to handle,
and eliminating the file makes building different Grist setups
a bit simpler.
* Remove `electron` package. It is barely used, just for some old
remnants of an older attempt at electron packaging. It was used
for two types, which I left at `any` for now. More code pruning is
no doubt possible here, but I'd rather do it when Electron packaging
has solidified.
* Add a hook for replacing the login system, and for adding some
extra middleware the login system may need.
* Add support for some more possible locations of Python, which
arise when a standalone version of it is included in the Electron
package. This isn't very general purpose, just configurations
that I found useful.
* Support using grist-core within a yarn workspace - the only tweak
needed was webpack related.
* Allow an external ID to be optionally associated with documents.
Summary: This is a backend part for the formula AI.
Test Plan: New tests
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: cyprien
Differential Revision: https://phab.getgrist.com/D3786
Summary:
New language selector on the Account page for logged-in users.
New icon for switching language for an anonymous user.
For anonymous users, language is stored in a cookie grist_user_locale.
Language is stored in user settings for authenticated users and takes
precedence over what is stored in the cookie.
Test Plan: New tests
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3766
Summary:
- Use newer flag in .npmrc to avoid warnings
- Fix check in WidgetRepository, useful for development but was broken
- Fix macSandboxExec for Macs that require libRosettaRuntime
- Make sure row count in Raw Data listing is visible when it takes more space
Test Plan: Tested manually
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3759
Summary:
Implement a checkbox that grants owners full access to tables by default, when creating new table/column rules.
* Checkbox appears above default rules.
* When set, a rule giving owners full access will be inserted in any new rule set started for tables or columns.
* The checkbox can be expanded to allow customization of the rules.
https://gristlabs.getgrist.com/doc/check-ins/p/3#a1.s7.r2251.c19
Test Plan: added tests
Reviewers: jarek
Reviewed By: jarek
Subscribers: anaisconce
Differential Revision: https://phab.getgrist.com/D3756
Summary:
Diff removes view-as pill in the document breadcrumbs and add new view-as banner.
Note: Banners are still missing mechanism to handle several banners. As of now both doc-usage and view-as banners could show up at the same time.
Test Plan: Refactored existing test.
Reviewers: jarek
Reviewed By: jarek
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3732
Summary:
for users who don't automatically have deep rights
to the document, provide them with attachment metadata only
for rows they have access to. This is a little tricky to
do efficiently. We provide attachment metadata when an
individual table is fetched, rather than on initial document
load, so we don't block that load on a full document scan.
We provide attachment metadata to a client when we see that
we are shipping rows mentioning particular attachments,
without making any effort to keep track of the metadata they
already have.
Test Plan: updated tests
Reviewers: dsagal, jarek
Reviewed By: dsagal, jarek
Differential Revision: https://phab.getgrist.com/D3722
Summary:
This is a first pass at snapshot support using the MinIO client, suitable
for use against a MinIO server or other S3-compatible storage (including
the original AWS S3).
In Grist Labs monorepo tests, it is run against AWS S3. It can be manually
configured to run again a MinIO server, and these tests pass. There are no
core tests just yet.
Next step would be to move external storage tests to core, and configure
workflow to run tests against a transient MinIO server.
Test Plan: applied same tests as for Azure and S3 (via AWS client)
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3729
Summary:
When a column is transformed, it creates two helper columns whose values are always
broadcasted to all clients. Now when there are some ACL rules, we are going to prune
those columns from messages sent to other connected clients.
Test Plan: Added new tests
Reviewers: dsagal, paulfitz
Reviewed By: dsagal, paulfitz
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3728
Summary:
When initially added in the User Manager, the support user
(e.g. support@getgrist.com) was misleadingly being annotated as
a free collaborator. This fixes the annotation to be "Grist support"
instead.
Test Plan: Browser test.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3730
Summary:
- New /webhooks event that lists all webhooks in a document (available for owners),
- Monitoring webhooks usage and saving it in memory or Redis,
- Loosening _usubscribe API endpoint, so that the information returned from the /webhook endpoint is enough to unsubscribe,
- Owners can remove webhook without the unsubscribe key.
The endpoint lists all webhooks that are registered in a document, not just webhooks from a single table.
There are two status fields. First for the webhook, second for the last request attempt.
Webhook can have 5 statuses: 'idle', 'sending', 'retrying', 'postponed', 'error', which roughly describes what the
sendLoop is currently doing. The 'error' status describes a situation when all request attempts failed and the queue needs
to be drained, so some requests were dropped.
The last request status can only be: 'success', 'failure' or 'rejected'. Rejected means that the last batch was dropped because the
queue was too long.
Test Plan: New and updated tests
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3727
Summary:
Adds a new UI for writing access rule memos.
Migrates old memos (written as Python comments) to the new UI.
Test Plan: Browser and migration tests.
Reviewers: jarek, dsagal
Reviewed By: jarek
Subscribers: dsagal, paulfitz
Differential Revision: https://phab.getgrist.com/D3726
Summary:
When access rules refer to tables and/or columns that no longer exist, offer convenient buttons to remove these rules.
It could alternatively be useful to generate errors when deleting tables or columns that are mentioned in access rules, and refuse to do so unless the access rules are updated first.
Test Plan: added and updated tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3718
Summary:
All users are treated as viewers for snapshot documents, since they
cannot reasonably be edited. This is a bit dubious and confusing now
that granular access rules exist. More urgently, owners of the trunk
document may be locked out of downloading a snapshot, and so also
locked out of replacing the trunk with a snapshot. This diff
explicitly gives an owner of a trunk document the right to download
its snapshots.
Test Plan: updated a snapshots test to something that fails without this diff
Reviewers: dsagal, georgegevoian
Reviewed By: dsagal, georgegevoian
Subscribers: jarek, dsagal
Differential Revision: https://phab.getgrist.com/D3721
Summary:
Creating an API endpoint to cancel any queued webhook messages from
a document.
Test Plan: Updated
Reviewers: paulfitz, georgegevoian
Reviewed By: paulfitz, georgegevoian
Differential Revision: https://phab.getgrist.com/D3713
Summary:
Writing results of the undo action to a database when the undo was caused by rejecting due to ACL checks.
This ensures that DB and sanbox are in sync in case of non-deterministic formulas.
Test Plan: Updated
Reviewers: georgegevoian, dsagal
Reviewed By: georgegevoian, dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3695
Summary: Document usage is now available for owners regardless ACL rules
Test Plan: Updated
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3717
Summary:
Rows in the _grist_Attachments table have a special lifecycle,
being created by a special method, and deleted via a special
process. All other modifications are now rejected, for simplicity.
Test Plan: added test
Reviewers: dsagal, jarek
Reviewed By: dsagal, jarek
Differential Revision: https://phab.getgrist.com/D3712
Summary:
This brings the treatment of metadata updates being broadcast to a user
into line with how it is treated when they first open the document.
Specifically, this fixes a bug where, for a document with a default access
rule denying access to everything, a user would not receive any
metadata updates.
Test Plan: added test; existing tests pass
Reviewers: jarek, dsagal
Reviewed By: jarek, dsagal
Differential Revision: https://phab.getgrist.com/D3711
Summary:
The sort and filter UI now has a more unified UI, with similar
capabilities that are accessible from different parts of Grist.
It's now also possible to pin individual filters to the filter bar,
which replaces the old toggle for showing all filters in the
filter bar.
Test Plan: Various tests (browser, migration, project).
Reviewers: jarek, dsagal
Reviewed By: jarek, dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3669
Summary:
View as feature uses example.com emails for simulated users. This can break
when such a user already exists in the home db. Here we pretend that these users
don't exist during ACL checks.
Test Plan: Updated and existing
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3705
Summary:
Snapshots can now only be listed for users with non-nuanced access
(no access rules, or owners on docs with rules). If a snapshot URL
leaks, or is shared by a user who can list snapshots, that URL
behaves as before -- it gives access to the snapshot according
to access rules in that snapshot.
Test Plan: added test
Reviewers: georgegevoian, dsagal
Reviewed By: georgegevoian, dsagal
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3698
Summary:
Attachments are a special case for granular access control. A user is now allowed to read a given attachment if they have read access to a cell containing its id. So when a user writes to a cell in an attachment column, it is important that they can only write the ids of cells to which they have access. This diff allows a user to add an attachment id in a cell if:
* The user already has access to that a attachment via some existing cell, or
* The user recently updated the attachment, or
* The attachment change is from an undo/redo of a previous action attributed to that user
Test Plan: Updated tests
Reviewers: georgegevoian, dsagal
Reviewed By: georgegevoian, dsagal
Differential Revision: https://phab.getgrist.com/D3681
Summary:
The /replace endpoint was built with home-level access control in mind. Updates needed:
* Only an owner can now replace a document. Only owners are permitted to change granular access rules, and a document replacement could change granular access rules.
* For the document being substituted in: the user must have complete access to view all material within it.
Test Plan: extended test
Reviewers: georgegevoian, dsagal
Reviewed By: georgegevoian, dsagal
Differential Revision: https://phab.getgrist.com/D3694
Summary:
Allow requests from untrusted origins instead of returning an error, but don't allow credentials (Cookie header) or API keys (Authorization header).
Allow setting the header `Content-type: application/json` as an alternative to `X-Requested-With: XMLHttpRequest` to make it easier for clients to make POST/PUT/PATCH/DELETE requests without authentication.
Discussion: https://grist.slack.com/archives/C0234CPPXPA/p1666355281535479
Test Plan: Added and updated DocApi tests. Tested manually how this affects requests made from a browser.
Reviewers: paulfitz, dsagal
Reviewed By: paulfitz, dsagal
Differential Revision: https://phab.getgrist.com/D3678
Summary:
New shortcuts for removing and adding rows.
For adding a row we now have Mod+(Shift)+Enter
For removing rows we now have Mod+Delete/Mod+Backspace
Before removing rows, the user is prompted to confirm, this prompt
can be dismissed and this setting can be remembered. User needs
to confirm only when using shortcut.
Old shortcuts are still active and shows information about this change.
This information is shown only once, after this shortcuts have default
behavior (zooming).
New users don't see this explanation.
Test Plan: Updated
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3655
Summary:
First iteration for comments system for Grist.
- Comments are stored in a generic metatable `_grist_Cells`
- Each comment is connected to a particular cell (hence the generic name of the table)
- Access level works naturally for records stored in this table
-- User can add/read comments for cells he can see
-- User can't update/remove comments that he doesn't own, but he can delete them by removing cells (rows/columns)
-- Anonymous users can't see comments at all.
- Each comment can have replies (but replies can't have more replies)
Comments are hidden by default, they can be enabled by COMMENTS=true env variable.
Some things for follow-up
- Avatars, currently the user's profile image is not shown or retrieved from the server
- Virtual rendering for comments list in creator panel. Currently, there is a limit of 200 comments.
Test Plan: New and existing tests
Reviewers: georgegevoian, paulfitz
Reviewed By: georgegevoian
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3509
Summary:
This makes two small tweaks based on a user's questions about sharing
sites publicly for a self-managed installation:
* The support user `support@getgrist.com` is made configurable with
`GRIST_SUPPORT_EMAIL`. This came up because only the support user
can share material with the special "everyone" user. This restriction
was added to avoid spam.
* Regardless of public sharing settings, for our SaaS we had
decided not to list public sites to anonymous users. That is
somewhat a question of taste, so a `GRIST_LIST_PUBLIC_SITES` flag
is added to override this choice.
Public sharing isn't in a well polished state, and this diff doesn't
advance that, in fact it adds a new wrinkle :-/
Test Plan: existing tests pass; manual testing
Reviewers: jarek
Reviewed By: jarek
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3663
Summary:
This makes the version shown when hovering on the Grist icon equal
the version set in package.json, for a grist-core build. Previously
the number shown was a hard-coded placeholder.
The Grist SaaS build has some build machinery dealing with the
version number that should be unaffected by this change for now.
Test Plan: tested manually with build_core.sh
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3659
Summary:
There is a new column in users table called ref (user reference).
It holds user's unique reference number that can be used for features
that require some kind of ownership logic (like comments).
Test Plan: Updated tests
Reviewers: georgegevoian, paulfitz
Reviewed By: georgegevoian, paulfitz
Differential Revision: https://phab.getgrist.com/D3641
Summary:
Adds a "Duplicate Table" menu option to the tables listed on
the Raw Data page. Clicking it opens a dialog that allows you to
make a copy of the table (with or without its data).
Test Plan: Python, server, and browser tests.
Reviewers: jarek, paulfitz
Reviewed By: jarek, paulfitz
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3619
Summary: Also clarifies that only lowercase letters are accepted.
Test Plan: manual
Reviewers: jarek
Reviewed By: jarek
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3647
Removes a comment now that `gvisor` works fine with grist-core, and is packaged in the docker image. Reorders possible sandbox flavors to de-emphasize `pynbox` since it isn't packaged in the docker image.
Summary:
Ref columns weren't filtred on section export.
Filters were applied to a display helper columns instead
of the actual columns.
Test Plan: Updated tests
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3644
Summary:
This diff adds a preview of the value of certain autocomplete suggestions, especially of the form `$foo.bar` or `user.email`. The main initial motivation was to show the difference between `$Ref` and `$Ref.DisplayCol`, but the feature is more general.
The client now sends the row ID of the row being edited (along with the table and column IDs which were already sent) to the server to fetch autocomplete suggestions. The returned suggestions are now tuples `(suggestion, example_value)` where `example_value` is a string or null. The example value is simply obtained by evaluating (in a controlled way) the suggestion in the context of the given record and the current user. The string representation is similar to the standard `repr` but dates and datetimes are formatted, and the whole thing is truncated for efficiency.
The example values are shown in the autocomplete popup separated from the actual suggestion by a number of spaces calculated to:
1. Clearly separate the suggestion from the values
2. Left-align the example values in most cases
3. Avoid having so much space such that connecting suggestions and values becomes visually difficult.
The tokenization of the row is then tweaked to show the example in light grey to deemphasise it.
Main discussion where the above was decided: https://grist.slack.com/archives/CDHABLZJT/p1661795588100009
The diff also includes various other small improvements and fixes:
- The autocomplete popup is much wider to make room for long suggestions, particularly lookups, as pointed out in https://phab.getgrist.com/D3580#inline-41007. The wide popup is the reason a fancy solution was needed to position the example values. I didn't see a way to dynamically resize the popup based on suggestions, and it didn't seem like a good idea to try.
- The `grist` and `python` labels previously shown on the right are removed. They were not helpful (https://grist.slack.com/archives/CDHABLZJT/p1659697086155179) and would get in the way of the example values.
- Fixed a bug in our custom tokenization that caused function arguments to be weirdly truncated in the middle: https://grist.slack.com/archives/CDHABLZJT/p1661956353699169?thread_ts=1661953258.342739&cid=CDHABLZJT and https://grist.slack.com/archives/C069RUP71/p1659696778991339
- Hide suggestions involving helper columns like `$gristHelper_Display` or `Table.lookupRecords(gristHelper_Display=` (https://grist.slack.com/archives/CDHABLZJT/p1661953258342739). The former has been around for a while and seems to be a mistake. The fix is simply to use `is_visible_column` instead of `is_user_column`. Since the latter is not used anywhere else, and using it in the first place seems like a mistake more than anything else, I've also removed the function to prevent similar mistakes in the future.
- Don't suggest private columns as lookup arguments: https://grist.slack.com/archives/CDHABLZJT/p1662133416652499?thread_ts=1661795588.100009&cid=CDHABLZJT
- Only fetch fresh suggestions specifically after typing `lookupRecords(` or `lookupOne(` rather than just `(`, as this would needlessly hide function suggestions which could still be useful to see the arguments. However this only makes a difference when there are still multiple matching suggestions, otherwise Ace hides them anyway.
Test Plan: Extended and updated several Python and browser tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3611
Summary:
This diff adds a new `BulkAddOrUpdateRecord` user action which is what is sounds like:
- A bulk version of the existing `AddOrUpdateRecord` action.
- Much more efficient for operating on many records than applying many individual actions.
- Column values are specified as maps from `colId` to arrays of values as usual.
- Produces bulk versions of `AddRecord` and `UpdateRecord` actions instead of many individual actions.
Examples of users wanting to use something like `AddOrUpdateRecord` with large numbers of records:
- https://grist.slack.com/archives/C0234CPPXPA/p1651789710290879
- https://grist.slack.com/archives/C0234CPPXPA/p1660743493480119
- https://grist.slack.com/archives/C0234CPPXPA/p1660333148491559
- https://grist.slack.com/archives/C0234CPPXPA/p1663069291726159
I tested what made many `AddOrUpdateRecord` actions slow in the first place. It was almost entirely due to producing many individual `AddRecord` user actions. About half of that time was for processing the resulting `AddRecord` doc actions. Lookups and updates were not a problem. With these changes, the slowness is gone.
The Python user action implementation is more complex but there are no surprises. The JS API now groups `records` based on the keys of `require` and `fields` so that `BulkAddOrUpdateRecord` can be applied to each group.
Test Plan: Update and extend Python and DocApi tests.
Reviewers: jarek, paulfitz
Reviewed By: jarek, paulfitz
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3642
Summary: Extend the way ACL resources are read in the server so that if a rule applies to a specific column then that rule also applies to helper columns belonging to that column, as well as helper columns belonging to fields which display that column. This is particularly intended for display columns of reference columns, but it also applies to conditional formatting rule columns.
Test Plan: Added a server test
Reviewers: paulfitz, jarek
Reviewed By: paulfitz, jarek
Differential Revision: https://phab.getgrist.com/D3628
Summary:
This endpoint has started to fail when called between a pair
of doc workers. The simplest fix is to simply remove it, it serves no
purpose.
Test Plan: added basic deployment test
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3636
XLSX export of active view / table
Co-authored-by: Louis Delbosc <louis.delbosc.prestataire@anct.gouv.fr>
Co-authored-by: Vincent Viers <vincent.viers@beta.gouv.fr>
Summary:
Adds initial implementation of dark mode. Preferences for dark mode are
available on the account settings page. Dark mode is currently a beta feature
as there are still some small bugs to squash and a few remaining UI elements
to style.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Subscribers: paulfitz, jarek
Differential Revision: https://phab.getgrist.com/D3587
Summary:
upgrade typeorm version, so Grist can run against newer versions of postgres.
Dusted off some old benchmarking code to verify that important queries don't get slower. They don't appear to, unlike for some intermediate versions of typeorm I tried in the past.
Most of the changes are because `findOne` changed how it interprets its arguments, and the value it returns when nothing is found. For the return value, I stuck with limiting its impact by emulating old behavior (returning undefined rather than null) rather than propagating the change out to parts of the code unrelated to the database.
Test Plan: existing tests pass; manual testing with postgres 10 and 14
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3613
Summary:
- The issue manifested as error "Cannot read property '0' of undefined" in some
cases, and as "Blocked by table read access rules" in others (instead of
limiting output to what's not blocked)
- Goes deeper: exports weren't respecting metadata censoring.
- The fix changes exports to use censored metadata, which addresses both errors above.
- Includes an improvement to column ordering in XLSX exports.
Test Plan: Add a server test for CSV and XLSX exports with access rules
Reviewers: paulfitz, georgegevoian
Reviewed By: paulfitz, georgegevoian
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3615
Summary:
Python 2 only needs to be supported for the sake of old documents and formulas. This doesn't apply to the separate sandboxes that parse files for imports. Using Python 3 only allows using newer libraries and library versions. In particular, the latest version of openpyxl doesn't support Python 2. This will also make it easier to make other similar changes in the future, such as replacing messytables with a modern library. See https://grist.slack.com/archives/C0234CPPXPA/p1661261829343999?thread_ts=1661260442.837959&cid=C0234CPPXPA
The latest openpyxl is better at handling a particular edge case with broken dates in Excel, but still doesn't quite do what we want, so we monkeypatch it. Discussion: https://grist.slack.com/archives/C02EGJ1FUCV/p1661440851911869?thread_ts=1661154219.515549&cid=C02EGJ1FUCV
Setting `preferredPythonVersion` to '3' in SafePythonComponent ensures that JS always creates import sandboxes that use Python 3. Within Python, a module used by all imports will raise an error in Python 2. Python unit tests of imports are now only run in Python 3, using the `load_tests` protocol of `unittest`.
Test Plan: Mostly existing tests. Added another strange date to the Excel fixture.
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3606
Summary:
Two summary table widgets that share a source table and have compatible groupby columns can be filter linked. This diff fixes a bug where this linking is broken when the source table is hidden by access rules. The source table data isn't needed for the linking, but its metadata is, and that metadata is censored by GranularAccess. To deal with this:
- `LinkConfig._assertValid` allows blank `tableId`s specifically for linking two summary tables.
- `LinkingState.filterColValues` gets the `colId`s of groupby columns from the summary table columns rather than the source table.
A closely related problem is that the titles of summary tables are incomplete when the source table is hidden, e.g. they just say `[by A]` instead of `Table1 [by A]`. To fix this, the raw view sections of source tables are 'uncensored' in GranularAccess.
Initially I also planned to uncensor the tableId of the source table, which seemed like a better and more general fix for the blank tableId problem. But several parts of client code use blank tableIds to know that a table should be hidden, so they were left as is.
Test Plan: Added an nbrowser test for summary table linking, and a server test for uncensoring the raw view section in GranularAccess.
Reviewers: georgegevoian, paulfitz
Reviewed By: georgegevoian, paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3608
Summary:
Fixes some bugs involving intervals, and updates RandomizedTimer to support both fixed and
randomized delays, and to better handle async callbacks.
* Fixed a bug where Throttle would queue up many pidusage calls due to the use of
setInterval, and the async nature of the calls.
* Fixed a but where RandomizedTimer (now just Interval) would not be disabled in
ActiveDoc on doc shutdown if initialization had not yet settled.
Test Plan: Tested manually.
Reviewers: jarek, dsagal
Reviewed By: jarek, dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3604
Summary:
getDataSize() call can be expensive and involve lots of disk reading. We can
avoid doing it repeatedly when the document isn't actually changing.
Test Plan: Should have no change in behavior except for timings.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3605
