Summary:
- `lookupRecords()` now allows efficient search in sorted results, with
the syntax `lookupRecords(..., order_by="-Date").find.le($Date)`. This will find the record with the nearest date that's <= `$Date`.
- The `find.*` methods are `le`, `lt`, `ge`, `gt`, and `eq`. All have O(log N) performance.
- `PREVIOUS(rec, group_by=..., order_by=...)` finds the previous record to rec, according to `group_by` / `order_by`, in amortized O(log N) time. For example, `PREVIOUS(rec, group_by="Account", order_by="Date")`.
- `PREVIOUS(rec, order_by=None)` finds the previous record in the full table, sorted by the `manualSort` column, to match the order visible in the unsorted table.
- `NEXT(...)` is just like `PREVIOUS(...)` but finds the next record.
- `RANK(rec, group_by=..., order_by=..., order="asc")` returns the rank of the record within the group, starting with 1. Order can be `"asc"` (default) or `"desc"`.
- The `order_by` argument in `lookupRecords`, and the new functions now supports tuples, as well as the "-" prefix to reverse order, e.g. `("Category", "-Date")`.
- New functions are only available in Python3, for a minor reason (to support keyword-only arguments for `group_by` and `order_by`) and also as a nudge to Python2 users to update.
- Includes fixes for several situations related to lookups that used to cause quadratic complexity.
Test Plan:
- New performance check that sorted lookups don't add quadratic complexity.
- Tests added for lookup find.* methods, and for PREVIOUS/NEXT/RANK.
- Tests added that renaming columns updates `order_by` and `group_by` arguments, and attributes on results (e.g. `PREVIOUS(...).ColId`) appropriately.
- Python3 tests can now produce verbose output when VERBOSE=1 and -v are given.
Reviewers: jarek, georgegevoian
Reviewed By: jarek, georgegevoian
Subscribers: paulfitz, jarek
Differential Revision: https://phab.getgrist.com/D4265
Automatically update dropdown condition formulas on Ref, RefList, Choice and ChoiceList columns when a column referred to has been renamed.
Also fixed column references in ACL formulas using the "$" notation not being properly renamed.
De-escalates to a normal user when the docker image is run as root.
Allows GRIST_DOCKER_USER and GRIST_DOCKER_GROUP to be passed to override the default de-escalation behaviour.
Backwards compatible with previous root installations.
--------
This change adds a new docker_entrypoint.sh, which when run as root de-escalates to the provided user, defaulting to grist:grist. This is similar to the approach used by the official postgres docker image.
To achieve backwards compatibility, it changes ownership of any files in `/persist` to the user it's given at runtime. Since the docker container is typically run as root, this should always work.
If the container is run as a standard user from the very start:
* It's the admin's responsibility to ensure `/persist` is writable by that user.
* `/grist` remains owned by root and is read-only.
Summary:
- Fixing port allocation in TestServer
- Extending logging in the Billing test
- Fixing negative rowIds support for add/remove actions
- Making FormulaEditor and CardView tests less flacky
Test Plan: Existing
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz, dsagal
Differential Revision: https://phab.getgrist.com/D4280
This is a new entrypoint, mostly intended for Docker, so we have one
simple process controlling the main Grist process. The purpose of this
is to be able to make Grist easily restartable with a new environment.
Summary:
- lookupOne/lookupRecords explain `sort_by` param better, and
link to more detailed article.
- Incorporate a typo fix from Help Center
- Fix the omission of TASTEME never having been documented.
Test Plan: Corresponding update to Help Center can be reviewed at https://github.com/gristlabs/grist-help/pull/351
Reviewers: jarek
Reviewed By: jarek
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D4269
Summary:
Adding new buttons to control the `timing` API and a way to view the results
using virtual table features.
Test Plan: Added new
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4252
Summary:
Presence of **kwargs syntax led to a Python error when handling column renames.
This change fixes it by ignoring **kwargs in lookup methods.
Test Plan: Added a test case
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D4247
Summary:
Dropdown conditions let you specify a predicate formula that's used to filter
choices and references in their respective autocomplete dropdown menus.
Test Plan: Python and browser tests (WIP).
Reviewers: jarek, paulfitz
Reviewed By: jarek
Subscribers: dsagal, paulfitz
Differential Revision: https://phab.getgrist.com/D4235
Summary:
Renaming table after converting Ref column to RefList didn't work. During table rename, all Refs columns
are converted briefly to Int columns which treats values stored in RefList columns as errors, and stores its
`repr` strings. This could be recovered back if the value stored in RefList column was a plain list, but if we had there
a RecordList object, the RefList column didn't know how to parse that.
Test Plan: Added test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4233
Summary:
- /timing/start endpoint to start collecting information
- /timing/stop endpoint to stop collecting
- /timing to retrive data gatherd so far
Timings are collected for all columns (including hidden/helpers/system)
Test Plan: Added new
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D4230
Summary:
If a table T contains a RefList:T showing RowId and there
are any empty cells, the table can't be removed.
Test Plan: Added new test
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D4215
Summary:
If the longer OpenAI model exceeds the OpenAPI context length, we now perform another retry with a
shorter variant of the formula prompt. The shorter prompt excludes non-referenced tables and lookup
method definitions, which should help reduce token usage in documents with larger schemas.
Test Plan: Server test.
Reviewers: JakubSerafin
Reviewed By: JakubSerafin
Subscribers: JakubSerafin
Differential Revision: https://phab.getgrist.com/D4184
Summary:
It was manifesting as error "docactions.[Bulk]UpdateRecord for
non-existent record #1", and due to manualSort column having internal
state that wasn't properly cleared.
Test Plan: Added a test case that fails without the fix.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: georgegevoian
Differential Revision: https://phab.getgrist.com/D4183
Summary:
Improvements
- Widget and column descriptions are now copied when duplicating a table.
- A Grist Plugin API command to open a Record Card is now available.
- New Card widgets set initial settings based on those used by their table's
Record Card.
Fixes
- Opening a reference in a Record Card from a Raw Data popup now opens
the correct reference.
Test Plan: Browser and python tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D4164
* run.sh: Replace pid with nodejs's one using exec
This notably ensures that "docker stop" sends a TERM signal to the node
process which can handle it gracefully and avoid document corruption.
* Use exec form for CMD in Dockerfile
---------
Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
Summary:
Forms improvements and following new design
- New headers
- New UI
- New right panel options
Test Plan: Tests updated
Reviewers: georgegevoian, dsagal
Reviewed By: georgegevoian
Subscribers: dsagal, paulfitz
Differential Revision: https://phab.getgrist.com/D4158
Summary: Adds `@functools.lru_cache` to `ts_to_dt` and `ts_to_date` which are used in `_make_rich_value` in Date/DateTime columns, i.e. every time such a column is accessed in a formula. I noticed that these operations are surprisingly expensive while working on https://phab.getgrist.com/D4075. This is just an easy way to potentially significantly speed up certain docs and formulas.
Test Plan:
Put this code in an engine test case:
```
def test_(self):
self.apply_user_action(["AddTable", "Table1", [
{"id": "A", "type": "DateTime:America/Chicago"},
]])
self.add_records("Table1", ["A"], [
[i] for i in range(1000)
])
formula = "for _ in range(1000): $A\nreturn 1"
import time
start = time.time()
self.add_column(
"Table1", formula, type="Any", isFormula=True, formula=formula
)
elapsed = time.time() - start
print(f"Took {elapsed:.2f} for formula {formula}")
```
The time goes from ~10s to ~1s. Similar tests showed no noticeable slowdown when caching had no effect.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D4099
Summary:
This gives a mechanism for controlling access control within a document that is distinct from (though implemented with the same machinery as) granular access rules.
It was hard to find a good way to insert this that didn't dissolve in a soup of complications, so here's what I went with:
* When reading rules, if there are shares, extra rules are added.
* If there are shares, all rules are made conditional on a "ShareRef" user property.
* "ShareRef" is null when a doc is accessed in normal way, and the row id of a share when accessed via a share.
There's no UI for controlling shares (George is working on it for forms), but you can do it by editing a `_grist_Shares` table in a document. Suppose you make a fresh document with a single page/table/widget, then to create an empty share you can do:
```
gristDocPageModel.gristDoc.get().docData.sendAction(['AddRecord', '_grist_Shares', null, {linkId: 'xyz', options: '{"publish": true}'}])
```
If you look at the home db now there should be something in the `shares` table:
```
$ sqlite3 -table landing.db "select * from shares"
+----+------------------------+------------------------+--------------+---------+
| id | key | doc_id | link_id | options |
+----+------------------------+------------------------+--------------+---------+
| 1 | gSL4g38PsyautLHnjmXh2K | 4qYuace1xP2CTcPunFdtan | xyz | ... |
+----+------------------------+------------------------+--------------+---------+
```
If you take the key from that (gSL4g38PsyautLHnjmXh2K in this case) and replace the document's urlId in its URL with `s.<key>` (in this case `s.gSL4g38PsyautLHnjmXh2K` then you can use the regular document landing page (it will be quite blank initially) or API endpoint via the share.
E.g. for me `http://localhost:8080/o/docs/s0gSL4g38PsyautLHnjmXh2K/share-inter-3` accesses the doc.
To actually share some material - useful commands:
```
gristDocPageModel.gristDoc.get().docData.getMetaTable('_grist_Views_section').getRecords()
gristDocPageModel.gristDoc.get().docData.sendAction(['UpdateRecord', '_grist_Views_section', 1, {shareOptions: '{"publish": true, "form": true}'}])
gristDocPageModel.gristDoc.get().docData.getMetaTable('_grist_Pages').getRecords()
gristDocPageModel.gristDoc.get().docData.sendAction(['UpdateRecord', '_grist_Pages', 1, {shareRef: 1}])
```
For a share to be effective, at least one page needs to have its shareRef set to the rowId of the share, and at least one widget on one of those pages needs to have its shareOptions set to {"publish": "true", "form": "true"} (meaning turn on sharing, and include form sharing), and the share itself needs {"publish": true} on its options.
I think special shares are kind of incompatible with public sharing, since by their nature (allowing access to all endpoints) they easily expose the docId, and changing that would be hard.
Test Plan: tests added
Reviewers: dsagal, georgegevoian
Reviewed By: dsagal, georgegevoian
Subscribers: jarek, dsagal
Differential Revision: https://phab.getgrist.com/D4144
Summary: Descriptions can now be set on Raw Data table sections.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D4131
Summary: Title
Test Plan: Fixed a test that was failing inconsistently
Reviewers: georgegevoian, paulfitz
Reviewed By: georgegevoian, paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4122
Summary:
Adds a new Record Card view section to each non-summary table, which can be from opened from various parts of the Grist UI to view and edit records in a popup card view.
Work is still ongoing, so the feature is locked away behind a flag; follow-up work is planned to finish up the implementation and add end-to-end tests.
Test Plan: Python and server tests. Browser tests will be included in a follow-up.
Reviewers: jarek, paulfitz
Reviewed By: jarek
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D4114
Summary: Call a new user action `RemoveTransformColumns` in ActiveDoc shutdown.
Test Plan: Added nbrowser test
Reviewers: georgegevoian, paulfitz
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D4107
Pyodide packages needed rebuilding, and the pyodide project has
moved on a bit so a new version number is needed. The new packages
have already been built and pushed to S3.
To verify, go to `sandbox/pyodide` and follow the README there.
Then at top level do `GRIST_SANDBOX_FLAVOR=pyodide`. Try to
create and edit a document using formulas. It should work.
Summary: While looking at webhooks code, I noticed that it calls `ActiveDoc.fetchQuery` (which I think typically leads to the sandbox method `fetch_table`) with a query on the `id` column, where the values could potentially be all row IDs in the table (e.g. if a new column was added, as in a recent discussion about webhooks gone wrong). This suggests that `fetch_table` should try to avoid using a list for values when possible. In practice this only starts to become an issue at about 10k rows, so I don't know if this has caused any real problems, but it seemed like something worth fixing.
Test Plan:
Extended unit tests for correctness. Tested performance manually. Made a doc with this formula:
```
import time
row_ids = list(Table2.table.row_ids)
query = {"id": row_ids}
start = time.time()
result = table.table._engine.fetch_table('Table2', query=query)
end = time.time()
assert result[1] == row_ids
end - start, len(row_ids)
```
Then put a bunch of rows in `Table2`. This diff made the returned elapsed time much less.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D4092
Summary:
Fixes a very specific bug reported here: https://grist.slack.com/archives/C069RUP71/p1694630242765769
The error occurred when:
1. Removing multiple columns simultaneously
2. Those columns were sources of groupby columns for a summary table (so removing them meant recreating the summary table and thus deleting its columns)
3. There was a display column for one of the columns that got deleted (either directly or indirectly) which was set to be automatically removed since it was no longer needed, but this failed because the column was already deleted as part of earlier table removal.
I fixed this by making `apply_auto_removes` remove table records last, so removing the display column wouldn't be a problem.
That fixed the original error, but then I noticed that trying to undo the removal could lead to another error (under even more specific circumstances). It's hard to see exactly why, but I can see that just 3 `RemoveColumn` user actions generated over 100 doc actions and corresponding undo actions, hence the difficulty in narrowing the problem down. This is partly because removing a single column would recreate a summary table, only for that table to be immediately replaced again when another column was removed. Making the frontend send a single `[BulkRemoveRecord, _grist_Tables_column, ...] ` leads to a more efficient and sensible process with about half as many doc actions and no undo error. I think this alone would also solve the original error, but the data engine change seems more generally helpful and worth keeping.
Test Plan: Added a Python test and an nbrowser test
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D4052
Test Plan: Added a test case that reproduces the bug and tests the fix
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D4057
Summary: Upgrades asttokens and uses the newly released support for astroid trees in `asttokens.ASTText` which is needed to deal with f-strings (as opposed to `asttokens.ASTTokens`).
Test Plan: Added a Python unit test
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D4027
Summary:
Previously, when a table was deleted, ref/reflist columns pointing at that table were deleted as well. This diff changes that, converting the columns to a suitable type instead.
See here for discussion and decisions: https://grist.slack.com/archives/C069RUP71/p1686060034848139
Test Plan: Added Python tests for most cases, and a DocApi test for where Python has to call JS code to convert the values.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D4011
A chardet dependency had changed, breaking a grist-electron upgrade.
So I reran the pyodide script to build the new one, and pushed it
to our CDN. This file changed. I don't see exactly where it plays
a role, but I thought I'd better commit it anyway.
Summary:
- Using a sample of data was causing poor detection if the sample were
cut mid-character. Switch to using line-based detection.
- Add a simple option for changing encoding. No convenient UI is offered
since config UI is auto-generated, but this at least makes it possible to
recover from bad guesses.
- Upgrades chardet library for good measure.
- Also fixes python3-building step, to more reliably rebuild Python
dependencies when requirements3.* files change.
Test Plan:
Added a python-side test case, and a browser test that encodings can
be switched, errors are displayed, and wrong encodings fail recoverably.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3979
Summary:
Replaced uses of asttokens.ASTTokens with asttokens.ASTText when working with plain `ast` trees, and use `atok.get_text_range` instead of `node.first_token`.
Upgraded asttokens in Python 2 (it was already upgraded in Python 3).
Test Plan: Added a test with f-strings.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D4001
