gristlabs_grist-core/sandbox
Alex Hall 1864b7ba5d (core) Add BulkAddOrUpdateRecord action for efficiency
Summary:
This diff adds a new `BulkAddOrUpdateRecord` user action which is what is sounds like:

- A bulk version of the existing `AddOrUpdateRecord` action.
- Much more efficient for operating on many records than applying many individual actions.
- Column values are specified as maps from `colId` to arrays of values as usual.
- Produces bulk versions of `AddRecord` and `UpdateRecord` actions instead of many individual actions.

Examples of users wanting to use something like `AddOrUpdateRecord` with large numbers of records:

- https://grist.slack.com/archives/C0234CPPXPA/p1651789710290879
- https://grist.slack.com/archives/C0234CPPXPA/p1660743493480119
- https://grist.slack.com/archives/C0234CPPXPA/p1660333148491559
- https://grist.slack.com/archives/C0234CPPXPA/p1663069291726159

I tested what made many `AddOrUpdateRecord` actions slow in the first place. It was almost entirely due to producing many individual `AddRecord` user actions. About half of that time was for processing the resulting `AddRecord` doc actions. Lookups and updates were not a problem. With these changes, the slowness is gone.

The Python user action implementation is more complex but there are no surprises. The JS API now groups `records` based on the keys of `require` and `fields` so that `BulkAddOrUpdateRecord` can be applied to each group.

Test Plan: Update and extend Python and DocApi tests.

Reviewers: jarek, paulfitz

Reviewed By: jarek, paulfitz

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3642
2022-09-28 17:58:33 +02:00
..
docker (core) support python3 in grist-core, and running engine via docker and/or gvisor 2021-07-28 09:02:32 -04:00
grist (core) Add BulkAddOrUpdateRecord action for efficiency 2022-09-28 17:58:33 +02:00
gvisor (core) add GVISOR_LIMIT_MEMORY to cap memory available in sandbox 2022-05-18 14:26:27 -04:00
gen_js_schema.py (core) Use GristObjCode in CellValue 2021-10-11 14:11:32 +02:00
install_tz.js (core) apply some dependabot suggestions 2022-09-07 14:15:34 -04:00
requirements3.txt Remove messytables dependency from xlsx import 2022-09-21 14:20:19 +02:00
requirements.txt (core) Run and test imports only in Python 3, upgrade openpyxl, fix weird date handling 2022-09-02 16:27:34 +02:00
run.sh (core) add machinery for self-managed flavor of Grist 2022-05-12 12:39:52 -04:00
watch.sh remove watch flag no longer available under newer webpack (#229) 2022-07-28 13:07:58 -04:00