Commit Graph

35 Commits

Author SHA1 Message Date
Alex Hall
1c43aed5dd (core) Upgrade sortedcontainers, python_dateutil, and html5lib for better Python 3.10 compatibility
Summary:
Based on https://github.com/gristlabs/grist-core/pull/251. It may not look like it, but there's very little going on in this diff:

- Tweak the DATEVALUE doctest for Python 2/3 compatibility.
- Mirrors the PR's changes to requirements3.txt in requirements.txt, i.e. make the same dependency upgrades in Python 2.
- Make the same upgrades in the thirdparty folder for the Python 2 nacl sandbox.

Test Plan: Updated one doctest for dateutil. Checked changelog of sortedcontainers. html5lib is only used by messytables and isn't actually relevant.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3609
2022-08-30 16:46:29 +02:00
Paul Fitzpatrick
63683f98cc (core) updates from grist-core 2022-08-26 17:29:25 -04:00
Yohan Boniface
ac9f2fbdd1 Update some python deps to prepare python>=3.10 compat
ABC classes have moved from collections to collections.abc since
python 3.3, and retro compat has ended with 3.8
2022-08-26 16:45:24 +02:00
Alex Hall
5f17dd0a06 (core) Convince grist-help that REQUEST is really unimplemented
Summary:
Change made to allow rebuilding functions in https://github.com/gristlabs/grist-help/pull/170

Deals the error described here: https://grist.slack.com/archives/C0234CPPXPA/p1660245534781629

Test Plan: Regenerating functions in grist-help works again.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3594
2022-08-23 19:25:27 +02:00
Alex Hall
31f54065f5 (core) Use standard library uuid.uuidv4 when possible for better randomness
Summary: Using the `random` module in the Grist `UUID()` function is not cryptographically secure, and is only necessary for the old pynbox (Python 2) sandbox which doesn't support `os.urandom`. This diff uses the `uuid.uuidv4()` function from the Python standard library when possible, which is more secure, only falling back to the old implementation when necessary.

Test Plan: Added Python unit tests to check both implementations.

Reviewers: dsagal

Subscribers: paulfitz, dsagal

Differential Revision: https://phab.getgrist.com/D3578
2022-08-12 16:45:11 +02:00
Alex Hall
083a0ec000 (core) Allow using MIN and MAX functions with dates/datetimes
Summary:
The MIN and MAX functions for formulas previously only considered numbers, ignoring other types, including dates. An example of this being a problem is here: https://community.getgrist.com/t/last-field-circularreferror-what-is-it/1114/4 . Using `MIN` on a column of dates would return 0 (the default) which gets converted to 1970-01-01. Users have to use `min` instead, which is confusing, and doesn't work when some values are empty.

This diff lets the functions operate on date and datetime values. A mixture of dates and datetimes is allowed, even though these cannot usually be compared in Python. Mixing dates and numbers will raise an exception.

Test Plan: Extended doctests

Reviewers: jarek, paulfitz

Reviewed By: jarek

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3560
2022-08-09 13:34:22 +02:00
Paul Fitzpatrick
7078922a65 (core) ensure randomness works when sandbox is cloned from a checkpoint
Summary:
This calls a new `initialize` method on the sandbox before we start
doing calculations with it, to make sure that `random.seed()` has
been called. Otherwise, if the sandbox is cloned from a checkpoint,
the seed will have been reset.

The `initialize` method includes the functionality previously done
by `set_doc_url` since it is also initialization/personalization and
this way we avoid introducing another round trip to the sandbox.

Test Plan: tested with grist-core configured to use gvisor

Reviewers: georgegevoian, dsagal

Reviewed By: georgegevoian, dsagal

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3549
2022-07-27 14:59:27 -04:00
Alex Hall
9fffb491f9 (core) External requests
Summary:
Adds a Python function `REQUEST` which makes an HTTP GET request. Behind the scenes it:

- Raises a special exception to stop trying to evaluate the current cell and just keep the existing value.
- Notes the request arguments which will be returned by `apply_user_actions`.
- Makes the actual request in NodeJS, which sends back the raw response data in a new action `RespondToRequests` which reevaluates the cell(s) that made the request.
- Wraps the response data in a class which mimics the `Response` class of the `requests` library.

In certain cases, this asynchronous flow doesn't work and the sandbox will instead synchronously call an exported JS method:

- When reevaluating a single cell to get a formula error, the request is made synchronously.
- When a formula makes multiple requests, the earlier responses are retrieved synchronously from files which store responses as long as needed to complete evaluating formulas. See https://grist.slack.com/archives/CL1LQ8AT0/p1653399747810139

Test Plan: Added Python and nbrowser tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: paulfitz, dsagal

Differential Revision: https://phab.getgrist.com/D3429
2022-06-17 21:53:20 +02:00
Alex Hall
1c89d08ea3 (core) Add a row to summary tables grouped by list column(s) corresponding to empty lists
Summary:
Adds some special handling to summary table and lookup logic:

- Source rows with empty choicelists/reflists get a corresponding summary row with an empty string/reference when grouping by that column, instead of excluding them from any group
- Adds a new `QueryOperation` 'empty' in the client which is used in `LinkingState`, `QuerySet`, and `recursiveMoveToCursorPos` to match empty lists in source tables against falsy values in linked summary tables.
- Adds a new parameter `match_empty` to the Python `CONTAINS` function so that regular formulas can implement the same behaviour as summary tables. See https://grist.slack.com/archives/C0234CPPXPA/p1654030490932119
- Uses the new `match_empty` argument in the formula generated for the `group` column when detaching a summary table.

Test Plan: Updated and extended Python and nbrowser tests of summary tables grouped by choicelists to test for new behaviour with empty lists.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3471
2022-06-09 23:38:14 +02:00
Alex Hall
c5ebd7db3d (core) Add PEEK() function to bypass circular dependencies
Summary:
Adds a Python function `PEEK()` for use in formulas which temporarily sets a new attribute `Engine._peeking` which disables the `_use_node` method, preventing dependency tracking and allowing the given expression to use outdated values. This allows circumventing circular reference errors. It's particularly meant for trigger formulas although it works in normal formulas as well. The expression is wrapped in a `lambda` by `codebuilder` for lazy evaluation.

Discussion: https://grist.slack.com/archives/C0234CPPXPA/p1653571024031359

Test Plan: Added a Python unit test for circular trigger formulas using PEEK.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3453
2022-06-02 19:50:14 +02:00
Alex Hall
dc9e53edc8 (core) Update the current time in formulas automatically every hour
Summary: Adds a special user action `UpdateCurrentTime` which invalidates an internal engine dependency node that doesn't belong to any table but is 'used' by the `NOW()` function. Applies the action automatically every hour.

Test Plan: Added a Python test for the user action. Tested the interval periodically applying the action manually: {F43312}

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3389
2022-04-28 21:07:40 +02:00
Alex Hall
2d0978559b (core) Replace compute stack and frames with _current_node and _current_row_id
Summary:
This is an attempt to optimise Engine._use_node. It doesn't seem to actually improve overall performance significantly, but it shouldn't make it worse, and I think it's an improvement to the code.

It turns out that there's no need to track a stack of compute frames any more. The only time we get close to nested evaluation, we set allow_evaluation=False to prevent it actually happening. So there's only one 'frame' during actual evaluation, which means we can get rid of the concept of frames entirely. This allows simplifying the code and letting the computer do less work in general.

Test Plan: this

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3310
2022-03-11 12:34:00 +02:00
Dmitry S
3136077636 (core) Update documentation of CURRENT_CONVERSION function
Summary: As requested by Anais in https://github.com/dsagal/grist-help/pull/132

Test Plan: Only a change to doc-comment (from which help docs are built).

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3267
2022-02-16 08:48:12 -05:00
Dmitry
25b7e6c245
Tweaks to documentation comments for user-facing Grist functions (#126)
- Update internal links in function documentation
- Remove emphasis in code blocks
- Remove trailing whitespace
2022-02-13 00:45:24 -05:00
Natalie Misasi
2611d05c39
Updates to Grist Functions (#125)
Clarify documentation for QUOTIENT, SUMPRODUCT, UUID, REPLACE, lookupOne, lookupRecords, and $group.
2022-02-13 00:38:24 -05:00
Alex Hall
5d671bf0b3 (core) New type conversion in the backend
Summary: This is https://phab.getgrist.com/D3205 plus some changes (https://github.com/dsagal/grist/compare/type-convert...type-convert-server?expand=1) that move the conversion process to the backend. A new user action ConvertFromColumn uses `call_external` so that the data engine can delegate back to ActiveDoc. Code for creating formatters and parsers is significantly refactored so that most of the logic is in `common` and can be used in different ways.

Test Plan: The original diff adds plenty of tests.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3240
2022-02-04 20:28:13 +02:00
Dmitry S
b37b8a9f6d (core) A few tiny documentation tweaks.
Summary:
- Improve readability of documentation of CONTAINS.
- Add leading underscore to Record._get_encodable_row_ids() to hide from
  public docs, and avoid interfering with user fields.
- Fix up lint errors

Test Plan: No behavior changes

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3209
2022-01-07 14:26:00 -05:00
Natalie Misasi
206e1550d5
Update lookup.py: include into CONTAINS a link to lookupRecords (#119) 2022-01-06 16:20:47 -05:00
Alex Hall
dc52c7fe0a (core) Add PY2_ROUND function for easier migration
Summary: As discussed in https://github.com/dsagal/grist-help/pull/112

Test Plan: doctest in function

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3147
2021-11-19 17:13:27 +02:00
Alex Hall
1ed4ca61ea (core) Account for timezone in TODAY()
Summary: return NOW(tz=tz).date()

Test Plan: None, curious to see if this fixes test_time_defaults when run near midnight.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3079
2021-10-21 18:27:35 +02:00
Alex Hall
0e5c2bee59 (core) Add CellError to explain when a formula refers to a cell with an error value
Summary:
get_cell_value wraps RaisedException with CellError to expand the error message for the user.

This is still pretty conceptual, the comments explain some things to think about, but it works and is an improvement.

Test Plan: Updated Python unit tests

Reviewers: dsagal, paulfitz

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2928
2021-07-30 22:58:45 +02:00
Alex Hall
04e5d90f86 (core) Barely working reference lists in frontend
Summary:
This makes it possible to set the type of a column to ReferenceList, but the UI is terrible

ReferenceList.ts is a mishmash of ChoiceList and Reference that sort of works but something about the CSS is clearly broken

ReferenceListEditor is just a text editor, you have to type in a JSON array of row IDs. Ignore the value that's present when you start editing. I can maybe try mashing together ReferenceEditor and ChoiceListEditor but it doesn't seem wise.
I think @georgegevoian should take over here. Reviewing the diff as it is to check for obvious issues is probably good but I don't think it's worth trying to land/merge anything.

Test Plan: none

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D2914
2021-07-23 18:41:44 +02:00
Alex Hall
f5981606e1 (core) Make CONTAINS a function for consistency with mkpydocs etc.
Summary: Having CONTAINS be a class is a pain, undoing that mistake now

Test Plan: none needed

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2929
2021-07-21 13:18:23 +02:00
Alex Hall
f7a9638992 (core) CONTAINS() and summarising by ChoiceList columns with flattening
Summary:
Added CONTAINS 'function' which can be used in lookups

Changed LookupMapColumn._row_key_map to use right=set so one row can have many keys when CONTAINS is used.

Use CONTAINS to implement group column in summary table, while helper column in source table can reference and create multiple rows in summary table, especially when summarising by ChoiceList columns.

Use itertools.product to generate all combinations of lookup keys and groupby values.

cleanup

Test Plan: Added python unit tests.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz, dsagal

Differential Revision: https://phab.getgrist.com/D2900
2021-07-19 16:35:35 +02:00
Alex Hall
305b133c59 (core) Remaining Python 3 compatibility changes
Summary: Biggest change is turning everything to unicode

Test Plan: The tests

Reviewers: dsagal, paulfitz

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2875
2021-06-25 12:00:58 +02:00
Alex Hall
16f297a250 (core) Simple Python 3 compatibility changes
Summary: Changes that move towards python 3 compatibility that are easy to review without much thought

Test Plan: The tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2873
2021-06-22 17:13:17 +02:00
Alex Hall
8a940676e9 (core) Generic tools for recording pycalls, deterministic mode.
Summary:
Replaces https://phab.getgrist.com/D2854

Refactoring of NSandbox:
- Simplify arguments to NSandbox.spawn. Only half the arguments were used depending on the flavour, adding a layer of confusion.
- Ensure the same environment variables are passed to both flavours of sandbox
- Simplify passing down environment variables.

Implement deterministic mode with libfaketime and a seeded random instance.
- Include static prebuilt libfaketime.so.1, may need another solution in future for other platforms.

Recording pycalls:
- Add script recordDocumentPyCalls.js to open a single document outside of tests.
- Refactor out recordPyCalls.ts to support various uses.
- Add afterEach hook to save all pycalls from server tests under $PYCALLS_DIR
- Make docTools usable without mocha.
- Add useLocalDoc and loadLocalDoc for loading non-fixture documents

Test Plan:
Made a document with formulas NOW() and UUID()
Compare two document openings in normal mode:

    diff <(test/recordDocumentPyCalls.js samples/d4W6NrzCMNVSVD6nWgNrGC.grist /dev/stdout) \
         <(test/recordDocumentPyCalls.js samples/d4W6NrzCMNVSVD6nWgNrGC.grist /dev/stdout)

Output:

    <                 1623407499.58132,
    ---
    >                 1623407499.60376,
    1195c1195
    <               "B": "bd2487f6-63c9-4f02-bbbc-5c0d674a2dc6"
    ---
    >               "B": "22e1a4fd-297f-4b86-91a2-bc42cc6da4b2"

`export DETERMINISTIC_MODE=1` and repeat. diff is empty!

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2857
2021-06-15 20:58:05 +02:00
Paul Fitzpatrick
0c5f7cf0a7 (core) add SELF_HYPERLINK() function for generating links to the current document
Summary:
 * Adds a `SELF_HYPERLINK()` python function, with optional keyword arguments to set a label, the page, and link parameters.
 * Adds a `UUID()` python function, since using python's uuid.uuidv4 hits a problem accessing /dev/urandom in the sandbox.  UUID makes no particular quality claims since it doesn't use an audited implementation.  A difficult to guess code is convenient for some use cases that `SELF_HYPERLINK()` enables.

The canonical URL for a document is mutable, but older versions generally forward.  So for implementation simplicity the document url is passed it on sandbox creation and remains fixed throughout the lifetime of the sandbox.  This could and should be improved in future.

The URL is passed into the sandbox as a `DOC_URL` environment variable.

The code for creating the URL is factored out of `Notifier.ts`. Since the url is a function of the organization as well as the document, some rejiggering is needed to make that information available to DocManager.

On document imports, the new document is registered in the database slightly earlier now, in order to keep the procedure for constructing the URL in different starting conditions more homogeneous.

Test Plan: updated test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2759
2021-03-18 19:37:07 -04:00
Dmitry S
ba9959b6af (core) Fix broken markdown link in the doc-comment of VLOOKUP
Test Plan: No code changes

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2752
2021-03-09 18:00:59 -05:00
Dmitry S
2b71cce88b (core) Fix CONCAT function that was breaking on presence of non-ascii characters
Summary: This was the only occurrence of the unicode() function that I could find.

Test Plan: Added a doctest case.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2750
2021-03-09 16:27:48 -05:00
Dmitry S
003029bf8a (core) Improve suggestions for formula autocomplete
Summary:
- Make suggestions less case-sensitive (not entirely case-insensitive, but
  allow top-level suggestions to match in all-lowercase)
- Add function signatures to suggestions for Grist functions.
- Excel-like functions that are present but not implemented are no longer
  offered as suggestions.

Test Plan:
Added a test case on python side, and a browser test case for how suggestions
are rendered and inserted.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2608
2020-09-12 00:21:29 -04:00
Dmitry S
2fbd3f1706 (core) Fix lookups in default formulas
Test Plan: TODO

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2603
2020-09-10 18:06:06 -04:00
Dmitry S
7a8debae16 (core) Improve object serialization, to help get RECORD data to Custom Widgets.
Summary:
- Change RECORD's dates_as_str default to False.
- Reimplement objtype encode_object/decode_object with less machinery.
- Implement encoding of dicts (with string keys).
- Make lists and dicts encode values recursively.
- Implement encoding/decoding in the client
- Decode automatically in plugins' fetchSelectedTable/Record, with an option to skip.

Test Plan: Tested manually, not sure what tests may be affected yet.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2593
2020-08-21 18:33:28 -04:00
Dmitry S
4e11b6a922 (core) Implement RECORD function that converts Records to Python dictionaries
Summary:
Examining a Record is currently difficult, because its columns are hard to
list (and to use), and CircularRef errors hard to avoid. The RECORD function
takes care of this mess to return a simple dictionary of values.

- Supports dates_as_iso=False flag to turn off the translation of date/datetime
  objects to strings.
- Supports expand_refs=True flag to apply RECORD() to encountered values of
  type Record, for a single level of nesting.

Test Plan: Added a unittest for RECORD()

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2582
2020-08-12 13:31:57 -04:00
Paul Fitzpatrick
b82eec714a (core) move data engine code to core
Summary:
this moves sandbox/grist to core, and adds a requirements.txt
file for reconstructing the content of sandbox/thirdparty.

Test Plan:
existing tests pass.
Tested core functionality manually.  Tested docker build manually.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2563
2020-07-29 08:57:25 -04:00