Commit Graph

16 Commits

Author SHA1 Message Date
Dmitry S
d5a4605d2a (core) Improve encoding detection for csv imports, and make encoding an editable option.
Summary:
- Using a sample of data was causing poor detection if the sample were
  cut mid-character. Switch to using line-based detection.
- Add a simple option for changing encoding. No convenient UI is offered
  since config UI is auto-generated, but this at least makes it possible to
  recover from bad guesses.
- Upgrades chardet library for good measure.

- Also fixes python3-building step, to more reliably rebuild Python
  dependencies when requirements3.* files change.

Test Plan:
Added a python-side test case, and a browser test that encodings can
be switched, errors are displayed, and wrong encodings fail recoverably.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3979
2023-08-24 09:50:52 -04:00
Alex Hall
5dfa9a542c (core) Upgrade to Python 3.11
Summary: Replaced mentions of Python 3.9 with 3.11

Test Plan: this

Reviewers: paulfitz, georgegevoian

Reviewed By: paulfitz, georgegevoian

Subscribers: dsagal, georgegevoian, paulfitz

Differential Revision: https://phab.getgrist.com/D3980
2023-08-04 18:19:40 +02:00
Alex Hall
ba16d50080 (core) Upgrade to latest versions of asttokens and wrapt
Summary:
Upgrades asttokens mainly because I suspected it would have helped with an error in a user's document (see https://grist.slack.com/archives/C0234CPPXPA/p1686145370484509) but we were unable to confirm that. Also adds a related test that I (wrongly) expected to have a similar problem before upgrading asttokens.

Upgrades wrapt for Python 3.11 support. Closes https://github.com/gristlabs/grist-core/issues/534.

Not upgrading the dependencies for Python 2 because pip is giving me errors when I try to install to the `thirdparty` folder. Both these upgrades are motivated by issues specific to Python 3 so this doesn't seem worth pursuing immediately.

Test Plan: Existing tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3973
2023-07-26 21:48:21 +02:00
Dmitry S
c0ed4a8a60 (core) Use a new docker image for tests in Jenkins
Summary:
The image Jenkins used up to now was from 2021. We updated buildtools/jenkins-ec2/Dockerfile since then but couldn't use new image; we can now that an issue with mounts in gvisor has been resolved.

The new image uses newer Chrome (111 vs 95); 111 is the latest version that works with the webdriver version we are using (after that one, copy-pasting tests fail).

In addition, this adds a more precise way to maintain python dependencies: by specifying top-level dependencies in the new core/sandbox/requirements3.in. An updated build step compiles that into requirements3.txt, and syncs venv to that using pip-sync.

This addresses the issue that previously, removing a module from dependencies would not have caused the build to remove it from sandbox_venv3.

Test Plan: Existing tests pass on the new image.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3952
2023-07-19 00:54:14 -04:00
Paul Fitzpatrick
66643a5e6b
add a pyodide-based "sandbox" flavor (#437)
This adds a new `GRIST_SANDBOX_FLAVOR=pyodide` option where the
version of Python used for the data engine is wasm, and so can
be run by node like the rest of the back end. It still runs as
a separate process.

There are a few small version changes made to packages to avoid
various awkwardnesses present in the current versions. All existing
tests pass.

This is very experimental. To use, you'll need something with
a bash shell and make. First do:
```
cd sandbox/pyodide
make setup           # README.md and Makefile have details
cd ..
```

Then running Grist as:
```
GRIST_SANDBOX_FLAVOR=pyodide yarn start
```
should work. Adding a formula with content:
```
import sys; return sys.version
```
should return a different Python version than other sandboxes.

The motivation for this work is to have a form of sandboxing
that will work on Windows for Grist Electron (for Linux we have
gvisor/runsc, for Mac we have sandbox-exec, but I haven't found
anything comparable for Windows).

It also brings a back-end-free version of Grist a bit closer, for
use-cases where that would make sense - such as serving a report
(in the form of a Grist document) on a static site.
2023-03-06 16:56:25 -05:00
Alex Hall
16a43edc2e (core) Upgrade friendly-traceback and its dependencies
Summary:
Upgrading the friendly-traceback package to include a fix that I specifically requested in https://github.com/friendly-traceback/friendly-traceback/issues/144 as a solution for the problem mentioned in https://grist.quip.com/HoSmAlvFax0j#MbTADAEcJb7 . Specifically, this shows a friendly explanation when using `len()` with a generator expression.

Also upgraded the dependencies `executing` and `stack_data` (which are mine) while I'm at it, although I don't expect this to really change anything.

Test Plan:
Existing tests. There was one test failure because of a new explanation about generic `Exception`s which I've suppressed.

Tested manually that the new explanation appears:

{F64605}

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3687
2022-10-28 14:44:14 +02:00
Dmitry S
0a8ce2178a (core) Add PHONE_FORMAT function using the phonenumberslight library
Summary:
Add phonenumberslite-8.12.57 to requirements
Implement PHONE_FORMAT function.

Test Plan: Added doctest test cases

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3665
2022-10-21 08:13:18 -04:00
Alex Hall
692ea4820f reorganise and comment on requirements files 2022-10-04 22:32:21 +02:00
Alex Hall
67c89f0d96 Remove requirements that were only there for messytables 2022-10-04 22:05:52 +02:00
Yohan Boniface
54703e2794 Remove messytables dependency from xlsx import 2022-09-21 14:20:19 +02:00
Alex Hall
42afb17e36 (core) Run and test imports only in Python 3, upgrade openpyxl, fix weird date handling
Summary:
Python 2 only needs to be supported for the sake of old documents and formulas. This doesn't apply to the separate sandboxes that parse files for imports. Using Python 3 only allows using newer libraries and library versions. In particular, the latest version of openpyxl doesn't support Python 2. This will also make it easier to make other similar changes in the future, such as replacing messytables with a modern library. See https://grist.slack.com/archives/C0234CPPXPA/p1661261829343999?thread_ts=1661260442.837959&cid=C0234CPPXPA

The latest openpyxl is better at handling a particular edge case with broken dates in Excel, but still doesn't quite do what we want, so we monkeypatch it. Discussion: https://grist.slack.com/archives/C02EGJ1FUCV/p1661440851911869?thread_ts=1661154219.515549&cid=C02EGJ1FUCV

Setting `preferredPythonVersion` to '3' in SafePythonComponent ensures that JS always creates import sandboxes that use Python 3. Within Python, a module used by all imports will raise an error in Python 2. Python unit tests of imports are now only run in Python 3, using the `load_tests` protocol of `unittest`.

Test Plan: Mostly existing tests. Added another strange date to the Excel fixture.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3606
2022-09-02 16:27:34 +02:00
Yohan Boniface
ac9f2fbdd1 Update some python deps to prepare python>=3.10 compat
ABC classes have moved from collections to collections.abc since
python 3.3, and retro compat has ended with 3.8
2022-08-26 16:45:24 +02:00
Alex Hall
49cb51bac5 (core) Error explanations from friendly-traceback
Summary: Extend formula error messages with explanations from https://github.com/friendly-traceback/friendly-traceback. Only for Python 3.

Test Plan: Updated several Python tests. In general, these require separate branches for Python 2 and 3.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3542
2022-08-12 19:45:00 +02:00
Alex Hall
6c90de4d62 (core) Switch excel import parsing from messytables+xlrd to openpyxl, and ignore empty rows
Summary:
Use openpyxl instead of messytables (which used xlrd internally) in import_xls.py.

Skip empty rows since excel files can easily contain huge numbers of them.

Drop support for xls files (which openpyxl doesn't support) in favour of the newer xlsx format.

Fix some details relating to python virtualenvs and dependencies, as Jenkins was failing to find new Python dependencies.

Test Plan: Mostly relying on existing tests. Updated various tests which referred to xls files instead of xlsx. Added a Python test for skipping empty rows.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3406
2022-05-12 14:43:21 +02:00
Alex Hall
5b16dd2e86 (core) Upgrade chardet
Summary: Upgrade chardet version from 2.3.0 to 4.0.0 to improve encoding detection when importing files with non-ascii characters.

Test Plan: the tests

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3080
2021-10-21 18:29:17 +02:00
Paul Fitzpatrick
bb8cb2593d (core) support python3 in grist-core, and running engine via docker and/or gvisor
Summary:
 * Moves essential plugins to grist-core, so that basic imports (e.g. csv) work.
 * Adds support for a `GRIST_SANDBOX_FLAVOR` flag that can systematically override how the data engine is run.
   - `GRIST_SANDBOX_FLAVOR=pynbox` is "classic" nacl-based sandbox.
   - `GRIST_SANDBOX_FLAVOR=docker` runs engines in individual docker containers. It requires an image specified in `sandbox/docker` (alternative images can be named with `GRIST_SANDBOX` flag - need to contain python and engine requirements). It is a simple reference implementation for sandboxing.
   - `GRIST_SANDBOX_FLAVOR=unsandboxed` runs whatever local version of python is specified by a `GRIST_SANDBOX` flag directly, with no sandboxing. Engine requirements must be installed, so an absolute path to a python executable in a virtualenv is easiest to manage.
   - `GRIST_SANDBOX_FLAVOR=gvisor` runs the data engine via gvisor's runsc. Experimental, with implementation not included in grist-core. Since gvisor runs on Linux only, this flavor supports wrapping the sandboxes in a single shared docker container.
 * Tweaks some recent express query parameter code to work in grist-core, which has a slightly different version of express (smoke test doesn't catch this since in Jenkins core is built within a workspace that has node_modules, and wires get crossed - in a dev environment the problem on master can be seen by doing `buildtools/build_core.sh /tmp/any_path_outside_grist`).

The new sandbox options do not have tests yet, nor does this they change the behavior of grist servers today. They are there to clean up and consolidate a collection of patches I've been using that were getting cumbersome, and make it easier to run experiments.

I haven't looked closely at imports beyond core.

Test Plan: tested manually against regular grist and grist-core, including imports

Reviewers: alexmojaki, dsagal

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D2942
2021-07-28 09:02:32 -04:00