Commit Graph

146 Commits

Author SHA1 Message Date
George Gevoian
0717ee627e (core) Relocate export urls to /download/
Summary:
Moves CSV and XLSX export urls under /download/, and
removes the document title query parameter which is now
retrieved from the backend.

Test Plan: No new tests. Existing tests that verify endpoints still function.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3010
2021-09-02 09:36:33 -07:00
George Gevoian
ef5da42378 (core) Update export CSV and Excel endpoints
Summary:
The endpoints for exporting CSV and Excel are now under
/api/docs/:docId/ and are forwarded to a doc worker for export.

The Share Menu has been updated to use the new endpoints.

Test Plan: No new tests. Existing tests that verify endpoints work correctly.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3007
2021-08-31 10:47:24 -07:00
George Gevoian
a6e08883e0 (core) Simple localization support and currency selector.
Summary:
- Grist document has a associated "locale" setting that affects how currency is formatted.
- Currency selector for number format.

Test Plan: not done

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D2977
2021-08-26 13:36:49 -07:00
Paul Fitzpatrick
e492dfdb22 (core) add experimental support for python3 in staging
Summary:
This adds `runsc` and `python3` to the grist-server images. For deployments with GRIST_EXPERIMENTAL_PLUGINS=1 (dev + staging but not prod) a hack is added to use `python3` under `runsc` for documents with a special title (`activate-python3-magic` or similar).

This will simplify experiments on behavior of this configuration under realistic conditions.

Hopefully, before landing this, I'll be able to switch to storing a python flag in a document options cell being added by @georgegevoian in a parallel diff, since using the doc title is super hacky :-).

Test Plan: tested manually on worker built locally

Reviewers: dsagal, alexmojaki

Reviewed By: dsagal, alexmojaki

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D2998
2021-08-26 09:39:26 -04:00
Paul Fitzpatrick
b3636b97e2 (core) Report memos consistently for blocked actions involving schema
Summary:
Currently actions blocked early because they could modify the
schema (e.g. changing formulas) do not report memo information
(comments in relevant rules).  This diff fixes that by using
more of the same code path in the two situations.  It also
adds information about what type of action was blocked to
error messages.

Test Plan: extended a test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2995
2021-08-24 08:03:58 -04:00
Paul Fitzpatrick
f53ab2cb30 (core) forbids edits when "view as" user is a viewer and access rules are permissive
Summary:
Currently, if access rules are set to allow edits unconditionally,
and an owner does "View As" a user who is a viewer only, they will
be allowed to make edits.  This catches that condition and adds a
test.

Test Plan: added test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D2991
2021-08-20 16:04:28 -04:00
Dmitry S
00c1a0c688 (core) Log the time taken by decodeActionFromRow() operations.
Summary:
Decoding large actions is a plausible culprit for hogging CPU time for
certain documents. To begin with, log the time taken for this operation,
so that we can tell if it's a problem in practice.

Test Plan: Should not affect any current behaviors

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2989
2021-08-20 11:28:33 -04:00
Paul Fitzpatrick
c561dad22d (core) lightly freshen the core readme, mentioning roadmap and forums etc.
Summary: This is a documentation update, and version bump on grist-core.

Test Plan: No code changes.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2982
2021-08-17 23:51:58 -04:00
Paul Fitzpatrick
f9630b3aa4 (core) clean up a collection of small problems affecting grist-core
Summary:
 * Remove adjustSession hack, interfering with loading docs under saml.
 * Allow the anonymous user to receive an empty list of workspaces for
   the merged org.
 * Behave better on first page load when org is in path - this used to
   fail because of lack of cookie.  This is very visible in grist-core,
   as a failure to load localhost:8484 on first visit.
 * Mark cookie explicitly as SameSite=Lax to remove a warning in firefox.
 * Make errorPages available in grist-core.

This changes the default behavior of grist-core to now start off in
anonymous mode, with an explicit sign-in step available.  If SAML is not configured,
the sign-in operation will unconditionally sign the user in as a default
user, without any password check or other security.  The user email is
taken from GRIST_DEFAULT_EMAIL if set.  This is a significant change, but
makes anonymous mode available in grist-core (which is convenient
for testing) and makes behavior with and without SAML much more consistent.

Test Plan: updated test; manual (time to start adding grist-core tests though!)

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2980
2021-08-17 21:44:50 -04:00
Alex Hall
e6e792655b (core) Add /columns endpoint to DocApi
Summary: Add /columns endpoint to DocApi

Test Plan: Added test

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2981
2021-08-17 23:20:52 +02:00
Paul Fitzpatrick
54beaede84 (core) revive saml support and test against Auth0
Summary:
SAML support had broken due to SameSite changes in browsers. This
makes it work again, and tests it against Auth0 (now owned by Okta).

Logging in and out works.  The logged out state is confusing, and may
not be complete.  The "Add Account" menu item doesn't work.
But with this, an important part of self-hosting becomes easier.

SAML support works also in grist-core, for site pages, but there
is a glitch on document pages that I'll look into separately.

Test Plan: tested manually

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2976
2021-08-16 17:36:09 -04:00
Alex Hall
34e9ad3498 (core) Add /records endpoint to DocApi with GET, POST, and PATCH
Summary:
Applies simple data transformations to the existing /data API.

Mimics the Airtable API. Designed in https://grist.quip.com/RZh9AEbPaj8x/Doc-API#FZfACAAZ9a0

Haven't done deletion because it seems like less of a priority and also not fully designed.

Test Plan: Added basic server tests similar to the /data tests. Haven't tested edge cases like bad input.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D2974
2021-08-12 18:02:56 +02:00
Alex Hall
7f1f8fc9e6 (core) Linking summary tables grouped by list columns
Summary:
Prefix keys of `LinkingState.filterColValues` with `_contains:` when the source column is a ChoiceList or ReferenceList.

This is parsed out to make a boolean `isContainsFilter` which is kept in each value of `QueryRefs.filterTuples` (previously `filterPairs`).

Then when converting back in `convertQueryFromRefs` we construct `Query.contains: {[colId: string]: boolean}`.

Finally `getFilterFunc` uses `Query.contains` to decide what kind of filtering to do.

This is not pretty, but the existing code is already very complex and it was hard to find something that wouldn't require touching loads of code just to make things compile.

Test Plan: Added a new nbrowser test and fixture, tests that selecting a source table by summary tables grouped by a choicelist column, non-list column, and both all filter the correct data.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2940
2021-08-10 20:41:24 +02:00
Alex Hall
4d526da58f (core) Move file import plugins into core/sandbox/grist
Summary:
Move all the plugins python code into the main folder with the core code.

Register file importing functions in the same main.py entrypoint as the data engine.

Remove options relating to different entrypoints and code directories. The only remaining plugin-specific option in NSandbox is the import directory/mount, i.e. where files to be parsed are placed.

Test Plan: this

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D2965
2021-08-09 18:37:14 +02:00
Paul Fitzpatrick
4a23b964ed (core) update read access for exceptional sessions
Summary:
Exceptional sessions had lost full read access to documents; this
restores it.  Exceptional sessions are used for system actions or
while creating documents.

Test Plan: added test

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D2966
2021-08-05 16:49:23 -04:00
Jarosław Sadziński
4ca47878ca (core) Adding import from google drive to the home screen
Summary: Importing from google drive from home screen (also for anonymous users)

Test Plan: Browser tests

Reviewers: dsagal, paulfitz

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2943
2021-08-05 20:46:11 +02:00
Jarosław Sadziński
6ed1d8dfea (core) Adding google drive plugin as a fallback for url plugin
Summary:
When importing from url, user types a url for google spreadsheet,
Grist will switch to Google Drive plugin to allow user to choose file manualy.

Test Plan: Browser tests

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2945
2021-08-04 17:59:13 +02:00
Alex Hall
5aed22dc1e (core) Remove dead code for fetching snapshots
Summary: Deletes code which was previously only used by SharedSharing.ts, which was deleted in D2894

Test Plan: no

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2960
2021-08-04 15:42:31 +02:00
Paul Fitzpatrick
750c78763e (core) add python2 to gvisor Dockerfile, for use in making comparisons
Summary:
This adds python2 to the gvisor sandbox image.  It can be used instead
of the default python3 by setting PYTHON_VERSION to 2 (or calling run.py with python2).
This is useful for making side-by-side comparisons with code running python3.

Test Plan: manual

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D2957
2021-08-03 17:51:31 -04:00
Jarosław Sadziński
521bbd9ac1 (core) Improving error messages on file imports
Summary: Improving error messages that get returned from "Import from URL" plugin.

Test Plan: browser tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: dsagal, alexmojaki

Differential Revision: https://phab.getgrist.com/D2946
2021-08-02 17:30:59 +02:00
Paul Fitzpatrick
6b3ac07ca7 (core) process GristDocAPI calls from custom widgets in the client
Summary:
Processing these calls in the client, rather than passing them on
to the backend, means that access rules are more straightforward to
apply.

An unrelated fix is included to filter _grist_ tables when fetched
individually - metadata could leak through this path.

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2954
2021-07-30 10:41:32 -04:00
Paul Fitzpatrick
bb8cb2593d (core) support python3 in grist-core, and running engine via docker and/or gvisor
Summary:
 * Moves essential plugins to grist-core, so that basic imports (e.g. csv) work.
 * Adds support for a `GRIST_SANDBOX_FLAVOR` flag that can systematically override how the data engine is run.
   - `GRIST_SANDBOX_FLAVOR=pynbox` is "classic" nacl-based sandbox.
   - `GRIST_SANDBOX_FLAVOR=docker` runs engines in individual docker containers. It requires an image specified in `sandbox/docker` (alternative images can be named with `GRIST_SANDBOX` flag - need to contain python and engine requirements). It is a simple reference implementation for sandboxing.
   - `GRIST_SANDBOX_FLAVOR=unsandboxed` runs whatever local version of python is specified by a `GRIST_SANDBOX` flag directly, with no sandboxing. Engine requirements must be installed, so an absolute path to a python executable in a virtualenv is easiest to manage.
   - `GRIST_SANDBOX_FLAVOR=gvisor` runs the data engine via gvisor's runsc. Experimental, with implementation not included in grist-core. Since gvisor runs on Linux only, this flavor supports wrapping the sandboxes in a single shared docker container.
 * Tweaks some recent express query parameter code to work in grist-core, which has a slightly different version of express (smoke test doesn't catch this since in Jenkins core is built within a workspace that has node_modules, and wires get crossed - in a dev environment the problem on master can be seen by doing `buildtools/build_core.sh /tmp/any_path_outside_grist`).

The new sandbox options do not have tests yet, nor does this they change the behavior of grist servers today. They are there to clean up and consolidate a collection of patches I've been using that were getting cumbersome, and make it easier to run experiments.

I haven't looked closely at imports beyond core.

Test Plan: tested manually against regular grist and grist-core, including imports

Reviewers: alexmojaki, dsagal

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D2942
2021-07-28 09:02:32 -04:00
Alex Hall
04e5d90f86 (core) Barely working reference lists in frontend
Summary:
This makes it possible to set the type of a column to ReferenceList, but the UI is terrible

ReferenceList.ts is a mishmash of ChoiceList and Reference that sort of works but something about the CSS is clearly broken

ReferenceListEditor is just a text editor, you have to type in a JSON array of row IDs. Ignore the value that's present when you start editing. I can maybe try mashing together ReferenceEditor and ChoiceListEditor but it doesn't seem wise.
I think @georgegevoian should take over here. Reviewing the diff as it is to check for obvious issues is probably good but I don't think it's worth trying to land/merge anything.

Test Plan: none

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D2914
2021-07-23 18:41:44 +02:00
Jarosław Sadziński
f8e4fe54ba (core) Fixing origin check during Google Authentication
Summary:
Fixing two bugs
- Google Auth Endpoint wasn't resolving protocol in a correct way
- Google Auth Popup was navigationg to endpoint url based on home url, which
  was diffent from current page origin

Test Plan: n/a

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D2937
2021-07-23 00:43:27 +02:00
Paul Fitzpatrick
95cc2eb282 (core) read document as owner in pre-fork mode, if have sufficient access to it
Summary:
This tweaks pre-fork mode to make the user's experience a bit more seamless.
Pre-fork mode is where the user has opened a document with intent to
fork it, but actual forking (with allocation of a new document id)
is postponed until they make their first change.

The tweak makes the user an owner for granular access purposes, if
forking is permitted.  So data visible only to owners because of
access rules will be visible to them.  As always, any edits would
go to a separate new copy.

A remaining tricky corner is what to do about "View As" functionality
on forks.  Fork sharing cannot be controlled, so the "Users -> View As"
functionality isn't available.  Perhaps the "Users" button on a fork
could encourage doing a save-copy and inviting users, or offer some
dummy users?  In any case, this diff doesn't change anything with
that corner.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2931
2021-07-21 14:52:31 -04:00
Jarosław Sadziński
291bcd17ff (core) Google auth endpoint has not responded with auth code
Summary:
Google Auth popup wasn't able to resolve origin from gristConfig.
Moving this reponsability to server side, where it gets calculated from initial request.

Test Plan: n/a

Reviewers: dsagal, paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2935
2021-07-21 20:18:04 +02:00
Jarosław Sadziński
08295a696b (core) Export to Excel and Send to drive
Summary:
Implementing export to excel and send to Google Drive feature.

As part of this feature few things were implemented:
- Server side google authentication exposed on url: (docs, docs-s, or localhost:8080)/auth/google
- Exporting grist documents as an excel file (xlsx)
- Storing exported grist document (in excel format) in Google Drive as a spreadsheet document.

Server side google authentication requires one new environmental variables
- GOOGLE_CLIENT_SECRET (required) used by authentication handler

Test Plan: Browser tests for exporting to excel.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2924
2021-07-21 16:36:00 +02:00
Paul Fitzpatrick
997be24a21 (core) add docs.options column to home db to store doc description, icon, openMode
Summary:
Bundles some new document options into a JSON column.
The icon option is treated somewhat gingerly.  It is intended, at
least initially, to store an image thumbnail for a document as a
url to hand-prepared assets (for examples and templates), so it is
locked down to a particular url prefix to avoid opening the door to
mischief.

Test Plan: added test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D2916
2021-07-15 21:51:05 -04:00
George Gevoian
e5eeb3ec80 (core) Add 'user' variable to trigger formulas
Summary:
The 'user' variable has a similar API to the one from access rules: it
contains properties about a user, such as their full name and email
address, as well as optional, user-defined attributes that are populated
via user attribute tables.

Test Plan: Python unit tests.

Reviewers: alexmojaki, paulfitz, dsagal

Reviewed By: alexmojaki, dsagal

Subscribers: paulfitz, dsagal, alexmojaki

Differential Revision: https://phab.getgrist.com/D2898
2021-07-15 15:18:32 -07:00
Dmitry S
6c114ef439 (core) Fix session handling when redirected to login when visiting a doc on a team site
Summary:
When redirecting to login, it's important to have a valid session set. This was
done by middleware that only applies to home pages. We need to set session to
live when redirecting in case of doc pages too.

Test Plan: Added a test case for fixed behavior by applying an existing case to doc pages too

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2915
2021-07-15 17:51:16 -04:00
Paul Fitzpatrick
6e15d44cf6 (core) start applying defenses for untrusted document uploads
Summary:
This applies some mitigations suggested by SQLite authors when
opening untrusted SQLite databases, as we do when Grist docs
are uploaded by the user.  See:
  https://www.sqlite.org/security.html#untrusted_sqlite_database_files

Steps implemented in this diff are:
  * Setting `trusted_schema` to off
  * Running a SQLite-level integrity check on uploads

Other steps will require updates to our node-sqlite3 fork, since they
are not available via the node-sqlite3 api (one more reason to migrate
to better-sqlite3).

I haven't yet managed to create a file that triggers an integrity
check failure without also being detected as corruption by sqlite
at a more basic level, so that is a TODO for testing.

Test Plan:
existing tests pass; need to come up with exploits to
actually test the defences and have not yet

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2909
2021-07-14 18:34:27 -04:00
Jarosław Sadziński
625fce5f65 (core) Refactoring google drive plugin
Summary:
Finishing implementation for google drive plugin.
- Refactoring plugin code to make it more robust and to follow grist ux
- Changing the way server hosts untrusted user content, from different domain to different port

Test Plan: Browser tests

Reviewers: dsagal, paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2881
2021-07-14 09:52:04 +02:00
George Gevoian
9592e3610b (core) Add 'value' to trigger formula autocomplete
Summary:
API signature for autocomplete updated to add column ID, which is
necessary for exposing correct types for 'value'.

Test Plan: Unit tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: jarek, alexmojaki

Differential Revision: https://phab.getgrist.com/D2896
2021-07-12 15:07:16 -07:00
Dmitry S
869b2f00ec (core) Remove LoginSession, which was mainly serving situations that are no longer used.
Summary:
In the past, Cognito sign-ins were intended to give authorization to some AWS
services (like SQS); various tokens were stored in the session for this
purpose. This is no longer used. Profiles from Cognito now serve a limited
purpose: first-time initialization of name and picture, and keeping track of
which login method was used. For these remaining needs, ScopedSession is
sufficient.

Test Plan:
Existing test pass. Tested manually that logins work with Google and
Email + Password. Tested manually that on a clean database, name and picture
are picked up from a Google Login.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2907
2021-07-12 13:04:00 -04:00
Dmitry S
f079ffdcb3 (core) Fix a log message about when a doc will close to be more accurate
Test Plan: Checked manually for a long-opening document that the time reported is correct.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2906
2021-07-12 09:51:54 -04:00
Paul Fitzpatrick
4222f1ed32 (core) communicate with sandbox via standard pipes
Summary:
This switches to using stdin/stdout for RPC calls to the sandbox, rather than specially allocated side channels. Plain text error information remains on stderr.

The motivation for the change is to simplify use of sandboxes, some of which support extra file descriptors and some of which don't.

The new style of communication is made the default, but I'm not committed to this, just that it be easy to switch to if needed. It is possible I'll need to switch the communication method again in the near future.

One reason not to make this default would be windows support, which is likely broken since stdin/stdout are by default in text mode.

Test Plan: existing tests pass

Reviewers: dsagal, alexmojaki

Reviewed By: dsagal, alexmojaki

Differential Revision: https://phab.getgrist.com/D2897
2021-07-12 06:45:47 -04:00
Alex Hall
ea01ca814d (core) Remove a bunch of dead code
Summary: Removed test/aws/, most of app/server/lib/, 3 dirs in app/lambda/, corresponding tests, and more!

Test Plan: a lot of this is quite the opposite...

Reviewers: dsagal, paulfitz

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2894
2021-07-01 18:38:21 +02:00
Alex Hall
84ddbc448b (core) Add test_replay for easily replaying data sent to the sandbox purely within python
Summary:
Run JS with a value for SANDBOX_BUFFERS_DIR, then run test_replay in python with the same value to replay just the python code.

See test_replay.py for more info.

Test Plan:
Record some data, e.g. `SANDBOX_BUFFERS_DIR=manual npm start` or `SANDBOX_BUFFERS_DIR=server ./test/testrun.sh server`.

Then run `SANDBOX_BUFFERS_DIR=server python -m unittest test_replay` from within `core/sandbox/grist` to replay the input from the JS.

Sample of the output will look like this:

```
Checking /tmp/sandbox_buffers/server/2021-06-16T15:13:59.958Z
True
Checking /tmp/sandbox_buffers/server/2021-06-16T15:16:37.170Z
True
Checking /tmp/sandbox_buffers/server/2021-06-16T15:14:22.378Z
True
```

Reviewers: paulfitz, dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2866
2021-06-30 16:56:09 +02:00
Paul Fitzpatrick
dca3abec1d (core) complete light sign-up flow for appsumo, and customize summaries
Summary:
Current appsumo sign-up flow doesn't reach the billing pages.
This diff nudges user on through that extra step.

It also tweaks plan summaries to say what special appsumo
features are in effect (member count prepaid for).

Test Plan: manual

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2882
2021-06-25 14:13:13 -04:00
Paul Fitzpatrick
36d5e7870e (core) streamline registration flow for new appsumo users
Summary:
This adds a new landing page for cognito sign-up, intended for
use by new appsumo users.

Their email address is pre-filled and locked down, and sign-up
is by entering a password.

The page is very crude compared to hosted cognito - especially
in error reporting! - but having the address filled in more
than makes up for that.

The flow does not quite connect with the new billing signup.
I think we can do that through the regular "welcome" process,
which will list the user's team site.  When the user visits
that site, we could detect that we are on a site with no
domain set yet and for which the user is a billing manager,
and trigger a visit to the appropriate billing page.

Test Plan: manual - hard to test through cognito email step

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2880
2021-06-25 10:47:10 -04:00
Alex Hall
305b133c59 (core) Remaining Python 3 compatibility changes
Summary: Biggest change is turning everything to unicode

Test Plan: The tests

Reviewers: dsagal, paulfitz

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2875
2021-06-25 12:00:58 +02:00
Dmitry S
6240fd6982 (core) Fix reporting of errors when saving cells by clicking away, and deduplicate memos.
Test Plan: Enhanced the test case for memos to check these cases too (fails without this fix).

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2876
2021-06-23 08:50:42 -04:00
Paul Fitzpatrick
cc04c6481a (core) add appsumo endpoints with stub implementations
Summary:
This adds appsumo /token and /notification endpoints, with some
tests.  The stub implementation is sufficient for AppSumo
activation to succeed (when exposed via port forwarding for testing).
It needs fleshing out:

 * Implement upgrade/downgrade/refund and stripe subscription.
 * Implement custom landing page and flow.

Test Plan: added tests

Reviewers: dsagal, georgegevoian

Reviewed By: dsagal

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D2864
2021-06-21 16:04:33 -04:00
Alex Hall
8a940676e9 (core) Generic tools for recording pycalls, deterministic mode.
Summary:
Replaces https://phab.getgrist.com/D2854

Refactoring of NSandbox:
- Simplify arguments to NSandbox.spawn. Only half the arguments were used depending on the flavour, adding a layer of confusion.
- Ensure the same environment variables are passed to both flavours of sandbox
- Simplify passing down environment variables.

Implement deterministic mode with libfaketime and a seeded random instance.
- Include static prebuilt libfaketime.so.1, may need another solution in future for other platforms.

Recording pycalls:
- Add script recordDocumentPyCalls.js to open a single document outside of tests.
- Refactor out recordPyCalls.ts to support various uses.
- Add afterEach hook to save all pycalls from server tests under $PYCALLS_DIR
- Make docTools usable without mocha.
- Add useLocalDoc and loadLocalDoc for loading non-fixture documents

Test Plan:
Made a document with formulas NOW() and UUID()
Compare two document openings in normal mode:

    diff <(test/recordDocumentPyCalls.js samples/d4W6NrzCMNVSVD6nWgNrGC.grist /dev/stdout) \
         <(test/recordDocumentPyCalls.js samples/d4W6NrzCMNVSVD6nWgNrGC.grist /dev/stdout)

Output:

    <                 1623407499.58132,
    ---
    >                 1623407499.60376,
    1195c1195
    <               "B": "bd2487f6-63c9-4f02-bbbc-5c0d674a2dc6"
    ---
    >               "B": "22e1a4fd-297f-4b86-91a2-bc42cc6da4b2"

`export DETERMINISTIC_MODE=1` and repeat. diff is empty!

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2857
2021-06-15 20:58:05 +02:00
George Gevoian
b94eb107d4 (core) Use individual choices for filtering choice lists
Test Plan: Wrote unit and browser tests that verify new behavior.

Reviewers: paulfitz, dsagal

Reviewed By: dsagal

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D2855
2021-06-11 09:34:06 -07:00
Paul Fitzpatrick
6f02987d10 (core) allow undos to be partial, if access control prohibits some part of them
Summary:
This is a somewhat experimental change, that will implement permitted parts of an undo if not all parts are permitted.  This is in preparation for trigger columns, where it may become common for a change in a record resulting in a change to an automatic change to another that the user cannot edit directly.  How to undo such an action is somewhat unclear.  One option is to undo the permitted parts, and then the triggers can rerun.

The general case is a bit of a can of worms, and feels adjacent to merging/rebasing etc.

Oh: it would probably be important in general to communicate to the user that an undo was partial, but this diff doesn't do that.  It would need some new plumbing.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2839
2021-06-10 18:26:11 -04:00
Alex Hall
2f3a0e0c7f (core) Showing censored values as a grey cell
Test Plan: Block read access to column A based on the condition rec.B == 1. Then setting B = 1 in a row makes the cell under A grey.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz, dsagal

Differential Revision: https://phab.getgrist.com/D2828
2021-06-07 13:11:41 +02:00
Paul Fitzpatrick
29c2b35dcc (core) speed up a step in document deletion
Summary:
The `_repairWorkspaceGuests` method is slow for workspaces with large numbers of documents.  It makes a query that produces a lot of rows.  The query itself is tolerable, but TypeORM processing uses enough CPU to be a likely culprit in some production instability.  This diff splits the query into two pieces that are logically independent, but which when combined were resulting in the number of rows being the product of the two pieces.  Once split, there is also a where clause that can be applied to one of the pieces.

The purpose of the method is to add every user that a document within a workspace is shared with to a "guest" group of the workspace itself.  The design of "guest" groups is not ideal, but this diff leaves the design unchanged and is intended only to speed up operation.

Made some small tweaks to the timing of a flakey test, and temporarily recreated the `samples` directory removed in a previous diff (this is currently breaking tests badly on a fresh worker without a `samples` directory lying around)

Test Plan: added test; existing tests pass

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D2844
2021-06-02 16:06:26 -04:00
Paul Fitzpatrick
c6265335af (core) make SELF_HYPERLINK urls cleaner
Summary:
This cleans up a few things about SELF_HYPERLINK urls:

 * Use `urlId` rather than `docId`.
 * Correctly merge personal org subdomain.
 * In dev environment, use clearer port number.

Test Plan: updated test

Reviewers: alexmojaki, dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2841
2021-05-28 08:54:37 -04:00
Paul Fitzpatrick
37698f9cb5 (core) apply access control to code view
Summary:
Names of private tables and columns were leaking via Code View.
This plugs that leak.

Test Plan: adds test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2840
2021-05-27 17:52:23 -04:00