Commit Graph

101 Commits

Author SHA1 Message Date
Alex Hall
0de0cb0f4a (core) Add PUT /records DocApi endpoint to AddOrUpdate records
Summary:
As designed in https://grist.quip.com/fZSrAnJKgO5j/Add-or-Update-Records-API

Current `POST /records` adds records, and `PATCH /records` updates them by row ID. This adds `PUT /records` to 'upsert' records, applying the AddOrUpdate user action. PUT was chosen because it's idempotent. Using a separate method (instead of inferring based on the request body) also cleanly separates validation, documentation, etc.

The name `require` for the new property was suggested by Paul because `where` isn't very clear when adding records.

Test Plan: New DocApi tests

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3251
2022-02-12 09:44:34 +02:00
Paul Fitzpatrick
4890a1fe89 (core) add free team site product
Summary:
This adds a Feature object that is an approximation of what we
plan for free team sites. It includes restrictions that are not
yet implemented, and an endpoint for testing.

Test Plan: added a test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3243
2022-02-04 09:00:51 -05:00
Paul Fitzpatrick
648cb67ee8 (core) freshen grist-admin doc-info and add a test for it
Summary:
The ./test/grist-admin doc-info command line tool was out of date
and not showing user access correctly anymore. This freshens the tool and
adds a small test for it.

Test Plan: Added test.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3182
2021-12-10 13:07:34 -05:00
Jarosław Sadziński
53bdd6c8e1 (core) Exposing more descriptive errors from exports
Summary:
Exports used to show generic message on error.
Adding error description to the message.

Test Plan: Updated tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3157
2021-11-30 17:26:32 +01:00
Paul Fitzpatrick
10a4cbb6bd (core) make document assignment endpoint available via /housekeeping api
Summary:
The /assign endpoint checks if a document is on the desired worker
and moves it if not. This is never done under regular operation, but
is useful when quarantining a misbehaving document.

The endpoint was failing to operate correctly if the requester did
not have access to the document. This diff makes the endpoint
accessible through a /housekeeping route, using the same pattern as
the /force-reload endpoint.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3109
2021-11-04 16:14:21 -04:00
Paul Fitzpatrick
35e18cc0ad (core) fix bug where sharing doc with everyone@ as viewer made it unlisted for site viewers
Summary:
Shares of the same role (e.g. viewer) at different levels could interact for a resource (e.g. a doc) shared with everyone@, potentially blocking the listing of that resource. This diff removes the interaction.

The permission of a user on a resource is calculated by finding all acl rules that link that resource to a group to which the user belongs, or to a group that has a subgroup to which the user belongs, etc, and then bitwise-or-ing the permissions on the acl rules. A later wrinkle was to allow public sharing via special users. A still later wrinkle was to avoid listing resources if they were only shared with the special everyone@ user, while allowing access to them if user has their full link. That wrinkle had a bug, where if e.g. a doc were shared with everyone@ as a viewer, and the org the doc was in was shared with someone@ as a viewer, and the doc inherited the org permissions via a workspace, then that doc would end up not being listed.

The fix is straightforward enough, but needs different code for postgres and sqlite, and is a bit verbose because we unwrap subgroups to a few levels rather than doing recursion (which looks cleaner but was slower in benchmarks).

Test Plan: added test that fails without this fix

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3095
2021-10-28 12:48:31 -04:00
Paul Fitzpatrick
f7c9919120 (core) annotate shares listed in UserManager for documents
Summary:
This gives more guidance to users when editing document shares in the UserManager dialog.

  * For a document on a team site, any shares with team members are marked `Team member`.
  * Shares that count as external collaborators are marked for documents on a team or personal site as `collaborator` (personal site) or `outside collaborator` (team site).
  * Collaborators are marked `1 of 2`, `2 of 2`, and then `limit exceeded`.
  * On a team site, links are offered for each collaborator to add them to the team. The links lead to a prefilled dialog for managing team membership which can be confirmed immediately, allowing the user to continue without interruption.
  * On a personal site, for the last collaborator and beyond, a link is added for creating a team. This isn't seamless since creating a team involves billing etc.

There's a small unrelated tweak in tests to remove a confusing import from `test/browser` in `test/server`.

One thing I didn't get to is checking if owner of doc is owner of site. If they aren't, they may try to add a member and be denied at that point - it would be more polite to change messaging earlier for them.

Test Plan: added and updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3083
2021-10-25 14:29:19 -04:00
Paul Fitzpatrick
d635c97686 (core) flesh out "View As" feature
Summary:
The users shown by the "View As" button are now drawn from more sources:
 * There are users the document is shared with. This has been rationalized, the behavior was somewhat erratic. If the user is not an owner of the document, the only user of this kind that will be listed is themselves.
 * There are users mentioned in any user attribute table keyed by Email. If name and access columns are present, those are respected, otherwise name is taken from email and access is set to "editors".
 * There are example users provided if there are not many other users available.

Test Plan: added and extended tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3045
2021-10-08 12:00:40 -04:00
Paul Fitzpatrick
383b8ffbf0 (core) add a tool for deleting a user
Summary:
This adds a `user:delete` target to the `cli.sh` tool. The desired user will be deleted from our database, from sendgrid, and from cognito.

There is code for scrubbing the user from team sites, but it isn't yet activated, I'm leaving finalizing and writing tests for it for follow-up.

Test Plan: tested manually

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3043
2021-09-29 12:08:23 -04:00
Alex Hall
3c4d71aeca (core) Initial webhooks implementation
Summary:
See https://grist.quip.com/VKd3ASF99ezD/Outgoing-Webhooks

- 2 new DocApi endpoints: _subscribe and _unsubscribe, not meant to be user friendly or publicly documented. _unsubscribe should be given the response from _subscribe in the body, e.g:

```
$ curl -X POST -H "Authorization: Bearer 8fd4dc59ecb05ab29ae5a183c03101319b8e6ca9" "http://localhost:8080/api/docs/6WYa23FqWxGNe3AR6DLjCJ/tables/Table2/_subscribe" -H "Content-type: application/json" -d '{"url": "https://webhook.site/a916b526-8afc-46e6-aa8f-a625d0d83ec3", "eventTypes": ["add"], "isReadyColumn": "C"}'
{"unsubscribeKey":"3246f158-55b5-4fc7-baa5-093b75ffa86c","triggerId":2,"webhookId":"853b4bfa-9d39-4639-aa33-7d45354903c0"}
$ curl -X POST -H "Authorization: Bearer 8fd4dc59ecb05ab29ae5a183c03101319b8e6ca9" "http://localhost:8080/api/docs/6WYa23FqWxGNe3AR6DLjCJ/tables/Table2/_unsubscribe" -H "Content-type: application/json" -d '{"unsubscribeKey":"3246f158-55b5-4fc7-baa5-093b75ffa86c","triggerId":2,"webhookId":"853b4bfa-9d39-4639-aa33-7d45354903c0"}'
{"success":true}
```

- New DB entity Secret to hold the webhook URL and unsubscribe key
- New document metatable _grist_Triggers subscribes to table changes and points to a secret to use for a webhook
- New file Triggers.ts processes action summaries and uses the two new tables to send webhooks.
- Also went on a bit of a diversion and made a typesafe subclass of TableData for metatables.

I think this is essentially good enough for a first diff, to keep the diffs manageable and to talk about the overall structure. Future diffs can add tests and more robustness using redis etc. After this diff I can also start building the Zapier integration privately.

Test Plan: Tested manually: see curl commands in summary for an example. Payloads can be seen in https://webhook.site/#!/a916b526-8afc-46e6-aa8f-a625d0d83ec3/0b9fe335-33f7-49fe-b90b-2db5ba53382d/1 . Great site for testing webhooks btw.

Reviewers: dsagal, paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3019
2021-09-23 14:35:39 +02:00
Paul Fitzpatrick
3e5a292cde (core) add tests for site deletion
Summary: This tests site deletion with and without a plan.

Test Plan: adding tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3017
2021-09-14 10:03:18 -04:00
Paul Fitzpatrick
ddcd08e147 (core) add a cli tool for deleting sites
Summary:
This adds a `site:delete` target to `cli.sh` for deleting sites. Sites should be specified by numeric org id, and for confirmation their name also needs to be given.

All the docs in the site are deleted permanently, and the workspaces, and the site, and the stripe customer (if any).

Test Plan: manual

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3015
2021-09-10 11:02:26 -04:00
George Gevoian
0717ee627e (core) Relocate export urls to /download/
Summary:
Moves CSV and XLSX export urls under /download/, and
removes the document title query parameter which is now
retrieved from the backend.

Test Plan: No new tests. Existing tests that verify endpoints still function.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3010
2021-09-02 09:36:33 -07:00
George Gevoian
ef5da42378 (core) Update export CSV and Excel endpoints
Summary:
The endpoints for exporting CSV and Excel are now under
/api/docs/:docId/ and are forwarded to a doc worker for export.

The Share Menu has been updated to use the new endpoints.

Test Plan: No new tests. Existing tests that verify endpoints work correctly.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3007
2021-08-31 10:47:24 -07:00
Paul Fitzpatrick
8b1ad588e9 (core) make treatment of emails consistent across /access endpoints
Summary:
Access endpoints were supposed to provide display versions of emails,
but in fact only the org endpoint was doing so.  This brings the
workspaces and docs endpoints into line, and adds tests.

Full user information is tweaked slightly to return an anonymous
flag only when anonymous.  This was already anticipated in the
FullUser type.

Test Plan: extended test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2999
2021-08-27 09:45:25 -04:00
Paul Fitzpatrick
f9630b3aa4 (core) clean up a collection of small problems affecting grist-core
Summary:
 * Remove adjustSession hack, interfering with loading docs under saml.
 * Allow the anonymous user to receive an empty list of workspaces for
   the merged org.
 * Behave better on first page load when org is in path - this used to
   fail because of lack of cookie.  This is very visible in grist-core,
   as a failure to load localhost:8484 on first visit.
 * Mark cookie explicitly as SameSite=Lax to remove a warning in firefox.
 * Make errorPages available in grist-core.

This changes the default behavior of grist-core to now start off in
anonymous mode, with an explicit sign-in step available.  If SAML is not configured,
the sign-in operation will unconditionally sign the user in as a default
user, without any password check or other security.  The user email is
taken from GRIST_DEFAULT_EMAIL if set.  This is a significant change, but
makes anonymous mode available in grist-core (which is convenient
for testing) and makes behavior with and without SAML much more consistent.

Test Plan: updated test; manual (time to start adding grist-core tests though!)

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2980
2021-08-17 21:44:50 -04:00
Paul Fitzpatrick
54beaede84 (core) revive saml support and test against Auth0
Summary:
SAML support had broken due to SameSite changes in browsers. This
makes it work again, and tests it against Auth0 (now owned by Okta).

Logging in and out works.  The logged out state is confusing, and may
not be complete.  The "Add Account" menu item doesn't work.
But with this, an important part of self-hosting becomes easier.

SAML support works also in grist-core, for site pages, but there
is a glitch on document pages that I'll look into separately.

Test Plan: tested manually

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2976
2021-08-16 17:36:09 -04:00
Dmitry S
1605e18f66 (core) Enable auto triggering of Welcome Tour, and various improvements.
Summary:
- Add showGristTour preference, and trigger tour automatically.
- Tour is only triggered for new and anonymous users on a personal org, with
  edit permission.

- Automatically open the right panel at tour start.
- Don't show tours on mobile, since that's not ready (popups are cut off
  and can't be dismissed)
- Cancel previous tour if a new one is somehow started.
- Remove #repeat- trigger hash tags from the URL when the tour starts.
- Ensure Help Center popup is positioned even when left panel is collapsed.
- Polish up the content of the last two cards in the tour.

Test Plan: Added test case for triggering and opening right panel.

Reviewers: alexmojaki, paulfitz

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D2955
2021-07-30 14:17:54 -04:00
George Gevoian
24fc3a2d00 (core) Redesign examples and templates UI
Summary:
The old Examples and Templates workspace is now
a page that pulls templates from a new public Grist Templates org.
The All Documents view will pull featured templates from that org, where
featured templates are simply pinned documents in Grist Templates. The
Examples and Templates page will also show the featured templates, as
well as the rest of the available templates organized by category. The
categories are equivalent to workspaces in Grist Templates, and are
generated dynamically.

Test Plan: Browser tests.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz, dsagal

Subscribers: dsagal, paulfitz, jarek

Differential Revision: https://phab.getgrist.com/D2930
2021-07-28 12:29:03 -07:00
Jarosław Sadziński
08295a696b (core) Export to Excel and Send to drive
Summary:
Implementing export to excel and send to Google Drive feature.

As part of this feature few things were implemented:
- Server side google authentication exposed on url: (docs, docs-s, or localhost:8080)/auth/google
- Exporting grist documents as an excel file (xlsx)
- Storing exported grist document (in excel format) in Google Drive as a spreadsheet document.

Server side google authentication requires one new environmental variables
- GOOGLE_CLIENT_SECRET (required) used by authentication handler

Test Plan: Browser tests for exporting to excel.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2924
2021-07-21 16:36:00 +02:00
Paul Fitzpatrick
1af99e9567 (core) link AppSumo activations with stripe, and support upgrades/downgrades
Summary:
This links AppSumo sign-ups with Stripe subscriptions
and our billing pages. Different AppSumo tiers are supported by
different coupons on the standard plan. Configuration of this
is in stripe, and then cached in the database.

The front end is tweaked just enough to make completing a sign-up
possible. It is not yet friendly.

Not covered includes:
 * Streamlining landing page.
 * Making billing pages git clearer summaries of AppSumo states.
 * Making flow through Cognito as graceful as possible - default
   probably doesn't meet AppSumo requirements.
 * Disabling site on cancellation/refund.
 * Downgrades when more seats in use than lower tier allows.

Test Plan: api-level tests added. No front-end tests yet.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2878
2021-06-24 10:18:42 -04:00
Paul Fitzpatrick
cc04c6481a (core) add appsumo endpoints with stub implementations
Summary:
This adds appsumo /token and /notification endpoints, with some
tests.  The stub implementation is sufficient for AppSumo
activation to succeed (when exposed via port forwarding for testing).
It needs fleshing out:

 * Implement upgrade/downgrade/refund and stripe subscription.
 * Implement custom landing page and flow.

Test Plan: added tests

Reviewers: dsagal, georgegevoian

Reviewed By: dsagal

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D2864
2021-06-21 16:04:33 -04:00
Paul Fitzpatrick
2feef7f780 (core) avoid typeorm's .save() method for relation with multi-column primary key
Summary:
A recently added stress test ("deletes documents reasonably quickly" in removedAt.ts) is sporadically failing under postgres.  It looks like typeorm's .save() method is in some way unreliable when setting a table with multi-column primary keys, via a ManyToMany relation. This diff replaces the .save() with explicit inserts/deletes.

I modified _repairWorkspaceGuests recently, so thought that change might have been the problem. However under the stress test, failures occur as often in _repairOrgGuests (not changed recently) as in _repairWorkspaceGuests (changed recently).

For reference, see schema diagram at https://grist.quip.com/wWpRAMe058Nl/Home-DB (the table being updated is `group_users`).

Possibly related issue: https://github.com/typeorm/typeorm/issues/4122

Test Plan:
After this change, stress test runs well on postgres locally (no failure 70 iterations); before it would fail on postgres within 3 iterations typically.

Separately: I gave a test that failed a little more time to return, and confirmed it was no slower on average, so I think it was unrelated.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D2848
2021-06-04 13:58:22 -04:00
Paul Fitzpatrick
29c2b35dcc (core) speed up a step in document deletion
Summary:
The `_repairWorkspaceGuests` method is slow for workspaces with large numbers of documents.  It makes a query that produces a lot of rows.  The query itself is tolerable, but TypeORM processing uses enough CPU to be a likely culprit in some production instability.  This diff splits the query into two pieces that are logically independent, but which when combined were resulting in the number of rows being the product of the two pieces.  Once split, there is also a where clause that can be applied to one of the pieces.

The purpose of the method is to add every user that a document within a workspace is shared with to a "guest" group of the workspace itself.  The design of "guest" groups is not ideal, but this diff leaves the design unchanged and is intended only to speed up operation.

Made some small tweaks to the timing of a flakey test, and temporarily recreated the `samples` directory removed in a previous diff (this is currently breaking tests badly on a fresh worker without a `samples` directory lying around)

Test Plan: added test; existing tests pass

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D2844
2021-06-02 16:06:26 -04:00
Paul Fitzpatrick
c6265335af (core) make SELF_HYPERLINK urls cleaner
Summary:
This cleans up a few things about SELF_HYPERLINK urls:

 * Use `urlId` rather than `docId`.
 * Correctly merge personal org subdomain.
 * In dev environment, use clearer port number.

Test Plan: updated test

Reviewers: alexmojaki, dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2841
2021-05-28 08:54:37 -04:00
Dmitry S
d1c1416d78 (core) Add rules to eslint to better match our coding conventions.
Summary:
We used tslint earlier, and on switching to eslint, some rules were not
transfered. This moves more rules over, for consistent conventions or helpful
warnings.

- Name private members with a leading underscore.
- Prefer interface over a type alias.
- Use consistent spacing around ':' in type annotations.
- Use consistent spacing around braces of code blocks.
- Use semicolons consistently at the ends of statements.
- Use braces around even one-liner blocks, like conditionals and loops.
- Warn about shadowed variables.

Test Plan: Fixed all new warnings. Should be no behavior changes in code.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2831
2021-05-24 12:56:18 -04:00
Paul Fitzpatrick
d0d3d3d0c9 (core) discount indirect changes for access control purposes
Summary:
This diff discounts indirect changes for access control purposes.  A UserAction that updates a cell A, which in turn causes changes in other dependent cells, will be considered a change to cell A for access control purposes.

The `engine.apply_user_actions` method now returns a `direct` array, with a boolean for each `stored` action, set to `true` if the action is attributed to the user or `false` if it is attributed to the engine.  `GranularAccess` ignores actions attributed to the engine when checking for edit rights.

Subtleties:
 * Removal of references to a removed row are considered direct changes.
 * Doesn't play well with undos as yet.  An action that indirectly modifies a cell the user doesn't have rights to may succeed, but it will not be reversible.

Test Plan: added tests, updated tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2806
2021-05-12 11:26:21 -04:00
Dmitry S
526b0ad33e (core) Configure more comprehensive eslint rules for Typescript
Summary:
- Update rules to be more like we've had with tslint
- Switch tsserver plugin to eslint (tsserver makes for a much faster way to lint in editors)
- Apply suggested auto-fixes
- Fix all lint errors and warnings in core/, app/, test/

Test Plan: Some behavior may change subtly (e.g. added missing awaits), relying on existing tests to catch problems.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2785
2021-04-26 18:54:55 -04:00
Paul Fitzpatrick
9d1bc5a518 (core) make AccessRules and FullCopies effective
Summary:
This allows `*SPECIAL:AccessRules` to give read access to the access rules to more users, and `*SPECIAL:FullCopies` to grant download/copy rights to more users.

This diff also changes forks to be owned by the user who forked them (previously they were an editor), since that feels more natural.

Test Plan: Added and updated tests.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2760
2021-03-25 15:05:26 -04:00
Paul Fitzpatrick
0c5f7cf0a7 (core) add SELF_HYPERLINK() function for generating links to the current document
Summary:
 * Adds a `SELF_HYPERLINK()` python function, with optional keyword arguments to set a label, the page, and link parameters.
 * Adds a `UUID()` python function, since using python's uuid.uuidv4 hits a problem accessing /dev/urandom in the sandbox.  UUID makes no particular quality claims since it doesn't use an audited implementation.  A difficult to guess code is convenient for some use cases that `SELF_HYPERLINK()` enables.

The canonical URL for a document is mutable, but older versions generally forward.  So for implementation simplicity the document url is passed it on sandbox creation and remains fixed throughout the lifetime of the sandbox.  This could and should be improved in future.

The URL is passed into the sandbox as a `DOC_URL` environment variable.

The code for creating the URL is factored out of `Notifier.ts`. Since the url is a function of the organization as well as the document, some rejiggering is needed to make that information available to DocManager.

On document imports, the new document is registered in the database slightly earlier now, in order to keep the procedure for constructing the URL in different starting conditions more homogeneous.

Test Plan: updated test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2759
2021-03-18 19:37:07 -04:00
Paul Fitzpatrick
587da4db97 (core) provide a way for an administrator to force reload of a document
Summary:
Adds POST /api/housekeeping/docs/:docId/force-reload, which allows the support user to force a document to reload (even if they don't otherwise have access to the document).

This could be done without a separate endpoint, but that turned out a bit messy, and there's some advantage to quarantining the exceptional authorization somewhere it could be easily elaborated.

Test Plan: adds test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2713
2021-01-25 14:16:53 -05:00
Paul Fitzpatrick
3ad9b18ddf (core) allow a doc owner to test access as a different user
Summary:
This adds back-end support for query parameters `aclAsUser_` and
`aclAsUserId_` which, when either is present, direct Grist to
process granular access control rules from the point of view
of that user (specified by email or id respectively).

Some front end support is added, in the form of a tag that
shows up when in this mode, and a way to cancel the mode.
No friendly way to initiate the mode is offered yet.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2704
2021-01-15 18:45:57 -05:00
Paul Fitzpatrick
438f259687 (core) start reconciling forking with granular access
Summary:
This allows a fork to be made by a user if:
 * That user is an owner of the document being forked, or
 * That user has full read access to the document being forked.

The bulk of the diff is reorganization of how forking is done.  ActiveDoc.fork is now responsible for creating a fork, not just a docId/urlId for the fork. Since fork creation should not be limited to the doc worker hosting the trunk, a helper endpoint is added for placing the fork.

The change required sanitizing worker allocation a bit, and allowed session knowledge to be removed from HostedStorageManager.

Test Plan: Added test; existing tests pass.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2700
2021-01-12 14:08:49 -05:00
Paul Fitzpatrick
68a682f876 (core) add housekeeping endpoints for cleaning doc snapshots+state
Summary:
This adds endpoints that allow the support user to remove unlisted
snapshots for a document, and to remove all action history for
a document.

This does increase what the support user can do, but not in a way
that would be particularly valuable to attack.  It would have some
destructive value, for removing history (removing unlisted
snapshots doesn't impact the user, by contrast).

This would simplify some maintenance operations.

Test Plan: added test for snapshots; tested states manually

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2699
2021-01-05 10:31:14 -05:00
Paul Fitzpatrick
3b3ae87ade (core) implement a safe mode for opening documents with rule problems
Summary:
Adds an "enter safe mode" option and explanation in modal that appears when a document fails to load, if user is owner. If "enter safe mode" is selected, document is reloaded on server in a special mode. Currently, the only difference is that if the acl rules fail to load, they are replaced with a fallback that grants full access to owners and no access to anyone else. An extra tag is shown to mark the document as safe mode, with an "x" for cancelling safe mode.

There are other ways a document could fail to load than just acl rules, so this is just a start.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2686
2020-12-14 13:04:13 -05:00
Paul Fitzpatrick
c1c17bf54e (core) distinguish open public documents from listing them
Summary:
getOrgWorkspaces and getWorkspaces had an unintended feature where
if a user had access to a workspace, they could list all publically
shared documents within that workspace.  This diff stops considering
resources shared with everyone@ when listing orgs or workspaces.
Resources shared with anon@ remain listed - this is how the example
workspace operates.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2671
2020-11-27 09:56:25 -05:00
Paul Fitzpatrick
d6ff1361cb (core) support GRIST_WORKER_GROUP to place worker into an exclusive group
Summary:
In an emergency, we may want to serve certain documents with "old" workers as we fix problems. This diff adds some support for that.

 * Creates duplicate task definitions and services for staging and production doc workers (called grist-docs-staging2 and grist-docs-prod2), pulling from distinct docker tags (staging2 and prod2). The services are set to have zero workers until we need them.
 * These new workers are started with a new env variable `GRIST_WORKER_GROUP` set to `secondary`.
 * The `GRIST_WORKER_GROUP` variable, if set, makes the worker available to documents in the named group, and only that group.
 * An unauthenticated `/assign` endpoint is added to documents which, when POSTed to, checks that the doc is served by a worker in the desired group for that doc (as set manually in redis), and if not frees the doc up for reassignment. This makes it possible to move individual docs between workers without redeployments.

The bash scripts added are a record of how the task definitions + services were created. The services could just have been copied manually, but the task definitions will need to be updated whenever the definitions for the main doc workers are updated, so it is worth scripting that.

For example, if a certain document were to fail on a new deployment of Grist, but rolling back the full deployment wasn't practical:
 * Set prod2 tag in docker to desired codebase for that document
 * Set desired_count for grist-docs-prod2 service to non-zero
 * Set doc-<docid>-group for that doc in redis to secondary
 * Hit /api/docs/<docid>/assign to move the doc to grist-docs-prod2

(If the document needs to be reverted to a previous snapshot, that currently would need doing manually - could be made simpler, but not in scope of this diff).

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2649
2020-11-02 15:46:46 -05:00
Paul Fitzpatrick
71519d9e5c (core) revamp snapshot inventory
Summary:
Deliberate changes:
 * save snapshots to s3 prior to migrations.
 * label migration snapshots in s3 metadata.
 * avoid pruning migration snapshots for a month.

Opportunistic changes:
 * Associate document timezone with snapshots, so pruning can respect timezones.
 * Associate actionHash/Num with snapshots.
 * Record time of last change in snapshots (rather than just s3 upload time, which could be a while later).

This ended up being a biggish change, because there was nowhere ideal to put tags (list of possibilities in diff).

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2646
2020-10-30 13:52:46 -04:00
Paul Fitzpatrick
c879393a8e (core) support adding user characteristic tables for granular ACLs
Summary:
This is a prototype for expanding the conditions that can be used in granular ACLs.

When processing ACLs, the following variables (called "characteristics") are now available in conditions:
 * UserID
 * Email
 * Name
 * Access (owners, editors, viewers)

The set of variables can be expanded by adding a "characteristic" clause.  This is a clause which specifies:
 * A tableId
 * The name of an existing characteristic
 * A colId
The effect of the clause is to expand the available characteristics with all the columns in the table, with values taken from the record where there is a match between the specified characteristic and the specified column.

Existing clauses are generalized somewhat to demonstrate and test the use these variables. That isn't the main point of this diff though, and I propose to leave generalizing+systematizing those clauses for a future diff.

Issues I'm not dealing with here:
 * How clauses combine.  (The scope on GranularAccessRowClause is a hack to save me worrying about that yet).
 * The full set of matching methods we'll allow.
 * Refreshing row access in clients when the tables mentioned in characteristic tables change.
 * Full CRUD permission control.
 * Default rules (part of combination).
 * Reporting errors in access rules.

That said, with this diff it is possible to e.g. assign a City to editors by their email address or name, and have only rows for those Cities be visible in their client. Ability to modify those rows, and remain updates about them, remains under incomplete control.

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2642
2020-10-19 13:33:47 -04:00
Dmitry S
986f469965 (core) Support "Public access" switch in the UI for docs.
Summary:
- The older "Anonymous access" option is renamed to "Public access" in the UI.
- This option was only present with GRIST_SUPPORT_ANON set.
- With GRIST_SUPPORT_ANON, the old behavior is kept: "Public access"
  option adds/removes anon@ user
- Without GRIST_SUPPORT_ANON (normal case), orgs/workspaces don't support
  "Public access" option. For documents, it adds/removes everyone@ user.

The latter is the main feature of interest. The GRIST_SUPPORT_ANON flag
is set for on-premise installs, and adds discoverability by anon users.

Test Plan: Added a test cases and checks in other tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2612
2020-09-18 19:14:33 -04:00
Dmitry S
166143557a (core) Show a welcome card when a user opens an example for the first time.
Summary:
- The card includes an image, a brief description, and a link to the tutorial.
- The left panel includes a link to the tutorial, and a button to reopen card.
- Card is collapsed and expanded with a little animation.
- Add a seenExamples pref for whether an example has been seen.
- Store the pref in localStorage for anon user.

Separately, added clearing of prefs of test users between tests, to avoid tests
affecting unrelated tests.

Test Plan: Added a browser test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2602
2020-09-09 23:08:50 -04:00
Dmitry S
0a5afd1f98 (core) Implement updated DocMenu UI: list/card mode and sort mode.
Summary:
- Add org-wide currentSort and currentView, saved as user preferences.
- Add per-workspace currentSort and currentView, backed by localStorage.
- Move localStorage-based observables to a separate file.
- Move hard-coded data about example docs to a separate file.
- Add UI for toggling sort and view mode.
- Removed unused features of buttonSelect to simplify it,
  and added support for light style of buttons.
- Added `parse` helper method to StringUnion, and use it in a few places where
  it simplifies code.
- Set `needRealOrg: true` in HomeDBManager.updateOrg() to fix saving prefs for
  mergedOrg.

Test Plan: WIP: Fixed some affected tests. New tests not yet written.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2587
2020-08-19 11:31:42 -04:00
Dmitry S
053d714655 (core) For getting access info, include the first-level doc and workspace users.
Summary:
When listing access on a doc or workspaces, include all users associated with
the resource or its parents.

Previously we only considered org-level users. This is normally sufficient
since doc and workspace users are automatically added as guests of the org. But
there are exceptions for special users (like everyone@), and generally, in case
of any divergence, it's important to list everyone who affects access
decisions.

Test Plan: Added a test that everyone@ user gets included in listings

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D2533
2020-08-05 00:26:04 -04:00
Paul Fitzpatrick
6b24d496db (core) add per-user per-org preferences to database
Summary:
Adds preferences to orgs.  There are a few flavors:
 * `userOrgPrefs`: these are specific to a certain user and a certain org.
 * `orgPrefs`: these are specific to a certain org, and apply to all users.
 * `userPrefs`: these are specific to a certain user, and apply to all orgs.

The three flavors of prefs are reported by `GET` for an org, and can be modified by `PATCH` for an org.  The user needs to have UPDATE rights to change `orgPrefs`, but can change `userOrgPrefs` and `userPrefs` without that right since the settings only affect themselves.

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2572
2020-08-04 15:20:13 -04:00
Paul Fitzpatrick
ee018ff183 (core) add more tests for inaccessible workspaces; fix doc count
Summary:
 * Checks that empty workspaces are listed correctly, including in
   cases where docs or workspaces have been made inaccessible to
   the user doing the listing.
 * Checks that when a document quota is in force, the count is
   correct, and not dependent on ACLs.
 * Fixes the document count used for document quotas, which in
   fact was not counting docs the current user did not have access
   to.

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2570
2020-07-31 12:55:21 -04:00
Paul Fitzpatrick
156b75133c (core) list inaccessible docs for editors/owners of workspaces
Summary:
This modifies the material listed in workspaces.  Previously,
material the user did not have access to was omitted.  Now, it
is included if the user has the right to delete the workspace.
This is to avoid scenarios where a user might try to delete a
workspace without being aware of the full consequences.

Test Plan: added tests; existing tests should pass

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2568
2020-07-30 23:05:15 -04:00
Dmitry S
2399baaca2 (core) When saving copies, allow saving to another org; update menus for making and saving copies.
Summary:
- Implemented selecting an org in some cases when using Save-Copy dialog.
- Unified previous 'Save Copy' menu into an enhanced "Share" menu.
- Renamed ExportMenu to ShareMenu, collect related code into it, and design the share button.
- Introduced trunkAccess property for forks, to know whether "Replace Original" is available.
- Simplified handling of fork() result, now that all code has been upgraded.
- Replaced 'Copy as Template' menu items with a checkbox in the Save-Copy dialog
- Removed copy links for examples in the DocMenu (to simplify, since not part of updated design)
- Updated the UI of the copying dialog.

Test Plan: Updated affected tests, added new test cases for copying when other orgs are a choice or not.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2561
2020-07-27 14:11:02 -04:00
Paul Fitzpatrick
b7b4b0229b (core) move some material to core that slipped through in a rebase
Summary:
This makes core independently buildable again, and adds a small
script to run as a sanity check.

Test Plan: checked that build_core.sh succeeds

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D2558
2020-07-23 11:29:05 -04:00
Paul Fitzpatrick
a27032df3e (core) add a test for unnecessary workspaces shown in trash, and fix issue
Summary:
Currently if I share a doc with a friend, and then soft-delete a doc
in the same workspace, that friend will see the workspace in their
trash (empty, but there).

This adds a test for the issue and resolves it by filtering out
docs at the sql level that used to be filtered out by javascript.

Test Plan: added test; existing tests pass

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2557
2020-07-23 11:04:33 -04:00
Dmitry S
671dc24214 (core) Allow the support user to access everyone's billing pages
Summary:
Give specialPermit to the support user for page loads and API requests needed
to serve billing pages.

Test Plan: Added new test cases

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2554
2020-07-22 15:45:47 -04:00
Paul Fitzpatrick
5ef889addd (core) move home server into core
Summary: This moves enough server material into core to run a home server.  The data engine is not yet incorporated (though in manual testing it works when ported).

Test Plan: existing tests pass

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2552
2020-07-21 20:39:10 -04:00