Commit Graph

62 Commits

Author SHA1 Message Date
Jarosław Sadziński
59942a23b6 (core) Limiting doc remove permission to owners.
Summary:
Guest editors added to a document were able to remove it. This limits this permission
by allowing only owners of a doc to delete it.

Test Plan: Updated

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal, anaisconce

Differential Revision: https://phab.getgrist.com/D3708
2022-11-30 23:37:18 +01:00
George Gevoian
efc3ba29d7 (core) Tweak autocomplete to only suggest team members
Summary: This changes the suggestions in the User Manager autocomplete.

Test Plan: Project tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3656
2022-10-18 08:08:05 -07:00
Jarosław Sadziński
bfd7243fe2 (core) Comments
Summary:
First iteration for comments system for Grist.
- Comments are stored in a generic metatable `_grist_Cells`
- Each comment is connected to a particular cell (hence the generic name of the table)
- Access level works naturally for records stored in this table
-- User can add/read comments for cells he can see
-- User can't update/remove comments that he doesn't own, but he can delete them by removing cells (rows/columns)
-- Anonymous users can't see comments at all.
- Each comment can have replies (but replies can't have more replies)

Comments are hidden by default, they can be enabled by COMMENTS=true env variable.
Some things for follow-up
- Avatars, currently the user's profile image is not shown or retrieved from the server
- Virtual rendering for comments list in creator panel. Currently, there is a limit of 200 comments.

Test Plan: New and existing tests

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3509
2022-10-17 13:38:21 +02:00
Paul Fitzpatrick
ab3cdb62ac (core) make the support account configurable, and allow listing public sites
Summary:
This makes two small tweaks based on a user's questions about sharing
sites publicly for a self-managed installation:

 * The support user `support@getgrist.com` is made configurable with
   `GRIST_SUPPORT_EMAIL`. This came up because only the support user
   can share material with the special "everyone" user. This restriction
   was added to avoid spam.
 * Regardless of public sharing settings, for our SaaS we had
   decided not to list public sites to anonymous users. That is
   somewhat a question of taste, so a `GRIST_LIST_PUBLIC_SITES` flag
   is added to override this choice.

Public sharing isn't in a well polished state, and this diff doesn't
advance that, in fact it adds a new wrinkle :-/

Test Plan: existing tests pass; manual testing

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3663
2022-10-14 14:23:59 -04:00
Jarosław Sadziński
9628253fd8 (core) Adding new column in users table "ref" with unique identifier.
Summary:
There is a new column in users table called ref (user reference).
It holds user's unique reference number that can be used for features
that require some kind of ownership logic (like comments).

Test Plan: Updated tests

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian, paulfitz

Differential Revision: https://phab.getgrist.com/D3641
2022-10-04 15:19:28 +02:00
Paul Fitzpatrick
d7b3fb972c (core) upgrade typeorm so we can support newer postgres
Summary:
upgrade typeorm version, so Grist can run against newer versions of postgres.

Dusted off some old benchmarking code to verify that important queries don't get slower. They don't appear to, unlike for some intermediate versions of typeorm I tried in the past.

Most of the changes are because `findOne` changed how it interprets its arguments, and the value it returns when nothing is found. For the return value, I stuck with limiting its impact by emulating old behavior (returning undefined rather than null) rather than propagating the change out to parts of the code unrelated to the database.

Test Plan: existing tests pass; manual testing with postgres 10 and 14

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3613
2022-09-02 15:34:21 -04:00
Jarosław Sadziński
ee109e9186 (core) Removing NEW_DEAL flag
Summary:
Removing NEW_DEAL flag checks and exposing all new deal features as default.
Also removing Pages.ts as it was moved to grist-core.

Test Plan: Existing and updated tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3572
2022-08-09 17:29:28 +02:00
George Gevoian
aeba738f7c (core) Add product for new personal plan
Summary:
Adds the new personal plan as a product that will be available
in the future. Can be enabled along with other plan-related via
an environment variable.

Test Plan: Browser tests and existing tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3533
2022-07-26 11:33:23 -07:00
Dmitry S
51ff72c15e (core) Faster builds all around.
Summary:
Building:
- Builds no longer wait for tsc for either client, server, or test targets. All use esbuild which is very fast.
- Build still runs tsc, but only to report errors. This may be turned off with `SKIP_TSC=1` env var.
- Grist-core continues to build using tsc.
- Esbuild requires ES6 module semantics. Typescript's esModuleInterop is turned
  on, so that tsc accepts and enforces correct usage.
- Client-side code is watched and bundled by webpack as before (using esbuild-loader)

Code changes:
- Imports must now follow ES6 semantics: `import * as X from ...` produces a
  module object; to import functions or class instances, use `import X from ...`.
- Everything is now built with isolatedModules flag. Some exports were updated for it.

Packages:
- Upgraded browserify dependency, and related packages (used for the distribution-building step).
- Building the distribution now uses esbuild's minification. babel-minify is no longer used.

Test Plan: Should have no behavior changes, existing tests should pass, and docker image should build too.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3506
2022-07-04 10:42:40 -04:00
Dmitry S
dd2eadc86e (core) Speed up and upgrade build.
Summary:
- Upgrades to build-related packages:
  - Upgrade typescript, related libraries and typings.
  - Upgrade webpack, eslint; add tsc-watch, node-dev, eslint_d.

- Build organization changes:
  - Build webpack from original typescript, transpiling only; with errors still
    reported by a background tsc watching process.

- Typescript-related changes:
  - Reduce imports of AWS dependencies (very noticeable speedup)
  - Avoid auto-loading global @types
  - Client code is now built with isolatedModules flag (for safe transpilation)
  - Use allowJs to avoid copying JS files manually.

- Linting changes
  - Enhance Arcanist ESLintLinter to run before/after commands, and set up to use eslint_d
  - Update eslint config, and include .eslintignore to avoid linting generated files.
  - Include a bunch of eslint-prompted and eslint-generated fixes
  - Add no-unused-expression rule to eslint, and fix a few warnings about it

- Other items:
  - Refactor cssInput to avoid circular dependency
  - Remove a bit of unused code, libraries, dependencies

Test Plan: No behavior changes, all existing tests pass. There are 30 tests fewer reported because `test_gpath.py` was removed (it's been unused for years)

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3498
2022-06-27 16:10:10 -04:00
Jarosław Sadziński
d92a761f6e (core) Product update popups and hosted stripe integration
Summary:
- Showing nudge to individual users to sign up for free team plan.
- Implementing billing page to upgrade from free team to pro.
- New modal with upgrade options and free team site signup.
- Integrating Stripe-hosted UI for checkout and plan management.

Test Plan: updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3456
2022-06-08 21:10:49 +02:00
Paul Fitzpatrick
b9a4b2b58f (core) add missing tsconfig file that affects IDEs
Summary:
I missed committing a file that is important for editing files comfortably in the ext directory in an IDE. This diff:

 * Adds tsconfig-base-ext.json - that was the only intended change
 * Unrelated: Forces all creation of connections to the home db through a new `getOrCreateConnection` method which changes the `busy_timeout` if using Sqlite. This was an attempt to fix random "database is locked" test failures. I believe multiple connections to the home db as an sqlite file do not happen in self-hosted Grist (where there is a single node process) or in our SaaS (where the database is in postgres). It does affect Grist started using `devServerMain.ts` (where multiple processes accessing same database are started) or various test configurations when extra database connections are opened.
 * Unrelated: I added a `busy_timeout` for session storage, when it uses Sqlite. Again, I don't believe this affects self-hosted Grist or our SaaS.
 * Tweaked a `BillingDiscount` test that looked perhaps vulnerable to a stripe request stalling.

I can't be sure my tweaks actually help, since I didn't succeed in replicating the failures. Update: looks like the "locked" error can still happen :(

Test Plan: manual

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3450
2022-05-27 13:48:58 -04:00
George Gevoian
a6063f570a (core) Polish Access Details
Summary:
Instead of showing a blank dialog for users whose access
is limited (e.g. public members), we now show the user's
role and a mention of whether their access is public.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3431
2022-05-19 09:45:26 -07:00
Jarosław Sadziński
0ab9e4a6a0 (core) Adding GristConnect login system
Summary:
New login system to allow simple SSO flow that is based on Discourse description that is available at:
https://meta.discourse.org/t/discourseconnect-official-single-sign-on-for-discourse-sso/13045

Test Plan: New core test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3418
2022-05-18 20:28:25 +02:00
Paul Fitzpatrick
e4d47a2f3c (core) add minimal support for activation keys
Summary: For grist-ee, expect an activation key in environment variable `GRIST_ACTIVATION` or in a file pointed to by `GRIST_ACTIVATION_FILE`. In absence of key, start a 30-day trial, during which a banner is shown. Once trial expires, installation goes into document-read-only mode.

Test Plan: added a test

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3426
2022-05-16 22:46:23 -04:00
George Gevoian
f48d579f64 (core) Add API endpoint to get site usage summary
Summary:
The summary includes a count of documents that are approaching
limits, in grace period, or delete-only. The endpoint is only accessible
to site owners, and is currently unused. A follow-up diff will add usage
banners to the site home page, which will use the response from the
endpoint to communicate usage information to owners.

Test Plan: Browser and server tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3420
2022-05-16 11:16:19 -07:00
Paul Fitzpatrick
20dd2fc70d (core) allow non-owners to remove themselves from sites/workspaces/docs
Summary:
For users who cannot otherwise change access to a resource, let
them remove themselves. Implemented via the standard endpoints
as a special exception that will process a request from a user
that would otherwise be denied, if the only contents of that
request are a removal of themselves.

Users who can change access are still not permitted to change their
own permissions or to remove themselves, as a precaution against
orphaning resources.

Test Plan: extended and updated tests

Reviewers: cyprien

Reviewed By: cyprien

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3367
2022-04-13 10:04:32 -04:00
George Gevoian
859c593448 (core) Add authSubject and authProvider to sessions
Summary:
This also updates Authorizer to link the authSubject
to Grist users if not previously linked. Linked subjects
are now used as the username for password-based logins,
instead of emails, which remain as a fallback.

Test Plan: Existing tests, and tested login flows manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3356
2022-04-11 11:42:02 -07:00
Paul Fitzpatrick
14f7e30e6f (core) add users.options.isConsultant flag, and omit such users from billing
Summary:
This adds an optional `isConsultant` flag to `users.options`, and an endpoint that allows the support user to turn it on or off. Users marked as consultants are not counted as billable members. Follows the example of existing `allowGoogleLogin` option.

Billable members are counted when members are added or removed from a site. Changing the `isConsultant` flag has no immediate or retroactive effect on billing. The number of users in stripe is now set unconditionally, rather than only when it has changed.

Notifications to billing managers are not aware of this billing nuance, but continue to report user counts that include consultants. The notifications link users to the billing page.

Test Plan: extended test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: anaisconce, jarek

Differential Revision: https://phab.getgrist.com/D3362
2022-04-11 10:26:31 -04:00
Alex Hall
59436d2bca (core) Grace period and delete-only mode when exceeding row limit
Summary:
Builds upon https://phab.getgrist.com/D3328

- Add HomeDB column `Document.gracePeriodStart`
- When the row count moves above the limit, set it to the current date. When it moves below, set it to null.
- Add DataLimitStatus type indicating if the document is approaching the limit, is in a grace period, or is in delete only mode if the grace period started at least 14 days ago. Compute it in ActiveDoc and send it to client when opening.
- Only allow certain user actions when in delete-only mode.

Follow-up tasks related to this diff:

- When DataLimitStatus in the client is non-empty, show a banner to the appropriate users.
- Only send DataLimitStatus to users with the appropriate access. There's no risk landing this now since real users will only see null until free team sites are released.
- Update DataLimitStatus immediately in the client when it changes, e.g. when user actions are applied or the product is changed. Right now it's only sent when the document loads.
- Update row limit, grace period start, and data limit status in ActiveDoc when the product changes, i.e. the user upgrades/downgrades.
- Account for data size when computing data limit status, not just row counts.

See also the tasks mentioned in https://phab.getgrist.com/D3331

Test Plan: Extended FreeTeam nbrowser test, testing the 4 statuses.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3331
2022-03-25 13:41:33 +02:00
Paul Fitzpatrick
134ae99e9a (core) add gvisor-based sandboxing to core
Summary:
This adds support for gvisor sandboxing in core. When Grist is run outside of a container, regular gvisor can be used (if on linux), and will run in rootless mode. When Grist is run inside a container, docker's default policy is insufficient for running gvisor, so a fork of gvisor is used that has less defence-in-depth but can run without privileges.

Sandboxing is automatically turned on in the Grist core container. It is not turned on automatically when built from source, since it is operating-system dependent.

This diff may break a complex method of testing Grist with gvisor on macs that I may have been the only person using. If anyone complains I'll find time on a mac to fix it :)

This diff includes a small "easter egg" to force document loads, primarily intended for developer use.

Test Plan: existing tests pass; checked that core and saas docker builds function

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3333
2022-03-24 17:04:49 -04:00
Alex Hall
546096fcc9 (core) Clean up and refactor uses of HomeDBManager.getDoc
Summary:
Firstly I just wanted some more consistency and less repetition in places where Documents are retrieved from the DB, so it's more obvious when code differs from the norm. Main changes for that part:

- Let HomeDBManager accept a `Request` directly and convert it to a `Scope`, and use this in a few places.
- `getScope` tries `req.docAuth.docId` if `req.params` doesn't have a docId.

I also refactored how `_createActiveDoc` gets the document URL, separating out getting the document from getting a URL for it. This is because I want to use that document object in a future diff, but I also just find it cleaner. Notable changes for that:

- Extracted a new method `HomeDBManager.getRawDocById` as an alternative to `getDoc` that's explicitly for when you only have a document ID.
- Removed the interface method `GristServer.getDocUrl` and its two implementations because it wasn't used elsewhere and it didn't really add anything on top of getting a doc (now done by `getRawDocById`) and `getResourceUrl`.
- Between `cachedDoc` and `getRawDocById` (which represent previously existing code paths) also try `getDoc(getScope(docSession.req))`, which is new, because it seems better to only `getRawDocById` as a last resort.

Test Plan: Existing tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3328
2022-03-24 13:42:36 +02:00
Alex Hall
ec8460b772 (core) Prune snapshots outside the window in product features
Summary:
- Add a method `getSnapshotWindow` to `IInventory` and `DocSnapshotInventory`. It returns a `SnapshotWindow`, which represents a duration of time for which we keep backups for a particular document.
- `DocSnapshotPruner` calls this method and passes the window to `shouldKeepSnapshots` to determine which document versions have fallen outside the window and should be pruned.
- The implementation passed to `DocSnapshotInventory` uses a new method `getDocProduct` in `HomeDBManager` which directly returns the `Product` associated with a document, given only the document ID. Other methods in `HomeDBManager` require passing more information, especially about a user, but `DocSnapshotPruner` only knows about document IDs.

Test Plan: Added a test for `getDocProduct` and a test for `DocSnapshotPruner` where `getSnapshotWindow` is specified.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3322
2022-03-18 18:48:14 +02:00
Paul Fitzpatrick
2563fb745a (core) make Grist easier to run with a single server
Summary:
This makes many small changes so that Grist is less fussy to run as a single instance behind a reverse proxy. Some users had difficulty with the self-connections Grist would make, due to internal network setup, and since these are unnecessary in any case in this scenario, they are now optimized away. Likewise some users had difficulties related to doc worker urls, which are now also optimized away. With these changes, users should be able to get a lot further on first try, at least far enough to open and edit documents.

The `GRIST_SINGLE_ORG` setting was proving a bit confusing, since it appeared to only work when set to `docs`. This diff
adds a check for whether the specified org exists, and if not, it creates it. This still depends on having a user email to make as the owner of the team, so there could be remaining difficulties there.

Test Plan: tested manually with nginx

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3299
2022-03-05 13:30:45 -05:00
George Gevoian
fa68b790bb (core) Remove code for unused welcome flows
Summary: Removes code that was marked for removal.

Test Plan: Existing tests still pass.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3289
2022-02-28 13:21:28 -08:00
George Gevoian
ff4e5d2769 (core) Send emails when 2FA settings are updated
Summary: When user 2FA status is changed, we now send out emails via SendGrid.

Test Plan: Server tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3280
2022-02-24 12:36:50 -08:00
George Gevoian
e264094412 (core) Add account page option to allow Google login
Summary:
Enabled by default, the new checkbox is only visible to
users logged in with email/password, and controls whether it is possible
to log in to the same account via a Google account
(with matching email). When disabled, CognitoClient will refuse logins
from Google if a Grist account with the same email exists.

Test Plan:
Server and browser tests for setting flag. Manual tests to verify
Cognito doesn't allow signing in with Google when flag is disabled.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3257
2022-02-14 16:56:23 -08:00
Paul Fitzpatrick
4890a1fe89 (core) add free team site product
Summary:
This adds a Feature object that is an approximation of what we
plan for free team sites. It includes restrictions that are not
yet implemented, and an endpoint for testing.

Test Plan: added a test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3243
2022-02-04 09:00:51 -05:00
Paul Fitzpatrick
648cb67ee8 (core) freshen grist-admin doc-info and add a test for it
Summary:
The ./test/grist-admin doc-info command line tool was out of date
and not showing user access correctly anymore. This freshens the tool and
adds a small test for it.

Test Plan: Added test.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3182
2021-12-10 13:07:34 -05:00
Paul Fitzpatrick
35e18cc0ad (core) fix bug where sharing doc with everyone@ as viewer made it unlisted for site viewers
Summary:
Shares of the same role (e.g. viewer) at different levels could interact for a resource (e.g. a doc) shared with everyone@, potentially blocking the listing of that resource. This diff removes the interaction.

The permission of a user on a resource is calculated by finding all acl rules that link that resource to a group to which the user belongs, or to a group that has a subgroup to which the user belongs, etc, and then bitwise-or-ing the permissions on the acl rules. A later wrinkle was to allow public sharing via special users. A still later wrinkle was to avoid listing resources if they were only shared with the special everyone@ user, while allowing access to them if user has their full link. That wrinkle had a bug, where if e.g. a doc were shared with everyone@ as a viewer, and the org the doc was in was shared with someone@ as a viewer, and the doc inherited the org permissions via a workspace, then that doc would end up not being listed.

The fix is straightforward enough, but needs different code for postgres and sqlite, and is a bit verbose because we unwrap subgroups to a few levels rather than doing recursion (which looks cleaner but was slower in benchmarks).

Test Plan: added test that fails without this fix

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3095
2021-10-28 12:48:31 -04:00
Paul Fitzpatrick
f7c9919120 (core) annotate shares listed in UserManager for documents
Summary:
This gives more guidance to users when editing document shares in the UserManager dialog.

  * For a document on a team site, any shares with team members are marked `Team member`.
  * Shares that count as external collaborators are marked for documents on a team or personal site as `collaborator` (personal site) or `outside collaborator` (team site).
  * Collaborators are marked `1 of 2`, `2 of 2`, and then `limit exceeded`.
  * On a team site, links are offered for each collaborator to add them to the team. The links lead to a prefilled dialog for managing team membership which can be confirmed immediately, allowing the user to continue without interruption.
  * On a personal site, for the last collaborator and beyond, a link is added for creating a team. This isn't seamless since creating a team involves billing etc.

There's a small unrelated tweak in tests to remove a confusing import from `test/browser` in `test/server`.

One thing I didn't get to is checking if owner of doc is owner of site. If they aren't, they may try to add a member and be denied at that point - it would be more polite to change messaging earlier for them.

Test Plan: added and updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3083
2021-10-25 14:29:19 -04:00
Paul Fitzpatrick
d635c97686 (core) flesh out "View As" feature
Summary:
The users shown by the "View As" button are now drawn from more sources:
 * There are users the document is shared with. This has been rationalized, the behavior was somewhat erratic. If the user is not an owner of the document, the only user of this kind that will be listed is themselves.
 * There are users mentioned in any user attribute table keyed by Email. If name and access columns are present, those are respected, otherwise name is taken from email and access is set to "editors".
 * There are example users provided if there are not many other users available.

Test Plan: added and extended tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3045
2021-10-08 12:00:40 -04:00
Paul Fitzpatrick
383b8ffbf0 (core) add a tool for deleting a user
Summary:
This adds a `user:delete` target to the `cli.sh` tool. The desired user will be deleted from our database, from sendgrid, and from cognito.

There is code for scrubbing the user from team sites, but it isn't yet activated, I'm leaving finalizing and writing tests for it for follow-up.

Test Plan: tested manually

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3043
2021-09-29 12:08:23 -04:00
Alex Hall
3c4d71aeca (core) Initial webhooks implementation
Summary:
See https://grist.quip.com/VKd3ASF99ezD/Outgoing-Webhooks

- 2 new DocApi endpoints: _subscribe and _unsubscribe, not meant to be user friendly or publicly documented. _unsubscribe should be given the response from _subscribe in the body, e.g:

```
$ curl -X POST -H "Authorization: Bearer 8fd4dc59ecb05ab29ae5a183c03101319b8e6ca9" "http://localhost:8080/api/docs/6WYa23FqWxGNe3AR6DLjCJ/tables/Table2/_subscribe" -H "Content-type: application/json" -d '{"url": "https://webhook.site/a916b526-8afc-46e6-aa8f-a625d0d83ec3", "eventTypes": ["add"], "isReadyColumn": "C"}'
{"unsubscribeKey":"3246f158-55b5-4fc7-baa5-093b75ffa86c","triggerId":2,"webhookId":"853b4bfa-9d39-4639-aa33-7d45354903c0"}
$ curl -X POST -H "Authorization: Bearer 8fd4dc59ecb05ab29ae5a183c03101319b8e6ca9" "http://localhost:8080/api/docs/6WYa23FqWxGNe3AR6DLjCJ/tables/Table2/_unsubscribe" -H "Content-type: application/json" -d '{"unsubscribeKey":"3246f158-55b5-4fc7-baa5-093b75ffa86c","triggerId":2,"webhookId":"853b4bfa-9d39-4639-aa33-7d45354903c0"}'
{"success":true}
```

- New DB entity Secret to hold the webhook URL and unsubscribe key
- New document metatable _grist_Triggers subscribes to table changes and points to a secret to use for a webhook
- New file Triggers.ts processes action summaries and uses the two new tables to send webhooks.
- Also went on a bit of a diversion and made a typesafe subclass of TableData for metatables.

I think this is essentially good enough for a first diff, to keep the diffs manageable and to talk about the overall structure. Future diffs can add tests and more robustness using redis etc. After this diff I can also start building the Zapier integration privately.

Test Plan: Tested manually: see curl commands in summary for an example. Payloads can be seen in https://webhook.site/#!/a916b526-8afc-46e6-aa8f-a625d0d83ec3/0b9fe335-33f7-49fe-b90b-2db5ba53382d/1 . Great site for testing webhooks btw.

Reviewers: dsagal, paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3019
2021-09-23 14:35:39 +02:00
Paul Fitzpatrick
3e5a292cde (core) add tests for site deletion
Summary: This tests site deletion with and without a plan.

Test Plan: adding tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3017
2021-09-14 10:03:18 -04:00
Paul Fitzpatrick
ddcd08e147 (core) add a cli tool for deleting sites
Summary:
This adds a `site:delete` target to `cli.sh` for deleting sites. Sites should be specified by numeric org id, and for confirmation their name also needs to be given.

All the docs in the site are deleted permanently, and the workspaces, and the site, and the stripe customer (if any).

Test Plan: manual

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3015
2021-09-10 11:02:26 -04:00
Paul Fitzpatrick
8b1ad588e9 (core) make treatment of emails consistent across /access endpoints
Summary:
Access endpoints were supposed to provide display versions of emails,
but in fact only the org endpoint was doing so.  This brings the
workspaces and docs endpoints into line, and adds tests.

Full user information is tweaked slightly to return an anonymous
flag only when anonymous.  This was already anticipated in the
FullUser type.

Test Plan: extended test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2999
2021-08-27 09:45:25 -04:00
Paul Fitzpatrick
f9630b3aa4 (core) clean up a collection of small problems affecting grist-core
Summary:
 * Remove adjustSession hack, interfering with loading docs under saml.
 * Allow the anonymous user to receive an empty list of workspaces for
   the merged org.
 * Behave better on first page load when org is in path - this used to
   fail because of lack of cookie.  This is very visible in grist-core,
   as a failure to load localhost:8484 on first visit.
 * Mark cookie explicitly as SameSite=Lax to remove a warning in firefox.
 * Make errorPages available in grist-core.

This changes the default behavior of grist-core to now start off in
anonymous mode, with an explicit sign-in step available.  If SAML is not configured,
the sign-in operation will unconditionally sign the user in as a default
user, without any password check or other security.  The user email is
taken from GRIST_DEFAULT_EMAIL if set.  This is a significant change, but
makes anonymous mode available in grist-core (which is convenient
for testing) and makes behavior with and without SAML much more consistent.

Test Plan: updated test; manual (time to start adding grist-core tests though!)

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2980
2021-08-17 21:44:50 -04:00
Dmitry S
1605e18f66 (core) Enable auto triggering of Welcome Tour, and various improvements.
Summary:
- Add showGristTour preference, and trigger tour automatically.
- Tour is only triggered for new and anonymous users on a personal org, with
  edit permission.

- Automatically open the right panel at tour start.
- Don't show tours on mobile, since that's not ready (popups are cut off
  and can't be dismissed)
- Cancel previous tour if a new one is somehow started.
- Remove #repeat- trigger hash tags from the URL when the tour starts.
- Ensure Help Center popup is positioned even when left panel is collapsed.
- Polish up the content of the last two cards in the tour.

Test Plan: Added test case for triggering and opening right panel.

Reviewers: alexmojaki, paulfitz

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D2955
2021-07-30 14:17:54 -04:00
George Gevoian
24fc3a2d00 (core) Redesign examples and templates UI
Summary:
The old Examples and Templates workspace is now
a page that pulls templates from a new public Grist Templates org.
The All Documents view will pull featured templates from that org, where
featured templates are simply pinned documents in Grist Templates. The
Examples and Templates page will also show the featured templates, as
well as the rest of the available templates organized by category. The
categories are equivalent to workspaces in Grist Templates, and are
generated dynamically.

Test Plan: Browser tests.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz, dsagal

Subscribers: dsagal, paulfitz, jarek

Differential Revision: https://phab.getgrist.com/D2930
2021-07-28 12:29:03 -07:00
Paul Fitzpatrick
1af99e9567 (core) link AppSumo activations with stripe, and support upgrades/downgrades
Summary:
This links AppSumo sign-ups with Stripe subscriptions
and our billing pages. Different AppSumo tiers are supported by
different coupons on the standard plan. Configuration of this
is in stripe, and then cached in the database.

The front end is tweaked just enough to make completing a sign-up
possible. It is not yet friendly.

Not covered includes:
 * Streamlining landing page.
 * Making billing pages git clearer summaries of AppSumo states.
 * Making flow through Cognito as graceful as possible - default
   probably doesn't meet AppSumo requirements.
 * Disabling site on cancellation/refund.
 * Downgrades when more seats in use than lower tier allows.

Test Plan: api-level tests added. No front-end tests yet.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2878
2021-06-24 10:18:42 -04:00
Paul Fitzpatrick
cc04c6481a (core) add appsumo endpoints with stub implementations
Summary:
This adds appsumo /token and /notification endpoints, with some
tests.  The stub implementation is sufficient for AppSumo
activation to succeed (when exposed via port forwarding for testing).
It needs fleshing out:

 * Implement upgrade/downgrade/refund and stripe subscription.
 * Implement custom landing page and flow.

Test Plan: added tests

Reviewers: dsagal, georgegevoian

Reviewed By: dsagal

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D2864
2021-06-21 16:04:33 -04:00
Paul Fitzpatrick
2feef7f780 (core) avoid typeorm's .save() method for relation with multi-column primary key
Summary:
A recently added stress test ("deletes documents reasonably quickly" in removedAt.ts) is sporadically failing under postgres.  It looks like typeorm's .save() method is in some way unreliable when setting a table with multi-column primary keys, via a ManyToMany relation. This diff replaces the .save() with explicit inserts/deletes.

I modified _repairWorkspaceGuests recently, so thought that change might have been the problem. However under the stress test, failures occur as often in _repairOrgGuests (not changed recently) as in _repairWorkspaceGuests (changed recently).

For reference, see schema diagram at https://grist.quip.com/wWpRAMe058Nl/Home-DB (the table being updated is `group_users`).

Possibly related issue: https://github.com/typeorm/typeorm/issues/4122

Test Plan:
After this change, stress test runs well on postgres locally (no failure 70 iterations); before it would fail on postgres within 3 iterations typically.

Separately: I gave a test that failed a little more time to return, and confirmed it was no slower on average, so I think it was unrelated.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D2848
2021-06-04 13:58:22 -04:00
Paul Fitzpatrick
29c2b35dcc (core) speed up a step in document deletion
Summary:
The `_repairWorkspaceGuests` method is slow for workspaces with large numbers of documents.  It makes a query that produces a lot of rows.  The query itself is tolerable, but TypeORM processing uses enough CPU to be a likely culprit in some production instability.  This diff splits the query into two pieces that are logically independent, but which when combined were resulting in the number of rows being the product of the two pieces.  Once split, there is also a where clause that can be applied to one of the pieces.

The purpose of the method is to add every user that a document within a workspace is shared with to a "guest" group of the workspace itself.  The design of "guest" groups is not ideal, but this diff leaves the design unchanged and is intended only to speed up operation.

Made some small tweaks to the timing of a flakey test, and temporarily recreated the `samples` directory removed in a previous diff (this is currently breaking tests badly on a fresh worker without a `samples` directory lying around)

Test Plan: added test; existing tests pass

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D2844
2021-06-02 16:06:26 -04:00
Paul Fitzpatrick
c6265335af (core) make SELF_HYPERLINK urls cleaner
Summary:
This cleans up a few things about SELF_HYPERLINK urls:

 * Use `urlId` rather than `docId`.
 * Correctly merge personal org subdomain.
 * In dev environment, use clearer port number.

Test Plan: updated test

Reviewers: alexmojaki, dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2841
2021-05-28 08:54:37 -04:00
Dmitry S
526b0ad33e (core) Configure more comprehensive eslint rules for Typescript
Summary:
- Update rules to be more like we've had with tslint
- Switch tsserver plugin to eslint (tsserver makes for a much faster way to lint in editors)
- Apply suggested auto-fixes
- Fix all lint errors and warnings in core/, app/, test/

Test Plan: Some behavior may change subtly (e.g. added missing awaits), relying on existing tests to catch problems.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2785
2021-04-26 18:54:55 -04:00
Paul Fitzpatrick
9d1bc5a518 (core) make AccessRules and FullCopies effective
Summary:
This allows `*SPECIAL:AccessRules` to give read access to the access rules to more users, and `*SPECIAL:FullCopies` to grant download/copy rights to more users.

This diff also changes forks to be owned by the user who forked them (previously they were an editor), since that feels more natural.

Test Plan: Added and updated tests.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2760
2021-03-25 15:05:26 -04:00
Paul Fitzpatrick
0c5f7cf0a7 (core) add SELF_HYPERLINK() function for generating links to the current document
Summary:
 * Adds a `SELF_HYPERLINK()` python function, with optional keyword arguments to set a label, the page, and link parameters.
 * Adds a `UUID()` python function, since using python's uuid.uuidv4 hits a problem accessing /dev/urandom in the sandbox.  UUID makes no particular quality claims since it doesn't use an audited implementation.  A difficult to guess code is convenient for some use cases that `SELF_HYPERLINK()` enables.

The canonical URL for a document is mutable, but older versions generally forward.  So for implementation simplicity the document url is passed it on sandbox creation and remains fixed throughout the lifetime of the sandbox.  This could and should be improved in future.

The URL is passed into the sandbox as a `DOC_URL` environment variable.

The code for creating the URL is factored out of `Notifier.ts`. Since the url is a function of the organization as well as the document, some rejiggering is needed to make that information available to DocManager.

On document imports, the new document is registered in the database slightly earlier now, in order to keep the procedure for constructing the URL in different starting conditions more homogeneous.

Test Plan: updated test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2759
2021-03-18 19:37:07 -04:00
Paul Fitzpatrick
3ad9b18ddf (core) allow a doc owner to test access as a different user
Summary:
This adds back-end support for query parameters `aclAsUser_` and
`aclAsUserId_` which, when either is present, direct Grist to
process granular access control rules from the point of view
of that user (specified by email or id respectively).

Some front end support is added, in the form of a tag that
shows up when in this mode, and a way to cancel the mode.
No friendly way to initiate the mode is offered yet.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2704
2021-01-15 18:45:57 -05:00
Paul Fitzpatrick
c1c17bf54e (core) distinguish open public documents from listing them
Summary:
getOrgWorkspaces and getWorkspaces had an unintended feature where
if a user had access to a workspace, they could list all publically
shared documents within that workspace.  This diff stops considering
resources shared with everyone@ when listing orgs or workspaces.
Resources shared with anon@ remain listed - this is how the example
workspace operates.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2671
2020-11-27 09:56:25 -05:00