Commit Graph

30 Commits

Author SHA1 Message Date
George Gevoian
a19ba0813a (core) Add telemetry
Test Plan: Server tests.

Reviewers: jarek

Differential Revision: https://phab.getgrist.com/D3818
2023-04-06 12:34:54 -04:00
George Gevoian
1ac4931c22 (core) Persist forks in home db
Summary:
Adds information about forks to the home db. This will be used
later by the UI to list forks of documents.

Test Plan: Browser and server tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3772
2023-02-20 22:46:36 -05:00
Jarosław Sadziński
fd02a00a0e Fixing all eslint's reported error 2023-01-03 17:22:58 +01:00
Louis Delbosc
eea2ef5cfb
Use url.hostname instead of url.host to allow host from environment variable (#326)
Co-authored-by <yohan.boniface@free.fr>
2022-10-25 14:59:17 -04:00
Louis Delbosc
49b1749e98
Add function to allow hosts from environment variables (#287)
* Add allowed host option to handle CORS requests
* Update readme with new GRIST_ALLOWED_HOSTS environment variable
2022-09-28 12:33:53 -04:00
Paul Fitzpatrick
f91f45b26d (core) support granular read access for attachments
Summary:
When a user requests to read the contents of an attachment, only allow the request if there exists a cell in an attachment column that contains the attachment and which they have read access to.

This does not cover:
 * Granular write access for attachments. In particular, a user who can write to any attachment column should be considered to have full read access to all attachment columns, currently.
 * Access control of attachment metadata such as name and format.

The implementation uses a sql query that requires a scan, and some notes on how this could be optimized in future. The web client was updated to specify the cell to check for access, and performance seemed fine in casual testing on a doc with 1000s of attachments. I'm not sure how performance would hold up as the set of access rules grows as well.

Test Plan: added tests

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3490
2022-07-07 07:22:02 -04:00
Dmitry S
51ff72c15e (core) Faster builds all around.
Summary:
Building:
- Builds no longer wait for tsc for either client, server, or test targets. All use esbuild which is very fast.
- Build still runs tsc, but only to report errors. This may be turned off with `SKIP_TSC=1` env var.
- Grist-core continues to build using tsc.
- Esbuild requires ES6 module semantics. Typescript's esModuleInterop is turned
  on, so that tsc accepts and enforces correct usage.
- Client-side code is watched and bundled by webpack as before (using esbuild-loader)

Code changes:
- Imports must now follow ES6 semantics: `import * as X from ...` produces a
  module object; to import functions or class instances, use `import X from ...`.
- Everything is now built with isolatedModules flag. Some exports were updated for it.

Packages:
- Upgraded browserify dependency, and related packages (used for the distribution-building step).
- Building the distribution now uses esbuild's minification. babel-minify is no longer used.

Test Plan: Should have no behavior changes, existing tests should pass, and docker image should build too.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3506
2022-07-04 10:42:40 -04:00
Jarosław Sadziński
d92a761f6e (core) Product update popups and hosted stripe integration
Summary:
- Showing nudge to individual users to sign up for free team plan.
- Implementing billing page to upgrade from free team to pro.
- New modal with upgrade options and free team site signup.
- Integrating Stripe-hosted UI for checkout and plan management.

Test Plan: updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3456
2022-06-08 21:10:49 +02:00
Jarosław Sadziński
0ab9e4a6a0 (core) Adding GristConnect login system
Summary:
New login system to allow simple SSO flow that is based on Discourse description that is available at:
https://meta.discourse.org/t/discourseconnect-official-single-sign-on-for-discourse-sso/13045

Test Plan: New core test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3418
2022-05-18 20:28:25 +02:00
George Gevoian
f48d579f64 (core) Add API endpoint to get site usage summary
Summary:
The summary includes a count of documents that are approaching
limits, in grace period, or delete-only. The endpoint is only accessible
to site owners, and is currently unused. A follow-up diff will add usage
banners to the site home page, which will use the response from the
endpoint to communicate usage information to owners.

Test Plan: Browser and server tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3420
2022-05-16 11:16:19 -07:00
Paul Fitzpatrick
4de5928396 (core) when redirecting, use protocol in APP_HOME_URL if available
Summary:
Currently, Grist behind a reverse proxy will generate many
needless redirects via `http`, and can't be used with only
port 443. This diff centralizes generation of these redirects
and uses the protocol in APP_HOME_URL if it is set.

Test Plan:
manually tested by rebuilding grist-core and
doing a reverse proxy deployment that had no support for
port 80. Prior to this change, there are lots of problems;
after, the site works as expected.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3400
2022-04-28 09:13:27 -04:00
George Gevoian
859c593448 (core) Add authSubject and authProvider to sessions
Summary:
This also updates Authorizer to link the authSubject
to Grist users if not previously linked. Linked subjects
are now used as the username for password-based logins,
instead of emails, which remain as a fallback.

Test Plan: Existing tests, and tested login flows manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3356
2022-04-11 11:42:02 -07:00
George Gevoian
4c5de16e2d (core) Include altSessionId in logs
Summary: Adds altSessionId to log output.

Test Plan: Tested manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3355
2022-04-08 16:40:34 -07:00
George Gevoian
6305811ca6 (core) Add new Grist login page
Summary:
Adds a new Grist login page to the login app, and replaces the
server-side Cognito Google Sign-In flow with Google's own OAuth flow.

Test Plan: Browser and server tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3332
2022-04-01 15:24:19 -07:00
Alex Hall
59436d2bca (core) Grace period and delete-only mode when exceeding row limit
Summary:
Builds upon https://phab.getgrist.com/D3328

- Add HomeDB column `Document.gracePeriodStart`
- When the row count moves above the limit, set it to the current date. When it moves below, set it to null.
- Add DataLimitStatus type indicating if the document is approaching the limit, is in a grace period, or is in delete only mode if the grace period started at least 14 days ago. Compute it in ActiveDoc and send it to client when opening.
- Only allow certain user actions when in delete-only mode.

Follow-up tasks related to this diff:

- When DataLimitStatus in the client is non-empty, show a banner to the appropriate users.
- Only send DataLimitStatus to users with the appropriate access. There's no risk landing this now since real users will only see null until free team sites are released.
- Update DataLimitStatus immediately in the client when it changes, e.g. when user actions are applied or the product is changed. Right now it's only sent when the document loads.
- Update row limit, grace period start, and data limit status in ActiveDoc when the product changes, i.e. the user upgrades/downgrades.
- Account for data size when computing data limit status, not just row counts.

See also the tasks mentioned in https://phab.getgrist.com/D3331

Test Plan: Extended FreeTeam nbrowser test, testing the 4 statuses.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3331
2022-03-25 13:41:33 +02:00
Alex Hall
546096fcc9 (core) Clean up and refactor uses of HomeDBManager.getDoc
Summary:
Firstly I just wanted some more consistency and less repetition in places where Documents are retrieved from the DB, so it's more obvious when code differs from the norm. Main changes for that part:

- Let HomeDBManager accept a `Request` directly and convert it to a `Scope`, and use this in a few places.
- `getScope` tries `req.docAuth.docId` if `req.params` doesn't have a docId.

I also refactored how `_createActiveDoc` gets the document URL, separating out getting the document from getting a URL for it. This is because I want to use that document object in a future diff, but I also just find it cleaner. Notable changes for that:

- Extracted a new method `HomeDBManager.getRawDocById` as an alternative to `getDoc` that's explicitly for when you only have a document ID.
- Removed the interface method `GristServer.getDocUrl` and its two implementations because it wasn't used elsewhere and it didn't really add anything on top of getting a doc (now done by `getRawDocById`) and `getResourceUrl`.
- Between `cachedDoc` and `getRawDocById` (which represent previously existing code paths) also try `getDoc(getScope(docSession.req))`, which is new, because it seems better to only `getRawDocById` as a last resort.

Test Plan: Existing tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3328
2022-03-24 13:42:36 +02:00
George Gevoian
9522438967 (core) Add Grist forgot password page
Summary:
The page isn't yet linked to from anywhere in the UI, but
will be soon, once the new login page is ready. The page
can still be accessed at login-[s].getgrist.com/forgot-password,
and the flow is similar to the one used by Cognito's hosted UI.

Also refactors much of the existing login app code into smaller
files with less duplication, tweaks password validation to be closer
to Cognito's requirements, and polishes various parts of the UI,
like the verified page CSS, and the form inputs.

Test Plan: Browser, server and project tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3296
2022-03-07 09:11:28 -08:00
George Gevoian
e264094412 (core) Add account page option to allow Google login
Summary:
Enabled by default, the new checkbox is only visible to
users logged in with email/password, and controls whether it is possible
to log in to the same account via a Google account
(with matching email). When disabled, CognitoClient will refuse logins
from Google if a Grist account with the same email exists.

Test Plan:
Server and browser tests for setting flag. Manual tests to verify
Cognito doesn't allow signing in with Google when flag is disabled.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3257
2022-02-14 16:56:23 -08:00
Alex Hall
0de0cb0f4a (core) Add PUT /records DocApi endpoint to AddOrUpdate records
Summary:
As designed in https://grist.quip.com/fZSrAnJKgO5j/Add-or-Update-Records-API

Current `POST /records` adds records, and `PATCH /records` updates them by row ID. This adds `PUT /records` to 'upsert' records, applying the AddOrUpdate user action. PUT was chosen because it's idempotent. Using a separate method (instead of inferring based on the request body) also cleanly separates validation, documentation, etc.

The name `require` for the new property was suggested by Paul because `where` isn't very clear when adding records.

Test Plan: New DocApi tests

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3251
2022-02-12 09:44:34 +02:00
Paul Fitzpatrick
d99db8d016 (core) move more tests to grist-core
Summary:
 * Tie build and run-time docker base images to a consistent version (buster)
 * Extend the test login system activated by GRIST_TEST_LOGIN to ease porting tests that currently rely on cognito (many)
 * Make org resets work in absence of billing endpoints
 * When in-memory session caches are used, add missing invalidation steps
 * Pass org information through sign-ups/sign-ins more carefully
 * For CORS, explicitly trust GRIST_HOST origin when set
 * Move some fixtures and tests to core, focussing on tests that cover existing failures or are in the set of tests run on deployments
 * Retain regular `test` target to run the test suite directly, without docker
 * Add a `test:smoke` target to run a single simple test without `GRIST_TEST_LOGIN` activated
 * Add a `test:docker` target to run the tests against a grist-core docker image - since tests rely on certain fixture teams/docs, added `TEST_SUPPORT_API_KEY` and `TEST_ADD_SAMPLES` flags to ease porting

The tests ported were `nbrowser` tests: `ActionLog.ts` (the first test I tend to port to anything, out of habit), `Fork.ts` (exercises a lot of doc creation paths), `HomeIntro.ts` (a lot of DocMenu exercise), and `DuplicateDocument.ts` (covers a feature known to be failing prior to this diff, the CORS tweak resolves it).

Test Plan: Manually tested via `buildtools/build_core.sh`. In follow up, I want to add running the `test:docker` target in grist-core's workflows. In jenkins, only the smoke test is run. There'd be an argument for running all tests, but they include particularly slow tests, and are duplicates of tests already run (in different configuration admittedly), so I'd like to try first just using them in grist-core to gate updates to any packaged version of Grist (the docker image currently).

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3176
2021-12-10 18:33:07 -05:00
Jarosław Sadziński
53bdd6c8e1 (core) Exposing more descriptive errors from exports
Summary:
Exports used to show generic message on error.
Adding error description to the message.

Test Plan: Updated tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3157
2021-11-30 17:26:32 +01:00
Paul Fitzpatrick
bb8cb2593d (core) support python3 in grist-core, and running engine via docker and/or gvisor
Summary:
 * Moves essential plugins to grist-core, so that basic imports (e.g. csv) work.
 * Adds support for a `GRIST_SANDBOX_FLAVOR` flag that can systematically override how the data engine is run.
   - `GRIST_SANDBOX_FLAVOR=pynbox` is "classic" nacl-based sandbox.
   - `GRIST_SANDBOX_FLAVOR=docker` runs engines in individual docker containers. It requires an image specified in `sandbox/docker` (alternative images can be named with `GRIST_SANDBOX` flag - need to contain python and engine requirements). It is a simple reference implementation for sandboxing.
   - `GRIST_SANDBOX_FLAVOR=unsandboxed` runs whatever local version of python is specified by a `GRIST_SANDBOX` flag directly, with no sandboxing. Engine requirements must be installed, so an absolute path to a python executable in a virtualenv is easiest to manage.
   - `GRIST_SANDBOX_FLAVOR=gvisor` runs the data engine via gvisor's runsc. Experimental, with implementation not included in grist-core. Since gvisor runs on Linux only, this flavor supports wrapping the sandboxes in a single shared docker container.
 * Tweaks some recent express query parameter code to work in grist-core, which has a slightly different version of express (smoke test doesn't catch this since in Jenkins core is built within a workspace that has node_modules, and wires get crossed - in a dev environment the problem on master can be seen by doing `buildtools/build_core.sh /tmp/any_path_outside_grist`).

The new sandbox options do not have tests yet, nor does this they change the behavior of grist servers today. They are there to clean up and consolidate a collection of patches I've been using that were getting cumbersome, and make it easier to run experiments.

I haven't looked closely at imports beyond core.

Test Plan: tested manually against regular grist and grist-core, including imports

Reviewers: alexmojaki, dsagal

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D2942
2021-07-28 09:02:32 -04:00
Jarosław Sadziński
f8e4fe54ba (core) Fixing origin check during Google Authentication
Summary:
Fixing two bugs
- Google Auth Endpoint wasn't resolving protocol in a correct way
- Google Auth Popup was navigationg to endpoint url based on home url, which
  was diffent from current page origin

Test Plan: n/a

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D2937
2021-07-23 00:43:27 +02:00
Paul Fitzpatrick
997be24a21 (core) add docs.options column to home db to store doc description, icon, openMode
Summary:
Bundles some new document options into a JSON column.
The icon option is treated somewhat gingerly.  It is intended, at
least initially, to store an image thumbnail for a document as a
url to hand-prepared assets (for examples and templates), so it is
locked down to a particular url prefix to avoid opening the door to
mischief.

Test Plan: added test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D2916
2021-07-15 21:51:05 -04:00
Paul Fitzpatrick
24e76b4abc (core) add endpoints for clearing snapshots and actions
Summary:
This adds a snapshots/remove and states/remove endpoint, primarily
for maintenance work rather than for the end user.  If some secret
gets into document history, it is useful to be able to purge it
in an orderly way.

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2694
2020-12-18 13:32:31 -05:00
Paul Fitzpatrick
27fd894fc7 (core) switch to newer download endpoint in client
Summary:
 * Fix old download endpoint to correctly pass org info in redirect.
 * Switch to use newer download endpoint in client.

Old endpoint not removed. I started doing that, but it is used in copying, and it struck me that I'm not sure what should happen when copying from a site document to "Personal" - should it be the Personal that is associated with docs.getgrist.com currently, of should it be the Personal that is associated with the email of the user on whatever-site-we-are-on.getgrist.com. So leaving that as separate work.

Test Plan: updated tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2639
2020-10-19 12:44:03 -04:00
Paul Fitzpatrick
45d2d5f897 (core) back-end support for tables that are accessible only by owners
Summary:
This makes it possible to serve a table or tables only to owners.

 * The _grist_ACLResources table is abused (temporarily) such that rows of the form `{colId: '~o', tableId}` are interpreted as meaning that `tableId` is private to owners.
 * Many websocket and api endpoints are updated to preserve the privacy of these tables.
 * In a document where some tables are private, a lot of capabilities are turned off for non-owners to avoid leaking info indirectly.
 * The client is tweaked minimally, to show '-' where a page with some private material would otherwise go.

No attempt is made to protect data from private tables pulled into non-private tables via formulas.

There are some known leaks remaining:
 * Changes to the schema of private tables are still broadcast to all clients (fixable).
 * Non-owner may be able to access snapshots or make forks or use other corners of API (fixable).
 * Changing name of table makes it public, since tableId in ACLResource is not updated (fixable).

Security will require some work, the attack surface is large.

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2604
2020-09-14 18:05:27 -04:00
Paul Fitzpatrick
20d8124f45 (core) support ?embed=true and &style=light for a clean embed experience
Summary:
This adds query parameters useful for tailoring the Grist experience, with an eye to embedding.

Setting `style=light` removes side and top bars, as a first pass at a focused view of a single document page (this would benefit from refining).

Setting `embed=true` has no significant effect just yet other than it restricts document access to viewer at most (this can be overridden by specifying `/m/default`).

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2585
2020-08-14 13:34:38 -04:00
Dmitry S
671dc24214 (core) Allow the support user to access everyone's billing pages
Summary:
Give specialPermit to the support user for page loads and API requests needed
to serve billing pages.

Test Plan: Added new test cases

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2554
2020-07-22 15:45:47 -04:00
Paul Fitzpatrick
5ef889addd (core) move home server into core
Summary: This moves enough server material into core to run a home server.  The data engine is not yet incorporated (though in manual testing it works when ported).

Test Plan: existing tests pass

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2552
2020-07-21 20:39:10 -04:00