Commit Graph

912 Commits

Author SHA1 Message Date
Dmitry S
4febd90758 (core) Fix an insidious bug in RefCountMap, manifesting as JS errors some time after import.
Summary:
After an import from inside a document, one minute later, an important
QuerySet would get disposed, leaving the view section in a bad state,
and manifesting as JS errors on subsequent operations. (Might not
*always* happen because switching pages would prevent it from
manifesting, I think.)

Bad state that I've seen after transforms is probably explainable as
this bug, which is unrelated. Reproduction was hard because who knew one
had to wait a minute?!

Test Plan:
Added a unittest for the fix in QuerySet, and a browser test that
fails without the fix (JS errors, bad state), and passes with.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2653
2020-11-06 09:24:56 -05:00
Dmitry S
e2226c3ab7 (core) Store formula values in DB, and include them into .stored/.undo fields of actions.
Summary:
- Introduce a new SQLiteDB migration, which adds DB columns for formula columns
- Newly added columns have the special ['P'] (pending) value in them
  (in order to show the usual "Loading..." on the first load that triggers the migration)
- Calculated values are added to .stored/.undo fields of user actions.
- Various changes made in the sandbox to include .stored/.undo in the right order.
- OnDemand tables ignore stored formula columns, replacing them with special SQL as before
- In particular, converting to OnDemand table leaves stale values in those
  columns, we should maybe clean those out.

Some tweaks on the side:
- Allow overriding chai assertion truncateThreshold with CHAI_TRUNCATE_THRESHOLD
- Rebuild python automatically in watch mode

Test Plan: Fixed various tests, updated some fixtures. Many python tests that check actions needed adjustments because actions moved from .stored to .undo. Some checks added to catch situations previously only caught in browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2645
2020-11-04 16:45:47 -05:00
Paul Fitzpatrick
3d3fe92bd0 (core) support access control on columns
Summary: Adds a granular access clause for columns. Permissions can be specified for a set of columns within a table. Permissions accumulate over clauses, in a way that is intended as a placeholder pending final design.

Test Plan: Added tests. Tested manually that updates to private columns are not sent to people who don't have access to them. There are a lot of extra tests needed and TODOs to be paid down after this experimental phase.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2651
2020-11-03 19:08:44 -05:00
Paul Fitzpatrick
d6ff1361cb (core) support GRIST_WORKER_GROUP to place worker into an exclusive group
Summary:
In an emergency, we may want to serve certain documents with "old" workers as we fix problems. This diff adds some support for that.

 * Creates duplicate task definitions and services for staging and production doc workers (called grist-docs-staging2 and grist-docs-prod2), pulling from distinct docker tags (staging2 and prod2). The services are set to have zero workers until we need them.
 * These new workers are started with a new env variable `GRIST_WORKER_GROUP` set to `secondary`.
 * The `GRIST_WORKER_GROUP` variable, if set, makes the worker available to documents in the named group, and only that group.
 * An unauthenticated `/assign` endpoint is added to documents which, when POSTed to, checks that the doc is served by a worker in the desired group for that doc (as set manually in redis), and if not frees the doc up for reassignment. This makes it possible to move individual docs between workers without redeployments.

The bash scripts added are a record of how the task definitions + services were created. The services could just have been copied manually, but the task definitions will need to be updated whenever the definitions for the main doc workers are updated, so it is worth scripting that.

For example, if a certain document were to fail on a new deployment of Grist, but rolling back the full deployment wasn't practical:
 * Set prod2 tag in docker to desired codebase for that document
 * Set desired_count for grist-docs-prod2 service to non-zero
 * Set doc-<docid>-group for that doc in redis to secondary
 * Hit /api/docs/<docid>/assign to move the doc to grist-docs-prod2

(If the document needs to be reverted to a previous snapshot, that currently would need doing manually - could be made simpler, but not in scope of this diff).

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2649
2020-11-02 15:46:46 -05:00
Dmitry S
275a35d03a (core) In Welcome questionnaire, add 'School' option, ask to enter company or school
Test Plan: Added some checks that company and use_school get recorded. (Live doc will need to be updated before release.)

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2650
2020-11-02 14:05:31 -05:00
Paul Fitzpatrick
71519d9e5c (core) revamp snapshot inventory
Summary:
Deliberate changes:
 * save snapshots to s3 prior to migrations.
 * label migration snapshots in s3 metadata.
 * avoid pruning migration snapshots for a month.

Opportunistic changes:
 * Associate document timezone with snapshots, so pruning can respect timezones.
 * Associate actionHash/Num with snapshots.
 * Record time of last change in snapshots (rather than just s3 upload time, which could be a while later).

This ended up being a biggish change, because there was nowhere ideal to put tags (list of possibilities in diff).

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2646
2020-10-30 13:52:46 -04:00
Paul Fitzpatrick
9287439e5a (core) give instructions on using Grist with docker
Summary: cleans up docker build and instructions.

Test Plan: docker image and instructions tested manually

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2648
2020-10-28 13:59:13 -04:00
Dmitry S
d7802bc7db (core) Support international addresses in the Billing form
Summary:
- When displaying, include the country code, and don't assume state is always present.
- When entering, include a country selector (defaulting to US), and
  make state/zip optional when non-US.
- Bring in an npm module with country codes.

Test Plan: Added a browser test case.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2647
2020-10-26 11:41:15 -04:00
Paul Fitzpatrick
c879393a8e (core) support adding user characteristic tables for granular ACLs
Summary:
This is a prototype for expanding the conditions that can be used in granular ACLs.

When processing ACLs, the following variables (called "characteristics") are now available in conditions:
 * UserID
 * Email
 * Name
 * Access (owners, editors, viewers)

The set of variables can be expanded by adding a "characteristic" clause.  This is a clause which specifies:
 * A tableId
 * The name of an existing characteristic
 * A colId
The effect of the clause is to expand the available characteristics with all the columns in the table, with values taken from the record where there is a match between the specified characteristic and the specified column.

Existing clauses are generalized somewhat to demonstrate and test the use these variables. That isn't the main point of this diff though, and I propose to leave generalizing+systematizing those clauses for a future diff.

Issues I'm not dealing with here:
 * How clauses combine.  (The scope on GranularAccessRowClause is a hack to save me worrying about that yet).
 * The full set of matching methods we'll allow.
 * Refreshing row access in clients when the tables mentioned in characteristic tables change.
 * Full CRUD permission control.
 * Default rules (part of combination).
 * Reporting errors in access rules.

That said, with this diff it is possible to e.g. assign a City to editors by their email address or name, and have only rows for those Cities be visible in their client. Ability to modify those rows, and remain updates about them, remains under incomplete control.

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2642
2020-10-19 13:33:47 -04:00
Paul Fitzpatrick
27fd894fc7 (core) switch to newer download endpoint in client
Summary:
 * Fix old download endpoint to correctly pass org info in redirect.
 * Switch to use newer download endpoint in client.

Old endpoint not removed. I started doing that, but it is used in copying, and it struck me that I'm not sure what should happen when copying from a site document to "Personal" - should it be the Personal that is associated with docs.getgrist.com currently, of should it be the Personal that is associated with the email of the user on whatever-site-we-are-on.getgrist.com. So leaving that as separate work.

Test Plan: updated tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2639
2020-10-19 12:44:03 -04:00
Dmitry S
ad7be0fd8d (core) Fix WelcomePage to use an explicit action URL, and parse submitted body in time to log it with errors.
Test Plan: Tested manually that "Preflight" error goes away in Safari.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2644
2020-10-19 11:40:39 -04:00
Dmitry S
0b1aa22ad9 (core) Ask the user some questions after they sign up and set their name.
Summary:
- Add a /welcome/info endpoint, to serve a page after /welcome/user
- Add a new forms module to factor out the styles that feel more natural for a web form.
- Simplify form submission using JSON with a BaseAPI helper.
- The POST submission to /welcome/info gets added to a Grist doc, using a
  specialPermit grant to gain access. A failure (e.g. missing doc) is logged
  but does not affect the user.

Test Plan: Added a test case.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2640
2020-10-15 23:36:00 -04:00
Dmitry S
5247521cb8 (core) Improve printing of tables, fix printing of charts, add a browser test.
Summary:
- Include column headers on each page for printing tables.
- Avoid page-breaks inside rows or cards of a card-list.
- Fix printing of charts that did not show up at all before.
- Add a browser test, not great, but somewhat functional.

Test Plan: New test, plus tested manually. Column headers work on Chrome and Firefox (not Safari).

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2636
2020-10-12 16:04:18 -04:00
Paul Fitzpatrick
a4929bde72 (core) add some row-level access control
Summary:
This implements a form of row-level access control where for a
given table, you may specify that only owners have access to
rows for which a given column has falsy values.

For simplicity:
 * Only owners may edit that table.
 * Non-owners with the document open will have forced
   reloads whenever the table is modified.

Baby steps...

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2633
2020-10-12 11:17:37 -04:00
Dmitry S
99ab09651e (core) Implement 'Print widget' option to print individual view sections.
Summary:
- Supports multi-page printing with some aggressive css overrides.
- Relies on a new function implemented by grist-plugin-api to print a
  multi-page CustomView.
- Renders all rows for printing for scrolly-based views.

Test Plan:
Doesn't seem possible to do a selenium test for printing. Tested
manually on Chrome, Firefox, and Safari.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2635
2020-10-10 00:35:33 -04:00
Dmitry S
d2ad5edc46 (core) Cleanup removing some old unused files, fixing logo.css, and removing #grist-app.
Summary:
- Move logo.css to core, since it's not included otherwise
- Remove unused old DocList and ViewLinker files.
- Remove #grist-app div that was only serving to supply a background

Test Plan: No changes of behavior, existing tests should pass.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2634
2020-10-09 17:04:09 -04:00
Paul Fitzpatrick
bd6a54e901 (core) mitigate csrf by requiring custom header for unsafe methods
Summary:
For methods other than `GET`, `HEAD`, and `OPTIONS`, allow cookie-based authentication only if a certain custom header is present.

Specifically, we check that `X-Requested-With` is set to `XMLHttpRequest`. This is somewhat arbitrary, but allows us to use https://expressjs.com/en/api.html#req.xhr.

A request send from a browser that sets a custom header will prompt a preflight check, giving us a chance to check if the origin is trusted.

This diff deals with getting the header in place. There will be more work to do after this:
 * Make sure that all important endpoints are checking origin.  Skimming code, /api endpoint check origin, and some but not all others.
 * Add tests spot-testing origin checks.
 * Check on cases that authenticate differently.
    - Check the websocket endpoint - it can be connected to from an arbitrary site; there is per-doc access control but probably better to lock it down more.
    - There may be old endpoints that authenticate based on knowledge of a client id rather than cookies.

Test Plan: added a test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2631
2020-10-08 14:19:25 -04:00
Paul Fitzpatrick
8dbcbba6b5 (core) fix invite links and add tests with APP_HOME_URL set
Summary:
Invite links broke when some base domain plumbing changed.
This fix updates them to be aware of the base domain,
and tests the Notifier class with APP_HOME_URL set to
make sure the environment variable has the expected effect.

Test Plan: added test, updated tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2630
2020-10-06 21:51:40 -04:00
Dmitry S
2d023377ce (core) Fix CustomView css to take full height of widget on all browsers including Safari
Summary: On Safari, in particular on mobile, the custom widget was truncated in height.

Test Plan: Tested manually on FF, Chrome, Safari (desktop) and Safari, Firefox (mobile)

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D2629
2020-10-06 13:18:50 -04:00
Dmitry S
90db5020c9 (core) Improve focus and keyboard shortcuts in modals.
Summary:
- Factor out focusing logic from Clipboard to FocusLayer.
- Generalize FocusLayer to support adding a temporary layer while a modal is open.
- Stop Mousetrap shortcuts while a modal is open.
- Refactor how Mousetrap's custom stopCallback is implemented to avoid
  needing to bundle knockout for mousetrap.

Test Plan: Added a test that Enter in a UserManager doesn't open a cell editor from underneath the modal.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2626
2020-10-03 22:56:00 -04:00
Paul Fitzpatrick
1654a2681f (core) move client code to core
Summary:
This moves all client code to core, and makes minimal fix-ups to
get grist and grist-core to compile correctly.  The client works
in core, but I'm leaving clean-up around the build and bundles to
follow-up.

Test Plan: existing tests pass; server-dev bundle looks sane

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2627
2020-10-02 13:24:21 -04:00
Dmitry S
bac070de91 (core) With ?aclUI=1 in the URL, UserManager for documents includes a button to open 'Access Rules'
Summary:
AccessRules class that implements that UI is intended to look vaguely like
detailed rules might look in the future, but only supports the very limited set
we have now.

In addition, UserManager and BillingPage code is separated into their own webpack bundles, to reduce the sizes of primary bundles, and relevant code from them is loaded asynchronously.

Also add two TableData methods: filterRowIds() and findMatchingRowId().

Test Plan: Only tested manually, proper automated tests don't seem warranted for this temporary UI.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2620
2020-09-29 23:15:20 -04:00
Paul Fitzpatrick
2edf64c132 (core) remove metrics
Summary: This removes some old metric code. There's also a user preference dialog that has a single option (whether to allow metrics) this is left in place with a dummy option. It could be ripped out as well, probably.

Test Plan: existing tests pass

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2622
2020-09-29 18:57:56 -04:00
Paul Fitzpatrick
e5b67fee7e (core) visualize simple differences between documents
Summary:
Render simple differences between documents.

 * Show cell changes.
 * Show cell conflicts.
 * Show row additions/deletions.

Doesn't support any schema changes, and is untested in the presence of schema changes.  Any widgets that access row data without using `cells` fields won't receive correct data.

Not addressed:
 * Rendering conflicts in mixed row addition/updating/deleting.
 * Column additions/deletions, option changes, etc.
 * Document level changes.
 * Table and column renames (though anticipated in ActionSummary structure).
 * Page-level changes.
 * Drawing attention to changes (marking changed pages+views, suppressing
   unchanged rows, etc).
 * Rendering differences in views other than GridView.
 * Adding UI for initiating a comparison.
 * Editing while comparing.

Replaces {D2600}

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2618
2020-09-29 15:29:40 -04:00
Dmitry S
a15187362c (core) Enhancements to the Public Access UI.
Summary:
- Show a 'Copy Link' button in UserManager.
- Add icons for Copy (to copy link), and also for Video (to open video
  tutorials in later onboarding changes)
- Add to UserManager a 'Public Access' member-like line for greater visibility
  and to allow changing role.
- In main document page, add a "public access" icon.
- On saving UserManager, re-fetch DocInfo to update "public access" icon.

Test Plan: TBD

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2617
2020-09-23 18:54:23 -04:00
Paul Fitzpatrick
b33641dd0c (core) support a ?compare=<docId2> option in document landing pages
Summary:
If a `compare` query parameter is supplied, this diff will load
the difference between the referenced document and the current
document into an observable in the GristDoc.

Nothing is done with the comparison yet.  Comparisons are not
yet live - they don't get updated if either document changes.

For convenience, `window.gristDocPageModel` is set as an easy
way to access the DocPageModel from the browser console.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2615
2020-09-21 15:47:22 -04:00
Dmitry S
dfdb54fcbd (core) When parsing ActionHistory into ActionGroups, avoid keeping many large actions in memory.
Summary:
Add a unittest that start the server with limited memory, and build just enough
of ActionHistory to crash the server before this fix, and not after.

Test Plan: Tested manually with various memory prints, and added a test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2616
2020-09-19 20:45:33 -04:00
Dmitry S
986f469965 (core) Support "Public access" switch in the UI for docs.
Summary:
- The older "Anonymous access" option is renamed to "Public access" in the UI.
- This option was only present with GRIST_SUPPORT_ANON set.
- With GRIST_SUPPORT_ANON, the old behavior is kept: "Public access"
  option adds/removes anon@ user
- Without GRIST_SUPPORT_ANON (normal case), orgs/workspaces don't support
  "Public access" option. For documents, it adds/removes everyone@ user.

The latter is the main feature of interest. The GRIST_SUPPORT_ANON flag
is set for on-premise installs, and adds discoverability by anon users.

Test Plan: Added a test cases and checks in other tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2612
2020-09-18 19:14:33 -04:00
Paul Fitzpatrick
87f2fd15fb (core) add more detail to /compare endpoint
Summary:
 * Extends `/api/docs/docId1/compare/docId2` endpoint with a `detail=1` option to include details of what changed in the document content.
 * Adds an `/api/docs/docId/compare?left=HASH&right=HASH` endpoint for comparing two versions of a single document. This is needed to implement the extension to `/api/docs/docId1/compare/docId2`.
 * Adds a `HashUtil` class to allow hash aliases like `HEAD` and `HEAD~`.

Everything is a bit crude:
 * Changes are expressed as ActionSummary objects, which aren't fully fleshed out.
 * Extra data about formula columns is inserted in an inflexible way.

This is extracted and cleaned up from https://phab.getgrist.com/D2600.

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2614
2020-09-18 16:31:29 -04:00
Paul Fitzpatrick
b44e4a94ab (core) support locking document structure to be controlled by owners only
Summary:
This is an incremental step in granular access control.  Using
a temporary `{colIds: '~o structure'}` representation in the
`_grist_ACLResources` table, the document structure can be set
to be controlled by owners only.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2613
2020-09-17 08:50:25 -04:00
Paul Fitzpatrick
2087ae5f67 (core) port DataRowModel and FieldBuilder to typescript
Summary:
This ports two classes touched by data-diffing branch to typescript, so that the code is easier to understand and modify.

DataRowModel is quite entangled with its base class, but porting it too got a little complicated.

Test Plan: existing tests pass

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2611
2020-09-16 10:41:13 -04:00
Paul Fitzpatrick
45d2d5f897 (core) back-end support for tables that are accessible only by owners
Summary:
This makes it possible to serve a table or tables only to owners.

 * The _grist_ACLResources table is abused (temporarily) such that rows of the form `{colId: '~o', tableId}` are interpreted as meaning that `tableId` is private to owners.
 * Many websocket and api endpoints are updated to preserve the privacy of these tables.
 * In a document where some tables are private, a lot of capabilities are turned off for non-owners to avoid leaking info indirectly.
 * The client is tweaked minimally, to show '-' where a page with some private material would otherwise go.

No attempt is made to protect data from private tables pulled into non-private tables via formulas.

There are some known leaks remaining:
 * Changes to the schema of private tables are still broadcast to all clients (fixable).
 * Non-owner may be able to access snapshots or make forks or use other corners of API (fixable).
 * Changing name of table makes it public, since tableId in ACLResource is not updated (fixable).

Security will require some work, the attack surface is large.

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2604
2020-09-14 18:05:27 -04:00
Dmitry S
166143557a (core) Show a welcome card when a user opens an example for the first time.
Summary:
- The card includes an image, a brief description, and a link to the tutorial.
- The left panel includes a link to the tutorial, and a button to reopen card.
- Card is collapsed and expanded with a little animation.
- Add a seenExamples pref for whether an example has been seen.
- Store the pref in localStorage for anon user.

Separately, added clearing of prefs of test users between tests, to avoid tests
affecting unrelated tests.

Test Plan: Added a browser test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2602
2020-09-09 23:08:50 -04:00
Paul Fitzpatrick
526fda4eba (core) make user role available in ActiveDoc methods
Summary: This makes the user's role (owner/editor/viewer) available in ActiveDoc methods. No use of that information is made yet, other than to log it.  The bulk of the diff is getting a handle on the various ways the methods can be called, and systematizing it a bit more.  In passing, access control is added to broadcasts of document changes, so users who no longer have access to a document do not receive changes if they still have the document open.

Test Plan: existing tests pass; test for broadcast access control added

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2599
2020-09-02 14:46:15 -04:00
Dmitry S
7a8debae16 (core) Improve object serialization, to help get RECORD data to Custom Widgets.
Summary:
- Change RECORD's dates_as_str default to False.
- Reimplement objtype encode_object/decode_object with less machinery.
- Implement encoding of dicts (with string keys).
- Make lists and dicts encode values recursively.
- Implement encoding/decoding in the client
- Decode automatically in plugins' fetchSelectedTable/Record, with an option to skip.

Test Plan: Tested manually, not sure what tests may be affected yet.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2593
2020-08-21 18:33:28 -04:00
Dmitry S
8240f8b3f0 (core) Show in the UI when docs are inaccessible or publicly accessible.
Summary:
- Add icons to indicate a publicly-accessible document
- Dim inaccessible DocMenu items

Test Plan: Added a browser test for rendering inaccessible and public docs

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2594
2020-08-21 15:46:29 -04:00
Paul Fitzpatrick
4a545c9f2a (core) make sharing with everyone@ on documents effective
Summary:
Sharing a document with everyone@ was effective at the api level,
but had two flaws in the web client:

 * A logged in user with no access at the org level could not access
   a publically shared doc within that org.
 * Likewise, for the anonymous user (but for a different reason).

This diff tweaks the web client to permit accessing a doc when
org information is unavailable.

It also changes how redirects happen for the anonymous user when
accessing a doc.  They now only happen once it has been confirmed
that the user does not have access to the doc.

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2591
2020-08-19 19:42:26 -04:00
Dmitry S
0a5afd1f98 (core) Implement updated DocMenu UI: list/card mode and sort mode.
Summary:
- Add org-wide currentSort and currentView, saved as user preferences.
- Add per-workspace currentSort and currentView, backed by localStorage.
- Move localStorage-based observables to a separate file.
- Move hard-coded data about example docs to a separate file.
- Add UI for toggling sort and view mode.
- Removed unused features of buttonSelect to simplify it,
  and added support for light style of buttons.
- Added `parse` helper method to StringUnion, and use it in a few places where
  it simplifies code.
- Set `needRealOrg: true` in HomeDBManager.updateOrg() to fix saving prefs for
  mergedOrg.

Test Plan: WIP: Fixed some affected tests. New tests not yet written.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2587
2020-08-19 11:31:42 -04:00
Paul Fitzpatrick
20d8124f45 (core) support ?embed=true and &style=light for a clean embed experience
Summary:
This adds query parameters useful for tailoring the Grist experience, with an eye to embedding.

Setting `style=light` removes side and top bars, as a first pass at a focused view of a single document page (this would benefit from refining).

Setting `embed=true` has no significant effect just yet other than it restricts document access to viewer at most (this can be overridden by specifying `/m/default`).

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2585
2020-08-14 13:34:38 -04:00
Dmitry S
48ca124f23 (core) Render unmarshallable values as non-errors, using their repr() strings.
Summary:
- Instead of sending an "UnmarshallableError" as an exception, introduce an
  "Unmarshallable" type of value, represented as ['U', repr(value)]
- Unmarshallable values are rendered using a bluish text color, no longer a
  pink background.
- Factor out ErrorDom to be simpler and cleaner.
- Add GristObjCode enum, and simplify related helpers.
- Use safe_repr() for when repr() itself fails
- Handle conversion errors using safe_repr() when str() fails

Test Plan: Added a test case based on a fixture covering a bunch of cases.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2584
2020-08-14 11:33:29 -04:00
Paul Fitzpatrick
ac5452c89f (core) add grist.onRecord and grist.onRecords event handlers
Summary:
This simplifies writing custom widgets that access selected
data.  To access the record at which the cursor is set, and
get any future changes to it as the cursor moves or data
changes, it suffices now to do:

```
grist.ready();
grist.onRecord(record => /* render */);
```

Similarly to access the set of selected records, and get any
changes, it suffices now to do:

```
grist.ready();
grist.onRecords(records => /* render */);
```

The `records` argument will be a list of objects, each of which
is a single record.  This is distinct from the column-based
representation favored in Grist up ontil now.  That remains
how methods like `fetchTable` or `fetchSelectedTable` represent
their results.  In the future, methods named like `fetchRecords`
or `fetchSelectedRecords` could be added that return lists.

Test Plan: extended tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2583
2020-08-13 14:34:23 -04:00
Dmitry S
4e20f7a8a2 (core) Add some media queries to improve printing
Summary:
- Hides left and right panels and the top bar
- Hides cursor and active-section highlight
- Hides "=" icon on formulas
- Nudges browser to include background for row/column headers, which is not
  otherwise included.

Still only what's visible is printed (e.g. large tables not paginated).

Test Plan: Tried a few pages manually on Firefox and Chrome.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2579
2020-08-10 10:36:34 -04:00
Dmitry S
053d714655 (core) For getting access info, include the first-level doc and workspace users.
Summary:
When listing access on a doc or workspaces, include all users associated with
the resource or its parents.

Previously we only considered org-level users. This is normally sufficient
since doc and workspace users are automatically added as guests of the org. But
there are exceptions for special users (like everyone@), and generally, in case
of any divergence, it's important to list everyone who affects access
decisions.

Test Plan: Added a test that everyone@ user gets included in listings

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D2533
2020-08-05 00:26:04 -04:00
Paul Fitzpatrick
0e131c2546 (core) do not allow anonymous user to set a name
Summary:
Only allow authorized users to set names. This excludes the anonymous user.

Seems to be a narrow issue isolated to the `POST /api/profile/user/name` endpoint, other `profile` posts/deletes are already restricted to authorized users.

Test Plan: added a test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2574
2020-08-04 17:56:13 -04:00
Paul Fitzpatrick
6b24d496db (core) add per-user per-org preferences to database
Summary:
Adds preferences to orgs.  There are a few flavors:
 * `userOrgPrefs`: these are specific to a certain user and a certain org.
 * `orgPrefs`: these are specific to a certain org, and apply to all users.
 * `userPrefs`: these are specific to a certain user, and apply to all orgs.

The three flavors of prefs are reported by `GET` for an org, and can be modified by `PATCH` for an org.  The user needs to have UPDATE rights to change `orgPrefs`, but can change `userOrgPrefs` and `userPrefs` without that right since the settings only affect themselves.

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2572
2020-08-04 15:20:13 -04:00
Dmitry S
30866c6c95 (core) Fix two issues combining to report misleading error when saving to an empty name
Summary:
1. The /import endpoint wasn't handling poor names like ".grist" as
   intended, instead trying to import them using the plugin-based imports.

2. The SaveCopy dialog was allowing users to save to an empty name,
   particularly bad because new docs now default to an empty name.

Error manifested as "Cannot parse data" to the user.
Reported in https://secure.helpscout.net/conversation/1242629116/292

Test Plan: Added tests for both parts of the fix.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2573
2020-08-03 19:53:29 -04:00
Paul Fitzpatrick
ee018ff183 (core) add more tests for inaccessible workspaces; fix doc count
Summary:
 * Checks that empty workspaces are listed correctly, including in
   cases where docs or workspaces have been made inaccessible to
   the user doing the listing.
 * Checks that when a document quota is in force, the count is
   correct, and not dependent on ACLs.
 * Fixes the document count used for document quotas, which in
   fact was not counting docs the current user did not have access
   to.

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2570
2020-07-31 12:55:21 -04:00
Paul Fitzpatrick
156b75133c (core) list inaccessible docs for editors/owners of workspaces
Summary:
This modifies the material listed in workspaces.  Previously,
material the user did not have access to was omitted.  Now, it
is included if the user has the right to delete the workspace.
This is to avoid scenarios where a user might try to delete a
workspace without being aware of the full consequences.

Test Plan: added tests; existing tests should pass

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2568
2020-07-30 23:05:15 -04:00
Paul Fitzpatrick
b82eec714a (core) move data engine code to core
Summary:
this moves sandbox/grist to core, and adds a requirements.txt
file for reconstructing the content of sandbox/thirdparty.

Test Plan:
existing tests pass.
Tested core functionality manually.  Tested docker build manually.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2563
2020-07-29 08:57:25 -04:00
Dmitry S
2399baaca2 (core) When saving copies, allow saving to another org; update menus for making and saving copies.
Summary:
- Implemented selecting an org in some cases when using Save-Copy dialog.
- Unified previous 'Save Copy' menu into an enhanced "Share" menu.
- Renamed ExportMenu to ShareMenu, collect related code into it, and design the share button.
- Introduced trunkAccess property for forks, to know whether "Replace Original" is available.
- Simplified handling of fork() result, now that all code has been upgraded.
- Replaced 'Copy as Template' menu items with a checkbox in the Save-Copy dialog
- Removed copy links for examples in the DocMenu (to simplify, since not part of updated design)
- Updated the UI of the copying dialog.

Test Plan: Updated affected tests, added new test cases for copying when other orgs are a choice or not.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2561
2020-07-27 14:11:02 -04:00
Paul Fitzpatrick
9b02d16bff (core) more grist-core cleanup
Summary:
 * Remove duplicate schema file
 * Move version file to a stub in grist-core
 * Simplify sandbox creation in grist-core (although not functional until sandbox code moved)
 * Add a minimal test for buildability

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2560
2020-07-23 16:21:08 -04:00
Paul Fitzpatrick
b7b4b0229b (core) move some material to core that slipped through in a rebase
Summary:
This makes core independently buildable again, and adds a small
script to run as a sanity check.

Test Plan: checked that build_core.sh succeeds

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D2558
2020-07-23 11:29:05 -04:00
Paul Fitzpatrick
b71f2f2a10 (core) add a deployment test for Import-from-URL, and fix underlying issue
Summary:
 * Adds a simple deployment test for the "Import from URL" button.
 * Makes server aware of plugin hostnames in the appropriate places.
 * Unrelated but convenient: allows following redirection when importing.

Test Plan:
Added tests. The `local_deployment` test works.  A modified
version of this works against `staging_deployment` (using a test url that
doesn't require redirection; also staging currently has a hot fix that can
hopefully be removed once the code fix included here is in).

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2556
2020-07-23 11:26:16 -04:00
Paul Fitzpatrick
a27032df3e (core) add a test for unnecessary workspaces shown in trash, and fix issue
Summary:
Currently if I share a doc with a friend, and then soft-delete a doc
in the same workspace, that friend will see the workspace in their
trash (empty, but there).

This adds a test for the issue and resolves it by filtering out
docs at the sql level that used to be filtered out by javascript.

Test Plan: added test; existing tests pass

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2557
2020-07-23 11:04:33 -04:00
Dmitry S
671dc24214 (core) Allow the support user to access everyone's billing pages
Summary:
Give specialPermit to the support user for page loads and API requests needed
to serve billing pages.

Test Plan: Added new test cases

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2554
2020-07-22 15:45:47 -04:00
Paul Fitzpatrick
4452a816ff (core) fix docker packaging after core shuffle
Summary:
The docker image was not building or running correctly
after breaking out more material into core.  This corrects
the necessary paths.

Test Plan:
tested by building locally with `./contain.sh ./build docker`
and running containers in development and production mode.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2555
2020-07-22 14:45:42 -04:00
Dmitry S
a19f19b503 (core) Changes to Billing to better handle error scenarios.
Summary:
- Bad status of a subscription is now reported, along with the last payment error, if any.
- Error caused when getting valueRemaining of a subscription in a bad state is now ignored.
- Certain kinds of errors from Stripe are now reported to the user in a
  friendlier way (avoiding statusCode of 500 when there is a better one)
- A wide range of Stripe errors are logged with metadata.
- Show a link to the Stripe-hosted last invoice, which seems useful generally,
  and also gives the user more options to pay after a payment failure.
- Get default_source along with customer to save a roundtrip to Stripe.
- Use a simpler (single) Stripe call for updating a customer’s card.
- Retry paying an invoice when updating a card when there is an unpaid invoice
  with a payment error.

Some refactoring included:
- Simplified ISubscriptionModel by extending IBillingSubscription.
- Factor out common portions from several Billing tests.

Add a test case for how some card errors are reported
Add bits to the new billing test, still WIP.

Test Plan:
Added a test suite for incomplete and past_due subscriptions, and a
test case one for better error reporting.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2553
2020-07-22 14:40:54 -04:00
Paul Fitzpatrick
5ef889addd (core) move home server into core
Summary: This moves enough server material into core to run a home server.  The data engine is not yet incorporated (though in manual testing it works when ported).

Test Plan: existing tests pass

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2552
2020-07-21 20:39:10 -04:00
Paul Fitzpatrick
c8c5afbbca freshen app/client/ui2018/cssVars.ts
The cssVars.ts file has changed to include some more knobs
for custom theming.  This commit updates the file, and
introduces a `stubs` directory for stubbing code that is
specific to our deployments of Grist and not of general interest.
2020-06-23 16:16:38 -04:00
Dmitry S
ad35f54b87 Update tsconfig files and switch to _build for outputs, for consistency with main grist repo 2020-05-22 02:14:28 -04:00
Dmitry S
a5fbc8fcd2 Add a fixture for the included component, to have something visible on the main page 2020-05-20 01:04:54 -04:00
Dmitry S
ec182792be Initial config with a few files that build on client and server side. 2020-05-20 00:50:46 -04:00