Summary:
- Replace unicode strings with byte strings when decoding values in sandbox.
- Columns that rely on float values should derive from NumericColumn, so
that set() ensures that a float is stored even if loading an int.
- Parse unmarshallable values (['U']) into an object that can be encoded
back to the same value (rather than info a RaisedException).
- Compare NaN's as equal for deciding whether a change is a no-op.
Unrelated:
- Removed a tiny bit of unhelpful logging
Test Plan:
Added a test case that reproduces several causes of Calculate
discrepancies by loading various values into various types of formula columns.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2676
Summary:
This implements row-level access control for outgoing messages, replacing the document reloading placeholder that was there before.
* Prior to broadcasting messages, GranularAccess is notified of actions+undo.
* While broadcasting messages to different sessions, if we find we need row level access control information, rows before and after the change are reconstructed.
* Messages are rewritten if rows that were previously forbidden are now allowed, and vice versa.
The diff is somewhat under-tested and under-optimized. Next step would be to implement row-level access control for incoming actions, which may result in some rejiggering of the code from this diff to avoid duplication of effort under some conditions.
Test Plan: added test
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2670
Summary:
- Added fields to _grist_ACLRules for the new Granular ACL representation
- Include a corresponding migration.
- Added ACLPermissions module with merging PermissionSets and converting to/from string.
- Implemented parsing of ACL formulas and compiling them into JS functions.
- Add automatic parsing of ACL formulas when ACLRules are added or updated.
- Convert GranularAccess to load and interpret new-style rules.
- Convert ACL UI to load and save new-style rules.
For now, no attempt to do anything better on the server or UI side, only to
reproduce previous behavior.
Test Plan: Added unittests for new files; fixed those for existing files.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2664
Summary:
This makes data diff rendering robust to changes in the names of tables.
It does not yet show information about those changes, but at least it
won't fail to show table content changes.
Added a missing case to ActionSummary concatenation that came up in
testing.
Test Plan: added test, updated test
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2661
Summary:
The new plans for granular access control are different and handled by
node.js. Some of the same tables will be reused, of which we never made
real use before except for expecting certain specific initial records.
This diff removes the old logic, replacing it with a stub that satisfies
the interface expected by other code.
It also removes several unused UserActions: AddUser/RemoveUser/
AddInstance/RemoveInstance.
Test Plan: Existing tests should pass.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2662
Summary:
Loading all user data to run a migration is risky (creates more than usual
memory pressure), and almost never needed (only one migration requires it).
This diff attempts to run migrations using only metadata (_grist_* tables),
but retries if the sandbox tells it that all data is needed.
The intent is for new migrations to avoid needing all data.
Test Plan: Added a somewhat contrived unittest.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2659
Summary:
With recent changes to action history, we can now remove the temporary
`finalRowContent` field from change details, since all the information
we need is now in the ActionSummary.
We also now have more information about the state of the common ancestor,
which previously we could not get either from ActionSummary or from
`finalRowContent`. We take advantage of that to flesh out rendering
differences where there are some changes locally and some changes
remotely.
There's still a lot more to do, this is just one step.
I have added a link to the UI for viewing the comparison. I wouldn't
want to advertise that link until diffs are robust to name changes.
Test Plan: added test, updated tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2658
Summary:
Type conversions and formula tranforms wait for the user and bundle multiple
actions. When an unrelated action is done (e.g. adding a page widget or a
column), we want to finalize the transform before applying it.
The approach turns out fairly complicated. There is an implicit queue of
bundles (which we don't let grow beyond 2, as that's too abnormal). Bundles may
be finalized by a user clicking something, or by an unrelated action/bundle, or
(as before) by transform DOM getting disposed.
- Updated RecordLayout to use bundleActions() helper
- Added support for nesting bundleActions inside another bundle (needed for
setting visibleCol during type change)
- In an unrelated tweak, when in debug-log in ActiveDoc, use a short representation of result.
Test Plan: Added a unittest for action bundling during type transform
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2655
Summary:
This fixes a two problems:
* A mistake in `KeyedMutex.runExclusive`.
* Logic about saving a document to s3 when the document is found to match what is already there.
`HostedStorageManager.flushDoc` could get caught in a loop if a document was uploaded to s3 and then, without any change to it, marked as dirty. Low level code would detect there was no change and skip the upload; but then the snapshotId could be unknown, causing an error and retries. This diff fixes that problem by discovering the snapshotId on downloads and tracking it. It also corrects a mutex problem that may have been creating the scenario. A small delay is added to `flushDoc` to mitigate the effect of similar problems in future. Exponential backoff would be good, but `flushDoc` is called in some situations where long delays would negatively impact worker shutdown or user work.
Test Plan: added tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2654
Summary:
- Introduce a new SQLiteDB migration, which adds DB columns for formula columns
- Newly added columns have the special ['P'] (pending) value in them
(in order to show the usual "Loading..." on the first load that triggers the migration)
- Calculated values are added to .stored/.undo fields of user actions.
- Various changes made in the sandbox to include .stored/.undo in the right order.
- OnDemand tables ignore stored formula columns, replacing them with special SQL as before
- In particular, converting to OnDemand table leaves stale values in those
columns, we should maybe clean those out.
Some tweaks on the side:
- Allow overriding chai assertion truncateThreshold with CHAI_TRUNCATE_THRESHOLD
- Rebuild python automatically in watch mode
Test Plan: Fixed various tests, updated some fixtures. Many python tests that check actions needed adjustments because actions moved from .stored to .undo. Some checks added to catch situations previously only caught in browser tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2645
Summary: Adds a granular access clause for columns. Permissions can be specified for a set of columns within a table. Permissions accumulate over clauses, in a way that is intended as a placeholder pending final design.
Test Plan: Added tests. Tested manually that updates to private columns are not sent to people who don't have access to them. There are a lot of extra tests needed and TODOs to be paid down after this experimental phase.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2651
Summary:
In an emergency, we may want to serve certain documents with "old" workers as we fix problems. This diff adds some support for that.
* Creates duplicate task definitions and services for staging and production doc workers (called grist-docs-staging2 and grist-docs-prod2), pulling from distinct docker tags (staging2 and prod2). The services are set to have zero workers until we need them.
* These new workers are started with a new env variable `GRIST_WORKER_GROUP` set to `secondary`.
* The `GRIST_WORKER_GROUP` variable, if set, makes the worker available to documents in the named group, and only that group.
* An unauthenticated `/assign` endpoint is added to documents which, when POSTed to, checks that the doc is served by a worker in the desired group for that doc (as set manually in redis), and if not frees the doc up for reassignment. This makes it possible to move individual docs between workers without redeployments.
The bash scripts added are a record of how the task definitions + services were created. The services could just have been copied manually, but the task definitions will need to be updated whenever the definitions for the main doc workers are updated, so it is worth scripting that.
For example, if a certain document were to fail on a new deployment of Grist, but rolling back the full deployment wasn't practical:
* Set prod2 tag in docker to desired codebase for that document
* Set desired_count for grist-docs-prod2 service to non-zero
* Set doc-<docid>-group for that doc in redis to secondary
* Hit /api/docs/<docid>/assign to move the doc to grist-docs-prod2
(If the document needs to be reverted to a previous snapshot, that currently would need doing manually - could be made simpler, but not in scope of this diff).
Test Plan: added tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2649
Summary:
Deliberate changes:
* save snapshots to s3 prior to migrations.
* label migration snapshots in s3 metadata.
* avoid pruning migration snapshots for a month.
Opportunistic changes:
* Associate document timezone with snapshots, so pruning can respect timezones.
* Associate actionHash/Num with snapshots.
* Record time of last change in snapshots (rather than just s3 upload time, which could be a while later).
This ended up being a biggish change, because there was nowhere ideal to put tags (list of possibilities in diff).
Test Plan: added tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2646
Summary:
This is a prototype for expanding the conditions that can be used in granular ACLs.
When processing ACLs, the following variables (called "characteristics") are now available in conditions:
* UserID
* Email
* Name
* Access (owners, editors, viewers)
The set of variables can be expanded by adding a "characteristic" clause. This is a clause which specifies:
* A tableId
* The name of an existing characteristic
* A colId
The effect of the clause is to expand the available characteristics with all the columns in the table, with values taken from the record where there is a match between the specified characteristic and the specified column.
Existing clauses are generalized somewhat to demonstrate and test the use these variables. That isn't the main point of this diff though, and I propose to leave generalizing+systematizing those clauses for a future diff.
Issues I'm not dealing with here:
* How clauses combine. (The scope on GranularAccessRowClause is a hack to save me worrying about that yet).
* The full set of matching methods we'll allow.
* Refreshing row access in clients when the tables mentioned in characteristic tables change.
* Full CRUD permission control.
* Default rules (part of combination).
* Reporting errors in access rules.
That said, with this diff it is possible to e.g. assign a City to editors by their email address or name, and have only rows for those Cities be visible in their client. Ability to modify those rows, and remain updates about them, remains under incomplete control.
Test Plan: added tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2642
Summary:
* Fix old download endpoint to correctly pass org info in redirect.
* Switch to use newer download endpoint in client.
Old endpoint not removed. I started doing that, but it is used in copying, and it struck me that I'm not sure what should happen when copying from a site document to "Personal" - should it be the Personal that is associated with docs.getgrist.com currently, of should it be the Personal that is associated with the email of the user on whatever-site-we-are-on.getgrist.com. So leaving that as separate work.
Test Plan: updated tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2639
Summary:
- Add a /welcome/info endpoint, to serve a page after /welcome/user
- Add a new forms module to factor out the styles that feel more natural for a web form.
- Simplify form submission using JSON with a BaseAPI helper.
- The POST submission to /welcome/info gets added to a Grist doc, using a
specialPermit grant to gain access. A failure (e.g. missing doc) is logged
but does not affect the user.
Test Plan: Added a test case.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2640
Summary:
This implements a form of row-level access control where for a
given table, you may specify that only owners have access to
rows for which a given column has falsy values.
For simplicity:
* Only owners may edit that table.
* Non-owners with the document open will have forced
reloads whenever the table is modified.
Baby steps...
Test Plan: added tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2633
Summary:
For methods other than `GET`, `HEAD`, and `OPTIONS`, allow cookie-based authentication only if a certain custom header is present.
Specifically, we check that `X-Requested-With` is set to `XMLHttpRequest`. This is somewhat arbitrary, but allows us to use https://expressjs.com/en/api.html#req.xhr.
A request send from a browser that sets a custom header will prompt a preflight check, giving us a chance to check if the origin is trusted.
This diff deals with getting the header in place. There will be more work to do after this:
* Make sure that all important endpoints are checking origin. Skimming code, /api endpoint check origin, and some but not all others.
* Add tests spot-testing origin checks.
* Check on cases that authenticate differently.
- Check the websocket endpoint - it can be connected to from an arbitrary site; there is per-doc access control but probably better to lock it down more.
- There may be old endpoints that authenticate based on knowledge of a client id rather than cookies.
Test Plan: added a test
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2631
Summary:
Invite links broke when some base domain plumbing changed.
This fix updates them to be aware of the base domain,
and tests the Notifier class with APP_HOME_URL set to
make sure the environment variable has the expected effect.
Test Plan: added test, updated tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2630
Summary:
This moves all client code to core, and makes minimal fix-ups to
get grist and grist-core to compile correctly. The client works
in core, but I'm leaving clean-up around the build and bundles to
follow-up.
Test Plan: existing tests pass; server-dev bundle looks sane
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2627
Summary:
AccessRules class that implements that UI is intended to look vaguely like
detailed rules might look in the future, but only supports the very limited set
we have now.
In addition, UserManager and BillingPage code is separated into their own webpack bundles, to reduce the sizes of primary bundles, and relevant code from them is loaded asynchronously.
Also add two TableData methods: filterRowIds() and findMatchingRowId().
Test Plan: Only tested manually, proper automated tests don't seem warranted for this temporary UI.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2620
Summary: This removes some old metric code. There's also a user preference dialog that has a single option (whether to allow metrics) this is left in place with a dummy option. It could be ripped out as well, probably.
Test Plan: existing tests pass
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2622
Summary:
Add a unittest that start the server with limited memory, and build just enough
of ActionHistory to crash the server before this fix, and not after.
Test Plan: Tested manually with various memory prints, and added a test.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2616
Summary:
* Extends `/api/docs/docId1/compare/docId2` endpoint with a `detail=1` option to include details of what changed in the document content.
* Adds an `/api/docs/docId/compare?left=HASH&right=HASH` endpoint for comparing two versions of a single document. This is needed to implement the extension to `/api/docs/docId1/compare/docId2`.
* Adds a `HashUtil` class to allow hash aliases like `HEAD` and `HEAD~`.
Everything is a bit crude:
* Changes are expressed as ActionSummary objects, which aren't fully fleshed out.
* Extra data about formula columns is inserted in an inflexible way.
This is extracted and cleaned up from https://phab.getgrist.com/D2600.
Test Plan: added tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2614
Summary:
This is an incremental step in granular access control. Using
a temporary `{colIds: '~o structure'}` representation in the
`_grist_ACLResources` table, the document structure can be set
to be controlled by owners only.
Test Plan: added test
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2613
Summary:
This makes it possible to serve a table or tables only to owners.
* The _grist_ACLResources table is abused (temporarily) such that rows of the form `{colId: '~o', tableId}` are interpreted as meaning that `tableId` is private to owners.
* Many websocket and api endpoints are updated to preserve the privacy of these tables.
* In a document where some tables are private, a lot of capabilities are turned off for non-owners to avoid leaking info indirectly.
* The client is tweaked minimally, to show '-' where a page with some private material would otherwise go.
No attempt is made to protect data from private tables pulled into non-private tables via formulas.
There are some known leaks remaining:
* Changes to the schema of private tables are still broadcast to all clients (fixable).
* Non-owner may be able to access snapshots or make forks or use other corners of API (fixable).
* Changing name of table makes it public, since tableId in ACLResource is not updated (fixable).
Security will require some work, the attack surface is large.
Test Plan: added tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2604
Summary: This makes the user's role (owner/editor/viewer) available in ActiveDoc methods. No use of that information is made yet, other than to log it. The bulk of the diff is getting a handle on the various ways the methods can be called, and systematizing it a bit more. In passing, access control is added to broadcasts of document changes, so users who no longer have access to a document do not receive changes if they still have the document open.
Test Plan: existing tests pass; test for broadcast access control added
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2599
Summary:
Sharing a document with everyone@ was effective at the api level,
but had two flaws in the web client:
* A logged in user with no access at the org level could not access
a publically shared doc within that org.
* Likewise, for the anonymous user (but for a different reason).
This diff tweaks the web client to permit accessing a doc when
org information is unavailable.
It also changes how redirects happen for the anonymous user when
accessing a doc. They now only happen once it has been confirmed
that the user does not have access to the doc.
Test Plan: added tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2591
Summary:
This adds query parameters useful for tailoring the Grist experience, with an eye to embedding.
Setting `style=light` removes side and top bars, as a first pass at a focused view of a single document page (this would benefit from refining).
Setting `embed=true` has no significant effect just yet other than it restricts document access to viewer at most (this can be overridden by specifying `/m/default`).
Test Plan: added tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2585
Summary:
1. The /import endpoint wasn't handling poor names like ".grist" as
intended, instead trying to import them using the plugin-based imports.
2. The SaveCopy dialog was allowing users to save to an empty name,
particularly bad because new docs now default to an empty name.
Error manifested as "Cannot parse data" to the user.
Reported in https://secure.helpscout.net/conversation/1242629116/292
Test Plan: Added tests for both parts of the fix.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2573
Summary:
this moves sandbox/grist to core, and adds a requirements.txt
file for reconstructing the content of sandbox/thirdparty.
Test Plan:
existing tests pass.
Tested core functionality manually. Tested docker build manually.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2563
Summary:
This makes core independently buildable again, and adds a small
script to run as a sanity check.
Test Plan: checked that build_core.sh succeeds
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D2558
Summary:
* Adds a simple deployment test for the "Import from URL" button.
* Makes server aware of plugin hostnames in the appropriate places.
* Unrelated but convenient: allows following redirection when importing.
Test Plan:
Added tests. The `local_deployment` test works. A modified
version of this works against `staging_deployment` (using a test url that
doesn't require redirection; also staging currently has a hot fix that can
hopefully be removed once the code fix included here is in).
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2556
Summary:
Give specialPermit to the support user for page loads and API requests needed
to serve billing pages.
Test Plan: Added new test cases
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2554
Summary:
The docker image was not building or running correctly
after breaking out more material into core. This corrects
the necessary paths.
Test Plan:
tested by building locally with `./contain.sh ./build docker`
and running containers in development and production mode.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2555
Summary: This moves enough server material into core to run a home server. The data engine is not yet incorporated (though in manual testing it works when ported).
Test Plan: existing tests pass
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2552
Dmitry S
Paul Fitzpatrick