Summary:
With recent changes to action history, we can now remove the temporary
`finalRowContent` field from change details, since all the information
we need is now in the ActionSummary.
We also now have more information about the state of the common ancestor,
which previously we could not get either from ActionSummary or from
`finalRowContent`. We take advantage of that to flesh out rendering
differences where there are some changes locally and some changes
remotely.
There's still a lot more to do, this is just one step.
I have added a link to the UI for viewing the comparison. I wouldn't
want to advertise that link until diffs are robust to name changes.
Test Plan: added test, updated tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2658
Summary:
This fixes a two problems:
* A mistake in `KeyedMutex.runExclusive`.
* Logic about saving a document to s3 when the document is found to match what is already there.
`HostedStorageManager.flushDoc` could get caught in a loop if a document was uploaded to s3 and then, without any change to it, marked as dirty. Low level code would detect there was no change and skip the upload; but then the snapshotId could be unknown, causing an error and retries. This diff fixes that problem by discovering the snapshotId on downloads and tracking it. It also corrects a mutex problem that may have been creating the scenario. A small delay is added to `flushDoc` to mitigate the effect of similar problems in future. Exponential backoff would be good, but `flushDoc` is called in some situations where long delays would negatively impact worker shutdown or user work.
Test Plan: added tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2654
Summary:
After an import from inside a document, one minute later, an important
QuerySet would get disposed, leaving the view section in a bad state,
and manifesting as JS errors on subsequent operations. (Might not
*always* happen because switching pages would prevent it from
manifesting, I think.)
Bad state that I've seen after transforms is probably explainable as
this bug, which is unrelated. Reproduction was hard because who knew one
had to wait a minute?!
Test Plan:
Added a unittest for the fix in QuerySet, and a browser test that
fails without the fix (JS errors, bad state), and passes with.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2653
Summary: Adds a granular access clause for columns. Permissions can be specified for a set of columns within a table. Permissions accumulate over clauses, in a way that is intended as a placeholder pending final design.
Test Plan: Added tests. Tested manually that updates to private columns are not sent to people who don't have access to them. There are a lot of extra tests needed and TODOs to be paid down after this experimental phase.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2651
Summary:
Deliberate changes:
* save snapshots to s3 prior to migrations.
* label migration snapshots in s3 metadata.
* avoid pruning migration snapshots for a month.
Opportunistic changes:
* Associate document timezone with snapshots, so pruning can respect timezones.
* Associate actionHash/Num with snapshots.
* Record time of last change in snapshots (rather than just s3 upload time, which could be a while later).
This ended up being a biggish change, because there was nowhere ideal to put tags (list of possibilities in diff).
Test Plan: added tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2646
Summary:
This is a prototype for expanding the conditions that can be used in granular ACLs.
When processing ACLs, the following variables (called "characteristics") are now available in conditions:
* UserID
* Email
* Name
* Access (owners, editors, viewers)
The set of variables can be expanded by adding a "characteristic" clause. This is a clause which specifies:
* A tableId
* The name of an existing characteristic
* A colId
The effect of the clause is to expand the available characteristics with all the columns in the table, with values taken from the record where there is a match between the specified characteristic and the specified column.
Existing clauses are generalized somewhat to demonstrate and test the use these variables. That isn't the main point of this diff though, and I propose to leave generalizing+systematizing those clauses for a future diff.
Issues I'm not dealing with here:
* How clauses combine. (The scope on GranularAccessRowClause is a hack to save me worrying about that yet).
* The full set of matching methods we'll allow.
* Refreshing row access in clients when the tables mentioned in characteristic tables change.
* Full CRUD permission control.
* Default rules (part of combination).
* Reporting errors in access rules.
That said, with this diff it is possible to e.g. assign a City to editors by their email address or name, and have only rows for those Cities be visible in their client. Ability to modify those rows, and remain updates about them, remains under incomplete control.
Test Plan: added tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2642
Summary:
* Fix old download endpoint to correctly pass org info in redirect.
* Switch to use newer download endpoint in client.
Old endpoint not removed. I started doing that, but it is used in copying, and it struck me that I'm not sure what should happen when copying from a site document to "Personal" - should it be the Personal that is associated with docs.getgrist.com currently, of should it be the Personal that is associated with the email of the user on whatever-site-we-are-on.getgrist.com. So leaving that as separate work.
Test Plan: updated tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2639
Summary:
- Add a /welcome/info endpoint, to serve a page after /welcome/user
- Add a new forms module to factor out the styles that feel more natural for a web form.
- Simplify form submission using JSON with a BaseAPI helper.
- The POST submission to /welcome/info gets added to a Grist doc, using a
specialPermit grant to gain access. A failure (e.g. missing doc) is logged
but does not affect the user.
Test Plan: Added a test case.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2640
Summary:
This implements a form of row-level access control where for a
given table, you may specify that only owners have access to
rows for which a given column has falsy values.
For simplicity:
* Only owners may edit that table.
* Non-owners with the document open will have forced
reloads whenever the table is modified.
Baby steps...
Test Plan: added tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2633
Summary:
For methods other than `GET`, `HEAD`, and `OPTIONS`, allow cookie-based authentication only if a certain custom header is present.
Specifically, we check that `X-Requested-With` is set to `XMLHttpRequest`. This is somewhat arbitrary, but allows us to use https://expressjs.com/en/api.html#req.xhr.
A request send from a browser that sets a custom header will prompt a preflight check, giving us a chance to check if the origin is trusted.
This diff deals with getting the header in place. There will be more work to do after this:
* Make sure that all important endpoints are checking origin. Skimming code, /api endpoint check origin, and some but not all others.
* Add tests spot-testing origin checks.
* Check on cases that authenticate differently.
- Check the websocket endpoint - it can be connected to from an arbitrary site; there is per-doc access control but probably better to lock it down more.
- There may be old endpoints that authenticate based on knowledge of a client id rather than cookies.
Test Plan: added a test
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2631
Summary:
- Factor out focusing logic from Clipboard to FocusLayer.
- Generalize FocusLayer to support adding a temporary layer while a modal is open.
- Stop Mousetrap shortcuts while a modal is open.
- Refactor how Mousetrap's custom stopCallback is implemented to avoid
needing to bundle knockout for mousetrap.
Test Plan: Added a test that Enter in a UserManager doesn't open a cell editor from underneath the modal.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2626
Summary:
This moves all client code to core, and makes minimal fix-ups to
get grist and grist-core to compile correctly. The client works
in core, but I'm leaving clean-up around the build and bundles to
follow-up.
Test Plan: existing tests pass; server-dev bundle looks sane
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2627
Summary:
AccessRules class that implements that UI is intended to look vaguely like
detailed rules might look in the future, but only supports the very limited set
we have now.
In addition, UserManager and BillingPage code is separated into their own webpack bundles, to reduce the sizes of primary bundles, and relevant code from them is loaded asynchronously.
Also add two TableData methods: filterRowIds() and findMatchingRowId().
Test Plan: Only tested manually, proper automated tests don't seem warranted for this temporary UI.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2620
Summary: This removes some old metric code. There's also a user preference dialog that has a single option (whether to allow metrics) this is left in place with a dummy option. It could be ripped out as well, probably.
Test Plan: existing tests pass
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2622
Summary:
Render simple differences between documents.
* Show cell changes.
* Show cell conflicts.
* Show row additions/deletions.
Doesn't support any schema changes, and is untested in the presence of schema changes. Any widgets that access row data without using `cells` fields won't receive correct data.
Not addressed:
* Rendering conflicts in mixed row addition/updating/deleting.
* Column additions/deletions, option changes, etc.
* Document level changes.
* Table and column renames (though anticipated in ActionSummary structure).
* Page-level changes.
* Drawing attention to changes (marking changed pages+views, suppressing
unchanged rows, etc).
* Rendering differences in views other than GridView.
* Adding UI for initiating a comparison.
* Editing while comparing.
Replaces {D2600}
Test Plan: added tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2618
Summary:
If a `compare` query parameter is supplied, this diff will load
the difference between the referenced document and the current
document into an observable in the GristDoc.
Nothing is done with the comparison yet. Comparisons are not
yet live - they don't get updated if either document changes.
For convenience, `window.gristDocPageModel` is set as an easy
way to access the DocPageModel from the browser console.
Test Plan: added test
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2615
Summary:
- The older "Anonymous access" option is renamed to "Public access" in the UI.
- This option was only present with GRIST_SUPPORT_ANON set.
- With GRIST_SUPPORT_ANON, the old behavior is kept: "Public access"
option adds/removes anon@ user
- Without GRIST_SUPPORT_ANON (normal case), orgs/workspaces don't support
"Public access" option. For documents, it adds/removes everyone@ user.
The latter is the main feature of interest. The GRIST_SUPPORT_ANON flag
is set for on-premise installs, and adds discoverability by anon users.
Test Plan: Added a test cases and checks in other tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2612
Summary:
* Extends `/api/docs/docId1/compare/docId2` endpoint with a `detail=1` option to include details of what changed in the document content.
* Adds an `/api/docs/docId/compare?left=HASH&right=HASH` endpoint for comparing two versions of a single document. This is needed to implement the extension to `/api/docs/docId1/compare/docId2`.
* Adds a `HashUtil` class to allow hash aliases like `HEAD` and `HEAD~`.
Everything is a bit crude:
* Changes are expressed as ActionSummary objects, which aren't fully fleshed out.
* Extra data about formula columns is inserted in an inflexible way.
This is extracted and cleaned up from https://phab.getgrist.com/D2600.
Test Plan: added tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2614
Summary:
This ports two classes touched by data-diffing branch to typescript, so that the code is easier to understand and modify.
DataRowModel is quite entangled with its base class, but porting it too got a little complicated.
Test Plan: existing tests pass
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2611
Summary:
- The card includes an image, a brief description, and a link to the tutorial.
- The left panel includes a link to the tutorial, and a button to reopen card.
- Card is collapsed and expanded with a little animation.
- Add a seenExamples pref for whether an example has been seen.
- Store the pref in localStorage for anon user.
Separately, added clearing of prefs of test users between tests, to avoid tests
affecting unrelated tests.
Test Plan: Added a browser test.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2602
Summary:
- Change RECORD's dates_as_str default to False.
- Reimplement objtype encode_object/decode_object with less machinery.
- Implement encoding of dicts (with string keys).
- Make lists and dicts encode values recursively.
- Implement encoding/decoding in the client
- Decode automatically in plugins' fetchSelectedTable/Record, with an option to skip.
Test Plan: Tested manually, not sure what tests may be affected yet.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2593
Summary:
- Add org-wide currentSort and currentView, saved as user preferences.
- Add per-workspace currentSort and currentView, backed by localStorage.
- Move localStorage-based observables to a separate file.
- Move hard-coded data about example docs to a separate file.
- Add UI for toggling sort and view mode.
- Removed unused features of buttonSelect to simplify it,
and added support for light style of buttons.
- Added `parse` helper method to StringUnion, and use it in a few places where
it simplifies code.
- Set `needRealOrg: true` in HomeDBManager.updateOrg() to fix saving prefs for
mergedOrg.
Test Plan: WIP: Fixed some affected tests. New tests not yet written.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2587
Summary:
This adds query parameters useful for tailoring the Grist experience, with an eye to embedding.
Setting `style=light` removes side and top bars, as a first pass at a focused view of a single document page (this would benefit from refining).
Setting `embed=true` has no significant effect just yet other than it restricts document access to viewer at most (this can be overridden by specifying `/m/default`).
Test Plan: added tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2585
Summary:
- Instead of sending an "UnmarshallableError" as an exception, introduce an
"Unmarshallable" type of value, represented as ['U', repr(value)]
- Unmarshallable values are rendered using a bluish text color, no longer a
pink background.
- Factor out ErrorDom to be simpler and cleaner.
- Add GristObjCode enum, and simplify related helpers.
- Use safe_repr() for when repr() itself fails
- Handle conversion errors using safe_repr() when str() fails
Test Plan: Added a test case based on a fixture covering a bunch of cases.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2584
Summary:
This simplifies writing custom widgets that access selected
data. To access the record at which the cursor is set, and
get any future changes to it as the cursor moves or data
changes, it suffices now to do:
```
grist.ready();
grist.onRecord(record => /* render */);
```
Similarly to access the set of selected records, and get any
changes, it suffices now to do:
```
grist.ready();
grist.onRecords(records => /* render */);
```
The `records` argument will be a list of objects, each of which
is a single record. This is distinct from the column-based
representation favored in Grist up ontil now. That remains
how methods like `fetchTable` or `fetchSelectedTable` represent
their results. In the future, methods named like `fetchRecords`
or `fetchSelectedRecords` could be added that return lists.
Test Plan: extended tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2583
Summary:
Adds preferences to orgs. There are a few flavors:
* `userOrgPrefs`: these are specific to a certain user and a certain org.
* `orgPrefs`: these are specific to a certain org, and apply to all users.
* `userPrefs`: these are specific to a certain user, and apply to all orgs.
The three flavors of prefs are reported by `GET` for an org, and can be modified by `PATCH` for an org. The user needs to have UPDATE rights to change `orgPrefs`, but can change `userOrgPrefs` and `userPrefs` without that right since the settings only affect themselves.
Test Plan: added tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2572
Summary:
- Implemented selecting an org in some cases when using Save-Copy dialog.
- Unified previous 'Save Copy' menu into an enhanced "Share" menu.
- Renamed ExportMenu to ShareMenu, collect related code into it, and design the share button.
- Introduced trunkAccess property for forks, to know whether "Replace Original" is available.
- Simplified handling of fork() result, now that all code has been upgraded.
- Replaced 'Copy as Template' menu items with a checkbox in the Save-Copy dialog
- Removed copy links for examples in the DocMenu (to simplify, since not part of updated design)
- Updated the UI of the copying dialog.
Test Plan: Updated affected tests, added new test cases for copying when other orgs are a choice or not.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2561
Summary:
* Remove duplicate schema file
* Move version file to a stub in grist-core
* Simplify sandbox creation in grist-core (although not functional until sandbox code moved)
* Add a minimal test for buildability
Test Plan: added test
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2560
Summary:
* Adds a simple deployment test for the "Import from URL" button.
* Makes server aware of plugin hostnames in the appropriate places.
* Unrelated but convenient: allows following redirection when importing.
Test Plan:
Added tests. The `local_deployment` test works. A modified
version of this works against `staging_deployment` (using a test url that
doesn't require redirection; also staging currently has a hot fix that can
hopefully be removed once the code fix included here is in).
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2556
Summary:
- Bad status of a subscription is now reported, along with the last payment error, if any.
- Error caused when getting valueRemaining of a subscription in a bad state is now ignored.
- Certain kinds of errors from Stripe are now reported to the user in a
friendlier way (avoiding statusCode of 500 when there is a better one)
- A wide range of Stripe errors are logged with metadata.
- Show a link to the Stripe-hosted last invoice, which seems useful generally,
and also gives the user more options to pay after a payment failure.
- Get default_source along with customer to save a roundtrip to Stripe.
- Use a simpler (single) Stripe call for updating a customer’s card.
- Retry paying an invoice when updating a card when there is an unpaid invoice
with a payment error.
Some refactoring included:
- Simplified ISubscriptionModel by extending IBillingSubscription.
- Factor out common portions from several Billing tests.
Add a test case for how some card errors are reported
Add bits to the new billing test, still WIP.
Test Plan:
Added a test suite for incomplete and past_due subscriptions, and a
test case one for better error reporting.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2553
Summary: This moves enough server material into core to run a home server. The data engine is not yet incorporated (though in manual testing it works when ported).
Test Plan: existing tests pass
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2552
