Summary:
Move all the plugins python code into the main folder with the core code.
Register file importing functions in the same main.py entrypoint as the data engine.
Remove options relating to different entrypoints and code directories. The only remaining plugin-specific option in NSandbox is the import directory/mount, i.e. where files to be parsed are placed.
Test Plan: this
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D2965
Summary: Disabling import sources from add menu when a user is not allowed to import.
Test Plan: browser tests
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D2970
Summary: Disabling "Add widget to page" on special pages like acl or code
Test Plan: browser tests
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2969
Summary:
Hides the workspace in the breadcrumbs menu if
the doc is unsaved and is not a fork. In practice,
this should usually be when an anonymous user creates a new
document.
Test Plan: Browser tests.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2967
Summary:
Exceptional sessions had lost full read access to documents; this
restores it. Exceptional sessions are used for system actions or
while creating documents.
Test Plan: added test
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D2966
Summary: Importing from google drive from home screen (also for anonymous users)
Test Plan: Browser tests
Reviewers: dsagal, paulfitz
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2943
Summary: UserProfile test fails randomly which is caused by the AccountWidget that loads its data asonchrynously.
Test Plan: Existing tests
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D2964
Summary:
When importing from url, user types a url for google spreadsheet,
Grist will switch to Google Drive plugin to allow user to choose file manualy.
Test Plan: Browser tests
Reviewers: paulfitz, dsagal
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2945
Summary: Deletes code which was previously only used by SharedSharing.ts, which was deleted in D2894
Test Plan: no
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2960
Summary:
This adds python2 to the gvisor sandbox image. It can be used instead
of the default python3 by setting PYTHON_VERSION to 2 (or calling run.py with python2).
This is useful for making side-by-side comparisons with code running python3.
Test Plan: manual
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D2957
Summary: Improving error messages that get returned from "Import from URL" plugin.
Test Plan: browser tests
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: dsagal, alexmojaki
Differential Revision: https://phab.getgrist.com/D2946
Summary:
get_cell_value wraps RaisedException with CellError to expand the error message for the user.
This is still pretty conceptual, the comments explain some things to think about, but it works and is an improvement.
Test Plan: Updated Python unit tests
Reviewers: dsagal, paulfitz
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2928
Summary:
- Add showGristTour preference, and trigger tour automatically.
- Tour is only triggered for new and anonymous users on a personal org, with
edit permission.
- Automatically open the right panel at tour start.
- Don't show tours on mobile, since that's not ready (popups are cut off
and can't be dismissed)
- Cancel previous tour if a new one is somehow started.
- Remove #repeat- trigger hash tags from the URL when the tour starts.
- Ensure Help Center popup is positioned even when left panel is collapsed.
- Polish up the content of the last two cards in the tour.
Test Plan: Added test case for triggering and opening right panel.
Reviewers: alexmojaki, paulfitz
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D2955
Summary:
Adds 'GristDocTour' as a possible value of urlState().docPage
GristDoc checks for this and converts it to a normal view record ID
It also then sets a flag showGristDocTour=true which tells Pages.ts to show the page in the sidebar
Otherwise the page is 'hidden' in the sidebar in the same way it would be if blocked by ACL rules
This all feels very hacky, but I don't know this code well enough to know if there's a better way. Hopefully this behaviour is temporary.
Test Plan: Tested manually, not sure if this is worth an automated test at this stage
Reviewers: paulfitz, dsagal
Reviewed By: paulfitz, dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D2953
Summary:
Processing these calls in the client, rather than passing them on
to the backend, means that access rules are more straightforward to
apply.
An unrelated fix is included to filter _grist_ tables when fetched
individually - metadata could leak through this path.
Test Plan: added tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2954
Summary:
Replace Finish button with Previous and an X to close
Add keyboard shortcuts to tour popups
Change last Next button to Finish instead of disabling, can be triggered by Enter key.
Allow closing the tour and reopening in the same place.
Test Plan: only manual, need to confirm desired behaviour
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2950
Summary: This mentions fiddle mode in FullCopies special permission.
Test Plan: visual, existing tests pass
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2952
Summary:
Temporarily adds a client-side check to hide the
Grist Templates org in the Save Copy menu. This will
be removed later once we update getOrgs to optionally
filter orgs that have no workspaces with write access.
Test Plan: Browser tests.
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D2951
Summary:
Checks for new special columns in GristDocTour: Link_Text, Link_URL, and Link_Icon.
No link is generated if Link_Text is blank or Link_URL cannot be parsed as a URL.
No icon is shown if Link_Icon is not the name of an icon in IconList.ts
Test Plan: Expanded tests, but they now assert things about HTML which may be brittle
Reviewers: georgegevoian, dsagal
Reviewed By: georgegevoian, dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D2947
Summary:
Staging tests were using an organization named Test Grist,
while local was using one named test-grist. Both are now
named Test Grist, which should help keep things more consistent
between local and deployment test runs.
The inherited line height of template docs in icon view was
different on Windows, cutting off part of the last line of the
description. The description line height should now be fixed
to a reasonable value.
Test Plan: Manually tested CSS fix on a Windows machine.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D2948
Summary:
The old Examples and Templates workspace is now
a page that pulls templates from a new public Grist Templates org.
The All Documents view will pull featured templates from that org, where
featured templates are simply pinned documents in Grist Templates. The
Examples and Templates page will also show the featured templates, as
well as the rest of the available templates organized by category. The
categories are equivalent to workspaces in Grist Templates, and are
generated dynamically.
Test Plan: Browser tests.
Reviewers: paulfitz, dsagal
Reviewed By: paulfitz, dsagal
Subscribers: dsagal, paulfitz, jarek
Differential Revision: https://phab.getgrist.com/D2930
Summary:
* Moves essential plugins to grist-core, so that basic imports (e.g. csv) work.
* Adds support for a `GRIST_SANDBOX_FLAVOR` flag that can systematically override how the data engine is run.
- `GRIST_SANDBOX_FLAVOR=pynbox` is "classic" nacl-based sandbox.
- `GRIST_SANDBOX_FLAVOR=docker` runs engines in individual docker containers. It requires an image specified in `sandbox/docker` (alternative images can be named with `GRIST_SANDBOX` flag - need to contain python and engine requirements). It is a simple reference implementation for sandboxing.
- `GRIST_SANDBOX_FLAVOR=unsandboxed` runs whatever local version of python is specified by a `GRIST_SANDBOX` flag directly, with no sandboxing. Engine requirements must be installed, so an absolute path to a python executable in a virtualenv is easiest to manage.
- `GRIST_SANDBOX_FLAVOR=gvisor` runs the data engine via gvisor's runsc. Experimental, with implementation not included in grist-core. Since gvisor runs on Linux only, this flavor supports wrapping the sandboxes in a single shared docker container.
* Tweaks some recent express query parameter code to work in grist-core, which has a slightly different version of express (smoke test doesn't catch this since in Jenkins core is built within a workspace that has node_modules, and wires get crossed - in a dev environment the problem on master can be seen by doing `buildtools/build_core.sh /tmp/any_path_outside_grist`).
The new sandbox options do not have tests yet, nor does this they change the behavior of grist servers today. They are there to clean up and consolidate a collection of patches I've been using that were getting cumbersome, and make it easier to run experiments.
I haven't looked closely at imports beyond core.
Test Plan: tested manually against regular grist and grist-core, including imports
Reviewers: alexmojaki, dsagal
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D2942
Summary:
Extracts code from showExampleCard into a generic function which is reused for document tours.
It handles reading and writing to user preferences for automatic showing and explicitly reopening.
Test Plan:
Manually tested that it automatically shows a tour just once and clicking to reopen works.
There's not much new functionality so there's little that needs testing. This is an initial version that's mostly internal and is likely to be polished for users in the future.
If I should still add tests, I'd like confirmation that the current behaviour is as desired.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2944
Summary:
Like the welcome tour, a special URL hash triggers startDocTour which uses data from a table GristDocTour to construct the appropriate popups.
This is the basic version described in https://grist.quip.com/sN2RAHI2dchm/Document-tours
Test Plan:
Added a new nbrowser test which tests the data produced by makeDocTour. The general behaviour of the UI and popups has hardly changed so existing tests cover that well enough.
The new test uses a new fixture document which you can open to easily experience the tour.
Error cases where there's no valid document tour are not tested because that behaviour is likely to change significantly and this feature is still quite 'private'.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: jarek, dsagal
Differential Revision: https://phab.getgrist.com/D2938
Summary:
This makes it possible to set the type of a column to ReferenceList, but the UI is terrible
ReferenceList.ts is a mishmash of ChoiceList and Reference that sort of works but something about the CSS is clearly broken
ReferenceListEditor is just a text editor, you have to type in a JSON array of row IDs. Ignore the value that's present when you start editing. I can maybe try mashing together ReferenceEditor and ChoiceListEditor but it doesn't seem wise.
I think @georgegevoian should take over here. Reviewing the diff as it is to check for obvious issues is probably good but I don't think it's worth trying to land/merge anything.
Test Plan: none
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: georgegevoian
Differential Revision: https://phab.getgrist.com/D2914
Summary:
Flaky Dates test failures related to the use of JQuery autocomplete for time
zones, which wasn't working well.
This diff replaces that autocomplete (as well as a similar select box in
DocumentSettings) with our newer autocomplete, adding some select-box like
behavior.
Most of the behavior is factored out into ACSelect, which could be more
generally useful.
Adds an option to autocomplete to keep options ordered according to their
initial order.
Unrelated: fix up usage of MultiHolder in Drafts to avoid 'already disposed'
warnings.
Test Plan: Fixed several affected tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D2919
Summary: Last document position was overwritting anchor link navigation.
Test Plan: Browser tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D2934
Summary:
Fixing two bugs
- Google Auth Endpoint wasn't resolving protocol in a correct way
- Google Auth Popup was navigationg to endpoint url based on home url, which
was diffent from current page origin
Test Plan: n/a
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D2937
Summary:
This tweaks pre-fork mode to make the user's experience a bit more seamless.
Pre-fork mode is where the user has opened a document with intent to
fork it, but actual forking (with allocation of a new document id)
is postponed until they make their first change.
The tweak makes the user an owner for granular access purposes, if
forking is permitted. So data visible only to owners because of
access rules will be visible to them. As always, any edits would
go to a separate new copy.
A remaining tricky corner is what to do about "View As" functionality
on forks. Fork sharing cannot be controlled, so the "Users -> View As"
functionality isn't available. Perhaps the "Users" button on a fork
could encourage doing a save-copy and inviting users, or offer some
dummy users? In any case, this diff doesn't change anything with
that corner.
Test Plan: added test
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2931
Summary:
Google Auth popup wasn't able to resolve origin from gristConfig.
Moving this reponsability to server side, where it gets calculated from initial request.
Test Plan: n/a
Reviewers: dsagal, paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2935
Summary:
Implementing export to excel and send to Google Drive feature.
As part of this feature few things were implemented:
- Server side google authentication exposed on url: (docs, docs-s, or localhost:8080)/auth/google
- Exporting grist documents as an excel file (xlsx)
- Storing exported grist document (in excel format) in Google Drive as a spreadsheet document.
Server side google authentication requires one new environmental variables
- GOOGLE_CLIENT_SECRET (required) used by authentication handler
Test Plan: Browser tests for exporting to excel.
Reviewers: paulfitz, dsagal
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2924
Summary: Adding a new element to a new Choice or ChoiceList column resulted in javascript error
Test Plan: Browser tests
Reviewers: georgegevoian, dsagal
Reviewed By: georgegevoian, dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D2932
Summary: Having CONTAINS be a class is a pain, undoing that mistake now
Test Plan: none needed
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2929
Summary: Remove repl.py, REPLTab.js, some wiring code, CSS, and a test in testscript.json.
Test Plan: NA
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2923
Summary:
- Fix flaky SamplesWS test, which was occasionally clicking a disabled button
- Increase timeout on backupSqliteDatabase test, which sometimes times out
- Fix a little flakiness in ExportSection test.
- Fix flaky Billing test by adjusting behavior slightly.
In Billing, when re-fetching subscription (e.g. on navigating back), we now
unset it while the fetch is pending, so that billing pages show a spinner or
"Fetching..." messages. This also gives tests something to wait for.
Also adjusts Billing styles on the "Fetching..." messages to make them look
different from regular text.
Test Plan: Mainly making exising tests more robust. Billing changes exercised by existing tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2920
Summary: In porting a js file to typescript, a dependency on tmp-promise was added - but core doesn't have that. This adds it, and also takes the opportunity to bump the grist-core version number.
Test Plan: manual (smoke test doesn't catch this)
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D2922
Summary:
Added CONTAINS 'function' which can be used in lookups
Changed LookupMapColumn._row_key_map to use right=set so one row can have many keys when CONTAINS is used.
Use CONTAINS to implement group column in summary table, while helper column in source table can reference and create multiple rows in summary table, especially when summarising by ChoiceList columns.
Use itertools.product to generate all combinations of lookup keys and groupby values.
cleanup
Test Plan: Added python unit tests.
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: paulfitz, dsagal
Differential Revision: https://phab.getgrist.com/D2900
Summary:
This diff brings in the new welcome tour. It builds upon `client/ui/OnBoardingPopup` that was committed to that purposes. Per this diff, the tour is accessible behind a flag and won't be visible to user: few caveats listed below needs to be adressed first.
This diff also brings few changes to onboarding module.
- allow to refer to element with selector
- usually dynamic selection of element sounds useful for when the
element does not exist yet when the tour starts. But the actual
reason when add it here, is to allow selecting the first cell.
- if the selector yields undefined (missing element), the popup
is simply skipped
- got rid of the internal registry to link between popup contents
and popup options. All is now define in the same interface. Registry
overall felt overkill and not needed.
- adds an option to show message as a simple modal that is centered
on the screen
This diff also brings the new welcome tour and hide it behind a flag
CAVEATS that need to be addressed in follow up commit:
- The url needs cleanup, #repeat-welcome-tour sticks to it and so even when navigating to home page. This could eventually become an issue: if user opens another document it would starts the onboarding tour again.
- For now you have to manually make sure the right panel is opened with the Column tab selected before starting the tour.
- On boarding tours were not designed with mobile support in mind. So probably a good idea to disable.
- Backend support needs to be done (persistence of first time user).
Test Plan:
Updated `projects/OnBoardingPopup` and adds new `nbrowser/welcomeTour`
To launch the tour:
- open any document
- open manually the right panel and the field tab
- append the flag `#repeat-welcome-tour` at the end of the url in the url bar and reload the page
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2917
Summary:
Dealing with some things that bothered and sometimes confused me:
Make Table.Record[Set] provide the table argument automatically
Remove the classes from UserTable because they're not used anywhere and the Table/UserTable distinction is already confusing. They're not documented for users and they don't show up in autocomplete.
Remove RecordSet.Record because it was confusing me where that attribute was being set, but also this means .Record will work properly for users with columns named 'Record'.
Test Plan: existing tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2913
Summary:
Choice columns can now add new choices directly
from the autocomplete menu. The autocomplete will
now highlight the first matching item, even if there are equally
ranked alternatives. No changes have been made to how the
autocomplete index is created, or how it scores items.
For choice and choice list columns, the filter menu will
now display values using their configured colors, similar to the
rest of the UI. Choice tokens throughout the UI now do a better
job of handling text overflow by showing an ellipsis whenever
there isn't enough space to show the full text of a choice.
Test Plan: Browser tests.
Reviewers: cyprien
Reviewed By: cyprien
Differential Revision: https://phab.getgrist.com/D2904
Summary:
Bundles some new document options into a JSON column.
The icon option is treated somewhat gingerly. It is intended, at
least initially, to store an image thumbnail for a document as a
url to hand-prepared assets (for examples and templates), so it is
locked down to a particular url prefix to avoid opening the door to
mischief.
Test Plan: added test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D2916
Summary:
The 'user' variable has a similar API to the one from access rules: it
contains properties about a user, such as their full name and email
address, as well as optional, user-defined attributes that are populated
via user attribute tables.
Test Plan: Python unit tests.
Reviewers: alexmojaki, paulfitz, dsagal
Reviewed By: alexmojaki, dsagal
Subscribers: paulfitz, dsagal, alexmojaki
Differential Revision: https://phab.getgrist.com/D2898
Summary:
When redirecting to login, it's important to have a valid session set. This was
done by middleware that only applies to home pages. We need to set session to
live when redirecting in case of doc pages too.
Test Plan: Added a test case for fixed behavior by applying an existing case to doc pages too
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2915
Summary:
This suspends service to a team site for which an AppSumo refund has been made, and nudges users to their free personal account.
I expect that a refund request would fail for a site where user is also paying us for extra seats.
Test Plan: tested manually
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2912
Summary:
This applies some mitigations suggested by SQLite authors when
opening untrusted SQLite databases, as we do when Grist docs
are uploaded by the user. See:
https://www.sqlite.org/security.html#untrusted_sqlite_database_files
Steps implemented in this diff are:
* Setting `trusted_schema` to off
* Running a SQLite-level integrity check on uploads
Other steps will require updates to our node-sqlite3 fork, since they
are not available via the node-sqlite3 api (one more reason to migrate
to better-sqlite3).
I haven't yet managed to create a file that triggers an integrity
check failure without also being detected as corruption by sqlite
at a more basic level, so that is a TODO for testing.
Test Plan:
existing tests pass; need to come up with exploits to
actually test the defences and have not yet
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2909
Summary:
Finishing implementation for google drive plugin.
- Refactoring plugin code to make it more robust and to follow grist ux
- Changing the way server hosts untrusted user content, from different domain to different port
Test Plan: Browser tests
Reviewers: dsagal, paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2881
Summary:
Error notifications include a "Report a problem" link, but the inclusion of
error message and stack trace was broken.
Test Plan: Tested manually and added a test case.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2911
Summary:
The sigma icon appears to the left of the table title
if the table is a summary table.
Test Plan: Updated browser tests.
Reviewers: jarek
Reviewed By: jarek
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D2895
Alex Hall
Jarosław Sadziński
George Gevoian
Paul Fitzpatrick
Dmitry S
Cyprien P