Summary:
- Substantial refactoring of the logic when the server fails to send some
messages to a client.
- Add seqId numbers to server messages to ensure reliable order.
- Add a needReload flag in clientConnect for a clear indication whent the
browser client needs to reload the app.
- Reproduce some potential failure scenarios in a test case (some of which
previously could have led to incorrectly ordered messages).
- Convert other Comm tests to typescript.
- Tweak logging of Comm and Client to be slightly more concise (in particular,
avoid logging sessionId)
Note that despite the big refactoring, this only addresses a fairly rare
situation, with websocket failures while server is trying to send to the
client. It includes no improvements for failures while the client is sending to
the server.
(I looked for an existing library that would take care of these issues. A relevant article I found is https://docs.microsoft.com/en-us/azure/azure-web-pubsub/howto-develop-reliable-clients, but it doesn't include a library for both ends, and is still in review. Other libraries with similar purposes did not inspire enough confidence.)
Test Plan: New test cases, which reproduce some previously problematic scenarios.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3470
Summary:
For self-hosted Grist, forward auth has proven useful, where
some proxy wrapped around Grist manages authentication, and
passes on user information to Grist in a trusted header.
The current implementation is adequate when Grist is the
only place where the user logs in or out, but is confusing
otherwise (see https://github.com/gristlabs/grist-core/issues/207).
Here we take some steps to broaden the scenarios Grist's
forward auth support can be used with:
* When a trusted header is present and is blank, treat
that as the user not being logged in, and don't look
any further for identity information. Specifically,
don't look in Grist's session information.
* Add a `GRIST_IGNORE_SESSION` flag to entirely prevent
Grist from picking up identity information from a cookie,
in order to avoid confusion between multiple login methods.
* Add tests for common scenarios.
Test Plan: added tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3482
Summary:
- Also converted sandboxUtil to typescript.
- The issue with %s manifested when a Python traceback contained "%s" in the
string; in that case the object with log metadata (e.g. docId) would
confusingly replace %s as if it were part of the message from Python.
Test Plan: Added a test case for the fix.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3486
Summary:
- Showing nudge to individual users to sign up for free team plan.
- Implementing billing page to upgrade from free team to pro.
- New modal with upgrade options and free team site signup.
- Integrating Stripe-hosted UI for checkout and plan management.
Test Plan: updated tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3456
Summary:
- Add app/common/CommTypes.ts to define types shared by client and server.
- Include @types/ws npm package
Test Plan: Intended to have no changes in behavior
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3467
Summary:
Introduces a new message type, docUsage, that's broadcast to all connected
clients whenever document usage is updated in ActiveDoc.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3451
Summary:
- Remove the empty-folder icon
- Add an "Invite team members" button for owners on empty team sites
- Add a "Browse Templates" button for all other cases on empty sites
- Update intro text for team, including a link to Sprouts
- Update intro text for personal/anon.
- Include a Free/Pro tag for team sites (for now, only "Free")
- Add a "Manage Team" button for owners on home page of all team sites.
- Polished the UI of UserManager: add a transition for the background, and
delay the appearance of the spinner for fast loads.
Test Plan: Fixed up the HomeIntro tests; added test case for Manage Team button
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3459
Summary:
I missed committing a file that is important for editing files comfortably in the ext directory in an IDE. This diff:
* Adds tsconfig-base-ext.json - that was the only intended change
* Unrelated: Forces all creation of connections to the home db through a new `getOrCreateConnection` method which changes the `busy_timeout` if using Sqlite. This was an attempt to fix random "database is locked" test failures. I believe multiple connections to the home db as an sqlite file do not happen in self-hosted Grist (where there is a single node process) or in our SaaS (where the database is in postgres). It does affect Grist started using `devServerMain.ts` (where multiple processes accessing same database are started) or various test configurations when extra database connections are opened.
* Unrelated: I added a `busy_timeout` for session storage, when it uses Sqlite. Again, I don't believe this affects self-hosted Grist or our SaaS.
* Tweaked a `BillingDiscount` test that looked perhaps vulnerable to a stripe request stalling.
I can't be sure my tweaks actually help, since I didn't succeed in replicating the failures. Update: looks like the "locked" error can still happen :(
Test Plan: manual
Reviewers: jarek
Reviewed By: jarek
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3450
Summary: Combines the code and behaviour of the existing endpoints `GET /records` (for the general shape of the result and the parameters for sort/filter/limit etc) and retrieving a specific attachment with `GET /attachments/:id` for handling fields specific to attachments.
Test Plan: Added a DocApi test. Also updated one test to use the new endpoint instead of raw `GET /tables/_grist_Attachments/records`.
Reviewers: cyprien
Reviewed By: cyprien
Subscribers: cyprien
Differential Revision: https://phab.getgrist.com/D3443
Summary:
This makes it possible to configure a SendGrid-based Notifier
instance via a JSON configuration file.
Test Plan: Tested manually.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3432
Summary:
DocApi tests occasionally fail in a confusing way due to redis
state. This resets the redis state at the beginning of the test.
Test Plan: existing tests pass
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3424
Summary:
Nearby deployment tests used the same user, which was
causing the full suite to fail. Repeated logins with a second
verification step (TOTP) fail in Cognito if the same verification code
is used twice. This happened to be the case since the gap
between tests wasn't long enough for a new code to be generated.
Test Plan: Manually verified deployment tests now pass.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3414
Summary:
Adds attachment and data size to the usage section of
the raw data page. Also makes in-document usage banners
update as user actions are applied, causing them to be
hidden/shown or updated based on the current state of
the document.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3395
Summary:
- Better focus on the widget title
- Adding columns only to the current view section
- New popup with options when user wants to delete a page
- New dialog to enter table name
- New table as a widget doesn't create a separate page
- Removing a table doesn't remove the primary view
Test Plan: Updated and new tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3410
Summary: Allow exceeding the daily API usage limit for a doc based on additional allocations for the current hour and minute. See the doc comment on getDocApiUsageKeysToIncr for details. This means that up to 5 redis keys may be relevant at a time for a single document.
Test Plan: Updated and expanded 'Daily API Limit' tests.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3368
Summary:
A new way for renaming tables.
- There is a new popup to rename section (where you can also rename the table)
- Renaming/Deleting page doesn't modify/delete the table.
- Renaming table can rename a page if the names match (and the page contains a section with that table).
- User can rename table in Raw Data UI in two ways - either on the listing or by using the section name popup
- As before, there is no way to change tableId - it is derived from a table name.
- When the section name is empty the table name is shown instead.
- White space for section name is allowed (to discuss) - so the user can just paste ' '.
- Empty name for a page is not allowed (but white space is).
- Some bugs related to deleting tables with attached summary tables (and with undoing this operation) were fixed (but not all of them yet).
Test Plan: Updated tests.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: georgegevoian
Differential Revision: https://phab.getgrist.com/D3360
Summary:
Summary columns now have their own conditional rules,
which are not shared with sister columns.
Test Plan: New test
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3388
Summary:
Call ActiveDoc.removeUnusedAttachments every hour using setInterval, and in ActiveDoc.shutdown (which also clears said interval).
Unrelated: small fix to my webhooks code which was creating a redis client on shutdown just to quit it.
Test Plan:
Tweaked DocApi test to remove expired attachments by force-reloading the doc, so that it removes them during shutdown. Extracted a new testing endpoint /verifyFiles to support this test (previously running that code only happened with `/removeUnused?verifyfiles=1`).
Tested the setInterval part manually.
Reviewers: paulfitz, dsagal
Reviewed By: paulfitz
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3387
Summary:
New command "Duplicate rows" is available in the Row/Card Context Menu and as a keyboard shortcut Ctrl+Alt+C.
- All selected rows are duplicated (even if only a single column is selected)
- Rows are inserted immediately after the last selected row (using manualSort value).
- Formulas and CENSORED fields are not copied.
Implemented on the UI level (no new action).
Test Plan: new test
Reviewers: cyprien
Reviewed By: cyprien
Differential Revision: https://phab.getgrist.com/D3371
Summary:
- Add a new parameter `Features.baseMaxAttachmentsBytesPerDocument` and set it to 1GB for the free team product.
- Add a method to DocStorage to calculate the total size of existing and used attachments.
- Add a migration to DocStorage adding an index to make the query in the above method fast.
- Check in ActiveDoc if uploading attachment(s) would exceed the product limit on that document.
Test Plan: Added test in `limits.ts` testing enforcement of the attachment limit.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3374
Summary:
Search now works on Raw Data Page.
- Search bar option 'Search on all pages' will change to 'Search on all tables' when on the Raw data page, and will allow searching through all tables.
- Little CSS adjustment for an overlay on Raw page (removes z-index as it is not needed, and conflicts with searchbar).
- Search bar option ('search on all') gets white background, little padding, and is moved 2 pixels up, this is needed for Raw page.
Test Plan: new and updated tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3376
Summary:
For users who cannot otherwise change access to a resource, let
them remove themselves. Implemented via the standard endpoints
as a special exception that will process a request from a user
that would otherwise be denied, if the only contents of that
request are a removal of themselves.
Users who can change access are still not permitted to change their
own permissions or to remove themselves, as a precaution against
orphaning resources.
Test Plan: extended and updated tests
Reviewers: cyprien
Reviewed By: cyprien
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3367
Summary: Adds methods to delete metadata rows based on timeDeleted. The flag expiredOnly determines if it only deletes attachments that were soft-deleted 7 days ago, or just all soft-deleted rows. Then any actual file data that doesn't have matching metadata is deleted.
Test Plan: DocApi test
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3364
Summary:
This also updates Authorizer to link the authSubject
to Grist users if not previously linked. Linked subjects
are now used as the username for password-based logins,
instead of emails, which remain as a fallback.
Test Plan: Existing tests, and tested login flows manually.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3356
Summary:
When linking table to a summary, the linking ended up broken after updating the summary group by columns.
This diff fixes that issue.
There were two issues:
1) some subscriptions were missing due to some .peek() calls instead of directly calling the observable.
2) the LinkingState instance was not being disposed.
3) the filterColValues was not updating after source data table has been loaded
Test Plan: Include new test file.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3358
Summary:
Redesigning color picker:
- Single color palette (no light/dark switch)
- Ability to remove color (new empty button)
New font options in the color picker.
Font options are available on:
- Default cell style
- Conditional rules styles
- Choice/ChoiceList editor and token field
- Filters for Choice/ChoiceList columns
Design document:
https://www.figma.com/file/bRTsb47VIOVBfJPj0qF3C9/Grist-Updates?node-id=415%3A8135
Test Plan: new and updated tests
Reviewers: georgegevoian, alexmojaki
Reviewed By: georgegevoian, alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3335
Summary:
- adds the `aggregate values` option to chart config
- aggregation is performed by turning table into summary table
- change columns options of xaxis and split series selectors to be the source table columns when `aggregate values` is on
- change xAxis and split series computed to hold colId instead of column id
- change GristDoc saveViewSection routine to preserve old sections viewFields and options
- Rename `Group data` into `split series`
quip doc: https://grist.quip.com/tAsCAuv8RiMa/Charts-data-aggregation#temp:C:QcK0ce13e1e8ae64048988f44f9c
Test Plan: Adds ChartAggregate.ts nbrowser test.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3336
Summary:
Builds on https://phab.getgrist.com/D3352
Add DocStorage.scanAttachmentsForUsageChanges to do fancy JSON query to find all attachment metadata rows whose soft deletion status needs updating.
Add ActiveDoc.updateUsedAttachments which uses the above and then applies the appropriate user action if needed to soft delete/undelete metadata rows.
Add endpoint in DocApi calling ActiveDoc method.
Test Plan: Added DocApi test
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3357
Summary: Adds a migration in preparation for future work on tracking and deleting attachments. This includes a `_grist_Attachments.timeDeleted` column which isn't used yet, and changing the storage format of user columns of type `Attachments`. DocStorage now treats Attachments like RefList in general (since they use JSON), which also prompted a tiny bit of refactoring.
Test Plan: Added a migration test case showing the change in format.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3352
Summary:
This annotates the plugin api sufficiently to generate some documentation
for it. See https://github.com/gristlabs/grist-help/pull/139
Contains some small code tweaks for things that caused typedoc some
trouble.
Test Plan: manual inspection of output
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3342
Summary:
Adds a new Grist login page to the login app, and replaces the
server-side Cognito Google Sign-In flow with Google's own OAuth flow.
Test Plan: Browser and server tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3332
* remove stray redis dependency in test
* tweak handling of database connection between tests
* upgrade node versions in tests, type guessing in node 10 has problems
Summary:
This shuffles some server tests to make them available in grist-core,
and adds a test for the `GRIST_PROXY_AUTH_HEADER` feature added in
https://github.com/gristlabs/grist-core/pull/165
It includes a fix for a header normalization issue for websocket connections.
Test Plan: added test
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3326
Summary:
Adding conditional formatting rules feature.
Each column can have multiple styling rules which are applied in order
when evaluated to a truthy value.
- The creator panel has a new section: Cell Style
- New user action AddEmptyRule for adding an empty rule
- New columns in _grist_Table_columns and fields
A new color picker will be introduced in a follow-up diff (as it is also
used in choice/choice list/filters).
Design document:
https://grist.quip.com/FVzfAgoO5xOF/Conditional-Formatting-Implementation-Design
Test Plan: new tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3282
Summary:
This diff introduces cursor features for raw data views:
- Restoring cursor position when the browser window is reloaded
- Restoring the last edit position when the browser window is reloaded
Test Plan: Added tests
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3314
Summary: Fix for error that happens when a search is resumed after one of the columns was hidden.
Test Plan: Added test that shows the error.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3309
Summary:
Updates simulateLogin to fill in the name field of
the test login page. Docker tests were failing because
users created via the test login page were falling back
to their email for their name.
Test Plan: N/A (fixing Docker tests)
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3315
Summary:
Importing a .grist document is implemented in a somewhat clunky way, in a multi-worker setup.
* First a random worker receives the upload, and updates Grist's various stores appropriately (database, redis, s3).
* Then a random worker is assigned to serve the document.
If the worker serving the document fails, there is a chance the it will end up assigned to the worker that handled its upload. Currently the worker will misbehave in this case. This diff:
* Ports a multi-worker test from test/home to run in test/s3, and adds a test simulating a bad scenario seen in the wild.
* Fixes persistence of any existing document checksum in redis when a worker is assigned.
* Adds a check when assigned a document to serve, and finding that document already cached locally. It isn't safe to rely only on the document checksum in redis, since that may have expired.
* Explicitly claims the document on the uploading worker, so this situation becomes even less likely to arise.
Test Plan: added test
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3305
Summary:
A staging test was failing because the util was throwing instead
of returning true or false.
Test Plan: N/A (fixing test)
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3301
Summary:
The old regex for login pages, /gristlogin/, used to handle
test login pages too, matching on a query param set in the
URL. A recent change to make the regex more specific broke this.
This fixes that regression by adding an additional check to
isOnLoginPage for the presence of the test login page.
Test Plan: N/A (fixing test)
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3300
Summary: Removes code that was marked for removal.
Test Plan: Existing tests still pass.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3289
Summary:
Core doesn't redirect to Cognito or our own sign-up page
when clicking 'sign up' on the welcome screen. Instead, it
redirects to the test login page.
Test Plan: N/A (fixing test)
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3298
Summary:
A selfish diff to tweak deployment tests so they pass on a
slightly smaller laptop screen that for whatever reason trips
a css breakpoint on the cognito login page.
Test Plan: manual
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3292
Summary:
The "Sign In" button can either redirect you to the Cognito login
page or the Grist sign-up page, depending on whether you've logged
in before. Since the HomeIntro tests can either be run as part of a full
deployment test suite, or individually, the test could redirect you to
both types of login pages.
Test Plan: N/A (fixing test)
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3291
Summary:
Removes code from simulateLogin that's no longer needed, and increases
the timeout on the util function that checks if we're on the Grist sign-up
page.
Test Plan: N/A (fixing test)
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3287
Summary:
This should resolve some staging test failures. The previous regexes
weren't quite specific enough, so random parts of a URL that had
traces of the real login URLs were causing isOnLoginPage() and
isOnGristLoginPage() to return true instead of false.
Test Plan: N/A (fixing tests)
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3283
Summary:
- Adds a dependency moment-guess (https://github.com/apoorv-mishra/moment-guess) to guess date formats from strings. However the npm package is missing source maps which leads to an ugly warning, so currently using a fork until https://github.com/apoorv-mishra/moment-guess/pull/22 is resolved.
- Adds guessDateFormat using moment-guess to determine the best candidate date format. The logic may be refined for e.g. lossless imports where the stakes are higher, but for now we're just trying to make type conversions smoother.
- Uses guessDateFormat to guess widget options when changing column type to date or datetime.
- Uses the date format of the original column when possible instead of guessing.
- Fixes a bug where choices were guessed based on the display column instead of the visible column, which made the guessed choices influenced by which values were referenced as well as completely broken when converting from reflist.
- @dsagal @georgegevoian This builds on https://phab.getgrist.com/D3265, currently unmerged. That diff was created first to alert to the change. Without it there would still be similar test failures/changes here as the date format would often be concretely guessed and saved as YYYY-MM-DD instead of being left as the default `undefined` which is shows as YYYY-MM-DD in the dropdown.
Test Plan: Added a unit test to `parseDate.ts`. Updated several browser tests which show the guessing in action during type conversion quite nicely.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: dsagal, georgegevoian
Differential Revision: https://phab.getgrist.com/D3264
Summary:
The rollout of the new sign-up page on prod caused a few tests
that previously expected to be on the Cognito sign-up page to fail.
Test Plan: N/A (fixing failing tests).
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3279
Summary:
Available at login.getgrist.com/signup, the new sign-up page
includes similar options available on the hosted Cognito sign-up
page, such as support for registering with Google. All previous
redirects to Cognito for sign-up should now redirect to the new
Grist sign-up page.
Login is still handled with the hosted Cognito login page, and there
is a link to go there from the new sign-up page.
Test Plan: Browser, project and server tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3249
Summary:
- Brings in a new utility `contextMenu` to open context menu next to the mouse position
- Use this utility to show a CellContextMenu, that sort of merge cell context menu and column context menu together.
- Show cell context menu on context click on any grid's cell.
- Also takes care of showing the row context menu for detail view on a context click that occurs on cells and not only on the row num header as it was the case prior to this diff.
- task: https://gristlabs.getgrist.com/doc/check-ins/p/5#a1.s9.r1529.c31
- discussion: https://grist.quip.com/ETGkAroLnc0Y/Cell-Context-Menu
{F40092}
Test Plan: - Adds project test and nbrowser for cell context menu and new cases for the detail row context menu.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3237
Summary: This is https://phab.getgrist.com/D3205 plus some changes (https://github.com/dsagal/grist/compare/type-convert...type-convert-server?expand=1) that move the conversion process to the backend. A new user action ConvertFromColumn uses `call_external` so that the data engine can delegate back to ActiveDoc. Code for creating formatters and parsers is significantly refactored so that most of the logic is in `common` and can be used in different ways.
Test Plan: The original diff adds plenty of tests.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3240
Summary: This is the first step towards raw data views, merely adding metadata without any UI. Every 'normal' table now has a widget referenced by `rawViewSectionRef`. It has no parent view/page and cannot actually be viewed for now. The widget is created during the AddTable user action, and the migration creates a widget for existing tables.
Test Plan: Many tests had to be updated, especially tests that listed all view sections and/or fields.
Reviewers: jarek, dsagal
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3232
Summary:
Updating filters when user renames labels in a choice/choice list column.
When there are unsaved filters they are reverted to orginal values (only
for the affected column).
Test Plan: new tests
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3230
Summary:
Improving experience when editing group-by column:
- Disable column rename
- Allow changing most widget properties:
- Color/Background
- Number format
- Date/DateTime format (but not the timezone)
- All toggle options (for toggle column)
- Remove Edit button on Choice Edit
- Changing the underlying column should reset all those options back to the original column.
Test Plan: nbrowser
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3216
Summary:
- Skip welcome questions in HomeIntro
- Make Funnel test less flaky, and increase its timeout
Test Plan: Both tests pass staging_deployment when run locally
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3218
Summary:
The user profile dialog is now a separate page, in preparation
for upcoming work to enable MFA. This commit also contains
some MFA changes, but the UI is currently disabled and the
implementation is limited to software tokens (TOTP) only.
Test Plan:
Updated browser tests for new profile page. Tests for MFAConfig
and CognitoClient will be added in a later diff, once the UI is enabled.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3199
Summary:
Adding configuration options for CustomWidgets.
Custom widgets can now store options (in JSON) in viewSection metadata.
Changes in grist-plugin-api:
- Adding onOptions handler, that will be invoked when the widget is ready and when the configuration is changed
- Adding WidgetAPI - new API to read and save a configuration for widget.
Changes in Grist:
- Rewriting CustomView code, and extracting code that is responsible for showing the iframe and registering Rpc.
- Adding Open Configuration button to Widget section in the Creator panel and in the section menu.
- Custom Widgets can implement "configure" method, to show configuration screen when requested.
Test Plan: Browser tests.
Reviewers: paulfitz, dsagal
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3185
Summary: When an editor is activated by typing, the active view should be scrolled to the active record.
Test Plan: new tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3196
Summary:
Addresses several issues:
- Error 'Cannot modify summary group-by column' when changing Text ->
ChoiceList in the presence of summary tables.
- Error 'ModifyColumn in unexpected position' when changing ChoiceList -> Text
in the presence of summary tables.
- Double-evaluation of trigger formulas in some cases.
Fixes include:
- Fixed verification that summary group-by columns match the underlying ones,
and added comments to explain.
- Avoid updating non-metadata lookups after each doc-action (early lookups
generated extra actions to populate summary tables, causing the 'ModifyColumn
in unexpected position' bug)
- When updating formulas, do update lookups first.
- Made a client-side tweak to avoid a JS error in case of some undos.
Solution to reduce lookups is based on https://phab.getgrist.com/D3069?vs=on&id=12445,
and tests for double-evaluation of trigger formulas are taken from there.
Add a new test case to protect against bugs caused by incorrect order of
evaluating #lookup columns.
Enhanced ChoiceList browser test to check a conversion scenario in the presence
of summary tables, previously triggering bugs.
Test Plan: Various tests added or enhanced.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3184
Summary:
This activates the new tests that can run on a grist-core image,
and will not push a new version to `latest` tag unless those tests
pass.
This also runs a smoke test in the regular CI tests, to confirm the
code continues to work without GRIST_TEST_LOGIN enabled.
Test Plan: tested on a fork of grist-core
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3186
Summary:
* Tie build and run-time docker base images to a consistent version (buster)
* Extend the test login system activated by GRIST_TEST_LOGIN to ease porting tests that currently rely on cognito (many)
* Make org resets work in absence of billing endpoints
* When in-memory session caches are used, add missing invalidation steps
* Pass org information through sign-ups/sign-ins more carefully
* For CORS, explicitly trust GRIST_HOST origin when set
* Move some fixtures and tests to core, focussing on tests that cover existing failures or are in the set of tests run on deployments
* Retain regular `test` target to run the test suite directly, without docker
* Add a `test:smoke` target to run a single simple test without `GRIST_TEST_LOGIN` activated
* Add a `test:docker` target to run the tests against a grist-core docker image - since tests rely on certain fixture teams/docs, added `TEST_SUPPORT_API_KEY` and `TEST_ADD_SAMPLES` flags to ease porting
The tests ported were `nbrowser` tests: `ActionLog.ts` (the first test I tend to port to anything, out of habit), `Fork.ts` (exercises a lot of doc creation paths), `HomeIntro.ts` (a lot of DocMenu exercise), and `DuplicateDocument.ts` (covers a feature known to be failing prior to this diff, the CORS tweak resolves it).
Test Plan: Manually tested via `buildtools/build_core.sh`. In follow up, I want to add running the `test:docker` target in grist-core's workflows. In jenkins, only the smoke test is run. There'd be an argument for running all tests, but they include particularly slow tests, and are duplicates of tests already run (in different configuration admittedly), so I'd like to try first just using them in grist-core to gate updates to any packaged version of Grist (the docker image currently).
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3176
Summary:
- Rather than translate from moment format to that of bootstrap-datepicker, use
the customization methods to format datepicker dates using moment directly.
- Fix issue with parseDate() when format includes tokens like Mo or Do
- Fix issue in parseDateTime() that could produce an off-by-one error in date
depending on local timezone.
- When opening DateEditor, show AltText value if present.
- Add crossorigin=anonymous to scripts that were missing it (including
bootstrap-datepicker), to ensure that errors from them are reported properly
rather than as 'Script error.'
Test Plan:
Added test cases to parseDate() test for low-level fixes; added a
browser test for the fixed DateEditor behavior.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3169
Summary:
Add function parseDateTime which parses a string containing both date and time componenents, intended for parsing pasted strings.
Add DateTimeParser subclass of ValueParser.
Test Plan: Extended parseDate.ts and CopyPaste.ts tests.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3152
Summary: Last position should be stored for document and user.
Test Plan: Updated tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3143
Summary:
This enables ptrace and modifies a nofile limit, to be consistent with staging. The nofile change isn't really needed, it was for gvisor's vfs2 which we are no longer using, it is just for consistency.
This diff also documents the PYTHON_VERSION_ON_CREATION=3 flag active in staging and ~~soon to be active in~~ production.
Test Plan: settings work in staging
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3125
Summary:
Adds links to manage team and go to billing account in
the org menu (opened by clicking the dropdown in the
top-left corner of Grist). Tweaks some wording of items
in both AppHeader and AccountWidget, and adds a link
to create a new team site to the Site Switcher in both
menus.
Also tweaks the UI of UserManager by adding
an animation when the manager is opened from the
doc access dialog.
Test Plan: Browser tests.
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3121
Summary:
Adding sort options for columns.
- Sort menu has a new option "More sort options" that opens up Sort left menu
- Each sort entry has an additional menu with 3 options
-- Order by choice index (for the Choice column, orders by choice position)
-- Empty last (puts empty values last in ascending order, first in descending order)
-- Natural sort (for Text column, compares strings with numbers as numbers)
Updated also CSV/Excel export and api sorting.
Most of the changes in this diff is a sort expression refactoring. Pulling out all the methods
that works on sortExpression array into a single namespace.
Test Plan: Browser tests
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: dsagal, alexmojaki
Differential Revision: https://phab.getgrist.com/D3077
Summary:
Handle reference columns in ViewFieldRec.valueParser.
Extracted code for reuse from ReferenceEditor to look up values in the visible column. While I was at it, also extracted a bit of common code from ReferenceEditor and ReferenceListEditor into a new class ReferenceUtils. More refactoring could be done in this area but it's out of scope.
Changed NTextEditor to use field.valueParser, which affects numeric and reference fields. In particular this means numbers are parsed on data entry, it doesn't change anything for references.
Test Plan:
Added more CopyPaste testing to test references.
Tested entering slightly formatted numbers in NumberFormatting.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3094
Summary: Adds parseDateStrict function based on parseDate, uses it in DateParser subclass of ValueParser.
Test Plan:
Tweaked parseDate test to check parseDateStrict.
Extended test in CopyPaste to test parsing dates as well as numbers.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3088
Summary:
Updates the preview table in Importer to show a diff of changes
when importing into an existing table and updating existing records.
Test Plan: Browser tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3060
Summary:
- Update cookie module, to support modern sameSite settings
- Add a new cookie, grist_sid_status with less-sensitive value, to let less-trusted subdomains know if user is signed in
- The new cookie is kept in-sync with the session cookie.
- For a user signed in once, allow auto-signin is appropriate.
- For a user signed in with multiple accounts, show a page to select which account to use.
- Move css stylings for rendering users to a separate module.
Test Plan: Added a test case with a simulated Discourse page to test redirects and account-selection page.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3047
Summary:
New environmental variable GOOGLE_DRIVE_SCOPE that modifies the scope
requested for Google Drive integration.
For prod it has value https://www.googleapis.com/auth/drive.file which leaves
current behavior (Grist is allowed only to access public files and for private
files - it fallbacks to Picker).
For staging it has value https://www.googleapis.com/auth/drive.readonly which
allows Grist to access all private files, and fallbacks to Picker only when the file is
neither public nor private).
Default value is https://www.googleapis.com/auth/drive.file
Test Plan: manual and existing tests
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3038
Summary:
This is preparatory work for running tests with the new sandbox in jenkins.
* Makes a base image that is now shared by grist servers and jenkins workers.
* Needed to allow jenkins to run `sudo runsc`.
* Converged on port 2020 for ssh to workers and servers.
Test Plan: added one runsc-based test and confirmed it was run
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3029
Summary: We still show up to 5 on regular-width screens.
Test Plan: Browser tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3008
Summary: It is important for linking to be maintained, or user could be gravely misled about material in other sections.
Test Plan: added test
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3003
Summary:
- Grist document has a associated "locale" setting that affects how currency is formatted.
- Currency selector for number format.
Test Plan: not done
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D2977
Summary:
Previously, using "Change Widget" allowed one to change the underlying table,
but would keep the linking settings. This could allow invalid settings which
would sometimes lead to JS errors. These manifested in production as
"UserError: Query error: n is not a function".
- Unset linking settings in this case, to avoid invalid values.
- In case invalid values are encountered (e.g. saved previously), treat them as
unset, to avoid JS errors.
- If an error does occur, report it with a stack trace.
Also, for testing, added 'selectBy' option to gristUtils helpers for using page-widget-picker.
Test Plan: Added test cases for resetting linking, and for ignoring invalid link settings.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D2993
Summary:
* Remove adjustSession hack, interfering with loading docs under saml.
* Allow the anonymous user to receive an empty list of workspaces for
the merged org.
* Behave better on first page load when org is in path - this used to
fail because of lack of cookie. This is very visible in grist-core,
as a failure to load localhost:8484 on first visit.
* Mark cookie explicitly as SameSite=Lax to remove a warning in firefox.
* Make errorPages available in grist-core.
This changes the default behavior of grist-core to now start off in
anonymous mode, with an explicit sign-in step available. If SAML is not configured,
the sign-in operation will unconditionally sign the user in as a default
user, without any password check or other security. The user email is
taken from GRIST_DEFAULT_EMAIL if set. This is a significant change, but
makes anonymous mode available in grist-core (which is convenient
for testing) and makes behavior with and without SAML much more consistent.
Test Plan: updated test; manual (time to start adding grist-core tests though!)
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2980
Summary: UserProfile test fails randomly which is caused by the AccountWidget that loads its data asonchrynously.
Test Plan: Existing tests
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D2964
Summary:
- Add showGristTour preference, and trigger tour automatically.
- Tour is only triggered for new and anonymous users on a personal org, with
edit permission.
- Automatically open the right panel at tour start.
- Don't show tours on mobile, since that's not ready (popups are cut off
and can't be dismissed)
- Cancel previous tour if a new one is somehow started.
- Remove #repeat- trigger hash tags from the URL when the tour starts.
- Ensure Help Center popup is positioned even when left panel is collapsed.
- Polish up the content of the last two cards in the tour.
Test Plan: Added test case for triggering and opening right panel.
Reviewers: alexmojaki, paulfitz
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D2955
Summary:
Staging tests were using an organization named Test Grist,
while local was using one named test-grist. Both are now
named Test Grist, which should help keep things more consistent
between local and deployment test runs.
The inherited line height of template docs in icon view was
different on Windows, cutting off part of the last line of the
description. The description line height should now be fixed
to a reasonable value.
Test Plan: Manually tested CSS fix on a Windows machine.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D2948
Summary:
The old Examples and Templates workspace is now
a page that pulls templates from a new public Grist Templates org.
The All Documents view will pull featured templates from that org, where
featured templates are simply pinned documents in Grist Templates. The
Examples and Templates page will also show the featured templates, as
well as the rest of the available templates organized by category. The
categories are equivalent to workspaces in Grist Templates, and are
generated dynamically.
Test Plan: Browser tests.
Reviewers: paulfitz, dsagal
Reviewed By: paulfitz, dsagal
Subscribers: dsagal, paulfitz, jarek
Differential Revision: https://phab.getgrist.com/D2930
Summary:
This applies some mitigations suggested by SQLite authors when
opening untrusted SQLite databases, as we do when Grist docs
are uploaded by the user. See:
https://www.sqlite.org/security.html#untrusted_sqlite_database_files
Steps implemented in this diff are:
* Setting `trusted_schema` to off
* Running a SQLite-level integrity check on uploads
Other steps will require updates to our node-sqlite3 fork, since they
are not available via the node-sqlite3 api (one more reason to migrate
to better-sqlite3).
I haven't yet managed to create a file that triggers an integrity
check failure without also being detected as corruption by sqlite
at a more basic level, so that is a TODO for testing.
Test Plan:
existing tests pass; need to come up with exploits to
actually test the defences and have not yet
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2909
Summary:
Finishing implementation for google drive plugin.
- Refactoring plugin code to make it more robust and to follow grist ux
- Changing the way server hosts untrusted user content, from different domain to different port
Test Plan: Browser tests
Reviewers: dsagal, paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2881
Summary:
User can freeze any number of columns, which will not move when a user scrolls grid horizontally.
Main use cases:
- Frozen columns don't move when a user scrolls horizontally
- The number of frozen columns is automatically persisted
- Readonly viewers see frozen columns and can modify them - but the change is not persisted
- On a small screen - frozen columns still moves to the left when scrolled, to reveal at least one column
- There is a single menu option - Toggle freeze - which offers the best action considering selected columns
- When a user clicks a single column - action to freeze/unfreeze is always there
- When a user clicks multiple columns - action is offered only where it makes sens (columns are near the frozen border)
Test Plan: Browser tests
Reviewers: dsagal, paulfitz
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2852
Summary:
Grist should not prevent read-only viewers from opening cell editors since they usually provide much more information than is visible in a cell.
Every editor was enhanced with a read-only mode that provides the same information available for an editor but doesn't allow to change the underlying data.
Test Plan: Browser tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2842
Summary:
Replaces https://phab.getgrist.com/D2854
Refactoring of NSandbox:
- Simplify arguments to NSandbox.spawn. Only half the arguments were used depending on the flavour, adding a layer of confusion.
- Ensure the same environment variables are passed to both flavours of sandbox
- Simplify passing down environment variables.
Implement deterministic mode with libfaketime and a seeded random instance.
- Include static prebuilt libfaketime.so.1, may need another solution in future for other platforms.
Recording pycalls:
- Add script recordDocumentPyCalls.js to open a single document outside of tests.
- Refactor out recordPyCalls.ts to support various uses.
- Add afterEach hook to save all pycalls from server tests under $PYCALLS_DIR
- Make docTools usable without mocha.
- Add useLocalDoc and loadLocalDoc for loading non-fixture documents
Test Plan:
Made a document with formulas NOW() and UUID()
Compare two document openings in normal mode:
diff <(test/recordDocumentPyCalls.js samples/d4W6NrzCMNVSVD6nWgNrGC.grist /dev/stdout) \
<(test/recordDocumentPyCalls.js samples/d4W6NrzCMNVSVD6nWgNrGC.grist /dev/stdout)
Output:
< 1623407499.58132,
---
> 1623407499.60376,
1195c1195
< "B": "bd2487f6-63c9-4f02-bbbc-5c0d674a2dc6"
---
> "B": "22e1a4fd-297f-4b86-91a2-bc42cc6da4b2"
`export DETERMINISTIC_MODE=1` and repeat. diff is empty!
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2857
Summary: Summaries test works only in a headless mode. In browser mode it failes as section is to narrow.
Test Plan: n/a
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2859
Summary:
We used tslint earlier, and on switching to eslint, some rules were not
transfered. This moves more rules over, for consistent conventions or helpful
warnings.
- Name private members with a leading underscore.
- Prefer interface over a type alias.
- Use consistent spacing around ':' in type annotations.
- Use consistent spacing around braces of code blocks.
- Use semicolons consistently at the ends of statements.
- Use braces around even one-liner blocks, like conditionals and loops.
- Warn about shadowed variables.
Test Plan: Fixed all new warnings. Should be no behavior changes in code.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2831
Summary: Grist document, when reloaded, is able to restore the latest cursor position and the editor state.
Test Plan: Browser test were created.
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D2808
Summary: Text editor for Integer and Numeric column was showing null or undefined when the underlying value was null.
Test Plan: Browser test
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2817
Summary: This treats newRec in the same way as rec in access formulas.
Test Plan: updated test for column renames; autocomplete checked manually.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2810
Summary:
- Supports auto-completion
- Supports various keyboard interactions to delete/select/copy-paste tokens
- Supports various mouse interactions to select and move tokens.
- Supports undo/redo while editing tokens.
Test Plan: stand-alone fixture and browser test verify the supported behaviors
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2789
Summary:
- New sort and filter button has several states
- Empty / unsaved / saved
- offers small save/revert button when unsaved
- Fix little issue with hanging tooltip when the refElem is disposed.
- The problem was that if you hover the save (or revert) button
and then click the button, it causes the button to disappear,
but the tooltip was staying.
Test Plan: Updated all tests to match the new UI.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: dsagal, paulfitz
Differential Revision: https://phab.getgrist.com/D2795
Summary: Select all functionality by clicking the upper left margin corner
Test Plan: Browser test created
Reviewers: dsagal, paulfitz
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2793
Summary:
This helps forcibly end mocha tests when they hang, and print out
something that may help debug the situation.
Also add the generated static/bundle.css file to core/.gitignore.
Also, avoid using npm-packages-offline-cache when building core, by avoiding use of .yarnrc which turns it on.
Test Plan: Tested manually
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2788
Summary:
- Update rules to be more like we've had with tslint
- Switch tsserver plugin to eslint (tsserver makes for a much faster way to lint in editors)
- Apply suggested auto-fixes
- Fix all lint errors and warnings in core/, app/, test/
Test Plan: Some behavior may change subtly (e.g. added missing awaits), relying on existing tests to catch problems.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2785
Summary:
* adds a smoke test to grist-core
* fixes a problem with highlight.js failing to load correctly
* skips survey for default user
* freshens docker build
Utility files in test/nbrowser are moved to core/test/nbrowser, so that gristUtils are available there. This increased the apparent size of the diff as "./" import paths needed replacing with "test/nbrowser/" paths. The utility files are untouched, except for the code to start a server - it now has a small grist-core specific conditional in it.
Test Plan: adds test
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2768
