Commit Graph

662 Commits

Author SHA1 Message Date
Paul Fitzpatrick
3a8e7032bc (core) updates from grist-core 2022-03-14 09:18:44 -04:00
George Gevoian
ad1b4f3cff (core) Record new user sign-ups
Summary:
Adds Google Tag Manager snippet to all login pages, and a new user
preference, recordSignUpEvent, that's set to true on first sign-in. The
client now checks for this preference, and if true, dynamically loads
Google Tag Manager to record a sign-up event. Afterwards, it removes
the preference.

Test Plan: Tested manually.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3319
2022-03-12 14:34:46 -08:00
Jarosław Sadziński
eff78ae2e1 (core) New UI for raw data views
Summary:
Creating new UI for raw data views based on design.
- Renaming left for follow up diff
- Link in the menu is hidden for now
To access raw UI, use /p/data URL.

Test Plan: new tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3306
2022-03-12 13:51:48 +01:00
Alex Hall
2d0978559b (core) Replace compute stack and frames with _current_node and _current_row_id
Summary:
This is an attempt to optimise Engine._use_node. It doesn't seem to actually improve overall performance significantly, but it shouldn't make it worse, and I think it's an improvement to the code.

It turns out that there's no need to track a stack of compute frames any more. The only time we get close to nested evaluation, we set allow_evaluation=False to prevent it actually happening. So there's only one 'frame' during actual evaluation, which means we can get rid of the concept of frames entirely. This allows simplifying the code and letting the computer do less work in general.

Test Plan: this

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3310
2022-03-11 12:34:00 +02:00
George Gevoian
f02174eb7e (core) Fix error when canceling import
Summary:
If cancel was clicked while a transform section was still being
generated in the Importer, an error was thrown. This refactors
the cancelImportFiles API action to take in the file upload id
in place of the entire DataSourceTransformed parameter, which
contains other values that are irrelevant to canceling. One of those
values, the transform section id, was causing the error to be thrown
since it was momentarily null.

Test Plan: Tested manually.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Differential Revision: https://phab.getgrist.com/D3317
2022-03-10 16:24:49 -08:00
Paul Fitzpatrick
00eb44fa00 v0.7.6 2022-03-09 10:11:30 -05:00
Alex Hall
77a5d31afe (core) More accurate data size measurement
Summary: As suggested by @dsagal in https://phab.getgrist.com/D3277#inline-36801, change to query `SUM(pgsize - unused)` instead of `SUM(pgsize)` to measure actual data size more accurately. Technically this doesn't reflect the database file size as accurately, but it should reflect sandbox memory usage better, and more importantly it should allow users to see data size decreasing when they delete stuff.

Test Plan: Tested manually by adding rows to a doc and looking at the logs. The data size is smaller and changes more granularly.

Reviewers: dsagal, paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3313
2022-03-09 12:04:16 +02:00
Alex Hall
222dc505eb (core) Generate Engine._autocomplete_context lazily
Summary: I ran the python tests through a profiler and found that just generating the autocomplete context was a major cost, and this would happen with every schema change. By only generating it when needed for autocomplete, the time for tests reduced from ~38s to ~33s.

Test Plan: this

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3312
2022-03-09 12:03:11 +02:00
Jarosław Sadziński
351d70d4fb (core) Serving widget info page from home url
Summary:
Custom widget into page is served from a homeUrl instead
of untrusted URL, which might be not used in grist-core.

Test Plan: manual test

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3307
2022-03-09 10:34:50 +01:00
Jarosław Sadziński
d2b82b84c7 (core) Fixing bug with resuming search on a hidden column.
Summary: Fix for error that happens when a search is resumed after one of the columns was hidden.

Test Plan: Added test that shows the error.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3309
2022-03-09 10:34:17 +01:00
George Gevoian
7ead97b913 (core) Fill name on test login page when possible
Summary:
Updates simulateLogin to fill in the name field of
the test login page. Docker tests were failing because
users created via the test login page were falling back
to their email for their name.

Test Plan: N/A (fixing Docker tests)

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3315
2022-03-08 20:29:39 -08:00
Paul Fitzpatrick
c4d3d7d3bb (core) be careful when reassigning a doc to a worker it was on before
Summary:
Importing a .grist document is implemented in a somewhat clunky way, in a multi-worker setup.

 * First a random worker receives the upload, and updates Grist's various stores appropriately (database, redis, s3).
 * Then a random worker is assigned to serve the document.

If the worker serving the document fails, there is a chance the it will end up assigned to the worker that handled its upload. Currently the worker will misbehave in this case. This diff:

 * Ports a multi-worker test from test/home to run in test/s3, and adds a test simulating a bad scenario seen in the wild.
 * Fixes persistence of any existing document checksum in redis when a worker is assigned.
 * Adds a check when assigned a document to serve, and finding that document already cached locally. It isn't safe to rely only on the document checksum in redis, since that may have expired.
 * Explicitly claims the document on the uploading worker, so this situation becomes even less likely to arise.

Test Plan: added test

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3305
2022-03-08 17:20:01 -05:00
Alex Hall
321019217d (core) Lossless imports
Summary:
- Removed string parsing and some type guessing code from parse_data.py. That logic is now implicitly done by ValueGuesser by leaving the initial column type as Any. parse_data.py mostly comes into play when importing files (e.g. Excel) containing values that already have types, i.e. numbers and dates.
- 0s and 1s are treated as numbers instead of booleans to keep imports lossless.
- Removed dateguess.py and test_dateguess.py.
- Changed what `guessDateFormat` does when multiple date formats work equally well for the given data, in order to be consistent with the old dateguess.py.
- Columns containing numbers are now always imported as Numeric, never Int.
- Removed `NullIfEmptyParser` because it was interfering with the new system. Its purpose was to avoid pointlessly changing a column from Any to Text when no actual data was inserted. A different solution to that problem was already added to `_ensure_column_accepts_data` in the data engine in a recent related diff.

Test Plan:
- Added 2 `nbrowser/Importer2` tests.
- Updated various existing tests.
- Extended testing of `guessDateFormat`. Added `guessDateFormats` to show how ambiguous dates are handled internally.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3302
2022-03-08 12:14:39 +02:00
George Gevoian
9522438967 (core) Add Grist forgot password page
Summary:
The page isn't yet linked to from anywhere in the UI, but
will be soon, once the new login page is ready. The page
can still be accessed at login-[s].getgrist.com/forgot-password,
and the flow is similar to the one used by Cognito's hosted UI.

Also refactors much of the existing login app code into smaller
files with less duplication, tweaks password validation to be closer
to Cognito's requirements, and polishes various parts of the UI,
like the verified page CSS, and the form inputs.

Test Plan: Browser, server and project tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3296
2022-03-07 09:11:28 -08:00
Paul Fitzpatrick
aa3fe975e7 (core) updates from grist-core 2022-03-07 09:59:50 -05:00
Cyprien P
76481d21e0 (core) Enable selecting x axis for the group data columns
Summary:
Until now, users could not pick the column that's currently serving as
the x axis.

To do that, chart needs to support having the x axis undefined.

This diff do just that:
  - allow x axis to be undefined
  - allow setting x axis from group data

Given that charts axis are stored as indexes of the section view
fields array, implementation required introduction of an extra chart
options: `isAxisUndefined`.

Test Plan: Updates existing test and adds new one.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3304
2022-03-07 10:49:18 +01:00
Paul Fitzpatrick
2563fb745a (core) make Grist easier to run with a single server
Summary:
This makes many small changes so that Grist is less fussy to run as a single instance behind a reverse proxy. Some users had difficulty with the self-connections Grist would make, due to internal network setup, and since these are unnecessary in any case in this scenario, they are now optimized away. Likewise some users had difficulties related to doc worker urls, which are now also optimized away. With these changes, users should be able to get a lot further on first try, at least far enough to open and edit documents.

The `GRIST_SINGLE_ORG` setting was proving a bit confusing, since it appeared to only work when set to `docs`. This diff
adds a check for whether the specified org exists, and if not, it creates it. This still depends on having a user email to make as the owner of the team, so there could be remaining difficulties there.

Test Plan: tested manually with nginx

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3299
2022-03-05 13:30:45 -05:00
Cyprien P
0da397ab90 (core) Fix js error when changing X Axis with chart with no series
Summary:
This error only occured for certain charts type. Some chart handled it correctly, others did not.

This brings a more general handling of this issue which should work for all charts.

Test Plan: New nbrowser test case.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3303
2022-03-04 14:06:19 +01:00
Alex Hall
5213972d24 (core) Guess numeric formatting options
Summary:
Change NumberParse.parse to return not just the parsed number but also information it gathered along the way about how the input string was formatted.

Use this in the new NumberParse.guessOptions to guess the actual widget options based on an array of strings.

Use NumberParse.guessOptions in TypeConversion (for when a user explicitly chooses to change type) and in ValueGuesser (for guesses about strings entered into empty columns).

Test Plan: Adds unit tests for NumberParse and ValueGuesser and updates the TypeChange2 nbrowser test.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3294
2022-03-03 21:32:03 +02:00
George Gevoian
cc1af85594 (core) Add missing .isPresent() to isOnTestLoginPage
Summary:
A staging test was failing because the util was throwing instead
of returning true or false.

Test Plan: N/A (fixing test)

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3301
2022-03-02 10:10:26 -08:00
Jarosław Sadziński
70373550cf (core) Shortcuts help page can be scrolled using a keyboard
Summary: Restoring keyboard navigation on the shortcuts popup.

Test Plan: manual tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3295
2022-03-02 11:43:05 +01:00
George Gevoian
10097c5423 (core) Add check for test login page in homeUtil.ts
Summary:
The old regex for login pages, /gristlogin/, used to handle
test login pages too, matching on a query param set in the
URL. A recent change to make the regex more specific broke this.

This fixes that regression by adding an additional check to
isOnLoginPage for the presence of the test login page.

Test Plan: N/A (fixing test)

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3300
2022-03-01 15:44:58 -08:00
George Gevoian
8f5ce9d610 (core) Wait for transform column before finalizing
Summary:
This adds a new private field to ColumnTransform that settles when
the transform column is done preparing. Finalize then waits for this
promise before proceeding.

A bug was occurring when the prepare promise was still pending,
but finalize was triggered indirectly, causing finalize to throw an error
due to the transform column not yet existing.

Test Plan: Tested manually.

Reviewers: jarek, dsagal

Reviewed By: jarek, dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3288
2022-03-01 13:59:14 -08:00
Alex Hall
599545fb11 (core) Fuller guessing of type and options when adding first data to blank columns
Summary:
Adds `common/ValueGuesser.ts` with logic for guessing column type and widget options (only for dates/datetimes) from an array of strings, and converting the strings to the guessed type in a lossless manner, so that converting back to Text gives the original values.

Changes `_ensure_column_accepts_data` in Python to call an exported JS method using the new logic where possible.

Test Plan: Added `test/common/ValueGuesser.ts` to unit test the core guessing logic and a DocApi end-to-end test for what happens to new columns.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3290
2022-03-01 22:00:45 +02:00
Paul Fitzpatrick
99ee495b4b v0.7.5 2022-02-28 16:52:44 -05:00
Jarosław Sadziński
ae6c857ac5 (core) Frozen columns reordering
Summary: Bug fix. Moving columns over the frozen set didn't work properly.

Test Plan: new tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3297
2022-02-28 22:52:37 +01:00
George Gevoian
fa68b790bb (core) Remove code for unused welcome flows
Summary: Removes code that was marked for removal.

Test Plan: Existing tests still pass.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3289
2022-02-28 13:21:28 -08:00
George Gevoian
83ba2957bf (core) Update failing HomeIntro core test
Summary:
Core doesn't redirect to Cognito or our own sign-up page
when clicking 'sign up' on the welcome screen. Instead, it
redirects to the test login page.

Test Plan: N/A (fixing test)

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3298
2022-02-28 12:23:28 -08:00
Paul Fitzpatrick
c91593e312 (core) make cognito logins for tests work on slightly smaller screens
Summary:
A selfish diff to tweak deployment tests so they pass on a
slightly smaller laptop screen that for whatever reason trips
a css breakpoint on the cognito login page.

Test Plan: manual

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3292
2022-02-25 15:04:45 -05:00
Cyprien P
c2070877b6 (core) Fix out of sync GROUP DATA and X Axis options when table change
Summary:
Fix this issue:
  When using “Change Widget” for a chart of a summary table, if I change group-by columns, the X-Axis dropdown becomes empty, and lists values from previous summary (not the actual summary table that’s now shown in the chart). Need to close and reopen creator panel to fix it.

Test Plan: Include nbrowser test

Reviewers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3284
2022-02-25 14:48:27 +01:00
Alex Hall
3445ecb64c (core) Search backwards with Shift+Enter
Summary: As simple as it sounds. Check ev.shiftKey in Enter key handler in search.ts.

Test Plan: Extended test/nbrowser/Search.ts

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3293
2022-02-25 14:07:15 +02:00
George Gevoian
a294eefdff (core) Update HomeIntro.ts to check for either login page
Summary:
The "Sign In" button can either redirect you to the Cognito login
page or the Grist sign-up page, depending on whether you've logged
in before. Since the HomeIntro tests can either be run as part of a full
deployment test suite, or individually, the test could redirect you to
both types of login pages.

Test Plan: N/A (fixing test)

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3291
2022-02-24 15:28:16 -08:00
George Gevoian
ff4e5d2769 (core) Send emails when 2FA settings are updated
Summary: When user 2FA status is changed, we now send out emails via SendGrid.

Test Plan: Server tests.

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3280
2022-02-24 12:36:50 -08:00
George Gevoian
8f1889d2d7 (core) Fix staging test that would timeout
Summary:
Removes code from simulateLogin that's no longer needed, and increases
the timeout on the util function that checks if we're on the Grist sign-up
page.

Test Plan: N/A (fixing test)

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3287
2022-02-23 19:03:41 -08:00
George Gevoian
ac910389bc (core) Make login URL regexes more specific
Summary:
This should resolve some staging test failures. The previous regexes
weren't quite specific enough, so random parts of a URL that had
traces of the real login URLs were causing isOnLoginPage() and
isOnGristLoginPage() to return true instead of false.

Test Plan: N/A (fixing tests)

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3283
2022-02-23 08:51:59 -08:00
Paul Fitzpatrick
accd640078 (core) add a user.SessionID value for trigger formulas and granular access rules
Summary:
This makes a `user.SessionID` value available in information about the user, for use with trigger formulas and granular access rules. The ID should be constant within a browser session for anonymous user. For logged in users it simply reflects their user id.

This ID makes it possible to write access rules and trigger formulas that allow different anonymous users to create, view, and edit their own records in a document.

For example, you could have a brain-storming document for puns, and allow anyone to add to it (without logging in), letting people edit their own records, but not showing the records to others until they are approved by a moderator. Without something like this, we could only let anonymous people add one field of a record, and not have a secure way to let them edit that field or others in the same record.

Also adds a `user.IsLoggedIn` flag in passing.

Test Plan: Added a test, updated tests. The test added is a mini-moderation doc, don't use it for real because it allows users to edit their entries after a moderator has approved them.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3273
2022-02-22 12:50:43 -05:00
George Gevoian
95592b81bd (core) Skip /welcome/user page for new users
Summary:
Since the new Grist sign-up page has a required field for
name, we can now skip the welcome page asking for the
same thing. Code and tests that can be removed later are
marked with TODOs.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3266
2022-02-22 08:38:22 -08:00
George Gevoian
36843e632b (core) Hide API key when it's not selected
Summary:
The API key is now hidden by default. Clicking the input
will select and reveal the key. As soon as the key loses
selection, it is hidden again.

Test Plan: Project test.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3270
2022-02-21 17:26:45 -08:00
Alex Hall
437d30bd9f (core) Log number of rows in user tables in data engine
Summary:
Adds a method Table._num_rows using an empty lookup map column.

Adds a method Engine.count_rows which adds them all up.

Returns the count after applying user actions to be logged by ActiveDoc.

Test Plan: Added a unit test in Python. Tested log message manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3275
2022-02-22 00:59:56 +02:00
Alex Hall
f1002c0e67 (core) Regularly log data size in DocStorage.applyStoredActions using sqlite dbstat
Summary:
- Small cleanup: Make DocStorage implement OnDemandStorage, and remove unused execWithBackup
- Upgrade to new versions (.3) of @gristlabs/sqlite3 and connect-sqlite3 to use dbstat
- Add _logDataSize method which queries dbstat, adding up pgsize for tables loaded into the data engine
- Only complete _logDataSize every 5 minutes using new field _lastLoggedDataSize

Test Plan: Tested manually

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3277
2022-02-22 00:59:04 +02:00
Alex Hall
4c935e7fb3 (core) Import Hashable from six.moves.collections_abc
Summary: Copy of https://github.com/gristlabs/grist-core/pull/136

Test Plan: this

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3278
2022-02-22 00:27:51 +02:00
Paul Fitzpatrick
21f2765a59 (core) updates from grist-core 2022-02-21 17:18:37 -05:00
Paul Fitzpatrick
88fe090032
Merge pull request #136 from tirkarthi/fix-abc
Import ABC through six.moves for Python 2/3 compatibility.
2022-02-21 17:18:05 -05:00
Alex Hall
5b352211c4 (core) Guess date format during type conversion
Summary:
- Adds a dependency moment-guess (https://github.com/apoorv-mishra/moment-guess) to guess date formats from strings. However the npm package is missing source maps which leads to an ugly warning, so currently using a fork until https://github.com/apoorv-mishra/moment-guess/pull/22 is resolved.
- Adds guessDateFormat using moment-guess to determine the best candidate date format. The logic may be refined for e.g. lossless imports where the stakes are higher, but for now we're just trying to make type conversions smoother.
- Uses guessDateFormat to guess widget options when changing column type to date or datetime.
- Uses the date format of the original column when possible instead of guessing.
- Fixes a bug where choices were guessed based on the display column instead of the visible column, which made the guessed choices influenced by which values were referenced as well as completely broken when converting from reflist.
- @dsagal @georgegevoian This builds on https://phab.getgrist.com/D3265, currently unmerged. That diff was created first to alert to the change. Without it there would still be similar test failures/changes here as the date format would often be concretely guessed and saved as YYYY-MM-DD instead of being left as the default `undefined` which is shows as YYYY-MM-DD in the dropdown.

Test Plan: Added a unit test to `parseDate.ts`. Updated several browser tests which show the guessing in action during type conversion quite nicely.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal, georgegevoian

Differential Revision: https://phab.getgrist.com/D3264
2022-02-21 22:39:47 +02:00
George Gevoian
481122b4d3 (core) Fix failing login tests
Summary:
The rollout of the new sign-up page on prod caused a few tests
that previously expected to be on the Cognito sign-up page to fail.

Test Plan: N/A (fixing failing tests).

Reviewers: alexmojaki

Reviewed By: alexmojaki

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3279
2022-02-21 11:50:30 -08:00
Paul Fitzpatrick
be8c053922 (core) updates from grist-core 2022-02-21 13:44:13 -05:00
Paul Fitzpatrick
e73e74b8ad
Merge pull request #141 from N6UDP/patch-1
Fix typo in email fallback for SAML
2022-02-20 15:11:00 -05:00
Lee Burton
9c47b9cdee
Fix typo in email fallback for SAML
It looks like nameId doesn't exist as a property but name_id does (as is used elsewhere in the function)
2022-02-20 02:39:30 -08:00
Paul Fitzpatrick
6d941fb5a3
Merge pull request #137 from EdwardBetts/spelling
Correct spelling mistakes
2022-02-19 21:19:31 -05:00
Edward Betts
d6e0e1fee3 Correct spelling mistakes 2022-02-19 09:46:49 +00:00