Commit Graph

24 Commits

Author SHA1 Message Date
Jonathan Perret
96b652fb52
Support HTTP long polling as an alternative to WebSockets (#859)
The motivation for supporting an alternative to WebSockets is that while all browsers supported by Grist offer native WebSocket support, some networking environments do not allow WebSocket traffic.

Engine.IO is used as the underlying implementation of HTTP long polling. The Grist client will first attempt a regular WebSocket connection, using the same protocol and endpoints as before, but fall back to long polling using Engine.IO if the WebSocket connection fails.

Include these changes:
- CORS websocket requests are now rejected as a stronger security measure. This shouldn’t affect anything in practice; but previously it could be possible to make unauthenticated websocket requests from another origin.
- GRIST_HOST variable no longer affects CORS responses (also should not affect anything in practice, as it wasn't serving a useful purpose)
2024-03-28 13:22:20 -04:00
Dmitry S
fc44a60edf (core) When reporting email in log metadata, use normalized email.
Summary:
There has been inconsistency in using display email vs normalized email, which
ends up creating some duplication in downstream analyses (e.g. the same user
showing up twice with different capitalization).

1. Add UserProfile.loginEmail field with normalized email to prefer, when set, over the inconsistently used UserProfile.email.
2. In one place where it's not available, normalize the display email manually.
3. Clean up some code in Client.ts.

Unrelated tweak to API Console to be clear when a URL parameter wasn't found (rather than show whatever happens to be the first value).

Several test robustness improvements:
- Misplaced parenthesis in gristWebDriverUtils has been causing optTimeout argument to be ignored in tests, and treated always as indefinite.
- Attempt to fix SortMenu test by ignoring (retrying with logging) errors in waitForServer, which include "script timeout" errors that come from a non-configurable selenium or chromedriver timeout.
- Attempt to improve onNewTab() helper, which plays a role in failing Billing tests.

Test Plan: Tested manually the capitalization of logged emails. Counting on existing tests to catch issues.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4188
2024-02-15 10:49:01 -05:00
Florent
5533b9b7ee
Fix server crash when client passes malformed JSON (#826)
* Fix server crash when client passes malformed JSON

* Take remarks into account

---------

Co-authored-by: Florent FAYOLLE <florent.fayolle@beta.gouv.fr>
2024-01-23 12:07:39 -05:00
Dmitry S
4d9bbf6263 (core) Exit more cleanly on unhandled errors, and handle errors writing to Clients.
Summary:
- Node has a strong recommendation to assume bad state and exit promptly on
  unhandled exceptions and rejections. We follow it, and only make an effort to
  clean up before exiting, and to log the error in a more standard way.

- The only case seen in recent month of an unhandled rejection was for
  attempting to write overly large JSON to a Client websocket. Ensure that's
  handled, and add a test case that artificially reproduces this scenario.

Test Plan:
Added a test case for failing write to Client, and a test case that unhandled
errors indeed kill the server but with an attempt at cleanup.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D4124
2023-12-01 09:42:00 -05:00
Dmitry S
526a5df157 (core) Manage memory used for websocket responses to reduce the risk of server crashes.
Summary:
- Implements MemoryPool for waiting on memory reservations.
- Uses MemoryPool to control memory used for stringifying JSON responses in Client.ts
- Limits total size of _missedMessages that may be queued for a particular client.
- Upgrades ws library, which may reduce memory usage, and allows pausing the websocket for testing.
  - The upgrade changed subtle behavior corners, requiring various fixes to code and tests.

- dos.ts:
  - Includes Paul's fixes and updates to the dos.ts script for manual stress-testing.
  - Logging tweaks, to avoid excessive dumps on uncaughtError, and include timestamps.

Test Plan:
- Includes a test that measures heap size, and fails without memory management.
- Includes a unittest for MemoryPool
- Some cleanup and additions to TestServer helper; in particular adds makeUserApi() helper used in multiple tests.
- Some fixes related to ws upgrade.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3974
2023-08-07 11:28:31 -04:00
George Gevoian
10f5f0cb37 (core) Add optional telemetry to grist-core
Summary:
Adds support for optional telemetry to grist-core.

A new environment variable, GRIST_TELEMETRY_LEVEL, controls the level of telemetry collected.

Test Plan: Server and unit tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal, anaisconce

Differential Revision: https://phab.getgrist.com/D3880
2023-06-07 12:00:51 -04:00
George Gevoian
1e873b4203 (core) Tweak telemetry
Summary: Adjusts the level of telemetry collected from Grist SaaS.

Test Plan: Tested manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3899
2023-05-19 13:06:15 -04:00
Jarosław Sadziński
9628253fd8 (core) Adding new column in users table "ref" with unique identifier.
Summary:
There is a new column in users table called ref (user reference).
It holds user's unique reference number that can be used for features
that require some kind of ownership logic (like comments).

Test Plan: Updated tests

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian, paulfitz

Differential Revision: https://phab.getgrist.com/D3641
2022-10-04 15:19:28 +02:00
Jarosław Sadziński
5219932a1f (core) i18
Summary:
Adding initial work for localization support.

Summary in https://grist.quip.com/OtZKA6RHdQ6T/Internationalization-and-Localization

Test Plan: Not yet

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3633
2022-09-29 18:02:09 +02:00
Alex Hall
1a6e1872de (core) Miscellaneous little logging improvements
Summary:
1. Log errors in `ActiveDoc.loadDoc` as errors, not just warnings, except for a common 'Cannot create fork' error caused by deployment tests.

2. Log the method name that had an error in `server/lib/Client.ts`.

Discussion: https://grist.slack.com/archives/CR8HZ4P9V/p1652364998893169

Following up on https://phab.getgrist.com/D3522

Test Plan: tested manually, particularly by running the nbrowser/Fork test that led to the initial noisy errors in Slack.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3525
2022-07-15 00:21:44 +02:00
Dmitry S
51ff72c15e (core) Faster builds all around.
Summary:
Building:
- Builds no longer wait for tsc for either client, server, or test targets. All use esbuild which is very fast.
- Build still runs tsc, but only to report errors. This may be turned off with `SKIP_TSC=1` env var.
- Grist-core continues to build using tsc.
- Esbuild requires ES6 module semantics. Typescript's esModuleInterop is turned
  on, so that tsc accepts and enforces correct usage.
- Client-side code is watched and bundled by webpack as before (using esbuild-loader)

Code changes:
- Imports must now follow ES6 semantics: `import * as X from ...` produces a
  module object; to import functions or class instances, use `import X from ...`.
- Everything is now built with isolatedModules flag. Some exports were updated for it.

Packages:
- Upgraded browserify dependency, and related packages (used for the distribution-building step).
- Building the distribution now uses esbuild's minification. babel-minify is no longer used.

Test Plan: Should have no behavior changes, existing tests should pass, and docker image should build too.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3506
2022-07-04 10:42:40 -04:00
Dmitry S
a91d493ffc (core) Fix issue with 'UNEXPECTED ORDER OF CALLBACKS' in Client.ts.
Summary:
- Substantial refactoring of the logic when the server fails to send some
  messages to a client.
- Add seqId numbers to server messages to ensure reliable order.
- Add a needReload flag in clientConnect for a clear indication whent the
  browser client needs to reload the app.
- Reproduce some potential failure scenarios in a test case (some of which
  previously could have led to incorrectly ordered messages).
- Convert other Comm tests to typescript.
- Tweak logging of Comm and Client to be slightly more concise (in particular,
  avoid logging sessionId)

Note that despite the big refactoring, this only addresses a fairly rare
situation, with websocket failures while server is trying to send to the
client. It includes no improvements for failures while the client is sending to
the server.

(I looked for an existing library that would take care of these issues. A relevant article I found is https://docs.microsoft.com/en-us/azure/azure-web-pubsub/howto-develop-reliable-clients, but it doesn't include a library for both ends, and is still in review. Other libraries with similar purposes did not inspire enough confidence.)

Test Plan: New test cases, which reproduce some previously problematic scenarios.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3470
2022-06-16 23:51:14 -04:00
Dmitry S
4f1cb53b29 (core) Converting server-side Comm.js to typescript
Summary:
- Add app/common/CommTypes.ts to define types shared by client and server.
- Include @types/ws npm package

Test Plan: Intended to have no changes in behavior

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3467
2022-06-07 15:47:17 -04:00
Paul Fitzpatrick
ce7eb05ed4 (core) get user.Name through same mechanism as user.id for websocket Client
Summary:
This avoids an extra database query to look up the user's current
name, by capturing it at the moment their user id is queried.

Test Plan: existing test for user.Name changes continues to pass

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3381
2022-04-14 12:49:35 -04:00
George Gevoian
4c5de16e2d (core) Include altSessionId in logs
Summary: Adds altSessionId to log output.

Test Plan: Tested manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3355
2022-04-08 16:40:34 -07:00
George Gevoian
6305811ca6 (core) Add new Grist login page
Summary:
Adds a new Grist login page to the login app, and replaces the
server-side Cognito Google Sign-In flow with Google's own OAuth flow.

Test Plan: Browser and server tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3332
2022-04-01 15:24:19 -07:00
Paul Fitzpatrick
accd640078 (core) add a user.SessionID value for trigger formulas and granular access rules
Summary:
This makes a `user.SessionID` value available in information about the user, for use with trigger formulas and granular access rules. The ID should be constant within a browser session for anonymous user. For logged in users it simply reflects their user id.

This ID makes it possible to write access rules and trigger formulas that allow different anonymous users to create, view, and edit their own records in a document.

For example, you could have a brain-storming document for puns, and allow anyone to add to it (without logging in), letting people edit their own records, but not showing the records to others until they are approved by a moderator. Without something like this, we could only let anonymous people add one field of a record, and not have a secure way to let them edit that field or others in the same record.

Also adds a `user.IsLoggedIn` flag in passing.

Test Plan: Added a test, updated tests. The test added is a mini-moderation doc, don't use it for real because it allows users to edit their entries after a moderator has approved them.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3273
2022-02-22 12:50:43 -05:00
Dmitry S
f2f4fe0eca (core) Add LogMethods helper and use it for more JSON data in logs. Reduce unhelpful logging.
Summary:
- Sharing, Client, DocClients, HostingStorageManager all include available info.
- In HostingStorageManager, log numSteps and maxStepTimeMs, in case that helps
  debug SQLITE_BUSY problem.
- Replace some action-bundle logging with a JSON version aggregating some info.
- Skip logging detailed list of actions in production.

Test Plan: Tested manually by eyeballing log output in dev environment.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3086
2021-10-25 10:25:18 -04:00
George Gevoian
a6e08883e0 (core) Simple localization support and currency selector.
Summary:
- Grist document has a associated "locale" setting that affects how currency is formatted.
- Currency selector for number format.

Test Plan: not done

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D2977
2021-08-26 13:36:49 -07:00
Dmitry S
869b2f00ec (core) Remove LoginSession, which was mainly serving situations that are no longer used.
Summary:
In the past, Cognito sign-ins were intended to give authorization to some AWS
services (like SQS); various tokens were stored in the session for this
purpose. This is no longer used. Profiles from Cognito now serve a limited
purpose: first-time initialization of name and picture, and keeping track of
which login method was used. For these remaining needs, ScopedSession is
sufficient.

Test Plan:
Existing test pass. Tested manually that logins work with Google and
Email + Password. Tested manually that on a clean database, name and picture
are picked up from a Google Login.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2907
2021-07-12 13:04:00 -04:00
Paul Fitzpatrick
47ea00dea3 (core) add user and docId to heartbeat logging
Summary: add user and docId to heartbeat logging

Test Plan: checked manually

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2784
2021-04-23 17:33:06 -04:00
Paul Fitzpatrick
6af811f7ab (core) give more detailed reasons for access denied when memos are present
Summary:
With this change, if a comment is added to an ACL formula, then that comment will be offered to the user if access is denied and that rule could potentially have granted access.

The code is factored so that when access is permitted, or when partially visible tables are being filtered, there is little overhead. Comments are gathered only when an explicit denial of access.

Test Plan: added tests, updated tests

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2730
2021-02-15 17:02:24 -05:00
Paul Fitzpatrick
438f259687 (core) start reconciling forking with granular access
Summary:
This allows a fork to be made by a user if:
 * That user is an owner of the document being forked, or
 * That user has full read access to the document being forked.

The bulk of the diff is reorganization of how forking is done.  ActiveDoc.fork is now responsible for creating a fork, not just a docId/urlId for the fork. Since fork creation should not be limited to the doc worker hosting the trunk, a helper endpoint is added for placing the fork.

The change required sanitizing worker allocation a bit, and allowed session knowledge to be removed from HostedStorageManager.

Test Plan: Added test; existing tests pass.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2700
2021-01-12 14:08:49 -05:00
Paul Fitzpatrick
5ef889addd (core) move home server into core
Summary: This moves enough server material into core to run a home server.  The data engine is not yet incorporated (though in manual testing it works when ported).

Test Plan: existing tests pass

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2552
2020-07-21 20:39:10 -04:00