5 Commits (a101337213f44ae6e016cfa775962d679290250b)

Author	SHA1	Message	Date
Dmitry S	fbae81648c	(core) Add options to /status health-check endpoints to check DB and Redis liveness. ... Summary: - /status accepts new optional query parameters: db=1, redis=1, and timeout=<ms> (defaults to 10_000). - These verify that the server can make trivial calls to DB/Redis, and that they return within the timeout. - New HealthCheck tests simulates DB and Redis problems. - Added resilience to Redis reconnects (helped by a test case that simulates disconnects) - When closing Redis-based session store, disconnect from Redis (to avoid hanging tests) Some associated test reorg: - Move stripeTools out of test/nbrowser, and remove an unnecessary dependency, to avoid starting up browser for gen-server tests. - Move TcpForwarder to its own file, to use in the new test. Test Plan: Added a new HealthCheck test that simulates DB and Redis problems. Reviewers: georgegevoian Reviewed By: georgegevoian Differential Revision: https://phab.getgrist.com/D4054	12 months ago
Dmitry S	526a5df157	(core) Manage memory used for websocket responses to reduce the risk of server crashes. ... Summary: - Implements MemoryPool for waiting on memory reservations. - Uses MemoryPool to control memory used for stringifying JSON responses in Client.ts - Limits total size of _missedMessages that may be queued for a particular client. - Upgrades ws library, which may reduce memory usage, and allows pausing the websocket for testing. - The upgrade changed subtle behavior corners, requiring various fixes to code and tests. - dos.ts: - Includes Paul's fixes and updates to the dos.ts script for manual stress-testing. - Logging tweaks, to avoid excessive dumps on uncaughtError, and include timestamps. Test Plan: - Includes a test that measures heap size, and fails without memory management. - Includes a unittest for MemoryPool - Some cleanup and additions to TestServer helper; in particular adds makeUserApi() helper used in multiple tests. - Some fixes related to ws upgrade. Reviewers: paulfitz Reviewed By: paulfitz Differential Revision: https://phab.getgrist.com/D3974	1 year ago
Dmitry S	51ff72c15e	(core) Faster builds all around. ... Summary: Building: - Builds no longer wait for tsc for either client, server, or test targets. All use esbuild which is very fast. - Build still runs tsc, but only to report errors. This may be turned off with `SKIP_TSC=1` env var. - Grist-core continues to build using tsc. - Esbuild requires ES6 module semantics. Typescript's esModuleInterop is turned on, so that tsc accepts and enforces correct usage. - Client-side code is watched and bundled by webpack as before (using esbuild-loader) Code changes: - Imports must now follow ES6 semantics: `import * as X from ...` produces a module object; to import functions or class instances, use `import X from ...`. - Everything is now built with isolatedModules flag. Some exports were updated for it. Packages: - Upgraded browserify dependency, and related packages (used for the distribution-building step). - Building the distribution now uses esbuild's minification. babel-minify is no longer used. Test Plan: Should have no behavior changes, existing tests should pass, and docker image should build too. Reviewers: georgegevoian Reviewed By: georgegevoian Subscribers: alexmojaki Differential Revision: https://phab.getgrist.com/D3506	2 years ago
Dmitry S	dd2eadc86e	(core) Speed up and upgrade build. ... Summary: - Upgrades to build-related packages: - Upgrade typescript, related libraries and typings. - Upgrade webpack, eslint; add tsc-watch, node-dev, eslint_d. - Build organization changes: - Build webpack from original typescript, transpiling only; with errors still reported by a background tsc watching process. - Typescript-related changes: - Reduce imports of AWS dependencies (very noticeable speedup) - Avoid auto-loading global @types - Client code is now built with isolatedModules flag (for safe transpilation) - Use allowJs to avoid copying JS files manually. - Linting changes - Enhance Arcanist ESLintLinter to run before/after commands, and set up to use eslint_d - Update eslint config, and include .eslintignore to avoid linting generated files. - Include a bunch of eslint-prompted and eslint-generated fixes - Add no-unused-expression rule to eslint, and fix a few warnings about it - Other items: - Refactor cssInput to avoid circular dependency - Remove a bit of unused code, libraries, dependencies Test Plan: No behavior changes, all existing tests pass. There are 30 tests fewer reported because `test_gpath.py` was removed (it's been unused for years) Reviewers: paulfitz Reviewed By: paulfitz Subscribers: paulfitz Differential Revision: https://phab.getgrist.com/D3498	2 years ago
Dmitry S	a91d493ffc	(core) Fix issue with 'UNEXPECTED ORDER OF CALLBACKS' in Client.ts. ... Summary: - Substantial refactoring of the logic when the server fails to send some messages to a client. - Add seqId numbers to server messages to ensure reliable order. - Add a needReload flag in clientConnect for a clear indication whent the browser client needs to reload the app. - Reproduce some potential failure scenarios in a test case (some of which previously could have led to incorrectly ordered messages). - Convert other Comm tests to typescript. - Tweak logging of Comm and Client to be slightly more concise (in particular, avoid logging sessionId) Note that despite the big refactoring, this only addresses a fairly rare situation, with websocket failures while server is trying to send to the client. It includes no improvements for failures while the client is sending to the server. (I looked for an existing library that would take care of these issues. A relevant article I found is https://docs.microsoft.com/en-us/azure/azure-web-pubsub/howto-develop-reliable-clients, but it doesn't include a library for both ends, and is still in review. Other libraries with similar purposes did not inspire enough confidence.) Test Plan: New test cases, which reproduce some previously problematic scenarios. Reviewers: paulfitz Reviewed By: paulfitz Differential Revision: https://phab.getgrist.com/D3470	2 years ago