6 Commits (103ebbb045e5770cb42967ac1deb1c2aaa56dd2b)

Author	SHA1	Message	Date
Dmitry S	51ff72c15e	(core) Faster builds all around. ... Summary: Building: - Builds no longer wait for tsc for either client, server, or test targets. All use esbuild which is very fast. - Build still runs tsc, but only to report errors. This may be turned off with `SKIP_TSC=1` env var. - Grist-core continues to build using tsc. - Esbuild requires ES6 module semantics. Typescript's esModuleInterop is turned on, so that tsc accepts and enforces correct usage. - Client-side code is watched and bundled by webpack as before (using esbuild-loader) Code changes: - Imports must now follow ES6 semantics: `import * as X from ...` produces a module object; to import functions or class instances, use `import X from ...`. - Everything is now built with isolatedModules flag. Some exports were updated for it. Packages: - Upgraded browserify dependency, and related packages (used for the distribution-building step). - Building the distribution now uses esbuild's minification. babel-minify is no longer used. Test Plan: Should have no behavior changes, existing tests should pass, and docker image should build too. Reviewers: georgegevoian Reviewed By: georgegevoian Subscribers: alexmojaki Differential Revision: https://phab.getgrist.com/D3506	2 years ago
Dmitry S	a91d493ffc	(core) Fix issue with 'UNEXPECTED ORDER OF CALLBACKS' in Client.ts. ... Summary: - Substantial refactoring of the logic when the server fails to send some messages to a client. - Add seqId numbers to server messages to ensure reliable order. - Add a needReload flag in clientConnect for a clear indication whent the browser client needs to reload the app. - Reproduce some potential failure scenarios in a test case (some of which previously could have led to incorrectly ordered messages). - Convert other Comm tests to typescript. - Tweak logging of Comm and Client to be slightly more concise (in particular, avoid logging sessionId) Note that despite the big refactoring, this only addresses a fairly rare situation, with websocket failures while server is trying to send to the client. It includes no improvements for failures while the client is sending to the server. (I looked for an existing library that would take care of these issues. A relevant article I found is https://docs.microsoft.com/en-us/azure/azure-web-pubsub/howto-develop-reliable-clients, but it doesn't include a library for both ends, and is still in review. Other libraries with similar purposes did not inspire enough confidence.) Test Plan: New test cases, which reproduce some previously problematic scenarios. Reviewers: paulfitz Reviewed By: paulfitz Differential Revision: https://phab.getgrist.com/D3470	2 years ago
Dmitry S	b57a211741	(core) Fix issue with sandboxUtil where %s in message would get interpreted wrongly ... Summary: - Also converted sandboxUtil to typescript. - The issue with %s manifested when a Python traceback contained "%s" in the string; in that case the object with log metadata (e.g. docId) would confusingly replace %s as if it were part of the message from Python. Test Plan: Added a test case for the fix. Reviewers: alexmojaki Reviewed By: alexmojaki Differential Revision: https://phab.getgrist.com/D3486	2 years ago
Paul Fitzpatrick	de703343d0	(core) disentangle some server tests, release to core, add GRIST_PROXY_AUTH_HEADER test ... Summary: This shuffles some server tests to make them available in grist-core, and adds a test for the `GRIST_PROXY_AUTH_HEADER` feature added in https://github.com/gristlabs/grist-core/pull/165 It includes a fix for a header normalization issue for websocket connections. Test Plan: added test Reviewers: georgegevoian Reviewed By: georgegevoian Differential Revision: https://phab.getgrist.com/D3326	2 years ago
Paul Fitzpatrick	c4fb40e0bd	(core) add gvisor-based sandbox to jenkins test worker image ... Summary: This is preparatory work for running tests with the new sandbox in jenkins. * Makes a base image that is now shared by grist servers and jenkins workers. * Needed to allow jenkins to run `sudo runsc`. * Converged on port 2020 for ssh to workers and servers. Test Plan: added one runsc-based test and confirmed it was run Reviewers: dsagal Reviewed By: dsagal Differential Revision: https://phab.getgrist.com/D3029	3 years ago
Paul Fitzpatrick	6e15d44cf6	(core) start applying defenses for untrusted document uploads ... Summary: This applies some mitigations suggested by SQLite authors when opening untrusted SQLite databases, as we do when Grist docs are uploaded by the user. See: https://www.sqlite.org/security.html#untrusted_sqlite_database_files Steps implemented in this diff are: * Setting `trusted_schema` to off * Running a SQLite-level integrity check on uploads Other steps will require updates to our node-sqlite3 fork, since they are not available via the node-sqlite3 api (one more reason to migrate to better-sqlite3). I haven't yet managed to create a file that triggers an integrity check failure without also being detected as corruption by sqlite at a more basic level, so that is a TODO for testing. Test Plan: existing tests pass; need to come up with exploits to actually test the defences and have not yet Reviewers: dsagal Reviewed By: dsagal Differential Revision: https://phab.getgrist.com/D2909	3 years ago