gristlabs_grist-core

Commit Graph

Author	SHA1	Message	Date
Dmitry S	2705d41c34	(core) Add timeouts to prevent ActiveDoc bad state during shutdown. Summary: Add two shutdown-related timeouts. 1. One is to limit the duration of any work that happens once shutdown begins. In particular, waiting for an update to current time could block indefinitely if the data engine is unresponsive. Such awaits are now limited to 5 seconds. 2. The other is to allow documents to get shutdown for inactivity even when some work takes forever. Certain work (e.g. applying user actions) generally prevents a document from shutting down while it's pending. This prevention is now limited to 5 minutes. Shutting down a doc while something is pending may break some assumptions, and lead to errors. The timeout is long to let us assume that the work is stuck, and that errors are better than waiting forever. Other changes: - Periodic ActiveDoc work (intervals) is now started when a doc finishes loading rather than in the constructor. The difference only showed up in tests which makes the intervals much shorter. - Move timeoutReached() utility function to gutil, and use it for isLongerThan(), since they are basically identical. Also makes sure that the timer in these is cleared in all cases. - Remove duplicate waitForIt implementation (previously had a copy in both test/server and core/test/server). - Change testUtil.captureLog to pass messages to its callback, to allow asserts on messages within the callback. Test Plan: Added new unittests for the new shutdowns, including a replication of a bad state that was possible during shutdown. Reviewers: paulfitz Reviewed By: paulfitz Subscribers: paulfitz Differential Revision: https://phab.getgrist.com/D4040	12 months ago
Dmitry S	d5a4605d2a	(core) Improve encoding detection for csv imports, and make encoding an editable option. Summary: - Using a sample of data was causing poor detection if the sample were cut mid-character. Switch to using line-based detection. - Add a simple option for changing encoding. No convenient UI is offered since config UI is auto-generated, but this at least makes it possible to recover from bad guesses. - Upgrades chardet library for good measure. - Also fixes python3-building step, to more reliably rebuild Python dependencies when requirements3.* files change. Test Plan: Added a python-side test case, and a browser test that encodings can be switched, errors are displayed, and wrong encodings fail recoverably. Reviewers: alexmojaki Reviewed By: alexmojaki Differential Revision: https://phab.getgrist.com/D3979	1 year ago
Dmitry S	526a5df157	(core) Manage memory used for websocket responses to reduce the risk of server crashes. Summary: - Implements MemoryPool for waiting on memory reservations. - Uses MemoryPool to control memory used for stringifying JSON responses in Client.ts - Limits total size of _missedMessages that may be queued for a particular client. - Upgrades ws library, which may reduce memory usage, and allows pausing the websocket for testing. - The upgrade changed subtle behavior corners, requiring various fixes to code and tests. - dos.ts: - Includes Paul's fixes and updates to the dos.ts script for manual stress-testing. - Logging tweaks, to avoid excessive dumps on uncaughtError, and include timestamps. Test Plan: - Includes a test that measures heap size, and fails without memory management. - Includes a unittest for MemoryPool - Some cleanup and additions to TestServer helper; in particular adds makeUserApi() helper used in multiple tests. - Some fixes related to ws upgrade. Reviewers: paulfitz Reviewed By: paulfitz Differential Revision: https://phab.getgrist.com/D3974	1 year ago
Jarosław Sadziński	da323fb741	(core) Floating formula editor Summary: Adding a way to detach an editor. Initially only implemented for the formula editor, includes redesign for the AI part. - Initially, the detached editor is tight with the formula assistant and both are behind GRIST_FORMULA_ASSISTANT flag, but this can be relaxed later on, as the detached editor can be used on its own. - Detached editor is only supported in regular fields and on the creator panel. It is not supported yet for conditional styles, due to preview limitations. - Old code for the assistant was removed completely, as it was only a temporary solution, but the AI conversation part was copied to the new one. - Prompting was not modified in this diff, it will be included in the follow-up with more test cases. Test Plan: Added only new tests; existing tests should pass. Reviewers: JakubSerafin Reviewed By: JakubSerafin Differential Revision: https://phab.getgrist.com/D3863	1 year ago
Dmitry S	51ff72c15e	(core) Faster builds all around. Summary: Building: - Builds no longer wait for tsc for either client, server, or test targets. All use esbuild which is very fast. - Build still runs tsc, but only to report errors. This may be turned off with `SKIP_TSC=1` env var. - Grist-core continues to build using tsc. - Esbuild requires ES6 module semantics. Typescript's esModuleInterop is turned on, so that tsc accepts and enforces correct usage. - Client-side code is watched and bundled by webpack as before (using esbuild-loader) Code changes: - Imports must now follow ES6 semantics: `import * as X from ...` produces a module object; to import functions or class instances, use `import X from ...`. - Everything is now built with isolatedModules flag. Some exports were updated for it. Packages: - Upgraded browserify dependency, and related packages (used for the distribution-building step). - Building the distribution now uses esbuild's minification. babel-minify is no longer used. Test Plan: Should have no behavior changes, existing tests should pass, and docker image should build too. Reviewers: georgegevoian Reviewed By: georgegevoian Subscribers: alexmojaki Differential Revision: https://phab.getgrist.com/D3506	2 years ago
Dmitry S	a91d493ffc	(core) Fix issue with 'UNEXPECTED ORDER OF CALLBACKS' in Client.ts. Summary: - Substantial refactoring of the logic when the server fails to send some messages to a client. - Add seqId numbers to server messages to ensure reliable order. - Add a needReload flag in clientConnect for a clear indication whent the browser client needs to reload the app. - Reproduce some potential failure scenarios in a test case (some of which previously could have led to incorrectly ordered messages). - Convert other Comm tests to typescript. - Tweak logging of Comm and Client to be slightly more concise (in particular, avoid logging sessionId) Note that despite the big refactoring, this only addresses a fairly rare situation, with websocket failures while server is trying to send to the client. It includes no improvements for failures while the client is sending to the server. (I looked for an existing library that would take care of these issues. A relevant article I found is https://docs.microsoft.com/en-us/azure/azure-web-pubsub/howto-develop-reliable-clients, but it doesn't include a library for both ends, and is still in review. Other libraries with similar purposes did not inspire enough confidence.) Test Plan: New test cases, which reproduce some previously problematic scenarios. Reviewers: paulfitz Reviewed By: paulfitz Differential Revision: https://phab.getgrist.com/D3470	2 years ago
Dmitry S	b57a211741	(core) Fix issue with sandboxUtil where %s in message would get interpreted wrongly Summary: - Also converted sandboxUtil to typescript. - The issue with %s manifested when a Python traceback contained "%s" in the string; in that case the object with log metadata (e.g. docId) would confusingly replace %s as if it were part of the message from Python. Test Plan: Added a test case for the fix. Reviewers: alexmojaki Reviewed By: alexmojaki Differential Revision: https://phab.getgrist.com/D3486	2 years ago
Paul Fitzpatrick	de703343d0	(core) disentangle some server tests, release to core, add GRIST_PROXY_AUTH_HEADER test Summary: This shuffles some server tests to make them available in grist-core, and adds a test for the `GRIST_PROXY_AUTH_HEADER` feature added in https://github.com/gristlabs/grist-core/pull/165 It includes a fix for a header normalization issue for websocket connections. Test Plan: added test Reviewers: georgegevoian Reviewed By: georgegevoian Differential Revision: https://phab.getgrist.com/D3326	3 years ago
Paul Fitzpatrick	c4fb40e0bd	(core) add gvisor-based sandbox to jenkins test worker image Summary: This is preparatory work for running tests with the new sandbox in jenkins. * Makes a base image that is now shared by grist servers and jenkins workers. * Needed to allow jenkins to run `sudo runsc`. * Converged on port 2020 for ssh to workers and servers. Test Plan: added one runsc-based test and confirmed it was run Reviewers: dsagal Reviewed By: dsagal Differential Revision: https://phab.getgrist.com/D3029	3 years ago
Paul Fitzpatrick	6e15d44cf6	(core) start applying defenses for untrusted document uploads Summary: This applies some mitigations suggested by SQLite authors when opening untrusted SQLite databases, as we do when Grist docs are uploaded by the user. See: https://www.sqlite.org/security.html#untrusted_sqlite_database_files Steps implemented in this diff are: * Setting `trusted_schema` to off * Running a SQLite-level integrity check on uploads Other steps will require updates to our node-sqlite3 fork, since they are not available via the node-sqlite3 api (one more reason to migrate to better-sqlite3). I haven't yet managed to create a file that triggers an integrity check failure without also being detected as corruption by sqlite at a more basic level, so that is a TODO for testing. Test Plan: existing tests pass; need to come up with exploits to actually test the defences and have not yet Reviewers: dsagal Reviewed By: dsagal Differential Revision: https://phab.getgrist.com/D2909	3 years ago

10 Commits (3dfe4be5f3733f98c70662a644efbd0951bf3655)