gristlabs_grist-core

Commit Graph

Author	SHA1	Message	Date
Dmitry S	51ff72c15e	(core) Faster builds all around. Summary: Building: - Builds no longer wait for tsc for either client, server, or test targets. All use esbuild which is very fast. - Build still runs tsc, but only to report errors. This may be turned off with `SKIP_TSC=1` env var. - Grist-core continues to build using tsc. - Esbuild requires ES6 module semantics. Typescript's esModuleInterop is turned on, so that tsc accepts and enforces correct usage. - Client-side code is watched and bundled by webpack as before (using esbuild-loader) Code changes: - Imports must now follow ES6 semantics: `import * as X from ...` produces a module object; to import functions or class instances, use `import X from ...`. - Everything is now built with isolatedModules flag. Some exports were updated for it. Packages: - Upgraded browserify dependency, and related packages (used for the distribution-building step). - Building the distribution now uses esbuild's minification. babel-minify is no longer used. Test Plan: Should have no behavior changes, existing tests should pass, and docker image should build too. Reviewers: georgegevoian Reviewed By: georgegevoian Subscribers: alexmojaki Differential Revision: https://phab.getgrist.com/D3506	2 years ago
Paul Fitzpatrick	302202b4fb	(core) freshen tests for python3 Summary: Recent python3 changes perturbed timing again, and a few more tests started failing. Contains an unrelated correction for gvisor running under docker (a useful configuration on macs for debugging gvisor problems, but not supported by throttling code). Test Plan: updated tests Reviewers: dsagal, alexmojaki Reviewed By: dsagal, alexmojaki Differential Revision: https://phab.getgrist.com/D3129	3 years ago
Paul Fitzpatrick	db34e1a6d9	(core) tweak throttling to work for gvisor/runsc Summary: Grist has, up to now, used a throttling mechanism that allows a sandbox free rein until it starts using above some threshold percentage of a cpu for some time - at that point, we start sending STOP and CONT signals on a duty cycle, with longer and longer STOPped periods until cpu usage is at a threshold. The general idea is to do short jobs quickly, while throttling long jobs (thus unfortunately making them even longer) in order to continue doing other short jobs quickly. The runsc sandbox is not a single process, there are in fact 5 per sandbox in our setup. Runsc can work with kvm or ptrace. Kvm is not available to us, so we use ptrace. With ptrace, there is one process that is the appropriate one to duty cycle, and another that needs to receive a signal in order to yield. This diff adds the necessary machinery. This is a conservative change, where I stick with our existing throttling mechanism and adapt it to the new sandbox. It would be reasonable to consider switching throttling. There's a lot the OS allows. We can set a quota for how much cpu a process can use within a given period, for example. However the overall behavior with that would be quite different to what we have, so feels like this would need more discussion. The implementation contains use of a linux utility `pgrep` since portability is not important (runsc is only available on linux) and there's no node api for enumerating children of a process. The diff contains some tweaks to `buildtools/contain.sh` to streamline experimenting with Grist and runsc on a mac. It is important for throttling that node and the sandbox processes are in the same process name space, if docker is in between them then some extra machinery is needed (a proxy throttler and a way to communicate with it) which I chose not to implement. Test Plan: added test; a lot of manual testing Reviewers: dsagal Reviewed By: dsagal Differential Revision: https://phab.getgrist.com/D3113	3 years ago

Author

SHA1

Message

Date

Dmitry S

51ff72c15e

(core) Faster builds all around.

Summary:
Building:
- Builds no longer wait for tsc for either client, server, or test targets. All use esbuild which is very fast.
- Build still runs tsc, but only to report errors. This may be turned off with `SKIP_TSC=1` env var.
- Grist-core continues to build using tsc.
- Esbuild requires ES6 module semantics. Typescript's esModuleInterop is turned
  on, so that tsc accepts and enforces correct usage.
- Client-side code is watched and bundled by webpack as before (using esbuild-loader)

Code changes:
- Imports must now follow ES6 semantics: `import * as X from ...` produces a
  module object; to import functions or class instances, use `import X from ...`.
- Everything is now built with isolatedModules flag. Some exports were updated for it.

Packages:
- Upgraded browserify dependency, and related packages (used for the distribution-building step).
- Building the distribution now uses esbuild's minification. babel-minify is no longer used.

Test Plan: Should have no behavior changes, existing tests should pass, and docker image should build too.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3506

Paul Fitzpatrick

302202b4fb

(core) freshen tests for python3

Summary:
Recent python3 changes perturbed timing again, and a few more tests started failing.

Contains an unrelated correction for gvisor running under docker (a useful configuration on macs for debugging gvisor problems, but not supported by throttling code).

Test Plan: updated tests

Reviewers: dsagal, alexmojaki

Reviewed By: dsagal, alexmojaki

Differential Revision: https://phab.getgrist.com/D3129

Paul Fitzpatrick

db34e1a6d9

(core) tweak throttling to work for gvisor/runsc

Summary:
Grist has, up to now, used a throttling mechanism that allows a sandbox free rein until it starts using above some threshold percentage of a cpu for some time - at that point, we start sending STOP and CONT signals on a duty cycle, with longer and longer STOPped periods until cpu usage is at a threshold. The general idea is to do short jobs quickly, while throttling long jobs (thus unfortunately making them even longer) in order to continue doing other short jobs quickly.

The runsc sandbox is not a single process, there are in fact 5 per sandbox in our setup. Runsc can work with kvm or ptrace. Kvm is not available to us, so we use ptrace. With ptrace, there is one process that is the appropriate one to duty cycle, and another that needs to receive a signal in order to yield. This diff adds the necessary machinery.

This is a conservative change, where I stick with our existing throttling mechanism and adapt it to the new sandbox. It would be reasonable to consider switching throttling. There's a lot the OS allows. We can set a quota for how much cpu a process can use within a given period, for example. However the overall behavior with that would be quite different to what we have, so feels like this would need more discussion.

The implementation contains use of a linux utility `pgrep` since portability is not important (runsc is only available on linux) and there's no node api for enumerating children of a process.

The diff contains some tweaks to `buildtools/contain.sh` to streamline experimenting with Grist and runsc on a mac. It is important for throttling that node and the sandbox processes are in the same process name space, if docker is in between them then some extra machinery is needed (a proxy throttler and a way to communicate with it) which I chose not to implement.

Test Plan: added test; a lot of manual testing

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D3113

3 Commits (654514c18d9121650d66dc9fe5c9cdf056d5df15)