Paul Fitzpatrick db34e1a6d9 (core) tweak throttling to work for gvisor/runsc ... Summary: Grist has, up to now, used a throttling mechanism that allows a sandbox free rein until it starts using above some threshold percentage of a cpu for some time - at that point, we start sending STOP and CONT signals on a duty cycle, with longer and longer STOPped periods until cpu usage is at a threshold. The general idea is to do short jobs quickly, while throttling long jobs (thus unfortunately making them even longer) in order to continue doing other short jobs quickly. The runsc sandbox is not a single process, there are in fact 5 per sandbox in our setup. Runsc can work with kvm or ptrace. Kvm is not available to us, so we use ptrace. With ptrace, there is one process that is the appropriate one to duty cycle, and another that needs to receive a signal in order to yield. This diff adds the necessary machinery. This is a conservative change, where I stick with our existing throttling mechanism and adapt it to the new sandbox. It would be reasonable to consider switching throttling. There's a lot the OS allows. We can set a quota for how much cpu a process can use within a given period, for example. However the overall behavior with that would be quite different to what we have, so feels like this would need more discussion. The implementation contains use of a linux utility `pgrep` since portability is not important (runsc is only available on linux) and there's no node api for enumerating children of a process. The diff contains some tweaks to `buildtools/contain.sh` to streamline experimenting with Grist and runsc on a mac. It is important for throttling that node and the sandbox processes are in the same process name space, if docker is in between them then some extra machinery is needed (a proxy throttler and a way to communicate with it) which I chose not to implement. Test Plan: added test; a lot of manual testing Reviewers: dsagal Reviewed By: dsagal Differential Revision: https://phab.getgrist.com/D3113		2021-11-04 17:23:43 -04:00
..
ACLFormula.ts	(core) add OWNERS='owners', EDITOR='editors', VIEWER='viewers' to condition formulas	2021-03-19 18:20:33 -04:00
ActionHistory.ts	(core) do not look at content of recent actions when loading documents	2021-09-29 11:27:02 -04:00
ActionHistoryImpl.ts	(core) do not look at content of recent actions when loading documents	2021-09-29 11:27:02 -04:00
ActionSummary.ts	(core) add an option to action summarization to preserve columns entirely	2021-10-28 21:52:19 -04:00
ActiveDoc.ts	(core) Adding sort options for columns.	2021-11-03 15:31:39 +01:00
ActiveDocImport.ts	(core) Add diff preview to Importer	2021-10-08 14:15:07 -07:00
AppEndpoint.ts	(core) Adding import from google drive to the home screen	2021-08-05 20:46:11 +02:00
Authorizer.ts	(core) For grist_sid*_status cookie, remember to set the path	2021-10-04 09:13:47 -04:00
BrowserSession.ts	(core) Implement DiscourseConnect to enable easy sign-in to community forum	2021-10-01 11:24:22 -04:00
checksumFile.ts	(core) move home server into core	2020-07-21 20:39:10 -04:00
Client.ts	(core) Add LogMethods helper and use it for more JSON data in logs. Reduce unhelpful logging.	2021-10-25 10:25:18 -04:00
Comm.js	(core) Simple localization support and currency selector.	2021-08-26 13:36:49 -07:00
dbUtils.ts	(core) move home server into core	2020-07-21 20:39:10 -04:00
DiscourseConnect.ts	(core) Implement DiscourseConnect to enable easy sign-in to community forum	2021-10-01 11:24:22 -04:00
DocApi.ts	(core) make document assignment endpoint available via /housekeeping api	2021-11-04 16:14:21 -04:00
DocApiTypes-ti.ts	(core) Adding schema validation for records endpoint	2021-10-18 21:40:50 +02:00
DocApiTypes.ts	(core) Adding schema validation for records endpoint	2021-10-18 21:40:50 +02:00
DocClients.ts	(core) Add LogMethods helper and use it for more JSON data in logs. Reduce unhelpful logging.	2021-10-25 10:25:18 -04:00
DocManager.ts	(core) open documents without blocking on data engine	2021-10-01 10:18:56 -04:00
DocPluginData.ts	(core) move home server into core	2020-07-21 20:39:10 -04:00
DocPluginManager.ts	(core) Move file import plugins into core/sandbox/grist	2021-08-09 18:37:14 +02:00
DocSession.ts	(core) Add LogMethods helper and use it for more JSON data in logs. Reduce unhelpful logging.	2021-10-25 10:25:18 -04:00
DocSnapshots.ts	(core) start reconciling forking with granular access	2021-01-12 14:08:49 -05:00
DocStorage.ts	(core) Use GristObjCode in CellValue	2021-10-11 14:11:32 +02:00
DocStorageManager.ts	(core) uncheck FullCopy special when copying/forking a document	2021-04-29 08:56:54 -04:00
docUtils.d.ts	(core) move home server into core	2020-07-21 20:39:10 -04:00
docUtils.js	(core) move home server into core	2020-07-21 20:39:10 -04:00
DocWorker.ts	(core) Add diff preview to Importer	2021-10-08 14:15:07 -07:00
DocWorkerMap.ts	(core) revive saml support and test against Auth0	2021-08-16 17:36:09 -04:00
ExcelFormatter.ts	(core) Simple localization support and currency selector.	2021-08-26 13:36:49 -07:00
ExpandedQuery.ts	(core) Use GristObjCode in CellValue	2021-10-11 14:11:32 +02:00
Export.ts	(core) Adding sort options for columns.	2021-11-03 15:31:39 +01:00
ExportCSV.ts	(core) Relocate export urls to /download/	2021-09-02 09:36:33 -07:00
ExportXLSX.ts	(core) Strip invalid characters from table name in excel import	2021-10-11 17:47:12 +02:00
expressWrap.ts	(core) give more detailed reasons for access denied when memos are present	2021-02-15 17:02:24 -05:00
ExternalStorage.ts	(core) Add rules to eslint to better match our coding conventions.	2021-05-24 12:56:18 -04:00
extractOrg.ts	(core) Refactoring google drive plugin	2021-07-14 09:52:04 +02:00
FileParserElement.ts	(core) move home server into core	2020-07-21 20:39:10 -04:00
filterUtils.ts	(core) uncheck FullCopy special when copying/forking a document	2021-04-29 08:56:54 -04:00
FlexServer.ts	(core) Process new user info in /welcome/info post without waiting for it to be written to the collecting document	2021-10-31 13:54:31 -04:00
GoogleAuth.ts	(core) Extending Google Drive integration scope	2021-10-01 10:47:12 +02:00
GoogleExport.ts	(core) support python3 in grist-core, and running engine via docker and/or gvisor	2021-07-28 09:02:32 -04:00
GoogleImport.ts	(core) Extending Google Drive integration scope	2021-10-01 10:47:12 +02:00
GranularAccess.ts	(core) Use GristObjCode in CellValue	2021-10-11 14:11:32 +02:00
GristServer.ts	(core) add a tool for deleting a user	2021-09-29 12:08:23 -04:00
gristSessions.ts	(core) Implement DiscourseConnect to enable easy sign-in to community forum	2021-10-01 11:24:22 -04:00
guessExt.ts	(core) move home server into core	2020-07-21 20:39:10 -04:00
HashUtil.ts	(core) add more detail to /compare endpoint	2020-09-18 16:31:29 -04:00
HostedMetadataManager.ts	(core) revamp snapshot inventory	2020-10-30 13:52:46 -04:00
HostedStorageManager.ts	(core) Add LogMethods helper and use it for more JSON data in logs. Reduce unhelpful logging.	2021-10-25 10:25:18 -04:00
IBilling.ts	(core) move home server into core	2020-07-21 20:39:10 -04:00
IChecksumStore.ts	(core) revamp snapshot inventory	2020-10-30 13:52:46 -04:00
ICreate.ts	(core) clean up a collection of small problems affecting grist-core	2021-08-17 21:44:50 -04:00
IDocStorageManager.ts	(core) uncheck FullCopy special when copying/forking a document	2021-04-29 08:56:54 -04:00
idUtils.ts	(core) start reconciling forking with granular access	2021-01-12 14:08:49 -05:00
IElectionStore.ts	(core) move some material to core that slipped through in a rebase	2020-07-23 11:29:05 -04:00
initialDocSql.ts	(core) open documents without blocking on data engine	2021-10-01 10:18:56 -04:00
INotifier.ts	(core) add a tool for deleting a user	2021-09-29 12:08:23 -04:00
ISandbox.ts	(core) get all tests working under python3/gvisor	2021-10-18 17:44:15 -04:00
IShell.ts	(core) move home server into core	2020-07-21 20:39:10 -04:00
ITestingHooks-ti.ts	(core) Implement DiscourseConnect to enable easy sign-in to community forum	2021-10-01 11:24:22 -04:00
ITestingHooks.ts	(core) Implement DiscourseConnect to enable easy sign-in to community forum	2021-10-01 11:24:22 -04:00
log.ts	(core) give instructions on using Grist with docker	2020-10-28 13:59:13 -04:00
LogMethods.ts	(core) Add LogMethods helper and use it for more JSON data in logs. Reduce unhelpful logging.	2021-10-25 10:25:18 -04:00
manifest.ts	(core) move home server into core	2020-07-21 20:39:10 -04:00
MinimalLogin.ts	(core) Implement DiscourseConnect to enable easy sign-in to community forum	2021-10-01 11:24:22 -04:00
NSandbox.ts	(core) tweak throttling to work for gvisor/runsc	2021-11-04 17:23:43 -04:00
OnDemandActions.ts	(core) Configure more comprehensive eslint rules for Typescript	2021-04-26 18:54:55 -04:00
PermissionInfo.ts	(core) Add 'user' variable to trigger formulas	2021-07-15 15:18:32 -07:00
Permit.ts	(core) revive saml support and test against Auth0	2021-08-16 17:36:09 -04:00
places.ts	(core) fix docker packaging after core shuffle	2020-07-22 14:45:42 -04:00
PluginEndpoint.ts	(core) move home server into core	2020-07-21 20:39:10 -04:00
PluginManager.ts	(core) Add rules to eslint to better match our coding conventions.	2021-05-24 12:56:18 -04:00
reportTimeTaken.ts	(core) Log the time taken by decodeActionFromRow() operations.	2021-08-20 11:28:33 -04:00
requestUtils.ts	(core) support python3 in grist-core, and running engine via docker and/or gvisor	2021-07-28 09:02:32 -04:00
RowAccess.ts	(core) Add rules to eslint to better match our coding conventions.	2021-05-24 12:56:18 -04:00
SafePythonComponent.ts	(core) Move file import plugins into core/sandbox/grist	2021-08-09 18:37:14 +02:00
SamlConfig.ts	(core) Implement DiscourseConnect to enable easy sign-in to community forum	2021-10-01 11:24:22 -04:00
SandboxControl.ts	(core) tweak throttling to work for gvisor/runsc	2021-11-04 17:23:43 -04:00
sandboxUtil.js	(core) move home server into core	2020-07-21 20:39:10 -04:00
sendAppPage.ts	(core) Extending Google Drive integration scope	2021-10-01 10:47:12 +02:00
ServerColumnGetters.ts	(core) Adding sort options for columns.	2021-11-03 15:31:39 +01:00
ServerLocale.ts	(core) Simple localization support and currency selector.	2021-08-26 13:36:49 -07:00
serverUtils.ts	(core) Add LogMethods helper and use it for more JSON data in logs. Reduce unhelpful logging.	2021-10-25 10:25:18 -04:00
Sessions.ts	(core) clean up a collection of small problems affecting grist-core	2021-08-17 21:44:50 -04:00
Sharing.ts	(core) Add LogMethods helper and use it for more JSON data in logs. Reduce unhelpful logging.	2021-10-25 10:25:18 -04:00
shortDesc.ts	(core) move home server into core	2020-07-21 20:39:10 -04:00
shutdown.js	(core) move home server into core	2020-07-21 20:39:10 -04:00
SQLiteDB.ts	(core) Migrate import code from data engine to Node	2021-10-04 10:27:00 -07:00
TableMetadataLoader.ts	(core) open documents without blocking on data engine	2021-10-01 10:18:56 -04:00
TagChecker.ts	(core) move home server into core	2020-07-21 20:39:10 -04:00
TestingHooks.ts	(core) Implement DiscourseConnect to enable easy sign-in to community forum	2021-10-01 11:24:22 -04:00
Throttle.ts	(core) tweak throttling to work for gvisor/runsc	2021-11-04 17:23:43 -04:00
TimeQuery.ts	(core) move home server into core	2020-07-21 20:39:10 -04:00
Triggers.ts	(core) Webhook event queue on redis	2021-11-03 22:43:35 +02:00
UnsafeNodeComponent.ts	(core) move home server into core	2020-07-21 20:39:10 -04:00
uploads.ts	(core) Extending Google Drive integration scope	2021-10-01 10:47:12 +02:00
WorkCoordinator.ts	(core) move home server into core	2020-07-21 20:39:10 -04:00