gristlabs_grist-core/sandbox/gvisor/get_checkpoint_path.sh

#!/usr/bin/env bash

# This defines a GRIST_CHECKPOINT environment variable, where we will store
# a sandbox checkpoint. The path is in principle arbitrary. In practice,
# it is helpful if it lies outside of the Grist repo (to avoid permission
# problems with docker users), but is distinct for each possible location
# of the Grist repo (to avoid collisions in distinct Jenkins jobs).
#
# Checkpointing is currently not supported by gvisor when running in
# rootless mode. Rootless mode is nevertheless the best way to run
# "mainstream" unpatched gvisor for Grist. If gvisor is unpatched
# (does not have the --unprivileged flag from
# https://github.com/google/gvisor/issues/4371#issuecomment-700917549)
# then we do not define GRIST_CHECKPOINT and checkpoints will not be
# used. If the host is linux, performance seems just fine; in other
# configurations we've seen about a second delay in initial load of
# python due to a relatively sluggish file system.
#
# So as part of figuring whether to allow checkpoints, this script
# determines the best flags to call gvisor with. It tries:
#   --unprivileged --ignore-cgroups   : for a newer rebased fork of gvisor
#   --unprivileged                    : for an older fork of gvisor
#   --rootless                        : unforked gvisor
# It leaves the flags in a GVISOR_FLAGS environment variable. This
# variable is respected by the sandbox/gvisor/run.py wrapper for running
# python in gvisor.

function check_gvisor {
  # If we already have working gvisor flags, return.
  if [[ -n "$GVISOR_FLAGS" ]]; then
    return
  fi
  # Check if a trivial command works under gvisor with the proposed flags.
  if runsc --network none "$@" do true 2> /dev/null; then
    export GVISOR_FLAGS="$@"
    export GVISOR_AVAILABLE=1
  fi
}

check_gvisor --unprivileged --ignore-cgroups
check_gvisor --unprivileged

# If we can't use --unprivileged, stick with --rootless and no checkpoint
if [[ -z "$GVISOR_FLAGS" ]]; then
  check_gvisor --rootless
else
  export GRIST_CHECKPOINT=/tmp/engine_$(echo $PWD | sed "s/[^a-zA-Z0-9]/_/g")
fi
(core) add machinery for self-managed flavor of Grist Summary: Currently, we have two ways that we deliver Grist. One is grist-core, which has simple defaults and is relatively easy for third parties to deploy. The second is our internal build for our SaaS, which is the opposite. For self-managed Grist, a planned paid on-premise version of Grist, I adopt the following approach: * Use the `grist-core` build mechanism, extending it to accept an overlay of extra code if present. * Extra code is supplied in a self-contained `ext` directory, with an `ext/app` directory that is of same structure as core `app` and `stubs/app`. * The `ext` directory also contains information about extra node dependencies needed beyond that of `grist-core`. * The `ext` directory is contained within our monorepo rather than `grist-core` since it may contain material not under the Apache license. Docker builds are achieved in our monorepo by using the `--build-context` functionality to add in `ext` during the regular `grist-core` build: ``` docker buildx build --load -t gristlabs/grist-ee --build-context=ext=../ext . ``` Incremental builds in our monorepo are achieved with the `build_core.sh` helper, like: ``` buildtools/build_core.sh /tmp/self-managed cd /tmp/self-managed yarn start ``` The initial `ext` directory contains material for snapshotting to S3. If you build the docker image as above, and have S3 access, you can do something like: ``` docker run -p 8484:8484 --env GRIST_SESSION_SECRET=a-secret \ --env GRIST_DOCS_S3_BUCKET=grist-docs-test \ --env GRIST_DOCS_S3_PREFIX=self-managed \ -v $HOME/.aws:/root/.aws -it gristlabs/grist-ee ``` This will start a version of Grist that is like `grist-core` but with S3 snapshots enabled. To release this code to `grist-core`, it would just need to move from `ext/app` to `app` within core. I tried a lot of ways of organizing self-managed Grist, and this was what made me happiest. There are a lot of trade-offs, but here is what I was looking for: * Only OSS-code in grist-core. Adding mixed-license material there feels unfair to people already working with the repo. That said, a possible future is to move away from our private monorepo to a public mixed-licence repo, which could have the same relationship with grist-core as the monorepo has. * Minimal differences between self-managed builds and one of our existing builds, ideally hewing as close to grist-core as possible for ease of documentation, debugging, and maintenance. * Ideally, docker builds without copying files around (the new `--build-context` functionality made that possible). * Compatibility with monorepo build. Expressing dependencies of the extra code in `ext` proved tricky to do in a clean way. Yarn/npm fought me every step of the way - everything related to optional dependencies was unsatisfactory in some respect. Yarn2 is flexible but smells like it might be overreach. In the end, organizing to install non-core dependencies one directory up from the main build was a good simple trick that saved my bacon. This diff gets us to the point of building `grist-ee` images conveniently, but there isn't a public repo people can go look at to see its source. This could be generated by taking `grist-core`, adding the `ext` directory to it, and pushing to a distinct repository. I'm not in a hurry to do that, since a PR to that repo would be hard to sync with our monorepo and `grist-core`. Also, we don't have any licensing text ready for the `ext` directory. So leaving that for future work. Test Plan: manual Reviewers: georgegevoian, alexmojaki Reviewed By: georgegevoian, alexmojaki Differential Revision: https://phab.getgrist.com/D3415 2 years ago			`#!/usr/bin/env bash`
(core) add gvisor-based sandboxing to core Summary: This adds support for gvisor sandboxing in core. When Grist is run outside of a container, regular gvisor can be used (if on linux), and will run in rootless mode. When Grist is run inside a container, docker's default policy is insufficient for running gvisor, so a fork of gvisor is used that has less defence-in-depth but can run without privileges. Sandboxing is automatically turned on in the Grist core container. It is not turned on automatically when built from source, since it is operating-system dependent. This diff may break a complex method of testing Grist with gvisor on macs that I may have been the only person using. If anyone complains I'll find time on a mac to fix it :) This diff includes a small "easter egg" to force document loads, primarily intended for developer use. Test Plan: existing tests pass; checked that core and saas docker builds function Reviewers: alexmojaki Reviewed By: alexmojaki Subscribers: alexmojaki Differential Revision: https://phab.getgrist.com/D3333 2 years ago
			`# This defines a GRIST_CHECKPOINT environment variable, where we will store`
			`# a sandbox checkpoint. The path is in principle arbitrary. In practice,`
			`# it is helpful if it lies outside of the Grist repo (to avoid permission`
			`# problems with docker users), but is distinct for each possible location`
			`# of the Grist repo (to avoid collisions in distinct Jenkins jobs).`
			`#`
			`# Checkpointing is currently not supported by gvisor when running in`
			`# rootless mode. Rootless mode is nevertheless the best way to run`
			`# "mainstream" unpatched gvisor for Grist. If gvisor is unpatched`
			`# (does not have the --unprivileged flag from`
			`# https://github.com/google/gvisor/issues/4371#issuecomment-700917549)`
			`# then we do not define GRIST_CHECKPOINT and checkpoints will not be`
			`# used. If the host is linux, performance seems just fine; in other`
			`# configurations we've seen about a second delay in initial load of`
			`# python due to a relatively sluggish file system.`
			`#`
			`# So as part of figuring whether to allow checkpoints, this script`
			`# determines the best flags to call gvisor with. It tries:`
			`# --unprivileged --ignore-cgroups : for a newer rebased fork of gvisor`
			`# --unprivileged : for an older fork of gvisor`
			`# --rootless : unforked gvisor`
			`# It leaves the flags in a GVISOR_FLAGS environment variable. This`
			`# variable is respected by the sandbox/gvisor/run.py wrapper for running`
			`# python in gvisor.`

			`function check_gvisor {`
			`# If we already have working gvisor flags, return.`
			`if [[ -n "$GVISOR_FLAGS" ]]; then`
			`return`
			`fi`
			`# Check if a trivial command works under gvisor with the proposed flags.`
			`if runsc --network none "$@" do true 2> /dev/null; then`
			`export GVISOR_FLAGS="$@"`
			`export GVISOR_AVAILABLE=1`
			`fi`
			`}`

			`check_gvisor --unprivileged --ignore-cgroups`
			`check_gvisor --unprivileged`

			`# If we can't use --unprivileged, stick with --rootless and no checkpoint`
			`if [[ -z "$GVISOR_FLAGS" ]]; then`
			`check_gvisor --rootless`
			`else`
			`export GRIST_CHECKPOINT=/tmp/engine_$(echo $PWD \| sed "s/[^a-zA-Z0-9]/_/g")`
			`fi`