(core) add machinery for self-managed flavor of Grist
Summary:
Currently, we have two ways that we deliver Grist. One is grist-core,
which has simple defaults and is relatively easy for third parties to
deploy. The second is our internal build for our SaaS, which is the
opposite. For self-managed Grist, a planned paid on-premise version
of Grist, I adopt the following approach:
* Use the `grist-core` build mechanism, extending it to accept an
overlay of extra code if present.
* Extra code is supplied in a self-contained `ext` directory, with
an `ext/app` directory that is of same structure as core `app`
and `stubs/app`.
* The `ext` directory also contains information about extra
node dependencies needed beyond that of `grist-core`.
* The `ext` directory is contained within our monorepo rather than
`grist-core` since it may contain material not under the Apache
license.
Docker builds are achieved in our monorepo by using the `--build-context`
functionality to add in `ext` during the regular `grist-core` build:
```
docker buildx build --load -t gristlabs/grist-ee --build-context=ext=../ext .
```
Incremental builds in our monorepo are achieved with the `build_core.sh` helper,
like:
```
buildtools/build_core.sh /tmp/self-managed
cd /tmp/self-managed
yarn start
```
The initial `ext` directory contains material for snapshotting to S3.
If you build the docker image as above, and have S3 access, you can
do something like:
```
docker run -p 8484:8484 --env GRIST_SESSION_SECRET=a-secret \
--env GRIST_DOCS_S3_BUCKET=grist-docs-test \
--env GRIST_DOCS_S3_PREFIX=self-managed \
-v $HOME/.aws:/root/.aws -it gristlabs/grist-ee
```
This will start a version of Grist that is like `grist-core` but with
S3 snapshots enabled. To release this code to `grist-core`, it would
just need to move from `ext/app` to `app` within core.
I tried a lot of ways of organizing self-managed Grist, and this was
what made me happiest. There are a lot of trade-offs, but here is what
I was looking for:
* Only OSS-code in grist-core. Adding mixed-license material there
feels unfair to people already working with the repo. That said,
a possible future is to move away from our private monorepo to
a public mixed-licence repo, which could have the same relationship
with grist-core as the monorepo has.
* Minimal differences between self-managed builds and one of our
existing builds, ideally hewing as close to grist-core as possible
for ease of documentation, debugging, and maintenance.
* Ideally, docker builds without copying files around (the new
`--build-context` functionality made that possible).
* Compatibility with monorepo build.
Expressing dependencies of the extra code in `ext` proved tricky to
do in a clean way. Yarn/npm fought me every step of the way - everything
related to optional dependencies was unsatisfactory in some respect.
Yarn2 is flexible but smells like it might be overreach. In the end,
organizing to install non-core dependencies one directory up from the
main build was a good simple trick that saved my bacon.
This diff gets us to the point of building `grist-ee` images conveniently,
but there isn't a public repo people can go look at to see its source. This
could be generated by taking `grist-core`, adding the `ext` directory
to it, and pushing to a distinct repository. I'm not in a hurry to do that,
since a PR to that repo would be hard to sync with our monorepo and
`grist-core`. Also, we don't have any licensing text ready for the `ext`
directory. So leaving that for future work.
Test Plan: manual
Reviewers: georgegevoian, alexmojaki
Reviewed By: georgegevoian, alexmojaki
Differential Revision: https://phab.getgrist.com/D3415
2022-05-12 15:24:48 +00:00
|
|
|
################################################################################
|
|
|
|
|
## The Grist source can be extended. This is a stub that can be overridden
|
|
|
|
|
## from command line, as:
|
|
|
|
|
## docker buildx build -t ... --build-context=ext=<path> .
|
|
|
|
|
## The code in <path> will then be built along with the rest of Grist.
|
|
|
|
|
################################################################################
|
|
|
|
|
FROM scratch as ext
|
|
|
|
|
|
2020-10-10 00:54:23 +00:00
|
|
|
################################################################################
|
2022-01-07 17:06:04 +00:00
|
|
|
## Javascript build stage
|
2020-10-10 00:54:23 +00:00
|
|
|
################################################################################
|
|
|
|
|
|
(core) move more tests to grist-core
Summary:
* Tie build and run-time docker base images to a consistent version (buster)
* Extend the test login system activated by GRIST_TEST_LOGIN to ease porting tests that currently rely on cognito (many)
* Make org resets work in absence of billing endpoints
* When in-memory session caches are used, add missing invalidation steps
* Pass org information through sign-ups/sign-ins more carefully
* For CORS, explicitly trust GRIST_HOST origin when set
* Move some fixtures and tests to core, focussing on tests that cover existing failures or are in the set of tests run on deployments
* Retain regular `test` target to run the test suite directly, without docker
* Add a `test:smoke` target to run a single simple test without `GRIST_TEST_LOGIN` activated
* Add a `test:docker` target to run the tests against a grist-core docker image - since tests rely on certain fixture teams/docs, added `TEST_SUPPORT_API_KEY` and `TEST_ADD_SAMPLES` flags to ease porting
The tests ported were `nbrowser` tests: `ActionLog.ts` (the first test I tend to port to anything, out of habit), `Fork.ts` (exercises a lot of doc creation paths), `HomeIntro.ts` (a lot of DocMenu exercise), and `DuplicateDocument.ts` (covers a feature known to be failing prior to this diff, the CORS tweak resolves it).
Test Plan: Manually tested via `buildtools/build_core.sh`. In follow up, I want to add running the `test:docker` target in grist-core's workflows. In jenkins, only the smoke test is run. There'd be an argument for running all tests, but they include particularly slow tests, and are duplicates of tests already run (in different configuration admittedly), so I'd like to try first just using them in grist-core to gate updates to any packaged version of Grist (the docker image currently).
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3176
2021-12-10 22:42:54 +00:00
|
|
|
FROM node:14-buster as builder
|
2020-10-10 00:54:23 +00:00
|
|
|
|
|
|
|
|
# Install all node dependencies.
|
(core) add machinery for self-managed flavor of Grist
Summary:
Currently, we have two ways that we deliver Grist. One is grist-core,
which has simple defaults and is relatively easy for third parties to
deploy. The second is our internal build for our SaaS, which is the
opposite. For self-managed Grist, a planned paid on-premise version
of Grist, I adopt the following approach:
* Use the `grist-core` build mechanism, extending it to accept an
overlay of extra code if present.
* Extra code is supplied in a self-contained `ext` directory, with
an `ext/app` directory that is of same structure as core `app`
and `stubs/app`.
* The `ext` directory also contains information about extra
node dependencies needed beyond that of `grist-core`.
* The `ext` directory is contained within our monorepo rather than
`grist-core` since it may contain material not under the Apache
license.
Docker builds are achieved in our monorepo by using the `--build-context`
functionality to add in `ext` during the regular `grist-core` build:
```
docker buildx build --load -t gristlabs/grist-ee --build-context=ext=../ext .
```
Incremental builds in our monorepo are achieved with the `build_core.sh` helper,
like:
```
buildtools/build_core.sh /tmp/self-managed
cd /tmp/self-managed
yarn start
```
The initial `ext` directory contains material for snapshotting to S3.
If you build the docker image as above, and have S3 access, you can
do something like:
```
docker run -p 8484:8484 --env GRIST_SESSION_SECRET=a-secret \
--env GRIST_DOCS_S3_BUCKET=grist-docs-test \
--env GRIST_DOCS_S3_PREFIX=self-managed \
-v $HOME/.aws:/root/.aws -it gristlabs/grist-ee
```
This will start a version of Grist that is like `grist-core` but with
S3 snapshots enabled. To release this code to `grist-core`, it would
just need to move from `ext/app` to `app` within core.
I tried a lot of ways of organizing self-managed Grist, and this was
what made me happiest. There are a lot of trade-offs, but here is what
I was looking for:
* Only OSS-code in grist-core. Adding mixed-license material there
feels unfair to people already working with the repo. That said,
a possible future is to move away from our private monorepo to
a public mixed-licence repo, which could have the same relationship
with grist-core as the monorepo has.
* Minimal differences between self-managed builds and one of our
existing builds, ideally hewing as close to grist-core as possible
for ease of documentation, debugging, and maintenance.
* Ideally, docker builds without copying files around (the new
`--build-context` functionality made that possible).
* Compatibility with monorepo build.
Expressing dependencies of the extra code in `ext` proved tricky to
do in a clean way. Yarn/npm fought me every step of the way - everything
related to optional dependencies was unsatisfactory in some respect.
Yarn2 is flexible but smells like it might be overreach. In the end,
organizing to install non-core dependencies one directory up from the
main build was a good simple trick that saved my bacon.
This diff gets us to the point of building `grist-ee` images conveniently,
but there isn't a public repo people can go look at to see its source. This
could be generated by taking `grist-core`, adding the `ext` directory
to it, and pushing to a distinct repository. I'm not in a hurry to do that,
since a PR to that repo would be hard to sync with our monorepo and
`grist-core`. Also, we don't have any licensing text ready for the `ext`
directory. So leaving that for future work.
Test Plan: manual
Reviewers: georgegevoian, alexmojaki
Reviewed By: georgegevoian, alexmojaki
Differential Revision: https://phab.getgrist.com/D3415
2022-05-12 15:24:48 +00:00
|
|
|
WORKDIR /grist
|
|
|
|
|
COPY package.json yarn.lock /grist/
|
2022-09-11 23:33:11 +00:00
|
|
|
RUN yarn install --frozen-lockfile --verbose --network-timeout 600000
|
(core) add machinery for self-managed flavor of Grist
Summary:
Currently, we have two ways that we deliver Grist. One is grist-core,
which has simple defaults and is relatively easy for third parties to
deploy. The second is our internal build for our SaaS, which is the
opposite. For self-managed Grist, a planned paid on-premise version
of Grist, I adopt the following approach:
* Use the `grist-core` build mechanism, extending it to accept an
overlay of extra code if present.
* Extra code is supplied in a self-contained `ext` directory, with
an `ext/app` directory that is of same structure as core `app`
and `stubs/app`.
* The `ext` directory also contains information about extra
node dependencies needed beyond that of `grist-core`.
* The `ext` directory is contained within our monorepo rather than
`grist-core` since it may contain material not under the Apache
license.
Docker builds are achieved in our monorepo by using the `--build-context`
functionality to add in `ext` during the regular `grist-core` build:
```
docker buildx build --load -t gristlabs/grist-ee --build-context=ext=../ext .
```
Incremental builds in our monorepo are achieved with the `build_core.sh` helper,
like:
```
buildtools/build_core.sh /tmp/self-managed
cd /tmp/self-managed
yarn start
```
The initial `ext` directory contains material for snapshotting to S3.
If you build the docker image as above, and have S3 access, you can
do something like:
```
docker run -p 8484:8484 --env GRIST_SESSION_SECRET=a-secret \
--env GRIST_DOCS_S3_BUCKET=grist-docs-test \
--env GRIST_DOCS_S3_PREFIX=self-managed \
-v $HOME/.aws:/root/.aws -it gristlabs/grist-ee
```
This will start a version of Grist that is like `grist-core` but with
S3 snapshots enabled. To release this code to `grist-core`, it would
just need to move from `ext/app` to `app` within core.
I tried a lot of ways of organizing self-managed Grist, and this was
what made me happiest. There are a lot of trade-offs, but here is what
I was looking for:
* Only OSS-code in grist-core. Adding mixed-license material there
feels unfair to people already working with the repo. That said,
a possible future is to move away from our private monorepo to
a public mixed-licence repo, which could have the same relationship
with grist-core as the monorepo has.
* Minimal differences between self-managed builds and one of our
existing builds, ideally hewing as close to grist-core as possible
for ease of documentation, debugging, and maintenance.
* Ideally, docker builds without copying files around (the new
`--build-context` functionality made that possible).
* Compatibility with monorepo build.
Expressing dependencies of the extra code in `ext` proved tricky to
do in a clean way. Yarn/npm fought me every step of the way - everything
related to optional dependencies was unsatisfactory in some respect.
Yarn2 is flexible but smells like it might be overreach. In the end,
organizing to install non-core dependencies one directory up from the
main build was a good simple trick that saved my bacon.
This diff gets us to the point of building `grist-ee` images conveniently,
but there isn't a public repo people can go look at to see its source. This
could be generated by taking `grist-core`, adding the `ext` directory
to it, and pushing to a distinct repository. I'm not in a hurry to do that,
since a PR to that repo would be hard to sync with our monorepo and
`grist-core`. Also, we don't have any licensing text ready for the `ext`
directory. So leaving that for future work.
Test Plan: manual
Reviewers: georgegevoian, alexmojaki
Reviewed By: georgegevoian, alexmojaki
Differential Revision: https://phab.getgrist.com/D3415
2022-05-12 15:24:48 +00:00
|
|
|
|
|
|
|
|
# Install any extra node dependencies (at root level, to avoid having to wrestle
|
|
|
|
|
# with merging them).
|
|
|
|
|
COPY --from=ext / /grist/ext
|
|
|
|
|
RUN \
|
|
|
|
|
mkdir /node_modules && \
|
|
|
|
|
cd /grist/ext && \
|
2022-09-11 23:33:11 +00:00
|
|
|
{ if [ -e package.json ] ; then yarn install --frozen-lockfile --modules-folder=/node_modules --verbose --network-timeout 600000 ; fi }
|
2020-10-10 00:54:23 +00:00
|
|
|
|
|
|
|
|
# Build node code.
|
(core) add machinery for self-managed flavor of Grist
Summary:
Currently, we have two ways that we deliver Grist. One is grist-core,
which has simple defaults and is relatively easy for third parties to
deploy. The second is our internal build for our SaaS, which is the
opposite. For self-managed Grist, a planned paid on-premise version
of Grist, I adopt the following approach:
* Use the `grist-core` build mechanism, extending it to accept an
overlay of extra code if present.
* Extra code is supplied in a self-contained `ext` directory, with
an `ext/app` directory that is of same structure as core `app`
and `stubs/app`.
* The `ext` directory also contains information about extra
node dependencies needed beyond that of `grist-core`.
* The `ext` directory is contained within our monorepo rather than
`grist-core` since it may contain material not under the Apache
license.
Docker builds are achieved in our monorepo by using the `--build-context`
functionality to add in `ext` during the regular `grist-core` build:
```
docker buildx build --load -t gristlabs/grist-ee --build-context=ext=../ext .
```
Incremental builds in our monorepo are achieved with the `build_core.sh` helper,
like:
```
buildtools/build_core.sh /tmp/self-managed
cd /tmp/self-managed
yarn start
```
The initial `ext` directory contains material for snapshotting to S3.
If you build the docker image as above, and have S3 access, you can
do something like:
```
docker run -p 8484:8484 --env GRIST_SESSION_SECRET=a-secret \
--env GRIST_DOCS_S3_BUCKET=grist-docs-test \
--env GRIST_DOCS_S3_PREFIX=self-managed \
-v $HOME/.aws:/root/.aws -it gristlabs/grist-ee
```
This will start a version of Grist that is like `grist-core` but with
S3 snapshots enabled. To release this code to `grist-core`, it would
just need to move from `ext/app` to `app` within core.
I tried a lot of ways of organizing self-managed Grist, and this was
what made me happiest. There are a lot of trade-offs, but here is what
I was looking for:
* Only OSS-code in grist-core. Adding mixed-license material there
feels unfair to people already working with the repo. That said,
a possible future is to move away from our private monorepo to
a public mixed-licence repo, which could have the same relationship
with grist-core as the monorepo has.
* Minimal differences between self-managed builds and one of our
existing builds, ideally hewing as close to grist-core as possible
for ease of documentation, debugging, and maintenance.
* Ideally, docker builds without copying files around (the new
`--build-context` functionality made that possible).
* Compatibility with monorepo build.
Expressing dependencies of the extra code in `ext` proved tricky to
do in a clean way. Yarn/npm fought me every step of the way - everything
related to optional dependencies was unsatisfactory in some respect.
Yarn2 is flexible but smells like it might be overreach. In the end,
organizing to install non-core dependencies one directory up from the
main build was a good simple trick that saved my bacon.
This diff gets us to the point of building `grist-ee` images conveniently,
but there isn't a public repo people can go look at to see its source. This
could be generated by taking `grist-core`, adding the `ext` directory
to it, and pushing to a distinct repository. I'm not in a hurry to do that,
since a PR to that repo would be hard to sync with our monorepo and
`grist-core`. Also, we don't have any licensing text ready for the `ext`
directory. So leaving that for future work.
Test Plan: manual
Reviewers: georgegevoian, alexmojaki
Reviewed By: georgegevoian, alexmojaki
Differential Revision: https://phab.getgrist.com/D3415
2022-05-12 15:24:48 +00:00
|
|
|
COPY tsconfig.json /grist
|
|
|
|
|
COPY tsconfig-ext.json /grist
|
|
|
|
|
COPY test/tsconfig.json /grist/test/tsconfig.json
|
2022-07-06 15:13:24 +00:00
|
|
|
COPY test/chai-as-promised.js /grist/test/chai-as-promised.js
|
(core) add machinery for self-managed flavor of Grist
Summary:
Currently, we have two ways that we deliver Grist. One is grist-core,
which has simple defaults and is relatively easy for third parties to
deploy. The second is our internal build for our SaaS, which is the
opposite. For self-managed Grist, a planned paid on-premise version
of Grist, I adopt the following approach:
* Use the `grist-core` build mechanism, extending it to accept an
overlay of extra code if present.
* Extra code is supplied in a self-contained `ext` directory, with
an `ext/app` directory that is of same structure as core `app`
and `stubs/app`.
* The `ext` directory also contains information about extra
node dependencies needed beyond that of `grist-core`.
* The `ext` directory is contained within our monorepo rather than
`grist-core` since it may contain material not under the Apache
license.
Docker builds are achieved in our monorepo by using the `--build-context`
functionality to add in `ext` during the regular `grist-core` build:
```
docker buildx build --load -t gristlabs/grist-ee --build-context=ext=../ext .
```
Incremental builds in our monorepo are achieved with the `build_core.sh` helper,
like:
```
buildtools/build_core.sh /tmp/self-managed
cd /tmp/self-managed
yarn start
```
The initial `ext` directory contains material for snapshotting to S3.
If you build the docker image as above, and have S3 access, you can
do something like:
```
docker run -p 8484:8484 --env GRIST_SESSION_SECRET=a-secret \
--env GRIST_DOCS_S3_BUCKET=grist-docs-test \
--env GRIST_DOCS_S3_PREFIX=self-managed \
-v $HOME/.aws:/root/.aws -it gristlabs/grist-ee
```
This will start a version of Grist that is like `grist-core` but with
S3 snapshots enabled. To release this code to `grist-core`, it would
just need to move from `ext/app` to `app` within core.
I tried a lot of ways of organizing self-managed Grist, and this was
what made me happiest. There are a lot of trade-offs, but here is what
I was looking for:
* Only OSS-code in grist-core. Adding mixed-license material there
feels unfair to people already working with the repo. That said,
a possible future is to move away from our private monorepo to
a public mixed-licence repo, which could have the same relationship
with grist-core as the monorepo has.
* Minimal differences between self-managed builds and one of our
existing builds, ideally hewing as close to grist-core as possible
for ease of documentation, debugging, and maintenance.
* Ideally, docker builds without copying files around (the new
`--build-context` functionality made that possible).
* Compatibility with monorepo build.
Expressing dependencies of the extra code in `ext` proved tricky to
do in a clean way. Yarn/npm fought me every step of the way - everything
related to optional dependencies was unsatisfactory in some respect.
Yarn2 is flexible but smells like it might be overreach. In the end,
organizing to install non-core dependencies one directory up from the
main build was a good simple trick that saved my bacon.
This diff gets us to the point of building `grist-ee` images conveniently,
but there isn't a public repo people can go look at to see its source. This
could be generated by taking `grist-core`, adding the `ext` directory
to it, and pushing to a distinct repository. I'm not in a hurry to do that,
since a PR to that repo would be hard to sync with our monorepo and
`grist-core`. Also, we don't have any licensing text ready for the `ext`
directory. So leaving that for future work.
Test Plan: manual
Reviewers: georgegevoian, alexmojaki
Reviewed By: georgegevoian, alexmojaki
Differential Revision: https://phab.getgrist.com/D3415
2022-05-12 15:24:48 +00:00
|
|
|
COPY app /grist/app
|
|
|
|
|
COPY stubs /grist/stubs
|
|
|
|
|
COPY buildtools /grist/buildtools
|
2021-04-02 23:11:27 +00:00
|
|
|
RUN yarn run build:prod
|
2020-10-10 00:54:23 +00:00
|
|
|
|
2022-01-07 17:06:04 +00:00
|
|
|
################################################################################
|
|
|
|
|
## Python collection stage
|
|
|
|
|
################################################################################
|
|
|
|
|
|
|
|
|
|
# Fetch python3.9 and python2.7
|
|
|
|
|
FROM python:3.9-slim-buster as collector
|
|
|
|
|
|
2020-10-10 00:54:23 +00:00
|
|
|
# Install all python dependencies.
|
|
|
|
|
ADD sandbox/requirements.txt requirements.txt
|
2022-01-07 17:06:04 +00:00
|
|
|
ADD sandbox/requirements3.txt requirements3.txt
|
2020-10-10 00:54:23 +00:00
|
|
|
RUN \
|
|
|
|
|
apt update && \
|
2022-04-03 23:43:29 +00:00
|
|
|
apt install -y --no-install-recommends python2 python-pip python-setuptools \
|
|
|
|
|
build-essential libxml2-dev libxslt-dev python-dev zlib1g-dev && \
|
|
|
|
|
pip2 install wheel && \
|
2022-01-07 17:06:04 +00:00
|
|
|
pip2 install -r requirements.txt && \
|
|
|
|
|
pip3 install -r requirements3.txt
|
2020-10-10 00:54:23 +00:00
|
|
|
|
2022-03-24 20:27:34 +00:00
|
|
|
################################################################################
|
|
|
|
|
## Sandbox collection stage
|
|
|
|
|
################################################################################
|
|
|
|
|
|
|
|
|
|
# Fetch gvisor-based sandbox. Note, to enable it to run within default
|
|
|
|
|
# unprivileged docker, layers of protection that require privilege have
|
|
|
|
|
# been stripped away, see https://github.com/google/gvisor/issues/4371
|
|
|
|
|
FROM gristlabs/gvisor-unprivileged:buster as sandbox
|
|
|
|
|
|
2020-10-10 00:54:23 +00:00
|
|
|
################################################################################
|
|
|
|
|
## Run-time stage
|
|
|
|
|
################################################################################
|
|
|
|
|
|
|
|
|
|
# Now, start preparing final image.
|
2021-11-30 04:14:50 +00:00
|
|
|
FROM node:14-buster-slim
|
2020-10-10 00:54:23 +00:00
|
|
|
|
2022-01-07 17:06:04 +00:00
|
|
|
# Install libexpat1, libsqlite3-0 for python3 library binary dependencies.
|
2022-03-24 20:27:34 +00:00
|
|
|
# Install pgrep for managing gvisor processes.
|
2022-01-07 17:06:04 +00:00
|
|
|
RUN \
|
|
|
|
|
apt-get update && \
|
2022-03-24 20:27:34 +00:00
|
|
|
apt-get install -y --no-install-recommends libexpat1 libsqlite3-0 procps && \
|
2022-01-07 17:06:04 +00:00
|
|
|
rm -rf /var/lib/apt/lists/*
|
|
|
|
|
|
|
|
|
|
# Keep all storage user may want to persist in a distinct directory
|
|
|
|
|
RUN mkdir -p /persist/docs
|
|
|
|
|
|
2020-10-10 00:54:23 +00:00
|
|
|
# Copy node files.
|
(core) add machinery for self-managed flavor of Grist
Summary:
Currently, we have two ways that we deliver Grist. One is grist-core,
which has simple defaults and is relatively easy for third parties to
deploy. The second is our internal build for our SaaS, which is the
opposite. For self-managed Grist, a planned paid on-premise version
of Grist, I adopt the following approach:
* Use the `grist-core` build mechanism, extending it to accept an
overlay of extra code if present.
* Extra code is supplied in a self-contained `ext` directory, with
an `ext/app` directory that is of same structure as core `app`
and `stubs/app`.
* The `ext` directory also contains information about extra
node dependencies needed beyond that of `grist-core`.
* The `ext` directory is contained within our monorepo rather than
`grist-core` since it may contain material not under the Apache
license.
Docker builds are achieved in our monorepo by using the `--build-context`
functionality to add in `ext` during the regular `grist-core` build:
```
docker buildx build --load -t gristlabs/grist-ee --build-context=ext=../ext .
```
Incremental builds in our monorepo are achieved with the `build_core.sh` helper,
like:
```
buildtools/build_core.sh /tmp/self-managed
cd /tmp/self-managed
yarn start
```
The initial `ext` directory contains material for snapshotting to S3.
If you build the docker image as above, and have S3 access, you can
do something like:
```
docker run -p 8484:8484 --env GRIST_SESSION_SECRET=a-secret \
--env GRIST_DOCS_S3_BUCKET=grist-docs-test \
--env GRIST_DOCS_S3_PREFIX=self-managed \
-v $HOME/.aws:/root/.aws -it gristlabs/grist-ee
```
This will start a version of Grist that is like `grist-core` but with
S3 snapshots enabled. To release this code to `grist-core`, it would
just need to move from `ext/app` to `app` within core.
I tried a lot of ways of organizing self-managed Grist, and this was
what made me happiest. There are a lot of trade-offs, but here is what
I was looking for:
* Only OSS-code in grist-core. Adding mixed-license material there
feels unfair to people already working with the repo. That said,
a possible future is to move away from our private monorepo to
a public mixed-licence repo, which could have the same relationship
with grist-core as the monorepo has.
* Minimal differences between self-managed builds and one of our
existing builds, ideally hewing as close to grist-core as possible
for ease of documentation, debugging, and maintenance.
* Ideally, docker builds without copying files around (the new
`--build-context` functionality made that possible).
* Compatibility with monorepo build.
Expressing dependencies of the extra code in `ext` proved tricky to
do in a clean way. Yarn/npm fought me every step of the way - everything
related to optional dependencies was unsatisfactory in some respect.
Yarn2 is flexible but smells like it might be overreach. In the end,
organizing to install non-core dependencies one directory up from the
main build was a good simple trick that saved my bacon.
This diff gets us to the point of building `grist-ee` images conveniently,
but there isn't a public repo people can go look at to see its source. This
could be generated by taking `grist-core`, adding the `ext` directory
to it, and pushing to a distinct repository. I'm not in a hurry to do that,
since a PR to that repo would be hard to sync with our monorepo and
`grist-core`. Also, we don't have any licensing text ready for the `ext`
directory. So leaving that for future work.
Test Plan: manual
Reviewers: georgegevoian, alexmojaki
Reviewed By: georgegevoian, alexmojaki
Differential Revision: https://phab.getgrist.com/D3415
2022-05-12 15:24:48 +00:00
|
|
|
COPY --from=builder /node_modules /node_modules
|
|
|
|
|
COPY --from=builder /grist/node_modules /grist/node_modules
|
|
|
|
|
COPY --from=builder /grist/_build /grist/_build
|
|
|
|
|
COPY --from=builder /grist/static /grist/static-built
|
2020-10-10 00:54:23 +00:00
|
|
|
|
2022-01-07 17:06:04 +00:00
|
|
|
# Copy python files.
|
|
|
|
|
COPY --from=collector /usr/bin/python2.7 /usr/bin/python2.7
|
|
|
|
|
COPY --from=collector /usr/lib/python2.7 /usr/lib/python2.7
|
|
|
|
|
COPY --from=collector /usr/local/lib/python2.7 /usr/local/lib/python2.7
|
|
|
|
|
COPY --from=collector /usr/local/bin/python3.9 /usr/bin/python3.9
|
|
|
|
|
COPY --from=collector /usr/local/lib/python3.9 /usr/local/lib/python3.9
|
|
|
|
|
COPY --from=collector /usr/local/lib/libpython3.9.* /usr/local/lib/
|
|
|
|
|
# Set default to python3
|
2022-03-24 20:27:34 +00:00
|
|
|
RUN \
|
|
|
|
|
ln -s /usr/bin/python3.9 /usr/bin/python && \
|
|
|
|
|
ln -s /usr/bin/python3.9 /usr/bin/python3 && \
|
|
|
|
|
ldconfig
|
|
|
|
|
|
|
|
|
|
# Copy runsc.
|
|
|
|
|
COPY --from=sandbox /runsc /usr/bin/runsc
|
2020-10-10 00:54:23 +00:00
|
|
|
|
|
|
|
|
# Add files needed for running server.
|
(core) add machinery for self-managed flavor of Grist
Summary:
Currently, we have two ways that we deliver Grist. One is grist-core,
which has simple defaults and is relatively easy for third parties to
deploy. The second is our internal build for our SaaS, which is the
opposite. For self-managed Grist, a planned paid on-premise version
of Grist, I adopt the following approach:
* Use the `grist-core` build mechanism, extending it to accept an
overlay of extra code if present.
* Extra code is supplied in a self-contained `ext` directory, with
an `ext/app` directory that is of same structure as core `app`
and `stubs/app`.
* The `ext` directory also contains information about extra
node dependencies needed beyond that of `grist-core`.
* The `ext` directory is contained within our monorepo rather than
`grist-core` since it may contain material not under the Apache
license.
Docker builds are achieved in our monorepo by using the `--build-context`
functionality to add in `ext` during the regular `grist-core` build:
```
docker buildx build --load -t gristlabs/grist-ee --build-context=ext=../ext .
```
Incremental builds in our monorepo are achieved with the `build_core.sh` helper,
like:
```
buildtools/build_core.sh /tmp/self-managed
cd /tmp/self-managed
yarn start
```
The initial `ext` directory contains material for snapshotting to S3.
If you build the docker image as above, and have S3 access, you can
do something like:
```
docker run -p 8484:8484 --env GRIST_SESSION_SECRET=a-secret \
--env GRIST_DOCS_S3_BUCKET=grist-docs-test \
--env GRIST_DOCS_S3_PREFIX=self-managed \
-v $HOME/.aws:/root/.aws -it gristlabs/grist-ee
```
This will start a version of Grist that is like `grist-core` but with
S3 snapshots enabled. To release this code to `grist-core`, it would
just need to move from `ext/app` to `app` within core.
I tried a lot of ways of organizing self-managed Grist, and this was
what made me happiest. There are a lot of trade-offs, but here is what
I was looking for:
* Only OSS-code in grist-core. Adding mixed-license material there
feels unfair to people already working with the repo. That said,
a possible future is to move away from our private monorepo to
a public mixed-licence repo, which could have the same relationship
with grist-core as the monorepo has.
* Minimal differences between self-managed builds and one of our
existing builds, ideally hewing as close to grist-core as possible
for ease of documentation, debugging, and maintenance.
* Ideally, docker builds without copying files around (the new
`--build-context` functionality made that possible).
* Compatibility with monorepo build.
Expressing dependencies of the extra code in `ext` proved tricky to
do in a clean way. Yarn/npm fought me every step of the way - everything
related to optional dependencies was unsatisfactory in some respect.
Yarn2 is flexible but smells like it might be overreach. In the end,
organizing to install non-core dependencies one directory up from the
main build was a good simple trick that saved my bacon.
This diff gets us to the point of building `grist-ee` images conveniently,
but there isn't a public repo people can go look at to see its source. This
could be generated by taking `grist-core`, adding the `ext` directory
to it, and pushing to a distinct repository. I'm not in a hurry to do that,
since a PR to that repo would be hard to sync with our monorepo and
`grist-core`. Also, we don't have any licensing text ready for the `ext`
directory. So leaving that for future work.
Test Plan: manual
Reviewers: georgegevoian, alexmojaki
Reviewed By: georgegevoian, alexmojaki
Differential Revision: https://phab.getgrist.com/D3415
2022-05-12 15:24:48 +00:00
|
|
|
ADD package.json /grist/package.json
|
|
|
|
|
ADD bower_components /grist/bower_components
|
|
|
|
|
ADD sandbox /grist/sandbox
|
|
|
|
|
ADD plugins /grist/plugins
|
|
|
|
|
ADD static /grist/static
|
|
|
|
|
|
|
|
|
|
# Finalize static directory
|
|
|
|
|
RUN \
|
|
|
|
|
mv /grist/static-built/* /grist/static && \
|
|
|
|
|
rmdir /grist/static-built
|
|
|
|
|
|
|
|
|
|
WORKDIR /grist
|
2020-10-10 00:54:23 +00:00
|
|
|
|
|
|
|
|
# Set some default environment variables to give a setup that works out of the box when
|
|
|
|
|
# started as:
|
|
|
|
|
# docker run -p 8484:8484 -it <image>
|
|
|
|
|
# Variables will need to be overridden for other setups.
|
2022-04-06 19:52:24 +00:00
|
|
|
#
|
|
|
|
|
# GRIST_SANDBOX_FLAVOR is set to unsandboxed by default, because it
|
|
|
|
|
# appears that the services people use to run docker containers have
|
|
|
|
|
# a wide variety of security settings and the functionality needed for
|
|
|
|
|
# sandboxing may not be possible in every case. For default docker
|
|
|
|
|
# settings, you can get sandboxing as follows:
|
|
|
|
|
# docker run --env GRIST_SANDBOX_FLAVOR=gvisor -p 8484:8484 -it <image>
|
|
|
|
|
#
|
2022-03-24 20:27:34 +00:00
|
|
|
ENV \
|
|
|
|
|
PYTHON_VERSION_ON_CREATION=3 \
|
|
|
|
|
GRIST_ORG_IN_PATH=true \
|
|
|
|
|
GRIST_HOST=0.0.0.0 \
|
|
|
|
|
GRIST_SINGLE_PORT=true \
|
|
|
|
|
GRIST_SERVE_SAME_ORIGIN=true \
|
|
|
|
|
GRIST_DATA_DIR=/persist/docs \
|
|
|
|
|
GRIST_INST_DIR=/persist \
|
|
|
|
|
GRIST_SESSION_COOKIE=grist_core \
|
|
|
|
|
GVISOR_FLAGS="-unprivileged -ignore-cgroups" \
|
2022-04-06 19:52:24 +00:00
|
|
|
GRIST_SANDBOX_FLAVOR=unsandboxed \
|
2022-03-24 20:27:34 +00:00
|
|
|
TYPEORM_DATABASE=/persist/home.sqlite3
|
|
|
|
|
|
2020-10-10 00:54:23 +00:00
|
|
|
EXPOSE 8484
|
2022-03-24 20:27:34 +00:00
|
|
|
|
|
|
|
|
CMD ./sandbox/run.sh
|
