2020-07-31 02:44:10 +00:00
# Grist
2021-05-15 16:12:36 +00:00
Grist is a modern relational spreadsheet. It combines the flexibility of a spreadsheet with the
2020-07-31 02:44:10 +00:00
robustness of a database to organize your data and make you more productive.
2022-01-26 21:01:30 +00:00
https://user-images.githubusercontent.com/118367/151245587-892e50a6-41f5-4b74-9786-fe3566f6b1fb.mp4
2022-01-07 17:06:04 +00:00
## Features
(By popular request: we have a specific write-up of [Grist vs Airtable ](https://www.getgrist.com/blog/grist-v-airtable/ ) that may be helpful).
Grist is a hybrid database/spreadsheet, meaning that:
- Columns work like they do in databases. They are named, and hold one kind of data.
- Columns can be filled by formula, spreadsheet-style, with automatic updates when referenced cells change.
Here are some specific feature highlights of Grist:
* Python formulas.
- Full [Python syntax is supported ](https://support.getgrist.com/formulas/#python ), and the standard library.
- Many [Excel functions ](https://support.getgrist.com/functions/ ) also available.
* A portable, self-contained format.
- Based on SQLite, the most widely deployed database engine.
- Any tool that can read SQLite can read numeric and text data from a Grist file.
- Great format for [backups ](https://support.getgrist.com/exports/#backing-up-an-entire-document ) that you can be confident you can restore in full.
- Great format for moving between different hosts.
* Convenient editing and formatting features.
- Choices and [choice lists ](https://support.getgrist.com/col-types/#choice-list-columns ), for adding colorful tags to records without fuss.
- [References ](https://support.getgrist.com/col-refs/#creating-a-new-reference-list-column ) and reference lists, for cross-referencing records in other tables.
- [Attachments ](https://support.getgrist.com/col-types/#attachment-columns ), to include media or document files in records.
- Dates and times, toggles, and special numerics such as currency all have specialized editors and formatting options.
2022-08-09 16:14:20 +00:00
- [Conditional Formatting ](https://support.getgrist.com/conditional-formatting/ ), letting you control the style of cells with formulas, to draw attention to important information.
2022-01-07 17:06:04 +00:00
* Great for dashboards, visualizations, and data entry.
- [Charts ](https://support.getgrist.com/widget-chart/ ) for visualization.
- [Summary tables ](https://support.getgrist.com/summary-tables/ ) for summing and counting across groups.
- [Widget linking ](https://support.getgrist.com/linking-widgets/ ) streamlines filtering and editing data.
Grist has a unique approach to visualization, where you can lay out and link distinct widgets to show together,
without cramming mixed material into a table.
- The [Filter bar ](https://support.getgrist.com/search-sort-filter/#filter-buttons ) is great for quick slicing and dicing.
* [Incremental imports ](https://support.getgrist.com/imports/#updating-existing-records ).
- So you can import a CSV of the last three months activity from your bank...
- ... and import new activity a month later without fuss or duplicates.
* Integrations.
- A [REST API ](https://support.getgrist.com/api/ ), [Zapier actions/triggers ](https://support.getgrist.com/integrators/#integrations-via-zapier ), and support from similar [integrators ](https://support.getgrist.com/integrators/ ).
- Import/export to Google drive, Excel format, CSV.
- Can link data with custom widgets hosted externally.
* [Many templates ](https://templates.getgrist.com/ ) to get you started, from investment research to organizing treasure hunts.
* Access control options.
- (You'll need SSO logins set up to make use of these options)
- Share [individual documents ](https://support.getgrist.com/sharing/ ), or workspaces, or [team sites ](https://support.getgrist.com/team-sharing/ ).
- Control access to [individual rows, columns, and tables ](https://support.getgrist.com/access-rules/ ).
- Control access based on cell values and user attributes.
* Can be self-maintained.
- Useful for intranet operation and specific compliance requirements.
2022-04-06 19:52:24 +00:00
* Sandboxing options for untrusted documents.
- On Linux or with docker, you can enable
(core) add machinery for self-managed flavor of Grist
Summary:
Currently, we have two ways that we deliver Grist. One is grist-core,
which has simple defaults and is relatively easy for third parties to
deploy. The second is our internal build for our SaaS, which is the
opposite. For self-managed Grist, a planned paid on-premise version
of Grist, I adopt the following approach:
* Use the `grist-core` build mechanism, extending it to accept an
overlay of extra code if present.
* Extra code is supplied in a self-contained `ext` directory, with
an `ext/app` directory that is of same structure as core `app`
and `stubs/app`.
* The `ext` directory also contains information about extra
node dependencies needed beyond that of `grist-core`.
* The `ext` directory is contained within our monorepo rather than
`grist-core` since it may contain material not under the Apache
license.
Docker builds are achieved in our monorepo by using the `--build-context`
functionality to add in `ext` during the regular `grist-core` build:
```
docker buildx build --load -t gristlabs/grist-ee --build-context=ext=../ext .
```
Incremental builds in our monorepo are achieved with the `build_core.sh` helper,
like:
```
buildtools/build_core.sh /tmp/self-managed
cd /tmp/self-managed
yarn start
```
The initial `ext` directory contains material for snapshotting to S3.
If you build the docker image as above, and have S3 access, you can
do something like:
```
docker run -p 8484:8484 --env GRIST_SESSION_SECRET=a-secret \
--env GRIST_DOCS_S3_BUCKET=grist-docs-test \
--env GRIST_DOCS_S3_PREFIX=self-managed \
-v $HOME/.aws:/root/.aws -it gristlabs/grist-ee
```
This will start a version of Grist that is like `grist-core` but with
S3 snapshots enabled. To release this code to `grist-core`, it would
just need to move from `ext/app` to `app` within core.
I tried a lot of ways of organizing self-managed Grist, and this was
what made me happiest. There are a lot of trade-offs, but here is what
I was looking for:
* Only OSS-code in grist-core. Adding mixed-license material there
feels unfair to people already working with the repo. That said,
a possible future is to move away from our private monorepo to
a public mixed-licence repo, which could have the same relationship
with grist-core as the monorepo has.
* Minimal differences between self-managed builds and one of our
existing builds, ideally hewing as close to grist-core as possible
for ease of documentation, debugging, and maintenance.
* Ideally, docker builds without copying files around (the new
`--build-context` functionality made that possible).
* Compatibility with monorepo build.
Expressing dependencies of the extra code in `ext` proved tricky to
do in a clean way. Yarn/npm fought me every step of the way - everything
related to optional dependencies was unsatisfactory in some respect.
Yarn2 is flexible but smells like it might be overreach. In the end,
organizing to install non-core dependencies one directory up from the
main build was a good simple trick that saved my bacon.
This diff gets us to the point of building `grist-ee` images conveniently,
but there isn't a public repo people can go look at to see its source. This
could be generated by taking `grist-core`, adding the `ext` directory
to it, and pushing to a distinct repository. I'm not in a hurry to do that,
since a PR to that repo would be hard to sync with our monorepo and
`grist-core`. Also, we don't have any licensing text ready for the `ext`
directory. So leaving that for future work.
Test Plan: manual
Reviewers: georgegevoian, alexmojaki
Reviewed By: georgegevoian, alexmojaki
Differential Revision: https://phab.getgrist.com/D3415
2022-05-12 15:24:48 +00:00
[gVisor ](https://github.com/google/gvisor ) sandboxing at the individual
document level.
2022-04-06 19:52:24 +00:00
- On OSX, you can use native sandboxing.
2020-07-31 02:44:10 +00:00
2021-08-18 02:20:00 +00:00
If you are curious about where Grist is going heading,
see [our roadmap ](https://github.com/gristlabs/grist-core/projects/1 ), drop a
question in [our forum ](https://community.getgrist.com ),
or browse [our extensive documentation ](https://support.getgrist.com ).
2022-01-07 17:06:04 +00:00
## Using Grist
2020-10-29 21:54:45 +00:00
2022-01-07 17:06:04 +00:00
There are docker images set up for individual use, or (with some
configuration) for self-hosting. Grist Labs offers a hosted service
2022-02-17 22:10:14 +00:00
at [docs.getgrist.com ](https://docs.getgrist.com ).
2020-10-28 17:37:36 +00:00
2022-04-06 19:52:24 +00:00
To get Grist running on your computer with [Docker ](https://www.docker.com/get-started ), do:
2020-10-28 17:37:36 +00:00
```sh
docker pull gristlabs/grist
docker run -p 8484:8484 -it gristlabs/grist
```
2022-01-07 17:06:04 +00:00
Then visit `http://localhost:8484` in your browser. You'll be able to create, edit, import,
and export documents. To preserve your work across docker runs, share a directory as `/persist` :
2020-10-28 17:37:36 +00:00
```sh
docker run -p 8484:8484 -v $PWD/persist:/persist -it gristlabs/grist
```
2022-02-17 22:10:14 +00:00
Get templates at [templates.getgrist.com ](https://templates.getgrist.com ) for payroll,
2022-01-07 17:06:04 +00:00
inventory management, invoicing, D& D encounter tracking, and a lot
more, or use any document you've created on
2022-02-17 22:10:14 +00:00
[docs.getgrist.com ](https://docs.getgrist.com ).
2020-10-29 21:54:45 +00:00
2022-01-07 17:06:04 +00:00
If you need to change the port Grist runs on, set a `PORT` variable, don't just change the
port mapping:
2020-10-29 21:54:45 +00:00
```
2022-01-07 17:06:04 +00:00
docker run --env PORT=9999 -p 9999:9999 -v $PWD/persist:/persist -it gristlabs/grist
```
2022-04-06 19:52:24 +00:00
To enable gVisor sandboxing, set `--env GRIST_SANDBOX_FLAVOR=gvisor` .
This should work with default docker settings, but may not work in all
environments.
2022-08-17 23:07:48 +00:00
You can find a lot more about configuring Grist, setting up authentication,
and running it on a public server in our
[Self-Managed Grist ](https://support.getgrist.com/self-managed/ ) handbook.
2022-01-07 17:06:04 +00:00
## Building from source
To build Grist from source, follow these steps:
2020-10-29 21:54:45 +00:00
2022-01-07 17:06:04 +00:00
yarn install
yarn run build:prod
yarn run install:python
yarn start
# Grist will be available at http://localhost:8484/
2020-10-29 21:54:45 +00:00
2022-04-06 19:52:24 +00:00
Grist formulas in documents will be run using Python executed directly on your
machine. You can configure sandboxing using a `GRIST_SANDBOX_FLAVOR`
environment variable.
* On OSX, `export GRIST_SANDBOX_FLAVOR=macSandboxExec`
uses the native `sandbox-exec` command for sandboxing.
* On Linux with [gVisor's runsc ](https://github.com/google/gvisor )
installed, `export GRIST_SANDBOX_FLAVOR=gvisor` is an option.
These sandboxing methods have been written for our own use at Grist Labs and
may need tweaking to work in your own environment - pull requests
very welcome here!
2022-01-07 17:06:04 +00:00
## Logins
Like git, Grist has features to track document revision history. So for full operation,
Grist expects to know who the user modifying a document is. Until it does, it operates
in a limited anonymous mode. To get you going, the docker image is configured so that
when you click on the "sign in" button Grist will attribute your work to `you@example.com` .
Change this by setting `GRIST_DEFAULT_EMAIL` :
```
docker run --env GRIST_DEFAULT_EMAIL=my@email -p 8484:8484 -v $PWD/persist:/persist -it gristlabs/grist
```
You can change your name in `Profile Settings` in
the [User Menu ](https://support.getgrist.com/glossary/#user-menu ).
2022-08-09 16:14:20 +00:00
For multi-user operation, or if you wish to access Grist across the
2022-08-17 23:07:48 +00:00
public internet, you'll want to connect it to your own Single Sign-On service.
There are a lot of ways to do this, including [SAML and forward authentication ](https://support.getgrist.com/self-managed/#how-do-i-set-up-authentication ).
Grist has been tested with [Authentik ](https://goauthentik.io/ ), [Auth0 ](https://auth0.com/ ),
and Google/Microsoft sign-ins via [Dex ](https://dexidp.io/ ).
2022-01-07 17:06:04 +00:00
## Why free and open source software
2020-10-29 21:54:45 +00:00
2022-01-07 17:06:04 +00:00
This repository, [grist-core ](https://github.com/gristlabs/grist-core ), is maintained by Grist
Labs. Our flagship product available at [getgrist.com ](https://www.getgrist.com ) is built from the code you see
here, combined with business-specific software designed to scale it to many users, handle billing,
etc.
Grist Labs is an open-core company. We offer Grist hosting as a
2022-08-17 23:07:48 +00:00
service, with free and paid plans. We also develop and sell
2022-01-07 17:06:04 +00:00
features related to Grist using a proprietary license, targeted at the
needs of enterprises with large self-managed installations. We see
data portability and autonomy as a key value Grist can bring to our
users, and `grist-core` as an essential means to deliver that. We are
committed to maintaining and improving the `grist-core` codebase, and
to be thoughtful about how proprietary offerings impact data portability
and autonomy.
2020-07-31 02:44:10 +00:00
By opening its source code and offering an [OSI ](https://opensource.org/ )-approved free license,
Grist benefits its users:
2022-01-07 17:06:04 +00:00
- **Developer community.** The freedom to examine source code, make bug fixes, and develop
new features is a big deal for a general-purpose spreadsheet-like product, where there is a
very long tail of features vital to someone somewhere.
- **Increased trust.** Because anyone can examine the source code, “ security by obscurity” is not
an option. Vulnerabilities in the code can be found by others and reported before they cause
2020-07-31 02:44:10 +00:00
damage.
2022-01-07 17:06:04 +00:00
- **Independence.** Grist is available to you regardless of the fortunes of the Grist Labs business,
since it is open source and can be self-hosted. Using our hosted solution is convenient, but you
are not locked in.
- **Price flexibility.** If you are low on funds but have time to invest, self-hosting is a great
option to have. And DIY users may have the technical savvy and motivation to delve in and make improvements,
which can benefit all users of Grist.
2020-07-31 02:44:10 +00:00
- **Extensibility.** For developers, having the source open makes it easier to build extensions (such as the
experimental [Custom Widget ](https://support.getgrist.com/widget-custom/ )). You can more easily
include Grist in your pipeline. And if a feature is missing, you can just take the source code and
2022-01-07 17:06:04 +00:00
build on top of it.
## Reviews
* [Grist on ProductHunt ](https://www.producthunt.com/posts/grist-2 )
* [Grist on AppSumo ](https://appsumo.com/products/grist/ ) (life-time deal is sold out)
* [Capterra ](https://www.capterra.com/p/232821/Grist/#reviews ), [G2 ](https://www.g2.com/products/grist/reviews ), [TrustRadius ](https://www.trustradius.com/products/grist/reviews )
2020-07-31 02:44:10 +00:00
2022-01-26 20:38:21 +00:00
## Environment variables
Grist can be configured in many ways. Here are the main environment variables it is sensitive to:
Variable | Purpose
-------- | -------
ALLOWED_WEBHOOK_DOMAINS | comma-separated list of permitted domains to use in webhooks (e.g. webhook.site,zapier.com)
APP_DOC_URL | doc worker url, set when starting an individual doc worker (other servers will find doc worker urls via redis)
APP_HOME_URL | url prefix for home api (home and doc servers need this)
APP_STATIC_URL | url prefix for static resources
2022-05-12 06:08:06 +00:00
APP_STATIC_INCLUDE_CUSTOM_CSS | set to "true" to include custom.css (from APP_STATIC_URL) in static pages
2022-01-26 20:38:21 +00:00
APP_UNTRUSTED_URL | URL at which to serve/expect plugin content.
2022-09-28 16:33:53 +00:00
GRIST_ADAPT_DOMAIN | set to "true" to support multiple base domains (careful, host header should be trustworthy)
GRIST_ALLOWED_HOSTS | comma-separated list of permitted domains origin for requests (e.g. my.site,another.com)
2022-01-26 20:38:21 +00:00
GRIST_APP_ROOT | directory containing Grist sandbox and assets (specifically the sandbox and static subdirectories).
GRIST_BACKUP_DELAY_SECS | wait this long after a doc change before making a backup
GRIST_DATA_DIR | directory in which to store document caches.
GRIST_DEFAULT_EMAIL | if set, login as this user if no other credentials presented
2022-02-08 18:40:48 +00:00
GRIST_DEFAULT_PRODUCT | if set, this controls enabled features and limits of new sites. See names of PRODUCTS in Product.ts.
2022-08-24 19:24:50 +00:00
GRIST_DEFAULT_LOCALE | Locale to use as fallback when Grist cannot honour the browser locale.
2022-01-26 20:38:21 +00:00
GRIST_DOMAIN | in hosted Grist, Grist is served from subdomains of this domain. Defaults to "getgrist.com".
GRIST_EXPERIMENTAL_PLUGINS | enables experimental plugins
2022-05-27 11:03:56 +00:00
GRIST_HIDE_UI_ELEMENTS | comma-separated list of parts of the UI to hide. Allowed names of parts: `helpCenter,billing,templates,multiSite,multiAccounts`
2022-01-26 20:38:21 +00:00
GRIST_HOME_INCLUDE_STATIC | if set, home server also serves static resources
GRIST_HOST | hostname to use when listening on a port.
GRIST_ID_PREFIX | for subdomains of form o-*, expect or produce o-${GRIST_ID_PREFIX}*.
2022-06-15 14:29:29 +00:00
GRIST_IGNORE_SESSION | if set, Grist will not use a session for authentication.
2022-01-26 20:38:21 +00:00
GRIST_INST_DIR | path to Grist instance configuration files, for Grist server.
2022-10-14 15:39:15 +00:00
GRIST_LIST_PUBLIC_SITES | if set to true, sites shared with the public will be listed for anonymous users. Defaults to false.
2022-01-26 20:38:21 +00:00
GRIST_MANAGED_WORKERS | if set, Grist can assume that if a url targeted at a doc worker returns a 404, that worker is gone
GRIST_MAX_UPLOAD_ATTACHMENT_MB | max allowed size for attachments (0 or empty for unlimited).
GRIST_MAX_UPLOAD_IMPORT_MB | max allowed size for imports (except .grist files) (0 or empty for unlimited).
GRIST_ORG_IN_PATH | if true, encode org in path rather than domain
2022-05-27 11:03:56 +00:00
GRIST_PAGE_TITLE_SUFFIX | a string to append to the end of the `<title>` in HTML documents. Defaults to `" - Grist"` . Set to `_blank` for no suffix at all.
2022-06-15 14:29:29 +00:00
GRIST_PROXY_AUTH_HEADER | header which will be set by a (reverse) proxy webserver with an authorized users' email. This can be used as an alternative to a SAML service. See also GRIST_FORWARD_AUTH_HEADER.
2022-01-26 20:38:21 +00:00
GRIST_ROUTER_URL | optional url for an api that allows servers to be (un)registered with a load balancer
GRIST_SERVE_SAME_ORIGIN | set to "true" to access home server and doc workers on the same protocol-host-port as the top-level page, same as for custom domains (careful, host header should be trustworthy)
GRIST_SESSION_COOKIE | if set, overrides the name of Grist's cookie
GRIST_SESSION_DOMAIN | if set, associates the cookie with the given domain - otherwise defaults to GRIST_DOMAIN
GRIST_SESSION_SECRET | a key used to encode sessions
2022-05-18 10:25:14 +00:00
GRIST_FORCE_LOGIN | when set to 'true' disables anonymous access
2022-01-26 20:38:21 +00:00
GRIST_SINGLE_ORG | set to an org "domain" to pin client to that org
GRIST_SUPPORT_ANON | if set to 'true', show UI for anonymous access (not shown by default)
2022-10-14 15:39:15 +00:00
GRIST_SUPPORT_EMAIL | if set, give a user with the specified email support powers. The main extra power is the ability to share sites, workspaces, and docs with all users in a listed way.
2022-01-26 20:38:21 +00:00
GRIST_THROTTLE_CPU | if set, CPU throttling is enabled
GRIST_USER_ROOT | an extra path to look for plugins in.
2022-05-18 10:25:14 +00:00
COOKIE_MAX_AGE | session cookie max age, defaults to 90 days; can be set to "none" to make it a session cookie
2022-01-26 20:38:21 +00:00
HOME_PORT | port number to listen on for REST API server; if set to "share", add API endpoints to regular grist port.
PORT | port number to listen on for Grist server
REDIS_URL | optional redis server for browser sessions and db query caching
Sandbox related variables:
Variable | Purpose
-------- | -------
GRIST_SANDBOX_FLAVOR | can be pynbox, unsandboxed, docker, or macSandboxExec. If set, forces Grist to use the specified kind of sandbox.
GRIST_SANDBOX | a program or image name to run as the sandbox. See NSandbox.ts for nerdy details.
PYTHON_VERSION | can be 2 or 3. If set, documents without an engine setting are assumed to use the specified version of python. Not all sandboxes support all versions.
PYTHON_VERSION_ON_CREATION | can be 2 or 3. If set, newly created documents have an engine setting set to python2 or python3. Not all sandboxes support all versions.
2022-06-15 14:29:29 +00:00
Forward authentication variables:
Variable | Purpose
-------- | -------
GRIST_FORWARD_AUTH_HEADER | if set, trust the specified header (e.g. "x-forwarded-user") to contain authorized user emails, and enable "forward auth" logins.
GRIST_FORWARD_AUTH_LOGIN_PATH | if GRIST_FORWARD_AUTH_HEADER is set, Grist will listen at this path for logins. Defaults to `/auth/login` .
GRIST_FORWARD_AUTH_LOGOUT_PATH | if GRIST_FORWARD_AUTH_HEADER is set, Grist will forward to this path when user logs out.
When using forward authentication, you may wish to also set the following variables:
* GRIST_FORCE_LOGIN=true to disable anonymous access.
* GRIST_IGNORE_SESSION=true to ignore any user identity information in a cookie.
Only do this if you use forward authentication on all paths.
You may not want to use forward authentication on all paths if it makes
signing in required, and you are trying to permit anonymous access.
GRIST_FORWARD_AUTH_HEADER is similar to GRIST_PROXY_AUTH_HEADER, but enables
a login system (assuming you have some forward authentication set up).
2022-01-26 20:38:21 +00:00
Google Drive integrations:
Variable | Purpose
-------- | -------
GOOGLE_CLIENT_ID | set to the Google Client Id to be used with Google API client
GOOGLE_CLIENT_SECRET| set to the Google Client Secret to be used with Google API client
GOOGLE_API_KEY | set to the Google API Key to be used with Google API client (accessing public files)
GOOGLE_DRIVE_SCOPE | set to the scope requested for Google Drive integration (defaults to drive.file)
Database variables:
Variable | Purpose
-------- | -------
TYPEORM_DATABASE | database filename for sqlite or database name for other db types
TYPEORM_HOST | host for db
TYPEORM_LOGGING | set to 'true' to see all sql queries
TYPEORM_PASSWORD | password to use
TYPEORM_PORT | port number for db if not the default for that db type
TYPEORM_TYPE | set to 'sqlite' or 'postgres'
TYPEORM_USERNAME | username to connect as
Testing:
Variable | Purpose
-------- | -------
2022-10-03 23:43:22 +00:00
GRIST_TESTING_SOCKET | a socket used for out-of-channel communication during tests only.
2022-01-26 20:38:21 +00:00
GRIST_TEST_HTTPS_OFFSET | if set, adds https ports at the specified offset. This is useful in testing.
2022-10-03 23:43:22 +00:00
GRIST_TEST_SSL_CERT | if set, contains filename of SSL certificate.
GRIST_TEST_SSL_KEY | if set, contains filename of SSL private key.
GRIST_TEST_LOGIN | allow fake unauthenticated test logins (suitable for dev environment only).
GRIST_TEST_ROUTER | if set, then the home server will serve a mock version of router api at /test/router
GREP_TESTS | pattern for selecting specific tests to run (e.g. `env GREP_TESTS=ActionLog yarn test` ).
## Tests
Tests are run automatically as part of CI when a PR is opened. However, it can be helpful to run tests locally
before pushing your changes to GitHub. First, you'll want to make sure you've installed all dependencies:
```
yarn install
yarn install:python
```
Then, you can run the main test suite like so:
```
yarn test
```
2022-10-03 23:46:33 +00:00
Python tests may also be run locally. (Note: currently requires Python 3.9.)
2022-10-03 23:43:22 +00:00
```
yarn test:python
```
For running specific tests, you can specify a pattern with the `GREP_TESTS` variable:
```
env GREP_TESTS=ChoiceList yarn test
env GREP_TESTS=summary yarn test:python
```
2022-01-26 20:38:21 +00:00
2022-01-07 17:06:04 +00:00
## License
2020-07-31 15:30:32 +00:00
This repository, `grist-core` , is released under the [Apache License, Version
2.0](http://www.apache.org/licenses/LICENSE-2.0), which is an
(core) add machinery for self-managed flavor of Grist
Summary:
Currently, we have two ways that we deliver Grist. One is grist-core,
which has simple defaults and is relatively easy for third parties to
deploy. The second is our internal build for our SaaS, which is the
opposite. For self-managed Grist, a planned paid on-premise version
of Grist, I adopt the following approach:
* Use the `grist-core` build mechanism, extending it to accept an
overlay of extra code if present.
* Extra code is supplied in a self-contained `ext` directory, with
an `ext/app` directory that is of same structure as core `app`
and `stubs/app`.
* The `ext` directory also contains information about extra
node dependencies needed beyond that of `grist-core`.
* The `ext` directory is contained within our monorepo rather than
`grist-core` since it may contain material not under the Apache
license.
Docker builds are achieved in our monorepo by using the `--build-context`
functionality to add in `ext` during the regular `grist-core` build:
```
docker buildx build --load -t gristlabs/grist-ee --build-context=ext=../ext .
```
Incremental builds in our monorepo are achieved with the `build_core.sh` helper,
like:
```
buildtools/build_core.sh /tmp/self-managed
cd /tmp/self-managed
yarn start
```
The initial `ext` directory contains material for snapshotting to S3.
If you build the docker image as above, and have S3 access, you can
do something like:
```
docker run -p 8484:8484 --env GRIST_SESSION_SECRET=a-secret \
--env GRIST_DOCS_S3_BUCKET=grist-docs-test \
--env GRIST_DOCS_S3_PREFIX=self-managed \
-v $HOME/.aws:/root/.aws -it gristlabs/grist-ee
```
This will start a version of Grist that is like `grist-core` but with
S3 snapshots enabled. To release this code to `grist-core`, it would
just need to move from `ext/app` to `app` within core.
I tried a lot of ways of organizing self-managed Grist, and this was
what made me happiest. There are a lot of trade-offs, but here is what
I was looking for:
* Only OSS-code in grist-core. Adding mixed-license material there
feels unfair to people already working with the repo. That said,
a possible future is to move away from our private monorepo to
a public mixed-licence repo, which could have the same relationship
with grist-core as the monorepo has.
* Minimal differences between self-managed builds and one of our
existing builds, ideally hewing as close to grist-core as possible
for ease of documentation, debugging, and maintenance.
* Ideally, docker builds without copying files around (the new
`--build-context` functionality made that possible).
* Compatibility with monorepo build.
Expressing dependencies of the extra code in `ext` proved tricky to
do in a clean way. Yarn/npm fought me every step of the way - everything
related to optional dependencies was unsatisfactory in some respect.
Yarn2 is flexible but smells like it might be overreach. In the end,
organizing to install non-core dependencies one directory up from the
main build was a good simple trick that saved my bacon.
This diff gets us to the point of building `grist-ee` images conveniently,
but there isn't a public repo people can go look at to see its source. This
could be generated by taking `grist-core`, adding the `ext` directory
to it, and pushing to a distinct repository. I'm not in a hurry to do that,
since a PR to that repo would be hard to sync with our monorepo and
`grist-core`. Also, we don't have any licensing text ready for the `ext`
directory. So leaving that for future work.
Test Plan: manual
Reviewers: georgegevoian, alexmojaki
Reviewed By: georgegevoian, alexmojaki
Differential Revision: https://phab.getgrist.com/D3415
2022-05-12 15:24:48 +00:00
[OSI ](https://opensource.org/ )-approved free software license.
See LICENSE.txt and NOTICE.txt for more information.
If you have received a version of Grist with an `ext` directory,
the material within it is separately licensed.
