From a7c0b31ea16cc45869b9891eb31190edf0aebd44 Mon Sep 17 00:00:00 2001 From: fflorent Date: Mon, 15 Apr 2024 17:06:07 +0200 Subject: [PATCH] Start documenting the databases --- documentation/database.md | 125 +++++ documentation/images/homedb-schema.svg | 609 +++++++++++++++++++++++++ 2 files changed, 734 insertions(+) create mode 100644 documentation/database.md create mode 100644 documentation/images/homedb-schema.svg diff --git a/documentation/database.md b/documentation/database.md new file mode 100644 index 00000000..1f9d0472 --- /dev/null +++ b/documentation/database.md @@ -0,0 +1,125 @@ +# Database + +First of all, let's explicit two databases that Grist manages: +1. The Home Database; +2. The Document Database (aka the grist document); + +The Home database is responsible for things related to the instance, like: + - the users and the groups registered on the instance; + - the billing; + - the organisations (aka sites), the workspaces; + - the documents metadata (id, name, workspace under which it is located...); + - the rights (ACL) to access to organisations, workspaces and documents (the access to the content of the document is controlled by the document itself); + +A Grist Document is a Sqlite database which contains data like: + - The tables, pages, views data; + - The ACL *inside* to access to all or part of tables (rows or columns); + +## The Document Database + +### Inspecting the Document + +A Grist Document (with the `.grist` extension) is actually a sqlite database. You may download a document like [this one](https://api.getgrist.com/o/templates/api/docs/keLK5sVeyfPkxyaXqijz2x/download?template=false&nohistory=false) and inspect its content using the `sqlite3` command: + +```` +$ sqlite3 Flashcards.grist +sqlite> .tables +Flashcards_Data _grist_TabBar +Flashcards_Data_summary_Card_Set _grist_TabItems +GristDocTour _grist_TableViews +_grist_ACLMemberships _grist_Tables +_grist_ACLPrincipals _grist_Tables_column +_grist_ACLResources _grist_Triggers +_grist_ACLRules _grist_Validations +_grist_Attachments _grist_Views +_grist_Cells _grist_Views_section +_grist_DocInfo _grist_Views_section_field +_grist_External_database _gristsys_Action +_grist_External_table _gristsys_ActionHistory +_grist_Filters _gristsys_ActionHistoryBranch +_grist_Imports _gristsys_Action_step +_grist_Pages _gristsys_FileInfo +_grist_REPL_Hist _gristsys_Files +_grist_Shares _gristsys_PluginData +```` + +:warning: Be sure to work on a copy of a document if you inspect its content, otherwise you may loose data. + +### The migrations + +The migrations are handled in the python sandbox in this code: +https://github.com/gristlabs/grist-core/blob/main/sandbox/grist/migrations.py + +For more information, please consult [the documentation for migrations](./migrations.md). + +## The Home Database + +The home database may either be a sqlite or a postgresql database depending on how the Grist instance has been installed. You may check the `TYPEORM_*` env variables in the [README](https://github.com/gristlabs/grist-core/blob/main/README.md). + +Unless otherwise configured, the home database is a sqlite file. In docker, it is stored in `/persist/home.sqlite3`. + +The schema below is the same (except minor differences in the column types) whatever the database type is. + +### The Schema + +As of 2024-04-15, the database schema is the following (it may have changed in the meantime): + +![Schema of the home database](./images/homedb-schema.svg) + +> [!NOTE] +> For simplicity's sake, we have removed tables related to the billing, nor to the migrations. + +If you want to generate the above schema by yourself, you may run the following command using [SchemaCrawler](https://www.schemacrawler.com/) ([a docker image is available for a quick run](https://www.schemacrawler.com/docker-image.html)): +````bash +# You may adapt the --database argument to fit with the actual file name +# You may also remove the `--grep-tables` option and all that follows to get the full schema. +$ schemacrawler --server=sqlite --database=landing.db --info-level=standard --portable-names --command=schema --output-format=svg --output-file=/tmp/graph.svg --grep-tables="products|billing_accounts|limits|billing_account_managers|activations|migrations" --invert-match +```` + +### `orgs` table + +Tables whose rows represent organisations (also called "Team sites"). + +| Column name | Description | +| ------------- | -------------- | +| id | The primary key | +| name | The name as displayed in the UI | +| domain | The part that should be added in the URL | +| owner | The id of the user who owns the org | +| host | ??? | + +### `workspaces` table + +Tables whose rows represent workspaces + +| Column name | Description | +| ------------- | -------------- | +| id | The primary key | +| name | The name as displayed in the UI | +| org_id | The organisation to which the workspace belongs | +| removed_at | If not null, stores the date when the workspaces has been placed in the trash (it will be hard deleted after 30 days) | + + +### `docs` table + +Tables whose rows represent documents + +| Column name | Description | +| ------------- | -------------- | +| id | The primary key | +| name | The name as displayed in the UI | +| workspace_id | The workspace to which the document belongs | +| is_pinned | Whether the document has been pinned or not | +| url_id | Short version of the `id`, as displayed in the URL | +| removed_at | If not null, stores the date when the workspaces has been placed in the trash (it will be hard deleted after 30 days) | +| options | Serialized options as described in [DocumentOptions](https://github.com/gristlabs/grist-core/blob/4567fad94787c20f65db68e744c47d5f44b932e4/app/common/UserAPI.ts#L125-L135) | +| grace_period_start | Specific to getgrist.com (TODO describe it) | +| usage | stats about the document (see [DocumentUsage](https://github.com/gristlabs/grist-core/blob/4567fad94787c20f65db68e744c47d5f44b932e4/app/common/DocUsage.ts)) | +| trunk_id | If set, the current document is a fork (as of 2024-04-15, only from a tutorial), and this column references the original document | +| type | If set, the current document is a special one (as specified in [DocumentType](https://github.com/gristlabs/grist-core/blob/4567fad94787c20f65db68e744c47d5f44b932e4/app/common/UserAPI.ts#L123)) | + + +### The migrations + +The database migrations are handled by TypeORM ([documentation](https://typeorm.io/migrations)). The migration files are located at `app/gen-server/migration` and are run at startup (so you don't have to worry about running them yourself). + diff --git a/documentation/images/homedb-schema.svg b/documentation/images/homedb-schema.svg new file mode 100644 index 00000000..eed9fc31 --- /dev/null +++ b/documentation/images/homedb-schema.svg @@ -0,0 +1,609 @@ + + + + + + +SchemaCrawler_Diagram + +generated by +SchemaCrawler 16.21.2 +generated on +2024-04-15 14:21:22 + + + +acl_rules_53bd8961 + +acl_rules + +[table] +id + +INTEGER NOT NULL + +auto-incremented +permissions + +INTEGER NOT NULL +type + +VARCHAR NOT NULL +workspace_id + +INTEGER +org_id + +INTEGER +doc_id + +VARCHAR +group_id + +INTEGER + + + + +docs_2f969a + +docs + +[table] +id + +VARCHAR NOT NULL +name + +VARCHAR NOT NULL +created_at + +DATETIME NOT NULL +updated_at + +DATETIME NOT NULL +workspace_id + +INTEGER +is_pinned + +BOOLEAN NOT NULL +url_id + +VARCHAR +removed_at + +DATETIME +options + +VARCHAR +grace_period_start + +DATETIME +usage + +VARCHAR +created_by + +INTEGER +trunk_id + +TEXT +type + +TEXT + + + + +acl_rules_53bd8961:w->docs_2f969a:e + + + + + + + + + + +groups_b63e4e33 + +groups + +[table] +id + +INTEGER NOT NULL + +auto-incremented +name + +VARCHAR NOT NULL + + + + +acl_rules_53bd8961:w->groups_b63e4e33:e + + + + + + + + + + +orgs_34a26e + +orgs + +[table] +id + +INTEGER NOT NULL + +auto-incremented +name + +VARCHAR NOT NULL +domain + +VARCHAR +created_at + +DATETIME NOT NULL +updated_at + +DATETIME NOT NULL +owner_id + +INTEGER +billing_account_id + +INTEGER +host + +VARCHAR + + + + +acl_rules_53bd8961:w->orgs_34a26e:e + + + + + + + + + + +workspaces_e61add + +workspaces + +[table] +id + +INTEGER NOT NULL + +auto-incremented +name + +VARCHAR NOT NULL +created_at + +DATETIME NOT NULL +updated_at + +DATETIME NOT NULL +org_id + +INTEGER +removed_at + +DATETIME + + + + +acl_rules_53bd8961:w->workspaces_e61add:e + + + + + + + + + + +aliases_c97dc35d + +aliases + +[table] +url_id + +VARCHAR NOT NULL +org_id + +INTEGER NOT NULL +doc_id + +VARCHAR +created_at + +DATETIME NOT NULL + + + + +aliases_c97dc35d:w->docs_2f969a:e + + + + + + + + + + +aliases_c97dc35d:w->orgs_34a26e:e + + + + + + + + + + +docs_2f969a:w->docs_2f969a:e + + + + + + + + + + +docs_2f969a:w->workspaces_e61add:e + + + + + + + + + + +users_6a70267 + +users + +[table] +id + +INTEGER NOT NULL + +auto-incremented +name + +VARCHAR NOT NULL +api_key + +VARCHAR +picture + +VARCHAR +first_login_at + +DATETIME +is_first_time_user + +INTEGER NOT NULL +options + +VARCHAR +connect_id + +VARCHAR +"ref" + +VARCHAR NOT NULL + + + + +docs_2f969a:w->users_6a70267:e + + + + + + + + + + +secrets_756efc22 + +secrets + +[table] +id + +VARCHAR NOT NULL +"value" + +VARCHAR NOT NULL +doc_id + +VARCHAR NOT NULL + + + + +secrets_756efc22:w->docs_2f969a:e + + + + + + + + + + +shares_ca2520d3 + +shares + +[table] +id + +INTEGER NOT NULL + +auto-incremented +key + +VARCHAR NOT NULL +doc_id + +VARCHAR NOT NULL +link_id + +VARCHAR NOT NULL +options + +VARCHAR NOT NULL + + + + +shares_ca2520d3:w->docs_2f969a:e + + + + + + + + + + +group_groups_dfa1d7f3 + +group_groups + +[table] +group_id + +INTEGER NOT NULL +subgroup_id + +INTEGER NOT NULL + + + + +group_groups_dfa1d7f3:w->groups_b63e4e33:e + + + + + + + + + + +group_groups_dfa1d7f3:w->groups_b63e4e33:e + + + + + + + + + + +group_users_41cb40a7 + +group_users + +[table] +group_id + +INTEGER NOT NULL +user_id + +INTEGER NOT NULL + + + + +group_users_41cb40a7:w->groups_b63e4e33:e + + + + + + + + + + +group_users_41cb40a7:w->users_6a70267:e + + + + + + + + + + +logins_be987289 + +logins + +[table] +id + +INTEGER NOT NULL + +auto-incremented +user_id + +INTEGER NOT NULL +email + +VARCHAR NOT NULL +display_email + +VARCHAR NOT NULL + + + + +logins_be987289:w->users_6a70267:e + + + + + + + + + + +id_dc7c64b2 +billing_accounts.id + + + +orgs_34a26e:w->id_dc7c64b2:e + + + + + + + + + + +orgs_34a26e:w->users_6a70267:e + + + + + + + + + + + +prefs_660170f + +prefs + +[table] +org_id + +INTEGER +user_id + +INTEGER +prefs + +VARCHAR NOT NULL + + + + +prefs_660170f:w->orgs_34a26e:e + + + + + + + + + + +prefs_660170f:w->users_6a70267:e + + + + + + + + + + +workspaces_e61add:w->orgs_34a26e:e + + + + + + + + + + +user_id_2d5fdf94 +billing_account_managers.user_id + + + +user_id_2d5fdf94:w->users_6a70267:e + + + + + + + + + +