Commit Graph

128 Commits

Author SHA1 Message Date
Jarosław Sadziński
d13b9b9019 (core) Billing for formula assistant
Summary:
Adding limits for AI calls and connecting those limits with a Stripe Account.

- New table in homedb called `limits`
- All calls to the AI are not routed through DocApi and measured.
- All products now contain a special key `assistantLimit`, with a default value 0
- Limit is reset every time the subscription has changed its period
- The billing page is updated with two new options that describe the AI plan
- There is a new popup that allows the user to upgrade to a higher plan
- Tiers are read directly from the Stripe product with a volume pricing model

Test Plan: Updated and added

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3907
2023-07-10 13:24:08 +02:00
George Gevoian
35237a5835 (core) Add Support Grist page and nudge
Summary:
Adds a new Support Grist page (accessible only in grist-core), containing
options to opt in to telemetry and sponsor Grist Labs on GitHub.

A nudge is also shown in the doc menu, which can be collapsed or permanently
dismissed.

Test Plan: Browser and server tests.

Reviewers: paulfitz, dsagal

Reviewed By: paulfitz

Subscribers: jarek, dsagal

Differential Revision: https://phab.getgrist.com/D3926
2023-07-04 17:36:59 -04:00
Paul Fitzpatrick
7e50467396 (core) tweak handler for aborted connections to work on modern node
Summary:
It became hard to detect aborted connections in node 16.
In node 14, req.on('close', ...) did the job. Thid diff adds a
work-around, until a better way is discovered or added.
Aborting a req will typically lead to 'close' being called
on the response, without writableFinished being set.

 - https://github.com/nodejs/node/issues/38924
 - https://github.com/nodejs/node/issues/40775

Test Plan:
existing DocApiForwarder test passes; manually
checking on various node versions.

Reviewers: JakubSerafin

Reviewed By: JakubSerafin

Differential Revision: https://phab.getgrist.com/D3923
2023-06-16 10:20:53 -04:00
Dmitry S
2740884e3c (core) Improve the look and behavior of /welcome/teams page (also shown for /welcome/start)
Summary:
- Move css module for the login page css to core/, to be reusable in core/ pages.
- Move /welcome/teams implementation to WelcomeSitePicker.ts
- List users for personal sites, as well as team sites.
- Add org param to setSessionActive() API method and end endpoint, to allow
  switching the specified org to another user.
- Add a little safety to getOrgUrl() function.

Test Plan: Added a test case for the new behaviors of the /welcome/teams page.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3914
2023-06-13 20:40:59 -04:00
George Gevoian
a460563daf (core) Polish telemetry code
Summary: Also fixes a few small bugs with telemetry collection.

Test Plan: Server and manual tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3915
2023-06-09 13:03:10 -04:00
George Gevoian
10f5f0cb37 (core) Add optional telemetry to grist-core
Summary:
Adds support for optional telemetry to grist-core.

A new environment variable, GRIST_TELEMETRY_LEVEL, controls the level of telemetry collected.

Test Plan: Server and unit tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal, anaisconce

Differential Revision: https://phab.getgrist.com/D3880
2023-06-07 12:00:51 -04:00
Dmitry S
d191859be7 (core) For exporting XLSX, do it memory-efficiently in a worker thread.
Summary:
- Excel exports were awfully memory-inefficient, causing occasional docWorker
  crashes. The fix is to use the "streaming writer" option of ExcelJS
  https://github.com/exceljs/exceljs#streaming-xlsx-writercontents. (Empirically
  on one example, max memory went down from 3G to 100M)
- It's also CPU intensive and synchronous, and can block node for tens of
  seconds. The fix is to use a worker-thread. This diff uses "piscina" library
  for a pool of threads.
- Additionally, adds ProcessMonitor that logs memory and cpu usage,
  particularly when those change significantly.
- Also introduces request cancellation, so that a long download cancelled by
  the user will cancel the work being done in the worker thread.

Test Plan:
Updated previous export tests; memory and CPU performance tested
manually by watching output of ProcessMonitor.

Difference visible in these log excerpts:

Before (total time to serve request 22 sec):
```
Telemetry processMonitor heapUsedMB=2187, heapTotalMB=2234, cpuAverage=1.13, intervalMs=17911
Telemetry processMonitor heapUsedMB=2188, heapTotalMB=2234, cpuAverage=0.66, intervalMs=5005
Telemetry processMonitor heapUsedMB=2188, heapTotalMB=2234, cpuAverage=0, intervalMs=5005
Telemetry processMonitor heapUsedMB=71, heapTotalMB=75, cpuAverage=0.13, intervalMs=5002
```
After (total time to server request 18 sec):
```
Telemetry processMonitor heapUsedMB=109, heapTotalMB=144, cpuAverage=0.5, intervalMs=5001
Telemetry processMonitor heapUsedMB=109, heapTotalMB=144, cpuAverage=1.39, intervalMs=5002
Telemetry processMonitor heapUsedMB=94, heapTotalMB=131, cpuAverage=1.13, intervalMs=5000
Telemetry processMonitor heapUsedMB=94, heapTotalMB=131, cpuAverage=1.35, intervalMs=5001
```
Note in "Before" that heapTotalMB goes up to 2GB in the first case, and "intervalMs" of 17 seconds indicates that node was unresponsive for that long. In the second case, heapTotalMB stays low, and the main thread remains responsive the whole time.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3906
2023-06-01 12:06:48 -04:00
Paul Fitzpatrick
7be0ee289d
support other SQLite wrappers, and various hooks needed by grist-static (#516) 2023-05-23 15:17:28 -04:00
George Gevoian
1e873b4203 (core) Tweak telemetry
Summary: Adjusts the level of telemetry collected from Grist SaaS.

Test Plan: Tested manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3899
2023-05-19 13:06:15 -04:00
George Gevoian
36f3fd0120 (core) Fix owner view access to snapshots
Summary:
Owners weren't able to access snapshots if access rules
that denied access to non-owners existed. The backend
was lowering snapshot document access to "viewers" as
part of implementing read-only behavior; this is now done
in the client, with document access for snapshots now
accurately reflecting the user's trunk access.

Additionally, sandboxes are no longer created for snapshots,
and background intervals aren't started for snapshots.

Test Plan: Browser test.

Reviewers: jarek, paulfitz

Reviewed By: jarek, paulfitz

Differential Revision: https://phab.getgrist.com/D3849
2023-04-17 00:16:59 -04:00
George Gevoian
900859854c (core) Convert site metric strings to ints
Summary:
Counts were being sent as strings to the Elasticsearch instance because the
SQL queries were returning strings for the aggregate columns instead of
integers.

Test Plan: N/A

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3859
2023-04-12 11:26:52 -04:00
George Gevoian
8a0bb4d4fe (core) Improve dark mode
Summary:
Enhances dark mode support for the formula editor, and adds support to
the color select popup. Also fixes some bugs with dark mode.

Test Plan: Tested manually.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3847
2023-04-12 01:58:48 -04:00
George Gevoian
a19ba0813a (core) Add telemetry
Test Plan: Server tests.

Reviewers: jarek

Differential Revision: https://phab.getgrist.com/D3818
2023-04-06 12:34:54 -04:00
Cyprien P
d8a063284a (core) Adds endpoint to update webhook
Summary:
Adds a new endpoint to update webhook.

Perform some refactoring to allow code reuse from endpoint allowing to _subscribe and _unsubscribe webhooks.

One aspect of webhook is that url are stored in the home db while the rest of the fields (tableRef, isReadyColRef, ...) are stored in sqlite. So care must be taken when updating fields, to properly rollback if anything should fail.

Follow up diff will bring UI to edit webhook list

Test Plan: Updated doc api server tests

Reviewers: jarek

Reviewed By: jarek

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3821
2023-03-31 19:26:02 +02:00
George Gevoian
be8e13df64 (core) Add initial tutorials implementation
Summary:
Documents can now be flagged as tutorials, which causes them to display
Markdown-formatted slides from a special GristDocTutorial table. Tutorial
documents are forked on open, and remember the last slide a user was on.
They can be restarted too, which prepares a new fork of the tutorial.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3813
2023-03-22 10:09:02 -04:00
Paul Fitzpatrick
cbf925aa00 (core) updates from grist-core 2023-02-21 10:49:19 -05:00
George Gevoian
1ac4931c22 (core) Persist forks in home db
Summary:
Adds information about forks to the home db. This will be used
later by the UI to list forks of documents.

Test Plan: Browser and server tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3772
2023-02-20 22:46:36 -05:00
Paul Fitzpatrick
3aba7f6208 (core) set up a trialing list and a call scheduled flag for Grist SaaS
Summary:
This adds support for the automation described in
https://grist.quip.com/rt3tAoEH4kuy/Automate-Sprouts-outreach-email

Test Plan: added tests; will need manual testing and iterative integration work

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3804
2023-02-20 15:32:58 -05:00
Paul Fitzpatrick
f7f76fb5e7
A set of tweaks to simplify electron packaging (#421)
* Replace `ormconfig.js` with a newer mechanism of configuring
    TypeORM that can be included in the source code properly.
    The path to `ormconfig.js` has always been awkward to handle,
    and eliminating the file makes building different Grist setups
    a bit simpler.
  * Remove `electron` package. It is barely used, just for some old
    remnants of an older attempt at electron packaging. It was used
    for two types, which I left at `any` for now. More code pruning is
    no doubt possible here, but I'd rather do it when Electron packaging
    has solidified.
  * Add a hook for replacing the login system, and for adding some
    extra middleware the login system may need.
  * Add support for some more possible locations of Python, which
    arise when a standalone version of it is included in the Electron
    package. This isn't very general purpose, just configurations
    that I found useful.
  * Support using grist-core within a yarn workspace - the only tweak
    needed was webpack related.
  * Allow an external ID to be optionally associated with documents.
2023-02-13 15:52:17 -05:00
Jarosław Sadziński
90d3ee037a (core) User language switcher
Summary:
New language selector on the Account page for logged-in users.
New icon for switching language for an anonymous user.

For anonymous users, language is stored in a cookie grist_user_locale.
Language is stored in user settings for authenticated users and takes
precedence over what is stored in the cookie.

Test Plan: New tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3766
2023-01-26 09:47:14 +01:00
Jarosław Sadziński
887cd388c0 (core) Adding creator as an owner for a new doc and ws
Summary:
By default editor inherits permission for a new document or workspace.
Now editor is added explicitly as an owner of a new doc or workspace.

Test Plan: Updated

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian, paulfitz

Subscribers: dsagal, paulfitz

Differential Revision: https://phab.getgrist.com/D3734
2023-01-09 17:56:48 +01:00
Cyprien P
cabac3d9d8 (core) Adds new view as banner
Summary:
Diff removes view-as pill in the document breadcrumbs and add new view-as banner.

Note: Banners are still missing mechanism to handle several banners. As of now both doc-usage and view-as banners could show up at the same time.

Test Plan: Refactored existing test.

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3732
2023-01-03 12:33:34 +01:00
Jarosław Sadziński
fa75c93d67 (core) Only owners should be able to rename a document.
Summary:
Checking SCHEMA_EDIT permission when user wants to
update document's name.

Test Plan: New test

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3733
2022-12-20 10:09:49 +01:00
Jarosław Sadziński
629fcccd5a (core) Adding /webhooks endpoint
Summary:
- New /webhooks event that lists all webhooks in a document (available for owners),
- Monitoring webhooks usage and saving it in memory or Redis,
- Loosening _usubscribe API endpoint, so that the information returned from the /webhook endpoint is enough to unsubscribe,
- Owners can remove webhook without the unsubscribe key.

The endpoint lists all webhooks that are registered in a document, not just webhooks from a single table.
There are two status fields. First for the webhook, second for the last request attempt.
Webhook can have 5 statuses: 'idle', 'sending', 'retrying', 'postponed', 'error', which roughly describes what the
sendLoop is currently doing. The 'error' status describes a situation when all request attempts failed and the queue needs
to be drained, so some requests were dropped.

The last request status can only be: 'success', 'failure' or 'rejected'. Rejected means that the last batch was dropped because the
queue was too long.

Test Plan: New and updated tests

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3727
2022-12-13 22:46:40 +01:00
Jarosław Sadziński
92d4fca855 (core) Adding DELETE /api/docs/webhooks/queue endpoint to clear the queue
Summary:
Creating an API endpoint to cancel any queued webhook messages from
a document.

Test Plan: Updated

Reviewers: paulfitz, georgegevoian

Reviewed By: paulfitz, georgegevoian

Differential Revision: https://phab.getgrist.com/D3713
2022-12-01 12:23:19 +01:00
Jarosław Sadziński
59942a23b6 (core) Limiting doc remove permission to owners.
Summary:
Guest editors added to a document were able to remove it. This limits this permission
by allowing only owners of a doc to delete it.

Test Plan: Updated

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal, anaisconce

Differential Revision: https://phab.getgrist.com/D3708
2022-11-30 23:37:18 +01:00
George Gevoian
efc3ba29d7 (core) Tweak autocomplete to only suggest team members
Summary: This changes the suggestions in the User Manager autocomplete.

Test Plan: Project tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3656
2022-10-18 08:08:05 -07:00
Jarosław Sadziński
bfd7243fe2 (core) Comments
Summary:
First iteration for comments system for Grist.
- Comments are stored in a generic metatable `_grist_Cells`
- Each comment is connected to a particular cell (hence the generic name of the table)
- Access level works naturally for records stored in this table
-- User can add/read comments for cells he can see
-- User can't update/remove comments that he doesn't own, but he can delete them by removing cells (rows/columns)
-- Anonymous users can't see comments at all.
- Each comment can have replies (but replies can't have more replies)

Comments are hidden by default, they can be enabled by COMMENTS=true env variable.
Some things for follow-up
- Avatars, currently the user's profile image is not shown or retrieved from the server
- Virtual rendering for comments list in creator panel. Currently, there is a limit of 200 comments.

Test Plan: New and existing tests

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3509
2022-10-17 13:38:21 +02:00
Paul Fitzpatrick
ab3cdb62ac (core) make the support account configurable, and allow listing public sites
Summary:
This makes two small tweaks based on a user's questions about sharing
sites publicly for a self-managed installation:

 * The support user `support@getgrist.com` is made configurable with
   `GRIST_SUPPORT_EMAIL`. This came up because only the support user
   can share material with the special "everyone" user. This restriction
   was added to avoid spam.
 * Regardless of public sharing settings, for our SaaS we had
   decided not to list public sites to anonymous users. That is
   somewhat a question of taste, so a `GRIST_LIST_PUBLIC_SITES` flag
   is added to override this choice.

Public sharing isn't in a well polished state, and this diff doesn't
advance that, in fact it adds a new wrinkle :-/

Test Plan: existing tests pass; manual testing

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3663
2022-10-14 14:23:59 -04:00
Jarosław Sadziński
db0b500197 (core) Migration that makes user.ref unique and non-nullable
Summary: Making user.ref column unique and non-nullable

Test Plan: manual and existing

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian, paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3654
2022-10-12 11:13:00 +02:00
Jarosław Sadziński
9628253fd8 (core) Adding new column in users table "ref" with unique identifier.
Summary:
There is a new column in users table called ref (user reference).
It holds user's unique reference number that can be used for features
that require some kind of ownership logic (like comments).

Test Plan: Updated tests

Reviewers: georgegevoian, paulfitz

Reviewed By: georgegevoian, paulfitz

Differential Revision: https://phab.getgrist.com/D3641
2022-10-04 15:19:28 +02:00
Paul Fitzpatrick
d7b3fb972c (core) upgrade typeorm so we can support newer postgres
Summary:
upgrade typeorm version, so Grist can run against newer versions of postgres.

Dusted off some old benchmarking code to verify that important queries don't get slower. They don't appear to, unlike for some intermediate versions of typeorm I tried in the past.

Most of the changes are because `findOne` changed how it interprets its arguments, and the value it returns when nothing is found. For the return value, I stuck with limiting its impact by emulating old behavior (returning undefined rather than null) rather than propagating the change out to parts of the code unrelated to the database.

Test Plan: existing tests pass; manual testing with postgres 10 and 14

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3613
2022-09-02 15:34:21 -04:00
George Gevoian
0c5441b176 (core) Add unquarantine command to admin CLI
Summary:
Adds a CLI command to un-quarantine an active document. Also tweaks the
name of related environment variable to avoid a naming conflict.

Test Plan: Server test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3583
2022-08-15 13:04:55 -07:00
George Gevoian
fbba6b8f52 (core) Add methods for quarantining documents
Summary:
Adds a new CLI command, doc, with a subcommand that quarantines
an active document. Adds a group query param to a housekeeping
endpoint for updating the document group prior to checking if a doc
needs to be reassigned. Both methods require support user credentials.

Test Plan: Server tests. (Additional testing will be done manually on staging.)

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3570
2022-08-09 09:26:48 -07:00
Jarosław Sadziński
ee109e9186 (core) Removing NEW_DEAL flag
Summary:
Removing NEW_DEAL flag checks and exposing all new deal features as default.
Also removing Pages.ts as it was moved to grist-core.

Test Plan: Existing and updated tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3572
2022-08-09 17:29:28 +02:00
George Gevoian
c54dde3dba (core) Populate doc title, description and thumbnail in app.html
Summary:
Fills in the title and description/thumbnail (for templates) in app.html if the
page being requested is for a document.

Test Plan: Tested manually.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3544
2022-07-27 13:57:59 -07:00
George Gevoian
aeba738f7c (core) Add product for new personal plan
Summary:
Adds the new personal plan as a product that will be available
in the future. Can be enabled along with other plan-related via
an environment variable.

Test Plan: Browser tests and existing tests.

Reviewers: jarek

Reviewed By: jarek

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3533
2022-07-26 11:33:23 -07:00
Paul Fitzpatrick
dd8d2e18f5 (core) add an access token mechanism to help with attachments in custom widgets
Summary:
With this, a custom widget can render an attachment by doing:
```
const tokenInfo = await grist.docApi.getAccessToken({readOnly: true});
const img = document.getElementById('the_image');
const id = record.C[0];  // get an id of an attachment
const src = `${tokenInfo.baseUrl}/attachments/${id}/download?auth=${tokenInfo.token}`;
img.setAttribute('src', src)
```

The access token expires after a few mins, so if a user right-clicks on an image
to save it, they may get access denied unless they refresh the page. A little awkward,
but s3 pre-authorized links behave similarly and it generally isn't a deal-breaker.

Test Plan: added tests

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3488
2022-07-19 11:55:18 -04:00
Paul Fitzpatrick
ec8ab598cb (core) add a yarn run cli tool, and add a sqlite gristify option
Summary:
This adds rudimentary support for opening certain SQLite files in Grist.

If you have a file such as `landing.db` in Grist, you can convert it to Grist format by doing (either in monorepo or grist-core):
```
yarn run cli -h
yarn run cli sqlite -h
yarn run cli sqlite gristify landing.db
```

The file is now openable by Grist. To actually do so with the regular Grist server, you'll need to either import it, or convert some doc you don't care about in the `samples/` directory to be a soft link to it (and then force a reload).

This implementation is a rudimentary experiment. Here are some awkwardnesses:
 * Only tables that happen to have a column called `id`, and where the column happens to be an integer, can be opened directly with Grist as it is today. That could be generalized, but it looked more than a Gristathon's worth of work, so I instead used SQLite views.
 * Grist will handle tables that start with an uncapitalized letter a bit erratically. You can successfully add columns, for example, but removing them will cause sadness - Grist will rename the table in a confused way.
 * I didn't attempt to deal with column names with spaces etc (though views could deal with those).
 * I haven't tried to do any fancy type mapping.
 * Columns with constraints can make adding new rows impossible in Grist, since Grist requires that a row can be added with just a single cell set.

Test Plan: added small test

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3502
2022-07-14 12:00:30 -04:00
Dmitry S
51ff72c15e (core) Faster builds all around.
Summary:
Building:
- Builds no longer wait for tsc for either client, server, or test targets. All use esbuild which is very fast.
- Build still runs tsc, but only to report errors. This may be turned off with `SKIP_TSC=1` env var.
- Grist-core continues to build using tsc.
- Esbuild requires ES6 module semantics. Typescript's esModuleInterop is turned
  on, so that tsc accepts and enforces correct usage.
- Client-side code is watched and bundled by webpack as before (using esbuild-loader)

Code changes:
- Imports must now follow ES6 semantics: `import * as X from ...` produces a
  module object; to import functions or class instances, use `import X from ...`.
- Everything is now built with isolatedModules flag. Some exports were updated for it.

Packages:
- Upgraded browserify dependency, and related packages (used for the distribution-building step).
- Building the distribution now uses esbuild's minification. babel-minify is no longer used.

Test Plan: Should have no behavior changes, existing tests should pass, and docker image should build too.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: alexmojaki

Differential Revision: https://phab.getgrist.com/D3506
2022-07-04 10:42:40 -04:00
Jarosław Sadziński
aefe451bab (core) Polishing upgrade plan UI
Summary:
- Update nudge boxes content and collapsing on personal and free team site
- New confirmation after upgrading from a free team site
- Refactoring ProductUpgrade code, splitting plans / modals and nudges

Test Plan: Manual and updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3481
2022-06-29 18:28:37 +02:00
Dmitry S
dd2eadc86e (core) Speed up and upgrade build.
Summary:
- Upgrades to build-related packages:
  - Upgrade typescript, related libraries and typings.
  - Upgrade webpack, eslint; add tsc-watch, node-dev, eslint_d.

- Build organization changes:
  - Build webpack from original typescript, transpiling only; with errors still
    reported by a background tsc watching process.

- Typescript-related changes:
  - Reduce imports of AWS dependencies (very noticeable speedup)
  - Avoid auto-loading global @types
  - Client code is now built with isolatedModules flag (for safe transpilation)
  - Use allowJs to avoid copying JS files manually.

- Linting changes
  - Enhance Arcanist ESLintLinter to run before/after commands, and set up to use eslint_d
  - Update eslint config, and include .eslintignore to avoid linting generated files.
  - Include a bunch of eslint-prompted and eslint-generated fixes
  - Add no-unused-expression rule to eslint, and fix a few warnings about it

- Other items:
  - Refactor cssInput to avoid circular dependency
  - Remove a bit of unused code, libraries, dependencies

Test Plan: No behavior changes, all existing tests pass. There are 30 tests fewer reported because `test_gpath.py` was removed (it's been unused for years)

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3498
2022-06-27 16:10:10 -04:00
Jarosław Sadziński
d92a761f6e (core) Product update popups and hosted stripe integration
Summary:
- Showing nudge to individual users to sign up for free team plan.
- Implementing billing page to upgrade from free team to pro.
- New modal with upgrade options and free team site signup.
- Integrating Stripe-hosted UI for checkout and plan management.

Test Plan: updated tests

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: paulfitz

Differential Revision: https://phab.getgrist.com/D3456
2022-06-08 21:10:49 +02:00
Paul Fitzpatrick
ff77ecc6c6 (core) deal with SQLITE_BUSY: database is locked during tests
Summary:
During testing, and in some dev configurations, we make connections to
the Grist home database from multiple processes, and store it in Sqlite.
This is a situation prone to file locks in Sqlite. We set a busy_timeout
parameter to help, but transactions may still fail immediately. This
diff adds retries to our patch of TypeORM for Sqlite transactions
(which without the patch are already flakey for other reasons
https://github.com/typeorm/typeorm/issues/1884#issuecomment-380767213).

Without this change, I can tickly SQLITE_BUSY problems fairly
reliably with the DocApi tests. With this change, I can't so far.

This change should not affect deployments of our SaaS (which use
Postgres for home db) or grist-core/grist-ee (which access home
db from a single process in default configuration).

Test Plan: repeated running of DocApi tests

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3466
2022-06-03 15:51:59 -04:00
Paul Fitzpatrick
dcaa2b4f29 (core) move DocUsageBanner to ext
Summary:
grist-ee build was failing since it didn't have a
DocUsageBanner implementation available. Made the implementation
added to monorepo available, since it will be useful to improve
the activation banner.

Test Plan: manaul

Reviewers: georgegevoian

Reviewed By: georgegevoian

Subscribers: georgegevoian

Differential Revision: https://phab.getgrist.com/D3452
2022-05-27 22:19:17 -04:00
Paul Fitzpatrick
b9a4b2b58f (core) add missing tsconfig file that affects IDEs
Summary:
I missed committing a file that is important for editing files comfortably in the ext directory in an IDE. This diff:

 * Adds tsconfig-base-ext.json - that was the only intended change
 * Unrelated: Forces all creation of connections to the home db through a new `getOrCreateConnection` method which changes the `busy_timeout` if using Sqlite. This was an attempt to fix random "database is locked" test failures. I believe multiple connections to the home db as an sqlite file do not happen in self-hosted Grist (where there is a single node process) or in our SaaS (where the database is in postgres). It does affect Grist started using `devServerMain.ts` (where multiple processes accessing same database are started) or various test configurations when extra database connections are opened.
 * Unrelated: I added a `busy_timeout` for session storage, when it uses Sqlite. Again, I don't believe this affects self-hosted Grist or our SaaS.
 * Tweaked a `BillingDiscount` test that looked perhaps vulnerable to a stripe request stalling.

I can't be sure my tweaks actually help, since I didn't succeed in replicating the failures. Update: looks like the "locked" error can still happen :(

Test Plan: manual

Reviewers: jarek

Reviewed By: jarek

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3450
2022-05-27 13:48:58 -04:00
George Gevoian
74ec9358da (core) Show usage banners in doc menu of free team sites
Summary:
Also fixes a minor CSS regression in UserManager where the
link to add a team member wasn't shown on a separate row.

Test Plan: Browser tests.

Reviewers: jarek

Reviewed By: jarek

Differential Revision: https://phab.getgrist.com/D3444
2022-05-26 15:01:35 -07:00
George Gevoian
a6063f570a (core) Polish Access Details
Summary:
Instead of showing a blank dialog for users whose access
is limited (e.g. public members), we now show the user's
role and a mention of whether their access is public.

Test Plan: Browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3431
2022-05-19 09:45:26 -07:00
Jarosław Sadziński
0ab9e4a6a0 (core) Adding GristConnect login system
Summary:
New login system to allow simple SSO flow that is based on Discourse description that is available at:
https://meta.discourse.org/t/discourseconnect-official-single-sign-on-for-discourse-sso/13045

Test Plan: New core test.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3418
2022-05-18 20:28:25 +02:00
Paul Fitzpatrick
e4d47a2f3c (core) add minimal support for activation keys
Summary: For grist-ee, expect an activation key in environment variable `GRIST_ACTIVATION` or in a file pointed to by `GRIST_ACTIVATION_FILE`. In absence of key, start a 30-day trial, during which a banner is shown. Once trial expires, installation goes into document-read-only mode.

Test Plan: added a test

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3426
2022-05-16 22:46:23 -04:00