* add support for conversational state to assistance endpoint
This refactors the assistance code somewhat, to allow carrying
along some conversational state. It extends the OpenAI-flavored
assistant to make use of that state to have a conversation.
The front-end is tweaked a little bit to allow for replies that
don't have any code in them (though I didn't get into formatting
such replies nicely).
Currently tested primarily through the runCompletion script,
which has been extended a bit to allow testing simulated
conversations (where an error is pasted in follow-up, or
an expected-vs-actual comparison).
Co-authored-by: George Gevoian <85144792+georgegevoian@users.noreply.github.com>
Summary:
- Webhooks form Triggers.ts should now use proxy if it's configured
- Proxy handling code separated to ProxyAgent.ts
- Tests for ProxyAgent
- Integration/API Tests for using Proxy in webhooks
- a bit of refactor - proxy test uses mostly the same codebase as DocApi.ts, but because last one if over 4000 lines long, I've put it into separated file, and extract some common parts (there is some duplicates tho)
- some cleanup in files that I've touched
Test Plan:
Manual test to check if proxy is used on the staging env
Automatic test checking if (fake) proxy was called
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3860
Summary:
- when grist table is exported, currency is check and introduced in cell format in the form of "[currency symbol] [value]" (for example: zł 10000, $ 5000) . It's not what some cultures should display currences, but it's close enought
- when no symbol is defined for the currency, currency 3 letters code is used instead
- when currency is unknown, we are falling back to "$"
Test Plan: - nbrowser test scenario added for that purpose, please check Currences.xlsx to see output format exported.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3886
Summary:
The URL /create-team-site on marketing site is unused, and no longer pointing to
anything functional.
The "efcr" product flavor has been defunct for a long time. Remove
references to it.
Test Plan: No tests should be affected
Reviewers: JakubSerafin
Reviewed By: JakubSerafin
Subscribers: JakubSerafin
Differential Revision: https://phab.getgrist.com/D3890
Summary:
When clicking the logo in the top-left corner, or finishing a tutorial, we
now direct users to the site they last visited, if possible. If unknown, a
new redirect endpoint, /welcome/home, is used instead, which directs users
to a sensible location based on the number of sites they have.
Test Plan: Browser tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3878
Summary:
- When importing into a Ref column, use lookupOne() formula for correct previews.
- When selecting columns to import into a Ref column, now a Numeric column like
'Order' will produce two options: "Order" and "Order (as row ID)".
- Fixes exports to correct the formatting of visible columns. This addresses multiple bugs:
1. Formatting wasn't used, e.g. a Ref showing a custom-formatted date was still presented as YYYY-MM-DD in CSVs.
2. Ref showing a Numeric column was formatted as if a row ID (e.g. `Table1[1.5]`), which is very wrong.
- If importing into a table that doesn't have a primary view, don't switch page after import.
Refactorings:
- Generalize GenImporterView to be usable in more cases; removed near-duplicated logic from node side
- Some other refactoring in importing code.
- Fix field/column option selection in ValueParser
- Add NUM() helper to turn integer-valued floats into ints, useful for "as row ID" lookups.
Test Plan: Added test cases for imports into reference columns, updated Exports test fixtures.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3875
Summary:
The conditions in the map/filter of attachments was faulty, causing
blank attachment types to slip through.
Test Plan: Tested manually.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3865
Summary:
Owners weren't able to access snapshots if access rules
that denied access to non-owners existed. The backend
was lowering snapshot document access to "viewers" as
part of implementing read-only behavior; this is now done
in the client, with document access for snapshots now
accurately reflecting the user's trunk access.
Additionally, sandboxes are no longer created for snapshots,
and background intervals aren't started for snapshots.
Test Plan: Browser test.
Reviewers: jarek, paulfitz
Reviewed By: jarek, paulfitz
Differential Revision: https://phab.getgrist.com/D3849
Summary:
Ports more test/browser tests from *.test.js (run using an old selenium setup) to *.ntest.js (run using newer setup).
Weird test failures happened due to a change in timing. Eventually tracked in down to billing changes in one test suite resulting in reloads in another test suite, since it turns out redis pub/sub channels are not scoped to the redis database specified in REDIS_URL, but are global:
https://redis.io/docs/manual/pubsub/#database--scoping.
Test Plan: Ported tests should run and pass
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3844
This closes a file left open during importing, not by the import itself, but by a SQLite integrity check. This was causing imports to fail on Windows (see https://github.com/gristlabs/grist-electron/issues/3)
Expose the log.add method, used in old Electron code. Without this change, using the Electron build of Grist fails if is configured for debug log output.
Summary:
Adds a new endpoint to update webhook.
Perform some refactoring to allow code reuse from endpoint allowing to _subscribe and _unsubscribe webhooks.
One aspect of webhook is that url are stored in the home db while the rest of the fields (tableRef, isReadyColRef, ...) are stored in sqlite. So care must be taken when updating fields, to properly rollback if anything should fail.
Follow up diff will bring UI to edit webhook list
Test Plan: Updated doc api server tests
Reviewers: jarek
Reviewed By: jarek
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3821
Summary:
The feature is behind a flag GRIST_FORMULA_ASSISTANT (must be "true"). But can be enabled in the
developer console by invoking GRIST_FORMULA_ASSISTANT.set(true).
Keys can be overriden in the document settings page.
Test Plan: For now just a stub test that checks if this feature is disabled by default.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3815
Summary:
This adds a nuanced redirecting endpoint. For example, on
docs.getgrist.com it does:
1) If logged in and no team site -> https://docs.getgrist.com/
2) If logged in and has team sites -> https://docs.getgrist.com/welcome/teams
3) If logged out but has a cookie -> /login, then 1 or 2
4) If entirely unknown -> /signup
Test Plan: added a test; tested behavior through logins manually
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3828
Summary:
Documents can now be flagged as tutorials, which causes them to display
Markdown-formatted slides from a special GristDocTutorial table. Tutorial
documents are forked on open, and remember the last slide a user was on.
They can be restarted too, which prepares a new fork of the tutorial.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3813
* place a new private ActiveDoc method in the expected order
This was tickling a lint failure on grist-core.
* reset an English translation that interferes with test currently
Summary:
A recent change perturbed some error handling when an ActiveDoc
is shutting down. It is important that errors get thrown when
attempting to replace a non-existent document. My bad in review
for not catching.
Test Plan: Snapshot test passes again
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: georgegevoian, cyprien
Differential Revision: https://phab.getgrist.com/D3824
Summary:
Porting script that run an evaluation against our formula dataset.
To test you need an openai key (see here: https://platform.openai.com/)
or hugging face (it should work as well), then checkout the branch and run
`OPENAI_API_KEY=<my_openai_api_key> node core/test/formula-dataset/runCompletion.js`
Test Plan:
Needs manually testing: so far there is no plan to make it part of CI.
The current score is somewhere around 34 successful prompts over a total of 47.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3816
Summary:
Warnings about deprecated shortcuts are no longer needed.
As a side effect it fixes a bug that caused those warnings to not persist its
state on pages with charts.
Test Plan: Removed
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3820
Summary:
- For python2, skip some tests of renaming which produce different results
because of an un-upgradable astroid version.
- Fix test affected by pyCall() having changed to async; avoid hanging timeout
callback in case of error.
Test Plan: All test cases should now pass (with 4 getting skipped)
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3819
This adds a new `GRIST_SANDBOX_FLAVOR=pyodide` option where the
version of Python used for the data engine is wasm, and so can
be run by node like the rest of the back end. It still runs as
a separate process.
There are a few small version changes made to packages to avoid
various awkwardnesses present in the current versions. All existing
tests pass.
This is very experimental. To use, you'll need something with
a bash shell and make. First do:
```
cd sandbox/pyodide
make setup # README.md and Makefile have details
cd ..
```
Then running Grist as:
```
GRIST_SANDBOX_FLAVOR=pyodide yarn start
```
should work. Adding a formula with content:
```
import sys; return sys.version
```
should return a different Python version than other sandboxes.
The motivation for this work is to have a form of sandboxing
that will work on Windows for Grist Electron (for Linux we have
gvisor/runsc, for Mac we have sandbox-exec, but I haven't found
anything comparable for Windows).
It also brings a back-end-free version of Grist a bit closer, for
use-cases where that would make sense - such as serving a report
(in the form of a Grist document) on a static site.
Summary:
This removes ormconfig.js from the saas build since it is no longer
needed (and has always been a pain point).
This expands some sendgrid logging to help figure out a problem.
Test Plan: existing tests should pass
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3808
Summary:
Adds information about forks to the home db. This will be used
later by the UI to list forks of documents.
Test Plan: Browser and server tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3772
* Replace `ormconfig.js` with a newer mechanism of configuring
TypeORM that can be included in the source code properly.
The path to `ormconfig.js` has always been awkward to handle,
and eliminating the file makes building different Grist setups
a bit simpler.
* Remove `electron` package. It is barely used, just for some old
remnants of an older attempt at electron packaging. It was used
for two types, which I left at `any` for now. More code pruning is
no doubt possible here, but I'd rather do it when Electron packaging
has solidified.
* Add a hook for replacing the login system, and for adding some
extra middleware the login system may need.
* Add support for some more possible locations of Python, which
arise when a standalone version of it is included in the Electron
package. This isn't very general purpose, just configurations
that I found useful.
* Support using grist-core within a yarn workspace - the only tweak
needed was webpack related.
* Allow an external ID to be optionally associated with documents.
Summary: This is a backend part for the formula AI.
Test Plan: New tests
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: cyprien
Differential Revision: https://phab.getgrist.com/D3786
Summary:
New language selector on the Account page for logged-in users.
New icon for switching language for an anonymous user.
For anonymous users, language is stored in a cookie grist_user_locale.
Language is stored in user settings for authenticated users and takes
precedence over what is stored in the cookie.
Test Plan: New tests
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3766
Summary:
- Use newer flag in .npmrc to avoid warnings
- Fix check in WidgetRepository, useful for development but was broken
- Fix macSandboxExec for Macs that require libRosettaRuntime
- Make sure row count in Raw Data listing is visible when it takes more space
Test Plan: Tested manually
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3759
Summary:
Implement a checkbox that grants owners full access to tables by default, when creating new table/column rules.
* Checkbox appears above default rules.
* When set, a rule giving owners full access will be inserted in any new rule set started for tables or columns.
* The checkbox can be expanded to allow customization of the rules.
https://gristlabs.getgrist.com/doc/check-ins/p/3#a1.s7.r2251.c19
Test Plan: added tests
Reviewers: jarek
Reviewed By: jarek
Subscribers: anaisconce
Differential Revision: https://phab.getgrist.com/D3756
Summary:
Diff removes view-as pill in the document breadcrumbs and add new view-as banner.
Note: Banners are still missing mechanism to handle several banners. As of now both doc-usage and view-as banners could show up at the same time.
Test Plan: Refactored existing test.
Reviewers: jarek
Reviewed By: jarek
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3732
Summary:
for users who don't automatically have deep rights
to the document, provide them with attachment metadata only
for rows they have access to. This is a little tricky to
do efficiently. We provide attachment metadata when an
individual table is fetched, rather than on initial document
load, so we don't block that load on a full document scan.
We provide attachment metadata to a client when we see that
we are shipping rows mentioning particular attachments,
without making any effort to keep track of the metadata they
already have.
Test Plan: updated tests
Reviewers: dsagal, jarek
Reviewed By: dsagal, jarek
Differential Revision: https://phab.getgrist.com/D3722
Summary:
This is a first pass at snapshot support using the MinIO client, suitable
for use against a MinIO server or other S3-compatible storage (including
the original AWS S3).
In Grist Labs monorepo tests, it is run against AWS S3. It can be manually
configured to run again a MinIO server, and these tests pass. There are no
core tests just yet.
Next step would be to move external storage tests to core, and configure
workflow to run tests against a transient MinIO server.
Test Plan: applied same tests as for Azure and S3 (via AWS client)
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3729
Summary:
When a column is transformed, it creates two helper columns whose values are always
broadcasted to all clients. Now when there are some ACL rules, we are going to prune
those columns from messages sent to other connected clients.
Test Plan: Added new tests
Reviewers: dsagal, paulfitz
Reviewed By: dsagal, paulfitz
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3728
Summary:
When initially added in the User Manager, the support user
(e.g. support@getgrist.com) was misleadingly being annotated as
a free collaborator. This fixes the annotation to be "Grist support"
instead.
Test Plan: Browser test.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3730
Summary:
- New /webhooks event that lists all webhooks in a document (available for owners),
- Monitoring webhooks usage and saving it in memory or Redis,
- Loosening _usubscribe API endpoint, so that the information returned from the /webhook endpoint is enough to unsubscribe,
- Owners can remove webhook without the unsubscribe key.
The endpoint lists all webhooks that are registered in a document, not just webhooks from a single table.
There are two status fields. First for the webhook, second for the last request attempt.
Webhook can have 5 statuses: 'idle', 'sending', 'retrying', 'postponed', 'error', which roughly describes what the
sendLoop is currently doing. The 'error' status describes a situation when all request attempts failed and the queue needs
to be drained, so some requests were dropped.
The last request status can only be: 'success', 'failure' or 'rejected'. Rejected means that the last batch was dropped because the
queue was too long.
Test Plan: New and updated tests
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3727
Summary:
Adds a new UI for writing access rule memos.
Migrates old memos (written as Python comments) to the new UI.
Test Plan: Browser and migration tests.
Reviewers: jarek, dsagal
Reviewed By: jarek
Subscribers: dsagal, paulfitz
Differential Revision: https://phab.getgrist.com/D3726
Summary:
When access rules refer to tables and/or columns that no longer exist, offer convenient buttons to remove these rules.
It could alternatively be useful to generate errors when deleting tables or columns that are mentioned in access rules, and refuse to do so unless the access rules are updated first.
Test Plan: added and updated tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D3718
