Summary:
If cancel was clicked while a transform section was still being
generated in the Importer, an error was thrown. This refactors
the cancelImportFiles API action to take in the file upload id
in place of the entire DataSourceTransformed parameter, which
contains other values that are irrelevant to canceling. One of those
values, the transform section id, was causing the error to be thrown
since it was momentarily null.
Test Plan: Tested manually.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3317
Summary:
- Removed string parsing and some type guessing code from parse_data.py. That logic is now implicitly done by ValueGuesser by leaving the initial column type as Any. parse_data.py mostly comes into play when importing files (e.g. Excel) containing values that already have types, i.e. numbers and dates.
- 0s and 1s are treated as numbers instead of booleans to keep imports lossless.
- Removed dateguess.py and test_dateguess.py.
- Changed what `guessDateFormat` does when multiple date formats work equally well for the given data, in order to be consistent with the old dateguess.py.
- Columns containing numbers are now always imported as Numeric, never Int.
- Removed `NullIfEmptyParser` because it was interfering with the new system. Its purpose was to avoid pointlessly changing a column from Any to Text when no actual data was inserted. A different solution to that problem was already added to `_ensure_column_accepts_data` in the data engine in a recent related diff.
Test Plan:
- Added 2 `nbrowser/Importer2` tests.
- Updated various existing tests.
- Extended testing of `guessDateFormat`. Added `guessDateFormats` to show how ambiguous dates are handled internally.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3302
Summary:
When two widgets are linked by same-record linking, and the source of that link is also filter-linked, then it will pick up default values from its own filter-link source, but the same-record-link target didn't. This fixes that so that default values are filled in intuitively.
Moved the logic of linkingState, linkingFilter, and getDefaultColValues from BaseView.js to LinkingState.ts and ViewSectionRec.ts. In particular getDefaultColValues is now a property of LinkingState which may be copied from the source view section for a same-record link.
Note that `ViewSectionRec.linkingFilter` no longer uses `computerBuilder` and thus doesn't ignore dependencies inside LinkingState any more. I couldn't figure out how to make `linkingFilter` a `pureComputed` (otherwise I get recursion errors) that ignores dependencies. In any case, it's now important to have a dependency on `srcSection.linkingState()` for `getDefaultColValues` to work correctly, so I think this is for the best.
Test Plan: Added a new nbrowser test and fixture.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3238
Summary:
- Adds a function `parseUserAction` for parsing strings in UserActions to `ValueParser.ts`
- Adds a boolean option `parseStrings` to use `parseUserAction` in `ActiveDoc.applyUserActions`, off by default.
- Uses `parseStrings` by default in DocApi (set `?noparse=true` in a request to disable) when adding/updating records through the `/data` or `/records` endpoints or in general with the `/apply` endpoint.
- Uses `parseStrings` for various actions in `ActiveDocImport`. Since most types are parsed in Python before these actions are constructed, this only affects references, which still look like errors in the import preview. Importing references can also easily still run into more complicated problems discussed in https://grist.slack.com/archives/C0234CPPXPA/p1639514844028200
Test Plan:
- Added tests to DocApi to compare behaviour with and without string parsing.
- Added a new browser test, fixture doc, and fixture CSV to test importing a file containing references.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3183
Summary:
Adding new destination "Skip" for multiple table imports.
Selecting this destination skips the import and makes the preview grayed out.
Test Plan: New Tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3181
Summary:
Updates the preview table in Importer to show a diff of changes
when importing into an existing table and updating existing records.
Test Plan: Browser tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3060
Summary:
The users shown by the "View As" button are now drawn from more sources:
* There are users the document is shared with. This has been rationalized, the behavior was somewhat erratic. If the user is not an owner of the document, the only user of this kind that will be listed is themselves.
* There are users mentioned in any user attribute table keyed by Email. If name and access columns are present, those are respected, otherwise name is taken from email and access is set to "editors".
* There are example users provided if there are not many other users available.
Test Plan: added and extended tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3045
Summary:
Finishing imports now occurs in Node instead of the
data engine, which makes it possible to import into
on-demand tables. Merging code was also refactored
and now uses a SQL query to diff source and destination
tables in order to determine what to update or add.
Also fixes a bug where incremental imports involving
Excel files with multiple sheets would fail due to the UI
not serializing merge options correctly.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3046
Summary:
With this diff, when a user opens a Grist document in a browser, they will be able to view its contents without waiting for the data engine to start up. Once the data engine starts, it will run a calculation and send any updates made. Changes to the document will be blocked until the engine is started and the initial calculation is complete.
The increase in responsiveness is useful in its own right, and also reduces the impact of an extra startup time in a candidate next-generation sandbox.
A small unrelated fix is included for `core/package.json`, to catch up with a recent change to `package.json`.
A small `./build schema` convenience is added to just rebuild the typescript schema file.
Test Plan: added test; existing tests pass - small fixes needed in some cases because of new timing
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3036
Summary:
New environmental variable GOOGLE_DRIVE_SCOPE that modifies the scope
requested for Google Drive integration.
For prod it has value https://www.googleapis.com/auth/drive.file which leaves
current behavior (Grist is allowed only to access public files and for private
files - it fallbacks to Picker).
For staging it has value https://www.googleapis.com/auth/drive.readonly which
allows Grist to access all private files, and fallbacks to Picker only when the file is
neither public nor private).
Default value is https://www.googleapis.com/auth/drive.file
Test Plan: manual and existing tests
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3038
Summary: Constructs a ClientQuery in a similar way to LinkingState to handle lists.
Test Plan: Extended SelectBySummary and SelectByRefList tests.
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3030
Summary:
This unsets the `direct` flag for actions emitted when summary tables are updated. That means those actions will be ignored for access control purposes. So if a user has the right to change a source table, the resulting changes to the summary won't result in the overall action bundle being forbidden.
I don't think I've actually seen the use case that inspired this issue being filed. I could imagine perhaps a user forbidden from creating rows globally making permitted updates that could add rows in a summary (and it being desirable to allow that).
Test Plan: added tests
Reviewers: jarek
Reviewed By: jarek
Subscribers: dsagal, alexmojaki, jarek
Differential Revision: https://phab.getgrist.com/D3022
Summary:
The import dialog now has an option to 'Update existing records',
which when checked will allow for selection of 1 or more fields
to match source and destination tables on.
If all fields match, then the matched record in the
destination table will be merged with the incoming record
from the source table. This means the incoming values will
replace the destination table values, unless the incoming
values are blank.
Additional merge strategies are implemented in the data
engine, but the import dialog only uses one of the
strategies currently. The others can be exposed in the UI
in the future, and tweak the behavior of how source
and destination values should be merged in different contexts,
such as when blank values exist.
Test Plan: Python and browser tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3020
Summary:
Prefix keys of `LinkingState.filterColValues` with `_contains:` when the source column is a ChoiceList or ReferenceList.
This is parsed out to make a boolean `isContainsFilter` which is kept in each value of `QueryRefs.filterTuples` (previously `filterPairs`).
Then when converting back in `convertQueryFromRefs` we construct `Query.contains: {[colId: string]: boolean}`.
Finally `getFilterFunc` uses `Query.contains` to decide what kind of filtering to do.
This is not pretty, but the existing code is already very complex and it was hard to find something that wouldn't require touching loads of code just to make things compile.
Test Plan: Added a new nbrowser test and fixture, tests that selecting a source table by summary tables grouped by a choicelist column, non-list column, and both all filter the correct data.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2940
Summary:
API signature for autocomplete updated to add column ID, which is
necessary for exposing correct types for 'value'.
Test Plan: Unit tests.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: jarek, alexmojaki
Differential Revision: https://phab.getgrist.com/D2896
Summary: Removed test/aws/, most of app/server/lib/, 3 dirs in app/lambda/, corresponding tests, and more!
Test Plan: a lot of this is quite the opposite...
Reviewers: dsagal, paulfitz
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2894
Summary:
This is a somewhat experimental change, that will implement permitted parts of an undo if not all parts are permitted. This is in preparation for trigger columns, where it may become common for a change in a record resulting in a change to an automatic change to another that the user cannot edit directly. How to undo such an action is somewhat unclear. One option is to undo the permitted parts, and then the triggers can rerun.
The general case is a bit of a can of worms, and feels adjacent to merging/rebasing etc.
Oh: it would probably be important in general to communicate to the user that an undo was partial, but this diff doesn't do that. It would need some new plumbing.
Test Plan: added test
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2839
Summary:
This particular combination of features is not built out - data will be
censored but changes to data will not. So the user will now get an error
if they try to do it. Existing rules of this kind will continue to
operate as before, and can be set via the api.
Test Plan: added test
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2751
Summary:
The goal is that those who can edit ACL rules can create or change rules for
any resource, even if the rules block their own ability to see the resource.
Test Plan: Added a browser test, and a server test for who can call the new method.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2703
Summary:
- Fix error-handling in bundleActions(), and wait for the full bundle to complete.
(The omissions here were making it impossibly to react to errors from inside bundleActions())
- Catch problematic rules early enough to undo them, by trying out ruleCollection.update()
on updated rules before the updates are applied.
- Added checkAclFormula() call to DocComm that checks parsing and compiling
formula, and reports errors.
- In UI, prevent saving if any aclFormulas are invalid, or while waiting for the to get checked.
- Also fixed some lint errors
Test Plan: Added a test case of error reporting in ACL formulas.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2689
Summary:
- Implemented selecting an org in some cases when using Save-Copy dialog.
- Unified previous 'Save Copy' menu into an enhanced "Share" menu.
- Renamed ExportMenu to ShareMenu, collect related code into it, and design the share button.
- Introduced trunkAccess property for forks, to know whether "Replace Original" is available.
- Simplified handling of fork() result, now that all code has been upgraded.
- Replaced 'Copy as Template' menu items with a checkbox in the Save-Copy dialog
- Removed copy links for examples in the DocMenu (to simplify, since not part of updated design)
- Updated the UI of the copying dialog.
Test Plan: Updated affected tests, added new test cases for copying when other orgs are a choice or not.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2561
Summary: This moves enough server material into core to run a home server. The data engine is not yet incorporated (though in manual testing it works when ported).
Test Plan: existing tests pass
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2552
