Summary:
Use openpyxl instead of messytables (which used xlrd internally) in import_xls.py.
Skip empty rows since excel files can easily contain huge numbers of them.
Drop support for xls files (which openpyxl doesn't support) in favour of the newer xlsx format.
Fix some details relating to python virtualenvs and dependencies, as Jenkins was failing to find new Python dependencies.
Test Plan: Mostly relying on existing tests. Updated various tests which referred to xls files instead of xlsx. Added a Python test for skipping empty rows.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3406
Summary:
A new way for renaming tables.
- There is a new popup to rename section (where you can also rename the table)
- Renaming/Deleting page doesn't modify/delete the table.
- Renaming table can rename a page if the names match (and the page contains a section with that table).
- User can rename table in Raw Data UI in two ways - either on the listing or by using the section name popup
- As before, there is no way to change tableId - it is derived from a table name.
- When the section name is empty the table name is shown instead.
- White space for section name is allowed (to discuss) - so the user can just paste ' '.
- Empty name for a page is not allowed (but white space is).
- Some bugs related to deleting tables with attached summary tables (and with undoing this operation) were fixed (but not all of them yet).
Test Plan: Updated tests.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: georgegevoian
Differential Revision: https://phab.getgrist.com/D3360
Summary:
This also enables the new Usage section for all sites. Currently,
it shows metrics for document row count, but only if the user
has full document read access. Otherwise, a message about
insufficient access is shown.
Test Plan: Browser tests.
Reviewers: jarek
Reviewed By: jarek
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3377
Summary:
Adds a `data-grist-col-ref` attribute to the copied HTML, then uses that when pasting to look up the source column and retrieve info about it. Copies the info into the target column if:
- The document is the same (the docId hash matches)
- The source column still exists and has the same type as when copied
- The source type isn't Text, because in that case it's nice if type guessing still happens
- The target column is empty, meaning it has type Any (we check earlier that it's not a formula column)
The info copied is the type, widgetOptions, and reference column settings (visible and display columns) but not conditional formatting.
The changes are mostly in a function `parsePasteForView` which is based on `BaseView._parsePasteForView` but ported to TypeScript in a new file `BaseView2.ts`.
Added a useraction `MaybeCopyDisplayFormula` exposing an existing Python function `maybe_copy_display_formula` because the target column needs a slightly different display formula.
Test Plan: Added a new nbrowser test file and fixture doc.
Reviewers: cyprien
Reviewed By: cyprien
Subscribers: jarek, dsagal
Differential Revision: https://phab.getgrist.com/D3344
Summary:
- When opening HelpScout beacon to an article ("answers"), avoid a 'navigate'
call to let the beacon show the previously open article.
- Work around a bug with reloading a page with a beacon article open: HelpScout
renders the last state without triggering usual events.
- Report errors to server when beacon fails to load.
- reportWarning() method now reports the message to the server.
Test Plan: Added a test case
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3345
Summary:
Adding conditional formatting rules feature.
Each column can have multiple styling rules which are applied in order
when evaluated to a truthy value.
- The creator panel has a new section: Cell Style
- New user action AddEmptyRule for adding an empty rule
- New columns in _grist_Table_columns and fields
A new color picker will be introduced in a follow-up diff (as it is also
used in choice/choice list/filters).
Design document:
https://grist.quip.com/FVzfAgoO5xOF/Conditional-Formatting-Implementation-Design
Test Plan: new tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3282
Summary:
Moves some auth-related UI components, like MFAConfig, out
of core, and adds a new ChangePasswordDialog component for
allowing direct password changes, replacing the old reset password
link to hosted Cognito.
Updates all MFA endpoints to use SRP for authentication.
Also refactors MFAConfig into smaller files, and polishes up some parts
of the UI to be more consistent with the login pages.
Test Plan: New server and deployment tests. Updated existing tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3311
Summary:
Creating new UI for raw data views based on design.
- Renaming left for follow up diff
- Link in the menu is hidden for now
To access raw UI, use /p/data URL.
Test Plan: new tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3306
Summary:
Enables configuration of multi-factor authentication from the
account page (for users who sign in with email/password), and adds
SMS as an authentication method.
Test Plan: Project, browser and server tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3215
Summary:
Improving experience when editing group-by column:
- Disable column rename
- Allow changing most widget properties:
- Color/Background
- Number format
- Date/DateTime format (but not the timezone)
- All toggle options (for toggle column)
- Remove Edit button on Choice Edit
- Changing the underlying column should reset all those options back to the original column.
Test Plan: nbrowser
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: alexmojaki
Differential Revision: https://phab.getgrist.com/D3216
Summary:
Previously, ref/reflist columns were formatted entirely based on their visible column, since they received values from the visible or display columns rather than the actual row IDs. This creates `ReferenceFormatter` and `ReferenceListFormatter` which still delegate most of the formatting work to a visible column formatter but fix a few issues:
- ReferenceList columns now actually use the options (e.g. date format) of the visible column to format their elements. Previously they were formatted generically because the visible column formatter wasn't expecting a list.
- Invalid references aren't formatted with an `#Invalid Ref` prefix.
- When the ref column displays the Row ID, it doesn't have a visible or display column. Previously this led to the references being formatted as just numbers in most cases, with special code in the widget to display them like `Table1[2]`. Now they are consistently formatted in that style throughout.
Test Plan: Updated existing tests.
Reviewers: jarek
Reviewed By: jarek
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3212
Summary: When an editor is activated by typing, the active view should be scrolled to the active record.
Test Plan: new tests
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3196
Summary:
The culprit was `series = uniqXValues(series);` because it creates new
series objects when they are used as keys to access error bars info
(`errorBars.get(line)`).
Fixed by making uniqXValues mutating series instead.
Test Plan: Adds a case to test error bars with bar charts.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3198
Summary:
Two bugs fixed:
1. On search, when the first result is in the active record, GridView wasn't scrolling to the active record.
2. When an active record was not visible, GridView wasn't scrolling to the active record when the column index was changed.
The problem was that the scrolling behavior was based only on rowIndex which isn't changed (and doesn't notify subscribers) when a column index changes or when the search highlights a cell.
This diff makes the computed depend also on the fieldIndex, and is introducing a new method that can scroll to the active record on demand (which is used by the search).
Test Plan: Updated tests.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3191
Summary:
Some things (like rendering cells) use the `visibleCol` for `createFormatter`, while other things (like `CopySelection`) used the `displayCol`. For references, the display column has type Any and doesn't know about the original formatting. This resulted in formatting being lost when copying from reference columns even though formatting was preserved when copying from the original (visible) column which looked identical. This diff fixes this and ensures that `createFormatter` is always used with the `visibleCol`. This was agreed on in https://grist.slack.com/archives/C0234CPPXPA/p1639571321043000
Additionally:
- Replaces the functions `createVisibleColFormatter` computed properties `visibleColFormatter` as suggested by a `TODO`.
- Extracts common code from `createVisibleColFormatter` in `ColumnRec` and `ViewFieldRec`
Test Plan: Fixed a test in CopyPaste which displayed the previous inconsistent behaviour.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D3189
Summary:
- Fix base href in HelpScout beacon when showing articles (in particular for Firefox)
- Show the 'Answers' tab normally except when reporting an error.
- Combine the "Give Feedback" and "Help Center" buttons into one that normally
opens the beacon (with a link to Help Center and to Community Forum), and a
smaller one that opens the Help Center site in a new tab.
- Update HELP_SCOUT_* env vars to use _V2 suffix, to allow them to coexist with
code using the previous beacon.
Test Plan: Updated the browser test to check the new behavior.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3170
Summary:
Following discussion in https://phab.getgrist.com/D3164:
- Change createParser to accept docData and one or two metadata row IDs and let it extract the metadata, so it's more easily usable in the server.
- Change ViewFieldRec.valueParser observable to a function createValueParser.
Test Plan: Existing tests.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3172
Summary: Refactoring in preparation for parsing strings from the API. The plan is that the API code will only need to do a server-side version of the code in ViewFieldRec.valueParser (minus ReferenceUtils) which is quite minimal.
Test Plan: Nothing extra here, I don't think it's needed. This stuff will get tested more in a future diff which changes the API.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3164
Summary: Add doc-id attribute to copied HTML columns next to column type. Only use the raw value (rather than the display value) when the parsed doc-id from pasted HTML matches the current document ID, similar to ensuring that the type matches. This only applies to references and reflists.
Test Plan: Extended CopyPaste.ts
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D3154
Summary:
- Donut charts is same as pie chart with few extra options to control size of the hole and to show/hide a big total in it.
- Add a new option type to tune a numeric options using a slider/spinner/keyboard.
- Add a new option type to tune a numeric options using a slider/keyboard
- Add a new .propWithDefault method to ObjObservable to allows to set a default value when options is undefined.
- mocha-webdriver's findContent does not work to find content in svg elements. So had to tweak original function into a sister function using .textContent instead.
Test Plan: Adds new tests
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: anaisconce, dsagal
Differential Revision: https://phab.getgrist.com/D3107
Summary:
Added a new object type code `l` (for lookup) which can be used in user actions as a temporary cell value in ref[list] columns and is immediately converted to a row ID in the data engine. The value contains the original raw string (to be used as alt text), the column ID to lookup (typically the visible column) and one or more values to lookup.
For reflists, valueParser now tries parsing the string first as JSON, then as a CSV row, and applies the visible column parsed to each item.
Both ref and reflists columns no longer format the parsed value when there's no matching reference, the original unparsed string is used as alttext instead.
Test Plan: Added another table "Multi-References" to CopyPaste test. Made that table and the References table test with and without table data loaded in the browser.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3118
Summary:
Adding sort options for columns.
- Sort menu has a new option "More sort options" that opens up Sort left menu
- Each sort entry has an additional menu with 3 options
-- Order by choice index (for the Choice column, orders by choice position)
-- Empty last (puts empty values last in ascending order, first in descending order)
-- Natural sort (for Text column, compares strings with numbers as numbers)
Updated also CSV/Excel export and api sorting.
Most of the changes in this diff is a sort expression refactoring. Pulling out all the methods
that works on sortExpression array into a single namespace.
Test Plan: Browser tests
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: dsagal, alexmojaki
Differential Revision: https://phab.getgrist.com/D3077
Summary:
- Grouping series may result in series with inconsistent number of values. This can result in inconsistent ordering of the bars displayed by plotly.
- This diff fixes it by consolidating grouped series by adding unll values for each missing xvalues in the series.
Here a is a minimal example of that bug:
{F36639}
Test Plan: Includes new nbrowser test.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D3085
Summary:
Handle reference columns in ViewFieldRec.valueParser.
Extracted code for reuse from ReferenceEditor to look up values in the visible column. While I was at it, also extracted a bit of common code from ReferenceEditor and ReferenceListEditor into a new class ReferenceUtils. More refactoring could be done in this area but it's out of scope.
Changed NTextEditor to use field.valueParser, which affects numeric and reference fields. In particular this means numbers are parsed on data entry, it doesn't change anything for references.
Test Plan:
Added more CopyPaste testing to test references.
Tested entering slightly formatted numbers in NumberFormatting.
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D3094
Summary:
- This should make these easier to work with and make changes to.
- Removes one unused method.
Test Plan: No changes of behavior, existing tests should pass.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3091
Summary:
When there is a link in a text cell (and formula cells), it will be
rendered with a little clickable icon wrapped in the anchor tag
with a proper link. Only links that starts with https? will be
rendered as links.
Links are shown in a Text and Formula fields, inside a GridView,
CardView and in the Import preview dialog.
Test Plan: Browser tests
Reviewers: alexmojaki
Reviewed By: alexmojaki
Subscribers: dsagal, alexmojaki
Differential Revision: https://phab.getgrist.com/D3070
Summary:
Searching for matching choices was using nativeCompare for a binary search,
while the list was sorted according to localeCompare. This was causing the
search to fail for some strings when the two orderings differ.
Test Plan: Tested manually
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3059
Summary:
New environmental variable GOOGLE_DRIVE_SCOPE that modifies the scope
requested for Google Drive integration.
For prod it has value https://www.googleapis.com/auth/drive.file which leaves
current behavior (Grist is allowed only to access public files and for private
files - it fallbacks to Picker).
For staging it has value https://www.googleapis.com/auth/drive.readonly which
allows Grist to access all private files, and fallbacks to Picker only when the file is
neither public nor private).
Default value is https://www.googleapis.com/auth/drive.file
Test Plan: manual and existing tests
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3038
Summary:
Bug reported by user: https://gristlabs.getgrist.com/doc/check-ins/p/3#a1.s7.r1183.c19p
Setting x axis to a column of type ChoiceList was breaking chart.
This diff fixes that by splitting the record into several records: one for each choice.
`test/nbrowser/ChartView1.ts` was becoming too big and long to run, so this diff introduces `test/nbrowser/ChartView2.ts` to add more test and `test/nbrowser/chartViewTestUtils.ts` to put all utilities or testing charts.
Test Plan: Adds new test.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3041
Summary:
This change prevents dragging tokens when the cursor is over the
delete button. Now, trying to drag after mousedown over the delete
button will do nothing if the cursor is released outside the delete
button, and will delete if the cursor is released inside the delete
button.
Test Plan: Tested manually.
Reviewers: alexmojaki
Reviewed By: alexmojaki
Differential Revision: https://phab.getgrist.com/D3024
Summary:
Chart view used to rely on the same view field configuration as used in any other widget.
This diff allows to explicitely select X-AXIS, Y-AXIS and group by column with column picker.
As charts supports several y-axis, we still use a draggable list to arrange them.
Diff also fix doc to the `insertPositions` function.
Test Plan: Updated the relevant test.
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3021
Summary:
Bar chart was a bit broken when there were redundant values on the X axis: the bars’s height maps to the sum of all the corresponding y values, when the data that shows up on hover is only the last one.
It seems that plotly does not support redundant values in the x axis and in all Plotly examples (implementation relies on plotly) x values only have unique values.
This diff, fixes by making sure x axis has unique values. If user actually wants to plot groups, they'll have to use a summary charts.
Test Plan: tested manually
Reviewers: georgegevoian
Reviewed By: georgegevoian
Differential Revision: https://phab.getgrist.com/D3011
Summary: It is important for linking to be maintained, or user could be gravely misled about material in other sections.
Test Plan: added test
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: dsagal
Differential Revision: https://phab.getgrist.com/D3003
Summary:
Adds Reference List as a widget type.
Reference List is similar to Choice List: multiple references can be added
to each cell through a similar editor, and the individual references
will always reflect their current value from the referenced table.
Test Plan: Browser tests.
Reviewers: dsagal
Reviewed By: dsagal
Subscribers: paulfitz, jarek, alexmojaki, dsagal
Differential Revision: https://phab.getgrist.com/D2959
Summary: Importing from google drive from home screen (also for anonymous users)
Test Plan: Browser tests
Reviewers: dsagal, paulfitz
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2943
Summary:
When importing from url, user types a url for google spreadsheet,
Grist will switch to Google Drive plugin to allow user to choose file manualy.
Test Plan: Browser tests
Reviewers: paulfitz, dsagal
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2945
Summary:
Flaky Dates test failures related to the use of JQuery autocomplete for time
zones, which wasn't working well.
This diff replaces that autocomplete (as well as a similar select box in
DocumentSettings) with our newer autocomplete, adding some select-box like
behavior.
Most of the behavior is factored out into ACSelect, which could be more
generally useful.
Adds an option to autocomplete to keep options ordered according to their
initial order.
Unrelated: fix up usage of MultiHolder in Drafts to avoid 'already disposed'
warnings.
Test Plan: Fixed several affected tests.
Reviewers: jarek
Reviewed By: jarek
Differential Revision: https://phab.getgrist.com/D2919
Summary:
Choice columns can now add new choices directly
from the autocomplete menu. The autocomplete will
now highlight the first matching item, even if there are equally
ranked alternatives. No changes have been made to how the
autocomplete index is created, or how it scores items.
For choice and choice list columns, the filter menu will
now display values using their configured colors, similar to the
rest of the UI. Choice tokens throughout the UI now do a better
job of handling text overflow by showing an ellipsis whenever
there isn't enough space to show the full text of a choice.
Test Plan: Browser tests.
Reviewers: cyprien
Reviewed By: cyprien
Differential Revision: https://phab.getgrist.com/D2904
Summary:
Error notifications include a "Report a problem" link, but the inclusion of
error message and stack trace was broken.
Test Plan: Tested manually and added a test case.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2911
Summary:
Includes overhauled choice configuration UI for choice and choice list
columns based on the TokenField library. Features include rich copy
and paste support, keyboard shortcuts for token manipulation, and
drag-and-drop support for arrangement.
Configured choice colors are visible throughout the application, such
as in the autocomplete window for both choice and choice list cells, and
in table cells directly.
Choice cells in particular are now styled closer to choice list cells,
and render their contents as colored tokens. Choice cells now also
use the improved autocomplete component that choice lists use, with
some room for future improvement (e.g. allowing new choice items to be
added inline like in choice list's autocomplete).
Also includes a minor fix for choice list cells where right align
was not working.
Test Plan: Browser tests updated.
Reviewers: jarek, dsagal
Reviewed By: jarek, dsagal
Subscribers: jarek
Differential Revision: https://phab.getgrist.com/D2890
Summary:
- Normally Reference columns can only be used for data entry once the target
table has loaded. When it shows RowID, we shouldn't need to wait.
- Also, fix pasting values between cells of a RowID-showing column.
Test Plan: Added a test for entering data before data has loaded.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2902
Summary:
Grist should not prevent read-only viewers from opening cell editors since they usually provide much more information than is visible in a cell.
Every editor was enhanced with a read-only mode that provides the same information available for an editor but doesn't allow to change the underlying data.
Test Plan: Browser tests
Reviewers: dsagal
Reviewed By: dsagal
Differential Revision: https://phab.getgrist.com/D2842
Summary:
We used tslint earlier, and on switching to eslint, some rules were not
transfered. This moves more rules over, for consistent conventions or helpful
warnings.
- Name private members with a leading underscore.
- Prefer interface over a type alias.
- Use consistent spacing around ':' in type annotations.
- Use consistent spacing around braces of code blocks.
- Use semicolons consistently at the ends of statements.
- Use braces around even one-liner blocks, like conditionals and loops.
- Warn about shadowed variables.
Test Plan: Fixed all new warnings. Should be no behavior changes in code.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2831
Summary:
- The parameter was inadvertently removed by userOverrideParams().
- when passed a function to `urlState().setHref(...)` or `urlState().makeUrl(...)` it is important that the function does not mutate the state that it receives as argument.
Link to the related task: https://gristlabs.getgrist.com/doc/check-ins/p/5#a1.s9.r791.c19
Test Plan: Adds test of the persistence by slightly modifying existing nbrowser/AccessRules2 tests.
Reviewers: paulfitz
Reviewed By: paulfitz
Subscribers: paulfitz
Differential Revision: https://phab.getgrist.com/D2820
Summary:
- Adds a new ChoiceList type, and widgets to view and edit it.
- Store in SQLite as a JSON string
- Support conversions between ChoiceList and other types
Test Plan: Added browser tests, and a test for how these values are stored
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2803
Summary:
- Supports auto-completion
- Supports various keyboard interactions to delete/select/copy-paste tokens
- Supports various mouse interactions to select and move tokens.
- Supports undo/redo while editing tokens.
Test Plan: stand-alone fixture and browser test verify the supported behaviors
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2789
Summary:
- Update rules to be more like we've had with tslint
- Switch tsserver plugin to eslint (tsserver makes for a much faster way to lint in editors)
- Apply suggested auto-fixes
- Fix all lint errors and warnings in core/, app/, test/
Test Plan: Some behavior may change subtly (e.g. added missing awaits), relying on existing tests to catch problems.
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2785
Summary:
- Switch code that compares user strings to use localeCompare() based on Intl.Collator.
- Use en-US locale for now. (Ideally should be a document property.)
- Note that with this change, sorting is also becoming case-insensitive (which
seems an improvement)
- Updated a sorted test fixture
- Updated a browser test with lots of unicode to expect different order.
- Added a bit of unicode to test ordering in Reference autocomplete dropdown.
Test Plan: Fixed / updated tests
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2758
Summary:
- When in View-As mode, clicking the Access Rules page now shows a tooltip
with a link to return to normal mode and open the Access Rules page.
- A "revert" button is shown next to the item with the same behavior.
- Implemented hoverTooltip() with various options. (It will have other uses.)
- Simplify creation of links based on UrlState:
- Allow merging with previous urlState using a function
- Add a helper function to merge in aclAsUser parameter.
- Add setHref() method to UrlState
Test Plan:
Added test cases:
- for tooltips generally in test/projects
- for updating UrlState using a callback
- for Access Rules tooltip and button behavior
Reviewers: paulfitz
Reviewed By: paulfitz
Differential Revision: https://phab.getgrist.com/D2749
