(core) Show example values in formula autocomplete

Summary:
This diff adds a preview of the value of certain autocomplete suggestions, especially of the form `$foo.bar` or `user.email`. The main initial motivation was to show the difference between `$Ref` and `$Ref.DisplayCol`, but the feature is more general.

The client now sends the row ID of the row being edited (along with the table and column IDs which were already sent) to the server to fetch autocomplete suggestions. The returned suggestions are now tuples `(suggestion, example_value)` where `example_value` is a string or null. The example value is simply obtained by evaluating (in a controlled way) the suggestion in the context of the given record and the current user. The string representation is similar to the standard `repr` but dates and datetimes are formatted, and the whole thing is truncated for efficiency.

The example values are shown in the autocomplete popup separated from the actual suggestion by a number of spaces calculated to:

1. Clearly separate the suggestion from the values
2. Left-align the example values in most cases
3. Avoid having so much space such that connecting suggestions and values becomes visually difficult.

The tokenization of the row is then tweaked to show the example in light grey to deemphasise it.

Main discussion where the above was decided: https://grist.slack.com/archives/CDHABLZJT/p1661795588100009

The diff also includes various other small improvements and fixes:

- The autocomplete popup is much wider to make room for long suggestions, particularly lookups, as pointed out in https://phab.getgrist.com/D3580#inline-41007. The wide popup is the reason a fancy solution was needed to position the example values. I didn't see a way to dynamically resize the popup based on suggestions, and it didn't seem like a good idea to try.
- The `grist` and `python` labels previously shown on the right are removed. They were not helpful (https://grist.slack.com/archives/CDHABLZJT/p1659697086155179) and would get in the way of the example values.
- Fixed a bug in our custom tokenization that caused function arguments to be weirdly truncated in the middle: https://grist.slack.com/archives/CDHABLZJT/p1661956353699169?thread_ts=1661953258.342739&cid=CDHABLZJT and https://grist.slack.com/archives/C069RUP71/p1659696778991339
- Hide suggestions involving helper columns like `$gristHelper_Display` or `Table.lookupRecords(gristHelper_Display=` (https://grist.slack.com/archives/CDHABLZJT/p1661953258342739). The former has been around for a while and seems to be a mistake. The fix is simply to use `is_visible_column` instead of `is_user_column`. Since the latter is not used anywhere else, and using it in the first place seems like a mistake more than anything else, I've also removed the function to prevent similar mistakes in the future.
- Don't suggest private columns as lookup arguments: https://grist.slack.com/archives/CDHABLZJT/p1662133416652499?thread_ts=1661795588.100009&cid=CDHABLZJT
- Only fetch fresh suggestions specifically after typing `lookupRecords(` or `lookupOne(` rather than just `(`, as this would needlessly hide function suggestions which could still be useful to see the arguments. However this only makes a difference when there are still multiple matching suggestions, otherwise Ace hides them anyway.

Test Plan: Extended and updated several Python and browser tests.

Reviewers: paulfitz

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D3611

sandbox/grist/engine.py

@@ -19,7 +19,7 @@ from sortedcontainers import SortedSet
 import acl
 import actions
 import action_obj
-from autocomplete_context import AutocompleteContext, lookup_autocomplete_options
+from autocomplete_context import AutocompleteContext, lookup_autocomplete_options, eval_suggestion
 from codebuilder import DOLLAR_REGEX
 import depend
 import docactions
@@ -1408,7 +1408,7 @@ class Engine(object):
     if not self._in_update_loop:
       self._bring_mlookups_up_to_date(doc_action)
 
-  def autocomplete(self, txt, table_id, column_id, user):
+  def autocomplete(self, txt, table_id, column_id, row_id, user):
     """
     Return a list of suggested completions of the python fragment supplied.
     """
@@ -1425,13 +1425,15 @@ class Engine(object):
         result = [
           txt + col_id + "="
           for col_id in lookup_table.all_columns
-          if column.is_user_column(col_id) or col_id == 'id'
+          if column.is_visible_column(col_id) or col_id == 'id'
         ]
         # Add specific complete lookups involving reference columns.
         result += [
           txt + option
           for option in lookup_autocomplete_options(lookup_table, table, reverse_only=False)
         ]
+        # Add a dummy empty example value for each result to produce the correct shape.
+        result = [(r, None) for r in result]
         return sorted(result)
 
     # replace $ with rec. and add a dummy rec object
@@ -1479,11 +1481,24 @@ class Engine(object):
             for option in lookup_autocomplete_options(lookup_table, table, reverse_only=True)
           ]
 
+    ### Add example values to all results where possible.
+    if row_id == "new":
+      row_id = table.row_ids.max()
+    rec = table.Record(row_id)
+    # Don't use the same user object as above because we don't want is_sample=True,
+    # which is only needed for the sake of suggesting completions.
+    # Here we want to show actual values.
+    user_obj = User(user, self.tables)
+    results = [
+      (result, eval_suggestion(result, rec, user_obj))
+      for result in results
+    ]
+
     # If we changed the prefix (expanding the $ symbol) we now need to change it back.
     if tweaked_txt != txt:
-      results = [txt + result[len(tweaked_txt):] for result in results]
+      results = [(txt + result[len(tweaked_txt):], value) for result, value in results]
     # pylint:disable=unidiomatic-typecheck
-    results.sort(key=lambda r: r[0] if type(r) == tuple else r)
+    results.sort(key=lambda r: r[0][0] if type(r[0]) == tuple else r[0])
     return results
 
   def _get_undo_checkpoint(self):