(core) Implement trigger formulas (generalizing default formulas)

Summary:
Trigger formulas can be calculated for new records, or for new records and
updates to certain fields, or all fields. They do not recalculate on open,
and they MAY be set directly by the user, including for data-cleaning.

- Column metadata now includes recalcWhen and recalcDeps fields.
- Trigger formulas are NOT recalculated on open or on schema changes.
- When recalcWhen is "never", formula isn't calculated even for new records.
- When recalcWhen is "allupdates", formula is calculated for new records and
  any manual (non-formula) updates to the record.
- When recalcWhen is "", formula is calculated for new records, and changes to
  recalcDeps fields (which may be formula fields or column itself).
- A column whose recalcDeps includes itself is a "data-cleaning" column; a
  value set by the user will still trigger the formula.
- All trigger-formulas receive a "value" argument (to support the case above).

Small changes
- Update RefLists (used for recalcDeps) when target rows are deleted.
- Add RecordList.__contains__ (for `rec in refList` or `id in refList` checks)
- Clarify that Calculate action has replaced load_done() in practice,
  and use it in tests too, to better match reality.

Left for later:
- UI for setting recalcWhen / recalcDeps.
- Implementation of actions such as "Recalculate for all cells".
- Allowing trigger-formulas access to the current user's info.

Test Plan: Added a comprehensive python-side test for various trigger combinations

Reviewers: paulfitz, alexmojaki

Reviewed By: paulfitz

Differential Revision: https://phab.getgrist.com/D2872

sandbox/grist/column.py

@@ -71,7 +71,7 @@ class BaseColumn(object):
     self.node = depend.Node(self.table_id, col_id)
     self._is_formula = col_info.is_formula
     self._is_private = bool(col_info.method) and getattr(col_info.method, 'is_private', False)
-    self.method = col_info.method
+    self.update_method(col_info.method)
 
     # Always initialize to include the special empty record at index 0.
     self.growto(1)
@@ -82,7 +82,11 @@ class BaseColumn(object):
     'method' function. The method may refer to variables in the generated "usercode" module, and
     it's important that all such references are to the rebuilt "usercode" module.
     """
-    self.method = method
+    if not self._is_formula and method:
+      # Include the current value of the cell as the third parameter (to default formulas).
+      self.method = lambda rec, table: method(rec, table, self.get_cell_value(int(rec)))
+    else:
+      self.method = method
 
   def is_formula(self):
     """
@@ -394,6 +398,21 @@ class BaseReferenceColumn(BaseColumn):
   def sample_value(self):
     return self._target_table.sample_record
 
+  def get_updates_for_removed_target_rows(self, target_row_ids):
+    """
+    Returns a list of pairs of (row_id, new_value) for values in this column that need to be
+    updated when target_row_ids are removed from the referenced table.
+    """
+    affected_rows = sorted(self._relation.get_affected_rows(target_row_ids))
+    return [(row_id, self._raw_get_without(row_id, target_row_ids)) for row_id in affected_rows]
+
+  def _raw_get_without(self, _row_id, _target_row_ids):
+    """
+    Returns a Ref or RefList cell value with the specified target_row_ids removed, assuming one of
+    them is actually present in the value. For References, it just leaves the default value.
+    """
+    return self.getdefault()
+
 
 class ReferenceColumn(BaseReferenceColumn):
   """
@@ -439,6 +458,14 @@ class ReferenceListColumn(BaseReferenceColumn):
       return typed_value
     return self._target_table.RecordSet(self._target_table, typed_value, self._relation)
 
+  def _raw_get_without(self, row_id, target_row_ids):
+    """
+    Returns the RefList cell value at row_id with the specified target_row_ids removed.
+    """
+    raw = self.raw_get(row_id)
+    if self.type_obj.is_right_type(raw):
+      raw = [r for r in raw if r not in target_row_ids] or None
+    return raw
 
 # Set up the relationship between usertypes objects and column objects.
 usertypes.BaseColumnType.ColType = DataColumn