(core) Python optimizations to speed up data engine

Summary:
- A bunch of optimizations guided by python profiling (esp. py-spy)
- Big one is optimizing Record/RecordSet attribute access
- Adds tracemalloc printout when running test_replay with PYTHONTRACEMALLOC=1 (on PY3)
  (but memory size is barely affected by these changes)

- Testing with RECORD_SANDBOX_BUFFERS_DIR, loading and calculating a particular
  very large doc (CRM), time taken improved from 73.9s to 54.8s (26% faster)

Test Plan: No behavior changes intended; relying on existing tests to verify that.

Reviewers: georgegevoian

Reviewed By: georgegevoian

Differential Revision: https://phab.getgrist.com/D3781
This commit is contained in:
Dmitry S
2023-02-04 11:20:13 -05:00
parent 7c448d746f
commit 9d4eeda480
8 changed files with 144 additions and 84 deletions

View File

@@ -60,6 +60,7 @@ class BaseColumn(object):
"""
def __init__(self, table, col_id, col_info):
self.type_obj = col_info.type_obj
self._is_right_type = self.type_obj.is_right_type
self._data = []
self.col_id = col_id
self.table_id = table.table_id
@@ -154,10 +155,14 @@ class BaseColumn(object):
else:
raise objtypes.CellError(self.table_id, self.col_id, row_id, raw.error)
return self._convert_raw_value(raw)
# Inline _convert_raw_value here because this is particularly hot code, called on every access
# of any data field in a formula.
if self._is_right_type(raw):
return self._make_rich_value(raw)
return self._alt_text(raw)
def _convert_raw_value(self, raw):
if self.type_obj.is_right_type(raw):
if self._is_right_type(raw):
return self._make_rich_value(raw)
return self._alt_text(raw)