gristlabs_grist-core/sandbox/grist/test_replay.py
Alex Hall 84ddbc448b (core) Add test_replay for easily replaying data sent to the sandbox purely within python
Summary:
Run JS with a value for SANDBOX_BUFFERS_DIR, then run test_replay in python with the same value to replay just the python code.

See test_replay.py for more info.

Test Plan:
Record some data, e.g. `SANDBOX_BUFFERS_DIR=manual npm start` or `SANDBOX_BUFFERS_DIR=server ./test/testrun.sh server`.

Then run `SANDBOX_BUFFERS_DIR=server python -m unittest test_replay` from within `core/sandbox/grist` to replay the input from the JS.

Sample of the output will look like this:

```
Checking /tmp/sandbox_buffers/server/2021-06-16T15:13:59.958Z
True
Checking /tmp/sandbox_buffers/server/2021-06-16T15:16:37.170Z
True
Checking /tmp/sandbox_buffers/server/2021-06-16T15:14:22.378Z
True
```

Reviewers: paulfitz, dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D2866
2021-06-30 16:56:09 +02:00

92 lines
3.4 KiB
Python

"""
Replay binary data sent from JS to reproduce behaviour in the sandbox.
This isn't really a test and it doesn't run under normal circumstances,
but it's convenient to run it alongside other tests to measure total coverage.
This is a tool to directly run some python code of interest to make it easier to do things like:
- Use a debugger within Python
- Measure Python code coverage from JS tests
- Rapidly iterate on Python code without having to repeatedly run the same JS
or write a Python test from scratch.
To use this, first set the environment variable RECORD_SANDBOX_BUFFERS_DIR to a directory path,
then run some JS code. For example you could run some tests,
or run `npm start` and then manually interact with a document in a way that triggers
desired behaviour in the sandbox.
This will store files like $RECORD_SANDBOX_BUFFERS_DIR/<subdirectory>/(input|output)
Each subdirectory corresponds to a single sandbox process so that replays are isolated.
JS tests can start many instances of the sandbox and thus create many subdirectories.
`input` contains the binary data sent from JS to Python, `output` contains the data sent back.
Currently, the name of each subdirectory is the time it was created.
Now run this test with the same value of RECORD_SANDBOX_BUFFERS_DIR. For each subdirectory,
it will read in `input` just as it would read the pipe from JS, and send output to a file
`new_output` in the same subdirectory. Then it will compare the data in `output` and `new_output`.
The outputs will usually match but there are many reasons they might differ:
- Functions registered in JS tests (e.g. via plugins) but not in the python unit tests.
- File paths in tracebacks.
- Slight differences between standard and NaCl interpreters.
- Functions involving randomness or time.
In any case the point is usually not whether or not the outputs match, but to directly run
just the python code of interest.
"""
from __future__ import print_function
import marshal
import os
import unittest
from main import run
from sandbox import Sandbox
def marshal_load_all(path):
result = []
with open(path, "rb") as f:
while True:
try:
result.append(marshal.load(f))
except EOFError:
break
return result
class TestReplay(unittest.TestCase):
maxDiff = None
def test_replay(self):
root = os.environ.get("RECORD_SANDBOX_BUFFERS_DIR")
if not root:
self.skipTest("RECORD_SANDBOX_BUFFERS_DIR not set")
for dirpath, dirnames, filenames in os.walk(root):
if "input" not in filenames:
continue
print("Checking " + dirpath)
input_path = os.path.join(dirpath, "input")
output_path = os.path.join(dirpath, "output")
new_output_path = os.path.join(dirpath, "new_output")
with open(input_path, "rb") as external_input:
with open(new_output_path, "wb") as external_output:
sandbox = Sandbox(external_input, external_output)
run(sandbox)
original_output = marshal_load_all(output_path)
# _send_to_js does two layers of marshalling,
# and NSandbox._onSandboxData parses one of those layers before writing,
# hence original_output is 'more parsed' than marshal_load_all(new_output_path)
new_output = [marshal.loads(b) for b in marshal_load_all(new_output_path)]
# It's usually not worth asserting a match, see comments at the top of the file
print("Match:", original_output == new_output)
# self.assertEqual(original_output, new_output)