(core) Use new asttokens.ASTText to support dollar signs inside f-strings

Summary:
Replaced uses of asttokens.ASTTokens with asttokens.ASTText when working with plain `ast` trees, and use `atok.get_text_range` instead of `node.first_token`.

Upgraded asttokens in Python 2 (it was already upgraded in Python 3).

Test Plan: Added a test with f-strings.

Reviewers: dsagal

Reviewed By: dsagal

Differential Revision: https://phab.getgrist.com/D4001
This commit is contained in:
Alex Hall
2023-08-23 12:45:14 +02:00
parent fe12562ad7
commit b9adcefcce
5 changed files with 29 additions and 16 deletions

View File

@@ -72,6 +72,12 @@ class TestCodeBuilder(test_engine.EngineTestCase):
self.assertEqual(make_body("'''test1'''\n\"\"\"test2\"\"\""),
"'''test1'''\nreturn \"\"\"test2\"\"\"")
if six.PY3:
self.assertEqual(
make_body("f'{$foo + 1 + $bar} 2 {3 + $baz}' + $foo2 + f'{4 + $bar2}!'"),
"return f'{rec.foo + 1 + rec.bar} 2 {3 + rec.baz}' + rec.foo2 + f'{4 + rec.bar2}!'"
)
# Test that we produce valid code when "$foo" occurs in invalid places.
if six.PY2:
raise_code = "raise SyntaxError('invalid syntax', ('usercode', 1, 5, u'foo($bar=1)'))"