(core) Modify prompt so that model may say it cannot help with certain requests.

Summary:
This tweaks the prompting so that the user's message is given on its own instead of as a docstring within Python. This is so that the prompt makes sense when:

- the user asks a question such as "Can you write me a formula which does ...?" rather than describing their formula as a docstring would, or
- the user sends a message that doesn't ask for a formula at all (https://grist.slack.com/archives/C0234CPPXPA/p1687699944315069?thread_ts=1687698078.832209&cid=C0234CPPXPA)

Also added wording for the model to refuse when the user asks for something that the model cannot do.

Because the code (and maybe in some cases the model) for non-ChatGPT models relies on the prompt consisting entirely of Python code produced by the data engine (which no longer contains the user's message) those code paths have been disabled for now. Updating them now seems like undesirable drag, I think it'd be better to revisit this when iteration/experimentation has slowed down and stabilised.

Test Plan:
Added entries to the formula dataset where the response shouldn't contain a formula, indicated by the value `1` for the new column `no_formula`.

This is somewhat successful, as the model does refuse to help in some of the new test cases, but not all. Performance on existing entries also seems a bit worse, but it's hard to distinguish this from random noise. Hopefully this can be remedied in the future with more work, e.g. automatic followup messages containing example inputs and outputs.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3936

app/server/lib/Assistance.ts

@@ -46,7 +46,7 @@ export interface AssistanceSchemaPromptV1Context {
 
 /**
  * A flavor of assistant for use with the OpenAI API.
- * Tested primarily with text-davinci-002 and gpt-3.5-turbo.
+ * Tested primarily with gpt-3.5-turbo.
  */
 export class OpenAIAssistant implements Assistant {
   private _apiKey: string;
@@ -60,8 +60,11 @@ export class OpenAIAssistant implements Assistant {
       throw new Error('OPENAI_API_KEY not set');
     }
     this._apiKey = apiKey;
-    this._model = process.env.COMPLETION_MODEL || "text-davinci-002";
+    this._model = process.env.COMPLETION_MODEL || "gpt-3.5-turbo-0613";
     this._chatMode = this._model.includes('turbo');
+    if (!this._chatMode) {
+      throw new Error('Only turbo models are currently supported');
+    }
     this._endpoint = `https://api.openai.com/v1/${this._chatMode ? 'chat/' : ''}completions`;
   }
 
@@ -72,25 +75,27 @@ export class OpenAIAssistant implements Assistant {
       if (messages.length === 0) {
         messages.push({
           role: 'system',
-          content: 'The user gives you one or more Python classes, ' +
-            'with one last method that needs completing. Write the ' +
-            'method body as a single code block, ' +
-            'including the docstring the user gave. ' +
-            'Just give the Python code as a markdown block, ' +
-            'do not give any introduction, that will just be ' +
-            'awkward for the user when copying and pasting. ' +
-            'You are working with Grist, an environment very like ' +
-            'regular Python except `rec` (like record) is used ' +
-            'instead of `self`. ' +
-            'Include at least one `return` statement or the method ' +
-            'will fail, disappointing the user. ' +
-            'Your answer should be the body of a single method, ' +
-            'not a class, and should not include `dataclass` or ' +
-            '`class` since the user is counting on you to provide ' +
-            'a single method. Thanks!'
+          content: 'You are a helpful assistant for a user of software called Grist. ' +
+            'Below are one or more Python classes. ' +
+            'The last method needs completing. ' +
+            "The user will probably give a description of what they want the method (a 'formula') to return. " +
+            'If so, your response should include the method body as Python code in a markdown block. ' +
+            'Do not include the class or method signature, just the method body. ' +
+            'If your code starts with `class`, `@dataclass`, or `def` it will fail. Only give the method body. ' +
+            'You can import modules inside the method body if needed. ' +
+            'You cannot define additional functions or methods. ' +
+            'The method should be a pure function that performs some computation and returns a result. ' +
+            'It CANNOT perform any side effects such as adding/removing/modifying rows/columns/cells/tables/etc. ' +
+            'It CANNOT interact with files/databases/networks/etc. ' +
+            'It CANNOT display images/charts/graphs/maps/etc. ' +
+            'If the user asks for these things, tell them that you cannot help. ' +
+            'The method uses `rec` instead of `self` as the first parameter.\n\n' +
+            '```python\n' +
+            await makeSchemaPromptV1(doc, request) +
+            '\n```',
         });
         messages.push({
-          role: 'user', content: await makeSchemaPromptV1(doc, request),
+          role: 'user', content: request.text,
         });
       } else {
         if (request.regenerate) {
@@ -257,10 +262,11 @@ function getAssistant() {
   if (process.env.OPENAI_API_KEY) {
     return new OpenAIAssistant();
   }
-  if (process.env.HUGGINGFACE_API_KEY) {
-    return new HuggingFaceAssistant();
-  }
-  throw new Error('Please set OPENAI_API_KEY or HUGGINGFACE_API_KEY');
+  // Maintaining this is too much of a burden for now.
+  // if (process.env.HUGGINGFACE_API_KEY) {
+  //   return new HuggingFaceAssistant();
+  // }
+  throw new Error('Please set OPENAI_API_KEY');
 }
 
 /**

documentation/llm.md

@@ -1,35 +1,35 @@
 # Using Large Language Models with Grist
 
 In this experimental Grist feature, originally developed by Alex Hall,
-you can hook up an AI model such as OpenAI's Codex to write formulas for
+you can hook up OpenAI's ChatGPT to write formulas for
 you. Here's how.
 
-First, you need an API key. You'll have best results currently with an
-OpenAI model. Visit https://openai.com/api/ and prepare a key, then
+First, you need an API key. Visit https://openai.com/api/ and prepare a key, then
 store it in an environment variable `OPENAI_API_KEY`.
 
-Alternatively, there are many non-proprietary models hosted on Hugging Face.
-At the time of writing, none can compare with OpenAI for use with Grist.
-Things can change quickly in the world of AI though. So instead of OpenAI,
-you can visit https://huggingface.co/ and prepare a key, then
-store it in an environment variable `HUGGINGFACE_API_KEY`.
-
 That's all the configuration needed!
 
 Currently it is only a backend feature, we are still working on the UI for it.
 
-## Trying other models
+## Hugging Face and other OpenAI models (deactivated)
 
-The model used will default to `text-davinci-002` for OpenAI. You can
-get better results by setting an environment variable `COMPLETION_MODEL` to
-`code-davinci-002` if you have access to that model.
+_Not currently available, needs some work to revive. These notes are only preserved as a reminder to ourselves of how this worked._
 
-The model used will default to `NovelAI/genji-python-6B` for
+~~To use a different OpenAI model such as `code-davinci-002` or `text-davinci-003`,
+set the environment variable `COMPLETION_MODEL` to the name of the model.~~
+
+~~Alternatively, there are many non-proprietary models hosted on Hugging Face.
+At the time of writing, none can compare with OpenAI for use with Grist.
+Things can change quickly in the world of AI though. So instead of OpenAI,
+you can visit https://huggingface.co/ and prepare a key, then
+store it in an environment variable `HUGGINGFACE_API_KEY`.~~
+
+~~The model used will default to `NovelAI/genji-python-6B` for
 Hugging Face. There's no particularly great model for this application,
 but you can try other models by setting an environment variable
 `COMPLETION_MODEL` to `codeparrot/codeparrot` or
-`NinedayWang/PolyCoder-2.7B` or similar.
+`NinedayWang/PolyCoder-2.7B` or similar.~~
 
-If you are hosting a model yourself, host it as Hugging Face does,
+~~If you are hosting a model yourself, host it as Hugging Face does,
 and use `COMPLETION_URL` rather than `COMPLETION_MODEL` to
-point to the model on your own server rather than Hugging Face.
+point to the model on your own server rather than Hugging Face.~~

sandbox/grist/formula_prompt.py

@@ -150,7 +150,7 @@ def class_schema(engine, table_id, exclude_col_id=None, lookups=False):
   return result
 
 
-def get_formula_prompt(engine, table_id, col_id, description,
+def get_formula_prompt(engine, table_id, col_id, _description,
                        include_all_tables=True,
                        lookups=True):
   result = ""
@@ -165,9 +165,7 @@ def get_formula_prompt(engine, table_id, col_id, description,
   result += "    @property\n"
   result += "    # rec is alias for self\n"
   result += "    def {}(rec) -> {}:\n".format(col_id, return_type)
-  result += '        """\n'
-  result += '{}\n'.format(indent(description, "        "))
-  result += '        """\n'
+  result += "        # Please fill in code only after this line, not the `def`\n"
   return result
 
 def indent(text, prefix, predicate=None):

sandbox/grist/test_formula_prompt.py

@@ -151,9 +151,7 @@ class Table2:
     @property
     # rec is alias for self
     def new_formula(rec) -> float:
-        """
-        description here
-        """
+        # Please fill in code only after this line, not the `def`
 ''')
 
   def test_get_formula_prompt(self):
@@ -183,9 +181,7 @@ class Table1:
     @property
     # rec is alias for self
     def text(rec) -> str:
-        """
-        description here
-        """
+        # Please fill in code only after this line, not the `def`
 ''')
 
     self.assert_prompt("Table2", "ref", '''\
@@ -199,9 +195,7 @@ class Table2:
     @property
     # rec is alias for self
     def ref(rec) -> Table1:
-        """
-        description here
-        """
+        # Please fill in code only after this line, not the `def`
 ''')
 
     self.assert_prompt("Table3", "reflist", '''\
@@ -219,9 +213,7 @@ class Table3:
     @property
     # rec is alias for self
     def reflist(rec) -> List[Table2]:
-        """
-        description here
-        """
+        # Please fill in code only after this line, not the `def`
 ''')
 
   def test_convert_completion(self):

test/formula-dataset/data/formula-dataset-index.csv

@@ -1,69 +1,76 @@
-table_id,col_id,doc_id,Description
-Contacts,Send_Email,hQHXqAQXceeQBPvRw5sSs1,"Link to compose an email, if there is one"
-Contacts,No_Notes,hQHXqAQXceeQBPvRw5sSs1,"Number of notes for this contact"
-Category,Contains_archived_project_,hQHXqAQXceeQBPvRw5sSs1,"Whether any projects in this category are archived"
-Tasks,Today,hQHXqAQXceeQBPvRw5sSs1,Needs to be done today (or every day)
-Tasks,Week_Day,hQHXqAQXceeQBPvRw5sSs1,Full name of deadline weekday
-Tasks,period,hQHXqAQXceeQBPvRw5sSs1,Whether this task was modified between (inclusive) the dates in the single row in Settings
-Expenses,Month,55Q2EtTbFvB1N6iizLh4Rk,e.g. 2022-01
-Payroll,Date_Range,5pHLanQNThxkEaEJHKJUf5,"The start date, followed by a dash (no spaces) and the end date if there is one. Dates are month/day with no leading zeroes."
-Payroll,Per_Hour,5pHLanQNThxkEaEJHKJUf5,The hourly rate of the latest rate for this role and person that started on or before this date
-Payroll,Payment,5pHLanQNThxkEaEJHKJUf5,"Total payment amount for hours worked, rounded to the nearest cent."
-Payroll_summary_Pay_Period_Person,Dates,5pHLanQNThxkEaEJHKJUf5,"All date ranges in the group, separated by a comma and a space"
-People,Full_Name,5pHLanQNThxkEaEJHKJUf5,"e.g. Doe, John"
-General_Ledger,Quarter,2YwYBWpREY2a1N2NV7cb55,e.g. 2020 Q4
-General_Ledger,Year,2YwYBWpREY2a1N2NV7cb55,"Just the year of the date, as a string"
-Time_Calculator,Time_Worked,np7TVHmuvFcHmo1K8h7Ur4,Formatted as hours:minutes. No leading zeroes for hours.
-Time_Calculator,Seconds_Worked,np7TVHmuvFcHmo1K8h7Ur4,"Number of seconds between start/end times, if they're both there"
-Funding_Source,Percentage,qprycQa2TVwajAe6Hb3bUZ,Ratio of the amount to the total across all rows
-Funding_Source_summary,Debt_to_Equity,qprycQa2TVwajAe6Hb3bUZ,Ratio of the total amounts in the group where the type is Debt vs Equity
-Invoices,Client,bReAxyLmzmEQfHF5L5Sc1e,Client's name followed by their address on the next line
-Invoices,Hours,bReAxyLmzmEQfHF5L5Sc1e,Total duration in hours across all time logs for this invoice
-Invoices,Due,bReAxyLmzmEQfHF5L5Sc1e,30 days after the invoice date
-Invoices,Invoice_ID,bReAxyLmzmEQfHF5L5Sc1e,Invoice date followed by the client's name in brackets
-Projects,Project_Name,bReAxyLmzmEQfHF5L5Sc1e,"Client name and project name, e.g. John Doe: Big project"
-Time_Log,Date,bReAxyLmzmEQfHF5L5Sc1e,Start date if there is one
-Time_Log,Duration_hrs_,bReAxyLmzmEQfHF5L5Sc1e,Duration (if there is one) in hours rounded to two decimal places
-Time_Log,Duration_min_,bReAxyLmzmEQfHF5L5Sc1e,"Number of minutes between start and end time. If either time is missing, leave blank. If end is before start, give 0."
-Filtered_By_Formula,LabelCount,9nNr9uQwoXWAvxcWQDygh6,"1 if the state is CA, otherwise 0"
-Objects,Address,pyMHqncEspfZN5zfShCwT8,"City and state, separated by comma space"
-Books,search_terms,hdXy57qLiyNf35oNLzzgBG,"Title and author name, with a space in between"
-BOM_Items,Cost,e4gEm7dt4cgBMkouVBNMeY,Total cost if both quantity and cost are given
-Bill_Of_Materials,Cost,e4gEm7dt4cgBMkouVBNMeY,Total cost
-All_Responses,Entry,qvND7WUcuNb2fU4n1vBJ7f,"Name and submitted date in the format ""Name - month-day"""
-All_Responses,Month,qvND7WUcuNb2fU4n1vBJ7f,Submitted month (full name) and year
-Cap_Table,Common_Stock,iXggjrCPHut9u2BuhJxJkk,"If the class is Options, RSUs, or Option Pool, return 0, otherwise return the fully diluted value."
-Cap_Table,Fully_Diluted,iXggjrCPHut9u2BuhJxJkk,"The granted amount, minus the total pool used if the class is Option Pool"
-Cap_Table,Fully_Diluted_,iXggjrCPHut9u2BuhJxJkk,Fully diluted as a fraction of the total
-Classes,Spots_Left,swLvb3Fic22gVzrdczcAoZ,or Full
-Classes,Count,swLvb3Fic22gVzrdczcAoZ,Number of enrollments for this class where the status is Confirmed
-All_Survey_Responses,Product_Experience_Score,4ktYzGV1mUipSiQFtkLGqm,"A number based on the experience:
+no_formula,table_id,col_id,doc_id,Description
+0,Contacts,Send_Email,hQHXqAQXceeQBPvRw5sSs1,"Link to compose an email, if there is one"
+0,Contacts,No_Notes,hQHXqAQXceeQBPvRw5sSs1,"Number of notes for this contact"
+0,Category,Contains_archived_project_,hQHXqAQXceeQBPvRw5sSs1,"Whether any projects in this category are archived"
+0,Tasks,Today,hQHXqAQXceeQBPvRw5sSs1,Needs to be done today (or every day)
+0,Tasks,Week_Day,hQHXqAQXceeQBPvRw5sSs1,Full name of deadline weekday
+0,Tasks,period,hQHXqAQXceeQBPvRw5sSs1,Whether this task was modified between (inclusive) the dates in the single row in Settings
+0,Expenses,Month,55Q2EtTbFvB1N6iizLh4Rk,e.g. 2022-01
+0,Payroll,Date_Range,5pHLanQNThxkEaEJHKJUf5,"The start date, followed by a dash (no spaces) and the end date if there is one. Dates are month/day with no leading zeroes."
+0,Payroll,Per_Hour,5pHLanQNThxkEaEJHKJUf5,The hourly rate of the latest rate for this role and person that started on or before this date
+0,Payroll,Payment,5pHLanQNThxkEaEJHKJUf5,"Total payment amount for hours worked, rounded to the nearest cent."
+0,Payroll_summary_Pay_Period_Person,Dates,5pHLanQNThxkEaEJHKJUf5,"All date ranges in the group, separated by a comma and a space"
+0,People,Full_Name,5pHLanQNThxkEaEJHKJUf5,"e.g. Doe, John"
+0,General_Ledger,Quarter,2YwYBWpREY2a1N2NV7cb55,e.g. 2020 Q4
+0,General_Ledger,Year,2YwYBWpREY2a1N2NV7cb55,"Just the year of the date, as a string"
+0,Time_Calculator,Time_Worked,np7TVHmuvFcHmo1K8h7Ur4,Formatted as hours:minutes. No leading zeroes for hours.
+0,Time_Calculator,Seconds_Worked,np7TVHmuvFcHmo1K8h7Ur4,"Number of seconds between start/end times, if they're both there"
+0,Funding_Source,Percentage,qprycQa2TVwajAe6Hb3bUZ,Ratio of the amount to the total across all rows
+0,Funding_Source_summary,Debt_to_Equity,qprycQa2TVwajAe6Hb3bUZ,Ratio of the total amounts in the group where the type is Debt vs Equity
+0,Invoices,Client,bReAxyLmzmEQfHF5L5Sc1e,Client's name followed by their address on the next line
+0,Invoices,Hours,bReAxyLmzmEQfHF5L5Sc1e,Total duration in hours across all time logs for this invoice
+0,Invoices,Due,bReAxyLmzmEQfHF5L5Sc1e,30 days after the invoice date
+0,Invoices,Invoice_ID,bReAxyLmzmEQfHF5L5Sc1e,Invoice date followed by the client's name in brackets
+0,Projects,Project_Name,bReAxyLmzmEQfHF5L5Sc1e,"Client name and project name, e.g. John Doe: Big project"
+0,Time_Log,Date,bReAxyLmzmEQfHF5L5Sc1e,Start date if there is one
+0,Time_Log,Duration_hrs_,bReAxyLmzmEQfHF5L5Sc1e,Duration (if there is one) in hours rounded to two decimal places
+0,Time_Log,Duration_min_,bReAxyLmzmEQfHF5L5Sc1e,"Number of minutes between start and end time. If either time is missing, leave blank. If end is before start, give 0."
+0,Filtered_By_Formula,LabelCount,9nNr9uQwoXWAvxcWQDygh6,"1 if the state is CA, otherwise 0"
+0,Objects,Address,pyMHqncEspfZN5zfShCwT8,"City and state, separated by comma space"
+0,Books,search_terms,hdXy57qLiyNf35oNLzzgBG,"Title and author name, with a space in between"
+0,BOM_Items,Cost,e4gEm7dt4cgBMkouVBNMeY,Total cost if both quantity and cost are given
+0,Bill_Of_Materials,Cost,e4gEm7dt4cgBMkouVBNMeY,Total cost
+1,Bill_Of_Materials,Cost,e4gEm7dt4cgBMkouVBNMeY,Calculate the mean cost and add a row showing the variance from the mean
+0,All_Responses,Entry,qvND7WUcuNb2fU4n1vBJ7f,"Name and submitted date in the format ""Name - month-day"""
+0,All_Responses,Month,qvND7WUcuNb2fU4n1vBJ7f,Submitted month (full name) and year
+0,Cap_Table,Common_Stock,iXggjrCPHut9u2BuhJxJkk,"If the class is Options, RSUs, or Option Pool, return 0, otherwise return the fully diluted value."
+0,Cap_Table,Fully_Diluted,iXggjrCPHut9u2BuhJxJkk,"The granted amount, minus the total pool used if the class is Option Pool"
+0,Cap_Table,Fully_Diluted_,iXggjrCPHut9u2BuhJxJkk,Fully diluted as a fraction of the total
+0,Classes,Spots_Left,swLvb3Fic22gVzrdczcAoZ,or Full
+0,Classes,Count,swLvb3Fic22gVzrdczcAoZ,Number of enrollments for this class where the status is Confirmed
+1,Classes,Count,swLvb3Fic22gVzrdczcAoZ,Add a row at the end with the total number
+0,All_Survey_Responses,Product_Experience_Score,4ktYzGV1mUipSiQFtkLGqm,"A number based on the experience:
 Very Dissatisfied: 1
 Somewhat Dissatisfied: 2
 Neutral: 3
 Somewhat Satisfied: 4
 Very Satisfied: 5"
-Time_Sheet_Entries_summary_Account_Employee_Month,Total_Spend,oGxD8EnzeVs6vSQK3QBrUv,Total hours worked times hourly rate
-Time_Sheets,Title,oGxD8EnzeVs6vSQK3QBrUv,Month number and employee full name separated by a space
-All_Products,SKU,sXsBGDTKau1F3fvxkCyoaJ,"Brand code, color code, and size, separated by dashes without spaces"
-All_Products,QTY_on_Order,sXsBGDTKau1F3fvxkCyoaJ,Total quantity minus total received quantity across all incoming order line items for this product
-All_Products,Stock_Alert,sXsBGDTKau1F3fvxkCyoaJ,"If the amount in stock and on order is more than 5: In Stock
+0,Time_Sheet_Entries_summary_Account_Employee_Month,Total_Spend,oGxD8EnzeVs6vSQK3QBrUv,Total hours worked times hourly rate
+0,Time_Sheets,Title,oGxD8EnzeVs6vSQK3QBrUv,Month number and employee full name separated by a space
+0,All_Products,SKU,sXsBGDTKau1F3fvxkCyoaJ,"Brand code, color code, and size, separated by dashes without spaces"
+0,All_Products,QTY_on_Order,sXsBGDTKau1F3fvxkCyoaJ,Total quantity minus total received quantity across all incoming order line items for this product
+0,All_Products,Stock_Alert,sXsBGDTKau1F3fvxkCyoaJ,"If the amount in stock and on order is more than 5: In Stock
 If it's 0: OUT OF STOCK
 Otherwise: Low Stock"
-Incoming_Order_Line_Items,Received_Qty,sXsBGDTKau1F3fvxkCyoaJ,"The quantity, but only if the order is received"
-Theaters,Latitude2,dKztiPYamcCpttT1LT1FnU,Coordinate before the comma
-Theaters,Longitude,dKztiPYamcCpttT1LT1FnU,Coordinate after the comma and space
-Families,Amount_Due,cJcSKdUC3nLNAv4wTjAxA6,"Total charged minus total paid, capped at 0"
-Families,Total_Applied,cJcSKdUC3nLNAv4wTjAxA6,Total charge for all paid sessions for this family
-Gifts_summary_Occasion_Who_Year,Over_Budget_,dr6epxpXUcy9rsFVUoXTEe,Did we spend more than the budget for this person?
-Gifts_summary_Year,Total_Budget,dr6epxpXUcy9rsFVUoXTEe,Total budget for all important dates this year
-Leases,Signer,5iMYwmESm33JpEECSqdZk2,The signing tenant for this lease
-Apartments,Have_Picture,5iMYwmESm33JpEECSqdZk2,Yes or No depending on if there's a picture
-Apartments,Current_Lease,5iMYwmESm33JpEECSqdZk2,The lease for this apartment whose current status is Active
-Current_Signers,Lease_Start_Date,5iMYwmESm33JpEECSqdZk2,The start date of the lease for this apartment whose current status is Active
-Leases,Lease_End_Date,5iMYwmESm33JpEECSqdZk2,Start date plus the lease term in years minus one day
-Tenancies,Minor,5iMYwmESm33JpEECSqdZk2,"1 if the age is less than 18, otherwise 0"
-Game_Schedule,Loser,1xJAp2uxM7tFCVUbEofKoF,The team that won fewer sets
-Standings,Win_Rate,1xJAp2uxM7tFCVUbEofKoF,Ratio of wins to total games
-Standings,Wins,1xJAp2uxM7tFCVUbEofKoF,Number of games won
-Prepare_Invoices,Due,9NH6D58FmxwPP43nw7uzQK,One month after the issued date if there is one
+0,Incoming_Order_Line_Items,Received_Qty,sXsBGDTKau1F3fvxkCyoaJ,"The quantity, but only if the order is received"
+0,Theaters,Latitude2,dKztiPYamcCpttT1LT1FnU,Coordinate before the comma
+1,Theaters,Latitude2,dKztiPYamcCpttT1LT1FnU,How can I see the coordinates on a map?
+0,Theaters,Longitude,dKztiPYamcCpttT1LT1FnU,Coordinate after the comma and space
+0,Families,Amount_Due,cJcSKdUC3nLNAv4wTjAxA6,"Total charged minus total paid, capped at 0"
+0,Families,Total_Applied,cJcSKdUC3nLNAv4wTjAxA6,Total charge for all paid sessions for this family
+0,Gifts_summary_Occasion_Who_Year,Over_Budget_,dr6epxpXUcy9rsFVUoXTEe,Did we spend more than the budget for this person?
+0,Gifts_summary_Year,Total_Budget,dr6epxpXUcy9rsFVUoXTEe,Total budget for all important dates this year
+0,Leases,Signer,5iMYwmESm33JpEECSqdZk2,The signing tenant for this lease
+1,Leases,Signer,5iMYwmESm33JpEECSqdZk2,Show the attached photo of the signing tenant
+0,Apartments,Have_Picture,5iMYwmESm33JpEECSqdZk2,Yes or No depending on if there's a picture
+0,Apartments,Current_Lease,5iMYwmESm33JpEECSqdZk2,The lease for this apartment whose current status is Active
+0,Current_Signers,Lease_Start_Date,5iMYwmESm33JpEECSqdZk2,The start date of the lease for this apartment whose current status is Active
+0,Leases,Lease_End_Date,5iMYwmESm33JpEECSqdZk2,Start date plus the lease term in years minus one day
+0,Tenancies,Minor,5iMYwmESm33JpEECSqdZk2,"1 if the age is less than 18, otherwise 0"
+0,Game_Schedule,Loser,1xJAp2uxM7tFCVUbEofKoF,The team that won fewer sets
+0,Standings,Win_Rate,1xJAp2uxM7tFCVUbEofKoF,Ratio of wins to total games
+0,Standings,Wins,1xJAp2uxM7tFCVUbEofKoF,Number of games won
+0,Prepare_Invoices,Due,9NH6D58FmxwPP43nw7uzQK,One month after the issued date if there is one
+1,Prepare_Invoices,Due,9NH6D58FmxwPP43nw7uzQK,Hello
+1,Prepare_Invoices,Due,9NH6D58FmxwPP43nw7uzQK,Can you help me?
+1,Prepare_Invoices,Due,9NH6D58FmxwPP43nw7uzQK,How do I create a new table?

test/formula-dataset/runCompletion_impl.ts

@@ -53,6 +53,7 @@ const TEMPLATE_URL = "https://grist-static.com/datasets/grist_dataset_formulai_2
 const oldFetch = DEPS.fetch;
 
 interface FormulaRec {
+  no_formula: string;
   table_id: string;
   col_id: string;
   doc_id: string;
@@ -170,6 +171,10 @@ where c.colId = ? and t.tableId = ?
           if (result.state) {
             history = result.state;
           }
+          if (rec.no_formula == "1") {
+            success = result.suggestedActions.length === 0;
+            return null;
+          }
           suggestedActions = result.suggestedActions;
           // apply modification
           const {actionNum} = await activeDoc.applyUserActions(session, suggestedActions);