Archives/gristlabs_grist-core
1
0
Fork 0
mirror of https://github.com/gristlabs/grist-core.git synced 2026-03-02 04:09:24 +00:00
Code Issues Projects Releases Wiki Activity

(core) Porting the AI evaluation script

Browse Source 
Summary:
Porting script that run an evaluation against our formula dataset.

To test you need an openai key (see here: https://platform.openai.com/)
or hugging face (it should work as well), then checkout the branch and run

`OPENAI_API_KEY=<my_openai_api_key> node core/test/formula-dataset/runCompletion.js`

Test Plan:
Needs manually testing: so far there is no plan to make it part of CI.

The current score is somewhere around 34 successful prompts over a total of 47.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3816
This commit is contained in:
Cyprien P
2023-03-15 09:52:17 +01:00
parent 2b2e19c5b5
commit 1ff93f89c2
8 changed files with 377 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files

55
test/formula-dataset/data/formula-dataset-index.csv Normal file
View File

@@ -0,0 +1,55 @@
table_id,col_id,doc_id,Description
Contacts,Send_Email,hQHXqAQXceeQBPvRw5sSs1,"Link to compose an email, if there is one"
Tasks,Today,hQHXqAQXceeQBPvRw5sSs1,Needs to be done today (or every day)
Tasks,Week_Day,hQHXqAQXceeQBPvRw5sSs1,Full name of deadline weekday
Expenses,Month,55Q2EtTbFvB1N6iizLh4Rk,e.g. 2022-01
Payroll,Date_Range,5pHLanQNThxkEaEJHKJUf5,"The start date, followed by a dash (no spaces) and the end date if there is one. Dates are month/day with no leading zeroes."
Payroll,Payment,5pHLanQNThxkEaEJHKJUf5,"Total payment amount for hours worked, rounded to the nearest cent."
Payroll_summary_Pay_Period_Person,Dates,5pHLanQNThxkEaEJHKJUf5,"All date ranges in the group, separated by a comma and a space"
People,Full_Name,5pHLanQNThxkEaEJHKJUf5,"e.g. Doe, John"
General_Ledger,Quarter,2YwYBWpREY2a1N2NV7cb55,e.g. 2020 Q4
General_Ledger,Year,2YwYBWpREY2a1N2NV7cb55,"Just the year of the date, as a string"
Time_Calculator,Time_Worked,np7TVHmuvFcHmo1K8h7Ur4,Formatted as hours:minutes. No leading zeroes for hours.
Time_Calculator,Seconds_Worked,np7TVHmuvFcHmo1K8h7Ur4,"Number of seconds between start/end times, if they're both there"
Funding_Source_summary,Debt_to_Equity,qprycQa2TVwajAe6Hb3bUZ,Ratio of the total amounts in the group where the type is Debt vs Equity
Invoices,Client,bReAxyLmzmEQfHF5L5Sc1e,Client's name followed by their address on the next line
Invoices,Due,bReAxyLmzmEQfHF5L5Sc1e,30 days after the invoice date
Invoices,Invoice_ID,bReAxyLmzmEQfHF5L5Sc1e,Invoice date followed by the client's name in brackets
Projects,Project_Name,bReAxyLmzmEQfHF5L5Sc1e,"Client name and project name, e.g. John Doe: Big project"
Time_Log,Date,bReAxyLmzmEQfHF5L5Sc1e,Start date if there is one
Time_Log,Duration_hrs_,bReAxyLmzmEQfHF5L5Sc1e,Duration (if there is one) in hours rounded to two decimal places
Time_Log,Duration_min_,bReAxyLmzmEQfHF5L5Sc1e,"Number of minutes between start and end time. If either time is missing, leave blank. If end is before start, give 0."
Filtered_By_Formula,LabelCount,9nNr9uQwoXWAvxcWQDygh6,"1 if the state is CA, otherwise 0"
Objects,Address,pyMHqncEspfZN5zfShCwT8,"City and state, separated by comma space"
Books,search_terms,hdXy57qLiyNf35oNLzzgBG,"Title and author name, with a space in between"
BOM_Items,Cost,e4gEm7dt4cgBMkouVBNMeY,Total cost if both quantity and cost are given
Bill_Of_Materials,Cost,e4gEm7dt4cgBMkouVBNMeY,Total cost
All_Responses,Entry,qvND7WUcuNb2fU4n1vBJ7f,"Name and submitted date in the format ""Name - month-day"""
All_Responses,Month,qvND7WUcuNb2fU4n1vBJ7f,Submitted month (full name) and year
Cap_Table,Common_Stock,iXggjrCPHut9u2BuhJxJkk,"If the class is Options, RSUs, or Option Pool, return 0, otherwise return the fully diluted value."
Cap_Table,Fully_Diluted,iXggjrCPHut9u2BuhJxJkk,"The granted amount, minus the total pool used if the class is Option Pool"
Cap_Table,Fully_Diluted_,iXggjrCPHut9u2BuhJxJkk,Fully diluted as a fraction of the total
Classes,Spots_Left,swLvb3Fic22gVzrdczcAoZ,or Full
All_Survey_Responses,Product_Experience_Score,4ktYzGV1mUipSiQFtkLGqm,"A number based on the experience:
Very Dissatisfied: 1
Somewhat Dissatisfied: 2
Neutral: 3
Somewhat Satisfied: 4
Very Satisfied: 5"
Time_Sheet_Entries_summary_Account_Employee_Month,Total_Spend,oGxD8EnzeVs6vSQK3QBrUv,Total hours worked times hourly rate
Time_Sheets,Title,oGxD8EnzeVs6vSQK3QBrUv,Month number and employee full name separated by a space
All_Products,SKU,sXsBGDTKau1F3fvxkCyoaJ,"Brand code, color code, and size, separated by dashes without spaces"
All_Products,Stock_Alert,sXsBGDTKau1F3fvxkCyoaJ,"If the amount in stock and on order is more than 5: In Stock
If it's 0: OUT OF STOCK
Otherwise: Low Stock"
Incoming_Order_Line_Items,Received_Qty,sXsBGDTKau1F3fvxkCyoaJ,"The quantity, but only if the order is received"
Theaters,Latitude2,dKztiPYamcCpttT1LT1FnU,Coordinate before the comma
Theaters,Longitude,dKztiPYamcCpttT1LT1FnU,Coordinate after the comma and space
Families,Amount_Due,cJcSKdUC3nLNAv4wTjAxA6,"Total charged minus total paid, capped at 0"
Gifts_summary_Occasion_Who_Year,Over_Budget_,dr6epxpXUcy9rsFVUoXTEe,Did we spend more than the budget for this person?
Apartments,Have_Picture,5iMYwmESm33JpEECSqdZk2,Yes or No depending on if there's a picture
Leases,Lease_End_Date,5iMYwmESm33JpEECSqdZk2,Start date plus the lease term in years minus one day
Tenancies,Minor,5iMYwmESm33JpEECSqdZk2,"1 if the age is less than 18, otherwise 0"
Game_Schedule,Loser,1xJAp2uxM7tFCVUbEofKoF,The team that won fewer sets
Standings,Win_Rate,1xJAp2uxM7tFCVUbEofKoF,Ratio of wins to total games
Prepare_Invoices,Due,9NH6D58FmxwPP43nw7uzQK,One month after the issued date if there is one
1 table_id col_id doc_id Description
2 Contacts Send_Email hQHXqAQXceeQBPvRw5sSs1 Link to compose an email, if there is one
3 Tasks Today hQHXqAQXceeQBPvRw5sSs1 Needs to be done today (or every day)
4 Tasks Week_Day hQHXqAQXceeQBPvRw5sSs1 Full name of deadline weekday
5 Expenses Month 55Q2EtTbFvB1N6iizLh4Rk e.g. 2022-01
6 Payroll Date_Range 5pHLanQNThxkEaEJHKJUf5 The start date, followed by a dash (no spaces) and the end date if there is one. Dates are month/day with no leading zeroes.
7 Payroll Payment 5pHLanQNThxkEaEJHKJUf5 Total payment amount for hours worked, rounded to the nearest cent.
8 Payroll_summary_Pay_Period_Person Dates 5pHLanQNThxkEaEJHKJUf5 All date ranges in the group, separated by a comma and a space
9 People Full_Name 5pHLanQNThxkEaEJHKJUf5 e.g. Doe, John
10 General_Ledger Quarter 2YwYBWpREY2a1N2NV7cb55 e.g. 2020 Q4
11 General_Ledger Year 2YwYBWpREY2a1N2NV7cb55 Just the year of the date, as a string
12 Time_Calculator Time_Worked np7TVHmuvFcHmo1K8h7Ur4 Formatted as hours:minutes. No leading zeroes for hours.
13 Time_Calculator Seconds_Worked np7TVHmuvFcHmo1K8h7Ur4 Number of seconds between start/end times, if they're both there
14 Funding_Source_summary Debt_to_Equity qprycQa2TVwajAe6Hb3bUZ Ratio of the total amounts in the group where the type is Debt vs Equity
15 Invoices Client bReAxyLmzmEQfHF5L5Sc1e Client's name followed by their address on the next line
16 Invoices Due bReAxyLmzmEQfHF5L5Sc1e 30 days after the invoice date
17 Invoices Invoice_ID bReAxyLmzmEQfHF5L5Sc1e Invoice date followed by the client's name in brackets
18 Projects Project_Name bReAxyLmzmEQfHF5L5Sc1e Client name and project name, e.g. John Doe: Big project
19 Time_Log Date bReAxyLmzmEQfHF5L5Sc1e Start date if there is one
20 Time_Log Duration_hrs_ bReAxyLmzmEQfHF5L5Sc1e Duration (if there is one) in hours rounded to two decimal places
21 Time_Log Duration_min_ bReAxyLmzmEQfHF5L5Sc1e Number of minutes between start and end time. If either time is missing, leave blank. If end is before start, give 0.
22 Filtered_By_Formula LabelCount 9nNr9uQwoXWAvxcWQDygh6 1 if the state is CA, otherwise 0
23 Objects Address pyMHqncEspfZN5zfShCwT8 City and state, separated by comma space
24 Books search_terms hdXy57qLiyNf35oNLzzgBG Title and author name, with a space in between
25 BOM_Items Cost e4gEm7dt4cgBMkouVBNMeY Total cost if both quantity and cost are given
26 Bill_Of_Materials Cost e4gEm7dt4cgBMkouVBNMeY Total cost
27 All_Responses Entry qvND7WUcuNb2fU4n1vBJ7f Name and submitted date in the format "Name - month-day"
28 All_Responses Month qvND7WUcuNb2fU4n1vBJ7f Submitted month (full name) and year
29 Cap_Table Common_Stock iXggjrCPHut9u2BuhJxJkk If the class is Options, RSUs, or Option Pool, return 0, otherwise return the fully diluted value.
30 Cap_Table Fully_Diluted iXggjrCPHut9u2BuhJxJkk The granted amount, minus the total pool used if the class is Option Pool
31 Cap_Table Fully_Diluted_ iXggjrCPHut9u2BuhJxJkk Fully diluted as a fraction of the total
32 Classes Spots_Left swLvb3Fic22gVzrdczcAoZ or Full
33 All_Survey_Responses Product_Experience_Score 4ktYzGV1mUipSiQFtkLGqm A number based on the experience: Very Dissatisfied: 1 Somewhat Dissatisfied: 2 Neutral: 3 Somewhat Satisfied: 4 Very Satisfied: 5
34 Time_Sheet_Entries_summary_Account_Employee_Month Total_Spend oGxD8EnzeVs6vSQK3QBrUv Total hours worked times hourly rate
35 Time_Sheets Title oGxD8EnzeVs6vSQK3QBrUv Month number and employee full name separated by a space
36 All_Products SKU sXsBGDTKau1F3fvxkCyoaJ Brand code, color code, and size, separated by dashes without spaces
37 All_Products Stock_Alert sXsBGDTKau1F3fvxkCyoaJ If the amount in stock and on order is more than 5: In Stock If it's 0: OUT OF STOCK Otherwise: Low Stock
38 Incoming_Order_Line_Items Received_Qty sXsBGDTKau1F3fvxkCyoaJ The quantity, but only if the order is received
39 Theaters Latitude2 dKztiPYamcCpttT1LT1FnU Coordinate before the comma
40 Theaters Longitude dKztiPYamcCpttT1LT1FnU Coordinate after the comma and space
41 Families Amount_Due cJcSKdUC3nLNAv4wTjAxA6 Total charged minus total paid, capped at 0
42 Gifts_summary_Occasion_Who_Year Over_Budget_ dr6epxpXUcy9rsFVUoXTEe Did we spend more than the budget for this person?
43 Apartments Have_Picture 5iMYwmESm33JpEECSqdZk2 Yes or No depending on if there's a picture
44 Leases Lease_End_Date 5iMYwmESm33JpEECSqdZk2 Start date plus the lease term in years minus one day
45 Tenancies Minor 5iMYwmESm33JpEECSqdZk2 1 if the age is less than 18, otherwise 0
46 Game_Schedule Loser 1xJAp2uxM7tFCVUbEofKoF The team that won fewer sets
47 Standings Win_Rate 1xJAp2uxM7tFCVUbEofKoF Ratio of wins to total games
48 Prepare_Invoices Due 9NH6D58FmxwPP43nw7uzQK One month after the issued date if there is one