(core) Porting the AI evaluation script

Summary:
Porting script that run an evaluation against our formula dataset.

To test you need an openai key (see here: https://platform.openai.com/)
or hugging face (it should work as well), then checkout the branch and run

`OPENAI_API_KEY=<my_openai_api_key> node core/test/formula-dataset/runCompletion.js`

Test Plan:
Needs manually testing: so far there is no plan to make it part of CI.

The current score is somewhere around 34 successful prompts over a total of 47.

Reviewers: paulfitz

Reviewed By: paulfitz

Subscribers: jarek

Differential Revision: https://phab.getgrist.com/D3816
This commit is contained in:
Cyprien P
2023-03-15 09:52:17 +01:00
parent 2b2e19c5b5
commit 1ff93f89c2
8 changed files with 377 additions and 37 deletions

View File

@@ -951,7 +951,7 @@ export function assertIsDefined<T>(name: string, value: T): asserts value is Non
* Calls function `fn`, passes any thrown errors to function `recover`, and finally calls `fn`
* once more if `recover` doesn't throw.
*/
export async function retryOnce<T>(fn: () => Promise<T>, recover: (e: unknown) => Promise<void>): Promise<T> {
export async function retryOnce<T>(fn: () => Promise<T>, recover: (e: unknown) => Promise<void>): Promise<T> {
try {
return await fn();
} catch (e) {
@@ -964,7 +964,7 @@ export function assertIsDefined<T>(name: string, value: T): asserts value is Non
* Checks if value is 'empty' (like null, undefined, empty string, empty array/set/map, empty object).
* Values like 0, true, false are not empty.
*/
export function notSet(value: any) {
export function notSet(value: any) {
return value === undefined || value === null || value === ''
|| (Array.isArray(value) && !value.length)
|| (typeof value === 'object' && !Object.keys(value).length)