gristlabs_grist-core/app/server
Alex Hall 7fd48364df (core) Improved error messages, retries, and handling of token limits in AI assistant
Summary:
In a nutshell:

- More specific and helpful error messages are shown to the user
- API requests are only retried when needed
- The system deals with reaching the maximum token limit better, especially by switching to a model with a bigger limit

In more detail:

- `COMPLETION_MODEL` configuration has been removed. By default `gpt-3.5-turbo-0613` is used which accepts 4k tokens. If that's not enough, `gpt-3.5-turbo-16k-0613` is used instead.
- Switching to the bigger model happens when either the prompt is too long by itself (the API immediately returns an error code) or the model reaches the 4k limit itself in the process of generating a response and thus returns an incomplete response. The latter case is made possible by removing the `max_tokens: 1500` in the request, which was very generous and would have lead to switching to the more expensive model more often than needed. The downside is that the user has to wait a bit longer for the response.
- If the bigger 16k token limit is also exceeded, the assistant immediately responds (instead of retrying as before) with an error message including suggestions. The suggestions include restarting the conversation if and only if the user has sent multiple messages.
- If a request fails because Grist has reached its OpenAI monthly billing quota, the assistant immediately responds (instead of retrying as before) with an error message suggesting that the user try again tomorrow.
- If a request fails for some other reason, the assistant retries, and if all attempts fail then the user is told to try again in a few minutes and is shown the exact error message, including the API response if there is one.
- Retrying only happens when an API request fails, whereas previously the system also retried errors from a much bigger scope which included calls to the sandbox. The downside is that the hugging face assistant no longer retries, although that code is currently disabled anyway.
- The assistant no longer waits an additional second after the final retry attempt fails.

Test Plan: Added a new server test file with several unit tests using faked OpenAI responses, including the happy path which wasn't really tested before.

Reviewers: dsagal

Reviewed By: dsagal

Subscribers: dsagal

Differential Revision: https://phab.getgrist.com/D3955
2023-07-18 16:01:37 +02:00
..
declarations (core) disentangle some server tests, release to core, add GRIST_PROXY_AUTH_HEADER test 2022-03-24 15:11:32 -04:00
lib (core) Improved error messages, retries, and handling of token limits in AI assistant 2023-07-18 16:01:37 +02:00
utils (core) sanitizing redis errors 2023-06-06 10:51:17 +02:00
companion.ts (core) Add Support Grist page and nudge 2023-07-04 17:36:59 -04:00
declarations.d.ts (core) Faster builds all around. 2022-07-04 10:42:40 -04:00
devServerMain.ts (core) Faster builds all around. 2022-07-04 10:42:40 -04:00
generateCheckpoint.ts (core) get all tests working under python3/gvisor 2021-10-18 17:44:15 -04:00
generateInitialDocSql.ts (core) Add optional telemetry to grist-core 2023-06-07 12:00:51 -04:00
localization.ts only offer languages that have been marked as substantially complete (#466) 2023-03-28 10:59:17 -04:00
mergedServerMain.ts (core) resolved some divergence in mergedServerMain 2023-07-10 07:50:52 -04:00
tsconfig.json (core) Simple localization support and currency selector. 2021-08-26 13:36:49 -07:00