[WIP] Implement support to BatchAPIs to gather evidence #687
Conversation
This class is used to submit batch calls to the OpenAI batch API
| data: list[dict[str,str]], | ||
| callbacks: list[Callable] | None = None, | ||
| name: str | None = None, | ||
| skip_system: bool = False, |
There was a problem hiding this comment.
I refactored out skip_system in #680, can you propagate that change to here?
| @@ -609,6 +618,10 @@ class Settings(BaseSettings): | |||
| " router_kwargs key with router kwargs as values." | |||
| ), | |||
| ) | |||
| use_batch_in_summary: bool = Field( | |||
| default=False, | |||
| description="Whether to use batch API for LLMs in summarization", | |||
There was a problem hiding this comment.
Can you add a few words on how the batches are actually formed
There was a problem hiding this comment.
Perhaps you can say something like:
Whether to use batch API for LLMs in summarization, which means multiple messages are sent in one API request.
There was a problem hiding this comment.
It was updated to:
"Whether to use batch API for LLMs in summarization, "
"which means multiple messages are sent in one API request "
"to the LLM provider's batch API."
"This option is only available for Claude(https://docs.anthropic.com/en/api/creating-message-batches)"
"and OpenAI (https://platform.openai.com/docs/guides/batch) chat models."
| } | ||
| ) | ||
|
|
||
| while batch.status != "completed": |
There was a problem hiding this comment.
we probably want "completed" and "failed" to be OpenAI enums here rather than free strings.
| batch = client.batches.retrieve(batch.id) | ||
| if batch.status == "failed": | ||
| raise Exception("Batch failed. \n\nReason: \n" + "\n".join([k.message for k in batch.errors.data])) | ||
| await asyncio.sleep(5) |
There was a problem hiding this comment.
Let's parameterize this waiting, and maybe make the default longer? like 30 second or 1 min polling?
We should probably add some debug/info logs here to track progress along with maybe a max-timeout which users can set.
Due to the parallel nature of gathering evidence and summarizing all candidate papers, we plan to use the batch API when possible.
Task list
get_summary_llmto decide which provider to use given the llm in the config