Skip to content

Conversation

ranjit41771
Copy link

Problem
The async_openai crate enforces strict deserialization for OpenAI-compatible streaming responses, but some APIs (e.g., Google's Gemini API) that partially emulate OpenAI's chat completion format may omit the index field in tool call deltas within streaming chunks. This leads to a Serde deserialization error like failed deserialization of: {...} when parsing responses where the index is not directly present in the tool call object (it's often at the choice level instead).For example, a Gemini tool call chunk might look like:
json { "choices": [ { "delta": { "tool_calls": [ { "function": { ... }, "id": "function-call-...", "type": "function" } ] }, "index": 0, "finish_reason": "tool_calls" } ], ... }

Here, the index is absent from the inner tool_calls array, causing the required u32 field in ChatCompletionMessageToolCallChunk to fail parsing. This breaks streaming workflows for users integrating with hybrid or third-party providers such as gemini.
SolutionChanged pub index: u32 to pub index: Option<u32> in ChatCompletionMessageToolCallChunk.
This allows graceful handling of missing index values without altering the struct's behavior for standard OpenAI responses (where index is typically provided and can be unwrapped safely).
No other fields or logic were modified, ensuring minimal impact on existing code.

Impact and BenefitsBackward Compatibility: OpenAI responses remain unaffected—index will still deserialize to Some(value).
Broader Compatibility: Enables seamless use with Gemini and potentially other OpenAI-like APIs (e.g., Anthropic, Mistral) that vary slightly in streaming formats. This reduces the need for user-side workarounds or forks.

Robustness: Makes the crate more resilient to real-world API variations, especially in agentic workflows involving tool calls (e.g., function invocations like get_ticket_types in the example error).
Community Value: As AI providers proliferate with OpenAI-inspired APIs, this change future-proofs the library for multi-provider setups, benefiting developers building cross-compatible chat/streaming apps.

…hunk struct to make it compatible with Gemini and OpenAI endpoint responses
@ranjit41771 ranjit41771 marked this pull request as draft September 20, 2025 08:35
@ranjit41771 ranjit41771 marked this pull request as ready for review September 20, 2025 08:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant