Skip to content

fix: enable failover during stream iteration#279

Open
kachelle wants to merge 2 commits intolaravel:0.xfrom
kachelle:feat/stream-failover
Open

fix: enable failover during stream iteration#279
kachelle wants to merge 2 commits intolaravel:0.xfrom
kachelle:feat/stream-failover

Conversation

@kachelle
Copy link
Contributor

@kachelle kachelle commented Mar 17, 2026

Summary

Streaming failover doesn't work because withModelFailover() only catches FailoverableException during the synchronous StreamableAgentResponse creation, not during the lazy generator iteration where HTTP errors
actually occur.

Problem

Promptable::stream() calls withModelFailover(), which wraps a try-catch around:

return $callback($provider, $model); // returns StreamableAgentResponse immediately

This succeeds instantly because StreamableAgentResponse uses a lazy generator - no HTTP call is made until the stream is iterated. By the time a provider returns 429 or 529 (during foreach iteration), execution is
outside the failover try-catch.

Passing provider: [Lab::Mistral, Lab::Anthropic] to stream() never actually fails over.

Solution

When multiple providers are configured, stream() now builds a StreamableAgentResponse with a failover-aware generator. The generator uses yield from to iterate each provider's stream, catching FailoverableException
during iteration and trying the next provider.

Single-provider calls use a direct path with no wrapping overhead.

Tests

  • test_stream_fails_over_to_next_provider_when_primary_is_rate_limited - verifies failover works and AgentFailedOver event is dispatched

  • test_stream_throws_last_exception_when_all_providers_fail - verifies the last exception is thrown when no provider succeeds

    Note: This fix works in conjunction with providers correctly classifying HTTP errors as FailoverableException. See feat: map Mistral HTTP 503 to PrismProviderOverloadedException prism-php/prism#959 for related work on mapping Mistral 503 responses.

This is only my 2nd PR to this project, happy to adjust the approach if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant