-
-
Notifications
You must be signed in to change notification settings - Fork 2
Closed
Labels
P2-highMajor functionality broken, no workaroundMajor functionality broken, no workaroundbugSomething isn't workingSomething isn't working
Description
Problem
When AI providers are rate-limited or circuit-broken, Layer 4 processing marks books as "All processing layers exhausted" even though the books are perfectly identifiable - they just couldn't be reached at that moment.
Merijeek reported (#150) a pile of "exhausted" books that were ALL previously identified successfully (McClellans, Alanson, Of Monsters and Mainframes, etc).
Root Cause
Two-part failure:
1. Double rate-limit check race condition
worker.py:378-385checks user rate limit, waits if needed ✓layer_ai_queue.py:100-103checks AGAIN and returns(0, 0)immediately if not allowed- Back in
worker.py:394,processed == 0incrementsempty_batch_count - After 3 empty batches → everything marked as exhausted (
worker.py:407-431)
2. Circuit-broken providers counted as "empty batch"
- If Gemini/OpenRouter are circuit-broken,
call_ai()returns empty results process_queue()returnsprocessed=0even though items exist in the queue- This counts toward the 3-strike exhaustion rule
- System doesn't distinguish "provider unavailable" from "genuinely unidentifiable"
Expected Behavior
- Rate-limited or circuit-broken batches should NOT count toward empty_batch_count
- If all providers are temporarily unavailable, system should WAIT for circuit breakers to clear, then retry
- Only count a batch as "empty" if providers were available and still couldn't identify
- Consider: requeue for retry later instead of permanent "exhausted" marking
Files Involved
library_manager/worker.py:370-443- Layer 4 loop with empty_batch_count logiclibrary_manager/pipeline/layer_ai_queue.py:100-103- Early return on rate limitlibrary_manager/providers/rate_limiter.py- Circuit breaker statelibrary_manager/providers/gemini.py:501-509- Gemini 429 handlinglibrary_manager/providers/bookdb.py:153-159- BookDB 429 handling
Reported By
@Merijeek in #150 (Feb 16) - books previously identified correctly now showing as exhausted
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
P2-highMajor functionality broken, no workaroundMajor functionality broken, no workaroundbugSomething isn't workingSomething isn't working