Skip to content

fix: Rate-limited batches falsely trigger 'all processing layers exhausted' #160

@deucebucket

Description

@deucebucket

Problem

When AI providers are rate-limited or circuit-broken, Layer 4 processing marks books as "All processing layers exhausted" even though the books are perfectly identifiable - they just couldn't be reached at that moment.

Merijeek reported (#150) a pile of "exhausted" books that were ALL previously identified successfully (McClellans, Alanson, Of Monsters and Mainframes, etc).

Root Cause

Two-part failure:

1. Double rate-limit check race condition

  • worker.py:378-385 checks user rate limit, waits if needed ✓
  • layer_ai_queue.py:100-103 checks AGAIN and returns (0, 0) immediately if not allowed
  • Back in worker.py:394, processed == 0 increments empty_batch_count
  • After 3 empty batches → everything marked as exhausted (worker.py:407-431)

2. Circuit-broken providers counted as "empty batch"

  • If Gemini/OpenRouter are circuit-broken, call_ai() returns empty results
  • process_queue() returns processed=0 even though items exist in the queue
  • This counts toward the 3-strike exhaustion rule
  • System doesn't distinguish "provider unavailable" from "genuinely unidentifiable"

Expected Behavior

  • Rate-limited or circuit-broken batches should NOT count toward empty_batch_count
  • If all providers are temporarily unavailable, system should WAIT for circuit breakers to clear, then retry
  • Only count a batch as "empty" if providers were available and still couldn't identify
  • Consider: requeue for retry later instead of permanent "exhausted" marking

Files Involved

  • library_manager/worker.py:370-443 - Layer 4 loop with empty_batch_count logic
  • library_manager/pipeline/layer_ai_queue.py:100-103 - Early return on rate limit
  • library_manager/providers/rate_limiter.py - Circuit breaker state
  • library_manager/providers/gemini.py:501-509 - Gemini 429 handling
  • library_manager/providers/bookdb.py:153-159 - BookDB 429 handling

Reported By

@Merijeek in #150 (Feb 16) - books previously identified correctly now showing as exhausted

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2-highMajor functionality broken, no workaroundbugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions