Saga Pattern & Distributed Transactions

[vadikko2]

🚀 New Features

Saga Pattern & Distributed Transactions

1. Saga Orchestrator: Introduced Saga and SagaTransaction to manage long-running distributed transactions across multiple services 2. or internal operations.
2. Step-Based Execution: Added SagaStepHandler interface. Each step defines an act (forward action) and compensate (rollback action) method.
3. Automatic Compensation: If any step fails, the Saga automatically triggers compensation for all previously completed steps in reverse order (LIFO).
4. State Persistence: Introduced ISagaStorage protocol with MemorySagaStorage implementation. Sagas now persist their state (RUNNING, COMPENSATING, FAILED, COMPLETED) and execution history, enabling crash recovery.
5. Recovery Tooling: Added recover_saga() utility to resume interrupted sagas from their last known consistent state.

🛡 Reliability & Recovery Strategy

This release implements a Strict Backward Recovery strategy to guarantee data consistency:

1. Point of No Return: Once a Saga enters COMPENSATING or FAILED status, forward execution is permanently disabled. Recovery will only complete pending compensations.
2. Zombie State Prevention: Prevents scenarios where a partial rollback conflicts with a retried forward execution.
3. Idempotency: The recovery mechanism automatically skips steps that were already successfully completed (or compensated) to prevent double-execution.
4. Resilient Compensation: Added configurable exponential backoff and retry mechanisms specifically for compensation steps (compensation_retry_count, compensation_retry_delay).

⚡️ Implementation Details

1. Type Safety: Full generic type support for SagaContext. The context is strictly typed throughout the transaction.
2. DI Integration: Step handlers are resolved dynamically via the Dependency Injection container, allowing seamless injection of repositories and API clients.
3. Middleware Support: Saga execution supports the existing middleware chain for logging, context propagation, etc.
4. Flexible Context Serialization: Support for Pydantic models, DataClasses, and standard dictionaries for state persistence.

📚 Documentation & Examples

1. Core Examples: Added examples/saga.py demonstrating success and failure scenarios.
2. FastAPI Integration: Added examples/fastapi_saga_sse.py showing how to stream Saga progress to clients via Server-Sent Events (SSE).
3. Recovery Guide: Added comprehensive documentation on how the recovery strategy works and how to implement storage backends.

🧪 Testing

1. Extensive test suite added covering complex edge cases:
2. Recovery from crashes during forward execution.
3. Recovery from crashes during compensation (ensuring the rollback completes).
4. Verification of the "Point of No Return" logic.
5. Idempotency checks for step re-execution.

---

This discussion was created from the release Saga Pattern & Distributed Transactions.

[akhundMurad]

Hey, I was planning to start improving my diator library and noticed that you already did amazing job here! Appreciate your hard work, don't give up 🙂

[vadikko2]

Thank you for your feedback, it is important for our development team!