System Design

Saga Pattern

Manage transactions that span multiple services without a central ACID coordinator — decomposing the operation into chained local transactions and defining compensating transactions to undo what was already done in case of failure.

Intent

A Saga is a sequence of local transactions, each in a different service, where the failure of any step triggers compensating transactions on the previous steps to reverse the effect. It's the practical answer to the consistency problem in distributed systems where each service has its own database.

In a monolith with a single database, an ACID transaction guarantees that either all the operations happen, or none do. In microservices, each service isolates its data in its own database — there's no way to use a single BEGIN and COMMIT spanning all of them. The Saga Pattern accepts this reality and works with eventual consistency instead of immediate consistency, trading distributed atomicity for a managed sequence of local transactions with a business-level rollback mechanism.

Problem

Consider an e-commerce flow: the customer places an order that needs to (1) create the order, (2) reserve stock in the inventory service, (3) process payment in the financial service, and (4) trigger the notification. Each of these steps lives in a service with its own database.

  • Two-Phase Commit (2PC) doesn't scale: the two-phase commit protocol requires every participant to lock resources during the prepare phase. In systems with dozens of services, this creates contention, increases latency, and creates a dependency on a central coordinator. Most modern microservices databases (DynamoDB, MongoDB, CockroachDB) don't support cross-service 2PC in a practical way.
  • Ignoring the problem creates silent inconsistency: if the payment fails after the stock reservation has already been confirmed, the system ends up with stock reserved for an order that will never exist — unless there's an explicit compensation mechanism.
  • Partially committed state is unavoidable: between the order confirmation and the payment confirmation, the system goes through intermediate states that need to be designed and communicated to the user coherently.

How it works

The structure of a Saga

Each step of the saga executes a local transaction in a single database and publishes an event or sends a message to the next participant. If a step fails, the saga executes compensating transactions on the previous steps, in reverse order.

Compensating transactions are not technical rollbacks — they're business operations that semantically undo the effect. "Cancel stock reservation" doesn't delete the database row: it creates a new operation that releases the reserved units. This is fundamental because, unlike a database rollback, the compensation happens after the original transaction has already been committed and potentially seen by other systems.

Choreography

Each service publishes events after completing its local transaction and reacts to events published by other services. There's no central coordinator — the flow emerges from the chain of reactions between the participants. 

  Order Service                 Inventory Service         Payment Service
       │                              │                           │
       │── OrderCreated ─────────────►│                           │
       │                              │── StockReserved ─────────►│
       │                              │                           │── PaymentProcessed
       │                              │                           │
       │                         [if it fails]              [if it fails]
       │◄── ReservationFailed ────────│     │◄── PaymentFailed ───┘
       │   (cancels order)            │    (cancels reservation)

Advantage: simple to implement for a few services, no single point of failure. Disadvantage: a saga's flow is spread across multiple services — understanding the full behavior requires correlating logs and events from several systems simultaneously.

Orchestration

A Saga Orchestrator — a dedicated process or service — explicitly coordinates the participants: it calls each service in sequence, waits for the response, and decides whether to proceed or trigger compensations in case of failure. The entire flow lives in a single place.

SAGA ORCHESTRATOR (coordinates the flow) Order Svc 1. create_order() Inventory Svc 2. reserve() Payment Svc 3. charge() cmd cmd cmd ✓ SUCCESS saga complete ✗ FAILURE e.g.: payment declined triggers compensations Compensations: cancel_reservation() cancel_order() (in reverse order) Orchestrator command Compensating transaction
Orchestrated Saga: the Orchestrator coordinates each service in sequence and triggers compensations in reverse order when a step fails.

Comparison: Choreography vs Orchestration

                    CHOREOGRAPHY         ORCHESTRATION
  ─────────────────────────────────────────────────────────
  Coordinator          None                 Saga Orchestrator
  Coupling             Low (via events)     Medium (to orchestrator)
  Observability        Difficult            Centralized in orch.
  Complexity           Low (few svcs)       Manageable (scales)
  Single point failure No                   Orchestrator (mitigate)
  Explicit flow        No                   Yes
  ─────────────────────────────────────────────────────────
  Suited for           2–3 services         4+ services
                       simple flows         complex flows

When to use

  • Microservices with isolated databases: when each service has its own database and it isn't possible — or desirable — to share a transactional database connection. Saga is the practical alternative to 2PC in this context.
  • Multi-service business transactions: flows such as checkout (order + stock + payment + notification), user onboarding (signup + email delivery + billing account creation), or bookings (flight + hotel + car).
  • Eventual consistency is acceptable: when the business tolerates temporary intermediate states — the order exists but the payment is still being processed — and compensation resolves the failure case acceptably.

When to avoid

  • Strong consistency is mandatory: financial operations that require true atomicity (debit and credit must be atomic) are better solved within a single service with a transactional database or with database-supported two-phase commit.
  • Too many participants: sagas with 10+ steps create complex compensation cascades that are hard to test. In these cases, reevaluating service granularity can be more effective.
  • Team without maturity in distributed systems: sagas require attention to idempotency, traceability, and failure handling in compensations. Without that maturity, the result is usually worse than a transactional monolith.

Pros and cons

Pros

  • Enables eventual consistency across microservices without locking resources (no global locks).
  • Each service maintains its autonomy and its own database — no infrastructure coupling.
  • Orchestration centralizes the flow, making observability, testing, and saga status tracking easier.
  • Scales better than 2PC in systems with many participants and high transaction volume.

Cons

  • Visible intermediate state: the system stays in partially committed states between steps, which needs to be communicated in the UX.
  • Compensation is harder than rollback: compensating transactions need to be designed, implemented, tested, and monitored separately.
  • Complex debugging: tracing the state of a saga distributed across multiple services requires a correlation ID, distributed tracing, and careful log attention.
  • Doesn't replace ACID where immediate consistency is required — it's a conscious trade-off, not a universal solution.

Common pitfalls

1. Non-idempotent compensation

If the compensation service fails after executing the operation but before confirming success, the orchestrator will retry. Without idempotency, the compensation is executed twice — the payment is refunded twice, the stock is released in duplicate. Every compensating transaction must be designed to produce the same result when executed multiple times.

Practice: use an idempotency key (saga ID + step ID) in compensating transactions. Before executing, check whether the compensation for that key has already been applied.

2. Intermediate state exposed to the user

Between step 1 (order created) and step 3 (payment confirmed), the order exists in the system but isn't complete. The user may see an order "awaiting confirmation" that's later canceled due to a payment failure. The UX needs to communicate these intermediate states clearly — an order being processed isn't a confirmed order.

3. Choreography without traceability

In choreography with many services, understanding a saga's current state requires correlating events from every participant. Without a correlation ID present in every event and log, and without distributed tracing (OpenTelemetry, Jaeger), support investigating a failed saga can take hours.

4. Compensation that fails

What if the compensating transaction itself fails? This is the hardest case. The saga enters an inconsistent state that requires intervention. The solution is to have automatic retry with backoff for compensations and, after retries are exhausted, alert the operations team with enough context for manual intervention. Never silently ignore a compensation failure.

Related architectures and patterns

The Outbox Pattern is frequently combined with Sagas: each saga step saves the result to the database and inserts the next command/event into the outbox table within the same local transaction, guaranteeing the event isn't lost even if the process fails after the commit.

Idempotency is a prerequisite for Sagas: both the normal steps and the compensations will be retried in case of network failure, and need to produce the same result on every attempt.

CQRS and Event Sourcing complement Sagas: the domain event is both the communication mechanism between participants (choreography) and the source of truth for the saga's state (the state can be rebuilt from the event log).