Outbox Pattern

Intent

The Outbox Pattern guarantees that data persisted in the database and events published to the message broker never fall out of sync, using the database transaction itself as the atomicity mechanism. The event doesn't go directly to the broker — it first goes to a database table within the same transaction that modifies the business data.

The word "outbox" comes from the email outbox analogy: messages you compose stay in the outbox before being sent. If the server crashes while you're composing, the message is saved and will be sent when the server comes back. The same principle applies here — the event stays "in the outbox" (a database table) until a separate process delivers it to the broker.

Problem

In event-driven systems, the business operation typically needs to do two things: persist the new state in the database and publish an event to notify other services. The problem is that the database and the message broker are independent resources — there's no way to include both in a single ACID transaction.

  // Naive code — two independent operations without atomicity
  await db.save(order);               // ← database commits here
  await broker.publish('OrderCreated', order);  // ← what if this fails?

  // Scenario 1: database commits, broker fails
  //   → order exists, event lost, other services never find out
  //   → silent inconsistency

  // Scenario 2: broker receives, database fails
  //   → event published, order doesn't exist
  //   → consumers process a phantom order

Any failure between the two operations — process crash, network timeout, broker error — leaves the system in an inconsistent state. And the problem is silent: there's no visible explicit error; the systems simply fall out of sync.

How it works

The Outbox table

The solution is to move the event publication inside the database transaction. Instead of publishing directly to the broker, the application inserts the event as a row in a table called outbox (or equivalent), within the same transaction that saves the business data.

  -- outbox table schema
  CREATE TABLE outbox (
    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    event_type  TEXT NOT NULL,             -- e.g.: 'OrderCreated'
    payload     JSONB NOT NULL,            -- serialized event data
    created_at  TIMESTAMPTZ DEFAULT NOW(),
    processed_at TIMESTAMPTZ              -- NULL = awaiting relay
  );

  -- Within the same transaction (local ACID):
  BEGIN;
    INSERT INTO orders (id, customer_id, total) VALUES (...);
    INSERT INTO outbox (event_type, payload)
      VALUES ('OrderCreated', '{"order_id": "...", "total": 99.90}');
  COMMIT;
  -- If the COMMIT fails, neither insertion happens.
  -- If the COMMIT succeeds, both insertions are guaranteed.

Message Relay: Polling vs CDC

The Message Relay is the process responsible for reading the outbox and publishing to the broker. There are two approaches:

• Polling: a process runs periodically (e.g., every 500ms), fetches rows from the outbox where processed_at IS NULL, publishes each event to the broker, and updates processed_at. Simple to implement, but introduces latency proportional to the polling interval and generates constant load on the database.
• CDC (Change Data Capture): captures inserts into the outbox table directly from the database's transaction log (e.g., Debezium reading PostgreSQL's WAL). Lower latency, no constant polling, and no additional query load. More complex to configure and operate.

At-least-once semantics

The Outbox Pattern guarantees events are never lost (no event is silently discarded), but doesn't guarantee exactly-once delivery. If the relay publishes the event and fails before marking it processed, it will republish on the next run. Consumers must be idempotent — processing the same event twice must not have a different effect than processing it once.

  -- Relay may publish duplicates. Consumers need idempotency:
  INSERT INTO order_stock (order_id, quantity)
  VALUES ($1, $2)
  ON CONFLICT (order_id) DO NOTHING;  -- idempotent: ignores duplicate

When to use

• Whenever you need atomicity between the database and the broker: if the business can't tolerate lost events (an order created without notifying the inventory service, for example), Outbox is the correct pattern.
• Event-driven systems with eventual consistency: when event consumers can process asynchronously and the relay's delivery latency (milliseconds to seconds) is acceptable.
• As the foundation for a Saga Pattern: each saga step can use the outbox to guarantee that the event triggering the next step is never lost — even if the process crashes right after the commit.

When to avoid

• Sub-millisecond latency required: the relay introduces additional latency (polling) or operational complexity (CDC). For real-time notifications with no tolerance for that delay, explore other approaches.
• No outbox cleanup job: without a periodic process that removes processed events, the table grows indefinitely. This operational cost needs to be accepted before adopting the pattern.

Pros and cons

Pros

• Atomicity between business data and event using only a local database transaction — no distributed transaction.
• Events are never lost: even if the broker is unavailable at write time, the event stays in the outbox and will be published when the broker comes back.
• Simple to implement with polling; adopting CDC can come later as an optimization.
• Compatible with any relational database that supports ACID transactions.

Cons

• At-least-once semantics: consumers need to be idempotent, which adds complexity.
• Additional latency: the relay introduces a delay between the commit and the event's delivery to the broker.
• Outbox table operation: periodic cleanup, monitoring relay lag, and CDC configuration are additional responsibilities.
• CDC requires attention to the database's WAL configuration (minimum retention compatible with the relay's speed).

Common pitfalls

1. Forgetting to clean up the outbox

Without a cleanup job (DELETE FROM outbox WHERE processed_at < NOW() - INTERVAL '7 days'), the table grows indefinitely. Polling queries get progressively slower and storage grows uncontrolled. Define the retention policy before going to production and monitor the table's size.

2. Relay without idempotency and unprepared consumers

The relay can publish the same event more than once in case of failure. If consumers aren't idempotent, operations like "reserve stock" or "send confirmation email" will be executed in duplicate. Document this semantics explicitly and ensure every outbox consumer implements idempotency.

3. CDC losing events due to WAL rotation

Debezium (and other CDC connectors) reads PostgreSQL's Write-Ahead Log. If the WAL is rotated before the CDC processes the outbox inserts — which can happen during periods of high write volume or if the CDC goes offline for a long time — the events are permanently lost. Configure wal_keep_size appropriately and monitor CDC lag.

4. High-frequency outbox messages without optimization

Polling every second on a table with millions of processed rows generates unnecessary load. Index the processed_at column (or use an indexed status column), limit the relay's batch size per run, and consider migrating to CDC as volume grows.

Rule of thumb: the outbox table should always have fewer than 1,000 unprocessed rows under normal operation. If the number grows, something is wrong with the relay — investigate before the lag causes noticeable inconsistencies.

Related architectures and patterns

• Saga Pattern
• Message Queues
• Idempotency
• Event-Driven Architecture
• CQRS

The Outbox Pattern is almost always used together with Sagas: each saga participant saves its result and inserts the event that triggers the next step into the outbox, within the same local transaction. This guarantees the saga's flow doesn't break due to a network failure between the commit and the publication.

In Event-Driven Architecture, the Outbox is the canonical solution to the dual-write problem: the application can't guarantee atomicity by writing to the database and publishing to the broker in separate operations.

CQRS with Event Sourcing frequently uses the Outbox or the event store itself as the publishing mechanism: the event is the source of truth for the state and also the trigger for updating the Read Model — but it's still necessary to guarantee the event reaches the broker reliably.