System Design

Eventual vs Strong Consistency

In distributed systems, replicated data can temporarily fall out of sync between nodes. Consistency defines how "current" the read a client receives is after a write — and the choice between models has direct consequences for availability, latency, and code complexity.

Intent

Consistency defines the guarantee about the value a client reads after a write has been confirmed on some node of the distributed system. In a database with replicas, the write happens on the primary but reads can be served from any replica — and asynchronous replicas can be behind. The question is: how much "behind" is acceptable for your use case?

This isn't a technical problem with a single right answer: it's a business trade-off. A bank balance system and a Twitter timeline feed have completely different answers to "how stale can a read be?"

Problem

When a system distributes data across multiple nodes to gain availability and performance, every write needs to be propagated to the replicas. That propagation has a cost: it takes time, it can fail, and it creates a window during which different replicas have different versions of the data.

  • If you wait for every replica to confirm before responding to the client (strong consistency): the write is slower, since it depends on the slowest replica in the set. If any replica is unreachable, you may need to refuse the write or the read.
  • If you confirm to the client as soon as the primary accepts (eventual consistency): the write is fast, but immediate reads from replicas may return the old value for a few milliseconds — or seconds under high load.
  • The problem is that both behaviors look correct in the development environment (local database, zero latency between nodes) and only become visible in production, under load, with real replicas.

How it works

Strong Consistency

After a write is confirmed, any subsequent read on any node of the system returns the updated value. To achieve this, the system needs some coordination mechanism: either the write is synchronous to every replica (all confirm before the client gets ok), or it uses a consensus protocol like Raft (used in etcd, CockroachDB, TiDB) or Paxos (used in Google Spanner, Zookeeper).

The cost is write latency: you pay the RTT to the slowest or geographically farthest replicas on every write. In multi-region systems with replicas in Europe and Asia, this can mean 150–300 ms of guaranteed write latency.

Eventual Consistency

After a write on the primary node, replicas will receive the update eventually — with no guarantee of when. Under normal network conditions, the lag is usually a few milliseconds to a few seconds. Under high load or network issues, it can be more.

The system confirms the write to the client as soon as the primary accepts it, without waiting for the replicas. Reads from replicas can return the old value until propagation completes. This allows higher availability and lower write latency, at the cost of potentially stale reads.

Intermediate models

Between the two extremes there are more specific guarantees that allow for more precise trade-offs:

  • Read-Your-Writes (read-after-write): you always read what you just wrote, even if other clients still see the old value. Implemented via sticky reads (the client always reads from the primary after writing) or causality tokens (the client carries the minimum version the server must have to serve the read).
  • Monotonic Reads: once the client has read a value, it will never read an older value. Important in systems where the user makes multiple reads — seeing data "go back in time" is confusing and creates bugs that are hard to reproduce.
  • Causal Consistency: operations that have a causal relationship (B depends on A) are seen in the correct order by every node. "You liked a post I haven't seen yet" is a causal consistency violation. Stronger than eventual, weaker than strong.
Eventual Consistency Node A (primary) Node B (replica) WRITE v=2 t=0ms READ v=1 stale! t=10ms async propagation ~50ms sync READ v=2 fresh! t=60ms time High availability · Low write latency Immediate read may return a stale value Use when: timeline, feeds, likes, catalog, feature flags CAP: keeps A (availability) during a partition Strong Consistency Node A (primary) Node B (replica) WRITE v=2 t=0ms waits for replicas synchronous write commit ack confirmed WRITE OK t=75ms READ v=2 guaranteed ✓ READ v=2 guaranteed ✓ Any read returns the updated value Write latency = replica RTT (+50–300 ms multi-region) Use when: balance, inventory, voting, critical stock
Eventual Consistency (left): write confirms fast, replica may return the old value for a few milliseconds. Strong Consistency (right): write only confirms after every replica syncs — higher latency, but any read on any node returns the updated value.

CAP Theorem

The CAP Theorem (Brewer, 2000) states that during a network Partition (P — nodes can't communicate with each other), a distributed system needs to choose between:

  • Consistency (C): every read returns the most recently written value, or an error. The system refuses to serve reads from nodes that might be outdated during the partition.
  • Availability (A): every request gets a response (possibly with stale data). The system keeps operating even if the nodes are on different sides of the partition.

Strong consistency → the system chooses C, sacrificing A during a partition. Eventual consistency → the system chooses A, keeping availability but possibly returning stale data.

PACELC — the more complete theorem

CAP only talks about partitions. PACELC (Daniel Abadi, 2012) adds: even without a partition, there's a trade-off between Latency (L) and Consistency (C). 

PACELC: if Partition → choose between A and C
        else (E) → trade-off between L and C

Examples:
  DynamoDB:   PA/EL  — prioritizes availability and low latency
  Cassandra:  PA/EL  — same; consistency configurable per operation
  PostgreSQL: PC/EC  — prioritizes consistency; more latency on writes
  Spanner:    PC/EC  — strong global consistency; relatively high latency
  MongoDB:    PA/EC  — available during partition; configurable (w:majority)

Many NoSQL databases let you configure the level per operation (DynamoDB ConsistentRead, Cassandra ConsistencyLevel), allowing different trade-offs for different reads within the same application.

When to use each model

  • Strong consistency: bank balances, e-commerce inventory with critical stock, voting systems, seat reservations, any operation where reading stale data has a real business consequence (selling the same seat twice, authorizing a withdrawal of money that doesn't exist).
  • Eventual consistency: social media timelines, view and like counters (a few seconds of lag is acceptable), product catalogs (seeing yesterday's price for a few ms is harmless), feature flags, recommendations, search — any data where staleness has negligible impact.
  • Read-Your-Writes: user profile — you just updated your avatar and want to see the change. For other users, eventual is fine, but for you specifically the guarantee matters.
  • Causal Consistency: comment threads (the reply to a comment must appear after the original comment), social activity feeds where causal order matters for context.

Pros and cons

Strong Consistency — Pros

  • Simple semantics: you never have to wonder "is this data up to date?"
  • Essential for financial operations and critical inventory.
  • Makes reasoning about the code easier: no need for "eventual retry" or "stale-read compensation" logic.

Strong Consistency — Cons

  • Higher write latency: the system waits for confirmation from every (or a majority of) replicas.
  • Lower availability during partitions: if replicas become unreachable, the system may refuse operations.
  • Doesn't scale well geographically: synchronous replication across regions adds 100–300 ms of write latency.

Eventual Consistency — Pros

  • High availability: the system keeps operating even with temporarily unreachable replicas.
  • Low write latency: confirms without waiting for replicas.
  • Scales well geographically: asynchronous replicas in any region with no impact on write latency.

Eventual Consistency — Cons

  • Complexity in application code: you need to handle the possibility of stale reads.
  • Unpredictable reads: under high load the lag can be seconds — the design must tolerate this explicitly.
  • Conflicts in concurrent writes: two nodes that accepted conflicting writes need a resolution strategy (LWW — Last Write Wins, CRDT, or manual resolution).

Common pitfalls

1. Assuming strong consistency on a database with asynchronous replicas

The most common mistake: the code writes to the database and immediately does a read — but the read goes to a read replica that hasn't processed the write yet. The user sees the old data. This looks like an intermittent bug because the lag varies, and in development (local database with no replicas) the bug never shows up.

Practice: in operations where Read-Your-Writes is needed, read from the primary (not the replica) immediately after writing. Modern ORMs have configuration for this (e.g., after_write_primary_reads in Prisma, read_preference=primary in MongoDB, or simply using the same connection as the write).

2. Treating "eventual" as "immediate in practice"

In development with a local database, propagation is instantaneous. In production, with real replicas and real load, the lag can be seconds. Architectures that assume "eventual takes less than 100 ms" work fine most of the time and fail silently when load increases or the network degrades. The design must tolerate lag of seconds, not milliseconds.

3. Not documenting the API's consistency model

If your read endpoint uses eventual consistency, document that explicitly. The caller needs to know it may receive data that's up to X seconds old, so it can decide whether that's acceptable for its use case. Undocumented consistency creates surprises in production and bugs that are hard to diagnose in integrations.

4. Confusing consistency with durability

Consistency defines how current the read you receive is. Durability defines whether a write survives failures (server crash, disk failure). They're orthogonal dimensions: a system can have eventual consistency and high durability (DynamoDB with replication across 3 zones) or strong consistency with configurable durability (MongoDB with w:1 vs w:majority). Confusing the two leads to mistaken architecture decisions.

Related architectures and patterns

Database Replication is the physical mechanism that implements the consistency models: synchronous replication enables strong consistency; asynchronous replication results in eventual consistency. Understanding consistency is essential to configuring replication correctly.

CQRS explicitly separates the write model (strong consistency in the Command Store) from the read model (eventual consistency in the projections/read models). This makes the consistency trade-off explicit in the architecture instead of implicit in the database configuration.

SQL vs NoSQL is strongly related to consistency: traditional SQL databases offer strong consistency by default (ACID); NoSQL databases historically prioritized availability and eventual consistency (BASE), although many modern ones (CockroachDB, Spanner, DynamoDB with transactions) offer configurable strong consistency.

Sharding adds another dimension: consistency across different shards is especially hard — a transaction that crosses two shards doesn't have a single source of truth, requiring distributed commit protocols to guarantee strong consistency.