System Design

Caching

Storing the result of a costly operation to serve future requests faster. It reduces latency, increases throughput, and decreases load on databases and origin services — at the cost of dealing with invalidation and consistency.

Intent

A cache is a faster storage layer that keeps the results of costly operations to avoid redoing them. The premise is simple: if a piece of data was computed or fetched once and will be requested again soon, storing it close to the consumer is cheaper than fetching it from the origin again.

The benefit is threefold: lower latency (the data reaches the client faster), higher throughput (more requests served with the same capacity), and reduced load on the origin (the database, external API, or compute service receives fewer calls). Caching is one of the most impactful performance techniques available and shows up in practically every layer of a modern system.

Problem

I/O operations — database access, calls to external APIs, disk reads — are orders of magnitude slower than in-memory operations. When these costly operations are repeated for every request, the system pays the full cost even when the result hasn't changed.

  • Accumulated latency: a 50 ms query is imperceptible in isolation, but when a page makes 20 queries to assemble the response, accumulated time exceeds 1 second. If most of those queries return the same data for different users, work is being repeated unnecessarily.
  • Origin overload during spikes: an event that multiplies requests by ten multiplies database queries by ten. Without caching, the origin infrastructure needs capacity for the peak load instead of the average load.
  • Rate limits on external APIs: third-party services often impose limits on requests per second or per hour. Without caching responses, the limit is reached faster and the system starts receiving errors or blocks.
  • Repeated computations: results of aggregations, rankings, reports, and heavy transformations are direct candidates: they cost a lot to compute and rarely change on every request.

How it works

Cache hierarchy

Caching exists at multiple layers, each with distinct latency and capacity. The closer to the processor, the faster — and smaller — the cache. 

  Faster / Lower capacity
  ┌──────────────────────────────────────────────────────────────┐
  │ L1/L2/L3 CPU cache      (~1-10 ns)    managed by the CPU     │
  │ RAM                     (~100 ns)     managed by the OS      │
  │ App cache — Redis/Memcached           (~1 ms over LAN)       │
  │ CDN  (~10-50 ms, geographic edge)     managed by the CDN     │
  │ Browser cache  (0 ms on HIT)          managed by the browser │
  └──────────────────────────────────────────────────────────────┘
  Slower / Higher capacity (origin: database, API)

In practice, the developer controls the layers from the application down: in-process cache, Redis/Memcached for a distributed cache shared between instances, a CDN for assets and HTTP responses, and HTTP headers for the browser cache.

Write and read strategies

The strategy defines the relationship between cache and origin: who populates the cache, when, and with what consistency guarantee. 

  CACHE-ASIDE (Lazy Loading)              WRITE-THROUGH

  Request                                 Write
      │                                       │
      ▼                                       ▼
  ┌────────┐  HIT                         ┌────────┐
  │ Cache  │──────────────► Response      │ Cache  │◄── simultaneous write
  └────┬───┘                              └────┬───┘
       │ MISS                                  │ simultaneous write
       ▼                                       ▼
  ┌────────┐                             ┌────────┐
  │   DB   │                             │   DB   │
  └────┬───┘                             └────────┘
       │ populates cache
       ▼                                 Cache is always fresh.
  ┌────────┐──────────────► Response     Every write has double latency.
  │ Cache  │                             Cache may hold data that's
  └────────┘                             never read (memory waste).

  App controls the cache explicitly.
  Cold start: miss on the 1st request.
  Risk: stale data if TTL is too long.


  WRITE-BEHIND (Write-Back)              READ-THROUGH

  Write                                   Request
      │                                       │
      ▼                                       ▼
  ┌────────┐◄── immediate write           ┌────────┐  HIT
  │ Cache  │                              │ Cache  │──────────► Response
  └────────┘                              └────┬───┘
       │ async (batch)                         │ MISS: cache fetches origin
       ▼                                       ▼
  ┌────────┐                             ┌────────┐
  │   DB   │                             │   DB   │
  └────────┘                             └────┬───┘
                                              │ populates cache
  Maximum write performance.                  ▼
  Risk: data loss if cache crashes      ┌────────┐──────────────► Response
  before persisting to the DB.          │ Cache  │
                                        └────────┘

                                        App doesn't distinguish cache from origin.

Eviction policies

A cache has finite capacity. When it's full, some data needs to be removed to make room for new data. The eviction policy defines which data is removed:

  • LRU (Least Recently Used): removes the data accessed longest ago. It's the default policy in most implementations (Redis, Memcached) because it reflects temporal locality well — recently used data is more likely to be used again.
  • LFU (Least Frequently Used): removes the data accessed least often. Better for stable access patterns where some data is accessed consistently much more than others.
  • TTL (Time To Live): not a capacity eviction policy — it's a maximum validity. Data expires after a defined time, regardless of usage. TTL is the main mechanism for temporal invalidation and works alongside LRU/LFU, not in their place.

Caching at different application layers

  • Browser cache: controlled by HTTP headers Cache-Control, ETag, and Last-Modified. Avoids requests to the server for static assets (JS, CSS, images) and responses that haven't changed. A browser HIT costs zero — the request never leaves the user's device.
  • CDN: geographically distributed caches close to users. Ideal for static assets, public API responses, and content with a defined TTL. Reduce latency by serving from the edge instead of the origin server and absorb massive traffic spikes.
  • Application cache — Redis/Memcached: shared between instances, persists beyond a single request's lifecycle. Used for user sessions, results of costly queries, responses from external APIs, results of heavy computations, and frequently read configuration data.
  • In-process cache: in-memory structures within the application's own process (HashMap, Map). Zero network latency, but not shared between instances. Suitable for immutable lookup data loaded at startup (configuration tables, domain enumerations).

When to use

  • Data read far more than written: e-commerce product pages, user profiles, system configuration — any data with a high read-to-write ratio. The cache returns the value many times for each time it's updated.
  • Costly, repetitive operations: queries with multiple joins, aggregations over large volumes, calls to external APIs with high latency. If the result serves more than one request, caching it splits the cost of producing it among all consumers.
  • Static or semi-static content: frontend assets (JS, CSS, images), catalog responses, listings that change a few times a day. A TTL of hours or days maximizes hit rate without the risk of showing critically outdated data.
  • Protecting the origin against spikes: when the origin database or service doesn't have capacity for the peak request volume, the cache absorbs the load and serves the data without increasing the origin's capacity.

When to avoid

  • Data that changes on every request: real-time balances, prices that fluctuate by the second, financial asset positions. Caching introduces a window of inconsistency that can have real consequences.
  • Data requiring mandatory strong consistency: in simultaneous banking transactions, the balance after a debit must be reflected immediately for the next operation. Eventual consistency isn't acceptable in critical financial operations.
  • When invalidation is more complex than the original problem: if the data relates to dozens of other entities and any change to any of them invalidates the cache, invalidation logic becomes a system in itself. In these cases, optimizing the original query may be more direct.
  • Data unique per user with a very high user volume: caching highly personalized data can consume more memory than the benefit it generates, since each user has their own cache entry with a hit rate close to one.

Pros and cons

Pros

  • Drastic latency reduction: data served from memory arrives orders of magnitude faster than from the origin.
  • Higher throughput with the same hardware: the database processes fewer queries, freeing capacity for writes and uncached queries.
  • Origin protection: the cache absorbs read spikes, protecting the database and APIs from overload.
  • Cost reduction on pay-per-request APIs: rate limits and per-call costs are diluted by the cache's hit rate.
  • Partial availability: some systems are configured to serve data from the cache even when the origin is unavailable (stale-while-revalidate).

Cons

  • Eventual consistency: there's a window between the origin's update and the cache's update where the data served is outdated.
  • Invalidation complexity: deciding when and how to invalidate the cache is one of the hardest problems in software engineering. Invalidating too early cancels the benefit; too late serves stale data.
  • Cold start: on the first request or after a restart, the cache is empty and all requests hit the origin simultaneously.
  • Additional infrastructure: Redis/Memcached have operational and financial cost, and need to be monitored, maintained, and operated.
  • Hard-to-reproduce bugs: behavior that depends on cache state is difficult to test, especially when it varies between hit and miss.

Common pitfalls

1. Cache stampede (thundering herd)

When a popular key's TTL expires, all requests that arrive simultaneously at that moment find a MISS and trigger the costly operation at the same time — exactly the problem the cache was supposed to prevent, now amplified. The database may receive dozens or hundreds of identical queries in a short interval, saturating under the sudden load.

Solutions: TTL jitter (expire at TTL + a random value, spreading expirations over time to prevent all popular keys from expiring together), mutex or lock (the first request to MISS acquires a lock and populates the cache; the others wait or receive stale data temporarily), and probabilistic early expiration (proactively revalidate the cache before it expires, with increasing probability as the TTL approaches zero).

2. Stale data from forgotten invalidation

The data is updated in the database, but the cache keeps serving the old version. This happens when invalidation is treated as an implementation detail instead of part of the write operation's contract. With Cache-Aside, every write operation needs to explicitly invalidate the affected keys — and that discipline needs to be maintained as new operations are added over time.

Rule of thumb: use TTL as a safety net (guarantees stale data eventually expires) and explicit invalidation on write as the primary mechanism (guarantees expiration happens at the right moment). Never rely on just one of the two in isolation.

3. Cache as source of truth

In a poorly implemented Write-Behind, writes go to the cache but the database isn't reliably updated. If the cache goes down — due to a restart, hardware failure, or aggressive eviction — the data is lost permanently. Cache is an acceleration layer, not a persistence layer. Treat it as disposable: the system should work correctly (just slower) without it.

4. Cache keys without a namespace

A Redis instance shared across environments (dev, staging, prod) or across different features can have key collisions. The key user:123 in the dev environment points to test data; in prod, to the real user. The key product:list from one feature can overwrite that of a different feature.

The solution is to prefix every key with a namespace: prod:catalog:product:list, dev:auth:user:123. Define the naming convention, document it, and apply it consistently across all code that interacts with the cache.

Related architectures and patterns

Horizontal scalability and caching are complementary: the cache reduces load on the origin, delaying or eliminating the need to scale the database vertically. When scaling out the application layer, a distributed cache (centralized Redis) ensures all instances share the same state instead of each keeping its own out-of-sync copy.

In CQRS, the Read Model often uses caching: read projections are precomputed, query-optimized data that can be stored in Redis. Updating the Read Model via domain events is a structured form of invalidation — the event signals that the projection needs to be recomputed.

Event-Driven Architecture offers a natural mechanism for cache invalidation: when a domain event is published (ProductUpdated, InventoryChanged), specialized consumers can invalidate or recompute the affected cache entries, decoupling invalidation from the original write operation.