System Design

Load Balancing

Distributes requests across multiple instances of a service to maximize throughput, minimize latency, and guarantee availability. Understanding distribution algorithms, operating layers, and failure behavior is essential for any system that needs horizontal scale.

Intent

A load balancer distributes requests across multiple instances of a service, preventing any individual instance from becoming overloaded while others sit idle. It's the central element that makes scale out possible in practice: without a single entry point that distributes load, scaling horizontally is just having multiple instances that aren't accessible in a unified way.

Besides distributing load, the load balancer fulfills additional functions: it detects unhealthy instances and stops sending traffic to them, drains connections from instances being removed without interrupting in-flight requests, and can route traffic based on request attributes — path, headers, cookies — for load balancers operating at the application layer.

Problem

Without a load-distribution mechanism, a service's horizontal growth creates problems that go beyond capacity:

  • Uneven distribution: without coordination, clients tend to always connect to the same instances — whether due to cached DNS, persistent connections, or manual configuration. Some instances become saturated while others sit idle.
  • No automatic failover: if an instance goes down, the traffic that reached it needs to be redirected automatically. Without a load balancer, the client receives an error and needs its own retry and service-discovery logic.
  • Zero-downtime deploy impossible: to update instances without interruption (rolling update, blue-green), a component that can remove old instances from the pool while adding new ones is required — a natural function of a load balancer.
  • Single entry point without a bottleneck: clients need a single, stable address for the service, regardless of how many instances exist behind it. The load balancer provides that virtual address and hides the internal topology.

How it works

  Client A ──┐
             │
  Client B ──┼──▶ ┌─────────────────────┐      ┌────────────┐
             │    │ Load Balancer       │─────▶│ Instance 1 │ (CPU: 30%)
  Client C ──┘    │                     │      └────────────┘
                  │ distribution        │      ┌────────────┐
                  │ algorithm +         │─────▶│ Instance 2 │ (CPU: 28%)
                  │ health checks       │      └────────────┘
                  │                     │      ┌────────────┐
                  │ (keeps only healthy │─────▶│ Instance 3 │ (CPU: 32%)
                  │  instances in pool) │      └────────────┘
                  └─────────────────────┘
                       ▲ single, stable
                       │ address for
                       │ clients

Distribution algorithms

The algorithm determines which instance receives each new request. The choice directly impacts load distribution and failure behavior:

  • Round Robin: distributes in a cyclic sequence — request 1 goes to instance 1, request 2 to instance 2, and so on, wrapping back to the start. Simple and effective when servers are homogeneous and requests have similar cost. Doesn't account for each instance's actual load.
  • Weighted Round Robin: the same as Round Robin, but with configurable weights per instance. An instance with weight 3 receives three times the requests of one with weight 1. Useful when instances have different capacities (e.g., during a rolling update where the new instance is still warming up).
  • Least Connections: routes to the instance with the fewest active connections at the moment. Better than Round Robin for requests of variable duration — avoids overloading an instance still processing long requests while others sit idle.
  • IP Hash: applies a hash function to the client's IP to determine the instance. The same client always reaches the same instance as long as the pool doesn't change. Guarantees session affinity without needing a cookie, but breaks distribution when an instance enters or leaves the pool — and every redistribution can send a client to a different instance.
  • Random: picks an instance at random. Surprisingly effective at sufficient scale — the law of large numbers produces an approximately uniform distribution. Requires no state on the load balancer.

Operating layers: L4 vs L7

Load balancers can operate at different layers of the OSI model, with distinct trade-offs between performance and functionality: 

  L4 — Transport Layer (TCP/UDP)
  ─────────────────────────────────────────────────────────────────
  Operates over TCP/UDP connections, without inspecting content.
  Routing based on: source/destination IP + port.
  Faster: lower processing overhead.
  Limitation: can't see path, headers, cookies, or HTTP body.

  Example: AWS NLB, HAProxy in TCP mode.

  L7 — Application Layer (HTTP/HTTPS)
  ─────────────────────────────────────────────────────────────────
  Inspects and understands the HTTP protocol.
  Routing based on: path (/api/* → service A, /static/* → CDN),
                    headers (Host, Authorization, Accept-Language),
                    cookies (for sticky session via cookie),
                    HTTP method, query string.
  More powerful: enables smart routing and TLS termination.
  Higher overhead: needs to parse the HTTP protocol.

  Example: AWS ALB, nginx, Traefik, HAProxy in HTTP mode.

Health checks

The load balancer needs to know which instances are healthy so it doesn't send traffic to failing instances. There are two complementary strategies:

  • Active (periodic probing): the load balancer sends periodic requests to a health endpoint on each instance (e.g., GET /health) and waits for a response. If the instance doesn't respond or returns an error within the configured timeout for N consecutive attempts, it's removed from the pool. When it responds successfully again, it's re-added.
  • Passive (detection via real errors): the load balancer monitors responses to clients' real requests. If an instance repeatedly returns 5xx errors, it's marked unhealthy and temporarily removed. Complements the active health check without adding synthetic traffic.

Sticky sessions (session affinity)

Sticky sessions guarantee that all of a client's requests always reach the same instance. The most common mechanism is a cookie inserted by the load balancer in the first response, identifying the instance that served that session. At L7, the load balancer reads that cookie on subsequent requests and routes to the same instance.

Sticky sessions are necessary for stateful applications that keep state in local memory. However, they create two problems: they break load balancing (a popular instance accumulates more connections than the others) and create a failure point (if the "stuck" instance goes down, the client loses its session anyway). The correct solution for stateful applications is to externalize state — not use sticky sessions as a crutch.

When to use

  • Multiple instances of a service: whenever there's more than one instance of the same application serving traffic, a load balancer is needed to distribute load and provide a single entry point.
  • High availability: the load balancer detects failures via health check and stops sending traffic to instances with problems, without manual intervention and without perceptible downtime for the user.
  • Zero-downtime deploy: rolling updates, blue-green deployments, and canary releases depend on the load balancer to gradually add new instances to the pool and remove old ones after draining in-flight connections.
  • Context-based routing (L7): when different paths or subdomains need to be routed to distinct services, an L7 load balancer eliminates the need for a separate API gateway for simple routing tasks.

When to avoid or adapt

  • Stateful applications without a plan to externalize state: adding a load balancer without making the application stateless (or without deliberately configuring sticky sessions) produces incorrect behavior. The load balancer isn't the solution — it's the exposure of the problem.
  • Volume so low the LB becomes unnecessary overhead: for internal services with very little load and tolerance for downtime, the operational overhead of maintaining a load balancer may not be justified. A simple DNS with multiple A records may be enough for service discovery without guaranteed availability.

Pros and cons

Pros

  • Distributes load automatically, without per-instance manual intervention.
  • Automatic failover via health checks: failing instances are removed from the pool without perceptible downtime.
  • Enables zero-downtime deploy with rolling update and blue-green.
  • L7 load balancer allows sophisticated routing by path, header, and cookie.
  • Centralizes TLS termination, simplifying certificates for the application service.
  • Hides the internal topology from clients: adding or removing instances is transparent to whoever consumes the service.

Cons

  • Single point of failure if there's no redundancy in the LB itself.
  • Additional latency overhead (usually milliseconds, but relevant in high-frequency calls).
  • Sticky sessions degrade load balancing and create dependency on a specific instance.
  • Shallow health checks can leave "alive" instances in the pool despite a logical defect.
  • Timeout, draining, and retry configuration requires attention — config errors cause subtle failures in production.

Common pitfalls

1. Sticky session masking local state

Using sticky session to "solve" the problem of a stateful application is a temporary fix that creates two new problems: load distribution stops being uniform (popular instances accumulate more sessions) and fault tolerance disappears (if the "stuck" instance goes down, the user loses their session anyway). Sticky session is acceptable as a transitional solution, but the correct fix is to externalize state — Redis for session, object storage for files — making the application genuinely stateless.

2. Shallow health check that doesn't detect real failures

A health check that only verifies the TCP port is open can declare an instance healthy even when it's in an error state: disconnected database, exhausted connection pool, timed-out external dependency. The health check endpoint should verify the instance's critical dependencies — database connectivity, disk space, queue status — and return an error when the instance isn't fit to serve real requests.

Rule of thumb: separate the liveness health check (is the instance alive?) from readiness (is the instance ready to receive traffic?). The load balancer should use readiness to decide whether to put the instance in the pool.

3. Forgetting connection draining when removing instances

Removing an instance from the pool immediately, without draining, cuts off all in-flight requests — users receive 502/504 errors for no apparent reason. Draining (or deregistration delay) instructs the load balancer to stop sending new requests to the instance, but wait for existing connections to finish before removing it definitively. The draining time should be longer than the service's maximum request timeout.

4. Single point of failure in the load balancer itself

Having a single load balancer eliminates the single point of failure in application instances, but creates a new single point of failure in the LB itself. The solution is to use an active/passive pair with automatic failover via VRRP, or delegate to a managed load balancer that already provides redundancy internally — such as AWS ALB or NLB, which are distributed by nature and scale automatically.

Related architectures and patterns

Horizontal vs Vertical Scalability is the architectural context where load balancing fits in: the load balancer is the operational mechanism that makes scale out possible. Without it, adding instances horizontally creates capacity without a unified access point.

Event-Driven Architecture complements load balancing in asynchronous systems: while the load balancer distributes synchronous HTTP requests, an EDA with message queues distributes asynchronous work among consumer workers — and those workers can be scaled horizontally behind a load balancer or managed directly by the message broker.

Caching and load balancing interact directly: a local cache per instance is problematic because different instances can have caches with different states. The solution is a centralized distributed cache (Redis, Memcached) that all instances behind the load balancer access in a shared way.