System Design

Horizontal vs Vertical Scalability

Two fundamental strategies for handling load growth: scale up (more power on the same machine) and scale out (more machines doing the same work). Understanding the limits and trade-offs of each approach is one of the most impactful infrastructure decisions in any system.

Intent

Scalability is a system's ability to handle load growth without performance degradation. When a system approaches its capacity limit — whether in requests per second, volume of data processed, or concurrent connections — there are two strategies to expand that capacity: grow vertically (a more powerful machine) or grow horizontally (more machines).

Scale up (vertical) means adding resources to the same instance: more CPU, more RAM, faster disks, faster network. The application doesn't change; the infrastructure that supports it grows. Scale out (horizontal) means adding new instances of the same application and distributing load between them through a load balancer. The infrastructure doesn't need to grow monolithically, but the application needs to be able to run in multiple copies simultaneously without inconsistencies.

Problem

Every system has a capacity ceiling. As load grows — more users, more data, more calls per second — concrete symptoms appear: rising latency, timeouts, memory errors, saturated CPU. The question isn't whether this ceiling will be reached, but when and how the system will respond to it.

  • Physical limit of scale up: there's a maximum hardware available on the market. Even with unlimited budget, there's a point where a single machine can't process more load — whether due to CPU limit, I/O limit, or lock contention on shared resources.
  • Non-linear cost: doubling a machine's resources doesn't cost double. The price grows exponentially as you approach the top of the hardware line. A 256-core server costs much more than eight 32-core servers with equivalent aggregate capacity.
  • Single point of failure: with a single instance, any hardware failure or planned maintenance takes the system down. There's no natural redundancy.
  • Local state as an obstacle to scale out: applications that keep state in memory (HTTP session, local cache, temporary files) work fine with one instance, but produce incorrect behavior when multiple instances need to serve the same user.

How it works

  SCALE UP (Vertical)                   SCALE OUT (Horizontal)

  ┌────────────────────────┐                  ┌───────────────────┐
  │     Single server      │                  │   Load Balancer   │
  │                        │                  └─────────┬─────────┘
  │ CPU: 4 → 16 → 64 cores │                            │ distributes requests
  │ RAM:  8 → 32 → 256 GB  │                 ┌──────────┼──────────┐
  │ Disk: HDD → SSD → NVMe │                 ▼          ▼          ▼
  │                        │             ┌───────┐  ┌───────┐  ┌───────┐
  │  ── same instance ──   │             │ App 1 │  │ App 2 │  │ App 3 │
  └────────────────────────┘             └───────┘  └───────┘  └───────┘
                                        (same image, identical instances)

  Limit: max available hardware       Limit: cost and complexity
  Failure point: single               Failure point: distributed (resilient)
  Code change: none                   Code change: requires stateless
  Elasticity: no                      Elasticity: native auto-scaling

Stateless vs Stateful: the precondition for scale out

Scale out requires the application to be stateless: each request must be processable by any instance without depending on data stored locally on a previous instance. An application that stores user session in memory, writes files locally, or keeps stateful connections cannot be scaled horizontally without breaking behavior.

The solution is to externalize state to shared services accessible by all instances:

  • HTTP session: Redis, Memcached, or a shared database instead of local memory.
  • Temporary files: object storage (S3, GCS) or a shared network volume (EFS, NFS) instead of the local filesystem.
  • Cache: centralized Redis or Memcached — a local cache per instance is inefficient and produces inconsistencies between instances.
  • Stateful connections (WebSockets, SSE): require the load balancer to use sticky sessions or the connection state to be managed by a centralized broker, such as Redis Pub/Sub.

Elasticity: scale out's exclusive benefit

Scale out enables auto-scaling: adding instances when load rises and removing them when load drops, paying only for the capacity in use. This is especially valuable for elastic workloads — systems with predictable spikes (end-of-day traffic, marketing campaigns, monthly closing) or unpredictable ones (viral content, a competitor's outage). With scale up, the large machine remains on and billed even during low-load periods, or needs to be resized manually — an operation that usually requires a maintenance window.

When to use

  • Scale up — relational database: databases like PostgreSQL and MySQL are hard to distribute horizontally in a transparent way. Scale up is the most direct route to increase capacity — more RAM for buffers, more cores for parallel queries, faster SSDs for I/O.
  • Scale up — legacy tools and stateful applications: when the code can't be modified to support multiple instances, scale up increases capacity without an architecture change.
  • Scale up — temporary, predictable spike: if the system needs more capacity for a short, defined period, temporarily provisioning a larger machine can be simpler than configuring and operating a cluster.
  • Scale out — stateless web services: REST APIs and HTTP applications without local state are natural candidates. Any instance can answer any request, making load balancing trivial.
  • Scale out — high availability: multiple instances eliminate the single point of failure. If one instance goes down, the others absorb the load without visible interruption to the user.
  • Scale out — unpredictable or elastic traffic: auto-scaling reacts to load variations within minutes, without manual intervention and with cost proportional to actual usage.
  • Scale out — multiple geographic regions: instances in distinct regions reduce latency for global users and provide geographic redundancy.

When to avoid

  • Avoid scale up alone in critical systems: a single instance, no matter how large, is a single point of failure. For systems that can't have downtime, isolated scale up isn't enough — at least one standby instance is needed.
  • Avoid scale out without making the application stateless first: adding instances of a stateful application solves nothing and introduces hard-to-reproduce bugs, where behavior varies depending on which instance handles the request.
  • Avoid scaling before measuring: scaling is expensive, operationally and financially. Identify the real bottleneck — CPU, memory, I/O, inefficient queries — before provisioning more resources. Often the solution is code optimization or adding a cache, not more hardware.

Pros and cons

Scale up — Pros

  • No change to the application's code or architecture.
  • Operationally simple: one instance, no synchronization between nodes.
  • Zero internal latency: no serialization of calls between instances.
  • Simple ACID transactions: no need for distributed coordination.

Scale up — Cons

  • Absolute physical limit: the most powerful hardware available has a ceiling.
  • Cost grows non-linearly near the top of the line.
  • Single point of failure: any hardware failure or maintenance brings the system down.
  • No elasticity: the machine stays on and billed even during low-load periods.

Scale out — Pros

  • No theoretical capacity limit: add instances as demand requires.
  • Natural high availability: multiple instances eliminate the single point of failure.
  • Elasticity: auto-scaling adds and removes instances on demand, with cost proportional to usage.
  • Smaller, homogeneous instances are cheaper and easier to replace than top-of-line machines.

Scale out — Cons

  • Requires the application to be stateless — a code and architecture change.
  • Higher operational complexity: load balancer, health checks, connection draining, cluster configuration.
  • Distributed problems: cache consistency, shared sessions, coordination between instances.
  • The database becomes a bottleneck if it isn't scaled along with the application layer.

Common pitfalls

1. Storing state in local memory and trying to scale out later

The most frequent pitfall: an application stores user session in local memory — a HashMap, a module-level variable, a stateful singleton object — and, when the time comes to scale, behavior breaks silently. Users whose requests reach different instances lose session, see another session's data, or receive inconsistent responses. Fixing this retroactively requires refactoring state logic scattered throughout the entire application.

Rule of thumb: design the application as stateless from the start. Store state in Redis or a database even while there's still a single instance. The cost of doing it right from the beginning is much lower than refactoring later under pressure.

2. Scaling vertically to the limit without noticing the code is blocking

More CPU and RAM don't help a single-threaded application that waits for I/O synchronously and in a blocking way. The server has 32 idle cores while the main thread waits for a database query to finish or an external call to respond. Before scaling anything, identify whether the bottleneck is resource (CPU, RAM, disk) or architecture (synchronous code, N+1 queries, absence of caching). Scaling doesn't replace optimization.

3. Scaling the application without scaling the database

It's common to add application instances and discover that the database becomes the new bottleneck. Ten application instances making concurrent queries against a database that previously served one instance can saturate the connection pool, the database's CPU, or the disk. Scaling out the application layer without a strategy for the database — read replicas, an external connection pool like PgBouncer, result caching, or sharding — simply trades one bottleneck for another.

4. Treating auto-scaling as a substitute for code optimization

Auto-scaling is a tool to absorb load variations, not to compensate for inefficient code. An N+1 query that makes 500 database calls per request will cost 500 times more with ten instances than with one. Infrastructure cost grows proportionally to code waste. Monitor latency and cost per request — not just uptime and availability.

Related architectures and patterns

Load Balancing is the mechanism that makes scale out possible in practice: the load balancer distributes requests among instances, detects failures via health checks, and drains connections from instances being removed. Without a load balancer, scaling horizontally is just having multiple instances that aren't accessible in a unified way.

The decision between Monolith and Microservices is directly tied to scalability strategy. Microservices allow parts of the system to scale independently — the payment service can have more instances than the reporting service — while a monolith requires scaling everything together, even if the bottleneck is just one specific module.

Caching is often the first step before scaling: reducing load on the database and application server with caching layers can delay or eliminate the need for scale out, at the cost of dealing with cache invalidation and eventual consistency.