System Design

Rate Limiting

Controlling the rate of requests a client can make to a service within a given period — protecting against abuse, low-intensity DDoS, and excessive consumption of shared resources.

Intent

Rate limiting imposes a ceiling on the rate at which a client can consume a service, guaranteeing that no client — by mistake or bad faith — can monopolize resources and degrade the experience for everyone else.

Without rate limiting, a single client that fires off an infinite request loop, an aggressive scraping bot, or a volumetric attack can completely saturate the service's processing capacity, making it unavailable to everyone. Rate limiting is the first line of defense against this kind of overload — simpler and more effective than sizing the infrastructure to absorb any possible spike.

Problem

Services exposed to multiple clients face challenges that rate limiting solves directly:

  • Abuse and scraping: bots that make thousands of requests per second to extract data, brute-force passwords, or simply overload the service.
  • Buggy clients: a client stuck in a retry loop without backoff can generate more requests than the entire normal client base combined.
  • Disproportionate consumption of paid APIs: in a per-request cost model, a single client with erratic behavior can generate unexpected costs for the provider.
  • Protecting expensive endpoints: PDF report generation, data export, mass email sending, calls to paid external APIs — operations that consume significant resources need to be limited independently of the rest.

How it works

Main algorithms

Token Bucket

A bucket stores up to N tokens (maximum capacity). Tokens are added at a fixed rate (e.g., 10 tokens per second). Each request consumes one token; if the bucket is empty, the request is rejected. If the bucket is full and no request arrives, tokens accumulate up to the maximum limit.

Advantage: allows legitimate bursts — an idle client accumulates tokens and can make N requests in a burst when it needs to. Disadvantage: a burst of up to N can happen at any moment, which can overload a downstream service that isn't prepared for spikes.

Leaky Bucket

Requests enter a queue and are processed at a constant rate (e.g., 10 req/second), independent of the arrival rate. If the queue fills up, new requests are rejected. The "faucet" leaks at a constant pace — hence the name.

Advantage: smooths bursts, protecting the downstream system from spikes. Disadvantage: valid burst requests wait in the queue, increasing perceived latency.

Fixed Window Counter

Counts requests within a fixed window (e.g., 100 req per minute, window resets every minute at second :00). Simple to implement with an atomic counter and TTL.

Classic problem: a client can make 100 requests in the last 5 seconds of one window and 100 in the first 5 seconds of the next — 200 requests in 10 seconds, double the intended limit.

Sliding Window Log

Records the timestamp of each request in a list. For each new request, it removes timestamps older than N seconds from the list and counts what remains. If the count exceeds the limit, it rejects the request. More precise than Fixed Window, but consumes more memory (one timestamp per request per client).

Sliding Window Counter

An efficient hybrid: keeps the counter for the current window and the previous window, calculating an approximation proportional to the overlap with the previous window. E.g.: if the previous window had 80 requests, the current window has 20, and we're 30% into the current window, the estimated count is 80 × 0.7 + 20 = 76.

Widely used in production: constant memory (two counters per client), without Fixed Window's burst vulnerability, and with good accuracy.

Token Bucket bucket: 6/10 tokens +10 tokens/s request -1 token 200 OK empty bucket 429 Allows burst: idle client accumulates tokens and can make N requests in a burst Fixed Window — Burst Problem Window 1 (0:00 – 0:59) Window 2 (1:00 – 1:59) 1:00 100 req 0:55–0:59 100 req 1:00–1:04 ✓ 100 accepted ✓ 100 accepted 200 requests in 10 seconds! Limit: 100/min — but both windows validate Solution: Sliding Window Counter approximates the count with weighted overlap from the previous window HTTP response headers (RFC 6585 + industry standard) HTTP/1.1 429 Too Many Requests X-RateLimit-Limit: 100 X-RateLimit-Remaining: 0 Retry-After: 42
Token Bucket allows controlled bursts (accumulated tokens). Fixed Window Counter has the classic vulnerability of accepting double the limit in two adjacent seconds at the window boundary — Sliding Window Counter solves this.

Rate limiting dimensions

Rate limiting can be applied across multiple dimensions, combined depending on the use case: 

  By IP            — basic protection against bots; penalizes users
                     behind corporate NAT (see pitfalls)
  By user          — limit per authenticated account; fairer
  By API key       — granular control for integrations and partners
  By endpoint      — different limits for /login (5/min) vs
                     /search (100/min)
  By tenant        — in multi-tenant SaaS, limit per organization
  Combined         — by IP AND by user AND by endpoint simultaneously

Implementation with Redis

Redis is the standard for rate limiting in distributed environments (multiple server instances). The counter needs to be shared — if each instance keeps its own counter, a client can make N requests per instance, multiplying the limit by the number of instances. 

  -- Fixed Window with Redis (Lua script for atomicity)
  local key = "rl:" .. client_id .. ":" .. window_start
  local count = redis.call("INCR", key)
  if count == 1 then
    redis.call("EXPIRE", key, window_size_seconds)
  end
  if count > limit then
    return 429
  end
  return 200

When to use

  • Public APIs: any API exposed to the internet should have rate limiting. Without it, a single misconfigured or malicious client can bring down the entire service.
  • Authentication endpoints: aggressive limiting on /login, /forgot-password, and OTP verification is the basic protection against brute force — 5 attempts per minute per IP is a reasonable starting point.
  • Expensive operations: PDF report generation, data export, mass email sending, calls to paid external APIs. Rate limiting protects both operational cost and availability.

When to be careful

  • Overly aggressive rate limiting on internal APIs: internal services that call each other at high frequency can suffer artificial degradation from poorly sized limits. Monitor and calibrate based on real production data before tightening limits.
  • Limiting only by IP in corporate contexts: hundreds of employees at a company may share a single public IP — rate limiting by IP treats them as a single client. Prefer limiting by authenticated user in these contexts.

Pros and cons

Pros

  • Protects service availability against abuse, client bugs, and low-intensity volumetric attacks.
  • Guarantees fairness among clients: no individual client monopolizes shared resources.
  • Protects operational costs in APIs charged per request or per expensive operation.
  • Relatively simple to implement with Redis and idiomatic in API Gateways (Kong, AWS API Gateway, Nginx).

Cons

  • Legitimate clients can be penalized if limits are poorly sized or if they share an IP with other clients.
  • Not sufficient defense against high-intensity volumetric DDoS attacks — for that, WAF and CDN with DDoS protection are needed.
  • Requires distributed state (Redis) in environments with multiple instances, adding latency and infrastructure dependency.
  • Fixed limits don't adapt to legitimate load variations — a more sophisticated algorithm (e.g., Token Bucket with a generous burst) is fairer.

Common pitfalls

1. Fixed Window with burst at the boundary

The most classic rate limiting problem: 100 requests in the last 5 seconds of a window plus 100 in the first 5 seconds of the next add up to 200 in 10 seconds, even though the limit is 100/minute. Use Sliding Window Counter to mitigate this — it's the option with the best balance between accuracy, memory cost, and implementation simplicity.

2. Rate limiting by IP on corporate networks

A company with 500 employees may have all of them going out through the same public IP range via NAT. Aggressive rate limiting by IP causes an employee making normal requests to be blocked because of a coworker's behavior. Always offer limits by authenticated user as an alternative.

3. No informative headers in the 429 response

A 429 Too Many Requests response without informative headers leaves the client in the dark: it doesn't know how many requests remain, when the limit resets, or how long to wait. The result is retry loops with arbitrary backoff or, worse, immediate retries that make the overload worse.

Always include: Retry-After (seconds until it can try again), X-RateLimit-Limit (configured limit), X-RateLimit-Remaining (requests remaining in the current window), and X-RateLimit-Reset (Unix timestamp of the next reset).

4. Rate limiting only at the edge without protecting internal services

If rate limiting exists only at the API Gateway, internal services that call each other directly (without going through the gateway) remain unprotected. A bug in service A that makes calls in a loop to service B can take down B even with the gateway protecting external traffic. Consider service-level rate limiting for critical internal communication.

Related architectures and patterns

The API Gateway is the most common place to implement rate limiting in microservices — centralizing the policy at a single point instead of duplicating the logic in each service. Kong, AWS API Gateway, NGINX, and Traefik offer rate limiting as a configurable native feature.

Circuit Breaker and Rate Limiting are complementary: Rate Limiting protects a service from overload caused by too many clients, while Circuit Breaker protects a client from continuing to call a service that's already failing. Together, they form a robust resilience layer.

Retries after receiving a 429 should respect Retry-After and be idempotent — the server must be prepared to receive the same request more than once without unwanted side effects.