System Design

Observability

The ability to understand a system's internal state from its external outputs — without needing to deploy new code to investigate a problem. The three pillars are Logs, Metrics, and Distributed Tracing.

Intent

Observability is the property of a system that lets you infer its internal state solely by analyzing the outputs it produces. In production you don't have access to a debugger — what you have are logs, metrics, and traces. Without proper instrumentation, investigating an incident becomes guesswork.

The term comes from control theory: a system is "observable" if its state can be determined from its external outputs over time. In the context of software, this means that when something goes wrong in production, you can answer: what happened? (logs), how often and how intensely? (metrics), and where exactly in the flow did it occur? (tracing). Without all three answers, the investigation operates in the dark.

Problem

Modern distributed systems — with dozens of microservices, synchronous and asynchronous calls, queues, caches, and databases — are inherently opaque. A user request can cross eight services before returning a response; when it fails, the cause could be in any one of them or in the interaction between them.

  • Uncorrelated logs: when each service writes independent logs without a common identifier, correlating the behavior of a specific request across services requires hours of manual grepping.
  • Average-only metrics: average latency can look healthy while 1% of users experience 10-second timeouts. Percentiles (p95, p99) reveal what the average hides.
  • No tracing, no bottleneck localization: knowing that "the order service got slow" is different from knowing that "the stock-lookup SQL query inside the order service took 800 ms on that specific call."

How it works

Logs — discrete event records

Logs are event records with a timestamp, severity level, and contextual data. JSON-structured logs are preferable to free text because they allow filtering, indexing, and querying in tools like Elasticsearch, Loki, or CloudWatch Logs Insights.

Every log should include a Correlation ID (also called a TraceID) — a unique identifier for the original request that stays constant as it crosses every service. Without this ID, correlating logs from different services is unfeasible at any real traffic volume. 

// Structured log — filterable and indexable
{"timestamp":"2024-01-15T14:23:01Z","level":"ERROR",
 "service":"order-svc","msg":"payment timeout",
 "traceId":"abc-123","userId":"u-456","durationMs":5032}

// Free-text log — hard to filter programmatically
[ERROR] 2024-01-15 14:23:01 Payment timeout for user u-456

Severity levels: DEBUG (development details), INFO (normal business events), WARN (abnormal but recoverable situation), ERROR (failure that requires attention). In production, use INFO as the default — DEBUG generates disproportionate volume and cost.

Metrics — aggregated numerical values

Metrics are numerical observations collected over time and aggregated into time series. They're more efficient than logs for trend analysis and alerting. The three fundamental types:

  • Counter: a monotonic value that only increases. E.g.: total requests, total errors. Useful for calculating rates (rate() in Prometheus). Never decreases — it only resets when the process restarts.
  • Gauge: an instantaneous value that goes up and down. E.g.: open connections, memory used, queue size. Captures the system's current state.
  • Histogram: distributes observations into configurable buckets and allows calculating percentiles (p50, p95, p99). Essential for latency, because the average hides the tail — p99 can be 100x the average in systems with a long-tail distribution.

The RED Method (by Tom Wilkie) defines the three essential metrics for any service: Rate (requests per second), Errors (error rate), and Duration (latency distribution). Any SLO starts here. The most common open source stack is Prometheus for collection and Grafana for visualization.

Distributed Tracing — request tracking

Distributed Tracing follows a request as it crosses multiple services. Each unit of work — an HTTP call, a database query, a cache operation — is a Span: an interval with a start time, duration, attributes, and a reference to the parent span. Related spans form a Trace.

The TraceID is propagated between services via HTTP headers. The current standard is W3C Trace Context with the traceparent header: 

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^ ^^
             ver        trace-id (16 bytes)        parent-span-id  flags

OpenTelemetry is the current standard for instrumentation — a vendor-neutral API and SDK that exports telemetry to any backend (Jaeger, Zipkin, Grafana Tempo, Datadog, Honeycomb). Adopting OpenTelemetry from the start avoids lock-in to observability tools.

At high volume, collecting 100% of traces is prohibitive in cost and overhead. Head-based sampling decides at the start of the request (e.g., 1–10% of requests); tail-based sampling decides at the end, retaining 100% of traces with errors and discarding the healthy ones — smarter, more complex to implement.

LOGS {"level":"INFO", "msg":"request started", "traceId":"abc-123"} {"level":"WARN", "msg":"slow query 800ms", "traceId":"abc-123"} {"level":"ERROR", "msg":"timeout 5032ms", "traceId":"abc-123"} {"level":"INFO", "status":500,"service":"order", "traceId":"abc-123"} DEBUG · INFO · WARN · ERROR Structured JSON + Correlation ID METRICS p99 p95 p50 0 time → p99 p95 p50 (median) Counter · Gauge · Histogram RED Method: Rate · Errors · Duration DISTRIBUTED TRACING API GW Order Auth DB 320 ms 296 ms 88 ms 44 ms TraceID: abc-123 4 spans · header traceparent W3C OpenTelemetry · Jaeger · Zipkin Per-request span waterfall 1–10% sampling at high volume TraceID: abc-123 The same TraceID appears in Logs + Metrics + Trace — correlation is what turns three silos into real observability
The three pillars of observability correlated by the same TraceID. Logs record discrete events, Metrics reveal trends and percentiles, Distributed Tracing pinpoints the exact bottleneck in the flow between services.

Correlation between the three pillars

The three pillars are valuable individually, but real observability emerges when they're correlated. The same TraceID that appears in the error log lets you navigate directly to the trace of that specific request, see which spans were slow, and then go to that service's latency metrics at the same instant. This fluid navigation (log → trace → metric) is what distinguishes observability from traditional monitoring.

When to use

  • Any service in production: the three pillars are a prerequisite, not a differentiator. Start with structured logs and RED metrics; add tracing later.
  • Microservices and distributed systems: tracing is especially valuable when a request crosses multiple services. It's impossible to manually correlate the behavior of 10 services without a TraceID and a span waterfall.
  • When defining SLOs: the pillars provide the data to measure SLIs — p99 latency, error rate, availability — and to verify whether the objectives are being met.

When to avoid (or size carefully)

  • Tracing with 100% sampling at high volume: on services with 10k req/s, storing every trace is prohibitive. Use head-based or tail-based sampling. Alerts and metrics don't need sampling.
  • DEBUG logs continuously in production: this level can generate GBs per hour and disproportionate ingestion cost. Use INFO as the default; DEBUG only during specific investigation windows.

Pros and cons

Pros

  • Drastically reduces MTTR: incidents are investigated with concrete data, not guesswork.
  • Enables SLOs based on real data — p99 latency, error rate, measured availability.
  • Tracing reveals bottlenecks invisible in isolated service metrics.
  • OpenTelemetry as a standard avoids lock-in to an observability vendor.

Cons

  • Cost: storing logs and traces at high volume can be significant — requires active retention and sampling management.
  • Instrumentation overhead: tracing adds latency (~1 ms) and CPU consumption per request. Sampling mitigates but doesn't eliminate it.
  • Setup complexity: configuring OpenTelemetry, Prometheus, Grafana, and Jaeger and correlating everything has a meaningful learning curve.
  • Retroactive instrumentation is much more expensive: adding observability to legacy code without dependency injection is orders of magnitude more work than instrumenting while developing.

Common pitfalls

1. Free-text logs

Unstructured logs — [ERROR] Payment failed for user 12345 — are readable for humans but impossible to filter programmatically. When you need "all errors for user X in the last 2 hours" on a system with 1k req/s, grepping free text isn't a viable strategy. Use structured JSON with indexable fields from day one.

2. Metrics without percentiles — the average trap

Average latency can look healthy while 1% of users suffer 30-second timeouts. This happens because the average is sensitive to volume (many fast requests dilute the slow ones) but insensitive to the tail. Always monitor p95 and p99. In user-facing SLAs, the relevant percentile is usually p99 — the "worst 1%" is the most impacted user.

Practice: set alerts on p99 latency and on the error rate per route. Average latency as an alert generates both false negatives (didn't alert when it should have) and false positives (alerted on isolated outliers).

3. Tracing without sampling — cost and overhead

Collecting 100% of traces on a service with 5,000 requests per second means storing 432 million traces per day. Beyond the storage cost, exporting traces itself adds network overhead. Use sampling: head-based (1–10%) for general cases; tail-based (retains 100% of traces with errors) for failure diagnosis.

4. Observability added after the incident

Instrumentation is much cheaper when implemented alongside the code. Adding observability after the fact — especially in legacy code without dependency injection — is orders of magnitude more expensive. Treat observability as a functional requirement: the acceptance criteria for any service should include "has structured logs, RED metrics, and traces with a propagated TraceID."

Related architectures and patterns

SLA, SLO, and SLI depend directly on observability: it's not possible to measure whether a p99 latency SLO is being met without instrumented histogram metrics and configured alerts.

Circuit Breaker is more effective when combined with metrics: the circuit's state (open/closed/half-open) should be an observable metric, and opening decisions should be visible in logs and traces.

API Gateway is a natural instrumentation point: positioned at the edge, it can inject a TraceID into every incoming request, collect RED metrics from all downstream services, and centralize access logs.

Message Queues add complexity to tracing: the propagation context (TraceID) needs to be serialized into the message metadata so the consumer can continue the same trace from the producer, even when processing the message minutes later.