Idempotency — Dev Helper Library

Intent

Idempotency guarantees that an operation can be executed any number of times with the same final effect as a single execution. This turns sensitive operations — payments, order creation, message sending — into operations safe for automatic retry, without the risk of duplicating side effects.

In mathematics, a function f is idempotent when f(f(x)) = f(x) for every x in its domain. In software systems, the concept translates to operations: executing the operation once or ten times should leave the system in the same final state. The second execution is a no-op — it causes no new effects.

Problem

Networks fail. Servers restart. Timeouts happen. In all of these cases, the client doesn't know whether the operation was executed or not — and the natural response is to try again. The problem is that "trying again" for operations with a side effect can cause undesired results:

Duplicate charge: the payment client sends a charge request, doesn't get a response (network timeout), and retries. If the server processed the first request but the response was lost on the network, the retry creates a second charge. The user is billed twice for a single purchase.
Duplicate order: a payment-confirmation webhook is delivered twice by the provider (normal behavior in at-least-once systems). The order system that processes the webhook creates two orders for the same purchase.
Duplicate email: a notification job fails halfway through and is re-run. Users who already received the email before the failure receive it again.
State inconsistency: a balance-update command executed twice debits the amount twice, even though the user performed the operation only once.

The fundamental problem: the system can't distinguish a new user intent from a repetition caused by a network failure. Without idempotency, retry is a source of silent bugs.

How it works

Naturally idempotent operations

Some operations are idempotent by definition — no additional mechanism is necessary:

GET: fetching a resource doesn't change server state. Executing it a hundred times returns the same result (assuming the state hasn't changed due to other operations).
PUT (full overwrite): sets a resource's state to a specific value. PUT /users/42 {"name": "John"} twice results in the same state as a single execution.
DELETE: removes a resource. A second DELETE of a nonexistent resource may return 404 instead of 200, but the server's state is the same: the resource doesn't exist.

Operations that are not idempotent by nature include POST (each call can create a new resource), counter increments (UPDATE balance = balance + 100), and any operation whose effect depends on the number of executions.

Idempotency Key — making non-idempotent operations safe

The standard technique for adding idempotency to non-idempotent operations is the Idempotency Key: a unique identifier generated by the client for each attempt at an operation (not for each retry — for each distinct user operation).

  Client generates: idempotency-key = uuid-v4() per user operation
  (the same UUID is reused across all retries of that operation)

  First execution:
  ┌──────────┐  POST /payments                ┌──────────────┐
  │  Client  │  Idempotency-Key: a1b2-c3d4  ─►│    Server    │
  └──────────┘                                └──────┬───────┘
                                                     │ New key?
                                                     │ YES
                                                     ▼
                                                         Processes the payment
                                                         Saves the result under key a1b2-c3d4
                                                         Returns: { status: "approved", id: 99 }

  Retry (timeout on the first attempt):
  ┌──────────┐  POST /payments                ┌──────────────┐
  │  Client  │  Idempotency-Key: a1b2-c3d4  ─►│    Server    │
  └──────────┘                                └──────┬───────┘
                                                     │ New key?
                                                     │ NO — already processed
                                                     ▼
                                                         Returns the cached result:
                                                         { status: "approved", id: 99 }
                                                         (does not reprocess the payment)

The server stores the result of the first execution associated with the key. Retries with the same key return the cached result without reprocessing the operation. From the client's point of view, both responses are identical — it doesn't need to know whether it was the first execution or a retry.

At-least-once + idempotency = safety

Message queue systems with at-least-once semantics guarantee the message will be delivered at least once — but may deliver it more than once in case of consumer failure or rebalancing. An idempotent consumer processes duplicates without consequences:

The message contains a unique event ID.
The consumer records IDs already processed.
Upon receiving the message, it checks whether the ID was already processed: if so, discard; if not, process and record.

Where to apply it

Payment endpoints: the highest-risk operation — charging twice has a direct consequence for the user and the business.
Order/resource creation: any endpoint that creates an entity with a real-world effect.
Sending notifications: email, SMS, push notification — the user shouldn't receive the same message multiple times.
Received webhooks: external providers can retry webhooks — the receiver needs to be idempotent.
Queue consumers with at-least-once: duplicates are delivered; the consumer must handle them.

When to use

Any operation exposed to automatic retry: network timeout, instance failure, retry configured on the HTTP client. If the client can try more than once, the server should be idempotent.
Queue consumers with at-least-once: Kafka, RabbitMQ, standard SQS — the delivery guarantee implies the possibility of duplicates. Handling duplicates is the consumer's responsibility.
Webhooks received from external systems: Stripe, GitHub, Shopify, and most webhook providers explicitly document that they may retry. The receiver should be idempotent by design.
Operations with an irreversible external side effect: charges, debits, inventory reservations, shipments. The cost of duplicating these effects is too high to depend solely on a single successful delivery.

When to avoid (or adapt)

When multiple effects are the intent: if the user clicks "add to cart" twice, they probably want two items — not one. In that case, the Idempotency Key should be generated explicitly by the user (or the action), not automatically by the client. The design should make clear what's accidental repetition and what's intent.
Read operations: GET is already idempotent by nature. Adding an Idempotency Key mechanism to read endpoints is overhead without benefit.

Pros and cons

Pros

Safe retry: the client can try again without fear of duplicating effects, simplifying error-handling logic.
Natural resilience: network failures, timeouts, and server restarts become recoverable events, not causes of inconsistency.
Compatibility with at-least-once: queue and webhook systems can be used safely without requiring the more expensive and complex exactly-once semantics.
Auditability: storing the Idempotency Key with the result creates a record of attempted operations, useful for debugging and auditing.

Cons

Key storage: the server needs to persist processed keys and associated results, adding storage and operational complexity.
Defining the key's scope: determining what constitutes "the same operation" vs. "a new operation" can be subtle and lead to bugs if done wrong.
External side effects are hard to control: the main operation may be idempotent, but calling external services (email, SMS) also needs to be, which depends on third-party capability.
Key expiration: deciding how long to keep keys requires analyzing the expected retry pattern — too short loses protection, too long increases storage.

Common pitfalls

1. Idempotency Key with the wrong scope

Using the user ID, the product ID, or any operation data as the idempotency key instead of a UUID generated per attempt is the most common mistake. If the key is the user ID, all of that user's attempts at different operations are treated as the same operation — the second payment of the month returns the cached result of the first.

The Idempotency Key should identify a specific attempt of a specific operation. The correct approach is: the client generates a new UUID for each distinct operation the user starts, and reuses that same UUID for every retry of that operation.

2. Expiring the key too early

If the Idempotency Key's cache expires in 5 minutes but the retry can come in 1 hour — for example, via a manually reprocessed dead letter queue — the protection disappears exactly when it would be needed. The key's retention window should cover the maximum possible period between the first attempt and the last retry, including delayed-processing scenarios.

3. Partial idempotency — duplicated side effects

The main operation (creating an order) is idempotent via the Idempotency Key, but the side effects (sending a confirmation email, firing a webhook to the ERP, creating an analytics event) aren't. The result: a single order created, but three confirmation emails, three webhooks fired, and three analytics events.

Rule of thumb: when making an operation idempotent, map every side effect it generates and evaluate the idempotency of each one. External side effects that can't be made idempotent should be moved out of the retry flow or protected with a state check before execution.

4. Confusing idempotency with concurrency safety

Idempotency protects against sequential retries — the same operation executed once and then again. It doesn't protect against concurrency: two simultaneous requests with different keys (two users clicking at the same time) or two processes processing the same queue message at the same time. For those scenarios, a distributed lock or an atomic transaction with a uniqueness check is required — idempotency solves a different problem.

Related architectures and patterns

Circuit Breaker and idempotency are complementary in a resilience strategy: the Circuit Breaker decides whether it's worth trying the call, and idempotency guarantees that, when the call is retried after a failure, the effect isn't duplicated. Together, they enable safe retry: the Circuit Breaker avoids retry when the service is degraded; idempotency protects when the retry is necessary.

In CQRS, idempotent commands are safer to operate in distributed systems: a "process payment" command that can be resent without a duplicated effect simplifies failure handling on the write side. The Event Store benefits too: events generated by idempotent commands don't duplicate even if the command is processed more than once.

Message Queues with at-least-once semantics are the most common context where idempotency is mandatory. The combination of queue + idempotent consumer is the de facto pattern for reliable message processing without requiring the complexity and cost of exactly-once semantics.