In short
An operation is idempotent when performing it repeatedly produces the same result as performing it once. It matters because retries are unavoidable: a network timeout does not tell you whether the request succeeded, so the client retries and the server may execute it twice. The standard fix is an idempotency key — a client-generated identifier the server records, so a repeat of the same key returns the original result instead of acting again.
The problem, precisely
A client sends a request. The server processes it. The response is lost on the way back — a timeout, a dropped connection, a load balancer restarting.
The client now knows nothing useful. It cannot distinguish the request never arrived from the request succeeded and the answer was lost. Those need opposite responses: retry, or do not.
So the client retries, because the alternative is silently losing work. And the server, which already did the job, does it again. That is a duplicate charge, a duplicate order, two invitation emails, two rows where there should be one.
This is not an edge case. It is the normal behaviour of unreliable networks, and it happens at scale continuously.
Where duplicates come from
Client retries after a timeout — the case above.
Users. Double-clicking a submit button, refreshing on a slow page, tapping back and submitting again.
Message queues. Most give at-least-once delivery. A consumer that crashes after doing the work but before acknowledging will see the message again, by design.
Webhooks. Providers retry on non-2xx responses, and sometimes on timeouts even when you succeeded. Any webhook handler will receive duplicates.
Background job frameworks. Retry on failure, and "failure" often includes jobs that actually completed.
Which operations are already safe
Some are naturally idempotent, and knowing which saves work.
Reads are trivially safe. Deletes by identifier usually are — deleting an already-deleted thing can succeed silently. Full replacement updates are, because setting a value to X twice leaves it at X.
The dangerous ones are creates and relative changes. Creating a record twice makes two records. "Add 10 to the balance" applied twice adds 20. Anything phrased as an increment rather than an assignment is a duplicate waiting to happen.
Where you can, express changes as assignments rather than increments — set status to shipped rather than advance status. That alone removes a category of bug.
Idempotency keys
For creates and anything with side effects, the standard solution is a key.
The client generates a unique identifier for the logical operation — not per attempt — and sends it with the request, typically in an Idempotency-Key header. Every retry of that same operation carries the same key.
The server then:
- Looks up the key. If it has a completed record, returns the stored response without doing the work again.
- If not, claims the key and processes the request.
- Stores the response against the key before replying.
The critical detail: the key must be claimed atomically, before the work happens. A unique constraint on the key column does this correctly. Checking whether a key exists and then inserting it is a race — two simultaneous retries both pass the check and both proceed, which is the exact bug you were preventing.
The details that get missed
Store the response, not just the fact. A retry needs to receive the same body and status as the original. Returning 200 OK with an empty body tells the client nothing about what was created.
Handle the in-flight case. If a key is claimed but not yet complete, a concurrent retry should get a clear "in progress" signal — commonly 409 — rather than waiting indefinitely or being told it succeeded.
Scope keys per account. Two customers must never collide, whatever they generate. Make the uniqueness constraint on the pair.
Expire them. Keys kept forever become an unbounded table. Twenty-four hours covers realistic retry windows for most systems.
Do not reuse a key for a different payload. If the same key arrives with different content, that is a client bug, and returning the original response silently hides it. Reject it explicitly.
Make the whole operation atomic where you can. If your handler writes three tables, do it in one transaction so a partial failure leaves nothing half-created for the retry to trip over.
When you cannot use a key
Sometimes the caller will not send one — a third-party webhook, a legacy integration.
Then derive uniqueness from the payload. Most webhook providers include an event identifier; record the ones you have processed and skip repeats. Where there is no identifier, a hash of the meaningful fields plus a timestamp window is a workable approximation.
For internal queues, a natural key often exists already: one order_shipped notification per order. A unique constraint on that pair does the same job as an idempotency key without one.
Testing it
This is easy to get wrong and easy to verify.
Send the same request twice sequentially and confirm one record and identical responses. Then send it twice concurrently — that is where naive check-then-insert implementations fail, and where the bug will find you in production. Then kill the process midway and retry, confirming the retry completes cleanly rather than tripping over a partial write.
If your handler survives those three, it is genuinely idempotent rather than accidentally so.
If you are building something where a duplicate would be expensive — payments, orders, notifications — book a call and we will look at the retry path with you.
Common questions
What does idempotent mean?
An operation is idempotent when performing it repeatedly has the same effect as performing it once. Reads, deletes by identifier, and full replacement updates are naturally idempotent. Creates and relative changes such as 'add 10 to the balance' are not, and those are where duplicate-request bugs occur.
What is an idempotency key?
A unique identifier the client generates for a logical operation and sends with the request, typically in an Idempotency-Key header. Every retry carries the same key, so the server can recognise a repeat, skip the work, and return the stored original response instead of acting twice.
How do you implement idempotency correctly?
Claim the key atomically before doing the work, using a unique constraint on the key column. Checking whether a key exists and then inserting it is a race condition — two concurrent retries both pass the check and both proceed. Store the full original response against the key so retries receive identical output.
Why do duplicate requests happen at all?
Because a client that times out cannot tell whether the request failed or succeeded with a lost response, so it retries. Users also double-click and refresh, message queues typically guarantee at-least-once delivery, and webhook providers retry on non-2xx responses and sometimes on timeouts even when you succeeded.
How do you handle idempotency for incoming webhooks?
Derive uniqueness from the payload, since the sender will not use your key scheme. Most providers include an event identifier — record which ones you have processed and skip repeats. Where none exists, a hash of the meaningful fields within a time window is a reasonable approximation.
