Idempotency: the problem everyone thinks they understand until the first duplicate arrives
Almost every engineer can define idempotency in an interview. Very few systems implement it correctly where it actually matters.
The definition of idempotency is simple: an operation is idempotent if running it multiple times produces the same result as running it once. Any engineer with a couple of years of experience can recite it. And yet it's one of the guarantees most frequently broken in production, not because nobody knows the definition, but because nobody stops to ask where, specifically, an operation can be duplicated.
Why duplicates are inevitable, not an edge case
In any system that communicates over a network, there's a fundamental gray zone: the client sends a request, the server processes it, but the response gets lost before making it back. From the client's perspective, the operation failed and should be retried. From the server's perspective, the operation already succeeded. That disagreement isn't a bug, it's a mathematical consequence of communicating over a network that doesn't guarantee delivery, and it happens constantly: timeouts, process restarts mid-request, message queues that redeliver when an ack doesn't arrive in time.
If the operation is "read some data," the duplicate doesn't matter. If the operation is "charge a card" or "decrement inventory," an unhandled duplicate is real money lost or incorrectly gained.
Where idempotency actually lives
The most common mistake is treating idempotency as a property of the code ("my function has no weird side effects") instead of a property of the protocol between two systems. Real idempotency requires the client and server to agree on something very concrete: an idempotency key, generated by the client, that uniquely identifies that specific attempt at the operation, not the operation in the abstract, but this particular attempt, including its retries.
With that key, the server can correctly answer the question that actually matters: "have I already processed this?" And if the answer is yes, return the already-computed result instead of running the operation again.
The nuances that separate a correct implementation from one that only looks correct
- The deduplication window has to survive as long as the client's reasonable retries. If a client can retry up to 24 hours after a prolonged network failure, and the idempotency record expires in 10 minutes, the protection is illusory for that case.
- Storing the key isn't enough, you also have to store the result. If the server only remembers "I've seen this key" but not what it responded the first time, the second attempt can't return a consistent response. It can only reject the request, which breaks the experience for the client.
- The check-and-mark operation has to be atomic. If two requests with the same key arrive almost simultaneously, and "check if it already exists" and "mark as processed" are two separate unlocked steps, both can pass the check before either marks anything, the same race condition that ruins any seat-reservation or inventory system.
The lesson
Idempotency doesn't fail because the team doesn't understand the concept. It fails because it gets implemented as a checkbox ("we added an idempotency table") without reasoning through the three conditions above, the ones that actually determine whether the protection holds in the worst case, not the happy path where everything arrives in order and on time, but the case where a request is delayed, duplicated, or arrives at the exact same millisecond as another.