A Timeout Tells You Nothing: Designing APIs That Are Safe to Retry

A payment request times out after 30 seconds. Your client library does the sensible thing and retries. The customer is charged twice.

Now find the bug. The network was fine. The database was fine. The payment provider was fine. Your retry logic did exactly what you told it to. Every component behaved correctly, and a customer was still charged twice.

This is not a bug you can fix by being more careful. It is a property of what a timeout is, and the only way out is to design for it.

The four outcomes you cannot tell apart#

When a request times out, here is everything you actually know: you sent bytes, and you did not get an answer. That is consistent with four very different realities.

1. The request never reached the server.
2. The server received it and died before doing anything.
3. The server charged the card, then died before responding.
4. The server charged the card, responded, and the response was lost on the way back.

In cases 1 and 2, retrying is exactly right. In cases 3 and 4, retrying charges the customer again.

The client cannot distinguish them. Not with a longer timeout, not with better error handling, not with a health check. The information simply is not there. This is the fundamental ambiguity of talking to another machine over a network, and no amount of engineering removes it.

So you have two options, and only two:

• Never retry. No duplicates, and you lose work every time a response goes missing.
• Retry, and make repeats harmless. Work is never lost, and the server must be built to absorb the duplicate.

Almost every system needs the second one. That is what idempotency is: not a nice-to-have, but the thing that makes the retry you were always going to write safe.

Some of this is already solved#

HTTP has an answer for part of the problem. An operation is idempotent if doing it many times has the same effect as doing it once, and RFC 9110 specifies which methods qualify:

• GET, HEAD, OPTIONS, TRACE are safe (no state change) and therefore idempotent.
• PUT and DELETE are idempotent but not safe. They change state, and doing it twice lands you in the same place as doing it once.
• POST and PATCH are not idempotent.

Two details worth knowing, because they come up constantly:

DELETE is idempotent in effect, not in response. The first call returns 204, the second returns 404. The resource is equally gone either way, which is what idempotency means. Clients should treat a 404 on a retried delete as success, not as a failure to handle.

PATCH is idempotent only if the patch is absolute. {"status": "shipped"} can be applied all day. {"op": "increment", "field": "attempts"} cannot. The method does not decide this. The payload does.

Which leaves the case that actually hurts: POST /payments means "create a new payment," so every call is by definition a new one. There is nothing in HTTP that makes that safe.

Idempotency keys, and the part people get wrong#

The fix is to let the client name the attempt, so the server can recognise a repeat of it.

The client generates a unique key once, before the first try, and sends the identical key on every retry of that same logical operation:

POST /payments HTTP/1.1
Idempotency-Key: 8f14e45f-ea0c-4c1e-9f31-2d9a1f5b7c33
Content-Type: application/json

{ "amount": 4999, "currency": "usd", "customer": "cus_123" }

Server-side:

1. Look up the key.
2. Not found        -> claim it, do the work, store (status + response body), return it.
3. Found, completed -> return the STORED response. Do not do the work again.
4. Found, in flight -> return 409, tell the client to retry shortly.

Step 3 is where implementations quietly break. On a repeat you must replay the original response, not merely skip the work. The client asked for a payment ID. If the retry returns 204 No Content because "we already did that," the client still has no payment ID, and your integration is broken in a way that looks like success.

Two ways to get the key itself wrong, both of which pass every test you are likely to write:

Generating it on the server. A key created inside the request handler identifies the request, not the attempt, so a retry generates a fresh one and looks brand new. I have seen this ship, because in tests nothing is ever retried.

Regenerating it per attempt. The same bug from the client side, usually caused by putting key generation inside the retry loop instead of outside it.

The key must be created by the client, once, before the first attempt, and held constant across every retry.

The race that ruins the naive version#

Here is the implementation nearly everyone writes first:

if store.get(key):
    return store.get(key).response
charge_card()
store.put(key, response)

Two retries arriving at the same moment both read "not found," and both charge the card:

Request A: lookup -> not found
Request B: lookup -> not found      <-- both passed the check
Request A: charge card
Request B: charge card              <-- charged twice

Reading before writing cannot fix this, because the gap between the read and the write is where the problem lives. The claim has to be atomic, and your database already has the primitive for it: a unique constraint.

CREATE TABLE idempotency_keys (
  key            TEXT PRIMARY KEY,      -- uniqueness enforced here
  request_hash   TEXT        NOT NULL,
  status         TEXT        NOT NULL,  -- 'in_progress' | 'completed'
  response_code  INT,
  response_body  JSONB,
  created_at     TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Whichever request wins this INSERT owns the work.
-- The loser gets a unique violation and returns 409.
INSERT INTO idempotency_keys (key, request_hash, status)
VALUES ($1, $2, 'in_progress');

The winner does the work and updates the row to completed with the response. The loser sees the constraint violation, returns 409, and the client comes back a moment later for the real answer. Redis SET key value NX gives you the same atomic claim if your key store lives elsewhere.

Where the record lives decides whether this actually works#

This is the detail that separates a design that holds up from one that fails at precisely the wrong moment.

Put the idempotency record in Redis and the payment in Postgres, and the two can drift apart: the charge commits, the process dies before Redis is updated, the retry arrives, and you charge again. You have built the appearance of idempotency without the guarantee, and it fails exactly when a crash happens, which is the only scenario it existed for.

Write the idempotency record in the same transaction as the side effect:

BEGIN;
  INSERT INTO payments (id, customer, amount, status)
       VALUES ($1, $2, $3, 'captured');
  UPDATE idempotency_keys
     SET status = 'completed', response_code = 201, response_body = $4
   WHERE key = $5;
COMMIT;

Now either both are durable or neither is. There is no window.

When the side effect is not in your database, because you are charging a real card at a payment provider, you cannot get that atomicity directly. So you push the key outward: pass your idempotency key through to the provider and let the duplicate be absorbed at the boundary that owns the money. This is why every serious payments API accepts one.

Same key, different body#

What if a client reuses a key with a different payload, perhaps from a bug that pins one key forever?

Store a hash of the request body next to the key and compare on repeat. Matching hash means a genuine retry, so replay the stored response. Different hash means the client is misusing the key, so reject it.

Silently returning the first response would be far worse than an error: the client believes it created a $10 refund and receives the record of a $4,999 charge. Stripe does exactly this comparison and errors when the parameters differ, which is the behaviour to copy.

Keys also cannot live forever. Pick a retention window, document it, and make sure it comfortably exceeds the longest retry schedule any client might use. Stripe keeps keys for at least 24 hours before pruning them, which is a sensible default to steal.

Often you do not need a key at all#

Idempotency keys are the general mechanism, not always the cheapest one. Before adding a table, check whether the operation can simply be made repeatable:

Absolute writes instead of relative ones. UPDATE orders SET status = 'shipped' is naturally repeatable. UPDATE accounts SET balance = balance - 50 is not. A surprising number of operations can be expressed as setting a value rather than changing one.

Client-generated resource IDs. Let the client choose the ID and PUT /orders/{client_uuid}. A repeat becomes an overwrite of identical data, and creation is idempotent for free.

Unique business constraints. A unique index on (customer_id, order_reference) makes duplicate orders impossible at the storage layer, no matter how many times the request arrives.

Conditional requests. If-Match with an ETag applies an update only against the state the client last saw. The second attempt fails with 412, which also solves lost updates.

Exactly-once is not a thing you can buy#

The obvious question at this point: why not use a system that guarantees exactly-once delivery and skip all of this?

Because exactly-once delivery across an unreliable network is not achievable. It reduces to the Two Generals Problem: no finite exchange of messages lets both sides become certain the other received the last one. The sender can never be sure, so it must either retry or not, and you are back to the same two options from the top of this article.

What products marketed as "exactly-once" provide is exactly-once processing within a closed system. Kafka's transactional producer can make consume-process-produce atomic because the offsets and the output both live inside Kafka and can share a transaction. The moment an external side effect is involved, charging a card, sending an email, calling a third-party API, the guarantee is gone. Kafka cannot roll back an email.

What every serious system actually runs is effectively-once: at-least-once delivery, plus consumers that make duplicates harmless. That is the pattern in this article, and it is what sits underneath every payment system you have used.

The mental model#

Stop thinking of retries as an error path and duplicates as an edge case. At any real scale, both are routine: every deploy interrupts in-flight work, every crash replays messages, every timeout is ambiguous.

So classify every operation that changes state into one of three buckets:

1. Naturally idempotent. Absolute writes, client-supplied IDs, unique constraints. Nothing more to do.
2. Made idempotent by key. Client-generated key, atomic claim, response replayed, record committed with the side effect.
3. Genuinely single-shot. This bucket should be almost empty, and anything left in it needs an explicit answer to "what happens when this arrives twice?"

A timeout tells you nothing. Build as though every request might arrive twice, because eventually it will.

---

This is Lesson 2.2 of Networking and API Design for System Design, a course on the communication layer: networking fundamentals, REST, GraphQL and gRPC, API gateways and rate limiting, API security, and real-time and asynchronous communication patterns. The delivery-semantics side of this argument, at-most-once against at-least-once and why exactly-once is an illusion, is covered in Module 5. Related reading: JWT Is Not a Session for the other API design decision people get wrong by default, and TIME_WAIT and Conntrack Exhaustion for what aggressive retry loops do to a connection table.