How to Design Reliable Retry Logic with Idempotency Keys

Why retries need care

Networks drop connections and servers return temporary errors, so integrations must retry. But a naïve retry loop can make things worse: it can charge a customer twice, create duplicate invoices, or flood an already struggling service.

Decide what is safe to retry

HTTP defines some methods — such as GET, PUT, and DELETE — as idempotent: repeating them has the same effect as sending them once (RFC 9110). POST is not, which is why creating payments or orders needs an idempotency key.

• Retry: network errors, 408, 429, 500, 502, 503, 504.
• Do not retry: 400, 401, 403, 404, 409, 422 — fix the request instead.

Back off exponentially, with jitter

Wait longer after each failure — for example a maximum of 200 ms, then 400 ms, 800 ms, and so on — up to a cap. “Full jitter” picks a random delay between zero and that maximum so many clients do not retry in lockstep. If the server sends a Retry-After header, use it.

Use one idempotency key per operation

Generate a unique key (a UUID works) when the logical operation starts, and send the same key on every retry. The server stores the key with the result; a repeated key returns the stored result rather than repeating the work. Generating a new key per attempt defeats the purpose.

Complete example

import { randomUUID } from 'node:crypto'

const RETRYABLE_STATUS = new Set([408, 429, 500, 502, 503, 504])
const sleep = (ms) => new Promise((r) => setTimeout(r, ms))

/** Full-jitter exponential backoff: random delay between 0 and min(cap, base * 2^attempt). */
export function backoffDelay(attempt, { baseMs = 200, capMs = 10_000 } = {}) {
  return Math.random() * Math.min(capMs, baseMs * 2 ** attempt)
}

export async function postWithRetry(url, body, { maxAttempts = 5, fetchImpl = fetch } = {}) {
  // One key per logical operation, reused on every retry of that operation.
  const idempotencyKey = randomUUID()
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    let res
    try {
      res = await fetchImpl(url, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json', 'Idempotency-Key': idempotencyKey },
        body: JSON.stringify(body),
      })
    } catch (err) {
      // Network error: the request may or may not have reached the server, so retrying is only safe with the key.
      if (attempt === maxAttempts - 1) throw err
      await sleep(backoffDelay(attempt))
      continue
    }
    if (res.ok) return res.json()
    if (!RETRYABLE_STATUS.has(res.status) || attempt === maxAttempts - 1) {
      throw new Error(`Request failed with status ${res.status}`)
    }
    // Respect Retry-After (seconds) when the server provides it.
    const retryAfter = Number(res.headers.get('retry-after'))
    await sleep(Number.isFinite(retryAfter) && retryAfter > 0 ? retryAfter * 1000 : backoffDelay(attempt))
  }
}

We tested this against a local server that returned 503, then 429 with Retry-After: 1, then succeeded. The client made three requests with one idempotency key, the server created exactly one record, and a 400 response failed immediately without retrying.

What the server must do

1. Store the idempotency key, a hash of the request body, and the response.
2. If the same key arrives with a different body, reject it — that is a client bug.
3. Expire keys after a reasonable window, and document that window.

Limitations

• Idempotency keys only help if the receiving API supports them; check its documentation.
• Retries add latency. Set an overall deadline so users are not left waiting.
• Log every final failure with enough context to investigate and replay it.