Retry

Intent

Some failures are transient: a 503 during a deploy, a 429 from a rate limiter, a connection dropped mid-handshake. Repeating the operation after a short wait turns them into successes. A retry policy decides which errors qualify, how many attempts are allowed, how much total time may be spent, and how the loop stops - including when the caller cancels.

Forces

• Transient vs permanent: a 503 usually heals; a 400 never will. Retrying permanent errors wastes time and hides the bug.
• Persistence vs amplification: every retry adds load to a dependency that is already struggling; N callers retrying M times are N×M requests.
• Idempotency vs duplication: if you cannot tell whether attempt one landed, repeating it may charge the card twice.
• Keep trying vs keep waiting: without a deadline, a retry loop is a hang with extra steps.

Therefore

Classify the error, and only for retryable ones repeat a bounded number of attempts with spacing, within an overall deadline, checking cancellation between attempts - then fail with the last error.

The naive version - for i in range(3): try: call() with no error classification, no spacing, no deadline, and no cancellation check - retries permanent failures, hammers the dependency at full speed, and ignores the caller giving up. Every one of those four omissions is a production incident.

Structure

sequenceDiagram
  participant C as Caller
  participant R as Retry loop
  participant D as Dependency
  C->>R: deliver(event, signal)
  R->>D: attempt 1
  D-->>R: 503, Retry-After 10 ms
  R->>R: retryable? budget left? not cancelled?
  R->>D: attempt 2 (after wait)
  D-->>R: 503, Retry-After 10 ms
  R->>D: attempt 3 (after wait)
  D-->>R: delivered
  R-->>C: done

Participants

• Retry loop (deliverWithRetry): owns the policy - retryable classification, maxAttempts, deadline, wait function, cancellation checks.
• Operation (deliver): the repeatable call. Must be idempotent or safely repeatable, and cancellable.
• Dependency (RemoteEndpoint): may answer with transient failures, ideally with Retry-After.
• Caller: passes a cancellation signal and receives the last error when the policy gives up.

Use it when

• The error is transient-looking - timeouts, connection resets, 429, 502, 503 - and the operation can be repeated safely.
• The operation is idempotent by nature (reads, conditional writes) or made repeatable with an idempotency key.
• A platform SDK documents transient failures and expects callers to retry (most cloud SDKs do).
• The caller can afford the added latency, or the work is background where latency is free.

Do not use it when

• The operation is not repeatable and has no idempotency key. A retried charge or a retried "create user" duplicates side effects; make it repeatable first (Command plus a durable queue with deduplication) or fail.
• The error is permanent: validation failures, missing resources, authorization denials, business rejections. Fix the input; do not repeat it.
• The dependency is down for an episode, not blipping. Retrying through an outage piles on load; a Circuit Breaker must stop the loop instead.
• The caller is interactive with a 1–2 s budget. Two attempts at most, then Fallback or fail.
• The HTTP client or SDK already retries. Wrapping a second loop around a built-in one multiplies attempts; configure one policy instead.
• The failure is a bug (a TypeError, a nil dereference). Retrying hides it and slows every future failure.

Application grill

Inspect first: the operations and whether each is idempotent and repeatable - that gate decides what may retry; the retry knobs already set (HTTP client defaults, SDK policies, queue visibility timeouts) that a second layer would multiply; how cancellation propagates between attempts.

Missing pieces: a small shared helper is enough; a library (p-retry, tenacity, resilience4j) pays off only when policy must be uniform across many call sites. Honoring Retry-After needs no infrastructure.

Ask before you build:

• Scale: what bounded attempt count, times callers, times failure rate, can the dependency absorb during its incident?
• Performance: does the retry sequence fit the caller's remaining deadline after the first attempt?
• Reliability: which errors are retryable (timeouts, 5xx) versus permanent (4xx), and where is that classified?
• Budget: stacked retries across layers multiply load in exactly the incident window - who audits the layers?
• Implementation: is the final failure surfaced with its attempt count and last error, and are retries marked in metrics?

Implementation steps

1. Make the operation repeatable: idempotency key on writes, conditional update, or an inherent read. If you cannot, stop here.
2. Classify errors explicitly. Retryable: timeouts, connection resets, 429, 502, 503. Terminal: 4xx (except 429), business errors, programming errors. Write the classifier as code, not as a habit.
3. Fix the bounds: maxAttempts (3–5), an overall deadline, and a per-attempt timeout (Timeout). Total wait is capped by the deadline, not by the schedule.
4. Space the attempts (a fixed short delay, or the server's Retry-After). The schedule discipline is Exponential Backoff.
5. Check cancellation before every wait and every attempt: an aborted signal must end the loop promptly instead of starting attempt N+1.
6. On give-up, surface the last error with the attempt count attached, so logs tell the whole story.
7. Emit metrics: attempts per call, outcomes, and give-up reasons. A rising retry rate is the first signal of a dependency degrading.

Example

A webhook delivery must survive a dependency that returns 503 with Retry-After while it restarts. Deliveries carry an idempotency key, attempts are bounded by count and by deadline, and a cancellation signal stops the loop between attempts.

// Errors the dependency raises; status decides retryability, not the message.
type HttpError = Error & { status: number; retryAfterMs: number | null };
const statusOf = (e: unknown) => (e as HttpError)?.status;
const isRetryable = (e: unknown) => [429, 500, 502, 503].includes(statusOf(e));

const sleep = (ms: number, signal: AbortSignal) =>
  new Promise<void>((resolve, reject) => {
    const t = setTimeout(resolve, ms);
    signal.addEventListener(
      "abort",
      () => {
        clearTimeout(t);
        reject(new Error("cancelled"));
      },
      { once: true },
    );
  });

interface DeliveryTarget {
  deliver(event: string, idempotencyKey: string): Promise<void>;
}

// The dependency: 503 with Retry-After while it restarts, then it serves.
class RemoteEndpoint implements DeliveryTarget {
  constructor(private failuresLeft: number) {}
  async deliver(event: string): Promise<void> {
    if (this.failuresLeft-- > 0) {
      const e = new Error("503 restarting") as HttpError;
      e.status = 503;
      e.retryAfterMs = 10;
      throw e;
    }
    console.log(`  [endpoint] delivered ${event}`);
  }
}

async function deliverWithRetry(
  target: DeliveryTarget,
  event: string,
  opts: { maxAttempts: number; deadlineMs: number; signal: AbortSignal },
): Promise<void> {
  const startedAt = Date.now();
  let lastError: unknown;
  for (let attempt = 1; attempt <= opts.maxAttempts; attempt++) {
    if (opts.signal.aborted) throw new Error("cancelled before attempt");
    try {
      await target.deliver(event, `evt:${event}:1`); // idempotency key
      return;
    } catch (e) {
      lastError = e;
      const wait = (e as HttpError).retryAfterMs ?? 10; // server hint first
      const withinBudget =
        Date.now() - startedAt + wait < opts.deadlineMs && isRetryable(e);
      if (!withinBudget) throw e; // give up: last error carries the story
      console.log(
        `  [retry] attempt ${attempt} failed (${(e as Error).message}); waiting ${wait} ms`,
      );
      await sleep(wait, opts.signal);
    }
  }
  throw lastError;
}

const endpoint = new RemoteEndpoint(2);
await deliverWithRetry(endpoint, "order.paid", {
  maxAttempts: 4,
  deadlineMs: 1_000,
  signal: AbortSignal.timeout(2_000),
});
// 2 failed attempts, then: [endpoint] delivered order.paid

try {
  await deliverWithRetry(new RemoteEndpoint(99), "order.refunded", {
    maxAttempts: 4,
    deadlineMs: 50, // budget runs out before the endpoint recovers
    signal: AbortSignal.timeout(2_000),
  });
} catch (e) {
  console.log(`gave up: ${(e as Error).message}`); // gave up: 503 restarting
}

The fixed 10 ms wait keeps this example honest about scope; in production that wait comes from the backoff schedule.

Trade-offs

Gains

• Transient failures disappear before anyone notices.
• The policy is one place: classification, bounds, and waits are testable and consistent.
• Giving up is loud: the last error plus attempt count tells the operator what happened.

Costs

• Load multiplies exactly when the dependency is least able to serve it.
• Duplicate side effects when idempotency was assumed instead of built.
• Latency grows: a caller that would fail in 100 ms now fails in attempts × waits.
• Two stacked retry layers (client library plus hand-rolled loop) multiply attempts silently.

Relations

• Exponential Backoff is the spacing policy this entry's waits should use in production.
• Timeout bounds each attempt; an attempt without a deadline can hang the whole loop.
• Circuit Breaker stops the loop when failures stop being transient; retrying an open breaker is the storm it prevents.
• Rate Limiting turns 429 into a schedule: its Retry-After is this policy's input, and a client-side budget prevents the refusals in the first place.
• Fallback is the caller's plan when attempts run out.
• Command moves delivery to a durable queue when the work must happen eventually but not synchronously.
• Decorator is the implementation shape: wrap the flaky operation, expose the same interface.

Agent checklist

• The operation is idempotent or made repeatable with an idempotency key before any retry exists.
• Retryable and terminal errors are classified in code, not by instinct; 4xx validation errors are never retried.
• Attempts are bounded by both count and overall deadline, and each attempt has its own timeout.
• Waits honor the server's Retry-After when present.
• Cancellation is checked before every attempt and wait, and mid-wait too.
• Giving up raises the last error with the attempt count, and retry rates are measurable.