# Retry

> Re-attempt only retryable, repeatable operations a bounded number of times, honour Retry-After, check cancellation between attempts, and surface the last error.

- kind: pattern
- category: resilience
- aliases: Retry Policy, Automatic Retry
- canonical: https://aipatterns.dev/design-patterns/retry
- inspired-by: https://learn.microsoft.com/en-us/azure/architecture/patterns/retry

## Intent

Some failures are transient: a 503 during a deploy, a 429 from a rate limiter, a connection dropped mid-handshake. Repeating the operation after a short wait turns them into successes. A retry policy decides which errors qualify, how many attempts are allowed, how much total time may be spent, and how the loop stops - including when the caller cancels.

## Forces

- **Transient** vs **permanent**: a 503 usually heals; a 400 never will. Retrying permanent errors wastes time and hides the bug.
- **Persistence** vs **amplification**: every retry adds load to a dependency that is already struggling; N callers retrying M times are N×M requests.
- **Idempotency** vs **duplication**: if you cannot tell whether attempt one landed, repeating it may charge the card twice.
- **Keep trying** vs **keep waiting**: without a deadline, a retry loop is a hang with extra steps.

## Therefore

**Classify the error, and only for retryable ones repeat a bounded number of attempts with spacing, within an overall deadline, checking cancellation between attempts - then fail with the last error.**

The naive version - `for i in range(3): try: call()` with no error classification, no spacing, no deadline, and no cancellation check - retries permanent failures, hammers the dependency at full speed, and ignores the caller giving up. Every one of those four omissions is a production incident.

## Structure

```mermaid
sequenceDiagram
  participant C as Caller
  participant R as Retry loop
  participant D as Dependency
  C->>R: deliver(event, signal)
  R->>D: attempt 1
  D-->>R: 503, Retry-After 10 ms
  R->>R: retryable? budget left? not cancelled?
  R->>D: attempt 2 (after wait)
  D-->>R: 503, Retry-After 10 ms
  R->>D: attempt 3 (after wait)
  D-->>R: delivered
  R-->>C: done
```

### Participants

- **Retry loop** (`deliverWithRetry`): owns the policy - retryable classification, `maxAttempts`, deadline, wait function, cancellation checks.
- **Operation** (`deliver`): the repeatable call. Must be idempotent or safely repeatable, and cancellable.
- **Dependency** (`RemoteEndpoint`): may answer with transient failures, ideally with `Retry-After`.
- **Caller**: passes a cancellation signal and receives the last error when the policy gives up.

## Use it when

- The error is transient-looking - timeouts, connection resets, 429, 502, 503 - and the operation can be repeated safely.
- The operation is idempotent by nature (reads, conditional writes) or made repeatable with an idempotency key.
- A platform SDK documents transient failures and expects callers to retry (most cloud SDKs do).
- The caller can afford the added latency, or the work is background where latency is free.

## Do not use it when

- The operation is not repeatable and has no idempotency key. A retried charge or a retried "create user" duplicates side effects; make it repeatable first ([Command](https://aipatterns.dev/design-patterns/command.md) plus a durable queue with deduplication) or fail.
- The error is permanent: validation failures, missing resources, authorization denials, business rejections. Fix the input; do not repeat it.
- The dependency is down for an episode, not blipping. Retrying through an outage piles on load; a [Circuit Breaker](https://aipatterns.dev/design-patterns/circuit-breaker.md) must stop the loop instead.
- The caller is interactive with a 1–2 s budget. Two attempts at most, then [Fallback](https://aipatterns.dev/design-patterns/fallback.md) or fail.
- The HTTP client or SDK already retries. Wrapping a second loop around a built-in one multiplies attempts; configure one policy instead.
- The failure is a bug (a `TypeError`, a nil dereference). Retrying hides it and slows every future failure.

## Application grill

**Inspect first:** the operations and whether each is idempotent and repeatable - that gate decides what may retry; the retry knobs already set (HTTP client defaults, SDK policies, queue visibility timeouts) that a second layer would multiply; how cancellation propagates between attempts.

**Missing pieces:** a small shared helper is enough; a library (p-retry, tenacity, resilience4j) pays off only when policy must be uniform across many call sites. Honoring Retry-After needs no infrastructure.

**Ask before you build:**

- **Scale:** what bounded attempt count, times callers, times failure rate, can the dependency absorb during its incident?
- **Performance:** does the retry sequence fit the caller's remaining deadline after the first attempt?
- **Reliability:** which errors are retryable (timeouts, 5xx) versus permanent (4xx), and where is that classified?
- **Budget:** stacked retries across layers multiply load in exactly the incident window - who audits the layers?
- **Implementation:** is the final failure surfaced with its attempt count and last error, and are retries marked in metrics?

## Implementation steps

1. Make the operation repeatable: idempotency key on writes, conditional update, or an inherent read. If you cannot, stop here.
2. Classify errors explicitly. Retryable: timeouts, connection resets, 429, 502, 503. Terminal: 4xx (except 429), business errors, programming errors. Write the classifier as code, not as a habit.
3. Fix the bounds: `maxAttempts` (3–5), an overall deadline, and a per-attempt timeout ([Timeout](https://aipatterns.dev/design-patterns/timeout.md)). Total wait is capped by the deadline, not by the schedule.
4. Space the attempts (a fixed short delay, or the server's `Retry-After`). The schedule discipline is [Exponential Backoff](https://aipatterns.dev/design-patterns/exponential-backoff.md).
5. Check cancellation before every wait and every attempt: an aborted signal must end the loop promptly instead of starting attempt N+1.
6. On give-up, surface the last error with the attempt count attached, so logs tell the whole story.
7. Emit metrics: attempts per call, outcomes, and give-up reasons. A rising retry rate is the first signal of a dependency degrading.

## Example

A webhook delivery must survive a dependency that returns 503 with `Retry-After` while it restarts. Deliveries carry an idempotency key, attempts are bounded by count and by deadline, and a cancellation signal stops the loop between attempts.

```ts title="deliver.ts"
// Errors the dependency raises; status decides retryability, not the message.
type HttpError = Error & { status: number; retryAfterMs: number | null };
const statusOf = (e: unknown) => (e as HttpError)?.status;
const isRetryable = (e: unknown) => [429, 500, 502, 503].includes(statusOf(e));

const sleep = (ms: number, signal: AbortSignal) =>
  new Promise<void>((resolve, reject) => {
    const t = setTimeout(resolve, ms);
    signal.addEventListener(
      "abort",
      () => {
        clearTimeout(t);
        reject(new Error("cancelled"));
      },
      { once: true },
    );
  });

interface DeliveryTarget {
  deliver(event: string, idempotencyKey: string): Promise<void>;
}

// The dependency: 503 with Retry-After while it restarts, then it serves.
class RemoteEndpoint implements DeliveryTarget {
  constructor(private failuresLeft: number) {}
  async deliver(event: string): Promise<void> {
    if (this.failuresLeft-- > 0) {
      const e = new Error("503 restarting") as HttpError;
      e.status = 503;
      e.retryAfterMs = 10;
      throw e;
    }
    console.log(`  [endpoint] delivered ${event}`);
  }
}

async function deliverWithRetry(
  target: DeliveryTarget,
  event: string,
  opts: { maxAttempts: number; deadlineMs: number; signal: AbortSignal },
): Promise<void> {
  const startedAt = Date.now();
  let lastError: unknown;
  for (let attempt = 1; attempt <= opts.maxAttempts; attempt++) {
    if (opts.signal.aborted) throw new Error("cancelled before attempt");
    try {
      await target.deliver(event, `evt:${event}:1`); // idempotency key
      return;
    } catch (e) {
      lastError = e;
      const wait = (e as HttpError).retryAfterMs ?? 10; // server hint first
      const withinBudget =
        Date.now() - startedAt + wait < opts.deadlineMs && isRetryable(e);
      if (!withinBudget) throw e; // give up: last error carries the story
      console.log(
        `  [retry] attempt ${attempt} failed (${(e as Error).message}); waiting ${wait} ms`,
      );
      await sleep(wait, opts.signal);
    }
  }
  throw lastError;
}

const endpoint = new RemoteEndpoint(2);
await deliverWithRetry(endpoint, "order.paid", {
  maxAttempts: 4,
  deadlineMs: 1_000,
  signal: AbortSignal.timeout(2_000),
});
// 2 failed attempts, then: [endpoint] delivered order.paid

try {
  await deliverWithRetry(new RemoteEndpoint(99), "order.refunded", {
    maxAttempts: 4,
    deadlineMs: 50, // budget runs out before the endpoint recovers
    signal: AbortSignal.timeout(2_000),
  });
} catch (e) {
  console.log(`gave up: ${(e as Error).message}`); // gave up: 503 restarting
}
```

```python title="deliver.py"
import time
from typing import Callable

class HttpError(Exception):
    def __init__(self, status: int, retry_after_ms: int | None = None) -> None:
        super().__init__(f"{status}")
        self.status, self.retry_after_ms = status, retry_after_ms

RETRYABLE = {429, 500, 502, 503}

class RemoteEndpoint:
    """503 with Retry-After while it restarts, then it serves."""

    def __init__(self, failures_left: int) -> None:
        self._failures_left = failures_left

    def deliver(self, event: str, idempotency_key: str) -> None:
        if self._failures_left > 0:
            self._failures_left -= 1
            raise HttpError(503, retry_after_ms=10)
        print(f"  [endpoint] delivered {event}")

def deliver_with_retry(
    target: RemoteEndpoint,
    event: str,
    max_attempts: int,
    deadline_ms: int,
    cancelled: Callable[[], bool],
) -> None:
    started_at = time.monotonic()
    last_error: Exception | None = None
    for attempt in range(1, max_attempts + 1):
        if cancelled():  # cancellation beats retrying
            raise InterruptedError("cancelled before attempt")
        try:
            target.deliver(event, f"evt:{event}:1")  # idempotency key
            return
        except HttpError as e:
            last_error = e
            wait = e.retry_after_ms if e.retry_after_ms is not None else 10
            in_budget = (
                (time.monotonic() - started_at) * 1000 + wait < deadline_ms
                and e.status in RETRYABLE
            )
            if not in_budget:
                raise  # give up: the last error carries the story
            print(f"  [retry] attempt {attempt} failed ({e}); waiting {wait} ms")
            time.sleep(wait / 1000)
    raise last_error  # type: ignore[misc]

deliver_with_retry(RemoteEndpoint(failures_left=2), "order.paid",
                   max_attempts=4, deadline_ms=1_000, cancelled=lambda: False)
# 2 failed attempts, then: [endpoint] delivered order.paid

try:
    deliver_with_retry(RemoteEndpoint(failures_left=99), "order.refunded",
                       max_attempts=4, deadline_ms=50, cancelled=lambda: False)
except HttpError as e:
    print(f"gave up: {e}")  # gave up: 503
```

```go title="deliver.go"
package main

import (
	"context"
	"errors"
	"fmt"
	"net/http"
	"time"
)

// HTTPError carries the status and the server's Retry-After hint.
type HTTPError struct {
	Status       int
	RetryAfter   time.Duration
}

func (e HTTPError) Error() string { return fmt.Sprintf("%d", e.Status) }

func retryable(status int) bool {
	return status == http.StatusTooManyRequests || status >= 500
}

// RemoteEndpoint 503s with Retry-After while it restarts, then it serves.
type RemoteEndpoint struct{ failuresLeft int }

func (r *RemoteEndpoint) Deliver(ctx context.Context, event, idempotencyKey string) error {
	if r.failuresLeft > 0 {
		r.failuresLeft--
		return HTTPError{Status: 503, RetryAfter: 10 * time.Millisecond}
	}
	fmt.Println("  [endpoint] delivered", event)
	return nil
}

func deliverWithRetry(ctx context.Context, target *RemoteEndpoint, event string) error {
	startedAt := time.Now()
	var lastErr error
	for attempt := 1; attempt <= 4; attempt++ {
		if err := ctx.Err(); err != nil { // cancellation beats retrying
			return err
		}
		err := target.Deliver(ctx, event, "evt:"+event+":1") // idempotency key
		if err == nil {
			return nil
		}
		var httpErr HTTPError
		lastErr = err
		wait := 10 * time.Millisecond
		if errors.As(err, &httpErr) {
			if httpErr.RetryAfter > 0 { // server hint first
				wait = httpErr.RetryAfter
			}
			if !retryable(httpErr.Status) {
				return err // permanent: do not repeat
			}
		}
		if time.Since(startedAt)+wait > time.Second {
			return err // overall deadline exceeded: give up
		}
		fmt.Printf("  [retry] attempt %d failed (%v); waiting %v\n", attempt, err, wait)
		select {
		case <-time.After(wait):
		case <-ctx.Done(): // cancelled mid-wait
			return ctx.Err()
		}
	}
	return lastErr
}

func main() {
	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
	defer cancel()

	if err := deliverWithRetry(ctx, &RemoteEndpoint{failuresLeft: 2}, "order.paid"); err != nil {
		fmt.Println("err:", err)
	} // 2 failed attempts, then: [endpoint] delivered order.paid

	err := deliverWithRetry(ctx, &RemoteEndpoint{failuresLeft: 99}, "order.refunded")
	fmt.Println("gave up:", err) // gave up: 503
}
```

```java title="Deliver.java"
public class Deliver {
    static final class HttpException extends RuntimeException {
        final int status;
        final long retryAfterMs;
        HttpException(int status, long retryAfterMs) {
            super(String.valueOf(status));
            this.status = status;
            this.retryAfterMs = retryAfterMs;
        }
    }

    /** 503 with Retry-After while it restarts, then it serves. */
    static final class RemoteEndpoint {
        private int failuresLeft;
        RemoteEndpoint(int failuresLeft) { this.failuresLeft = failuresLeft; }

        void deliver(String event, String idempotencyKey) {
            if (failuresLeft > 0) {
                failuresLeft--;
                throw new HttpException(503, 10);
            }
            System.out.println("  [endpoint] delivered " + event);
        }
    }

    static void deliverWithRetry(RemoteEndpoint target, String event,
                                 int maxAttempts, long deadlineMs) throws InterruptedException {
        long startedAt = System.nanoTime();
        RuntimeException lastError = null;
        for (int attempt = 1; attempt <= maxAttempts; attempt++) {
            if (Thread.currentThread().isInterrupted()) // cancellation beats retrying
                throw new InterruptedException("cancelled before attempt");
            try {
                target.deliver(event, "evt:" + event + ":1"); // idempotency key
                return;
            } catch (HttpException e) {
                lastError = e;
                long wait = e.retryAfterMs() > 0 ? e.retryAfterMs() : 10; // hint first
                boolean retryable = e.status() == 429 || e.status() >= 500;
                boolean withinBudget =
                        (System.nanoTime() - startedAt) / 1_000_000 + wait < deadlineMs;
                if (!retryable || !withinBudget) throw e; // give up
                System.out.println("  [retry] attempt " + attempt + " failed ("
                        + e.getMessage() + "); waiting " + wait + " ms");
                Thread.sleep(wait);
            }
        }
        throw lastError;
    }

    public static void main(String[] args) throws InterruptedException {
        deliverWithRetry(new RemoteEndpoint(2), "order.paid", 4, 1_000);
        // 2 failed attempts, then: [endpoint] delivered order.paid

        try {
            deliverWithRetry(new RemoteEndpoint(99), "order.refunded", 4, 50);
        } catch (HttpException e) {
            System.out.println("gave up: " + e.getMessage()); // gave up: 503
        }
    }
}
```

The fixed 10 ms wait keeps this example honest about scope; in production that wait comes from the backoff schedule.

## Trade-offs

**Gains**
- Transient failures disappear before anyone notices.
- The policy is one place: classification, bounds, and waits are testable and consistent.
- Giving up is loud: the last error plus attempt count tells the operator what happened.

**Costs**
- Load multiplies exactly when the dependency is least able to serve it.
- Duplicate side effects when idempotency was assumed instead of built.
- Latency grows: a caller that would fail in 100 ms now fails in attempts × waits.
- Two stacked retry layers (client library plus hand-rolled loop) multiply attempts silently.

## Relations

- [Exponential Backoff](https://aipatterns.dev/design-patterns/exponential-backoff.md) is the spacing policy this entry's waits should use in production.
- [Timeout](https://aipatterns.dev/design-patterns/timeout.md) bounds each attempt; an attempt without a deadline can hang the whole loop.
- [Circuit Breaker](https://aipatterns.dev/design-patterns/circuit-breaker.md) stops the loop when failures stop being transient; retrying an open breaker is the storm it prevents.
- [Rate Limiting](https://aipatterns.dev/design-patterns/rate-limiting.md) turns `429` into a schedule: its `Retry-After` is this policy's input, and a client-side budget prevents the refusals in the first place.
- [Fallback](https://aipatterns.dev/design-patterns/fallback.md) is the caller's plan when attempts run out.
- [Command](https://aipatterns.dev/design-patterns/command.md) moves delivery to a durable queue when the work must happen eventually but not synchronously.
- [Decorator](https://aipatterns.dev/design-patterns/decorator.md) is the implementation shape: wrap the flaky operation, expose the same interface.

## Agent checklist

- [ ] The operation is idempotent or made repeatable with an idempotency key before any retry exists.
- [ ] Retryable and terminal errors are classified in code, not by instinct; 4xx validation errors are never retried.
- [ ] Attempts are bounded by both count and overall deadline, and each attempt has its own timeout.
- [ ] Waits honor the server's `Retry-After` when present.
- [ ] Cancellation is checked before every attempt and wait, and mid-wait too.
- [ ] Giving up raises the last error with the attempt count, and retry rates are measurable.