# Circuit Breaker > Count consecutive failures to a dependency, refuse calls while it looks broken, and let one probe test recovery - so a dead service cannot eat its callers. - kind: pattern - category: resilience - aliases: Breaker - canonical: https://aipatterns.dev/design-patterns/circuit-breaker - inspired-by: https://learn.microsoft.com/en-us/azure/architecture/patterns/circuit-breaker ## Intent Wrap calls to a dependency in a small state machine. While closed, calls pass and failures are counted. After failures cross a threshold the breaker opens: calls fail immediately without touching the dependency, giving it room to recover and freeing the callers. After a cool-off, the breaker half-opens and lets one probe through; success closes it again, failure reopens it. ## Forces - **Hopeful retries** vs **retry storms**: when every caller retries a failing dependency, each one multiplies the load exactly when the dependency is least able to serve it. - **Fail fast** vs **fail wrongly**: while open, callers get instant errors - including for work that might have succeeded. - **Sensitivity** vs **flapping**: a threshold too low trips on single blips; too high lets a dead dependency consume pools and threads first. - **Shared state** vs **local view**: with ten instances, one instance's closed breaker says nothing about what the other nine saw. ## Therefore **Track recent failures per dependency and trip into an open state at a threshold, fail calls fast while open, then admit a limited probe after a cool-off to test recovery.** The breaker is a [State](https://aipatterns.dev/design-patterns/state.md) machine with three states - closed, open, half-open - sitting in front of one dependency, the same position a [Proxy](https://aipatterns.dev/design-patterns/proxy.md) takes. While open it raises a distinct error so callers can react: serve a [Fallback](https://aipatterns.dev/design-patterns/fallback.md), queue the work, or surface a clean "temporarily unavailable". ## Structure ```mermaid stateDiagram-v2 [*] --> closed closed --> open : consecutive failures reach threshold open --> halfOpen : cool-off elapses halfOpen --> closed : probe succeeds halfOpen --> open : probe fails ``` ### Participants - **Breaker** (`Breaker`): the guard. Holds the state, the failure count, the threshold, the cool-off, and the clock. Raises `CircuitOpenError` while open. - **Dependency** (`PaymentGateway`): the remote call the breaker guards. One breaker per dependency and operation. - **Caller**: calls the breaker as if it were the dependency; on `CircuitOpenError` it degrades instead of waiting. - **Clock** (`now`): injected so transitions are testable without sleeping. ## Use it when - A dependency fails in episodes - a bad deploy, a stalled database, an overload - and callers would otherwise wait or retry through the whole episode. - Callers pile up behind a failing dependency: threads held, connection pools drained, latency cascading upstream. - You need the dependency to be left alone to recover: open means zero traffic, which is the only thing that lets an overloaded service recover. - A burst of failures means "stop" rather than "try again" - paying before you know the gateway is down is worse than failing the checkout. ## Do not use it when - The failure is rare and returns instantly. An occasional 500 that comes back in 30 ms costs nothing; a retry handles it. A breaker adds state to guard a problem you do not have. - The operation must be attempted (or explicitly deferred) for correctness - capturing a payment. Failing fast to the user is wrong; queue the work with [Command](https://aipatterns.dev/design-patterns/command.md) and a durable queue instead. - Calls are local and cannot hang. No network, no failure mode worth a state machine. - Your framework ships one ([Resilience4j](https://resilience4j.readme.io), .NET `Microsoft.Extensions.Http.Resilience`, Go libraries). Configure it; the entry below shows what it does inside, not that you should hand-roll it in production paths. - Every failure means "this request is bad" (validation). The breaker would open on your clients' bugs. Breakers are for dependencies failing, not callers. ## Application grill **Inspect first:** the calls to the failing dependency and the client library wrapping them; whether one process or many instances serve the traffic, which decides per-instance versus shared breaker state; existing resilience libraries before adding a second. **Missing pieces:** single instance: a small in-process breaker (a counter and three states) or a library (opossum in Node, resilience4j on the JVM, Polly in .NET). Many instances tripping independently still hammer the dependency as a herd; shared state in Redis makes them open together and adds a datastore with its own failure modes - start per-instance. **Ask before you build:** - **Scale:** how many instances run, and is per-instance tripping acceptable or must the fleet open as one? - **Performance:** what latency counts as a failure - is there a per-dependency timeout budget feeding the breaker? - **Reliability:** is fail-fast acceptable to callers while open, and what does the single half-open probe test? - **Budget:** is shared breaker state worth a second datastore, or does per-instance cover the failure pattern? - **Implementation:** how do on-call engineers see that a breaker is open (a metric, a log, an alert)? ## Implementation steps 1. Give the wrapped calls deadlines first ([Timeout](https://aipatterns.dev/design-patterns/timeout.md)); a hang must count as a failure or the breaker never trips. 2. Classify errors: count dependency failures (5xx, timeouts, connection errors); do not count caller errors (4xx, business rejections). 3. Store per breaker: state, consecutive failure count, threshold (5 is a common start), cool-off (10–60 s), and the time it opened. Key it by dependency and operation - the payment breaker must not trip because shipping is down. 4. On failure in closed state, increment; at the threshold, open and record the time. On success, reset the count. 5. While open, fail immediately with a distinct `CircuitOpenError` - do not touch the dependency, do not let callers retry it. 6. After the cool-off, admit a limited probe (one call, or a small rate): success closes the breaker and resets the count; failure reopens and restarts the cool-off. 7. Decide the state's scope: per process instance is simplest and each instance trips independently; a shared store gives one view at the cost of a read on every call. 8. Log or emit a metric on every transition; an open breaker is an incident signal, not a silent optimization. ## Example Checkout charges a payment gateway. The gateway's deploy is broken from t=1000 to t=2000 on our injected clock. Two failed charges trip the breaker; the third caller fails instantly without touching the gateway; after the 5-second cool-off a probe succeeds and traffic resumes. ```ts title="payments.ts" interface PaymentGateway { charge(cents: number): string; } class CircuitOpenError extends Error { constructor() { super("circuit open: dependency is down"); this.name = "CircuitOpenError"; } } // The dependency: 503 while its deploy is broken, then it recovers. class FlakyGateway implements PaymentGateway { constructor( private readonly brokenUntil: number, private readonly now: () => number, ) {} charge(cents: number): string { if (this.now() < this.brokenUntil) throw new Error("503 unavailable"); return `charged ${cents}`; } } class Breaker implements PaymentGateway { private state: "closed" | "open" | "halfOpen" = "closed"; private failures = 0; private openedAt = 0; constructor( private readonly inner: PaymentGateway, private readonly threshold: number, // consecutive failures before opening private readonly coolOffMs: number, // how long the breaker stays open private readonly now: () => number = Date.now, ) {} charge(cents: number): string { if (this.state === "open") { if (this.now() - this.openedAt < this.coolOffMs) throw new CircuitOpenError(); // fail fast: no call is made this.state = "halfOpen"; // cool-off over: this call is the probe } try { const result = this.inner.charge(cents); this.failures = 0; this.state = "closed"; return result; } catch (e) { const probe = this.state === "halfOpen"; this.state = "closed"; if (probe || ++this.failures >= this.threshold) this.trip(); throw e; } } private trip() { this.state = "open"; this.openedAt = this.now(); this.failures = 0; console.log(` [breaker] open at t=${this.openedAt}`); } } let clock = 1_000; const breaker = new Breaker(new FlakyGateway(2_000, () => clock), 2, 5_000, () => clock); try { breaker.charge(500); } catch {} // 503, failure 1 try { breaker.charge(500); } catch {} // 503, failure 2 -> opens try { breaker.charge(500); } catch (e) { console.log(`${(e as Error).name}: no gateway call was made`); // CircuitOpenError: no gateway call was made } clock += 6_000; // cool-off (5 s) has passed by t=7000 console.log(breaker.charge(500)); // probe succeeds -> closed, charged 500 ``` ```python title="payments.py" from typing import Callable, Protocol class CircuitOpenError(Exception): pass class PaymentGateway(Protocol): def charge(self, cents: int) -> str: ... class FlakyGateway: """503 while its deploy is broken, then it recovers.""" def __init__(self, broken_until: int, now: Callable[[], int]) -> None: self._broken_until, self._now = broken_until, now def charge(self, cents: int) -> str: if self._now() < self._broken_until: raise RuntimeError("503 unavailable") return f"charged {cents}" class Breaker: def __init__( self, inner: PaymentGateway, threshold: int, cool_off_ms: int, now: Callable[[], int], ) -> None: self._inner, self._threshold = inner, threshold self._cool_off, self._now = cool_off_ms, now self._state, self._failures, self._opened_at = "closed", 0, 0 def charge(self, cents: int) -> str: if self._state == "open": if self._now() - self._opened_at < self._cool_off: raise CircuitOpenError("circuit open: dependency is down") self._state = "halfOpen" # cool-off over: this call is the probe try: result = self._inner.charge(cents) self._failures, self._state = 0, "closed" return result except Exception as e: probe = self._state == "halfOpen" self._state = "closed" self._failures += 1 if probe or self._failures >= self._threshold: self._trip() raise e def _trip(self) -> None: self._state, self._opened_at, self._failures = "open", self._now(), 0 print(f" [breaker] open at t={self._opened_at}") clock = 1_000 now = lambda: clock # injected clock; reassigning clock moves time for both objects breaker = Breaker( FlakyGateway(broken_until=2_000, now=now), threshold=2, cool_off_ms=5_000, now=now ) for _ in range(2): try: breaker.charge(500) except RuntimeError: pass # 503, failures 1 and 2 -> opens try: breaker.charge(500) except CircuitOpenError as e: print(f"{type(e).__name__}: no gateway call was made") clock += 6_000 # cool-off (5 s) has passed print(breaker.charge(500)) # probe succeeds -> closed, charged 500 ``` ```go title="payments.go" package main import ( "errors" "fmt" "time" ) type PaymentGateway interface { Charge(cents int) (string, error) } var errUnavailable = errors.New("503 unavailable") // FlakyGateway 503s while its deploy is broken, then recovers. type FlakyGateway struct { brokenUntil time.Time now func() time.Time } func (g FlakyGateway) Charge(cents int) (string, error) { if g.now().Before(g.brokenUntil) { return "", errUnavailable } return fmt.Sprintf("charged %d", cents), nil } // ErrCircuitOpen is returned while the breaker is open. Callers must not retry it. var ErrCircuitOpen = errors.New("circuit open: dependency is down") type breakerState int const ( closed breakerState = iota open halfOpen ) type Breaker struct { inner PaymentGateway threshold int coolOff time.Duration now func() time.Time state breakerState failures int openedAt time.Time } func NewBreaker(inner PaymentGateway, threshold int, coolOff time.Duration, now func() time.Time) *Breaker { return &Breaker{inner: inner, threshold: threshold, coolOff: coolOff, now: now} } func (b *Breaker) Charge(cents int) (string, error) { if b.state == open { if b.now().Sub(b.openedAt) < b.coolOff { return "", ErrCircuitOpen // fail fast: no call is made } b.state = halfOpen // cool-off over: this call is the probe } result, err := b.inner.Charge(cents) probe := b.state == halfOpen b.state = closed switch { case err == nil: b.failures = 0 case probe: b.trip() default: b.failures++ if b.failures >= b.threshold { b.trip() } } return result, err } func (b *Breaker) trip() { b.state, b.openedAt, b.failures = open, b.now(), 0 fmt.Printf(" [breaker] open at t=%d\n", b.openedAt.UnixMilli()) } func main() { clock := time.UnixMilli(1_000) now := func() time.Time { return clock } breaker := NewBreaker( FlakyGateway{brokenUntil: time.UnixMilli(2_000), now: now}, 2, 5*time.Second, now) for i := 0; i < 2; i++ { breaker.Charge(500) // 503, failures 1 and 2 -> opens } if _, err := breaker.Charge(500); errors.Is(err, ErrCircuitOpen) { fmt.Println("CircuitOpenError: no gateway call was made") } clock = clock.Add(6 * time.Second) // cool-off (5 s) has passed if result, err := breaker.Charge(500); err == nil { fmt.Println(result) // probe succeeds -> closed, charged 500 } } ``` ```java title="Payments.java" import java.time.Instant; import java.util.function.LongSupplier; public class Payments { interface PaymentGateway { String charge(int cents); } static final class CircuitOpenException extends RuntimeException { CircuitOpenException() { super("circuit open: dependency is down"); } } /** 503 while its deploy is broken, then it recovers. */ record FlakyGateway(long brokenUntil, LongSupplier now) implements PaymentGateway { public String charge(int cents) { if (now.getAsLong() < brokenUntil) throw new IllegalStateException("503 unavailable"); return "charged " + cents; } } static final class Breaker implements PaymentGateway { private enum State { CLOSED, OPEN, HALF_OPEN } private final PaymentGateway inner; private final int threshold; private final long coolOffMs; private final LongSupplier now; private State state = State.CLOSED; private int failures = 0; private long openedAt = 0; Breaker(PaymentGateway inner, int threshold, long coolOffMs, LongSupplier now) { this.inner = inner; this.threshold = threshold; this.coolOffMs = coolOffMs; this.now = now; } public String charge(int cents) { if (state == State.OPEN) { if (now.getAsLong() - openedAt < coolOffMs) throw new CircuitOpenException(); // fail fast: no call is made state = State.HALF_OPEN; // cool-off over: this call is the probe } try { String result = inner.charge(cents); failures = 0; state = State.CLOSED; return result; } catch (RuntimeException e) { boolean probe = state == State.HALF_OPEN; state = State.CLOSED; if (probe || ++failures >= threshold) trip(); throw e; } } private void trip() { state = State.OPEN; openedAt = now.getAsLong(); failures = 0; System.out.println(" [breaker] open at t=" + openedAt); } } public static void main(String[] args) { long[] clock = {1_000}; Breaker breaker = new Breaker( new FlakyGateway(2_000, () -> clock[0]), 2, 5_000, () -> clock[0]); for (int i = 0; i < 2; i++) { try { breaker.charge(500); } catch (RuntimeException ignored) { } // 503 x2 -> opens } try { breaker.charge(500); } catch (CircuitOpenException e) { System.out.println("CircuitOpenException: no gateway call was made"); } clock[0] += 6_000; // cool-off (5 s) has passed System.out.println(breaker.charge(500)); // probe succeeds -> closed, charged 500 } } ``` In concurrent code the same logic needs a lock (or atomics) around the state, and "one probe" must mean one at a time: admit a single in-flight probe and reject or queue the rest until it resolves. ## Trade-offs **Gains** - A failing dependency stops receiving traffic, which is often the only condition under which it recovers. - Callers fail in microseconds instead of queuing behind a dead service. - `CircuitOpenError` makes the outage explicit and branchable. **Costs** - State must be scoped and, under concurrency, synchronised; instances trip independently unless you share state. - Thresholds and cool-offs need tuning per dependency; wrong ones flap or open too late. - While open, work that would have succeeded is rejected. - A probe can re-trip the breaker and extend the outage window. ## Relations - [State](https://aipatterns.dev/design-patterns/state.md): the breaker is a three-state machine; this entry implements it with an enum because the states are few, but the State pattern applies as transitions grow. - [Proxy](https://aipatterns.dev/design-patterns/proxy.md): the breaker stands in front of the dependency with the same interface; unlike a classic proxy it is usually applied by the caller, like a [Decorator](https://aipatterns.dev/design-patterns/decorator.md). - [Timeout](https://aipatterns.dev/design-patterns/timeout.md): feeds it - a hang is a failure, so calls need deadlines before the breaker can count them. - [Retry](https://aipatterns.dev/design-patterns/retry.md): a retry loop must treat `CircuitOpenError` as terminal; retrying an open breaker is the storm the breaker exists to stop. - [Fallback](https://aipatterns.dev/design-patterns/fallback.md): what callers serve while the breaker is open. - [Command](https://aipatterns.dev/design-patterns/command.md): work that must not be dropped while the breaker is open goes to a durable queue instead of the caller's error path. ## Agent checklist - [ ] One breaker per dependency and operation, not one global breaker. - [ ] Calls carry timeouts first, so hangs count as failures. - [ ] Caller errors (4xx, validation) do not advance the failure count. - [ ] `CircuitOpenError` is a distinct type, and no retry loop in the path retries it. - [ ] The cool-off admits a limited probe, and success resets the failure count. - [ ] Transitions are logged or emitted as metrics, and the state's scope (per instance or shared) is a written decision.