Circuit Breaker

Intent

Wrap calls to a dependency in a small state machine. While closed, calls pass and failures are counted. After failures cross a threshold the breaker opens: calls fail immediately without touching the dependency, giving it room to recover and freeing the callers. After a cool-off, the breaker half-opens and lets one probe through; success closes it again, failure reopens it.

Forces

• Hopeful retries vs retry storms: when every caller retries a failing dependency, each one multiplies the load exactly when the dependency is least able to serve it.
• Fail fast vs fail wrongly: while open, callers get instant errors - including for work that might have succeeded.
• Sensitivity vs flapping: a threshold too low trips on single blips; too high lets a dead dependency consume pools and threads first.
• Shared state vs local view: with ten instances, one instance's closed breaker says nothing about what the other nine saw.

Therefore

Track recent failures per dependency and trip into an open state at a threshold, fail calls fast while open, then admit a limited probe after a cool-off to test recovery.

The breaker is a State machine with three states - closed, open, half-open - sitting in front of one dependency, the same position a Proxy takes. While open it raises a distinct error so callers can react: serve a Fallback, queue the work, or surface a clean "temporarily unavailable".

Structure

stateDiagram-v2
  [*] --> closed
  closed --> open : consecutive failures reach threshold
  open --> halfOpen : cool-off elapses
  halfOpen --> closed : probe succeeds
  halfOpen --> open : probe fails

Participants

• Breaker (Breaker): the guard. Holds the state, the failure count, the threshold, the cool-off, and the clock. Raises CircuitOpenError while open.
• Dependency (PaymentGateway): the remote call the breaker guards. One breaker per dependency and operation.
• Caller: calls the breaker as if it were the dependency; on CircuitOpenError it degrades instead of waiting.
• Clock (now): injected so transitions are testable without sleeping.

Use it when

• A dependency fails in episodes - a bad deploy, a stalled database, an overload - and callers would otherwise wait or retry through the whole episode.
• Callers pile up behind a failing dependency: threads held, connection pools drained, latency cascading upstream.
• You need the dependency to be left alone to recover: open means zero traffic, which is the only thing that lets an overloaded service recover.
• A burst of failures means "stop" rather than "try again" - paying before you know the gateway is down is worse than failing the checkout.

Do not use it when

• The failure is rare and returns instantly. An occasional 500 that comes back in 30 ms costs nothing; a retry handles it. A breaker adds state to guard a problem you do not have.
• The operation must be attempted (or explicitly deferred) for correctness - capturing a payment. Failing fast to the user is wrong; queue the work with Command and a durable queue instead.
• Calls are local and cannot hang. No network, no failure mode worth a state machine.
• Your framework ships one (Resilience4j, .NET Microsoft.Extensions.Http.Resilience, Go libraries). Configure it; the entry below shows what it does inside, not that you should hand-roll it in production paths.
• Every failure means "this request is bad" (validation). The breaker would open on your clients' bugs. Breakers are for dependencies failing, not callers.

Application grill

Inspect first: the calls to the failing dependency and the client library wrapping them; whether one process or many instances serve the traffic, which decides per-instance versus shared breaker state; existing resilience libraries before adding a second.

Missing pieces: single instance: a small in-process breaker (a counter and three states) or a library (opossum in Node, resilience4j on the JVM, Polly in .NET). Many instances tripping independently still hammer the dependency as a herd; shared state in Redis makes them open together and adds a datastore with its own failure modes - start per-instance.

Ask before you build:

• Scale: how many instances run, and is per-instance tripping acceptable or must the fleet open as one?
• Performance: what latency counts as a failure - is there a per-dependency timeout budget feeding the breaker?
• Reliability: is fail-fast acceptable to callers while open, and what does the single half-open probe test?
• Budget: is shared breaker state worth a second datastore, or does per-instance cover the failure pattern?
• Implementation: how do on-call engineers see that a breaker is open (a metric, a log, an alert)?

Implementation steps

1. Give the wrapped calls deadlines first (Timeout); a hang must count as a failure or the breaker never trips.
2. Classify errors: count dependency failures (5xx, timeouts, connection errors); do not count caller errors (4xx, business rejections).
3. Store per breaker: state, consecutive failure count, threshold (5 is a common start), cool-off (10–60 s), and the time it opened. Key it by dependency and operation - the payment breaker must not trip because shipping is down.
4. On failure in closed state, increment; at the threshold, open and record the time. On success, reset the count.
5. While open, fail immediately with a distinct CircuitOpenError - do not touch the dependency, do not let callers retry it.
6. After the cool-off, admit a limited probe (one call, or a small rate): success closes the breaker and resets the count; failure reopens and restarts the cool-off.
7. Decide the state's scope: per process instance is simplest and each instance trips independently; a shared store gives one view at the cost of a read on every call.
8. Log or emit a metric on every transition; an open breaker is an incident signal, not a silent optimization.

Example

Checkout charges a payment gateway. The gateway's deploy is broken from t=1000 to t=2000 on our injected clock. Two failed charges trip the breaker; the third caller fails instantly without touching the gateway; after the 5-second cool-off a probe succeeds and traffic resumes.

interface PaymentGateway {
  charge(cents: number): string;
}

class CircuitOpenError extends Error {
  constructor() {
    super("circuit open: dependency is down");
    this.name = "CircuitOpenError";
  }
}

// The dependency: 503 while its deploy is broken, then it recovers.
class FlakyGateway implements PaymentGateway {
  constructor(
    private readonly brokenUntil: number,
    private readonly now: () => number,
  ) {}
  charge(cents: number): string {
    if (this.now() < this.brokenUntil) throw new Error("503 unavailable");
    return `charged ${cents}`;
  }
}

class Breaker implements PaymentGateway {
  private state: "closed" | "open" | "halfOpen" = "closed";
  private failures = 0;
  private openedAt = 0;

  constructor(
    private readonly inner: PaymentGateway,
    private readonly threshold: number, // consecutive failures before opening
    private readonly coolOffMs: number, // how long the breaker stays open
    private readonly now: () => number = Date.now,
  ) {}

  charge(cents: number): string {
    if (this.state === "open") {
      if (this.now() - this.openedAt < this.coolOffMs)
        throw new CircuitOpenError(); // fail fast: no call is made
      this.state = "halfOpen"; // cool-off over: this call is the probe
    }
    try {
      const result = this.inner.charge(cents);
      this.failures = 0;
      this.state = "closed";
      return result;
    } catch (e) {
      const probe = this.state === "halfOpen";
      this.state = "closed";
      if (probe || ++this.failures >= this.threshold) this.trip();
      throw e;
    }
  }

  private trip() {
    this.state = "open";
    this.openedAt = this.now();
    this.failures = 0;
    console.log(`  [breaker] open at t=${this.openedAt}`);
  }
}

let clock = 1_000;
const breaker = new Breaker(new FlakyGateway(2_000, () => clock), 2, 5_000, () => clock);

try { breaker.charge(500); } catch {} // 503, failure 1
try { breaker.charge(500); } catch {} // 503, failure 2 -> opens
try {
  breaker.charge(500);
} catch (e) {
  console.log(`${(e as Error).name}: no gateway call was made`);
  // CircuitOpenError: no gateway call was made
}

clock += 6_000; // cool-off (5 s) has passed by t=7000
console.log(breaker.charge(500)); // probe succeeds -> closed, charged 500

In concurrent code the same logic needs a lock (or atomics) around the state, and "one probe" must mean one at a time: admit a single in-flight probe and reject or queue the rest until it resolves.

Trade-offs

Gains

• A failing dependency stops receiving traffic, which is often the only condition under which it recovers.
• Callers fail in microseconds instead of queuing behind a dead service.
• CircuitOpenError makes the outage explicit and branchable.

Costs

• State must be scoped and, under concurrency, synchronised; instances trip independently unless you share state.
• Thresholds and cool-offs need tuning per dependency; wrong ones flap or open too late.
• While open, work that would have succeeded is rejected.
• A probe can re-trip the breaker and extend the outage window.

Relations

• State: the breaker is a three-state machine; this entry implements it with an enum because the states are few, but the State pattern applies as transitions grow.
• Proxy: the breaker stands in front of the dependency with the same interface; unlike a classic proxy it is usually applied by the caller, like a Decorator.
• Timeout: feeds it - a hang is a failure, so calls need deadlines before the breaker can count them.
• Retry: a retry loop must treat CircuitOpenError as terminal; retrying an open breaker is the storm the breaker exists to stop.
• Fallback: what callers serve while the breaker is open.
• Command: work that must not be dropped while the breaker is open goes to a durable queue instead of the caller's error path.

Agent checklist

• One breaker per dependency and operation, not one global breaker.
• Calls carry timeouts first, so hangs count as failures.
• Caller errors (4xx, validation) do not advance the failure count.
• CircuitOpenError is a distinct type, and no retry loop in the path retries it.
• The cool-off admits a limited probe, and success resets the failure count.
• Transitions are logged or emitted as metrics, and the state's scope (per instance or shared) is a written decision.