Circuit breaker

What is a circuit breaker?

A circuit breaker sits between your application and a dependency it calls, such as an external API, an iPaaS connector, or a payment provider, and watches how those calls go. While calls succeed it stays out of the way. Once failures cross a threshold it stops sending calls for a while and fails fast: every new call returns an error or a fallback straight away rather than waiting on a service that is already down.

The name comes from the electrical fuse. When too much current flows the circuit trips to prevent a fire; here, when too many calls fail, the breaker trips to stop the damage spreading.

It protects two sides at once. The failing dependency gets room to recover instead of being hammered by fresh calls, and the caller is spared a slow dependency tying up threads and database connections on every timeout until those run out and take down unrelated parts of the system. That is how one slow API becomes a cascading failure across a whole data pipeline.

The three states

A circuit breaker is a small state machine with three states.

• Closed. The normal state. Calls pass through to the dependency and the breaker counts recent failures. When failures cross the threshold, it trips to open.

• Open. Calls are rejected immediately, with no attempt to reach the dependency. A timer runs for the length of the open period, and when it expires the breaker moves to half-open.

• Half-open. A limited number of trial calls are let through to check whether the dependency has recovered. If they succeed, the breaker closes and traffic flows normally again. If any fails, it snaps back to open and restarts the timer.

The half-open step stops a recovering service from being flooded the instant it comes back, letting it prove itself on a few calls before you send the full load.

A worked example

Say you wrap calls to a shipping-rate API. You set the breaker to open when the failure rate reaches 50% over the last 100 calls, to stay open for 60 seconds, then to allow 10 trial calls in half-open. During a provider outage the failure rate climbs past 50%, the breaker opens, and for those 60 seconds your checkout returns a cached rate or a clear error in milliseconds instead of hanging on each timeout. Then it tries the 10 trial calls and either closes or opens again. Set that threshold too low and the breaker trips on normal blips; too high and it reacts too late.

Circuit breakers and retries

A retry and a circuit breaker answer opposite questions. A retry assumes the failure is temporary and tries the same call again. A circuit breaker assumes the failure is not temporary and stops trying for a while. They are meant to work together.

Put the retry inside the breaker, and make the retry give up the moment the breaker is open. If your retry policy keeps trying while the breaker is open, you rebuild the exact retry storm the breaker exists to stop, and the dependency never gets its quiet window to recover. Repeating calls also needs the operation to be safe to repeat, which is where idempotence comes in.

A circuit breaker is one of a small family of resilience controls. Retry policy, exponential backoff, and rate limit each handle a different failure mode, and real systems usually combine a few of them.