A Circuit Breaker in Front of the Model Router

There is a circuit breaker in front of our model router. It exists because of a specific failure shape that only shows up when a provider is unwell rather than down.

A provider that is down is easy. Connections refuse, you catch it, you fail over, everyone goes home. A provider that is degraded is the expensive case: requests are accepted, held, and eventually time out. You do not learn anything for thirty seconds, and you learn it once per request.

With live conversations in flight, that is the whole problem. Every conversation that starts during the bad window independently discovers the outage by waiting.

The shape of the failure

Without a breaker, each request pays full price for information the last request already had. Twenty conversations means twenty thirty-second waits, twenty timeouts, twenty failovers, and twenty people wondering what happened to the agent.

Retries make it worse, not better. A degraded provider is usually degraded because it is overloaded, and the correct-looking response to a timeout is to try again, which is a load test dressed as error handling.

A breaker's job is to make the first few requests pay for the discovery and let everything after that fail immediately and cheaply.

The numbers

failureThreshold:      3
resetSeconds:          120
minOpenSeconds:        10
probeIntervalSeconds:  5
probeLockSeconds:      4
recoverySuccesses:     2

Three failures opens the circuit. Once open, requests for that model stop being attempted and route elsewhere.

Three, not one. A single timeout is noise: a network blip, a slow region, one unlucky request. Opening on one failure means flapping, and a breaker that flaps is worse than none, because now you have two sources of instability instead of one.

Ten seconds minimum open. This is the one that is not obvious. Even if a probe succeeds immediately, the circuit stays open for ten seconds. A provider recovering from overload is fragile, and the reward for detecting recovery in the first second is dumping your entire held load onto it and knocking it over again. The minimum is a deliberate refusal to be clever.

Two successes to close. One success can be luck. Two consecutive successes is weak evidence, which is the right amount of evidence for a decision that is cheap to revisit.

The probe lock

State lives in Redis, keyed per model:

model:circuit:<model>
model:failures:<model>
model:probe-lock:<model>

The first two are the obvious ones. The third is the one that gets forgotten.

We run several service instances. Every one of them sees the circuit open, and every one of them wants to know whether the provider is back. Without coordination, "send a probe every five seconds" becomes every instance probing every five seconds, and a struggling provider gets a synchronised burst from your entire fleet on a five second cycle.

So probing takes a short lock with a four second lease. One instance probes, the rest read the result. The lease is shorter than the interval, so an instance that dies mid-probe releases the lock by expiry rather than wedging the circuit permanently open.

That lock is the difference between a breaker that protects a struggling provider and one that is a distributed denial of service with good intentions.

Open state is cached in memory

The set of open circuits is held in a local Set and refreshed in bulk, not read from Redis per request.

Checking whether a circuit is open has to be effectively free. If it costs a network round trip, you have added latency to every request in exchange for saving latency during outages, and outages are rare. The in-memory read is stale by up to one refresh interval, which is fine: being a few seconds late to notice a provider recovered costs almost nothing, and the failure counters that decide when to open are still authoritative in Redis where every instance contributes to the same count.

Consistency where the decision is made, speed where the decision is read.

What it does not do

It does not know why the provider failed. Timeout, rate limit, 500, malformed response: all the same event. Classifying failures sounds smarter and mostly produces a rules engine that is wrong in novel ways during exactly the incidents where you need it to be simple.

It does not rescue the conversation that was in flight when the circuit opened. Those requests fail and the fallback handles them. The breaker protects the ones that have not started yet.

It does not replace timeouts. It is downstream of them. A breaker with no timeout underneath it never observes a failure at all, because nothing ever returns.

The reason it is worth the code

A circuit breaker is roughly a hundred lines and adds a piece of distributed state to your system, which is not free. The justification is narrow and it is this: without one, the cost of an outage scales with your traffic.

More customers means more conversations means more independent thirty-second discoveries of the same fact. The breaker makes that cost constant. Three requests learn, everything else is told.

That is the entire trade, and it is a good one.