We ran a checkout pipeline that fronted a legacy payments gateway. The gateway had a 3-second timeout and, once every few weeks, it would wedge for ten minutes. The client pool backed up, request queues filled, and within 30 seconds the entire web tier was burning goroutines on calls that were never going to succeed. Every downstream retry made the recovery slower. A circuit breaker fixed that incident; this post is the version we actually shipped, not the textbook version.

The core contract is three states. Closed: traffic flows, we count failures. Open: we short-circuit every call, optionally with a fast-fail error, and start a cooldown timer. Half-Open: after the cooldown, we probe with a limited number of trial requests; success flips us back to Closed, a single failure sends us back to Open. The two decisions that separate a useful breaker from a dangerous one are: what counts as a failure, and how the counts are tracked.

Failure must be defined at the caller’s boundary, not by HTTP status alone. A 503 from a healthy load balancer is a real upstream failure; a 4xx caused by a bad request is the caller’s bug and should not trip the breaker, or one misbehaving client will take down a service for everyone. We settled on: connection errors, timeouts, and 5xx count against the window. Everything else passes through.

For accounting, the naive approach locks a sync.Mutex around a slice of timestamps. It works but it serializes every request through a mutex and burns allocations under load. Our hot path uses a fixed-size ring buffer of uint64 buckets plus sync/atomic for the counters. Each bucket is a second of the sliding window; a background cleanup advances the window cheaply instead of filtering a slice on every call. At 2,000 rps this kept the Allow() path to roughly 40ns per call with zero allocations, versus about 1.2us when we benchmarked the mutex-plus-slice version.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
package breaker

import (
	"sync/atomic"
	"time"
)

type bucket struct {
	ts       int64
	failures uint64
	total    uint64
}

type window struct {
	buckets [60]bucket // one per second, 60s window
}

// record advances the window and bumps the counters atomically.
func (w *window) record(failure bool) {
	now := time.Now().Unix()
	idx := int(now % 60)
	b := &w.buckets[idx]
	if atomic.LoadInt64(&b.ts) != now {
		// First write of this second: stamp it and zero the counts.
		// Concurrent writers racing the stamp both write now+0, which is
		// idempotent, then fall through to the atomic increments.
		atomic.StoreInt64(&b.ts, now)
		atomic.StoreUint64(&b.failures, 0)
		atomic.StoreUint64(&b.total, 0)
	}
	atomic.AddUint64(&b.total, 1)
	if failure {
		atomic.AddUint64(&b.failures, 1)
	}
}

// failureRate skips buckets outside the sliding 60s window.
func (w *window) failureRate(now time.Time) float64 {
	var f, t uint64
	for i := 0; i < len(w.buckets); i++ {
		b := &w.buckets[i]
		if now.Unix()-atomic.LoadInt64(&b.ts) > 60 {
			continue
		}
		f += atomic.LoadUint64(&b.failures)
		t += atomic.LoadUint64(&b.total)
	}
	if t == 0 {
		return 0
	}
	return float64(f) / float64(t)
}

Two tuning rules that cost us the most uptime:

Thresholds must be proportional to traffic. A 50% failure-rate threshold is fine at steady state but dangerous on a low-traffic service: five failed health checks out of ten requests trips the breaker during a deploy that would have succeeded. We use max(10, volume-scaled) semantics — trip only when you have both a minimum sample count and a failure ratio, computed on a rolling 60-second window. The minimum-sample guard alone prevented most of our false positives.

Half-open probing is a rate limit, not a binary coin flip. Letting exactly one probe through per cooldown period means a service that needs 200ms to drain its queue fails your one probe and resets the whole cooldown. We send a small batch (10% of normal peak concurrency, capped at 32) and trip Open again only if that batch’s failure rate exceeds the same threshold. Recovery goes from minutes of flapping to a single 30–60s settling period.

Every state transition is a first-class event, not a log line. We emit a metrics counter on each transition and alert on Closed→Open within one minute. The most useful debugging signal we found was the Open duration: if it consistently exceeds the configured cooldown, the downstream has a real queue-drain problem, not a blip. That led us to add jittered backoff to the caller’s retries — the breaker keeps upstream load to near zero, but the clients that received fast-fail errors still retried in lockstep and created thundering-herd rediscovery.

One more production note: don’t use a package that hides state from you. We run with a small homegrown breaker precisely because we need to inspect bucket snapshots and probe settings at runtime via a debug endpoint. The dependency-free requirement wasn’t ideology — it was because in an outage we wanted to read the breaker state without fighting a library’s API. Wire the breaker around a shared http.Client.Transport, expose State(), Snapshot(), and Reset(), and you’ll be the person who gets paged at 2am and actually knows what the dashboard means.

Circuit breakers are the cheapest form of bulkhead. One Allow() call, one counter, and a timed probe is enough to keep a wedged dependency from becoming a platform outage.