A Circuit Breaker Implementation in Go That Saved Our Uptime

The incident

A downstream payment API started returning 503s for 30 seconds. Our service kept retrying, exhausting connection pools, which caused timeouts in callers up the chain. A 30-second blip became a 4-minute partial outage.

The fix: circuit breaker

type CircuitBreaker struct {
    mu            sync.Mutex
    state         State
    failureCount  int
    threshold     int
    lastFailure   time.Time
    cooldown      time.Duration
}

func (cb *CircuitBreaker) Call(fn func() error) error {
    cb.mu.Lock()
    if cb.state == StateOpen {
        if time.Since(cb.lastFailure) > cb.cooldown {
            cb.state = StateHalfOpen
        } else {
            cb.mu.Unlock()
            return ErrCircuitOpen
        }
    }
    cb.mu.Unlock()

    err := fn()

    cb.mu.Lock()
    defer cb.mu.Unlock()
    if err != nil {
        cb.failureCount++
        cb.lastFailure = time.Now()
        if cb.failureCount >= cb.threshold {
            cb.state = StateOpen
        }
        return err
    }
    // Success — reset
    cb.failureCount = 0
    cb.state = StateClosed
    return nil
}

What I learned

The three-state model (Closed → Open → Half-Open → Closed) prevents the retry storm problem. The cooldown period is the critical parameter — too short and you hammer a recovering service; too long and you reject requests unnecessarily.