Circuit breakers: preventing cascading failures

In distributed architectures, failures are inevitable. What kills production systems is not a single failing service, but cascading failure.

If your recommendation service begins taking 10 seconds per request instead of 50ms, the API gateway continues waiting. Connection pools fill up, worker threads get blocked, memory spikes, and soon the checkout and login services crash too.

A Circuit Breaker prevents this by automatically cutting off requests to an unhealthy dependency before it starves the rest of the system.

The Three States of a Circuit Breaker

Just like an electrical breaker in your home, the software circuit breaker wraps remote calls and tracks failure rates across three states:

                  Failures > Threshold
       ┌─────────────────────────────────────────┐
       ▼                                         │
 ┌───────────┐    Success     ┌───────────┐      │
 │  CLOSED   │ ─────────────> │ HALF-OPEN │      │
 │ (Normal)  │ <───────────── │ (Testing) │      │
 └───────────┘    Test Pass   └───────────┘      │
       ▲                            ▲            │
       │         Timeout Passed     │            │
       └────────────────────────────┴────────────┘
                                      OPEN (Failing Fast)

1. Closed (Normal Operation)

Requests flow through directly to the downstream service.
The breaker counts successes, failures, and slow calls in a rolling time window.
As long as failure rates stay below the configured threshold (e.g. < 50% errors over 10 seconds), the circuit remains Closed.

2. Open (Fail-Fast Mode)

Once failures exceed the threshold, the circuit trips Open.
All subsequent calls to the downstream service are immediately aborted without ever making a network request.
The caller receives an instant fallback response or cached data.
Benefit: Gives the struggling downstream service time to recover and frees up upstream threads instantly.

3. Half-Open (Canary Testing)

After a sleep window (e.g. 30 seconds), the breaker transitions to Half-Open.
It permits a small number of canary probe requests to pass through.
If the canary requests succeed, the breaker resets to Closed.
If any canary request fails, the breaker trips back to Open for another cooldown period.

Code Example: Using a Breaker with Fallbacks

import CircuitBreaker from 'opossum';

async function fetchRecommendations(userId: string) {
  const res = await fetch(`https://recs.internal/users/${userId}`, { timeout: 1000 });
  return res.json();
}

const breaker = new CircuitBreaker(fetchRecommendations, {
  timeout: 1000,           // If action takes longer than 1s, trigger failure
  errorThresholdPercentage: 50, // When 50% of requests fail, trip open
  resetTimeout: 30000      // Wait 30s before trying Half-Open
});

// Graceful fallback when the breaker is OPEN
breaker.fallback(() => ({ items: ['default_trending_item_1', 'default_trending_item_2'] }));

const recommendations = await breaker.fire(currentUserId);

Essential Best Practices

Technique	Purpose
Fast Timeouts	Never let an HTTP request wait indefinitely; set aggressive timeouts (500ms–2s)
Meaningful Fallbacks	Return cached data, empty arrays, or degraded features instead of 500 crashes
Bulkhead Pattern	Isolate connection pools so one failing service cannot consume all network sockets
Observability & Alerts	Emit metrics whenever a circuit trips `OPEN` so DevOps is alerted before users notice