Circuit Breakers¶

Circuit breakers prevent cascade failures by stopping traffic to unhealthy workers. They're essential for maintaining system stability when workers fail.

Overview¶

Why Circuit Breakers?¶

Without circuit breakers, a failing worker can cause:

1. Wasted requests: Requests sent to failing workers timeout
2. Increased latency: Clients wait for timeouts before retry
3. Resource exhaustion: Connections pile up to dead workers
4. Cascade failures: Retry storms overwhelm remaining workers

Circuit breakers fail fast—they detect failing workers and stop sending traffic immediately.

How It Works¶

Each worker has its own circuit breaker with three states:

State Transitions¶

Closed → Open¶

The circuit opens when:

consecutive_failures >= failure_threshold

A single successful request resets consecutive_failures to zero. --cb-window-duration-secs is accepted and validated but is not consumed by the state machine — failures are tracked with a running consecutive-failure counter rather than a sliding window.

Open → Half-Open¶

Once the circuit has been open for --cb-timeout-duration-secs, the next check of the worker's state (for example, during worker selection) moves it to half-open.

Half-Open → Closed¶

If --cb-success-threshold consecutive requests succeed, the circuit closes and normal operation resumes.

Half-Open → Open¶

If any request fails during half-open, the circuit reopens immediately and the open timeout starts over.

What Counts as a Failure¶

The breaker learns from the status of each response a worker sends back. Since v1.10, capacity pushback is not a failure:

• Why 429 is excluded. A busy engine answers 429 to say "not now". Counting that as a failure would open the breakers of the busiest workers during a load spike and push their traffic onto the rest. The gateway still retries a 429 when retries are enabled, but the response records no breaker sample in either direction, so it cannot close a half-open breaker either.
• gRPC workers. gRPC status codes are mapped to HTTP statuses first: RESOURCE_EXHAUSTED becomes 429 (pushback), UNAVAILABLE becomes 503 and DEADLINE_EXCEEDED becomes 504 (failures), and INVALID_ARGUMENT becomes 400 (a success for the breaker).
• Gateway sheds never count. Responses the gateway produces without a worker answering, such as admission sheds, overload protection sheds (worker_overload_protection_shed), and no_available_workers, record nothing on any breaker.
• 503 is still a failure by default, because a 503 cannot be told apart from an outage. If a backend uses 503 for backpressure, add it to that worker's capacity codes.

Per-Worker Overrides¶

The resilience block of a worker spec (for example, in a POST /workers request body; see the Admin API) overrides the breaker settings for that worker:

{
  "url": "http://gpu1:8000",
  "resilience": {
    "capacity_status_codes": [429, 503],
    "cb_failure_threshold": 5,
    "cb_timeout_secs": 30
  }
}

A status list you set replaces the default instead of adding to it, so keep 429 in capacity_status_codes if you still want it treated as pushback.

Configuration¶

smg launch \
  --worker-urls http://w1:8000 http://w2:8000 \
  --cb-failure-threshold 5 \
  --cb-success-threshold 2 \
  --cb-timeout-duration-secs 30

Parameters¶

The thresholds must be at least 1, and the durations must be greater than 0.

Configuration Examples¶

smg launch \
  --cb-failure-threshold 3 \
  --cb-timeout-duration-secs 30

smg launch \
  --cb-failure-threshold 20 \
  --cb-success-threshold 5 \
  --cb-timeout-duration-secs 120

Tuning Guidelines¶

Example Scenarios¶

Normal Operation¶

Worker Fails¶

Once every worker for the model has an open circuit, requests get 503 no_available_workers. For external provider workers the code is service_unavailable.

Recovery¶

Monitoring¶

Metrics¶

The worker label is the worker URL.

Useful PromQL Queries¶

# Current circuit breaker states
smg_worker_cb_state

# Workers with open circuits
count(smg_worker_cb_state == 1)

# State transitions rate
rate(smg_worker_cb_transitions_total[5m])

# Failure share per worker
sum by (worker) (rate(smg_worker_cb_outcomes_total{outcome="failure"}[5m]))
  / sum by (worker) (rate(smg_worker_cb_outcomes_total[5m]))

What to Watch¶

Alerting Example¶

groups:
  - name: smg-circuit-breakers
    rules:
      - alert: CircuitBreakerOpen
        expr: smg_worker_cb_state == 1
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Circuit breaker open for {{ $labels.worker }}"

      - alert: AllCircuitsOpen
        # Removed workers report -1, so compare against live workers only.
        expr: count(smg_worker_cb_state == 1) == count(smg_worker_cb_state >= 0)
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "All worker circuit breakers are open"

Interaction with Other Features¶

Retries¶

• For local workers, each retry attempt runs worker selection again, so a worker whose circuit is open is skipped.
• If every worker is unavailable, the attempt gets 503 no_available_workers.
• A half-open worker is selected like any other; its outcomes decide whether the circuit closes or reopens.
• A 429 from a worker is retried when retries are enabled, but leaves that worker's breaker untouched.

Health Checks¶

Circuit breakers and health checks are independent gates: health checks probe workers in the background, while circuit breakers react to real request outcomes. A worker receives traffic only when both allow it, and when overload protection, if enabled, has not vetoed it:

Disabling Circuit Breakers¶

In some cases, you may want to disable circuit breakers:

smg launch --worker-urls http://w1:8000 --disable-circuit-breaker

The breakers keep recording outcomes, so the metrics still move, but no circuit opens unless a worker sets its own cb_failure_threshold in its resilience block.

What's Next?¶