Health Checks¶

Background health checks continuously probe every worker, take unhealthy workers out of the selection pool without waiting for requests to fail, and return them when they recover.

Overview¶

Why Health Checks?¶

Without proactive health checks:

• Reactive detection: Failures only discovered when real requests fail
• Wasted requests: Multiple requests may fail before worker is marked unhealthy
• Slower recovery: No way to know when a worker has recovered without trying it

With health checks:

• Proactive detection: Unhealthy workers leave the pool without waiting for requests to fail
• Fast recovery: Workers rejoin the pool as soon as they're healthy
• Fewer wasted requests: Real requests only go to verified healthy workers

How It Works¶

SMG probes each registered worker on its own schedule. A newly registered worker is probed immediately, then once every --health-check-interval-secs. Probes run concurrently, up to 128 at a time, so a slow worker does not delay the others.

Probes by Worker Type¶

The ZMQ probe is also what reconnects a restarted engine, so health checks stay on for ZMQ workers even when they are disabled in configuration. Under --upstream-http2, HTTP probes use the same protocol as request traffic: HTTP/2 for workers that negotiated it at registration, HTTP/1.1 for the rest (see Request Streaming).

The HTTP probe sends the API key in the Authorization header, as regular-mode HTTP requests do, so on a plain http:// worker URL the key crosses the network unencrypted.

Worker States¶

HTTP and gRPC workers become Ready as soon as their registration completes, because registration has already reached the worker. ZMQ workers stay Pending until the engine completes its handshake, then become Ready immediately.

The smg_worker_health gauge collapses these to 1 (Ready) and 0 (anything else), so existing dashboards continue to work.

State Transitions¶

The failure count restarts when a worker enters NotReady, so a Ready worker that stops answering reaches Failed after 4 × --health-failure-threshold consecutive failed probes: 12 probes, about 12 minutes at the default 60-second interval.

Failed is not terminal. A failed worker that is not removed keeps being probed, so an engine that restarts on the same address rejoins without being registered again.

Worker Auto-Recovery¶

--remove-unhealthy-workers (alias --worker-auto-recovery) decides what happens to a worker that reaches Failed:

The default follows --service-discovery. With discovery on, removal is followed by re-registration, so recovery completes on its own. Without discovery, nothing would add a removed worker back, so the default keeps failed workers in place instead of permanently shrinking a static fleet.

smg launch (the Python launcher from pip, and the container image) also accepts the alias, as --worker-auto-recovery and --no-worker-auto-recovery. The smg binary has no --no- form; turn auto-recovery off there with --remove-unhealthy-workers=false.

Removal targets are resolved from registered worker URLs and pinned to the failed worker's own revision, so only that registration is removed: sibling DP ranks behind the same address stay registered. Workers imported from mesh peers are never removed locally; the gateway that owns them manages them.

Configuration¶

smg \
  --worker-urls http://w1:8000 http://w2:8000 \
  --health-check-interval-secs 60 \
  --health-failure-threshold 3 \
  --health-success-threshold 2 \
  --health-check-timeout-secs 5 \
  --health-check-endpoint /health

Parameters¶

Per-Worker Overrides¶

Workers registered through the worker API can override the gateway defaults in a health block of the worker spec; unset fields keep the gateway values. The same block can be changed later with PATCH /workers/{worker_id}.

{
  "url": "http://worker-3:8000",
  "health": {
    "check_interval_secs": 15,
    "timeout_secs": 3,
    "failure_threshold": 5,
    "success_threshold": 1,
    "drain_settle_secs": 30
  }
}

The probe endpoint and the auto-recovery setting apply gateway-wide. Workers created by service discovery use the gateway defaults.

Recommended Configurations¶

smg \
  --health-check-interval-secs 10 \
  --health-failure-threshold 2 \
  --health-check-timeout-secs 3

smg \
  --health-check-interval-secs 120 \
  --health-failure-threshold 5 \
  --health-success-threshold 3

smg \
  --health-check-interval-secs 30 \
  --health-failure-threshold 3 \
  --health-success-threshold 2 \
  --health-check-timeout-secs 5

smg --disable-health-check

Worker Health Endpoint¶

SMG expects HTTP workers to provide a health endpoint that returns:

• 2xx status code: Worker is healthy
• Any other status or timeout: Worker is unhealthy

Example Health Endpoint (vLLM)¶

vLLM workers expose /health by default:

# vLLM automatically provides /health endpoint
vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000

Example Health Endpoint (SGLang)¶

SGLang workers expose /health by default:

# SGLang automatically provides /health endpoint
python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --port 8000

Custom Health Endpoint¶

If your worker uses a different health endpoint:

smg \
  --worker-urls http://worker:8000 \
  --health-check-endpoint /api/health

Gateway Probe Endpoints¶

The probes above are how SMG checks its workers. For Kubernetes, load balancers, and uptime monitors, SMG answers its own probes:

/readiness returns 503 with one of these reasons:

The readiness decision is recomputed from worker registry events (and at least once per second) and served from memory, so probes answer in constant time regardless of fleet size. Probe responses, including the expected 503s, are logged at DEBUG level, so polling does not flood the log with errors.

Set --health-check-port <port> to also serve /liveness, /readiness, and /health on a dedicated plain-HTTP listener with its own runtime thread, so a saturated gateway cannot starve its probes. The routes stay available on the main port. See Dedicated Probe Port for a Kubernetes example.

Interaction with Circuit Breakers¶

Health checks and circuit breakers are independent gates, and a worker is selected only when both allow it:

Key differences:

• Health checks: Proactive background monitoring (no request impact)
• Circuit breakers: Reactive detection based on real request failures

Both are recommended for production deployments. The hash-based policies (consistent_hashing and prefix_hash) also skip a worker whose circuit breaker is open, because the router removes unavailable workers before the policy runs.

Monitoring¶

Metrics¶

Useful PromQL Queries¶

# Current health status per worker
smg_worker_health

# Count of unhealthy workers
count(smg_worker_health == 0)

# Health check success rate
rate(smg_worker_health_checks_total{result="success"}[5m]) /
rate(smg_worker_health_checks_total[5m])

# Failed checks per minute
rate(smg_worker_health_checks_total{result="failure"}[1m]) * 60

Alert Thresholds¶

Alerting Example¶

groups:
  - name: smg-health-checks
    rules:
      - alert: WorkerUnhealthy
        expr: smg_worker_health == 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Worker {{ $labels.worker }} is unhealthy"

      - alert: MajorityUnhealthy
        # Removed workers report -1, so count only registered ones
        expr: count(smg_worker_health == 0) > count(smg_worker_health >= 0) / 2
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Majority of workers are unhealthy"

Tuning Guidelines¶

What's Next?¶