Retries¶

SMG retries failed attempts with exponential backoff and jitter, so transient worker failures are absorbed by the gateway instead of reaching clients, without overwhelming workers that are recovering.

Overview¶

Why Retries?¶

Transient failures are common in distributed systems:

• Network timeouts: Temporary network congestion or packet loss
• Worker overload: Temporary capacity limits (429 responses)
• Intermittent errors: Brief service interruptions during deployments
• Connection issues: Worker restart or network partition

Without retries, every transient failure becomes a client-visible error. With retries, SMG handles these automatically.

How a Retry Works¶

1. SMG dispatches the attempt and checks the response status as soon as the worker answers, before any of the body reaches the client.
2. If the status is retryable, the response is not marked terminal, and attempts remain, SMG waits for the backoff delay.
3. The next attempt runs worker selection again. Workers that are unhealthy, have an open circuit breaker, or are vetoed by overload protection are skipped. The policy can still pick the same worker if it remains eligible, for example under cache_aware or consistent_hashing affinity.
4. When a response is not retried, or no attempts remain, SMG returns it to the client.

--retry-max-retries counts attempts, including the first: the default 5 allows the initial attempt plus up to four retries, and 1 disables retries.

A worker that has returned a success status and started streaming its response is never retried, even if the stream fails later.

HTTP-router responses relayed from a worker carry an x-smg-routed-worker-id header with the URL of the worker that produced them (the decode worker in PD mode), so a client can see where the final attempt landed.

Where Retries Apply¶

Exponential Backoff with Jitter¶

SMG uses exponential backoff with jitter to space out retry attempts:

base  = min(initial_backoff_ms * backoff_multiplier ^ n, max_backoff_ms)
delay = base * (1 + uniform(-jitter_factor, +jitter_factor))

n is 0 before the first retry, 1 before the second, and so on. The cap applies before jitter, and SMG refuses to start if --retry-jitter-factor is outside 0.0–1.0.

Example Progression¶

With default settings (5 attempts, so up to 4 retries):

Why Jitter?¶

Without jitter, if multiple requests fail simultaneously, they all retry at exactly the same time—potentially overwhelming the recovering service. Jitter spreads out retries randomly to prevent this "thundering herd" problem.

Retryable Status Codes¶

SMG retries responses with these status codes. The list is global: per-worker resilience settings cannot add or remove codes from it.

SMG's own errors use these codes too. In regular HTTP mode, a failed connection to the worker and an upstream timeout are both a 500, and in PD mode a transport failure on either leg is a 502; on the HTTP and gRPC routers, "no available workers" (every candidate unhealthy or circuit-open) is a 503.

Requests with other status codes (e.g., 400 Bad Request, 401 Unauthorized) are not retried because they would likely fail again.

Never Retried¶

Some responses carry a retryable status but go straight back to the client, because another attempt cannot succeed within a backoff window:

• Worker overload sheds: 503 with error code worker_overload_protection_shed and a Retry-After header. The overload veto only changes when worker loads are polled again. See Overload Protection.
• Tenant rate-limit denials (gRPC router): 429 with tenant_rate_limit_exceeded. A tenant's budget is reserved once per request, before dispatch, so a denial is returned with its Retry-After, and retry attempts never reserve again. See Tenant Rate Limiting.
• Admission rejections: 429 admission_queue_full and 503 admission_queue_timeout come from the admission layer in front of the router and never reach the retry loop. See Rate Limiting.
• A few other gateway errors that another attempt cannot fix are marked terminal the same way.

Configuration¶

smg launch \
  --worker-urls http://w1:8000 http://w2:8000 \
  --retry-max-retries 5 \
  --retry-initial-backoff-ms 50 \
  --retry-max-backoff-ms 30000 \
  --retry-backoff-multiplier 1.5 \
  --retry-jitter-factor 0.2

Parameters¶

Per-Worker Overrides¶

The worker spec's resilience block accepts retry fields (max_retries, initial_backoff_ms, max_backoff_ms, backoff_multiplier, jitter_factor, and disable_retry), but in v1.11.0 they have no effect. Worker registration leaves them out of the spec it stores on the worker, and a per-model retry policy is only created from retry fields in that stored spec, so none is created and every request uses the router flags above. Use --retry-* and --disable-retries instead.

The same block's circuit-breaker settings do apply, including two status-code lists that affect the circuit breaker, not retries: retryable_status_codes (statuses counted as circuit-breaker failures) and capacity_status_codes (statuses treated as capacity pushback). Narrowing either list never makes a status non-retryable. See Interaction with Circuit Breakers.

Recommended Configurations¶

smg launch \
  --retry-max-retries 2 \
  --retry-initial-backoff-ms 10 \
  --retry-max-backoff-ms 100

smg launch \
  --retry-max-retries 3 \
  --retry-initial-backoff-ms 100 \
  --retry-backoff-multiplier 2.0

smg launch \
  --retry-max-retries 10 \
  --retry-initial-backoff-ms 100 \
  --retry-max-backoff-ms 60000 \
  --retry-backoff-multiplier 2.0

smg launch --disable-retries

Retries and Request Bodies¶

The HTTP router can only retry a request whose body it buffered. For each request it decides whether to buffer the body or stream it to the worker unread:

• With retries enabled, a body that nothing else needs to parse is buffered up to --max-buffered-request-bytes (default 1 MiB) so it stays retryable. A larger body streams to the worker and gets a single attempt.
• Requests the router must parse anyway, for example under the default cache_aware policy, always buffer and keep their retries.
• With retries disabled, eligible bodies stream at any size.

Raise --max-buffered-request-bytes to keep larger requests retryable, at the cost of router memory. smg_router_request_body_path_total{reason="retry_forfeited"} counts requests that gave up their retries to stream. See Request Streaming and Upstream Connections for the full decision.

Pre-Response Resends¶

Separately from these retries, the HTTP router resends a request once, immediately, when the send failed before the worker produced any response, most often because the backend had already closed the pooled connection. The resend uses no backoff and no retry attempt, happens even with --disable-retries, and is counted in smg_router_upstream_send_retries_total. Timeouts, streamed bodies, and upstream bodies of 1 MiB or more are not resent. See Resending Pre-Response Failures.

Interaction with Circuit Breakers¶

Retries and circuit breakers meet in worker selection:

• Open circuits are skipped. Every attempt, including the first, chooses only among workers whose circuit is closed or half-open, so a retry never goes to a worker whose circuit has opened in the meantime. A half-open circuit does not throttle traffic: its worker is as selectable as one with a closed circuit.
• "No available workers." If every candidate is unhealthy or circuit-open, the attempt fails with 503 no_available_workers. The HTTP router (regular and PD) retries it like any other retryable failure: it backs off and selects again, and a circuit may have moved to half-open by then. The gRPC pipeline selects workers for the first attempt before its retry loop, so there the 503 is returned at once unless it happens on a retry.
• Every attempt is recorded. Each attempt's status counts toward the circuit breaker of the worker that served it.
• Capacity pushback is retried but not counted. A 429 (by default) is retried, but records neither a failure nor a success on the breaker, so a busy worker's circuit does not open under load. Per worker, resilience.capacity_status_codes replaces that list, for example to treat 503 as pushback. Whether a capacity status is retried still follows the global retryable list.
• Client-caused aborts are not counted. Aborted streamed uploads (408, 413, 400) say nothing about the worker.

See Circuit Breakers for thresholds and state transitions.

Monitoring¶

Metrics¶

worker_type is regular, prefill, or decode for local workers and external for the OpenAI provider router.

Useful PromQL Queries¶

# Retries per second
sum(rate(smg_worker_retries_total[5m]))

# Requests that ran out of attempts, per second
sum(rate(smg_worker_retries_exhausted_total[5m]))

# Average backoff delay
sum(rate(smg_worker_retry_backoff_seconds_sum[5m]))
  / sum(rate(smg_worker_retry_backoff_seconds_count[5m]))

# 99th percentile backoff per retry number, over the last minute
# (a summary: SMG exports precomputed quantiles, not _bucket series)
max by (attempt) (smg_worker_retry_backoff_seconds{quantile="0.99"})

What to Watch¶

Tuning Guidelines¶

What's Next?¶