Overload Protection¶

Inference engines accept work past saturation. Once an engine's running batch is full, new requests wait in its scheduler queue, and every request routed there waits a little longer. Load balancing spreads traffic relative to the other workers, so when the whole fleet saturates, every engine's queue deepens together until latency collapses, and routing alone never turns a request away. Worker overload protection puts an absolute, per-worker ceiling on the load each engine reports: a worker at or above its ceiling is taken out of routing until its load drops back under, and a request whose every candidate worker is over the ceiling is rejected immediately with a 503 and a Retry-After hint instead of joining an engine queue.

Overload protection is opt-in. The load monitoring that feeds it is on by default.

Overview¶

Why Overload Protection?¶

Without it:

1. Engines queue past saturation: a worker whose batch is full keeps accepting requests into its waiting queue, so each request routed to it waits longer before its first token.
2. Relative balancing never refuses: load-aware policies pick the least-loaded worker, but when every worker is saturated they still pick one.
3. Gateway admission only counts its own traffic: rate limiting and the priority scheduler count requests in flight through this gateway instance. They cannot see a KV cache filled by long prompts or by another gateway replica sharing the same workers.

Overload protection reads what the engine itself reports, which is the same no matter how many gateway replicas share a worker.

How It Differs from Other Load Controls¶

SMG has several controls that react to load. They act at different points and do different things when they trip:

The layers stack: admission control bounds what the gateway accepts, the routing policy decides where a request goes, and overload protection decides which workers may receive it at all.

How It Works¶

Signals¶

Each load report is scored on two signals:

A worker is overloaded when either signal is at or above its threshold. Token usage is compared as a mean, not rank by rank: a saturated DP rank vetoes the worker only if it lifts the mean to the threshold, and the mean can reach the threshold while some ranks are still below it. The mean is the same signal --balance-token-usage-threshold reads, applied as an absolute ceiling instead of a fleet-wide spread.

Where the Signals Come From¶

The gateway's load monitor collects one report per worker per poll. The source depends on how the worker is connected:

Polling¶

• Workers are polled in groups (same model, worker type, and connection mode) every --load-monitor-interval seconds (default 10), with a floor of 1 second. The worker spec also has a load_monitor_interval_secs field, but in v1.11.0 worker registration does not copy it onto the worker, so it has no effect; use --load-monitor-interval.
• Only Ready workers are polled. HTTP load requests time out after 5 seconds.
• The overload check runs once per poll of each worker, against that worker's own thresholds, and the verdict is stored on the worker. Selection only reads the stored flag, so the check adds nothing per request. Only a poll can set the flag. A poll can also clear it, and so can the events listed under When a Worker Does Not Report.
• There is no hysteresis: the first report under every threshold clears the veto.

When a Worker Does Not Report¶

Overload protection fails open. A poll that produces no report (an unsupported backend, a timeout, an error, or an empty report) clears the worker's veto: no fresh signal means no opinion, and the worker stays routable. The veto is also cleared when a worker leaves Ready, is removed, or is replaced (PUT or PATCH on /workers/{worker_id}), and for every worker when the monitor has to rebuild its state (logged as a warning).

A field the backend leaves out reads as 0, so that signal never trips. --worker-overload-token-usage only works for backends that report KV usage.

Configuration¶

smg launch \
  --worker-urls http://w1:8000 http://w2:8000 \
  --worker-overload-protection \
  --worker-overload-waiting-requests 16

Gateway Thresholds¶

Either threshold on its own enables protection; --worker-overload-protection is not required. An explicit --worker-overload-token-usage overrides the 0.9 default. With all three unset (and no per-worker blocks), protection is off and routing is unchanged.

The Python launchers accept the same flags; under smg serve they take the --router- prefix (for example --router-worker-overload-protection). See the Configuration Reference for every flag.

Per-Worker Thresholds¶

Fleets that mix GPU types or model sizes saturate at different points. Give a worker its own thresholds with an overload block in its worker spec when you register it through the Admin API:

curl -X POST http://localhost:30000/workers \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "http://gpu-small:8000",
    "overload": {
      "waiting_requests": 8,
      "token_usage": 0.85
    }
  }'

• Per signal: a field set in the block wins; an omitted field falls back to the gateway value.
• Enables protection on its own: a block turns protection on for its worker even when every gateway flag is unset.
• Resolved at registration: out-of-range values fail that worker's registration. The POST is still answered with 202 Accepted, but the worker is not added.
• Not patchable: PATCH /workers/{worker_id} cannot change the block and keeps it as is. To change it, send the complete spec with PUT /workers/{worker_id}, which re-runs registration (data-parallel workers need DELETE and a new POST instead).
• Visible: GET /workers returns the block as part of each worker's spec.

Workers without a block, including those from --worker-urls and service discovery, use the gateway thresholds. External API workers (OpenAI and other providers) have no load feed, so gateway thresholds are not applied to them.

Behavior¶

Excluding a Worker¶

While a worker is overloaded:

• Every routing policy skips it, on the HTTP and gRPC routers, for regular workers and for each leg of a prefill/decode pair. Hash-based and sticky policies treat it like an unhealthy worker and route the key elsewhere while the veto lasts.
• Requests already running on it are unaffected. The veto only applies to selection.
• It returns to routing on the first poll whose report is under every threshold.

Shedding Requests¶

When every worker the request could have been routed to (the pool selection drew from: same model, worker type, and transport) is overloaded, the gateway rejects the request at once instead of queueing it:

HTTP/1.1 503 Service Unavailable
Content-Type: application/json
Retry-After: 10
X-SMG-Error-Code: worker_overload_protection_shed

{
  "error": {
    "type": "Service Unavailable",
    "code": "worker_overload_protection_shed",
    "message": "All workers for model 'meta-llama/Llama-3.1-8B-Instruct' are overloaded",
    "param": null
  }
}

• Retry-After is the gateway's --load-monitor-interval in whole seconds (at least 1), because a vetoed worker's load is not re-checked before the next poll. Clients should wait that long before retrying.
• Not retried internally: the gateway's retry layer never retries a shed, so the answer comes back immediately rather than after rounds of backoff against the same verdict.
• Trustworthy code: X-SMG-Error-Code is set only by the gateway and is stripped from responses forwarded from workers, so it always marks a decision this gateway made.
• Dispatch-time re-check: if the chosen worker is flagged between selection and dispatch, the request is shed with the message Worker '<url>' for model '<model>' became overloaded before dispatch. The gateway sheds instead of re-selecting; the flag is only set at poll cadence, so this window is rare.
• Mixed causes are not a shed: if some candidates are out for another reason (unhealthy, or circuit breaker open) and the rest are overloaded, the request gets the ordinary 503 no_available_workers.

Load Monitoring¶

The load monitor feeds overload protection, the load-aware routing policies (cache_aware, power_of_two, least_load), GET /loads, the engine_load field of GET /workers, and the smg_engine_* gauges. Since v1.10.0 it polls every worker group from registration onward, whether or not anything consumes the data. That was a behavior change: gateways running a load-blind policy, which never polled before, now poll every worker at --load-monitor-interval.

See Load Monitoring Configuration for the full reference.

PD Decode Admission Window¶

In gRPC prefill/decode disaggregation, SGLang-lineage engines (SGLang, TokenSpeed) start a bootstrap deadline on the prefill leg as soon as a request lands, and it only clears once the decode engine has admitted the request. Decode admission is bounded by the engine's running window (--max-running-requests / --max-num-seqs), so a burst wider than that window lets prefill deadlines expire while requests wait in the decode queue.

The gateway therefore never has more disaggregated requests in flight to a decode worker than that worker's running window holds (counted per gateway instance):

• Room in the window: the request is dispatched immediately.
• Window full: the request waits in the gateway for up to --pd-admission-wait-secs (default 30). If no room frees, it is shed with the same 503 worker_overload_protection_shed response and Retry-After header.
• Wider than the window: a batched request that needs more rooms than the window holds is shed immediately.
• 0 sheds as soon as the window is full, without waiting.
• No window reported: workers whose engine reports no running window are never gated.

This gate is on by default and independent of the overload thresholds. Keep the wait well under the engine's bootstrap deadline (120 s on TokenSpeed) so a request that waits still dispatches with time to spare. smg_pd_admission_waits_total counts dispatches that waited and were admitted; smg_pd_admission_sheds_total counts sheds. See PD Disaggregation for the rest of the PD request path.

Monitoring¶

Metrics¶

The other smg_engine_* gauges (running requests, generation throughput, prefix cache hit rate, and the PD transfer gauges) come from the same poll. When a worker is removed, its smg_engine_* series are set to -1, because metric series cannot be deleted. The full list is in the Metrics Reference.

Useful PromQL Queries¶

# Workers currently excluded, per model
smg_workers_overloaded

# Shed rate by stage
sum by (stage) (
  rate(smg_worker_overload_shed_total[5m])
)

# Mean KV usage per worker (the token usage signal)
avg by (worker, model) (smg_engine_token_usage >= 0)

# Waiting requests per worker (the queue signal)
sum by (worker, model) (smg_engine_waiting_requests >= 0)

Alerting Example¶

groups:
  - name: smg-overload-protection
    rules:
      - alert: WorkersOverloaded
        expr: smg_workers_overloaded > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "{{ $value }} worker(s) for {{ $labels.model }} excluded by overload protection"

      - alert: OverloadShedding
        expr: sum(rate(smg_worker_overload_shed_total[5m])) > 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "SMG is shedding requests with worker_overload_protection_shed"

Production Tuning¶

Recommended Configurations¶

smg launch \
  --worker-overload-protection

smg launch \
  --worker-overload-protection \
  --worker-overload-waiting-requests 16

smg launch --worker-overload-protection
# then register small workers with
# "overload": {"waiting_requests": 8}

smg launch \
  --worker-overload-protection \
  --load-monitor-interval 5

The values above are starting points, not recommendations for your hardware. Size the waiting-requests ceiling from smg_engine_waiting_requests at healthy peak load, and set it above that normal peak.

Tuning Guidelines¶

Overload protection sheds; it does not queue. Keep rate limiting or the priority scheduler in front of it to bound what the gateway accepts, and make sure clients honor Retry-After.

What's Next?¶