Load Balancing¶

SMG provides ten load balancing policies to distribute requests across workers. Set the policy with --policy (default cache_aware):

smg --worker-urls http://w1:8000 http://w2:8000 --policy cache_aware

Policy Comparison¶

Cache-Aware (Recommended)¶

The production default. Routes each request to a worker that already holds its prefix; when no worker holds enough of it, or the holder is far busier than the rest, it routes to the worker with the lowest expected wait (the least_load score).

smg \
  --policy cache_aware \
  --worker-urls http://w1:8000 http://w2:8000 \
  --cache-threshold 0.3 \
  --balance-abs-threshold 64 \
  --balance-rel-threshold 1.5

Best for multi-turn conversations, RAG applications, and batch processing with shared templates. See Cache-Aware Routing for KV-event mode, the hash index, and KV-pressure options.

Power of Two Choices¶

Samples two random workers and routes to the one with the lower expected wait, using the same formula as least_load with its default tuning. Good load distribution while reading only two workers' load per request.

smg --policy power_of_two --worker-urls http://w1:8000 http://w2:8000

Best for large or heterogeneous fleets where cache locality doesn't matter.

Least Load¶

Routes to the worker with the lowest expected wait: the token work queued on it divided by its generation throughput, plus a penalty that grows as its KV cache fills. Work sent since the worker's last load report is counted too, so bursts spread out between polls. It is intended for gRPC workers, where each request's token count is known.

smg launch \
  --policy least_load \
  --worker-urls grpc://worker1:50051 grpc://worker2:50052 \
  --model-path meta-llama/Llama-3.1-8B-Instruct

When every worker is at the waiting-queue cap, the request fails with 503 instead of deepening a backlog. See Least Load for the scoring model.

Consistent Hashing¶

Header-based routing with minimal redistribution on scaling. Routes based on X-SMG-Routing-Key header or implicit keys (Authorization, X-Forwarded-For, Cookie).

smg --policy consistent_hashing --worker-urls http://w1:8000 http://w2:8000

Routing Headers¶

Priority: X-SMG-Target-Worker > body rid (with --routing-key-override) > routing-key header > implicit keys > random fallback

Best for session affinity and user-to-worker pinning.

Prefix Hash¶

A lightweight alternative to full cache-aware routing. Hashes the start of each request (its first --prefix-token-count tokens, or four times as many characters when it carries no token IDs) onto a consistent hash ring, and moves a request off its ring worker only when that worker's in-flight load is clearly above average.

smg \
  --policy prefix_hash \
  --worker-urls http://w1:8000 http://w2:8000 \
  --prefix-token-count 256 \
  --prefix-hash-load-factor 1.25

A valid X-SMG-Routing-Key header replaces the prompt as the hash key. Lower memory than cache_aware with predictable O(log n) performance.

Bucket¶

Routes prefill requests by length in PD mode: each prefill worker owns a range of request sizes, and the ranges adapt to recent traffic every 5 seconds. Only the prefill leg gets buckets, so set it with --prefill-policy: --decode-policy bucket is rejected at startup, the Python launcher also rejects --policy bucket, and anywhere else bucket picks a random worker.

smg launch \
  --pd-disaggregation \
  --prefill http://prefill1:8000 9001 \
  --prefill http://prefill2:8000 9002 \
  --decode http://decode1:8000 \
  --decode http://decode2:8000 \
  --prefill-policy bucket \
  --decode-policy power_of_two

Both flags are shared with cache_aware. Best for PD disaggregation where prefill workers handle different request sizes.

Manual¶

Pins each routing key to a worker and keeps it there until that worker becomes unavailable or the key goes unused for --max-idle-secs. Keys come from the X-SMG-Routing-Key header, or from the request body's rid when --routing-key-override is also enabled. Requests without a key are placed by the assignment mode and not pinned.

smg launch \
  --policy manual \
  --worker-urls http://w1:8000 http://w2:8000 \
  --assignment-mode min_load \
  --max-idle-secs 14400 \
  --eviction-interval 120

To pin conversations on top of another policy, such as cache_aware, use --routing-key-override (alias --sticky-sessions) instead. See Sticky Sessions and Routing Keys.

Best for stateful chat where context is stored on workers.

Round Robin¶

Rotates through the available workers in order, keeping a separate rotation for each set of candidate workers (for example, each PD prefill's decode partners), so every worker in a set gets an even share.

smg --policy round_robin --worker-urls http://w1:8000 http://w2:8000

Random¶

Each available worker has equal probability of selection. Zero state overhead.

smg --policy random --worker-urls http://w1:8000 http://w2:8000

Passthrough¶

Sends every request to the first available worker, with no balancing and no KV-event subscription. Use it when the gateway fronts a single worker; with more workers, the others receive traffic only while the first is unavailable. smg serve switches to it automatically when it launches one worker.

smg launch --policy passthrough --worker-urls http://w1:8000

Choosing a Policy¶

Next Steps¶

• Load Balancing Concepts — Detailed policy architecture, advantages/limitations, scenario guides
• Cache-Aware Routing Concepts — Radix tree architecture and routing algorithm deep dive
• PD Disaggregation — Separate prefill and decode policies with --prefill-policy and --decode-policy
• Tokenizer Caching — Reduce tokenization overhead with two-level caching