Sticky Sessions and Routing Keys¶

A multi-turn conversation runs fastest when each turn lands on the worker that served the turn before it, because that worker still holds the conversation's prefix in its KV cache. Sticky sessions give SMG that property: each request carries a routing key, taken from the request body's rid or from a header, and SMG pins the key to one worker for as long as the key stays in use.

SMG provides stickiness in two forms:

• The manual policy pins every keyed request in its own key-to-worker map.
• The routing-key override (--routing-key-override, alias --sticky-sessions) adds the same pinning on top of any other policy. Keyed requests stick; requests without a key, and by default the first request for each new key, are still placed by the configured policy, such as cache_aware.

Overview¶

Deriving the Routing Key¶

Key Precedence¶

For each request, SMG uses the first of these that yields a key:

1. Body rid (override enabled only): the rid with its lineage suffixes stripped.
2. Routing-key headers: the first header named in --routing-key-headers that carries a valid value.
3. No key: the configured policy places the request, and nothing is pinned.

A body rid wins even when a routing-key header is also present, so a proxy that stamps a unique header value on every request cannot split a conversation. This holds on every policy, including manual and consistent_hashing.

Body rid Lineage¶

rid is an SGLang-style request ID accepted in the body of chat completions, completions, messages, /generate, embeddings, classify, and rerank requests (for rerank, the first ID of a list). The Responses API has no rid, so Responses requests can pin only through a header. SMG reads rid for routing only when --routing-key-override is enabled.

SMG removes one trailing retry suffix, then one trailing turn suffix:

<base>[_t<digits>][_r<digits>]  ->  <base>

• A suffix is an underscore, a lowercase t (turn) or r (retry), and one or more ASCII digits, at the very end of the rid.
• The retry suffix is removed first, so it has to come last: conv_t2_r1 becomes conv, while conv_r1_t2 becomes conv_r1.
• Each suffix is removed at most once.
• If nothing would be left, the whole rid is the key.
• If the resulting key is longer than 128 bytes, SMG ignores the rid and falls back to the headers.

Stripping only computes the key; the request keeps its full rid. On gRPC workers outside PD mode, SMG sends the rid as the engine request ID, so the suffixes let every turn and retry carry a distinct ID while sharing one key.

Routing-Key Headers¶

--routing-key-headers takes an ordered list of header names; the default is x-smg-routing-key. Each name must be a valid HTTP header name, or SMG refuses to start, and names match case-insensitively. For each request, SMG walks the list and takes the first header whose value is:

• non-empty,
• valid UTF-8, and
• at most 128 bytes.

A value that fails these checks is skipped without an error, and SMG tries the next name. With the override enabled, header keys get the same lineage stripping as a rid, so a proxy that forwards conv_t2 in a header pins the same entry as a body rid of conv_t2.

# Read an upstream proxy's header first, and keep the SMG header as a fallback
smg launch --worker-urls http://w1:8000 http://w2:8000 \
  --routing-key-override \
  --routing-key-headers x-routing-key x-smg-routing-key

Configuration¶

Enable the override on top of any policy, here cache_aware:

smg launch --worker-urls http://w1:8000 http://w2:8000 http://w3:8000 \
  --policy cache_aware \
  --routing-key-override

Then give each turn of a conversation a rid that shares a base:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "rid": "chat-7f3a_t1",
    "messages": [{"role": "user", "content": "Plan a three-day trip to Kyoto."}]
  }'

The next turn sends "rid": "chat-7f3a_t2", and a retry of that turn sends "rid": "chat-7f3a_t2_r1". All three pin under the key chat-7f3a. A client that cannot set rid sends -H "x-smg-routing-key: chat-7f3a" instead.

--assignment-mode, --max-idle-secs, and --eviction-interval apply to both the manual policy and the override. See the Configuration Reference for every routing option.

Manual Policy or Override?¶

Choose the override to keep another policy, usually cache_aware, in charge of where conversations start. Choose manual when placement should depend only on the assignment mode.

How the Override Composes with Each Policy¶

Requests without a key go to the configured policy unchanged. Under delegate assignment, the override's default, the first request of a conversation is placed exactly as it would be without the override (by prefix match under cache_aware, for example), and later turns follow it. Pinning keeps the worker's cached prefix reachable on every turn; it does not make the engine keep that cache any longer.

In PD mode, the prefill and decode legs are pinned independently.

Assignment Modes and Eviction¶

Assignment Modes¶

--assignment-mode decides where a key goes the first time SMG sees it, and where it moves when none of its remembered workers is available.

min_load and min_group break ties at random, and each gateway replica counts only the requests it dispatched. The default is random for --policy manual and delegate for the override. One flag sets both, so leave --assignment-mode unset to keep delegation under the override.

Reuse and Failover¶

Each key remembers up to two workers: a primary and one failover candidate. A request goes to the first remembered worker that is available: healthy, with its circuit breaker not open, and not vetoed by overload protection. Adding workers never moves a key.

When no remembered worker is available, SMG places the request again and records the result:

• Under the override in delegate mode, the configured policy picks the worker, which becomes the new primary; the previous primary is kept as the failover candidate.
• In the other modes, and in every mode under --policy manual, the assignment mode picks the worker and adds it behind the existing candidate, dropping the older one when two are already remembered. The earlier worker stays first, so the key returns to it as soon as it is available again.

In-Flight Threshold¶

Under the override, when a key already has 2 requests in flight on its pinned worker, SMG places the next concurrent request for that key again: through the configured policy in delegate mode, otherwise through the assignment mode. The request goes wherever that placement lands. The threshold triggers a new placement; it is not a hard limit. The placement can pick the pinned worker again, for example when cache_aware finds the conversation's prefix there, so a key can have more than 2 requests in flight on one worker. The pin moves only if the placement picks a different worker that has fewer than 2 of the key's requests in flight, so fleet-wide pressure cannot walk a conversation away from the worker that holds its prefix. Other traffic on the worker never triggers a new placement, and each gateway replica counts its own requests. The manual policy has no such threshold.

Idle Eviction¶

Every --eviction-interval seconds (default 120), SMG removes pins that have not been used for --max-idle-secs (default 14400, four hours). Each request that reuses a pin refreshes it. A key whose pin was evicted is placed like a new key on its next request.

The map has no size limit: it holds one entry per key used within the idle window (per model and PD leg under the override), and each entry remembers at most two workers. Setting --max-idle-secs to 0 disables eviction, and the map then grows without bound. --eviction-interval 0 also disables it, but SMG refuses to start with that value when the policy is cache_aware, the default.

Scoping¶

• By model (override): a key used with several models keeps an independent pin for each, so a conversation that switches models cannot evict its own pins. The manual policy's map is not scoped by model.
• By PD leg: prefill and decode pins are independent, under both manual and the override.
• By gateway replica: pins live in each replica's memory. They are not shared through mesh HA, and a restart clears them. With several replicas, send each conversation to the same replica upstream, or use consistent_hashing, which computes the same placement on every replica that sees the same workers as available.
• Self-hosted workers only: requests to external providers and to the Realtime API are placed by SMG's least-load selector and are never pinned.

Request Body Streaming¶

SMG can stream a large request body straight to an HTTP worker instead of buffering it, when nothing in the gateway needs to read the body; see Request Streaming. Sticky routing changes that decision:

• Override enabled: SMG buffers every request body so it can read the rid, even when the request carries routing hints. These requests are counted in smg_router_request_body_path_total{path="buffered", reason="routing_key_override"}.
• Override disabled: a valid x-smg-routing-tokens header lets cache_aware and bucket place the request without its body, so a large body can stream. A routing-key header does not lift that requirement.

The hint header format is in the Request Headers reference.

Observability¶

Routed Worker Header¶

Inference responses from HTTP workers carry x-smg-routed-worker-id: the URL of the worker that served the request, as registered with the gateway, including the @<rank> suffix for data-parallel workers. For a prefill/decode pair it names the decode worker. Compare it across turns to confirm that a conversation sticks. Responses from gRPC workers do not carry it.

curl -s -D - -o /dev/null http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-smg-routing-key: user-1234" \
  -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "Hi"}]}' \
  | grep -i x-smg-routed-worker-id
# x-smg-routed-worker-id: http://w2:8000

Metrics¶

The manual policy and the override both report through the smg_manual_policy_* metrics. The full list is in the Metrics Reference.

Debug Logs¶

At --log-level debug, every decision the override makes logs one Sticky routing decision line with the key source (rid or header), the key (prefixed internally with the model and PD leg), the branch, the chosen worker, and the model ID. An eviction sweep that removes pins logs ManualPolicy TTL eviction at info.

Troubleshooting¶

What's Next?¶