High Availability¶

Run several SMG routers as one cluster. With --enable-mesh, routers form a peer-to-peer mesh: they track each other's membership with a SWIM-style gossip protocol, replicate the workers each router registers, and share cache-aware routing updates, so every router routes with the cluster's workers and cache affinity.

The mesh does not steer client traffic. Put the routers behind a load balancer or a Kubernetes Service; that is what moves clients to the remaining routers when one fails.

Overview¶

Mesh Architecture¶

                      clients
                         │
         load balancer or Kubernetes Service
        ┌────────────────┼────────────────┐
        ▼                ▼                ▼
  ┌──────────┐     ┌──────────┐     ┌──────────┐
  │ router A │◀───▶│ router B │◀───▶│ router C │   mesh: gRPC on --mesh-port
  └────┬─────┘     └────┬─────┘     └────┬─────┘
       └────────────────┼────────────────┘
                        ▼
                     workers

Every router is equal: there is no leader and no quorum. Each pair of routers (A and C included) keeps one sync stream over the mesh port, and state converges without coordination.

Configuration¶

Quick Start¶

Start the first router, then point the next one at it with --mesh-peer-urls:

# Router 1 (bootstrap)
smg launch --enable-mesh --mesh-advertise-host 10.0.0.1 --mesh-port 39527 \
    --worker-urls http://10.0.1.1:8000 http://10.0.1.2:8000

# Router 2, joining router 1
smg launch --enable-mesh --mesh-advertise-host 10.0.0.2 --mesh-port 39527 \
    --mesh-peer-urls 10.0.0.1:39527

Router 2 has no workers of its own; it imports router 1's workers through the mesh.

Command Line Options¶

Python Entrypoint¶

The Python CLI (smg launch) and the Docker image's entrypoint (python3 -m smg.launch_router) accept the same mesh flags with the same defaults and address checks, so container args can pass them directly:

smg launch --enable-mesh --mesh-host 0.0.0.0 --mesh-advertise-host 10.0.0.11 --mesh-port 39527

smg serve takes router flags with a --router- prefix, for example --router-enable-mesh and --router-mesh-advertise-host.

Basic Configuration¶

smg launch --enable-mesh \
--mesh-server-name node1 \
--mesh-host 0.0.0.0 \
--mesh-advertise-host 10.0.0.11 \
--mesh-port 39527 \
--worker-urls http://10.0.1.1:8000 http://10.0.1.2:8000

smg launch --enable-mesh \
--mesh-server-name node2 \
--mesh-host 0.0.0.0 \
--mesh-advertise-host 10.0.0.12 \
--mesh-port 39527 \
--mesh-peer-urls 10.0.0.11:39527 \
--worker-urls http://10.0.1.1:8000 http://10.0.1.2:8000

smg launch --enable-mesh \
--mesh-server-name node3 \
--mesh-host 0.0.0.0 \
--mesh-advertise-host 10.0.0.13 \
--mesh-port 39527 \
--mesh-peer-urls 10.0.0.11:39527 \
--worker-urls http://10.0.1.1:8000 http://10.0.1.2:8000

Gossip Protocol¶

Membership and Failure Detection¶

1. Probe: Every second, each router pings one random peer that is not DOWN or LEAVING. The ping carries the sender's membership table, and the receiver adopts any entry with a higher version.
2. Indirect probe: If the ping fails, the router asks up to three other ALIVE peers to ping the target (ping-req).
3. Suspect, then down: If no probe reaches the target, it moves from ALIVE to SUSPECTED, or from SUSPECTED to DOWN, and the new status is broadcast to every ALIVE peer.
4. Sync stream: After a successful probe of an ALIVE peer, the router whose name sorts first opens the pair's sync stream if none is open.

Node Status States¶

Failure Detection Timing¶

These values are fixed; v1.11 has no flags to tune them.

State Synchronization¶

Synchronized State Types¶

Because each router enforces rate limits on its own, the cluster-wide limit is roughly the per-router limit times the number of routers. Sticky-session assignments are also per router, so the same routing key can map to different workers on different routers.

Worker Registry Sync¶

Each router publishes the workers it registered itself, whether from --worker-urls, the worker API, or service discovery, whenever one is added, changes status, or is removed. Peers import them into their own registries:

• Imports are health-checked locally. A peer rebuilds the worker from its published spec and probes it itself. The owner's health flag sets the import's initial state and afterwards only nudges it: healthy promotes a pending or not-ready import to ready, and unhealthy demotes a ready one.
• Workers are matched by URL. A router that registered a URL itself keeps its own worker and ignores peers' copies. Give a worker the same URL on every router, and never a loopback address: a peer that imports http://127.0.0.1:8000 routes to its own host.
• Only the owner removes a worker. Removing a worker on the router that registered it removes it everywhere. Deleting an imported worker through another router's API lasts at most until that router's next reconcile pass (every 30 seconds) imports it again.
• A departed router's workers stay. If a router leaves for good, its peers keep its workers registered and keep health-checking them.
• API keys stay local. The published spec leaves out the worker's api_key.

Cache-Aware State Sync¶

The cache_aware policy keeps an approximate prefix tree per model: a string tree for HTTP requests and a token tree for gRPC requests. With the mesh on, every routing decision made through these trees is shared:

1. Publish: After choosing a worker, the router records it in its own tree and queues a delta: a hash of the request's prefix path plus the worker URL. Once per gossip round, each model's queued deltas go to every connected peer as one batch.
2. Apply: A peer that already knows that prefix path adds the worker to it in its own tree.
3. Repair: A peer that does not know the path, such as a router that just started, asks a random ALIVE peer for its whole tree for that model and tree type and replays it. The tree arrives in pages of up to about 2 MiB; a repair that makes no progress for 5 seconds is retried, preferably with another peer, up to 3 times. Unknown prefixes for the same model and tree type that arrive meanwhile are folded into the repair in flight.

The default policy and per-model cache-aware policies take part, with no flag beyond --enable-mesh. In v1.11 the prefill, decode, and encode policies of disaggregated mode do not publish their inserts: the router creates them after the mesh attaches its sync adapter.

What consistency to expect:

• Approximate. Deltas and repairs move the trees toward each other, but they are not guaranteed to match: a dropped delta is not resent, and a repair starts only when a later delta names a path the peer does not know. Each router still evicts its own tree on its own schedule (--eviction-interval, --max-tree-size); evictions are not shared.
• Best effort. A delta batch that a peer's stream cannot accept is dropped, not resent. The peer catches up through repair the next time it sees a prefix it does not know.
• Survives a router failure. Routing decisions that a failed router already shared stay in its peers' trees, so traffic that moves to them keeps that cache affinity.

Not synchronized: event-driven cache-aware routing (KV events), which each router builds from the workers' KV event streams, and the hash placement index (--cache-index hash).

How State Is Replicated¶

Worker-state merges order updates with Lamport clocks rather than wall-clock time, so clock skew between routers does not decide which update wins.

Deployment Patterns¶

Cluster Size¶

Any number of routers works. There is no leader election or quorum, so odd counts are not required, and each router keeps serving with the state it has. Every pair of routers keeps one sync stream, so N routers hold N×(N−1)/2 streams and each router sends its updates to N−1 peers.

Kubernetes Deployment¶

StatefulSet Configuration¶

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: smg
  namespace: inference
spec:
  serviceName: smg-mesh
  replicas: 3
  selector:
    matchLabels:
      app: smg
  template:
    metadata:
      labels:
        app: smg
      annotations:
        sglang.ai/mesh-port: "39527"
    spec:
      serviceAccountName: smg
      containers:
      - name: smg
        image: ghcr.io/smg-project/smg:latest
        args:
        - --service-discovery
        - --service-discovery-namespace=inference
        - --selector=app=sglang-worker
        - --service-discovery-port=8000
        - --router-selector=app=smg
        - --enable-mesh
        - --mesh-server-name=$(POD_NAME)
        - --mesh-host=0.0.0.0
        - --mesh-advertise-host=$(POD_IP)
        - --mesh-port=39527
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        ports:
        - containerPort: 30000
          name: http
        - containerPort: 39527
          name: mesh

Each router discovers the sglang-worker pods itself and finds its peers through the app=smg label. The stable pod names of a StatefulSet double as mesh names, and the smg service account needs the same Pod permissions as worker discovery.

Headless Service¶

apiVersion: v1
kind: Service
metadata:
  name: smg-mesh
  namespace: inference
spec:
  clusterIP: None
  selector:
    app: smg
  ports:
  - port: 39527
    name: mesh

This is the StatefulSet's governing Service. Router discovery dials Pod IPs, so the mesh does not use these DNS names; expose the HTTP port to clients through a regular Service or load balancer.

Kubernetes Router Discovery¶

Router discovery watches Pods that match --router-selector and writes them straight into the mesh membership table, replacing --mesh-peer-urls. It runs as its own task alongside worker discovery.

smg launch --enable-mesh \
    --mesh-server-name "$POD_NAME" \
    --mesh-advertise-host "$POD_IP" \
    --service-discovery \
    --service-discovery-namespace inference \
    --selector app=sglang-worker \
    --router-selector app=smg

How Pods map to node status:

• Running and Ready: ALIVE at the Pod IP and mesh port
• Not Ready (already known and not DOWN): SUSPECTED
• Deleted or terminating: DOWN

Monitoring¶

Mesh Metrics¶

Mesh metrics are served with the other gateway metrics on the Prometheus port (default 29000).

Each sync stream is opened by the router whose name sorts first and counted by the router that accepts it, so a router reports router_mesh_peer_connections only for peers whose names sort before its own, and router_mesh_sync_round_duration_seconds only for peers whose names sort after it. Scrape every router and sum: a fully connected mesh of N routers has N×(N−1)/2 streams. Accepting a stream also sets an empty-peer series to 1 that normally stays at 1, so filter with peer!="". After you remove a router for good, its series stays at 0 on the routers that accepted its streams until they restart.

Alerting Rules¶

groups:
- name: smg-mesh
  rules:
  # A fully connected mesh of N routers has N*(N-1)/2 sync streams.
  # 3 routers -> 3 streams. Scrape every router.
  - alert: SMGMeshStreamsMissing
    expr: (count(router_mesh_peer_connections{peer!=""} == 1) or vector(0)) < 3
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "SMG mesh has fewer sync streams than expected"

  - alert: SMGMeshStreamDown
    expr: router_mesh_peer_connections{peer!=""} == 0
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "Mesh sync stream from router {{ $labels.peer }} is down"

Best Practices¶

Troubleshooting¶

Common Issues¶

Debug Logging¶

RUST_LOG=warn,smg=info,smg_mesh=debug,smg::mesh=debug,smg::mesh_discovery=debug \
    smg launch --enable-mesh ...

Gossip, sync streams, and CRDT merges log under smg_mesh; the worker and cache-tree sync adapters under smg::mesh; router discovery under smg::mesh_discovery. RUST_LOG replaces the filter built from --log-level, so keep a base level in it.

Verify Cluster Health¶

# Membership: every gossip round logs this router's node table at INFO
kubectl -n inference logs smg-0 | grep 'Status:' | tail -n 1

# Worker sync: every router lists the workers its peers registered
curl -s http://<router-ip>:30000/workers | jq -r '.workers[] | "\(.url) healthy=\(.is_healthy)"'

# Sync streams across all routers: N*(N-1)/2 when fully connected
count(router_mesh_peer_connections{peer!=""} == 1)

What's Next?¶