Getting Started¶

Shepherd Model Gateway (SMG) routes and manages LLM traffic across workers. This page gives you a fast path to a working gateway, then points you to feature-specific setup guides.

Install¶

pip install smg

cargo install --locked smg

# Debian/Ubuntu
sudo apt-get install -y build-essential pkg-config libssl-dev protobuf-compiler
# macOS (Homebrew)
brew install protobuf openssl@3

docker pull lightseekorg/smg:latest
# Same image on GitHub Container Registry
docker pull ghcr.io/smg-project/smg:latest

docker pull ghcr.io/smg-project/smg:<version>-<engine>-<engine_version>

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"
# Clone and build (needs the protoc and OpenSSL packages from the Cargo tab)
git clone https://github.com/smg-project/smg.git
cd smg
cargo build --release

pip install maturin
make python-install

Step 1: Start SMG¶

Choose one of these startup paths.

Option A: All-in-one with smg serve¶

smg serve launches backend worker process(es) and then starts SMG with generated worker URLs.

smg serve \
  --backend vllm \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --data-parallel-size 2 \
  --host 0.0.0.0 \
  --port 30000

smg serve \
  --backend trtllm \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --data-parallel-size 2 \
  --host 0.0.0.0 \
  --port 30000

smg serve \
  --backend tokenspeed \
  --connection-mode zmq \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --router-model-path meta-llama/Llama-3.1-8B-Instruct \
  --data-parallel-size 2 \
  --host 0.0.0.0 \
  --port 30000

smg serve \
  --backend sglang \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --data-parallel-size 2 \
  --connection-mode grpc \
  --host 0.0.0.0 \
  --port 30000

This starts --data-parallel-size worker replicas, waits for each to become healthy, then starts the gateway. With --connection-mode zmq there is no wait: the gateway starts at once, and /readiness returns 503 until a worker has completed its handshake and the tokenizer has loaded.

To connect vLLM to its engine core directly instead of through gRPC, add --connection-mode zmq and --router-model-path <model>. See ZMQ Direct Workers.

Gateway options take a --router- prefix (for example --router-policy or --router-model-path); other flags are passed to the engine. When the backend's own CLI also defines --host or --port, its definition replaces the one above: with --backend sglang the router defaults to 127.0.0.1:30000, and with vLLM in http mode to 0.0.0.0:8000. Pass both flags to be explicit.

Option B: Launch gateway only with smg launch¶

Use this when workers are already running or managed by another platform.

For gRPC workers:

smg launch \
  --worker-urls grpc://localhost:50051 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --policy round_robin \
  --host 0.0.0.0 \
  --port 30000

For HTTP workers:

smg launch \
  --worker-urls http://localhost:8000 \
  --policy round_robin \
  --host 0.0.0.0 \
  --port 30000

Step 2: Verify Core Endpoints¶

Health:

curl http://localhost:30000/health
curl http://localhost:30000/readiness

OpenAI-compatible chat completions:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}]
  }'

Responses API:

curl http://localhost:30000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "input": "Say hello in one sentence."
  }'

Step 3: Choose Your Setup Track¶

Core Deployment¶

• Multiple Workers
• gRPC Workers
• ZMQ Direct Workers
• PD Disaggregation
• Service Discovery

Operations and Security¶

• Monitoring
• Logging
• TLS
• Control Plane Auth
• Control Plane Operations

Reliability and Data¶

• Reliability Controls
• Data Connections
• Tokenization and Parsing APIs

Advanced Features¶

• Load Balancing
• KV Events Cache-Aware Routing
• Tokenizer Caching
• MCP in Responses API
• External Providers
• RL Control Plane

Worker Startup Recipes (Standalone)¶

Use these when workers are not started via smg serve. Each command starts one worker; register it with smg launch --worker-urls using a grpc:// or http:// URL, as in Option B. The SMG engine images for vLLM, SGLang, TensorRT-LLM, and TokenSpeed already contain the engine and the SMG gRPC servicer. SMG v1.11.0's CI starts workers the same way (plus test-specific flags) on vLLM 0.27.1, SGLang 0.5.20, TensorRT-LLM 1.3.0rc24, and a pinned TokenSpeed commit.

# gRPC worker (grpc://<host>:50051)
python -m vllm.entrypoints.grpc_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051 \
  --tensor-parallel-size 1
# HTTP worker (http://<host>:8000)
python -m vllm.entrypoints.openai.api_server 
  --model meta-llama/Llama-3.1-8B-Instruct 
  --host 0.0.0.0 
  --port 8000

# gRPC worker (grpc://<host>:50051)
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051 \
  --smg-grpc-mode
# HTTP worker (http://<host>:8000)
python -m sglang.launch_server 
  --model-path meta-llama/Llama-3.1-8B-Instruct 
  --host 0.0.0.0 
  --port 8000

# gRPC worker (grpc://<host>:50051)
python -m tensorrt_llm.commands.serve serve \
  meta-llama/Llama-3.1-8B-Instruct \
  --grpc \
  --host 0.0.0.0 \
  --port 50051 \
  --backend pytorch \
  --tp_size 1

# gRPC worker (grpc://<host>:50051)
python -m smg_grpc_servicer.tokenspeed \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051 \
  --grammar-backend xgrammar

# gRPC worker (grpc://<host>:50051)
python -m smg_grpc_servicer.mlx.server \
  --model mlx-community/Qwen3-0.6B-4bit \
  --host 0.0.0.0 \
  --port 50051

For ZMQ direct workers (ipc:// URLs), the engine runs headless and connects to the gateway instead; see ZMQ Workers.

PD Disaggregation Workers¶

For prefill-decode disaggregation, start separate prefill and decode workers, each on its own GPUs (on one host, pin them with CUDA_VISIBLE_DEVICES):

# Prefill worker
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051 \
  --smg-grpc-mode \
  --disaggregation-mode prefill \
  --disaggregation-bootstrap-port 8998
# Decode worker
python -m sglang.launch_server 
  --model-path meta-llama/Llama-3.1-8B-Instruct 
  --host 0.0.0.0 
  --port 50061 
  --smg-grpc-mode 
  --disaggregation-mode decode

smg launch \
  --pd-disaggregation \
  --prefill grpc://localhost:50051 8998 \
  --decode grpc://localhost:50061 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000

# Prefill worker
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --disaggregation-mode prefill \
  --disaggregation-bootstrap-port 8998
# Decode worker
python -m sglang.launch_server 
  --model-path meta-llama/Llama-3.1-8B-Instruct 
  --host 0.0.0.0 
  --port 8001 
  --disaggregation-mode decode

smg launch \
  --pd-disaggregation \
  --prefill http://localhost:8000 8998 \
  --decode http://localhost:8001 \
  --host 0.0.0.0 \
  --port 30000

# Prefill worker
VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
python -m vllm.entrypoints.grpc_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051 \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}'
# Decode worker
VLLM_NIXL_SIDE_CHANNEL_PORT=5601 
python -m vllm.entrypoints.grpc_server 
  --model meta-llama/Llama-3.1-8B-Instruct 
  --host 0.0.0.0 
  --port 50052 
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}'

smg launch \
  --pd-disaggregation \
  --prefill grpc://localhost:50051 \
  --decode grpc://localhost:50052 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000

# Prefill worker
VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}'
# Decode worker
VLLM_NIXL_SIDE_CHANNEL_PORT=5601 
vllm serve meta-llama/Llama-3.1-8B-Instruct 
  --host 0.0.0.0 
  --port 8001 
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}'

smg launch --pd-disaggregation --host 0.0.0.0 --port 30000

curl -X POST http://localhost:30000/workers \
  -H "Content-Type: application/json" \
  -d '{"url": "http://localhost:8000", "worker_type": "prefill", "kv_connector": "NixlConnector"}'
curl -X POST http://localhost:30000/workers 
  -H "Content-Type: application/json" 
  -d '{"url": "http://localhost:8001", "worker_type": "decode", "kv_connector": "NixlConnector"}'

See PD Disaggregation for the Mooncake backend, TokenSpeed and EPD, Kubernetes discovery, and scaling.

Send a Request¶

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "user", "content": "What is the capital of France?"}
    ],
    "max_tokens": 50
  }'

Expected response:

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The capital of France is Paris."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 14,
    "completion_tokens": 8,
    "total_tokens": 22
  }
}

Verify Health¶

# Gateway health
curl http://localhost:30000/health

# Worker status
curl http://localhost:30000/workers

Deploy with Docker¶

For local deployment, run the gateway image and point it at your worker. Container arguments go to the image's entrypoint, python3 -m smg.launch_router, so they are the same flags as smg launch from pip:

docker run -d \
  --name smg \
  -p 30000:30000 \
  -p 29000:29000 \
  --add-host=host.docker.internal:host-gateway \
  lightseekorg/smg:latest \
  --worker-urls http://host.docker.internal:8000 \
  --policy cache_aware

The gateway listens on 0.0.0.0:30000 and serves Prometheus metrics on 0.0.0.0:29000 by default. --add-host makes host.docker.internal resolve to the Docker host on Linux; Docker Desktop provides that name already. The container runs as a non-root user (UID 65532), so files you mount into it must be readable by that user. ghcr.io/smg-project/smg:latest is the same image.

Verify:

docker ps | grep smg
curl http://localhost:30000/health

All-in-one with engine images¶

Engine images use smg as their entrypoint, so pass serve to start the worker and the gateway in one container. Set --host 0.0.0.0: smg serve binds the gateway to 127.0.0.1 by default, which a published port cannot reach. The examples use the v1.11.0 images; the Docker tab under Install lists the engine tags each release builds.

docker run -d --gpus all \
  --name smg \
  -p 30000:30000 \
  -v /path/to/models:/models \
  ghcr.io/smg-project/smg:1.11.0-sglang-v0.5.20 \
  serve \
  --backend sglang \
  --model-path /models/meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000

docker run -d --gpus all \
  --name smg \
  -p 30000:30000 \
  -v /path/to/models:/models \
  ghcr.io/smg-project/smg:1.11.0-vllm-v0.27.1 \
  serve \
  --backend vllm \
  --model /models/meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000

Each engine image sets SMG_DEFAULT_BACKEND to its engine, so --backend is optional. Workers connect over gRPC by default. TensorRT-LLM images take --backend trtllm and support only gRPC. TokenSpeed needs --connection-mode zmq and --router-model-path (see ZMQ Workers). The release TokenSpeed images can't do this: the SMG 1.7.0 they contain has no TokenSpeed backend or ZMQ mode in smg serve.

Verify:

curl http://localhost:30000/health
curl http://localhost:30000/v1/models

Deploy to Kubernetes (Quick Start)¶

Run SMG in the cluster with the Helm chart, or with your own manifests.

Helm¶

The chart is published to GHCR as an OCI artifact, versioned with SMG releases (1.9.0 and later):

helm install smg oci://ghcr.io/smg-project/charts/smg \
  --version 1.11.0 \
  --namespace inference --create-namespace \
  -f values.yaml

A minimal values.yaml either lists workers or discovers them:

router:
  policy: cache_aware
  workerUrls:
    - http://worker-1.inference.svc:8000
    - http://worker-2.inference.svc:8000

router:
  policy: cache_aware
  serviceDiscovery:
    enabled: true
    selector: "app=sglang-worker"
    port: 8000

The chart creates these resources (names assume the release is called smg):

A few chart behaviors to know:

• The router container runs the gateway image, so it accepts the same flags as smg launch from pip. router.extraArgs appends flags the values file does not cover.
• The chart's schema limits router.policy to cache_aware, round_robin, power_of_two, manual, random, and prefix_hash.
• global.image.tag overrides the gateway version, which defaults to the chart's appVersion.
• Engine images for workers[] resolve to ghcr.io/<repository>:<tag>, and the repository defaults to global.image.repository (lightseekorg/smg). GHCR has no engine release images under that name after 1.9.0, so set image.repository: smg-project/smg next to each worker's image.tag. The chart's worker examples still pin 1.3.3 images.

Verify:

helm test smg -n inference
kubectl -n inference port-forward svc/smg-router 30000:80 &
curl http://localhost:30000/workers

The chart README, values.yaml, and examples cover engine workers, multiple models, mesh HA, monitoring, and ingress.

Without Helm¶

Start SMG with service discovery:

smg launch \
  --service-discovery \
  --selector app=sglang-worker \
  --service-discovery-namespace inference \
  --service-discovery-port 8000 \
  --policy cache_aware

In a pod, pass the same flags as container args to the gateway image. SMG needs permission to watch pods in the namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: smg-discovery
  namespace: inference
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]

Bind the Role to the gateway pod's service account with a RoleBinding; Service Discovery has the full manifests. Without --service-discovery-namespace, SMG watches pods in all namespaces and needs a ClusterRole instead.

Verify:

kubectl get pods -n inference -l app=sglang-worker
curl http://localhost:30000/workers

Navigate by Category¶

Core Setup¶

• Multiple Workers — connect local or external worker endpoints
• gRPC Workers — gateway-side tokenization, parsing, and tool handling
• ZMQ Direct Workers — same-host engines with no engine API server in the path
• PD Disaggregation — split prefill and decode paths
• Service Discovery — Kubernetes pod-based worker registration

Operations¶

• Monitoring — Prometheus metrics, tracing, and alerts
• Logging — structured logs and aggregation patterns
• TLS — HTTPS gateway configuration
• Control Plane Auth — secure worker/tokenizer/WASM management endpoints

Reliability and Data¶

• Reliability Controls — concurrency limits, retries, and circuit breakers
• Data Connections — history backend setup for PostgreSQL, Redis, and Oracle
• Tokenization and Parsing APIs — tokenize, detokenize, and parser endpoints

Advanced Features¶

• Load Balancing — policy selection and tuning
• KV Events Cache-Aware Routing — prefix routing driven by engine KV cache events
• Tokenizer Caching — L0/L1 cache setup for gRPC mode
• MCP in Responses API — configure and execute MCP tools through /v1/responses
• External Providers — route to OpenAI, Anthropic, Gemini, and xAI backends
• RL Control Plane — pause, refit, and resume inference engines from an RL training loop

Troubleshooting¶