Multiple Workers¶

SMG can route across many workers simultaneously — local inference servers, remote cloud APIs, or a mix of both. This guide covers how to add workers and balance traffic across them.

Supported Worker Types¶

SMG connects to workers over HTTP, gRPC, or ZMQ, and supports both local inference servers and remote API providers:

ZMQ workers (ipc://) run on the same host as SMG, which connects directly to the engine core with no engine API server in between. See ZMQ Direct Workers.

Static Workers via CLI¶

Pass multiple URLs to --worker-urls:

smg \
  --worker-urls http://worker1:8000 http://worker2:8000 http://worker3:8000 \
  --policy round_robin \
  --host 0.0.0.0 \
  --port 30000

For gRPC workers, use the grpc:// scheme and provide --model-path so the gateway can load the tokenizer:

smg \
  --worker-urls grpc://worker1:50051 grpc://worker2:50052 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --policy round_robin

See gRPC Workers for details on what gRPC mode enables.

Cloud API Workers¶

Route to a cloud provider by setting --backend to its router and passing the provider's base URL, without a /v1 suffix. Callers send their own provider key with each request, and SMG forwards it upstream (bring your own key). SMG does not read provider key variables such as OPENAI_API_KEY.

smg \
  --backend openai \
  --worker-urls https://api.openai.com \
  --host 0.0.0.0 \
  --port 30000

smg \
  --backend anthropic \
  --worker-urls https://api.anthropic.com \
  --host 0.0.0.0 \
  --port 30000

smg \
  --backend openai \
  --worker-urls https://api.x.ai \
  --host 0.0.0.0 \
  --port 30000

smg \
  --backend gemini \
  --worker-urls https://generativelanguage.googleapis.com \
  --host 0.0.0.0 \
  --port 30000

The OpenAI-compatible router (--backend openai) also adjusts each request for the provider that serves its model, detected from the model name (for example, xAI handling for grok* models).

A worker can also carry a stored key: pass api_key when you add a worker through the API, or set --api-key, which the --worker-urls workers take as their key. With --api-key set, SMG lists the provider's models with that key when it registers the worker (unless the provider's admin key variable, such as OPENAI_ADMIN_KEY, is set), and callers must authenticate to the gateway with it, which puts it in the Authorization header that SMG forwards upstream. See External Providers for how stored and caller keys combine.

Dynamic Workers with IGW Mode¶

The worker API (POST /workers, GET /workers, DELETE /workers/{worker_id}) is available in every mode. Inference Gateway (IGW) mode (--enable-igw) also runs every router at once (HTTP and gRPC, regular and disaggregated, plus the provider routers) and picks one per request from the workers that serve the requested model, so one gateway can mix worker types. Start it without --worker-urls and add workers at runtime:

smg --enable-igw --host 0.0.0.0 --port 30000

Add a worker¶

curl -X POST http://localhost:30000/workers \
  -H "Content-Type: application/json" \
  -d '{"url": "http://worker1:8000"}'

Response (202 Accepted, with a Location header):

{
  "status": "accepted",
  "worker_id": "0199a3c2-7f1e-7c52-9b1e-5d7c3a2f4e10",
  "url": "http://worker1:8000",
  "location": "/workers/0199a3c2-7f1e-7c52-9b1e-5d7c3a2f4e10",
  "message": "Worker addition queued for background processing"
}

The worker_id is a UUID. SMG registers the worker in the background: GET /workers lists it once registration finishes, and until then GET /workers/{worker_id} (the location) shows the job's status.

List workers¶

curl http://localhost:30000/workers

Remove a worker¶

curl -X DELETE http://localhost:30000/workers/{worker_id}

Worker configuration options¶

POST /workers takes a worker spec. Only url is required; the gateway probes the worker and discovers the rest. A spec with commonly used options:

{
  "url": "http://worker:8000",
  "runtime_type": "vllm",
  "api_key": "optional-key",
  "labels": {"region": "us-east"},
  "health": {"check_interval_secs": 30},
  "overload": {"token_usage": 0.9}
}

The Admin API reference lists every field, the validation rules, and how PATCH and PUT change a registered worker.

Realtime-capable workers¶

To route the Realtime API — the WebSocket /v1/realtime endpoint, WebRTC /v1/realtime/calls, and the realtime REST endpoints — through the HTTP router to a local worker, mark that worker with the well-known realtime label. Only workers labeled realtime: "true" receive realtime traffic, so SMG never proxies a realtime connection to a worker that can't serve it.

The worker must itself expose an OpenAI-compatible realtime endpoint. For example, vLLM serving a speech model with the realtime task (such as Qwen/Qwen3-ASR-1.7B) exposes ws://<worker>/v1/realtime for streaming transcription.

curl -X POST http://localhost:30000/workers \
  -H "Content-Type: application/json" \
  -d '{
    "url": "http://asr-worker:8000",
    "runtime": "vllm",
    "labels": {"realtime": "true"}
  }'

The same worker also serves batch transcription via POST /v1/audio/transcriptions, which the HTTP router forwards without requiring the realtime label.

Verify¶

# List connected workers
curl http://localhost:30000/workers

# Check health
curl http://localhost:30000/health

# Send a request
curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Next Steps¶

• Monitoring — Track request rates, latency, and worker health
• gRPC Workers — Enable tokenization, chat templates, and tool parsing at the gateway
• PD Disaggregation — Separate prefill and decode onto specialized workers