Configuration Reference¶

Every smg launch flag in SMG v1.11.0, with its default, accepted values, and environment variable. Sections mostly follow the headings that smg launch --help prints.

Configuration Methods¶

SMG reads its configuration from:

1. Command-line flags (highest priority)
2. Environment variables, only for the flags that declare one (the Oracle ATP_* variables, JWT_ISSUER, JWT_AUDIENCE, JWT_JWKS_URI, and CONTROL_PLANE_API_KEYS) and for the multimodal flags with SMG_* fallbacks, plus the environment-only settings in the Environment Variable Reference
3. Built-in defaults (lowest priority)

One exception: RUST_LOG, when set to a valid filter, replaces the log filter that --log-level would build. See Logging Configuration.

The smg command that pip install smg provides parses flags with its own Python parser, which differs in a few places. See Python Launcher Differences.

Running the Gateway¶

Commands and Binary Names¶

Flag Syntax¶

• Space-separated lists after one flag: --worker-urls, --selector, --prefill-selector, --decode-selector, --encode-selector, --router-selector, --routing-key-headers, --request-id-headers, --storage-context-headers, --cors-allowed-origins, --prometheus-duration-buckets, --mesh-peer-urls.
• One value per flag (repeat the flag for more): --prefill, --decode, --encode, --model-alias, --tenant-api-key, --jwt-role-mapping, --control-plane-api-keys. For example, --jwt-role-mapping A=admin B=user fails to parse; write --jwt-role-mapping A=admin --jwt-role-mapping B=user.
• Comma-separated: --cache-boundaries 2048,8192.
• Parsed outside clap: --prefill and --encode take an optional trailing bootstrap port (--prefill URL [PORT]), which clap cannot express, so the binary parses them itself before clap runs. They work on every entrypoint but do not appear in smg launch --help, and they need the space form: --prefill=URL is rejected.
• Hidden aliases: --model for --model-path and --runtime for --backend parse but are not listed in --help. Visible aliases are noted next to each flag below.

Worker Configuration¶

Where the gateway listens and which workers it registers at startup.

Examples:

--worker-urls http://worker1:8000 http://worker2:8000
--worker-urls http://[::1]:8000 http://192.168.1.1:8000  # IPv6 and IPv4
--worker-urls grpc://worker1:50051  # gRPC mode
--worker-urls ipc:///tmp/smg-zmq/engine-0  # same-host ZMQ

Routing Policy Configuration¶

Routing flags configure the default policy (--policy). In PD mode, --prefill-policy and --decode-policy build their policies from the same flag values. For how each policy behaves, see Load Balancing.

Load Balancing Policy¶

Recommendation: Use cache_aware for LLM workloads to maximize KV cache hit rates.

Cache-Aware Policy Options¶

The event-driven options apply when workers publish KV cache events; see KV Events Cache-Aware Routing.

Least Load Policy Options¶

These apply wherever least_load is the policy, including as --prefill-policy or --decode-policy. The policy routes to the worker with the lowest expected wait, (queued_tokens + inflight_tokens) / throughput + kv_pressure_weight × k / (1 − k), where k is the worker's KV-cache usage.

Prefix Hash Policy Options¶

Manual Policy and Sticky Sessions¶

--policy manual pins each routing key to a worker. --routing-key-override applies the same sticky map on top of any other policy. manual and consistent_hashing already key on the routing key themselves, so the override adds no second map to them; they read its rid-derived key directly. Both share the options below, and both sweep their map every --eviction-interval seconds. See Sticky Sessions.

Data Parallelism and Multi-Model Options¶

Worker Overload Protection¶

Absolute per-worker ceilings, evaluated once per load report rather than per request. A worker at or above a ceiling leaves routing until its load report drops below it. When every candidate worker is overloaded, the request is shed immediately with 503 and error code worker_overload_protection_shed instead of queueing. The shed carries Retry-After set to --load-monitor-interval (at least 1 second), and the gateway does not retry it. See Overload Protection.

An overload block on a worker spec overrides these values per signal, and turns protection on for that worker even when all three flags are unset. Protection needs load reports, so the load monitor keeps polling workers that have it on (see Load Monitoring Configuration).

How the overload settings differ from the other load thresholds:

RL Control Plane Configuration¶

See RL Control Plane.

PD Disaggregation Configuration¶

Prefill-decode (PD) disaggregation runs prefill and decode on separate workers. Encode-prefill-decode (EPD) mode adds encode workers that run the vision tower. See PD Disaggregation.

Enable PD or EPD Mode¶

Prefill Servers¶

The token after the URL is read as the bootstrap port only if it is a port number or none; leaving it out also means no bootstrap port. Used in PD and EPD modes.

Examples:

--prefill http://prefill1:30001 9001 \
--prefill http://prefill2:30002 9002 \
--prefill http://prefill3:30003 none  # No bootstrap port

Decode Servers¶

Example:

--decode http://decode1:30003 \
--decode http://decode2:30004

Encode Servers¶

EPD mode only.

PD-Specific Policies¶

Worker Startup Configuration¶

These govern every worker registration, not only PD workers, although --help lists them under PD Disaggregation.

Load Monitoring Configuration¶

The load monitor polls every worker group for load reports. The reports feed the load-aware policies (cache_aware, least_load, power_of_two), worker overload protection, and the smg_engine_* gauges. --help also lists the PD admission wait under this heading; that gate uses the running window each decode engine advertises at registration rather than the load reports.

Multimodal Configuration¶

These settings apply where the gateway fetches and preprocesses media itself: the gRPC pipeline that serves gRPC and ZMQ workers. Most of the flags fall back to an SMG_* environment variable when unset, so a value resolves as flag, then environment variable, then built-in default. See Multimodal.

Tensor Transport¶

Controls how the router ships preprocessed multimodal tensors (image/video encoder inputs and model-specific tensors) to gRPC workers. This does not affect accuracy: the inline and shared-memory paths produce byte-identical tensors. The TokenSpeed and vLLM gRPC paths can use shared memory; only TokenSpeed workers pull pixels over RDMA. ZMQ workers always receive tensors inline.

Resolution precedence (highest first): CLI flag, SMG_MM_* environment variable, built-in default.

The worker spec has matching multimodal_tensor_transport and multimodal_shm_min_bytes fields, but v1.11.0 does not copy them onto registered workers, so per-worker values have no effect. The legacy SMG_TOKENSPEED_MM_* names still work but log a migration warning.

On the worker side, the vLLM and TokenSpeed gRPC servicers read TOKENSPEED_UNLINK_MM_SHM_AFTER_READ (default on: unlink each /dev/shm segment after the worker reads it; 0, false, or no keeps the segments), and the TokenSpeed servicer also reads TOKENSPEED_LOG_MM_TIMING (worker-side timing logs).

Media Processing and Limits¶

RDMA Pixel Lane¶

The RDMA lane lets TokenSpeed workers, including EPD encode workers, pull cached pixels from the gateway over NIXL instead of receiving them inline; vLLM workers cannot pull RDMA payloads yet. It needs a gateway built with the mm-rdma Cargo feature and NIXL; in other builds every transfer falls back to inline. The lane is on when --multimodal-tensor-transport rdma (or the legacy --mm-pixel-rdma) is set and a listener IP is configured.

Environment-only RDMA settings, read by the gateway. The TokenSpeed worker reads the two timeouts under the same names with the same defaults, so set them identically on both sides.

Deprecated Environment Fallbacks¶

--mm-processing, --mm-pixel-cache-mb, --mm-pixel-rdma, --rdma-listen-ip, --rdma-slot-ttl-s, and --log-mm-timing replace settings that used to be environment-only. Their SMG_* fallbacks still work, but a value taken from the environment logs a deprecation warning stating that env support ends in the next minor release. Move these settings to the flags.

Environment-Only Media Settings¶

Service Discovery (Kubernetes)¶

Watches Kubernetes pods and registers the matching ones as workers. Enabling service discovery automatically enables IGW mode. Discovery is not available with --backend openai, anthropic, or gemini. See Service Discovery.

Prefill bootstrap ports come from the fixed sglang.ai/bootstrap-port pod annotation.

Example:

--selector app=sglang-worker tier=inference

Model Aliases¶

--model-alias accepts an extra client-facing model name for a served model (format <alias>=<canonical>, repeatable). It applies to every locally registered worker whose model ID equals the canonical side, including workers registered by Kubernetes service discovery. Matching is case-sensitive. Aliased requests are sent to the canonical model; backend responses are passed through unchanged. /v1/models lists canonical IDs only. An alias must differ from its canonical ID, and mapping one alias to two different canonical models fails at startup.

Aliases do not participate in Kubernetes Pod selection or model discovery. A discovered worker's canonical model ID continues to come from the backend served_model_name (or from --model-id-from); aliases only add names that resolve to that canonical ID after registration.

Example:

smg launch \
  --service-discovery ... \
  --model-alias GLM-5.2-Coding=GLM-5.2 \
  --model-alias glm-5.2=GLM-5.2

Logging Configuration¶

RUST_LOG, when set to a valid filter, replaces the filter built from --log-level:

RUST_LOG=smg=debug,hyper=warn smg launch ...

See Configure Logging.

Prometheus Metrics Configuration¶

Default buckets: 0.001 0.005 0.01 0.025 0.05 0.1 0.25 0.5 1 2.5 5 10 15 30 45 60 90 120 180 240 300 480 900 1200 1800 2700 3600 5400 7200.

Example:

--prometheus-duration-buckets 0.001 0.005 0.01 0.025 0.05 0.1 0.25 0.5 1.0 2.5 5.0 10.0

See the Metrics Reference.

Request Handling Configuration¶

--stream-request-bodies-over, which existed briefly in pre-release builds, was removed before v1.10.0; v1.11.0 rejects it as an unknown argument. The per-request decision described under --max-buffered-request-bytes replaces it.

Examples:

--cors-allowed-origins http://localhost:3000 https://example.com
--request-id-headers x-request-id x-trace-id x-correlation-id

Storage Context Headers¶

Example:

--storage-context-headers x-tenant-id=tenant_id x-user-id=user_id

This lets storage hooks read values such as tenant_id and user_id from the request context without hard-coding specific headers in the gateway. Each context key can be mapped from only one header.

Only map headers that are injected or sanitized by a trusted upstream. Client-supplied headers can otherwise spoof storage hook request context values.

Rate Limiting Configuration¶

Gateway-wide admission control for the inference routes. For behavior and sizing guidance, see Rate Limiting.

Priority Scheduler Configuration¶

Priority-aware admission, disabled by default. The YAML format, request header, and response codes are in the Priority Scheduler Reference.

Tenant Rate Limit Configuration¶

Per-tenant LLM token and request rate limits, disabled by default. The YAML format is in the Tenant Rate Limiting Reference; see also Tenant Rate Limiting.

Retry Configuration¶

Gateway-level retries of failed worker requests. See Retries.

Backoff Formula:

delay = min(initial_backoff * multiplier^attempt, max_backoff)
delay = delay * (1 + random(-jitter_factor, +jitter_factor))

attempt is 0 before the first retry.

Circuit Breaker Configuration¶

Per-worker circuit breakers. See Circuit Breakers.

Circuit Breaker States:

• Closed: Normal operation, counting consecutive failures
• Open: All requests to the worker fail fast
• Half-Open: Requests pass through unthrottled; --cb-success-threshold consecutive successes close the circuit, and any failure reopens it

Health Check Configuration¶

Active health probes of each worker. See Health Checks.

Tokenizer Configuration¶

Downloads from the Hugging Face Hub use HF_TOKEN when it is set. See Tokenizer Caching.

Parser Configuration¶

Both flags take a registered parser name; an unknown name fails startup. For each request, a parser named in the model's card wins, then these flags, then auto-detection from the model ID. See Tokenization and Parsing APIs.

Names use underscores: --reasoning-parser deepseek-r1 fails; use deepseek_r1.

MCP Configuration¶

MCP Config Path¶

Only the file's servers list is applied. A server entry's own proxy: block sets that server's outbound proxy; otherwise MCP connections use the standard HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, and NO_PROXY variables (upper- or lowercase). v1.11.0 ignores the top-level proxy: block and never reads MCP_HTTP_PROXY, MCP_HTTPS_PROXY, or MCP_NO_PROXY. See Model Context Protocol (MCP).

Backend Configuration¶

Storage Configuration¶

Settings for the --history-backend databases. See Data Connections.

Oracle Database¶

The connection uses --oracle-dsn when it is set; otherwise --oracle-wallet-path and --oracle-tns-alias are both required.

PostgreSQL Database¶

Redis Database¶

The Rust CLI declares no environment variables for PostgreSQL or Redis. POSTGRES_DB_URL, POSTGRES_POOL_MAX, REDIS_URL, REDIS_POOL_MAX, and REDIS_RETENTION_DAYS are read only by the Python launcher.

Schema Migrations¶

The Oracle and PostgreSQL backends track schema versions and apply pending migrations at startup only when auto_migrate is true. Set it in the --schema-config YAML; when the YAML leaves it out, or there is no YAML, DB_AUTO_MIGRATE=true (or 1) turns it on. With auto_migrate off and migrations pending, startup fails and prints the SQL to apply by hand.

TLS/mTLS Security Configuration¶

Server TLS¶

For HTTPS on the gateway:

Client mTLS¶

The Rust CLI has no flags for a client certificate or CA bundle toward workers. The Python launcher provides --client-cert-path and --client-key-path (set together) and --ca-cert-paths; see Python Launcher Differences and Configure TLS.

OpenTelemetry Configuration¶

Example:

smg launch --enable-trace --otlp-traces-endpoint jaeger:4317

Control Plane Authentication¶

Authentication for the admin routes: worker management, cache and tokenizer operations, WASM modules, and /v1/rl when enabled. When neither API keys nor JWT are configured here, those routes fall back to the shared --api-key (see Data Plane Authentication). See Authentication and Control Plane Auth.

--help also lists --api-key under this heading; it is the shared data plane key described in the next section.

API key example:

--control-plane-api-keys 'key1:Admin:admin:secret123' \
--control-plane-api-keys 'key2:ReadOnly:user:secret456'

JWT role mapping example:

--jwt-role-mapping 'Gateway.Admin=admin' --jwt-role-mapping 'Gateway.User=user'

Data Plane Authentication¶

Example:

smg launch \
  --worker-urls http://worker:8000 \
  --api-key shared-secret \
  --tenant-api-key team-red:red-secret \
  --tenant-api-key team-blue:blue-secret

Mesh Server Configuration¶

High-availability mesh networking for multi-router coordination. See High Availability.

Router pods can also discover each other through --router-selector (see Service Discovery (Kubernetes)).

Example:

smg launch \
  --enable-mesh \
  --mesh-server-name router-1 \
  --mesh-advertise-host 192.168.1.10 \
  --mesh-port 39527 \
  --mesh-peer-urls 192.168.1.11:39527

WebRTC Configuration¶

Used by the WebRTC realtime route (POST /v1/realtime/calls).

Runtime Configuration¶

Controls the tokio async runtime that backs request handling.

By default the runtime is container-aware. Tokio sizes its worker pool to std::thread::available_parallelism(), which on Rust 1.95+ already reads the cgroup CPU quota — so under a Kubernetes limits.cpu the worker count matches the pod's quota, not the host's core count. No extra configuration is needed for the default to be right under a CPU limit.

Do not set an inflated TOKIO_WORKER_THREADS (for example a fixed 32). That overrides the container-aware default and oversubscribes worker threads against the cores the scheduler actually grants, causing scheduler thrash, tail-latency spikes, and /health starvation. Leaving it unset is the correct production configuration.

Worker Threads¶

Explicit async runtime worker-thread count. Leave unset to use tokio's container-aware default above; set it only to pin an explicit count (overriding the cgroup-quota-derived default and any TOKIO_WORKER_THREADS value).

Python Launcher Differences¶

pip install smg installs a Python smg command, not the Rust binary. Its smg launch, and python -m smg.launch_router, parse flags with bindings/python/src/smg/router_args.py and then start the gateway through the Python bindings. The gateway image's entrypoint is python3 -m smg.launch_router and the engine images' entrypoint is this Python smg, so docker run arguments go through this parser too. Most flags match the Rust CLI. The differences in v1.11.0:

• Entry points: the Python smg needs a subcommand (launch or serve); there is no start alias and no subcommand-less form. Under smg serve, most router flags take a --router- prefix (for example --router-policy), and --router-disable-arg-fallback stops them from falling back to same-named backend flags.
• Python-only flags: --client-cert-path, --client-key-path, and --ca-cert-paths (mTLS toward workers); --bucket-adjust-interval-secs (default 5; the Rust CLI fixes 5 seconds); --control-plane-audit-enabled (audit logging is off unless set, while the Rust CLI enables it and offers --disable-audit-logging); and --no-remove-unhealthy-workers / --no-worker-auto-recovery in place of =false.
• Renamed flags: --eviction-interval-secs (default 60) instead of --eviction-interval (default 120); --oracle-connect-descriptor instead of --oracle-dsn; --oracle-username instead of --oracle-user; --postgres-pool-max instead of --postgres-pool-max-size; --redis-pool-max instead of --redis-pool-max-size. When both a TNS alias and a DSN are given, Python uses the alias, while the Rust CLI uses the DSN.
• Different values: --policy and --decode-policy do not accept bucket (--prefill-policy does), and --prefill-policy and --decode-policy accept passthrough. --backend accepts only sglang, vllm, tokenspeed, openai, and anthropic, and defaults to sglang, which behaves like leaving the Rust flag unset. --reasoning-parser and --tool-call-parser are checked against the registered names at parse time.
• Environment variables: Python reads POSTGRES_DB_URL, POSTGRES_POOL_MAX, REDIS_URL, REDIS_POOL_MAX, and REDIS_RETENTION_DAYS as defaults for its storage flags; the Rust CLI does not. Both read the Oracle ATP_* variables. JWT_ISSUER, JWT_AUDIENCE, JWT_JWKS_URI, and CONTROL_PLANE_API_KEYS are Rust-only.
• Multiple values: --control-plane-api-keys and --jwt-role-mapping accept several values after one flag in Python; the Rust CLI takes one value per flag.
• Not available in Python: --disable-audit-logging, --drain-settle-secs, --engine-metrics, --jwt-role-claim, --kv-indexer-ttl-secs, --kv-indexer-max-entries, --pd-pairing-mode, the four --priority-scheduler-* flags, --runtime-worker-threads, the hidden --runtime alias of --backend, --tenant-api-key, --trust-tenant-header, --tenant-header-name, the two --tenant-rate-limit-* flags, --webrtc-bind-addr, and --webrtc-stun-server.

Configuration Examples¶

Minimal Configuration¶

smg launch --worker-urls http://localhost:8000

High-Throughput Configuration¶

smg launch \
  --worker-urls http://w1:8000 http://w2:8000 http://w3:8000 http://w4:8000 \
  --policy cache_aware \
  --max-concurrent-requests 200 \
  --queue-size 400 \
  --queue-timeout-secs 60 \
  --retry-max-retries 3

Low-Latency Configuration¶

smg launch \
  --worker-urls http://w1:8000 http://w2:8000 \
  --policy power_of_two \
  --max-concurrent-requests 50 \
  --queue-size 25 \
  --queue-timeout-secs 5 \
  --health-check-interval-secs 5 \
  --request-timeout-secs 30

Least-Load Routing with Overload Protection¶

smg launch \
  --worker-urls http://w1:8000 http://w2:8000 http://w3:8000 \
  --policy least_load \
  --least-load-max-waiting-requests 32 \
  --worker-overload-protection \
  --worker-overload-waiting-requests 64 \
  --load-monitor-interval 5

least_load skips a worker once its reported queue, plus requests dispatched since its last load report, reaches 32. Overload protection removes a worker from routing at 90% KV-cache usage (the --worker-overload-protection default) or at 64 queued requests. When all three workers are over a ceiling, requests get an immediate 503 with Retry-After: 5.

Sticky Sessions on Cache-Aware Routing¶

smg launch \
  --worker-urls http://w1:8000 http://w2:8000 \
  --policy cache_aware \
  --sticky-sessions \
  --sticky-key-idle-secs 3600

PD Disaggregated Mode¶

smg launch \
  --pd-disaggregation \
  --prefill http://prefill1:30001 9001 \
  --prefill http://prefill2:30002 9002 \
  --decode http://decode1:30003 \
  --decode http://decode2:30004 \
  --prefill-policy cache_aware \
  --decode-policy round_robin

Kubernetes Service Discovery¶

smg launch \
  --service-discovery \
  --selector app=sglang-worker \
  --service-discovery-namespace inference \
  --service-discovery-port 8000 \
  --policy cache_aware

High-Availability Mesh¶

# Router 1
smg launch \
  --enable-mesh \
  --mesh-server-name router-1 \
  --mesh-advertise-host 192.168.1.10 \
  --mesh-port 39527 \
  --mesh-peer-urls 192.168.1.11:39527 \
  --worker-urls http://worker1:8000

# Router 2
smg launch \
  --enable-mesh \
  --mesh-server-name router-2 \
  --mesh-advertise-host 192.168.1.11 \
  --mesh-port 39527 \
  --mesh-peer-urls 192.168.1.10:39527 \
  --worker-urls http://worker2:8000

Secure Production Configuration¶

smg launch \
  --service-discovery \
  --selector app=sglang-worker \
  --service-discovery-namespace inference \
  --policy cache_aware \
  --max-concurrent-requests 100 \
  --tls-cert-path /etc/certs/server.crt \
  --tls-key-path /etc/certs/server.key \
  --jwt-issuer https://login.microsoftonline.com/tenant/v2.0 \
  --jwt-audience api://smg-gateway \
  --jwt-role-mapping 'Gateway.Admin=admin' \
  --jwt-role-mapping 'Gateway.User=user' \
  --enable-trace \
  --otlp-traces-endpoint jaeger:4317 \
  --health-check-port 8081 \
  --host 0.0.0.0 \
  --port 443

With Tokenizer and Parsers¶

smg launch \
  --worker-urls grpc://localhost:50051 \
  --model-path Qwen/Qwen3-8B \
  --tokenizer-cache-enable-l0 \
  --tokenizer-cache-l0-max-entries 50000 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen

With Database Backend¶

# PostgreSQL
smg launch \
  --worker-urls http://localhost:8000 \
  --history-backend postgres \
  --postgres-db-url "postgres://user:pass@localhost:5432/smg" \
  --postgres-pool-max-size 32

# Redis
smg launch \
  --worker-urls http://localhost:8000 \
  --history-backend redis \
  --redis-url "redis://localhost:6379" \
  --redis-pool-max-size 32 \
  --redis-retention-days 7

Environment Variable Reference¶

Flag Variables¶

These variables back a flag; the flag wins when both are set.

Gateway Settings¶

Multimodal Variables¶

Described in Multimodal Configuration.

Python Launcher Variables¶

Read only by the Python launcher (see Python Launcher Differences); the Rust smg binary ignores them.