ZMQ Direct Workers¶

When SMG and the inference engine share a host, SMG can connect straight to the engine core over ZMQ. No engine HTTP server or gRPC servicer sits in the path: the engine runs headless (scheduling and model execution only), SMG does everything else — tokenization, chat templates, parsing, stop handling, and routing — and the two exchange token IDs over local ipc:// sockets. Direct ZMQ connections were added in v1.10.0.

When to Use ZMQ¶

Compared with gRPC, the direct path drops the servicer process, a serialization round-trip, and a context switch per message. Use it when the gateway and engine run on one machine. Use gRPC when the engine runs on another host, or when you need disaggregation, KV-event routing, embeddings, or engine admin operations (see Limits).

How It Works¶

For a worker URL ipc://<path>:

1. SMG binds two data-plane sockets, ipc://<path>-in.sock (requests) and ipc://<path>-out.sock (outputs), plus a handshake listener at tcp://127.0.0.1:<port>, where the port is derived from <path> (see Handshake Address).
2. The headless engine dials the handshake address. SMG answers with the data-plane addresses (a HELLO, INIT, READY exchange); the engine connects to them and reports its context length and data-parallel size.
3. SMG marks the worker ready as soon as the handshake completes.
4. Requests go to the engine as token IDs. Outputs come back as token IDs, with the engine's scheduler load piggybacked on the output batches.

The engine never sees text, so SMG also does the work that the engine's API server or gRPC servicer would otherwise do:

Supported Engines¶

Both engines use the same handshake, so SMG can't tell which one dialed in. Declare the runtime with --backend (startup workers) or runtime_type (worker API). A ZMQ worker without a runtime is treated as vLLM, and SMG logs runtime_type unspecified for ZMQ worker; defaulting to vLLM EngineCore.

SGLang, TensorRT-LLM, and MLX can't use ZMQ in v1.11.0. smg serve accepts --connection-mode zmq only with --backend vllm or --backend tokenspeed, and the gateway rejects any other runtime with ZMQ worker ... has unsupported runtime ...: only vllm and tokenspeed are supported over the ZMQ direct backend. Connect those engines over gRPC.

SMG doesn't check engine versions at connect time, and its decoders accept fields that newer engine releases append. SMG's CI runs the ZMQ suites against vLLM 0.27.1 and a pinned TokenSpeed revision.

Quick Start with smg serve¶

smg serve --connection-mode zmq starts headless engines and the gateway together:

smg serve \
  --backend vllm \
  --connection-mode zmq \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --router-model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000

smg serve \
  --backend tokenspeed \
  --connection-mode zmq \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --router-model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000

What smg serve sets up in ZMQ mode:

The launcher builds these engine commands. If you pass a flag the launcher sets, your copy is dropped, except for the two TokenSpeed defaults above, where your value is used:

# vLLM
python -m vllm.entrypoints.cli.main serve <model> --headless \
  --data-parallel-size 1 --data-parallel-size-local 1 \
  --data-parallel-address 127.0.0.1 --data-parallel-rpc-port <handshake port>

# TokenSpeed
python -m tokenspeed.cli serve --headless --model <model> \
  --port <worker port> --dist-init-addr 127.0.0.1:<store port> \
  --data-parallel-address 127.0.0.1 --data-parallel-rpc-port <handshake port> \
  --zmq-engine-index 0 --grammar-backend xgrammar --enable-output-logprobs

To put several data-parallel engines behind one worker, start the engines yourself and use smg launch --zmq-engine-count (see Data-Parallel Engines).

Manual Setup with smg launch¶

Socket Path¶

Pick a path for each worker, for example ipc:///tmp/smg-zmq/engine-0. SMG creates the parent directory with mode 0700 if it doesn't exist. An existing directory must be a real directory (not a symlink) owned by the user SMG runs as, or SMG refuses to bind, so don't place sockets directly in a shared directory such as /tmp. The engine must be able to reach the directory: run it as the same user, or create the directory yourself with the access the engine needs.

Before binding, SMG deletes any socket files already at <path>-in.sock and <path>-out.sock, treating them as leftovers from an earlier gateway run. Give each gateway process its own socket paths: a second gateway configured with the same worker URL would delete the first one's live sockets.

Handshake Address¶

SMG binds the handshake on 127.0.0.1 at a port in 20000–29999, computed from the path after ipc:// as 20000 + FNV-1a-64(path) mod 10000. Compute it before starting the engine:

python3 - <<'EOF'
path = "/tmp/smg-zmq/engine-0"  # the worker URL without "ipc://"
h = 0xCBF29CE484222325
for b in path.encode():
    h = ((h ^ b) * 0x100000001B3) & 0xFFFFFFFFFFFFFFFF
print(20000 + h % 10000)
EOF

For this path it prints 22670. SMG also logs the address when it binds:

Binding ZMQ client for worker ipc:///tmp/smg-zmq/engine-0 (handshake=tcp://127.0.0.1:22670, engines=1)

Two paths can hash to the same port; SMG rejects the second worker at registration. Workers added through the worker API can set a fixed zmq_handshake_address instead.

Start the Engine¶

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --headless \
  --data-parallel-size 1 \
  --data-parallel-size-local 1 \
  --data-parallel-address 127.0.0.1 \
  --data-parallel-rpc-port 22670

python -m tokenspeed.cli serve \
  --headless \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --data-parallel-address 127.0.0.1 \
  --data-parallel-rpc-port 22670 \
  --zmq-engine-index 0 \
  --grammar-backend xgrammar \
  --enable-output-logprobs

Add your usual engine flags, such as --tensor-parallel-size.

Start the Gateway¶

smg launch \
  --worker-urls ipc:///tmp/smg-zmq/engine-0 \
  --backend vllm \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000

Use --backend tokenspeed for a TokenSpeed engine. --model-path is required: it names the model and loads the tokenizer. SMG binds the sockets as soon as it registers the worker, then waits up to 600 seconds for each handshake message, which gives the engine time to load the model and profile its KV cache. If the handshake fails, the next health probe starts a new attempt.

To serve several engines, list one ipc:// URL per engine. They are independent workers balanced by --policy.

Register Workers at Runtime¶

Add a ZMQ worker to a running gateway with POST /workers. The ipc:// scheme selects ZMQ, so connection_mode can be left out:

curl -X POST http://localhost:30000/workers \
  -H "Content-Type: application/json" \
  -d '{
    "url": "ipc:///tmp/smg-zmq/engine-1",
    "runtime_type": "tokenspeed"
  }'

ZMQ-related worker fields:

zmq_handshake_address pairs a worker with an engine that dials a fixed address. A TokenSpeed engine started without --data-parallel-rpc-port dials tcp://127.0.0.1:30500, so this worker pairs with it:

curl -X POST http://localhost:30000/workers \
  -H "Content-Type: application/json" \
  -d '{
    "url": "ipc:///tmp/smg-zmq/tokenspeed-0",
    "runtime_type": "tokenspeed",
    "zmq_handshake_address": "tcp://127.0.0.1:30500"
  }'

Registration runs in the background (the API answers 202 Accepted), so a rejected worker shows up only in the gateway log; see Troubleshooting. For the full request schema, see Worker Management.

Data-Parallel Engines¶

There are two ways to run data parallelism over ZMQ:

• Replicas: several single-engine workers, each with its own socket path. The gateway's routing policy spreads requests across them. smg serve --data-parallel-size N sets this up.
• Grouped worker: one worker URL and one socket set, with N data-parallel engines behind it. The gateway sees one worker and picks the engine (rank) for each request itself.

Configure a Grouped Worker¶

Start the engine group with its data-parallel size, then tell the gateway how many engines to wait for:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --headless \
  --data-parallel-size 2 \
  --data-parallel-size-local 2 \
  --data-parallel-address 127.0.0.1 \
  --data-parallel-rpc-port 22670
smg launch 
  --worker-urls ipc:///tmp/smg-zmq/engine-0 
  --backend vllm 
  --model-path meta-llama/Llama-3.1-8B-Instruct 
  --zmq-engine-count 2

python -m tokenspeed.cli serve 
  --headless 
  --model meta-llama/Llama-3.1-8B-Instruct 
  --data-parallel-size 2 
  --dist-init-addr 127.0.0.1:31233 
  --data-parallel-address 127.0.0.1 
  --data-parallel-rpc-port 22670 
  --zmq-engine-index 0 
  --grammar-backend xgrammar 
  --enable-output-logprobs

smg launch 
  --worker-urls ipc:///tmp/smg-zmq/engine-0 
  --backend tokenspeed 
  --model-path meta-llama/Llama-3.1-8B-Instruct 
  --zmq-engine-count 2

• --zmq-engine-count applies to every ipc:// URL in --worker-urls and must be a positive integer. For a worker added through the API, set "dp_size" on the worker instead.
• The engine process needs N times its tensor-parallel size in GPUs.
• The gateway must wait for exactly as many engines as dial in. An extra engine can fail the handshake with duplicate HELLO ... after INIT phase; a missing one leaves the handshake waiting until it times out.
• Grouped workers can't be combined with --dp-aware, which tries to expand the group into one worker per rank and fails with cannot be dp-aware expanded. Single-engine ZMQ workers register normally under --dp-aware.

Rank Selection¶

Output batches carry the producing rank's scheduler load: running requests, waiting requests, and KV-cache usage (vLLM's scheduler stats, or TokenSpeed's load snapshot). For each request, SMG scores every rank in the group and sends the request to the lowest score:

score = max(requests SMG has in flight on the rank, running + waiting)
        + waiting × 6 × max(0, kv_cache_usage − 0.5)

• The in-flight floor keeps a stale report from making a busy rank look idle, so a burst of requests between reports spreads across ranks.
• Engines report load only while they produce output. When a rank's last in-flight request finishes, SMG resets that rank's running and waiting counts to zero and keeps its last reported KV usage.
• Ties rotate across ranks. A TokenSpeed build that sends no load snapshot is scored on SMG's in-flight counts alone.
• vLLM MoE models run their data-parallel ranks in lockstep and pause the whole group when idle. SMG sends the wake-up that vLLM's data-parallel coordinator process would otherwise send, so no coordinator runs. Ranks of dense models run independently.

Rank selection happens inside the connection. Routing policies see the group as one worker, and requests are never pinned to a rank.

Worker Lifecycle¶

• Health checks stay on for ZMQ workers even with --disable-health-check or a per-worker disable_health_check, because the probe is what reconnects a restarted engine. SMG logs Ignoring disabled health checks for ZMQ worker ....
• With --remove-unhealthy-workers, a ZMQ worker whose engine stays down is removed and nothing re-adds it. The flag is off by default unless service discovery is enabled; leave it off for ZMQ workers so they rejoin when their engine restarts.
• The 300-second silence limit is fixed. A single request whose prefill takes longer than that on an otherwise idle engine is failed as if the engine had died.
• Updating a ZMQ worker's properties, such as labels or priority, keeps its live connection.

Features and Limits¶

Supported over ZMQ¶

Limits¶

Verify¶

# Ready once the handshake has completed and the tokenizer has loaded
curl http://localhost:30000/readiness

# Transport, runtime, and status of each worker
curl -s http://localhost:30000/workers | jq '.workers[] | {url, connection_mode, runtime_type, status}'

Expected worker entry:

{
  "url": "ipc:///tmp/smg-zmq/engine-0",
  "connection_mode": "zmq",
  "runtime_type": "vllm",
  "status": "ready"
}

Send a request:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Gateway metrics label ZMQ workers with connection_mode="zmq":

curl -s http://localhost:29000/metrics | grep 'connection_mode="zmq"'

Troubleshooting¶

Next Steps¶

• gRPC Workers — Engines on other hosts, disaggregation, and KV-event routing
• Multiple Workers — Mix worker types and register workers at runtime
• Load Balancing — Policies for spreading traffic across ZMQ replicas
• Architecture Overview — Where the ZMQ path sits in the gateway
• Configuration Reference — --worker-urls and related worker flags
• Monitoring — Gateway and engine-load metrics