gRPC Workers¶

When workers connect via gRPC instead of HTTP, SMG becomes a full OpenAI-compatible server: it handles tokenization, chat templates, reasoning extraction, and tool calling at the gateway level. Workers run raw inference only.

For vLLM or TokenSpeed engines on the same host as the gateway, the ZMQ direct backend is an alternative: it runs the same gateway pipeline over local IPC sockets.

What gRPC Mode Enables¶

In HTTP mode, SMG is a smart proxy that handles routing and failover only. In gRPC mode, SMG takes over the full request processing pipeline.

Start a gRPC Worker¶

The vLLM, SGLang, TokenSpeed and MLX gRPC servers come from the smg-grpc-servicer Python package. Install the extra for your engine: pip install "smg-grpc-servicer[vllm]", [sglang] or [mlx]. For TokenSpeed, install TokenSpeed first, then smg-grpc-servicer. SMG's engine container images already include the package. TensorRT-LLM serves gRPC from its own serve command.

python -m vllm.entrypoints.grpc_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051 \
  --smg-grpc-mode

# Enable XGrammar guided decoding for forced tool calls and response_format
echo "guided_decoding_backend: xgrammar" > trtllm-config.yaml
python -m tensorrt_llm.commands.serve serve 
  meta-llama/Llama-3.1-8B-Instruct 
  --grpc 
  --host 0.0.0.0 
  --port 50051 
  --backend pytorch 
  --extra_llm_api_options trtllm-config.yaml

python -m smg_grpc_servicer.tokenspeed 
  --model meta-llama/Llama-3.1-8B-Instruct 
  --host 0.0.0.0 
  --port 50051 
  --grammar-backend xgrammar

python -m smg_grpc_servicer.mlx.server \
  --model mlx-community/Qwen3-0.6B-4bit \
  --host 0.0.0.0 \
  --port 50051

Connect SMG¶

Point SMG at the gRPC worker with a grpc:// URL. SMG detects the engine on its own by probing for SGLang, vLLM, TensorRT-LLM, TokenSpeed and MLX in turn:

smg launch \
  --worker-urls grpc://localhost:50051 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000

The API is still OpenAI-compatible, so clients send the same requests as with HTTP workers:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "user", "content": "Hello!"}
    ]
  }'

Multiple gRPC Workers¶

smg launch \
  --worker-urls grpc://worker1:50051 grpc://worker2:50052 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --policy round_robin

Reasoning Extraction¶

For thinking models (DeepSeek-R1, Qwen3, and others), SMG moves chain-of-thought content into a separate field:

smg launch \
  --worker-urls grpc://worker:50051 \
  --model-path deepseek-ai/DeepSeek-R1 \
  --reasoning-parser deepseek_r1

SMG picks the parser from the model name by default; --reasoning-parser overrides that for every model. Separation is on by default, so a plain request is enough:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-R1",
    "messages": [{"role": "user", "content": "What is 25 * 37?"}]
  }'

The response includes both fields:

{
  "choices": [{
    "message": {
      "role": "assistant",
      "content": "925",
      "reasoning_content": "Let me calculate 25 * 37 step by step..."
    }
  }]
}

Send "separate_reasoning": false to keep the reasoning in content.

Supported Reasoning Parsers¶

base and passthrough are also registered, but only for explicit use. The gRPC Pipeline page lists the name patterns each parser is auto-detected from.

Tool Calling¶

In gRPC mode, SMG parses function calls from model output:

smg launch \
  --worker-urls grpc://worker:50051 \
  --model-path meta-llama/Llama-3.2-3B-Instruct \
  --tool-call-parser llama

To have the gateway execute MCP tools on /v1/responses, add --mcp-config-path; see MCP in Responses API.

Supported Tool Call Parsers¶

passthrough (no parsing) is also registered. Formats and name patterns are in the gRPC Pipeline reference.

Per-Model Parsers¶

--tool-call-parser and --reasoning-parser apply to every model. When one gateway serves several model families, set parsers per model through the worker API instead. Labels on the worker registration override the flags for that worker's model:

curl -X POST http://localhost:30000/workers \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -d '{
    "url": "grpc://glm-worker:50051",
    "labels": {"tool_parser": "glm47_moe", "reasoning_parser": "glm45"}
  }'

An unknown parser name fails the registration. See Parser Selection for the full precedence rules.

gpt-oss (Harmony) Vocab¶

Serving gpt-oss models over gRPC uses the Harmony encoding, whose vocab (o200k_base.tiktoken) is fetched when a gpt-oss worker registers — from a local directory if TIKTOKEN_ENCODINGS_BASE is set, otherwise downloaded and cached (TIKTOKEN_RS_CACHE_DIR, default $TMPDIR/tiktoken-rs-cache). If the vocab can't be loaded, that worker's registration fails and retries; other models are unaffected.

For air-gapped deployments, pre-seed the file and point SMG at it:

export TIKTOKEN_ENCODINGS_BASE=/opt/tiktoken   # contains o200k_base.tiktoken

HTTP vs gRPC: When to Use Which¶

Troubleshooting¶

Next Steps¶

• gRPC Pipeline Concepts — Pipeline architecture, parser selection, and all supported parsers
• ZMQ Workers — Same-host vLLM and TokenSpeed engines over the ZMQ direct backend
• Multimodal Pipeline — Image and video processing on the router or the worker
• Tokenizer Caching — Two-level cache for reduced CPU overhead
• MCP in Responses API — Configure Model Context Protocol servers for /v1/responses
• PD Disaggregation — Separate prefill and decode with gRPC workers