gRPC Pipeline¶

When workers connect over gRPC, SMG runs the whole OpenAI-compatible serving pipeline itself. It renders the chat template, tokenizes, builds constrained-decoding grammars, detokenizes, extracts reasoning, parses tool calls, and runs MCP tool loops. Workers only run inference on token IDs. Workers on the ZMQ direct backend go through the same pipeline.

Overview¶

Pipeline Architecture¶

Responsibility Comparison¶

The same pipeline serves Chat Completions, the Messages API and the Responses API. gpt-oss models use a separate Harmony pipeline. SMG selects it when a worker's model card lists the GptOssForCausalLM architecture or the gpt_oss model type, or when the model name contains gpt-oss. Harmony has its own encoding and output channels, so the chat templates and parser registries on this page don't apply to it.

Parser Selection¶

Every request resolves one tool-call parser and one reasoning parser for its model. SMG checks these sources in order:

1. Per-model override: the tool_parser / reasoning_parser of the model's card on its workers.
2. Gateway flag: --tool-call-parser / --reasoning-parser, which applies to every model.
3. Automatic detection from the model name.

The gateway checks the flag values when it starts. An unknown --tool-call-parser or --reasoning-parser value stops startup with unknown tool-call parser '<name>' or unknown reasoning parser '<name>'; the pip smg launch rejects it earlier as an invalid choice. Names must match a registered parser exactly (deepseek_r1, not deepseek-r1).

Per-Model Overrides¶

The process-wide flags suit a gateway that serves one model family. To serve several families from one gateway, give each model its own parsers when you register its workers. Either form works:

• Labels: tool_parser and reasoning_parser keys in the worker's labels.
• Model card: tool_parser and reasoning_parser fields on an explicit entry in the worker's models list. An explicit card keeps its own values; labels only fill the fields it leaves empty.

curl -X POST http://localhost:30000/workers \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -d '{
    "url": "grpc://coder-worker:50051",
    "models": [
      {"id": "my-coder", "tool_parser": "qwen_xml", "reasoning_parser": "qwen3"}
    ]
  }'

Registration fails if an override names an unknown parser (worker <url> declares unknown tool_parser '<name>' for model '<id>'). Workers of the same model should agree. If they don't, for example during a rolling upgrade, SMG logs a warning at registration and uses the lexicographically smallest name. Kubernetes service discovery sets no parser labels, so discovered workers rely on the flags. See the Admin API for the worker endpoints.

Automatic Detection¶

Without an override or a flag, SMG matches the request's model field against name patterns. It uses the canonical name, after model aliases resolve. Matching is case-insensitive and finds the pattern anywhere in the name, so Qwen/Qwen3.8-2.4T-A95B matches qwen3.8. The two registries break ties differently:

• Tool-call parsers: the longest matching pattern wins, so deepseek-v4.1 beats deepseek-v4. If nothing matches, SMG doesn't parse tool calls and returns the text as content.
• Reasoning parsers: the first match in the reference table's order wins, which is why the specific DeepSeek, Qwen-thinking, Kimi and MiniMax patterns come before the broad qwen, kimi and minimax ones. If nothing matches, reasoning stays in content.

The reference tables under Reasoning Parsers and Tool Call Parsers list every pattern.

Chat Templates and Native Renderers¶

In gRPC mode the gateway builds the prompt text itself. Most models use a Jinja2 chat template. SMG takes it from a model card's chat_template path or --chat-template, then from a chat_template.json or .jinja file in the model directory, and finally from the chat_template field of tokenizer_config.json. Templates render with minijinja 2.24, so two Python Jinja2 behaviors work: conditional expressions inside keyword arguments (namespace(name=x if x else '')) compile, and booleans and none render as True, False and None (smg-project/smg#2277).

Native Renderers¶

Some checkpoints ship a Python prompt encoder instead of a Jinja template. For these, SMG uses a native Rust port of that encoder, tested against the reference output. The tokenizer picks the renderer from the config.json next to the tokenizer files. The DeepSeek and Kimi-K3 renderers replace the Jinja template entirely, so --chat-template has no effect on those checkpoints.

DeepSeek-V4 checkpoints share one config.json, but the 0731 refresh changed what the effort levels render. SMG tells the revisions apart from the checkpoint's encoding/encoding_dsv4.py. If that file is missing, it looks for a 0731 marker in the model path, and otherwise assumes the original encoding (smg-project/smg#2080). A request goes through SMG's DeepSeek request profile when a /-separated part of its model name is deepseek-v4-flash, deepseek-v4-pro, deepseek-v4.1-flash or deepseek-flash (in any case), as in deepseek-ai/DeepSeek-V4-Flash. The profile turns thinking on unless the request turns it off, which overrides the V4 renderer's default (smg-project/smg#2671). Kimi-K3 appends a response-channel stub after the assistant header. The engine receives the stub, but the prompt_tokens that clients see leave it out, matching Moonshot's billing (smg-project/smg#2564).

Thinking and Reasoning Effort¶

Reasoning controls reach the template as template variables. The request fields are documented in the OpenAI-Compatible API.

• Template kwargs: reasoning_effort (or thinking.effort, which takes precedence), tool_choice and response_format are passed to the template as kwargs of the same name. An entry in chat_template_kwargs overrides any of them.
• Thinking on or off: thinking.type (enabled or disabled) sets the preference. Without it, a reasoning_effort of "none" or "minimal" means off. SMG writes the preference under the key the template actually reads: enable_thinking (Qwen3, GLM, Nemotron), thinking (DeepSeek, Kimi), or thinking_mode set to "enabled" or "disabled" (MiniMax-M3). A value you pass for that key in chat_template_kwargs wins.
• Native effort names: the DeepSeek-V4, V4.1 and Kimi-K3 renderers turn the values in the table above into their own effort prompt text.
• Parser arming: the reasoning parser follows the same decision. When thinking is on, it starts in reasoning mode, so the prompt and the parser agree.

Reasoning Parsers¶

Reasoning parsers separate chain-of-thought from the final answer. Models that emit thinking tokens before their response need one.

Configuration¶

Supported Parsers¶

Complete Parser Reference¶

All 19 registered reasoning parsers, in auto-detection order (the first match wins):

"Starts in reasoning" is the parser's own default. The gateway also starts a parser in reasoning mode when a request has thinking on for a template or renderer with a thinking toggle, such as Qwen3, GLM, Nemotron, DeepSeek-V4 and V4.1, and Kimi-K3. The model's first tokens then count as reasoning.

Output Format¶

Reasoning separation is on by default (separate_reasoning defaults to true). The extracted text is returned in reasoning_content:

{
  "choices": [{
    "message": {
      "role": "assistant",
      "content": "The answer is 42.",
      "reasoning_content": "Let me think step by step..."
    }
  }]
}

Set "separate_reasoning": false in a request to leave the reasoning text in content. When streaming, a parser holds back text that might be the start of a marker, such as a trailing </thi. At the end of the stream it releases that text to the correct field instead of dropping it (smg-project/smg#2523).

Tool Call Parsers¶

Tool call parsers turn each model's native tool-call syntax into OpenAI tool_calls.

Configuration¶

Supported Parsers¶

Complete Parser Reference¶

All 24 registered tool-call parsers. For auto-detection, the longest matching pattern wins. The last column shows the constraint sent for a forced tool_choice when the parser is set by --tool-call-parser or an override (see Constrained Decoding).

Parsing Behavior¶

• Schema-aware arguments: the XML-style parsers (qwen_xml and its aliases, glm45_moe, glm47_moe, minimax_m2, minimax_m3) convert argument values to the types the tool's JSON schema declares. minimax_m3 also resolves properties declared under oneOf, anyOf or allOf. It turns an empty container element into [] or {}, and recovers the missing closing tag of an empty nested container (smg-project/smg#2370, smg-project/smg#2422, smg-project/smg#2567).
• Special tokens: when a request carries tools and tool_choice isn't none, the gateway decodes with special tokens kept, so the parser sees its trigger tokens. Under a JSON-schema constraint the output has no trigger tokens, so the request's own skip_special_tokens applies.
• Streaming: arguments stream as deltas. If a parser buffers text as a possible tool call and it never becomes one, the text is sent as content instead of being dropped. The json, llama, mistral, qwen and cohere parsers do this (smg-project/smg#2271), as do the DeepSeek DSML parsers (smg-project/smg#2525) and minimax_m3, which releases a false tool-call start as soon as it can no longer match (smg-project/smg#2423). qwen_xml keeps each call's arguments separate, even when several calls arrive in one chunk or a value contains } (smg-project/smg#2490).
• Tool call IDs: call_ plus 24 hex characters by default. For model names containing kimi, IDs follow the Kimi reference format instead: functions.NAME:N, or NAME_N when the name also contains k3. N counts tool calls across the whole conversation (smg-project/smg#2104).

Tool Execution Flow¶

1. Parse: the resolved parser extracts calls from the model output. Text outside the calls stays in content.
2. Return: Chat Completions returns the calls in tool_calls with finish_reason: "tool_calls", unless the engine stopped for length or an error. The Messages API returns tool_use blocks with stop_reason: "tool_use".
3. Execute (Responses API): on /v1/responses, the gateway runs calls to MCP tools itself, appends the results to the conversation, and resumes generation. The request's max_tool_calls, capped by the gateway, bounds the loop. See MCP.

Constrained Decoding (XGrammar)¶

When a request forces a tool call or asks for structured output, SMG attaches a constraint to the gRPC request. The engine compiles it with its guided-decoding backend (XGrammar) and enforces it token by token, so the output matches the declared schemas before SMG's parsers ever see it.

Where It Runs in the Pipeline¶

In the regular gRPC pipeline, the tool constraint is generated after the prompt is rendered and tokenized (model_gateway/src/routers/grpc/regular/stages/chat/preparation.rs):

1. Filter tools by tool_choice
2. Render the prompt (chat template or native renderer)
3. Tokenize
4. Process multimodal inputs (see Multimodal Pipeline)
5. Build the tool constraint (generate_tool_constraint)
6. Build the stop decoder, then worker selection and dispatch

The Harmony pipeline (gpt-oss) builds its structural tags before encoding (model_gateway/src/routers/grpc/harmony/stages/preparation.rs). It rejects a request that combines a forced tool call with response_format.

When a Constraint Is Sent¶

Two parsers add rules on top of their structural tag:

• glm47_moe: when thinking is on, GLM-4.7-family templates end the prompt inside <think>. SMG then wraps the GLM tag in a reasoning prefix: free text that must close with </think>, with tool-call tokens excluded, followed by the forced call. The model reasons first, and the call can't end up inside the thinking block where the reasoning parser would absorb it. On SGLang workers the gateway then turns off SGLang's own require_reasoning deferral for that request (smg-project/smg#2550). GLM requests without tools, or with tool_choice auto or none, carry no grammar (smg-project/smg#2549, smg-project/smg#2550).
• kimi_k3: each argument grammar carries the tool schema's $defs / definitions block, so parameters that use $ref compile instead of failing the request (smg-project/smg#2392).

Combining with Structured Output¶

A request carries at most one constraint. response_format becomes a JSON schema (json_object is {"type": "object"}), and text adds nothing. The regex and ebnf request extensions each add a constraint of their own. For SGLang, vLLM and TokenSpeed workers, the gateway rejects a request that sets more than one of these with 400 invalid_request_parameters. If a forced tool call applies as well, the gateway keeps the output-format constraint and drops the tool one for SGLang, TensorRT-LLM and TokenSpeed workers, and keeps only the tool constraint for vLLM workers.

Enforcement¶

The constraint travels in the request's SamplingParams.constraint oneof (crates/grpc_client/proto/). The oneof has json_schema, regex, ebnf_grammar (grammar on vLLM) and structural_tag fields, and vLLM also accepts json_object and choice. TensorRT-LLM receives the same constraint as GuidedDecodingParams. The engine needs a grammar backend to enforce it:

• TokenSpeed: start it with --grammar-backend xgrammar. Its default is none.
• TensorRT-LLM: set guided_decoding_backend: xgrammar in its --extra_llm_api_options file.
• MLX: doesn't support constraints. It rejects a chat request with a forced tool call or a response_format (400) and ignores the chat regex and ebnf extensions.

See gRPC Workers for the launch commands.

Configuration¶

Parser CLI Options¶

MCP Integration¶

When MCP is configured, the gateway executes MCP tool calls on /v1/responses:

smg launch \
  --worker-urls grpc://worker:50051 \
  --model-path meta-llama/Llama-3.2-3B-Instruct \
  --tool-call-parser llama \
  --mcp-config-path /path/to/mcp.json

See the MCP Guide for detailed configuration.

Recommended Configurations¶

smg launch \
  --model-path deepseek-ai/DeepSeek-R1 \
  --reasoning-parser deepseek_r1 \
  --worker-urls grpc://worker1:50051

smg launch \
  --model-path meta-llama/Llama-3.2-3B-Instruct \
  --tool-call-parser llama \
  --mcp-config-path /config/mcp.json \
  --worker-urls grpc://worker:50051

smg launch \
  --model-path Qwen/Qwen3-32B \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen \
  --mcp-config-path /config/mcp.json \
  --tokenizer-cache-enable-l0 \
  --tokenizer-cache-enable-l1 \
  --worker-urls grpc://worker:50051

Monitoring¶

Pipeline Metrics¶

This pipeline records the metrics below. The smg_router_* series carry router_type="grpc"; the MCP metrics have no router_type label:

See the Metrics Reference for every metric and label.

Debug Logging¶

RUST_LOG, when set, replaces the filter built from --log-level. Targets are crate and module paths. Keep a base level such as warn,smg=info so the rest of the gateway still logs:

# gRPC pipeline stages (preparation, worker selection, response processing)
RUST_LOG=warn,smg=info,smg::routers::grpc=debug smg launch ...

# Tool-call and reasoning parsers
RUST_LOG=warn,smg=info,tool_parser=debug,reasoning_parser=debug smg launch ...

# Tokenizers, chat templates and native renderers
RUST_LOG=warn,smg=info,llm_tokenizer=debug smg launch ...

Troubleshooting¶

What's Next?¶