Multimodal Pipeline¶

With gRPC workers, SMG runs the multimodal front end for the engine. It fetches each image or video, preprocesses it the way the model's reference processor does, expands the prompt's placeholder tokens to the right length, and sends the encoder tensors to the worker. vLLM gRPC workers can instead receive the media URLs and process them on the engine.

Overview¶

Request Flow¶

SMG accepts media on these APIs:

Media URLs can be http://, https://, or base64 data: URLs.

On the gRPC path, a media request goes through these steps:

1. Plan and validate. SMG collects the media parts in prompt order and picks the model's processor from the model name and its config.json model_type. A model with no processor, a modality the model does not take, or more items than the per-request limit is rejected before anything is fetched.
2. Render and check anchors. The chat template renders one placeholder anchor per item (for example <|image_pad|> for Qwen3-VL). After tokenizing, SMG checks that the prompt holds exactly one anchor per item.
3. Choose where to process. --mm-processing decides between the router and a vLLM worker (see Where Media Is Processed).
4. Fetch and decode. Items are fetched concurrently. Images are decoded and videos are sampled into frames.
5. Preprocess and expand. The model's processor turns each item into encoder tensors and a feature-token count. SMG replaces each anchor with the model's full placeholder run, including any per-frame markers and timestamps.
6. Select a worker. The request is refused if the selected engine does not accept one of its modalities.
7. Assemble and send. The tensors are serialized for that engine and sent inline, through shared memory, or over RDMA. With --multimodal-max-inflight-bytes set, the request first reserves room for its media.

The processor configuration (config.json, preprocessor_config.json, and, when present, processor_config.json and video_preprocessor_config.json) comes from the tokenizer bundle the gRPC worker serves, or from the tokenizer source (a local directory or a Hugging Face download).

Where Media Is Processed¶

--mm-processing decides, per request, whether SMG preprocesses the media or forwards media references to a vLLM gRPC worker that processes them itself (smg-project/smg#2399, #2400). Only vLLM gRPC workers process references; every other engine receives router-preprocessed media.

A request is forwardable when every media part is an image_url or video_url without per-item hints (fps, max_long_side_pixel). Audio parts are never forwarded. The flag falls back to SMG_MM_PROCESSING (deprecated; env support ends in the next minor release), then auto. An unreadable SMG_MM_PROCESSING stops the gateway at startup.

In auto, SMG consults every registered worker of the model, healthy or not, so a health flap does not move a model between modes. Each decision is counted in smg_mm_processing_total{model,mode,reason}, with reason config, plan_not_forwardable, model_not_opted_in, auto_none (no registered workers), auto_uniform, or auto_mixed.

On the worker path:

• The request carries the unexpanded prompt (one anchor per item) and the media URLs in prompt order.
• An inline data: URL larger than SMG_IMAGE_MAX_INPUT_BYTES or SMG_VIDEO_MAX_INPUT_BYTES is refused with media_ref_too_large.
• Every URL scheme must be one the selected worker advertises in its mm_media_ref_schemes label: http, https, and data, plus file when vLLM runs with --allowed-local-media-path.
• Direct ZMQ workers and EPD encode workers never take references.

Worker-Side Processing on vLLM¶

A vLLM gRPC worker processes references when its servicer runs a media processor:

SMG_VLLM_MM_PROCESSOR=inprocess \
  vllm serve Qwen/Qwen3-VL-8B-Instruct --grpc --allowed-media-domains example.com

The servicer reads each setting below from its --mm-* flag when the launcher passes the flags through (smg-project/smg#2626), and otherwise from the matching SMG_VLLM_MM_* variable. vLLM's gRPC launcher (vllm serve --grpc) does not pass them, so set the variables; a value read from a variable logs a deprecation warning, and env support is due to end in the next minor release.

SMG_VLLM_MM_MAX_VIDEO_FRAMES (env only; default 0, which leaves it to vLLM's --media-io-kwargs) caps the frames a video is sampled to.

The worker advertises its processor in the mm_processor, mm_processor_source, and mm_media_ref_schemes labels, which appear in GET /workers. An engine started with --language-model-only never advertises one. vLLM's own --allowed-media-domains, --allowed-local-media-path, --media-io-kwargs, --limit-mm-per-prompt, and VLLM_*_FETCH_TIMEOUT govern fetching on the worker; without --allowed-media-domains the worker fetches from any host.

Failures keep their cause (smg-project/smg#2596). The caller's own mistakes (a bad URL, a disallowed host, an oversized payload) come back as 400. Transient failures (a fetch timeout, a refused connection, an origin 5xx, a saturated worker, a sidecar that is down or overloaded) come back as retryable 503s. A sidecar timeout is not retried, because the worker already spent the whole budget on that input (smg-project/smg#2624).

Redis Media-Processing Sidecar¶

The sidecar moves fetching and processing out of the vLLM process (smg-project/smg#2401). It is a GPU-free process that runs next to the worker with a private Redis: it pops jobs, fetches with vLLM's MediaConnector, runs vLLM's multimodal processor over the unexpanded prompt, and pushes full tensors back.

pip install "smg-grpc-servicer[vllm,vllm-redis]"

python -m smg_grpc_servicer.vllm.mm_sidecar --model Qwen/Qwen3-VL-8B-Instruct \
  --redis-url redis://127.0.0.1:6379/0 --allowed-media-domains example.com

SMG_VLLM_MM_PROCESSOR=redis SMG_VLLM_MM_REDIS_URL=redis://127.0.0.1:6379/0 \
  vllm serve Qwen/Qwen3-VL-8B-Instruct --grpc

• The sidecar takes --redis-url (falls back to SMG_VLLM_MM_REDIS_URL, then localhost), --namespace (falls back to SMG_VLLM_MM_SIDECAR_NAMESPACE), --concurrency (default 2), and vLLM's engine flags. It has no timeout flag: the worker's --mm-sidecar-timeout-ms travels with each job as its deadline (smg-project/smg#2651).
• The worker and the sidecar must agree on the model, vLLM version, dtype, video backend, media and processor kwargs, and --limit-mm-per-prompt (pass it to both; the sidecar's limit is the one that applies). The worker advertises mm_processor=redis only while a sidecar with a matching fingerprint keeps its hello key alive (refreshed every 5 seconds, 15-second TTL). SMG reads the label when it registers the worker, so start the sidecar first: a worker registered while no sidecar was up keeps the model on the router path in auto until that worker is registered again.
• Jobs and results travel over Redis lists under smg:mm:v1:{namespace}, and results expire after 120 seconds. An encoded result at or above SMG_VLLM_MM_MAX_RESULT_BYTES (default 512 MiB, lowered to Redis's proto-max-bulk-len when that is smaller) is not pushed; the request fails with a 400 whose message includes media_too_large.

Supported Models¶

SMG picks a model's processor from the model name and its config.json model_type. These families have vision support in v1.11.0, with their built-in per-request limits:

¹ When the checkpoint declares the tokens its video layout needs.

² vLLM currently sizes some MiniMax-M3 images differently from SMG, so answers on the two paths can differ (smg-project/smg#2609).

³ Kimi-K3 media stay on the router path: vLLM's Kimi-K3 processor expands a different anchor than the one SMG renders (smg-project/smg#2644).

Recent additions include Qwen4-Exp (smg-project/smg#2327), MiniMax-M3 images and video (#2371, #2581), GLM-5.3-Flash (#2349), and DeepSeek-V4.1 (#2526). Audio (audio_url, input_audio) is accepted for Qwen3-Omni, Qwen3-ASR, and Inkling, on TokenSpeed workers only.

Engine Support¶

vLLM and TokenSpeed take several modalities in one request, such as images and video together. A modality the selected engine does not take is refused with 400 multimodal_not_supported.

Video¶

Decoding¶

SMG decodes video on the gateway. The default build runs ffprobe and ffmpeg, which must be on the gateway's PATH; without them a video request fails with ffmpeg executable not found. Builds with the opencv-video feature decode with OpenCV first and fall back to ffmpeg. The SMG container image is built with opencv-video and ships ffmpeg; the PyPI wheels are built without opencv-video.

Concurrent decodes share one CPU budget: each ffmpeg or OpenCV decode gets fewer threads as more run at once (smg-project/smg#2583).

Frame Sampling¶

fps and max_long_side_pixel are MiniMax-M3 extensions to the OpenAI content part, and SMG applies them for any model. image_url.max_long_side_pixel (a positive multiple of 28) caps an image's long side the same way. A part with any of these hints is never forwarded to a worker.

{"type": "video_url", "video_url": {"url": "https://example.com/clip.mp4", "fps": 1.0}}

Models whose reference processor lays video out frame by frame get the same layout from SMG: Qwen3-VL, GLM-5.3-Flash, and MiniMax-M3 prompts carry one block per temporal frame, with a timestamp. For MiniMax-M3, SMG also applies the pixel budget per frame and stamps each frame with the reference timestamp (smg-project/smg#2574, #2575).

Several clips in one request are preprocessed in parallel (smg-project/smg#2595), and on vLLM a request can carry images and video together (smg-project/smg#2581).

Limits and Safety¶

Media Counts¶

Each model has built-in per-request limits (see Supported Models). To change them:

Neither setting enables a modality the model does not support. When you raise a limit, raise the engine's own limit to match. On the worker path, the vLLM worker also enforces its own item limits.

Sizes and Timeouts¶

In-Flight Media Budget¶

--multimodal-max-inflight-bytes caps the bytes of preprocessed media SMG holds for engines at once (smg-project/smg#2585). A request that fits waits up to 2 seconds for room and then gets 429 multimodal_inflight_budget. A request larger than the whole budget gets 413 multimodal_payload_too_large right away. A waiting request still holds its media, and the waiting queue is capped at one budget (a request that would overfill it gets 429 right away), so size memory for about twice the value. Unset leaves the budget unbounded; 0 is refused.

Errors¶

Multimodal errors use the standard error body ({"error": {"type": ..., "code": ..., "message": ...}}), and the code is also sent in the X-SMG-Error-Code header.

Prefill-decode deployments add three codes, listed under Prefill-Decode Disaggregation.

Performance¶

• Fetch once. Parts that name the same media with the same settings share one fetch and decode (smg-project/smg#2586).
• Data URL fast path. data:image/... URLs go straight to the base64 decoder without a full URL parse, which matters for large inline images (smg-project/smg#2648).
• Parallel preprocessing. Modality batches are preprocessed concurrently. Within a batch, processors built on the Qwen-VL pipeline (Qwen2-VL through Qwen3-VL, Qwen3-Omni, MiniMax-M3) split the images across cores (smg-project/smg#2582), and video clips run in parallel (smg-project/smg#2595).
• Pixel cache. --mm-pixel-cache-mb gives SMG a host-memory LRU cache of preprocessed images, keyed by the image's content hash and a fingerprint of the model and its preprocessing config. It serves single-image requests and, for the Qwen2-VL, Qwen2.5-VL, and Qwen3-VL family processors, requests with fewer than 32 images, where each image is cached on its own and a repeated image is preprocessed once (smg-project/smg#2602). The cache is off by default.
• JPEG decoding. When libturbojpeg is installed on the gateway host, JPEGs are decoded with libjpeg-turbo using Pillow's defaults, so the pixels match what vLLM computes. Without it, SMG uses a pure-Rust decoder that can differ by a few levels per pixel.

Tensor Transport¶

--multimodal-tensor-transport chooses how preprocessed tensors reach the worker:

A failed shared-memory write falls back to inline and is counted in smg_mm_shm_write_failures_total. The two flags fall back to SMG_MM_TENSOR_TRANSPORT and SMG_MM_SHM_MIN_BYTES (legacy names SMG_TOKENSPEED_MM_TENSOR_TRANSPORT and SMG_TOKENSPEED_MM_SHM_MIN_BYTES). The transport is chosen gateway-wide. The worker spec has multimodal_tensor_transport and multimodal_shm_min_bytes fields, and SMG consults them for the worker that runs the vision encoder (the encode worker in EPD), but in v1.11.0 worker registration does not copy them onto the worker, so they have no effect (see the Admin API). The RDMA lane is also on or off for the whole gateway.

RDMA Pixel Lane¶

The RDMA lane stages each pixel tensor in a pre-registered host-memory arena and hands the TokenSpeed worker a small descriptor; the worker pulls the pixels with a one-sided RDMA read.

• Build the smg binary with the mm-rdma Cargo feature (NIXL). A default build compiles an inert exporter, so every tensor stays inline.
• Turn the lane on with --multimodal-tensor-transport rdma (or the legacy --mm-pixel-rdma), and set --rdma-listen-ip to the gateway's RDMA address. Without a listen IP, the lane stays off.
• The arena defaults to 64 slots of 32 MiB (2 GiB). SMG_RDMA_POOL_SLOTS and SMG_RDMA_SLOT_BYTES resize it (a slot must hold one image's pixels), and SMG_RDMA_LISTEN_PORT (default 18515) sets the listener port.
• A slot the worker never releases is reclaimed after a TTL derived from the worker's longest hold: SMG_RDMA_LANDING_WAIT_S (default 120) plus SMG_RDMA_READ_TIMEOUT_S (default 60) plus 30 seconds. --rdma-slot-ttl-s overrides it only with a value longer than that hold.
• A tensor that cannot be staged falls back to shared memory or inline.

Encoder Input Width¶

vLLM workers receive the pixel tensor as float32 unless SMG_VLLM_ENCODER_INPUT_DTYPE asks for bfloat16 or float16, or the worker carries a multimodal_encoder_dtype label (smg-project/smg#2591). The vLLM servicer accepts half-precision media since smg-project/smg#2590. TokenSpeed workers receive the width they advertise, or bfloat16 when they advertise none; SMG_TOKENSPEED_ENCODER_INPUT_DTYPE and the per-modality SMG_TOKENSPEED_IMAGE_ENCODER_INPUT_DTYPE, SMG_TOKENSPEED_VIDEO_ENCODER_INPUT_DTYPE, and SMG_TOKENSPEED_AUDIO_ENCODER_INPUT_DTYPE override it. SGLang always receives float32.

Other Paths¶

Direct ZMQ Backends¶

vLLM and TokenSpeed workers reached over the direct ZMQ backend take router-preprocessed media, sent inline (smg-project/smg#2056, #2539). Media references, shared memory, and RDMA are not available on this wire. TokenSpeed over ZMQ refuses media whose tensors include image_grid_thw or video_grid_thw (the M-RoPE families, such as Qwen-VL); serve those models over gRPC. See ZMQ Workers.

Prefill-Decode Disaggregation¶

With prefill-decode disaggregation, media are normally processed once, for the prefill leg:

• Router path. The prefill worker gets the pixels. On vLLM and TokenSpeed, the decode worker gets each item's identity (content hashes, placeholder ranges, and M-RoPE grid tensors) but not the pixels; on SGLang it gets no media data (smg-project/smg#2243, #2365, #2366). The exception is vLLM with n > 1: there is no KV handoff, so the decode leg keeps the full payload and encodes the media itself. On vLLM, the decode worker keys its cached KV blocks by the content hashes (through the cache salt when no grid tensors travel), so two different images behind the same text cannot share cached KV blocks.
• Worker path (vLLM). The prefill worker processes the references and returns the processed identity with its result. SMG builds the decode leg from it, so the decode worker does not fetch or process the media again (smg-project/smg#2627). With n > 1 there is no KV handoff, so the decode leg keeps the references and processes them itself.
• Language-model-only decode workers. vLLM decode workers started with --language-model-only report supports_vision=false and get the prefill-expanded prompt plus the content hashes, with no media payload (smg-project/smg#2640). Three combinations cannot be served and are refused with a non-retryable 400: models that need M-RoPE grids (pd_decode_language_model_only_mrope), media references (pd_decode_language_model_only_media_refs), and n > 1 (pd_decode_language_model_only_n_samples).

See PD Disaggregation for how the legs are paired and dispatched.

EPD (Encode-Prefill-Decode)¶

--epd-disaggregation (gRPC TokenSpeed workers only) adds encode workers, listed with --encode <url> [bootstrap_port]. SMG still fetches and preprocesses the media, then sends each item's pixels to an encode worker picked by --encode-policy (default consistent_hashing). The encode worker runs the vision tower and ships the embeddings to the prefill worker over Mooncake, so the prefill request carries no pixels. Pixels reach encode workers inline, through shared memory, or over the RDMA lane.

Observability¶

--log-mm-timing (env fallback SMG_LOG_MM_TIMING, deprecated) logs smg_mm_timing lines at INFO: a per-request breakdown (media_fetch_decode_ms, preprocess_ms, token_expand_ms, total_ms) plus video decode and tensor transport events.

GET /workers shows the labels behind these decisions: mm_processor, mm_processor_source, mm_media_ref_schemes, supports_vision, and shm_namespace_id. See the Metrics Reference for every metric.

Configuration Summary¶

--mm-processing, --mm-pixel-cache-mb, --mm-pixel-rdma, --rdma-listen-ip, --rdma-slot-ttl-s, and --log-mm-timing replace SMG_* variables that still work as deprecated fallbacks until the next minor release (smg-project/smg#2625). With smg serve, prefix router flags with --router-, for example --router-mm-processing router. See Multimodal Configuration for the full list.

Example¶

Start a vLLM gRPC worker with a vision model and point SMG at it:

python -m vllm.entrypoints.grpc_server \
  --model Qwen/Qwen3-VL-8B-Instruct \
  --host 0.0.0.0 \
  --port 50051

smg launch \
  --worker-urls grpc://localhost:50051 \
  --model-path Qwen/Qwen3-VL-8B-Instruct \
  --port 30000

Send an image with a chat completion:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-VL-8B-Instruct",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "image_url", "image_url": {"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"}},
        {"type": "text", "text": "Describe this image in one sentence."}
      ]
    }],
    "max_tokens": 64
  }'

usage.prompt_tokens includes the expanded image placeholder run. To send a local file, pass it as a base64 data URL such as data:image/jpeg;base64,....

What's Next?¶