OpenAI-Compatible API Reference¶

SMG provides a fully OpenAI-compatible API, allowing you to use existing OpenAI client libraries with your self-hosted inference workers.

Base URL¶

http://localhost:30000/v1

Authentication¶

SMG supports optional API key authentication:

curl http://localhost:30000/v1/chat/completions \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '...'

Enable authentication with --api-key:

smg launch --worker-urls http://worker:8000 --api-key "your-api-key"

For real multi-tenant separation (e.g. per-tenant rate limiting), configure one key per tenant instead with --tenant-api-key tenant_id:key (repeatable). Each key resolves to its own tenant identity; --api-key remains available as a single shared fallback key whose callers all share one identity.

smg launch --worker-urls http://worker:8000 \
  --tenant-api-key team-red:red-secret \
  --tenant-api-key team-blue:blue-secret

Callers authenticate the same way as with --api-key — a Bearer token in the Authorization header — just using their own tenant's key instead of the shared one:

curl http://localhost:30000/v1/chat/completions \
  -H "Authorization: Bearer red-secret" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

This request is attributed to team-red, distinct from a request authenticated with blue-secret. A missing key, or one that doesn't match any configured --api-key or --tenant-api-key value, is rejected with 401 Unauthorized and an empty body. Tenant keys authenticate serving endpoints only — /v1/chat/completions, /v1/completions, /v1/embeddings, etc. They do not grant access to admin/management routes (/workers, /flush_cache, and similar); that surface requires either the shared --api-key or control-plane authentication.

GET /v1/models is a public route and answers without a key (see List Models).

Endpoints¶

Chat Completions¶

Create a chat completion.

POST /v1/chat/completions

Request Body¶

Extension Fields¶

SMG also accepts these SGLang-style extensions on Chat Completions:

Fields SMG does not model are kept and forwarded unchanged to HTTP workers, and so are unknown stream_options keys such as SGLang's step_usage_chunks. The SGLang-native /generate endpoint likewise keeps its nested sampling_params.custom_params object. Typed sampling numbers are forwarded without float widening: "top_p": 0.95 reaches HTTP workers, PD legs, and external providers as 0.95, not 0.949999988079071.

Message Object¶

Content Parts¶

A content part whose type SMG does not model, or a known type with a malformed payload, is forwarded unchanged to HTTP workers. gRPC workers reject it with 400 unsupported_content_part, and the gpt-oss (Harmony) path with 400 harmony_build_failed. Which workers accept which media is covered in Multimodal.

Thinking and Reasoning Effort¶

Two fields control reasoning. reasoning_effort is OpenAI's effort level. thinking is a typed object shared by the Kimi, MiniMax, z.ai, and DeepSeek dialects:

The effective effort is thinking.effort when present, otherwise reasoning_effort. Leaving thinking out changes nothing: the chat template's own default applies (Kimi K3, for example, thinks by default), subject to reasoning_effort. The exception is the DeepSeek V4 profile, which turns thinking on unless the request turns it off (see Defaults and Rewrites). A thinking object without type expresses no toggle, but its effort still counts.

On gRPC workers SMG renders the chat template itself. The template receives the effective effort as its reasoning_effort variable, and SMG decides whether the model starts in reasoning mode from the first of these that applies:

1. an explicit thinking toggle in chat_template_kwargs;
2. for templates with their own effort levels, the effort (chat_template_kwargs.reasoning_effort, else the effective effort): none and minimal turn thinking off, a native level turns it on;
3. thinking.type (enabled on, disabled off);
4. the effective effort, where none and minimal turn thinking off.

If none applies, the template default stands. The template and the reasoning parser use the same decision. On gpt-oss (Harmony) models SMG reads reasoning_effort and clamps it to the three levels Harmony knows: none and minimal become low, xhigh and max become high, and an unknown value becomes medium. HTTP workers receive both fields as sent unless a provider profile rewrites them, and provider profiles restrict the accepted values per model family.

Structured Output¶

response_format accepts:

• {"type": "text"}
• {"type": "json_object"}
• {"type": "json_schema", "json_schema": {"name": "...", "schema": {...}, "strict": true}}

For json_schema, name must not be empty and schema must be a JSON object ({} is allowed). A string, array, boolean, number, or null schema is rejected with 400 before the request reaches any worker, on every backend, unless the HTTP router streams the request to the worker unparsed (see Request Streaming).

Example Request¶

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What is the capital of France?"}
    ],
    "max_tokens": 100,
    "temperature": 0.7
  }'

Response¶

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1705312345,
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The capital of France is Paris."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 25,
    "completion_tokens": 8,
    "total_tokens": 33
  }
}

Usage¶

On gRPC workers SMG builds usage from the engine's counters; HTTP workers return the engine's own usage object. The speculative-decoding fields appear only when the engine reports proposed draft tokens. SMG reads them from SGLang, TokenSpeed, vLLM (started with --per-request-spec-decode-metrics), and TensorRT-LLM (once its servicer fills the field); MLX reports none.

Streaming Response¶

With "stream": true, responses are sent as Server-Sent Events:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

Response (abbreviated):

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1705312345,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1705312345,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[{"index":0,"delta":{"content":"Hello!"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1705312345,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1705312345,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[],"usage":{"prompt_tokens":9,"completion_tokens":2,"total_tokens":11}}

data: [DONE]

An error before the first chunk is an ordinary JSON error response with its own status; when a worker rejects a streaming request, SMG keeps the worker's content type (application/json, not text/event-stream) so SSE clients still see the error body. If generation fails after a gRPC stream has started, SMG sends data: {"error": {"message": "...", "type": "internal_error"}} followed by data: [DONE].

Stream Usage¶

On HTTP workers stream_options, including keys SMG does not model, is forwarded to the engine, which produces the usage chunks. The Kimi profile turns include_usage on for streaming requests that leave it unset, on every backend; an explicit false is kept.

The DeepSeek V4 profile uses DeepSeek's stream format on gRPC workers instead: aggregate usage rides on the final finish chunk whether or not include_usage is set, and no usage-only chunk follows. With include_usage: true, earlier chunks carry "usage": null. continuous_usage_stats does not apply, and if the backend stream ends before every choice completes, the finish chunk carries no aggregate usage.

Completions¶

Create a text completion (legacy API).

POST /v1/completions

Request Body¶

Completions takes the same extension fields as Chat Completions except the chat-only ones (continue_final_message, separate_reasoning, stream_reasoning, chat_template_kwargs), plus a json_schema string constraint. On gRPC workers each prompt in an array is its own engine request.

Example Request¶

curl http://localhost:30000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "prompt": "The quick brown fox",
    "max_tokens": 50
  }'

Response¶

{
  "id": "cmpl-abc123",
  "object": "text_completion",
  "created": 1705312345,
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "choices": [
    {
      "text": " jumps over the lazy dog.",
      "index": 0,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 4,
    "completion_tokens": 7,
    "total_tokens": 11
  }
}

List Models¶

List available models.

GET /v1/models

Example Request¶

curl http://localhost:30000/v1/models

Response¶

{
  "object": "list",
  "data": [
    {
      "id": "meta-llama/Llama-3.1-8B-Instruct",
      "object": "model",
      "created": 0,
      "owned_by": "self_hosted"
    }
  ]
}

The list covers the models served by self-hosted workers, each once under its canonical ID. Names configured with --model-alias are accepted in requests but are not listed. created is always 0. owned_by is self_hosted for locally hosted workers, or the provider name (for example openai, anthropic, xai, gemini) for upstream providers. When no self-hosted worker serves any model, the endpoint returns 503 with the plain-text body No models available.

The endpoint needs no key. If the caller does send a bearer token (or an x-api-key header) that is not one of the gateway's own keys, SMG treats it as the caller's own provider key: it asks the registered external providers' /v1/models with that token and returns the first non-empty list, falling back to the self-hosted list. A gateway key (--api-key or --tenant-api-key) is never forwarded to a provider.

Audio Transcriptions¶

Transcribe an audio file (batch).

POST /v1/audio/transcriptions

The request is multipart/form-data:

A malformed form, a missing or empty file, a missing model, an out-of-range temperature, or an unparseable stream gets a plain-text 400 before routing. Other form fields are ignored.

curl http://localhost:30000/v1/audio/transcriptions \
  -F file=@audio.wav \
  -F model=Qwen/Qwen3-ASR-1.7B

What serves the request depends on the worker:

On gRPC, a model belongs to the Qwen3-ASR family when its ID, or a serving worker's model ID or model-identity label, contains qwen3-asr or qwen3_asr (ignoring case). Audio input over gRPC is currently accepted by TokenSpeed workers only; other gRPC runtimes answer 400 multimodal_not_supported. The gRPC endpoint is whole-file only:

response_format: json returns {"text": "..."} and text returns the plain transcript. Sampling is greedy unless temperature is set, and a given language pins the transcript language.

Realtime API¶

SMG proxies the OpenAI Realtime API to a realtime-capable worker. Both the OpenAI router (to an upstream provider) and the HTTP router (to a local worker labeled realtime: "true") support it. SMG relays frames verbatim, so the worker must speak the OpenAI Realtime protocol — for local workers, for example vLLM serving an ASR model with the realtime task.

GET /v1/realtime requires the model query parameter and answers 400 without it. POST /v1/realtime/calls takes the model from the model query parameter for an application/sdp body, or from the session JSON in a multipart/form-data body.

WebSocket example¶

# pip install websockets
import asyncio, websockets

async def main():
    url = "ws://localhost:30000/v1/realtime?model=Qwen/Qwen3-ASR-1.7B"
    headers = {"Authorization": "Bearer your-api-key"}
    async with websockets.connect(url, additional_headers=headers) as ws:
        # Send realtime events (session.update, input_audio_buffer.append, ...)
        # and receive transcription/response events from the worker.
        ...

asyncio.run(main())

Provider Profiles¶

Some model families come with a vendor contract that differs from the OpenAI baseline. SMG picks a provider profile from the request's model and applies that profile's defaults and validation to Chat Completions requests before routing, on every backend.

Profile Selection¶

A profile matches when any /-separated segment of the model ID starts with one of its markers, ignoring case, so moonshotai/Kimi-K2-Instruct and /models/kimi-k3 both select Kimi.

Selection reads the model name as sent, before --model-alias resolution: an alias that doesn't match a marker itself gets the OpenAI baseline. Message-level extension fields that belong to another profile (such as Kimi's message tools sent to a non-Kimi model) are dropped before dispatch, with one warning logged per request.

Profile defaults and validation apply to /v1/chat/completions requests that pass through SMG's request validation. They are not applied to /v1/responses, and a request the HTTP router streams to the worker unparsed skips them (see Request Streaming).

Validation Rules¶

A violation is rejected with 400. The response has error.type set to invalid_request_error and error.code set to 400 (as for every validation failure), and error.message carries the rule's message:

{
  "error": {
    "message": "__all__: invalid thinking.type: only enabled or disabled is allowed for this model",
    "type": "invalid_request_error",
    "code": 400
  }
}

The rule code in the last column names the rule in SMG's source and tests; only context_length_exceeded is returned as error.code.

z.ai and DeepSeek V4 check both effort spellings, even the one that thinking.effort shadows.

Defaults and Rewrites¶

Error Responses¶

Error Format¶

Errors that SMG generates use this envelope and repeat code in the X-SMG-Error-Code response header:

{
  "error": {
    "type": "Bad Request",
    "code": "context_length_exceeded",
    "message": "This model's maximum context length is 2048 tokens. However, your request has 2310 input tokens. Please reduce the length of the input.",
    "param": null
  }
}

type is the HTTP status's reason phrase (Bad Request, Not Found, Too Many Requests, Service Unavailable, ...). Request parsing and validation failures use a different shape and carry no X-SMG-Error-Code header:

A 401 from API-key authentication has an empty body. Errors from HTTP workers are relayed with the worker's status and body. SMG never forwards a worker's X-SMG-Error-Code header, so that header always means SMG itself produced the error.

Error Codes¶

On gRPC workers, an engine finish reason of error (outside the OpenAI set) becomes an error response or stream error; clients never receive "finish_reason": "error".

Client Libraries¶

Python (OpenAI SDK)¶

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="your-api-key"  # or "not-needed" if auth disabled
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "user", "content": "Hello!"}
    ]
)

print(response.choices[0].message.content)

JavaScript/TypeScript¶

import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'http://localhost:30000/v1',
  apiKey: 'your-api-key'
});

const response = await client.chat.completions.create({
  model: 'meta-llama/Llama-3.1-8B-Instruct',
  messages: [
    { role: 'user', content: 'Hello!' }
  ]
});

console.log(response.choices[0].message.content);

cURL¶

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-api-key" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Request Headers¶

Header names are case-insensitive. SMG reads the following request headers:

When SMG proxies a request to an HTTP worker, it forwards only these client headers: Authorization, anthropic-version, anthropic-beta, X-Request-ID, X-Correlation-ID, headers starting with X-Request-ID-, traceparent, tracestate, and X-SMG-Routing-Key.

Routing Hint Headers¶

A client, or a proxy in front of SMG that has already tokenized the prompt, can pass the routing signal in a header so SMG does not need the request body to choose a worker. Hints affect only worker placement. A malformed or over-limit hint is ignored and the request routes as if the header were absent; it never causes an error.

# X-SMG-Routing-Tokens carries the prompt's first token IDs
curl http://localhost:30000/v1/completions \
  -H "Content-Type: application/json" \
  -H "X-SMG-Routing-Tokens: 128000,791,4062,14198,39935" \
  -d @request.json

With a valid X-SMG-Routing-Tokens hint, the text-routing policies (cache_aware, bucket) no longer need the body to place the request, so a large body can stream to the worker instead of being buffered; see Request Streaming. When --routing-key-override is enabled, SMG always buffers the body to read its rid, and hints do not change that.

Response Headers¶

Rate Limiting¶

SMG does not send X-RateLimit-* headers. A request shed by admission control, rate limiting, or overload protection gets 429 or 503 with the standard error envelope, the X-SMG-Error-Code header, and usually a Retry-After header in seconds:

Wait at least Retry-After seconds before retrying. For configuration, see Rate Limiting, Tenant Rate Limiting, Priority Scheduling, and Overload Protection.