Admin API Reference¶

SMG provides administrative endpoints for managing tokenizers, workers, WASM modules, and engine caches, plus read-only model and server information.

Tokenizer Management¶

Manage tokenizers for text processing and tokenization.

Add Tokenizer¶

POST /v1/tokenizers

Adds a new tokenizer from a local path or HuggingFace model ID.

Request Body:

{
  "name": "llama3-tokenizer",
  "source": "meta-llama/Meta-Llama-3-8B",
  "chat_template_path": "/path/to/template.jinja"
}

Response: 202 Accepted

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "pending",
  "message": "Tokenizer 'llama3-tokenizer' registration job submitted. Loading from: meta-llama/Meta-Llama-3-8B"
}

Loading runs in the background. Poll Get Tokenizer Status with the returned id.

Response: 409 Conflict when a tokenizer with this name is already registered. The id is the existing tokenizer's:

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "failed",
  "message": "Tokenizer 'llama3-tokenizer' already exists"
}

List Tokenizers¶

GET /v1/tokenizers

Returns all registered tokenizers.

Response: 200 OK

{
  "tokenizers": [
    {
      "id": "550e8400-e29b-41d4-a716-446655440000",
      "name": "llama3-tokenizer",
      "source": "meta-llama/Meta-Llama-3-8B",
      "vocab_size": 128256
    }
  ]
}

Get Tokenizer¶

GET /v1/tokenizers/{tokenizer_id}

Returns details for a specific tokenizer. {tokenizer_id} is the tokenizer's ID or its name.

Response: 200 OK

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "name": "llama3-tokenizer",
  "source": "meta-llama/Meta-Llama-3-8B",
  "vocab_size": 128256
}

Response: 404 Not Found

{
  "error": {
    "message": "Tokenizer 'llama3-tokenizer' not found",
    "type": "tokenizer_not_found"
  }
}

Get Tokenizer Status¶

GET /v1/tokenizers/{tokenizer_id}/status

Returns the loading status of a tokenizer. A loaded tokenizer can be looked up by ID or name; a job that is still queued, loading, or failed only by the ID that Add Tokenizer returned.

Response: 200 OK

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "completed",
  "message": "Tokenizer 'llama3-tokenizer' is loaded and ready",
  "vocab_size": 128256
}

vocab_size is present only for completed. A failed status is kept for about five minutes. After that, and for an ID that SMG does not know:

Response: 404 Not Found

{
  "error": {
    "message": "Tokenizer '550e8400-e29b-41d4-a716-446655440000' not found and no pending job",
    "type": "not_found"
  }
}

Remove Tokenizer¶

DELETE /v1/tokenizers/{tokenizer_id}

Removes a tokenizer. {tokenizer_id} is the tokenizer's ID or its name.

Response: 200 OK

{
  "success": true,
  "message": "Tokenizer 'llama3-tokenizer' removed successfully"
}

Response: 404 Not Found

{
  "success": false,
  "message": "Tokenizer 'llama3-tokenizer' not found"
}

Worker Management¶

Register, inspect, update, and remove backend workers at runtime.

worker_id is the UUID that POST /workers returns. Changes are asynchronous: the gateway checks the request, queues a job, and answers 202 Accepted. Follow the job with GET /workers/{worker_id}.

Create Worker¶

POST /workers

Registers a worker. The body is a worker spec; only url is required. The URL scheme selects the transport:

The scheme must be lowercase. Send the spec as JSON:

curl -X POST http://localhost:30000/workers \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -d @worker.json

{
  "url": "http://gpu1:8000",
  "api_key": "worker-secret-key",
  "labels": {"region": "us-east"}
}

{
  "url": "grpc://gpu2:50051",
  "runtime_type": "sglang",
  "models": [
    {
      "id": "meta-llama/Llama-3.1-8B-Instruct",
      "aliases": ["llama-3.1-8b"],
      "tool_parser": "llama"
    }
  ]
}

{
  "url": "ipc:///tmp/smg-zmq/engine-31000",
  "runtime_type": "vllm",
  "labels": {"model_path": "Qwen/Qwen3-8B"}
}

{
  "url": "http://gpu3:8000",
  "runtime_type": "vllm",
  "overload": {
    "waiting_requests": 32,
    "token_usage": 0.85
  },
  "http_pool": {
    "http2": true,
    "connect_timeout_secs": 5
  },
  "health": {
    "drain_settle_secs": 30
  }
}

Response: 202 Accepted, with a Location header equal to location.

{
  "status": "accepted",
  "worker_id": "01997a4e-3c2b-7f1d-9a8e-2b6c4d5e7f80",
  "url": "http://gpu1:8000",
  "location": "/workers/01997a4e-3c2b-7f1d-9a8e-2b6c4d5e7f80",
  "message": "Worker addition queued for background processing"
}

202 means the request passed the checks listed under Worker Errors. Probing, engine detection, and the remaining validation run in the background job; Get Worker shows how to follow it.

Worker Spec¶

The body of POST /workers and PUT /workers/{worker_id}. Only url is required. The nested blocks (health, overload, http_pool, resilience) are partial: a field you leave out falls back to the gateway default. Unknown fields are ignored rather than rejected, so check the spelling. A known field with the wrong type, or an unknown enum value, is rejected with 422.

Identity and transport

Models and labels

Model card fields (models[]):

Cards also accept display_name, hf_model_type, architectures, provider, tokenizer_path, metadata, id2label, and num_labels.

Labels with built-in meaning:

Routing metadata

Health checks (health): per-worker overrides of the gateway health-check flags. See Health Checks.

Overload thresholds (overload): self-hosted workers only. See Overload Protection.

Either field turns overload protection on for this worker, even when the gateway flags leave it off; a field you leave out uses the gateway value. The worker returns to routing once a load report is below both thresholds. An out-of-range value fails registration. PATCH cannot change this block; use PUT.

HTTP client (http_pool): applies to HTTP workers, including external providers. See Request Streaming.

Resilience (resilience): per-worker circuit-breaker settings. See Circuit Breakers.

ZMQ workers: see ZMQ Workers.

Registration of a ZMQ worker fails when the runtime is not vllm or tokenspeed, when worker_type is not regular (disaggregated workers need gRPC), when no model ID is available, when dp_size is above 1 on a gateway running with --dp-aware, or when the handshake address is not tcp:// or is already bound by another ZMQ worker. Setting zmq_handshake_address on a non-ZMQ worker also fails registration. Health checks stay on for ZMQ workers, because the probe is what reconnects a restarted engine.

PD disaggregation and KV transfer: see PD Disaggregation.

health, http_pool, and resilience take effect but are not echoed back by GET /workers; overload is.

Worker Errors¶

Worker endpoints report errors as {"error": "<message>", "code": "<CODE>"}, not in the shape described under Error Responses:

{
  "error": "Invalid value for field 'worker_url': 10.0.0.5:8000 - URL must start with a lowercase http://, https://, grpc://, grpcs://, or ipc:// scheme",
  "code": "BAD_REQUEST"
}

A body that does not parse never reaches these checks. The gateway answers with a plain-text body: 415 when Content-Type: application/json is missing, 400 for malformed JSON, and 422 (Failed to deserialize the JSON body into the target type: ...) for a missing url, a wrong type, or an unknown value such as "worker_type": "prefil".

Other problems surface in the background job, after the 202:

• overload.waiting_requests of 0, or overload.token_usage outside (0.0, 1.0]
• the ZMQ constraints listed under Worker Spec
• an unknown tool_parser or reasoning_parser
• a worker that never answers, or http_pool.http2: true for a worker that only speaks HTTP/1.1

List Workers¶

GET /workers
GET /workers?model={model_id}

Returns every registered worker. model keeps only workers that serve that model ID or alias; workers without a model list match any model. The stats counts cover the returned workers (encode workers are counted in total only). Workers whose registration job is still running are not listed.

Response: 200 OK (abridged: discovered labels and card fields vary by engine)

{
  "workers": [
    {
      "id": "01997a4e-3c2b-7f1d-9a8e-2b6c4d5e7f80",
      "model_id": "meta-llama/Llama-3.1-8B-Instruct",
      "url": "grpc://gpu2:50051",
      "models": [
        {
          "id": "meta-llama/Llama-3.1-8B-Instruct",
          "aliases": ["llama-3.1-8b"],
          "model_type": ["chat", "completions", "responses", "tools"],
          "context_length": 131072,
          "tool_parser": "llama"
        }
      ],
      "worker_type": "regular",
      "connection_mode": "grpc",
      "runtime_type": "sglang",
      "labels": {
        "model_path": "meta-llama/Llama-3.1-8B-Instruct",
        "tp_size": "1"
      },
      "priority": 50,
      "cost": 1.0,
      "max_connection_attempts": 20,
      "is_healthy": true,
      "status": "ready",
      "load": 2,
      "http2": false
    }
  ],
  "total": 1,
  "stats": {
    "prefill_count": 0,
    "decode_count": 0,
    "regular_count": 1
  }
}

Each worker object carries its spec fields (never api_key) plus:

Get Worker¶

GET /workers/{worker_id}

Returns one worker in the same shape as a GET /workers entry. While a job for the worker is queued or running, or for about five minutes after one fails, the response also carries job_status. A job that succeeds clears it.

While POST /workers is still registering the worker, the response is a placeholder: only id, url, status, and job_status are meaningful, and the other fields hold defaults.

Response: 200 OK

{
  "id": "01997a4e-3c2b-7f1d-9a8e-2b6c4d5e7f80",
  "url": "grpc://gpu2:50051",
  "worker_type": "regular",
  "connection_mode": "http",
  "runtime_type": "unspecified",
  "priority": 50,
  "cost": 1.0,
  "max_connection_attempts": 20,
  "is_healthy": false,
  "status": "pending",
  "load": 0,
  "http2": false,
  "job_status": {
    "job_type": "AddWorker",
    "worker_url": "grpc://gpu2:50051",
    "status": "processing",
    "message": null,
    "timestamp": 1790250060
  }
}

Registration keeps probing a worker that does not answer, every --worker-startup-check-interval seconds (default 30), until --worker-startup-timeout-secs (default 1800) runs out. If a POST /workers job fails, the gateway releases the reserved ID: GET /workers/{worker_id} returns 404, and the reason appears only in the gateway log (Failed job: type=AddWorker ...). A failed PATCH, PUT, or DELETE job leaves the worker in place with a failed job_status.

Update Worker (partial)¶

PATCH /workers/{worker_id}

Changes a few fields in place. Omitted fields keep their current values. The worker keeps its status and connections, and nothing is probed again.

Request Body:

{
  "priority": 75,
  "labels": {"tier": "gold"},
  "api_key": "new-api-key",
  "health": {"check_interval_secs": 15, "drain_settle_secs": 60}
}

Any other field, such as overload, http_pool, resilience, or models, is ignored. Use PUT to change those.

In v1.11.0, a PATCH also rebuilds the worker's circuit breaker with built-in thresholds (5 failures, 2 successes, 30 seconds), ignoring the gateway's circuit-breaker flags and any resilience overrides until a PUT re-registers the worker.

Response: 202 Accepted

{
  "status": "accepted",
  "worker_id": "01997a4e-3c2b-7f1d-9a8e-2b6c4d5e7f80",
  "message": "Worker update queued for background processing"
}

Replace Worker (full)¶

PUT /workers/{worker_id}

Replaces the worker's spec and re-runs the full registration workflow (probe, engine detection, metadata discovery) under the same ID. The body is a complete worker spec. Fields you omit take their defaults rather than keeping their current values, so resend any labels, overload, or other settings you want to keep.

The url must equal the worker's current URL; to move a worker, DELETE it and POST the new URL. PUT is refused with 400 when the gateway runs with --dp-aware, because one spec expands to one worker per data-parallel rank; use DELETE and POST there too. If the worker is deleted or replaced again before the job runs, the job fails rather than overwrite the newer state.

Response: 202 Accepted, with the same body as PATCH.

Delete Worker¶

DELETE /workers/{worker_id}

Removes a worker. A ready worker first moves to draining: it receives no new requests, and the gateway waits health.drain_settle_secs (default --drain-settle-secs, 5 seconds) so in-flight requests can finish before it removes the worker. A worker in any other status is removed without waiting.

Response: 202 Accepted

{
  "status": "accepted",
  "worker_id": "01997a4e-3c2b-7f1d-9a8e-2b6c4d5e7f80",
  "message": "Worker removal queued for background processing"
}

Cache Management¶

Flush the engines' KV caches, and read the engine load the gateway has polled from its workers.

Flush Cache¶

POST /flush_cache

Asks every registered HTTP and gRPC worker to drop its KV prefix cache. The calls go out in parallel, and the response reports the outcome per worker. Requires admin authentication.

The fan-out does not filter by runtime: external-provider workers registered over HTTP receive the POST too, and appear under failed if they reject it. The endpoint only calls the workers; it does not reset the gateway's own cache-aware routing state.

Response: 200 OK when every worker that was called succeeds (or there was nothing to call), 206 Partial Content when at least one fails, including when all of them fail.

{
  "status": "success",
  "message": "Successfully flushed cache on all 3 workers",
  "workers_flushed": 3,
  "total_http_workers": 2,
  "total_grpc_workers": 1,
  "total_zmq_workers_skipped": 0,
  "total_workers": 3
}

On a partial failure, status is "partial_success" and two more fields list the outcome, successful (worker URLs) and failed ({worker, error} entries):

{
  "status": "partial_success",
  "message": "Cache flush: 2 succeeded, 1 failed (1 ZMQ workers skipped: no cache-flush RPC)",
  "workers_flushed": 2,
  "total_http_workers": 2,
  "total_grpc_workers": 1,
  "total_zmq_workers_skipped": 1,
  "total_workers": 4,
  "successful": ["http://gpu1:8000", "grpc://gpu3:50051"],
  "failed": [
    {
      "worker": "http://gpu2:8000",
      "error": "flush_cache failed for worker http://gpu2:8000/flush_cache: HTTP 500 Internal Server Error"
    }
  ]
}

Get Loads¶

GET /loads
GET /get_loads

Returns the engine load the gateway's load monitor last collected: one entry per worker per DP rank, plus a fleet-wide aggregate. The body uses the same schema engines report on /v1/loads, with every entry tagged by worker and worker_type.

The response is built from the monitor's cached snapshot, the same numbers the routing policies act on, so no request reaches a worker. Values can be up to one poll interval old (--load-monitor-interval, 10 seconds by default). Serving from the cache is also what makes the route safe to poll from another SMG gateway that registers this one as a worker.

• Auth: /loads is a public route, served without auth like /health. /get_loads is a deprecated alias that stays on the admin tier and requires admin authentication.
• Filter: ?model=<model_id> limits the response to workers serving that model.
• Missing workers: workers without a current report (not Ready yet, no load source, or a failed last poll) are left out rather than reported as idle. Before the first poll, loads is empty and aggregate is absent.

Response: 200 OK

{
  "timestamp": "2026-09-24T18:20:11.482913+00:00",
  "version": "smg-1.11.0",
  "dp_rank_count": 2,
  "loads": [
    {
      "worker": "http://gpu1:8000",
      "worker_type": "regular",
      "dp_rank": 0,
      "num_running_reqs": 12,
      "num_waiting_reqs": 3,
      "num_waiting_uncached_tokens": 5120,
      "num_total_reqs": 15,
      "num_used_tokens": 48000,
      "max_total_num_tokens": 131072,
      "token_usage": 0.37,
      "gen_throughput": 1850.5,
      "cache_hit_rate": 0.62,
      "utilization": 0.41,
      "max_running_requests": 256
    },
    {
      "worker": "http://gpu2:8000",
      "worker_type": "regular",
      "dp_rank": 0,
      "num_running_reqs": 9,
      "num_waiting_reqs": 0,
      "num_waiting_uncached_tokens": 0,
      "num_total_reqs": 9,
      "num_used_tokens": 30000,
      "max_total_num_tokens": 131072,
      "token_usage": 0.23,
      "gen_throughput": 1420.0,
      "cache_hit_rate": 0.55,
      "utilization": 0.3,
      "max_running_requests": 256
    }
  ],
  "aggregate": {
    "total_running_reqs": 21,
    "total_waiting_reqs": 3,
    "total_reqs": 24,
    "avg_token_usage": 0.3,
    "avg_throughput": 1635.25,
    "avg_utilization": 0.355
  }
}

The same per-worker report appears as engine_load on GET /workers and GET /workers/{worker_id}. It is unrelated to their load field, which counts requests the gateway has in flight to the worker. Overload Protection explains where each backend's report comes from.

Model Information¶

Query model and server information. These are public routes and need no authentication.

List Models¶

GET /v1/models

Returns the models that the registered self-hosted workers serve, read from the gateway's worker registry rather than from the workers. A caller that sends its own provider key gets the external providers' model list instead, fetched with that key. See List Models for the details.

Response: 200 OK

{
  "object": "list",
  "data": [
    {
      "id": "meta-llama/Llama-3.1-8B-Instruct",
      "object": "model",
      "created": 0,
      "owned_by": "self_hosted"
    }
  ]
}

created is always 0. With no self-hosted model to list, the route answers 503 with the plain-text body No models available.

Get Model Info¶

GET /get_model_info

Forwards the request to the first healthy regular HTTP worker (the first healthy prefill worker in HTTP PD mode) and returns that worker's answer. SGLang serves this route, and its answer includes fields such as model_path, tokenizer_path, and is_generation. An engine without the route, such as vLLM, answers 404. In regular mode the worker's status and body pass through unchanged; in HTTP PD mode an error status comes back in the gateway's error envelope. With no healthy worker to ask, the route answers 503.

Get Server Info¶

GET /get_server_info

Forwards the request the same way as Get Model Info and returns the worker's answer. SGLang returns its server arguments together with scheduler state and its version.

WASM Module Management¶

Manage WebAssembly plugins. Modules are registered from files accessible to the gateway process; the request body contains descriptors with paths, not binary payloads.

These routes need --enable-wasm. Without it, each answers 500 Internal Server Error with an empty body.

Add WASM Module¶

POST /wasm

Registers one or more WASM modules.

Request Body: JSON WasmModuleAddRequest

{
  "modules": [
    {
      "name": "custom-middleware",
      "file_path": "/etc/smg/wasm/custom-middleware.wasm",
      "module_type": "Middleware",
      "attach_points": [
        {"Middleware": "OnRequest"},
        {"Middleware": "OnResponse"}
      ]
    }
  ]
}

The only supported module_type today is Middleware. Valid Middleware attach points are OnRequest, OnResponse, and OnError, but OnError modules never run in v1.11.0: the middleware calls only OnRequest and OnResponse modules, and skips OnResponse for streaming responses.

Response: 200 OK on full success, 400 Bad Request if any module failed to register. The response body echoes every requested module with an add_result field indicating success (carrying the assigned UUID) or failure (carrying the error message).

{
  "modules": [
    {
      "name": "custom-middleware",
      "file_path": "/etc/smg/wasm/custom-middleware.wasm",
      "module_type": "Middleware",
      "attach_points": [
        {"Middleware": "OnRequest"},
        {"Middleware": "OnResponse"}
      ],
      "add_result": {
        "Success": "550e8400-e29b-41d4-a716-446655440000"
      }
    }
  ]
}

List WASM Modules¶

GET /wasm

Returns all registered WASM modules together with aggregate execution metrics.

Response: 200 OK

{
  "modules": [
    {
      "module_uuid": "550e8400-e29b-41d4-a716-446655440000",
      "module_meta": {
        "name": "custom-middleware",
        "file_path": "/etc/smg/wasm/custom-middleware.wasm",
        "sha256_hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
        "size_bytes": 65536,
        "created_at": "2024-01-15T12:00:00.000000000Z",
        "last_accessed_at": "2024-01-15T12:05:00.000000000Z",
        "access_count": 40,
        "attach_points": [
          {"Middleware": "OnRequest"}
        ]
      }
    }
  ],
  "metrics": {
    "total_executions": 40,
    "successful_executions": 40,
    "failed_executions": 0,
    "total_execution_time_ms": 120,
    "max_execution_time_ms": 8,
    "average_execution_time_ms": 3.0
  }
}

metrics covers every module. average_execution_time_ms is total_execution_time_ms divided by total_executions, and is left out until a module has run.

Remove WASM Module¶

DELETE /wasm/{module_uuid}

Removes a WASM module. The body is a plain text status message, not JSON.

Response: 200 OK

Module removed successfully

On failure returns 400 Bad Request with the error text as the body. A module_uuid that is not a UUID gets 400 with an empty body.

Error Responses¶

The admin endpoints do not share one error format. The body depends on the part of SMG that rejects the request:

Errors that the gateway generates itself, for example when no worker is available, use this envelope and repeat code in the X-SMG-Error-Code response header:

{
  "error": {
    "type": "Service Unavailable",
    "code": "no_workers",
    "message": "No workers are available",
    "param": null
  }
}

type is the reason phrase of the HTTP status (Not Found, Service Unavailable, ...), and param is always null. The inference endpoints use the same envelope; see Error Responses in the OpenAI-compatible API reference.

Authentication¶

The tokenizer, worker, WASM, cache flush, and deprecated /get_loads endpoints on this page are control-plane routes. Send the credential as Authorization: Bearer <token>:

1. With control-plane auth configured (--control-plane-api-keys, or --jwt-issuer with --jwt-audience): a control-plane API key or JWT whose role is admin. A missing or invalid credential gets 401, and one with another role gets 403. The shared --api-key is not accepted.
2. Without it: the shared --api-key. Per-tenant --tenant-api-key keys are rejected, so a gateway with tenant keys but no --api-key answers 401 to every call. With no keys configured at all, the routes are open.

Public endpoints need no authentication: the health probes, /loads, /engine_metrics, and the Model Information routes. See Auth Model for every route's tier.