Tenant Rate Limiting Reference¶

Precise contract for per-tenant token/request rate limiting: every configuration knob, the YAML policy schema, and the exact response shape. For how it works conceptually — the reserve/settle model, retry handling, n>1 accounting — see Tenant Rate Limiting.

Tenant rate limiting is disabled by default and, when enabled, only enforced on the gRPC router's Chat, Generate, Completion, and Messages endpoints (Harmony-mode chat included). It does not apply to the Responses endpoint, embeddings, classify, audio transcriptions, or the HTTP/external-provider routers.

Enabling it¶

smg \
  --worker-urls grpc://w1:9000 grpc://w2:9000 \
  --tenant-rate-limit-enabled \
  --tenant-rate-limit-config /etc/smg/tenant-rate-limit.yaml

CLI flags¶

Both flags belong to the Rust smg binary. The Python launcher (smg launch from pip, and the container image) does not accept them in v1.11.0.

YAML configuration¶

default_policy:
  tokens_per_minute: 100000
  requests_per_minute: 600

tenants:
  - tenant_key: "auth:team-red"
    tokens_per_minute: 500000
    requests_per_minute: 3000
    model_rules:
      - rule_id: gpt4-cap
        matcher:
          type: exact
          value: gpt-4
        tokens_per_minute: 50000
        requests_per_minute: 300
      - rule_id: legacy-models
        matcher:
          type: prefix
          value: "legacy-"
        tokens_per_minute: 10000
        requests_per_minute: 100

  - tenant_key: "anonymous"
    tokens_per_minute: 5000
    requests_per_minute: 60

A tenant not listed under tenants uses default_policy, with its own buckets: tenants never share a bucket. Tenant keys are the same canonical keys SMG resolves elsewhere in the request path — auth:<id>, header:<id>, ip:<address>, or anonymous — there's no separate tenant-identity system for this feature. A --tenant-api-key team-red:<key> caller resolves to auth:team-red; a caller using the shared --api-key resolves to auth: followed by the hex SHA-256 of that key.

default_policy / tenants[] fields¶

model_rules[] fields¶

At most one model rule applies per request. An exact match wins over any prefix match; the longest matching prefix wins among competing prefixes. Rules never stack with each other — only with the tenant-global limits. When a rule applies, a reservation must be affordable in both the tenant-global scope and the rule's scope, or it's denied.

Validation¶

Checked once, at load, before the gateway ever serves traffic on this config:

• default_policy must not set tenant_key; every entry under tenants must.
• Tenant keys must be non-empty, have no surrounding whitespace, be unique across tenants, and be a canonical serving-path tenant key (auth:, header:, ip:-prefixed, or exactly anonymous) — a bare ID copy-pasted without its prefix is rejected rather than silently never matching.
• A header: tenant key requires --trust-tenant-header, since without it no request resolves to a header: tenant.
• tokens_per_minute and requests_per_minute must be > 0, on every scope (default_policy, each tenant, each model rule).
• rule_id must match [A-Za-z0-9._-]+, and be unique within its tenant (or default_policy).
• matcher.value must be non-empty and have no surrounding whitespace.
• No two rules within the same tenant may share the same exact value, or the same prefix value (an exact and a prefix rule may share the same literal string — they're different match kinds).
• Unknown YAML fields anywhere in the document are rejected rather than silently ignored (so a typo like tenant: instead of tenants: fails loudly instead of compiling to an empty override list).

Any validation failure is the same as an unparsable file: the gateway refuses to start.

Response codes¶

The response body is the gateway's standard JSON error envelope:

{
  "error": {
    "type": "Too Many Requests",
    "code": "tenant_rate_limit_exceeded",
    "message": "Tenant rate limit exceeded for this request",
    "param": null
  }
}

How admission is computed¶

Each affected scope (tenant-global, and the matching model rule if any) is a continuous-refill bucket with two independent counters:

• Capacity = the configured tokens_per_minute / requests_per_minute value itself. The bucket starts full.
• Refill rate = capacity ÷ 60, applied continuously (not in discrete per-minute resets).
• Reserve debits the estimated input-token count from the token counter and a flat 1 from the request counter, only if both counters can currently afford it. A request is denied if either can't.
• Settle applies a signed delta — (real input tokens + real completion tokens) − estimated tokens — to the token counter only. A response that used more than estimated can push the counter temporarily negative (debt); the request counter is never trued up, since it was always exactly 1.

Observability¶

There are currently no dedicated Prometheus metrics for tenant rate-limit decisions — admissions and settlement deltas are not exposed as counters or histograms today. Denials are visible as the 429 responses themselves, which the generic smg_http_responses_total counter records with status_code="429" and error_code="tenant_rate_limit_exceeded".

See also¶