llm-d - GoModel

GoModel’s llmd provider sends OpenAI-compatible requests to an llm-d Router and adds trusted scheduling metadata for the Endpoint Picker (EPP). Use this provider when GoModel is the authenticated API gateway in front of llm-d.

Prerequisites

A running llm-d deployment with an HTTPRoute that reaches the Router/EPP.
The model name exposed by that route.
Network access from GoModel to the Router service.

The llm-d quickstart creates an EPP service named from the guide release. Use the service name from your own deployment.

Configure with environment variables

LLMD_BASE_URL=http://quickstart-epp.llm-d.svc.cluster.local/v1
LLMD_MODELS=Qwen/Qwen2.5-0.5B-Instruct
GOMODEL_MASTER_KEY=change-me

# Optional controls
LLMD_INFERENCE_OBJECTIVE=standard-traffic
LLMD_FAIRNESS_FROM_USER_PATH=true
# LLMD_API_KEY=router-token

LLMD_BASE_URL is required and should include /v1. LLMD_API_KEY is optional; set it when the Gateway in front of llm-d requires bearer authentication. When LLMD_INFERENCE_OBJECTIVE is omitted, GoModel does not send an inference objective header. Set it when the llm-d deployment defines multiple inference objectives and this GoModel provider should select one of them. Declare LLMD_MODELS when your route does not serve GET /v1/models. GoModel uses that list as the provider’s model inventory.

Configure with YAML

providers:
  llmd:
    type: llmd
    base_url: "http://quickstart-epp.llm-d.svc.cluster.local/v1"
    inference_objective: "standard-traffic"
    fairness_from_user_path: true
    models:
      - id: "Qwen/Qwen2.5-0.5B-Instruct"

fairness_from_user_path defaults to true. GoModel derives the fairness ID from the effective user_path, after authentication and key policy have been applied. Set it to false if another trusted layer sets fairness metadata.

A raw X-GoModel-User-Path supplied by a client is not enough to set the fairness ID. Bind the path to a managed auth key or an extension identity so GoModel can treat it as an effective authenticated path.

Do not forward client-supplied llm-d scheduling headers from an untrusted edge. A client could otherwise select another objective or fairness group. The dedicated provider strips llm-d and Gateway API Inference Extension control headers on passthrough requests, then injects the configured values.

For the objective and fairness ID, GoModel sends both the current x-llm-d-* headers and the deprecated x-gateway-* aliases with the same trusted values. This supports stable llm-d 0.8 deployments while using the canonical names recognized by llm-d 0.9 and later. When EPP flow control returns a 429 on a translated inference route, GoModel preserves the x-llm-d-request-dropped-reason response header so clients can distinguish rejection from post-dispatch eviction.

Verify

First create a managed GoModel API key in the dashboard, bind its user_path to /team/alpha, and assign the returned value to TEAM_ALPHA_GOMODEL_KEY. The key binding, rather than a client-asserted header, establishes the fairness ID used by this request.

curl -s http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer ${TEAM_ALPHA_GOMODEL_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llmd/Qwen/Qwen2.5-0.5B-Instruct",
    "messages": [{"role": "user", "content": "Reply with exactly ok."}]
  }'

GoModel removes the outer llmd/ routing qualifier before sending the model name upstream. Slash-shaped Hugging Face model IDs remain intact.

Supported routes

The provider implements chat completions, Responses, embeddings, and model listing through the OpenAI-compatible API. Passthrough is enabled by default for the other routes in the llm-d HTTP API reference, including OpenAI completions, Anthropic Messages, and vLLM Generate:

curl -s http://localhost:8080/p/llmd/completions \
  -H "Authorization: Bearer change-me" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "prompt": "Hello"
  }'

The separate llm-d Batch Gateway is not exposed as GoModel’s native batch API. Files, audio, and Responses lifecycle utility endpoints are also not advertised by this provider.

Multiple llm-d routes

Use suffixed variables to register independent Router services:

LLMD_PROD_BASE_URL=http://prod-epp.llm-d.svc.cluster.local/v1
LLMD_PROD_MODELS=Qwen/Qwen3-32B
LLMD_PROD_INFERENCE_OBJECTIVE=production

LLMD_DEV_BASE_URL=http://dev-epp.llm-d.svc.cluster.local/v1
LLMD_DEV_MODELS=Qwen/Qwen2.5-0.5B-Instruct
LLMD_DEV_INFERENCE_OBJECTIVE=development

These variables register llmd-prod and llmd-dev. Select them with model names such as llmd-dev/Qwen/Qwen2.5-0.5B-Instruct.

​Prerequisites

​Configure with environment variables

​Configure with YAML

​Verify

​Supported routes

​Multiple llm-d routes

Prerequisites

Configure with environment variables

Configure with YAML

Verify

Supported routes

Multiple llm-d routes