Run the OpenAI-compatible chat front¶

Topics: operations, requests

Before starting, have:

• A running Horde with the /v2/generate/chat routes reachable over HTTP from the front.
• Access to the Horde's primary Redis server.
• At least one text worker with chat enabled.
• The project's dependencies installed with uv sync.

End state: a python -m horde.oai process answering GET /healthz, GET /v1/models and POST /v1/chat/completions, subscribed to the horde's chat completion signals, and serving its API documentation at /docs, /redoc and /openapi.json.

The front is a separate ASGI process. It does not hold a database connection or import anything from the Flask application: it submits each chat request to the horde's native /v2/generate/chat/async, waits for the horde's Redis signal on horde:chat:<request id> (polling the status route as a fallback), and returns an OpenAI response.

Client authentication and attribution¶

A client's Authorization: Bearer <key> becomes the horde apikey. A request without one uses the anonymous key. The front's Client-Agent header is <application>:<version>:<user agent> via horde-oai-front. Horde statistics use it to attribute traffic to the application and distinguish it from direct native calls.

The application value is selected in this order: X-Title, X-OpenRouter-Title, the host of HTTP-Referer, then the host of Referer. These values use version 0. Without these headers, the value is horde-oai-front with the Horde version. The user agent is the client's User-Agent, or unknown.

Control characters and colons are removed from each part. The whole value is at most 256 characters, with the user agent truncated first.

1. Configure the front¶

Every setting is an environment variable, read at start through pydantic-settings. A .env file in the working directory is read too. A variable that is set but invalid stops the start with a validation error that identifies it.

Connections and listening¶

Admission and request limits¶

Request budgets and cancellation¶

Streaming and status polling¶

Connection pools and proxy addresses¶

The proxy passkey belongs to a service account: PUT /v2/users/<user id> with {"generate_proxy_passkey": true}, sent with that account's API key, returns it. Without it, the horde sees every request as coming from the front's address, and one abusive client can put that address in timeout for all of them.

Verification: uv run python -c "from horde.oai.settings import load_settings; print(load_settings())" prints the settings used by the front, without the Redis URL or the passkey. An invalid setting produces an error with the variable's name.

Reversal: unset the variables.

2. Start the front¶

From the repository root:

uv run python -m horde.oai

With docker compose, the aihorde_oai service runs the same command from the horde image and publishes port 7002:

docker compose up -d aihorde_oai

Verification: the log shows Subscribed to chat signals on horde:chat:*, and

curl -s http://127.0.0.1:7002/healthz

returns {"live": true, "ready": true, "signal_subscribed": true, "inflight": 0}. ready: false means the process is at HORDE_OAI_MAX_INFLIGHT or stopping. signal_subscribed: false means Redis is unset or unreachable; requests still complete, but only at the fallback poll interval.

The interactive API documentation is at http://127.0.0.1:7002/docs, and / redirects there. /redoc shows the same description, and /openapi.json is the OpenAPI document itself. /healthz is left out of it, since only operators call it.

Reversal: stop the process, or docker compose stop aihorde_oai. On a clean stop the front waits up to HORDE_OAI_SHUTDOWN_DRAIN_SECONDS for open requests, then cancels every request still open on the horde within HORDE_OAI_SHUTDOWN_CANCEL_SECONDS before closing its Redis subscription. A request whose earlier cancel failed or timed out, even one with a returned response, gets its cancel sent once more. A cancel still in progress is not sent twice.

While the process runs, the same failed cancels are sent again every HORDE_OAI_CANCEL_RETRY_SECONDS. At most 10,000 requests with returned responses are held for this. Past that the oldest is dropped with a warning.

After a crash, open connections are dropped. Their Horde jobs run to completion and are charged. Their replies are not collected.

3. Check the model list¶

curl -s http://127.0.0.1:7002/v1/models | jq '.data[].id'

Verification: the names match the horde's GET /v2/status/models?type=text&chat=true, which lists only models that an online chat worker serves. An empty list means no chat worker is online. GET /v1/models/<model name> returns one entry. The name may contain slashes. The list is cached for five seconds per process.

Each entry is built from the model's chat object in that list:

The limits and parameters are availability hints: a request can still lack an eligible worker a moment later.

4. Send a chat request¶

curl -s http://127.0.0.1:7002/v1/chat/completions \
  -H "Authorization: Bearer <your-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<model-name from step 3>",
    "messages": [{"role": "user", "content": "Say hello."}],
    "max_tokens": 64
  }' | jq .

Verification: the response is a chat.completion. Its id is chatcmpl- plus the Horde request id, and its model is the model that generated the reply. x_horde includes request_id, worker_id, worker_name, kudos (what the request cost), charge_basis (the Horde's pricing basis, passed through unchanged), estimated_usage and, when the horde supplies them, queue_wait_seconds and context_adjustments.

estimated_usage: true means the worker did not report token counts and usage is an estimate at three characters per token. x_horde.notice appears when the worker did not return visible text. A sampling setting or the model choice is the likely cause. Each choice includes the worker's native_finish_reason beside finish_reason.

The front logs one line per request: Chat request <id> finished: streamed=False submitted_at=... woke_at=... woke_by=signal waited=...s. woke_by=poll on most requests means signals are not arriving.

Streaming check¶

Add "stream": true (and optionally "stream_options": {"include_usage": true}) to receive server-sent events instead: : horde queue_position=7 wait_time=42 comments while the request waits, then a role chunk, content chunks, a finish chunk with x_horde, the usage chunk with choices: [] when asked for, and data: [DONE]. A plain request gets its content as the worker posts progress. The text of each post is spread over the estimated interval until the next post. A request with tools or structured output gets the whole text in one chunk. The response includes Cache-Control: no-cache, X-Accel-Buffering: no and x-should-retry: false.

The same request works from an OpenAI SDK by pointing its base URL at http://<front-host>:7002/v1 and using the horde API key as the SDK's API key.

Reversal: none needed. A request the client abandons is cancelled on the horde.

5. Check the front against the OpenAI SDK¶

The conformance module drives the pinned official openai Python SDK against the real front and the real native app, each served on a loopback port, with a fake chat worker answering every job, so it does not need an inference backend. It runs under the integration test stack, which provides Postgres, the front's Redis subscription and object storage for chat images:

AI_HORDE_REQUIRE_TEST_DEPS=1 bash tests/run_with_test_stack.sh --   .venv/Scripts/python.exe -m pytest tests/integration/test_chat_front_sdk.py -p no:cacheprovider

On Linux or macOS the interpreter is .venv/bin/python.

The runner works from Git Bash as from Linux or macOS. When the Garage bootstrap fails it exits with an error and does not run the command, so a test run does not silently provision separate databases in place of the stack.

Verification: every test passes: the model list includes the worker's chat model and leaves out a model that is not served by any online worker. A completion and a streamed completion return the worker's reply, finish_reason stop and the worker's token counts. logit_bias is a BadRequestError whose param is logit_bias. A model without an online worker returns 503 no_eligible_worker at once. And an image sent as a data URL reaches the worker, which declares vision and the url image transport, as a signed http(s) URL.

Reversal: the run creates and drops its database schema. The test stack stays up for later runs unless run_with_test_stack.sh is given --down.

Request keys the front accepts¶

The front accepts model (or models), messages, stream, stream_options, max_tokens, max_completion_tokens, temperature, top_p, frequency_penalty, presence_penalty, stop, seed, reasoning_effort, n, tools, tool_choice, parallel_tool_calls and response_format. It applies streaming options locally and forwards generation parameters to the Horde. max_tokens and max_completion_tokens must agree when both are sent and lie between 16 and 16384. With neither, the horde generates up to 512 tokens. n must be 1. stop is a string or up to four strings. stream_options needs "stream": true.

Message roles are system, developer, user, assistant and tool. developer is stored as system, so the developer role loses precedence over system messages. An assistant message may include tool_calls and a tool message answers one by tool_call_id (see the chat front reference). Content is a string or an array of text parts. A user message may also include image_url parts (see Image inputs).

Stock clients send some keys at their defaults. These are accepted and dropped: empty or null tools, null tool_choice or parallel_tool_calls, response_format: {"type": "text"}, logprobs: false, top_logprobs: 0, store: false and service_tier "auto" or "default". user and metadata are accepted and unused.

A non-null tool_choice, including "none", or parallel_tool_calls without tools returns 400 ChatRequestConflict. Unsupported top-level keys and content part types return 400 UnsupportedChatParameter with the path in param, such as messages[1].content[0].type=input_audio. Unknown keys inside messages, tool definitions and tool_choice are dropped. See Parameter screening.

The horde object takes worker restrictions and context options, along with conformance, reasoning visibility, availability and kudos limits. dry_run and operation_key are refused on the front. See Horde options for the complete list and defaults.

Image inputs¶

A user message may include {"type": "image_url", "image_url": {"url": ..., "detail": ...}} parts, with url a data:image/...;base64, URL or an http(s) URL and detail optionally auto, low or high. The front forwards them unchanged. It refuses before the submit a request with more than 20 image parts (400 TooManyExtraSourceImages), a data URL that decodes to more than 5 MiB, or data URLs that decode to more than 25 MiB in all (400 SourceImageSizeExceeded, with param the path of the part's url). Data URLs do not count toward HORDE_OAI_MAX_TEXT_BYTES.

The front does not have the user's trust status. The Horde therefore enforces the 5 MiB total for users who are neither trusted nor patrons. It fetches, checks and stores each image, charges the extra source image tax per image, refuses images over the user's total (400 SourceImageSizeExceeded), and sends the image only to a worker that declares the vision feature and an image transport.

Image refusals pass through with the horde's rc as code, such as SourceImageUrlRefused for a URL at a private address or SourceImageResolutionExceeded, and 503 ChatImageStorageUnavailable when the image store refuses the image. A model whose list entry shows image in architecture.input_modalities has such a worker online.

Errors the front returns¶

Every error is an OpenAI envelope, {"error": {"message", "type", "param", "code", "metadata"}}, with param and code written as null when they do not apply. metadata holds what is known: request_id, the horde's rc, retry_after, and submit_outcome: "unknown" with the request's operation_key when a submit was lost and recovery did not settle it.

A Horde error keeps the Horde's status, and its rc becomes code. A horde 429 keeps Retry-After and the X-RateLimit-* headers, and has code rate_limit_exceeded when the Horde does not send an rc, such as for its per-route request limit. A horde 503 without Retry-After (such as ChatAdmissionDisabled) gets Retry-After: 30. A status read that ends the wait with a horde error other than 404, 429 or a 5xx, such as 401 or 403, cancels the request before the error is returned. A 404 means the horde no longer holds it.

Once the horde holds a request, or may hold one, every error includes x-should-retry: false, so OpenAI SDKs do not retry into a second charged job. Errors before the submit do not include the header and are safe to retry.

A streamed request's headers go out after the first status read, so a refusal at that read (a declaration or limit gap, queue_unavailable, or a status error) is the HTTP error in the table, as on a non-streamed request. In a stream, an error after the headers arrives as one data: event with the same envelope and no choice, then data: [DONE]. A streamed request has a separate budget, HORDE_OAI_STREAM_TIMEOUT_SECONDS.

A client retry after a lost response is a new request and can create a second charged job. The front does not deduplicate requests.

Lost submits¶

The front sends each client request with a fresh UUID as horde.operation_key. The horde claims the key for the submitting user when it activates the request. A repeated key with the same body returns the first request's {id, kudos} with status 200, without queuing or charging anything.

When a submit was sent and a response did not arrive (a read timeout, a reset or a dropped connection), the front sends the identical body with the same key again to the next root, up to HORDE_OAI_SUBMIT_RECOVERY_ATTEMPTS times and only while the request's budget lasts. When every root is out of the rotation after the failure, the attempt goes to them anyway. Then:

• A response with an id, 200 for an existing claim or 202 when the lost submit did not reach the Horde, continues the request as any submit does. The log shows Recovered the lost submit of operation <key> on attempt <n>.
• A connection failure, a 429 or a 5xx moves on to the next attempt.
• OperationKeyConflict returns 502, as in the table above.
• Any other horde refusal is passed through with x-should-retry: false, submit_outcome: "unknown" and the key, since the lost submit may still have queued the request.
• With the attempts or the budget spent, the response is 502 HordeUnavailable with submit_outcome: "unknown" and the key. The key finds the request, if one exists, in waiting_prompts.operation_key.

A refused connection on the first submit is not a lost submit: the request did not reach a horde, and the 502 does not include an x-should-retry header.

Two metrics count these cases (see Chat front reference):

Neither metric measures prevented duplicate requests from clients. A claim match shows that the Horde queued a request whose submit response was lost. A retry hint shows that a client SDK retried, for any cause.

Export telemetry¶

The front exports OpenTelemetry traces (one server span per request, /healthz excluded, and a client span per call to the horde that the horde's spans continue), metrics, and its Loguru log records as OpenTelemetry logs, through the same environment variables as the horde's Flask application. With none of them set it does not export anything.

PYROSCOPE_ENABLED=true starts Pyroscope continuous profiling with the horde's PYROSCOPE_* variables (see horde.telemetry). The application name is the front's service name, so its profiles are separate from the horde's.

Verification: with an endpoint set, a collector receives horde.oai.* metrics within the SDK's export interval after the first request, one Chat request <id> finished: ... log record per finished request, and a trace per request whose root is the front's POST /v1/chat/completions span with the horde's POST /api/v2/generate/chat/async span beneath it. The metric and log field lists are in the Chat front reference.

Reversal: unset the variables and restart the front.