Send chat completions through the OpenAI-compatible front¶

Topics: chat, requests

Before starting, have:

• An OpenAI chat completions client with configurable base URL and bearer token.
• A Horde chat front's base URL, used below as OAI_BASE_URL. The public service is https://oai.aihorde.net/v1.
• A Horde API key, used below as HORDE_API_KEY, or the anonymous key 0000000000.

End state: the client sends POST /v1/chat/completions to the Horde, streamed or not, with tools, images and structured output where a served model supports them, and it handles the Horde's refusals and retry headers without creating duplicate charged jobs.

The front returns 400 for unsupported top-level parameters. Unknown keys inside messages, tools definitions and a tool_choice object are dropped, as OpenAI does. The reasons are in the chat requests explanation and ADR 0029. The tutorial walks the same features with the Python SDK from scratch.

Configure base URL, key and attribution¶

Set the client's base URL to OAI_BASE_URL and its API key to the Horde API key. The client sends it as Authorization: Bearer <key>. With the Python SDK:

from openai import OpenAI

client = OpenAI(base_url="<OAI_BASE_URL>", api_key="<HORDE_API_KEY>")

A request without Authorization uses the anonymous key, and so does Bearer 0000000000. Anonymous requests are served after requests from users with kudos (ADR 0035). An Authorization header that is not Bearer <key> is 401 InvalidAuthorizationHeader.

The Horde attributes traffic by Client-Agent, which the front composes from your headers as <application>:<version>:<user agent> via horde-oai-front. The application is your X-Title header, else X-OpenRouter-Title, else the host of HTTP-Referer, else the host of Referer. The user agent is your User-Agent. Send X-Title with your application's name so Horde statistics count your traffic under it:

client = OpenAI(base_url="<OAI_BASE_URL>", api_key="<HORDE_API_KEY>", default_headers={"X-Title": "<your-app>"})

Verification: GET /v1/models with the key returns 200:

curl -s -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer $HORDE_API_KEY" "$OAI_BASE_URL/models"

prints 200. A malformed header prints 401.

Choose a model from the model card¶

GET /v1/models lists models served by at least one online chat worker. Models without an online worker are absent. Model ids are the Horde's model names and may contain slashes. An id containing : is 404 model_not_found, since the front does not serve model variants.

curl -s "$OAI_BASE_URL/models" | jq '.data[] | {id, context_length, max: .top_provider.max_completion_tokens, supported_parameters, x_horde}'

The fields are the maxima and union across the model's workers, and they are availability hints: a request at the largest max_tokens fits only the workers that declare it, and the workers online can change before the request is picked up. A request for a model without an online worker ends at once with 503 no_eligible_worker. A request that cannot be served by the online workers ends with 503 no_eligible_worker after the front's grace, or at once with 404 no_worker_supports_features when the model's online workers lack a declaration it needs (see Errors and retries).

Verification: your configured model appears in .data[].id, with x_horde.workers at least 1 and every optional parameter sent by your client listed in supported_parameters.

Accept any of several models¶

Send models, a list of up to 5 model ids, in place of model when any of them will do. The request is served by whichever listed model an online worker runs, and the response model reports the one that served it:

curl -s "$OAI_BASE_URL/chat/completions" -H "Authorization: Bearer $HORDE_API_KEY" -H "Content-Type: application/json" \
  -d '{"models": ["<model-a>", "<model-b>"], "messages": [{"role": "user", "content": "Hi"}]}' | jq .model

Every listed id has to exist on the Horde: an unknown one is 404 with param at its entry, such as models[1], and the request is not queued. More than 5 ids, or an empty one, is 400 UnsupportedChatParameter. Sending both model and models is 400 ChatRequestConflict. The kudos quote uses the first listed model. The generation is charged at the model that served it.

Verification: the printed model is one of the listed ids.

Request keys the front accepts¶

Completion limit¶

max_tokens and max_completion_tokens accept 16 to 16384 and must agree when both are sent. The default is 512 when neither is sent. See Deliberate deviations for the minimum of 16. The limit covers both reasoning and the answer (see Reasoning budget).

A value above the largest length declared by an online worker for the model is lowered to that length and reported in x_horde.context_adjustments.

Above the free output allowance, the API key must have enough kudos to cover the request. Otherwise the request is refused with 403 KudosUpfront. The allowance is 512 to 4096 by default, depending on the load on eligible workers.

Reasoning effort values¶

none, minimal, low, medium, high or xhigh, forwarded unchanged. A worker has to declare the reasoning_effort feature for any value, none included. When you leave it out, any chat worker may serve the request, and a worker that declares the feature receives medium, OpenAI's default, which koboldcpp applies as a thinking budget of about half of max_tokens. none disables thinking. How much reasoning the other values allow is the worker's mapping. Any other value is 400 BadRequest with param reasoning_effort.

Message roles¶

Message roles are system, developer, user, assistant and tool, with these keys:

A developer message is sent to the worker as system. Content is a string or an array of text parts. A user message may also include image_url parts.

Any other key on a message, and any unknown key inside a tool call, its function object or a content part, is dropped and not forwarded, as OpenAI ignores it. A name on a tool message is dropped. tool_calls on a message other than an assistant message is refused, as OpenAI refuses it.

Client defaults¶

Stock clients send some keys at their defaults. The front accepts these and drops them before the Horde sees them: empty or null tools, null tool_choice, null parallel_tool_calls, response_format: {"type": "text"}, logprobs: false, top_logprobs: 0, store: false, and service_tier "auto" or "default".

Verification: a request with only model, messages and the keys above returns 200 with a chat.completion body.

Keys the front refuses¶

Any other key, and any of the dropped keys at another value, returns 400 UnsupportedChatParameter, with param containing the key's path. Examples include logit_bias, logprobs: true, functions, function_call, audio, modalities, prediction, web_search_options and sampler extras such as rep_pen, top_k and min_p.

An unknown key inside stream_options or response_format is refused at its path, as OpenAI refuses it. tool_calls on a message other than an assistant message is refused at messages[<i>].tool_calls, and an unsupported role at messages[<i>].role. A refused content part is reported with its type, such as messages[0].content[0].type=input_audio.

A value OpenAI refuses by range is 400 with OpenAI's message and param: temperature outside 0 to 2, top_p outside 0 to 1, a penalty outside -2 to 2, n or a token limit below 1, and more than four stop sequences are UnsupportedChatParameter, such as Invalid 'temperature': decimal above maximum value. Expected a value <= 2, but got 2.5 instead. An empty messages is BadRequest with OpenAI's message. A token limit from 1 to 15 is refused with the Horde floor of 16.

Other 400 codes:

Each refusal happens before the request is queued, so it is not charged and is safe to correct and resend.

Verification:

curl -s "$OAI_BASE_URL/chat/completions" -H "Authorization: Bearer $HORDE_API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "<model>", "messages": [{"role": "user", "content": "Hi"}], "logit_bias": {"50256": -100}}' | jq .error

returns "code": "UnsupportedChatParameter" and "param": "logit_bias". The Python SDK raises BadRequestError with param == "logit_bias".

Sampling defaults and the horde object¶

The worker receives every sampling value, so the worker's defaults do not apply. An omitted key is sent with the neutral value: temperature 1.0, top_p 1.0, frequency_penalty 0, presence_penalty 0, and no repetition penalty, top_k or min_p filtering. max_tokens defaults to 512. Send temperature and top_p explicitly when the model's recommended settings differ.

The horde object includes the Horde options OpenAI does not have a key for:

Any other key in horde, including dry_run and operation_key, is 400 UnsupportedChatParameter. With the Python SDK, send the object as extra_body={"horde": {...}}. Other clients add it to the JSON body. Restrictions narrow the eligible workers, so a combination that is not supported by any online worker ends with no_eligible_worker, or a declaration or limit error when the cause is reported.

Context fitting options¶

truncate_oldest grows the context as grow does, then drops whole turns from the oldest forward until the messages fit, keeping every system and developer message and the latest user message with what follows it. grow raises the context to fit the messages, up to the largest chat context an online worker for the model declares, and refuses messages that do not fit at that bound. reject refuses messages that do not fit the context.

Kudos estimate cap¶

A positive integer cap on the request's kudos estimate. A request whose estimate at current pricing exceeds it is refused with 400 max_kudos_exceeded before it is queued, so it is not charged. metadata.kudos_estimate and metadata.max_kudos hold both values. The estimate assumes the full max_tokens is generated, while the charge is for the output estimate, so the check is conservative.

Check Horde options¶

Verification: x_horde.worker_id of the response is one of the only ids, and x_horde.context_adjustments lists a worker_max_length entry when max_tokens was lowered, {"from", "to", "reason": "grow"} when the context grew and a truncate_oldest entry when the oldest turns were dropped, null otherwise.

A long conversation can be sent whole. Under the default truncate_oldest, the Horde drops what does not fit:

completion = client.chat.completions.create(
    model="<MODEL>",
    messages=conversation,
)
for adjustment in completion.x_horde["context_adjustments"] or []:
    if adjustment["reason"] == "truncate_oldest":
        print(f"{adjustment['dropped_messages']} oldest messages were dropped")

Reasoning¶

A reasoning model, such as Qwen3, generates reasoning before the reply. The answer in content does not include that reasoning. When the worker submitted reasoning, the assistant message has it in reasoning_content beside content, as DeepSeek and OpenRouter return it. A stream sends it as delta.reasoning_content chunks before the content of the same worker post, so a client that reads the field shows progress during a long reasoning phase. The OpenAI SDK keeps the field in model_extra, streamed and whole. To hide the reasoning, as OpenAI's chat API does, send:

{"horde": {"include_reasoning": false}}

Either way, usage.completion_tokens_details.reasoning_tokens reports how many of the completion tokens were reasoning when the worker counted them. The option does not change the work, its cost or the operation the request is for.

Verification: a request to a reasoning model returns choices[0].message.reasoning_content. The same request with include_reasoning: false returns a message without a reasoning_content key.

Reasoning budget¶

The model generates its reasoning within max_tokens, together with the answer, and the reasoning is charged like the answer. Set max_tokens large enough for both.

When you leave out reasoning_effort, a worker that declares the feature receives medium. On koboldcpp that is a thinking budget of about half of max_tokens. When the reasoning reaches it, the thinking is closed and the answer follows. A worker that does not declare the feature does not receive a value, and the model reasons without a cap. An omitted max_tokens defaults to 512 in both cases.

A reply with finish_reason length, an empty content and reasoning in reasoning_content ran out of tokens while thinking. It cannot be continued, because the OpenAI API cannot resume reasoning. Reasoning sent back inside an assistant message becomes a previous turn, and reasoning_content on a request message is dropped. Send a fresh request with a larger max_tokens or a lower reasoning_effort. none disables thinking.

Images¶

A user message may include {"type": "image_url", "image_url": {"url": ..., "detail": ...}} parts. url is a data:image/...;base64, URL or a public http(s) URL. detail is optional (auto, low, high) and passed to the worker. Send images only to a model whose architecture.input_modalities includes image.

The Horde fetches a URL once at submit, with 2-second timeouts and at most 3 redirects, and stores each image for the worker. Each image adds 5 kudos to the request's cost. Image parts in any other role are refused with their path, such as messages[1].content[0].type=image_url.

Verification: the reply describes the image, and x_horde.kudos is higher than for the same request without it.

Tools and tool_choice¶

tools holds up to 128 functions, {"type": "function", "function": {"name", "description", "parameters", "strict"}}, each name 1 to 64 characters of A-Z, a-z, 0-9, _ and -. Other keys on an entry or its function are dropped. The parameters schema is forwarded unchanged.

tool_choice is none, auto, required, or {"type": "function", "function": {"name": "<name>"}} to force one function. Other keys in that object are dropped. parallel_tool_calls is a boolean forwarded to the worker with the tools. tool_choice and parallel_tool_calls without tools are 400 ChatRequestConflict, as under OpenAI. A parameters schema may nest at most 10 levels below its root. A model's card lists tools when a worker serves them.

A tool round:

1. Send the request with tools. A reply with calls has finish_reason tool_calls (or stop under a forced tool_choice), message.content null when there is no text, and message.tool_calls, each {"id", "type": "function", "function": {"name", "arguments"}} with arguments a JSON string.
2. Run each call.
3. Send the history again with the assistant message replayed as {"role": "assistant", "content": null, "tool_calls": [...]}, then one {"role": "tool", "tool_call_id": "<id>", "content": "<result>"} per call.

The tool history must follow OpenAI's rule: the tool messages come right after the assistant message and answer each of its call ids exactly once, in any order. A message arriving before every call is answered, a second answer to one id, or a tool message with no open calls is 400 UnsupportedChatParameter with param messages[<i>].role. An id that was not issued in the preceding assistant message has param messages[<i>].tool_call_id. The messages use OpenAI's format.

With the Python SDK, appending the returned message object to the history replays it correctly, including a message returned by get_final_completion() from a stream: the front drops the fields added by the SDK (refusal, parsed, tool_calls[].index, function.parsed_arguments and similar).

Verification: the first reply's message.tool_calls[0].function.name is the function asked for, and the follow-up reply has finish_reason stop with text and no tool_calls.

Conformance¶

By default ("conformance": "strict") a request goes only to workers that declare the guarantees it relies on: an enforced tool_choice for required or a forced function, schema adherence for a json_schema format with strict: true, and refusal of a prompt that does not fit (the worker does not trim it). A request requiring a guarantee that is not declared by any online worker serving the model ends at once with 404 no_worker_supports_features, and the message lists the guarantee.

To accept workers that may deviate, for example a backend that renders the same prompt for every tool_choice, send:

{"horde": {"conformance": "best_effort"}}

Under best_effort the reply may not follow those parts of the contract. Features and limits still apply in both modes: a worker that clamps the temperature when tools are present does not receive a tools request above its limit, and a tools request above the limit of every online worker ends at once with 404 no_worker_supports_features. tool_choice: "none" does not need a guarantee, since the worker does not receive tool definitions under it.

Verification: a request with a forced tool_choice that ended with 404 no_worker_supports_features under strict, on a model whose card does not list tool_choice, is picked up under best_effort.

Structured output¶

response_format takes {"type": "json_object"} or {"type": "json_schema", "json_schema": {"name", "schema", "description", "strict"}}, with schema required. A model's card lists response_format when one of its workers serves either type. It does not say which, so a json_schema request can lack an eligible worker on a model whose workers serve only json_object.

With "strict": true the schema follows OpenAI's structural rules: every object schema, nested ones included, sets "additionalProperties": false and lists all its properties in required. A schema breaking them is 400 UnsupportedChatParameter with param response_format and OpenAI's message. A tool with "strict": true in its function follows the same rules for its parameters, refused with param tools[i].function.parameters. A schema may nest at most 10 levels below its root, strict or not.

A request may offer tools and set a response_format together. The format applies only to the final text answer, as under OpenAI. Only a worker declaring tools_with_response_format serves such a request, under either conformance mode. When the feature is not declared by any of the model's online workers, the request ends at once with 404 no_worker_supports_features.

OpenAI refuses a json_object request whose messages do not mention JSON. The Horde serves it. To get OpenAI's refusal, send {"horde": {"openai_guards": true}}.

The compact JSON of tools, tool_choice and response_format together may be at most 256 KiB.

Verification: json.loads(choices[0].message.content) succeeds, and for json_schema the document has the schema's required keys.

Streaming¶

With "stream": true the response is server-sent events in this order:

1. Keepalive comment lines while the request waits, such as : horde queue_position=7 wait_time=42, at an interval the deployment sets (5 seconds by default).
2. A role chunk, delta: {"role": "assistant"}.
3. A reasoning chunk with the whole delta.reasoning_content, when the model reasoned, unless you sent horde.include_reasoning: false.
4. A content chunk with the whole delta.content, when the reply has text.
5. A tool-call chunk with delta.tool_calls, when the reply has calls: every call whole, each with its 0-based index, id, type and function, and no content in the same chunk.
6. A finish chunk, delta: {}, with finish_reason, native_finish_reason and x_horde.
7. A usage chunk with choices: [] and usage, when stream_options.include_usage is true.
8. data: [DONE].

A request without tools or a non-text response_format can receive content during generation if the worker posts progress. The stream starts with a role chunk. Each progress post, usually sent about once a second, is paced into smaller chunks. Reasoning precedes the content from the same post unless horde.include_reasoning is false. Before the finish chunk, the front sends any final text that was not covered by the relayed chunks.

For tools, structured output or a worker without progress posts, the worker generates the complete reply first. The text then arrives in one content chunk.

Every chunk uses the same id and system_fingerprint. The finish and usage chunks include the served model. An SDK accumulating tool-call deltas by index reconstructs the calls from the single tool-call chunk. In the Python SDK, client.chat.completions.stream(...) with get_final_completion() returns the assembled message.

A stream's headers go out after the front's first status read of the request. A refusal at that read (queue_unavailable, no_worker_supports_features, no_worker_supports_limit, or a Horde error such as 404) is an HTTP error with its status, body and Retry-After, as for a non-streamed request. Once the headers are out the status is 200 whatever the outcome, and a failure is one data: {"error": {...}} event, with the same envelope as an error response and no choices, followed by data: [DONE]. A hand-written parser checks each data: payload for a top-level error.

Live content can end in such an event after some text arrived: code ChatStreamDiverged when the worker stopped and the Horde would start the reply over with another, or when the final reply does not continue the text sent, and ChatRequestFaulted when the request faults. Discard the partial text. The request is not retried on the same stream. The OpenAI SDKs raise on the event.

Verification: a successful stream ends with a finish chunk containing x_horde.request_id, a usage chunk when stream_options.include_usage is true, and data: [DONE]. A buffered text reply without reasoning has four JSON chunks (role, content, finish, usage). Live progress can produce more content and reasoning chunks.

Errors and retries¶

Every error is {"error": {"message", "type", "param", "code", "metadata"}}, with param and code null when they do not apply and metadata with the Horde's request_id and return code where known (ADR 0034). A refusal from the Horde keeps its status and has its return code (see README_return_codes.md at the repository root) as code.

A Horde 429 keeps Retry-After and the X-RateLimit-* headers, and has code rate_limit_exceeded when the Horde does not send a return code. A native payload validation failure, which does not include a return code, is 400 BadRequest with the failing key as param, such as temperature or max_tokens.

The front's own codes:

A request that faulted after the Horde refused its workers' replies reports the last refusal:

{
  "error": {
    "message": "The AI Horde reports request <request-id> as faulted after refusing a worker's reply (ChatSubmitNonconformant): The request's 'response_format' is 'json_object', so the reply content has to be JSON.",
    "type": "api_error",
    "param": null,
    "code": "ChatRequestFaulted",
    "metadata": {"request_id": "<request-id>", "rc": "ChatSubmitNonconformant"}
  }
}

A request whose worker reported why it faulted the job reports that reason:

{
  "error": {
    "message": "The AI Horde reports request <request-id> as faulted after its worker reported a fault (ChatWorkerFault): The worker could not fit the prompt into its context.",
    "type": "api_error",
    "param": null,
    "code": "ChatRequestFaulted",
    "metadata": {"request_id": "<request-id>", "rc": "ChatWorkerFault"}
  }
}

Retry rules:

• An error without x-should-retry came before the Horde held the request. The request was not queued or charged, and a retry after Retry-After is safe.
• An error with x-should-retry: false disables automatic retries in OpenAI SDKs. Usually the Horde held the request, or may have. A retry is a new request and can create a second charged job, since the front does not deduplicate client requests. The front also sets this header on no_eligible_worker refusals before queueing, to prevent immediate retries against an unchanged worker pool.
• no_eligible_worker means the request as sent does not have a worker online: check the model card for the model's workers and supported_parameters, and change the request or wait for workers. The same request fails the same way until a worker comes online.
• queue_unavailable states the pool at the first status read: eligible worker threads, the estimated wait and the budget left. Retry after Retry-After, or send availability_mode: "wait" to queue for the budget.
• no_worker_supports_features lists what is missing. Drop or change the feature, lower the temperature, or send horde.conformance: "best_effort" when the missing values are guarantees. The same request fails the same way until a worker declaring them comes online.
• A stream always includes x-should-retry: false.

Verification: a request for a known model that is not served by any online worker returns 503 with "code": "no_eligible_worker" at once. The message states that the model is not served by any worker:

curl -s -i "$OAI_BASE_URL/chat/completions" -H "Authorization: Bearer $HORDE_API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "<an unserved model>", "messages": [{"role": "user", "content": "Hi"}]}' | grep -i -E "^HTTP|x-should-retry|retry-after|no_eligible"

Limits¶

max_tokens is the limit clients most often exceed. It is an upper bound in the OpenAI API. A client default above every online worker's declared length is lowered to the largest length declared by an online worker for the model. The same applies to the 512 sent when max_tokens is omitted. x_horde.context_adjustments then holds {"from": <requested>, "to": <declared>, "reason": "worker_max_length"}. A reply that needs more tokens ends with finish_reason length. When the workers change their declarations after the submit, the request can still end with 400 no_worker_supports_limit and param max_tokens.

Verification: a request above the card's max_completion_tokens is served, and its x_horde.context_adjustments holds a worker_max_length entry whose to is the card's value.

Deliberate deviations¶

The front differs from OpenAI on purpose in these points:

• max_tokens and max_completion_tokens below 16 are refused, where OpenAI accepts any value from 1. 16 is the minimum generation length of the Horde's native text API, and raising a smaller limit would generate more tokens than the client asked for. The message states the floor.
• param paths into messages are written messages[2].role, where OpenAI writes messages.[2].role. Paths into tools match OpenAI's form, such as tools[0].function.parameters.
• Refusals use the front's codes (UnsupportedChatParameter, ChatRequestConflict, BadRequest) in place of OpenAI's per-failure codes such as decimal_above_max_value. Message and param follow OpenAI's.
• system_fingerprint is a short hash of the serving worker id and model, null when the reply does not include a worker id. It changes when another worker serves the model. It does not change when a worker changes its backend under the same id.
• A stream includes each reply's tool calls whole in one chunk, where OpenAI streams the arguments in fragments.
• The first streamed chunk includes only role in its delta, with no content key.
• Live content arrives in batches of about a second, and only for requests without tools or structured output.
• A json_object request whose messages do not mention JSON is served, where OpenAI refuses it, unless horde.openai_guards is set.
• Reasoning text is returned as reasoning_content, where OpenAI does not return it. The OpenAI SDK keeps the field in model_extra, and horde.include_reasoning: false hides it.

Reversal¶

A client request does not change Horde state beyond the request itself. Point the client's base URL back at its previous provider to stop using the Horde. A request the client abandons by closing the connection is cancelled on the Horde. A streamed request cancelled while a worker generates is charged for the content the worker streamed before the cancel, priced as a reply of that length.

Related¶

• Chat front reference: request keys, message roles, response, stream and model card shapes.
• Chat requests reference: the native contract behind the front, including worker eligibility and pricing.
• Add chat completions to a web frontend: the same contract from a browser client.