Skip to content

Chat requests on the Horde

Topics: chat, requests, workers

  • A chat request sends an OpenAI messages array to a worker. The worker renders the conversation with the loaded model's chat template. Chat requests share the text queue, pricing and limits.
  • Workers opt in on every text pop. The Horde routes chat requests by the declared model, limits and required features.
  • A separate OpenAI-compatible front serves /v1/chat/completions through the native chat routes. Redis signals notify the front when a reply is available.
  • Unsupported parameters are refused with the key in the error message. Unknown keys inside messages, tool definitions and tool_choice are dropped, as OpenAI does.
  • Plain streamed requests receive text as the worker posts progress. Other requests receive the complete reply after generation. Each request produces one choice.

Every field is specified in the chat requests reference and the chat front reference. Each design choice below links to an ADR with the rationale and rejected alternatives.

Prompt strings and model templates

A native text request contains one prompt string, which the worker sends to the model as written. Instruction-tuned models answer well only when a conversation is rendered in the chat template they were trained on, with that model's role markers and turn separators. The client has to render the prompt with the correct template for the serving model and backend. An incorrect template can reduce reply quality without producing an error.

With a structured conversation, the worker renders the input using the loaded model's template. The request can also include tool definitions, tool results and images as separate fields. The shape chosen is OpenAI's chat-completions body, since it is what client libraries, SDKs and agent frameworks already send.

A prompt string translated from the conversation at submit was the alternative, and is how the pilot proxy works. It does not need a native change, but the horde would then own every model's chat template and its maintenance, and tools, images and reported usage would not have a path to the worker (ADR 0019).

How a chat request travels

A request passes through the front, the native API and a volunteer worker, with Postgres, Redis and object storage between them. The front does not hold a database connection. It returns results from the native status route or, while a stream is open, from the retained progress stream.

flowchart LR
    C[OpenAI client] -->|/v1/chat/completions| F[Chat front<br/>ASGI process]
    N[Native client] -->|/v2/generate/chat/*| H
    F -->|chat async, status, cancel<br/>service passkey| H[Native API<br/>Flask replicas]
    H --> P[(Postgres<br/>TextWaitingPrompt is_chat<br/>processing_gens)]
    H -->|images, offloaded messages| O[(Object storage)]
    H -->|completion signal,<br/>progress leases and streams| R[(Redis)]
    R -->|signal, XREAD| F
    W[Text worker<br/>chat declaration] -->|text pop, progress, submit| H
    W -->|signed URL fetch| O

The sequence below shows one request. The opt blocks apply only to streamed requests without tools or structured output. The loop runs only when the worker posts progress. Other requests use the final status.

sequenceDiagram
    participant C as Client
    participant F as Front
    participant H as Native API
    participant R as Redis
    participant W as Worker
    C->>F: POST /v1/chat/completions
    F->>F: per-source and global caps, validation
    F->>H: POST /v2/generate/chat/async (operation key)
    H->>H: moderation, required features, kudos, row
    H-->>F: request id
    F->>R: subscribe horde:chat:id
    F->>H: first status read
    Note over F: declaration gap, limits gap or fail_fast refusal<br/>is an HTTP error here, before any SSE headers
    W->>H: text pop with chat declaration
    H->>H: eligibility, narrow tools, sign URLs
    opt streamed plain request
        H->>R: write progress lease
    end
    H-->>W: chat job (messages or messages_url)
    opt streamed plain request
        loop about once a second
            W->>H: POST text/progress (seq, delta)
            H->>R: check lease, append to stream
            R-->>F: XREAD entry
            F-->>C: SSE content chunk
        end
    end
    W->>H: structured submit
    H->>H: verify reply, store generation, charge and reward
    H->>R: publish generation, done
    R-->>F: signal
    F->>H: status read
    F-->>C: completion, or the final chunks and [DONE]

The front submits over HTTP, subscribes to the request's completion signal, reads the status once when woken, and returns an OpenAI response. A native client can call the chat routes directly and poll the status route.

Design choices

Chat as a text request subtype

A chat request is a TextWaitingPrompt with a messages column and an is_chat discriminator (ADR 0019). Queue position, kudos, upfront kudos, concurrency limits, statistics, countermeasures and moderation apply because the code that applies them to text requests runs unchanged. The discriminator keeps the route families apart: a chat id is not found on the text routes, and the reverse.

A separate request class and table would have given chat a single representation, at the price of a second branch in every text behaviour. The cost of the chosen option is a second representation: the row's prompt column holds the conversation flattened into a string, read by the token estimate, kudos and moderation and not sent to a worker, and every validation rewrite of that string has to be mirrored onto the messages (Flattened prompt).

A separate front woken by Redis

An OpenAI call holds its connection open until the reply exists, and a horde request can wait minutes for a worker. Waiting inside the Flask application would pin a waitress thread per open call, threads that worker pops and submits share. The front is a separate ASGI process with one event loop, so a wait costs a coroutine (ADR 0023). It does not hold a database connection: the outcome lives in Postgres and the front reads it through the status route.

The native side publishes a small event on horde:chat:<request id> when a generation lands and when a request ends (Completion signal). Publishing is best effort and a message does not include text, so a lost message delays the response until the front's fallback poll and does not change it. Polling alone was the alternative. Its status load grows with the number of open waits.

A request on the front moves through the states below. After submission, any exit other than a completed reply or a fault cancels the Horde request. Abandoned requests therefore leave the queue (Errors the front returns):

stateDiagram-v2
    [*] --> Admitted: per-source and global caps
    [*] --> Refused: 503 FrontAtCapacity
    Admitted --> Validated: body read and checked
    Admitted --> Refused: 400, 401, 404, 408 or 413
    Validated --> Submitted: native submit<br/>lost responses recovered by operation key
    Validated --> Refused: native refusal or 502
    Submitted --> Waiting: first status read passes
    Submitted --> Cancelled: gap or fail_fast refusal<br/>HTTP error before any stream opens
    Waiting --> Waiting: signal, recheck or fallback poll<br/>relayed deltas on a stream
    Waiting --> Answered: done with a generation
    Waiting --> Cancelled: deadline 408, grace 503,<br/>no generation 502, client gone
    Waiting --> Faulted: 502 ChatRequestFaulted
    Answered --> [*]
    Refused --> [*]
    Cancelled --> [*]
    Faulted --> [*]

Once a request reaches Submitted, every error the front returns includes x-should-retry: false, since a retry would be a second charged request. A submit with a lost response is the one case the front repeats on its own: it sends the identical body with the same operation key, and the Horde returns the existing request (ADR 0026):

sequenceDiagram
    participant C as Client
    participant F as Front
    participant H as Native API
    C->>F: POST /v1/chat/completions
    F->>H: submit, operation key K
    H->>H: request queued, K claimed for the user
    H--xF: response lost (timeout or reset)
    F->>H: same body, same K, next root
    H-->>F: 200, the first request's id and kudos
    Note over H: nothing queued or charged again
    F-->>C: the completion
    C->>F: client retry after a lost response
    F->>H: submit, new operation key K2
    Note over H: a second, charged request

The front does not deduplicate client retries: a fresh client request gets a fresh key, so only the front's repeat of a submit is recovered.

The costs are operational. The front is one more process to deploy, drain and monitor, a request belongs to the process that accepted it, and a front that crashes leaves its open requests to run, be charged and go uncollected.

Capability declaration and routing by features

A worker states what it serves on every text pop: chat_jobs, the model_id, chat_limits, chat_features and image_transports (ADR 0022). A capability is a property of the loaded model and backend: whether a projector is loaded, whether the template renders tools, how much context the backend really has. A bridge version string cannot express those, which rules out the version table the horde uses for image features. The horde does not probe workers. Server-side probes would turn claims into evidence, at the price of worker time before any job.

Request requirements are computed once. required_chat_features derives the request's required features when it is created, counting a field at its documented default as absent, and stores them on the row. Eligibility compares the worker's declaration with that stored list and does not load a candidate's messages (Eligibility).

Two properties follow. A worker that predates chat does not send chat_jobs, so it does not receive a job without a prompt. A worker that declares chat keeps receiving text jobs under its text limits, so opting in does not take text work away. A chat-only worker sets prompt_jobs: false. The cost is trust: a misconfigured worker can declare a feature it does not honour.

Guarantees and the conformance mode

A backend can accept every field and still deviate from the contract on purpose: it may render the same prompt for every tool_choice, clamp the temperature when tools are present, or trim a prompt that does not fit. Each is a reasonable local choice, and over the horde the requester sees only the reply. A capability says the worker acts on an input. A guarantee says the reply follows that part of the contract exactly (ADR 0036). Both share the chat_features list, and a numeric limit such as tools_max_temperature joins chat_limits.

A request is strict unless its horde object selects best_effort, because the clients that cannot opt in are the plain OpenAI clients using OpenAI behaviour. Under strict, required_chat_features adds the required guarantees to the stored list, so eligibility compares them like features. Under best_effort the request does not require a guarantee. Limits apply in both modes, since a clamped value changes the request, and a mode governs only how faithfully the reply follows the request.

The contract is enforced at routing, dispatch and submission. Routing sends a request only to a worker whose declaration covers it. Dispatch narrows what a job includes where the contract allows: under tool_choice: none the worker does not receive tool definitions, so none does not need a guarantee, and a tool_choice that specifies a function reaches the worker as that one function under required. The stored request keeps the client's values, and the submit checks the reply against them.

flowchart TD
    A[Request created<br/>chat_required_features stored] --> B{Route: does an online<br/>worker's declaration cover<br/>features, guarantees, limits?}
    B -->|no worker declares it| G[404 declaration gap<br/>or 400 limits gap]
    B -->|model does not have an online worker| Q[503 at submission<br/>request is not queued]
    B -->|yes| D[Narrow: build the job<br/>tool_choice none sends no tools<br/>a specified function goes as required]
    D --> S{Verify the submit against<br/>the stored request}
    S -->|conforms| OK[Generation stored,<br/>charged and rewarded]
    S -->|refused| RF[Job faulted on the worker<br/>request goes back to the queue]
    RF -->|another attempt| B
    RF -->|repeated faults| X[Request faulted<br/>the front returns 502]

Reply verification rejects detectable contract violations even when the worker declares the required guarantees. It can verify only properties visible in the submitted payload.

Under strict, verification refuses a tool call under none, a missing call or a call to another function under a forced choice, and content that is not the requested JSON or does not match a strict schema. Under strict with parallel_tool_calls: false, only the first call is retained. Because this reduction applies to every worker's reply, routing does not require single_tool_call_enforced.

Under either mode a call to a function outside the client's tools, or two calls sharing an id, is refused, because a client cannot dispatch or respond to such a reply. A refused submit faults the job on the worker and counts towards faulting the request, so the check rejects only detectable violations of the response contract. A trimmed prompt or a hidden tool-selection pass does not show in the reply and is not verified.

Prompt fit on the worker

The guarantee prompt_overflow_refused promises that a worker faults a conversation that does not fit instead of trimming it. Only the worker can keep this promise. The horde does not run a tokenizer and sizes context at 3 characters per token (CHARS_PER_TOKEN_ESTIMATE in horde/classes/kobold/request_fit.py), and a trimmed prompt does not show in the reply. The worker's count has to be at least the count used by generation. A lower count lets a prompt pass the fit check and still overflow, and the backend then trims it.

An exact counter renders the conversation with the backend's template and tokenizes it. Its count matches the count used by generation when both render with the same template and the same BOS token, and falls short when they differ. The minimal worker in tools/chat_worker_example/ therefore compares its counter with the backend's usage.prompt_tokens for one probe conversation at startup. It declares the guarantee only when the counter is not short.

A backend without a counter route can still back the guarantee with an upper bound. A byte-level BPE tokenizer emits at least one UTF-8 byte per token, so the byte length of the messages' JSON bounds the token count of the message text. The JSON keys around each message stand in for its role markup.

The template's fixed part is the BOS token, any system prompt injected by the template, and the generation prompt. The worker measures it once from the probe's usage.prompt_tokens and adds it to the bound. The probe does not include tools, so it does not measure a tool preamble, and the bound backs the guarantee for tool requests only when the operator adds the preamble's size with --template-overhead.

The bound costs capacity. It counts one token per byte, which for ASCII text is about three times the horde's estimate of 3 characters per token, and more for text outside ASCII. The horde sends a worker conversations that fit by its own estimate, so a worker on the byte bound faults long conversations with prompt_overflow.

A worker that aborts more than 20 jobs in an hour is put into maintenance. The alternative is to trim the prompt, which keeps the job and breaks the guarantee, so a worker that trims does not declare it and serves only best_effort requests. The counters and the check script are in Write a minimal chat worker.

Strict handling of unsupported parameters

A key or value that is not implemented is refused with 400 UnsupportedChatParameter, and the message states the key (ADR 0029). A field at its documented default counts as omitted, so stock clients that send defaults unprompted still work, and the front drops a short, fixed list of default no-ops. Capability fields (tools, response_format, image parts) are not dropped, since they change the result.

Worker availability is checked separately. A request for a model that is not served by any online chat worker is refused at once with 503 no_eligible_worker. A max_tokens above every serving worker's declared length is lowered to the largest declared length. Other availability failures depend on the cause. A missing feature or guarantee produces an immediate 404, and an exceeded size limit produces 400. When the status does not report such a declaration gap, the front returns 503 if is_possible remains false after the grace period. See No eligible worker for admission refusals and status checks.

Where OpenAI refuses a request shape, the horde refuses it too, with OpenAI's message and param: a tool_choice or parallel_tool_calls without tools, a tool history whose calls are not answered right after the assistant message that made them, and a strict schema that leaves an object open or a property optional. A client written against OpenAI then fails the same way on both services. The native route and the front use the same rules from horde/chat_contract.py.

Some OpenAI refusals apply only to OpenAI's backend, such as refusing a json_object request without a mention of JSON in the messages. The Horde accepts these requests. To enable those refusals, set horde.openai_guards.

Messages and their nested objects, the tools definitions and the tool_choice object are the exception: an unknown key inside them is dropped and not forwarded. Every other object, and the top level, refuses unknown keys. The split mirrors OpenAI. OpenAI's API ignores unknown keys inside those objects and refuses them at the top level and inside response_format and stream_options. It also refuses tool_calls on a message other than an assistant message, which the front refuses too. The OpenAI Python SDK also replays its accumulated streamed reply with SDK-specific fields, so refusing them would break the SDK's tool loop.

Dropping unsupported fields by default, as OpenRouter does, was the alternative (ADR 0030). It produces fewer errors and hides a changed answer in a field that most clients do not read.

Clients must stop sending unsupported fields. A value that is a no-op at its default can be added to the front's default table, by key, as a deliberate change.

Pricing by the text output estimate

A chat generation is charged and rewarded through the text formula, on the characters of the reply (content, reasoning, and each tool call's name and arguments) divided by four, floored at 20 tokens and capped at the requested max_length (ADR 0027, Pricing). A short reply to a large max_tokens is charged for the short reply. The basis is computed in one function and stored on every generation as charge_basis: output_estimate.

A streamed generation cancelled by the client is settled on the same basis on the content posted by the worker before the cancel, which the Horde measured, so a volunteer worker is paid for what it turned in.

Pricing on the token counts workers report was the alternative (ADR 0028). It is fairer per reply and trusts volunteer reports that are not audited, which would give workers a reason to over-report. The counts are stored and shown on every generation and do not enter the charge, so the alternative can be evaluated on real data. The cost of the estimate is accuracy: characters approximate tokens poorly for code and non-English text, and the floor charges a near-empty reply as 20 tokens.

A client can bound what a request may cost with horde.max_kudos. The native route compares it with the dry-run quote, which assumes the full max_length, and refuses before anything is queued, held or uploaded. The charge follows the reply's output estimate, which is usually smaller, so the check is conservative and a refused request might have cost less (Kudos cap).

Reasoning

A reasoning model generates reasoning before the answer. The worker submits it in reasoning_content, separate from content, so a client that does not expect reasoning receives a clean answer, and a block cut off by max_tokens cannot become the reply. The front returns it as reasoning_content, so a client that reads that field shows progress while the model generates reasoning. A request that sets horde.include_reasoning to false does not receive it. The native status shows it either way.

Reasoning is generated output, and OpenAI bills it as completion tokens, so its characters are counted with the content and tool calls, capped at the requested max_length. A reply whose whole budget went to reasoning is charged and rewarded for that budget, and the same count is the worker's speed sample. The worker's reasoning_tokens count is shown and does not change the charge, as with every other reported count (Reasoning).

Reasoning and the answer share max_tokens, and the OpenAI API cannot resume reasoning. A reply cut off while thinking cannot be continued. In a follow-up request, the earlier reasoning is a previous turn. The thinking and the answer therefore have to fit in one request.

koboldcpp gives an absent reasoning_effort an unlimited thinking budget, so a small max_tokens could be spent on reasoning alone. For that reason a worker that declares the feature receives medium, OpenAI's default, when the client omits the value (ADR 0033). On koboldcpp that caps the thinking at about half of max_tokens. The default lowers the chance of an empty reply and does not guarantee an answer.

Admission against the online pool

Workers are volunteers, and the models online change minute to minute. By default a request waits for a worker for the front's whole budget. A client that prefers an early answer sets horde.availability_mode: "fail_fast", and the front checks the first status read: zero eligible worker threads, or a wait_time over the budget left, refuses the request with 503 queue_unavailable and a Retry-After, and cancels it. The refusal describes the pool at that moment and leaves the retry to the client. The same request may be served once workers come online.

The estimate counts the queued work ahead that the request's eligible workers could take and divides it by their throughput (Queue position and wait time), so it is an approximation, and a request may still start sooner or wait longer.

A refusal at the first read arrives before a stream opens, as a plain HTTP error, so a streaming client handles it as it would any other refused request. The decision waits up to about 2 seconds for a status that includes queue information, since a request just submitted is absent from the horde's cached queue positions until they refresh (No eligible worker).

Images by the declared transport

The horde fetches, checks and normalises each image at submit, stores it as JPEG, and keeps only a reference in the row (Images). At pop, after the commits that release the queue's row lock, it sends the images using the worker's declared transport (ADR 0032). A worker that declares url receives a signed download URL per image and fetches it. A worker that declares only base64 receives the bytes inline.

Inlining base64 read from object storage under the row lock was the alternative: a network read inside the lock, and megabytes in a JSON response. The base64 transport reads the store after the lock is released. Storing base64 in the row would load image bytes on every queue scan. With URLs, the worker must be able to reach object storage, and a failed fetch is a faulted job.

A deployment can also offload a large request whole: the messages, images inlined, are stored as one JSON object at submit, and a worker that declares messages_url receives one short-lived signed URL in place of messages (Messages offload). The worker then makes one self-contained fetch and does not manage partial downloads, and large bodies stay off the pop response. The transport is chosen per generation, from what the request stored and what the popping worker declared:

flowchart TD
    P[Worker pops a chat job] --> O{Request offloaded and<br/>worker declares messages_url?}
    O -->|yes| MU[messages_url<br/>one signed URL, 120 s]
    O -->|no| I{Request has images?}
    I -->|no| IN[messages inline]
    I -->|yes| T{Worker declares url?}
    T -->|yes| U[messages inline<br/>one signed URL per image]
    T -->|no, base64 only| B[messages inline<br/>image bytes inlined]

Sampling defaults and the developer role

Every chat payload includes every sampling field, with OpenAI's default where the client left one out (ADR 0033). An omission then means the same thing on every worker, which a worker's local defaults (koboldcpp's include a repetition penalty) would not give. The cost is that an operator's tuned defaults do not apply to chat jobs.

A developer message is stored and sent as system, so templates that predate the role can render it. The rewrite loses the role's precedence over system, and the horde does not advertise a developer-role feature while it holds.

Errors in the OpenAI envelope

The front returns every error as {"error": {"message", "type", "param", "code", "metadata"}}, with the horde's return code as code, and uses OpenRouter's status vocabulary where OpenAI does not define a status, such as 502 for a worker or generation failure and 503 for no eligible worker (ADR 0034). An envelope mapped onto only 400 and 500 would have made capacity failures indistinguishable from server faults.

After a stream's headers are sent the status is fixed, so a later error arrives as one SSE event with the envelope without a choice, and the front does not make up a finish reason. The full table is in Errors the front returns.

Anonymous access

A request without Authorization uses the anonymous key, as the native API allows (ADR 0035). Anonymous access allows use without registration. Requiring a key would remove that access. Abuse is bounded by the existing anonymous limits, the lowest kudos priority, and per-source caps on the front, which aggregate users behind one address.

Streaming

A streamed request without tools whose response_format is absent or text gets its content as the worker posts progress, about one post a second. The front spreads each post's text evenly over the estimated interval until the next post, so the client sees a steady flow of text. Without pacing, each post would arrive as one lump. Pacing adds up to about one post interval of latency. More frequent worker posts would cost the Horde more progress writes.

The batches are kept in a retained Redis stream per request. Redis avoids progress writes to the primary database, but progress can be lost on failover (ADR 0024). Postgres progress rows were the alternative, durable and paid for in primary database writes on the connection pool pops and submits share (ADR 0025).

A lease in Redis, written at pop, authenticates a generation's posts, so the progress route does not read Postgres. The lease's life is the streamed generation's life as the worker sees it (Streaming progress):

stateDiagram-v2
    [*] --> Leased: pop of a streamed job
    Leased --> Leased: post accepted (200)<br/>or resynced after 409
    Leased --> Submitted: worker submits the whole reply
    Leased --> Ended: generation faults
    Leased --> Cancelled: client or front cancels
    Leased --> PostsStopped: 413 cap reached
    PostsStopped --> Submitted: worker still submits
    Submitted --> [*]: lease deleted, stream left to expire
    Ended --> [*]: ended marker, lease deleted
    Cancelled --> [*]: settled on the posted text<br/>next acknowledgement includes cancelled true<br/>worker stops and does not submit

A cancel settles the generation on the content posted by the worker before it, so the worker is paid for that text and the requester charged for it. The settlement does not count a dropped job, so a worker that stops on the flag is not penalised for a client leaving. The front relays entries as they arrive and reconciles them with the final reply: a reply equal to or extending the relayed text completes the stream, and one that diverges ends it with an error event.

A request with tools or a non-text response_format waits for the whole reply as a non-streamed request does, then sends it as server-sent events: a role chunk, one content chunk, a tool-call chunk when there are calls, a finish chunk, optional usage, and [DONE] (Response shape).

Service selection

Horde behaviour without an OpenAI key (worker restrictions, trusted or validated workers, context sizing) travels in a horde object in the request body, sent through SDK extra body fields (ADR 0020). A request may list acceptable models in models in place of model. Any listed model may serve the request, and the response reports which model generated the reply. The list states what the client accepts, so reordering the list does not change the operation fingerprint. An unknown entry rejects the whole request.

Under context_fit: truncate_oldest a history too long for any served context loses its oldest whole turns and keeps every system and developer message and the latest user turn. Moderation screens the full submitted messages before any message is dropped. For the request's lifetime its row stores the kept messages, which the worker receives, and, as on every request, the full flattened submission in submitted_prompt.

The model id is the exact native model name. The front reserves the colon, so a model id that contains one is refused as unknown with 404 model_not_found.

Base-URL profiles were the alternative (ADR 0021): one configuration per application, at the price of a resolver and revision store, invisible to model pickers, and lost by clients that hardcode /v1.

What a client can rely on

  • Identity. A completion's id is chatcmpl- plus the Horde request id, and x_horde.request_id includes the id alone. Use it for support or a native status read.
  • Charges. A request costs what the same reply would cost as a text request, reported in x_horde.kudos with its charge_basis. Reported token counts do not change it. A lost submit inside the front is recovered by operation key without a second charge (ADR 0026). A retry by the client is a new request and can create another charge.
  • Request handling. Required features are checked against worker declarations. Unsupported parameters are refused. Under strict, a reply's tool calls follow tool_choice and parallel_tool_calls, and its content follows response_format, whatever the worker declared.
  • Stream order. A stream's chunks arrive in the order above, and an error after the headers arrives as one error event followed by [DONE].
  • Retry control. Once the Horde may hold a request, errors include x-should-retry: false, so SDKs do not retry automatically. A later client retry can create a second charged job.
  • Failure behaviour. A worker that cannot serve a job faults it, and the job returns to the queue until the request's fault limit is exceeded. A faulted request gets 502.

Availability refusals depend on the cause:

Cause Front response
Model without an online chat worker Immediate 503 no_eligible_worker
Missing required declaration Immediate 404 no_worker_supports_features
Only a context or output size limit exceeded Immediate 400 no_worker_supports_limit
Other lack of eligible workers after the grace 503 no_eligible_worker, with cancellation
Insufficient eligible threads or excessive estimated wait under fail_fast 503 queue_unavailable at the first status check, with cancellation

See availability errors for precedence and request timing.

What a client cannot rely on

  • Delta streaming. It covers plain text requests only. A request with tools or structured output streams the reply whole after generation, and a relayed stream is best effort, since a lost stream still ends with the full reply.
  • Multiple choices. n other than 1 is refused.
  • Worker selection by quantisation or pinned model. horde.quantizations and horde.pinned_model are unknown keys and return 400 UnsupportedChatParameter.
  • Identical answers across workers. The defaults are the same everywhere. Backends, quantisations and templates are not.
  • Verified capabilities. A feature is honoured by a worker that declared it, and the declaration is the worker's claim. Only what the reply itself shows is checked. A trimmed prompt or a clamped sampler is not.
  • Token counts. Native status returns the worker's reported counts, or null for missing counts. The front estimates missing counts and marks them with x_horde.estimated_usage.