Serve chat jobs from a text worker¶

Topics: chat, workers

To serve chat jobs, add a chat declaration to text pop, render each conversation with the loaded model's chat template, then submit a structured assistant message. The worker can continue serving text jobs through the same loop.

Before starting, have:

• A text worker using POST /api/v2/generate/text/pop and POST /api/v2/generate/text/submit with a registered API key.
• A backend that renders the loaded model's chat template and counts the rendered tokens.
• A Horde at <HORDE_URL> with the /api/v2/generate/chat routes enabled.

After integration, the model appears in GET <HORDE_URL>/api/v2/status/models?type=text&chat=true with the declared capabilities and limits. The worker completes supported chat jobs and faults jobs that it cannot serve.

Use the chat requests reference for field definitions. Python workers can use horde_sdk.worker.chat_declaration: declare_from_backend, ChatJobPayload, build_chat_reply_submit and build_chat_fault_submit.

For an end-to-end run with koboldcpp, follow Serve your first chat job. For the design rationale, see Chat requests on the Horde. To send client requests, see Send chat completions.

One chat job in full¶

A worker adds its chat declaration to every pop, beside the usual text fields. The example in step 1 declares two optional features, the limits and the model, without image support. It serves best_effort requests. To serve default strict requests, also implement and declare prompt_overflow_refused, as described under Features and guarantees.

A chat job arrives with chat: true and the conversation in payload.messages:

{
  "payload": {
    "messages": [
      {"role": "system", "content": "You are a terse assistant."},
      {"role": "user", "content": [{"type": "text", "text": "Name one colour."}]}
    ],
    "temperature": 0.5,
    "top_p": 1.0,
    "frequency_penalty": 0.0,
    "presence_penalty": 0.0,
    "rep_pen": 1.0,
    "top_k": 0,
    "min_p": 0.0,
    "stop_sequence": ["\nuser:"],
    "max_length": 512,
    "max_context_length": 2048,
    "n": 1
  },
  "id": "<job-id>",
  "ids": ["<job-id>"],
  "messages": [],
  "ttl": 150,
  "extra_source_images": null,
  "skipped": {},
  "softprompt": null,
  "model": "<model-name>",
  "chat": true
}

The worker renders payload.messages with its chat template, generates, and submits an assistant message:

{
  "id": "<job-id>",
  "message": {"role": "assistant", "content": "Blue."},
  "finish_reason": "stop",
  "model_id": "<model-name>",
  "state": "ok",
  "prompt_tokens": 21,
  "completion_tokens": 3,
  "native_finish_reason": "eos_token"
}

The submit returns the worker's earned kudos:

{"reward": "<kudos>"}

Every other body a worker meets (tools, images, offloaded messages, faults, progress posts, refusals and worker details) is in Chat wire examples, checked against the running code. Write a minimal chat worker is a complete worker for an OpenAI-compatible backend in one file.

The pop loop¶

The steps below build one pop loop. A pass through it, with the step that covers each part:

flowchart TD
    P[Pop with the chat declaration<br/>step 1] --> J{Job returned?}
    J -->|no| P
    J -->|yes| K{chat: true?<br/>step 2}
    K -->|no| T[Serve the text job<br/>step 8]
    K -->|yes| M{messages_url?}
    M -->|yes| FM[Fetch the messages<br/>within 120 s, step 5]
    M -->|no| IM{Image URLs?}
    IM -->|yes| FI[Fetch each image<br/>step 5]
    IM -->|no| R
    FM --> R[Render and fit the conversation<br/>steps 3 and 4]
    FI --> R
    R -->|does not fit, cannot render| F[Fault with a reason<br/>step 7]
    R --> G[Generate]
    G --> S{stream_progress: true?}
    S -->|yes| PP[Post progress about once a second<br/>step 6]
    PP -->|404 ChatJobAborted or cancelled true| X[Stop generating<br/>and do not submit]
    PP -->|404 ChatProgressNotStreaming,<br/>413, repeated 503| G2[Stop posting, keep generating]
    PP -->|generation finished| SB
    G2 --> SB
    S -->|no| SB[Submit the structured reply<br/>step 6]
    G -->|backend fails| F
    SB -->|ChatSubmitInvalid,<br/>ChatSubmitNonconformant,<br/>ChatModelMismatch| F
    SB -->|AbortedGen: job ended| P
    SB --> P
    F --> P
    T --> P
    X --> P

A failed fetch is a fault too, sent without fault_reason.

1. Declare chat on pop¶

Add five fields to every pop body, beside the usual name, models, max_length and max_context_length (Chat declaration on pop):

{
  "chat_jobs": true,
  "chat_features": ["seed", "presence_penalty"],
  "chat_limits": {"max_context_length": 4096, "max_length": 512},
  "model_id": "<model-name>",
  "image_transports": []
}

Chat opt-in¶

chat_jobs: true opts in. The Horde stores the declaration from every pop, so a pop without chat_jobs withdraws it, and the worker does not receive a chat job from that pop.

Features and guarantees¶

chat_features lists the backend's supported optional features. Declare a feature only when the backend implements it. The Horde does not test declarations (ADR 0022). A request that needs a feature waits for a worker that declares it, so an undeclared feature costs only the requests that need it. Unknown values are ignored.

The same list includes guarantees: a guarantee states that the backend follows one part of the OpenAI contract exactly, where it could otherwise deviate on purpose (ADR 0036). A request is strict unless the client opts into best_effort, and a strict request that needs a guarantee waits for a worker that declares it. Declare a guarantee only when the backend implements the behaviour. Operator configuration can withhold a guarantee. It cannot establish a guarantee that the backend does not implement.

Every strict request needs prompt_overflow_refused, so a worker that does not declare it serves only best_effort requests.

Chat limits¶

chat_limits applies to chat jobs. The pop's max_context_length and max_length still apply to text jobs. Set chat_limits.max_context_length to the backend's actual context size for one conversation, rendered template and reply together, and chat_limits.max_length to the most tokens it generates for one reply. A request that exceeds either limit is not sent to the worker.

Set the optional chat_limits.tools_max_temperature when the backend lowers the temperature while tools are present: a request that forwards tools at a higher temperature is then not sent to the worker, under either conformance mode, since a clamped value changes the request. Leave it out when the backend applies every temperature as sent.

Model id¶

model_id is the worker's chat model, and it has to be one of models. The native route treats it as an opaque string. The OpenAI-compatible front reserves : and returns 404 for any model id that contains one, so a model name with a colon can be served natively and cannot be reached through the front (ADR 0020). Choose a name without a colon.

Image transports¶

image_transports lists how the worker accepts images: url, base64 or both. Leave it out, or send it empty, when the worker does not accept images (see step 5). Add messages_url to receive a large request's messages as one URL (see Fetch offloaded messages). It can be declared without vision.

Text job opt-in¶

prompt_jobs is optional and defaults to true. Send "prompt_jobs": false with chat_jobs: true to serve chat jobs only: the worker then does not receive text jobs, does not make a text request possible, and may leave out max_length and max_context_length.

Declaration checks and withdrawal¶

Verification: the pop returns 200. A pop that sets chat_jobs without chat_limits or model_id, with a negative chat_limits.tools_max_temperature, or with a model_id outside models, or that sets prompt_jobs: false without chat_jobs, returns 400 ChatDeclarationInvalid and its message says which field is wrong. Within about ten seconds (the model cache refresh) the model list shows the declaration:

curl -s "<HORDE_URL>/api/v2/status/models?type=text&chat=true" | jq '.[] | select(.name == "<model-name>") | .chat'

The chat object reports workers, the largest max_context_length and max_length among the online chat workers serving the model, and the sorted union of their features (Chat model list). Covered by tests/integration/test_statistics.py, test_text_models_carry_the_chat_declarations_only_when_asked_for, and tests/integration/test_chat.py, test_an_invalid_chat_declaration_is_refused_on_pop.

The worker's details show the horde's stored declaration from the last pop:

curl -s "<HORDE_URL>/api/v2/workers/name/<worker-name>" | jq '{accepts_chat_jobs, accepts_prompt_jobs, chat}'

accepts_chat_jobs and accepts_prompt_jobs are the pop's chat_jobs and prompt_jobs. chat holds model_id, features, limits and image_transports as the pop sent them, without the feature and transport names unknown to the horde, and is absent while the worker does not accept chat jobs (Worker details). Worker details are cached for 30 seconds, so a changed declaration can take that long to show. Covered by test_worker_details_read_back_the_chat_declaration_of_the_last_pop.

Reversal: pop without chat_jobs. The worker stops receiving chat jobs from that pop on. A chat job already assigned keeps the model and limits recorded at assignment, so the worker still submits it under the old model_id (test_the_assigned_model_and_limits_survive_a_later_change_of_declaration).

2. Recognise a chat job¶

A chat job's pop response includes chat: true at the top level. A text job's response does not include a chat key and is unchanged, so branch on that key (Chat job pop response).

A chat job's payload differs from a text payload in these ways:

• messages holds the conversation. Roles are system, user, assistant and tool. A client's developer message arrives as system (ADR 0033). An assistant message may include tool_calls, a tool message includes tool_call_id, and a user message may include image_url parts.
• There is no prompt. The horde stores a flattened copy of the conversation for its estimates and does not send it to workers (Flattened prompt).
• temperature, top_p, frequency_penalty, presence_penalty, rep_pen, top_k and min_p are always present, with OpenAI's defaults where the client left them out. Apply them as sent and ignore the backend's local defaults, so one request means the same thing on every worker.
• stop_sequence and seed are present only when the client sent them.
• reasoning_effort is present when you declare the reasoning_effort feature. It is the client's value unchanged, or medium, OpenAI's default, when the client did not send one. A worker that does not declare the feature does not receive the key. The client's max_length covers the reasoning and the answer. When the limit cuts off the thinking, submit the reasoning as reasoning_content and leave content empty.
• max_length and max_context_length are validated budgets within the worker's chat_limits. Generate at most max_length tokens and render within max_context_length.
• tools, tool_choice, parallel_tool_calls and response_format are present only when the request applies them (see step 4).
• stream_progress: true is present only when the client streams the reply (see Streaming progress).

Verification: log the pop's chat key and the payload keys. A chat payload has messages or messages_url, and does not include prompt. Covered by test_a_chat_request_completes_with_its_structured_reply_and_signals, which checks the whole payload, and test_a_developer_message_reaches_the_worker_as_a_system_message.

3. Render and fit the conversation¶

Render messages with the chat template of the loaded model, tokenise the result, and check that the prompt tokens plus max_length fit max_context_length. When they do not, fault the job (see step 7). Never drop older messages to make it fit: the client sent that conversation and would receive a reply to a different one without knowing.

The horde sizes the context from a character estimate of the flattened conversation, so its sizing is approximate. The exact token count depends on the worker's template and tokenizer, so the worker must check fit. See Chat job pop response.

The fit check must not run side effects. A backend's request transform may do more than render. Koboldcpp's chat transform, when it formats without a Jinja template, runs a short in-process generation to choose a tool when tool_choice is auto or required. Running that transform to estimate the prompt could generate once more for every chat job with tools, outside the job's budget.

koboldcpp's embedded worker therefore does not render for the check. It counts the tokens of the JSON of messages, tools and response_format, adds 32 tokens, 16 for each message and 512 for each image, and faults the job with prompt_overflow when that count plus max_length exceeds its context.

Verification: send a conversation longer than chat_limits.max_context_length allows. The worker submits {"id": "<job-id>", "state": "faulted", "model_id": "<model-name>", "fault_reason": "prompt_overflow"} and does not generate.

4. Serve tools and structured output¶

tools, tool_choice, parallel_tool_calls and response_format arrive as the client sent them, and only when they change the answer (Tools), with two narrowings: under tool_choice: none the payload does not include a tool key, and under a tool_choice that specifies a function only that function arrives, with tool_choice: "required". Pass them to the backend unchanged.

• Declare tools only when the template renders tool definitions, assistant tool calls and tool results, and the worker returns the model's calls as tool_calls. A request needs tools when it offers tools under a tool_choice other than none, and also when its history alone contains a tool call or a tool message, since the template has to render those turns.
• Declare json_object only when the backend constrains the reply to one JSON object. A grammar for JSON in general is not enough: koboldcpp's shared JSON grammar is rooted at an array for tool calls, so for json_object it roots the grammar at an object.
• Declare json_schema only when the backend constrains the reply to the schema in response_format.json_schema.schema.
• Declare tools_with_response_format only when the backend serves a request with both tool definitions and a response_format: the model may still call a tool, and the format constrains only the final text answer, as under OpenAI. A request that forwards tools beside a response format needs it under either conformance mode. A backend whose output grammar for the format forbids the tool-call syntax, or whose tool grammar replaces the format, does not declare it.

Submit tool calls with the id, type and function the backend minted, and content null when the model only called tools (see step 6).

Verification: a request with tools waits while only workers without tools are online, its status reports is_possible: false, and the first pop declaring tools receives it with the tools in the payload. Covered by test_a_tools_request_waits_for_a_worker_declaring_tools_and_is_not_possible_without_one, test_a_tool_history_alone_needs_a_worker_declaring_tools and test_a_json_schema_request_goes_only_to_a_worker_declaring_json_schema.

5. Serve images¶

Declare vision only while the loaded model reads images, and declare at least one image transport with it. A request with images goes only to a worker that declares both (Images).

At pop, each image_url part's url is replaced according to the declared transports:

• With url declared, it is a signed object storage URL, valid for the job's TTL and at least 600 seconds. Fetch it before generating, bound the bytes read, and take the media type from the bytes. The horde stores every image as JPEG, and an object store may return a generic Content-Type, so the header is not a reliable media type.
• With base64 alone, it is a data:image/jpeg;base64, URL with the bytes inline.

The horde prefers url when a worker declares both (ADR 0032). A fetch that fails is a faulted job (see step 7).

Verification: a request with an image part reaches the worker as a signed http(s) URL when it declares url, and does not reach a worker without vision and a transport. Covered by tests/integration/test_chat_images.py, test_a_url_worker_receives_a_signed_url_valid_for_at_least_the_job_ttl, test_a_base64_worker_receives_a_jpeg_data_url_built_after_the_queue_lock_is_released and test_a_worker_without_vision_or_a_transport_never_pops_an_image_request.

Fetch offloaded messages¶

A horde with the messages offload enabled stores a large request's whole messages array as one object at submit (Messages offload). Declare messages_url in image_transports, beside url or base64 when the worker declares vision, to receive such a request as one URL:

• The payload includes messages_url in place of messages. Fetch it right after the pop: the URL is valid for 120 seconds.
• The body is the JSON messages array, each image part's url a data:image/jpeg;base64, URL, so the worker does not need a further fetch. It is served either plain or with Content-Encoding: gzip. Decode by the response's Content-Encoding header. Python's urllib does not decode gzip on its own.
• Cap the decoded body at 36 MiB, the chat request body cap, which bounds the body at every request limit.
• A request that was not offloaded still arrives with messages inline, and a worker that does not declare messages_url receives offloaded requests inline too.

A fetch that fails is a faulted job (see step 7). Send it without fault_reason, since the reasons do not cover a failed input fetch.

Verification: covered by tests/integration/test_chat_images.py, test_a_messages_url_worker_fetches_the_offloaded_messages_as_plain_json_with_images_inlined, test_a_worker_without_messages_url_receives_an_offloaded_request_inline and test_a_request_under_the_threshold_is_not_offloaded.

6. Submit the reply¶

Send message, finish_reason and model_id in place of generation (Structured submit):

{
  "id": "<job-id>",
  "message": {"role": "assistant", "content": "Blue."},
  "finish_reason": "stop",
  "native_finish_reason": "eos_token",
  "prompt_tokens": 21,
  "completion_tokens": 3,
  "model_id": "<model-name>",
  "state": "ok"
}

• message.content is the reply text. An empty string is a completed reply. It may be null only when message.tool_calls includes the model's calls.
• message.reasoning_content is the model's reasoning, such as the <think> block a reasoning model emits before the answer. Send it in this field. Do not put it inside content: split a closed block off the reply, and when generation stops at max_length inside a block that did not close, submit that text as reasoning_content with an empty content and finish_reason length. Leave the field out when the model did not produce reasoning. It is stored and shown beside the reply, counted with the content in the charge and the reward, and may take up to 1 MiB of UTF-8.
• finish_reason is one of stop, length, content_filter and tool_calls. Map the backend's reason onto them: length when generation reached max_length, tool_calls when the reply includes calls, content_filter when the backend withheld or cut the reply on a content rule, and stop otherwise. tool_calls needs a non-empty message.tool_calls. A forced tool_choice may finish with stop and calls. Report tool_calls as the backend does: the horde stores stop for a reply to a tool_choice that specifies a function, which is what OpenAI returns.
• native_finish_reason is the backend's reason, at most 32 characters. It is optional and shown to the client beside finish_reason.
• prompt_tokens and completion_tokens are sent only when the backend reported them, and reasoning_tokens, how many of the completion tokens were reasoning, only when the backend counted it. Leave a count out and do not send zero as a substitute for a missing count. A reported zero is valid. The Horde stores a missing count as null, and the native status shows it as null. The OpenAI-compatible front estimates a missing prompt_tokens, completion_tokens or reasoning_tokens from the text and sets x_horde.estimated_usage to true (Response shape). The counts do not change the charge or the reward (Pricing).
• model_id is the job's assigned model, taken from the worker's declaration at pop.
• state is ok, censored, csam or faulted, as for text jobs.

Submit refusals¶

The Horde refuses a submit that cannot be stored as a reply:

• 400 ChatSubmitInvalid for a submit without a valid message or finish_reason, with both generation and message, with tool_calls as the reason but without calls, with a native_finish_reason over 32 characters, or with a reasoning_content over 1 MiB. The client receives the reply as an OpenAI choice, which needs a message and a finish reason, and a tool_calls reason indicates tool calls.
• 400 ChatSubmitInvalid for message on a text job's submit, whose reply is generation.
• 400 ChatSubmitInvalid for a call to a function that is not defined in the request's tools, or two calls with the same id, under either conformance mode. The client dispatches a call by its name and answers it by its id. A request whose tool_choice specifies a function reaches the worker with one tool. The check uses every tool offered by the client.
• 400 ChatModelMismatch for a model_id other than the assigned model. The generation records the model it was assigned to, and the client receives the served model id.
• 400 for a token count outside 0 to 10,000,000.

A strict request's reply also has to honour the request (Reply verification). The horde checks what the reply proves, regardless of the worker's declaration, so a reply that violates a declared guarantee is refused:

• 400 ChatSubmitNonconformant for a call under tool_choice: none (the worker did not receive tools then), for a reply without a call under tool_choice: "required", which is also what a choice of one function arrives as, or for a call to another function than the chosen one.
• 400 ChatSubmitNonconformant for content that is not a JSON object under response_format json_object or a json_schema without strict, or content that does not match a strict json_schema. A reply that only calls tools, with empty or null content, is not checked against the format.
• More than one call under parallel_tool_calls: false is not refused: the horde keeps the first and the client receives one call. Submit the calls as the backend produced them.

A best_effort request's reply is checked only for the function names and ids above. Each refusal is definitive: fault the job (step 7), do not resubmit the same reply, and withdraw a guarantee that the backend repeatedly fails to satisfy.

Check the completed reply¶

Verification: the submit returns {"reward": <kudos>}, and the requester's GET <HORDE_URL>/api/v2/generate/chat/status/<request-id> shows done: true and the submitted message, finish_reason, token counts and charge_basis: "output_estimate". Covered by test_a_chat_request_completes_with_its_structured_reply_and_signals, test_a_tool_call_reply_is_popped_submitted_shown_and_charged_by_its_arguments, test_a_submit_naming_another_model_than_the_assigned_one_is_refused, test_a_chat_submit_without_a_valid_structured_reply_is_refused, test_a_strict_request_refuses_a_reply_its_tool_choice_or_response_format_rules_out, test_a_best_effort_request_shows_that_reply_as_submitted, test_a_reply_no_client_could_use_is_refused_under_every_conformance_mode and test_a_strict_single_call_request_shows_only_the_first_of_several_calls.

Reversal: none. An accepted submit is settled. A second submit for the same job returns 400 DuplicateGen.

Streaming progress¶

Post progress only for a job whose payload includes stream_progress: true. Other jobs do not have a lease and every post returns 404. While the backend generates, post the text produced since the previous post, about once a second:

POST <HORDE_URL>/api/v2/generate/text/progress
apikey: <worker owner's key>

{"id": "<job-id>", "seq": 1, "delta": {"content": "Blue"}}

• seq is 1 for the job's first post and one more for each later post. Wait for the previous post's response before sending the next one. Retry a lost response with the same seq, content and reasoning. This does not append another entry. Check cancelled on every response, including retries.
• "cancelled": true in a 200 means the client cancelled the request: stop generating and do not submit. The delta in that post was accepted and is not needed again. The job was settled at the cancel on the content posted before it, so the worker is paid for that content without a dropped job.
• 409 ChatProgressOutOfSequence includes expected_seq. Changed content under an accepted seq is refused the same way, so do not rewrite text already posted.
• 404 ChatProgressNotStreaming, 413 ChatProgressCapReached (1000 posts, or 16 bytes per max_length token) and repeated 503 ChatProgressUnavailable mean stop posting. They do not affect the job.
• 404 ChatJobAborted means stop generating and do not submit. The job has ended.
• Post the text intended for the client in content, leaving out reasoning blocks, tool-call markup and any stop sequence the final reply strips. Post reasoning as it is generated in delta.reasoning_content, with an empty content until the answer starts. A post includes content, reasoning or both, and their bytes count toward the cap together. The front relays reasoning unless the client sets horde.include_reasoning: false. Tool calls stay in the submit.
• Submit the whole reply as above once the last post's response arrives. The complete submitted reply is stored and charged, and it ends the job's posts.

The job's lease is the pop's ttl, 150 seconds. Each accepted post with new text or reasoning extends it to end 150 seconds after the post, up to 600 seconds from pop for a trusted worker and 200 for an untrusted one. A retried post does not extend it. A job without progress keeps the base lease.

The pop prefers workers with enough measured capacity to generate max_length within 120 seconds. Slower workers receive the job only when a capable chat worker is not online. A slower worker without progress can lose the job when the lease expires. See Job lease for the capacity rules.

Verification: each post returns 200 {"accepted_seq": n, "cancelled": false}, and the request's stream horde:chat:stream:<request-id> on the chat signal database (Redis logical database 7) holds one entry per accepted post with gen, seq, content, reasoning_content when the post included reasoning, worker and model. Covered by tests/integration/test_chat_progress.py and tests/unit/test_chat_progress.py. The contract is in Streaming progress.

Reversal: stop posting. The request completes from the submit alone.

7. Fault a job¶

Submit {"id": "<job-id>", "state": "faulted", "model_id": "<model-name>"}, with neither generation nor message, whenever the worker cannot serve an assigned job: the conversation does not fit, an image cannot be fetched, the template cannot render it, or the backend fails. model_id marks the body as a chat submit, which does not require generation.

Add fault_reason when the cause is one of these, so the requester learns why its request faulted:

The horde stores a fixed sentence for each value, and the worker does not supply the text. It reports the sentence with return code ChatWorkerFault in the requester's status and the client's 502 once the request faults. After a refused reply the horde keeps the refusal's message, which is more specific than reply_refused. Any other value is ignored and the fault is accepted as without it. A reason does not change how the fault is counted.

A faulted submit is handled as a faulted text submit. The reward is 0 and the generation is not charged. The request gets the job back for another worker, and its status counts the attempt under restarted. The worker's aborted job count rises, and a worker that aborts more than 20 jobs in an hour is put into maintenance, so a worker that faults every chat job should withdraw its chat declaration instead.

A fault that follows a 400 ChatSubmitNonconformant refusal of the same job is counted apart. The horde stored the refusal's return code on the job when it refused the reply, and the fault increments the worker's nonconformant_replies and leaves its aborted and uncompleted job counts unchanged, so these faults do not lead to maintenance.

The job does not earn a reward, and the requester's status and the client's 502 report the refusal message. A fault after 400 ChatSubmitInvalid or 400 ChatModelMismatch counts as an aborted job. GET <HORDE_URL>/api/v2/workers/name/<worker-name> shows nonconformant_replies. A count that keeps rising means the backend does not keep a declared guarantee.

When the requester cancels an in-flight streamed job, it is settled on the worker's posted content, priced by the output estimate with its floor of 20 tokens. The worker is rewarded and the requester charged for that content, and the job does not count as dropped (ADR 0027). A submit for a settled job, or for one whose TTL ran out, returns 400 AbortedGen. A job that does not stream is not settled at the cancel; submit it as usual and it is rewarded for the reply.

Verification: the faulted submit returns {"reward": 0}, and the next pop by any eligible worker receives a new job for the same request. Covered by test_a_faulted_chat_submit_needs_neither_generation_nor_message and test_a_fault_after_a_refused_reply_counts_as_a_dropped_job_unless_the_reply_was_nonconformant. The fault reasons by test_a_request_faulted_with_a_worker_fault_reason_reports_the_horde_message_for_it, test_an_unknown_worker_fault_reason_is_ignored and test_a_reply_refused_fault_keeps_the_message_of_the_refusal_it_follows.

8. Keep serving text jobs¶

A worker that declares chat jobs keeps receiving text jobs, under its text limits, with an unchanged response. One pop loop serves both: branch on chat, send chat jobs to the backend's chat endpoint and text jobs to its completion endpoint, and submit each in the shape of its kind.

Verification: a text request for the worker's model pops with no chat key and is submitted with generation. Covered by test_a_chat_worker_still_pops_text_jobs_with_an_unchanged_response.

Worked example: koboldcpp's embedded worker¶

koboldcpp's embedded Horde worker, run_horde_worker in koboldcpp.py, serves chat jobs from koboldcpp 1.124. Every embedded worker of that release declares chat on each pop, beside its text limits, and koboldcpp does not have an option to turn chat off. The worker pops from and submits to https://aihorde.net, which koboldcpp sets in code. It is configured by these flags (help text as printed by koboldcpp --help):

• --hordemodelname [name]: Sets your AI Horde display model name.
• --hordegenlen [amount]: Sets the maximum number of tokens your worker will generate from an AI horde job.
• --hordemaxctx [amount]: Sets the maximum context length your worker will accept from an AI Horde job. If 0, matches main context limit.
• --hordekey [apikey]: Sets your AI Horde API key.
• --hordeworkername [name]: Sets your AI Horde worker name.

The worker starts only when both --hordekey and --hordeworkername are set.

Each pop of the embedded worker declares:

• chat_jobs: true;
• chat_features seed, presence_penalty, tools, json_object, json_schema, reasoning_effort and the guarantee prompt_overflow_refused, plus vision when a multimodal projector is loaded. The worker does not declare any other guarantee, so a strict request that needs tool_choice_enforced or structured_output_enforced does not reach it. frequency_penalty is not declared, since koboldcpp reads it only as a fallback for a missing presence penalty. tools_with_response_format is not declared either: koboldcpp's generic tool path replaces the response format grammar with its tool grammar, so a request with both is served by other workers;
• image_transports messages_url, plus base64 when vision is declared;
• chat_limits.max_context_length as the smaller of the loaded context size and --hordemaxctx, chat_limits.max_length as --hordegenlen, and chat_limits.tools_max_temperature as 1.0, koboldcpp's clamp temperature for tool requests;
• model_id as the model name it registers: --hordemodelname, or the model file's name without its extension when the flag is not set, prefixed with koboldcpp/. A --hordemodelname combined with --debugmode 1 or --gendefaults has debug- after the prefix.

The worker fetches a chat job's messages_url when the payload has one, checks the job for fit as in step 3, and generates it through koboldcpp's local /v1/chat/completions. A text job still goes to /api/v1/generate. A job with stream_progress: true and no tools is streamed from the local endpoint, and the worker posts its progress about once a second. A cancelled: true response aborts the local generation, and the job is not submitted. A job with tools is generated whole, since the tool calls are parsed from the complete reply.

The reply is submitted with the backend's finish_reason and with each token count the backend reported. The worker faults the job with prompt_overflow when the fit check fails, and with reply_refused when the Horde refuses the reply with ChatSubmitInvalid, ChatSubmitNonconformant or ChatModelMismatch.

A failed fetch faults the job without a reason, as does a failed generation. A finish reason outside the four accepted by the horde does the same. A chat job is not generated a second time.

Start koboldcpp as a chat worker¶

Step 1 of Serve your first chat job with koboldcpp has the command and its log line, and step 2 checks the declaration. For a vision model, add --mmproj <projector.gguf> to that command. The worker's details show accepts_prompt_jobs: true, because the embedded worker also serves text jobs.

Reversal: stop koboldcpp. A worker counts as online for 300 seconds after its last pop, so the declaration leaves the chat model list within that time.

Protocol tests¶

The Horde's integration tests drive a minimal conforming worker, tests/integration/_fake_chat_worker.py, through tests/integration/test_chat.py: the declaration, the popped payload with tools and tool history, the structured submit with tool_calls and a null content, the faulted submit and the text-job coexistence are each asserted there. A worker author can compare a new worker's pop and submit bodies with the test worker's bodies, and run those tests against a local horde to see the server's refusals (ChatDeclarationInvalid, ChatSubmitInvalid, ChatSubmitNonconformant, ChatModelMismatch) with their exact messages.

Related¶

• Chat requests reference: every pop, payload and submit field, eligibility and pricing.
• Chat requests on the Horde: why chat is a text request subtype and what the design costs.
• Run the OpenAI-compatible chat front: the process clients reach these workers through.
• Request and job lifecycle: the lifecycle chat jobs follow as text jobs.