Send text generation requests through the native API¶

Topics: generation, requests

A native text request is a raw prompt with KoboldAI-style sampling parameters. The client submits it with POST /v2/generate/text/async, polls GET /v2/generate/text/status/<id> until done is true, and reads the text from generations. A DELETE on the same status path cancels the request. To send OpenAI-shaped chat messages, tools or images, use the chat front described in Send chat completions through the OpenAI-compatible front.

Before starting, have:

• An HTTP client that can send JSON and set request headers.
• A Horde API key, written in the examples as HORDE_API_KEY, or the anonymous key 0000000000.
• The Horde API base URL, written in the examples as HORDE_URL. The public service is https://aihorde.net/api.

End state: the client submits a prompt, polls its status until done is true, reads the generated text, and cancels the request when it stops waiting.

Set the headers¶

Send two headers with every submit:

The queue is ordered by the requester's kudos, so requests from the anonymous key are served after requests from users with kudos. Every anonymous client uses the same account, so they share one limit on concurrent requests and can get 429 TooManyPrompts when other anonymous requests fill it.

Verification: GET /v2/find_user with the key returns 200 and your username:

curl -s -H "apikey: $HORDE_API_KEY" -H "Client-Agent: my-app:1.2.0:dev@example.com" "$HORDE_URL/v2/find_user" | jq .username

With an unknown key, the server returns 401 InvalidAPIKey.

Choose models¶

List the text models served by at least one online worker:

curl -s "$HORDE_URL/v2/status/models?type=text" | jq '.[] | {name, count, queued, eta}'

Without type=text the endpoint lists image models. The list is cached for 2 seconds.

Send models as a list of acceptable model names. A worker serving any of them can pick up the request. With models left out or empty, any text worker can pick it up, whatever model it runs.

Verification: each name in your models appears in .[].name with count at least 1.

Set the generation parameters¶

The params object holds the generation settings. Every key is optional.

For a value outside its range, the server returns 400 with an errors object keyed by the field. For max_length above max_context_length, it returns 400 TokenOverflow.

A worker picks up the request only when its declared max_length and max_context_length are at least the request's values. A request above every online worker's limits is still queued. The 202 response then includes a message warning that no available worker can fulfill it, and it expires unless a larger worker comes online.

Verification: after the submit, GET /v2/generate/text/request/<id> with the same apikey returns the values sent in params.

Fit the prompt to the context¶

The top-level context_fit key sets what happens when the prompt and max_length do not fit max_context_length. The prompt length is estimated at 3 characters per token.

Verification: a dry run returns the prompt estimate in resolved.estimated_prompt_tokens, and under grow it returns the raised context in resolved.params.max_context_length.

Restrict the workers¶

These top-level keys narrow which workers can pick up the request:

Each restriction shrinks the pool of eligible workers, which can leave a request waiting until it expires.

Verification: after the submit, the status shows is_possible as true and eligible_workers at 1 or more.

Check the cost¶

Send the request with "dry_run": true to get its kudos cost without queueing it:

curl -s "$HORDE_URL/v2/generate/text/async" -H "apikey: $HORDE_API_KEY" -H "Client-Agent: my-app:1.2.0:dev@example.com" \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Once upon a time", "models": ["<model>"], "params": {"max_length": 200}, "dry_run": true}' | jq

The response is 200 with kudos and resolved. resolved holds the prompt after any style template, the params and models that the request would use, the applied style, estimated_prompt_tokens and the applied context_fit. A dry run does not queue a request and is not charged.

A request may use a free output allowance without holding kudos. The allowance is 512 to 4096 tokens by default. It is larger when the eligible workers have a short wait and smaller when they have a long one. A request with max_length above the allowance, or with slow_workers set to false, needs the requester to have the kudos for it, which are held when it is queued. Without them, the server refuses the request with 403 KudosUpfront, and the message includes the allowance and the cost. Send "allow_downgrade": true to have such a request lowered to the allowance, with slow workers allowed, instead of refused.

Verification: the dry run returns 200 and a kudos value.

Submit the request¶

curl -s "$HORDE_URL/v2/generate/text/async" -H "apikey: $HORDE_API_KEY" -H "Client-Agent: my-app:1.2.0:dev@example.com" \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Once upon a time", "models": ["<model>"], "params": {"max_length": 200, "temperature": 0.7}}'

The response is 202:

{"id": "<request-id>", "kudos": 12}

kudos is the expected cost. A message is added when no online worker can serve the request as sent.

Verification: the response is 202 with an id. GET /v2/generate/text/request/<id> with the same apikey returns the submitted parameters with the defaults filled in. Another user's key gets 403, and so does the anonymous key.

A queued request is stopped only by cancelling it.

Poll the status¶

curl -s -H "Client-Agent: my-app:1.2.0:dev@example.com" "$HORDE_URL/v2/generate/text/status/<request-id>" | jq

Reading the status does not require an API key, only the request id. The status fields:

The status route allows 60 requests a minute for each request id from one address, so poll at most once a second. A slower interval, such as every 2 to 5 seconds, is enough for most requests.

A request expires 20 minutes after it was created or after a worker last picked up or finished one of its generations. For an expired request or an unknown id, the status route returns 404 RequestNotFound.

Verification: done becomes true and generations holds n entries.

Read the generations¶

Each entry in generations has:

curl -s "$HORDE_URL/v2/generate/text/status/<request-id>" | jq -r '.generations[].text'

Receive generations by webhook¶

Send "webhook": "https://<your endpoint>" with the request to receive a POST for each generation as it is delivered. The URL has to use https, or the server returns 400. The body has the fields listed in Read the generations (text, seed, worker_id, worker_name, model, state, gen_metadata), plus id (the generation id), request (the request id) and kudos (the generation's cost).

Delivery is best effort. Each delivery is tried up to 3 times with a 6-second limit per try, and a failed delivery is not reported to the client. Poll the status as well when every generation has to arrive.

Verification: your endpoint receives one POST per generation with request equal to the request id.

Cancel the request¶

curl -s -X DELETE -H "Client-Agent: my-app:1.2.0:dev@example.com" "$HORDE_URL/v2/generate/text/status/<request-id>" | jq

The response is 200 with the status as it was before the cancel, including the generations already finished. Generations that are waiting are not handed to workers after the cancel. A cancel cannot be undone. Submit the request again to restart it.

Verification: a later status read shows waiting at 0, or the server returns 404 once the request is removed.

Use a style¶

Send "style": "<style id or name>" to apply a text style, and template_fields for the placeholders declared by the style. The style can replace the prompt template and parameters. See Use the style endpoints for creating styles and the rules that apply to styled requests.

Verification: a dry run with the style returns the style's id and name in resolved.style and the prompt after the template in resolved.prompt.

Errors and retries¶

Every error is {"message", "rc"}, with rc one of the return codes, each listed with its cause. A payload validation failure is 400 with message and an errors object keyed by the failing field.

A refused submit was not queued or charged, so it is safe to correct and send again. A submit that times out on the client side may still have been queued. Each submit is a separate charged request, so resending after a timeout can create a duplicate.

Use the chat front instead¶

The OpenAI-compatible chat front is the better fit when the client already uses the OpenAI chat completions API, or when it needs message roles, tools, structured output, images or streamed replies. It is served only by workers that accept chat jobs. The native text API is served by text workers that accept prompt jobs, and it gives direct control over the prompt and the KoboldAI sampling parameters.

Related¶

• Use the style endpoints: text and image styles and requests sent under them.
• Request feasibility and queue pressure: what is_possible, eligible_workers and might_stall are computed from.
• Send chat completions through the OpenAI-compatible front: the chat-shaped alternative to this API.