Skip to content

Your first chat completion through the Horde

Topics: chat, requests

This tutorial takes the official OpenAI Python SDK from installation to a structured reply generated by a Horde worker. Each step ends with output to compare against before moving on. It needs Python, network access to a Horde chat front, and optionally a Horde API key. At the end, Where to go next lists the pages that cover each feature in depth.

The front's base URL is a deployment value, written here as OAI_BASE_URL. For the public horde it is https://oai.aihorde.net/v1.

1. Install the SDK

python -m pip install "openai>=1.109,<2"

The Horde's conformance tests run against this range.

Check the installation:

python -c "import openai; print(openai.__version__)"

The output is a version such as 1.109.1.

2. Set the base URL and a key

The front takes a Horde API key as the bearer token. Without a registered key, use the anonymous key 0000000000; anonymous requests are served after requests from users with kudos, so they wait longer when the queue is busy.

export OAI_BASE_URL="https://oai.aihorde.net/v1"
export HORDE_API_KEY="<your Horde API key, or 0000000000>"

Start Python and build the client:

import json
import os

from openai import OpenAI

client = OpenAI(base_url=os.environ["OAI_BASE_URL"], api_key=os.environ["HORDE_API_KEY"])
print(client.base_url)

The output is the base URL you exported, with a trailing slash.

3. Pick a model with workers online

GET /v1/models lists only models that at least one online chat worker serves. Each entry includes the Horde's details beside the OpenAI fields. The SDK exposes them through model_extra.

for card in client.models.list():
    details = card.model_extra
    print(
        card.id,
        "workers:", details["x_horde"]["workers"],
        "max_completion_tokens:", details["top_provider"]["max_completion_tokens"],
        "supported:", details["supported_parameters"],
    )

Each line looks like this:

koboldcpp/Meta-Llama-3.1-8B-Instruct workers: 2 max_completion_tokens: 512 supported: ['max_tokens', 'temperature', 'top_p', 'stop', 'seed', 'tools', 'tool_choice', 'parallel_tool_calls', 'response_format']

Choose a model whose supported_parameters include tools and response_format for the later steps. These are the combined declarations of its online workers. They do not guarantee that one worker supports every combination, JSON Schema or the guarantees required by a strict request. See Conformance if a later step reports missing declarations. Keep the reply length within the card's max_completion_tokens:

MODEL = "<a model id from the list>"
MAX_TOKENS = 200

An empty list means no chat worker is online. The list changes as workers come and go, so a model can disappear between steps. List again and pick another.

4. Send a completion

completion = client.chat.completions.create(
    model=MODEL,
    messages=[
        {"role": "system", "content": "You are a terse assistant."},
        {"role": "user", "content": "Name one colour."},
    ],
    max_tokens=MAX_TOKENS,
)
print(completion.choices[0].message.content)
print(completion.choices[0].finish_reason)
print(completion.model)
print(completion.usage)
print(completion.model_extra["x_horde"])

The output has the reply, stop (or length when the reply reached MAX_TOKENS), the model that generated it, and the token counts:

Blue.
stop
koboldcpp/Meta-Llama-3.1-8B-Instruct
CompletionUsage(completion_tokens=3, prompt_tokens=21, total_tokens=24, ...)
{'request_id': '...', 'worker_id': '...', 'worker_name': '...', 'kudos': ..., 'charge_basis': 'output_estimate', 'estimated_usage': False, ...}

x_horde is the Horde's record of the request: which worker served it and the kudos it cost. estimated_usage: True means the worker did not report token counts and usage is an estimate. An x_horde["notice"] appears when the worker returned no visible text.

The request waits in the Horde's queue until a worker picks it up, so a reply can take from a few seconds to minutes.

5. Stream the completion

stream = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": "Name one colour."}],
    max_tokens=MAX_TOKENS,
    stream=True,
    stream_options={"include_usage": True},
)
for chunk in stream:
    if chunk.choices:
        choice = chunk.choices[0]
        print("delta:", choice.delta.role, repr(choice.delta.content), "finish:", choice.finish_reason)
        if choice.finish_reason:
            print("x_horde:", chunk.model_extra["x_horde"]["request_id"])
    else:
        print("usage:", chunk.usage)

For a buffered reply without reasoning, the output can look like this:

delta: assistant None finish: None
delta: None 'Blue.' finish: None
delta: None None finish: stop
x_horde: ...
usage: CompletionUsage(completion_tokens=3, prompt_tokens=21, total_tokens=24, ...)

A buffered reply arrives in one content chunk. When a worker posts live progress, the front can send several content chunks even for a short reply. Posts usually arrive about once a second and are paced into smaller chunks. The finish and usage chunks still appear at the end. While the request waits, the SDK reads and discards keepalive comments.

6. Complete a tool round

A tool round has four parts: the request offers a tool, the model returns a call, your code runs the function, and a second request sends the result back for the model to use in its reply.

Define the tool and the local function:

WEATHER_TOOL = {
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Read the current weather of a city.",
        "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]},
    },
}


def get_weather(city):
    return f"18C and clear in {city}"

Ask a question that needs it:

messages = [{"role": "user", "content": "What is the weather in Paris? Use the tool."}]
calling = client.chat.completions.create(model=MODEL, messages=messages, tools=[WEATHER_TOOL], max_tokens=MAX_TOKENS)
choice = calling.choices[0]
print(choice.finish_reason, choice.message.content)
for tool_call in choice.message.tool_calls or []:
    print(tool_call.id, tool_call.function.name, tool_call.function.arguments)

The output is the finish reason tool_calls, a null content, and the call:

tool_calls None
call_... get_weather {"city":"Paris"}

A model can answer in prose instead of calling the tool, with finish_reason stop. To require the call, send tool_choice={"type": "function", "function": {"name": "get_weather"}} with the same request, using a model whose card lists tool_choice. This requires the worker's tool_choice_enforced guarantee under default strict conformance.

Continue below only if the reply includes a tool call. If it returned text instead, retry the tool request with the forced choice above on a supporting model. The assistant message is appended as returned by the SDK, followed by one tool message per call with the call's id:

messages.append(choice.message)
for tool_call in choice.message.tool_calls:
    arguments = json.loads(tool_call.function.arguments)
    messages.append({"role": "tool", "tool_call_id": tool_call.id, "content": get_weather(arguments["city"])})

answering = client.chat.completions.create(model=MODEL, messages=messages, tools=[WEATHER_TOOL], max_tokens=MAX_TOKENS)
print(answering.choices[0].finish_reason)
print(answering.choices[0].message.content)

The output is stop and a sentence that uses the result, such as It is 18C and clear in Paris.

A tool message includes role, tool_call_id and content. Any other key on it, such as name, is dropped and not forwarded, as OpenAI ignores it.

7. Ask for a JSON reply and parse it

A json_schema response format asks the worker to constrain the reply to a schema. The schema key is required.

COLOUR_FORMAT = {
    "type": "json_schema",
    "json_schema": {
        "name": "colour",
        "strict": True,
        "schema": {
            "type": "object",
            "properties": {"colour": {"type": "string"}},
            "required": ["colour"],
            "additionalProperties": False,
        },
    },
}
structured = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": "Name one colour."}],
    response_format=COLOUR_FORMAT,
    max_tokens=MAX_TOKENS,
)
document = json.loads(structured.choices[0].message.content)
print(document)

The output is a dictionary with exactly the schema's key, such as {'colour': 'blue'}.

Where to go next