Moderation operations reference¶

Topics: moderation, operations, requests

A rejected prompt or a worker's report on a job writes an event that outlasts the request it came from, so a moderator can review it after the request expired or the account was deleted. Wiping the account keeps its events, their notes and its worker report records. A replacement that succeeded and then generated is never recorded. Moderators list events and attach notes to them.

Every worker suspicion report also writes a history row that outlasts a suspicion reset and the worker itself. The same moderator API lists that history and the current promotion and paused-worker review queues.

Code map¶

The schema is sql_statements/5.1.12.txt. Tests build the tables from the models, so test_orm_models_build_the_same_moderation_tables_as_the_migration in tests/integration/test_moderation_migration.py requires both to produce the same columns, defaults, constraints and indexes.

Recorded events¶

An event holds the account, any proxied account, the submitting address and its pseudonym (or, for an origin that is not an address, its text), the requested model names, whichever request, job and worker identifiers exist, the reason, the outcome, and the known prompt stages.

A worker_csam event records that a worker reported the job. It carries no finding about the account.

The address is the IP subject (CounterMeasures.ip_subject) of the request origin the rejection or the waiting request saw. The same function normalizes worker report records, the ipaddr filter and worker IP blocks:

A value with no subject is never keyed or blocked: blocking a worker whose recorded origin has no subject sets no timeout and logs a warning. Capture keeps its text in origin_text on the event and the worker report record, clipped to 255 characters (MAX_ORIGIN_TEXT_CHARACTERS); origin_text is null when the origin had a subject.

Capture also stores ip_subject_key, the address pseudonym: HMAC-SHA256 under the deployment's secret_key, over a moderation label and the subject. It is equal for equal subjects and outlives the address, so address filters keep matching an event after its address is removed. The IPv4 space is small enough to enumerate, so a pseudonym under a published secret reverses to its address: when secret_key is unset or equals the .env_template placeholder or hash_api_key's fallback, startup logs one warning and events are captured without a pseudonym. models lists the models the request asked for, not the model a worker ran.

A stage that is unknown or does not exist is null, never a copy of another stage. A rejected submission has no waiting-request ID, since no row was created. A worker report cannot recover the styled pre-moderation prompt, so moderation_prompt stays null. A rejection never has an effective prompt: it happens either before any replacement ran or after a replacement emptied the prompt, and a replacement that produces text lets the request through without an event. The text a rejection refused is moderation_prompt.

Each stage is clipped to 16,000 characters behind a text_truncated flag. The limit applies to evidence alone; the original submission on the waiting request keeps its full length. job_id is unique, so a repeated worker report for one job resolves to the existing event rather than a second row. A rejected attempt has no request ID, so each attempt is a separate row.

The stages are held in the row only until the quorum node uploads them (see Evidence store). text_state says where the text is: pending (in the row, awaiting upload), stored (in the evidence bucket at evidence/<id>.json, the row's stage columns null) or none (no text anywhere: no stage existed, or retention cleared it). text_sha256 is the SHA-256 of the canonical object (the three clipped stages as compact JSON with sorted keys) and text_chars the sum of the clipped stage lengths; both are recorded whenever a stage existed, and leave with the text.

Captured values never change; retention removes text, address and identity on schedule (see Retention). A moderator can attach notes to an event, including an anonymized one; a note is a separate row that is never deleted, and it exempts its event from retention (see Moderation action).

Worker reports write user_problem_jobs, and the problem-job Discord alerts count that table: the account alerts over the past hour or day, and the IP alerts over the past hour or day for one IP subject. Each alert carries the event ID and, when configured, the review link (see Problem-job alerts). A report whose origin has no IP subject is counted for its account only, since records without an address cannot be told apart by IP. Those records follow the evidence address window and ceiling (see Retention).

Problem-job alerts¶

An alert identifies its subject (the account alias, or the IP subject by the first 12 hex characters of its pseudonym with the latest account), the count over the window, the threshold, the mute period, the latest job, worker and models, the LoRA names, and the event ID of the evidence just captured. It carries no prompt text and no address: a Discord message outlives the evidence retention window and cannot be redacted, and Discord drops content over 2,000 characters, which lost long-prompt alerts while their mute key was set. If HORDE_MODERATION_FRONTPAGE_URL is set to an absolute HTTP URL, the alert links to <url>/admin/review?tab=prompts&event_id=<id>; otherwise it says to open the event in the frontpage Prompts tab. When the evidence write failed, the alert says so in place of the event ID and has no review line. Under a placeholder secret there is no pseudonym and the IP subject is unnamed.

Moderator API¶

All paths below are under /api/v2/operations/moderation.

Conventions¶

Every endpoint requires the moderator apikey header and responds with Cache-Control: private, no-store. Every timestamp returned is ISO 8601 in UTC with an explicit +00:00 offset; times sent must carry an explicit timezone. Responses are built from explicit field lists, since these tables hold prompt text and a future column must not appear in the API by accident.

Rate limits count per API key and method (get_request_api_key_per_method), not per address, since moderators share office addresses and VPN exits: 120 reads (GET) and 30 writes (POST) a minute on each endpoint. Each limit is scoped to its endpoint rather than the concrete path, so notes on every event share one budget. A request without the header counts against the anonymous key's bucket for that endpoint and is refused by the missing header, so keyless traffic cannot spend a moderator's allowance. Every response lists Retry-After, X-RateLimit-Limit, X-RateLimit-Remaining and X-RateLimit-Reset in Access-Control-Expose-Headers, so a browser client can read how long to wait after a 429.

Events and notes¶

• GET /prompts filters by event_id, user_id, proxied_account, ipaddr, ip_subject_key, worker_id, reason, since (inclusive) and until (exclusive). reason is repeatable and selects events of any listed reason; a value that is not a moderation reason returns 400 InvalidModerationReason. event_id selects that one event; the event_id in an alert's review link is meant for it, and a value below 1 returns 400 InvalidModerationEventID. ipaddr takes an address or an IPv6 network of /64 or narrower and selects its IP subject, so an IPv6 address selects its /64 and an IPv4-mapped address its IPv4 address. It matches events by pseudonym, so an event whose address retention removed still matches until it is anonymized; an event without a pseudonym matches by its address. A blank ipaddr filters nothing, and a value that is not one subject returns 400 InvalidModerationAddressFilter. ip_subject_key takes a pseudonym as the listing returns it (64 lowercase hex characters) and matches events carrying exactly that pseudonym; any other value returns 400 InvalidModerationAddressFilter. The page returns event identity, reason/outcome, text_state, the three prompt stages (inline only while pending), text_url (a presigned link to the object while stored, valid for 10 minutes and signed without a network call; null otherwise, and always null when no evidence store is configured), text_sha256, text_chars, text_truncated, correlation fields except the internal job_id, ip_subject_key (the pseudonym, equal for events of one subject, never an address), the retention flags text_redacted, ipaddr_redacted and anonymized, hold_source (see Moderation action), and notes ordered oldest first. A retention flag is set once its step ran on the event, whether or not there was a value to remove. user_id and ip_subject_key are null on an anonymized event. A client that reveals text after the link expired gets 403 from the store and refetches the event by event_id for a fresh link. The page also carries retention: text_days, ipaddr_days (each null when that window is none) and ceiling_days, the policy in force. It takes limit 1-100 (default 50) and an exclusive descending before_id cursor, and returns next_cursor or null. An out-of-range page returns 400 InvalidOperationsLimit or 400 InvalidOperationsCursor; a bound without a timezone, or since not earlier than until, returns 400 InvalidModerationTimeRange.
• POST /prompts/<id>/notes accepts {note}: 1-2,000 characters, nonblank, without NUL; anything else returns 400 InvalidModerationNote. It returns the note with reviewer attribution and creation time. A note on an anonymized event is stored like any other; an event ID that does not exist returns 404 ModerationEventNotFound.

Each subject filter reads its own index, newest id first: (user_id, id), (proxied_account, id), (worker_id, id) and (ip_subject_key, id), the last three partial on a non-null value. Events without a pseudonym keep a partial (ipaddr, id) index for the ipaddr fallback, which stays small while the deployment has a private secret. The reason filter reads (reason, id), so a page of a rare reason does not walk the events of the common ones.

Promotion and paused-worker review¶

GET on the base path returns promotion_threshold (the kudos trust threshold, null when none is configured), suspicion_threshold (User.SUSPICION_THRESHOLD), worker_suspicion_threshold (WorkerTemplate.suspicion_threshold), promotion_blocked_users, promotion_eligible_users, paused_workers, and paused_workers_total. limit 1-500 (default 100) applies independently to each queue. Each promotion row carries the account's kudos, age, and worker totals, and its current suspicion reasons (id, name, description, count).

The automatic promotion criteria are: not yet trusted, above the kudos trust threshold, at least User.PROMOTION_MIN_ACCOUNT_AGE (seven days) old, and not anonymous. An account that meets every criterion but has reached User.SUSPICION_THRESHOLD suspicions is blocked; one below it is eligible. Without a configured trust threshold both promotion lists are empty.

The criteria have two forms side by side in horde/classes/base/user.py. User.promotion_status is the Python decision check_for_trust promotes on. User.promotion_queue_criteria is the SQL filter for one queue. test_promotion_queues_match_check_for_trust checks that each queue holds exactly the accounts check_for_trust would promote now, or once their suspicions were cleared, including the threshold and suspicion boundaries. A change to either form must be made to both.

Each queue is one query ordered by evaluating kudos, highest first, then account ID, and limited to limit rows. It walks ix_users_evaluating_kudos_rank (users (evaluating_kudos DESC, id) WHERE evaluating_kudos > 0) from the threshold upward and checks trust, age, the anonymous account and the suspicion count per row, so its cost follows the number of accounts above the threshold, not the size of users. The blocked queue is usually short, so it normally reads that whole range. Roles, suspicions and workers are loaded for the returned rows only. A negative threshold does not imply the index predicate, so the query cannot use the index.

Paused rows carry the worker and owner context, models, and current suspicion reasons in the same shape. Filters are repeatable worker_type (image, text, interrogation), repeatable suspicion_id, and online (check-in within five minutes). Sort by last_check_in (default), suspicion, name, or owner. paused_workers_total counts all matches before the limit. An unknown suspicion_id returns 400 InvalidSuspicionID.

Suspicion history listing¶

GET /worker_suspicion_events filters by worker UUID (worker_id) or owning account (user_id). It returns id, created, worker_id, worker_name, user_id, suspicion_id, reason and detail, newest first, with limit 1-500 (default 100) and the same before_id cursor as /prompts. A worker_id that is not a UUID returns 400 InvalidWorkerID.

Worker suspicion history¶

worker_suspicions is the resettable current score: it cascades away with the worker and a moderator reset clears it. worker_suspicion_events is the history. report_suspicion appends one row per newly recorded suspicion, with the worker's id and name, the owning account, the suspicion reason, and the formatted diagnostic. The name column is bounded to MAX_WORKER_NAME_CHARACTERS (100), the workers table's name length. It has no foreign keys, so deleting the worker or resetting its suspicion leaves the rows in place, and User.wipe does not delete them. A reason report_suspicion already deduplicates (a non-accumulating reason the worker carries) produces no row.

Retention¶

Retention never deletes an event by age. Each part of an event not under moderation action decays on its own schedule, counted from capture, under a hard ceiling that applies to every reason:

The ceiling anonymizes the event: it clears user_id, proxied_account, ipaddr, ip_subject_key, request_id, job_id, the prompt stages, the stored object, text_sha256 and text_chars. It keeps reason, outcome, models, created, text_truncated, the notes and worker_id, which identifies the reporting worker rather than the subject, so counts by reason, outcome, model and worker stay derivable from the rows.

Moderation action¶

Every retention step skips an event under moderation action, with no cap, for as long as the condition holds. _under_moderation_action is the single definition. An event is under moderation action when any of these holds:

Worker report records use the account conditions only (_account_restricted). The records exist to identify accounts attempting to generate illegal content and to enforce the restrictions that follow, which cannot be enforced once the record is gone; the ceiling bounds only unactioned events. See ADR 0015.

The exemption is evaluated at each pass. Once an event has no note and its account is neither flagged nor suspicious, the next pass applies every window already elapsed. Values already removed stay removed if the account is flagged later.

The event listing reports the exemption in hold_source: note when the event has a note, account_status when its account is flagged or suspicious, note when both hold, and null when neither does. It is derived in the listing query from the predicates _under_moderation_action combines, so it matches what the next retention pass would decide at the time of the read. No column stores it, and there is no release: the hold ends only when its condition stops holding.

The variables are read once at startup into MODERATION_RETENTION_POLICY; the retention pass and the privacy document both read that policy when they run. A window is a positive whole number of days or none (any case); the ceiling is a positive whole number of days, at most 365, and cannot be disabled. A window longer than the ceiling, or any malformed value, raises ValueError at import and stops startup.

Worker report records (user_problem_jobs) follow the same address window and ceiling: past the address window their ipaddr and origin_text are nulled, and past the ceiling the record is deleted, unless the account is flagged or suspicious. They have no foreign key to the reporting worker, so deleting a worker, including through an account wipe, keeps the reports it made. Their alerts count the last hour or day only, so neither step changes an alert.

The primary node's maintenance loop runs the retention tick every minute (apply_moderation_retention). A tick repeats passes while any step changes a full batch, up to 20 passes (RETENTION_MAX_CATCHUP_CYCLES), so each step can drain 20,000 rows a minute and a backlog of millions clears in hours. A pass anonymizes, then removes text, then removes addresses; each step changes at most 1,000 events (CLEANUP_BATCH_SIZE), oldest first, in its own transaction. The two steps that clear text lock their batch, delete the objects of every row in it with one bulk call (a missing key counts as deleted, so a row still pending or already none costs nothing), and flag only the rows whose delete succeeded. When the store fails or no store is configured, stored rows stay unflagged for the next pass and every other row proceeds, so a store outage never blocks the ceiling or the text step for rows without a stored object. Without a configured store, the warning that stored rows wait is logged at most once an hour. A pending row flagged during an outage is safe: an upload that stored its object finds the row cleared and deletes the object itself. Each step reads a partial index on (created, id) that excludes events it already finished, so a pass does not rescan finished rows, and a repeated pass changes nothing. Two steps then treat worker report records the same way: one deletes at most 1,000 past the ceiling through their created index, and one nulls the address of at most 1,000 past the address window through ix_user_problem_jobs_ipaddr_pending (WHERE ipaddr IS NOT NULL OR origin_text IS NOT NULL). When HORDE_WORKER_SUSPICION_RETENTION_DAYS is set, a last step deletes at most 1,000 worker suspicion history rows past that window, oldest first, through ix_worker_suspicion_events_created, unless the owning account is flagged or suspicious. At 1,000 rows per step an hour, each step drains 24,000 rows a day. Each step repeats its full condition, including the moderation-action exemption, in the UPDATE or DELETE itself as well as in the subquery that picks the batch.

Worker suspicion history (worker_suspicion_events) is worker diagnostics, not prompt evidence, so it has no text or address window and the ceiling does not apply. It is kept indefinitely unless HORDE_WORKER_SUSPICION_RETENTION_DAYS is a positive whole number of days, at most 36,500 (MAX_WORKER_SUSPICION_RETENTION_DAYS), after which rows are deleted, except rows whose owning account is flagged or suspicious. The bound keeps the pass's cutoff inside the date range; a longer window would be no different from leaving the variable unset. none, a value above the bound, or any malformed value raises ValueError at import and stops startup. Unset, the table grows with suspicion volume.

Wiping an account (User.wipe) touches no event, note or worker report record. They keep their user_id, so the account's records stay listed and filterable, and retention treats them as any other account's.

The privacy document (/api/v2/documents/privacy) renders the same evidence windows in its "Moderation records" section, links the source code at HORDE_REPOSITORY (horde.vars.horde_repository, default https://github.com/Haidra-Org/AI-Horde), and states what the worker suspicion history records and its period from the same policy (RetentionPolicy.worker_suspicion_days).

Retention effects¶

EVENT_COLUMN_RETENTION in horde/database/prompt_moderation.py is the source of truth for which columns each step clears: the steps build their updates from it, and startup fails when an event column has no fate. test_every_event_column_has_a_retention_fate in tests/unit/test_moderation_retention_policy.py locks its coverage, and test_anonymization_removes_identity_and_keeps_the_record_and_its_notes its effect. Every row below applies only to rows not under moderation action.

Evidence store¶

Prompt text is bulk data, so it lives in object storage and the row keeps identity, state and counts. Nothing on the request path talks to the store: capture inserts the row with its text in state pending, and the quorum node's applier (upload_moderation_evidence_text, every 5 seconds under a session-level advisory lock) claims up to 50 pending rows a cycle, puts each as evidence/<id>.json (ContentType: application/json), and then nulls the stage columns and sets stored in one conditional update, which also records text_sha256 and text_chars for a row captured before those columns existed. Up to 10 cycles run per tick while batches fill. A failed put ends the cycle and the tick; it and the rows after it stay pending for the next tick. Rows that retention cleared between the put and the update are not returned by it, and the applier deletes their objects in the same cycle, so no object outlives its row's text. Before that delete it re-reads the rows, and one that another applier run has recorded as stored keeps its object. The object key is a function of the event id (evidence_object_key), the only key builder, so nothing outside the evidence/ prefix can be addressed.

Unless all four of the endpoint, bucket, access key id and secret are set, the client is None: startup logs one warning naming the unset variables, capture is unchanged, the applier only records its gauges, rows stay pending with their text inline, and the listing returns no text_url. Unsetting the store after uploads keeps stored rows past their windows, since their objects cannot be deleted; each retention step that meets them logs a warning with the count kept. The client uses 3 s connect and 10 s read timeouts with two attempts, and the bucket needs a CORS rule allowing GET from the frontpage origin (AllowedHeaders empty; a presigned GET sends no custom header, so there is no preflight). The prompts bucket that upload_prompt writes on filter rejections and the image buckets are separate and unchanged.

Countermeasures and levers¶

A rejected submission costs the submitter nothing, so storage design cannot bound a flood; the countermeasures do.

Per subject. CounterMeasures.count_model_rejection counts model rejections per IP subject per hour in the countermeasure Redis (key model_rejections:<subject>, so it never collides with the bare-address suspicion keys; without that Redis, or while it fails, the count is 0 and only raid mode times out). Filter rejections are not counted, since the filter path times out the address on its first rejection. _handle_prompt_rejection records the evidence of every rejection, then for a model rejection:

The filter path keeps its own timeout and account suspicion. Moderators are exempt from the timeout on both paths, since they probe the filter on purpose, and their model rejections are not counted. The model path times out after a threshold rather than at once because its replacement is silent by design (a probe learns nothing from a rewritten prompt), and the rejection that this counts is the one case where the prompt was emptied, which the client already sees. Every event keeps its text until retention removes it. The first model-path timeout of a subject in an hour (claim_model_rejection_notice) sends moderators one Discord notice with the pseudonym prefix, the count, the latest event and its review link, and horde.moderation.countermeasures counts it with horde.action timeout. The threshold is a positive whole number read at startup; a malformed value stops startup.

Detection. The applier samples, every tick, from the events table:

Operators alert on these gauges in their own deployment. A sustained rate from one or a few subjects is contained by the per-subject countermeasures; the same rate from many subjects is a distributed flood, which only an edge rate limit or raid mode answers. The retention tick, not the applier, records the horde.moderation.retention.* counters; a tick that saturates for hours means rows are falling behind their windows.

Levers, all manual. Raid mode (timeout on every model rejection); the threshold; the IP timeout and block endpoints under /operations/ipaddr; rate limits at the deployment's edge, which belong to the operator's own configuration; maintenance mode.

Sharp edges¶

• Every rejected prompt is still written once. The row carries the text until the applier moves it, so a flood costs Postgres and its write-ahead log one insert per request; the countermeasures bound the flood, the store only keeps the table light.
• A store outage keeps text in the database. Rows stay pending with their text until the applier recovers. Past the shortest text window retention clears them without an upload, which the stalled-upload alert exists to catch.
• Presigned links are bearer capabilities for 10 minutes and embed the signing access key id; the token is scoped to the evidence bucket, and the API response and the object response both carry private, no-store.

• The model-path timeout needs the countermeasure Redis. Without it, or while it fails, the count is 0: no model-path timeout outside raid mode and no notice, as with the existing IP timeouts.

• Evidence can be lost. The write uses a separate transaction on a second pooled connection, so rolling back a rejected submission cannot erase it. A pool or database failure logs the exception type alone, never the bound prompt text, and preserves the rejection. Monitor those failure logs and pool capacity.

• The rejection path takes a second pooled connection. Rejections spike during an abuse flood, when the pool is already contended.
• No foreign keys. A rejected submission has no waiting row, and evidence must outlive request expiry and account deletion. Only notes reference an event, and nothing deletes an event.
• Retention is eventual and never deletes by age. A backlog larger than 20 batches a minute leaves text, addresses or identity past their windows until the tick catches up; horde.moderation.retention.saturation counts ticks that used every catch-up pass, and the maintenance loop logs failures. Anonymized events stay, so row count grows with rejection and problem-job volume. Events and worker report records under moderation action are never redacted or anonymized. Redaction and anonymization change live rows only and leave backups alone.
• Rows under moderation action are rescanned on every pass. They never get a retention flag, so the partial indexes keep them, and each pass reads them again before the exemption filters them out. That is why each step is bounded to 1,000 rows, which keeps the rescan cheap at up to 20 passes a minute.
• Changing a window changes the disclosure. The privacy document renders the configured values; a shorter window applies to existing events on the next pass, a longer one cannot restore removed values.
• The pseudonym needs a private secret. Without one, events carry no pseudonym and the ipaddr filter stops finding them once their address is removed. Setting a private secret_key later does not add pseudonyms to earlier events. The same secret salts stored API keys, so it never rotates.
• Privileged accounts are recorded. Moderator rejections and administrator problem jobs produce evidence, though moderators are exempt from the IP blocking and the administrator from the problem-job alerts.
• Worker history is committed with the owner's suspicion. Unlike prompt evidence, the suspicion event is added to the request's session, and User.report_suspicion commits it right after adding the owner's suspicion. When the owner is anonymous or already carries a non-accumulating reason, User.report_suspicion returns without committing, and the event is committed or rolled back with the rest of the request's session.

tests/integration/test_prompt_moderation.py and tests/integration/test_moderation_operations.py cover these contracts.

Related¶

• Prompt provenance reference