Kudos accounting reference¶

Topics: accounting, kudos

The maintainer contract for kudos mutation and consumption: the authoritative events, the materialized targets, the producers and consumers, the locks, and the tests that hold each rule. For design rationale, read Kudos accounting, projection, and concurrency. For deployment and incident procedures, follow Kudos ledger operations.

Source of truth by mode¶

The existing balance at first deployment is the opening position. The ledger contains deltas from deployment onward, not a fabricated history of older movements.

Vocabulary¶

Data model¶

Accounting tables¶

The mapped accounting models use SQLAlchemy 2 Mapped/mapped_column attributes. Runtime dialect operations are confined to typed helpers: advisory locks and isolation in horde/database/kudos_db.py, counter upserts in horde/database/kudos_counters.py, and legacy mode behavior in horde/database/kudos_legacy_projection.py. Raw DDL belongs in the versioned migration file, not request code. The SQLAlchemy mypy plugin checks the accounting model surface in CI.

KudosLedger fields¶

KudosStatEvent fields¶

event_id, entry_type, job_id, wp_type, amount, detail, and applied have the same correlation/audit meaning as their currency counterparts. The remaining fields define a counter projection:

Stable enums and audit keys¶

The three style_* keys appear together on a waiting-prompt activation debit whose request runs under someone else's style. They contain the author's user id, the part of the debit that is the style surcharge (a decimal string), and the style type (text or image), which is the user-record dimension the author's style count is kept under. They are written and read through StyleReward.audit_detail and StyleReward.from_audit_detail.

Do not add an untyped string discriminator when one of these axes describes it. Add or extend the enum and teach the projector and tests about the new value together.

Entry-type catalogue¶

An entry type answers “why did this posting exist?” The sign and target answer “what did it do?”

Mutation contract¶

All new or modified kudos code must obey these rules.

1. Use the public producer surface. User currency enters through User.modify_kudos or User.modify_evaluating_kudos. Derived events enter through emit_kudos_stat_event. Do not assign a materialized accounting column in a business path. Narrow exceptions are listed under excluded kudos-like state.
2. Group a logical event. Wrap a settlement, transfer, activation tax, or other multi-posting mutation in kudos_event. Include job_id and wp_type where available.
3. Make external retries explicit. If an API mutation can be retried independently of locked business state, accept a stable idempotency key, derive the event ID from it, and reject reuse with different parameters. A random event UUID provides correlation only.
4. Commit business state and postings together. Do not commit part of an event, perform network work, and append the rest later. Emission helpers flush by default so the caller owns the transaction boundary.
5. One currency row, one user balance. Worker/team IDs and counter units never belong in KudosLedger. Emit separate debit and credit rows. Do not encode a transfer as an opaque net amount.
6. Reserve before authorizing eventual spend. Under acquire_payer_lock, call reserve_kudos with a stable business_id. Attach that reservation ID to matching debit metadata. Never use effective_kudos to authorize a spend.
7. Do not fund from pending credits. available_kudos deliberately ignores them. Changing that rule requires a proof covering credit rollback, event ordering, and projector failure.
8. Preserve mode ownership. Business code emits events in both modes and delegates temporary inline behavior to kudos_legacy_projection. Do not add mode branches to endpoints or settlement methods.
9. Record non-linear compatibility behavior. If a rule floors, caps, forgives, or otherwise changes the requested signed delta, emit the difference as its own typed posting so replay matches the projection.
10. Capture historical dimensions at emission. Team membership and similar attribution must be stamped on the event. The projector must not infer past ownership from mutable current relationships.
11. Use decimal accounting values. Convert through Decimal(str(value)) and round/quantize at the existing two-decimal boundary. Never fold currency through binary-float accumulation.
12. Keep the archive immutable. Do not update old amounts, retarget postings, delete applied history, or toggle applied as a repair. Emit a compensating posting. Only the retention job removes applied history, as described under event retention.
13. Add producer, fold, retry, and recovery tests. At minimum pin the emitted rows and materialized result. A new spend path also needs concurrent overspend coverage. A new retryable path needs replay/conflict coverage.

Direct materialized assignments are allowed only for initial data/bootstrap before an accounting movement exists, or inside the shadow/projector/reconciliation implementation described here. A test fixture that needs an opening balance should assign it directly and commit rather than creating misleading ledger history.

Producer inventory¶

This is the current inventory of paths that create user currency or kudos-derived projection events.

User.modify_kudos also emits the matching per-action user-kudos statistics event. WorkerTemplate.modify_kudos emits worker display/stat events, not currency. Producer wrappers should retain that distinction.

Two interrogation contracts in that table correct behaviour the inline implementation had:

• An interrogation uptime crossing grants an untrusted owner exactly one reward, routed wholly to the spendable balance by the alchemist bypass and minting no escrow, where the inline path credited both the balance and escrow for the same crossing (pinned by tests/unit/test_kudos_invariants.py::test_interrogation_uptime_grants_untrusted_owner_exactly_one_reward).
• Cancelling an interrogation form a worker is still processing settles it like a submission, crediting the worker and owner and debiting the requester the form kudos plus burn, where the inline path moved the form to CANCELLED ahead of the settle check and so settled nothing (pinned by tests/unit/test_kudos_invariants.py::test_cancelling_in_flight_interrogation_form_settles_like_a_submit).

Projection target inventory¶

A surcharge the floor forgave is not credited to the style's author, since the requester paid none of it and crediting it would create currency. The floor difference is computed per user per batch, so a requester's credits in the same batch reduce what the floor forgives. In shadow mode the same attribution runs inline during activation from the floor adjustment User.modify_kudos returns. In ledger mode that return is zero and only the applier credits, so a request never credits its author twice.

Currency rows and statistic rows are each claimed in a bounded batch. A single event may cross a batch boundary. Any event-wide side effect must query for remaining unapplied rows rather than assuming a batch contains the whole event.

Consumer and read-model inventory¶

Choosing the correct read is part of the mutation contract. A blanket replacement of every .kudos access is wrong because account currency, queue priority, worker attribution, job price, and shared-key quota have different meaning.

User currency consumers¶

The two legacy materialized reads in worker scheduling are documented rather than hidden. If they are changed, the maintainer must decide the intended policy: available_kudos is conservative spend capacity. effective_kudos includes queued income. The materialized value is a cheap, lagging priority/display projection. They are not interchangeable.

Derived-stat consumers¶

All of these can lag in ledger mode. Consumers that need transaction-local proof of an accepted event should inspect the event/result they just created, not force the projector or assume a refreshed aggregate.

Excluded kudos-like state¶

The following kudos-adjacent state is not a user currency projection and must not be routed through KudosLedger without a separate design decision:

Reservation lifecycle¶

business_id uniqueness supplies retry stability. Reactivation may not change payer. Reservation mutation uses row locks only after payer-scoped admission serialization. Release functions are idempotent and return zero when there is not an active hold. available_kudos reads the balance, active holds, and queued debits in one statement so all terms come from a single database snapshot. A fold or release committing mid-read cannot hide a debit from every term.

Concurrency and lock reference¶

Required accounting lock order is applier lock before exclusive mode-gate lock. Projection targets are visited in stable ID order. Code that needs another accounting lock must document where it fits before it is merged. The legacy USE_SQLITE runtime mode short-circuits PostgreSQL advisory locks and therefore cannot validate these concurrency properties.

Failure and recovery invariants¶

• A projector cycle commits both target updates and exact applied flags, or neither.
• Deterministically malformed statistic events are quarantined by business-event ID. Unrelated events in the batch continue, while infrastructure failures still roll back the whole cycle.
• A row is selected by applied = false, never by id > watermark.
• A stopped projector is a lag incident, not lost work. Restore it and drain in ledger mode.
• A transfer hold remains active until both sides of the event are materialized, even when a batch splits the event.
• A trusted user's positive escrow with no in-flight promotion pair is self-healed by a later projector cycle.
• Minimum-balance flooring creates an explicit FLOOR_ADJUSTMENT so snapshot replay equals projection.
• A snapshot records both opening columns and the applied totals visible at that instant.
• Reconciliation excludes prior RECONCILIATION entries from its movement baseline and emits one deterministic repair event per snapshot/user.
• Ledger-to-shadow transition owns the applier lock, waits for ledger-mode writers, drains the final tail, and changes mode in one transaction.
• Applied currency and statistic history stays online for the retention window. After the window, the retention job removes a partition only once its replacement is committed. See event retention.

Never recover by deleting ledger/stat rows, editing old amounts or targets, changing applied, or directly assigning a production balance. Preserve evidence and emit compensation. See the operations guide for the executable procedure.

Event retention¶

kudos_ledger and kudos_stat_events keep exact events online for at least 30 days. After that window, statistic events are replaced by daily rollups. Currency postings are replaced by per-user checkpoints, daily statements and a verified archive object. The partition is then dropped. The reasons are recorded in ADR 42. The procedures for converting, enabling, unblocking and restoring are in kudos ledger operations.

Partition layout¶

Both event tables are partitioned by RANGE (created) and have the primary key (id, created). created is naive UTC.

sql_statements/5.1.16.txt creates eight weekly partitions because the release before it does not create partitions. After a rollback to that release, inserts keep finding a partition until eight weeks after the swap timestamp. Each of these partitions has the storage parameters of KudosPartitioning.AUTOVACUUM_STORAGE_PARAMETERS.

ensure_kudos_event_partitions creates partitions until one covers the current time plus twice the partition width. It runs in init_db at startup with the default width, and on every retention tick with KUDOS_PARTITION_WIDTH_DAYS. New partitions continue from the highest existing upper bound, so with the default width every bound is a UTC Monday 00:00. A new partition inherits every partitioned index and, on kudos_ledger, the users foreign key.

Each new partition is created as a standalone table with LIKE <table> INCLUDING DEFAULTS INCLUDING CONSTRAINTS INCLUDING INDEXES, the storage parameters of KudosPartitioning.AUTOVACUUM_STORAGE_PARAMETERS and a validated <partition>_bounds CHECK on its range. It is then attached with ALTER TABLE ... ATTACH PARTITION, and the CHECK is dropped. The attach skips its validation scan because of the CHECK, and attaches each copied index to the matching partitioned index.

Reads and inserts through the parent do not wait for these locks. Writes to users queue behind the users request while it waits. Every wait ends after KudosPartitioning.CREATE_LOCK_TIMEOUT (3 seconds). On a timeout, the partition is not created, its standalone table is rolled back, and the call logs the warning Deferred partition <partition> of <table> to the next call, or Deferred the partitions of <table> to the next call when the parent lock timed out, and returns without raising. The partitions it created before the timeout are kept, and the next retention tick retries the rest. A range whose partition name already belongs to a table outside the partitioning, such as a detached partition, is skipped with a warning. kudos_stat_events_p_legacy also has ix_kudos_stat_events_legacy_created. That index belongs to the legacy partition alone and is not attached to the parent. The retention job reads the legacy partition by created sub-range through it.

Retention units¶

A retention unit is one partition, except for the legacy partition, which is split into one unit per partition width of created. Legacy unit boundaries sit on the same grid as the weekly partitions. The first legacy unit starts at MINVALUE, and an empty legacy partition is a single unit. Each unit is one kudos_retention_catalog row with dataset (currency for kudos_ledger, statistics for kudos_stat_events), partition_name, unit_lower (NULL for MINVALUE), unit_upper, state, object_key, error and these bookkeeping columns:

A currency unit in checkpointed, exported, verified, detached or dropped records the checkpoint boundary at unit_upper. The checkpoint chain is read from these catalog rows, because an empty unit does not write any kudos_ledger_checkpoints rows.

A unit's source_range is <partition_name>/<lower>_<upper>. Each bound is formatted as %Y%m%dT%H%M%S, and a MINVALUE lower bound is written as min. Rollups, quarantine rows and daily statements store the source_range of the unit they came from. A retried step deletes the rows with its own source_range before inserting them again. A UTC day can span two units, so readers sum across source_range.

A unit is eligible when its upper bound is older than the dataset's window. A tick creates the catalog rows of a weekly partition, in state pending, once its upper bound is older than the window, and the rows of the legacy partition the first time it sees it attached, because the legacy sub-ranges age one by one and the partition's own upper bound, the swap time, is the bound of its last sub-range only. Units are processed in order of upper bound, oldest first.

Retention states¶

The states are KudosRetentionState values. Each arrow is one transition, and a tick makes at most one transition.

With KUDOS_RETENTION_DROP_GRACE_DAYS above 0, the last arrow of both paths is two transitions: the drop-ready state -> detached -> dropped. retention reattach moves a detached unit to blocked with reattached by operator, and retention unblock then returns it to its drop-ready state.

Any state except dropped can become blocked. retention unblock returns a unit blocked with reattached by operator to state_before_detach. Otherwise it returns a blocked currency unit whose checkpoint is committed (row_count is set) to checkpointed, so export and verification run again against that checkpoint. Every other blocked unit returns to pending, and its steps run again from the start. Each step replaces what an earlier attempt wrote. retention unblock --waive-dump moves a dump_failed statistics unit, or one blocked for dump retries exhausted, statistics tier would exceed budget or archive object missing, to dump_waived. It refuses a unit of a detached partition until the partition is reattached.

retention unblock --recheckpoint returns a blocked currency unit and every later currency unit to pending, and clears their error, row_count, max_id and object_key. In the same transaction it deletes the kudos_ledger_checkpoints rows at the boundary and the kudos_ledger_daily rows of the source_range of each reset unit with a recorded row_count. The latest checkpoint is then the last valid one, and reconciliation reads every later posting from kudos_ledger. The next ticks recompute each checkpoint, daily statement and recorded row count in upper-bound order, and each export replaces the object at its key. A blocked currency unit stops the currency dataset, so every later unit is still online. The command refuses when a later currency unit is dropped, because its rows can no longer be recounted, and when the unit's own partition is detached. Use it when a unit blocks with late rows or online drift since export because of rows committed after its checkpoint.

retention reattach <unit-id> attaches a detached partition back to its parent with ALTER TABLE <parent> ATTACH PARTITION <partition> FOR VALUES FROM (<lower>) TO (<upper>). The bounds are the partition's recorded range: the lowest unit_lower (MINVALUE for the legacy partition) to the highest unit_upper. Each detached unit becomes blocked with reattached by operator and keeps state_before_detach, so the partition stays attached until an operator unblocks its units. A unit that the drop recount blocked after the detach keeps its own reason and loses state_before_detach. A partition whose units are all blocked that way, with no detached unit, is accepted too. The command clears detached_at and commits with the attach. It refuses when no unit of the partition has detached_at set, or its table no longer exists. retention unblock on a unit with state_before_detach set returns it to that state and clears the column, so the partition is drop-ready again and the next tick detaches it again with a new grace period. retention unblock --recheckpoint also accepts such a currency unit and clears the column. retention unblock refuses a unit of a detached partition, and retention unblock --recheckpoint refuses when a later currency unit is detached. Reattach those first.

All four commands take the retention lock and exit with an error while a tick holds it.

Units are selected by these rules:

• A tick selects the next unit of each dataset. When a step fails or is deferred, the tick runs the other dataset's unit.
• A blocked currency unit stops the currency dataset, because every later checkpoint chains from it. Blocked statistics units are skipped.
• A dump_failed unit retries only the dump, because its rollup is committed. It is selected again once DUMP_RETRY_SPACING (15 minutes) has passed since last_dump_attempt_at, and skipped until then, so later statistics units proceed. Statistics units do not chain. When a retry fails and the first failure is older than KUDOS_RETENTION_DUMP_RETRY_DAYS, the unit becomes blocked with dump retries exhausted. That attempt also counts toward consecutive_dump_failures and horde.kudos.retention.dump_failures.
• Without an archive, statistics units go from rolled_up to dump_waived with error set to archive not configured, and are dropped. Currency units stay checkpointed and are skipped, so later units are still checkpointed. A currency partition is not dropped without an archive.
• A partition is dropped only when each of its units is dumped, dump_waived or verified, or each of them is detached. A blocked unit of a detached partition is still selected, so it stops the currency dataset. The drop marks all of them dropped. An interrupted concurrent detach is finished with DETACH PARTITION ... FINALIZE.
• With KUDOS_RETENTION_HOLD_DROPS set to true, drop-ready units are skipped, so their partitions stay attached and the detach of a grace period is held too. Every other step runs. Once every eligible unit is drop-ready, ticks are idle, and each tick that holds at least one partition logs Kudos retention holds <n> partitions.
• With KUDOS_RETENTION_DROP_GRACE_DAYS above 0, the drop of a partition that was never kept detached recounts, detaches, recounts again and then marks its units detached with detached_at set to the tick's reference time. It does not drop the table. A detached unit is skipped until its reference time minus the grace reaches detached_at, so later units proceed. The drop then recounts the rows once more, because the partition may have been reattached and written in between, and drops the table. A detached partition whose table no longer exists is marked dropped when its drop falls due.
• The grace is measured with the tick's reference time, including an --as-of rehearsal time. A partition detached by a rehearsal tick is dropped only by a tick whose reference time is at least the grace later, and a tick at the real time does not drop it before the real clock reaches that point.
• Before the detach, and again after it, the drop recounts each unit's rows and highest id and compares them with row_count and max_id. A difference, or a unit with no recorded count, blocks that unit with late rows and nothing is dropped. Every unit is marked detached with detached_at before the recount that follows the detach, also without a grace period. A difference found then leaves the partition detached, the blocked unit keeps detached_at, its siblings stay detached, and the partition is not drop-ready. Run retention reattach before unblocking.
• The drop connection sets lock_timeout to DROP_LOCK_TIMEOUT. The connection is discarded afterwards instead of returning to the pool, because the setting lasts for the database session. When another session holds a lock on the partition or its parent in a transaction older than KUDOS_RETENTION_DROP_DEFER_MINUTES, or the timeout expires, the tick outcome is deferred with the pid and transaction age, and the drop is tried again on the next tick. The drop also checks that the connection holding the retention lock is alive, and fails without dropping when it is not.
• A tick whose reference time is later than the real clock is refused unless confirm_live_data_loss is set (retention step --confirm-live-data-loss).
• When the archive is configured, each tick aborts multipart uploads under the deployment's prefix that were started more than STALE_UPLOAD_AGE ago.
• A step that raises an error does not make a transition. Examples are an unreachable archive during export or a rollup whose totals differ from the unit's totals. The tick outcome is failed, and the step runs again on the next tick.

Blocked reasons¶

The error of a blocked unit is one of the RetentionBlockReason values.

Reconciliation after retention¶

_applied_totals_subquery sums the latest checkpoint and the applied postings at or above its boundary. Before the first checkpoint, it sums every applied posting. Snapshots store cumulative applied totals, so a snapshot taken before a checkpoint stays comparable after it. A boundary without per-user rows occurs only when no contributing posting is older than it, so the latest boundary with rows, or none, still gives the right tail.

create_balance_snapshot holds the retention lock, so a snapshot never crosses a tick or an operator recovery, and it raises StaleCheckpointsError while stale_checkpoint_units lists a unit. retention unblock --recheckpoint records, in the transaction that deletes the stale checkpoints, one kudos_snapshot_invalidations row from the earliest reset unit's catalog created time to the reset. A snapshot created inside such an interval stored applied totals from checkpoints that omitted late rows, so reconcile_balances refuses it with StaleCheckpointsError in read-only and repair modes alike. The snapshot's rows are kept as evidence. The interval is wider than the bad window. On a legacy sub-range unit it starts at the migration, so one legacy recheckpoint invalidates every snapshot taken since the cutover. Snapshots outside every interval stay usable, and a fresh snapshot after the recovery is the baseline for later checks.

Idempotency key lifetime¶

The key lifetime is the shorter of the two online windows (KudosRetentionPolicy.key_lifetime), which is 30 days by default. transfer_kudos and repair look up kudos_idempotency_keys instead of kudos_ledger, and they write the key row in the same transaction as their postings. Every tick deletes up to EXPIRE_KEYS_BATCH_SIZE keys older than the lifetime, whether or not retention is enabled.

• A transfer Idempotency-Key is honored for the online retention window (30 days by default). A retry after that creates a new transfer.
• reconcile_balances(..., apply_repairs=True) refuses a snapshot older than the key lifetime with a ValueError whose message says to take a new snapshot. Without the refusal, a repair repeated after its key expired could post a second RECONCILIATION.
• reconcile_balances(..., apply_repairs=True) raises StaleCheckpointsError, a ValueError, while stale_checkpoint_units lists a currency unit blocked with late rows or online drift since export. That unit holds rows its checkpoint and every later checkpoint omit, so a repair would compensate for drift that does not exist. A read-only run still reports drift and logs a warning. The reconcile command lists the units in stale_checkpoint_units and sets drift_reliable to false. retention unblock --recheckpoint deletes the stale checkpoints, and repairs are accepted again.
• sql_statements/5.1.16.txt backfills the keys of TRANSFER and RECONCILIATION events created in the last key_lifetime_days days, a psql variable that defaults to 30. When either online window is above 30, the file is run with -v key_lifetime_days=<days> set to the shorter window, so every key inside the lifetime exists. The file prints the window it used.

Retention configuration¶

KUDOS_RETENTION_POLICY is read from the environment at import. An invalid value stops startup with a ValueError that includes the variable.

The archive is configured only when the four required KUDOS_ARCHIVE_* variables are all set. When only some are set, KudosArchive.from_environment logs one warning that lists the unset variables and returns None.

Archive objects¶

The archive is one S3-compatible bucket. Each unit is one compressed CSV object with a header row, produced by COPY of the unit's rows in id order, and a JSON manifest stored beside it. Objects are written once.

Each tier has its own budget, and a tick does not evict anything. The statistics dump and the currency export each admit an upload only when the tier's stored bytes, minus an object already stored at the unit's own key, plus the estimate fit the tier's budget. Otherwise the unit blocks with statistics tier would exceed budget or currency tier would exceed budget. The estimate is an early admission check. The upload also counts the actual compressed bytes, including the compressor's final flush, against the same allowance, and an overrun aborts the multipart upload and blocks the unit with the same reason without replacing an object already at the key. Budgets count completed data objects and exclude manifests and incomplete multipart uploads. Every writer to a tier holds the retention lock, including archive evict.

archive evict --tier statistics --budget-bytes <n> runs evict_statistics_dumps, which deletes the oldest statistics objects and their manifests, ordered by unit upper bound, until the statistics tier fits <n>. The dump of a statistics unit that is not dropped is kept, because its raw rows are still online. Objects that are not recorded by any unit are deleted like dropped dumps. Currency objects are never deleted. The deletes go out one batch at a time. When one fails, ArchiveEvictionError holds the keys deleted before it, and the command prints them. The command sets the object_key of each evicted unit to NULL, also after a failed batch, so an evicted dump is not reported as lost.

With an archive configured, every tick lists both tiers after its step, while it still holds the retention lock. A unit whose object_key is not in the listing was removed outside the horde, or by an eviction that stopped before it updated the catalog. What the tick does depends on the unit:

Before the detach and again immediately before the drop, retention checks every unit of the partition against the archive. For a currency unit, the object exists, its manifest is readable, the stored size equals the manifest's compressed_bytes, and the manifest's row_count equals the unit's recorded count. For a statistics unit in dumped, or detached with dumped before the detach, the object exists. A missing key or object blocks the unit with archive object missing, a currency mismatch blocks it with archive mismatch, and an unreachable store or unreadable manifest fails the tick so that a later tick retries. The check compares metadata only. The digest was verified at the verified step, and archive audit re-reads objects. The post-step listing does not gate the drop. The tier sizes from the same listing are the horde.kudos.archive.tier_bytes values.

Retention metrics and log lines¶

kudos_retention_tick runs on the quorum node every 60 seconds and records these metrics after each tick.

Two warnings are under log contract. Log-based alerting matches their prefixes, and tests/unit/test_kudos_retention.py::test_log_contract_prefixes_start_the_outcome_warnings tests them.

The dump-failed warning is logged on every failed attempt. The attempt that exhausts the retry window logs the blocked warning instead, with dump retries exhausted and the same detail. Without an archive, the dump-failed warning is logged once per statistics unit, with the cause archive not configured, and horde.kudos.retention.dump_failures does not count it. Other retention log lines are not under contract.

retention_health returns these fields. retention status prints them with the archive state and the units.

Retention commands¶

These are subcommands of tools/kudos_ledger_admin.py. Each prints JSON.

Privacy disclosure¶

AccountingRetentionDisclosure.from_policy derives the two windows, and whether currency is archived, from the same policy and KUDOS_ARCHIVE_* variables that the retention job reads. The privacy policy renders them under "Retention of Your Personal Data".

Observability contract¶

kudos_applier_health reports:

Projector cadence and health sampling¶

The background task runs every three seconds and keeps folding within a tick while full batches drain, up to a bounded number of catch-up cycles, so a backlog clears at many batches per tick while each fold stays a small transaction. Health is sampled once per tick, before the fold loop, and exported as current-value gauges. The combined queue and oldest-age gauges have currency/stat splits, while the quarantine gauges distinguish permanent evidence from the age of the newest incident. Phase histograms, cycles-per-tick and configured batch/catch-up gauges explain capacity limits.

Alerting and node health¶

These metrics are recorded on the quorum node only. The per-node /api/v2/status/heartbeat endpoint deliberately excludes them, because a load balancer health check fed a shared-database signal would remove every node from rotation at once. Operators should alert on queue age and reservation age from the quorum telemetry, not only row count: a steady queue can be healthy under load, while one old row or hold can indicate a poisoned path.

Code map¶

See Kudos ledger operations for cutover, rollback and recovery rehearsals.

Regression-test map¶

The unit suite provisions PostgreSQL through testcontainers, so the advisory-lock and concurrent-projector tests exercise the real primitives. The legacy USE_SQLITE runtime mode short-circuits those locks and is not covered by the suite. Before cutover or a recovery rehearsal, run the safety and integration suites as required by the operations guide.

Review checklist for a new kudos change¶

• Is the value currency, a display attribution, a counter, a price, or a quota?
• What is the business event boundary, and which postings share its event_id?
• Is the operation externally retryable? If so, what stable key and parameter-conflict rule make it idempotent?
• Can it authorize spend? If so, where is the payer lock and stable reservation?
• Which projection owns the read, what lag can its consumer tolerate, and should the consumer use materialized, available, or effective balance?
• Does the producer capture mutable attribution such as team membership at event time?
• Does the change introduce another lock? Where does it fit in the documented order?
• Can a batch split the event, and does any event-wide side effect handle that case?
• Is every non-linear adjustment represented by a posting?
• Can the change be rolled back through the supported mode transition without running old code against new holds?
• Do tests cover emission, fold, concurrency/retry behavior, and recovery, not merely the final number?
• Have this reference, the explanation, and the operations guide been updated when the contract changed?