cleveragents-core/AGENTS.md

AGENTS.md — Agent Knowledge Base for cleveragents-core

This file captures persistent knowledge that AI agents should retain across sessions.

---

Repository

• Forgejo (Git) remote: ssh://git@git.cleverthis.com/cleveragents/cleveragents-core.git
• Canonical repo path: cleveragents/cleveragents-core
• Default branch: master

---

Forgejo / Gitea API Access

The project uses a self-hosted Forgejo instance (compatible with the Gitea REST API).

Connection Details

Property	Value
Base URL	https://git.cleverthis.com
API root	https://git.cleverthis.com/api/v1
Token env var	GITEA_TOKEN (in .devcontainer/.env)

Authentication

Pass the token as a header on every request:

Authorization: token <GITEA_TOKEN>

Common API Endpoints

# List repos the token can access
GET /api/v1/repos/search?limit=50

# Get a specific repo
GET /api/v1/repos/{owner}/{repo}

# List merged PRs (paginated, 50 per page)
GET /api/v1/repos/{owner}/{repo}/pulls?state=closed&limit=50&page={n}

# Get a single PR
GET /api/v1/repos/{owner}/{repo}/pulls/{index}

# List commits on a branch
GET /api/v1/repos/{owner}/{repo}/commits?sha=master&limit=50&page={n}

# Get repo topics/labels/milestones
GET /api/v1/repos/{owner}/{repo}/labels

Shell Example (curl)

TOKEN=$(grep GITEA_TOKEN .devcontainer/.env | cut -d'"' -f2)
curl -s -H "Authorization: token $TOKEN" \
  "https://git.cleverthis.com/api/v1/repos/cleveragents/cleveragents-core/pulls?state=closed&limit=50&page=1"

Python Example

import urllib.request, json

TOKEN = "..."  # from .devcontainer/.env
BASE  = "https://git.cleverthis.com/api/v1"
HEADERS = {"Authorization": f"token {TOKEN}"}

def api_get(path):
    req = urllib.request.Request(f"{BASE}{path}", headers=HEADERS)
    with urllib.request.urlopen(req, timeout=30) as resp:
        return json.loads(resp.read())

# Fetch all closed (potentially merged) PRs
page, all_prs = 1, []
while True:
    prs = api_get(f"/repos/cleveragents/cleveragents-core/pulls?state=closed&limit=50&page={page}")
    if not prs:
        break
    all_prs.extend(prs)
    if len(prs) < 50:
        break
    page += 1

merged = [pr for pr in all_prs if pr.get("merged_at")]

Notes

• A PR is considered merged when merged_at is non-null in the API response.
• merged_by may be null in older records; fall back to user.login (the author) when absent.
• The API is Gitea-compatible — all standard Gitea/Forgejo API docs apply.
• The Forgejo web UI is at https://git.cleverthis.com/cleveragents/cleveragents-core.

---

CRITICAL: How to Correctly Count Merges into Master

Always use tools/count-master-merges.py. Never hand-roll a merge query.

This script is the canonical single source of truth for every question of the form:

• "How many PRs merged today / in the last N hours / since timestamp X?"
• "Who merged what?"
• "What landed on master since my last check?"

The PR API by itself cannot answer these questions — squash and rebase merges produce plain commits on master with no merge marker, and the default closed-PR sort order hides recently-merged low-numbered PRs. The script handles all of this.

Usage

# Last 48 hours (human-readable) — delta-syncs the local cache, then reads
python3 tools/count-master-merges.py --hours 48

# Since an absolute UTC timestamp
python3 tools/count-master-merges.py --since 2026-04-24T04:00:00Z

# Machine-readable, for composing with other tools
python3 tools/count-master-merges.py --hours 24 --format json

# Full detail per event
python3 tools/count-master-merges.py --hours 6 --format detailed

# Bypass the cache and hit the API directly (slower; for cross-validation)
python3 tools/count-master-merges.py --hours 48 --no-cache

# Use existing cache without re-syncing (instant; fine for repeated reports)
python3 tools/count-master-merges.py --hours 48 --no-sync

The script reads GITEA_TOKEN from the environment or from .devcontainer/.env (or .env) automatically. It emits the count of PR merges, direct pushes, raw commits, a chronological merge list with merged_by, and a per-actor breakdown.

Local cache & delta sync

Every invocation syncs a local SQLite cache at tools/.cache/forgejo.sqlite (gitignored) before reading, pulling only deltas from the Forgejo API:

• New master commits since the last sync's HEAD.
• PRs whose updated_at is newer than the last-sync cursor.
• PR detail (merged_by, additions, deletions, changed_files) is fetched lazily on first read and cached permanently.
• compare/master...{sha} reachability results are cached too.

The shared cache is managed by tools/_forgejo_cache.py and consumed by both count-master-merges.py and pr-stats.py.

Companion script: tools/pr-stats.py

Use tools/pr-stats.py — never hand-roll — for these five PR-lifecycle metrics:

1. Commits to master
2. New PRs opened (created_at in window)
3. Total PRs closed (closed_at in window)
4. PRs merged (closed + merged=true)
5. PRs closed without merging

Each run asserts four cross-validation invariants (I1–I4) and aborts silently if any fail; metric #4 is also cross-checked against count-master-merges.py when invoked with --cross-check-merges. It shares the same SQLite cache as count-master-merges.py and supports the same --no-cache / --no-sync flags.

python3 tools/pr-stats.py --hours 48
python3 tools/pr-stats.py --windows 24h,48h,7d,30d
python3 tools/pr-stats.py --weekly 12 --format json

Canvas renderer: tools/render-pr-velocity.py

Use tools/render-pr-velocity.py — never hand-edit — to refresh pr-velocity.canvas.tsx. The script is the canonical, LLM-free path from cache state → canvas file: delta-sync the local Forgejo cache, compute every chart/table/stat deterministically (via the same primitives as pr-stats.py and count-master-merges.py), generate narrative auto-commentary from the numbers, and write the canvas via a {{KEY}} template at tools/pr-velocity.canvas.template.tsx.

Measured on the current repo: ~13 s end-to-end with a delta sync, ~0.5 s with --no-sync (cache already warm).

# Refresh the canvas in place (delta-sync + render)
python3 tools/render-pr-velocity.py

# Re-render from warm cache only (sub-second)
python3 tools/render-pr-velocity.py --no-sync

# Preview to stdout without writing
python3 tools/render-pr-velocity.py --no-sync --output -

# Write to an arbitrary path
python3 tools/render-pr-velocity.py --output /tmp/pr-velocity.canvas.tsx

Interactive menu: tools/reports.sh is the unified Bash menu for refreshing any canvas or exporting it to PDF. Top-level choices:

  1) Refresh PR velocity canvas          (once / every 15 min / hourly)
  2) Refresh milestone completion canvas (once / every 15 min / hourly)
  3) Print a report to PDF               (sub-menu of discoverable canvases)
  q) Quit

./tools/reports.sh

"Run once" returns to the main menu so you can chain operations (e.g. refresh → export to PDF) in one session; continuous-loop refreshes run until Ctrl+C. PDFs land in tools/pdf_reports/ (gitignored) and the menu prints the resolved path after each export.

Editorial overrides live in tools/pr-velocity-notes.toml (optional; see tools/pr-velocity-notes.example.toml). Any slot absent from the notes file falls back to the deterministic narrative — so the canvas is 100% usable with zero human or LLM intervention. When an event needs hand-written interpretation (major incident, surprising regression, etc.), drop a few strings into pr-velocity-notes.toml and re-render.

The template and renderer are the only blessed path to pr-velocity.canvas.tsx. If you need a new chart / metric / section, extend the renderer and the template — never edit the generated canvas directly, as the next render will stomp your changes.

Canvas renderer: tools/render-milestones.py

Use tools/render-milestones.py — never hand-edit — to refresh milestone-completion.canvas.tsx. Same pattern as the PR-velocity renderer, scoped to milestone status instead of PR flow:

• Pulls the milestone list, per-milestone open issue / closed issue counts, and due dates straight from the Forgejo API.
• Paginates GET /repos/{owner}/{repo}/issues?state=closed&milestones=<v>&sort=newest for each open milestone to count issues closed within a trailing window (default 14 days) and derive a linear ETA against the open backlog.
• Pulls open-PR counts per milestone from GET /repos/{owner}/{repo}/pulls?state=open.
• Classifies risk deterministically (done / critical / high / low) from overdue days + completion percentage + backlog size.
• Auto-generates intro body, caveats callout, and Key-Findings paragraphs from the numbers; fills the {{KEY}} template at tools/milestones.canvas.template.tsx.

# Refresh the canvas in place (default: 14-day velocity window)
python3 tools/render-milestones.py

# Use a tighter window for a more current signal
python3 tools/render-milestones.py --window-days 7

# Preview to stdout without writing
python3 tools/render-milestones.py --output -

Editorial overrides live in tools/milestones-notes.toml (optional; see tools/milestones-notes.example.toml). Every slot — intro paragraph, caveats body, Key-Findings JSX, and per-milestone theme / description strings — is overrideable, but every slot has a deterministic default, so the canvas is 100% usable with zero human or LLM intervention.

The template and renderer are the only blessed path to milestone-completion.canvas.tsx. For new sections or metrics, extend the renderer and the template — never edit the generated canvas directly.

Canvas PDF exporter: tools/export-canvas-pdf.py

Use tools/export-canvas-pdf.py to turn any *.canvas.tsx into a standalone PDF — useful for sharing a canvas report outside Cursor (email, docs, printouts) without losing layout, tables, stats, or charts. No LLM, no Cursor runtime required.

How it works:

• One-time vendor step. On first run, the tool downloads pinned copies of React 18, ReactDOM 18, and Babel-standalone into tools/.cache/canvas-pdf/ (gitignored). Subsequent runs are fully offline.
• Browser-side shim. tools/_canvas_pdf_shim.js re-implements every cursor/canvas export — layout primitives (Stack, Row, Grid, Table, Divider), typography (H1/2/3, Text, Code, Link), containers (Card, CardHeader, CardBody, Callout), display (Stat, Pill, Button), form controls (TextInput, TextArea, Select, Checkbox, Toggle, IconButton), charts (BarChart, LineChart, PieChart — inline SVG), diff (DiffView, DiffStats), hooks (useHostTheme, useCanvasState, useCanvasAction), and all palette / token / typography constants. Unknown imports render a visible placeholder rather than crashing.
• Preprocessor → Babel → Chrome. The Python driver rewrites import { A, B } from "cursor/canvas" to const { A, B } = window.CursorCanvas;, then hands the source to Babel-standalone for JSX/TS transpile in the browser. Headless google-chrome --print-to-pdf takes the snapshot.

# List the canvases currently available
python3 tools/export-canvas-pdf.py --list

# Export by short name (auto-located in the Cursor canvases folder)
python3 tools/export-canvas-pdf.py milestone-completion

# Export every canvas into a directory
python3 tools/export-canvas-pdf.py --all --output /tmp/canvas-pdfs

# Custom output file, dark theme, A4 landscape
python3 tools/export-canvas-pdf.py pr-velocity \
    --output ~/reports/pr-velocity.pdf \
    --theme dark --format A4 --landscape

# Keep the intermediate HTML for debugging
python3 tools/export-canvas-pdf.py milestone-completion --keep-html

Flags: --format (Letter / Legal / Tabloid / A3 / A4 / A5), --landscape, --margin-mm, --scale, --wait-ms (bump if charts look unfinished), --keep-html (debug), --quiet, --canvas-dir (default: Cursor's canvases folder).

Requires google-chrome (or chromium) on PATH; one-time internet for the vendor step. Pure Python otherwise.

Cache maintenance is also exposed directly via tools/forgejo-sync.py:

# Pull deltas (fast, safe to run anytime)
python3 tools/forgejo-sync.py

# Truncate and resync from scratch (use after suspected force-push or schema drift)
python3 tools/forgejo-sync.py --full

# After a sync, back-fill merged_by detail for every merged PR (one-shot, permanent)
python3 tools/forgejo-sync.py --backfill

# Show cache stats without syncing
python3 tools/forgejo-sync.py --stats

Performance: on the current repo (2100+ commits, 1400+ PRs), a cold seed takes ~2.5 minutes, a steady-state delta sync takes ~7 seconds, and a 12-week weekly breakdown drops from >5 minutes (uncached, times out) to ~0.4 seconds. If the cache is ever suspected of drift, delete tools/.cache/forgejo.sqlite* and re-run — the scripts will self-seed.

All Paths Code Can Land on Master (reference)

The script accounts for all of these:

Path	Commit message marker	Detection strategy
Forgejo UI merge commit	Merge pull request 'title'	message match
Forgejo UI squash merge	none (plain commit)	SHA cross-reference
Forgejo UI rebase merge	none (N plain commits)	SHA cross-reference
Forgejo auto-merge	depends on strategy	above, as applicable
CLI git merge → push	Merge branch 'X'	message match
CLI rebase / cherry-pick / direct push	none	fallthrough = direct push
Batch-merge PR	chore(merge): batch merge of N pull requests	matched as merge commit
Integration-branch merge (PR base ≠ master, later merged to master)	varies	Phase 2 reachability check via compare/master...{merge_commit_sha} — if zero commits ahead, the PR's code is on master and emits an integration-branch-merge event

Why the script is required (the two traps it solves)

1. Squash and rebase merges are invisible from commit messages alone. They land as plain commits indistinguishable from direct pushes. Example: PR #1228 was merged by HAL9000 via squash merge with commit message fix(tests): update resource_dag.robot... — you cannot tell from the message that it was a PR merge. The script cross-references every commit SHA against the PR API's head.sha and merge_commit_sha fields to catch these.

2. The closed-PR API default sort hides recent merges of old PRs. Default order is PR number descending. A recently-merged low-numbered PR (e.g. #1228) would be on page ~400+ under default ordering. The script uses sort=recentupdate so recently-merged PRs always appear on page 1 regardless of PR number.

If you ever find yourself writing a closed-PR scan loop or merge-marker regex by hand, stop and run the script instead. Every bug we've hit in merge counting has come from bypassing it.

Correctness guarantees (as of 2026-04-25)

The script is designed so that no PR merge or commit-to-master can be silently dropped. Specifically:

• Simultaneous merges are not collapsed. Commits are grouped by second-level committer timestamp, and each distinct PR in a group gets its own event. A previous minute-level grouping silently discarded the 2nd..Nth PR when multiple PRs merged in the same minute (61 events were lost in a single 30-day window under that bug).
• Rebase merges stay as single events. When exactly one PR is matched in a second-group, every commit in that group is attributed to that PR regardless of which specific commit SHA matched. N-commit rebase merges therefore emit one event with commit_count = N, not N separate events.
• No closed-PR page cap within the window. The closed-PR scan uses sort=recentupdate and does not stop until two consecutive full pages contain zero in-window merges. There is no fixed 30-page ceiling.
• No early-exit on out-of-order commits. The commits scan continues until an entire page falls outside the window, so a single commit returned out of strict chronological order cannot truncate the scan.
• Unparseable timestamps are warned, not silently dropped.
• Network flakes are retried with exponential backoff (3 tries).
• Integration-branch merges are counted. Phase 2 checks every in-window closed+merged PR whose SHA didn't surface on the master commit stream, and emits an integration-branch-merge event when compare/master...{sha} confirms the PR's code is reachable from master HEAD. This captures PRs whose base.ref != 'master' (integration branches later merged to master via fast-forward or merge-commit) as long as the PR closed and its code actually landed on master.

Known theoretical miss paths (what the script still cannot detect)

These are fundamental API limitations, not script bugs:

• Force-pushes to master rewrite history; any commit removed from master is gone from the commits endpoint and cannot be counted. (Expected behavior.)
• In-flight race during scan. If a new commit is pushed to master while the script is paginating through commits, the page boundary shifts and the bottom of one page may be missed. Re-run the script to pick up anything missed; real incidence is negligible outside extreme push rates.
• Cherry-picked commits produce a new SHA on master that does not match any PR's head.sha or merge_commit_sha. They are correctly classified as direct pushes. If the question is "did this PR's code land on master?" via cherry-pick, the answer is yes but the PR will not appear in the list.
• Integration branch merged via squash/rebase to master. If an integration branch was squashed or rebased into master (rather than fast-forwarded or merge-committed), the integration branch's commits — including each sub-PR's merge_commit_sha — are orphaned and replaced by new SHAs on master. The Phase 2 reachability check correctly reports those PRs as "not on master" because, technically, their specific commit SHAs are no longer reachable from master even though the code is. In that case the integration PR itself is counted (it has a merge_commit_sha that IS on master), but the individual sub-PRs are not. This is the same pattern as a batch-merge PR and is correct behavior: we cannot attribute code that was squashed/rebased across layers without ambiguity.
• Multi-PR, same-second merges with mixed squash + rebase. If PR A (squash, 1 commit) and PR B (rebase, N commits) commit in the same second, the script attributes 1 commit to PR A, 1 commit to PR B (its head), and the remaining N-1 rebased commits become a separate direct-push event. Neither PR is missed, but the direct-push count is slightly inflated in this edge case. Not observed in any historical window to date.
• Merged-but-open PRs. The closed-PR scan uses state=closed. Gitea merge is atomic (merge → close) so this is not reachable in practice.

---

Merge invariant (auto-agents pipeline)

The auto-agents merge pipeline is governed by a hard correctness invariant captured by the deterministic merge driver in tools/merge_drive.py. The plan is described in detail in .cursor/plans/improve_auto-agents_pipeline_a31d3945.plan.md; this section captures the operational summary that agents need to read on every session.

The invariant (non-negotiable)

Every commit that lands on master must come from a SHA whose CI passed against the exact current master. Specifically:

1. The PR is rebased onto current master HEAD.
2. CI passes on the rebased head SHA against all required status contexts.
3. Master must not have advanced between CI completion and the merge call.
4. The merge call uses head_commit_id as an optimistic lock so any push between our check and the merge fails the merge with HTTP 409.

If any of (1)–(4) is violated, two PRs that pass individually can still combine into a broken master state. The continuous invariant verifier in tools/verify_invariant.py audits master every 15 minutes and opens an auto/invariant-violation issue on any breach.

Why the LLM merge supervisor is decommissioned

merge_drive.py is the sole merger of PRs in this repo. The historical pr-merge-supervisor / pr-merge-worker LLM pair has been removed from the managed fleet in .opencode/agents/auto-agents.md; nothing in the repo launches it anymore. The decommissioning is mandatory for the invariant to hold:

• The LLM-side merge_pr.ts calls Forgejo's merge endpoint with merge_when_checks_succeed=true. That accepts a rebased head SHA and lets Forgejo merge it whenever its CI clears, with no coordination with the driver's train-build, no bisect-on-failure, and no cooperation with the auto/claimed-merge ownership label.
• Running both concurrently would race on the same PRs, void the driver's train CI invariants (the LLM worker's automerge can fire between the driver's "CI green" check and its merge call, advancing master out from under the driver), and silently leak merges that bypass MERGE_DRIVER_RESTART_LIMIT, anti-starvation cooldowns, and the auto/needs-implementer escalation path.

The supervisor and worker prompt files (pr-merge-supervisor.md, pr-merge-worker.md) and the underlying TypeScript scripts (merge_pr.ts, rebase_pr.ts) are intentionally left on disk. They are not referenced by any agent flow today, but they remain available for two narrow, deliberate uses:

1. The operator-facing tools/claim_pr.py / claim_pr.ts flow (interactive, human-in-the-loop) shares the auto/claimed-merge label with the driver and is allowed to coexist.
2. Future implementations of the driver may shell out to merge_pr.ts for the actual POST /pulls/<n>/merge call rather than re-implement that HTTP interaction.

Do not re-add pr-merge-supervisor to auto-agents.md's managed fleet. The note in that file explains the risk inline so the constraint is visible at the point of edit.

Tier 1 mutual-respect claim contract (2026-05-04)

The merge driver's auto/claimed-merge contract has been extended to a three-way symmetric mutex so reviewer / implementer workers never comment on PRs being merged, and the merge driver never rebases a PR that an implementer or reviewer is actively working on. The contract is:

1. Workers claim before working. pr-review-worker.md (Startup step 3) and implementation-worker.md (Startup step 5, only when work_type=pr_fix) invoke claim_pr.ts with --action claim and a kind-specific TTL (1800 s for reviewer, 7200 s for implementer). The script is idempotent; re-claiming an already-claimed PR is a soft no-op.
2. Workers release before exiting. Both worker prompts have a "Release and Exit" subsection that funnels every termination path (success, early-exit, dispatcher failure) through a single claim_pr.ts --action release call.
3. Wrappers exclude claimed PRs. The 5 reviewer and 2 implementer list_prs_* wrapper scripts in .opencode/skills/auto-agents-system/scripts/ pass excludeClaimed: true so any PR carrying any of the three auto/claimed-* labels is hidden from the corresponding supervisor pool. This is the load-bearing fix for the "comments on a mid-train PR" race — once merge_drive.py adds auto/claimed-merge, the reviewer / implementer pools cannot pick that PR.
4. Driver sweeps every claim label. merge_drive.py:sweep_all_expired_claims runs at the start of every cycle and releases stale claims for all three labels (not just auto/claimed-merge). If a worker crashes between claim and release, the label is released after the TTL so the PR re-enters the candidate pool — the system is self-healing. The cycle output dict carries swept_by_label so operators can see per-label orphan rates in the cycle log.
5. Bash + skill permissions are explicit. Both worker prompts gain one bash allow (npx --yes tsx*claim_pr.ts*) and one skill allow (auto-agents-system). The pre-existing *api/v1/repos/*/labels*: deny rules are unchanged — the issue-level labels endpoint (/issues/{n}/labels) used by claim/release is not matched by that pattern.

The contract assumes both workers correctly run the release step on every termination path. The TTL sweep is the safety net, not the primary release mechanism — under normal operation, claims are released within seconds of work completing, not at the end of the TTL window.

This is the minimum viable extension of the deterministic merge driver to the reviewer / implementer pools.

Tier 2 deterministic dispatchers (2026-05-06)

The reviewer and implementer queues now have host-level Python dispatchers:

• tools/dispatch_review.py polls the same five review work groups the decommissioned pr-review-supervisor used, claims PRs with auto/claimed-reviewer, invokes pr-review-worker through OpenCode HTTP, releases the claim, and records dispatch_review_cycles telemetry.
• tools/dispatch_implementer.py polls failing-CI PRs, unaddressed REQUEST_CHANGES PRs, then open issues, claims PR work with auto/claimed-implementer, invokes implementation-worker, releases the claim, and records dispatch_implementer_cycles telemetry. Issue implementation is not claimed because no PR exists yet.

The LLM workers still perform the irreducible work: code review judgment and code generation. The dispatchers replace the long-running LLM supervisor loops (implementation-supervisor.md and pr-review-supervisor.md, both decommissioned 2026-05-09) with deterministic queueing, liveness, claim, timeout, and telemetry behavior. The legacy supervisor agent files are gone from .opencode/agents/ and the dispatchers no longer carry the coexistence-refusal guards that briefly bridged the two layers — the hard-switch was an explicit operator decision (see CHANGELOG.md 2026-05-09 entry).

Claim ownership semantics. Forgejo's label-add API is idempotent — it returns 200 whether the label was newly attached or already present — so the dispatchers cannot infer ownership from the add call alone. Before each claim, _dispatch_runtime.claim_work_item GETs the issue's labels and refuses (terminal_state="already-claimed") when any auto/claimed-* label is already attached. This means a PR already held by the merge driver, the conflict driver, or a sibling dispatcher is left untouched: the worker is not invoked and the release path is not run, so we never strip a claim we did not just attach. The pre-check has the same TOCTOU window as claim_pr.ts (documented in conflict-drive-plan.md § 3.3.1); the worker layer's idempotent claim helper deduplicates work in the rare racing case.

Worker session outcomes. The dispatchers call _opencode_worker.run_session_blocking, an outcome-agnostic entry point whose SessionResult.status is one of completed / timeout / transport-error. Reviewer / implementer workers do not need to emit the conflict-driver {outcome: resolved | unresolvable | rebase-failed} JSON to be classified as successful — a clean session exit is sufficient. The conflict driver still uses run_worker_blocking, which thin-wraps the session runner and adds the JSON outcome classification.

Cycle failure budget. Each dispatcher's outer loop tracks consecutive cycle exceptions against cycle_failure_budget (default 5, overridable via REVIEW_DISPATCHER_CYCLE_FAILURE_BUDGET / IMPLEMENTER_DISPATCHER_CYCLE_FAILURE_BUDGET). Transient failures — tsx-time exceptions, malformed JSON from a work-group script, Forgejo 4xx blips — are logged and the loop continues. Once the budget is exhausted the driver exits with code 2 so a supervising launcher (systemd, opencode-builder.sh, etc.) can restart it with fresh state, matching the behavior of merge_drive.py and conflict_drive.py.

The supervisor side of that contract ships in this repo as scripts/dispatchers-launcher.sh and contrib/systemd/cleveragents-dispatchers.service. The launcher reaps each child, restarts on non-zero exit with backoff, enforces a per-child crash-loop limit (DISPATCHERS_RESTART_LIMIT × DISPATCHERS_RESTART_LIMIT_WINDOW_S), and forwards SIGTERM for a clean shutdown so a systemctl stop lands as success.

Claim-time terminal states. In addition to the session lifecycle states above, the dispatcher records:

• claim-failed — Forgejo refused our label-add (e.g., the label is not defined on the repo).
• already-claimed — another worker already holds an auto/claimed-* label on the item; we did not claim and did not release.
• labels-fetch-failed — the labels GET returned non-200 (auth / permission / repo-gone). Treated as a foreign claim: refused rather than silently attaching a label we have no permission to manage. The HTTP status is captured in claim_result.label_fetch_status for operator triage.

Long-worker liveness contract (2026-05-07). The dispatcher's heartbeat is refreshed inside the OpenCode polling loop, not just between cycles. _dispatch_runtime.dispatch_one passes an on_poll callback to _opencode_worker.run_session_blocking; the callback fires once per status-poll iteration (every ~2 s) and writes the heartbeat file. Without this, a 30-minute review session would leave the heartbeat file untouched, and scripts/dispatchers-launcher.sh / cleveragents-dispatchers.service would treat the dispatcher as hung and SIGTERM it mid-cycle — orphaning the OpenCode session and the auto/claimed-* lock. Callback exceptions are logged and swallowed inside run_session_blocking, so a transient heartbeat-write failure (disk full, EROFS, EACCES) cannot mask a successful worker completion. The same wiring is plumbed into tools/conflict_drive.py for the conflict-resolver-worker invocation.

In-flight cycle row visibility (2026-05-07, schema v5). The Tier 2 cycle tables (dispatch_review_cycles, dispatch_implementer_cycles) carry a row from cycle start, not from cycle end. _dispatch_runtime.run_one_cycle calls begin_cycle (INSERT with ended_at IS NULL) before any worker dispatch and finish_cycle (UPDATE) in a try/finally so a crash mid-cycle still resolves the row. The schema migration drops the ended_at NOT NULL constraint and adds a UNIQUE INDEX on cycle_id; pre-existing rows are preserved verbatim. The /api/health telemetry endpoint surfaces the most-recent in-flight cycle per dispatcher (in_flight_cycle: {cycle_id, started_at, session_id, elapsed_s}) and a running_long_worker flag that fires when the heartbeat is older than 600 s AND a matching pid is alive — points at a callback regression rather than a normal long worker. The Drivers and Overview tabs render in-flight rows with a tinted background and an "in flight" pill so a stuck cycle is impossible to miss.

Cross-cutting agent instructions (2026-05-07)

opencode.json carries an instructions array that OpenCode appends to every session's system prompt — every agent, every subagent, every tier dispatcher. Use it for rules that must hold uniformly across the fleet so we do not have to copy/paste the same section into 42 agent definition files.

The wired location in this repo is .opencode/instructions/*.md. Each file in that directory is a single, focused rule set:

• .opencode/instructions/bash-commands.md — explains how the OpenCode bash permission engine matches against the raw, unexpanded command string and why chained commands (&&, ||, ;, |), command substitution ($(...)), and bare variable assignments (WORK_DIR=...) always fall through to the default deny rule. Includes wrong / right examples and a recovery recipe for permission denied errors.

When you add a new file under .opencode/instructions/, the existing glob in opencode.json picks it up automatically — no further wiring needed. OpenCode reads instructions at session creation, so a running OpenCode server must be restarted (or the file must already exist before its agent sessions are spawned) for changes to take effect. Restart the OpenCode server after adding or editing files here.

When to add a file here vs. updating an individual agent: if the rule applies to every agent that uses the affected tool (bash, edit, webfetch, etc.) put it here. If it is specific to a single agent's domain (e.g., the reviewer's review-format requirements), keep it in that agent's .opencode/agents/<name>.md body.

Tier 0A audit findings (2026-05-02)

The audit in tools/forgejo_audit.py was run against cleveragents/cleveragents-core and produced these findings:

• dismiss_stale_approvals is already false on the master branch protection rule. 1A is a no-op; the reapprove-if-clean-rebase fallback is not required. No follow-up todo is generated.
• Canonical merge-bot identity is Forgejo <forgejo@cleverthis.com>. Any commit on master whose committer email is anything else is a direct push (Path B revert candidate, or unauthorized — 2E alerts unconditionally).
• merge_pr.ts sets merge_when_checks_succeed=true unconditionally. This is the documented invariant-violation risk — verified empirically in 0A as part of pre-condition P1.
• default_delete_branch_after_merge is false on the repo. The merge driver and tools/migrate_to_new_driver.py sweep abandoned auto/train/<ts> branches older than 2 hours.
• Required CI contexts on master include: CI / lint, CI / typecheck, CI / unit_tests, CI / integration_tests, CI / e2e_tests, CI / coverage, plus CI / build, CI / docker, CI / quality, and CI / security (with wildcards). The driver fetches this list from branch_protections/master at startup and refreshes every 15 min — it does not hardcode the set.
• push_whitelist_usernames is [freemo] — only Jeff can direct-push to master. HAL9000 / HAL9001 cannot, by branch protection.

To re-run the audit: python3 tools/forgejo_audit.py. The empirical pre-conditions (P1, P2, P5) require live test PRs and are listed in the script's report output for manual verification — they are intentionally not automated against the production repo.

Path A vs Path B revert procedures

When 2E opens an auto/invariant-violation issue, the bad commit must be backed out:

• Default — Path A (revert PR through 0B). Open a new PR with git revert <bad_sha> against master, labelled auto/revert. The operator (typically Jeff) approves the revert PR manually — no automated rubber-stamp. The driver picks it up on the next loop and merges it through the standard 0B path. Latency: one full cycle (~30-45 min). Invariant-clean: the revert is CI-validated against the broken master.
• Operator override — Path B (direct revert push, emergency only). Used when the operator suspects the merge driver itself is the cause of the violation, or when Path A latency is unacceptable (e.g., master CI is broken for everyone). Stop the driver (systemctl stop merge-driver), then git revert <bad_sha> && git push origin master from a privileged identity (Jeff's account, with branch-protection bypass). Restart the driver only after the violation root cause is understood. Latency: < 5 min. 2E will alert on the direct-push commit (it has no exemption logic). The operator manually closes the alert after confirming the revert worked, OR 2E's auto-close function (same cron) closes it automatically on the next 15-min tick once the bad commit's effect is no longer in master.

The Path B privileged identity is freemo (Jeff), per the push whitelist.

Driver environment variables

The merge driver in tools/merge_drive.py reads these environment variables. All except GITEA_TOKEN are optional — sensible defaults are baked in. The list mirrors _load_env() and Config in that file; if you add a new env var to the driver, add it here too.

Env var	Required	Default	Purpose
GITEA_TOKEN	yes	—	Forgejo PAT for the merge-bot identity used for read access and merge calls. Falls back to a GITEA_TOKEN=... line in ~/.cleveragents/forgejo-token if the env var is unset.
FORGEJO_REVIEWER_PAT	yes when MERGE_DRIVER_MAX_N >= 2	—	Separate PAT used to auto-approve umbrella train PRs (Tier 2D). Should belong to a reviewer-bot identity distinct from the merge-bot so the umbrella PR satisfies the "reviewer ≠ author" branch protection rule.
MERGE_DRIVER_MAX_N	no	1	Max train size. 1 = single-PR mode (the safe rollout default). Raise to 2 only after 0B has been stable ≥ 1 week with zero auto/invariant-violation issues, then to 5, then to 10 once 1B telemetry shows clean-batch rate ≥ 70 %.
MERGE_DRIVER_CI_TIMEOUT_S	no	3600 (1 h)	Hard ceiling for waiting on CI of a rebased head SHA before releasing the PRs back to the pool with auto/ci-timeout. Re-tune to roughly 2 × P95(CI completion) once 1B has a week of data.
MERGE_DRIVER_CI_POLL_INTERVAL_S	no	30	Sleep between status-check polls while waiting for CI on a rebased SHA.
MERGE_DRIVER_POLL_INTERVAL_S	no	30	Outer-loop sleep when pick_candidates returned nothing.
MERGE_DRIVER_LOCK_PATH	no	merge-driver.lock	Path for the single-instance fcntl.flock. Resolved with the fallback chain ${MERGE_DRIVER_LOCK_PATH} → ${XDG_RUNTIME_DIR}/merge-driver.lock → /tmp/merge-driver.lock.
MERGE_DRIVER_HEARTBEAT_PATH	no	merge-driver.heartbeat	Liveness file the driver touches every loop iteration. Same fallback chain as the lock.
MERGE_DRIVER_WORK_DIR	no	${TMPDIR}/merge-driver-work	Persistent local clone the driver uses for rebases and fast-forward checks. Wiped and re-cloned on a corrupt-state error; never assume contents survive a restart.
MERGE_DRIVER_COOLDOWN_MINUTES	no	30	How long a PR is excluded after being released as ci-timeout / restart-throttled. Anti-starvation: PRs that fail repeatedly are dropped to the back of the queue rather than blocking the train.
MERGE_DRIVER_API_RETRIES	no	3	Retry attempts for idempotent reads. State-changing writes are retried at most once and only on connection-level errors.
MERGE_DRIVER_REQUEST_TIMEOUT_S	no	30	Per-request timeout for every Forgejo API call.
MERGE_DRIVER_RESTART_LIMIT	no	8	Max number of CI runs a single train cycle may invalidate (master-advanced restarts + 409 retries). When exhausted the cycle releases as restart-budget-exhausted and the constituent PRs receive auto/restart-throttled.
MERGE_DRIVER_CLAIM_TTL_SECONDS	no	MERGE_DRIVER_CI_TIMEOUT_S + 1800	TTL on auto/claimed-merge. The sweep_expired_claims pass at the start of every cycle releases any claim comment older than this so a crashed predecessor doesn't leave PRs stuck claimed.
MERGE_DRIVER_LOG_LEVEL	no	INFO	Python logging level for merge_drive.py: DEBUG/INFO/WARNING/ERROR. Stderr is the destination unless a parent process (systemd, supervisord) has already configured the root logger.
VERIFY_INVARIANT_LOG_LEVEL	no	INFO (or DEBUG if --progress is passed)	Python logging level for verify_invariant.py. The --progress CLI flag flips the default to DEBUG; setting this env var overrides both.

The Tier 2 dispatchers follow the same conventions: REVIEW_DISPATCHER_* and IMPLEMENTER_DISPATCHER_* cover LOCK_PATH, HEARTBEAT_PATH, CYCLE_SECONDS, MAX_ITEMS_PER_CYCLE, WORKER_TIMEOUT_SECONDS, CLAIM_TTL_SECONDS, API_RETRIES, REQUEST_TIMEOUT_S, SCRIPT_TIMEOUT_SECONDS, CYCLE_FAILURE_BUDGET, and LOG_LEVEL. dispatch_review.py reads FORGEJO_REVIEWER_PAT (falling back to GITEA_TOKEN for local tests); dispatch_implementer.py reads FORGEJO_PAT (falling back to GITEA_TOKEN). Both read OPENCODE_SERVER_URL (default http://127.0.0.1:4096) and write heartbeat files through the same fallback chain as merge_drive.py.

For deterministic-dispatcher runs that don't need the watchdog session, start OpenCode in server-only mode: OPENCODE_BUILDER_SERVER_ONLY=1 bash scripts/opencode-builder.sh. That keeps the HTTP worker API available (the dispatchers need it to run the LLM workers per work item) without spinning up the auto-agents watchdog session at all. Skipping the watchdog is fine in deployments where the host's process supervisor and log aggregator are already monitoring dispatcher health.

Bot-generated commit conventions

The merge driver commits via the Forgejo API merge endpoint. Specific commits it generates (umbrella PR titles, train branch commits, etc.) are exempted from strict conventional-commit footer rules:

• Umbrella PR title format: chore(merge): batch merge of N pull requests (#A, #B, #C).
• Bot-authored commits (committer email is the merge-bot identity above) do not require an ISSUES CLOSED: #N footer — they are mechanical merge artifacts, not feature work. The reviewer worker's footer check exempts them via the committer-email match.

Operational tooling

These scripts implement the pipeline. Cross-link them so newcomers can discover the system from AGENTS.md alone:

• tools/forgejo_audit.py — read-only Tier 0A audit. Re-run before any rollout step to confirm Forgejo pre-conditions still hold.
• tools/setup_auto_labels.py — provisions the auto/* label set at org level. Idempotent; safe to re-run.
• tools/setup_branch_protection.py — Tier 0C: enforces push_whitelist_usernames=[freemo] so only Jeff (and Path B reverts) can land commits on master without going through the driver.
• tools/audit_branch_protection.py — Tier 1A: dry-run / live flip of dismiss_stale_approvals.
• tools/migrate_to_new_driver.py — Tier 0B.5 one-shot: clears stale auto/claimed-* labels, cancels merge_when_checks_succeed schedules, sweeps abandoned auto/train/<ts> branches > 2 h old.
• tools/merge_drive.py — Tier 0B + 2D: the deterministic merge driver. Single-instance lock + heartbeat.
• tools/conflict_drive.py — Tier 1.5: deterministic conflict-resolution driver. Owns the auto/needs-conflict-resolution label, runs git rebase against master in a fresh clone, delegates to the conflict-resolver-worker subagent only when the rebase actually conflicts, force-pushes with --force-with-lease, escalates to auto/needs-implementer after three definite failures in 24h. Single-instance lock + heartbeat.
• tools/dispatch_review.py — Tier 2: deterministic reviewer dispatcher. Polls existing review work-group wrappers, claims auto/claimed-reviewer, invokes pr-review-worker, releases claims, and records dispatch_review_cycles.
• tools/dispatch_implementer.py — Tier 2: deterministic implementer dispatcher. Polls existing implementer / issue work-group wrappers, claims auto/claimed-implementer for PR work, invokes implementation-worker, releases claims, and records dispatch_implementer_cycles.
• tools/inject_synthetic_conflict.py — local-only test helper: opens a PR on a fork with a guaranteed add/add or modify-vs-stub conflict (modes: trivial, multi-commit, unresolvable). Used by Phase B fork-mode validation and the opt-in tests/auto_agents/integration/test_conflict_resolution_e2e.py.
• tools/verify_invariant.py — Tier 2E: cron job (every 15 min) that audits new master commits and opens or auto-closes auto/invariant-violation issues.
• tools/flag_stale_prs.py — Tier 1C: flags PRs idle ≥ 28 days with unaddressed REQUEST_CHANGES. Assigns freemo and applies auto/stale-inactivity.
• tools/local_ci_gate.sh — Tier 1G: wraps the canonical 6-gate nox pipeline (lint → typecheck → unit_tests → integration_tests → e2e_tests → coverage_report). Run before pushing any PR branch. --fast skips the slowest two gates for the inner loop; a final full run is still required before commit.
• .opencode/skills/auto-agents-system/scripts/claim_pr.ts — Tier 2B: claim/release helper that the deterministic dispatchers (tools/dispatch_review.py, tools/dispatch_implementer.py) and tools/merge_drive.py invoke to mark a PR with an auto/claimed-{merge,implementer,reviewer} label and TTL comment. Operator workflows can also invoke it directly via tools/claim_pr.py. list_prs.ts and every list_prs_* wrapper accept the matching --exclude-claimed opt-in flag (off by default).
• tools/setup_test_fork.py — sentinel testing: bootstrap a personal fork (creates the fork via the Forgejo forks endpoint, mirrors branch protection, provisions auto/* labels at the repo scope). Idempotent.
• tools/duplicate_prs_to_fork.py — sentinel testing: duplicates the N most-recent open upstream PRs (head branch + body + safe-listed labels) into the personal fork under tests/sentinel-<N>-<safe-branch> and tags them auto/sentinel. Use to populate the fork before exercising the driver / verifier / preflight against an isolated workspace.
• tools/preflight_phase01.sh — pre-production validation harness; honours FORGEJO_OWNER, FORGEJO_REPO, FORGEJO_API_BASE so the same script can be pointed at the canonical repo or a personal-fork test repo.

Targeting a personal-fork test repo

All Python pipeline tools (merge_drive.py, conflict_drive.py, dispatch_review.py, dispatch_implementer.py, verify_invariant.py, forgejo_audit.py, audit_branch_protection.py, flag_stale_prs.py, setup_auto_labels.py, setup_branch_protection.py, migrate_to_new_driver.py, inject_synthetic_conflict.py) read FORGEJO_OWNER / FORGEJO_REPO / FORGEJO_API_BASE (and FORGEJO_ORG / FORGEJO_DEFAULT_BRANCH / FORGEJO_MERGE_BOT_EMAILS / FORGEJO_PUSH_WHITELIST where applicable) at module-import time. Defaults preserve current canonical-repo behaviour. Set them to retarget the entire stack:

export FORGEJO_OWNER=drew
export FORGEJO_REPO=cleveragents-core
export FORGEJO_PUSH_WHITELIST=drew
# all subsequent invocations now hit drew/cleveragents-core

The verifier's cursor file is automatically partitioned by <owner>.<repo> so a fork run never clobbers the canonical cursor.

Forgejo API quirks observed during sentinel rollout

• GET /repos/{owner}/{repo}/pulls?head=owner:branch is silently ignored by Forgejo (unlike GitHub). The endpoint returns every open PR. Code that needs to find a PR by head-branch must list pulls page-by-page and match head.ref client-side (see existing_fork_pr in tools/duplicate_prs_to_fork.py).
• Newly created forks return HTTP 202 Accepted from POST /repos/<upstream>/forks — the fork is not queryable immediately. tools/setup_test_fork.py:wait_for_fork polls until GET /repos/<owner>/<repo> returns 200 before applying branch protection or labels.
• Forks have has_actions: false and has_issues: false by default. Workflow YAML files at .forgejo/workflows/*.yml are copied with the repo, but with has_actions=false no runs ever fire on push or pull_request. With has_issues=false, verify_invariant.py cannot file auto/invariant-violation issues. setup_test_fork.py PATCHes both to true after fork creation. The Actions API endpoint (/actions/runs, /actions/runners) returns HTTP 404 ("target couldn't be found") while has_actions=false, which is a useful signal — once it returns {"workflow_runs":[],"total_count":0} you know the runtime is on.
• Required-check contexts use a wildcard convention. Forgejo appends (push) or (pull_request) to every status context name, so upstream's branch protection on master uses entries like CI / lint* — the trailing * absorbs the event suffix. The setup_test_fork.py mirror logic preserves the wildcards verbatim; merge_drive.py:required_checks_passed matches them via fnmatch.
• Forgejo status field is status, not state. GitHub Statuses API uses state; Forgejo uses status. merge_drive.py reads s.get("status") or s.get("state") to handle both. New tools that consume statuses must do the same.
• Forks do not inherit org-level labels. setup_test_fork.py always provisions the canonical auto/* set at the repo scope so the driver / reviewer / implementer can set/read their labels.
• Upstream repository labels (e.g. Type/Task) are not auto-created in the fork; duplicate_prs_to_fork.py only attaches labels that already exist in the fork. This is by design — sentinels need only the auto/* operational labels to exercise the pipeline.

Bootstrap env-var convention

setup_test_fork.py is the bootstrap for the test environment, so it deliberately uses separate env vars from the rest of the pipeline to avoid the chicken-and-egg ambiguity of "which repo are we talking about right now":

Var	Read by	Purpose
FORGEJO_UPSTREAM_OWNER / FORGEJO_UPSTREAM_REPO	setup_test_fork.py only	The repo being forked from. Defaults to canonical cleveragents/cleveragents-core. CLI override: --upstream-owner / --upstream-repo.
FORGEJO_OWNER / FORGEJO_REPO	every other tool (driver, verifier, preflight, duplication, etc.)	The repo being operated on. Set to your fork (drew) when running fork-mode preflight or sentinel exercises.

The intended workflow is:

# 1) Bootstrap the fork ONCE with drew's admin PAT (no .env override
#    needed at this step — drew creates the fork and promotes HAL9000
#    to admin via PUT /collaborators).
GITEA_TOKEN=<drew-admin-pat> tools/setup_test_fork.py

# 2) From here on, every downstream tool runs as HAL9000 via the
#    launcher (matches Jeff's confirmed production env).
source tools/launch_fork.sh
export FORGEJO_PUSH_WHITELIST=drew
tools/preflight_phase01.sh
tools/duplicate_prs_to_fork.py --count 30

For step 2 there is also a one-liner helper that pins all the fork-mode env vars at once and validates the target repo before exporting anything:

source tools/launch_fork.sh
# Repo target:
#   FORGEJO_OWNER, FORGEJO_REPO, FORGEJO_URL, FORGEJO_API_BASE,
#   FORGEJO_DEFAULT_BRANCH, FORGEJO_ORG.
# Auto-agents env contract (all 8 required by .opencode/agents/auto-agents.md):
#   GIT_USER_NAME, GIT_USER_EMAIL,
#   FORGEJO_PAT, FORGEJO_USERNAME, FORGEJO_PASSWORD,
#   FORGEJO_REVIEWER_PAT, FORGEJO_REVIEWER_USERNAME, FORGEJO_REVIEWER_PASSWORD.
# Driver-side compat (synthesized from FORGEJO_PAT):
#   GITEA_TOKEN — so the deterministic Python tools that still read
#                 this name authenticate as HAL9000 too. Pre-export
#                 GITEA_TOKEN before sourcing to override.
# Optional pass-through:
#   CA_MAX_PARALLEL_WORKERS.
# The launcher resolves canonical names (FORGEJO_*, GIT_*) from the
# HAL_9000_*/HAL_9001_* aliases in .devcontainer/.env, derives
# FORGEJO_USERNAME / FORGEJO_REVIEWER_USERNAME via /api/v1/user whoami,
# and refuses to export anything if (a) any required value is unresolved
# or (b) the fork target fails a `GET /repos/<owner>/<repo>` validation
# check (must exist, be a fork, grant push to the calling user).
# Override the fork target with:
#   FORK_OWNER=... FORK_REPO=... source tools/launch_fork.sh

Always prefer source tools/launch_fork.sh over hand-exporting the fork-mode vars. Two reasons:

1. The LLM orchestrator (.opencode/agents/auto-agents.md) falls back to git remote get-url origin for FORGEJO_OWNER/FORGEJO_REPO when its env vars are unset. The canonical clone's origin points at upstream, so a fork-mode launch with stale environment silently targets upstream.
2. The auto-agents agent will exit immediately if any of its 8 required env vars are missing (GIT_USER_NAME, GIT_USER_EMAIL, FORGEJO_PAT, FORGEJO_USERNAME, FORGEJO_PASSWORD, FORGEJO_REVIEWER_PAT, FORGEJO_REVIEWER_USERNAME, FORGEJO_REVIEWER_PASSWORD). The launcher surfaces that failure up front instead of 30 minutes into a run.

After sourcing, the canonical LLM-side launcher is scripts/opencode-builder.sh (which boots an OpenCode server, opens an auto-agents session, and keeps it nudged forever). The driver side is python3 tools/merge_drive.py. Both must be running for end-to-end coverage.

Two-identity convention (production-fidelity)

The pipeline runs on two Forgejo identities — the same two production runs with, per Jeff's confirmed env at 2026-05-05. There is no "operator admin" third identity; everything goes through HAL9000 + HAL9001:

Identity	Forgejo account	Repo permission	env vars set by launcher	.env keys consumed
Primary bot (HAL9000)	HAL9000	admin	FORGEJO_PAT, FORGEJO_USERNAME, FORGEJO_PASSWORD, GIT_USER_NAME, GIT_USER_EMAIL, GITEA_TOKEN (synthesized)	HAL_9000_API_KEY, HAL_9000_FORGEJO_PASSWORD, GIT_USER_NAME, GIT_USER_EMAIL
Reviewer bot (HAL9001)	HAL9001	write	FORGEJO_REVIEWER_PAT, FORGEJO_REVIEWER_USERNAME, FORGEJO_REVIEWER_PASSWORD	HAL_9001_API_KEY, HAL_9001_FORGEJO_PASSWORD

HAL9000 does everything — LLM-worker git activity (commits, comments, reviews), the deterministic merge driver (tools/merge_drive.py), the verifier, and every preflight script. HAL9001 has exactly one job: auto-approving HAL9000's umbrella PRs (Forgejo blocks self-approval, so this must be a different account from HAL9000).

The launcher synthesizes GITEA_TOKEN=$FORGEJO_PAT so the deterministic Python tools (which still read GITEA_TOKEN by name) authenticate as HAL9000. To override with a different identity (e.g. drew's admin PAT for a one-time fork-setup task), pre-export GITEA_TOKEN before sourcing the launcher.

The LLM agents use the web password (*_PASSWORD) for two purposes documented in .opencode/skills/forgejo-api/: basic-auth token CRUD on /users/{username}/tokens, and cookie-based web login for CI-action log scraping (REST API does not expose action logs).

FORGEJO_USERNAME / FORGEJO_REVIEWER_USERNAME are auto-derived from each PAT via /api/v1/user whoami if unset, so the operator only has to set the PATs and passwords.

Fork-mode prerequisites the launcher does not check (operator must ensure these once, up front):

1. HAL9000 must be added as an admin collaborator on the fork (PUT /repos/<owner>/<repo>/collaborators/HAL9000 with {"permission": "admin"}). Production has HAL9000 as admin on cleveragents/cleveragents-core, and audit_branch_protection.py plus setup_branch_protection.py --dry-run in the preflight read /branch_protections, which is admin-only. Without admin those steps fail with HTTP 403.
2. HAL9001 must be added as a write collaborator on the fork (same endpoint, "permission": "write"). HAL9001 only auto-approves PRs, which is a write-level operation — it does not need admin.
3. The fork's apply_to_admins branch protection should be true (PATCH the branch-protection rule). Without it HAL9000 — which is admin on the fork — would silently bypass required-status-check gates and the merge driver would land merges that haven't passed CI. With apply_to_admins=true HAL9000 follows the same rules every other contributor does.

All three prerequisites are documented as one-time setup in CHANGELOG.md under the prelaunch fork environment fixes entry.

Production launch story (and how to reproduce it against a fork)

There is no production launch config (cron, systemd, Compose, workflow) checked into this repo. The deployed schedules and unit files live on whichever operator host runs the canonical pipeline. What the repo does tell us:

Component	Production invocation pattern	Trigger
tools/merge_drive.py	Long-running daemon: python3 tools/merge_drive.py until SIGTERM	Hinted as a host-level systemd unit named merge-driver (only the systemctl stop merge-driver reference exists in this repo; the unit file does not)
tools/verify_invariant.py	Single-shot per invocation: python3 tools/verify_invariant.py [--mode observe|alert] [--dry-run]	Cron @ 15-min cadence (documented narratively; cron entry is on the operator host)
tools/flag_stale_prs.py	Single-shot: python3 tools/flag_stale_prs.py [--days N]	Cron @ "every few hours" (cadence documented narratively; entry is on the operator host)
Reviewer / Implementer dispatchers	Long-running deterministic Python daemons: tools/dispatch_review.py and tools/dispatch_implementer.py (started together by scripts/dispatchers-launcher.sh or the equivalent systemd unit). They invoke the LLM workers (pr-review-worker, implementation-worker) per work item via OpenCode HTTP. The legacy pr-review-supervisor.md / implementation-supervisor.md LLM agents that previously owned this orchestration were decommissioned 2026-05-09 — their files are gone from .opencode/agents/. Merge orchestration is its own deterministic daemon, tools/merge_drive.py.	Hinted as host-level systemd units; the dispatcher launcher is scripts/dispatchers-launcher.sh and a sample unit ships in contrib/systemd/cleveragents-dispatchers.service.
auto-agents watchdog	Single OpenCode session running the watchdog-only auto-agents agent. Reads dispatcher heartbeat files and logs WARNING when either goes stale. Does NOT restart anything (host-level process supervisor's job).	scripts/opencode-builder.sh (creates/attaches to OpenCode and starts the session).

To reproduce the production pipeline against the fork, source the launcher once per shell and run the same commands the production host runs, in three separate shells:

# All three shells start with the same preamble. Both the driver and
# the LLM fleet authenticate as HAL9000 (production-fidelity); HAL9001
# is used only for the umbrella-PR auto-approval path inside the
# driver.
source tools/launch_fork.sh
export FORGEJO_PUSH_WHITELIST=drew

# Shell A — driver daemon (deterministic merges + invariant proofs)
python3 tools/merge_drive.py     # hold open

# Shell B — verifier on a 15-min cron equivalent
while true; do
  python3 tools/verify_invariant.py --mode alert
  sleep 900
done

# Shell C — Reviewer + implementer dispatchers (deterministic Python).
# These are the daemons that replaced the legacy LLM supervisors
# (decommissioned 2026-05-09). The launcher reaps each child and
# applies a per-child crash-loop limit.
scripts/dispatchers-launcher.sh  # hold open; SIGTERM for clean stop

# Shell C' (optional) — auto-agents watchdog session. The dispatchers
# above own all orchestration; this session only watches their
# heartbeat files and logs WARNINGs when either goes stale. NOT
# required for correct operation — the host-level process supervisor
# (systemd / runit / docker) is the one that actually restarts
# dispatchers; this is just an in-cluster log channel.
scripts/opencode-builder.sh      # hold open; Ctrl+C to stop

# Shell D — conflict-resolution driver daemon (deterministic rebase +
# LLM-assisted conflict resolution; release path is also deterministic)
python3 tools/conflict_drive.py  # hold open

# Shell E (optional but strongly recommended) — telemetry console.
# Loopback-only by default; consolidates daemon health, cycle
# telemetry, PR label flow, OpenCode sessions, and token/cost into
# a single browser tab. Reads the same per-(owner,repo) cache the
# drivers write to.
python3 .opencode/telemetry/server.py   # http://127.0.0.1:8765

Shell E — pipeline telemetry console

/.opencode/telemetry/server.py is a stdlib-only HTTP server that exposes everything the drivers emit (SQLite merge_cycle / conflict_drive_cycles / llm_activity, daemon PIDs + heartbeats, live Forgejo PRs by auto/* label, OpenCode sessions, token+cost rollup) in a tabbed web UI. Source the same launcher the drivers use so the console reads the same per-(owner, repo) cache file the drivers wrote to:

source tools/launch_fork.sh                     # or canonical-mode env
python3 .opencode/telemetry/server.py           # 127.0.0.1:8765

Per-tab polling — inactive tabs don't fetch — server load on a long-lived browser session stays flat. Default bind is loopback-only; exposing on a public interface requires putting auth in front via a reverse proxy. Console never writes to Forgejo, OpenCode, or the SQLite cache. Full operator notes: .opencode/telemetry/README.md.

Shell D — conflict-resolution driver

tools/conflict_drive.py is the deterministic sibling of merge_drive.py that owns the auto/needs-conflict-resolution label end-to-end. Each cycle it picks the oldest open PR carrying that label, runs a deterministic git rebase origin/master in a fresh clone, and only delegates to conflict-resolver-worker (a single-shot OpenCode subagent) when the rebase actually conflicts. The driver verifies the resolution, force-pushes with --force-with-lease, and removes the label. On unrecoverable cases (e.g. a delete-vs-rewrite conflict the worker cannot resolve semantically) it commits the conflict markers, retains the label, and after three definite failures in 24h escalates to auto/needs-implementer for a human implementer.

Operator-facing notes:

1. Committer attribution. The resolution commit is authored by the PR author (preserved) but committed as Forgejo (matches merge_drive convention; HAL9000 is the bot identity for the merge, not the rebase).
2. unresolvable outcome. When the worker exits unresolvable the driver still pushes a branch, but that branch contains the conflict markers committed verbatim. CI will fail; merge_drive.py will refuse to merge it. The point of this mode is to give a human implementer the exact rebase frontier to fix in one place — not five separate PRs whose git log no longer matches master's tip.
3. Approval drift. If a PR loses approval mid-cycle (reviewer dismisses, author force-pushes), conflict_drive.py still resolves the conflict. The merge gate stays at merge_drive.py, which will refuse the merge until approvals are re-collected. The two drivers are intentionally independent on this axis.

Required TTL constraint

MERGE_DRIVER_CLAIM_TTL_SECONDS must be at least CONFLICT_DRIVER_WORKER_TIMEOUT_SECONDS + 300. conflict_drive.py asserts this at startup and aborts (sys.exit(2)) if the constraint is violated. The 300 s buffer covers the deterministic rebase, the post-worker verification (git status / git diff --check / git merge-base --is-ancestor), the force-with-lease push, and label cleanup. Without this buffer, the merge driver could time out a PR's claim while the conflict driver is still in the verification phase, opening a window for two drivers to push to the same head.

auto/blocked-by-deps semantics

merge_drive.pr_is_eligible queries /issues/{n}/blocks and applies auto/blocked-by-deps (orange d93f0b) when the PR has any open blocking issue. The driver removes the label automatically the moment all dependencies close — operators do not need to flip the label manually to unstick a PR. The label exists for visibility (it explains to the reviewer fleet why a PR is being skipped) and to keep the release-path scoreboard accurate; the merge driver short-circuits on the dependency check before consuming any further budget on that PR.

How to run the local conflict-driver integration test

The integration test in tests/auto_agents/integration/test_conflict_resolution_e2e.py is opt-in and requires a Forgejo PAT — it is intentionally not wired into Forgejo CI (the Actions runner has no fork PAT). Run it locally against your fork:

export CONFLICT_DRIVER_INTEGRATION_TEST=1
export FORGEJO_OWNER=<your-fork-owner>     # e.g. drew
export GITEA_TOKEN=<your-fork-PAT>         # write access to the fork

pytest -xvs tests/auto_agents/integration/test_conflict_resolution_e2e.py

The test uses tools/inject_synthetic_conflict.py to open a guaranteed conflict PR, runs python3 tools/conflict_drive.py --once against it, polls until auto/needs-conflict-resolution clears, and asserts the head SHA advanced. A finalizer closes the synthetic PR and deletes the head branch on test exit (success or failure).

If the test process is killed with SIGKILL (or your machine reboots) before the finalizer fires, clean up the orphaned PR by hand:

gh -R <fork-owner>/cleveragents-core pr close <pr-number> --delete-branch

`scripts/opencode-builder.sh` boots an OpenCode server (or attaches to
one already running on `127.0.0.1:4096`), creates a session with the
`auto-agents` agent, and runs an infinite `continue`-nudge loop that
also dismisses any blocking question prompts the agent raises. As of
2026-05-09 the `auto-agents` agent is a watchdog-only session — it
reads dispatcher heartbeat files and logs WARNINGs; it does NOT
launch or restart anything. The reviewer / implementer dispatchers
themselves run as deterministic Python daemons in Shell C (above).
Merge orchestration is **intentionally absent** from any LLM fleet —
it is performed by the deterministic `merge_drive.py` daemon in
Shell A.

#### Bot-identity considerations for fork-mode

`tools/merge_drive.py` always sets the local-clone git config to
`user.name=Forgejo` / `user.email=forgejo@cleverthis.com` regardless
of which token is used. Combined with `verify_invariant.py`'s
default `MERGE_BOT_EMAILS={"forgejo@cleverthis.com"}`, fork-mode
runs do **not** need an `FORGEJO_MERGE_BOT_EMAILS` override —
driver-merged commits are correctly recognised as bot-merges in
both modes.

The smoke commit on the sentinel branch (committed as
`drew <drew.morris@cleverthis.com>`) is *not* a driver-style merge;
it was a manual no-op push to verify Actions fired. The driver's
own commits will be authored as the Forgejo bot identity.

The reviewer and implementer dispatchers
(`tools/dispatch_review.py`, `tools/dispatch_implementer.py`) read
their `--owner` / `--repo` from `FORGEJO_OWNER` / `FORGEJO_REPO`
(`launch_fork.sh` exports these to point at the fork), so a
fork-mode dispatcher launched in a separate shell sees only PRs in
the fork and never collides with the production-canonical
dispatchers running on the operator host. The legacy
`implementation-supervisor.md` and `pr-review-supervisor.md` agents
that previously hard-coded `cleveragents/cleveragents-core` were
deleted 2026-05-09; fork-mode isolation for them is moot — they no
longer exist. (`pr-merge-supervisor.md` was decommissioned earlier
in the same vein; same conclusion.)

### Auto-* label registry

Provisioned by `tools/setup_auto_labels.py`. Mutate this list in the
script, not by hand:

| Label | Set by | Cleared by | Meaning |
|-------|--------|------------|---------|
| `auto/claimed-merge` | `merge_drive.py` / `claim_pr.ts` | end of cycle / `claim_pr.ts release` / driver TTL sweep | Currently being processed by the merge driver. Swept by `merge_drive.py:sweep_all_expired_claims` every cycle. |
| `auto/claimed-implementer` | `implementation-worker.md` (Tier 1: Startup step 5) via `claim_pr.ts` | `implementation-worker.md` "Release and Exit" / driver TTL sweep | Currently being processed by an implementer worker. Set on entry, cleared on exit; the merge driver's `sweep_all_expired_claims` releases stuck labels after `claim_ttl_seconds`. |
| `auto/claimed-reviewer` | `pr-review-worker.md` (Tier 1: Startup step 3) via `claim_pr.ts` | `pr-review-worker.md` "Release and Exit" / driver TTL sweep | Currently being processed by a reviewer worker. Set on entry, cleared on exit; the merge driver's `sweep_all_expired_claims` releases stuck labels after `claim_ttl_seconds`. |
| `auto/ci-timeout` | `merge_drive.py` (release path) | next merge attempt that fully completes | Last cycle hit `MERGE_DRIVER_CI_TIMEOUT_S`; PR excluded for `MERGE_DRIVER_COOLDOWN_MINUTES`. |
| `auto/restart-throttled` | `merge_drive.py` (Budget exhaustion) | next clean cycle | Train repeatedly lost master-tempo races. Same cooldown as above. |
| `auto/needs-implementer` | `pr-review-worker.md` (Tier 1F) / `conflict_drive.py` (3rd definite failure in 24h) | next implementer commit | ≥ 5 `REQUEST_CHANGES` cycles, or CI failing on a non-trivial pattern, or conflict driver gave up after the 24h definite-failure budget was exhausted. Escalation flag for `tier-dispatcher`. |
| `auto/needs-conflict-resolution` | `merge_drive.py` (rebase failed during merge cycle) | `conflict_drive.py` on successful resolution + force-push | PR is rebase-blocked against current master. Owned by `tools/conflict_drive.py`. |
| `auto/blocked-by-deps` | `merge_drive.py:pr_is_eligible` (via `_pr_has_open_dependencies`) | `merge_drive.py:pr_is_eligible` (auto-cleanup when all blockers close) | One or more open issue dependencies (`/issues/{n}/blocks`). Removed automatically next cycle when the last blocker closes — operators do not need to flip this manually. |
| `auto/stale-inactivity` | `flag_stale_prs.py` | new commit on PR | PR idle ≥ 28 days with unaddressed `REQUEST_CHANGES`. Assigned to `freemo`. |
| `auto/invariant-violation` | `verify_invariant.py` (issue label) | `verify_invariant.py` auto-close, or operator | Audit found a master commit that did not pass CI on the merged SHA. |
| `auto/driver-down` | external monitor | operator after restart | Liveness heartbeat stale > 5 min. |
| `auto/revert` | operator (Path A) | merged | Revert PR opened in response to an `auto/invariant-violation`. |
| `auto/sentinel` | `duplicate_prs_to_fork.py` | operator (delete fork PR) | Sentinel PR duplicated from upstream into a personal fork for pipeline testing. Lives only in the fork. |

---

## Known Users / Contributors

| Login | Role |
|---|---|
| `freemo` | Jeffrey Phillips Freeman — Founder & CTO |
| `brent.edwards` | Contributor |
| `CoreRasurae` | Contributor |
| `hamza.khyari` | Contributor |
| `HAL9000` | AI agent user |
| `hurui200320` | Contributor |
| `aditya` | Contributor |
| `mngrif` | Contributor |
| `eugen.thaci` | Contributor |

---

## Rules

- Read this file at the start of every new chat.
- After every substantive change, add an entry to `tools/CHANGELOG` with a date and a suggested commit message.
- **Always use `tools/count-master-merges.py` to count PR merges or list master activity.** Never hand-roll a PR API scan, never read from the PR API alone, and never rely on `"Merge pull request"` commit message matching. If the script is missing a feature you need, extend the script — do not bypass it.
- **Always use `tools/render-pr-velocity.py` to refresh `pr-velocity.canvas.tsx`.** Do not hand-edit the generated canvas — it will be overwritten on the next render. Hand-written commentary and custom Key-Findings cards belong in `tools/pr-velocity-notes.toml`; new charts or metrics belong in the renderer + template.
- **Always use `tools/render-milestones.py` to refresh `milestone-completion.canvas.tsx`.** Do not hand-edit the generated canvas — the next render will stomp your changes. Hand-written intro / caveats / Key-Findings text and per-milestone description overrides belong in `tools/milestones-notes.toml`; new metrics, classification rules, or sections belong in the renderer + template.

## graphify

This project has a knowledge graph at graphify-out/ with god nodes, community structure, and cross-file relationships.

When the user types `/graphify`, invoke the `skill` tool with `skill: "graphify"` before doing anything else.

Rules:
- ALWAYS read graphify-out/GRAPH_REPORT.md before reading any source files, running grep/glob searches, or answering codebase questions. The graph is your primary map of the codebase.
- IF graphify-out/wiki/index.md EXISTS, navigate it instead of reading raw files
- For cross-module "how does X relate to Y" questions, prefer `graphify query "<question>"`, `graphify path "<A>" "<B>"`, or `graphify explain "<concept>"` over grep — these traverse the graph's EXTRACTED + INFERRED edges instead of scanning files
- After modifying code, run `graphify update .` to keep the graph current (AST-only, no API cost).