cleveragents-core/tools/controller/deploy/RUNBOOK.md

Controller Ops Runbook

Operational guide for the cleveragents controller (master + workers). Pair with the systemd units in systemd/ and the env-file examples in the same directory.

Architecture in 90 seconds

• Master singleton per (owner, repo): runs discovery, the state machine tick, the reaper, the pickup-guard, periodic reconciliation, and Forgejo writes. Enforced by one systemd unit per repo.
• Workers (N per machine, M machines): dequeue pending workflow_attempts, spawn a per-attempt MCP subprocess + OpenCode session, write the result. Multi-machine safe via the v9 PR-level lock with TTL + heartbeat.
• Postgres is the shared substrate. SQLite is supported only for single-host dev — multi-machine deployments must use Postgres.
• OpenCode is the LLM driver. Each per-attempt subprocess spawns a session against the OpenCode HTTP API.

Prerequisites

• Linux host with systemd 245+ (uses Type=simple template units).
• Python 3.13 + uv installed; project checked out at /opt/cleveragents-core (or symlinked there) and a virtualenv at /opt/cleveragents-core/.venv.
• Postgres 14+ reachable from every controller host. The DB user needs CREATE on the controller database (the app issues CREATE TABLE IF NOT EXISTS on first run).
• OpenCode server reachable from every worker host.
• A dedicated unix user cleveragents:cleveragents. Owns /opt/cleveragents-core, /var/lib/cleveragents, and /var/log/cleveragents (the latter two are reserved for runtime state and logs).
• Forgejo PAT for a bot account with read access to PRs/issues
  • write access to labels, comments, reviews, and merges.

First-time setup

1. Create the OS bits:

   sudo useradd --system --create-home --shell /usr/sbin/nologin cleveragents
   sudo install -d -o cleveragents -g cleveragents \
       /var/lib/cleveragents /var/log/cleveragents
   sudo install -d -o root -g root -m 0755 /etc/cleveragents
   

2. Drop in the systemd units + env files:

   sudo cp tools/controller/deploy/systemd/cleveragents-controller-master.service \
       /etc/systemd/system/
   sudo cp tools/controller/deploy/systemd/cleveragents-controller-worker@.service \
       /etc/systemd/system/
   sudo install -m 0640 -o root -g cleveragents \
       tools/controller/deploy/systemd/master.env.example \
       /etc/cleveragents/master.env
   sudo install -m 0640 -o root -g cleveragents \
       tools/controller/deploy/systemd/worker.env.example \
       /etc/cleveragents/worker.env
   sudo systemctl daemon-reload
   

3. Edit the env files — set OWNER/REPO/DB URL/Forgejo PAT/opt-in label. The example files document every variable. Mode 0640 keeps the Forgejo PAT off other users.

4. Create the opt-in label on Forgejo: Visit https://<forgejo>/<owner>/<repo>/labels and add a label matching CONTROLLER_OPT_IN_LABEL (default controller-managed). This is the migration mechanism — only labeled PRs/issues are discovered.

5. Initialise the database (one-shot, master does it):

   sudo systemctl start cleveragents-controller-master
   sudo journalctl -u cleveragents-controller-master -n 50
   # Expect: "controller backfill: first-time for <owner>/<repo>"
   

6. Start the worker pool (sized per your throughput needs):

   sudo systemctl enable --now cleveragents-controller-worker@implementer-1
   sudo systemctl enable --now cleveragents-controller-worker@reviewer-1
   sudo systemctl enable --now cleveragents-controller-worker@estimator-1
   

   For role-specialised pools, drop a per-instance override file at /etc/cleveragents/worker.implementer-1.env setting CLEVERAGENTS_WORKER_ROLES=implementer.

7. Verify:

   sudo systemctl status cleveragents-controller-master
   sudo systemctl status 'cleveragents-controller-worker@*'
   psql "$CLEVERAGENTS_DB_URL" -c \
       "SELECT current_state, COUNT(*) FROM workflows GROUP BY current_state;"
   

Day-to-day operations

Where the logs are

• Master: journalctl -u cleveragents-controller-master -f
• One worker: journalctl -u cleveragents-controller-worker@implementer-1 -f
• All workers: journalctl -u 'cleveragents-controller-worker@*' -f

What "healthy" looks like

• Master logs every iteration at INFO if anything transitioned. A quiet master (no transitions for minutes) means there's no work ready — that's normal between PR pushes.
• Workers log per dequeue + per finalize. A worker that polls every 5s + finds nothing logs at DEBUG; INFO at startup + every successful attempt.
• Useful query to spot stuck workflows:

  SELECT workflow_id, current_state, last_transition_at, current_tier
  FROM workflows
  WHERE current_state NOT IN ('MERGED', 'ABANDONED', 'STUCK', 'CREATED_PR')
    AND last_transition_at < NOW() - INTERVAL '1 hour'
  ORDER BY last_transition_at;
  

Pausing management of one PR

Remove the opt-in label from the PR on Forgejo. The next reconciliation tick (within CONTROLLER_RECONCILIATION_INTERVAL_S, default 300s) detects the missing label and transitions the workflow to ABANDONED with reason opt-in-label-removed. The controller stops touching it.

To re-enable: re-add the label. Next discovery tick rediscovers it as a new DISCOVERED workflow.

Adding capacity

Just enable more worker units:

sudo systemctl enable --now cleveragents-controller-worker@implementer-2

The dequeue protocol is FOR UPDATE SKIP LOCKED on Postgres, so adding workers scales linearly without lock contention.

Restarting cleanly

• Master: sudo systemctl restart cleveragents-controller-master. The SIGTERM handler flips the stop_event; the current tick finishes; the loop exits within 30s.
• Workers: sudo systemctl restart cleveragents-controller-worker@implementer-1. In-flight attempts get a graceful stop via the heartbeat thread ceasing — the master's reaper resets the attempt to pending after the TTL. New workers pick it up.

Incident response

"A workflow is STUCK"

1. Find it:

   SELECT workflow_id, owner, repo, entity_number, current_state,
          last_transition_at
   FROM workflows
   WHERE current_state = 'STUCK';
   

2. Inspect the most recent event row for context:

   SELECT ts, event_type, from_state, to_state, payload
   FROM controller_events
   WHERE workflow_id = <id>
   ORDER BY ts DESC LIMIT 10;
   

3. Common reasons:
   • pickup-exhausted — the attempt was reset to pending too many times (default 3, CONTROLLER_PICKUP_GUARD_MAX_PICKUPS). Cause: workers are dying mid-attempt. Check worker logs around the attempt's created_at.
   • opt-in-label-removed — operator removed the label. Re-add it on Forgejo or treat this as intentional abandonment.
   • pr-not-found-on-forgejo — Forgejo returned 404 during reconciliation. PR may have been deleted; verify by hand.
   • branch-protection — merge attempt hit a branch-protection rule the bot account isn't allowed to bypass. Fix on Forgejo.
4. To unstick a workflow (operator action):

   UPDATE workflows
   SET current_state = 'IMPLEMENTING',
       last_transition_at = NOW(),
       entered_state_at = NOW()
   WHERE workflow_id = <id>;
   INSERT INTO controller_events
       (workflow_id, ts, event_type, from_state, to_state, payload,
        forgejo_write_pending, replay_attempts)
   VALUES (<id>, NOW(), 'operator-unstick', 'STUCK', 'IMPLEMENTING',
           '{"reason":"<short reason>","operator":"<your name>"}', 0, 0);
   

   The next tick will schedule a fresh implementer attempt.

"MERGING is stuck"

Look for merging_retry_count >= 5 in the workflows table — that means the merge endpoint has 5xx'd 5 times in a row and the master gave up. Inspect the controller_events payload for the actual HTTP error. Most common cause: Forgejo restart mid-merge — just operator- unstick to retry once Forgejo is back.

"Workers are dequeuing but nothing happens"

Check the OpenCode server first — workers depend on a healthy OpenCode at CONTROLLER_OPENCODE_URL. The worker log will show "OpenCode transport error" or "OpenCode session timed out" for each failed attempt.

"Multiple machines, but one machine takes all the work"

Confirm the dequeue is using FOR UPDATE SKIP LOCKED:

EXPLAIN UPDATE workflow_attempts SET status='in_progress', ...
WHERE attempt_id = (SELECT attempt_id FROM workflow_attempts
                    WHERE status='pending' ORDER BY created_at
                    FOR UPDATE SKIP LOCKED LIMIT 1);

On SQLite (single-host dev), the lock is BEGIN IMMEDIATE — multiple machines pointed at the same SQLite file will serialize and one machine will appear to monopolize. This is expected; switch to Postgres for multi-machine.

Database recovery

The controller's DB is the source of truth — Forgejo is the journal. If the DB is lost:

1. Restore from your Postgres backup if you have one.
2. Otherwise: start fresh. Master backfill will discover every open PR/issue with the opt-in label. Active workflows lose their attempt history — they re-enter at DISCOVERED → ANALYZING. No Forgejo state is lost (PRs + comments + reviews are intact).

Migration playbook (Phase 2)

Goal: bring one PR under controller management without exposing the broader repo.

1. Pick a low-risk PR with no active human reviewers.
2. Add the controller-managed label on Forgejo.
3. Watch the master log — next discovery tick rediscovers it as DISCOVERED. The state machine drives ANALYZING → IMPLEMENTING.
4. Monitor the controller_events table for transitions:

   SELECT ts, from_state, to_state, payload
   FROM controller_events
   WHERE workflow_id = (
       SELECT workflow_id FROM workflows
        WHERE entity_number = <pr_number> AND kind='pr'
   )
   ORDER BY ts;
   

5. If anything goes wrong: remove the label. The next reconciliation tick ABANDONS it cleanly.
6. After a successful run, add the label to a second PR. Etc.

When you're confident, run with --no-opt-in-label to manage all PRs unconditionally.

Tunables cheat sheet

Variable	Default	When to change
CONTROLLER_MASTER_TICK_INTERVAL_S	30	Lower for snappier responses; raises DB load
CONTROLLER_REAPER_INTERVAL_S	60	Match to your worker heartbeat (default 30s) ×2
CONTROLLER_RECONCILIATION_INTERVAL_S	300	Lower if operators frequently modify PRs out-of-band
CONTROLLER_WORKER_POLL_INTERVAL_S	5	Lower for snappier dequeue; raises DB load
CONTROLLER_HEARTBEAT_INTERVAL_S	30	Must be ≪ lock_ttl_seconds (master writes 600s)
CONTROLLER_MAX_CONCURRENT_WORKER_THREADS_PER_MACHINE	1	Pool-thread count inside the one worker process (not extra processes); bump toward your OpenCode per-instance concurrency ceiling
CONTROLLER_TIER_*_TIMEOUT_S	600 / 1200 / 1800	Bump if tier-2 attempts hit the wall on large PRs
CONTROLLER_OPT_IN_LABEL	controller-managed	Rename if you collide with an existing label

What this controller does NOT do (yet)

• Replace stub CI parsers (robot_framework / bandit / semgrep / vulture / radon / slipcover / build). Those gates emit parser-pending-* CIFailures and don't extract structured findings yet — the raw log is still surfaced to the implementer.
• HA (multi-master). The plan is one master per (owner, repo); HA via lease-based leadership is deferred.
• Per-PR custom configs (different tier ceilings, custom timeouts). Everything is global env config in v1.

Phase 2 trial — known gaps to handle by hand

The controller infrastructure can drive a PR from DISCOVERED → IMPLEMENTING → AWAITING_CI, but two pieces are stubbed and need operator intervention for end-to-end validation:

MCP-to-OpenCode transport (response-builder MCPs not yet wired)

The controller's response-builder MCPs (implementer-builder etc.) aren't registered in .opencode/opencode.json, so the agent can't call implementer_finalize via MCP. The prompt builder gives the agent a FALLBACK: write JSON output to {workspace_dir}/{role}_output.json directly via the Write tool. The controller polls that path. Trial impact: agents that follow the fallback instruction work fine. If an agent ignores the fallback

• tries the MCP path, the attempt times out after finalize_timeout_s (default 30s) and gets recorded as worker-internal-error. Watch the log for "did not emit canonical output within Ns" messages.

CI status polling — autonomous (no manual SQL needed)

The master loop runs run_ci_status_poll_tick every CONTROLLER_CI_STATUS_POLL_INTERVAL_S (default 60s). For each workflow in AWAITING_CI, it calls Forgejo's /commits/{sha}/status combined-status endpoint and applies the matching state transition:

Forgejo state	Event fired	Next state
success / neutral / skipped / warning	ci_green	REVIEWING
failure / error / cancelled / timed_out / stale	ci_red_retry_same_tier	IMPLEMENTING
pending / queued / in_progress / action_required	(no-op)	stays AWAITING_CI

Operators no longer need the manual SQL workaround. If CI hangs permanently (broken integration, runner outage), the ci_polling_exhausted timeout (default 2h) still fires as the safety net → STUCK.

To tune for trial speed:

CONTROLLER_CI_STATUS_POLL_INTERVAL_S=10   # poll every 10s
CONTROLLER_AWAITING_CI_TIMEOUT_S=300      # 5-min timeout instead of 2h

Trial checklist

1. Set FORGEJO_URL + FORGEJO_TOKEN + CLEVERAGENTS_DB_URL on master
   • worker hosts.
2. Pick a low-risk PR; add the controller-managed label.
3. Start the master: systemctl start cleveragents-controller-master.
4. Watch the journal: journalctl -u cleveragents-controller-master -f.
5. Expect within ~30s:
   • Discovery picks up the PR (DiscoveredEntity logged)
   • Promoter advances DISCOVERED → ANALYZING
   • Scheduler enqueues an estimator attempt
   • Worker dequeues + spawns the estimator agent
6. Expect within tier-0 timeout (~10 min):
   • Estimator emits JSON → ANALYZING → IMPLEMENTING
   • Scheduler enqueues an implementer attempt
   • Worker spawns the implementer agent
   • Implementer commits + pushes → workflow → AWAITING_CI
7. CI runs in Forgejo Actions; the master's CI-status poller fires every 60s (or whatever you set CONTROLLER_CI_STATUS_POLL_INTERVAL_S to) — when CI goes green the workflow advances to REVIEWING automatically.
8. The reviewer attempt runs; if verdict='approve', the workflow transitions to MERGING and the controller calls Forgejo's merge endpoint.
9. Confirm MERGED state via DB:

   SELECT current_state FROM workflows WHERE entity_number = <pr>;