The launch script now starts merge_drive by default (T5-7: it is the
controller's singleton merge stage — APPROVED -> MERGING -> MERGED).
Previously the script killed merge_drive as a "legacy" process but
never started it, so controller workflows dead-ended at APPROVED.
merge_drive moves into the CONTROLLER process group; --no-merge skips
it. Also adds the worker thread-pool concurrency knob (default 2 for
the trial harness).
Renames the misleadingly-named env var
CONTROLLER_MAX_CONCURRENT_WORKERS_PER_MACHINE ->
CONTROLLER_MAX_CONCURRENT_WORKER_THREADS_PER_MACHINE: it sizes a
ThreadPoolExecutor inside one worker process, it does not spawn
worker processes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Continuing the round-3 deep-pass cleanup. Three more run-blockers
+ one robustness fix.
RB5 — MERGING handler never invoked from master loop:
``run_merging_tick`` was exported by the master package but no caller
fired it. Workflows that transition to MERGING (via reviewer
approval) would sit there indefinitely with no Forgejo merge call.
Fix:
- ``master/loop.py`` accepts a ``merging_args=(owner, repo,
merge_callback)`` kwarg. When set, the tick fires every iteration
(cheap if no workflows in MERGING).
- ``MasterTickReport`` gains ``merging: MergingHandlerReport | None``.
- ``master/__main__.py`` wires it from the Forgejo callback bundle.
RB6 — periodic discovery never fires:
``run_discovery`` was only called at startup via
``run_startup_backfill`` + the ``--discovery-only-once`` smoke flag.
PRs created after master startup would not be discovered until the
master restarted.
Fix:
- ``master/loop.py`` accepts ``discovery_args=(owner, repo, list_prs,
list_issues)`` or the 5-tuple with kwargs. Periodic tick on its
own cadence (``CONTROLLER_DISCOVERY_INTERVAL_S``, default 30s).
- ``MasterTickReport`` gains ``discovery: DiscoveryReport | None``.
- ``master/__main__.py`` wires it + threads ``require_opt_in_label``
through.
RB-robust — worker calls create_all defensively:
Master is normally responsible for schema creation (workers run
After= it via systemd ordering). But if the worker is started in
isolation (test / local dev / unit ordering broken), it'd crash on
the first query against missing tables.
Fix:
- ``worker/__main__.py`` calls ``create_all(engine)`` after
``build_engine``. ``create_all`` is idempotent (CREATE TABLE IF
NOT EXISTS); safe to call from both master + worker.
- ``cleveragents-controller-worker@.service`` adds
``After=cleveragents-controller-master.service`` +
``Wants=cleveragents-controller-master.service`` so systemd
enforces the start ordering in production.
Total: 703 controller tests pass (no test changes; all new wiring
is exercised by master_main_loop tests via the new kwargs).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Deployment surface for the controller. Operators copy the unit files
+ env examples to /etc/, customize, and `systemctl enable --now`.
Layout (under tools/controller/deploy/):
- systemd/cleveragents-controller-master.service — master singleton.
One unit per (owner, repo); plan v9 singleton constraint enforced
by deployment.
- systemd/cleveragents-controller-worker@.service — worker template.
`systemctl enable cleveragents-controller-worker@implementer-1`
spins up one instance; scale by adding instances. Per-instance
env override at /etc/cleveragents/worker.<inst>.env (optional)
specialises CLEVERAGENTS_WORKER_ROLES per instance.
- systemd/master.env.example — every env var the master reads,
documented. Copy to /etc/cleveragents/master.env, mode 0640.
- systemd/worker.env.example — every env var the workers read.
- RUNBOOK.md — operator guide:
* Prereqs (Linux + systemd 245+, Python 3.13 + uv, Postgres 14+,
OpenCode server, dedicated cleveragents user).
* First-time setup (7 ordered steps from useradd to first PR).
* Day-to-day ops (where logs live, healthy queries, role pool
sizing, label-based pause, clean restart).
* Incident response (6 named scenarios: STUCK workflows, MERGING
retry exhaustion, no-OpenCode, dequeue starvation, DB loss,
operator-unstick procedure).
* Migration playbook (Phase 2: opt one PR in via label,
monitor controller_events, escalate to full management).
* Tunables cheat sheet covering all 8 env vars + their tradeoffs.
- README.md — index linking the above.
Hardening on both units: NoNewPrivileges, PrivateTmp,
ProtectSystem=strict, ProtectHome, narrow ReadWritePaths, kernel
+ control-group protections. RestartPreventExitStatus=2 prevents
systemd loop-restart on misconfiguration (exit 2 = bad env).
No code changes; verified via `systemd-analyze verify` (unit syntax
parses; only "venv path doesn't exist on dev host" warnings, which
are expected). Test suite: 569/569 pass, 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
drew