a21466add232d59cdec1604e09d58ca05659a623
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 drew
|
eb01eb0172 |
feat(controller): dual-mode launcher (fork/prod) + DB-mode validator
Adds the operator surface for switching the controller pipeline between
the personal fork (drew/cleveragents-core) and the canonical repo
(cleveragents/cleveragents-core) via a MODE env + --prod CLI flag,
backed by safety primitives that make a wrong-mode launch loud rather
than silent.
run-controller-state-machine-pipeline.sh: --prod flag and MODE env
(primary home: .devcontainer/.env) select fork vs prod. After resolving
MODE, the launcher auto-sources the matching overlay file
(.devcontainer/.env.{fork,prod}) and asserts MODE didn't drift during
the source step. The drift assertion uses a readonly snapshot under an
obscure variable name so a stray ``MODE=fork`` in .env.prod aborts the
launch with a clear bash error rather than silently demoting the run.
CONTROLLER_RUN_DIR_ROOT now overrides the trial /tmp path so prod can
use a persistent /var/lib/cleveragents/run dir.
tools/launch_prod.sh (new): sibling to launch_fork.sh with the opposite
safety primitive — affirmative GET /repos/{owner}/{repo} that asserts
the target is non-fork, exists, isn't archived, and the bot has push.
On any failure, no env is exported. Honors HAL_* aliases for parity
with launch_fork.sh and prints a hard-to-miss PROD-MODE banner.
tools/controller/deploy/validate_db_mode.py (new): stamps a _mode_marker
table on each SQLite db (controller DB + telemetry cache) on first use,
asserts a match on every subsequent launch, and moves mismatched files
aside as <name>.<prior-mode>.bak.<ts> — never deletes. The --adopt flag
lets an operator grandfather in already-good pre-marker data without
losing history. Wired into the launcher's startup sequence before
OpenCode and the master start.
tools/_cache_path.py (new): single source of truth for the per-(owner,
repo) Forgejo cache file convention. .opencode/telemetry/server.py and
the launcher both delegate here so the dual-source-truth drift risk is
eliminated. tools/_pipeline_cache.py and tools/controller/db/models.py
documented as not owning the _mode_marker table so future migrations
leave it alone.
.opencode/telemetry/server.py: hosts the llm_activity scraper as a
background subprocess thread (60s cadence, --since-hours 1 in steady
state, full backfill on first tick). Re-homes the cost-telemetry data
path after the pr_state_warmer was retired by the controller migration
— without this the Cost tab freezes when the warmer's loop is gone.
Subprocess (not in-process) for isolation; failures swallowed.
opencode.json: local-claude provider's baseURL now reads
{env:LOCAL_PROXY_URL} instead of the literal http://127.0.0.1:3456/v1,
matching the apiKey pattern already in use.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
f454670a0e |
chore(controller): launch merge_drive from the pipeline script; rename worker-concurrency env var
The launch script now starts merge_drive by default (T5-7: it is the controller's singleton merge stage — APPROVED -> MERGING -> MERGED). Previously the script killed merge_drive as a "legacy" process but never started it, so controller workflows dead-ended at APPROVED. merge_drive moves into the CONTROLLER process group; --no-merge skips it. Also adds the worker thread-pool concurrency knob (default 2 for the trial harness). Renames the misleadingly-named env var CONTROLLER_MAX_CONCURRENT_WORKERS_PER_MACHINE -> CONTROLLER_MAX_CONCURRENT_WORKER_THREADS_PER_MACHINE: it sizes a ThreadPoolExecutor inside one worker process, it does not spawn worker processes. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
3ca794be75 |
feat(controller): autonomous CI status polling — closes the last trial gap
The Phase 2 trial previously required operator-intervention SQL to
advance workflows from AWAITING_CI → REVIEWING (no automated CI
status polling). This commit wires the missing tick so the trial
runs end-to-end without manual help.
Components:
- ``master/forgejo_http.py``: new ``get_ci_status`` callback wraps
Forgejo's ``/commits/{sha}/status`` combined-status endpoint;
added to ``ForgejoCallbacks``.
- ``master/ci_status_poll.py`` (NEW): ``run_ci_status_poll_tick``
scans AWAITING_CI workflows, fetches CI status keyed on the
latest implementer attempt's ``head_sha_after``, and applies
state transitions via ``apply_event``. TOCTOU-defended UPDATE
(``WHERE current_state='AWAITING_CI'``) + per-row exception
isolation.
- ``master/loop.py``: new ``ci_status_poll_args=(owner, repo,
get_ci_status)`` kwarg + ``ci_status_poll_interval_s`` config
(default 60s) + ``MasterTickReport.ci_status_poll`` field.
- ``master/__main__.py``: threads ``callbacks.get_ci_status`` into
the loop.
State mapping (Forgejo combined-status state → event):
- success / neutral / skipped / warning → ci_green → REVIEWING
- failure / error / cancelled / timed_out / stale →
ci_red_retry_same_tier → IMPLEMENTING
- pending / queued / in_progress / action_required → no-op (wait)
- None / unknown / fetch failure → no-op (transient)
The ``ci_polling_exhausted`` timeout (default 2h) remains as the
safety net for CI that genuinely never reports.
Tests (+14 in test_master_ci_status_poll.py):
- Happy paths (success→green, failure→red, pending→wait)
- Error paths (callback raises; workflow without head_sha)
- Event row shape (event_type='ci-green'/'ci-red', reason payload)
- Extended state mapping (cancelled, neutral, in_progress)
- Other-repo isolation
- LoopIntegration end-to-end via master_main_loop with safety timer
RUNBOOK updated: removed the manual SQL workaround; added the
autonomous CI poll's tunables.
Total: 726 controller tests pass (+14 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
b4ebf6a2bd |
docs(controller): RUNBOOK — Phase 2 trial gaps + manual workarounds
Document the two known gaps that need operator intervention for an
end-to-end trial:
1. MCP-to-OpenCode transport: response-builder MCPs aren't yet
registered in opencode.json. Agents use the FALLBACK file-write
path (``{workspace_dir}/{role}_output.json``) per the prompt
instructions. The agent_runner polls both channels.
2. CI status polling not wired: AWAITING_CI exits only via
ci_polling_exhausted timeout (default 2h → STUCK). To advance
during the trial, the operator runs a SQL UPDATE to manually
transition AWAITING_CI → REVIEWING (snippet in the RUNBOOK).
Added a step-by-step Trial checklist covering: env vars, label
the PR, expected log timeline, the manual SQL to advance past
AWAITING_CI, and the verification query.
Both gaps have follow-up phases queued (1m for MCP wiring, 1n for
CI poller). The trial as described validates the full controller
state machine + most of the worker substrate; only the CI poll +
the MCP transport are operator-intervention paths.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
251eeb21ff |
fix(controller): more pipeline run-blockers — merging tick, periodic discovery, worker create_all, systemd ordering
Continuing the round-3 deep-pass cleanup. Three more run-blockers + one robustness fix. RB5 — MERGING handler never invoked from master loop: ``run_merging_tick`` was exported by the master package but no caller fired it. Workflows that transition to MERGING (via reviewer approval) would sit there indefinitely with no Forgejo merge call. Fix: - ``master/loop.py`` accepts a ``merging_args=(owner, repo, merge_callback)`` kwarg. When set, the tick fires every iteration (cheap if no workflows in MERGING). - ``MasterTickReport`` gains ``merging: MergingHandlerReport | None``. - ``master/__main__.py`` wires it from the Forgejo callback bundle. RB6 — periodic discovery never fires: ``run_discovery`` was only called at startup via ``run_startup_backfill`` + the ``--discovery-only-once`` smoke flag. PRs created after master startup would not be discovered until the master restarted. Fix: - ``master/loop.py`` accepts ``discovery_args=(owner, repo, list_prs, list_issues)`` or the 5-tuple with kwargs. Periodic tick on its own cadence (``CONTROLLER_DISCOVERY_INTERVAL_S``, default 30s). - ``MasterTickReport`` gains ``discovery: DiscoveryReport | None``. - ``master/__main__.py`` wires it + threads ``require_opt_in_label`` through. RB-robust — worker calls create_all defensively: Master is normally responsible for schema creation (workers run After= it via systemd ordering). But if the worker is started in isolation (test / local dev / unit ordering broken), it'd crash on the first query against missing tables. Fix: - ``worker/__main__.py`` calls ``create_all(engine)`` after ``build_engine``. ``create_all`` is idempotent (CREATE TABLE IF NOT EXISTS); safe to call from both master + worker. - ``cleveragents-controller-worker@.service`` adds ``After=cleveragents-controller-master.service`` + ``Wants=cleveragents-controller-master.service`` so systemd enforces the start ordering in production. Total: 703 controller tests pass (no test changes; all new wiring is exercised by master_main_loop tests via the new kwargs). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
a19b609554 |
feat(controller): Phase 1l — systemd units + ops runbook
Deployment surface for the controller. Operators copy the unit files
+ env examples to /etc/, customize, and `systemctl enable --now`.
Layout (under tools/controller/deploy/):
- systemd/cleveragents-controller-master.service — master singleton.
One unit per (owner, repo); plan v9 singleton constraint enforced
by deployment.
- systemd/cleveragents-controller-worker@.service — worker template.
`systemctl enable cleveragents-controller-worker@implementer-1`
spins up one instance; scale by adding instances. Per-instance
env override at /etc/cleveragents/worker.<inst>.env (optional)
specialises CLEVERAGENTS_WORKER_ROLES per instance.
- systemd/master.env.example — every env var the master reads,
documented. Copy to /etc/cleveragents/master.env, mode 0640.
- systemd/worker.env.example — every env var the workers read.
- RUNBOOK.md — operator guide:
* Prereqs (Linux + systemd 245+, Python 3.13 + uv, Postgres 14+,
OpenCode server, dedicated cleveragents user).
* First-time setup (7 ordered steps from useradd to first PR).
* Day-to-day ops (where logs live, healthy queries, role pool
sizing, label-based pause, clean restart).
* Incident response (6 named scenarios: STUCK workflows, MERGING
retry exhaustion, no-OpenCode, dequeue starvation, DB loss,
operator-unstick procedure).
* Migration playbook (Phase 2: opt one PR in via label,
monitor controller_events, escalate to full management).
* Tunables cheat sheet covering all 8 env vars + their tradeoffs.
- README.md — index linking the above.
Hardening on both units: NoNewPrivileges, PrivateTmp,
ProtectSystem=strict, ProtectHome, narrow ReadWritePaths, kernel
+ control-group protections. RestartPreventExitStatus=2 prevents
systemd loop-restart on misconfiguration (exit 2 = bad env).
No code changes; verified via `systemd-analyze verify` (unit syntax
parses; only "venv path doesn't exist on dev host" warnings, which
are expected). Test suite: 569/569 pass, 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|