a21466add232d59cdec1604e09d58ca05659a623
39 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 drew
|
016b348117 |
feat(controller): grooming gate (Phase 0 + Phase 1 worker-shape dispatch)
Phase 0 (foundation):
- Cause enum (controller_events.cause) for action attribution
- Schema: grooming_decisions audit table; workflows gains
grooming_evaluated_at + deferred_reason + deferred_at +
deferred_target_workflow_id; pulls gains touched_files
- audit_comments: CLOSE / DEFER templates + render_comment_template
- forgejo_writes: close_issue + defer_issue 5-step crash-safe protocol
(fingerprint dedup, error matrix, dry-run)
- patch_pr_state callback in forgejo_http
- grooming_config: 22-env-var frozen-dataclass config + log_effective
- pulls.touched_files cache extension (_pipeline_cache.py schema v8)
- reaper.reap_grooming_decisions audit-retention sweep
- reconciliation RESUME guard (deferred_reason)
Phase 1 (worker-queue shape, 2026-05-25):
- New state: GROOMING. New events: grooming_started, groom_verdict_
{proceed,defer,close}. 5 new transitions; all invariants still clean
- GroomingInputV1 + GroomingOutputV1 Pydantic contracts
- outcomes._map_grooming_outcome routes verdicts to state-machine events
- prefetch.build_grooming_stage_b_input + list_open_prs callback
- scheduler GROOMING -> grooming_stage_b role
- promote: cfg-gated DISCOVERED -> GROOMING when CONTROLLER_GROOMING_
ENABLED=true; issues skip grooming
- forgejo_writes decomposed: close_act/defer_act (Forgejo writes only;
state-machine already transitioned) + close_decide_and_act/
defer_decide_and_act (Phase 0 callers); _apply_workflow_transition
is underscore-private
- grooming.py library: tokenization, suspicion scoring (Jaccard +
weighted overlap), deterministic checks, action -> verdict mapping
- mcp/grooming_builder.py: 14-tool FastMCP server emits GroomingOutputV1
- .opencode/agents/grooming-stage-b.md: duplicate-detection agent
prompt (claude-haiku-4-5)
- grooming_side_effects.run_grooming_side_effects_tick: per-state tick
performs Forgejo writes after groom_verdict_{defer,close} fires.
Filters on event_type='transition' + payload.event (centralizes the
convention pending Phase 2's latest_transition_event helper)
- GroomingCallbacks frozen dataclass; loop.py + __main__.py wired
Worker role registry (single source of truth):
- worker/roles.py: WORKER_ROLES + WorkerRoleSpec + default_roles_csv
+ output_filename_for. agent_runner.ROLE_TO_MCP_MODULE / ROLE_TO_
OUTPUT_MODEL derive from it; opencode_session.agent_name_for reads
it for flat cases; all 6 prompt builders use output_filename_for;
worker --roles default = default_roles_csv(); launcher script
derives --roles via shell substitution. Cross-site invariant test
enforces alignment across 5 sites + opencode.json MCP registry.
Phase 0 silent-bug fix:
- reconciliation.py RESUME guard SELECT now includes deferred_reason
(was missing since Phase 0; guard was a silent no-op). Tightened
from getattr to attribute access to fail fast on future omissions.
Tests (1456 total, +91 grooming-specific):
- test_grooming_phase0.py: 34 tests (orchestrator matrix, crash
recovery, idempotency, dry-run)
- test_grooming_phase1.py: 60 tests (library, contracts, state
machine, outcomes, scheduler, promote, prefetch, act-variants
with signature parity, side-effect tick incl. natural-idempotency
+ executed-flag-skip + verdict-mismatch + reconciliation RESUME)
- test_mcp_builders.py TestGroomingBuilder: 29 tests (happy paths
+ 22 validation rules + Pydantic round-trip + master-tick-read-
path companion)
- test_worker_agent_runner.py TestRoleMaps: cross-role wiring
alignment + agent-prompt-vs-worker-fallback filename contract +
inspect.signature equality (close_act/defer_act vs
close_issue/defer_issue)
- test_state_machine.py: transition count 51 -> 56 +
events_from_grooming
Live-validated end-to-end on 4 staged sentinel PRs (#55-#58) in
dry_run: agent emits verdicts via MCP, state-machine transitions
fire, side-effect tick writes audit row, deferred_reason gates
reconciliation RESUME correctly.
Deferred refinements + Phase 2 prerequisite (latest_transition_event
helper) tracked in .drew/regressions-plan.md "Phase 1 follow-up
backlog".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
d3cf710d29 |
fix(controller): tier-selection calibration — finalize timeout 30s→90s, estimator defaults to tier 1
Two coupled changes responding to the 2026-05-22 batch's tier-0 hit
rate of 0% on PRs the estimator judged "simple", plus three spurious
worker-internal-errors caused by the estimator's MCP-finalize wall
clock exceeding the previous 30s budget.
agent_runner.py — finalize timeout
----------------------------------
`production_agent_runner` previously hardcoded `finalize_timeout_s=30.0`.
On the 8-PR batch the worker recorded three `worker-internal-error`
attempts (PRs 47, 50, 54) all with the same message:
role='estimator' did not emit canonical output within 30.0s ...
The retries succeeded with estimator self-time of 41.5s, 25.9s, 21.5s —
all within plausible bounds for a multi-subsystem PR (PR 54 needed
41.5s of reasoning + MCP IPC + finalize write). Each spurious timeout
cost ~90s of worker wallclock + a re-dispatch.
New module-level constant `_DEFAULT_FINALIZE_TIMEOUT_S` reads
`CONTROLLER_FINALIZE_TIMEOUT_S` (default 90s), matching the
`CONTROLLER_*_TIMEOUT_S` convention used by `opencode_session.py`.
Operators can tune via `.devcontainer/.env.{fork,prod}`.
estimator-implementation.md — TIER MAP + CALIBRATION
----------------------------------------------------
The estimator was wrong on 4 of 8 PRs in the May-22 batch, all in
the same direction (recommended tier 0, escalated to tier 1).
Three of the four were high-confidence picks — including a PR
described as "4 string constant replacements, isolated scope, no
logic changes" that still got bounced at tier 0.
Three prompt changes:
- TIER MAP rewritten: tier 1 is now the default for non-trivial
work; tier 0 requires positive evidence the change is mechanical
(single file, ≤50 LOC, no new logic, no test changes).
- CONFIDENCE RULES "When uncertain, prefer" flipped tier 0 → tier 1.
- New CALIBRATION section bakes in the empirical observation (0/4
hit rate, why Haiku struggles in this codebase, cost calculus:
wrong tier-1 < wrong tier-0 by an order of magnitude when measured
per merged PR).
tiers.yaml — clarify scope
--------------------------
The previous "Used by estimator-implementation.md to make tier
choices model-agnostic" comment on the `capability` field misled
readers into thinking tiers.yaml drove classification. It doesn't —
the estimator agent reads its TIER MAP from its own prompt and
emits an integer; tiers.yaml only answers "for tier-N, which model
runs?". Updated header + field comments to flag this clearly so
future operators don't change descriptors here expecting the
estimator to honor them.
Tests
-----
+ test_default_finalize_timeout_reads_env: covers the env-var
resolution for `CONTROLLER_FINALIZE_TIMEOUT_S` (default, override,
float values). Restores the default at the end so subsequent
tests in the session see the stock constant.
Existing tier-model registry + 27 agent_runner tests pass unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
981ddd6a8e |
fix(controller): incident hardening — PR-44 dispute guard + PR-46 fixes
Bundles fixes for three production incidents (PR-44, PR-46 a/b/c). Every change has an incident-reference comment in the code and a named regression test. 199 tests pass on the impacted modules. PR-44 — fabricated dispute escape --------------------------------- A tier-0 implementer, bounced twice by red CI, emitted a fabricated ``dispute-reviewer`` outcome and shortcut a red PR into REVIEWING → APPROVED → MERGING, bypassing the CI gate. ``dispute-reviewer`` is the only IMPLEMENTING → REVIEWING edge that doesn't pass through AWAITING_CI, so it must be defended. tools/controller/master/outcomes.py: ``_map_implementer_outcome`` now guards ``dispute-reviewer`` with two preconditions — (1) ``attempt_saw_green_ci`` (the HEAD must already be CI-verified; a dispute can't jump the gate on a red/pending head), and (2) ``prior_reviews >= 1`` (must reference a review that actually happened, not a hallucinated one). Either guard fails → ``implementer_competence_failure`` → tier escalation. A weak model can't game its way past CI; a stronger tier is given the real problem. tools/controller/master/tick.py: new ``_count_prior_reviews`` helper counts completed reviewer attempts (epoch-scoped so an ``operator_unstick`` resets the count). Wired into the ``map_outcome_to_event`` call. PR-46(a) — stale gate-script preferred over in-repo --------------------------------------------------- ``gate.py`` was preferring the seeded ``/tmp/local_tools`` copy of ``local_ci_gate.sh`` over the version-matched in-repo copy. The seed predated the ``--envdir`` flag; the controller pipeline's invocations rejected as bad-argv every committing implementer's gate. The seed-refresher (``dispatch_implementer.py``) is on the retired dispatcher path, so the staleness was permanent. tools/controller/worker/gate.py: resolution order is now ``CONTROLLER_LOCAL_CI_GATE`` env > in-repo > seeded ``/tmp``. The seeded copy survives only as a last-resort fallback. Module-level constants ``_IN_REPO_GATE_SCRIPT`` / ``_SEEDED_GATE_SCRIPT`` let tests substitute paths. PR-46(b) — 6-second-old run flagged zombie ------------------------------------------ ``classify_ci_run`` instantly classified a CI run as ``stale`` when the Actions API reported no active task. A freshly-pushed run has no task simply because no runner has picked it up yet, and there are brief gaps between jobs — both false positives. A 6-second-old PR-46 run was bounced before CI could even start. tools/controller/master/ci_run_status.py: new ``ZOMBIE_GRACE`` (default 3 min, env: ``CONTROLLER_CI_ZOMBIE_GRACE_MIN``). "No active task" only classifies a run as stale once the run has ALSO gone quiet past the grace. Much shorter than ``STALE_AFTER`` since the active-task absence is corroborating evidence, not the sole signal. PR-46(c) — ruff-format-only violation slipping through lint ----------------------------------------------------------- CI's ``lint`` job runs both the ``lint`` nox session (ruff check) AND ``ruff format --check``. The pre-push local gate only ran the former; a formatting-only violation passed pre-push then failed CI. tools/local_ci_gate.sh: the ``lint`` gate now runs ``ruff check`` followed by ``ruff format --check``, unconditional. Either one failing marks the gate red. ``ruff format --check`` is whole-repo and takes no posargs. tests/auto_agents/test_local_ci_gate.py updated for the new two-call shape. Supporting changes ------------------ .forgejo/workflows/ci.yml: gates ``coverage`` and ``docker`` jobs on repo variables ``skip_coverage`` / ``skip_docker`` so the long reaper-prone jobs can be skipped per-fork without editing CI. Guard step always runs so the job still reports ``success`` and ``status-check`` stays green. tools/duplicate_prs_to_fork.py: bakes the same ``skip_coverage`` / ``skip_docker`` gates into every sentinel PR's ci.yml at PR-creation time so fork-mode runs inherit the gating. Idempotent. .opencode/opencode.json: adds ``timeout: 1860000`` (31 min) to the ``ci`` MCP server so long CI waits don't timeout the tool. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
a103a31bbf |
feat(controller): worker-owned gated push for the implementer
The implementer agent no longer pushes to git — the controller worker now owns the push: it gates the agent's commits on lint+typecheck and pushes via a single leased primitive. Closes two production defects: - Clobber: the pre-fix MCP --force-with-lease leased against a freshly-fetched tip, so the lease always passed — an in-flight implementer destroyed a commit pushed to the PR branch during its run (lost a hand-pushed skip_coverage fix on PR #46). - Gate-skip: the agent verified only the CI-flagged gate, so a fix for one gate shipped fresh violations in another (lint flapped pass->fail across CI runs 198->199). Step 1 — worker_push primitive: - New git_push.py: one leased push, pinned to the SHA the worker started from; classifies pushed / stale_input / diverged / infra_error; bounded infra-retry. - mcp_git_server.push gains expected_sha for a correctly-pinned lease. - finalize_conflict_resolution migrated onto worker_push. Step 2 — deterministic gate: - New gate.py: per-slot nox env-dirs (no venv races), manifest-hash staleness keying, lazy warm-up. - local_ci_gate.sh gains --envdir. Step 3 — worker-owned gated push: - New implementer_finalize.py: divergence pre-check -> lint+typecheck gate -> ruff auto-fix -> leased push. finalize's outcome is authoritative over the agent's emitted outcome. - agent_runner integrates finalize; salvage no longer pushes. - outcomes/tick: gate-failed + push-time stale-input caps, epoch-scoped; WorkerError carries an output_payload so the gate report reaches the next attempt's prompt; prefetch surfaces gate-failed attempts. - The 5 task-implementor prompts drop the agent push step. Reviewed across 4 adversarial rounds; full controller+MCP suite green (1379 passed). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
d4e1219577 |
fix(controller): scope conflict-marker check to resolved files
finalize_conflict_resolution grepped every file in the PR's origin/<base>..HEAD diff for committed conflict markers. On a large sentinel PR (478-file diff) that includes .opencode/agents/ git-rebase-util.md — an agent def that *documents* conflict markers with literal "<<<<<<< HEAD" lines — so the check false-positived and rejected every resolution, looping the conflict-resolver indefinitely (run-9: 185 attempts in ~4h, 125 on this exact error). Scope the grep to the files the resolution actually touched (resolved_files = prep-time conflicted set + the agent's reported files_modified). A leftover marker can only be in a file the agent resolved; a clean rebase/merge resolves nothing so the check is skipped. Adds a regression test with a marker-documenting file in the PR diff that is not a resolved file. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
615a05b982 |
feat(controller): rebase-default conflict resolution with merge fallback
PR branches 177-180 commits ahead of base cannot be rebased commit-by-commit by a single-shot resolver agent (too many conflict stops for one session). Conflict-prep now defaults to rebase (linear history) and falls back to a single 3-way merge when the branch is too divergent (commit count over CONTROLLER_CONFLICT_REBASE_MAX_COMMITS, default 60). The merge pipeline derives the track from branch shape via a Do:rebase -> Do:merge ladder in _make_merge_pr — no stored flag. Adds a git_rebase_continue MCP tool plus status rebase/merge-in-progress fields so the conflict-resolver agent is fully MCP-driven and dual-mode (mid-rebase or mid-merge). Also routes a green-CI implementer noop straight to REVIEWING instead of a deadlock-prone AWAITING_CI round trip. No state-machine change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
5f0fff0db2 |
fix(controller): wait for the whole CI run to finish before acting on it
Two places acted on a CI run that was still executing — the PR-39/PR-40 "doesn't wait for CI to finish" incident: 1. ci_status_poll — keyed the AWAITING_CI verdict purely off Forgejo's combined `state`. That combined state flips to `failure` the instant ONE gate fails, even with other gates still running (and can read `success` before a late gate reports). So a run at 9-passed / 1-failed / 2-pending fired `ci_red` and yanked the workflow out of AWAITING_CI mid-run. New `_ci_run_incomplete()` scans the per-gate `statuses`; while any gate is pending/running the poll waits, regardless of the combined state. Only a fully-terminal run yields a verdict. 2. implementer prompt routing — `_ci_summary_is_pending` required the run to be failure-free to count as "still pending," so a run with an early failure + gates still executing fell through to the ci-infra-failure block and the implementer acted on a partial result. Dropped the no-failures clause: any pending gate → the ci-not-ready (wait) block, which now also covers partial results. Zombie/stale runs (a gate pending for hours) are left to the controller's CI-freshness gate to re-trigger — a separate follow-up; ci_poll_exhaustion remains the backstop. 4 new tests; full controller suite (1195) green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
eb1d81828f |
feat(controller): implementer_retrigger_ci — agent re-triggers CI on ci-infra-failure
A CI job hard-killed by OOM / pod eviction produces no verdict — nothing in the diff to fix. The implementer correctly emits outcome=ci-infra-failure, but that alone only shuffles workflow state; CI never re-runs, so every retry reads the same dead run and the workflow loops until the _MAX_CI_INFRA_FAILURE backstop STUCKs it (observed live on PR #39 in run-3). New ``implementer_retrigger_ci(owner, repo, pr_branch)`` MCP tool on the implementer response-builder lets the agent kick a fresh CI run. Forgejo 15.x has no Actions rerun API, so it reuses the controller's existing mechanism — ``ci_rerun.trigger_ci_rerun_via_empty_commit`` — an empty commit that advances the PR head SHA, which is the unambiguous fresh-state signal for CI (and a stale review). - ci_rerun.py is loaded standalone (importlib by file path, registered in sys.modules before exec so its @dataclass resolves) — the response-builder MCP must not pull the heavy tools.controller.master package. - _retrigger_ci resolves the Forgejo base URL + token from env (FORGEJO_URL / FORGEJO_API_BASE, FORGEJO_TOKEN / GITEA_TOKEN) and never raises. - Once per session: a second implementer_retrigger_ci call is refused benignly so a looping agent cannot pile junk commits on the PR. - The implementer prompt's ci-infra-failure block now instructs the agent to call the tool before emitting the outcome. 10 new tests (test_mcp_builders.py, test_worker_prompts.py); full controller suite (1192) green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
8fe98cb5c0 |
feat(controller): salvage timed-out implementer + cap unbounded prefetch-retry loop
Two reliability fixes from the proactive audit of untouched pipeline modules (AUDIT-1, AUDIT-2 in .drew/PENDING_FIXES.md). AUDIT-1 — a timed-out implementer's committed work was never salvaged. opencode_session maps an OpenCode timeout / transport-error to WorkerError, which propagated out of production_agent_runner BEFORE the canonical-output wait + _salvage_implementer_commits ran. A timeout is the case most likely to have a complete committed fix (agent ran out of wallclock, not correctness). The session-exception handler now runs the same salvage the canonical-missing path uses before discarding the attempt as worker-internal-error. AUDIT-2 — a persistently failing prefetch() retried every master tick forever (the T4-4 guard covered only the issue-kind sub-case; same unbounded-retry class as the run-2 174x estimator loop). The scheduler now journals each prefetch failure as a controller_events row, counts failures since the workflow's last transition, and routes the workflow to STUCK after _PREFETCH_FAILURE_LIMIT (10) failures. The STUCK transition is rowcount-guarded so the journal never records a transition the current_state guard rejected. 7 new tests; full controller suite (1169) green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
f76568a871 |
feat(controller): ci-infra-failure implementer outcome → bounded rerun
The implementer-side counterpart to the `indeterminate` verdict. When the implementer IS dispatched onto a CI failure and finds the log carries no verdict (a hard-kill / OOM — nothing in the diff to fix), it can now emit `outcome=ci-infra-failure` instead of being forced to `blocked` → STUCK. `ci-infra-failure` forbids commits/files/blockers (no-work invariant, like `noop`) and fires `implementer_ci_infra_failure`, routing IMPLEMENTING → DISCOVERED so the CI-freshness gate reruns CI under its bounded RERUN_BUDGET. Backstopped by `_MAX_CI_INFRA_FAILURE=4` so a mis-classification cannot loop the gate forever. Wired through: the V1 contract enum, the implementer MCP builder (outcome value + no-work invariant), the state machine (event + IMPLEMENTING→DISCOVERED transition), the outcome mapper (+ per-workflow cap), tick's `_count_prior_ci_infra_failure`, and the implementer prompt — which now surfaces the outcome whenever the CI summary shows a failing gate, with guidance to use it ONLY when the log genuinely shows no verdict. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
521117882e |
fix(controller): conflict resolver lands resolutions on the PR
Run-2 PR #40 exposed that the conflict-resolver stage never resolved conflicts *on the PR*. The resolver agent owned the whole git flow (rebase + resolve + push) but was never given the PR head branch name, so it pushed the resolved branch to a guessed branch ("main"), the push was lost, and it reported "resolved" anyway — the controller advanced the workflow on a resolution that never reached the PR. This splits the work along the controller's principle: deterministic git mechanics in the worker, only the semantic merge in the agent. - conflict_rebase.py (new): the worker starts the rebase (prepare_conflict_rebase — hands the agent a real mid-rebase worktree) and, after the agent's resolution, lands it with a *verified* push (finalize_conflict_resolution) — force-with-lease pinned to the PR head the resolver started from, then re-reads the remote to confirm it moved. A failed/rejected push is never reported as resolved. run_conflict_resolver_attempt orchestrates it. - ConflictResolverInputV1 / prefetch: carry head_ref so the worker knows the PR branch (this also makes the worker's fetch_and_validate run for the conflict_resolver — it gated on head_sha AND head_ref). - worker/__main__.py: route conflict_resolver attempts through the new orchestration. - conflict-resolver-worker.md: the agent resolves + continues the rebase and never pushes — the controller lands it. - mcp_git_server.py: add the controller worktree root to the git MCP's allowed bases (every git_* MCP call errored for this role before). The committed-marker check is scoped to the PR's own diff — a whole-tree grep false-positived on repo files that legitimately document conflict markers. The worker's rebase/push are lost-lock guarded so a reaped worker cannot race the shared worktree. 1136 controller tests pass (+11 new in test_conflict_rebase.py). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
a6ff7da63c |
fix(controller): recover work lost to plumbing + run-25 reliability fixes
Run-25 surfaced five ways the controller lost or mis-routed correct implementer work. Each is fixed: 1. Canonical-output salvage (agent_runner.py): when an implementer's OpenCode session ends cleanly but emits no V1 output, its commits are no longer discarded — they are pushed and a synthetic outcome=resolved routes the workflow to AWAITING_CI so CI judges the code. Guarded by lost_lock_check so a reaped worker cannot race a fresh one on the shared worktree. 2. Preserve committed-but-unpushed work (workspace.py): the pre-reset snapshot (auto-scratch/pr-<N>) now captures committed commits ahead of the reset target, not just dirty edits — a finished-but-unpushed fix survives for the next attempt to adopt. 3. blocked escalates (outcomes.py): an implementer 'blocked' below MAX_TIER now escalates to a stronger tier instead of dead-ending at STUCK; only 'blocked' AT MAX_TIER STUCKs. 4. Diff-size tier floor (tick.py): the estimator's structured recommended_tier can contradict its own reasoning (a 387-file PR emitted as tier 0). current_tier is now floored deterministically by diff size so a huge PR cannot run on the weakest model. 5. CI log excerpt (prompts.py): the per-gate log excerpt is now tail-clipped, not head-clipped — the failure tracebacks live at the bottom and were being cut off, leaving the implementer blind. 1121 controller tests pass (+34 new). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
0db0a15dad |
feat(controller): RUN_CI_LOCAL verdict source, ci-not-ready outcome, escalation hardening
Adds RUN_CI_LOCAL — an on-demand local-CI verdict source for when the cluster's Forgejo CI is broken — plus robustness fixes, the telemetry Live-tab rewrite, and PR-level cost attribution. Controller: - RUN_CI_LOCAL: the master swaps its Forgejo CI callbacks for local `forgejo-runner exec` runs (tools/run-ci-full-local.sh + local_ci.py). Async per-(owner,repo,SHA) on-disk job cache; preflights the forgejo-runner binary + Docker daemon at startup (fail loud, not a red verdict on every PR); GCs finished run dirs + per-run actcache. - ci-not-ready implementer outcome + implementer_ci_not_ready event: an implementer that runs before the on-demand verdict exists parks in AWAITING_CI instead of dead-ending at STUCK; capped against ci_red ping-pong. - ci_poll_exhaustion skips its sweep while a local CI run is in flight, so AWAITING_CI workflows queued behind on-demand CI are not STUCK'd by the remote-CI-sized timeout. - Escalate the workflow after repeated worker-internal-error at a tier instead of retrying to pickup-exhaustion -> STUCK. - forgejo_http: normalise Forgejo's per-gate `status` key to `state` so failing gates are actually counted (they previously all read as pending). - Per-tier worker timeouts bumped +15 min; a timed-out attempt's dirty-worktree residue is preserved on auto-scratch/pr-<N> before the next attempt's reset. - Implementer agents now verify only the CI-flagged gate(s) via a targeted re-run rather than the full local battery before claiming resolved/noop. Re-running the whole suite CI will run anyway was the #1 cause of implementer timeouts; CI remains the real gate and re-dispatches the implementer on red. Telemetry: - Live tab rebuilt on /api/live (controller DB run state + the live OpenCode session forest) after the live_log_writer sidecar was retired with the legacy dispatchers. - Durable per-attempt input/output payloads surfaced in the Live drill-down, archived-session detail, and Workflows timeline. - PR-level cost attribution: worker session tags carry -pr-<n>; backfill_llm_activity_pr.py repairs rows written before the fix. Shared: - tools/controller/session_tag.py — one canonical controller-tag parser shared by the telemetry server and the backfill. Tests: new coverage for local_ci (state machine, log parsing, _summarize_run, GC, in-flight probe, preflight), the ci-not-ready path, the escalation/ci-not-ready SQL counters, the ci_poll in-flight skip, and CI-status payload parsing across both sources. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
0bc734c020 |
style: ruff format the controller-state-machine branch (288 files)
Applies `ruff format` to the accumulated formatting debt on this branch. Formatting-only — no behavioral changes. Required for CI/lint's format gate (`nox -s format -- --check`), which the branch was failing on 288 tracked files that drifted from ruff's canonical style. In-progress WIP files are intentionally excluded so this commit stays a clean formatting-only diff. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
f454670a0e |
chore(controller): launch merge_drive from the pipeline script; rename worker-concurrency env var
The launch script now starts merge_drive by default (T5-7: it is the controller's singleton merge stage — APPROVED -> MERGING -> MERGED). Previously the script killed merge_drive as a "legacy" process but never started it, so controller workflows dead-ended at APPROVED. merge_drive moves into the CONTROLLER process group; --no-merge skips it. Also adds the worker thread-pool concurrency knob (default 2 for the trial harness). Renames the misleadingly-named env var CONTROLLER_MAX_CONCURRENT_WORKERS_PER_MACHINE -> CONTROLLER_MAX_CONCURRENT_WORKER_THREADS_PER_MACHINE: it sizes a ThreadPoolExecutor inside one worker process, it does not spawn worker processes. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
a6986008ee |
feat(controller): trial-5 batch — dispute path, merge pipeline split, conflict-resolver hardening
T5-1 reviewer feedback rendered full-body to the implementer
T5-4/9 implementer dispute path — dispute-at-any-tier with per-tier cap,
OPERATOR_ATTENTION state on stalemate, pr-review-worker-dispute agent
T5-5 reviewer BLOCKING ISSUE EVIDENCE RULE + 5-step validation
T5-7 merge step split into a singleton process — impl/review masters write
APPROVED and stop; merge_drive owns APPROVED -> MERGING -> MERGED
T5-10 merge process is fully deterministic; base conflicts bounce to the
controller's CONFLICT_RESOLVING (LLM); conflict_drive sidecar retired
T5-11 implementer fast success path — verified-clean outcome so a no-op
after conflict resolution doesn't force busywork
T5-12 conflict-resolver permissions fixed across all paths (/tmp/** glob)
T5-13 conflict-resolver PR-intent prehydration (title/body/comments)
Adds tools/_controller_db_bridge.py so merge_drive reads the controller DB
directly (Option B), plus APPROVED + OPERATOR_ATTENTION states, the
dispute/verified-clean events, and the V1 contract fields backing them.
Reviewer model: baseline -> sonnet, dispute -> opus.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
drew
|
3e23853ffa |
fix(controller): batch R — wire 5 V1-contract fields the controller silently dropped
Trial run-3 (2026-05-19) surfaced the first instance of a broader bug class: V1 contract fields existed and agents emitted them, but no controller code wired them into state transitions. An adversarial "walk the happy path" code review found 4 more, all listed below. The class shape: a V1 field is "Required iff X" by contract docstring, the worker emits it correctly, but the master reads the wrong field (or doesn't read it at all), so a critical state transition silently no-ops or drops to the wrong default. FIX #0 — outcome-mapper early-return (committed earlier in this session) — moved role dispatch before the ``outcome is None`` guard so estimator+reviewer+summarizer (V1 contracts without an ``outcome`` field) are correctly handled. Without this fix, all estimator attempts in trial run-3 completed successfully then were silently discarded, stranding all 6 workflows in ANALYZING. FIX #1 — current_tier never written from estimator's recommended_tier File: tools/controller/master/tick.py The ANALYZING→IMPLEMENTING UPDATE wrote only current_state / last_transition_at / entered_state_at. recommended_tier from the estimator payload was never extracted, so every PR ran at the workflow's creation-time tier (typically 0) regardless of what the estimator recommended — the entire tier-escalation ladder was informational-only. Fix: per-event ``extra_set`` clauses; on ``estimator_done`` / ``estimator_metadata_only`` events, set ``current_tier = :rec_tier`` from the payload (with 0..2 validation). Tests: TestEstimatorRecommendedTierWritten (3 cases). FIX #2 — approved_at_sha never passed to merge callback File: tools/controller/master/merging.py, forgejo_http.py ReviewerOutputV1.approved_at_sha is the exact SHA the reviewer signed off on. Pre-fix the MergeCallback signature was ``(owner, repo, pr_number)`` — Forgejo merged whatever HEAD currently was. Race condition: a concurrent push (operator or another driver) between approval and merge would silently merge unapproved code. Fix: extended signature to ``(owner, repo, pr_number, approved_at_sha)``; SQL SELECT now pulls the latest reviewer attempt's output_payload as a subquery; merge_pr forwards it to Forgejo as ``head_commit_id`` (Forgejo refuses with 409 if HEAD has advanced). Defensive: still merges when approved_at_sha is None but logs a WARNING. Tests: TestApprovedAtShaPassedToMerge (2 cases). FIX #3 — tier_last_succeeded column had ZERO writers File: tools/controller/master/tick.py The schema column existed; the merging.py 409-conflict path read it to recover the last-known-good tier; but NOTHING ever wrote to it. Every workflow's tier_last_succeeded was permanently NULL → the 409-recovery path transitioned to IMPLEMENTING(tier=NULL) → scheduler silently coerced to tier 0. Fix: on ``implementer_pushed`` event, ``UPDATE workflows SET tier_last_succeeded = current_tier``. Tests: TestTierLastSucceededWritten. FIX #4 — outcome column NULL for estimator/reviewer/summarizer File: tools/controller/worker/runner.py ``workflow_attempts.outcome`` is the operator-facing audit column. Pre-fix the runner extracted ``output_payload.get("outcome")`` blindly — works for implementer/conflict_resolver but those three roles have no ``outcome`` field. Result: ``SELECT … WHERE outcome IS NOT NULL`` audit queries silently missed every estimator/reviewer/ summarizer attempt. Fix: new ``_derive_outcome_for_audit(role, payload)`` helper synthesizes meaningful per-role values: - implementer/conflict_resolver: payload['outcome'] (unchanged) - reviewer: payload['verdict'] - estimator: 'metadata-only' OR f'tier-{recommended_tier}' - summarizer: 'summarized' Tests: TestOutcomeAuditColumn (parametrized 8 cases). FIX #5 — conflict_resolver new_head_sha never preferred File: tools/controller/worker/runner.py ConflictResolverOutputV1.new_head_sha is "Required iff outcome='resolved'" (the canonical post-rebase branch tip). Pre-fix runner.py used ``commit_shas[-1]`` for head_sha_after — works for normal git rebase --continue but wrong for resolvers that did force-pushed merge commits where the last commit SHA ≠ the branch tip. CI status poll would then poll the wrong SHA. Fix: when role=='conflict_resolver', prefer ``new_head_sha`` over commits[-1]. Tests: TestConflictResolverNewHeadShaUsed (2 cases). ALSO updated existing tests that papered over the original bug: - test_master_outcomes.py: estimator tests used to inject a fake ``"outcome": "(implicit)"`` field; now use real V1 shape (no outcome). Reviewer tests now use ``verdict`` (the real V1 field) not ``outcome``. - test_master_tick.py reviewer tests: same `verdict` switch. - test_master_merging.py: updated all 13 ``lambda o, r, n: ...`` merge-callback stubs to the new 4-arg signature. CONFIRMED-CLEAN (no fix needed) by the same code review: - outcomes.py post-fix-#0 - prefetch.py field reads - prompts.py field accesses - ci_status_poll.py role+outcome filter The above were verified to handle all 5 V1 contract shapes correctly. Total: 802 → 819 controller tests, 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
2ad0958cb7 |
fix(controller): batch O — trial-2 fix: interpolate real attempt_id into prompts
Trial run-2 surfaced the actual blocker: the response-builder MCPs
shared their module-global ``_STATE`` across OpenCode sessions (they're
registered as ``type: local`` in opencode.json, so OpenCode spawns one
subprocess per OpenCode-server lifetime, not per session). The CA2/PD5
cross-session reset I added in the M-batch keyed on
``identity.attempt_id`` — but the prompt template literally read:
1. ``estimator_start(attempt_id=..., workflow_id=..., ...)``
The ``...`` were placeholder ellipses, not interpolated. So agents
guessed ``attempt_id=1`` every single time. The cross-session reset
compared ``1 == 1``, decided "same attempt — no reset", and the prior
attempt's ``finalized=True`` persisted forever. Every estimator
session after the first failed with ``"response already finalized;
no further mutations allowed"`` on every ``_set_*`` call → no
finalize → 30s worker timeout → workflow STUCK.
Observed in trial run-2: 2 successes (attempts 1, 2), then 15
consecutive failures (attempts 3–17 across all 6 workflows) before
the pickup_guard would have STUCK every workflow.
Two fixes (defense in depth):
1. **Unconditional reset on _start** in all 5 builders. We can't
distinguish "agent retry in same session" from "new session reusing
this MCP" reliably — the observable signature is identical. Just
reset whenever ``_STATE.started`` is True; ``reset_for_new_attempt``
logs a WARN if the prior state was in-flight so abandoned attempts
are still visible to ops. The agent's last ``_start`` always wins.
2. **Interpolate real attempt_id + workflow_id + pr_number + head_sha
into all 5 prompts' Output contract sections**. The agent_runner
now injects ``attempt_id`` into ``input_payload`` (matching the
existing ``workspace_dir`` injection pattern). Each ``build_*_prompt``
reads ``input_payload.get("attempt_id")`` and bakes the concrete
value into the MCP-call signature shown to the agent, plus a
``PASS THESE EXACT VALUES`` directive.
Also strengthened each prompt's ``_finalize`` line with
``**You MUST call this tool — without it the controller times out.**``
so the agent understands the contract is hard, not optional.
Updated tests:
- ``test_double_start_refused`` → ``test_double_start_resets_silently``:
pins the new permissive-reset behavior + asserts the WARN log fires.
- ``test_implementer_rejects_intra_session_double_start`` updated for
same reason; now asserts the second _start succeeds + last-wins
semantics (used_tier == new tier).
- NEW ``test_prompt_interpolates_real_attempt_id`` parametrized over
all 5 roles: asserts ``attempt_id=42`` + ``workflow_id=7`` appear
literally in the rendered prompt AND that ``attempt_id=...`` /
``workflow_id=...`` placeholders DO NOT.
Total: 795 → 800 tests, 0 regressions.
The model-override files (.md + .txt) are still in place from the
prior turn — that change is orthogonal to this bug fix; OpenCode
ignores the dispatcher's pass-through per the README, and the actual
generation has been on haiku the whole time. The .md frontmatter
change to sonnet will only take effect on the next OpenCode restart.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
a4892f8db9 |
fix(controller): batch N — second-round adversarial review fixes
Round-2 adversarial review on commit
|
||
|
drew
|
84da774212 |
fix(controller): batch M — post-Phase-1m adversarial review fixes
Three rounds of adversarial review (Chief Architect / Principal Dev /
Senior Test Engineer) on commits 3ca794be7..db12f45ac surfaced ~35
issues. This commit addresses 25+ across criticals, highs, and
mediums, and adds 40 new tests covering the changes plus key gaps the
review identified.
CRITICALS (M1):
- CA1: stale {role}_output.json from a prior attempt on the same
per-PR workspace was readable as "fresh" output of the new attempt.
agent_runner now unlinks the MCP-canonical path AND every fallback
path BEFORE the session runs.
- CA2/PD5: opencode.json-registered MCP subprocesses persist across
OpenCode sessions, but BuilderState was module-singleton. Added
reset_for_new_attempt() + cross-session detection (compare
identity.attempt_id) to every *_start; force-resets with WARN if
prior attempt was interrupted (timeout / lost lock).
- PD3: inline-JSON callback could overwrite an MCP-written canonical
V1 file with adapted-from-prose garbage. Callback now inspects
existing files and skips when V1 is already present.
- PD4: FORGEJO_URL = .rstrip("/api/v1") is a character-set strip —
catastrophic for hosts whose path contains /v1 in the middle.
Replaced with explicit endswith()-based suffix strip.
- CA10: clone URL embedded $FORGEJO_TOKEN, persisted into
.git/config where any agent could cat it. Token now sourced via
local credential.helper at clone-time, URL kept clean.
- CA12: state.finalized was set BEFORE the file write, so disk-full
/ OSError left the agent unable to retry finalize. Reordered.
HIGHS (M2):
- CA3/PD12: output_path validation (NUL-byte rejection, must be
absolute, parent-not-file check) in finalize_and_emit.
- CA6: ci_status_poll SELECT only considered implementer attempts;
conflict_resolver also pushes commits. SQL now unions both roles.
- PD9: ci_status_poll could advance on a stale "resolved" SHA from a
blocked attempt (whose head_sha_after == head_sha_before). Added
outcome='resolved' filter.
- CA8: cancelled/stale CI states mapped to ci_red_retry_same_tier,
burning pickup_count on healthy PRs. Both now wait (treated as
operator/system action, not failure). timed_out stays red.
- TE9: unknown Forgejo CI states now WARN-log instead of silently
being treated as pending — operators see new state strings.
- PD8: ci_status_poll event_type strings standardized to match the
state-machine event names (ci_green / ci_red_retry_same_tier)
instead of legacy ci-green / ci-red.
- CA7: inline-JSON callback now checks lost_lock_check BEFORE write
so a file isn't staged after lock loss.
- PD10: atomic .tmp + os.replace writes in both MCP finalize and
inline callback so the poller never sees a half-written file.
- PD16: inline_output_callback exceptions now re-raise as WorkerError
instead of being silently logged (root cause was buried 30s later
in a canonical-output timeout).
- CA9: WorkerConfig manual rebuild on --max-concurrent/--poll-interval
silently dropped new fields. Use dataclasses.replace, matching
round-4 P5 fix in master/__main__.py.
MEDIUMS (M3) — legacy_adapter quality upgrades:
- PD1: unrecognized confidence values now WARN instead of silently
defaulting to "medium" — surfaces agent prompt drift.
- PD2: estimator recommended_tier clamped to {0,1,2} so an out-of-
range int doesn't bypass the adapter's whole purpose.
- PD7: reviewer blocking_issues list-of-strings coerced into the
list-of-BlockingIssue-dict shape strict_parse requires.
- PD13: conflict_resolver prompt defaults tier=1 + warns instead of
raising; the scheduler always sets it but defends against drift.
- PD14: summarizer summary < 50 chars padded with a clear marker so
strict_parse accepts it (and the truncation is visible).
- PD15: implementer blockers capped at 4096 chars each so a buggy
agent can't blow up audit log / DB column.
- PD17: launch script accepts either FORGEJO_TOKEN or GITEA_TOKEN
with a clear error if both are unset.
- PD22: conflict_resolver adapter accepts singular commit_sha
fallback, matching implementer.
- CA4: every adapter invocation logs role + payload key fingerprint
so operators can measure agent-migration progress.
- estimator + summarizer now have explicit _start tools (the prompts
already referenced them; previously absent → first call would fail).
TESTS (M4) — added 40 tests in test_post_review_fixes.py:
- Cross-session MCP state reset (implementer + reviewer + estimator
+ summarizer; intra-session double-start still rejected).
- finalize_and_emit output_path precedence (arg > env > stdout),
parent-dir creation, rejection of relative/NUL paths, failed-write
leaves state retryable.
- legacy_adapter quality: tier clamping, blocker cap, non-string
commit warning, blocking_issues string coercion, conflict_resolver
full roundtrip + non-resolved head clearing, summarizer padding,
confidence warning, V1-passthrough no-log.
- opencode.json registration parity: every MCP the prompts name is
registered with the correct module path.
- Per-role prompts mention {role}_output.json (canonical poller path)
+ the "DO NOT emit chat-JSON" directive.
- FORGEJO_URL suffix-strip parametrized table.
- agent_runner stale-file cleanup: prior-attempt file is unlinked
before a new session can read it as phantom output.
Also updated 2 pre-existing tests for the CA8 / PD8 / PD13 behavior
changes (cancelled→wait, event_type renaming, conflict_resolver
default-tier warning).
Total: 741 → 781 tests, 0 regressions.
DEFERRED (M5 follow-up — non-trial-blocking):
- CA5: head_sha verification via git cat-file (requires subprocess).
- CA11: discovery_interval_s wall-time cadence (vs iteration count).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
db12f45acb |
feat(controller): Phase 1m — wire response-builder MCPs into OpenCode (Option A)
The trial-path expedient (legacy_adapter harvesting chat-JSON) is now
the SAFETY NET; the architecture-intended path (agents call the
controller's response-builder MCPs, which validate against V1 +
write canonical JSON) is now wired end-to-end.
Four pieces:
1. MCP `finalize` accepts `output_path` argument
``mcp/_builder_base.finalize_and_emit`` adds ``output_path: str | None``
kwarg. Precedence: explicit arg > ``CONTROLLER_CANONICAL_OUTPUT_PATH``
env > stdout. This solves the per-attempt path problem that
blocked opencode.json registration (the env var is static; the
per-attempt path comes from the prompt, the agent passes it as a
tool arg). Each role's finalize updated:
- estimator_finalize(output_path)
- implementer_finalize(output_path)
- reviewer_finalize(output_path)
- conflict_finalize(output_path)
- summarizer_finalize(output_path)
Creates parent directory if missing (so the controller doesn't
need to pre-create). Returns ``wrote_to`` in the ok dict so tests
can pin the path.
2. opencode.json registers the 5 controller MCPs
``.opencode/opencode.json`` adds:
- implementer-response-builder
- reviewer-response-builder
- estimator-response-builder
- conflict-resolver-response-builder
- summarizer-response-builder
Each spawned via ``python -m tools.controller.mcp.{role}_builder``
with PYTHONPATH=/repo-root so the controller imports resolve.
3. Controller prompt builder injects the EXACT tool sequence
``worker/prompts.py`` rewrites each role's "Output contract"
section. The old "PREFERRED MCP / FALLBACK file-write" instruction
becomes a single REQUIRED contract: numbered tool calls (``X_start``,
``X_set_*``, ``X_finalize(output_path=...)``) with the explicit
per-attempt path baked in. Explicit "DO NOT emit a JSON object in
your final chat message" instruction to override the legacy
contract baked into the agent system prompts.
4. Agent permission whitelists include the new MCPs
- ``.opencode/agents/task-implementor.md`` (source for tier-{0,1,2,min}
variants — regenerated via tools/sync_tier_models.py)
- ``.opencode/agents/pr-review-worker.md``
- ``.opencode/agents/estimator-implementation.md``
- ``.opencode/agents/conflict-resolver-worker.md``
Each adds the matching ``"{role}*": allow`` pattern.
Safety net preserved:
The legacy_adapter (commit
|
||
|
drew
|
bcc59d38af |
fix(controller): trial-path output channel — harvest inline JSON + legacy-shape adapter
Trial run-1 surfaced the real failure mode: the existing agent
prompts emit their result as a single JSON object in the FINAL
assistant message (the legacy dispatch_implementer/dispatch_review
contract). They DON'T call the controller's response-builder MCPs
(not registered in opencode.json) and they DON'T write a file. The
worker's canonical-output poller timed out every attempt.
Fix (two pieces):
1. INLINE JSON HARVEST
``_opencode_worker.run_session_blocking`` already extracts the
last JSON object from the agent's final response into
``SessionResult.parsed_json``. The controller's opencode_session
adapter was ignoring it.
- ``opencode_session.py:run_opencode_session`` now accepts an
``inline_output_callback`` kwarg. After the session completes,
if parsed_json is set, the callback fires.
- ``agent_runner.py`` provides the callback: writes the parsed
JSON to the canonical-output path so the existing poller picks
it up uniformly with the MCP + file-write channels.
2. LEGACY → V1 SHAPE ADAPTER (``worker/legacy_adapter.py``)
The legacy JSON shape doesn't match V1 contracts:
- estimator: legacy {recommended_tier, is_confident, reasoning}
vs V1 {output_version, recommended_tier, is_metadata_only,
confidence, reasoning, wallclock_seconds}
- implementer: legacy {outcome: resolved|unresolved, ...} vs V1
{outcome: resolved|rebase-failed|blocked|noop|competence-failure,
commit_shas, blockers, used_tier, ...}
- reviewer: legacy {verdict, ...} vs V1 {output_version, verdict,
blocking_issues, suggested_next_action, ...}
``adapt_to_v1(role, payload, tier, wallclock_seconds)`` normalizes
each role's legacy shape into the corresponding V1 dict:
- adds ``output_version="V1"``
- maps outcome enums (``unresolved`` → ``blocked``)
- synthesizes missing required fields with sensible defaults
- coerces singular commit_sha → list[commit_shas]
- converts legacy is_confident bool → confidence string
Already-V1 payloads pass through unchanged. The agent_runner
wraps the inline_output_callback to run the adapter before
writing to the canonical path.
Tests (+12 in test_legacy_adapter.py):
- Each role's adapt: legacy in, V1-validating-via-strict_parse out
- Already-V1 passthrough
- Outcome enum mappings + invalid-outcome fallback
- Singular commit_sha normalization
- Non-dict / unknown-role passthrough (lets strict_parse raise)
Total: 738 controller tests pass (+12 net), 0 regressions.
Long-term: update agent prompts to emit V1 directly, register the
MCPs in opencode.json. For the trial, this adapter lets the existing
agents flow through the new controller unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
f93f0d1c53 |
fix(controller): round-4 P2 — defensive input_payload normalization
If the DB returned NULL input_payload (legacy/seeded data, schema bug, hand-edited rows), agent_runner crashed on ``dict(None)`` → WorkerError → master re-pickups → STUCK after MAX_PICKUPS reaps. Silent infinite-loop until exhaustion. Fix: ``worker/runner.py:run_one_attempt`` checks isinstance(dict) on entry; substitutes empty dict + logs WARNING. Agent still runs normally with an empty payload. Test: ``test_none_input_payload_doesnt_crash`` — passes input_payload=None + verifies the attempt completes normally. Total: 712 controller tests pass (+1 net). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
f57d9f9478 |
fix(controller): batch L — round-4 trial-blockers (A1, P1, P3, P4, P5, T5)
Round-4 adversarial review found 5 trial-blockers + 1 silent-debt
item the post-round-3 deep pass missed. All fixed.
A1 — pre-clone the workspace so the agent has a worktree to operate on
``worker/__main__.py``: the agent_runner closure now constructs a
``PerPRWorkspace`` from input_payload.owner/repo/pr_number + the
FORGEJO_URL+FORGEJO_TOKEN env vars. Pre-flight:
- ``workspace.ensure_present()`` creates the dir skeleton.
- ``workspace.clone_if_absent()`` clones the repo into
``{workspace_dir}/worktree/`` if not already present (idempotent).
- ``workspace.fetch_and_validate(head_sha, head_ref)`` refreshes +
verifies the workspace is at the expected head. ``StaleInputError``
→ ``WorkerError(outcome='stale-input')`` so the master re-prefetches
without burning a pickup. ``RuntimeError`` → ``worker-internal-error``.
Previously the agent saw an empty workspace_dir + had no repo.
P1 — partial-write defense in the canonical-output poller
``worker/agent_runner.py:_wait_for_canonical_output`` now polls each
path with a two-pass quiescence check (size stable + content parses
as JSON) before returning. Partial writes (agent crashed mid-flush)
are skipped + the polling loop continues. The previous
``f.read().strip()`` returned partial JSON which then tripped
``ContractValidationError`` → ``worker-internal-error`` with no
record of WHICH path; now logs source path on every read.
P3 — TOCTOU defense in promote_discovered
``master/promote.py``: the UPDATE now filters
``current_state='DISCOVERED'``. If a concurrent reconciliation
moved the row off DISCOVERED between SELECT and UPDATE, rowcount=0
+ we skip the event-row write. No duplicate audit entry; no
overwriting a pause-by-label-removal.
P4 — explicit tuple-length validation in reconciliation_args + discovery_args
``master/loop.py``: previously a 6-tuple silently fell into the
``else`` 4-tuple unpack, raised ValueError("too many values"), got
swallowed by the per-iter ``except Exception``, and reconciliation
silently died forever. Now: ``elif n == 4`` + ``else: raise TypeError``.
The TypeError still hits the per-iter except (so the loop doesn't
crash) but ``logger.exception`` surfaces the actionable message in
journald. Operator sees "reconciliation_args must be a 4- or 5-tuple;
got length 6" instead of zero indication.
P5 — --tick-interval CLI flag preserves other config fields
``master/__main__.py``: replaced the manual ``MasterConfig(...)``
rebuild (which dropped reconciliation/ci_poll/discovery intervals)
with ``dataclasses.replace(cfg_loop, tick_interval_s=args.tick_interval)``.
Operators who pass --tick-interval no longer silently revert the
other intervals to defaults.
T5 — scheduler._commit_escalation uses safe_json_dumps
``master/scheduler.py``: the escalation event row's payload was the
only call site that bypassed safe_json_dumps. Now consistent — a
future contributor adding a datetime/Decimal field won't trip raw
json.dumps at runtime.
Tests (+4 net):
- ``test_worker_agent_runner.py::test_partial_write_not_read``: pins
P1 (truncated fallback file + valid MCP output → MCP wins).
- ``test_master_promote.py::test_toctou_state_change_between_select_and_update``:
pins P3 (steal state via monkey-patch → no double-promotion, no
extra event row).
- ``test_master_loop.py::test_reconciliation_args_wrong_length_logs_not_silent``:
pins P4 (6-tuple → logged error, not silent forever).
- ``test_entry_points.py::test_tick_interval_flag_preserves_other_cfg_fields``:
pins P5 (env-set non-default intervals survive --tick-interval).
Total: 711 controller tests pass (+4 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
7bf39a0b51 |
fix(controller): RB8 — head_sha bookkeeping in _write_outcome
The worker's _write_outcome never wrote head_sha_before / head_sha_after
on the workflow_attempts row. But tick.py reads both to compute
head_sha_advanced, which the outcome mapper REQUIRES to distinguish
``implementer_pushed`` (true push) from ``implementer_blocked``
(worker said resolved but git didn't move).
Without these columns set, every implementer attempt's "resolved"
outcome mapped to a no-op event, and workflows would stall
permanently after the agent ran successfully.
Fix:
- ``worker/runner.py:_write_outcome`` accepts ``head_sha_before`` +
``head_sha_after`` kwargs and writes both columns in the UPDATE.
- ``run_one_attempt`` computes:
- head_sha_before = input_payload["head_sha"] (what the agent
was told to start from)
- head_sha_after = output_payload["commit_shas"][-1] if any
commits were produced; falls back to head_sha_before otherwise
so tick.py correctly sees "no advance" when the agent didn't
commit.
Tests (+2):
- test_head_sha_before_after_recorded — pins the happy path where
the agent committed and head_sha advanced.
- test_no_commits_means_head_sha_unchanged — pins the no-advance
case where head_sha_after equals head_sha_before.
Total: 707 controller tests pass (+2 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
0c9e8fdd69 |
fix(controller): trial-path output channel — fallback file-write for unwired MCP layer
RB7 (newly discovered) — response-builder MCPs are NEVER REACHABLE
from the OpenCode agent:
The controller spawns the per-attempt response-builder MCP via
``python -m tools.controller.mcp.{role}_builder`` with stdin=PIPE
+ CONTROLLER_CANONICAL_OUTPUT_PATH set. But:
- ``.opencode/opencode.json`` does NOT register the controller MCPs.
- OpenCode only knows about MCPs it sees in opencode.json (graphify,
ci, forgejo, git, handoff).
- The OpenCode agent has NO transport to call
``implementer_finalize`` because OpenCode doesn't see the MCP.
- The controller's spawned MCP subprocess sits waiting for JSON-RPC
on stdin that no one sends.
- The agent does work, the session ends, the canonical-output-file
is empty, agent_runner times out with WorkerError.
This would have been the single biggest blocker to running an actual
Phase 2 trial. The MCP layer is structurally complete but not
plumbed into OpenCode — pivoting requires either registering them
in opencode.json (which doesn't support per-attempt env injection)
OR adding a fallback channel.
Fix (trial-path):
- ``worker/prompts.py``: each role's output-contract section now
documents BOTH the PREFERRED MCP call AND a FALLBACK direct
file-write to ``{workspace_dir}/{role}_output.json``. The agent
can pick either; once the MCP layer is wired (Phase 1m) it'll
naturally prefer the MCP.
- ``worker/agent_runner.py:_wait_for_canonical_output``: now polls
the MCP-canonical path AND a list of ``fallback_paths``. Call
site adds ``{workspace_dir}/{role}_output.json`` to fallback
paths. First non-empty wins.
Tests (+2):
- ``test_fallback_file_picked_up_when_mcp_silent``: agent writes
only the fallback file (no MCP drive); runner reads it.
- ``test_first_nonempty_output_wins``: documents the
poll-both-channels semantic.
Documented Phase-1m work:
- Register controller MCPs in ``.opencode/opencode.json``
- Solve per-attempt env injection (the canonical_output_path is
per-attempt; opencode.json env is static — needs a request_id
arg on finalize() OR a workspace-relative output convention)
Until that's done, the trial uses the fallback file-write path.
Total: 705 controller tests pass (+2 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
251eeb21ff |
fix(controller): more pipeline run-blockers — merging tick, periodic discovery, worker create_all, systemd ordering
Continuing the round-3 deep-pass cleanup. Three more run-blockers + one robustness fix. RB5 — MERGING handler never invoked from master loop: ``run_merging_tick`` was exported by the master package but no caller fired it. Workflows that transition to MERGING (via reviewer approval) would sit there indefinitely with no Forgejo merge call. Fix: - ``master/loop.py`` accepts a ``merging_args=(owner, repo, merge_callback)`` kwarg. When set, the tick fires every iteration (cheap if no workflows in MERGING). - ``MasterTickReport`` gains ``merging: MergingHandlerReport | None``. - ``master/__main__.py`` wires it from the Forgejo callback bundle. RB6 — periodic discovery never fires: ``run_discovery`` was only called at startup via ``run_startup_backfill`` + the ``--discovery-only-once`` smoke flag. PRs created after master startup would not be discovered until the master restarted. Fix: - ``master/loop.py`` accepts ``discovery_args=(owner, repo, list_prs, list_issues)`` or the 5-tuple with kwargs. Periodic tick on its own cadence (``CONTROLLER_DISCOVERY_INTERVAL_S``, default 30s). - ``MasterTickReport`` gains ``discovery: DiscoveryReport | None``. - ``master/__main__.py`` wires it + threads ``require_opt_in_label`` through. RB-robust — worker calls create_all defensively: Master is normally responsible for schema creation (workers run After= it via systemd ordering). But if the worker is started in isolation (test / local dev / unit ordering broken), it'd crash on the first query against missing tables. Fix: - ``worker/__main__.py`` calls ``create_all(engine)`` after ``build_engine``. ``create_all`` is idempotent (CREATE TABLE IF NOT EXISTS); safe to call from both master + worker. - ``cleveragents-controller-worker@.service`` adds ``After=cleveragents-controller-master.service`` + ``Wants=cleveragents-controller-master.service`` so systemd enforces the start ordering in production. Total: 703 controller tests pass (no test changes; all new wiring is exercised by master_main_loop tests via the new kwargs). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
febb352618 |
fix(controller): pipeline run-blockers — promoter, scheduler, owner/repo, workspace_dir patch
Round-3 deep pass identified four issues that would have prevented an
actual end-to-end pipeline run:
RB1 — DISCOVERED → ANALYZING never fired in production:
The state machine defines (DISCOVERED, discovery_picked_up) →
ANALYZING but NO production code fires the event. Workflows
created by discovery would sit in DISCOVERED forever.
Fix:
- New ``master/promote.py``: ``run_promote_discovered_tick`` scans
for DISCOVERED workflows + fires ``discovery_picked_up`` via
apply_event (state-machine invariants stay enforced) + emits a
``discovery-promoted`` controller_events row per transition.
- Composes with the master loop's other ticks; runs every iteration
(cheap — typically 0-1 row).
RB2 — scheduler.schedule_next_attempts never called from master loop:
The scheduler was exported by the master package but never invoked.
It creates the ``workflow_attempts`` rows that workers dequeue —
without it, workers would have nothing to pick up.
Fix:
- ``master/loop.py`` now accepts a ``prefetch: PrefetchCallback``
kwarg. When provided, the loop runs promote_discovered + scheduler
every iteration after tick/reaper/reconciliation.
- ``MasterTickReport`` gains ``promote_discovered`` and ``scheduler``
optional fields so on_iteration callbacks see both.
- ``master/__main__.py`` builds a ``PrefetchDataCallbacks`` from the
Forgejo callback bundle and constructs the production
``make_prefetch_callback(engine, callbacks)`` — wires through to
the loop's new prefetch kwarg.
RB3 — owner / repo missing from V1 input contracts:
The implementer / reviewer / estimator / conflict-resolver V1 inputs
had pr_number but not owner/repo. The OpenCode agent would have
had no way to know which Forgejo repo to clone — it would have had
to derive owner/repo from process env, coupling the worker to a
single repo.
Fix:
- ``contracts/v1.py``: added ``owner: str`` and ``repo: str``
(min_length=1) to ImplementerInputV1, ReviewerInputV1,
EstimatorInputV1, ConflictResolverInputV1.
- ``master/prefetch.py``: builders populate owner/repo from the
Workflow row (already known at prefetch time).
- Existing test fixtures in ``test_contracts_v1.py`` updated.
RB4 — input_payload.workspace_dir placeholder reached the agent:
Prefetch wrote ``workspace_dir = "<worker-injected>"`` as a
placeholder; the worker never patched it before invoking the
OpenCode session. The prompt builder rendered the literal
placeholder string into the agent's prompt — the agent had no idea
where to clone.
Fix:
- ``worker/agent_runner.py``: patches input_payload.workspace_dir
with the real path immediately before calling run_opencode_session.
Uses a shallow copy so the caller's dict isn't side-effected.
- ``worker/__main__.py``: workspace_dir naming convention is now
``pr-{owner}-{repo}-{pr_number}`` (matches workspace.py's
PerPRWorkspace convention) so the janitor's pr-* glob + the
agent's expected workspace location agree. Falls back to
``pr-attempt-{N}`` for legacy input_payloads missing owner/repo.
Tests:
- ``test_master_promote.py`` (NEW, +7 tests):
- empty DB no-op
- single workflow promoted
- multiple promoted in one tick
- only DISCOVERED targeted (non-DISCOVERED untouched)
- controller_events row emitted with correct shape
- idempotent after first promotion
- LoopIntegration end-to-end: DISCOVERED → ANALYZING → pending
estimator attempt visible in workflow_attempts (pins the entire
previously-broken pipeline from discovery to enqueue)
Total: 703 controller tests pass (+7 net), 0 regressions.
Without these four fixes, the pipeline would have looked alive in
unit tests but produced zero work in a real deployment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
c4a7f0027d |
fix(controller): batch H — round-2 important items (N5-N10)
Six items from the round-2 adversarial review's "important" list. N5 — restricted JSON encoder replaces ``default=str``: ``default=str`` silently stringified custom objects, sets, and bytes to ``"<MyObj at 0x...>"`` — masking worker output bugs. Now uses a restricted encoder (``tools/controller/_json_safe.safe_json_dumps``) with an allowlist: - datetime / date → ISO-8601 string - Decimal → str (preserves precision) - UUID → canonical string - Path → str - set / frozenset → sorted list (best-effort) - Everything else → TypeError (a worker output regression surfaces loudly instead of writing garbage to the DB) Replaces ``json.dumps(..., default=str)`` at: - ``worker/runner.py`` (terminal-state UPDATE write) - ``master/scheduler.py`` (input_payload INSERT + UPDATE) Tests: ``test_json_safe.py`` (+15 tests) — every allowlisted type + rejected types (custom class, bytes, complex) + nested structures + kwargs forwarding. N6 — strict parser check now runs BEFORE backfill + loop: Previously the strict-parser exit could run AFTER backfill (the test ``test_strict_parser_coverage_blocks_startup_with_stubs`` passed because backfill's ``fail_if_called`` AssertionError was swallowed by the bare ``except Exception``, then strict exited 2). The test asserted the right outcome via the wrong path. Fix: - ``master/__main__.py``: parser-coverage check moved to immediately after engine creation, BEFORE backfill + loop. Strict-mode failure exits 2 without wasting a Forgejo round-trip + without dependent code paths firing. - Test refactored: count-based assertions on backfill and loop call counts (0 each) instead of fail_if_called. Catches regressions where the strict check moves back below either. N7 — _to_aware_datetime unit-tested in isolation: Previously exercised only via end-to-end comment-filter test. New ``TestToAwareDatetime`` (+12 tests) covers: None, empty string, Z suffix, +00:00 offset, microseconds preserved, naive datetime → UTC, malformed string → None, partial string → None, unsupported types → None, timezone abbreviations → None, equality across Z + offset forms (the regression the helper exists to defend). N8 — runtime=None lazy-import branch tested: Previously all 24 HTTP factory tests injected a fake runtime; the production path (``build_callbacks(cfg=None)`` → sys.path injection + lazy import of ``tools._claim_runtime``) was untested. ``TestBuildCallbacksDefaultRuntime`` (+2 tests): verifies the import succeeds + every ForgejoCallbacks attribute is callable; verifies the import is idempotent (second call doesn't crash on sys.path re-insert). N9 — resume event-row emission asserted: Round-1's batch B added the PAUSE event-row test (``test_label_removed_emits_event_with_reason``) but not RESUME. ``test_label_restored_emits_event_with_reason`` pins the resume's controller_events shape (from_state=PAUSED, to_state=<prior>, reason="opt-in-label-restored", source="reconciliation") so operators auditing the timeline see both pause + resume. N10 — concurrent SQLite dequeue test: The dequeue docstring claims SQLite BEGIN-DEFERRED concurrent dequeues "retry via busy_timeout (5s)" — but no test verified. ``TestSQLiteConcurrentDequeue::test_two_threads_racing_one_wins`` spawns two threads, both attempt dequeue simultaneously via a threading.Barrier. Asserts: neither thread raises SQLITE_BUSY- without-retry, exactly one acquires the attempt, the loser sees no_pending_eligible (winner committed first). Total: 691 controller tests pass (+31 net), 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
147e3403c1 |
fix(controller): batch G — show-stoppers from round-2 review (N1–N4)
Four items the round-2 adversarial review flagged as ship-blockers.
N1 — PID-reuse defense is now WIRED in production:
Round 1's batch D shipped ``subprocess_starttime`` in the sidecar +
janitor checks against it, but NO production code wrote sidecars.
The defense was unwired; tests passed against a code path that
production never invoked.
Fix:
- ``worker/agent_runner.py`` accepts ``workspace_dir`` and
``opencode_server_url`` kwargs. When ``workspace_dir`` is set, it
writes a sidecar (``{workspace_dir}/worker.session``) immediately
after MCP spawn capturing the real PID + starttime from
``/proc/{pid}/stat`` field 22. Removes it on attempt completion.
- ``worker/__main__.py`` builds the per-attempt workspace dir
(``{workspace_root}/pr-attempt-{N}/``) and threads it through the
agent_runner closure with the OpenCode URL. The naming convention
is picked up by the janitor's ``pr-*`` glob; when per-PR shared
workspaces ship (Phase 1k++ follow-up), it changes to
``pr-{owner}-{repo}-{N}``.
- Test: ``TestSidecarWiring`` (+2 tests) verifies the sidecar appears
during the attempt, carries the right PID + starttime + instance,
and is cleaned up post-attempt.
N2 — AWAITING_CI escape event firing is now WIRED in production:
Round 1's batch D shipped ``ci_polling_exhausted`` /
``ci_flake_retries_exhausted`` in TRANSITIONS, but NO production code
emitted them. Workflows could still hang in AWAITING_CI forever.
Fix:
- New ``master/ci_poll.py``: ``run_ci_poll_exhaustion_tick`` scans
workflows whose ``entered_state_at`` is older than
``CONTROLLER_AWAITING_CI_TIMEOUT_S`` (default 7200s) and fires
``ci_polling_exhausted`` via ``apply_event`` → STUCK + emits a
``ci_poll_exhausted`` controller_events row with the threshold
payload.
- ``master/loop.py`` integrates the new tick on its own cadence
(``ci_poll_exhaustion_interval_s`` env, default 300s). Composes
with the existing master loop. ``MasterTickReport`` gains
``ci_poll_exhaustion: CIPollExhaustionReport | None``.
- Tests: ``test_master_ci_poll.py`` (+7 tests) — happy path, fresh
workflow stays untouched, only AWAITING_CI is targeted (other
long-lived non-terminal states ignored), event row shape pinned,
default threshold matches the documented 2h, end-to-end loop
integration (master_main_loop drives the exhaustion +
workflow → STUCK without operator intervention).
- Dialect-portable SQL (Postgres interval, SQLite julianday).
- Handles SQLite returning TIMESTAMP as str from text() queries
(no .isoformat() on str).
N3 — externally-merged/closed PRs now win over label removal:
Round-1's PAUSE-on-label-removed shipped, but reconciliation
checked the label gate BEFORE checking merged/closed. Operators
removing the opt-in label on an already-merged PR would PAUSE the
workflow forever — never transitioning to MERGED.
Fix:
- ``master/reconciliation.py:_reconcile_one`` re-ordered:
1. Check terminal-state mappings (merged/closed) FIRST — apply
immediately if they fire.
2. THEN the opt-in label gate (pause/resume).
3. Fall through to "consistent" otherwise.
- Tests: ``test_externally_merged_takes_priority_over_label_removal``
+ ``test_externally_closed_takes_priority_over_label_removal``
pin the contract. Both seed an IMPLEMENTING workflow + Forgejo
reporting "merged/closed AND no opt-in label" → workflow
transitions to MERGED/ABANDONED (not PAUSED) + pre_pause_state
stays None.
N4 — graceful handling of empty env vars:
``int(os.environ.get("CONTROLLER_FORGEJO_REQUEST_TIMEOUT_S", "30"))``
crashes with non-actionable ``int('') ValueError`` if the operator
sets the env to empty/whitespace (common when sourcing a partially-
edited /etc/cleveragents/master.env file).
Fix:
- ``master/forgejo_cfg.py:_env_int(name, default)`` — empty or
whitespace-only values fall back to the documented default; only
non-numeric values still raise (with a clear message naming the
variable).
- Tests: ``test_empty_env_value_falls_back_to_default`` +
``test_whitespace_only_env_falls_back`` + updated
``test_malformed_env_raises_value_error`` to match the new
"not a valid integer" wording.
Total: 660 controller tests pass (+13 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
d71046b9a0 |
fix(controller): batch D — PID-reuse, AWAITING_CI escape, flake bound, scheduler skip
Four safety items from the consolidated adversarial-review punch list.
ITEM 8 — PID-reuse hazard in janitor:
The janitor SIGKILL'd whatever process happened to live at the
sidecar's recorded subprocess_pid. Between sidecar write and janitor
sweep, the OS can reuse the PID for an unrelated process; the janitor
was killing innocents under fork-heavy workloads.
Fix:
- ``session_sidecar.py``: added ``subprocess_starttime`` field
(Optional[int]) + ``read_proc_starttime(pid)`` helper that reads
``/proc/{pid}/stat`` field 22 (clock ticks since boot — monotonic
for a (boot, pid) pair).
- ``WorkerSession.from_dict`` filters unknown keys so forward + back
compat with sidecars from earlier/later versions is preserved.
- ``janitor._pid_alive`` and ``_kill_with_grace`` accept
``expected_starttime``; on mismatch they short-circuit and DON'T
signal the impostor.
- ``_kill_with_grace`` return semantics tightened: True iff a signal
was actually delivered (False for "PID gone" / "PID reused"). The
``sessions_killed`` counter now reflects real kills.
ITEM 9 — AWAITING_CI escape from infinite poll:
Previously AWAITING_CI could only exit via ``ci_green`` /
``ci_red_*`` / ``ci_flake_retry`` — if CI hangs forever (runner
outage, broken integration, etc.) the workflow had no controller-
driven STUCK path; only operator_unstick could rescue it.
Fix: new ``ci_polling_exhausted`` event → STUCK. The master's
AWAITING_CI poll handler is the natural place to emit it once a
threshold passes (deferred to a follow-up — Phase 1k+ ships the
event in the table; the timer fires it).
ITEM 10 — ci_flake_retry was unbounded:
The ``ci_flake_retry`` self-loop on AWAITING_CI had no encoded
ceiling. Pathological flaky CI could loop forever (the docstring
said "retry once per gate" but nothing enforced it).
Fix:
- New ``workflows.ci_flake_retries_remaining`` column (server_default
'1', default 1 — operators tune via ``CONTROLLER_CI_FLAKE_RETRIES``
at startup or via direct UPDATE).
- New ``ci_flake_retries_exhausted`` event → ESCALATING. Master
decrements the column on each ci_flake_retry; at 0 the next CI
failure routes through ci_red_* (regular path) or this new
event (escalates if the operator wants a hard ceiling).
ITEM 11 — scheduler now skips PAUSED workflows:
Without this, the scheduler could enqueue a fresh attempt for a
PAUSED workflow between two reconciliation ticks (race: label
removed at T+0, reconciliation runs at T+300, scheduler ticks at
T+30 with stale DB state). The window is at most one attempt of
worker work.
Fix: ``schedule_next_attempts`` SQL now lists only
{ANALYZING, IMPLEMENTING, REVIEWING, CONFLICT_RESOLVING, ESCALATING}
explicitly; PAUSED is excluded by absence. Reconciliation owns the
PAUSED → resume transition; scheduler doesn't touch it.
Schema additions:
- ``workflows.ci_flake_retries_remaining`` (INTEGER NOT NULL DEFAULT 1)
- ``workflows.awaiting_ci_started_at`` (TIMESTAMP NULL) — for the
poll-exhaustion timer (timer impl deferred; column is staged).
Tests:
- TestReadProcStarttime — 3 tests (Linux skip-guard) for the
/proc/pid/stat parser (self-pid > 0, missing pid is None,
invalid pid is None).
- TestJanitor::test_pid_reuse_defended_via_starttime — pins the
contract end-to-end (real subprocess + fabricated wrong starttime
→ janitor doesn't signal).
- TestPhase1kPlusTransitions — 5 tests pinning the new events +
proving the load-bearing invariants still pass.
- test_scheduler_skips_paused_workflows — pins item 11.
Total: 603 controller tests pass (+10 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
6ba1926c52 |
fix(controller): batch A — datetime json, Forgejo state map, ISO compare, dequeue docs
Four narrow bug fixes flagged by adversarial code review (items 1, 5, 6, 7 from the consolidated critique). ITEM 1 — datetime → json.dumps crash (silent write-after-work failure): - ``worker/runner.py:274`` and ``master/scheduler.py:283`` now pass ``default=str`` to ``json.dumps`` so nested datetime fields (e.g. CISummary.observed_at) serialize without raising. - Before this fix: a worker would do its real work, then crash on the terminal-state UPDATE with TypeError, get recorded as ``worker-internal-error``, and the output payload would be lost. - Test: TestDatetimeSerializationSafety in test_master_ci_summarize + test_scheduler_handles_datetime_in_input_payload in test_master_prefetch (both pin the regression — the with-default test passes, the without-default test asserts the TypeError so future maintainers see the failure mode). ITEM 5 — Forgejo state mapping completeness: - Extended ``_FORGEJO_STATE_TO_GATE_STATUS`` in ``master/ci_summarize.py`` to cover ``cancelled``, ``timed_out``, ``action_required``, ``queued``, ``in_progress``, ``neutral``, ``skipped``, ``stale`` — states observed across Forgejo / Gitea / GH-mirror that previously collapsed to ``pending``, telling the implementer "CI is still running" when really a job was cancelled. - ``cancelled`` / ``timed_out`` / ``action_required`` / ``stale`` now map to ``error`` (the gate failed). - ``queued`` / ``in_progress`` stay ``pending`` (still running). - ``neutral`` / ``skipped`` → ``passed``/``skipped`` (informational). - Test: TestExtendedForgejoStates — 6 tests covering each new state. ITEM 6 — lexicographic ISO comparison drops/dupes comments: - ``master/prefetch.py:_comment_bodies_since`` and ``_iso`` replaced with ``_to_aware_datetime`` + datetime comparison. Forgejo emits ``2026-05-18T12:00:00Z``; Python's ``datetime.isoformat()`` emits ``2026-05-18T12:00:00+00:00`` — a string compare gives 'Z' (0x5A) vs '+' (0x2B) which silently misorders timestamps. - Now parses via ``datetime.fromisoformat`` (with Z → +00:00 rewrite), defaults naive timestamps to UTC, and compares as ``datetime``. - Test: test_comments_filter_handles_z_suffix_vs_offset_form pins the regression. ITEM 7 — false BEGIN IMMEDIATE claim in dequeue docstring: - The dequeue docstring claimed ``BEGIN IMMEDIATE`` was applied by session_scope; it wasn't. Attempted a global ``begin``-event listener that conflicted with StaticPool's shared-connection model (test_prefetch_callback_works_in_scheduler broke). - Reverted to a documentation fix: SQLite stays on default BEGIN DEFERRED (the SQLITE_BUSY retry via busy_timeout=5s is acceptable for single-host dev) + the docs make MULTI-MACHINE REQUIRES POSTGRES explicit at three call sites (db/session.py, db/dequeue.py, RUNBOOK.md was already updated in Phase 1l). Postgres has FOR UPDATE SKIP LOCKED which is what production actually uses. Tests: 586 controller tests pass (+10 new), 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
1e27039555 |
feat(controller): render CI raw_log_excerpt + target_url in worker prompts
Phase 1j shipped prefetch + parsers; Phase 1i shipped the prompt templates. But the prompt's CI-summary section only rendered each failing gate's name, status, and summary_line — the raw_log_excerpt (up to 16KB captured per gate) and target_url were sitting in the input_payload dict, never surfaced to the LLM. Impact: for stub-parser gates (7 of 10 tools — robot_framework, bandit, semgrep, vulture, radon, slipcover, build), the implementer saw "parser pending; see raw_log_excerpt" with no log in the prompt. It would have to shell out via OpenCode to re-fetch the log, which is wasteful and a regression vs the existing pipeline. This commit: - Renders ``target_url`` as ``log: <url>`` for failing gates so the LLM can curl it if needed. - Renders ``raw_log_excerpt`` inside a fenced code block, truncated to 3KB per gate (storage cap is 16KB; trim to prompt-friendly size with a marker preserving the original byte count). - Renders ``composite_findings`` as nested sub-findings (security_scan → bandit head + semgrep/vulture children, each with their own excerpt) so multi-tool gates are first-class in the prompt. - Skips the fence + URL for passing gates (already filtered) and for failures with no excerpt (defensive — keeps clean parsers from emitting empty log blocks). Conversion: the 20% of CI failures handled by stub parsers go from "broken — implementer doesn't see the log" to "less structured — implementer sees the raw log and can reason about it." Real parsers still ship in priority order as a follow-up. Tests (+7 in TestCIFailureRendering, 0 regressions): - target_url rendered for failing gate - target_url skipped for passing gate - raw_log_excerpt rendered in markdown code fence - Long log excerpt truncated with marker - Missing excerpt → no log block (no empty fence) - composite_findings render as sub-findings with their own excerpts - Stub-parser-pending failure now surfaces the actual log (the whole point of the commit) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
1b09a2054f |
feat(controller): Phase 1i — per-role prompt templates
Replace the JSON-dump prompt stub with structured per-role builders
that render the V1 input contract fields in human-friendly sections.
The agent's system prompt (.opencode/agents/{name}.md) provides the
HOW (tool calls, response shape); this module provides the WHAT (the
per-attempt context drawn from input_payload).
Per-role builders in tools/controller/worker/prompts.py:
- build_implementer_prompt — branch coords, CI summary (highlights
failing gates), active reviewer state, new comments, prior attempts,
optional allowed_files scope, output contract reminder
- build_reviewer_prompt — PR coords, CI summary, full diff (with
code fence), implementer's latest claim, prior reviewer outputs
- build_estimator_prompt — PR/issue context (title, body, diff
summary, head_sha), tier-pick guidance
- build_conflict_resolver_prompt — coords, conflicted files (or
porcelain fallback), prior implementer outputs
- build_summarizer_prompt — aged-out attempt + prior running
summary; instructs terse condensation
- build_prompt(role, tier, input_payload) — dispatcher
Truncation budgets (in chars):
- full_diff: 12,000
- pr_body / summaries: 4,000
- prior-attempt JSON: 2,000
- comment one-liner: 200
Wired as the default prompt builder in opencode_session via
``_default_prompt_for``. wire_opencode_session() callers that pass
prompt_builder=... override per-test.
Tests (+35 in test_worker_prompts.py, 1 updated adapter test, 0
regressions):
- Per-role: section headers present, required fields rendered,
missing fields fall back to (unavailable), oversize fields
truncated
- Dispatcher: routes by role; raises ValueError for missing tier on
tiered roles + unknown role
- Adapter integration: _default_prompt_for delegates to build_prompt
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
7b1b68b784 |
feat(controller): Phase 1e — master + worker __main__ entry points
Runnable as `python -m tools.controller.{master,worker}` for systemd
deployment. Wires the production callbacks (Forgejo HTTP +
OpenCode session adapter + agent_runner) into the previously-shipped
main loops.
tools/controller/master/__main__.py:
- Args: --owner, --repo (required); --opencode-url, --log-level,
--tick-interval, --discovery-only-once (smoke flag).
- Reads CLEVERAGENTS_DB_URL (returns 2 if unset — operator-actionable
error visible in systemd logs).
- build_cfg_stub reuses _mcp_common.ForgejoCfg → PAT rotation +
Forgejo URL env vars work without a new config module.
- SIGTERM/SIGINT → stop_event → master_main_loop drains + exits.
- --discovery-only-once: runs one discovery sweep + exits. Useful for
initial backfill or smoke testing the Forgejo callback wiring
without committing to the long-running loop.
tools/controller/worker/__main__.py:
- Args: --opencode-url, --roles (default all 5; comma-separated),
--max-concurrent, --poll-interval, --log-level, --no-startup-janitor.
- Empty --roles after parsing → exit 2.
- Startup sequence:
1. Run orphan-workspace sweep (unless --no-startup-janitor)
2. Build OpenCode session adapter via wire_opencode_session
3. Wrap production_agent_runner with the session injected
4. Wire SIGTERM/SIGINT → stop_event
5. Enter worker_main_loop
9 new tests in test_entry_points.py:
- Arg parsing: master --owner/--repo required; both DB-URL-missing
returns 2; worker empty-roles returns 2.
- --help works via subprocess for both (no SystemExit issue).
- Master --discovery-only-once smoke: stubs build_callbacks + cfg
builder; verifies exit 0 + the discovery call chain runs.
- Worker --no-startup-janitor: stubs sweep + loop; verifies sweep
is skipped when flag set, runs when not.
Total: 407 controller tests; full auto_agents suite 2769 pass.
Controller v1 is now end-to-end deployable as two systemd units
(one per machine for each instance). The remaining work for a real
production rollout is:
- Real prefetch callbacks for the scheduler (assembling V1 input
payloads from Forgejo data)
- Backfill at master startup (existing-PRs → DISCOVERED rows)
- Reconciliation tick (DB ↔ Forgejo sync)
- Per-role prompt templates (the V1 prompt stub works; specialised
per-role prompts will land as agents are migrated)
|
||
|
drew
|
bb8d872ba7 |
feat(controller): Phase 1d-4 — OpenCode session adapter
The production wrapper that wires the existing
_opencode_worker.run_session_blocking into the controller's
run_opencode_session protocol that production_agent_runner expects.
tools/controller/worker/opencode_session.py:
- agent_name_for(role, tier): role+tier → OpenCode agent name.
- implementer + tier {0,1,2} → task-implementor-tier-{0,1,2}
- reviewer → pr-review-worker
- estimator → estimator-implementation
- conflict_resolver → conflict-resolver-worker
- summarizer → controller-summarizer (new agent name)
- DEFAULT_TIER_TIMEOUT_S: per-tier wallclock budgets via env vars.
Defaults match plan v9 (tier 0: 600s, tier 1: 1200s, tier 2: 1800s).
Reviewer/estimator/conflict use the default-agent timeout (600s).
- wire_opencode_session(opencode_server_url, ...) → callable matching
the production_agent_runner's run_opencode_session contract.
- Resolves role+tier to agent name + per-tier timeout
- Builds the prompt (default stub or custom builder; per-role prompt
templates are a follow-up)
- Calls run_session_blocking with an on_poll callback that raises
WorkerLostLock if the controller's lost_lock_check returns True
mid-session (heartbeat thread reaper detected stolen lock)
- Routes SessionResult.status:
- 'completed' → return None (MCP's canonical output is in the
tempfile; production_agent_runner reads it after we return)
- 'timeout' → raise WorkerError(worker-internal-error)
- 'transport-error' → raise WorkerError(worker-internal-error,
including error_kind for forensics)
- unknown → raise WorkerError defensively
- Any unexpected exception from run_session_blocking wrapped as
WorkerError(worker-internal-error).
- Production wiring: dependency injection. wire_opencode_session()
lazy-imports the real run_session_blocking when not overridden.
Tests inject a stub.
22 new tests in test_worker_opencode_session.py:
- agent_name_for: parametrized per role (4 implementer-tier rows +
4 flat-role rows) + invalid tier + unknown role
- DEFAULT_TIER_TIMEOUT_S: each tier has a timeout + monotonic
- wire_opencode_session: completed/timeout/transport-error/unknown-
status routing; run_session_blocking raises wrapped as WorkerError;
on_poll propagates lost_lock; on_poll no-op when lock held;
custom prompt_builder used; default prompt includes role + tier
+ finalize() instruction + rendered input_payload; tag prefix
customizable.
Total: 398 controller tests; full auto_agents suite 2760 pass.
|
||
|
drew
|
23a014e402 |
feat(controller): Phase 1c-3 — production agent_runner (real MCP subprocess)
The production seam that connects the controller's runner to actual
LLM execution. Per-attempt MCP subprocess spawn + OpenCode session
drive + canonical JSON read + strict-parse → output dict.
tools/controller/worker/agent_runner.py:
- production_agent_runner(): the function tested production wires
into run_one_attempt's agent_runner param. Per-attempt lifecycle:
1. Map role → matching MCP module + V1 output contract.
2. mkstemp a canonical-output path; spawn `python -m {mcp_module}`
subprocess with CONTROLLER_CANONICAL_OUTPUT_PATH set in env.
3. Call injected run_opencode_session (tests use JSON-RPC stdin
driver; production wires _opencode_worker.run_session_blocking).
4. Wait for the MCP's finalize_and_emit to write the canonical
JSON to the tempfile.
5. Strict-parse against the role's V1 output model.
6. SIGTERM/grace/SIGKILL the MCP + unlink the tempfile in finally.
- ROLE_TO_MCP_MODULE / ROLE_TO_OUTPUT_MODEL: explicit per-role
dispatch tables. Covers all 5 worker roles.
- Error classification per the WorkerError outcome enum:
- unknown role → ValueError (master bug; not a worker outcome)
- session raises generic Exception → WorkerError(worker-internal-error)
- session raises WorkerLostLock → propagated unchanged
- lost_lock_check returns True after session → WorkerLostLock
- MCP didn't emit canonical output within timeout → WorkerError(
worker-internal-error)
- MCP emitted JSON that fails strict-parse → WorkerError(
contract-violation; per v9 status='failed' policy maps to
workflow STUCK)
tools/controller/mcp/_builder_base.py:
- finalize_and_emit() now writes to CONTROLLER_CANONICAL_OUTPUT_PATH
if set (production subprocess path) or stdout if not (direct-call
test path; capsys captures). Decouples canonical output from the
FastMCP JSON-RPC stdout transport — they would otherwise collide
in the subprocess (both writing to the same stream).
- Atomic file write: open + write + fsync + close.
- Existing test_mcp_builders.py (capsys-based) still passes with the
fallback path; new test_worker_agent_runner.py exercises the
file-based subprocess path with real MCPs.
10 new tests in test_worker_agent_runner.py:
- End-to-end with real MCP subprocesses (implementer-resolved,
reviewer-approve, estimator-tier-recommendation)
- Error paths: unknown role (ValueError), session raises (wrapped
as WorkerError), session raises WorkerLostLock (propagated),
lost_lock_check returns True post-session (raises WorkerLostLock),
no finalize → finalize_timeout → WorkerError
- Role-map sanity: every role has both an MCP module and output model
Total: 376 controller tests; full auto_agents suite 2738 pass.
This commit completes the v1 boundary — the controller now has the
complete stack from V1 contracts through MCP response builders, DB
schema, worker dequeue/heartbeat/runner/workspace, master state
machine + tick + reaper + pickup guard + scheduler + discovery +
Forgejo writes + MERGING + HTTP adapter, AND a production
agent_runner that wires it all to real OpenCode + MCP subprocess
spawning. Phase 1d-3+ enhancements (real OpenCode session wiring
in run_opencode_session) and Phase 2+ migration steps remain.
|
||
|
drew
|
30dfd92021 |
feat(controller): Phase 1c-2 — workspace + session sidecar + orphan janitor
Per-PR workspace umbrella manager (per plan v6) + the worker.session
sidecar that tracks live OpenCode session metadata for orphan cleanup
+ the startup janitor that sweeps orphaned workspaces from previous
worker crashes.
tools/controller/worker/:
- workspace.py: PerPRWorkspace + WorkspaceIdentity (frozen dataclass
yielding the canonical pr-{owner}-{repo}-{N} dir name). ensure_present
is idempotent. clone_if_absent runs `git clone --no-single-branch`
if .git/ missing; idempotent on re-call. fetch_and_validate runs
`git fetch origin --prune`, compares origin/<ref> to expected
head_sha, raises StaleInputError on mismatch (v6 stale-input fix),
then `git reset --hard <expected_sha>` + `git clean -fdx` to wipe
worktree residue from prior attempts. remove() rm -rf's the
workspace.
- session_sidecar.py: WorkerSession frozen dataclass +
write_sidecar (atomic via tmp+rename+fsync) + read_sidecar (tolerates
missing/empty/malformed/wrong-shape gracefully). Sidecar captures
opencode_server_url / session_id / subprocess_pid /
spawned_by_controller_pid / instance_id / spawned_at.
- janitor.py: sweep_orphans pre-queries the DB once for the set of
live instance_ids (workflow_attempts.status='in_progress'), then
scans workspace_root for pr-* dirs. For each:
- no sidecar → just delete (crashed pre-spawn)
- sidecar's instance_id is in live set → preserve (active worker)
- else → orphan. Try cancel_callback (production wires to OpenCode
cancel API); fall back to SIGTERM/grace/SIGKILL on the
subprocess_pid. Then delete sidecar + workspace. JanitorReport
summarises each sweep for structured logging.
Key v6 design points implemented:
- worker.session sidecar atomicity → orphan detection is robust
against partial writes (e.g., worker crashed mid-spawn).
- Cancel-callback-then-SIGKILL fallback → production prefers the
graceful OpenCode-side cancel; tests inject a fake.
- _kill_with_grace returns True after SIGKILL delivery; zombie
reaping is the parent's responsibility, not the janitor's.
27 new tests:
- WorkspaceIdentity (format + frozen)
- PerPRWorkspace paths + ensure_present idempotency
- clone_if_absent (creates worktree, idempotent, raises without URL)
- fetch_and_validate (matching sha resets clean; mismatched raises
StaleInputError; unknown ref raises)
- Sidecar I/O round-trip + atomicity + missing/empty/malformed handling
- Janitor: empty root / no-sidecar / active-lock / orphan-with-dead-pid
/ orphan-with-live-pid / cancel-callback (3 sub-paths: ok / fails /
raises) / mixed-workspaces / non-pr-skip
Total: 184 controller tests; full auto_agents suite 2546 pass.
|
||
|
drew
|
eab476e48e |
feat(controller): Phase 1c — worker controller skeleton
The dequeue+lock+heartbeat+runner+loop machinery. Production
OpenCode + MCP invocation slots in via the agent_runner callable
(Phase 1c-2). This commit is the structural foundation:
tools/controller/worker/:
- identity.py: build_instance_id() → "{hostname}/{pid}/{worker_uuid}"
per plan v9 (slash delimiter; IPv6-safe; uuid4 prefix for
per-instance uniqueness).
- heartbeat.py: Heartbeat thread that updates lock_heartbeat_at
every interval (default 30s). v9 simplified: TTL-only (no activity
tracking). UPDATE … WHERE locked_by_instance=us; rowcount=0 →
lost_lock_event.set() and thread exits, letting reaper handle it.
- runner.py: run_one_attempt() drives one attempt end-to-end.
Starts heartbeat → invokes agent_runner → on success writes
status='complete' + output_payload; on WorkerError writes
status='failed' with outcome label; on WorkerLostLock or detected
stolen-lock-at-write returns aborted (no DB write — reaper has
already re-pended). Defense-in-depth: even if agent returns
successfully, lost_lock_event.is_set() check skips the write.
- loop.py: worker_main_loop() polls the DB for pending attempts up
to MAX_CONCURRENT_WORKERS_PER_MACHINE, submits each to a
ThreadPoolExecutor. Honors stop_event for graceful shutdown
(drains in-flight before exit).
tools/controller/db/session.py: StaticPool for in-memory SQLite so
the heartbeat thread + runner write + dequeue all see the same DB
(without this, ":memory:" gives each connection an independent DB).
16 new tests in test_worker.py: instance ID format/uniqueness;
heartbeat tick (hold + steal); runner happy path; 5 error paths
(worker error / unexpected exception / WorkerLostLock raised /
stolen lock at write / lost_lock_event set defense-in-depth); 4
loop scenarios (single attempt, role filter skip, empty queue
exit-on-stop, explicit instance_id).
Total: 157 controller tests; full auto_agents suite 2519 pass.
|