a21466add232d59cdec1604e09d58ca05659a623
184 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 drew
|
a21466add2 |
feat(controller): Phase 4 metadata-hygiene + round-2 adversarial fixes
Five deterministic, idempotent Phase 4 checks: 1. completed_not_closed — close linked issues on MERGED 2. closing_keyword_fixup — add Closes #N to PR bodies 3. label_sync_from_issue — copy Priority/Type/MoSCoW labels 4. state_label_inference — sync State/* label to current_state 5. milestone_assignment — copy milestone from linked issue All default-off via CONTROLLER_METADATA_HYGIENE_ENABLED + per-check granular env flags. Dry-run mode shares the grooming CONTROLLER_GROOMING_DRY_RUN flag. Round-1 fixes (applied before this commit): - False-positive idempotency lock (executed=True on skip) - Unbounded MERGED scan → LEFT JOIN candidate query - Duplicate _classify_forgejo_status → import from forgejo_writes - Bare-ref regex too broad ([#42](url) misread) → add [ lookbehind - Wrong audit stage → 'metadata_hygiene' Round-2 adversarial fixes (3 architect, 4 principal, 7 test engineer): - completed_not_closed: executed=True only when ALL refs close; partial success writes executed=False so remaining issues retry - milestone_assignment: was calling get_pr_details (hits /pulls/, returns 404 for plain issues) → now uses get_issue_state (/issues/{n}) so milestone fetch works for all issue types - label_sync failure path: write executed=False audit row for observability; pre-fix left no audit trail for persistent failures - _BARE_REF_RE: add ( to lookbehind to exclude (#42) link destinations - state_label_inference: re-read current_state inside inner session to avoid stale-snapshot spurious label writes across session boundaries - _last_synced_state: add decision_id DESC tiebreaker for same-second wall-clock rows - dry-run completed_not_closed: separate early-return path to avoid inflating completed_not_closed_executed counter 71 tests (54 round-1 + 17 round-2): - TestCompletedNotClosedPartialSuccess (3) — partial/zero/full success - TestLabelSyncAdjustLabelsFailure (2) — failure audit + retry - TestStateLabelAdjustLabelsFailure (2) — no executed=1 on failure - TestStateLabelInferenceTerminalWorkflows (3) — MERGED/ABANDONED sync - TestLastSyncedStateDryRunThenReal (2) — dry-run → real-run - TestClosingKeywordFixupBareRefAlreadyCovered (2) — candidates subtraction - TestErrorPathHandlingRound2 (3) — label_sync + state_label errors Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
drew
|
4d969eaf2b |
feat(controller): Phase 3 — Gate 3 reviewer-abandon
When the reviewer finishes a review and judges the work fundamentally unworkable (implementation surfaced misdiagnosis, obsoleted-by-other-work, or irreducible complexity), it can now emit verdict='abstain' + suggested_next_action='abandon' with a Gate-3 abandon_reason_category. The controller routes REVIEWING → ABANDONED and (when the kill switch is on) performs the Forgejo close via the reviewer-abandon side-effect tick — no implementer/CI/merge cycles. Wired with the same defense-in-depth pattern Phase 2 established: MCP setter validation + outcomes mapper dispatch with confidence gating + Pydantic atomicity validator + side-effect tick with audit-trail attribution (cause=REVIEWER_ABANDON, event_type='reviewer_abandon'). Default-off CONTROLLER_GATE3_ABANDON_ENABLED kill switch so a fresh deploy is audit-only until the operator explicitly enables Forgejo writes. Bundled refactor: hoisted the 9 Gate-2 + 3 Gate-3-exclusive abandon categories into tools/controller/contracts/abandon_categories.py (triggered by Phase 3 per the plan's follow-up backlog). Both gates now consume the shared frozensets; doc-contract tests grep each agent prompt against the canonical list. Adversarial review (2 rounds): caught + fixed MCP cross-check ordering (atomicity FIRST so missing-setter shows actionable error), confidence=None symmetric downgrade across both gates, dead blocking-issues extraction in _run_close, idempotency clock-collision in the test, low-vs-missing reason-string conflation, and several test-quality gaps. 4064/4071 tests passing (7 pre-existing failures unrelated to Phase 3). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
a91df787d7 |
feat(controller): Phase 2 — Gate 2 estimator-abandon
Adds the second of three abandon gates: the estimator (Gate 2) can
mark a work item fundamentally unworkable, transitioning the
workflow ANALYZING -> ABANDONED and triggering a Forgejo close via
Phase 1's decomposed close_act orchestrator. Catches abandon cases
at the cheapest LLM stage, before implementer/reviewer tiers fire.
Substantive:
- EstimatorOutputV1: additive verdict + abandon_reason_category +
abandon_reason_detail fields (pre-Phase-2 outputs still parse).
@model_validator enforces abandon-requires-category atomicity at
parse time — third defense layer beyond MCP setter + outcomes
mapper
- state_machine: estimator_abandon event + (ANALYZING,
estimator_abandon) -> ABANDONED. 57 transitions; invariants clean
- mcp/estimator_builder: estimator_set_verdict setter validates
verdict enum + 9-category whitelist (scope_intractable,
intent_wrong, security_regression, deprecated_dependency,
breaks_protected_invariants, out_of_scope, low_value,
unmaintained_path, policy_violation) + cross-field rules
- outcomes._map_estimator_outcome: dispatch verdict='abandon'
-> estimator_abandon, with confidence-low downgrade to
estimator_done (honors the agent prompt's documented "high or
medium" requirement)
- estimator_abandon_side_effects.py: per-state side-effect tick
modeled on grooming_side_effects.py; invokes close_act with
cause=Cause.ESTIMATOR_ABANDON + event_type='estimator_abandon'
- _events.py: shared latest_transition_event +
workflows_with_latest_transition_in helpers; dialect-aware
payload['event'] extraction (SQLite json_extract +
PostgreSQL ->>); centralizes the event_type='transition' +
payload['event'] convention that side-effect ticks consume
- gate2_abandon_config.py: CONTROLLER_GATE2_ABANDON_ENABLED kill
switch (default false). Fresh Phase 2 deploys are audit-only
until operator explicitly enables; dry_run shared with grooming
for unified safe-rollout staging
- .opencode/agents/estimator-implementation.md: GATE 2 ABANDON
section with 9-category criteria + low_value disqualifier ("PR
cites an issue/ticket -> route to reviewer instead")
Round-2 adversarial-review fixes (all required pre-commit):
- forgejo_writes.close_issue / close_act: NEW cause + event_type
kwargs (defaults preserve Phase 1 grooming behavior; Phase 2
callsite overrides). Fixes audit-trail attribution: telemetry
queries SELECT WHERE cause='estimator_abandon' now return the
right rows. Phase 1 regression test pins the grooming defaults
- tick.py operator_unstick lookback: dialect-aware json_extract
fix (Phase 1 carry-over bug; would silently no-op on PostgreSQL)
- grooming_side_effects.py: idempotency filter now keys on
check_name set (grooming check_names only) so a Phase 1 close
and a Phase 2 close on the same workflow don't cross-cancel
Tests (+50): TestEstimatorOutputV1Phase2,
TestEstimatorAbandonStateMachine, TestMapEstimatorOutcomePhase2
(including confidence-low downgrade), TestEstimatorSetVerdict
(all 9 categories + cross-field rules), TestEventsHelper,
TestEstimatorAbandonSideEffectTick (including
test_close_writes_estimator_abandon_cause_and_event_type pinning
the audit-trail attribution, and Phase 1 regression guard).
Doc-contract test asserts all 9 categories appear in the agent
prompt. 1509/1509 passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
016b348117 |
feat(controller): grooming gate (Phase 0 + Phase 1 worker-shape dispatch)
Phase 0 (foundation):
- Cause enum (controller_events.cause) for action attribution
- Schema: grooming_decisions audit table; workflows gains
grooming_evaluated_at + deferred_reason + deferred_at +
deferred_target_workflow_id; pulls gains touched_files
- audit_comments: CLOSE / DEFER templates + render_comment_template
- forgejo_writes: close_issue + defer_issue 5-step crash-safe protocol
(fingerprint dedup, error matrix, dry-run)
- patch_pr_state callback in forgejo_http
- grooming_config: 22-env-var frozen-dataclass config + log_effective
- pulls.touched_files cache extension (_pipeline_cache.py schema v8)
- reaper.reap_grooming_decisions audit-retention sweep
- reconciliation RESUME guard (deferred_reason)
Phase 1 (worker-queue shape, 2026-05-25):
- New state: GROOMING. New events: grooming_started, groom_verdict_
{proceed,defer,close}. 5 new transitions; all invariants still clean
- GroomingInputV1 + GroomingOutputV1 Pydantic contracts
- outcomes._map_grooming_outcome routes verdicts to state-machine events
- prefetch.build_grooming_stage_b_input + list_open_prs callback
- scheduler GROOMING -> grooming_stage_b role
- promote: cfg-gated DISCOVERED -> GROOMING when CONTROLLER_GROOMING_
ENABLED=true; issues skip grooming
- forgejo_writes decomposed: close_act/defer_act (Forgejo writes only;
state-machine already transitioned) + close_decide_and_act/
defer_decide_and_act (Phase 0 callers); _apply_workflow_transition
is underscore-private
- grooming.py library: tokenization, suspicion scoring (Jaccard +
weighted overlap), deterministic checks, action -> verdict mapping
- mcp/grooming_builder.py: 14-tool FastMCP server emits GroomingOutputV1
- .opencode/agents/grooming-stage-b.md: duplicate-detection agent
prompt (claude-haiku-4-5)
- grooming_side_effects.run_grooming_side_effects_tick: per-state tick
performs Forgejo writes after groom_verdict_{defer,close} fires.
Filters on event_type='transition' + payload.event (centralizes the
convention pending Phase 2's latest_transition_event helper)
- GroomingCallbacks frozen dataclass; loop.py + __main__.py wired
Worker role registry (single source of truth):
- worker/roles.py: WORKER_ROLES + WorkerRoleSpec + default_roles_csv
+ output_filename_for. agent_runner.ROLE_TO_MCP_MODULE / ROLE_TO_
OUTPUT_MODEL derive from it; opencode_session.agent_name_for reads
it for flat cases; all 6 prompt builders use output_filename_for;
worker --roles default = default_roles_csv(); launcher script
derives --roles via shell substitution. Cross-site invariant test
enforces alignment across 5 sites + opencode.json MCP registry.
Phase 0 silent-bug fix:
- reconciliation.py RESUME guard SELECT now includes deferred_reason
(was missing since Phase 0; guard was a silent no-op). Tightened
from getattr to attribute access to fail fast on future omissions.
Tests (1456 total, +91 grooming-specific):
- test_grooming_phase0.py: 34 tests (orchestrator matrix, crash
recovery, idempotency, dry-run)
- test_grooming_phase1.py: 60 tests (library, contracts, state
machine, outcomes, scheduler, promote, prefetch, act-variants
with signature parity, side-effect tick incl. natural-idempotency
+ executed-flag-skip + verdict-mismatch + reconciliation RESUME)
- test_mcp_builders.py TestGroomingBuilder: 29 tests (happy paths
+ 22 validation rules + Pydantic round-trip + master-tick-read-
path companion)
- test_worker_agent_runner.py TestRoleMaps: cross-role wiring
alignment + agent-prompt-vs-worker-fallback filename contract +
inspect.signature equality (close_act/defer_act vs
close_issue/defer_issue)
- test_state_machine.py: transition count 51 -> 56 +
events_from_grooming
Live-validated end-to-end on 4 staged sentinel PRs (#55-#58) in
dry_run: agent emits verdicts via MCP, state-machine transitions
fire, side-effect tick writes audit row, deferred_reason gates
reconciliation RESUME correctly.
Deferred refinements + Phase 2 prerequisite (latest_transition_event
helper) tracked in .drew/regressions-plan.md "Phase 1 follow-up
backlog".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
d3cf710d29 |
fix(controller): tier-selection calibration — finalize timeout 30s→90s, estimator defaults to tier 1
Two coupled changes responding to the 2026-05-22 batch's tier-0 hit
rate of 0% on PRs the estimator judged "simple", plus three spurious
worker-internal-errors caused by the estimator's MCP-finalize wall
clock exceeding the previous 30s budget.
agent_runner.py — finalize timeout
----------------------------------
`production_agent_runner` previously hardcoded `finalize_timeout_s=30.0`.
On the 8-PR batch the worker recorded three `worker-internal-error`
attempts (PRs 47, 50, 54) all with the same message:
role='estimator' did not emit canonical output within 30.0s ...
The retries succeeded with estimator self-time of 41.5s, 25.9s, 21.5s —
all within plausible bounds for a multi-subsystem PR (PR 54 needed
41.5s of reasoning + MCP IPC + finalize write). Each spurious timeout
cost ~90s of worker wallclock + a re-dispatch.
New module-level constant `_DEFAULT_FINALIZE_TIMEOUT_S` reads
`CONTROLLER_FINALIZE_TIMEOUT_S` (default 90s), matching the
`CONTROLLER_*_TIMEOUT_S` convention used by `opencode_session.py`.
Operators can tune via `.devcontainer/.env.{fork,prod}`.
estimator-implementation.md — TIER MAP + CALIBRATION
----------------------------------------------------
The estimator was wrong on 4 of 8 PRs in the May-22 batch, all in
the same direction (recommended tier 0, escalated to tier 1).
Three of the four were high-confidence picks — including a PR
described as "4 string constant replacements, isolated scope, no
logic changes" that still got bounced at tier 0.
Three prompt changes:
- TIER MAP rewritten: tier 1 is now the default for non-trivial
work; tier 0 requires positive evidence the change is mechanical
(single file, ≤50 LOC, no new logic, no test changes).
- CONFIDENCE RULES "When uncertain, prefer" flipped tier 0 → tier 1.
- New CALIBRATION section bakes in the empirical observation (0/4
hit rate, why Haiku struggles in this codebase, cost calculus:
wrong tier-1 < wrong tier-0 by an order of magnitude when measured
per merged PR).
tiers.yaml — clarify scope
--------------------------
The previous "Used by estimator-implementation.md to make tier
choices model-agnostic" comment on the `capability` field misled
readers into thinking tiers.yaml drove classification. It doesn't —
the estimator agent reads its TIER MAP from its own prompt and
emits an integer; tiers.yaml only answers "for tier-N, which model
runs?". Updated header + field comments to flag this clearly so
future operators don't change descriptors here expecting the
estimator to honor them.
Tests
-----
+ test_default_finalize_timeout_reads_env: covers the env-var
resolution for `CONTROLLER_FINALIZE_TIMEOUT_S` (default, override,
float values). Restores the default at the end so subsequent
tests in the session see the stock constant.
Existing tier-model registry + 27 agent_runner tests pass unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
981ddd6a8e |
fix(controller): incident hardening — PR-44 dispute guard + PR-46 fixes
Bundles fixes for three production incidents (PR-44, PR-46 a/b/c). Every change has an incident-reference comment in the code and a named regression test. 199 tests pass on the impacted modules. PR-44 — fabricated dispute escape --------------------------------- A tier-0 implementer, bounced twice by red CI, emitted a fabricated ``dispute-reviewer`` outcome and shortcut a red PR into REVIEWING → APPROVED → MERGING, bypassing the CI gate. ``dispute-reviewer`` is the only IMPLEMENTING → REVIEWING edge that doesn't pass through AWAITING_CI, so it must be defended. tools/controller/master/outcomes.py: ``_map_implementer_outcome`` now guards ``dispute-reviewer`` with two preconditions — (1) ``attempt_saw_green_ci`` (the HEAD must already be CI-verified; a dispute can't jump the gate on a red/pending head), and (2) ``prior_reviews >= 1`` (must reference a review that actually happened, not a hallucinated one). Either guard fails → ``implementer_competence_failure`` → tier escalation. A weak model can't game its way past CI; a stronger tier is given the real problem. tools/controller/master/tick.py: new ``_count_prior_reviews`` helper counts completed reviewer attempts (epoch-scoped so an ``operator_unstick`` resets the count). Wired into the ``map_outcome_to_event`` call. PR-46(a) — stale gate-script preferred over in-repo --------------------------------------------------- ``gate.py`` was preferring the seeded ``/tmp/local_tools`` copy of ``local_ci_gate.sh`` over the version-matched in-repo copy. The seed predated the ``--envdir`` flag; the controller pipeline's invocations rejected as bad-argv every committing implementer's gate. The seed-refresher (``dispatch_implementer.py``) is on the retired dispatcher path, so the staleness was permanent. tools/controller/worker/gate.py: resolution order is now ``CONTROLLER_LOCAL_CI_GATE`` env > in-repo > seeded ``/tmp``. The seeded copy survives only as a last-resort fallback. Module-level constants ``_IN_REPO_GATE_SCRIPT`` / ``_SEEDED_GATE_SCRIPT`` let tests substitute paths. PR-46(b) — 6-second-old run flagged zombie ------------------------------------------ ``classify_ci_run`` instantly classified a CI run as ``stale`` when the Actions API reported no active task. A freshly-pushed run has no task simply because no runner has picked it up yet, and there are brief gaps between jobs — both false positives. A 6-second-old PR-46 run was bounced before CI could even start. tools/controller/master/ci_run_status.py: new ``ZOMBIE_GRACE`` (default 3 min, env: ``CONTROLLER_CI_ZOMBIE_GRACE_MIN``). "No active task" only classifies a run as stale once the run has ALSO gone quiet past the grace. Much shorter than ``STALE_AFTER`` since the active-task absence is corroborating evidence, not the sole signal. PR-46(c) — ruff-format-only violation slipping through lint ----------------------------------------------------------- CI's ``lint`` job runs both the ``lint`` nox session (ruff check) AND ``ruff format --check``. The pre-push local gate only ran the former; a formatting-only violation passed pre-push then failed CI. tools/local_ci_gate.sh: the ``lint`` gate now runs ``ruff check`` followed by ``ruff format --check``, unconditional. Either one failing marks the gate red. ``ruff format --check`` is whole-repo and takes no posargs. tests/auto_agents/test_local_ci_gate.py updated for the new two-call shape. Supporting changes ------------------ .forgejo/workflows/ci.yml: gates ``coverage`` and ``docker`` jobs on repo variables ``skip_coverage`` / ``skip_docker`` so the long reaper-prone jobs can be skipped per-fork without editing CI. Guard step always runs so the job still reports ``success`` and ``status-check`` stays green. tools/duplicate_prs_to_fork.py: bakes the same ``skip_coverage`` / ``skip_docker`` gates into every sentinel PR's ci.yml at PR-creation time so fork-mode runs inherit the gating. Idempotent. .opencode/opencode.json: adds ``timeout: 1860000`` (31 min) to the ``ci`` MCP server so long CI waits don't timeout the tool. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
eb01eb0172 |
feat(controller): dual-mode launcher (fork/prod) + DB-mode validator
Adds the operator surface for switching the controller pipeline between
the personal fork (drew/cleveragents-core) and the canonical repo
(cleveragents/cleveragents-core) via a MODE env + --prod CLI flag,
backed by safety primitives that make a wrong-mode launch loud rather
than silent.
run-controller-state-machine-pipeline.sh: --prod flag and MODE env
(primary home: .devcontainer/.env) select fork vs prod. After resolving
MODE, the launcher auto-sources the matching overlay file
(.devcontainer/.env.{fork,prod}) and asserts MODE didn't drift during
the source step. The drift assertion uses a readonly snapshot under an
obscure variable name so a stray ``MODE=fork`` in .env.prod aborts the
launch with a clear bash error rather than silently demoting the run.
CONTROLLER_RUN_DIR_ROOT now overrides the trial /tmp path so prod can
use a persistent /var/lib/cleveragents/run dir.
tools/launch_prod.sh (new): sibling to launch_fork.sh with the opposite
safety primitive — affirmative GET /repos/{owner}/{repo} that asserts
the target is non-fork, exists, isn't archived, and the bot has push.
On any failure, no env is exported. Honors HAL_* aliases for parity
with launch_fork.sh and prints a hard-to-miss PROD-MODE banner.
tools/controller/deploy/validate_db_mode.py (new): stamps a _mode_marker
table on each SQLite db (controller DB + telemetry cache) on first use,
asserts a match on every subsequent launch, and moves mismatched files
aside as <name>.<prior-mode>.bak.<ts> — never deletes. The --adopt flag
lets an operator grandfather in already-good pre-marker data without
losing history. Wired into the launcher's startup sequence before
OpenCode and the master start.
tools/_cache_path.py (new): single source of truth for the per-(owner,
repo) Forgejo cache file convention. .opencode/telemetry/server.py and
the launcher both delegate here so the dual-source-truth drift risk is
eliminated. tools/_pipeline_cache.py and tools/controller/db/models.py
documented as not owning the _mode_marker table so future migrations
leave it alone.
.opencode/telemetry/server.py: hosts the llm_activity scraper as a
background subprocess thread (60s cadence, --since-hours 1 in steady
state, full backfill on first tick). Re-homes the cost-telemetry data
path after the pr_state_warmer was retired by the controller migration
— without this the Cost tab freezes when the warmer's loop is gone.
Subprocess (not in-process) for isolation; failures swallowed.
opencode.json: local-claude provider's baseURL now reads
{env:LOCAL_PROXY_URL} instead of the literal http://127.0.0.1:3456/v1,
matching the apiKey pattern already in use.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
a103a31bbf |
feat(controller): worker-owned gated push for the implementer
The implementer agent no longer pushes to git — the controller worker now owns the push: it gates the agent's commits on lint+typecheck and pushes via a single leased primitive. Closes two production defects: - Clobber: the pre-fix MCP --force-with-lease leased against a freshly-fetched tip, so the lease always passed — an in-flight implementer destroyed a commit pushed to the PR branch during its run (lost a hand-pushed skip_coverage fix on PR #46). - Gate-skip: the agent verified only the CI-flagged gate, so a fix for one gate shipped fresh violations in another (lint flapped pass->fail across CI runs 198->199). Step 1 — worker_push primitive: - New git_push.py: one leased push, pinned to the SHA the worker started from; classifies pushed / stale_input / diverged / infra_error; bounded infra-retry. - mcp_git_server.push gains expected_sha for a correctly-pinned lease. - finalize_conflict_resolution migrated onto worker_push. Step 2 — deterministic gate: - New gate.py: per-slot nox env-dirs (no venv races), manifest-hash staleness keying, lazy warm-up. - local_ci_gate.sh gains --envdir. Step 3 — worker-owned gated push: - New implementer_finalize.py: divergence pre-check -> lint+typecheck gate -> ruff auto-fix -> leased push. finalize's outcome is authoritative over the agent's emitted outcome. - agent_runner integrates finalize; salvage no longer pushes. - outcomes/tick: gate-failed + push-time stale-input caps, epoch-scoped; WorkerError carries an output_payload so the gate report reaches the next attempt's prompt; prefetch surfaces gate-failed attempts. - The 5 task-implementor prompts drop the agent push step. Reviewed across 4 adversarial rounds; full controller+MCP suite green (1379 passed). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
6178be3aa7 |
feat(models): single-source model registry via models.yaml + sync_models.py
.opencode/models/models.yaml is now the ONE file humans edit to assign a model to an agent. tools/sync_models.py regenerates every derived surface — the .opencode/models/*.txt files, opencode.json's `agent` block, and each non-tier agent's .md `model:` frontmatter — so an assignment cannot drift across surfaces. `--check` verifies with no writes and is enforced in CI by test_model_registry_in_sync_with_manifest. Hardened after adversarial review: - Deletes orphan <agent>.txt files left behind when an override is dropped from the manifest. The dispatcher's resolver reads <agent>.txt before default.txt, so a stale file would silently pin the old model. Tier .txt files are left to sync_tier_models.py. - Rejects a manifest key that does not name a real agent (no matching .opencode/agents/<name>.md) instead of silently appending a bogus opencode.json entry and leaving the real agent on the default model. - Validates the regenerated opencode.json BEFORE writing it, so a bad render aborts cleanly instead of corrupting the file on disk. Tier-ladder agents (task-implementor-tier-*) remain governed separately by tiers.yaml + sync_tier_models.py and are passed through untouched. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
d9919e783b |
feat(merge-drive): post the reviewer's approval to Forgejo before merge
The controller's reviewer stage records verdict=approve only in the
controller DB — it never posts a Forgejo review. Branch protection's
required-approvals gate then 405s the merge ('not allowed to merge: Does
not have enough approvals') for a controller-managed single PR, because
nothing reflects the controller's approval to Forgejo (PR #45). The merge
driver only auto-approved umbrella PRs for multi-PR trains; single PRs
got nothing.
merge_train now posts a real Forgejo APPROVED review for a controller-
managed single PR (len(prs) == 1 with a _controller_workflow_id) right
before the merge endpoint, via _post_approval_review:
- posted as the reviewer identity (HAL9001 via cfg.reviewer_pat) — a
non-author approval, which is what required-approvals demands;
- pinned to the exact rebased SHA about to merge (commit_id);
- idempotent — _pr_has_approval skips the post if an APPROVED review
already exists, so a merge_train recursion does not spam duplicates;
- best-effort + logged — a failed post does not abort the merge; the
WARNING is the operator signal.
The umbrella-PR auto-approval is refactored onto the same helper.
12 new tests in test_merge_drive.py; full merge_drive suites green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
7fdac7ded0 |
fix(merge-drive): correct CI-wait — newest-per-context statuses + zombie-CI detection
wait_for_ci could never see CI finish on a green commit. fetch_commit_statuses
flattened Forgejo's newest-first /commits/{sha}/statuses *history* with
last-write-wins, so each context kept its OLDEST object — the initial
`pending` posted when the job was queued — and a fully-green SHA read as
all-pending. PR #45 hung against a green commit until the 1 h ci_timeout_s.
fetch_commit_statuses now keeps the newest status per context by created_at.
A required check that never decides — a job a misconfigured branch protection
requires but no workflow emits, or a gate left pending by a crashed runner —
also pinned the wait to the full ci_timeout_s. wait_for_ci now consults
Forgejo's Actions API via ci_run_is_live: once the run is confirmed dead for
_ZOMBIE_CONFIRM_POLLS consecutive polls while a required check is still
undecided, it ends with the verdict the finished gates produced
(_zombie_verdict — passing / failing / timeout). Mirrors the controller's
zombie-CI detection in ci_run_status.py.
37 new/updated tests in test_merge_drive.py; full merge_drive suites green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
bea16f7130 |
fix(controller): scope STUCK-gating counters to the operator_unstick epoch
operator_unstick is the deliberate operator escape hatch that requeues a STUCK/APPROVED/PAUSED workflow back to DISCOVERED. It reset the state but not the per-workflow failure tallies, so a requeued workflow inherited every historical worker-error / ci-not-ready / ci-infra-failure / dispute / contract-violation / resolved-conflict count and re-tripped a backstop cap almost immediately (observed: a PR re-STUCK after a single fresh ci-not-ready because 5 stale ones from prior runs were still counted). All seven prior-outcome counters now accept an epoch_start and, when the workflow has been operator-requeued, count only attempts created after the most recent operator_unstick event — so a requeue genuinely starts the backstop budgets fresh. The automatic DISCOVERED loops (ci_infra_recheck, ci_red_retry) are not operator_unstick events, so their caps still bound those loops. operator_unstick is matched under both conventions seen in the wild (bare event_type, or a transition whose payload names it); the comparison is wrapped in datetime() to be immune to timestamp-format drift. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
14f79ee61b |
fix(controller): count only resolved conflict cycles toward the STUCK budget
The v6 "3+ conflicts -> STUCK" guard measured every conflict_resolver attempt at the tier, including failed/blocked/errored ones. A burst of transient worker errors (e.g. the run-9 marker false-positive) inflated the count so that a workflow whose conflict was genuinely resolved on the first real pass was routed to STUCK instead of IMPLEMENTING. _count_conflict_resolver_attempts now counts only complete+resolved attempts -- each resolved attempt is exactly one conflict cycle, since the workflow leaves CONFLICT_RESOLVING on success and only re-enters when a new conflict appears. Failed attempts are retries of the same unresolved conflict and stay bounded by the pickup guard. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
d4e1219577 |
fix(controller): scope conflict-marker check to resolved files
finalize_conflict_resolution grepped every file in the PR's origin/<base>..HEAD diff for committed conflict markers. On a large sentinel PR (478-file diff) that includes .opencode/agents/ git-rebase-util.md — an agent def that *documents* conflict markers with literal "<<<<<<< HEAD" lines — so the check false-positived and rejected every resolution, looping the conflict-resolver indefinitely (run-9: 185 attempts in ~4h, 125 on this exact error). Scope the grep to the files the resolution actually touched (resolved_files = prep-time conflicted set + the agent's reported files_modified). A leftover marker can only be in a file the agent resolved; a clean rebase/merge resolves nothing so the check is skipped. Adds a regression test with a marker-documenting file in the PR diff that is not a resolved file. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
615a05b982 |
feat(controller): rebase-default conflict resolution with merge fallback
PR branches 177-180 commits ahead of base cannot be rebased commit-by-commit by a single-shot resolver agent (too many conflict stops for one session). Conflict-prep now defaults to rebase (linear history) and falls back to a single 3-way merge when the branch is too divergent (commit count over CONTROLLER_CONFLICT_REBASE_MAX_COMMITS, default 60). The merge pipeline derives the track from branch shape via a Do:rebase -> Do:merge ladder in _make_merge_pr — no stored flag. Adds a git_rebase_continue MCP tool plus status rebase/merge-in-progress fields so the conflict-resolver agent is fully MCP-driven and dual-mode (mid-rebase or mid-merge). Also routes a green-CI implementer noop straight to REVIEWING instead of a deadlock-prone AWAITING_CI round trip. No state-machine change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
8c9a0c50bb |
fix(controller): green-CI noop routes forward instead of ABANDONED
An implementer dispatched against a fully-green CI that emits `noop`
("nothing to fix") was mapped to competence-failure -> ESCALATING ->
ABANDONED at MAX_TIER. PR-39 and PR-40 dead-ended exactly this way:
12/12 CI gates green, workflow ABANDONED, solely because the
implementer said `noop` instead of the synonymous `verified-clean`.
The tick now reads the attempt's input_payload ci_summary; a `noop`
whose attempt saw an unambiguously green CI (overall success, zero
failed, zero pending, >=1 passed) routes via `implementer_verified`
-> AWAITING_CI (-> ci_green -> REVIEWING) -- the same forward path
`verified-clean` already takes. A non-green `noop` (red / pending /
unknown / no CI) keeps the competence-failure -> escalate behavior.
- outcomes.py: `attempt_saw_green_ci` param gates the noop branch.
- tick.py: `_input_ci_summary` + `_ci_summary_is_green` helpers.
- tests: green->AWAITING_CI, red/none->ESCALATING, helper unit tests.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
drew
|
14e592ddd5 |
feat(controller): zombie-CI detection — stop waiting on dead CI runs
A CI gate stuck `pending` is ambiguous: the job may genuinely be running, or the run may be dead (a crashed runner, an Actions job whose terminal commit-status was never posted — `CI / status-check` zombies routinely here). The "wait for the whole run to finish" fix then waited forever on the dead case (PR #36: a `status-check` gate pending for 8 h while the run had actually finished RED 8 h earlier). New `ci_run_status.classify_ci_run` resolves a still-pending run to `complete` / `running` / `stale` via two checks, authoritative-first: 1. ACTIVE-RUN — `get_action_tasks` asks Forgejo's Actions API directly whether a task for the commit is still running; catches a dead run immediately, regardless of age. 2. AGE — if no gate has updated in > CONTROLLER_CI_STALE_AFTER_MIN (default 90) the run has stopped; the fallback when the Actions API is unavailable. A `stale` run is no longer waited on: the verdict is taken from the gates that DID finish (`terminal_verdict`) — any failure → red, all pass → green, fully-dead → red. Applied in both `ci_status_poll` (the AWAITING_CI verdict) and `ci_summarize` (the implementer's summary — zombie pending gates drop out of `gates_pending`/`overall_state`) so the two agree and never ping-pong. New `get_action_tasks` Forgejo callback wired through forgejo_http → __main__ → the poll and the prefetch path. 23 new tests; full controller suite (1222) green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
2fef8cb9f9 |
fix(controller): strip Actions log timestamps before CI parsing
Every implementer was handed `findings=0` for its red gates even when
the parser's own summary line counted real failures ("Pyright: 10
errors", "Behave: 2 scenarios failed"). Root cause: GitHub/Forgejo
Actions prefixes EVERY log line with an ISO-8601 timestamp
("2026-05-20T18:04:41.1555784Z "). Every parser extracts findings with
^-anchored regexes against the raw tool output (pyright
`^file:line:col - error:`, behave `^\s+...feature:N Scenario:`); the
timestamp prefix pushes that content off the line start so the anchors
never match — while the UN-anchored summary-count regex still matches,
producing the misleading "summary says N, findings=[]" state.
Fix: new `strip_log_timestamps()` in ci_summary_parsers/_base.py,
applied once centrally in `ci_summarize._safe_fetch` — the single
chokepoint feeding every parser and the no-parser raw-log excerpt.
Lines without a timestamp prefix are left untouched, so it is safe
unconditionally; it also trims ~28 chars/line off the implementer's
raw-log excerpt.
5 new tests; full controller suite (1200) green.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
drew
|
5f0fff0db2 |
fix(controller): wait for the whole CI run to finish before acting on it
Two places acted on a CI run that was still executing — the PR-39/PR-40 "doesn't wait for CI to finish" incident: 1. ci_status_poll — keyed the AWAITING_CI verdict purely off Forgejo's combined `state`. That combined state flips to `failure` the instant ONE gate fails, even with other gates still running (and can read `success` before a late gate reports). So a run at 9-passed / 1-failed / 2-pending fired `ci_red` and yanked the workflow out of AWAITING_CI mid-run. New `_ci_run_incomplete()` scans the per-gate `statuses`; while any gate is pending/running the poll waits, regardless of the combined state. Only a fully-terminal run yields a verdict. 2. implementer prompt routing — `_ci_summary_is_pending` required the run to be failure-free to count as "still pending," so a run with an early failure + gates still executing fell through to the ci-infra-failure block and the implementer acted on a partial result. Dropped the no-failures clause: any pending gate → the ci-not-ready (wait) block, which now also covers partial results. Zombie/stale runs (a gate pending for hours) are left to the controller's CI-freshness gate to re-trigger — a separate follow-up; ci_poll_exhaustion remains the backstop. 4 new tests; full controller suite (1195) green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
22b76b2834 |
fix(worker): correct misleading "model override" log line
The worker's _resolve_role_model resolves a model from .opencode/models/<agent>.txt (or the default.txt fallback) and passes it on POST /session — but OpenCode does NOT consume that field for generation; it generates from its startup-cached opencode.json agent.<name>.model (the footgun documented in .opencode/models/README.md). The old line "model override for agent=X -> Y" read as if Y were the model in effect. For a tier agent with no per-agent .txt file it logged "-> default.txt(haiku)", which falsely looked like every implementer tier was downgraded to haiku — when opencode.json's task-implementor-tier-2 block correctly pins opus (verified: run-3 tier-2 session archive records claude-opus-4-6). Reworded to "POST /session model hint=Y ... observability only; OpenCode generates from opencode.json's agent.X.model". Also corrected the module docstring's stale "takes effect ... no restart required" claim. Log + docstring only — no behaviour change. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
eb1d81828f |
feat(controller): implementer_retrigger_ci — agent re-triggers CI on ci-infra-failure
A CI job hard-killed by OOM / pod eviction produces no verdict — nothing in the diff to fix. The implementer correctly emits outcome=ci-infra-failure, but that alone only shuffles workflow state; CI never re-runs, so every retry reads the same dead run and the workflow loops until the _MAX_CI_INFRA_FAILURE backstop STUCKs it (observed live on PR #39 in run-3). New ``implementer_retrigger_ci(owner, repo, pr_branch)`` MCP tool on the implementer response-builder lets the agent kick a fresh CI run. Forgejo 15.x has no Actions rerun API, so it reuses the controller's existing mechanism — ``ci_rerun.trigger_ci_rerun_via_empty_commit`` — an empty commit that advances the PR head SHA, which is the unambiguous fresh-state signal for CI (and a stale review). - ci_rerun.py is loaded standalone (importlib by file path, registered in sys.modules before exec so its @dataclass resolves) — the response-builder MCP must not pull the heavy tools.controller.master package. - _retrigger_ci resolves the Forgejo base URL + token from env (FORGEJO_URL / FORGEJO_API_BASE, FORGEJO_TOKEN / GITEA_TOKEN) and never raises. - Once per session: a second implementer_retrigger_ci call is refused benignly so a looping agent cannot pile junk commits on the PR. - The implementer prompt's ci-infra-failure block now instructs the agent to call the tool before emitting the outcome. 10 new tests (test_mcp_builders.py, test_worker_prompts.py); full controller suite (1192) green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
9d20f865d3 |
feat(controller): pre-review mergeable gate — skip doomed reviews
When CI goes green, route to REVIEWING only if the PR still merges cleanly into base. If a cheap Forgejo mergeable check shows the base advanced while CI ran, route AWAITING_CI → CONFLICT_RESOLVING instead — handing the conflict to the LLM conflict_resolver BEFORE the expensive reviewer pass, since code that must be rebased gets re-CI'd and re-reviewed afterwards anyway. - new event `pre_review_base_conflict` + transition (AWAITING_CI → CONFLICT_RESOLVING) - ci_status_poll: `_decide_green_event` gate behind a new optional `get_pr_details` callback; conservative — only an explicit mergeable=false diverts, an unknown/uncomputed bit falls through to ci_green so a fresh PR is never false-routed - gate-fired event rows carry `mergeable` in the payload so the false-positive rate is observable from controller_events - wired through loop.py (5th ci_status_poll_args element) + __main__ One API call, no LLM — cheap+frequent detection gating the expensive+rare conflict_resolver/CI/reviewer stages. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
8fe98cb5c0 |
feat(controller): salvage timed-out implementer + cap unbounded prefetch-retry loop
Two reliability fixes from the proactive audit of untouched pipeline modules (AUDIT-1, AUDIT-2 in .drew/PENDING_FIXES.md). AUDIT-1 — a timed-out implementer's committed work was never salvaged. opencode_session maps an OpenCode timeout / transport-error to WorkerError, which propagated out of production_agent_runner BEFORE the canonical-output wait + _salvage_implementer_commits ran. A timeout is the case most likely to have a complete committed fix (agent ran out of wallclock, not correctness). The session-exception handler now runs the same salvage the canonical-missing path uses before discarding the attempt as worker-internal-error. AUDIT-2 — a persistently failing prefetch() retried every master tick forever (the T4-4 guard covered only the issue-kind sub-case; same unbounded-retry class as the run-2 174x estimator loop). The scheduler now journals each prefetch failure as a controller_events row, counts failures since the workflow's last transition, and routes the workflow to STUCK after _PREFETCH_FAILURE_LIMIT (10) failures. The STUCK transition is rowcount-guarded so the journal never records a transition the current_state guard rejected. 7 new tests; full controller suite (1169) green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
58307bbdab |
refactor(ci-logs): retire the dead legacy {sha}.json cache layer
Follow-up to the B1 unification: with `fetch_pr_failure_logs` now a
view over the `get_ci_logs` bundle, the legacy `{sha}.json` cache had
no readers left. Remove it wholesale rather than leave it orphaned.
- `_ci_logs.py`: delete `_cache_covers_all_current_failures`,
`_record_failure`, `_read_cache`, `_write_cache`, `cache_path` — all
zero-caller after B1. `invalidate` re-pointed onto the bundle cache
(`bundle_cache_path`) so it stays a working API. Module docstring
rewritten to describe the bundle-as-single-store reality.
- `local_ci.py`: `_write_ci_logs_cache` no longer writes the legacy
`{sha}.json` — `put_local_bundle` already populates the bundle that
`fetch_pr_failure_logs` projects, so the MCP tool still sees local
CI logs. One write path, not two.
- Tests re-pointed onto `bundle_cache_path`; `ruff format` applied.
No behavior change — only dead code removed and the docstring
brought current.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
drew
|
f76568a871 |
feat(controller): ci-infra-failure implementer outcome → bounded rerun
The implementer-side counterpart to the `indeterminate` verdict. When the implementer IS dispatched onto a CI failure and finds the log carries no verdict (a hard-kill / OOM — nothing in the diff to fix), it can now emit `outcome=ci-infra-failure` instead of being forced to `blocked` → STUCK. `ci-infra-failure` forbids commits/files/blockers (no-work invariant, like `noop`) and fires `implementer_ci_infra_failure`, routing IMPLEMENTING → DISCOVERED so the CI-freshness gate reruns CI under its bounded RERUN_BUDGET. Backstopped by `_MAX_CI_INFRA_FAILURE=4` so a mis-classification cannot loop the gate forever. Wired through: the V1 contract enum, the implementer MCP builder (outcome value + no-work invariant), the state machine (event + IMPLEMENTING→DISCOVERED transition), the outcome mapper (+ per-workflow cap), tick's `_count_prior_ci_infra_failure`, and the implementer prompt — which now surfaces the outcome whenever the CI summary shows a failing gate, with guidance to use it ONLY when the log genuinely shows no verdict. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
6ab6df319c |
feat(controller): indeterminate-CI verdict for no-verdict (OOM) failures
Diagnosed live on PR 39 (STUCK 2026-05-20): both CI gates ran 17 minutes, the captured log was <3 minutes and ended mid-execution (`still running` / a just-launched runner) with ZERO error markers. A hard process kill (OOM-killer / pod eviction) cannot flush a buffer, print a traceback or emit an exit code — it leaves a *hole*, not a phrase. The classifier called this `fresh_real`, the implementer was sent to "fix" a failure with nothing to fix, and the PR dead-ended at blocked → STUCK. New `indeterminate` verdict: a failing gate whose FULL log carries no terminal verdict marker (`##[error]`, test summary, Traceback, exit code) AND ends mid-execution. Both conditions required — "no marker" alone over-fires on a real failure whose tool output isn't in the marker set (e.g. `ruff format`). Routed to a bounded rerun, same as `infra_broken`/`stale`, via `ci_status_poll` and `ci_gate`. Also fixes an infra-signature regression exposed by the full-log change: the bare step-name signatures (`git fetch`, `Set up job`, `actions/checkout`) matched the checkout/setup preamble of EVERY job log — against a full untruncated log they turned every failing run into `infra_broken`. Replaced with genuine error-text signatures (`could not read from remote`, `download action repository failed`, `unable to access`, `failed to connect`). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
6cd78bffda |
refactor(ci-logs): unify CI-log cache — one fetch path, run-aware freeze
Three cache-layer fixes so every consumer reads CI logs through one
store (`get_ci_logs`), and the cache cannot serve a stale verdict.
B1 — `fetch_pr_failure_logs` is now a thin VIEW over `get_ci_logs`.
It was a parallel fetch path with its own `{sha}.json` cache, own
backoff and 4000-char tail truncation — a second cache that could
drift from the bundle. It now projects the unified `{sha}.full.json`
bundle to the legacy failing-jobs-with-tails envelope. Retained for
the `ci_fetch_pr_failure_logs` MCP tool + legacy prefetch callers.
B2 — a partial (in-flight) bundle re-fetches on a minimum interval
(`CI_LOGS_PARTIAL_MIN_REFETCH_S`, default 60s) instead of on every
call, bounding polling load on the Forgejo UI endpoints.
B3 — a terminal+clean bundle is frozen only while its `run_id`
matches the live run. The cache is keyed by head_sha, but a SHA can
be re-run; a cached bundle whose run_id no longer matches is treated
as stale and re-fetched (last-run-wins). Resolving the live run
costs one cheap ci-detail call; the per-job log fetches are still
skipped on a hit. A `source="local"` RUN_CI_LOCAL bundle keeps its
no-fetch fast path.
`mcp_ci_server`'s `source` probe re-pointed onto the bundle cache.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
drew
|
c4a11901f5 |
feat(ci): unified CI-log cache — phase 2 (consumer migration + RUN_CI_LOCAL)
Builds on
|
||
|
drew
|
27289ea4b7 |
feat(ci): unified CI-log cache — get_ci_logs + implementer path (phase 1)
The implementer's ci_summary carried an empty raw_log_excerpt for every failed gate (diagnosed via PR 39): prefetch._build_ci_summary passed a no-op log fetcher, and ci_summarize._gate_to_nox_session never stripped Forgejo's "(pull_request)" event suffix, so every PR gate fell through to NoParserAvailable with no log fetched at all. Phase 1 — the implementer-facing path: - _ci_logs.get_ci_logs(): unified entry point — every job of a run, full untruncated logs, one cache. `partial` marks an in-flight run; a terminal + clean bundle is frozen forever. Reuses the existing session-cookie login + exponential backoff. Additive — fetch_pr_failure_logs and its 9 consumers are untouched. - ci_summarize: strip the "(pull_request)" event suffix so gates resolve to their nox parser; _no_parser_failure now carries the raw log instead of hardcoding "". - prefetch._build_ci_summary + forgejo_http + __main__: wire a real get_ci_logs-backed log_fetcher so each failed gate's raw_log_excerpt is filled from the cache. - 9 new tests; full controller suite green (1169 passed). Deferred to later phases: re-point the freshness gate / ci_status_poll onto get_ci_logs, RUN_CI_LOCAL into the same cache, and (head_sha, run_id, attempt) multi-attempt keying. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
521117882e |
fix(controller): conflict resolver lands resolutions on the PR
Run-2 PR #40 exposed that the conflict-resolver stage never resolved conflicts *on the PR*. The resolver agent owned the whole git flow (rebase + resolve + push) but was never given the PR head branch name, so it pushed the resolved branch to a guessed branch ("main"), the push was lost, and it reported "resolved" anyway — the controller advanced the workflow on a resolution that never reached the PR. This splits the work along the controller's principle: deterministic git mechanics in the worker, only the semantic merge in the agent. - conflict_rebase.py (new): the worker starts the rebase (prepare_conflict_rebase — hands the agent a real mid-rebase worktree) and, after the agent's resolution, lands it with a *verified* push (finalize_conflict_resolution) — force-with-lease pinned to the PR head the resolver started from, then re-reads the remote to confirm it moved. A failed/rejected push is never reported as resolved. run_conflict_resolver_attempt orchestrates it. - ConflictResolverInputV1 / prefetch: carry head_ref so the worker knows the PR branch (this also makes the worker's fetch_and_validate run for the conflict_resolver — it gated on head_sha AND head_ref). - worker/__main__.py: route conflict_resolver attempts through the new orchestration. - conflict-resolver-worker.md: the agent resolves + continues the rebase and never pushes — the controller lands it. - mcp_git_server.py: add the controller worktree root to the git MCP's allowed bases (every git_* MCP call errored for this role before). The committed-marker check is scoped to the PR's own diff — a whole-tree grep false-positived on repo files that legitimately document conflict markers. The worker's rebase/push are lost-lock guarded so a reaped worker cannot race the shared worktree. 1136 controller tests pass (+11 new in test_conflict_rebase.py). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
3106230ef8 |
fix(controller): cap estimator worker-internal-error retries
An estimator whose session ends without emitting canonical output fails with worker-internal-error. Unlike the implementer, the estimator has no escalation path (it runs pre-tier) and no salvage (it produces no git artifact) — so a flaky estimator session just re-enqueues, with nothing to stop it. Run-2 burned 174 consecutive estimator worker-internal-error attempts on one PR. After _ESTIMATOR_WORKER_ERROR_LIMIT (3) such failures the workflow now STUCKs for operator attention via estimator_failed_twice — symmetric with the implementer worker-error escalation cap. - outcomes.py: the cap + the prior_estimator_worker_errors param. - tick.py: _count_prior_estimator_worker_errors (per-workflow count; the estimator runs pre-tier, so tier is not a meaningful axis). - state_machine.py: estimator_failed_twice description corrected — the event had no emitter before this; strict-parse failures route via contract-violation -> pickup_exhausted, not this event. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
4b184b5a29 |
fix(merge): bypass stale-label veto for controller-managed PRs
merge_drive's pr_is_eligible vetoed any PR carrying auto/needs-* / auto/stale-inactivity labels as "labelled-needs-attention". The T5-7 controller-managed bypass only covered the auto/ready-to-merge gate, so a controller-APPROVED PR still carrying a stale auto/* reflection label was silently dropped from every merge cycle — run-25/26 PR #40 sat APPROVED-but-unmergeable for hours behind "picked 0 candidates" with no diagnostic. For a controller-managed PR the controller DB is the source of truth, so the auto/needs-* veto no longer applies to it; auto/unstable still does (operator override). If the PR genuinely doesn't merge, merge_drive discovers that during the merge attempt and routes it MERGING -> CONFLICT_RESOLVING via merge_base_conflict. pick_candidates now also logs each skip reason, so a stale label can never silently produce "picked 0 candidates" again. Validated live: #40 went from vetoed-forever to picked up -> MERGING -> CONFLICT_RESOLVING. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
f92822affd |
fix(sentinel): bake the CI OOM-prevention fix into duplicated test PRs
duplicate_prs_to_fork.py reset the fork's master to upstream and pushed raw upstream PR heads — neither carries the noxfile/ci.yml parallelism cap, so every sentinel PR's CI risked the 64-worker OOM-kill the pipeline itself hit. The fix is now committed onto the fork's master and merged into each sentinel branch, so it is in effect for the sentinel's CI run while staying invisible in the sentinel PR diff (it sits in the common ancestor of base and head). A merge conflict — a PR that itself edits the capped lines — aborts cleanly and pushes the raw head. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
a6ff7da63c |
fix(controller): recover work lost to plumbing + run-25 reliability fixes
Run-25 surfaced five ways the controller lost or mis-routed correct implementer work. Each is fixed: 1. Canonical-output salvage (agent_runner.py): when an implementer's OpenCode session ends cleanly but emits no V1 output, its commits are no longer discarded — they are pushed and a synthetic outcome=resolved routes the workflow to AWAITING_CI so CI judges the code. Guarded by lost_lock_check so a reaped worker cannot race a fresh one on the shared worktree. 2. Preserve committed-but-unpushed work (workspace.py): the pre-reset snapshot (auto-scratch/pr-<N>) now captures committed commits ahead of the reset target, not just dirty edits — a finished-but-unpushed fix survives for the next attempt to adopt. 3. blocked escalates (outcomes.py): an implementer 'blocked' below MAX_TIER now escalates to a stronger tier instead of dead-ending at STUCK; only 'blocked' AT MAX_TIER STUCKs. 4. Diff-size tier floor (tick.py): the estimator's structured recommended_tier can contradict its own reasoning (a 387-file PR emitted as tier 0). current_tier is now floored deterministically by diff size so a huge PR cannot run on the weakest model. 5. CI log excerpt (prompts.py): the per-gate log excerpt is now tail-clipped, not head-clipped — the failure tracebacks live at the bottom and were being cut off, leaving the implementer blind. 1121 controller tests pass (+34 new). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
3c4d4b3362 |
fix(ci): cap test parallelism to prevent OOM-kills + extend CI-wait timeout
The behave/pabot test sessions derived worker count from the host CPU count; on a 64-core runner that spawned ~64 workers per job, oversubscribed RAM, and the run was OOM-killed mid-suite (SIGKILL / exit 137) — a red job with no test summary. - noxfile.py: cap auto-derived parallelism at _MAX_DEFAULT_PROCESSES=8; TEST_PROCESSES still overrides for hosts that can take more. - .forgejo/workflows/ci.yml: pin TEST_PROCESSES=8 on the unit_tests and integration jobs. - run-controller-state-machine-pipeline.sh: bump CONTROLLER_AWAITING_CI_TIMEOUT_S 2400s -> 4500s — capped (8-worker) CI runs longer, so the old 40-min ci_poll_exhaustion timeout risked STUCKing workflows whose CI was still legitimately running. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
0db0a15dad |
feat(controller): RUN_CI_LOCAL verdict source, ci-not-ready outcome, escalation hardening
Adds RUN_CI_LOCAL — an on-demand local-CI verdict source for when the cluster's Forgejo CI is broken — plus robustness fixes, the telemetry Live-tab rewrite, and PR-level cost attribution. Controller: - RUN_CI_LOCAL: the master swaps its Forgejo CI callbacks for local `forgejo-runner exec` runs (tools/run-ci-full-local.sh + local_ci.py). Async per-(owner,repo,SHA) on-disk job cache; preflights the forgejo-runner binary + Docker daemon at startup (fail loud, not a red verdict on every PR); GCs finished run dirs + per-run actcache. - ci-not-ready implementer outcome + implementer_ci_not_ready event: an implementer that runs before the on-demand verdict exists parks in AWAITING_CI instead of dead-ending at STUCK; capped against ci_red ping-pong. - ci_poll_exhaustion skips its sweep while a local CI run is in flight, so AWAITING_CI workflows queued behind on-demand CI are not STUCK'd by the remote-CI-sized timeout. - Escalate the workflow after repeated worker-internal-error at a tier instead of retrying to pickup-exhaustion -> STUCK. - forgejo_http: normalise Forgejo's per-gate `status` key to `state` so failing gates are actually counted (they previously all read as pending). - Per-tier worker timeouts bumped +15 min; a timed-out attempt's dirty-worktree residue is preserved on auto-scratch/pr-<N> before the next attempt's reset. - Implementer agents now verify only the CI-flagged gate(s) via a targeted re-run rather than the full local battery before claiming resolved/noop. Re-running the whole suite CI will run anyway was the #1 cause of implementer timeouts; CI remains the real gate and re-dispatches the implementer on red. Telemetry: - Live tab rebuilt on /api/live (controller DB run state + the live OpenCode session forest) after the live_log_writer sidecar was retired with the legacy dispatchers. - Durable per-attempt input/output payloads surfaced in the Live drill-down, archived-session detail, and Workflows timeline. - PR-level cost attribution: worker session tags carry -pr-<n>; backfill_llm_activity_pr.py repairs rows written before the fix. Shared: - tools/controller/session_tag.py — one canonical controller-tag parser shared by the telemetry server and the backfill. Tests: new coverage for local_ci (state machine, log parsing, _summarize_run, GC, in-flight probe, preflight), the ci-not-ready path, the escalation/ci-not-ready SQL counters, the ci_poll in-flight skip, and CI-status payload parsing across both sources. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
0bc734c020 |
style: ruff format the controller-state-machine branch (288 files)
Applies `ruff format` to the accumulated formatting debt on this branch. Formatting-only — no behavioral changes. Required for CI/lint's format gate (`nox -s format -- --check`), which the branch was failing on 288 tracked files that drifted from ruff's canonical style. In-progress WIP files are intentionally excluded so this commit stays a clean formatting-only diff. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
a2fcecb28d |
feat(controller): CI-freshness gate — re-trigger stale/infra CI instead of dead-ending
A discovered PR with stale CI (every job failed at the checkout step —
a git-fetch connection reset, pure infra, no code ran) burned an
estimator + a tier-2 implementer and dead-ended at STUCK. The pipeline
had no notion of CI freshness and never triggered CI — only polled.
New early master tick (ci_gate) runs before DISCOVERED->ANALYZING
promotion. For each DISCOVERED pr-kind workflow it classifies the CI
via ci_freshness.classify_ci_result:
- infra_broken — failed; the failing jobs' LOG content carries a
checkout/setup signature (curl 56, expected 'packfile', ...).
Logs are fetched via _ci_logs (session-cookie auth).
- stale — failed; newest status older than CONTROLLER_CI_MAX_AGE_S
(default 6h). Timestamp-based, log-independent — catches an old
failure even when Forgejo has purged its logs.
- no_ci / pending / fresh_real — handled accordingly.
infra_broken/stale/no_ci -> push an empty commit to the PR branch
(Forgejo 15.0.2 has no rerun API), routing DISCOVERED -> AWAITING_CI
(new event discovery_ci_rerun_triggered). The existing AWAITING_CI
poller then gets a real verdict. A reran CI that is ALSO infra/stale
routes AWAITING_CI -> DISCOVERED (new event ci_infra_recheck) so the
gate re-handles it; bounded by a rerun budget of 3, then STUCK.
Also wires the existing CI summarizer into prefetch so workers stop
receiving ci_summary=null.
1095 controller tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
drew
|
f454670a0e |
chore(controller): launch merge_drive from the pipeline script; rename worker-concurrency env var
The launch script now starts merge_drive by default (T5-7: it is the controller's singleton merge stage — APPROVED -> MERGING -> MERGED). Previously the script killed merge_drive as a "legacy" process but never started it, so controller workflows dead-ended at APPROVED. merge_drive moves into the CONTROLLER process group; --no-merge skips it. Also adds the worker thread-pool concurrency knob (default 2 for the trial harness). Renames the misleadingly-named env var CONTROLLER_MAX_CONCURRENT_WORKERS_PER_MACHINE -> CONTROLLER_MAX_CONCURRENT_WORKER_THREADS_PER_MACHINE: it sizes a ThreadPoolExecutor inside one worker process, it does not spawn worker processes. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
012dabcebe |
chore(controller): track the controller-pipeline launch script in tools/
Moved from .drew/ (gitignored, operator-personal) so the launch script is versioned. REPO_ROOT now derives from the script's own location instead of a hardcoded absolute path, and the DB-URL comment documents .devcontainer/.env as the standard place to pin a stable, persistent CLEVERAGENTS_DB_URL (the per-run sqlite path is now just the fallback for an isolated one-off run). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
4e0b1bbdbf |
fix(controller): second review pass — close remaining MERGING stranding paths
The first post-commit fix only handled merge-error-* strings and missed that build_train's single-PR failure reasons flow straight into terminal_state via _release. Four were unmapped (no-head-ref, rev-parse-fetched-head-failed, rebase-timeout, push-failed-force-lease-mismatch) — same stranding bug class. - _resolve_bridge_event no longer returns None. Any unrecognised terminal_state falls back to merge_retry_exhausted (-> STUCK): operator-visible and recoverable beats silently orphaned in MERGING. The three static build_train reasons are mapped explicitly; the catch-all logs loudly so an unanticipated state is still visible. - merge-error-* routing refined: 404 -> merge_external_action (reconcile via PR state), 422 -> merge_base_conflict (CONFLICT_- RESOLVING re-rebases) instead of blanket STUCK. - Hard-crash reaper: merge_drive is a singleton and run_one_cycle is serial, so any workflow in MERGING at cycle start is the residue of a crashed prior cycle. run_one_cycle now reclaims them via merge_interrupted (MERGING -> APPROVED) before picking candidates. Adds bridge.list_merging_workflows. - run_one_cycle warns if a claimed PR is absent from pr_terminal_states instead of silently using the aggregate. 3309 tests pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
17cd91d275 |
fix(controller): post-commit review — close 3 merge-bridge stranding bugs
The trial-5 merge-pipeline split could strand workflows in MERGING
forever on its main paths. The controller has no MERGING handler
(merge_drive owns it), so any terminal_state that maps to no event
leaves the workflow orphaned. Three such holes, all caught by the
post-commit multi-perspective review:
1. merge_train emits "merge-error-{403,5xx,...}" for any non-2xx/409
Forgejo merge POST — none were mapped. Added _resolve_bridge_event:
403 -> branch_protection_blocked, all else -> retry_exhausted (STUCK).
2. run_one_cycle applied one shared outcome.terminal_state to every
claimed PR. A bisected train returns "bisected" (unmapped) so all
PRs stranded; a mixed train could tell a merged PR merge_base_conflict.
CycleOutcome now carries pr_terminal_states and events emit per-PR.
3. Graceful shutdown mid-merge ("stopped") was unmapped. Added the
merge_interrupted event (MERGING -> APPROVED) so the workflow
returns to the handoff state for clean re-pickup.
Also fixes a stale comment in _controller_db_bridge.py (merge_base_conflict
routes to CONFLICT_RESOLVING, not IMPLEMENTING) and adds a drift-guard
test cross-checking the bridge's transition map against the canonical
state machine — that drift is what produced the stale comment.
3304 tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
drew
|
a6986008ee |
feat(controller): trial-5 batch — dispute path, merge pipeline split, conflict-resolver hardening
T5-1 reviewer feedback rendered full-body to the implementer
T5-4/9 implementer dispute path — dispute-at-any-tier with per-tier cap,
OPERATOR_ATTENTION state on stalemate, pr-review-worker-dispute agent
T5-5 reviewer BLOCKING ISSUE EVIDENCE RULE + 5-step validation
T5-7 merge step split into a singleton process — impl/review masters write
APPROVED and stop; merge_drive owns APPROVED -> MERGING -> MERGED
T5-10 merge process is fully deterministic; base conflicts bounce to the
controller's CONFLICT_RESOLVING (LLM); conflict_drive sidecar retired
T5-11 implementer fast success path — verified-clean outcome so a no-op
after conflict resolution doesn't force busywork
T5-12 conflict-resolver permissions fixed across all paths (/tmp/** glob)
T5-13 conflict-resolver PR-intent prehydration (title/body/comments)
Adds tools/_controller_db_bridge.py so merge_drive reads the controller DB
directly (Option B), plus APPROVED + OPERATOR_ATTENTION states, the
dispute/verified-clean events, and the V1 contract fields backing them.
Reviewer model: baseline -> sonnet, dispute -> opus.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
drew
|
ab8a5e5bfb |
fix(controller): T4-4 — route issue workflows to STUCK when role has no prefetch
Trial-4 observed wf=6 (entity #34, kind='issue') promoted ANALYZING→ IMPLEMENTING by the estimator, but build_implementer_input raises ``ValueError("implementer prefetch needs kind='pr', got 'issue'")``. The scheduler logged WARNING and retried every ~5s forever — log spam + workflow never reached a terminal state. Pre-fix: 2026-05-18 21:13:43 WARNING ... prefetch failed ... 2026-05-18 21:13:50 WARNING ... prefetch failed ... 2026-05-18 21:13:55 WARNING ... prefetch failed ... (50+ identical lines over the trial) Fix: before calling prefetch, scheduler checks the workflow's kind column. If kind='issue' and role in {implementer, reviewer, conflict_resolver}, the workflow is routed to STUCK with reason='issue-not-supported (T4-4)'. Operator-visible controller_events row tags this as 'issue-not-supported'. Future work: IssueImplementerOutputV1 contract already exists in contracts/v1.py:362; a follow-up batch can add ``build_issue_ implementer_input`` + remove this short-circuit. For the trial we just need issues to stop wedging. Test: TestIssueWorkflowRoutedToStuck in test_batch_s_fixes.py. 833 controller tests pass (was 832; +1). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
46764b841b |
fix(controller): batch S — 11 fixes from trial-4 finding + 2nd code-review pass
Trial-4 ran the controller past ANALYZING for the first time. One
critical regression surfaced live (T4-1 — scheduler dup-attempt
enqueue), and a parallel agent-driven code review found 8 more bugs
across "races + error paths + agent quality" classes. Batch S
addresses 11 of those.
LIVE-OBSERVED REGRESSION (trial-4 2026-05-19 00:49):
T4-1 — Scheduler enqueues duplicate estimator after workflow advanced
File: tools/controller/master/scheduler.py
The "already pending" check filtered on
``status IN ('pending', 'in_progress')`` — missing the brief window
where the prior attempt is ``status='complete'`` but tick hasn't yet
processed its outcome. Result: scheduler enqueues a 2nd estimator/etc.;
when its outcome fires from a now-advanced state, IllegalTransition
→ STUCK. Hit wf=1 in trial-4. Fix: also skip when an unprocessed
``complete`` attempt exists (finished_at > w.last_transition_at).
E-5 — IllegalTransition over-aggressive STUCKing
File: tools/controller/master/tick.py
Companion fix to T4-1. Even if T4-1 escapes in some other path (or
a worker delays writing outcome past tick), a stale-outcome
(workflow already advanced via parallel path like ci_status_poll or
reconciliation) shouldn't STUCK. New ``_is_stale_role_outcome``
helper recognizes "this role's outcome arrived after the workflow
moved past its origin state" → consume the attempt, bump
last_transition_at, continue. Only genuine state corruption → STUCK.
E-3 — Corrupted output_payload silently wedges workflow
File: tools/controller/master/tick.py
Pre-fix _decode_output_payload swallowed json.JSONDecodeError →
mapper returned None → tick bumped last_transition_at but workflow
never moved. Operator had no signal. Now raises
``CorruptedOutputPayload`` → tick routes to STUCK with reason.
E-1 — Contract-violation routed to STUCK on first attempt
File: tools/controller/master/outcomes.py + tick.py
v9 spec promised retry-once-with-corrective-prompt for
contract-violation; the column ``strict_parse_retries`` existed but
nothing read/incremented it. _map_failed_outcome now takes
``prior_contract_violations`` count (queried in tick.py); STUCKs
only when count ≥ _CONTRACT_VIOLATION_RETRY_LIMIT (2). First two
violations re-enqueue.
R-1 — Reconciliation flipped workflow state mid-attempt
File: tools/controller/master/reconciliation.py
Worker holding a lock + heartbeating; reconciliation flipped current_state
to MERGED/ABANDONED based on Forgejo; tick.py then skipped the
worker's eventual write (terminal-state exclusion). Worker's output
lost. Fix: _apply_transition first checks for in_progress attempts
on the same workflow and defers if any exist.
R-2 + R-8 — _apply_transition lacks current_state guard
Files: reconciliation.py + merging.py
Same pattern as ci_status_poll's existing TOCTOU defense. UPDATE now
filters ``WHERE current_state = :from_state``; on rowcount=0, skip
the event row. Prevents racing ticks from over-writing each other.
R-4 — Reaper UPDATE didn't re-check heartbeat freshness
File: tools/controller/reaper.py
A worker's healthy heartbeat between reaper's SELECT and UPDATE
would be silently overwritten; the worker's later _write_outcome
(filtered on locked_by_instance) returned rowcount=0 → output lost.
UPDATE now includes the same freshness filter as the SELECT, so
fresh heartbeats protect the row.
E-8 — merging_retry_count not reset on STUCK/abandoned paths
File: tools/controller/master/merging.py
Pre-fix, only 200/409/422 paths reset the counter. 403 (branch
protection), 404 (externally closed → ABANDONED), retry-exhausted
(STUCK) leaked stale counts. If operator unsticks a STUCK workflow
back through MERGING, the stale count made it STUCK again sooner
than expected. All terminal-state-changing paths now reset.
A-1 — commit_shas validation accepted any ≥7-char string
File: tools/controller/mcp/implementer_builder.py
Tightened to ``re.fullmatch(r"[0-9a-f]{7,40}")``. Pre-fix an agent
could pass any 7+ char string; head_sha_advanced accepted the
hallucination; CI poll then 404'd on the fake SHA forever (until
2h ci_poll_exhaustion).
A-2 — merging-409 → IMPLEMENTING(tier=NULL) trap
File: tools/controller/master/merging.py
On 409 the handler routes to IMPLEMENTING(tier=tier_last_succeeded);
if that's NULL (e.g., metadata-only → REVIEWING → approve → 409 path
where implementer_pushed never fired), the MCP rejects tier=NULL →
contract-violation → STUCK. Fix: default to current_tier when
tier_last_succeeded is NULL.
A-9 — Reviewer cross-field check: verdict ↔ suggested_next_action
File: tools/controller/mcp/reviewer_builder.py
Agent could set verdict=approve + suggested_next_action=abandon; the
master fired reviewer_approve regardless. New _VERDICT_ACTION_COMPAT
map enforces compatible pairs at finalize.
TESTS:
- New file ``test_batch_s_fixes.py`` with 13 regression tests, one
per fix class.
- Updated test_master_outcomes.py for the new contract-violation
retry behavior.
- Updated test_master_prefetch.py fixture to bump last_transition_at
past the seeded attempts (the new T4-1 filter would otherwise
correctly identify the pre-seeded attempts as unprocessed).
Total: 819 → 832 tests, 0 regressions.
DEFERRED to a follow-up batch (per PENDING_FIXES.md):
- 8 MEDIUM items (error UX, dead fields, signal-loss in error paths)
- 5 LOW items (agent quality polish)
- 4 CONFIRMED-CLEAN (no fix needed)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
3e23853ffa |
fix(controller): batch R — wire 5 V1-contract fields the controller silently dropped
Trial run-3 (2026-05-19) surfaced the first instance of a broader bug class: V1 contract fields existed and agents emitted them, but no controller code wired them into state transitions. An adversarial "walk the happy path" code review found 4 more, all listed below. The class shape: a V1 field is "Required iff X" by contract docstring, the worker emits it correctly, but the master reads the wrong field (or doesn't read it at all), so a critical state transition silently no-ops or drops to the wrong default. FIX #0 — outcome-mapper early-return (committed earlier in this session) — moved role dispatch before the ``outcome is None`` guard so estimator+reviewer+summarizer (V1 contracts without an ``outcome`` field) are correctly handled. Without this fix, all estimator attempts in trial run-3 completed successfully then were silently discarded, stranding all 6 workflows in ANALYZING. FIX #1 — current_tier never written from estimator's recommended_tier File: tools/controller/master/tick.py The ANALYZING→IMPLEMENTING UPDATE wrote only current_state / last_transition_at / entered_state_at. recommended_tier from the estimator payload was never extracted, so every PR ran at the workflow's creation-time tier (typically 0) regardless of what the estimator recommended — the entire tier-escalation ladder was informational-only. Fix: per-event ``extra_set`` clauses; on ``estimator_done`` / ``estimator_metadata_only`` events, set ``current_tier = :rec_tier`` from the payload (with 0..2 validation). Tests: TestEstimatorRecommendedTierWritten (3 cases). FIX #2 — approved_at_sha never passed to merge callback File: tools/controller/master/merging.py, forgejo_http.py ReviewerOutputV1.approved_at_sha is the exact SHA the reviewer signed off on. Pre-fix the MergeCallback signature was ``(owner, repo, pr_number)`` — Forgejo merged whatever HEAD currently was. Race condition: a concurrent push (operator or another driver) between approval and merge would silently merge unapproved code. Fix: extended signature to ``(owner, repo, pr_number, approved_at_sha)``; SQL SELECT now pulls the latest reviewer attempt's output_payload as a subquery; merge_pr forwards it to Forgejo as ``head_commit_id`` (Forgejo refuses with 409 if HEAD has advanced). Defensive: still merges when approved_at_sha is None but logs a WARNING. Tests: TestApprovedAtShaPassedToMerge (2 cases). FIX #3 — tier_last_succeeded column had ZERO writers File: tools/controller/master/tick.py The schema column existed; the merging.py 409-conflict path read it to recover the last-known-good tier; but NOTHING ever wrote to it. Every workflow's tier_last_succeeded was permanently NULL → the 409-recovery path transitioned to IMPLEMENTING(tier=NULL) → scheduler silently coerced to tier 0. Fix: on ``implementer_pushed`` event, ``UPDATE workflows SET tier_last_succeeded = current_tier``. Tests: TestTierLastSucceededWritten. FIX #4 — outcome column NULL for estimator/reviewer/summarizer File: tools/controller/worker/runner.py ``workflow_attempts.outcome`` is the operator-facing audit column. Pre-fix the runner extracted ``output_payload.get("outcome")`` blindly — works for implementer/conflict_resolver but those three roles have no ``outcome`` field. Result: ``SELECT … WHERE outcome IS NOT NULL`` audit queries silently missed every estimator/reviewer/ summarizer attempt. Fix: new ``_derive_outcome_for_audit(role, payload)`` helper synthesizes meaningful per-role values: - implementer/conflict_resolver: payload['outcome'] (unchanged) - reviewer: payload['verdict'] - estimator: 'metadata-only' OR f'tier-{recommended_tier}' - summarizer: 'summarized' Tests: TestOutcomeAuditColumn (parametrized 8 cases). FIX #5 — conflict_resolver new_head_sha never preferred File: tools/controller/worker/runner.py ConflictResolverOutputV1.new_head_sha is "Required iff outcome='resolved'" (the canonical post-rebase branch tip). Pre-fix runner.py used ``commit_shas[-1]`` for head_sha_after — works for normal git rebase --continue but wrong for resolvers that did force-pushed merge commits where the last commit SHA ≠ the branch tip. CI status poll would then poll the wrong SHA. Fix: when role=='conflict_resolver', prefer ``new_head_sha`` over commits[-1]. Tests: TestConflictResolverNewHeadShaUsed (2 cases). ALSO updated existing tests that papered over the original bug: - test_master_outcomes.py: estimator tests used to inject a fake ``"outcome": "(implicit)"`` field; now use real V1 shape (no outcome). Reviewer tests now use ``verdict`` (the real V1 field) not ``outcome``. - test_master_tick.py reviewer tests: same `verdict` switch. - test_master_merging.py: updated all 13 ``lambda o, r, n: ...`` merge-callback stubs to the new 4-arg signature. CONFIRMED-CLEAN (no fix needed) by the same code review: - outcomes.py post-fix-#0 - prefetch.py field reads - prompts.py field accesses - ci_status_poll.py role+outcome filter The above were verified to handle all 5 V1 contract shapes correctly. Total: 802 → 819 controller tests, 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
751dbfed33 |
fix(agents): batch Q — tier-variant model injection + Option B (drop dead .txt files)
Two related fixes from empirical testing of OpenCode's model
resolution against the controller's tier-escalation ladder.
ROOT-CAUSE FINDING (empirical, 2026-05-18)
==========================================
Spun up a probe agent (.opencode/agents/model-probe.md), asked the
model to self-identify, and tried three model-routing mechanisms:
1. **POST /session ``model`` in body**: OpenCode 0.x silently
IGNORES this — all probes returned ``openai/gpt-5.3-chat-latest``
(OpenCode's fallback default), not the requested haiku/sonnet/opus.
2. **.md frontmatter ``model:`` line** (after OpenCode restart):
HONORED — pinning to claude-haiku-4-5 yielded haiku, pinning to
sonnet yielded sonnet, etc.
3. **opencode.json ``agent.<name>.model``**: HONORED — same result
as .md frontmatter.
CONSEQUENCE: pre-batch-Q the tier variants had NO ``model:`` in
their .md frontmatter; the model lived only in
``.opencode/models/task-implementor-tier-<N>.txt`` files that the
dispatcher passed via POST /session body. Since OpenCode ignores
that pass-through, ALL FOUR tier variants ran on the SAME default
model (gpt-5.3 in this configuration) — the entire tier-escalation
ladder was cosmetic for model selection. The trial-2 sessions
logged ``model override -> claude-haiku-4-5`` but the actual
generation was on something else entirely.
WHAT THIS COMMIT DOES
=====================
1. **sync_tier_models.py rewrite** (already in batch P, refined here):
inject ``model: <providerID/modelID>`` line into each generated
tier variant's .md frontmatter. This is the mechanism OpenCode
actually reads at startup. The model values come from tiers.yaml
(source of truth).
2. **Drop the dead .opencode/models/task-implementor-tier-*.txt
files** (Option B): the dispatcher pass-through they fed was
empirically dead — OpenCode doesn't read the model from POST
/session. ``sync_tier_models.py`` now removes any stale .txt
files on each run (so a developer can't accidentally re-create
them).
3. **Test updates**:
- ``test_no_stale_variant_txt_files_remain``: pins that the .txt
files stay deleted (was ``test_every_tier_has_a_variant_txt``).
- ``test_each_variant_md_carries_correct_model_from_manifest``:
pins that the .md frontmatter model: matches tiers.yaml (was
``test_each_variant_txt_matches_manifest_model``).
- ``test_every_agent_file_reference_in_opencode_json_resolves``:
relaxed to skip when opencode.json has no agent block (which is
the Option B steady state). Still pins {file:...} resolution
for any future use.
- ``test_each_variant_md_matches_renderer_output_for_its_tier``:
updated to call the new renderer signature ``render_task_
implementer_variant(source, model)`` (was the byte-copy
identity test, retired because variants now differ by the
injected model: line).
- ``TestTaskImplementorVariantsAreByteIdentical`` → renamed
``test_variants_identical_except_for_model_line``: strips the
model: line via regex and asserts the rest is byte-identical.
OPERATOR WORKFLOW (unchanged surface)
=====================================
To swap a tier's model:
1. Edit ``.opencode/models/tiers.yaml`` (one line)
2. ``python3 tools/sync_tier_models.py`` (regenerates .md; removes
any stale .txt)
3. Commit both files
4. Restart OpenCode (it caches .md frontmatter at startup)
What's still ahead (deferred):
- ``_resolve_role_model()`` in tools/_opencode_worker.py is now
proven dead code (reads .txt files that don't exist; injects
model into POST /session body that OpenCode ignores). Should be
deleted in a follow-up — kept now to minimize blast radius.
- The misleading "if generation uses a different model, restart
OpenCode so opencode.json's {file:...} re-resolves" log line in
``_opencode_worker.py:1569`` is wrong post-Option-B; should be
retired with the dead code above.
- Whether opencode.json's agent block supports a ``permission`` field
is the gate for an even deeper simplification (Option D). Skipped
for now per operator direction; the controller path uses .md
frontmatter for permissions.
3148 tests pass, 4 skipped, 0 failures.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
2ad0958cb7 |
fix(controller): batch O — trial-2 fix: interpolate real attempt_id into prompts
Trial run-2 surfaced the actual blocker: the response-builder MCPs
shared their module-global ``_STATE`` across OpenCode sessions (they're
registered as ``type: local`` in opencode.json, so OpenCode spawns one
subprocess per OpenCode-server lifetime, not per session). The CA2/PD5
cross-session reset I added in the M-batch keyed on
``identity.attempt_id`` — but the prompt template literally read:
1. ``estimator_start(attempt_id=..., workflow_id=..., ...)``
The ``...`` were placeholder ellipses, not interpolated. So agents
guessed ``attempt_id=1`` every single time. The cross-session reset
compared ``1 == 1``, decided "same attempt — no reset", and the prior
attempt's ``finalized=True`` persisted forever. Every estimator
session after the first failed with ``"response already finalized;
no further mutations allowed"`` on every ``_set_*`` call → no
finalize → 30s worker timeout → workflow STUCK.
Observed in trial run-2: 2 successes (attempts 1, 2), then 15
consecutive failures (attempts 3–17 across all 6 workflows) before
the pickup_guard would have STUCK every workflow.
Two fixes (defense in depth):
1. **Unconditional reset on _start** in all 5 builders. We can't
distinguish "agent retry in same session" from "new session reusing
this MCP" reliably — the observable signature is identical. Just
reset whenever ``_STATE.started`` is True; ``reset_for_new_attempt``
logs a WARN if the prior state was in-flight so abandoned attempts
are still visible to ops. The agent's last ``_start`` always wins.
2. **Interpolate real attempt_id + workflow_id + pr_number + head_sha
into all 5 prompts' Output contract sections**. The agent_runner
now injects ``attempt_id`` into ``input_payload`` (matching the
existing ``workspace_dir`` injection pattern). Each ``build_*_prompt``
reads ``input_payload.get("attempt_id")`` and bakes the concrete
value into the MCP-call signature shown to the agent, plus a
``PASS THESE EXACT VALUES`` directive.
Also strengthened each prompt's ``_finalize`` line with
``**You MUST call this tool — without it the controller times out.**``
so the agent understands the contract is hard, not optional.
Updated tests:
- ``test_double_start_refused`` → ``test_double_start_resets_silently``:
pins the new permissive-reset behavior + asserts the WARN log fires.
- ``test_implementer_rejects_intra_session_double_start`` updated for
same reason; now asserts the second _start succeeds + last-wins
semantics (used_tier == new tier).
- NEW ``test_prompt_interpolates_real_attempt_id`` parametrized over
all 5 roles: asserts ``attempt_id=42`` + ``workflow_id=7`` appear
literally in the rendered prompt AND that ``attempt_id=...`` /
``workflow_id=...`` placeholders DO NOT.
Total: 795 → 800 tests, 0 regressions.
The model-override files (.md + .txt) are still in place from the
prior turn — that change is orthogonal to this bug fix; OpenCode
ignores the dispatcher's pass-through per the README, and the actual
generation has been on haiku the whole time. The .md frontmatter
change to sonnet will only take effect on the next OpenCode restart.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
a4892f8db9 |
fix(controller): batch N — second-round adversarial review fixes
Round-2 adversarial review on commit
|
||
|
drew
|
84da774212 |
fix(controller): batch M — post-Phase-1m adversarial review fixes
Three rounds of adversarial review (Chief Architect / Principal Dev /
Senior Test Engineer) on commits 3ca794be7..db12f45ac surfaced ~35
issues. This commit addresses 25+ across criticals, highs, and
mediums, and adds 40 new tests covering the changes plus key gaps the
review identified.
CRITICALS (M1):
- CA1: stale {role}_output.json from a prior attempt on the same
per-PR workspace was readable as "fresh" output of the new attempt.
agent_runner now unlinks the MCP-canonical path AND every fallback
path BEFORE the session runs.
- CA2/PD5: opencode.json-registered MCP subprocesses persist across
OpenCode sessions, but BuilderState was module-singleton. Added
reset_for_new_attempt() + cross-session detection (compare
identity.attempt_id) to every *_start; force-resets with WARN if
prior attempt was interrupted (timeout / lost lock).
- PD3: inline-JSON callback could overwrite an MCP-written canonical
V1 file with adapted-from-prose garbage. Callback now inspects
existing files and skips when V1 is already present.
- PD4: FORGEJO_URL = .rstrip("/api/v1") is a character-set strip —
catastrophic for hosts whose path contains /v1 in the middle.
Replaced with explicit endswith()-based suffix strip.
- CA10: clone URL embedded $FORGEJO_TOKEN, persisted into
.git/config where any agent could cat it. Token now sourced via
local credential.helper at clone-time, URL kept clean.
- CA12: state.finalized was set BEFORE the file write, so disk-full
/ OSError left the agent unable to retry finalize. Reordered.
HIGHS (M2):
- CA3/PD12: output_path validation (NUL-byte rejection, must be
absolute, parent-not-file check) in finalize_and_emit.
- CA6: ci_status_poll SELECT only considered implementer attempts;
conflict_resolver also pushes commits. SQL now unions both roles.
- PD9: ci_status_poll could advance on a stale "resolved" SHA from a
blocked attempt (whose head_sha_after == head_sha_before). Added
outcome='resolved' filter.
- CA8: cancelled/stale CI states mapped to ci_red_retry_same_tier,
burning pickup_count on healthy PRs. Both now wait (treated as
operator/system action, not failure). timed_out stays red.
- TE9: unknown Forgejo CI states now WARN-log instead of silently
being treated as pending — operators see new state strings.
- PD8: ci_status_poll event_type strings standardized to match the
state-machine event names (ci_green / ci_red_retry_same_tier)
instead of legacy ci-green / ci-red.
- CA7: inline-JSON callback now checks lost_lock_check BEFORE write
so a file isn't staged after lock loss.
- PD10: atomic .tmp + os.replace writes in both MCP finalize and
inline callback so the poller never sees a half-written file.
- PD16: inline_output_callback exceptions now re-raise as WorkerError
instead of being silently logged (root cause was buried 30s later
in a canonical-output timeout).
- CA9: WorkerConfig manual rebuild on --max-concurrent/--poll-interval
silently dropped new fields. Use dataclasses.replace, matching
round-4 P5 fix in master/__main__.py.
MEDIUMS (M3) — legacy_adapter quality upgrades:
- PD1: unrecognized confidence values now WARN instead of silently
defaulting to "medium" — surfaces agent prompt drift.
- PD2: estimator recommended_tier clamped to {0,1,2} so an out-of-
range int doesn't bypass the adapter's whole purpose.
- PD7: reviewer blocking_issues list-of-strings coerced into the
list-of-BlockingIssue-dict shape strict_parse requires.
- PD13: conflict_resolver prompt defaults tier=1 + warns instead of
raising; the scheduler always sets it but defends against drift.
- PD14: summarizer summary < 50 chars padded with a clear marker so
strict_parse accepts it (and the truncation is visible).
- PD15: implementer blockers capped at 4096 chars each so a buggy
agent can't blow up audit log / DB column.
- PD17: launch script accepts either FORGEJO_TOKEN or GITEA_TOKEN
with a clear error if both are unset.
- PD22: conflict_resolver adapter accepts singular commit_sha
fallback, matching implementer.
- CA4: every adapter invocation logs role + payload key fingerprint
so operators can measure agent-migration progress.
- estimator + summarizer now have explicit _start tools (the prompts
already referenced them; previously absent → first call would fail).
TESTS (M4) — added 40 tests in test_post_review_fixes.py:
- Cross-session MCP state reset (implementer + reviewer + estimator
+ summarizer; intra-session double-start still rejected).
- finalize_and_emit output_path precedence (arg > env > stdout),
parent-dir creation, rejection of relative/NUL paths, failed-write
leaves state retryable.
- legacy_adapter quality: tier clamping, blocker cap, non-string
commit warning, blocking_issues string coercion, conflict_resolver
full roundtrip + non-resolved head clearing, summarizer padding,
confidence warning, V1-passthrough no-log.
- opencode.json registration parity: every MCP the prompts name is
registered with the correct module path.
- Per-role prompts mention {role}_output.json (canonical poller path)
+ the "DO NOT emit chat-JSON" directive.
- FORGEJO_URL suffix-strip parametrized table.
- agent_runner stale-file cleanup: prior-attempt file is unlinked
before a new session can read it as phantom output.
Also updated 2 pre-existing tests for the CA8 / PD8 / PD13 behavior
changes (cancelled→wait, event_type renaming, conflict_resolver
default-tier warning).
Total: 741 → 781 tests, 0 regressions.
DEFERRED (M5 follow-up — non-trial-blocking):
- CA5: head_sha verification via git cat-file (requires subprocess).
- CA11: discovery_interval_s wall-time cadence (vs iteration count).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|