a21466add232d59cdec1604e09d58ca05659a623
2536 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 drew
|
a21466add2 |
feat(controller): Phase 4 metadata-hygiene + round-2 adversarial fixes
Five deterministic, idempotent Phase 4 checks: 1. completed_not_closed — close linked issues on MERGED 2. closing_keyword_fixup — add Closes #N to PR bodies 3. label_sync_from_issue — copy Priority/Type/MoSCoW labels 4. state_label_inference — sync State/* label to current_state 5. milestone_assignment — copy milestone from linked issue All default-off via CONTROLLER_METADATA_HYGIENE_ENABLED + per-check granular env flags. Dry-run mode shares the grooming CONTROLLER_GROOMING_DRY_RUN flag. Round-1 fixes (applied before this commit): - False-positive idempotency lock (executed=True on skip) - Unbounded MERGED scan → LEFT JOIN candidate query - Duplicate _classify_forgejo_status → import from forgejo_writes - Bare-ref regex too broad ([#42](url) misread) → add [ lookbehind - Wrong audit stage → 'metadata_hygiene' Round-2 adversarial fixes (3 architect, 4 principal, 7 test engineer): - completed_not_closed: executed=True only when ALL refs close; partial success writes executed=False so remaining issues retry - milestone_assignment: was calling get_pr_details (hits /pulls/, returns 404 for plain issues) → now uses get_issue_state (/issues/{n}) so milestone fetch works for all issue types - label_sync failure path: write executed=False audit row for observability; pre-fix left no audit trail for persistent failures - _BARE_REF_RE: add ( to lookbehind to exclude (#42) link destinations - state_label_inference: re-read current_state inside inner session to avoid stale-snapshot spurious label writes across session boundaries - _last_synced_state: add decision_id DESC tiebreaker for same-second wall-clock rows - dry-run completed_not_closed: separate early-return path to avoid inflating completed_not_closed_executed counter 71 tests (54 round-1 + 17 round-2): - TestCompletedNotClosedPartialSuccess (3) — partial/zero/full success - TestLabelSyncAdjustLabelsFailure (2) — failure audit + retry - TestStateLabelAdjustLabelsFailure (2) — no executed=1 on failure - TestStateLabelInferenceTerminalWorkflows (3) — MERGED/ABANDONED sync - TestLastSyncedStateDryRunThenReal (2) — dry-run → real-run - TestClosingKeywordFixupBareRefAlreadyCovered (2) — candidates subtraction - TestErrorPathHandlingRound2 (3) — label_sync + state_label errors Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
drew
|
4d969eaf2b |
feat(controller): Phase 3 — Gate 3 reviewer-abandon
When the reviewer finishes a review and judges the work fundamentally unworkable (implementation surfaced misdiagnosis, obsoleted-by-other-work, or irreducible complexity), it can now emit verdict='abstain' + suggested_next_action='abandon' with a Gate-3 abandon_reason_category. The controller routes REVIEWING → ABANDONED and (when the kill switch is on) performs the Forgejo close via the reviewer-abandon side-effect tick — no implementer/CI/merge cycles. Wired with the same defense-in-depth pattern Phase 2 established: MCP setter validation + outcomes mapper dispatch with confidence gating + Pydantic atomicity validator + side-effect tick with audit-trail attribution (cause=REVIEWER_ABANDON, event_type='reviewer_abandon'). Default-off CONTROLLER_GATE3_ABANDON_ENABLED kill switch so a fresh deploy is audit-only until the operator explicitly enables Forgejo writes. Bundled refactor: hoisted the 9 Gate-2 + 3 Gate-3-exclusive abandon categories into tools/controller/contracts/abandon_categories.py (triggered by Phase 3 per the plan's follow-up backlog). Both gates now consume the shared frozensets; doc-contract tests grep each agent prompt against the canonical list. Adversarial review (2 rounds): caught + fixed MCP cross-check ordering (atomicity FIRST so missing-setter shows actionable error), confidence=None symmetric downgrade across both gates, dead blocking-issues extraction in _run_close, idempotency clock-collision in the test, low-vs-missing reason-string conflation, and several test-quality gaps. 4064/4071 tests passing (7 pre-existing failures unrelated to Phase 3). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
a91df787d7 |
feat(controller): Phase 2 — Gate 2 estimator-abandon
Adds the second of three abandon gates: the estimator (Gate 2) can
mark a work item fundamentally unworkable, transitioning the
workflow ANALYZING -> ABANDONED and triggering a Forgejo close via
Phase 1's decomposed close_act orchestrator. Catches abandon cases
at the cheapest LLM stage, before implementer/reviewer tiers fire.
Substantive:
- EstimatorOutputV1: additive verdict + abandon_reason_category +
abandon_reason_detail fields (pre-Phase-2 outputs still parse).
@model_validator enforces abandon-requires-category atomicity at
parse time — third defense layer beyond MCP setter + outcomes
mapper
- state_machine: estimator_abandon event + (ANALYZING,
estimator_abandon) -> ABANDONED. 57 transitions; invariants clean
- mcp/estimator_builder: estimator_set_verdict setter validates
verdict enum + 9-category whitelist (scope_intractable,
intent_wrong, security_regression, deprecated_dependency,
breaks_protected_invariants, out_of_scope, low_value,
unmaintained_path, policy_violation) + cross-field rules
- outcomes._map_estimator_outcome: dispatch verdict='abandon'
-> estimator_abandon, with confidence-low downgrade to
estimator_done (honors the agent prompt's documented "high or
medium" requirement)
- estimator_abandon_side_effects.py: per-state side-effect tick
modeled on grooming_side_effects.py; invokes close_act with
cause=Cause.ESTIMATOR_ABANDON + event_type='estimator_abandon'
- _events.py: shared latest_transition_event +
workflows_with_latest_transition_in helpers; dialect-aware
payload['event'] extraction (SQLite json_extract +
PostgreSQL ->>); centralizes the event_type='transition' +
payload['event'] convention that side-effect ticks consume
- gate2_abandon_config.py: CONTROLLER_GATE2_ABANDON_ENABLED kill
switch (default false). Fresh Phase 2 deploys are audit-only
until operator explicitly enables; dry_run shared with grooming
for unified safe-rollout staging
- .opencode/agents/estimator-implementation.md: GATE 2 ABANDON
section with 9-category criteria + low_value disqualifier ("PR
cites an issue/ticket -> route to reviewer instead")
Round-2 adversarial-review fixes (all required pre-commit):
- forgejo_writes.close_issue / close_act: NEW cause + event_type
kwargs (defaults preserve Phase 1 grooming behavior; Phase 2
callsite overrides). Fixes audit-trail attribution: telemetry
queries SELECT WHERE cause='estimator_abandon' now return the
right rows. Phase 1 regression test pins the grooming defaults
- tick.py operator_unstick lookback: dialect-aware json_extract
fix (Phase 1 carry-over bug; would silently no-op on PostgreSQL)
- grooming_side_effects.py: idempotency filter now keys on
check_name set (grooming check_names only) so a Phase 1 close
and a Phase 2 close on the same workflow don't cross-cancel
Tests (+50): TestEstimatorOutputV1Phase2,
TestEstimatorAbandonStateMachine, TestMapEstimatorOutcomePhase2
(including confidence-low downgrade), TestEstimatorSetVerdict
(all 9 categories + cross-field rules), TestEventsHelper,
TestEstimatorAbandonSideEffectTick (including
test_close_writes_estimator_abandon_cause_and_event_type pinning
the audit-trail attribution, and Phase 1 regression guard).
Doc-contract test asserts all 9 categories appear in the agent
prompt. 1509/1509 passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
016b348117 |
feat(controller): grooming gate (Phase 0 + Phase 1 worker-shape dispatch)
Phase 0 (foundation):
- Cause enum (controller_events.cause) for action attribution
- Schema: grooming_decisions audit table; workflows gains
grooming_evaluated_at + deferred_reason + deferred_at +
deferred_target_workflow_id; pulls gains touched_files
- audit_comments: CLOSE / DEFER templates + render_comment_template
- forgejo_writes: close_issue + defer_issue 5-step crash-safe protocol
(fingerprint dedup, error matrix, dry-run)
- patch_pr_state callback in forgejo_http
- grooming_config: 22-env-var frozen-dataclass config + log_effective
- pulls.touched_files cache extension (_pipeline_cache.py schema v8)
- reaper.reap_grooming_decisions audit-retention sweep
- reconciliation RESUME guard (deferred_reason)
Phase 1 (worker-queue shape, 2026-05-25):
- New state: GROOMING. New events: grooming_started, groom_verdict_
{proceed,defer,close}. 5 new transitions; all invariants still clean
- GroomingInputV1 + GroomingOutputV1 Pydantic contracts
- outcomes._map_grooming_outcome routes verdicts to state-machine events
- prefetch.build_grooming_stage_b_input + list_open_prs callback
- scheduler GROOMING -> grooming_stage_b role
- promote: cfg-gated DISCOVERED -> GROOMING when CONTROLLER_GROOMING_
ENABLED=true; issues skip grooming
- forgejo_writes decomposed: close_act/defer_act (Forgejo writes only;
state-machine already transitioned) + close_decide_and_act/
defer_decide_and_act (Phase 0 callers); _apply_workflow_transition
is underscore-private
- grooming.py library: tokenization, suspicion scoring (Jaccard +
weighted overlap), deterministic checks, action -> verdict mapping
- mcp/grooming_builder.py: 14-tool FastMCP server emits GroomingOutputV1
- .opencode/agents/grooming-stage-b.md: duplicate-detection agent
prompt (claude-haiku-4-5)
- grooming_side_effects.run_grooming_side_effects_tick: per-state tick
performs Forgejo writes after groom_verdict_{defer,close} fires.
Filters on event_type='transition' + payload.event (centralizes the
convention pending Phase 2's latest_transition_event helper)
- GroomingCallbacks frozen dataclass; loop.py + __main__.py wired
Worker role registry (single source of truth):
- worker/roles.py: WORKER_ROLES + WorkerRoleSpec + default_roles_csv
+ output_filename_for. agent_runner.ROLE_TO_MCP_MODULE / ROLE_TO_
OUTPUT_MODEL derive from it; opencode_session.agent_name_for reads
it for flat cases; all 6 prompt builders use output_filename_for;
worker --roles default = default_roles_csv(); launcher script
derives --roles via shell substitution. Cross-site invariant test
enforces alignment across 5 sites + opencode.json MCP registry.
Phase 0 silent-bug fix:
- reconciliation.py RESUME guard SELECT now includes deferred_reason
(was missing since Phase 0; guard was a silent no-op). Tightened
from getattr to attribute access to fail fast on future omissions.
Tests (1456 total, +91 grooming-specific):
- test_grooming_phase0.py: 34 tests (orchestrator matrix, crash
recovery, idempotency, dry-run)
- test_grooming_phase1.py: 60 tests (library, contracts, state
machine, outcomes, scheduler, promote, prefetch, act-variants
with signature parity, side-effect tick incl. natural-idempotency
+ executed-flag-skip + verdict-mismatch + reconciliation RESUME)
- test_mcp_builders.py TestGroomingBuilder: 29 tests (happy paths
+ 22 validation rules + Pydantic round-trip + master-tick-read-
path companion)
- test_worker_agent_runner.py TestRoleMaps: cross-role wiring
alignment + agent-prompt-vs-worker-fallback filename contract +
inspect.signature equality (close_act/defer_act vs
close_issue/defer_issue)
- test_state_machine.py: transition count 51 -> 56 +
events_from_grooming
Live-validated end-to-end on 4 staged sentinel PRs (#55-#58) in
dry_run: agent emits verdicts via MCP, state-machine transitions
fire, side-effect tick writes audit row, deferred_reason gates
reconciliation RESUME correctly.
Deferred refinements + Phase 2 prerequisite (latest_transition_event
helper) tracked in .drew/regressions-plan.md "Phase 1 follow-up
backlog".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
bc36fea757 | enhanced claude.md file for more efficient operation | ||
|
drew
|
d3cf710d29 |
fix(controller): tier-selection calibration — finalize timeout 30s→90s, estimator defaults to tier 1
Two coupled changes responding to the 2026-05-22 batch's tier-0 hit
rate of 0% on PRs the estimator judged "simple", plus three spurious
worker-internal-errors caused by the estimator's MCP-finalize wall
clock exceeding the previous 30s budget.
agent_runner.py — finalize timeout
----------------------------------
`production_agent_runner` previously hardcoded `finalize_timeout_s=30.0`.
On the 8-PR batch the worker recorded three `worker-internal-error`
attempts (PRs 47, 50, 54) all with the same message:
role='estimator' did not emit canonical output within 30.0s ...
The retries succeeded with estimator self-time of 41.5s, 25.9s, 21.5s —
all within plausible bounds for a multi-subsystem PR (PR 54 needed
41.5s of reasoning + MCP IPC + finalize write). Each spurious timeout
cost ~90s of worker wallclock + a re-dispatch.
New module-level constant `_DEFAULT_FINALIZE_TIMEOUT_S` reads
`CONTROLLER_FINALIZE_TIMEOUT_S` (default 90s), matching the
`CONTROLLER_*_TIMEOUT_S` convention used by `opencode_session.py`.
Operators can tune via `.devcontainer/.env.{fork,prod}`.
estimator-implementation.md — TIER MAP + CALIBRATION
----------------------------------------------------
The estimator was wrong on 4 of 8 PRs in the May-22 batch, all in
the same direction (recommended tier 0, escalated to tier 1).
Three of the four were high-confidence picks — including a PR
described as "4 string constant replacements, isolated scope, no
logic changes" that still got bounced at tier 0.
Three prompt changes:
- TIER MAP rewritten: tier 1 is now the default for non-trivial
work; tier 0 requires positive evidence the change is mechanical
(single file, ≤50 LOC, no new logic, no test changes).
- CONFIDENCE RULES "When uncertain, prefer" flipped tier 0 → tier 1.
- New CALIBRATION section bakes in the empirical observation (0/4
hit rate, why Haiku struggles in this codebase, cost calculus:
wrong tier-1 < wrong tier-0 by an order of magnitude when measured
per merged PR).
tiers.yaml — clarify scope
--------------------------
The previous "Used by estimator-implementation.md to make tier
choices model-agnostic" comment on the `capability` field misled
readers into thinking tiers.yaml drove classification. It doesn't —
the estimator agent reads its TIER MAP from its own prompt and
emits an integer; tiers.yaml only answers "for tier-N, which model
runs?". Updated header + field comments to flag this clearly so
future operators don't change descriptors here expecting the
estimator to honor them.
Tests
-----
+ test_default_finalize_timeout_reads_env: covers the env-var
resolution for `CONTROLLER_FINALIZE_TIMEOUT_S` (default, override,
float values). Restores the default at the end so subsequent
tests in the session see the stock constant.
Existing tier-model registry + 27 agent_runner tests pass unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
aa088f3c9e |
chore(env): add .env.example + .env.{fork,prod} as no-secret templates
Three env files for the dual-mode controller launcher, all containing placeholder values only: - .devcontainer/.env.example — full setup template. Operators copy this to .devcontainer/.env (gitignored) and fill in real HAL9000 / HAL9001 PATs + passwords, LLM provider keys, and the MODE selector. Documents every variable the auto-agents pipeline reads with the same incident / contract / RFC references as the live file. - .devcontainer/.env.fork — fork-mode overlay defaults (target = drew/cleveragents-core, trial-fast intervals, MERGE_DRIVER_LOG_LEVEL= DEBUG, /tmp run dirs). Loaded by the controller launcher when MODE=fork. - .devcontainer/.env.prod — prod-mode overlay defaults (target = cleveragents/cleveragents-core, slower production cadences, MERGE_DRIVER_LOG_LEVEL=INFO, persistent /var/lib/cleveragents paths). Loaded when MODE=prod or --prod is passed. No real credentials or tokens in any of these files. They stay committed only as long as that remains true — if a real secret value ever lands here, the file MUST be moved back behind the .gitignore rule on line 92. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
981ddd6a8e |
fix(controller): incident hardening — PR-44 dispute guard + PR-46 fixes
Bundles fixes for three production incidents (PR-44, PR-46 a/b/c). Every change has an incident-reference comment in the code and a named regression test. 199 tests pass on the impacted modules. PR-44 — fabricated dispute escape --------------------------------- A tier-0 implementer, bounced twice by red CI, emitted a fabricated ``dispute-reviewer`` outcome and shortcut a red PR into REVIEWING → APPROVED → MERGING, bypassing the CI gate. ``dispute-reviewer`` is the only IMPLEMENTING → REVIEWING edge that doesn't pass through AWAITING_CI, so it must be defended. tools/controller/master/outcomes.py: ``_map_implementer_outcome`` now guards ``dispute-reviewer`` with two preconditions — (1) ``attempt_saw_green_ci`` (the HEAD must already be CI-verified; a dispute can't jump the gate on a red/pending head), and (2) ``prior_reviews >= 1`` (must reference a review that actually happened, not a hallucinated one). Either guard fails → ``implementer_competence_failure`` → tier escalation. A weak model can't game its way past CI; a stronger tier is given the real problem. tools/controller/master/tick.py: new ``_count_prior_reviews`` helper counts completed reviewer attempts (epoch-scoped so an ``operator_unstick`` resets the count). Wired into the ``map_outcome_to_event`` call. PR-46(a) — stale gate-script preferred over in-repo --------------------------------------------------- ``gate.py`` was preferring the seeded ``/tmp/local_tools`` copy of ``local_ci_gate.sh`` over the version-matched in-repo copy. The seed predated the ``--envdir`` flag; the controller pipeline's invocations rejected as bad-argv every committing implementer's gate. The seed-refresher (``dispatch_implementer.py``) is on the retired dispatcher path, so the staleness was permanent. tools/controller/worker/gate.py: resolution order is now ``CONTROLLER_LOCAL_CI_GATE`` env > in-repo > seeded ``/tmp``. The seeded copy survives only as a last-resort fallback. Module-level constants ``_IN_REPO_GATE_SCRIPT`` / ``_SEEDED_GATE_SCRIPT`` let tests substitute paths. PR-46(b) — 6-second-old run flagged zombie ------------------------------------------ ``classify_ci_run`` instantly classified a CI run as ``stale`` when the Actions API reported no active task. A freshly-pushed run has no task simply because no runner has picked it up yet, and there are brief gaps between jobs — both false positives. A 6-second-old PR-46 run was bounced before CI could even start. tools/controller/master/ci_run_status.py: new ``ZOMBIE_GRACE`` (default 3 min, env: ``CONTROLLER_CI_ZOMBIE_GRACE_MIN``). "No active task" only classifies a run as stale once the run has ALSO gone quiet past the grace. Much shorter than ``STALE_AFTER`` since the active-task absence is corroborating evidence, not the sole signal. PR-46(c) — ruff-format-only violation slipping through lint ----------------------------------------------------------- CI's ``lint`` job runs both the ``lint`` nox session (ruff check) AND ``ruff format --check``. The pre-push local gate only ran the former; a formatting-only violation passed pre-push then failed CI. tools/local_ci_gate.sh: the ``lint`` gate now runs ``ruff check`` followed by ``ruff format --check``, unconditional. Either one failing marks the gate red. ``ruff format --check`` is whole-repo and takes no posargs. tests/auto_agents/test_local_ci_gate.py updated for the new two-call shape. Supporting changes ------------------ .forgejo/workflows/ci.yml: gates ``coverage`` and ``docker`` jobs on repo variables ``skip_coverage`` / ``skip_docker`` so the long reaper-prone jobs can be skipped per-fork without editing CI. Guard step always runs so the job still reports ``success`` and ``status-check`` stays green. tools/duplicate_prs_to_fork.py: bakes the same ``skip_coverage`` / ``skip_docker`` gates into every sentinel PR's ci.yml at PR-creation time so fork-mode runs inherit the gating. Idempotent. .opencode/opencode.json: adds ``timeout: 1860000`` (31 min) to the ``ci`` MCP server so long CI waits don't timeout the tool. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
eb01eb0172 |
feat(controller): dual-mode launcher (fork/prod) + DB-mode validator
Adds the operator surface for switching the controller pipeline between
the personal fork (drew/cleveragents-core) and the canonical repo
(cleveragents/cleveragents-core) via a MODE env + --prod CLI flag,
backed by safety primitives that make a wrong-mode launch loud rather
than silent.
run-controller-state-machine-pipeline.sh: --prod flag and MODE env
(primary home: .devcontainer/.env) select fork vs prod. After resolving
MODE, the launcher auto-sources the matching overlay file
(.devcontainer/.env.{fork,prod}) and asserts MODE didn't drift during
the source step. The drift assertion uses a readonly snapshot under an
obscure variable name so a stray ``MODE=fork`` in .env.prod aborts the
launch with a clear bash error rather than silently demoting the run.
CONTROLLER_RUN_DIR_ROOT now overrides the trial /tmp path so prod can
use a persistent /var/lib/cleveragents/run dir.
tools/launch_prod.sh (new): sibling to launch_fork.sh with the opposite
safety primitive — affirmative GET /repos/{owner}/{repo} that asserts
the target is non-fork, exists, isn't archived, and the bot has push.
On any failure, no env is exported. Honors HAL_* aliases for parity
with launch_fork.sh and prints a hard-to-miss PROD-MODE banner.
tools/controller/deploy/validate_db_mode.py (new): stamps a _mode_marker
table on each SQLite db (controller DB + telemetry cache) on first use,
asserts a match on every subsequent launch, and moves mismatched files
aside as <name>.<prior-mode>.bak.<ts> — never deletes. The --adopt flag
lets an operator grandfather in already-good pre-marker data without
losing history. Wired into the launcher's startup sequence before
OpenCode and the master start.
tools/_cache_path.py (new): single source of truth for the per-(owner,
repo) Forgejo cache file convention. .opencode/telemetry/server.py and
the launcher both delegate here so the dual-source-truth drift risk is
eliminated. tools/_pipeline_cache.py and tools/controller/db/models.py
documented as not owning the _mode_marker table so future migrations
leave it alone.
.opencode/telemetry/server.py: hosts the llm_activity scraper as a
background subprocess thread (60s cadence, --since-hours 1 in steady
state, full backfill on first tick). Re-homes the cost-telemetry data
path after the pr_state_warmer was retired by the controller migration
— without this the Cost tab freezes when the warmer's loop is gone.
Subprocess (not in-process) for isolation; failures swallowed.
opencode.json: local-claude provider's baseURL now reads
{env:LOCAL_PROXY_URL} instead of the literal http://127.0.0.1:3456/v1,
matching the apiKey pattern already in use.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
a103a31bbf |
feat(controller): worker-owned gated push for the implementer
The implementer agent no longer pushes to git — the controller worker now owns the push: it gates the agent's commits on lint+typecheck and pushes via a single leased primitive. Closes two production defects: - Clobber: the pre-fix MCP --force-with-lease leased against a freshly-fetched tip, so the lease always passed — an in-flight implementer destroyed a commit pushed to the PR branch during its run (lost a hand-pushed skip_coverage fix on PR #46). - Gate-skip: the agent verified only the CI-flagged gate, so a fix for one gate shipped fresh violations in another (lint flapped pass->fail across CI runs 198->199). Step 1 — worker_push primitive: - New git_push.py: one leased push, pinned to the SHA the worker started from; classifies pushed / stale_input / diverged / infra_error; bounded infra-retry. - mcp_git_server.push gains expected_sha for a correctly-pinned lease. - finalize_conflict_resolution migrated onto worker_push. Step 2 — deterministic gate: - New gate.py: per-slot nox env-dirs (no venv races), manifest-hash staleness keying, lazy warm-up. - local_ci_gate.sh gains --envdir. Step 3 — worker-owned gated push: - New implementer_finalize.py: divergence pre-check -> lint+typecheck gate -> ruff auto-fix -> leased push. finalize's outcome is authoritative over the agent's emitted outcome. - agent_runner integrates finalize; salvage no longer pushes. - outcomes/tick: gate-failed + push-time stale-input caps, epoch-scoped; WorkerError carries an output_payload so the gate report reaches the next attempt's prompt; prefetch surfaces gate-failed attempts. - The 5 task-implementor prompts drop the agent push step. Reviewed across 4 adversarial rounds; full controller+MCP suite green (1379 passed). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
6178be3aa7 |
feat(models): single-source model registry via models.yaml + sync_models.py
.opencode/models/models.yaml is now the ONE file humans edit to assign a model to an agent. tools/sync_models.py regenerates every derived surface — the .opencode/models/*.txt files, opencode.json's `agent` block, and each non-tier agent's .md `model:` frontmatter — so an assignment cannot drift across surfaces. `--check` verifies with no writes and is enforced in CI by test_model_registry_in_sync_with_manifest. Hardened after adversarial review: - Deletes orphan <agent>.txt files left behind when an override is dropped from the manifest. The dispatcher's resolver reads <agent>.txt before default.txt, so a stale file would silently pin the old model. Tier .txt files are left to sync_tier_models.py. - Rejects a manifest key that does not name a real agent (no matching .opencode/agents/<name>.md) instead of silently appending a bogus opencode.json entry and leaving the real agent on the default model. - Validates the regenerated opencode.json BEFORE writing it, so a bad render aborts cleanly instead of corrupting the file on disk. Tier-ladder agents (task-implementor-tier-*) remain governed separately by tiers.yaml + sync_tier_models.py and are passed through untouched. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
d9919e783b |
feat(merge-drive): post the reviewer's approval to Forgejo before merge
The controller's reviewer stage records verdict=approve only in the
controller DB — it never posts a Forgejo review. Branch protection's
required-approvals gate then 405s the merge ('not allowed to merge: Does
not have enough approvals') for a controller-managed single PR, because
nothing reflects the controller's approval to Forgejo (PR #45). The merge
driver only auto-approved umbrella PRs for multi-PR trains; single PRs
got nothing.
merge_train now posts a real Forgejo APPROVED review for a controller-
managed single PR (len(prs) == 1 with a _controller_workflow_id) right
before the merge endpoint, via _post_approval_review:
- posted as the reviewer identity (HAL9001 via cfg.reviewer_pat) — a
non-author approval, which is what required-approvals demands;
- pinned to the exact rebased SHA about to merge (commit_id);
- idempotent — _pr_has_approval skips the post if an APPROVED review
already exists, so a merge_train recursion does not spam duplicates;
- best-effort + logged — a failed post does not abort the merge; the
WARNING is the operator signal.
The umbrella-PR auto-approval is refactored onto the same helper.
12 new tests in test_merge_drive.py; full merge_drive suites green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
7fdac7ded0 |
fix(merge-drive): correct CI-wait — newest-per-context statuses + zombie-CI detection
wait_for_ci could never see CI finish on a green commit. fetch_commit_statuses
flattened Forgejo's newest-first /commits/{sha}/statuses *history* with
last-write-wins, so each context kept its OLDEST object — the initial
`pending` posted when the job was queued — and a fully-green SHA read as
all-pending. PR #45 hung against a green commit until the 1 h ci_timeout_s.
fetch_commit_statuses now keeps the newest status per context by created_at.
A required check that never decides — a job a misconfigured branch protection
requires but no workflow emits, or a gate left pending by a crashed runner —
also pinned the wait to the full ci_timeout_s. wait_for_ci now consults
Forgejo's Actions API via ci_run_is_live: once the run is confirmed dead for
_ZOMBIE_CONFIRM_POLLS consecutive polls while a required check is still
undecided, it ends with the verdict the finished gates produced
(_zombie_verdict — passing / failing / timeout). Mirrors the controller's
zombie-CI detection in ci_run_status.py.
37 new/updated tests in test_merge_drive.py; full merge_drive suites green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
bea16f7130 |
fix(controller): scope STUCK-gating counters to the operator_unstick epoch
operator_unstick is the deliberate operator escape hatch that requeues a STUCK/APPROVED/PAUSED workflow back to DISCOVERED. It reset the state but not the per-workflow failure tallies, so a requeued workflow inherited every historical worker-error / ci-not-ready / ci-infra-failure / dispute / contract-violation / resolved-conflict count and re-tripped a backstop cap almost immediately (observed: a PR re-STUCK after a single fresh ci-not-ready because 5 stale ones from prior runs were still counted). All seven prior-outcome counters now accept an epoch_start and, when the workflow has been operator-requeued, count only attempts created after the most recent operator_unstick event — so a requeue genuinely starts the backstop budgets fresh. The automatic DISCOVERED loops (ci_infra_recheck, ci_red_retry) are not operator_unstick events, so their caps still bound those loops. operator_unstick is matched under both conventions seen in the wild (bare event_type, or a transition whose payload names it); the comparison is wrapped in datetime() to be immune to timestamp-format drift. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
14f79ee61b |
fix(controller): count only resolved conflict cycles toward the STUCK budget
The v6 "3+ conflicts -> STUCK" guard measured every conflict_resolver attempt at the tier, including failed/blocked/errored ones. A burst of transient worker errors (e.g. the run-9 marker false-positive) inflated the count so that a workflow whose conflict was genuinely resolved on the first real pass was routed to STUCK instead of IMPLEMENTING. _count_conflict_resolver_attempts now counts only complete+resolved attempts -- each resolved attempt is exactly one conflict cycle, since the workflow leaves CONFLICT_RESOLVING on success and only re-enters when a new conflict appears. Failed attempts are retries of the same unresolved conflict and stay bounded by the pickup guard. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
d4e1219577 |
fix(controller): scope conflict-marker check to resolved files
finalize_conflict_resolution grepped every file in the PR's origin/<base>..HEAD diff for committed conflict markers. On a large sentinel PR (478-file diff) that includes .opencode/agents/ git-rebase-util.md — an agent def that *documents* conflict markers with literal "<<<<<<< HEAD" lines — so the check false-positived and rejected every resolution, looping the conflict-resolver indefinitely (run-9: 185 attempts in ~4h, 125 on this exact error). Scope the grep to the files the resolution actually touched (resolved_files = prep-time conflicted set + the agent's reported files_modified). A leftover marker can only be in a file the agent resolved; a clean rebase/merge resolves nothing so the check is skipped. Adds a regression test with a marker-documenting file in the PR diff that is not a resolved file. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
615a05b982 |
feat(controller): rebase-default conflict resolution with merge fallback
PR branches 177-180 commits ahead of base cannot be rebased commit-by-commit by a single-shot resolver agent (too many conflict stops for one session). Conflict-prep now defaults to rebase (linear history) and falls back to a single 3-way merge when the branch is too divergent (commit count over CONTROLLER_CONFLICT_REBASE_MAX_COMMITS, default 60). The merge pipeline derives the track from branch shape via a Do:rebase -> Do:merge ladder in _make_merge_pr — no stored flag. Adds a git_rebase_continue MCP tool plus status rebase/merge-in-progress fields so the conflict-resolver agent is fully MCP-driven and dual-mode (mid-rebase or mid-merge). Also routes a green-CI implementer noop straight to REVIEWING instead of a deadlock-prone AWAITING_CI round trip. No state-machine change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
8c9a0c50bb |
fix(controller): green-CI noop routes forward instead of ABANDONED
An implementer dispatched against a fully-green CI that emits `noop`
("nothing to fix") was mapped to competence-failure -> ESCALATING ->
ABANDONED at MAX_TIER. PR-39 and PR-40 dead-ended exactly this way:
12/12 CI gates green, workflow ABANDONED, solely because the
implementer said `noop` instead of the synonymous `verified-clean`.
The tick now reads the attempt's input_payload ci_summary; a `noop`
whose attempt saw an unambiguously green CI (overall success, zero
failed, zero pending, >=1 passed) routes via `implementer_verified`
-> AWAITING_CI (-> ci_green -> REVIEWING) -- the same forward path
`verified-clean` already takes. A non-green `noop` (red / pending /
unknown / no CI) keeps the competence-failure -> escalate behavior.
- outcomes.py: `attempt_saw_green_ci` param gates the noop branch.
- tick.py: `_input_ci_summary` + `_ci_summary_is_green` helpers.
- tests: green->AWAITING_CI, red/none->ESCALATING, helper unit tests.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
drew
|
14e592ddd5 |
feat(controller): zombie-CI detection — stop waiting on dead CI runs
A CI gate stuck `pending` is ambiguous: the job may genuinely be running, or the run may be dead (a crashed runner, an Actions job whose terminal commit-status was never posted — `CI / status-check` zombies routinely here). The "wait for the whole run to finish" fix then waited forever on the dead case (PR #36: a `status-check` gate pending for 8 h while the run had actually finished RED 8 h earlier). New `ci_run_status.classify_ci_run` resolves a still-pending run to `complete` / `running` / `stale` via two checks, authoritative-first: 1. ACTIVE-RUN — `get_action_tasks` asks Forgejo's Actions API directly whether a task for the commit is still running; catches a dead run immediately, regardless of age. 2. AGE — if no gate has updated in > CONTROLLER_CI_STALE_AFTER_MIN (default 90) the run has stopped; the fallback when the Actions API is unavailable. A `stale` run is no longer waited on: the verdict is taken from the gates that DID finish (`terminal_verdict`) — any failure → red, all pass → green, fully-dead → red. Applied in both `ci_status_poll` (the AWAITING_CI verdict) and `ci_summarize` (the implementer's summary — zombie pending gates drop out of `gates_pending`/`overall_state`) so the two agree and never ping-pong. New `get_action_tasks` Forgejo callback wired through forgejo_http → __main__ → the poll and the prefetch path. 23 new tests; full controller suite (1222) green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
2fef8cb9f9 |
fix(controller): strip Actions log timestamps before CI parsing
Every implementer was handed `findings=0` for its red gates even when
the parser's own summary line counted real failures ("Pyright: 10
errors", "Behave: 2 scenarios failed"). Root cause: GitHub/Forgejo
Actions prefixes EVERY log line with an ISO-8601 timestamp
("2026-05-20T18:04:41.1555784Z "). Every parser extracts findings with
^-anchored regexes against the raw tool output (pyright
`^file:line:col - error:`, behave `^\s+...feature:N Scenario:`); the
timestamp prefix pushes that content off the line start so the anchors
never match — while the UN-anchored summary-count regex still matches,
producing the misleading "summary says N, findings=[]" state.
Fix: new `strip_log_timestamps()` in ci_summary_parsers/_base.py,
applied once centrally in `ci_summarize._safe_fetch` — the single
chokepoint feeding every parser and the no-parser raw-log excerpt.
Lines without a timestamp prefix are left untouched, so it is safe
unconditionally; it also trims ~28 chars/line off the implementer's
raw-log excerpt.
5 new tests; full controller suite (1200) green.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
drew
|
5f0fff0db2 |
fix(controller): wait for the whole CI run to finish before acting on it
Two places acted on a CI run that was still executing — the PR-39/PR-40 "doesn't wait for CI to finish" incident: 1. ci_status_poll — keyed the AWAITING_CI verdict purely off Forgejo's combined `state`. That combined state flips to `failure` the instant ONE gate fails, even with other gates still running (and can read `success` before a late gate reports). So a run at 9-passed / 1-failed / 2-pending fired `ci_red` and yanked the workflow out of AWAITING_CI mid-run. New `_ci_run_incomplete()` scans the per-gate `statuses`; while any gate is pending/running the poll waits, regardless of the combined state. Only a fully-terminal run yields a verdict. 2. implementer prompt routing — `_ci_summary_is_pending` required the run to be failure-free to count as "still pending," so a run with an early failure + gates still executing fell through to the ci-infra-failure block and the implementer acted on a partial result. Dropped the no-failures clause: any pending gate → the ci-not-ready (wait) block, which now also covers partial results. Zombie/stale runs (a gate pending for hours) are left to the controller's CI-freshness gate to re-trigger — a separate follow-up; ci_poll_exhaustion remains the backstop. 4 new tests; full controller suite (1195) green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
22b76b2834 |
fix(worker): correct misleading "model override" log line
The worker's _resolve_role_model resolves a model from .opencode/models/<agent>.txt (or the default.txt fallback) and passes it on POST /session — but OpenCode does NOT consume that field for generation; it generates from its startup-cached opencode.json agent.<name>.model (the footgun documented in .opencode/models/README.md). The old line "model override for agent=X -> Y" read as if Y were the model in effect. For a tier agent with no per-agent .txt file it logged "-> default.txt(haiku)", which falsely looked like every implementer tier was downgraded to haiku — when opencode.json's task-implementor-tier-2 block correctly pins opus (verified: run-3 tier-2 session archive records claude-opus-4-6). Reworded to "POST /session model hint=Y ... observability only; OpenCode generates from opencode.json's agent.X.model". Also corrected the module docstring's stale "takes effect ... no restart required" claim. Log + docstring only — no behaviour change. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
eb1d81828f |
feat(controller): implementer_retrigger_ci — agent re-triggers CI on ci-infra-failure
A CI job hard-killed by OOM / pod eviction produces no verdict — nothing in the diff to fix. The implementer correctly emits outcome=ci-infra-failure, but that alone only shuffles workflow state; CI never re-runs, so every retry reads the same dead run and the workflow loops until the _MAX_CI_INFRA_FAILURE backstop STUCKs it (observed live on PR #39 in run-3). New ``implementer_retrigger_ci(owner, repo, pr_branch)`` MCP tool on the implementer response-builder lets the agent kick a fresh CI run. Forgejo 15.x has no Actions rerun API, so it reuses the controller's existing mechanism — ``ci_rerun.trigger_ci_rerun_via_empty_commit`` — an empty commit that advances the PR head SHA, which is the unambiguous fresh-state signal for CI (and a stale review). - ci_rerun.py is loaded standalone (importlib by file path, registered in sys.modules before exec so its @dataclass resolves) — the response-builder MCP must not pull the heavy tools.controller.master package. - _retrigger_ci resolves the Forgejo base URL + token from env (FORGEJO_URL / FORGEJO_API_BASE, FORGEJO_TOKEN / GITEA_TOKEN) and never raises. - Once per session: a second implementer_retrigger_ci call is refused benignly so a looping agent cannot pile junk commits on the PR. - The implementer prompt's ci-infra-failure block now instructs the agent to call the tool before emitting the outcome. 10 new tests (test_mcp_builders.py, test_worker_prompts.py); full controller suite (1192) green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
9d20f865d3 |
feat(controller): pre-review mergeable gate — skip doomed reviews
When CI goes green, route to REVIEWING only if the PR still merges cleanly into base. If a cheap Forgejo mergeable check shows the base advanced while CI ran, route AWAITING_CI → CONFLICT_RESOLVING instead — handing the conflict to the LLM conflict_resolver BEFORE the expensive reviewer pass, since code that must be rebased gets re-CI'd and re-reviewed afterwards anyway. - new event `pre_review_base_conflict` + transition (AWAITING_CI → CONFLICT_RESOLVING) - ci_status_poll: `_decide_green_event` gate behind a new optional `get_pr_details` callback; conservative — only an explicit mergeable=false diverts, an unknown/uncomputed bit falls through to ci_green so a fresh PR is never false-routed - gate-fired event rows carry `mergeable` in the payload so the false-positive rate is observable from controller_events - wired through loop.py (5th ci_status_poll_args element) + __main__ One API call, no LLM — cheap+frequent detection gating the expensive+rare conflict_resolver/CI/reviewer stages. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
8fe98cb5c0 |
feat(controller): salvage timed-out implementer + cap unbounded prefetch-retry loop
Two reliability fixes from the proactive audit of untouched pipeline modules (AUDIT-1, AUDIT-2 in .drew/PENDING_FIXES.md). AUDIT-1 — a timed-out implementer's committed work was never salvaged. opencode_session maps an OpenCode timeout / transport-error to WorkerError, which propagated out of production_agent_runner BEFORE the canonical-output wait + _salvage_implementer_commits ran. A timeout is the case most likely to have a complete committed fix (agent ran out of wallclock, not correctness). The session-exception handler now runs the same salvage the canonical-missing path uses before discarding the attempt as worker-internal-error. AUDIT-2 — a persistently failing prefetch() retried every master tick forever (the T4-4 guard covered only the issue-kind sub-case; same unbounded-retry class as the run-2 174x estimator loop). The scheduler now journals each prefetch failure as a controller_events row, counts failures since the workflow's last transition, and routes the workflow to STUCK after _PREFETCH_FAILURE_LIMIT (10) failures. The STUCK transition is rowcount-guarded so the journal never records a transition the current_state guard rejected. 7 new tests; full controller suite (1169) green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
58307bbdab |
refactor(ci-logs): retire the dead legacy {sha}.json cache layer
Follow-up to the B1 unification: with `fetch_pr_failure_logs` now a
view over the `get_ci_logs` bundle, the legacy `{sha}.json` cache had
no readers left. Remove it wholesale rather than leave it orphaned.
- `_ci_logs.py`: delete `_cache_covers_all_current_failures`,
`_record_failure`, `_read_cache`, `_write_cache`, `cache_path` — all
zero-caller after B1. `invalidate` re-pointed onto the bundle cache
(`bundle_cache_path`) so it stays a working API. Module docstring
rewritten to describe the bundle-as-single-store reality.
- `local_ci.py`: `_write_ci_logs_cache` no longer writes the legacy
`{sha}.json` — `put_local_bundle` already populates the bundle that
`fetch_pr_failure_logs` projects, so the MCP tool still sees local
CI logs. One write path, not two.
- Tests re-pointed onto `bundle_cache_path`; `ruff format` applied.
No behavior change — only dead code removed and the docstring
brought current.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
drew
|
f76568a871 |
feat(controller): ci-infra-failure implementer outcome → bounded rerun
The implementer-side counterpart to the `indeterminate` verdict. When the implementer IS dispatched onto a CI failure and finds the log carries no verdict (a hard-kill / OOM — nothing in the diff to fix), it can now emit `outcome=ci-infra-failure` instead of being forced to `blocked` → STUCK. `ci-infra-failure` forbids commits/files/blockers (no-work invariant, like `noop`) and fires `implementer_ci_infra_failure`, routing IMPLEMENTING → DISCOVERED so the CI-freshness gate reruns CI under its bounded RERUN_BUDGET. Backstopped by `_MAX_CI_INFRA_FAILURE=4` so a mis-classification cannot loop the gate forever. Wired through: the V1 contract enum, the implementer MCP builder (outcome value + no-work invariant), the state machine (event + IMPLEMENTING→DISCOVERED transition), the outcome mapper (+ per-workflow cap), tick's `_count_prior_ci_infra_failure`, and the implementer prompt — which now surfaces the outcome whenever the CI summary shows a failing gate, with guidance to use it ONLY when the log genuinely shows no verdict. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
6ab6df319c |
feat(controller): indeterminate-CI verdict for no-verdict (OOM) failures
Diagnosed live on PR 39 (STUCK 2026-05-20): both CI gates ran 17 minutes, the captured log was <3 minutes and ended mid-execution (`still running` / a just-launched runner) with ZERO error markers. A hard process kill (OOM-killer / pod eviction) cannot flush a buffer, print a traceback or emit an exit code — it leaves a *hole*, not a phrase. The classifier called this `fresh_real`, the implementer was sent to "fix" a failure with nothing to fix, and the PR dead-ended at blocked → STUCK. New `indeterminate` verdict: a failing gate whose FULL log carries no terminal verdict marker (`##[error]`, test summary, Traceback, exit code) AND ends mid-execution. Both conditions required — "no marker" alone over-fires on a real failure whose tool output isn't in the marker set (e.g. `ruff format`). Routed to a bounded rerun, same as `infra_broken`/`stale`, via `ci_status_poll` and `ci_gate`. Also fixes an infra-signature regression exposed by the full-log change: the bare step-name signatures (`git fetch`, `Set up job`, `actions/checkout`) matched the checkout/setup preamble of EVERY job log — against a full untruncated log they turned every failing run into `infra_broken`. Replaced with genuine error-text signatures (`could not read from remote`, `download action repository failed`, `unable to access`, `failed to connect`). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
6cd78bffda |
refactor(ci-logs): unify CI-log cache — one fetch path, run-aware freeze
Three cache-layer fixes so every consumer reads CI logs through one
store (`get_ci_logs`), and the cache cannot serve a stale verdict.
B1 — `fetch_pr_failure_logs` is now a thin VIEW over `get_ci_logs`.
It was a parallel fetch path with its own `{sha}.json` cache, own
backoff and 4000-char tail truncation — a second cache that could
drift from the bundle. It now projects the unified `{sha}.full.json`
bundle to the legacy failing-jobs-with-tails envelope. Retained for
the `ci_fetch_pr_failure_logs` MCP tool + legacy prefetch callers.
B2 — a partial (in-flight) bundle re-fetches on a minimum interval
(`CI_LOGS_PARTIAL_MIN_REFETCH_S`, default 60s) instead of on every
call, bounding polling load on the Forgejo UI endpoints.
B3 — a terminal+clean bundle is frozen only while its `run_id`
matches the live run. The cache is keyed by head_sha, but a SHA can
be re-run; a cached bundle whose run_id no longer matches is treated
as stale and re-fetched (last-run-wins). Resolving the live run
costs one cheap ci-detail call; the per-job log fetches are still
skipped on a hit. A `source="local"` RUN_CI_LOCAL bundle keeps its
no-fetch fast path.
`mcp_ci_server`'s `source` probe re-pointed onto the bundle cache.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
drew
|
c4a11901f5 |
feat(ci): unified CI-log cache — phase 2 (consumer migration + RUN_CI_LOCAL)
Builds on
|
||
|
drew
|
27289ea4b7 |
feat(ci): unified CI-log cache — get_ci_logs + implementer path (phase 1)
The implementer's ci_summary carried an empty raw_log_excerpt for every failed gate (diagnosed via PR 39): prefetch._build_ci_summary passed a no-op log fetcher, and ci_summarize._gate_to_nox_session never stripped Forgejo's "(pull_request)" event suffix, so every PR gate fell through to NoParserAvailable with no log fetched at all. Phase 1 — the implementer-facing path: - _ci_logs.get_ci_logs(): unified entry point — every job of a run, full untruncated logs, one cache. `partial` marks an in-flight run; a terminal + clean bundle is frozen forever. Reuses the existing session-cookie login + exponential backoff. Additive — fetch_pr_failure_logs and its 9 consumers are untouched. - ci_summarize: strip the "(pull_request)" event suffix so gates resolve to their nox parser; _no_parser_failure now carries the raw log instead of hardcoding "". - prefetch._build_ci_summary + forgejo_http + __main__: wire a real get_ci_logs-backed log_fetcher so each failed gate's raw_log_excerpt is filled from the cache. - 9 new tests; full controller suite green (1169 passed). Deferred to later phases: re-point the freshness gate / ci_status_poll onto get_ci_logs, RUN_CI_LOCAL into the same cache, and (head_sha, run_id, attempt) multi-attempt keying. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
521117882e |
fix(controller): conflict resolver lands resolutions on the PR
Run-2 PR #40 exposed that the conflict-resolver stage never resolved conflicts *on the PR*. The resolver agent owned the whole git flow (rebase + resolve + push) but was never given the PR head branch name, so it pushed the resolved branch to a guessed branch ("main"), the push was lost, and it reported "resolved" anyway — the controller advanced the workflow on a resolution that never reached the PR. This splits the work along the controller's principle: deterministic git mechanics in the worker, only the semantic merge in the agent. - conflict_rebase.py (new): the worker starts the rebase (prepare_conflict_rebase — hands the agent a real mid-rebase worktree) and, after the agent's resolution, lands it with a *verified* push (finalize_conflict_resolution) — force-with-lease pinned to the PR head the resolver started from, then re-reads the remote to confirm it moved. A failed/rejected push is never reported as resolved. run_conflict_resolver_attempt orchestrates it. - ConflictResolverInputV1 / prefetch: carry head_ref so the worker knows the PR branch (this also makes the worker's fetch_and_validate run for the conflict_resolver — it gated on head_sha AND head_ref). - worker/__main__.py: route conflict_resolver attempts through the new orchestration. - conflict-resolver-worker.md: the agent resolves + continues the rebase and never pushes — the controller lands it. - mcp_git_server.py: add the controller worktree root to the git MCP's allowed bases (every git_* MCP call errored for this role before). The committed-marker check is scoped to the PR's own diff — a whole-tree grep false-positived on repo files that legitimately document conflict markers. The worker's rebase/push are lost-lock guarded so a reaped worker cannot race the shared worktree. 1136 controller tests pass (+11 new in test_conflict_rebase.py). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
3106230ef8 |
fix(controller): cap estimator worker-internal-error retries
An estimator whose session ends without emitting canonical output fails with worker-internal-error. Unlike the implementer, the estimator has no escalation path (it runs pre-tier) and no salvage (it produces no git artifact) — so a flaky estimator session just re-enqueues, with nothing to stop it. Run-2 burned 174 consecutive estimator worker-internal-error attempts on one PR. After _ESTIMATOR_WORKER_ERROR_LIMIT (3) such failures the workflow now STUCKs for operator attention via estimator_failed_twice — symmetric with the implementer worker-error escalation cap. - outcomes.py: the cap + the prior_estimator_worker_errors param. - tick.py: _count_prior_estimator_worker_errors (per-workflow count; the estimator runs pre-tier, so tier is not a meaningful axis). - state_machine.py: estimator_failed_twice description corrected — the event had no emitter before this; strict-parse failures route via contract-violation -> pickup_exhausted, not this event. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
4b184b5a29 |
fix(merge): bypass stale-label veto for controller-managed PRs
merge_drive's pr_is_eligible vetoed any PR carrying auto/needs-* / auto/stale-inactivity labels as "labelled-needs-attention". The T5-7 controller-managed bypass only covered the auto/ready-to-merge gate, so a controller-APPROVED PR still carrying a stale auto/* reflection label was silently dropped from every merge cycle — run-25/26 PR #40 sat APPROVED-but-unmergeable for hours behind "picked 0 candidates" with no diagnostic. For a controller-managed PR the controller DB is the source of truth, so the auto/needs-* veto no longer applies to it; auto/unstable still does (operator override). If the PR genuinely doesn't merge, merge_drive discovers that during the merge attempt and routes it MERGING -> CONFLICT_RESOLVING via merge_base_conflict. pick_candidates now also logs each skip reason, so a stale label can never silently produce "picked 0 candidates" again. Validated live: #40 went from vetoed-forever to picked up -> MERGING -> CONFLICT_RESOLVING. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
f92822affd |
fix(sentinel): bake the CI OOM-prevention fix into duplicated test PRs
duplicate_prs_to_fork.py reset the fork's master to upstream and pushed raw upstream PR heads — neither carries the noxfile/ci.yml parallelism cap, so every sentinel PR's CI risked the 64-worker OOM-kill the pipeline itself hit. The fix is now committed onto the fork's master and merged into each sentinel branch, so it is in effect for the sentinel's CI run while staying invisible in the sentinel PR diff (it sits in the common ancestor of base and head). A merge conflict — a PR that itself edits the capped lines — aborts cleanly and pushes the raw head. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
a6ff7da63c |
fix(controller): recover work lost to plumbing + run-25 reliability fixes
Run-25 surfaced five ways the controller lost or mis-routed correct implementer work. Each is fixed: 1. Canonical-output salvage (agent_runner.py): when an implementer's OpenCode session ends cleanly but emits no V1 output, its commits are no longer discarded — they are pushed and a synthetic outcome=resolved routes the workflow to AWAITING_CI so CI judges the code. Guarded by lost_lock_check so a reaped worker cannot race a fresh one on the shared worktree. 2. Preserve committed-but-unpushed work (workspace.py): the pre-reset snapshot (auto-scratch/pr-<N>) now captures committed commits ahead of the reset target, not just dirty edits — a finished-but-unpushed fix survives for the next attempt to adopt. 3. blocked escalates (outcomes.py): an implementer 'blocked' below MAX_TIER now escalates to a stronger tier instead of dead-ending at STUCK; only 'blocked' AT MAX_TIER STUCKs. 4. Diff-size tier floor (tick.py): the estimator's structured recommended_tier can contradict its own reasoning (a 387-file PR emitted as tier 0). current_tier is now floored deterministically by diff size so a huge PR cannot run on the weakest model. 5. CI log excerpt (prompts.py): the per-gate log excerpt is now tail-clipped, not head-clipped — the failure tracebacks live at the bottom and were being cut off, leaving the implementer blind. 1121 controller tests pass (+34 new). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
c34a0d1584 |
feat(telemetry): per-stage (role) cost breakdown in the Cost tab
Adds a "stage / role" Group-by option alongside model and PR. Token spend is bucketed by pipeline stage (implementer / reviewer / estimator / conflict_resolver), derived from the session tag rather than the per-row agent — subagent turns inherit the wrapper session's tag, so their tokens roll into the stage that spawned them instead of scattering across sub-agent names. Legacy AUTO-* tags fold into the same buckets as controller tags; probe/overhead sessions sink to "other". - server.py: _stage_from_tag helper + a group_by="role" branch in _api_cost that SUMs per (session_tag, model, provider) so per-model pricing stays exact, then collapses to one row per stage. - app.js/index.html: new dropdown option; renderCost generalized so the pr and role aggregate views share layout. - 11 new tests, incl. a regression pin that a subagent turn rolls into its wrapper stage. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
3c4d4b3362 |
fix(ci): cap test parallelism to prevent OOM-kills + extend CI-wait timeout
The behave/pabot test sessions derived worker count from the host CPU count; on a 64-core runner that spawned ~64 workers per job, oversubscribed RAM, and the run was OOM-killed mid-suite (SIGKILL / exit 137) — a red job with no test summary. - noxfile.py: cap auto-derived parallelism at _MAX_DEFAULT_PROCESSES=8; TEST_PROCESSES still overrides for hosts that can take more. - .forgejo/workflows/ci.yml: pin TEST_PROCESSES=8 on the unit_tests and integration jobs. - run-controller-state-machine-pipeline.sh: bump CONTROLLER_AWAITING_CI_TIMEOUT_S 2400s -> 4500s — capped (8-worker) CI runs longer, so the old 40-min ci_poll_exhaustion timeout risked STUCKing workflows whose CI was still legitimately running. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
0db0a15dad |
feat(controller): RUN_CI_LOCAL verdict source, ci-not-ready outcome, escalation hardening
Adds RUN_CI_LOCAL — an on-demand local-CI verdict source for when the cluster's Forgejo CI is broken — plus robustness fixes, the telemetry Live-tab rewrite, and PR-level cost attribution. Controller: - RUN_CI_LOCAL: the master swaps its Forgejo CI callbacks for local `forgejo-runner exec` runs (tools/run-ci-full-local.sh + local_ci.py). Async per-(owner,repo,SHA) on-disk job cache; preflights the forgejo-runner binary + Docker daemon at startup (fail loud, not a red verdict on every PR); GCs finished run dirs + per-run actcache. - ci-not-ready implementer outcome + implementer_ci_not_ready event: an implementer that runs before the on-demand verdict exists parks in AWAITING_CI instead of dead-ending at STUCK; capped against ci_red ping-pong. - ci_poll_exhaustion skips its sweep while a local CI run is in flight, so AWAITING_CI workflows queued behind on-demand CI are not STUCK'd by the remote-CI-sized timeout. - Escalate the workflow after repeated worker-internal-error at a tier instead of retrying to pickup-exhaustion -> STUCK. - forgejo_http: normalise Forgejo's per-gate `status` key to `state` so failing gates are actually counted (they previously all read as pending). - Per-tier worker timeouts bumped +15 min; a timed-out attempt's dirty-worktree residue is preserved on auto-scratch/pr-<N> before the next attempt's reset. - Implementer agents now verify only the CI-flagged gate(s) via a targeted re-run rather than the full local battery before claiming resolved/noop. Re-running the whole suite CI will run anyway was the #1 cause of implementer timeouts; CI remains the real gate and re-dispatches the implementer on red. Telemetry: - Live tab rebuilt on /api/live (controller DB run state + the live OpenCode session forest) after the live_log_writer sidecar was retired with the legacy dispatchers. - Durable per-attempt input/output payloads surfaced in the Live drill-down, archived-session detail, and Workflows timeline. - PR-level cost attribution: worker session tags carry -pr-<n>; backfill_llm_activity_pr.py repairs rows written before the fix. Shared: - tools/controller/session_tag.py — one canonical controller-tag parser shared by the telemetry server and the backfill. Tests: new coverage for local_ci (state machine, log parsing, _summarize_run, GC, in-flight probe, preflight), the ci-not-ready path, the escalation/ci-not-ready SQL counters, the ci_poll in-flight skip, and CI-status payload parsing across both sources. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
0bc734c020 |
style: ruff format the controller-state-machine branch (288 files)
Applies `ruff format` to the accumulated formatting debt on this branch. Formatting-only — no behavioral changes. Required for CI/lint's format gate (`nox -s format -- --check`), which the branch was failing on 288 tracked files that drifted from ruff's canonical style. In-progress WIP files are intentionally excluded so this commit stays a clean formatting-only diff. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
a2fcecb28d |
feat(controller): CI-freshness gate — re-trigger stale/infra CI instead of dead-ending
A discovered PR with stale CI (every job failed at the checkout step —
a git-fetch connection reset, pure infra, no code ran) burned an
estimator + a tier-2 implementer and dead-ended at STUCK. The pipeline
had no notion of CI freshness and never triggered CI — only polled.
New early master tick (ci_gate) runs before DISCOVERED->ANALYZING
promotion. For each DISCOVERED pr-kind workflow it classifies the CI
via ci_freshness.classify_ci_result:
- infra_broken — failed; the failing jobs' LOG content carries a
checkout/setup signature (curl 56, expected 'packfile', ...).
Logs are fetched via _ci_logs (session-cookie auth).
- stale — failed; newest status older than CONTROLLER_CI_MAX_AGE_S
(default 6h). Timestamp-based, log-independent — catches an old
failure even when Forgejo has purged its logs.
- no_ci / pending / fresh_real — handled accordingly.
infra_broken/stale/no_ci -> push an empty commit to the PR branch
(Forgejo 15.0.2 has no rerun API), routing DISCOVERED -> AWAITING_CI
(new event discovery_ci_rerun_triggered). The existing AWAITING_CI
poller then gets a real verdict. A reran CI that is ALSO infra/stale
routes AWAITING_CI -> DISCOVERED (new event ci_infra_recheck) so the
gate re-handles it; bounded by a rerun budget of 3, then STUCK.
Also wires the existing CI summarizer into prefetch so workers stop
receiving ci_summary=null.
1095 controller tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
drew
|
e1bcdd3d21 | Small telemetry cost UI tweak to center content in cell | ||
|
drew
|
f454670a0e |
chore(controller): launch merge_drive from the pipeline script; rename worker-concurrency env var
The launch script now starts merge_drive by default (T5-7: it is the controller's singleton merge stage — APPROVED -> MERGING -> MERGED). Previously the script killed merge_drive as a "legacy" process but never started it, so controller workflows dead-ended at APPROVED. merge_drive moves into the CONTROLLER process group; --no-merge skips it. Also adds the worker thread-pool concurrency knob (default 2 for the trial harness). Renames the misleadingly-named env var CONTROLLER_MAX_CONCURRENT_WORKERS_PER_MACHINE -> CONTROLLER_MAX_CONCURRENT_WORKER_THREADS_PER_MACHINE: it sizes a ThreadPoolExecutor inside one worker process, it does not spawn worker processes. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
1058341dd1 |
feat(telemetry): per-1M-token rate annotations + friendly uninitialized-DB note
The cost table now shows the resolved in/out/cached per-1M-token rates in parens next to each priced model, so an operator can see how a row's cost was derived. And controller-DB query failures distinguish "DB reachable but no schema yet" (the normal pre-first-run state, before the master runs create_all) from a real query error, instead of surfacing a raw SQL "no such table" dump. |
||
|
drew
|
012dabcebe |
chore(controller): track the controller-pipeline launch script in tools/
Moved from .drew/ (gitignored, operator-personal) so the launch script is versioned. REPO_ROOT now derives from the script's own location instead of a hardcoded absolute path, and the DB-URL comment documents .devcontainer/.env as the standard place to pin a stable, persistent CLEVERAGENTS_DB_URL (the per-run sqlite path is now just the fallback for an isolated one-off run). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
4e0b1bbdbf |
fix(controller): second review pass — close remaining MERGING stranding paths
The first post-commit fix only handled merge-error-* strings and missed that build_train's single-PR failure reasons flow straight into terminal_state via _release. Four were unmapped (no-head-ref, rev-parse-fetched-head-failed, rebase-timeout, push-failed-force-lease-mismatch) — same stranding bug class. - _resolve_bridge_event no longer returns None. Any unrecognised terminal_state falls back to merge_retry_exhausted (-> STUCK): operator-visible and recoverable beats silently orphaned in MERGING. The three static build_train reasons are mapped explicitly; the catch-all logs loudly so an unanticipated state is still visible. - merge-error-* routing refined: 404 -> merge_external_action (reconcile via PR state), 422 -> merge_base_conflict (CONFLICT_- RESOLVING re-rebases) instead of blanket STUCK. - Hard-crash reaper: merge_drive is a singleton and run_one_cycle is serial, so any workflow in MERGING at cycle start is the residue of a crashed prior cycle. run_one_cycle now reclaims them via merge_interrupted (MERGING -> APPROVED) before picking candidates. Adds bridge.list_merging_workflows. - run_one_cycle warns if a claimed PR is absent from pr_terminal_states instead of silently using the aggregate. 3309 tests pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
17cd91d275 |
fix(controller): post-commit review — close 3 merge-bridge stranding bugs
The trial-5 merge-pipeline split could strand workflows in MERGING
forever on its main paths. The controller has no MERGING handler
(merge_drive owns it), so any terminal_state that maps to no event
leaves the workflow orphaned. Three such holes, all caught by the
post-commit multi-perspective review:
1. merge_train emits "merge-error-{403,5xx,...}" for any non-2xx/409
Forgejo merge POST — none were mapped. Added _resolve_bridge_event:
403 -> branch_protection_blocked, all else -> retry_exhausted (STUCK).
2. run_one_cycle applied one shared outcome.terminal_state to every
claimed PR. A bisected train returns "bisected" (unmapped) so all
PRs stranded; a mixed train could tell a merged PR merge_base_conflict.
CycleOutcome now carries pr_terminal_states and events emit per-PR.
3. Graceful shutdown mid-merge ("stopped") was unmapped. Added the
merge_interrupted event (MERGING -> APPROVED) so the workflow
returns to the handoff state for clean re-pickup.
Also fixes a stale comment in _controller_db_bridge.py (merge_base_conflict
routes to CONFLICT_RESOLVING, not IMPLEMENTING) and adds a drift-guard
test cross-checking the bridge's transition map against the canonical
state machine — that drift is what produced the stale comment.
3304 tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
drew
|
a6986008ee |
feat(controller): trial-5 batch — dispute path, merge pipeline split, conflict-resolver hardening
T5-1 reviewer feedback rendered full-body to the implementer
T5-4/9 implementer dispute path — dispute-at-any-tier with per-tier cap,
OPERATOR_ATTENTION state on stalemate, pr-review-worker-dispute agent
T5-5 reviewer BLOCKING ISSUE EVIDENCE RULE + 5-step validation
T5-7 merge step split into a singleton process — impl/review masters write
APPROVED and stop; merge_drive owns APPROVED -> MERGING -> MERGED
T5-10 merge process is fully deterministic; base conflicts bounce to the
controller's CONFLICT_RESOLVING (LLM); conflict_drive sidecar retired
T5-11 implementer fast success path — verified-clean outcome so a no-op
after conflict resolution doesn't force busywork
T5-12 conflict-resolver permissions fixed across all paths (/tmp/** glob)
T5-13 conflict-resolver PR-intent prehydration (title/body/comments)
Adds tools/_controller_db_bridge.py so merge_drive reads the controller DB
directly (Option B), plus APPROVED + OPERATOR_ATTENTION states, the
dispute/verified-clean events, and the V1 contract fields backing them.
Reviewer model: baseline -> sonnet, dispute -> opus.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
drew
|
751f660f30 |
refactor(telemetry): realign dashboard with the controller state machine
The legacy dispatchers are retired; the controller state machine is now the system of record. This brings the telemetry console in line. Daemon health: - _DAEMON_SPECS now probes controller_master / controller_worker (drops dispatch_review, dispatch_implementer, conflict_drive, verify_invariant; keeps merge_drive + opencode_builder). - Removed the dispatcher-specific in-flight-cycle machinery (_last_in_flight_cycle, _DAEMON_CYCLE_TABLES); the controller tracks in-flight work via workflow_attempts.status. New controller-DB endpoints + Workflows tab: - /api/workflows — state-machine inventory with per-workflow attempt counts and a state summary. - /api/workflow_events — per-workflow state-transition timeline unioned with its attempts. - /api/attention — STUCK / OPERATOR_ATTENTION / PAUSED backlog plus recently reaped attempts, surfaced on the Overview tab. - New Workflows tab with a state filter and a click-through timeline. Existing surfaces: - /api/prs joins the live Forgejo PR list with controller current_state; filter is now by state, not retired auto/* labels. - /api/cycles keeps only the merge driver; retired driver names return a note pointing at /api/workflows instead of 500ing. - Drivers tab trimmed to merge_cycle; About copy updated. _iso() coerces DB timestamps (Python datetime on Postgres, ISO strings on SQLite raw queries) so the controller-DB endpoints work on both. Tests: dispatcher-cycle tests replaced with controller-DB coverage (workflows inventory + filter + repo isolation, attention backlog, workflow timeline, PR state join). 18 passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
27dbb3230c |
feat(telemetry): per-PR cost rollup + Phases tab for per-phase wallclock
- /api/cost?group_by=pr aggregates spend per PR across models; two-stage SQL+Python rollup preserves per-model pricing accuracy when a PR uses multiple models. - /api/phases reads worker phase wallclock from the controller DB (workflow_attempts joined to workflows, scoped to REPO_OWNER/REPO_NAME) and CI job-time from the legacy ci_gate_events table. CI is tagged kind=ci_job_time so share% reflects worker wallclock only. - New Phases tab in the dashboard with phase/PR group-by toggle. - Controller engine cache is thread-safe (double-check + Lock) and uses pool_pre_ping to recycle stale Postgres connections. - Empty/missing controller DB surfaces an instrumentation-pending note rather than 500ing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |