a21466add232d59cdec1604e09d58ca05659a623
12 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 drew
|
016b348117 |
feat(controller): grooming gate (Phase 0 + Phase 1 worker-shape dispatch)
Phase 0 (foundation):
- Cause enum (controller_events.cause) for action attribution
- Schema: grooming_decisions audit table; workflows gains
grooming_evaluated_at + deferred_reason + deferred_at +
deferred_target_workflow_id; pulls gains touched_files
- audit_comments: CLOSE / DEFER templates + render_comment_template
- forgejo_writes: close_issue + defer_issue 5-step crash-safe protocol
(fingerprint dedup, error matrix, dry-run)
- patch_pr_state callback in forgejo_http
- grooming_config: 22-env-var frozen-dataclass config + log_effective
- pulls.touched_files cache extension (_pipeline_cache.py schema v8)
- reaper.reap_grooming_decisions audit-retention sweep
- reconciliation RESUME guard (deferred_reason)
Phase 1 (worker-queue shape, 2026-05-25):
- New state: GROOMING. New events: grooming_started, groom_verdict_
{proceed,defer,close}. 5 new transitions; all invariants still clean
- GroomingInputV1 + GroomingOutputV1 Pydantic contracts
- outcomes._map_grooming_outcome routes verdicts to state-machine events
- prefetch.build_grooming_stage_b_input + list_open_prs callback
- scheduler GROOMING -> grooming_stage_b role
- promote: cfg-gated DISCOVERED -> GROOMING when CONTROLLER_GROOMING_
ENABLED=true; issues skip grooming
- forgejo_writes decomposed: close_act/defer_act (Forgejo writes only;
state-machine already transitioned) + close_decide_and_act/
defer_decide_and_act (Phase 0 callers); _apply_workflow_transition
is underscore-private
- grooming.py library: tokenization, suspicion scoring (Jaccard +
weighted overlap), deterministic checks, action -> verdict mapping
- mcp/grooming_builder.py: 14-tool FastMCP server emits GroomingOutputV1
- .opencode/agents/grooming-stage-b.md: duplicate-detection agent
prompt (claude-haiku-4-5)
- grooming_side_effects.run_grooming_side_effects_tick: per-state tick
performs Forgejo writes after groom_verdict_{defer,close} fires.
Filters on event_type='transition' + payload.event (centralizes the
convention pending Phase 2's latest_transition_event helper)
- GroomingCallbacks frozen dataclass; loop.py + __main__.py wired
Worker role registry (single source of truth):
- worker/roles.py: WORKER_ROLES + WorkerRoleSpec + default_roles_csv
+ output_filename_for. agent_runner.ROLE_TO_MCP_MODULE / ROLE_TO_
OUTPUT_MODEL derive from it; opencode_session.agent_name_for reads
it for flat cases; all 6 prompt builders use output_filename_for;
worker --roles default = default_roles_csv(); launcher script
derives --roles via shell substitution. Cross-site invariant test
enforces alignment across 5 sites + opencode.json MCP registry.
Phase 0 silent-bug fix:
- reconciliation.py RESUME guard SELECT now includes deferred_reason
(was missing since Phase 0; guard was a silent no-op). Tightened
from getattr to attribute access to fail fast on future omissions.
Tests (1456 total, +91 grooming-specific):
- test_grooming_phase0.py: 34 tests (orchestrator matrix, crash
recovery, idempotency, dry-run)
- test_grooming_phase1.py: 60 tests (library, contracts, state
machine, outcomes, scheduler, promote, prefetch, act-variants
with signature parity, side-effect tick incl. natural-idempotency
+ executed-flag-skip + verdict-mismatch + reconciliation RESUME)
- test_mcp_builders.py TestGroomingBuilder: 29 tests (happy paths
+ 22 validation rules + Pydantic round-trip + master-tick-read-
path companion)
- test_worker_agent_runner.py TestRoleMaps: cross-role wiring
alignment + agent-prompt-vs-worker-fallback filename contract +
inspect.signature equality (close_act/defer_act vs
close_issue/defer_issue)
- test_state_machine.py: transition count 51 -> 56 +
events_from_grooming
Live-validated end-to-end on 4 staged sentinel PRs (#55-#58) in
dry_run: agent emits verdicts via MCP, state-machine transitions
fire, side-effect tick writes audit row, deferred_reason gates
reconciliation RESUME correctly.
Deferred refinements + Phase 2 prerequisite (latest_transition_event
helper) tracked in .drew/regressions-plan.md "Phase 1 follow-up
backlog".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
a103a31bbf |
feat(controller): worker-owned gated push for the implementer
The implementer agent no longer pushes to git — the controller worker now owns the push: it gates the agent's commits on lint+typecheck and pushes via a single leased primitive. Closes two production defects: - Clobber: the pre-fix MCP --force-with-lease leased against a freshly-fetched tip, so the lease always passed — an in-flight implementer destroyed a commit pushed to the PR branch during its run (lost a hand-pushed skip_coverage fix on PR #46). - Gate-skip: the agent verified only the CI-flagged gate, so a fix for one gate shipped fresh violations in another (lint flapped pass->fail across CI runs 198->199). Step 1 — worker_push primitive: - New git_push.py: one leased push, pinned to the SHA the worker started from; classifies pushed / stale_input / diverged / infra_error; bounded infra-retry. - mcp_git_server.push gains expected_sha for a correctly-pinned lease. - finalize_conflict_resolution migrated onto worker_push. Step 2 — deterministic gate: - New gate.py: per-slot nox env-dirs (no venv races), manifest-hash staleness keying, lazy warm-up. - local_ci_gate.sh gains --envdir. Step 3 — worker-owned gated push: - New implementer_finalize.py: divergence pre-check -> lint+typecheck gate -> ruff auto-fix -> leased push. finalize's outcome is authoritative over the agent's emitted outcome. - agent_runner integrates finalize; salvage no longer pushes. - outcomes/tick: gate-failed + push-time stale-input caps, epoch-scoped; WorkerError carries an output_payload so the gate report reaches the next attempt's prompt; prefetch surfaces gate-failed attempts. - The 5 task-implementor prompts drop the agent push step. Reviewed across 4 adversarial rounds; full controller+MCP suite green (1379 passed). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
14e592ddd5 |
feat(controller): zombie-CI detection — stop waiting on dead CI runs
A CI gate stuck `pending` is ambiguous: the job may genuinely be running, or the run may be dead (a crashed runner, an Actions job whose terminal commit-status was never posted — `CI / status-check` zombies routinely here). The "wait for the whole run to finish" fix then waited forever on the dead case (PR #36: a `status-check` gate pending for 8 h while the run had actually finished RED 8 h earlier). New `ci_run_status.classify_ci_run` resolves a still-pending run to `complete` / `running` / `stale` via two checks, authoritative-first: 1. ACTIVE-RUN — `get_action_tasks` asks Forgejo's Actions API directly whether a task for the commit is still running; catches a dead run immediately, regardless of age. 2. AGE — if no gate has updated in > CONTROLLER_CI_STALE_AFTER_MIN (default 90) the run has stopped; the fallback when the Actions API is unavailable. A `stale` run is no longer waited on: the verdict is taken from the gates that DID finish (`terminal_verdict`) — any failure → red, all pass → green, fully-dead → red. Applied in both `ci_status_poll` (the AWAITING_CI verdict) and `ci_summarize` (the implementer's summary — zombie pending gates drop out of `gates_pending`/`overall_state`) so the two agree and never ping-pong. New `get_action_tasks` Forgejo callback wired through forgejo_http → __main__ → the poll and the prefetch path. 23 new tests; full controller suite (1222) green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
c4a11901f5 |
feat(ci): unified CI-log cache — phase 2 (consumer migration + RUN_CI_LOCAL)
Builds on
|
||
|
drew
|
27289ea4b7 |
feat(ci): unified CI-log cache — get_ci_logs + implementer path (phase 1)
The implementer's ci_summary carried an empty raw_log_excerpt for every failed gate (diagnosed via PR 39): prefetch._build_ci_summary passed a no-op log fetcher, and ci_summarize._gate_to_nox_session never stripped Forgejo's "(pull_request)" event suffix, so every PR gate fell through to NoParserAvailable with no log fetched at all. Phase 1 — the implementer-facing path: - _ci_logs.get_ci_logs(): unified entry point — every job of a run, full untruncated logs, one cache. `partial` marks an in-flight run; a terminal + clean bundle is frozen forever. Reuses the existing session-cookie login + exponential backoff. Additive — fetch_pr_failure_logs and its 9 consumers are untouched. - ci_summarize: strip the "(pull_request)" event suffix so gates resolve to their nox parser; _no_parser_failure now carries the raw log instead of hardcoding "". - prefetch._build_ci_summary + forgejo_http + __main__: wire a real get_ci_logs-backed log_fetcher so each failed gate's raw_log_excerpt is filled from the cache. - 9 new tests; full controller suite green (1169 passed). Deferred to later phases: re-point the freshness gate / ci_status_poll onto get_ci_logs, RUN_CI_LOCAL into the same cache, and (head_sha, run_id, attempt) multi-attempt keying. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
521117882e |
fix(controller): conflict resolver lands resolutions on the PR
Run-2 PR #40 exposed that the conflict-resolver stage never resolved conflicts *on the PR*. The resolver agent owned the whole git flow (rebase + resolve + push) but was never given the PR head branch name, so it pushed the resolved branch to a guessed branch ("main"), the push was lost, and it reported "resolved" anyway — the controller advanced the workflow on a resolution that never reached the PR. This splits the work along the controller's principle: deterministic git mechanics in the worker, only the semantic merge in the agent. - conflict_rebase.py (new): the worker starts the rebase (prepare_conflict_rebase — hands the agent a real mid-rebase worktree) and, after the agent's resolution, lands it with a *verified* push (finalize_conflict_resolution) — force-with-lease pinned to the PR head the resolver started from, then re-reads the remote to confirm it moved. A failed/rejected push is never reported as resolved. run_conflict_resolver_attempt orchestrates it. - ConflictResolverInputV1 / prefetch: carry head_ref so the worker knows the PR branch (this also makes the worker's fetch_and_validate run for the conflict_resolver — it gated on head_sha AND head_ref). - worker/__main__.py: route conflict_resolver attempts through the new orchestration. - conflict-resolver-worker.md: the agent resolves + continues the rebase and never pushes — the controller lands it. - mcp_git_server.py: add the controller worktree root to the git MCP's allowed bases (every git_* MCP call errored for this role before). The committed-marker check is scoped to the PR's own diff — a whole-tree grep false-positived on repo files that legitimately document conflict markers. The worker's rebase/push are lost-lock guarded so a reaped worker cannot race the shared worktree. 1136 controller tests pass (+11 new in test_conflict_rebase.py). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
0bc734c020 |
style: ruff format the controller-state-machine branch (288 files)
Applies `ruff format` to the accumulated formatting debt on this branch. Formatting-only — no behavioral changes. Required for CI/lint's format gate (`nox -s format -- --check`), which the branch was failing on 288 tracked files that drifted from ruff's canonical style. In-progress WIP files are intentionally excluded so this commit stays a clean formatting-only diff. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
a2fcecb28d |
feat(controller): CI-freshness gate — re-trigger stale/infra CI instead of dead-ending
A discovered PR with stale CI (every job failed at the checkout step —
a git-fetch connection reset, pure infra, no code ran) burned an
estimator + a tier-2 implementer and dead-ended at STUCK. The pipeline
had no notion of CI freshness and never triggered CI — only polled.
New early master tick (ci_gate) runs before DISCOVERED->ANALYZING
promotion. For each DISCOVERED pr-kind workflow it classifies the CI
via ci_freshness.classify_ci_result:
- infra_broken — failed; the failing jobs' LOG content carries a
checkout/setup signature (curl 56, expected 'packfile', ...).
Logs are fetched via _ci_logs (session-cookie auth).
- stale — failed; newest status older than CONTROLLER_CI_MAX_AGE_S
(default 6h). Timestamp-based, log-independent — catches an old
failure even when Forgejo has purged its logs.
- no_ci / pending / fresh_real — handled accordingly.
infra_broken/stale/no_ci -> push an empty commit to the PR branch
(Forgejo 15.0.2 has no rerun API), routing DISCOVERED -> AWAITING_CI
(new event discovery_ci_rerun_triggered). The existing AWAITING_CI
poller then gets a real verdict. A reran CI that is ALSO infra/stale
routes AWAITING_CI -> DISCOVERED (new event ci_infra_recheck) so the
gate re-handles it; bounded by a rerun budget of 3, then STUCK.
Also wires the existing CI summarizer into prefetch so workers stop
receiving ci_summary=null.
1095 controller tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
drew
|
a6986008ee |
feat(controller): trial-5 batch — dispute path, merge pipeline split, conflict-resolver hardening
T5-1 reviewer feedback rendered full-body to the implementer
T5-4/9 implementer dispute path — dispute-at-any-tier with per-tier cap,
OPERATOR_ATTENTION state on stalemate, pr-review-worker-dispute agent
T5-5 reviewer BLOCKING ISSUE EVIDENCE RULE + 5-step validation
T5-7 merge step split into a singleton process — impl/review masters write
APPROVED and stop; merge_drive owns APPROVED -> MERGING -> MERGED
T5-10 merge process is fully deterministic; base conflicts bounce to the
controller's CONFLICT_RESOLVING (LLM); conflict_drive sidecar retired
T5-11 implementer fast success path — verified-clean outcome so a no-op
after conflict resolution doesn't force busywork
T5-12 conflict-resolver permissions fixed across all paths (/tmp/** glob)
T5-13 conflict-resolver PR-intent prehydration (title/body/comments)
Adds tools/_controller_db_bridge.py so merge_drive reads the controller DB
directly (Option B), plus APPROVED + OPERATOR_ATTENTION states, the
dispute/verified-clean events, and the V1 contract fields backing them.
Reviewer model: baseline -> sonnet, dispute -> opus.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
drew
|
febb352618 |
fix(controller): pipeline run-blockers — promoter, scheduler, owner/repo, workspace_dir patch
Round-3 deep pass identified four issues that would have prevented an
actual end-to-end pipeline run:
RB1 — DISCOVERED → ANALYZING never fired in production:
The state machine defines (DISCOVERED, discovery_picked_up) →
ANALYZING but NO production code fires the event. Workflows
created by discovery would sit in DISCOVERED forever.
Fix:
- New ``master/promote.py``: ``run_promote_discovered_tick`` scans
for DISCOVERED workflows + fires ``discovery_picked_up`` via
apply_event (state-machine invariants stay enforced) + emits a
``discovery-promoted`` controller_events row per transition.
- Composes with the master loop's other ticks; runs every iteration
(cheap — typically 0-1 row).
RB2 — scheduler.schedule_next_attempts never called from master loop:
The scheduler was exported by the master package but never invoked.
It creates the ``workflow_attempts`` rows that workers dequeue —
without it, workers would have nothing to pick up.
Fix:
- ``master/loop.py`` now accepts a ``prefetch: PrefetchCallback``
kwarg. When provided, the loop runs promote_discovered + scheduler
every iteration after tick/reaper/reconciliation.
- ``MasterTickReport`` gains ``promote_discovered`` and ``scheduler``
optional fields so on_iteration callbacks see both.
- ``master/__main__.py`` builds a ``PrefetchDataCallbacks`` from the
Forgejo callback bundle and constructs the production
``make_prefetch_callback(engine, callbacks)`` — wires through to
the loop's new prefetch kwarg.
RB3 — owner / repo missing from V1 input contracts:
The implementer / reviewer / estimator / conflict-resolver V1 inputs
had pr_number but not owner/repo. The OpenCode agent would have
had no way to know which Forgejo repo to clone — it would have had
to derive owner/repo from process env, coupling the worker to a
single repo.
Fix:
- ``contracts/v1.py``: added ``owner: str`` and ``repo: str``
(min_length=1) to ImplementerInputV1, ReviewerInputV1,
EstimatorInputV1, ConflictResolverInputV1.
- ``master/prefetch.py``: builders populate owner/repo from the
Workflow row (already known at prefetch time).
- Existing test fixtures in ``test_contracts_v1.py`` updated.
RB4 — input_payload.workspace_dir placeholder reached the agent:
Prefetch wrote ``workspace_dir = "<worker-injected>"`` as a
placeholder; the worker never patched it before invoking the
OpenCode session. The prompt builder rendered the literal
placeholder string into the agent's prompt — the agent had no idea
where to clone.
Fix:
- ``worker/agent_runner.py``: patches input_payload.workspace_dir
with the real path immediately before calling run_opencode_session.
Uses a shallow copy so the caller's dict isn't side-effected.
- ``worker/__main__.py``: workspace_dir naming convention is now
``pr-{owner}-{repo}-{pr_number}`` (matches workspace.py's
PerPRWorkspace convention) so the janitor's pr-* glob + the
agent's expected workspace location agree. Falls back to
``pr-attempt-{N}`` for legacy input_payloads missing owner/repo.
Tests:
- ``test_master_promote.py`` (NEW, +7 tests):
- empty DB no-op
- single workflow promoted
- multiple promoted in one tick
- only DISCOVERED targeted (non-DISCOVERED untouched)
- controller_events row emitted with correct shape
- idempotent after first promotion
- LoopIntegration end-to-end: DISCOVERED → ANALYZING → pending
estimator attempt visible in workflow_attempts (pins the entire
previously-broken pipeline from discovery to enqueue)
Total: 703 controller tests pass (+7 net), 0 regressions.
Without these four fixes, the pipeline would have looked alive in
unit tests but produced zero work in a real deployment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
6ba1926c52 |
fix(controller): batch A — datetime json, Forgejo state map, ISO compare, dequeue docs
Four narrow bug fixes flagged by adversarial code review (items 1, 5, 6, 7 from the consolidated critique). ITEM 1 — datetime → json.dumps crash (silent write-after-work failure): - ``worker/runner.py:274`` and ``master/scheduler.py:283`` now pass ``default=str`` to ``json.dumps`` so nested datetime fields (e.g. CISummary.observed_at) serialize without raising. - Before this fix: a worker would do its real work, then crash on the terminal-state UPDATE with TypeError, get recorded as ``worker-internal-error``, and the output payload would be lost. - Test: TestDatetimeSerializationSafety in test_master_ci_summarize + test_scheduler_handles_datetime_in_input_payload in test_master_prefetch (both pin the regression — the with-default test passes, the without-default test asserts the TypeError so future maintainers see the failure mode). ITEM 5 — Forgejo state mapping completeness: - Extended ``_FORGEJO_STATE_TO_GATE_STATUS`` in ``master/ci_summarize.py`` to cover ``cancelled``, ``timed_out``, ``action_required``, ``queued``, ``in_progress``, ``neutral``, ``skipped``, ``stale`` — states observed across Forgejo / Gitea / GH-mirror that previously collapsed to ``pending``, telling the implementer "CI is still running" when really a job was cancelled. - ``cancelled`` / ``timed_out`` / ``action_required`` / ``stale`` now map to ``error`` (the gate failed). - ``queued`` / ``in_progress`` stay ``pending`` (still running). - ``neutral`` / ``skipped`` → ``passed``/``skipped`` (informational). - Test: TestExtendedForgejoStates — 6 tests covering each new state. ITEM 6 — lexicographic ISO comparison drops/dupes comments: - ``master/prefetch.py:_comment_bodies_since`` and ``_iso`` replaced with ``_to_aware_datetime`` + datetime comparison. Forgejo emits ``2026-05-18T12:00:00Z``; Python's ``datetime.isoformat()`` emits ``2026-05-18T12:00:00+00:00`` — a string compare gives 'Z' (0x5A) vs '+' (0x2B) which silently misorders timestamps. - Now parses via ``datetime.fromisoformat`` (with Z → +00:00 rewrite), defaults naive timestamps to UTC, and compares as ``datetime``. - Test: test_comments_filter_handles_z_suffix_vs_offset_form pins the regression. ITEM 7 — false BEGIN IMMEDIATE claim in dequeue docstring: - The dequeue docstring claimed ``BEGIN IMMEDIATE`` was applied by session_scope; it wasn't. Attempted a global ``begin``-event listener that conflicted with StaticPool's shared-connection model (test_prefetch_callback_works_in_scheduler broke). - Reverted to a documentation fix: SQLite stays on default BEGIN DEFERRED (the SQLITE_BUSY retry via busy_timeout=5s is acceptable for single-host dev) + the docs make MULTI-MACHINE REQUIRES POSTGRES explicit at three call sites (db/session.py, db/dequeue.py, RUNBOOK.md was already updated in Phase 1l). Postgres has FOR UPDATE SKIP LOCKED which is what production actually uses. Tests: 586 controller tests pass (+10 new), 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
dfdfbf762b |
feat(controller): Phase 1h — prefetch callbacks (V1 input assembly)
Per plan v9, the master assembles the worker input_payload at attempt-enqueue time so the worker dequeues a ready-to-use payload with no extra Forgejo I/O of its own. This phase ships per-role V1-input builders + a factory matching the scheduler's PrefetchCallback protocol: - build_implementer_input → ImplementerInputV1 shape (head_sha, head_ref, base_branch, active_reviews, pr_comments_since_last_attempt, prior_attempts, diff_summary) - build_reviewer_input → ReviewerInputV1 shape (full_diff, prior_implementer_attempts, implementer_claim, prior_reviews) - build_estimator_input → EstimatorInputV1 shape (pr_title, pr_body, diff_summary) — works for both PR and issue kinds - build_conflict_resolver_input → ConflictResolverInputV1 shape with conflicted_files=[] stub (worker patches via git rebase) - make_prefetch_callback(engine, callbacks) → routes by role; returns (payload, "V1") matching the scheduler's PrefetchCallback signature Forgejo HTTP wiring adds four new callbacks (get_pr_details, get_pr_diff, list_pr_reviews, list_pr_comments) plumbed through ForgejoCallbacks. Worker-side patches (post-dequeue, pre-validation): - attempt_id, attempt_number (known from dequeue) - workspace_dir (worker filesystem path) - wallclock_budget_s (worker config) What this phase DOES NOT yet produce: - ci_summary / failing_gates — Phase 1j (deterministic CI summarizer) - Issue-kind estimator's title/body — needs list_issue_details callback (defer to future phase) - conflict_resolver's actual conflicted_files — needs worker-side git rebase + conflict-parse pass Tests (+26 in test_master_prefetch.py, 0 regressions): - Per-role shape validation + V1 contract parse after worker patches - Prior-attempts merge (verbatim cap=3, oldest-first, total count) - Active-reviews projection (filters invalid states/missing user) - pr_comments_since_last_attempt filtering by finished_at - Factory routes by role; unknown role raises - Scheduler integration end-to-end (real prefetch → real INSERT) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |