a21466add232d59cdec1604e09d58ca05659a623
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 drew
|
016b348117 |
feat(controller): grooming gate (Phase 0 + Phase 1 worker-shape dispatch)
Phase 0 (foundation):
- Cause enum (controller_events.cause) for action attribution
- Schema: grooming_decisions audit table; workflows gains
grooming_evaluated_at + deferred_reason + deferred_at +
deferred_target_workflow_id; pulls gains touched_files
- audit_comments: CLOSE / DEFER templates + render_comment_template
- forgejo_writes: close_issue + defer_issue 5-step crash-safe protocol
(fingerprint dedup, error matrix, dry-run)
- patch_pr_state callback in forgejo_http
- grooming_config: 22-env-var frozen-dataclass config + log_effective
- pulls.touched_files cache extension (_pipeline_cache.py schema v8)
- reaper.reap_grooming_decisions audit-retention sweep
- reconciliation RESUME guard (deferred_reason)
Phase 1 (worker-queue shape, 2026-05-25):
- New state: GROOMING. New events: grooming_started, groom_verdict_
{proceed,defer,close}. 5 new transitions; all invariants still clean
- GroomingInputV1 + GroomingOutputV1 Pydantic contracts
- outcomes._map_grooming_outcome routes verdicts to state-machine events
- prefetch.build_grooming_stage_b_input + list_open_prs callback
- scheduler GROOMING -> grooming_stage_b role
- promote: cfg-gated DISCOVERED -> GROOMING when CONTROLLER_GROOMING_
ENABLED=true; issues skip grooming
- forgejo_writes decomposed: close_act/defer_act (Forgejo writes only;
state-machine already transitioned) + close_decide_and_act/
defer_decide_and_act (Phase 0 callers); _apply_workflow_transition
is underscore-private
- grooming.py library: tokenization, suspicion scoring (Jaccard +
weighted overlap), deterministic checks, action -> verdict mapping
- mcp/grooming_builder.py: 14-tool FastMCP server emits GroomingOutputV1
- .opencode/agents/grooming-stage-b.md: duplicate-detection agent
prompt (claude-haiku-4-5)
- grooming_side_effects.run_grooming_side_effects_tick: per-state tick
performs Forgejo writes after groom_verdict_{defer,close} fires.
Filters on event_type='transition' + payload.event (centralizes the
convention pending Phase 2's latest_transition_event helper)
- GroomingCallbacks frozen dataclass; loop.py + __main__.py wired
Worker role registry (single source of truth):
- worker/roles.py: WORKER_ROLES + WorkerRoleSpec + default_roles_csv
+ output_filename_for. agent_runner.ROLE_TO_MCP_MODULE / ROLE_TO_
OUTPUT_MODEL derive from it; opencode_session.agent_name_for reads
it for flat cases; all 6 prompt builders use output_filename_for;
worker --roles default = default_roles_csv(); launcher script
derives --roles via shell substitution. Cross-site invariant test
enforces alignment across 5 sites + opencode.json MCP registry.
Phase 0 silent-bug fix:
- reconciliation.py RESUME guard SELECT now includes deferred_reason
(was missing since Phase 0; guard was a silent no-op). Tightened
from getattr to attribute access to fail fast on future omissions.
Tests (1456 total, +91 grooming-specific):
- test_grooming_phase0.py: 34 tests (orchestrator matrix, crash
recovery, idempotency, dry-run)
- test_grooming_phase1.py: 60 tests (library, contracts, state
machine, outcomes, scheduler, promote, prefetch, act-variants
with signature parity, side-effect tick incl. natural-idempotency
+ executed-flag-skip + verdict-mismatch + reconciliation RESUME)
- test_mcp_builders.py TestGroomingBuilder: 29 tests (happy paths
+ 22 validation rules + Pydantic round-trip + master-tick-read-
path companion)
- test_worker_agent_runner.py TestRoleMaps: cross-role wiring
alignment + agent-prompt-vs-worker-fallback filename contract +
inspect.signature equality (close_act/defer_act vs
close_issue/defer_issue)
- test_state_machine.py: transition count 51 -> 56 +
events_from_grooming
Live-validated end-to-end on 4 staged sentinel PRs (#55-#58) in
dry_run: agent emits verdicts via MCP, state-machine transitions
fire, side-effect tick writes audit row, deferred_reason gates
reconciliation RESUME correctly.
Deferred refinements + Phase 2 prerequisite (latest_transition_event
helper) tracked in .drew/regressions-plan.md "Phase 1 follow-up
backlog".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
981ddd6a8e |
fix(controller): incident hardening — PR-44 dispute guard + PR-46 fixes
Bundles fixes for three production incidents (PR-44, PR-46 a/b/c). Every change has an incident-reference comment in the code and a named regression test. 199 tests pass on the impacted modules. PR-44 — fabricated dispute escape --------------------------------- A tier-0 implementer, bounced twice by red CI, emitted a fabricated ``dispute-reviewer`` outcome and shortcut a red PR into REVIEWING → APPROVED → MERGING, bypassing the CI gate. ``dispute-reviewer`` is the only IMPLEMENTING → REVIEWING edge that doesn't pass through AWAITING_CI, so it must be defended. tools/controller/master/outcomes.py: ``_map_implementer_outcome`` now guards ``dispute-reviewer`` with two preconditions — (1) ``attempt_saw_green_ci`` (the HEAD must already be CI-verified; a dispute can't jump the gate on a red/pending head), and (2) ``prior_reviews >= 1`` (must reference a review that actually happened, not a hallucinated one). Either guard fails → ``implementer_competence_failure`` → tier escalation. A weak model can't game its way past CI; a stronger tier is given the real problem. tools/controller/master/tick.py: new ``_count_prior_reviews`` helper counts completed reviewer attempts (epoch-scoped so an ``operator_unstick`` resets the count). Wired into the ``map_outcome_to_event`` call. PR-46(a) — stale gate-script preferred over in-repo --------------------------------------------------- ``gate.py`` was preferring the seeded ``/tmp/local_tools`` copy of ``local_ci_gate.sh`` over the version-matched in-repo copy. The seed predated the ``--envdir`` flag; the controller pipeline's invocations rejected as bad-argv every committing implementer's gate. The seed-refresher (``dispatch_implementer.py``) is on the retired dispatcher path, so the staleness was permanent. tools/controller/worker/gate.py: resolution order is now ``CONTROLLER_LOCAL_CI_GATE`` env > in-repo > seeded ``/tmp``. The seeded copy survives only as a last-resort fallback. Module-level constants ``_IN_REPO_GATE_SCRIPT`` / ``_SEEDED_GATE_SCRIPT`` let tests substitute paths. PR-46(b) — 6-second-old run flagged zombie ------------------------------------------ ``classify_ci_run`` instantly classified a CI run as ``stale`` when the Actions API reported no active task. A freshly-pushed run has no task simply because no runner has picked it up yet, and there are brief gaps between jobs — both false positives. A 6-second-old PR-46 run was bounced before CI could even start. tools/controller/master/ci_run_status.py: new ``ZOMBIE_GRACE`` (default 3 min, env: ``CONTROLLER_CI_ZOMBIE_GRACE_MIN``). "No active task" only classifies a run as stale once the run has ALSO gone quiet past the grace. Much shorter than ``STALE_AFTER`` since the active-task absence is corroborating evidence, not the sole signal. PR-46(c) — ruff-format-only violation slipping through lint ----------------------------------------------------------- CI's ``lint`` job runs both the ``lint`` nox session (ruff check) AND ``ruff format --check``. The pre-push local gate only ran the former; a formatting-only violation passed pre-push then failed CI. tools/local_ci_gate.sh: the ``lint`` gate now runs ``ruff check`` followed by ``ruff format --check``, unconditional. Either one failing marks the gate red. ``ruff format --check`` is whole-repo and takes no posargs. tests/auto_agents/test_local_ci_gate.py updated for the new two-call shape. Supporting changes ------------------ .forgejo/workflows/ci.yml: gates ``coverage`` and ``docker`` jobs on repo variables ``skip_coverage`` / ``skip_docker`` so the long reaper-prone jobs can be skipped per-fork without editing CI. Guard step always runs so the job still reports ``success`` and ``status-check`` stays green. tools/duplicate_prs_to_fork.py: bakes the same ``skip_coverage`` / ``skip_docker`` gates into every sentinel PR's ci.yml at PR-creation time so fork-mode runs inherit the gating. Idempotent. .opencode/opencode.json: adds ``timeout: 1860000`` (31 min) to the ``ci`` MCP server so long CI waits don't timeout the tool. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
db12f45acb |
feat(controller): Phase 1m — wire response-builder MCPs into OpenCode (Option A)
The trial-path expedient (legacy_adapter harvesting chat-JSON) is now
the SAFETY NET; the architecture-intended path (agents call the
controller's response-builder MCPs, which validate against V1 +
write canonical JSON) is now wired end-to-end.
Four pieces:
1. MCP `finalize` accepts `output_path` argument
``mcp/_builder_base.finalize_and_emit`` adds ``output_path: str | None``
kwarg. Precedence: explicit arg > ``CONTROLLER_CANONICAL_OUTPUT_PATH``
env > stdout. This solves the per-attempt path problem that
blocked opencode.json registration (the env var is static; the
per-attempt path comes from the prompt, the agent passes it as a
tool arg). Each role's finalize updated:
- estimator_finalize(output_path)
- implementer_finalize(output_path)
- reviewer_finalize(output_path)
- conflict_finalize(output_path)
- summarizer_finalize(output_path)
Creates parent directory if missing (so the controller doesn't
need to pre-create). Returns ``wrote_to`` in the ok dict so tests
can pin the path.
2. opencode.json registers the 5 controller MCPs
``.opencode/opencode.json`` adds:
- implementer-response-builder
- reviewer-response-builder
- estimator-response-builder
- conflict-resolver-response-builder
- summarizer-response-builder
Each spawned via ``python -m tools.controller.mcp.{role}_builder``
with PYTHONPATH=/repo-root so the controller imports resolve.
3. Controller prompt builder injects the EXACT tool sequence
``worker/prompts.py`` rewrites each role's "Output contract"
section. The old "PREFERRED MCP / FALLBACK file-write" instruction
becomes a single REQUIRED contract: numbered tool calls (``X_start``,
``X_set_*``, ``X_finalize(output_path=...)``) with the explicit
per-attempt path baked in. Explicit "DO NOT emit a JSON object in
your final chat message" instruction to override the legacy
contract baked into the agent system prompts.
4. Agent permission whitelists include the new MCPs
- ``.opencode/agents/task-implementor.md`` (source for tier-{0,1,2,min}
variants — regenerated via tools/sync_tier_models.py)
- ``.opencode/agents/pr-review-worker.md``
- ``.opencode/agents/estimator-implementation.md``
- ``.opencode/agents/conflict-resolver-worker.md``
Each adds the matching ``"{role}*": allow`` pattern.
Safety net preserved:
The legacy_adapter (commit
|
||
|
drew
|
2658deee94 |
feat(auto-agents): PR State Warmer substrate + supporting infra
Adds a long-lived sidecar (pr_state_warmer.py) that polls Forgejo's /pulls endpoint every 30s and writes the full PR snapshot to a shared SQLite store, eliminating the dispatcher's per-cycle cold-cache stalls (24-30s rebuilds on flaky cycles) and the silent 50-PR pagination cap on the legacy single-page fetch. Substrate - tools/_pr_state_cache.py — SQLite store with (owner, repo) PK, WAL mode, additive v2→v3 migration (comments_refreshed_updated_at), bounded fcntl.flock migration lock, threading.Lock for per-process init, @_with_reheal decorator (catches OperationalError no-such- table + DatabaseError corruption with file quarantine), atomic TEMP-table chunking for >32k seen-set, _normalize_updated_at to canonicalize Forgejo tz-marker drift - tools/pr_state_warmer.py — poll/upsert/vanish/comments-refresh loop with fcntl.flock singleton (rejects second warmer), bounded comments-refresh cap, persistent deferral via SQL pending query, PermissionError-tolerant lock setup, cold-start log suppression - tools/_pr_classification_cache.py — three-layer fall-through (warmer cache → list cache → live fetch) with staleness gate (PR_STATE_WARMER_STALE_AFTER_S floored at 30s in prod) Comments cache hardening - Bot-filter at write time drops bot status/claim/release/sentinel while preserving **Implementation Attempt** markers (94.6% reduction on bot-heavy PRs like #30's 19k-comment thread) - _normalize_since_cursor strips microsecond precision before building ?since= query (fixes the live-observed Forgejo HTTP 422 bug on PRs #25 + #28); handles uppercase Z, lowercase z, ±HH:MM offsets (including non-zero like +05:30), naive ISO - Lazy migration of legacy null-key by_author entries on _read_cache - _newest_cursor walks tail-back skipping malformed entries Supporting infrastructure (cumulative dmpipeline-v2 work) - Telemetry server: SSE live tail, run-sessions enumeration, cost/token tracking, app.js UI rewrite with collapsible sections - MCP servers (mcp_ci_server, mcp_forgejo_server, mcp_git_server, mcp_handoff_server, mcp_graphify_server) for opencode worker context access - Live log writer (tools/live_log_writer.py) — SSE-streaming dispatcher event log - Tier-dispatcher escalation flow with prompts trimmed for budget - Shared bot-logins resolver (tools/_bot_logins.py) replacing two drift-prone copies - token_usage_audit.py for opencode cost analysis Tests - 2259 passing across 65 changed/new files - New suites: test_pr_state_cache, test_pr_state_warmer, test_pr_state_warmer_integration, test_pr_classification_cache, test_pr_list_cache_backoff, test_mcp_* (5 servers), test_live_log_writer_sse, test_telemetry_run_sessions, test_review_post_ready_label - Test_pr_comments_cache expanded with bot-filter coverage, cursor-normalization regression pins, format-drift, atomicity, failed-comments-not-stamped (silent-data-loss class) - Parametrized @_with_reheal coverage across 7 wrapped APIs - Real fault-inject atomicity test for chunked mark_vanished path via Connection wrapper class - Subprocess-based singleton flock test (cross-process contract) - Event-driven SIGTERM-mid-poll test (no fixed-sleep flake) Architecture notes - Schema v3 migration is additive (ALTER ADD COLUMN); v0/v1 still need destructive rebuild because pre-v2 column shape lacks owner/repo. Cross-process drop-table-ping-pong prevented by the fcntl migration lock + per-process _initialized flag. - Comments-refresh deferral is persistent via comments_refreshed_updated_at column — survives warmer restart, picks up next cycle even if PR didn't change again. Replaces in-memory changed_numbers list. - Rollback path: PR_STATE_WARMER_PREFER=0 bypasses the warmer cache and reverts to live-fetch behavior. PR_STATE_CACHE_DISABLE=1 short-circuits the warmer process at startup. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |