a21466add232d59cdec1604e09d58ca05659a623
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 drew
|
0bc734c020 |
style: ruff format the controller-state-machine branch (288 files)
Applies `ruff format` to the accumulated formatting debt on this branch. Formatting-only — no behavioral changes. Required for CI/lint's format gate (`nox -s format -- --check`), which the branch was failing on 288 tracked files that drifted from ruff's canonical style. In-progress WIP files are intentionally excluded so this commit stays a clean formatting-only diff. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
b80f5f7c3a |
fix(auto-agents): reviewer data-handling — five live-observed regressions
Five independent bug fixes around the reviewer's data pipeline, all live-observed on 2026-05-17 run-1 (PR #35 / PR-shaped traffic generally). Bundled because they share the same data-correctness intent and one fixture-update covers two of them. 1) **PR diff: distinguish "fetch failed" from "PR has no changes."** ``tools/_pr_diff.py``. The old code conflated both cases into ``unavailable=True``, which forced ``data_complete=False`` and blocked APPROVED verdicts. Live-observed on PR #35 created by the new_issue worker without any code changes — Forgejo returns HTTP 200 + empty body for a head==base PR, and the reviewer was incorrectly told the diff was unavailable. 2) **PR-state cache: refresh body + labels on the unchanged- ``updated_at`` branch.** ``tools/_pr_state_cache.py``. Forgejo label add/remove mutations do NOT bump ``updated_at``, so the warmer's cached PR object would carry stale labels for as long as the PR sat idle. Downstream consumers (cycle-cap, claim sweeps, filter exclusions) would never see them. Cheap fix — same row, two extra columns refreshed. 3) **Comments cache: URL-encode the ``since=`` cursor.** ``tools/_pr_comments_cache.py`` + matching test update. The ``+`` in ``+00:00`` decodes to a space on Forgejo's query- string parser, producing 422 errors. Live-observed on PR #35 run-1: 6 consecutive 422s on the same clean ``+00:00`` cursor before the cache backed off for 30 min. ``urllib.parse.quote`` with ``safe=''`` quotes every non-alphanumeric so ``+`` → ``%2B``, ``:`` → ``%3A``. Test updated to ``unquote`` the captured path before substring-matching. 4) **Reviewer prompt: surface the clone fallback.** ``tools/_review_prompt.py``. Adds an inline note in the pre-fetched diff section explaining the two diff sources (inline-truncated vs pre-cloned worktree) and the ``REVIEW_DISPATCHER_DIFF_MAX_BYTES`` cap. Closes a reviewer- side confusion where the model didn't know it could read source files from disk when the inline diff was truncated. 5) **Reviewer agent contract: truncated diff + clone IS data-complete.** ``.opencode/agents/pr-review-worker.md``. The prior wording said ``truncated=True`` forced ``data_complete= False`` and blocked APPROVED — but with the pre-cloned worktree available, the reviewer DOES have full code access and APPROVED should remain valid. Updated guidance now distinguishes "truncated but clone present" (APPROVED OK) from "truncated AND no clone" (COMMENT / REQUEST_CHANGES only). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
2658deee94 |
feat(auto-agents): PR State Warmer substrate + supporting infra
Adds a long-lived sidecar (pr_state_warmer.py) that polls Forgejo's /pulls endpoint every 30s and writes the full PR snapshot to a shared SQLite store, eliminating the dispatcher's per-cycle cold-cache stalls (24-30s rebuilds on flaky cycles) and the silent 50-PR pagination cap on the legacy single-page fetch. Substrate - tools/_pr_state_cache.py — SQLite store with (owner, repo) PK, WAL mode, additive v2→v3 migration (comments_refreshed_updated_at), bounded fcntl.flock migration lock, threading.Lock for per-process init, @_with_reheal decorator (catches OperationalError no-such- table + DatabaseError corruption with file quarantine), atomic TEMP-table chunking for >32k seen-set, _normalize_updated_at to canonicalize Forgejo tz-marker drift - tools/pr_state_warmer.py — poll/upsert/vanish/comments-refresh loop with fcntl.flock singleton (rejects second warmer), bounded comments-refresh cap, persistent deferral via SQL pending query, PermissionError-tolerant lock setup, cold-start log suppression - tools/_pr_classification_cache.py — three-layer fall-through (warmer cache → list cache → live fetch) with staleness gate (PR_STATE_WARMER_STALE_AFTER_S floored at 30s in prod) Comments cache hardening - Bot-filter at write time drops bot status/claim/release/sentinel while preserving **Implementation Attempt** markers (94.6% reduction on bot-heavy PRs like #30's 19k-comment thread) - _normalize_since_cursor strips microsecond precision before building ?since= query (fixes the live-observed Forgejo HTTP 422 bug on PRs #25 + #28); handles uppercase Z, lowercase z, ±HH:MM offsets (including non-zero like +05:30), naive ISO - Lazy migration of legacy null-key by_author entries on _read_cache - _newest_cursor walks tail-back skipping malformed entries Supporting infrastructure (cumulative dmpipeline-v2 work) - Telemetry server: SSE live tail, run-sessions enumeration, cost/token tracking, app.js UI rewrite with collapsible sections - MCP servers (mcp_ci_server, mcp_forgejo_server, mcp_git_server, mcp_handoff_server, mcp_graphify_server) for opencode worker context access - Live log writer (tools/live_log_writer.py) — SSE-streaming dispatcher event log - Tier-dispatcher escalation flow with prompts trimmed for budget - Shared bot-logins resolver (tools/_bot_logins.py) replacing two drift-prone copies - token_usage_audit.py for opencode cost analysis Tests - 2259 passing across 65 changed/new files - New suites: test_pr_state_cache, test_pr_state_warmer, test_pr_state_warmer_integration, test_pr_classification_cache, test_pr_list_cache_backoff, test_mcp_* (5 servers), test_live_log_writer_sse, test_telemetry_run_sessions, test_review_post_ready_label - Test_pr_comments_cache expanded with bot-filter coverage, cursor-normalization regression pins, format-drift, atomicity, failed-comments-not-stamped (silent-data-loss class) - Parametrized @_with_reheal coverage across 7 wrapped APIs - Real fault-inject atomicity test for chunked mark_vanished path via Connection wrapper class - Subprocess-based singleton flock test (cross-process contract) - Event-driven SIGTERM-mid-poll test (no fixed-sleep flake) Architecture notes - Schema v3 migration is additive (ALTER ADD COLUMN); v0/v1 still need destructive rebuild because pre-v2 column shape lacks owner/repo. Cross-process drop-table-ping-pong prevented by the fcntl migration lock + per-process _initialized flag. - Comments-refresh deferral is persistent via comments_refreshed_updated_at column — survives warmer restart, picks up next cycle even if PR didn't change again. Replaces in-memory changed_numbers list. - Rollback path: PR_STATE_WARMER_PREFER=0 bypasses the warmer cache and reverts to live-fetch behavior. PR_STATE_CACHE_DISABLE=1 short-circuits the warmer process at startup. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
dc4e9368f2 |
feat(auto-agents): block-store substrate + DRY refactors
Externalise large prompt sections (PR diff, comments, CI failure
logs, reviews, linked issues) into a cross-process SQLite-backed
block store so the worker can recover original content when an
intermediate tier-* agent's summarisation strips inline sections.
New substrate
-------------
- ``tools/_block_store.py`` — SQLite WAL, per-row 1MB cap, 1h TTL,
janitor (startup + opportunistic per-hour in register()),
threading.Lock around the periodic-sweep gate.
- ``tools/_block_prompt.py`` — registration glue + ``## Available
blocks`` Markdown table renderer.
- ``tools/_prefetch_section.py`` — single-source-of-truth section
registry; collapses the duplicate ``_register_*_blocks`` helpers
the reviewer and implementer previously kept in lockstep.
- ``tools/mcp_block_store_server.py`` — FastMCP wrapper exposing
``block_fetch`` / ``block_list`` / ``block_register`` /
``block_invalidate`` to agents. Uses the expanded ``_mcp_common``
helpers.
- ``tools/_implementer_escalation_helpers.py`` — pure helpers
extracted from ``dispatch_implementer.py`` (~180 lines off the
3284-line file); takes ``claim_runtime`` as a kwarg for clean DI.
DRY refactors
-------------
- ``tools/_backoff.py`` — shared ``Backoff`` dataclass collapses the
three near-identical exponential-backoff state machines in
``_pr_comments_cache``, ``_ci_logs``, ``_pr_classification_cache``.
- ``tools/_mcp_common.py`` — expanded with ``bootstrap_loader``,
``error_envelope``, ``make_main`` so each MCP server's prelude is
three lines.
- ``tools/_pr_diff.build_diff_section_full`` — returns a 4-tuple
including the raw diff body so the reviewer's block-store
registration reuses the bytes instead of doing a second HTTP fetch.
Wiring
------
- ``_review_prompt.build_review_prompt`` builds a ``PrefetchSection``
registry via ``_review_sections``, registers them, and renders the
``## Available blocks`` table at the end of the prompt.
- ``_implementer_prefetch._fetch_pr_context`` /
``fetch_new_issue_context`` build the equivalent registry via
``_implementer_sections`` and stamp ``result.block_refs`` for the
prompt builder to read.
- ``_implementer_prompt`` builders include
``_build_available_blocks_section(result)`` in all three flows.
- ``dispatch_review.main`` + ``dispatch_implementer.main`` call
``_block_store.janitor()`` at startup; the per-call opportunistic
janitor in ``register()`` keeps the file bounded between restarts.
Agent contract updates
----------------------
- ``.opencode/agents/task-implementor.md`` +
``.opencode/agents/pr-review-worker.md``:
- ``block_store*`` permission
- new "Block-store substrate" paragraph explaining
``block_fetch`` / ``block_list`` as the summarisation recovery
path.
Tests
-----
- ``test_block_store.py`` — 48 tests pinning every public contract
(register/fetch/list/invalidate/janitor, key whitelist, TTL,
size cap, WAL durability).
- ``test_block_prompt`` — covered transitively via the e2e test.
- ``test_block_store_recovery_e2e.py`` — builds a real prompt via
``build_pr_fix_prompt``, applies a heading-bounded summariser stub
(``_summarise_inline_sections``), asserts inline content is stripped
yet block keys survive and ``block_fetch`` recovers original content.
Plus a ``block_list`` fallback test for the worst case where the
table itself was summarised away.
- ``test_mcp_block_store_server.py`` — 27 wrapper-contract tests.
- ``test_mcp_block_store_transport.py`` — spawns the actual server
subprocess via ``mcp.client.stdio`` and exercises the JSON-RPC
transport round-trip in ~1.5s. Catches FastMCP schema /
serialisation bugs the in-process tests miss.
- ``test_backoff.py`` — 14 behaviour-focused tests of the shared
``Backoff`` curve.
- ``test_pr_comments_cache.py`` + ``test_ci_logs.py`` — deleted the
now-redundant ``TestComputeNextAttemptAfter`` / ``TestBackoffActive``
/ ``TestBackoffHelpers`` classes; ``test_backoff`` covers them.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
355af84fb1 |
refactor(auto-agents): hard-switch supervisor decommission + implementer parity
Combines the 2026-05-09 hard-switch decommissioning of the LLM
implementation/pr-review supervisors with the Phase 2/3/4/5b
implementer parity work (prefetch + preclone + telemetry + operator-
status comments) and the third/fourth-round critique cleanup.
Removed
- .opencode/agents/implementation-supervisor.md (340 LoC)
- .opencode/agents/pr-review-supervisor.md (348 LoC)
- _dispatch_runtime.assert_no_legacy_supervisor +
detect_legacy_supervisor_sessions and the SUPERVISOR_TAGS /
SUPERVISOR_OVERRIDE_ENV plumbing in both dispatchers, along with
the five supervisor-coexistence tests in test_dispatch_runtime.py
- _watchdog_helpers.parse_truthy_env + watchdog_check.py
--check-env mode + their dedicated unit tests (the legacy
DISPATCHERS_RUNNING gate had no callers after the watchdog
rewrite became unconditional)
Added
- tools/_implementer_prefetch.py — pre-dispatch Forgejo fetches
(PR/issue body, diff, CI status, comments, reviews, linked
issues, Epic) per work group
- tools/_implementer_prompt.py — pure-function prompt assembly
with UNTRUSTED CONTENT fences and shared
PR_COMPLIANCE_CHECKLIST / OUTPUT_CONTRACT
- tools/_phase4_telemetry.py — extractor + JSONL sink for the
Phase 4 plan metrics
- tools/_status_comments.py — per-fingerprint operator-status
comment substrate, namespaced for reviewer + implementer
- _dispatch_runtime.SessionContext dataclass + SIGTERM/SIGINT
cooperative claim release with synchronous handler
- TestSupervisorAgentsDecommissioned and
TestAutoAgentsMdIsWatchdogOnly anti-regression lints (glob over
*supervisor*.md in .opencode/agents/, plus body keyword bans
and bash allow-list lint)
- pyproject.toml `slow` marker registration for the subprocess
SIGTERM smoke test
- tests/auto_agents/fixtures/{phase4-acceptance.yaml,
phase4-session-output-sample.txt}
Rewritten
- .opencode/agents/auto-agents.md from supervisor-fleet manager
(~545 LoC) to dispatcher heartbeat watchdog (~184 LoC); host
init system / process manager (systemd / runit / docker) is now
the explicit restart authority instead of "host-level process
supervisor"
- AGENTS.md production-launch story (Shells A-D) reflects the
deterministic-Python orchestration boundary; the bot-identity
fork-mode paragraph reads from FORGEJO_OWNER / FORGEJO_REPO
env vars instead of the deleted hard-coded supervisor flags
- tools/launch_fork.sh header documents three host-level entry
points (dispatchers-launcher.sh, opencode-builder.sh,
merge_drive.py)
- worker self-descriptions (implementation-worker.md,
pr-review-worker.md) refer to the dispatcher / merge driver
instead of the deleted supervisors; session-health-quick-util.md
and async-agent-util.md treat -SUP-suffixed sessions as
flag-and-escalate signals
Tests: 1006 passed, 3 skipped, 0 failed under tests/auto_agents/.
Lint: zero new ruff errors on touched files; three pre-existing
errors in tools/_pr_diff.py at lines blamed to 2026-05-07.
Operator note: the only in-process rollback knob for prefetch
issues is IMPLEMENTER_DISPATCHER_PREFETCH=0 (and the matching
IMPLEMENTER_DISPATCHER_PRECLONE=0). Anything beyond that is git
revert of this commit. Residual doc surface in the
auto-agents-system and supervised-workers skill READMEs is
documentation-only; the agent files those READMEs reference no
longer exist.
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
drew
|
f3b10a5e72 |
refactor(auto-agents): rename review clone/diff substrate for implementer parity
Phase 0 of the implementer-parity plan: rename the formerly review-only
pre-clone + diff-fetch helpers so the implementer dispatcher can share
them in upcoming phases without forking the substrate or hauling
review-specific naming into a different work-group registry.
Renames (git rename-detected):
- tools/_review_clone.py -> tools/_pr_clone.py
- tools/_review_diff.py -> tools/_pr_diff.py
Extractions (new shared modules):
- tools/_pr_prompt.py UNTRUSTED-CONTENT marker helpers
(fence_markers, redact_marker,
wrap_untrusted_section) for Phase 2
pre-fetch consumers.
- tools/_commit_lint.py lint_commit_message + CONVENTIONAL_TYPES
+ _bot_committer_email so both
self-validation CLIs reuse one lint.
- tools/_validate_cli_common.py DiffResult / DiffErrorKind /
diff_from_worktree /
commit_from_worktree / _excerpt_*
/ resolve_base_ref / emit_error so
implementer_validate (Phase 1) does
not duplicate ~280 lines of CLI
plumbing.
Slim:
- tools/_review_validate_helpers.py shrinks 499 -> 223 lines and now
owns only review-specific validate_position_in_diff +
draft_strict_checks. Re-exports the moved helpers so existing test
monkeypatches (helpers.diff_from_worktree, helpers.subprocess) keep
working without churn.
API surface change:
- prepare_pr_worktree(cfg, n, sha, *, kind="review") -- back-compat
default; implementer dispatcher passes kind="implementer" in Phase 3.
- _worktree_base / _is_preclone_disabled now consult per-kind env
vars (REVIEW_DISPATCHER_WORKTREE_BASE vs.
IMPLEMENTER_DISPATCHER_WORKTREE_BASE; matching DISABLE_PRECLONE
toggles). Mirror is shared per repo regardless of kind.
- WorktreeHandle gains a `kind` field (defaulted) so cleanup paths
can discriminate.
Bug fix discovered along the way:
- _validate_cli_common.emit_error froze stream=sys.stdout at
function-definition time, which made pytest's capsys invisible to
the JSON error output. Now resolves sys.stdout at call time.
Tests:
- New tests/auto_agents/test_shared_substrate.py (14 tests) covers:
every public symbol the legacy modules exposed, helper-re-export
identity preservation, per-kind worktree-base + disable-toggle
semantics, the kind back-compat default, and the new _pr_prompt
fence/redact helpers. Also guards against the deleted
_review_clone.py / _review_diff.py reappearing on disk.
- All 24 importing call-sites updated; full reviewer test suite
remains green (637 passed, 3 skipped).
- `git grep '_review_clone\|_review_diff' tools/` returns zero
matches, the plan's Phase 0 exit criterion.
Sets up Phase 1 (implementer-helpers skill), Phase 2 (pre-fetch
parity), and Phase 3 (pre-cloned implementer worktrees) with no
shared reviewer-only code paths.
ISSUES CLOSED: #N/A
Co-authored-by: Cursor <cursoragent@cursor.com>
|