62d4e8f07d5cab70c3861eb8efd3ebdf64ef068c
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 drew
|
2658deee94 |
feat(auto-agents): PR State Warmer substrate + supporting infra
Adds a long-lived sidecar (pr_state_warmer.py) that polls Forgejo's /pulls endpoint every 30s and writes the full PR snapshot to a shared SQLite store, eliminating the dispatcher's per-cycle cold-cache stalls (24-30s rebuilds on flaky cycles) and the silent 50-PR pagination cap on the legacy single-page fetch. Substrate - tools/_pr_state_cache.py — SQLite store with (owner, repo) PK, WAL mode, additive v2→v3 migration (comments_refreshed_updated_at), bounded fcntl.flock migration lock, threading.Lock for per-process init, @_with_reheal decorator (catches OperationalError no-such- table + DatabaseError corruption with file quarantine), atomic TEMP-table chunking for >32k seen-set, _normalize_updated_at to canonicalize Forgejo tz-marker drift - tools/pr_state_warmer.py — poll/upsert/vanish/comments-refresh loop with fcntl.flock singleton (rejects second warmer), bounded comments-refresh cap, persistent deferral via SQL pending query, PermissionError-tolerant lock setup, cold-start log suppression - tools/_pr_classification_cache.py — three-layer fall-through (warmer cache → list cache → live fetch) with staleness gate (PR_STATE_WARMER_STALE_AFTER_S floored at 30s in prod) Comments cache hardening - Bot-filter at write time drops bot status/claim/release/sentinel while preserving **Implementation Attempt** markers (94.6% reduction on bot-heavy PRs like #30's 19k-comment thread) - _normalize_since_cursor strips microsecond precision before building ?since= query (fixes the live-observed Forgejo HTTP 422 bug on PRs #25 + #28); handles uppercase Z, lowercase z, ±HH:MM offsets (including non-zero like +05:30), naive ISO - Lazy migration of legacy null-key by_author entries on _read_cache - _newest_cursor walks tail-back skipping malformed entries Supporting infrastructure (cumulative dmpipeline-v2 work) - Telemetry server: SSE live tail, run-sessions enumeration, cost/token tracking, app.js UI rewrite with collapsible sections - MCP servers (mcp_ci_server, mcp_forgejo_server, mcp_git_server, mcp_handoff_server, mcp_graphify_server) for opencode worker context access - Live log writer (tools/live_log_writer.py) — SSE-streaming dispatcher event log - Tier-dispatcher escalation flow with prompts trimmed for budget - Shared bot-logins resolver (tools/_bot_logins.py) replacing two drift-prone copies - token_usage_audit.py for opencode cost analysis Tests - 2259 passing across 65 changed/new files - New suites: test_pr_state_cache, test_pr_state_warmer, test_pr_state_warmer_integration, test_pr_classification_cache, test_pr_list_cache_backoff, test_mcp_* (5 servers), test_live_log_writer_sse, test_telemetry_run_sessions, test_review_post_ready_label - Test_pr_comments_cache expanded with bot-filter coverage, cursor-normalization regression pins, format-drift, atomicity, failed-comments-not-stamped (silent-data-loss class) - Parametrized @_with_reheal coverage across 7 wrapped APIs - Real fault-inject atomicity test for chunked mark_vanished path via Connection wrapper class - Subprocess-based singleton flock test (cross-process contract) - Event-driven SIGTERM-mid-poll test (no fixed-sleep flake) Architecture notes - Schema v3 migration is additive (ALTER ADD COLUMN); v0/v1 still need destructive rebuild because pre-v2 column shape lacks owner/repo. Cross-process drop-table-ping-pong prevented by the fcntl migration lock + per-process _initialized flag. - Comments-refresh deferral is persistent via comments_refreshed_updated_at column — survives warmer restart, picks up next cycle even if PR didn't change again. Replaces in-memory changed_numbers list. - Rollback path: PR_STATE_WARMER_PREFER=0 bypasses the warmer cache and reverts to live-fetch behavior. PR_STATE_CACHE_DISABLE=1 short-circuits the warmer process at startup. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
dc4e9368f2 |
feat(auto-agents): block-store substrate + DRY refactors
Externalise large prompt sections (PR diff, comments, CI failure
logs, reviews, linked issues) into a cross-process SQLite-backed
block store so the worker can recover original content when an
intermediate tier-* agent's summarisation strips inline sections.
New substrate
-------------
- ``tools/_block_store.py`` — SQLite WAL, per-row 1MB cap, 1h TTL,
janitor (startup + opportunistic per-hour in register()),
threading.Lock around the periodic-sweep gate.
- ``tools/_block_prompt.py`` — registration glue + ``## Available
blocks`` Markdown table renderer.
- ``tools/_prefetch_section.py`` — single-source-of-truth section
registry; collapses the duplicate ``_register_*_blocks`` helpers
the reviewer and implementer previously kept in lockstep.
- ``tools/mcp_block_store_server.py`` — FastMCP wrapper exposing
``block_fetch`` / ``block_list`` / ``block_register`` /
``block_invalidate`` to agents. Uses the expanded ``_mcp_common``
helpers.
- ``tools/_implementer_escalation_helpers.py`` — pure helpers
extracted from ``dispatch_implementer.py`` (~180 lines off the
3284-line file); takes ``claim_runtime`` as a kwarg for clean DI.
DRY refactors
-------------
- ``tools/_backoff.py`` — shared ``Backoff`` dataclass collapses the
three near-identical exponential-backoff state machines in
``_pr_comments_cache``, ``_ci_logs``, ``_pr_classification_cache``.
- ``tools/_mcp_common.py`` — expanded with ``bootstrap_loader``,
``error_envelope``, ``make_main`` so each MCP server's prelude is
three lines.
- ``tools/_pr_diff.build_diff_section_full`` — returns a 4-tuple
including the raw diff body so the reviewer's block-store
registration reuses the bytes instead of doing a second HTTP fetch.
Wiring
------
- ``_review_prompt.build_review_prompt`` builds a ``PrefetchSection``
registry via ``_review_sections``, registers them, and renders the
``## Available blocks`` table at the end of the prompt.
- ``_implementer_prefetch._fetch_pr_context`` /
``fetch_new_issue_context`` build the equivalent registry via
``_implementer_sections`` and stamp ``result.block_refs`` for the
prompt builder to read.
- ``_implementer_prompt`` builders include
``_build_available_blocks_section(result)`` in all three flows.
- ``dispatch_review.main`` + ``dispatch_implementer.main`` call
``_block_store.janitor()`` at startup; the per-call opportunistic
janitor in ``register()`` keeps the file bounded between restarts.
Agent contract updates
----------------------
- ``.opencode/agents/task-implementor.md`` +
``.opencode/agents/pr-review-worker.md``:
- ``block_store*`` permission
- new "Block-store substrate" paragraph explaining
``block_fetch`` / ``block_list`` as the summarisation recovery
path.
Tests
-----
- ``test_block_store.py`` — 48 tests pinning every public contract
(register/fetch/list/invalidate/janitor, key whitelist, TTL,
size cap, WAL durability).
- ``test_block_prompt`` — covered transitively via the e2e test.
- ``test_block_store_recovery_e2e.py`` — builds a real prompt via
``build_pr_fix_prompt``, applies a heading-bounded summariser stub
(``_summarise_inline_sections``), asserts inline content is stripped
yet block keys survive and ``block_fetch`` recovers original content.
Plus a ``block_list`` fallback test for the worst case where the
table itself was summarised away.
- ``test_mcp_block_store_server.py`` — 27 wrapper-contract tests.
- ``test_mcp_block_store_transport.py`` — spawns the actual server
subprocess via ``mcp.client.stdio`` and exercises the JSON-RPC
transport round-trip in ~1.5s. Catches FastMCP schema /
serialisation bugs the in-process tests miss.
- ``test_backoff.py`` — 14 behaviour-focused tests of the shared
``Backoff`` curve.
- ``test_pr_comments_cache.py`` + ``test_ci_logs.py`` — deleted the
now-redundant ``TestComputeNextAttemptAfter`` / ``TestBackoffActive``
/ ``TestBackoffHelpers`` classes; ``test_backoff`` covers them.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
355af84fb1 |
refactor(auto-agents): hard-switch supervisor decommission + implementer parity
Combines the 2026-05-09 hard-switch decommissioning of the LLM
implementation/pr-review supervisors with the Phase 2/3/4/5b
implementer parity work (prefetch + preclone + telemetry + operator-
status comments) and the third/fourth-round critique cleanup.
Removed
- .opencode/agents/implementation-supervisor.md (340 LoC)
- .opencode/agents/pr-review-supervisor.md (348 LoC)
- _dispatch_runtime.assert_no_legacy_supervisor +
detect_legacy_supervisor_sessions and the SUPERVISOR_TAGS /
SUPERVISOR_OVERRIDE_ENV plumbing in both dispatchers, along with
the five supervisor-coexistence tests in test_dispatch_runtime.py
- _watchdog_helpers.parse_truthy_env + watchdog_check.py
--check-env mode + their dedicated unit tests (the legacy
DISPATCHERS_RUNNING gate had no callers after the watchdog
rewrite became unconditional)
Added
- tools/_implementer_prefetch.py — pre-dispatch Forgejo fetches
(PR/issue body, diff, CI status, comments, reviews, linked
issues, Epic) per work group
- tools/_implementer_prompt.py — pure-function prompt assembly
with UNTRUSTED CONTENT fences and shared
PR_COMPLIANCE_CHECKLIST / OUTPUT_CONTRACT
- tools/_phase4_telemetry.py — extractor + JSONL sink for the
Phase 4 plan metrics
- tools/_status_comments.py — per-fingerprint operator-status
comment substrate, namespaced for reviewer + implementer
- _dispatch_runtime.SessionContext dataclass + SIGTERM/SIGINT
cooperative claim release with synchronous handler
- TestSupervisorAgentsDecommissioned and
TestAutoAgentsMdIsWatchdogOnly anti-regression lints (glob over
*supervisor*.md in .opencode/agents/, plus body keyword bans
and bash allow-list lint)
- pyproject.toml `slow` marker registration for the subprocess
SIGTERM smoke test
- tests/auto_agents/fixtures/{phase4-acceptance.yaml,
phase4-session-output-sample.txt}
Rewritten
- .opencode/agents/auto-agents.md from supervisor-fleet manager
(~545 LoC) to dispatcher heartbeat watchdog (~184 LoC); host
init system / process manager (systemd / runit / docker) is now
the explicit restart authority instead of "host-level process
supervisor"
- AGENTS.md production-launch story (Shells A-D) reflects the
deterministic-Python orchestration boundary; the bot-identity
fork-mode paragraph reads from FORGEJO_OWNER / FORGEJO_REPO
env vars instead of the deleted hard-coded supervisor flags
- tools/launch_fork.sh header documents three host-level entry
points (dispatchers-launcher.sh, opencode-builder.sh,
merge_drive.py)
- worker self-descriptions (implementation-worker.md,
pr-review-worker.md) refer to the dispatcher / merge driver
instead of the deleted supervisors; session-health-quick-util.md
and async-agent-util.md treat -SUP-suffixed sessions as
flag-and-escalate signals
Tests: 1006 passed, 3 skipped, 0 failed under tests/auto_agents/.
Lint: zero new ruff errors on touched files; three pre-existing
errors in tools/_pr_diff.py at lines blamed to 2026-05-07.
Operator note: the only in-process rollback knob for prefetch
issues is IMPLEMENTER_DISPATCHER_PREFETCH=0 (and the matching
IMPLEMENTER_DISPATCHER_PRECLONE=0). Anything beyond that is git
revert of this commit. Residual doc surface in the
auto-agents-system and supervised-workers skill READMEs is
documentation-only; the agent files those READMEs reference no
longer exist.
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
drew
|
f3b10a5e72 |
refactor(auto-agents): rename review clone/diff substrate for implementer parity
Phase 0 of the implementer-parity plan: rename the formerly review-only
pre-clone + diff-fetch helpers so the implementer dispatcher can share
them in upcoming phases without forking the substrate or hauling
review-specific naming into a different work-group registry.
Renames (git rename-detected):
- tools/_review_clone.py -> tools/_pr_clone.py
- tools/_review_diff.py -> tools/_pr_diff.py
Extractions (new shared modules):
- tools/_pr_prompt.py UNTRUSTED-CONTENT marker helpers
(fence_markers, redact_marker,
wrap_untrusted_section) for Phase 2
pre-fetch consumers.
- tools/_commit_lint.py lint_commit_message + CONVENTIONAL_TYPES
+ _bot_committer_email so both
self-validation CLIs reuse one lint.
- tools/_validate_cli_common.py DiffResult / DiffErrorKind /
diff_from_worktree /
commit_from_worktree / _excerpt_*
/ resolve_base_ref / emit_error so
implementer_validate (Phase 1) does
not duplicate ~280 lines of CLI
plumbing.
Slim:
- tools/_review_validate_helpers.py shrinks 499 -> 223 lines and now
owns only review-specific validate_position_in_diff +
draft_strict_checks. Re-exports the moved helpers so existing test
monkeypatches (helpers.diff_from_worktree, helpers.subprocess) keep
working without churn.
API surface change:
- prepare_pr_worktree(cfg, n, sha, *, kind="review") -- back-compat
default; implementer dispatcher passes kind="implementer" in Phase 3.
- _worktree_base / _is_preclone_disabled now consult per-kind env
vars (REVIEW_DISPATCHER_WORKTREE_BASE vs.
IMPLEMENTER_DISPATCHER_WORKTREE_BASE; matching DISABLE_PRECLONE
toggles). Mirror is shared per repo regardless of kind.
- WorktreeHandle gains a `kind` field (defaulted) so cleanup paths
can discriminate.
Bug fix discovered along the way:
- _validate_cli_common.emit_error froze stream=sys.stdout at
function-definition time, which made pytest's capsys invisible to
the JSON error output. Now resolves sys.stdout at call time.
Tests:
- New tests/auto_agents/test_shared_substrate.py (14 tests) covers:
every public symbol the legacy modules exposed, helper-re-export
identity preservation, per-kind worktree-base + disable-toggle
semantics, the kind back-compat default, and the new _pr_prompt
fence/redact helpers. Also guards against the deleted
_review_clone.py / _review_diff.py reappearing on disk.
- All 24 importing call-sites updated; full reviewer test suite
remains green (637 passed, 3 skipped).
- `git grep '_review_clone\|_review_diff' tools/` returns zero
matches, the plan's Phase 0 exit criterion.
Sets up Phase 1 (implementer-helpers skill), Phase 2 (pre-fetch
parity), and Phase 3 (pre-cloned implementer worktrees) with no
shared reviewer-only code paths.
ISSUES CLOSED: #N/A
Co-authored-by: Cursor <cursoragent@cursor.com>
|