dc4e9368f2
Externalise large prompt sections (PR diff, comments, CI failure
logs, reviews, linked issues) into a cross-process SQLite-backed
block store so the worker can recover original content when an
intermediate tier-* agent's summarisation strips inline sections.
New substrate
-------------
- ``tools/_block_store.py`` — SQLite WAL, per-row 1MB cap, 1h TTL,
janitor (startup + opportunistic per-hour in register()),
threading.Lock around the periodic-sweep gate.
- ``tools/_block_prompt.py`` — registration glue + ``## Available
blocks`` Markdown table renderer.
- ``tools/_prefetch_section.py`` — single-source-of-truth section
registry; collapses the duplicate ``_register_*_blocks`` helpers
the reviewer and implementer previously kept in lockstep.
- ``tools/mcp_block_store_server.py`` — FastMCP wrapper exposing
``block_fetch`` / ``block_list`` / ``block_register`` /
``block_invalidate`` to agents. Uses the expanded ``_mcp_common``
helpers.
- ``tools/_implementer_escalation_helpers.py`` — pure helpers
extracted from ``dispatch_implementer.py`` (~180 lines off the
3284-line file); takes ``claim_runtime`` as a kwarg for clean DI.
DRY refactors
-------------
- ``tools/_backoff.py`` — shared ``Backoff`` dataclass collapses the
three near-identical exponential-backoff state machines in
``_pr_comments_cache``, ``_ci_logs``, ``_pr_classification_cache``.
- ``tools/_mcp_common.py`` — expanded with ``bootstrap_loader``,
``error_envelope``, ``make_main`` so each MCP server's prelude is
three lines.
- ``tools/_pr_diff.build_diff_section_full`` — returns a 4-tuple
including the raw diff body so the reviewer's block-store
registration reuses the bytes instead of doing a second HTTP fetch.
Wiring
------
- ``_review_prompt.build_review_prompt`` builds a ``PrefetchSection``
registry via ``_review_sections``, registers them, and renders the
``## Available blocks`` table at the end of the prompt.
- ``_implementer_prefetch._fetch_pr_context`` /
``fetch_new_issue_context`` build the equivalent registry via
``_implementer_sections`` and stamp ``result.block_refs`` for the
prompt builder to read.
- ``_implementer_prompt`` builders include
``_build_available_blocks_section(result)`` in all three flows.
- ``dispatch_review.main`` + ``dispatch_implementer.main`` call
``_block_store.janitor()`` at startup; the per-call opportunistic
janitor in ``register()`` keeps the file bounded between restarts.
Agent contract updates
----------------------
- ``.opencode/agents/task-implementor.md`` +
``.opencode/agents/pr-review-worker.md``:
- ``block_store*`` permission
- new "Block-store substrate" paragraph explaining
``block_fetch`` / ``block_list`` as the summarisation recovery
path.
Tests
-----
- ``test_block_store.py`` — 48 tests pinning every public contract
(register/fetch/list/invalidate/janitor, key whitelist, TTL,
size cap, WAL durability).
- ``test_block_prompt`` — covered transitively via the e2e test.
- ``test_block_store_recovery_e2e.py`` — builds a real prompt via
``build_pr_fix_prompt``, applies a heading-bounded summariser stub
(``_summarise_inline_sections``), asserts inline content is stripped
yet block keys survive and ``block_fetch`` recovers original content.
Plus a ``block_list`` fallback test for the worst case where the
table itself was summarised away.
- ``test_mcp_block_store_server.py`` — 27 wrapper-contract tests.
- ``test_mcp_block_store_transport.py`` — spawns the actual server
subprocess via ``mcp.client.stdio`` and exercises the JSON-RPC
transport round-trip in ~1.5s. Catches FastMCP schema /
serialisation bugs the in-process tests miss.
- ``test_backoff.py`` — 14 behaviour-focused tests of the shared
``Backoff`` curve.
- ``test_pr_comments_cache.py`` + ``test_ci_logs.py`` — deleted the
now-redundant ``TestComputeNextAttemptAfter`` / ``TestBackoffActive``
/ ``TestBackoffHelpers`` classes; ``test_backoff`` covers them.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
173 lines
6.2 KiB
Python
173 lines
6.2 KiB
Python
"""Pure helpers extracted from :mod:`dispatch_implementer`'s
|
|
in-cycle tier escalation loop.
|
|
|
|
The orchestrator (``_post_session_action_with_escalation``) stays in
|
|
the dispatcher because it has tight coupling to telemetry, post-
|
|
session action wiring, and the WORK_GROUPS registry. The functions
|
|
here have no such coupling — they're side-effecting OS / git /
|
|
HTTP calls or pure data transforms — so they extract cleanly and
|
|
each gains its own test surface in :mod:`tests.auto_agents.test_dispatch_implementer`.
|
|
|
|
The dispatcher imports the public names below and delegates verbatim.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
import os
|
|
import subprocess
|
|
import sys
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
_TOOLS_DIR = str(Path(__file__).resolve().parent)
|
|
if _TOOLS_DIR not in sys.path:
|
|
sys.path.insert(0, _TOOLS_DIR)
|
|
from _loader import ( # noqa: E402 type: ignore[import-not-found]
|
|
load_sibling as _load_sibling,
|
|
)
|
|
|
|
_logger = logging.getLogger("implementer_escalation_helpers")
|
|
|
|
|
|
def fetch_pr_state(cfg: Any, pr_number: int, *, claim_runtime: Any) -> str:
|
|
"""GET the PR state. Returns ``"open"`` / ``"closed"`` / ``"merged"``.
|
|
Conservative fallback to ``"open"`` on any failure so the escalation
|
|
loop doesn't spuriously end a cycle on a transient blip.
|
|
|
|
``claim_runtime`` is the HTTP shim module the caller is using; the
|
|
dispatcher passes its own reference so tests that fresh-reload
|
|
``_claim_runtime`` see their fresh instance, not whatever was
|
|
cached when this helper module first imported."""
|
|
if cfg.dry_run or pr_number <= 0:
|
|
return "open"
|
|
try:
|
|
response = claim_runtime.get(
|
|
f"/repos/{cfg.owner}/{cfg.repo}/pulls/{int(pr_number)}", cfg
|
|
)
|
|
except (OSError, ValueError) as exc:
|
|
_logger.warning(
|
|
"PR state fetch failed for #%s (treating as 'open'): %s",
|
|
pr_number, exc,
|
|
)
|
|
return "open"
|
|
if int(response.get("status") or 0) != 200:
|
|
return "open"
|
|
body = response.get("body") or {}
|
|
if not isinstance(body, dict):
|
|
return "open"
|
|
state = body.get("state")
|
|
if isinstance(state, str) and state:
|
|
# Forgejo exposes "merged" via the `merged` boolean on
|
|
# state=closed PRs; honour that distinction.
|
|
if state == "closed" and body.get("merged") is True:
|
|
return "merged"
|
|
return state
|
|
return "open"
|
|
|
|
|
|
def reset_worktree_to_pinned_sha(handle: Any) -> bool:
|
|
"""Reset the pre-cloned worktree to the SHA captured at prefetch
|
|
time. Returns ``True`` on success, ``False`` on any failure
|
|
(logged WARNING; caller continues — the next session's discover
|
|
falls through to ``git-isolator-util``).
|
|
|
|
Defends against worker-side state corruption: missing worktree
|
|
dir (the worker may have ``rm -rf``'d it), stale ``.git/*.lock``
|
|
files from SIGKILL'd worker git ops.
|
|
"""
|
|
if handle is None:
|
|
return False
|
|
path = getattr(handle, "path", None)
|
|
pinned_sha = getattr(handle, "head_sha", None)
|
|
if not path or not pinned_sha:
|
|
return False
|
|
sha_short = str(pinned_sha)[:12]
|
|
if not Path(path).is_dir():
|
|
_logger.warning(
|
|
"worktree disappeared before reset for SHA %s at %s — "
|
|
"the prior worker session likely deleted it. Escalation "
|
|
"will continue; the next session's "
|
|
"``implementer-workspace.py discover`` will fall through "
|
|
"to ``git-isolator-util`` for a fresh clone.",
|
|
sha_short, path,
|
|
)
|
|
return False
|
|
|
|
try:
|
|
git_dir = Path(path) / ".git"
|
|
if git_dir.is_dir():
|
|
for lock in git_dir.glob("*.lock"):
|
|
if lock.is_file():
|
|
lock.unlink()
|
|
_logger.info(
|
|
"removed stale .git/%s at %s before reset",
|
|
lock.name, path,
|
|
)
|
|
except OSError:
|
|
pass
|
|
|
|
try:
|
|
subprocess.run(
|
|
["git", "-C", str(path), "reset", "--hard", str(pinned_sha)],
|
|
check=True, capture_output=True, timeout=30, text=True,
|
|
)
|
|
subprocess.run(
|
|
["git", "-C", str(path), "clean", "-xfdq"],
|
|
check=True, capture_output=True, timeout=30, text=True,
|
|
)
|
|
return True
|
|
except subprocess.CalledProcessError as exc:
|
|
# CalledProcessError.__str__ shows only exit code; include
|
|
# captured stderr so the operator can diagnose (vanished SHA
|
|
# vs. lock contention vs. permissions).
|
|
_logger.warning(
|
|
"worktree reset to pinned SHA %s failed at %s: "
|
|
"exit=%s stderr=%r; escalation continues",
|
|
sha_short, path, exc.returncode,
|
|
(exc.stderr or "").strip()[:400],
|
|
)
|
|
return False
|
|
except subprocess.SubprocessError as exc:
|
|
_logger.warning(
|
|
"worktree reset to pinned SHA %s raised %s at %s; "
|
|
"escalation continues",
|
|
sha_short, type(exc).__name__, path,
|
|
)
|
|
return False
|
|
|
|
|
|
def per_tier_worker_timeout(cfg: Any, tier: int) -> int:
|
|
"""Resolve the worker timeout for ``tier``. Per-tier override via
|
|
``IMPLEMENTER_DISPATCHER_WORKER_TIMEOUT_TIER_{N}_SECONDS``; falls
|
|
back to ``cfg.worker_timeout_seconds``.
|
|
|
|
Higher tiers historically need more wallclock (tier-2 escalation
|
|
can legitimately run for hours); operators tune this per-slot,
|
|
not per-model, so it stays correct across model swaps."""
|
|
env_name = f"IMPLEMENTER_DISPATCHER_WORKER_TIMEOUT_TIER_{int(tier)}_SECONDS"
|
|
raw = os.environ.get(env_name)
|
|
if raw:
|
|
try:
|
|
return max(60, int(raw))
|
|
except ValueError:
|
|
_logger.warning(
|
|
"%s=%r is not an int; falling back to global timeout",
|
|
env_name, raw,
|
|
)
|
|
return int(cfg.worker_timeout_seconds)
|
|
|
|
|
|
def terminal_state_from_session(session: Any) -> str:
|
|
"""Mirror dispatch_one's terminal_state derivation: ``completed``
|
|
for the happy path, raw ``status`` otherwise (``timeout`` /
|
|
``transport-error``)."""
|
|
return "completed" if session.status == "completed" else session.status
|
|
|
|
|
|
__all__ = (
|
|
"fetch_pr_state",
|
|
"per_tier_worker_timeout",
|
|
"reset_worktree_to_pinned_sha",
|
|
"terminal_state_from_session",
|
|
)
|