Files
cleveragents-core/tools/_block_prompt.py
T
drew dc4e9368f2 feat(auto-agents): block-store substrate + DRY refactors
Externalise large prompt sections (PR diff, comments, CI failure
logs, reviews, linked issues) into a cross-process SQLite-backed
block store so the worker can recover original content when an
intermediate tier-* agent's summarisation strips inline sections.

New substrate
-------------

- ``tools/_block_store.py`` — SQLite WAL, per-row 1MB cap, 1h TTL,
  janitor (startup + opportunistic per-hour in register()),
  threading.Lock around the periodic-sweep gate.
- ``tools/_block_prompt.py`` — registration glue + ``## Available
  blocks`` Markdown table renderer.
- ``tools/_prefetch_section.py`` — single-source-of-truth section
  registry; collapses the duplicate ``_register_*_blocks`` helpers
  the reviewer and implementer previously kept in lockstep.
- ``tools/mcp_block_store_server.py`` — FastMCP wrapper exposing
  ``block_fetch`` / ``block_list`` / ``block_register`` /
  ``block_invalidate`` to agents. Uses the expanded ``_mcp_common``
  helpers.
- ``tools/_implementer_escalation_helpers.py`` — pure helpers
  extracted from ``dispatch_implementer.py`` (~180 lines off the
  3284-line file); takes ``claim_runtime`` as a kwarg for clean DI.

DRY refactors
-------------

- ``tools/_backoff.py`` — shared ``Backoff`` dataclass collapses the
  three near-identical exponential-backoff state machines in
  ``_pr_comments_cache``, ``_ci_logs``, ``_pr_classification_cache``.
- ``tools/_mcp_common.py`` — expanded with ``bootstrap_loader``,
  ``error_envelope``, ``make_main`` so each MCP server's prelude is
  three lines.
- ``tools/_pr_diff.build_diff_section_full`` — returns a 4-tuple
  including the raw diff body so the reviewer's block-store
  registration reuses the bytes instead of doing a second HTTP fetch.

Wiring
------

- ``_review_prompt.build_review_prompt`` builds a ``PrefetchSection``
  registry via ``_review_sections``, registers them, and renders the
  ``## Available blocks`` table at the end of the prompt.
- ``_implementer_prefetch._fetch_pr_context`` /
  ``fetch_new_issue_context`` build the equivalent registry via
  ``_implementer_sections`` and stamp ``result.block_refs`` for the
  prompt builder to read.
- ``_implementer_prompt`` builders include
  ``_build_available_blocks_section(result)`` in all three flows.
- ``dispatch_review.main`` + ``dispatch_implementer.main`` call
  ``_block_store.janitor()`` at startup; the per-call opportunistic
  janitor in ``register()`` keeps the file bounded between restarts.

Agent contract updates
----------------------

- ``.opencode/agents/task-implementor.md`` +
  ``.opencode/agents/pr-review-worker.md``:
  - ``block_store*`` permission
  - new "Block-store substrate" paragraph explaining
    ``block_fetch`` / ``block_list`` as the summarisation recovery
    path.

Tests
-----

- ``test_block_store.py`` — 48 tests pinning every public contract
  (register/fetch/list/invalidate/janitor, key whitelist, TTL,
  size cap, WAL durability).
- ``test_block_prompt`` — covered transitively via the e2e test.
- ``test_block_store_recovery_e2e.py`` — builds a real prompt via
  ``build_pr_fix_prompt``, applies a heading-bounded summariser stub
  (``_summarise_inline_sections``), asserts inline content is stripped
  yet block keys survive and ``block_fetch`` recovers original content.
  Plus a ``block_list`` fallback test for the worst case where the
  table itself was summarised away.
- ``test_mcp_block_store_server.py`` — 27 wrapper-contract tests.
- ``test_mcp_block_store_transport.py`` — spawns the actual server
  subprocess via ``mcp.client.stdio`` and exercises the JSON-RPC
  transport round-trip in ~1.5s. Catches FastMCP schema /
  serialisation bugs the in-process tests miss.
- ``test_backoff.py`` — 14 behaviour-focused tests of the shared
  ``Backoff`` curve.
- ``test_pr_comments_cache.py`` + ``test_ci_logs.py`` — deleted the
  now-redundant ``TestComputeNextAttemptAfter`` / ``TestBackoffActive``
  / ``TestBackoffHelpers`` classes; ``test_backoff`` covers them.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-16 22:55:52 -04:00

161 lines
5.4 KiB
Python

"""Prompt-side glue for the block store: register prefetched sections
and render the ``## Available blocks`` table both reviewer and
implementer prompts embed.
The renderer's output is part of the agent's reading contract; one
implementation keeps the format stable across both prompts.
Best-effort: registration failures (size cap, SQLite hiccup, disabled
env) log at WARN and drop the ref for that section — the inline
prompt content is still authoritative.
"""
from __future__ import annotations
import json
import logging
import sqlite3
import sys
from dataclasses import dataclass
from pathlib import Path
from typing import Any
_TOOLS_DIR = str(Path(__file__).resolve().parent)
if _TOOLS_DIR not in sys.path:
sys.path.insert(0, _TOOLS_DIR)
from _loader import ( # noqa: E402 type: ignore[import-not-found]
load_sibling as _load_sibling,
)
_block_store = _load_sibling("_block_store", "_block_store.py")
_logger = logging.getLogger("block_prompt")
@dataclass(frozen=True)
class BlockRef:
"""One row in the ``## Available blocks`` table.
``key`` is what the worker passes to ``block_fetch``. ``bytes`` lets
the worker decide if a round-trip is worth it. ``description`` is
one short human-readable line (e.g. ``"50 PR comments"``).
"""
key: str
block_type: str
bytes: int
description: str
def register_section_block(
*,
pr_number: int | None,
head_sha: str | None,
block_type: str,
content: str,
description: str,
ttl_s: int | None = None,
) -> BlockRef | None:
"""Register a section. ``None`` return is a soft skip — caller
falls back to the inline prompt section. ``pr_number=None`` for
issue-only work uses an ``issue-{block_type}-{sha}`` key shape."""
if not isinstance(content, str) or not content:
return None
try:
if pr_number is not None:
key = _block_store.make_pr_block_key(
pr_number, block_type, head_sha,
)
else:
safe = _block_store.sanitise_sha_suffix(head_sha)
sha_suffix = f"-{safe}" if safe else ""
key = f"issue-{block_type}{sha_suffix}"
except _block_store.BlockStoreError as exc:
_logger.warning(
"block key build failed (block_type=%s, pr=%s, sha=%s): %s",
block_type, pr_number, head_sha, exc,
)
return None
try:
info = _block_store.register(
key, content,
ttl_s=ttl_s,
source="dispatcher",
block_type=block_type,
pr_number=pr_number,
)
except _block_store.BlockStoreError as exc:
# Most common cause: content over the cap. WARN, return
# None, and let the prompt fall through to inline-only.
_logger.warning(
"block register failed (key=%s, bytes=%s): %s",
key, len(content.encode("utf-8")), exc,
)
return None
except (OSError, sqlite3.DatabaseError) as exc:
# SQLite I/O failure (disk full, locked, corrupt). Substrate
# is best-effort — log and let the prompt fall through to
# the inline section. Programmer errors (TypeError, KeyError)
# propagate so the test suite catches them.
_logger.warning(
"block register I/O failure (key=%s): %s: %s",
key, type(exc).__name__, exc,
)
return None
return BlockRef(
key=info["key"],
block_type=block_type,
bytes=int(info["bytes"]),
description=description,
)
def serialise_json_block(value: Any) -> str:
"""Canonical JSON encoding for block content: sorted keys, 2-space
indent, default str fallback. Centralised so dispatcher writes
and any test that needs the same shape stay consistent."""
return json.dumps(value, indent=2, sort_keys=True, default=str)
def render_available_blocks_section(refs: list[BlockRef]) -> str:
"""Render the ``## Available blocks`` table, or ``""`` if empty.
One row per block (~80 chars) so summarisation leaves keys intact."""
if not refs:
return ""
rows: list[str] = []
for ref in sorted(refs, key=lambda r: (r.block_type, r.key)):
rows.append(
f"| `{ref.key}` | {ref.block_type} | {ref.bytes} | "
f"{ref.description} |"
)
table = "| key | type | bytes | description |\n" \
"|-----|------|-------|-------------|\n" + "\n".join(rows)
return f"""## Available blocks (fetch via `block_store` MCP if summarised)
The dispatcher pre-fetched every section above AND registered the
content into the cross-process block store. If an intermediate agent
summarised any inline section before it reached you, call the
``block_store`` MCP's ``block_fetch`` tool with the key from the table
below — block_fetch always returns the dispatcher's original
content for the current cycle, regardless of what your prompt now
shows.
Block IDs survive summarisation; raw content does not. When the inline
section seems truncated or you cannot find a detail you expect to be
present (e.g. a specific failing assertion in the CI logs section),
fetch the block by key BEFORE giving up or invoking heavy fallbacks
like ``ci_run_local_gate``.
{table}
Default block TTL is 1 hour — if you've been working past that,
re-fetch may return ``error: block 'X' expired``; that's fine,
fall back to the inline section."""
__all__ = (
"BlockRef",
"register_section_block",
"render_available_blocks_section",
"serialise_json_block",
)