drew
dc4e9368f2
Externalise large prompt sections (PR diff, comments, CI failure
logs, reviews, linked issues) into a cross-process SQLite-backed
block store so the worker can recover original content when an
intermediate tier-* agent's summarisation strips inline sections.
New substrate
-------------
- ``tools/_block_store.py`` — SQLite WAL, per-row 1MB cap, 1h TTL,
janitor (startup + opportunistic per-hour in register()),
threading.Lock around the periodic-sweep gate.
- ``tools/_block_prompt.py`` — registration glue + ``## Available
blocks`` Markdown table renderer.
- ``tools/_prefetch_section.py`` — single-source-of-truth section
registry; collapses the duplicate ``_register_*_blocks`` helpers
the reviewer and implementer previously kept in lockstep.
- ``tools/mcp_block_store_server.py`` — FastMCP wrapper exposing
``block_fetch`` / ``block_list`` / ``block_register`` /
``block_invalidate`` to agents. Uses the expanded ``_mcp_common``
helpers.
- ``tools/_implementer_escalation_helpers.py`` — pure helpers
extracted from ``dispatch_implementer.py`` (~180 lines off the
3284-line file); takes ``claim_runtime`` as a kwarg for clean DI.
DRY refactors
-------------
- ``tools/_backoff.py`` — shared ``Backoff`` dataclass collapses the
three near-identical exponential-backoff state machines in
``_pr_comments_cache``, ``_ci_logs``, ``_pr_classification_cache``.
- ``tools/_mcp_common.py`` — expanded with ``bootstrap_loader``,
``error_envelope``, ``make_main`` so each MCP server's prelude is
three lines.
- ``tools/_pr_diff.build_diff_section_full`` — returns a 4-tuple
including the raw diff body so the reviewer's block-store
registration reuses the bytes instead of doing a second HTTP fetch.
Wiring
------
- ``_review_prompt.build_review_prompt`` builds a ``PrefetchSection``
registry via ``_review_sections``, registers them, and renders the
``## Available blocks`` table at the end of the prompt.
- ``_implementer_prefetch._fetch_pr_context`` /
``fetch_new_issue_context`` build the equivalent registry via
``_implementer_sections`` and stamp ``result.block_refs`` for the
prompt builder to read.
- ``_implementer_prompt`` builders include
``_build_available_blocks_section(result)`` in all three flows.
- ``dispatch_review.main`` + ``dispatch_implementer.main`` call
``_block_store.janitor()`` at startup; the per-call opportunistic
janitor in ``register()`` keeps the file bounded between restarts.
Agent contract updates
----------------------
- ``.opencode/agents/task-implementor.md`` +
``.opencode/agents/pr-review-worker.md``:
- ``block_store*`` permission
- new "Block-store substrate" paragraph explaining
``block_fetch`` / ``block_list`` as the summarisation recovery
path.
Tests
-----
- ``test_block_store.py`` — 48 tests pinning every public contract
(register/fetch/list/invalidate/janitor, key whitelist, TTL,
size cap, WAL durability).
- ``test_block_prompt`` — covered transitively via the e2e test.
- ``test_block_store_recovery_e2e.py`` — builds a real prompt via
``build_pr_fix_prompt``, applies a heading-bounded summariser stub
(``_summarise_inline_sections``), asserts inline content is stripped
yet block keys survive and ``block_fetch`` recovers original content.
Plus a ``block_list`` fallback test for the worst case where the
table itself was summarised away.
- ``test_mcp_block_store_server.py`` — 27 wrapper-contract tests.
- ``test_mcp_block_store_transport.py`` — spawns the actual server
subprocess via ``mcp.client.stdio`` and exercises the JSON-RPC
transport round-trip in ~1.5s. Catches FastMCP schema /
serialisation bugs the in-process tests miss.
- ``test_backoff.py`` — 14 behaviour-focused tests of the shared
``Backoff`` curve.
- ``test_pr_comments_cache.py`` + ``test_ci_logs.py`` — deleted the
now-redundant ``TestComputeNextAttemptAfter`` / ``TestBackoffActive``
/ ``TestBackoffHelpers`` classes; ``test_backoff`` covers them.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
90 lines
2.8 KiB
Python
90 lines
2.8 KiB
Python
"""Shared exponential-backoff state machine for the cache modules.
|
|
|
|
Three caches (:mod:`_pr_comments_cache`, :mod:`_ci_logs`,
|
|
:mod:`_pr_classification_cache`) previously each carried a copy of
|
|
the same ``next_attempt_after = now + min(BASE * 2**(failures-1), MAX)``
|
|
machinery. Centralised here so a future tweak (jitter, cap, etc.)
|
|
lands once.
|
|
|
|
Usage::
|
|
|
|
BACKOFF = Backoff(
|
|
base_env="CI_LOGS_BACKOFF_BASE_S",
|
|
max_env="CI_LOGS_BACKOFF_MAX_S",
|
|
base_default=60, max_default=1800,
|
|
)
|
|
next_iso = BACKOFF.next_attempt_after(failures=3, now_dt=now)
|
|
if BACKOFF.is_active(cached, now_dt=now): return cached, False
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import datetime as _dt
|
|
import os
|
|
from dataclasses import dataclass
|
|
from typing import Any
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class Backoff:
|
|
"""Bundle the four env-tunable knobs that govern one cache's
|
|
backoff. Frozen because instances are module-level constants;
|
|
each cache module instantiates one at import time."""
|
|
|
|
base_env: str
|
|
max_env: str
|
|
base_default: int = 60
|
|
max_default: int = 1800
|
|
|
|
def base_s(self) -> int:
|
|
return _read_positive_int_env(self.base_env, self.base_default)
|
|
|
|
def max_s(self) -> int:
|
|
return _read_positive_int_env(self.max_env, self.max_default)
|
|
|
|
def next_attempt_after(
|
|
self, failures: int, now_dt: _dt.datetime,
|
|
) -> str | None:
|
|
"""Return an ISO-8601 timestamp ``failures`` retries out, or
|
|
``None`` if ``failures <= 0`` (no backoff window required).
|
|
|
|
The exponent is clamped at 16 (2**16 ~ 18 hours of base*65536)
|
|
to avoid integer overflow on adversarial inputs; the final
|
|
delay is then capped at ``max_s``."""
|
|
if failures <= 0:
|
|
return None
|
|
raw_delay_s = self.base_s() * (2 ** min(failures - 1, 16))
|
|
delay_s = min(raw_delay_s, self.max_s())
|
|
return (now_dt + _dt.timedelta(seconds=delay_s)).isoformat()
|
|
|
|
def is_active(
|
|
self, cached: dict[str, Any] | None, now_dt: _dt.datetime,
|
|
) -> bool:
|
|
"""True iff ``cached["next_attempt_after"]`` parses as an ISO
|
|
datetime in the future. Returns ``False`` for missing /
|
|
malformed / past values so a corrupt cache row falls through
|
|
to a fresh live fetch rather than getting stuck in stale
|
|
backoff."""
|
|
if not cached:
|
|
return False
|
|
raw = cached.get("next_attempt_after")
|
|
if not raw:
|
|
return False
|
|
try:
|
|
deadline = _dt.datetime.fromisoformat(str(raw))
|
|
except (ValueError, TypeError):
|
|
return False
|
|
return now_dt < deadline
|
|
|
|
|
|
def _read_positive_int_env(name: str, default: int) -> int:
|
|
raw = os.environ.get(name)
|
|
if raw:
|
|
try:
|
|
return max(1, int(raw))
|
|
except ValueError:
|
|
pass
|
|
return default
|
|
|
|
|
|
__all__ = ("Backoff",)
|