dc4e9368f2
Externalise large prompt sections (PR diff, comments, CI failure
logs, reviews, linked issues) into a cross-process SQLite-backed
block store so the worker can recover original content when an
intermediate tier-* agent's summarisation strips inline sections.
New substrate
-------------
- ``tools/_block_store.py`` — SQLite WAL, per-row 1MB cap, 1h TTL,
janitor (startup + opportunistic per-hour in register()),
threading.Lock around the periodic-sweep gate.
- ``tools/_block_prompt.py`` — registration glue + ``## Available
blocks`` Markdown table renderer.
- ``tools/_prefetch_section.py`` — single-source-of-truth section
registry; collapses the duplicate ``_register_*_blocks`` helpers
the reviewer and implementer previously kept in lockstep.
- ``tools/mcp_block_store_server.py`` — FastMCP wrapper exposing
``block_fetch`` / ``block_list`` / ``block_register`` /
``block_invalidate`` to agents. Uses the expanded ``_mcp_common``
helpers.
- ``tools/_implementer_escalation_helpers.py`` — pure helpers
extracted from ``dispatch_implementer.py`` (~180 lines off the
3284-line file); takes ``claim_runtime`` as a kwarg for clean DI.
DRY refactors
-------------
- ``tools/_backoff.py`` — shared ``Backoff`` dataclass collapses the
three near-identical exponential-backoff state machines in
``_pr_comments_cache``, ``_ci_logs``, ``_pr_classification_cache``.
- ``tools/_mcp_common.py`` — expanded with ``bootstrap_loader``,
``error_envelope``, ``make_main`` so each MCP server's prelude is
three lines.
- ``tools/_pr_diff.build_diff_section_full`` — returns a 4-tuple
including the raw diff body so the reviewer's block-store
registration reuses the bytes instead of doing a second HTTP fetch.
Wiring
------
- ``_review_prompt.build_review_prompt`` builds a ``PrefetchSection``
registry via ``_review_sections``, registers them, and renders the
``## Available blocks`` table at the end of the prompt.
- ``_implementer_prefetch._fetch_pr_context`` /
``fetch_new_issue_context`` build the equivalent registry via
``_implementer_sections`` and stamp ``result.block_refs`` for the
prompt builder to read.
- ``_implementer_prompt`` builders include
``_build_available_blocks_section(result)`` in all three flows.
- ``dispatch_review.main`` + ``dispatch_implementer.main`` call
``_block_store.janitor()`` at startup; the per-call opportunistic
janitor in ``register()`` keeps the file bounded between restarts.
Agent contract updates
----------------------
- ``.opencode/agents/task-implementor.md`` +
``.opencode/agents/pr-review-worker.md``:
- ``block_store*`` permission
- new "Block-store substrate" paragraph explaining
``block_fetch`` / ``block_list`` as the summarisation recovery
path.
Tests
-----
- ``test_block_store.py`` — 48 tests pinning every public contract
(register/fetch/list/invalidate/janitor, key whitelist, TTL,
size cap, WAL durability).
- ``test_block_prompt`` — covered transitively via the e2e test.
- ``test_block_store_recovery_e2e.py`` — builds a real prompt via
``build_pr_fix_prompt``, applies a heading-bounded summariser stub
(``_summarise_inline_sections``), asserts inline content is stripped
yet block keys survive and ``block_fetch`` recovers original content.
Plus a ``block_list`` fallback test for the worst case where the
table itself was summarised away.
- ``test_mcp_block_store_server.py`` — 27 wrapper-contract tests.
- ``test_mcp_block_store_transport.py`` — spawns the actual server
subprocess via ``mcp.client.stdio`` and exercises the JSON-RPC
transport round-trip in ~1.5s. Catches FastMCP schema /
serialisation bugs the in-process tests miss.
- ``test_backoff.py`` — 14 behaviour-focused tests of the shared
``Backoff`` curve.
- ``test_pr_comments_cache.py`` + ``test_ci_logs.py`` — deleted the
now-redundant ``TestComputeNextAttemptAfter`` / ``TestBackoffActive``
/ ``TestBackoffHelpers`` classes; ``test_backoff`` covers them.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
171 lines
6.9 KiB
Python
171 lines
6.9 KiB
Python
"""Shared helpers for the project's MCP servers.
|
|
|
|
Each ``tools/mcp_*_server.py`` is a per-server process (per the design
|
|
in ``docs/development/...`` — graphify, ci, forgejo, git, handoff,
|
|
block_store), but they all share:
|
|
|
|
1. **Forgejo runtime context.** The ``_claim_runtime`` helpers expect
|
|
a ``RuntimeContext`` protocol object with ``token``,
|
|
``request_timeout_s``, ``api_retries``, ``claim_ttl_seconds``,
|
|
``owner``, ``repo``. Three MCPs (``ci``, ``forgejo``, ``git``) all
|
|
need to build one from env. Factored out so a PAT rotation or
|
|
default-tweak lands in one place.
|
|
|
|
2. **Identity selection (HAL9000 vs HAL9001).** Most write operations
|
|
use the worker identity (``FORGEJO_PAT`` — HAL9000), but the
|
|
reviewer's formal review submissions and umbrella-PR approvals use
|
|
the reviewer identity (``FORGEJO_REVIEWER_PAT`` — HAL9001). Passing
|
|
``reviewer=True`` to :class:`ForgejoCfg` produces a context with
|
|
the HAL9001 token while leaving owner/repo/timeouts alone.
|
|
|
|
3. **Sibling-module loader bootstrap.** Every MCP server inserts
|
|
``tools/`` into ``sys.path`` and imports ``load_sibling`` to load
|
|
non-package siblings (``_claim_runtime``, ``_review_fetch``, etc.).
|
|
:func:`bootstrap_loader` does the path insert + import in one call.
|
|
|
|
4. **Error envelope shape.** Tools return ``{"error": str, **fields}``
|
|
so the agent's parser branches once on a single key. Centralised
|
|
here as :func:`error_envelope`.
|
|
|
|
5. **Process entrypoint.** Every server's ``main()`` is the same
|
|
try/run/KeyboardInterrupt/Exception wrapper. :func:`make_main`
|
|
builds one bound to the server name for stderr labelling.
|
|
|
|
Kept deliberately small. If something only one MCP needs, it lives in
|
|
that MCP's file — not here.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
import sys
|
|
from pathlib import Path
|
|
from types import ModuleType
|
|
from typing import Any, Callable
|
|
|
|
|
|
# ─── Runtime tuning ────────────────────────────────────────────────
|
|
# Per-call HTTP timeout for Forgejo API requests issued from the MCP
|
|
# servers. The default matches what the dispatchers use for similar
|
|
# read paths; bump via env if a slow Forgejo causes spurious tool
|
|
# errors during a long worker cycle.
|
|
_FORGEJO_TIMEOUT_S = int(os.environ.get("MCP_FORGEJO_TIMEOUT_S", "15"))
|
|
_FORGEJO_RETRIES = int(os.environ.get("MCP_FORGEJO_RETRIES", "3"))
|
|
|
|
|
|
class ForgejoCfg:
|
|
"""Minimal RuntimeContext satisfying ``_claim_runtime``'s
|
|
protocol, plus the ``owner`` / ``repo`` fields the ``_review_*``
|
|
helpers read directly.
|
|
|
|
Constructed per-call (cheap — just env reads) so a PAT rotation
|
|
on the dispatcher side is picked up without restarting the MCP
|
|
server. The ``reviewer`` flag selects HAL9001's PAT instead of
|
|
HAL9000's — used by tools that act on behalf of the umbrella
|
|
approver identity (notably the reviewer worker's review-submit
|
|
path).
|
|
"""
|
|
|
|
def __init__(self, *, reviewer: bool = False) -> None:
|
|
if reviewer:
|
|
self.token = os.environ.get("FORGEJO_REVIEWER_PAT", "")
|
|
else:
|
|
self.token = os.environ.get("FORGEJO_PAT", "")
|
|
self.request_timeout_s = _FORGEJO_TIMEOUT_S
|
|
self.api_retries = _FORGEJO_RETRIES
|
|
# Unused by reads but required by the RuntimeContext protocol.
|
|
# Defaults match the implementer dispatcher's own claim_ttl.
|
|
self.claim_ttl_seconds = int(os.environ.get("MCP_CLAIM_TTL_S", "7200"))
|
|
# ``_review_*`` helpers read these directly off cfg rather
|
|
# than the module-level fallbacks in ``_claim_runtime``.
|
|
self.owner = os.environ.get("FORGEJO_OWNER", "cleveragents")
|
|
self.repo = os.environ.get("FORGEJO_REPO", "cleveragents-core")
|
|
|
|
|
|
def require_token(cfg: ForgejoCfg, identity_label: str) -> str | None:
|
|
"""Return ``None`` if ``cfg.token`` is set; otherwise an
|
|
error-message string the MCP tool can return verbatim to the
|
|
agent. Centralised so every tool prints the same actionable
|
|
message ("set X env var") instead of opaque HTTP-401 errors."""
|
|
if cfg.token:
|
|
return None
|
|
env_name = (
|
|
"FORGEJO_REVIEWER_PAT" if identity_label == "reviewer" else "FORGEJO_PAT"
|
|
)
|
|
return (
|
|
f"{env_name} not set in MCP environment "
|
|
f"(identity={identity_label}). The MCP server inherits env from "
|
|
"the OpenCode process that spawned it — restart OpenCode after "
|
|
"sourcing tools/launch_fork.sh."
|
|
)
|
|
|
|
|
|
# ─── Sibling-module loader ──────────────────────────────────────────
|
|
|
|
|
|
def bootstrap_loader() -> Callable[[str, str], ModuleType]:
|
|
"""Insert ``tools/`` into ``sys.path`` and return ``load_sibling``.
|
|
|
|
Every MCP server starts with the same 3-line dance:
|
|
|
|
sys.path.insert(0, str(Path(__file__).parent))
|
|
from _loader import load_sibling
|
|
|
|
Wrapped here so a server's prelude becomes one expression::
|
|
|
|
load_sibling = bootstrap_loader()
|
|
_claim_runtime = load_sibling("_claim_runtime", "_claim_runtime.py")
|
|
"""
|
|
tools_dir = str(Path(__file__).resolve().parent)
|
|
if tools_dir not in sys.path:
|
|
sys.path.insert(0, tools_dir)
|
|
from _loader import load_sibling # noqa: E402 — deliberately late
|
|
return load_sibling
|
|
|
|
|
|
# ─── Error envelope ────────────────────────────────────────────────
|
|
|
|
|
|
def error_envelope(error: str, **fields: Any) -> dict[str, Any]:
|
|
"""Uniform error envelope shape used by every MCP tool.
|
|
|
|
Tools return ``{"error": str, **fields}`` so the agent's parser
|
|
branches once on the ``error`` key. Tools that return collections
|
|
on success include the empty collection in ``fields`` so the
|
|
agent's parsing code stays the same on error and success.
|
|
"""
|
|
return {"error": error, **fields}
|
|
|
|
|
|
# ─── Process entrypoint ────────────────────────────────────────────
|
|
|
|
|
|
def make_main(server: Any, server_name: str) -> Callable[[], int]:
|
|
"""Build a ``main()`` for an MCP server bound to ``server``.
|
|
|
|
Returns a zero-arg function suitable for ``sys.exit(main())``.
|
|
Same shape every MCP used to inline: KeyboardInterrupt → 0,
|
|
any other Exception → log to stderr + return 1.
|
|
|
|
``server_name`` appears in the stderr fatal line; OpenCode logs
|
|
that to its server output so operators can tell which MCP died.
|
|
"""
|
|
def main() -> int:
|
|
try:
|
|
server.run()
|
|
except KeyboardInterrupt:
|
|
return 0
|
|
except Exception as exc: # noqa: BLE001 — top-of-process catch
|
|
print(f"{server_name}: fatal: {exc!r}", file=sys.stderr)
|
|
return 1
|
|
return 0
|
|
return main
|
|
|
|
|
|
__all__ = (
|
|
"ForgejoCfg",
|
|
"bootstrap_loader",
|
|
"error_envelope",
|
|
"make_main",
|
|
"require_token",
|
|
)
|