Files
cleveragents-core/tools/mcp_graphify_server.py
T
drew 2658deee94 feat(auto-agents): PR State Warmer substrate + supporting infra
Adds a long-lived sidecar (pr_state_warmer.py) that polls Forgejo's
/pulls endpoint every 30s and writes the full PR snapshot to a
shared SQLite store, eliminating the dispatcher's per-cycle
cold-cache stalls (24-30s rebuilds on flaky cycles) and the silent
50-PR pagination cap on the legacy single-page fetch.

Substrate
- tools/_pr_state_cache.py  — SQLite store with (owner, repo) PK,
  WAL mode, additive v2→v3 migration (comments_refreshed_updated_at),
  bounded fcntl.flock migration lock, threading.Lock for per-process
  init, @_with_reheal decorator (catches OperationalError no-such-
  table + DatabaseError corruption with file quarantine), atomic
  TEMP-table chunking for >32k seen-set, _normalize_updated_at to
  canonicalize Forgejo tz-marker drift
- tools/pr_state_warmer.py — poll/upsert/vanish/comments-refresh
  loop with fcntl.flock singleton (rejects second warmer), bounded
  comments-refresh cap, persistent deferral via SQL pending query,
  PermissionError-tolerant lock setup, cold-start log suppression
- tools/_pr_classification_cache.py — three-layer fall-through
  (warmer cache → list cache → live fetch) with staleness gate
  (PR_STATE_WARMER_STALE_AFTER_S floored at 30s in prod)

Comments cache hardening
- Bot-filter at write time drops bot status/claim/release/sentinel
  while preserving **Implementation Attempt** markers (94.6%
  reduction on bot-heavy PRs like #30's 19k-comment thread)
- _normalize_since_cursor strips microsecond precision before
  building ?since= query (fixes the live-observed Forgejo HTTP 422
  bug on PRs #25 + #28); handles uppercase Z, lowercase z, ±HH:MM
  offsets (including non-zero like +05:30), naive ISO
- Lazy migration of legacy null-key by_author entries on _read_cache
- _newest_cursor walks tail-back skipping malformed entries

Supporting infrastructure (cumulative dmpipeline-v2 work)
- Telemetry server: SSE live tail, run-sessions enumeration,
  cost/token tracking, app.js UI rewrite with collapsible sections
- MCP servers (mcp_ci_server, mcp_forgejo_server, mcp_git_server,
  mcp_handoff_server, mcp_graphify_server) for opencode worker
  context access
- Live log writer (tools/live_log_writer.py) — SSE-streaming
  dispatcher event log
- Tier-dispatcher escalation flow with prompts trimmed for budget
- Shared bot-logins resolver (tools/_bot_logins.py) replacing two
  drift-prone copies
- token_usage_audit.py for opencode cost analysis

Tests
- 2259 passing across 65 changed/new files
- New suites: test_pr_state_cache, test_pr_state_warmer,
  test_pr_state_warmer_integration, test_pr_classification_cache,
  test_pr_list_cache_backoff, test_mcp_* (5 servers),
  test_live_log_writer_sse, test_telemetry_run_sessions,
  test_review_post_ready_label
- Test_pr_comments_cache expanded with bot-filter coverage,
  cursor-normalization regression pins, format-drift, atomicity,
  failed-comments-not-stamped (silent-data-loss class)
- Parametrized @_with_reheal coverage across 7 wrapped APIs
- Real fault-inject atomicity test for chunked mark_vanished path
  via Connection wrapper class
- Subprocess-based singleton flock test (cross-process contract)
- Event-driven SIGTERM-mid-poll test (no fixed-sleep flake)

Architecture notes
- Schema v3 migration is additive (ALTER ADD COLUMN); v0/v1 still
  need destructive rebuild because pre-v2 column shape lacks
  owner/repo. Cross-process drop-table-ping-pong prevented by the
  fcntl migration lock + per-process _initialized flag.
- Comments-refresh deferral is persistent via
  comments_refreshed_updated_at column — survives warmer restart,
  picks up next cycle even if PR didn't change again. Replaces
  in-memory changed_numbers list.
- Rollback path: PR_STATE_WARMER_PREFER=0 bypasses the warmer
  cache and reverts to live-fetch behavior. PR_STATE_CACHE_DISABLE=1
  short-circuits the warmer process at startup.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 10:02:53 -04:00

239 lines
8.3 KiB
Python

#!/usr/bin/env python3
"""MCP server wrapping the local graphify code-knowledge-graph CLI.
Spawned by OpenCode via ``.opencode/opencode.json``'s ``mcp`` block
and exposed to agents that have ``mcp__graphify*: allow`` in their
permission block.
Replaces the v2 ``.opencode/plugins/graphify.js`` reminder plugin
(which broke deny-by-default bash allowlists by prepending
``echo "..." && `` to every command — see the plugin file's header
comment for the post-mortem) and the per-agent boilerplate that
allowed ``"graphify *"`` and granted ``external_directory`` read
access to ``graphify-out/``. Agents now need a single permission
line — ``"mcp__graphify*": allow`` — to use the graph.
Tools exposed
-------------
``report(head_lines: int = 200)``
Return the first ``head_lines`` lines of ``GRAPH_REPORT.md`` (god
nodes + community structure + cross-module summary). The default
matches the historical ``cat … | head -200`` pattern that the
task-implementor used to run on every session.
``query(question: str, budget: int = 2000)``
BFS traversal of the graph for a natural-language question.
Returns ``graphify``'s text output verbatim — typically node
citations with file:line references, ranked by graph distance.
``path(a: str, b: str)``
Shortest path between two concept nodes (e.g. between
``SessionContext`` and ``post_session_action``).
``explain(concept: str)``
Neighborhood summary for a single node.
State
-----
Stateless. Every call re-reads ``graphify-out/graph.json`` — which
``graphify update`` (run by the user / git commit hook) rewrites on
code changes. No caching here; the underlying CLI is fast and the
graph is small.
Configuration
-------------
``GRAPHIFY_OUT_DIR`` (default
``/home/drew/repos/cleveragents-core/graphify-out``)
Where ``graph.json`` and ``GRAPH_REPORT.md`` live.
``GRAPHIFY_BIN`` (default ``graphify`` on PATH)
The graphify CLI binary. Override if multiple installs exist.
``GRAPHIFY_TIMEOUT_S`` (default ``30``)
Per-call subprocess timeout in seconds. ``query`` and ``path``
can take a few seconds on larger graphs; the default leaves
headroom without letting a hung CLI stall a worker session.
"""
from __future__ import annotations
import os
import shutil
import subprocess
import sys
from pathlib import Path
from mcp.server.fastmcp import FastMCP
sys.path.insert(0, str(Path(__file__).parent))
from _mcp_common import make_main # noqa: E402
# ─── Configuration ─────────────────────────────────────────────────
GRAPHIFY_OUT = Path(
os.environ.get(
"GRAPHIFY_OUT_DIR",
"/home/drew/repos/cleveragents-core/graphify-out",
)
)
GRAPHIFY_BIN = os.environ.get("GRAPHIFY_BIN", "graphify")
GRAPHIFY_TIMEOUT_S = int(os.environ.get("GRAPHIFY_TIMEOUT_S", "30"))
GRAPH_JSON = GRAPHIFY_OUT / "graph.json"
GRAPH_REPORT = GRAPHIFY_OUT / "GRAPH_REPORT.md"
# ─── Server ────────────────────────────────────────────────────────
server = FastMCP("graphify")
def _check_environment() -> str | None:
"""Return a human-readable error string if the environment is not
usable, or ``None`` if everything is in place. Called at the top
of every tool so the agent gets a clear, actionable message
instead of an opaque subprocess failure.
"""
if not GRAPHIFY_OUT.is_dir():
return (
f"graphify-out directory not found at {GRAPHIFY_OUT}. "
"Set GRAPHIFY_OUT_DIR or run `graphify update <repo>` to generate it."
)
if not GRAPH_JSON.is_file():
return (
f"graph.json not found at {GRAPH_JSON}. "
"Run `graphify update <repo>` to (re-)generate the graph."
)
if shutil.which(GRAPHIFY_BIN) is None and not Path(GRAPHIFY_BIN).is_file():
return (
f"graphify CLI not found (looked for {GRAPHIFY_BIN!r}). "
"Set GRAPHIFY_BIN or install with `uv tool install graphifyy`."
)
return None
def _run_graphify(args: list[str]) -> str:
"""Invoke the graphify CLI and return its stdout. On non-zero
exit or timeout, returns a formatted error string the agent can
read directly — never raises through MCP, so a single misuse
doesn't crash the server."""
cmd = [GRAPHIFY_BIN, *args]
try:
result = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=GRAPHIFY_TIMEOUT_S,
check=False,
)
except subprocess.TimeoutExpired:
return (
f"ERROR: graphify timed out after {GRAPHIFY_TIMEOUT_S}s. "
f"Command: {' '.join(cmd)!r}"
)
except FileNotFoundError:
return (
f"ERROR: graphify binary not found at {GRAPHIFY_BIN!r}. "
"Set GRAPHIFY_BIN env var or install graphify."
)
if result.returncode != 0:
return (
f"ERROR: graphify exited {result.returncode}. "
f"stderr: {result.stderr.strip()[:500]}"
)
return result.stdout
@server.tool()
def report(head_lines: int = 200) -> str:
"""First N lines of GRAPH_REPORT.md (god nodes, communities,
surprising cross-module connections). Call this once per session
to orient before grep/find — the report tells you the shape of
the codebase. Default 200 lines matches the historical
``cat … | head -200`` pattern.
"""
env_err = _check_environment()
if env_err is not None:
return f"ERROR: {env_err}"
if not GRAPH_REPORT.is_file():
return (
f"ERROR: GRAPH_REPORT.md not found at {GRAPH_REPORT}. "
"Run `graphify update <repo>` to regenerate it."
)
try:
with GRAPH_REPORT.open("r", encoding="utf-8") as fh:
lines = [next(fh) for _ in range(max(1, head_lines))]
except StopIteration:
# File is shorter than head_lines — fall through with what we have.
pass
except OSError as exc:
return f"ERROR: reading {GRAPH_REPORT}: {exc}"
return "".join(lines)
@server.tool()
def query(question: str, budget: int = 2000) -> str:
"""BFS traversal of the graph for a natural-language question.
Returns ranked node citations with file:line references, capped
at ``budget`` tokens. Use this **instead of** ``grep -r`` for
cross-module questions ("how does X relate to Y", "what depends
on Z", "what's downstream of file F"). Default budget 2000 is
the same default the CLI uses.
"""
env_err = _check_environment()
if env_err is not None:
return f"ERROR: {env_err}"
if not question.strip():
return "ERROR: question must be non-empty"
return _run_graphify(
[
"query",
question,
"--graph",
str(GRAPH_JSON),
"--budget",
str(max(1, int(budget))),
]
)
@server.tool()
def path(a: str, b: str) -> str:
"""Shortest path between two concept nodes in the graph. Use
when you need to understand the dependency chain between two
specific things (e.g. ``path("SessionContext",
"post_session_action")``).
"""
env_err = _check_environment()
if env_err is not None:
return f"ERROR: {env_err}"
if not a.strip() or not b.strip():
return "ERROR: both `a` and `b` must be non-empty"
return _run_graphify(["path", a, b, "--graph", str(GRAPH_JSON)])
@server.tool()
def explain(concept: str) -> str:
"""Plain-language neighborhood summary for a single concept node
(its direct neighbors, types of edges, file:line citations). Use
when you've located a node and want to understand what's around
it before reading source.
"""
env_err = _check_environment()
if env_err is not None:
return f"ERROR: {env_err}"
if not concept.strip():
return "ERROR: concept must be non-empty"
return _run_graphify(["explain", concept, "--graph", str(GRAPH_JSON)])
# No startup fail-fast on missing graphify-out: the operator may
# spawn OpenCode before running the first ``graphify update``, and
# :func:`_check_environment` gives a clearer in-band error than a
# startup crash.
main = make_main(server, "mcp_graphify_server")
if __name__ == "__main__":
sys.exit(main())