016b348117
Phase 0 (foundation):
- Cause enum (controller_events.cause) for action attribution
- Schema: grooming_decisions audit table; workflows gains
grooming_evaluated_at + deferred_reason + deferred_at +
deferred_target_workflow_id; pulls gains touched_files
- audit_comments: CLOSE / DEFER templates + render_comment_template
- forgejo_writes: close_issue + defer_issue 5-step crash-safe protocol
(fingerprint dedup, error matrix, dry-run)
- patch_pr_state callback in forgejo_http
- grooming_config: 22-env-var frozen-dataclass config + log_effective
- pulls.touched_files cache extension (_pipeline_cache.py schema v8)
- reaper.reap_grooming_decisions audit-retention sweep
- reconciliation RESUME guard (deferred_reason)
Phase 1 (worker-queue shape, 2026-05-25):
- New state: GROOMING. New events: grooming_started, groom_verdict_
{proceed,defer,close}. 5 new transitions; all invariants still clean
- GroomingInputV1 + GroomingOutputV1 Pydantic contracts
- outcomes._map_grooming_outcome routes verdicts to state-machine events
- prefetch.build_grooming_stage_b_input + list_open_prs callback
- scheduler GROOMING -> grooming_stage_b role
- promote: cfg-gated DISCOVERED -> GROOMING when CONTROLLER_GROOMING_
ENABLED=true; issues skip grooming
- forgejo_writes decomposed: close_act/defer_act (Forgejo writes only;
state-machine already transitioned) + close_decide_and_act/
defer_decide_and_act (Phase 0 callers); _apply_workflow_transition
is underscore-private
- grooming.py library: tokenization, suspicion scoring (Jaccard +
weighted overlap), deterministic checks, action -> verdict mapping
- mcp/grooming_builder.py: 14-tool FastMCP server emits GroomingOutputV1
- .opencode/agents/grooming-stage-b.md: duplicate-detection agent
prompt (claude-haiku-4-5)
- grooming_side_effects.run_grooming_side_effects_tick: per-state tick
performs Forgejo writes after groom_verdict_{defer,close} fires.
Filters on event_type='transition' + payload.event (centralizes the
convention pending Phase 2's latest_transition_event helper)
- GroomingCallbacks frozen dataclass; loop.py + __main__.py wired
Worker role registry (single source of truth):
- worker/roles.py: WORKER_ROLES + WorkerRoleSpec + default_roles_csv
+ output_filename_for. agent_runner.ROLE_TO_MCP_MODULE / ROLE_TO_
OUTPUT_MODEL derive from it; opencode_session.agent_name_for reads
it for flat cases; all 6 prompt builders use output_filename_for;
worker --roles default = default_roles_csv(); launcher script
derives --roles via shell substitution. Cross-site invariant test
enforces alignment across 5 sites + opencode.json MCP registry.
Phase 0 silent-bug fix:
- reconciliation.py RESUME guard SELECT now includes deferred_reason
(was missing since Phase 0; guard was a silent no-op). Tightened
from getattr to attribute access to fail fast on future omissions.
Tests (1456 total, +91 grooming-specific):
- test_grooming_phase0.py: 34 tests (orchestrator matrix, crash
recovery, idempotency, dry-run)
- test_grooming_phase1.py: 60 tests (library, contracts, state
machine, outcomes, scheduler, promote, prefetch, act-variants
with signature parity, side-effect tick incl. natural-idempotency
+ executed-flag-skip + verdict-mismatch + reconciliation RESUME)
- test_mcp_builders.py TestGroomingBuilder: 29 tests (happy paths
+ 22 validation rules + Pydantic round-trip + master-tick-read-
path companion)
- test_worker_agent_runner.py TestRoleMaps: cross-role wiring
alignment + agent-prompt-vs-worker-fallback filename contract +
inspect.signature equality (close_act/defer_act vs
close_issue/defer_issue)
- test_state_machine.py: transition count 51 -> 56 +
events_from_grooming
Live-validated end-to-end on 4 staged sentinel PRs (#55-#58) in
dry_run: agent emits verdicts via MCP, state-machine transitions
fire, side-effect tick writes audit row, deferred_reason gates
reconciliation RESUME correctly.
Deferred refinements + Phase 2 prerequisite (latest_transition_event
helper) tracked in .drew/regressions-plan.md "Phase 1 follow-up
backlog".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
819 lines
32 KiB
Python
819 lines
32 KiB
Python
"""Production agent_runner adapter — spawns the role-builder MCP +
|
|
the OpenCode session + returns the parsed worker output.
|
|
|
|
The injected ``agent_runner`` parameter to ``run_one_attempt``
|
|
(``runner.py``) is what production wires to this module. Tests use a
|
|
synthetic agent_runner.
|
|
|
|
Lifecycle per attempt (plan v9 per-attempt subprocess model):
|
|
1. Spawn the matching role-builder MCP subprocess (one of
|
|
``tools/controller/mcp/{role}_builder.py``) with stdout captured
|
|
so we can read the finalize'd JSON.
|
|
2. Tell the OpenCode session to run the role's agent, pointed at the
|
|
spawned MCP server (path passed via env / args).
|
|
3. Wait for the OpenCode session to complete + the MCP subprocess
|
|
to emit one JSON line on stdout.
|
|
4. Strict-parse against the V1 output contract for the role.
|
|
5. Cleanup: terminate the MCP subprocess (in case it didn't exit
|
|
cleanly after finalize).
|
|
|
|
The actual OpenCode session call is parameterized as
|
|
``run_opencode_session`` so this module is testable without a live
|
|
OpenCode. Production wires it to ``tools/_opencode_worker.run_session_blocking``.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
import os
|
|
import signal
|
|
import subprocess
|
|
import sys
|
|
import threading
|
|
from collections.abc import Callable
|
|
from dataclasses import dataclass
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
from ..contracts.parse import ContractValidationError, strict_parse
|
|
from .implementer_finalize import (
|
|
WORKER_ERROR_OUTCOMES,
|
|
finalize_implementer_attempt,
|
|
)
|
|
from .legacy_adapter import adapt_to_v1
|
|
from .roles import WORKER_ROLES
|
|
from .runner import WorkerError, WorkerLostLock
|
|
from .session_sidecar import (
|
|
WorkerSession,
|
|
now_iso,
|
|
read_proc_starttime,
|
|
write_sidecar,
|
|
)
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
|
|
# Public role-to-MCP-module map. Kept as a top-level constant so the
|
|
# existing many call sites (and tests) keep working unchanged; the
|
|
# values are derived from the ``roles.WORKER_ROLES`` registry — that's
|
|
# the single source of truth. See ``roles.py`` for the "how to add a
|
|
# role" instructions.
|
|
ROLE_TO_MCP_MODULE: dict[str, str] = {
|
|
role: spec.mcp_module for role, spec in WORKER_ROLES.items()
|
|
}
|
|
|
|
# Public role-to-output-model map. Same derivation pattern.
|
|
ROLE_TO_OUTPUT_MODEL: dict[str, type] = {
|
|
role: spec.output_model for role, spec in WORKER_ROLES.items()
|
|
}
|
|
|
|
|
|
# Callable signature for the injected OpenCode session runner.
|
|
# Inputs: role, mcp_subprocess (so the session can spawn-as-child or
|
|
# communicate via stdio), input_payload (already validated).
|
|
# Returns: nothing — the MCP subprocess is what emits the canonical
|
|
# JSON on its own stdout. The session is just the driver.
|
|
#
|
|
# Production wires this to a thin wrapper around
|
|
# ``_opencode_worker.run_session_blocking``.
|
|
OpenCodeSessionRunner = Callable[..., None]
|
|
|
|
|
|
@dataclass
|
|
class MCPSpawnResult:
|
|
"""Captured MCP subprocess + its stdout reader thread."""
|
|
|
|
process: subprocess.Popen
|
|
stdout_buffer: list[str] # mutable; appended by reader thread
|
|
reader_thread: threading.Thread
|
|
|
|
|
|
# Time the worker waits for an MCP to emit its canonical JSON output before
|
|
# treating the attempt as worker-internal-error and re-dispatching. Bumped
|
|
# from 30s to 90s after the 2026-05-22 batch surfaced spurious estimator
|
|
# timeouts at ~30s+ (real estimator self-time observed up to 41s on
|
|
# multi-subsystem PRs; the worker wall-clock includes MCP IPC + finalize
|
|
# write, which adds seconds on top). Env-overridable for operators tuning
|
|
# tighter / looser bounds; matches the CONTROLLER_*_TIMEOUT_S convention
|
|
# used by ``opencode_session.py``.
|
|
_DEFAULT_FINALIZE_TIMEOUT_S = float(
|
|
os.environ.get("CONTROLLER_FINALIZE_TIMEOUT_S", "90")
|
|
)
|
|
|
|
|
|
def production_agent_runner(
|
|
*,
|
|
attempt_id: int,
|
|
role: str,
|
|
tier: int | None,
|
|
input_payload: dict[str, Any],
|
|
instance_id: str,
|
|
lost_lock_check: Callable[[], bool],
|
|
run_opencode_session: OpenCodeSessionRunner,
|
|
python_executable: str = sys.executable,
|
|
mcp_startup_timeout_s: float = 5.0,
|
|
finalize_timeout_s: float = _DEFAULT_FINALIZE_TIMEOUT_S,
|
|
workspace_dir: Path | None = None,
|
|
opencode_server_url: str = "",
|
|
finalize_implementer: Callable[..., Any] = finalize_implementer_attempt,
|
|
) -> dict[str, Any]:
|
|
"""The production agent_runner used by run_one_attempt.
|
|
|
|
Maps the role → matching MCP module + output contract; spawns
|
|
the MCP as a subprocess; calls the injected ``run_opencode_session``
|
|
to drive the LLM session; reads the MCP's emitted JSON; strict-
|
|
parses; returns the parsed dict.
|
|
|
|
Tests can call this directly with a synthetic ``run_opencode_session``
|
|
that pokes the MCP's stdin (or otherwise simulates a session). The
|
|
integration with ``_opencode_worker.run_session_blocking`` is the
|
|
production wiring.
|
|
|
|
Raises:
|
|
ValueError: unknown role.
|
|
WorkerError: MCP didn't start / didn't emit valid JSON /
|
|
strict-parse failed twice.
|
|
WorkerLostLock: ``lost_lock_check`` returned True during the run.
|
|
"""
|
|
if role not in ROLE_TO_MCP_MODULE:
|
|
raise ValueError(f"unknown role {role!r}; cannot select MCP")
|
|
mcp_module = ROLE_TO_MCP_MODULE[role]
|
|
model_class = ROLE_TO_OUTPUT_MODEL[role]
|
|
|
|
# Production: per-attempt canonical output path. MCP's
|
|
# finalize_and_emit writes the canonical JSON HERE rather than
|
|
# to stdout (which is shared with the FastMCP JSON-RPC transport).
|
|
import tempfile
|
|
|
|
out_fd, out_path = tempfile.mkstemp(
|
|
prefix=f"controller-canonical-{attempt_id}-",
|
|
suffix=".json",
|
|
)
|
|
os.close(out_fd)
|
|
|
|
spawn = _spawn_mcp_subprocess(
|
|
mcp_module,
|
|
python_executable=python_executable,
|
|
startup_timeout_s=mcp_startup_timeout_s,
|
|
canonical_output_path=out_path,
|
|
)
|
|
|
|
# Patch the input_payload so the prompt builder can interpolate
|
|
# real values (workspace_dir + attempt_id) into the agent's
|
|
# prompt. Without these, the prompt has literal "..." placeholders
|
|
# in the MCP tool-call signatures — the agent then guesses
|
|
# ``attempt_id=1`` every time, the cross-session reset can't tell
|
|
# sessions apart, stale MCP state persists, and every attempt
|
|
# after the first fails with "already finalized". (Real bug
|
|
# observed in trial run-2.) We mutate a shallow copy so the
|
|
# caller's dict isn't side-effected.
|
|
input_payload = dict(input_payload)
|
|
input_payload["attempt_id"] = attempt_id
|
|
if workspace_dir is not None:
|
|
input_payload["workspace_dir"] = str(workspace_dir)
|
|
|
|
# Phase 1k++ (N1): write a per-attempt sidecar so the startup
|
|
# janitor can detect orphaned MCP subprocesses + workspaces if
|
|
# this worker crashes. The starttime is read AFTER spawn so it
|
|
# reflects the actual /proc/{pid}/stat field 22 the janitor
|
|
# later compares against (defends against PID reuse).
|
|
sidecar_path: Path | None = None
|
|
if workspace_dir is not None:
|
|
try:
|
|
sidecar_path = Path(workspace_dir) / "worker.session"
|
|
write_sidecar(
|
|
sidecar_path,
|
|
WorkerSession(
|
|
opencode_server_url=opencode_server_url,
|
|
session_id="", # populated by the session adapter if/when known
|
|
subprocess_pid=spawn.process.pid,
|
|
spawned_by_controller_pid=os.getpid(),
|
|
instance_id=instance_id,
|
|
spawned_at=now_iso(),
|
|
subprocess_starttime=read_proc_starttime(spawn.process.pid),
|
|
),
|
|
)
|
|
except OSError as exc:
|
|
logger.warning(
|
|
"sidecar write failed for attempt_id=%s: %s — janitor "
|
|
"won't be able to detect orphans for this attempt",
|
|
attempt_id,
|
|
exc,
|
|
)
|
|
sidecar_path = None
|
|
|
|
# CA1 stale-file cleanup: the per-PR workspace dir is reused across
|
|
# attempts on the same PR, so a prior attempt's canonical output
|
|
# would be picked up by the poller as if it were this attempt's
|
|
# output. Wipe BOTH the MCP canonical path (defensive) and the
|
|
# fallback paths before the session starts. The fallback filename
|
|
# comes from ``roles.output_filename_for`` — the same source the
|
|
# per-role prompt builders use, so the worker and agent can never
|
|
# disagree about the filename.
|
|
from .roles import output_filename_for
|
|
|
|
fallback_paths: list[str] = []
|
|
if workspace_dir is not None:
|
|
fallback_paths.append(
|
|
str(Path(workspace_dir) / output_filename_for(role))
|
|
)
|
|
for stale in [out_path] + fallback_paths:
|
|
try:
|
|
os.unlink(stale)
|
|
except FileNotFoundError:
|
|
pass
|
|
except OSError as exc:
|
|
# PD-R2-7: a stale file we can't remove is a worse failure
|
|
# mode than aborting — the poller would read it as if it
|
|
# were this attempt's output. Hard-fail so the attempt
|
|
# surfaces the underlying permissions/filesystem bug.
|
|
raise WorkerError(
|
|
f"could not unlink stale output file {stale} before session: {exc}",
|
|
outcome="worker-internal-error",
|
|
) from exc
|
|
|
|
try:
|
|
# Hand the OpenCode session the MCP we just spawned + the
|
|
# input payload. The session's job: drive the LLM through
|
|
# its tool-calling sequence; the MCP captures the assembled
|
|
# state + emits canonical JSON via its finalize() tool.
|
|
#
|
|
# Trial-path: legacy-style agents emit a single JSON object
|
|
# as their final response message; the OpenCode worker
|
|
# extracts it into ``SessionResult.parsed_json``. We hand
|
|
# the opencode_session adapter a callback that writes that
|
|
# JSON to the canonical-output file so the post-session poller
|
|
# picks it up uniformly with the MCP + file-write channels.
|
|
import json as _json_mod
|
|
|
|
import time as _time
|
|
|
|
_session_start = _time.monotonic()
|
|
|
|
def _capture_inline_json(parsed: dict) -> None:
|
|
# CA7: skip if we lost the lock — file written after lock-loss
|
|
# would poison a subsequent attempt on this same workspace.
|
|
if lost_lock_check():
|
|
logger.info(
|
|
"inline-output capture skipped (lost lock) "
|
|
"for attempt_id=%s role=%s",
|
|
attempt_id,
|
|
role,
|
|
)
|
|
return
|
|
# PD3: if the MCP already wrote canonical V1 to out_path,
|
|
# do NOT overwrite. The MCP path is authoritative; the
|
|
# inline-chat path is a fallback for agents that don't call
|
|
# the MCP. Inspect any existing file at out_path first.
|
|
for p in [out_path] + list(fallback_paths):
|
|
try:
|
|
with open(p, "r", encoding="utf-8") as f:
|
|
existing = f.read().strip()
|
|
except (FileNotFoundError, OSError):
|
|
continue
|
|
if not existing:
|
|
continue
|
|
try:
|
|
existing_obj = _json_mod.loads(existing)
|
|
except (TypeError, ValueError):
|
|
continue
|
|
if (
|
|
isinstance(existing_obj, dict)
|
|
and existing_obj.get("output_version") == "V1"
|
|
):
|
|
logger.debug(
|
|
"MCP already emitted V1 to %s; skipping inline "
|
|
"capture (would overwrite authoritative output)",
|
|
p,
|
|
)
|
|
return
|
|
# Adapt the legacy-shape JSON to the V1 contract before
|
|
# writing. Without this, strict_parse against the V1
|
|
# model rejects the legacy shape (missing output_version,
|
|
# different outcome enum, etc.).
|
|
wallclock = _time.monotonic() - _session_start
|
|
adapted = adapt_to_v1(
|
|
role,
|
|
parsed,
|
|
tier=tier,
|
|
wallclock_seconds=wallclock,
|
|
)
|
|
# PD10: atomic write via .tmp + os.replace so the poller
|
|
# never sees a half-written file.
|
|
tmp_path = f"{out_path}.tmp"
|
|
try:
|
|
with open(tmp_path, "w", encoding="utf-8") as f:
|
|
f.write(_json_mod.dumps(adapted))
|
|
f.write("\n")
|
|
f.flush()
|
|
os.fsync(f.fileno())
|
|
os.replace(tmp_path, out_path)
|
|
except OSError as exc:
|
|
logger.warning(
|
|
"inline-output capture write failed (%s): canonical "
|
|
"poller will fall through to other channels",
|
|
exc,
|
|
)
|
|
try:
|
|
os.unlink(tmp_path)
|
|
except OSError:
|
|
pass
|
|
|
|
# ``canonical`` holds the role's V1 output JSON once it lands —
|
|
# whether the agent emitted it, or it was salvaged from the
|
|
# worktree after a session failure. None until then.
|
|
canonical: str | None = None
|
|
try:
|
|
run_opencode_session(
|
|
role=role,
|
|
tier=tier,
|
|
input_payload=input_payload,
|
|
mcp_process=spawn.process,
|
|
attempt_id=attempt_id,
|
|
instance_id=instance_id,
|
|
lost_lock_check=lost_lock_check,
|
|
inline_output_callback=_capture_inline_json,
|
|
)
|
|
except WorkerLostLock:
|
|
raise
|
|
except Exception as exc:
|
|
# The session raised — most commonly a tier-budget timeout
|
|
# (opencode_session maps timeout / transport-error to
|
|
# WorkerError). A timeout is the case MOST likely to have a
|
|
# complete committed fix sitting in the worktree: the agent
|
|
# ran out of wallclock, not correctness. Before discarding
|
|
# the attempt as worker-internal-error, run the SAME salvage
|
|
# the canonical-missing path uses — otherwise a timed-out
|
|
# implementer's committed work never reaches CI, because the
|
|
# session exception propagates before the canonical-output
|
|
# wait below ever runs (audit finding #1).
|
|
salvaged = _salvage_implementer_commits(
|
|
role=role,
|
|
input_payload=input_payload,
|
|
workspace_dir=workspace_dir,
|
|
tier=tier,
|
|
wallclock_seconds=_time.monotonic() - _session_start,
|
|
lost_lock_check=lost_lock_check,
|
|
)
|
|
if salvaged is None:
|
|
# Nothing safe to salvage — wrap so the runner's outer
|
|
# handler classifies properly.
|
|
raise WorkerError(
|
|
f"OpenCode session raised: {exc}",
|
|
outcome="worker-internal-error",
|
|
) from exc
|
|
logger.warning(
|
|
"attempt_id=%s role=%s: OpenCode session raised (%s); "
|
|
"salvaged %d commit(s) from the worktree, pushed, and "
|
|
"synthesized outcome=resolved so CI verifies the work",
|
|
attempt_id,
|
|
role,
|
|
exc,
|
|
len(salvaged.get("commit_shas") or []),
|
|
)
|
|
canonical = _json_mod.dumps(salvaged)
|
|
|
|
if canonical is None:
|
|
# The session returned normally. Wait for the canonical V1
|
|
# output to land in one of its channels (MCP path, fallback
|
|
# file, inline-chat capture).
|
|
if lost_lock_check():
|
|
raise WorkerLostLock("lost lock during MCP session")
|
|
canonical = _wait_for_canonical_output(
|
|
spawn,
|
|
out_path,
|
|
timeout_s=finalize_timeout_s,
|
|
fallback_paths=fallback_paths,
|
|
)
|
|
if canonical is None:
|
|
# The OpenCode session ended but no canonical V1 output
|
|
# landed in ANY channel (MCP path, fallback file, inline-
|
|
# chat capture). For an implementer this is often a
|
|
# finished agent that simply never emitted its report —
|
|
# while its actual work, a git commit, sits in the
|
|
# worktree. Discarding the attempt (run-25 did exactly
|
|
# this, losing a correct fix) is worse than salvaging:
|
|
# push the committed work and synthesize a minimal V1
|
|
# output so the controller routes to AWAITING_CI and lets
|
|
# CI judge the code.
|
|
salvaged = _salvage_implementer_commits(
|
|
role=role,
|
|
input_payload=input_payload,
|
|
workspace_dir=workspace_dir,
|
|
tier=tier,
|
|
wallclock_seconds=_time.monotonic() - _session_start,
|
|
lost_lock_check=lost_lock_check,
|
|
)
|
|
if salvaged is None:
|
|
raise WorkerError(
|
|
f"role={role!r} did not emit canonical output within "
|
|
f"{finalize_timeout_s}s (looked at MCP path {out_path} "
|
|
f"+ fallback paths {fallback_paths})",
|
|
outcome="worker-internal-error",
|
|
)
|
|
logger.warning(
|
|
"attempt_id=%s role=%s: canonical output missing; "
|
|
"salvaged %d commit(s) from the worktree, pushed, and "
|
|
"synthesized outcome=resolved so CI verifies the work",
|
|
attempt_id,
|
|
role,
|
|
len(salvaged.get("commit_shas") or []),
|
|
)
|
|
canonical = _json_mod.dumps(salvaged)
|
|
|
|
try:
|
|
parsed = strict_parse(model_class, canonical)
|
|
except ContractValidationError as exc:
|
|
raise WorkerError(
|
|
f"MCP emitted JSON that failed {model_class.__name__} "
|
|
f"strict-parse: {exc}",
|
|
outcome="contract-violation",
|
|
) from exc
|
|
|
|
result = parsed.model_dump()
|
|
# Implementer push is worker-owned: the agent commits, the worker
|
|
# gates (lint + typecheck, with a ruff auto-fix pass) and pushes
|
|
# via the leased worker_push primitive. finalize's outcome is
|
|
# AUTHORITATIVE — it supersedes whatever the agent emitted.
|
|
if role == "implementer" and workspace_dir is not None:
|
|
fin = finalize_implementer(
|
|
worktree=Path(workspace_dir) / "worktree",
|
|
input_payload=input_payload,
|
|
agent_output=result,
|
|
tier=tier,
|
|
lost_lock_check=lost_lock_check,
|
|
)
|
|
if fin.outcome in WORKER_ERROR_OUTCOMES:
|
|
# gate-failed / stale-input / worker-internal-error →
|
|
# the runner's status='failed' re-enqueue machinery.
|
|
# output_payload carries the gate report so the next
|
|
# attempt's prompt sees the specific failures.
|
|
raise WorkerError(
|
|
fin.detail or fin.outcome,
|
|
outcome=fin.outcome,
|
|
output_payload=fin.output_payload,
|
|
)
|
|
result = fin.output_payload
|
|
return result
|
|
finally:
|
|
_terminate_mcp_subprocess(spawn)
|
|
# Clean up the canonical output file (best-effort).
|
|
try:
|
|
os.unlink(out_path)
|
|
except OSError:
|
|
pass
|
|
# Sidecar cleanup: the attempt completed (success or failure),
|
|
# so the janitor shouldn't see it as orphaned. Best-effort —
|
|
# if removal fails, the janitor's PID-alive check will skip
|
|
# it cleanly on the next sweep.
|
|
if sidecar_path is not None:
|
|
try:
|
|
sidecar_path.unlink()
|
|
except OSError:
|
|
pass
|
|
|
|
|
|
# ─── MCP subprocess lifecycle ────────────────────────────────────────
|
|
|
|
|
|
def _spawn_mcp_subprocess(
|
|
mcp_module: str,
|
|
*,
|
|
python_executable: str,
|
|
startup_timeout_s: float,
|
|
canonical_output_path: str,
|
|
) -> MCPSpawnResult:
|
|
"""Spawn ``python -m {mcp_module}`` with stdout captured via a
|
|
background reader thread.
|
|
|
|
``canonical_output_path`` is the file the MCP writes its canonical
|
|
JSON to (via the CONTROLLER_CANONICAL_OUTPUT_PATH env var).
|
|
Stdout is still captured (for transport/debug logging) but the
|
|
authoritative output goes to the file.
|
|
"""
|
|
repo_root = str(Path(__file__).resolve().parents[3])
|
|
env = dict(os.environ)
|
|
# Prepend repo_root to PYTHONPATH so ``-m tools.controller.mcp....`` works.
|
|
existing_pp = env.get("PYTHONPATH", "")
|
|
env["PYTHONPATH"] = (
|
|
f"{repo_root}{os.pathsep}{existing_pp}" if existing_pp else repo_root
|
|
)
|
|
env["CONTROLLER_CANONICAL_OUTPUT_PATH"] = canonical_output_path
|
|
proc = subprocess.Popen(
|
|
[python_executable, "-m", mcp_module],
|
|
stdin=subprocess.PIPE,
|
|
stdout=subprocess.PIPE,
|
|
stderr=subprocess.PIPE,
|
|
text=True,
|
|
env=env,
|
|
cwd=repo_root,
|
|
)
|
|
|
|
stdout_buffer: list[str] = []
|
|
reader_thread = threading.Thread(
|
|
target=_drain_stream_into_list,
|
|
args=(proc.stdout, stdout_buffer),
|
|
name=f"mcp-stdout-{proc.pid}",
|
|
daemon=True,
|
|
)
|
|
reader_thread.start()
|
|
|
|
# Best-effort startup check: did the process die immediately?
|
|
# Don't actually wait the full timeout — production OpenCode sessions
|
|
# are long-running; we just verify the subprocess is alive.
|
|
if proc.poll() is not None:
|
|
raise WorkerError(
|
|
f"MCP subprocess died immediately (returncode={proc.returncode})",
|
|
outcome="worker-internal-error",
|
|
)
|
|
|
|
return MCPSpawnResult(
|
|
process=proc,
|
|
stdout_buffer=stdout_buffer,
|
|
reader_thread=reader_thread,
|
|
)
|
|
|
|
|
|
def _drain_stream_into_list(stream, target: list[str]) -> None:
|
|
"""Read lines from ``stream`` into ``target`` until EOF.
|
|
|
|
Runs in a background thread. Each line (without trailing newline)
|
|
is appended. Robust against the stream closing mid-read.
|
|
"""
|
|
try:
|
|
for line in iter(stream.readline, ""):
|
|
if not line:
|
|
break
|
|
target.append(line.rstrip("\n"))
|
|
except (ValueError, OSError):
|
|
# Stream closed; thread exits.
|
|
pass
|
|
|
|
|
|
def _wait_for_canonical_output(
|
|
spawn: MCPSpawnResult,
|
|
out_path: str,
|
|
*,
|
|
timeout_s: float,
|
|
fallback_paths: list[str] | None = None,
|
|
) -> str | None:
|
|
"""Poll the canonical-output file. Returns its content on first
|
|
non-empty read, or None on timeout.
|
|
|
|
``fallback_paths`` are additional paths to poll alongside
|
|
``out_path``. Used during the trial phase to support direct
|
|
file-write output from agents that can't call the MCP (the
|
|
response-builder MCPs aren't wired into OpenCode yet — see
|
|
prompts.py for the contract the agent follows).
|
|
|
|
The MCP writes the file atomically (open + write + fsync + close)
|
|
in its finalize_and_emit. We poll until the file exists + has
|
|
content OR the subprocess has been gone long enough to indicate
|
|
it's not going to write.
|
|
"""
|
|
import json as _json
|
|
import os as _os
|
|
import time
|
|
|
|
paths_to_poll = [out_path] + list(fallback_paths or [])
|
|
|
|
def _try_read_stable(p: str) -> str | None:
|
|
"""Read ``p`` only if size is stable across two stats AND the
|
|
content parses as JSON. Defends against partial writes (agent
|
|
crashed mid-flush; the file has truncated text). Returns the
|
|
content string on success; None if the file is missing, empty,
|
|
in-flight, or unparseable.
|
|
|
|
Round-4 P1 fix: the previous version did ``f.read().strip()``
|
|
and returned anything non-empty — a half-written file's
|
|
partial JSON would then trip ``ContractValidationError`` →
|
|
``worker-internal-error`` with no record of WHICH path.
|
|
"""
|
|
try:
|
|
sz1 = _os.stat(p).st_size
|
|
except (FileNotFoundError, OSError):
|
|
return None
|
|
if sz1 == 0:
|
|
return None
|
|
# Give the writer a moment in case the flush is still landing.
|
|
time.sleep(0.02)
|
|
try:
|
|
sz2 = _os.stat(p).st_size
|
|
except (FileNotFoundError, OSError):
|
|
return None
|
|
if sz1 != sz2:
|
|
return None # size still moving; wait for the next poll
|
|
try:
|
|
with open(p, "r", encoding="utf-8") as f:
|
|
content = f.read().strip()
|
|
except (FileNotFoundError, OSError):
|
|
return None
|
|
if not content:
|
|
return None
|
|
# Validate it parses as JSON — partial writes often produce
|
|
# syntactically broken content that strict_parse would later
|
|
# reject with a confusing error far from the read site.
|
|
try:
|
|
_json.loads(content)
|
|
except (TypeError, ValueError):
|
|
return None
|
|
return content
|
|
|
|
def _try_read_any() -> tuple[str | None, str | None]:
|
|
"""Returns (content, source_path) of the first stable+valid
|
|
path, or (None, None) if nothing's ready."""
|
|
for p in paths_to_poll:
|
|
content = _try_read_stable(p)
|
|
if content:
|
|
return content, p
|
|
return None, None
|
|
|
|
start = time.monotonic()
|
|
last_source: str | None = None
|
|
while True:
|
|
content, source = _try_read_any()
|
|
if content:
|
|
last_source = source
|
|
logger.debug(
|
|
"canonical output read from %s (%d bytes)",
|
|
source,
|
|
len(content),
|
|
)
|
|
return content
|
|
|
|
# Subprocess gone + file still empty → give it one more brief
|
|
# window for the final flush to land, then give up.
|
|
if spawn.process.poll() is not None:
|
|
time.sleep(0.1)
|
|
content, source = _try_read_any()
|
|
if source:
|
|
logger.debug(
|
|
"canonical output read from %s after subprocess exit",
|
|
source,
|
|
)
|
|
return content
|
|
|
|
if time.monotonic() - start > timeout_s:
|
|
logger.warning(
|
|
"canonical output timeout after %.1fs (looked at: %s); "
|
|
"last successful read source: %s",
|
|
timeout_s,
|
|
paths_to_poll,
|
|
last_source,
|
|
)
|
|
return None
|
|
time.sleep(0.05)
|
|
|
|
|
|
def _terminate_mcp_subprocess(spawn: MCPSpawnResult) -> None:
|
|
"""Send SIGTERM + 2s grace + SIGKILL to ensure cleanup. Drain
|
|
the reader thread."""
|
|
proc = spawn.process
|
|
if proc.poll() is None:
|
|
try:
|
|
proc.terminate()
|
|
except ProcessLookupError:
|
|
pass
|
|
try:
|
|
proc.wait(timeout=2.0)
|
|
except subprocess.TimeoutExpired:
|
|
try:
|
|
proc.kill()
|
|
except ProcessLookupError:
|
|
pass
|
|
try:
|
|
proc.wait(timeout=2.0)
|
|
except subprocess.TimeoutExpired:
|
|
pass
|
|
# Reader thread will exit on its own when stdout closes.
|
|
spawn.reader_thread.join(timeout=2.0)
|
|
|
|
|
|
# ─── canonical-output salvage ────────────────────────────────────────
|
|
|
|
|
|
def _git_in_worktree(worktree: Path, *args: str, timeout: float = 120.0) -> str:
|
|
"""Run a git command in ``worktree``; return stdout. Raises
|
|
``RuntimeError`` on non-zero exit or timeout."""
|
|
try:
|
|
result = subprocess.run(
|
|
["git", "-C", str(worktree), *args],
|
|
capture_output=True,
|
|
text=True,
|
|
check=True,
|
|
timeout=timeout,
|
|
)
|
|
except subprocess.CalledProcessError as exc:
|
|
raise RuntimeError(
|
|
f"git {' '.join(args)} failed: {(exc.stderr or '').strip()}"
|
|
) from exc
|
|
except subprocess.TimeoutExpired as exc:
|
|
raise RuntimeError(f"git {' '.join(args)} timed out") from exc
|
|
return result.stdout
|
|
|
|
|
|
def _salvage_implementer_commits(
|
|
*,
|
|
role: str,
|
|
input_payload: dict[str, Any],
|
|
workspace_dir: Path | None,
|
|
tier: int | None,
|
|
wallclock_seconds: float,
|
|
lost_lock_check: Callable[[], bool],
|
|
) -> dict[str, Any] | None:
|
|
"""Recover a finished-but-unreported implementer attempt.
|
|
|
|
When an implementer's OpenCode session ends cleanly but no
|
|
canonical V1 output is emitted, the agent's actual work — a git
|
|
commit — is still in the worktree. Rather than discard the attempt
|
|
as worker-internal-error (run-25 did exactly this, losing a correct
|
|
fix), enumerate those commits and synthesize a minimal advisory
|
|
``ImplementerOutputV1`` with ``outcome=resolved``.
|
|
``finalize_implementer_attempt`` then takes this as advisory input,
|
|
runs the lint+typecheck gate, and pushes — salvage itself no longer
|
|
pushes.
|
|
|
|
``lost_lock_check`` is honored before touching the worktree and
|
|
again immediately before the push: the canonical-output wait can
|
|
span the lock TTL, and if the lock was reaped a fresh worker may
|
|
already own this workspace — racing its ``git reset``/``clean`` with
|
|
our git ops would corrupt the shared worktree. This mirrors
|
|
``_capture_inline_json``'s CA7 guard.
|
|
|
|
Returns the synthetic output dict on a successful salvage, or None
|
|
when there is nothing safe to salvage (not an implementer, lock
|
|
lost, no commits, or the push could not land — in which case the
|
|
caller raises worker-internal-error and the next attempt adopts the
|
|
work from auto-scratch/pr-<N>).
|
|
"""
|
|
if role != "implementer" or workspace_dir is None:
|
|
return None
|
|
if lost_lock_check():
|
|
logger.info(
|
|
"salvage skipped: lock lost — the workspace may have a new "
|
|
"owner; touching the shared worktree now would race it"
|
|
)
|
|
return None
|
|
worktree = Path(workspace_dir) / "worktree"
|
|
if not (worktree / ".git").exists():
|
|
return None
|
|
base = input_payload.get("head_sha")
|
|
head_ref = input_payload.get("head_ref")
|
|
if not isinstance(base, str) or not base:
|
|
return None
|
|
if not isinstance(head_ref, str) or not head_ref:
|
|
return None
|
|
try:
|
|
head = _git_in_worktree(worktree, "rev-parse", "HEAD").strip()
|
|
except RuntimeError as exc:
|
|
logger.warning("salvage: could not read worktree HEAD: %s", exc)
|
|
return None
|
|
if not head or head == base:
|
|
return None # agent produced no commits — nothing to salvage
|
|
try:
|
|
# --reverse → oldest-first, so commit_shas[-1] is the branch
|
|
# tip; runner.py derives head_sha_after from commit_shas[-1].
|
|
commit_shas = _git_in_worktree(
|
|
worktree, "rev-list", "--reverse", f"{base}..{head}"
|
|
).split()
|
|
files_touched = [
|
|
line.strip()
|
|
for line in _git_in_worktree(
|
|
worktree, "diff", "--name-only", f"{base}..{head}"
|
|
).splitlines()
|
|
if line.strip()
|
|
]
|
|
except RuntimeError as exc:
|
|
logger.warning("salvage: could not enumerate commits/files: %s", exc)
|
|
return None
|
|
if not commit_shas or not files_touched:
|
|
return None
|
|
# The push is NOT done here — finalize_implementer_attempt gates
|
|
# (lint + typecheck) and pushes the worktree commits via the leased
|
|
# worker_push primitive. Salvage now only RECOVERS the commit list
|
|
# when the agent emitted no canonical output; the synthesized dict
|
|
# below is advisory input that finalize supersedes.
|
|
return {
|
|
"output_version": "V1",
|
|
"outcome": "resolved",
|
|
"files_touched": files_touched,
|
|
"commit_shas": commit_shas,
|
|
"confidence": "low",
|
|
"blockers": [],
|
|
"used_tier": tier if isinstance(tier, int) and tier in (0, 1, 2) else 0,
|
|
"disputed_review_id": None,
|
|
"disputed_blocker_index": None,
|
|
"dispute_evidence": None,
|
|
"verified_at_sha": None,
|
|
"wallclock_seconds": max(0.0, wallclock_seconds),
|
|
}
|
|
|
|
|
|
__all__ = [
|
|
"MCPSpawnResult",
|
|
"OpenCodeSessionRunner",
|
|
"ROLE_TO_MCP_MODULE",
|
|
"ROLE_TO_OUTPUT_MODEL",
|
|
"production_agent_runner",
|
|
]
|