d4e1219577e01f270aca90e2706cfdcd523100cd
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 drew
|
0db0a15dad |
feat(controller): RUN_CI_LOCAL verdict source, ci-not-ready outcome, escalation hardening
Adds RUN_CI_LOCAL — an on-demand local-CI verdict source for when the cluster's Forgejo CI is broken — plus robustness fixes, the telemetry Live-tab rewrite, and PR-level cost attribution. Controller: - RUN_CI_LOCAL: the master swaps its Forgejo CI callbacks for local `forgejo-runner exec` runs (tools/run-ci-full-local.sh + local_ci.py). Async per-(owner,repo,SHA) on-disk job cache; preflights the forgejo-runner binary + Docker daemon at startup (fail loud, not a red verdict on every PR); GCs finished run dirs + per-run actcache. - ci-not-ready implementer outcome + implementer_ci_not_ready event: an implementer that runs before the on-demand verdict exists parks in AWAITING_CI instead of dead-ending at STUCK; capped against ci_red ping-pong. - ci_poll_exhaustion skips its sweep while a local CI run is in flight, so AWAITING_CI workflows queued behind on-demand CI are not STUCK'd by the remote-CI-sized timeout. - Escalate the workflow after repeated worker-internal-error at a tier instead of retrying to pickup-exhaustion -> STUCK. - forgejo_http: normalise Forgejo's per-gate `status` key to `state` so failing gates are actually counted (they previously all read as pending). - Per-tier worker timeouts bumped +15 min; a timed-out attempt's dirty-worktree residue is preserved on auto-scratch/pr-<N> before the next attempt's reset. - Implementer agents now verify only the CI-flagged gate(s) via a targeted re-run rather than the full local battery before claiming resolved/noop. Re-running the whole suite CI will run anyway was the #1 cause of implementer timeouts; CI remains the real gate and re-dispatches the implementer on red. Telemetry: - Live tab rebuilt on /api/live (controller DB run state + the live OpenCode session forest) after the live_log_writer sidecar was retired with the legacy dispatchers. - Durable per-attempt input/output payloads surfaced in the Live drill-down, archived-session detail, and Workflows timeline. - PR-level cost attribution: worker session tags carry -pr-<n>; backfill_llm_activity_pr.py repairs rows written before the fix. Shared: - tools/controller/session_tag.py — one canonical controller-tag parser shared by the telemetry server and the backfill. Tests: new coverage for local_ci (state machine, log parsing, _summarize_run, GC, in-flight probe, preflight), the ci-not-ready path, the escalation/ci-not-ready SQL counters, the ci_poll in-flight skip, and CI-status payload parsing across both sources. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
a6986008ee |
feat(controller): trial-5 batch — dispute path, merge pipeline split, conflict-resolver hardening
T5-1 reviewer feedback rendered full-body to the implementer
T5-4/9 implementer dispute path — dispute-at-any-tier with per-tier cap,
OPERATOR_ATTENTION state on stalemate, pr-review-worker-dispute agent
T5-5 reviewer BLOCKING ISSUE EVIDENCE RULE + 5-step validation
T5-7 merge step split into a singleton process — impl/review masters write
APPROVED and stop; merge_drive owns APPROVED -> MERGING -> MERGED
T5-10 merge process is fully deterministic; base conflicts bounce to the
controller's CONFLICT_RESOLVING (LLM); conflict_drive sidecar retired
T5-11 implementer fast success path — verified-clean outcome so a no-op
after conflict resolution doesn't force busywork
T5-12 conflict-resolver permissions fixed across all paths (/tmp/** glob)
T5-13 conflict-resolver PR-intent prehydration (title/body/comments)
Adds tools/_controller_db_bridge.py so merge_drive reads the controller DB
directly (Option B), plus APPROVED + OPERATOR_ATTENTION states, the
dispute/verified-clean events, and the V1 contract fields backing them.
Reviewer model: baseline -> sonnet, dispute -> opus.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
drew
|
84da774212 |
fix(controller): batch M — post-Phase-1m adversarial review fixes
Three rounds of adversarial review (Chief Architect / Principal Dev /
Senior Test Engineer) on commits 3ca794be7..db12f45ac surfaced ~35
issues. This commit addresses 25+ across criticals, highs, and
mediums, and adds 40 new tests covering the changes plus key gaps the
review identified.
CRITICALS (M1):
- CA1: stale {role}_output.json from a prior attempt on the same
per-PR workspace was readable as "fresh" output of the new attempt.
agent_runner now unlinks the MCP-canonical path AND every fallback
path BEFORE the session runs.
- CA2/PD5: opencode.json-registered MCP subprocesses persist across
OpenCode sessions, but BuilderState was module-singleton. Added
reset_for_new_attempt() + cross-session detection (compare
identity.attempt_id) to every *_start; force-resets with WARN if
prior attempt was interrupted (timeout / lost lock).
- PD3: inline-JSON callback could overwrite an MCP-written canonical
V1 file with adapted-from-prose garbage. Callback now inspects
existing files and skips when V1 is already present.
- PD4: FORGEJO_URL = .rstrip("/api/v1") is a character-set strip —
catastrophic for hosts whose path contains /v1 in the middle.
Replaced with explicit endswith()-based suffix strip.
- CA10: clone URL embedded $FORGEJO_TOKEN, persisted into
.git/config where any agent could cat it. Token now sourced via
local credential.helper at clone-time, URL kept clean.
- CA12: state.finalized was set BEFORE the file write, so disk-full
/ OSError left the agent unable to retry finalize. Reordered.
HIGHS (M2):
- CA3/PD12: output_path validation (NUL-byte rejection, must be
absolute, parent-not-file check) in finalize_and_emit.
- CA6: ci_status_poll SELECT only considered implementer attempts;
conflict_resolver also pushes commits. SQL now unions both roles.
- PD9: ci_status_poll could advance on a stale "resolved" SHA from a
blocked attempt (whose head_sha_after == head_sha_before). Added
outcome='resolved' filter.
- CA8: cancelled/stale CI states mapped to ci_red_retry_same_tier,
burning pickup_count on healthy PRs. Both now wait (treated as
operator/system action, not failure). timed_out stays red.
- TE9: unknown Forgejo CI states now WARN-log instead of silently
being treated as pending — operators see new state strings.
- PD8: ci_status_poll event_type strings standardized to match the
state-machine event names (ci_green / ci_red_retry_same_tier)
instead of legacy ci-green / ci-red.
- CA7: inline-JSON callback now checks lost_lock_check BEFORE write
so a file isn't staged after lock loss.
- PD10: atomic .tmp + os.replace writes in both MCP finalize and
inline callback so the poller never sees a half-written file.
- PD16: inline_output_callback exceptions now re-raise as WorkerError
instead of being silently logged (root cause was buried 30s later
in a canonical-output timeout).
- CA9: WorkerConfig manual rebuild on --max-concurrent/--poll-interval
silently dropped new fields. Use dataclasses.replace, matching
round-4 P5 fix in master/__main__.py.
MEDIUMS (M3) — legacy_adapter quality upgrades:
- PD1: unrecognized confidence values now WARN instead of silently
defaulting to "medium" — surfaces agent prompt drift.
- PD2: estimator recommended_tier clamped to {0,1,2} so an out-of-
range int doesn't bypass the adapter's whole purpose.
- PD7: reviewer blocking_issues list-of-strings coerced into the
list-of-BlockingIssue-dict shape strict_parse requires.
- PD13: conflict_resolver prompt defaults tier=1 + warns instead of
raising; the scheduler always sets it but defends against drift.
- PD14: summarizer summary < 50 chars padded with a clear marker so
strict_parse accepts it (and the truncation is visible).
- PD15: implementer blockers capped at 4096 chars each so a buggy
agent can't blow up audit log / DB column.
- PD17: launch script accepts either FORGEJO_TOKEN or GITEA_TOKEN
with a clear error if both are unset.
- PD22: conflict_resolver adapter accepts singular commit_sha
fallback, matching implementer.
- CA4: every adapter invocation logs role + payload key fingerprint
so operators can measure agent-migration progress.
- estimator + summarizer now have explicit _start tools (the prompts
already referenced them; previously absent → first call would fail).
TESTS (M4) — added 40 tests in test_post_review_fixes.py:
- Cross-session MCP state reset (implementer + reviewer + estimator
+ summarizer; intra-session double-start still rejected).
- finalize_and_emit output_path precedence (arg > env > stdout),
parent-dir creation, rejection of relative/NUL paths, failed-write
leaves state retryable.
- legacy_adapter quality: tier clamping, blocker cap, non-string
commit warning, blocking_issues string coercion, conflict_resolver
full roundtrip + non-resolved head clearing, summarizer padding,
confidence warning, V1-passthrough no-log.
- opencode.json registration parity: every MCP the prompts name is
registered with the correct module path.
- Per-role prompts mention {role}_output.json (canonical poller path)
+ the "DO NOT emit chat-JSON" directive.
- FORGEJO_URL suffix-strip parametrized table.
- agent_runner stale-file cleanup: prior-attempt file is unlinked
before a new session can read it as phantom output.
Also updated 2 pre-existing tests for the CA8 / PD8 / PD13 behavior
changes (cancelled→wait, event_type renaming, conflict_resolver
default-tier warning).
Total: 741 → 781 tests, 0 regressions.
DEFERRED (M5 follow-up — non-trial-blocking):
- CA5: head_sha verification via git cat-file (requires subprocess).
- CA11: discovery_interval_s wall-time cadence (vs iteration count).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
bcc59d38af |
fix(controller): trial-path output channel — harvest inline JSON + legacy-shape adapter
Trial run-1 surfaced the real failure mode: the existing agent
prompts emit their result as a single JSON object in the FINAL
assistant message (the legacy dispatch_implementer/dispatch_review
contract). They DON'T call the controller's response-builder MCPs
(not registered in opencode.json) and they DON'T write a file. The
worker's canonical-output poller timed out every attempt.
Fix (two pieces):
1. INLINE JSON HARVEST
``_opencode_worker.run_session_blocking`` already extracts the
last JSON object from the agent's final response into
``SessionResult.parsed_json``. The controller's opencode_session
adapter was ignoring it.
- ``opencode_session.py:run_opencode_session`` now accepts an
``inline_output_callback`` kwarg. After the session completes,
if parsed_json is set, the callback fires.
- ``agent_runner.py`` provides the callback: writes the parsed
JSON to the canonical-output path so the existing poller picks
it up uniformly with the MCP + file-write channels.
2. LEGACY → V1 SHAPE ADAPTER (``worker/legacy_adapter.py``)
The legacy JSON shape doesn't match V1 contracts:
- estimator: legacy {recommended_tier, is_confident, reasoning}
vs V1 {output_version, recommended_tier, is_metadata_only,
confidence, reasoning, wallclock_seconds}
- implementer: legacy {outcome: resolved|unresolved, ...} vs V1
{outcome: resolved|rebase-failed|blocked|noop|competence-failure,
commit_shas, blockers, used_tier, ...}
- reviewer: legacy {verdict, ...} vs V1 {output_version, verdict,
blocking_issues, suggested_next_action, ...}
``adapt_to_v1(role, payload, tier, wallclock_seconds)`` normalizes
each role's legacy shape into the corresponding V1 dict:
- adds ``output_version="V1"``
- maps outcome enums (``unresolved`` → ``blocked``)
- synthesizes missing required fields with sensible defaults
- coerces singular commit_sha → list[commit_shas]
- converts legacy is_confident bool → confidence string
Already-V1 payloads pass through unchanged. The agent_runner
wraps the inline_output_callback to run the adapter before
writing to the canonical path.
Tests (+12 in test_legacy_adapter.py):
- Each role's adapt: legacy in, V1-validating-via-strict_parse out
- Already-V1 passthrough
- Outcome enum mappings + invalid-outcome fallback
- Singular commit_sha normalization
- Non-dict / unknown-role passthrough (lets strict_parse raise)
Total: 738 controller tests pass (+12 net), 0 regressions.
Long-term: update agent prompts to emit V1 directly, register the
MCPs in opencode.json. For the trial, this adapter lets the existing
agents flow through the new controller unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
1b09a2054f |
feat(controller): Phase 1i — per-role prompt templates
Replace the JSON-dump prompt stub with structured per-role builders
that render the V1 input contract fields in human-friendly sections.
The agent's system prompt (.opencode/agents/{name}.md) provides the
HOW (tool calls, response shape); this module provides the WHAT (the
per-attempt context drawn from input_payload).
Per-role builders in tools/controller/worker/prompts.py:
- build_implementer_prompt — branch coords, CI summary (highlights
failing gates), active reviewer state, new comments, prior attempts,
optional allowed_files scope, output contract reminder
- build_reviewer_prompt — PR coords, CI summary, full diff (with
code fence), implementer's latest claim, prior reviewer outputs
- build_estimator_prompt — PR/issue context (title, body, diff
summary, head_sha), tier-pick guidance
- build_conflict_resolver_prompt — coords, conflicted files (or
porcelain fallback), prior implementer outputs
- build_summarizer_prompt — aged-out attempt + prior running
summary; instructs terse condensation
- build_prompt(role, tier, input_payload) — dispatcher
Truncation budgets (in chars):
- full_diff: 12,000
- pr_body / summaries: 4,000
- prior-attempt JSON: 2,000
- comment one-liner: 200
Wired as the default prompt builder in opencode_session via
``_default_prompt_for``. wire_opencode_session() callers that pass
prompt_builder=... override per-test.
Tests (+35 in test_worker_prompts.py, 1 updated adapter test, 0
regressions):
- Per-role: section headers present, required fields rendered,
missing fields fall back to (unavailable), oversize fields
truncated
- Dispatcher: routes by role; raises ValueError for missing tier on
tiered roles + unknown role
- Adapter integration: _default_prompt_for delegates to build_prompt
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
bb8d872ba7 |
feat(controller): Phase 1d-4 — OpenCode session adapter
The production wrapper that wires the existing
_opencode_worker.run_session_blocking into the controller's
run_opencode_session protocol that production_agent_runner expects.
tools/controller/worker/opencode_session.py:
- agent_name_for(role, tier): role+tier → OpenCode agent name.
- implementer + tier {0,1,2} → task-implementor-tier-{0,1,2}
- reviewer → pr-review-worker
- estimator → estimator-implementation
- conflict_resolver → conflict-resolver-worker
- summarizer → controller-summarizer (new agent name)
- DEFAULT_TIER_TIMEOUT_S: per-tier wallclock budgets via env vars.
Defaults match plan v9 (tier 0: 600s, tier 1: 1200s, tier 2: 1800s).
Reviewer/estimator/conflict use the default-agent timeout (600s).
- wire_opencode_session(opencode_server_url, ...) → callable matching
the production_agent_runner's run_opencode_session contract.
- Resolves role+tier to agent name + per-tier timeout
- Builds the prompt (default stub or custom builder; per-role prompt
templates are a follow-up)
- Calls run_session_blocking with an on_poll callback that raises
WorkerLostLock if the controller's lost_lock_check returns True
mid-session (heartbeat thread reaper detected stolen lock)
- Routes SessionResult.status:
- 'completed' → return None (MCP's canonical output is in the
tempfile; production_agent_runner reads it after we return)
- 'timeout' → raise WorkerError(worker-internal-error)
- 'transport-error' → raise WorkerError(worker-internal-error,
including error_kind for forensics)
- unknown → raise WorkerError defensively
- Any unexpected exception from run_session_blocking wrapped as
WorkerError(worker-internal-error).
- Production wiring: dependency injection. wire_opencode_session()
lazy-imports the real run_session_blocking when not overridden.
Tests inject a stub.
22 new tests in test_worker_opencode_session.py:
- agent_name_for: parametrized per role (4 implementer-tier rows +
4 flat-role rows) + invalid tier + unknown role
- DEFAULT_TIER_TIMEOUT_S: each tier has a timeout + monotonic
- wire_opencode_session: completed/timeout/transport-error/unknown-
status routing; run_session_blocking raises wrapped as WorkerError;
on_poll propagates lost_lock; on_poll no-op when lock held;
custom prompt_builder used; default prompt includes role + tier
+ finalize() instruction + rendered input_payload; tag prefix
customizable.
Total: 398 controller tests; full auto_agents suite 2760 pass.
|