751dbfed337bc18ec6dd96cc7b3f241bb3207acb
22 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 drew
|
2ad0958cb7 |
fix(controller): batch O — trial-2 fix: interpolate real attempt_id into prompts
Trial run-2 surfaced the actual blocker: the response-builder MCPs
shared their module-global ``_STATE`` across OpenCode sessions (they're
registered as ``type: local`` in opencode.json, so OpenCode spawns one
subprocess per OpenCode-server lifetime, not per session). The CA2/PD5
cross-session reset I added in the M-batch keyed on
``identity.attempt_id`` — but the prompt template literally read:
1. ``estimator_start(attempt_id=..., workflow_id=..., ...)``
The ``...`` were placeholder ellipses, not interpolated. So agents
guessed ``attempt_id=1`` every single time. The cross-session reset
compared ``1 == 1``, decided "same attempt — no reset", and the prior
attempt's ``finalized=True`` persisted forever. Every estimator
session after the first failed with ``"response already finalized;
no further mutations allowed"`` on every ``_set_*`` call → no
finalize → 30s worker timeout → workflow STUCK.
Observed in trial run-2: 2 successes (attempts 1, 2), then 15
consecutive failures (attempts 3–17 across all 6 workflows) before
the pickup_guard would have STUCK every workflow.
Two fixes (defense in depth):
1. **Unconditional reset on _start** in all 5 builders. We can't
distinguish "agent retry in same session" from "new session reusing
this MCP" reliably — the observable signature is identical. Just
reset whenever ``_STATE.started`` is True; ``reset_for_new_attempt``
logs a WARN if the prior state was in-flight so abandoned attempts
are still visible to ops. The agent's last ``_start`` always wins.
2. **Interpolate real attempt_id + workflow_id + pr_number + head_sha
into all 5 prompts' Output contract sections**. The agent_runner
now injects ``attempt_id`` into ``input_payload`` (matching the
existing ``workspace_dir`` injection pattern). Each ``build_*_prompt``
reads ``input_payload.get("attempt_id")`` and bakes the concrete
value into the MCP-call signature shown to the agent, plus a
``PASS THESE EXACT VALUES`` directive.
Also strengthened each prompt's ``_finalize`` line with
``**You MUST call this tool — without it the controller times out.**``
so the agent understands the contract is hard, not optional.
Updated tests:
- ``test_double_start_refused`` → ``test_double_start_resets_silently``:
pins the new permissive-reset behavior + asserts the WARN log fires.
- ``test_implementer_rejects_intra_session_double_start`` updated for
same reason; now asserts the second _start succeeds + last-wins
semantics (used_tier == new tier).
- NEW ``test_prompt_interpolates_real_attempt_id`` parametrized over
all 5 roles: asserts ``attempt_id=42`` + ``workflow_id=7`` appear
literally in the rendered prompt AND that ``attempt_id=...`` /
``workflow_id=...`` placeholders DO NOT.
Total: 795 → 800 tests, 0 regressions.
The model-override files (.md + .txt) are still in place from the
prior turn — that change is orthogonal to this bug fix; OpenCode
ignores the dispatcher's pass-through per the README, and the actual
generation has been on haiku the whole time. The .md frontmatter
change to sonnet will only take effect on the next OpenCode restart.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
a4892f8db9 |
fix(controller): batch N — second-round adversarial review fixes
Round-2 adversarial review on commit
|
||
|
drew
|
84da774212 |
fix(controller): batch M — post-Phase-1m adversarial review fixes
Three rounds of adversarial review (Chief Architect / Principal Dev /
Senior Test Engineer) on commits 3ca794be7..db12f45ac surfaced ~35
issues. This commit addresses 25+ across criticals, highs, and
mediums, and adds 40 new tests covering the changes plus key gaps the
review identified.
CRITICALS (M1):
- CA1: stale {role}_output.json from a prior attempt on the same
per-PR workspace was readable as "fresh" output of the new attempt.
agent_runner now unlinks the MCP-canonical path AND every fallback
path BEFORE the session runs.
- CA2/PD5: opencode.json-registered MCP subprocesses persist across
OpenCode sessions, but BuilderState was module-singleton. Added
reset_for_new_attempt() + cross-session detection (compare
identity.attempt_id) to every *_start; force-resets with WARN if
prior attempt was interrupted (timeout / lost lock).
- PD3: inline-JSON callback could overwrite an MCP-written canonical
V1 file with adapted-from-prose garbage. Callback now inspects
existing files and skips when V1 is already present.
- PD4: FORGEJO_URL = .rstrip("/api/v1") is a character-set strip —
catastrophic for hosts whose path contains /v1 in the middle.
Replaced with explicit endswith()-based suffix strip.
- CA10: clone URL embedded $FORGEJO_TOKEN, persisted into
.git/config where any agent could cat it. Token now sourced via
local credential.helper at clone-time, URL kept clean.
- CA12: state.finalized was set BEFORE the file write, so disk-full
/ OSError left the agent unable to retry finalize. Reordered.
HIGHS (M2):
- CA3/PD12: output_path validation (NUL-byte rejection, must be
absolute, parent-not-file check) in finalize_and_emit.
- CA6: ci_status_poll SELECT only considered implementer attempts;
conflict_resolver also pushes commits. SQL now unions both roles.
- PD9: ci_status_poll could advance on a stale "resolved" SHA from a
blocked attempt (whose head_sha_after == head_sha_before). Added
outcome='resolved' filter.
- CA8: cancelled/stale CI states mapped to ci_red_retry_same_tier,
burning pickup_count on healthy PRs. Both now wait (treated as
operator/system action, not failure). timed_out stays red.
- TE9: unknown Forgejo CI states now WARN-log instead of silently
being treated as pending — operators see new state strings.
- PD8: ci_status_poll event_type strings standardized to match the
state-machine event names (ci_green / ci_red_retry_same_tier)
instead of legacy ci-green / ci-red.
- CA7: inline-JSON callback now checks lost_lock_check BEFORE write
so a file isn't staged after lock loss.
- PD10: atomic .tmp + os.replace writes in both MCP finalize and
inline callback so the poller never sees a half-written file.
- PD16: inline_output_callback exceptions now re-raise as WorkerError
instead of being silently logged (root cause was buried 30s later
in a canonical-output timeout).
- CA9: WorkerConfig manual rebuild on --max-concurrent/--poll-interval
silently dropped new fields. Use dataclasses.replace, matching
round-4 P5 fix in master/__main__.py.
MEDIUMS (M3) — legacy_adapter quality upgrades:
- PD1: unrecognized confidence values now WARN instead of silently
defaulting to "medium" — surfaces agent prompt drift.
- PD2: estimator recommended_tier clamped to {0,1,2} so an out-of-
range int doesn't bypass the adapter's whole purpose.
- PD7: reviewer blocking_issues list-of-strings coerced into the
list-of-BlockingIssue-dict shape strict_parse requires.
- PD13: conflict_resolver prompt defaults tier=1 + warns instead of
raising; the scheduler always sets it but defends against drift.
- PD14: summarizer summary < 50 chars padded with a clear marker so
strict_parse accepts it (and the truncation is visible).
- PD15: implementer blockers capped at 4096 chars each so a buggy
agent can't blow up audit log / DB column.
- PD17: launch script accepts either FORGEJO_TOKEN or GITEA_TOKEN
with a clear error if both are unset.
- PD22: conflict_resolver adapter accepts singular commit_sha
fallback, matching implementer.
- CA4: every adapter invocation logs role + payload key fingerprint
so operators can measure agent-migration progress.
- estimator + summarizer now have explicit _start tools (the prompts
already referenced them; previously absent → first call would fail).
TESTS (M4) — added 40 tests in test_post_review_fixes.py:
- Cross-session MCP state reset (implementer + reviewer + estimator
+ summarizer; intra-session double-start still rejected).
- finalize_and_emit output_path precedence (arg > env > stdout),
parent-dir creation, rejection of relative/NUL paths, failed-write
leaves state retryable.
- legacy_adapter quality: tier clamping, blocker cap, non-string
commit warning, blocking_issues string coercion, conflict_resolver
full roundtrip + non-resolved head clearing, summarizer padding,
confidence warning, V1-passthrough no-log.
- opencode.json registration parity: every MCP the prompts name is
registered with the correct module path.
- Per-role prompts mention {role}_output.json (canonical poller path)
+ the "DO NOT emit chat-JSON" directive.
- FORGEJO_URL suffix-strip parametrized table.
- agent_runner stale-file cleanup: prior-attempt file is unlinked
before a new session can read it as phantom output.
Also updated 2 pre-existing tests for the CA8 / PD8 / PD13 behavior
changes (cancelled→wait, event_type renaming, conflict_resolver
default-tier warning).
Total: 741 → 781 tests, 0 regressions.
DEFERRED (M5 follow-up — non-trial-blocking):
- CA5: head_sha verification via git cat-file (requires subprocess).
- CA11: discovery_interval_s wall-time cadence (vs iteration count).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
db12f45acb |
feat(controller): Phase 1m — wire response-builder MCPs into OpenCode (Option A)
The trial-path expedient (legacy_adapter harvesting chat-JSON) is now
the SAFETY NET; the architecture-intended path (agents call the
controller's response-builder MCPs, which validate against V1 +
write canonical JSON) is now wired end-to-end.
Four pieces:
1. MCP `finalize` accepts `output_path` argument
``mcp/_builder_base.finalize_and_emit`` adds ``output_path: str | None``
kwarg. Precedence: explicit arg > ``CONTROLLER_CANONICAL_OUTPUT_PATH``
env > stdout. This solves the per-attempt path problem that
blocked opencode.json registration (the env var is static; the
per-attempt path comes from the prompt, the agent passes it as a
tool arg). Each role's finalize updated:
- estimator_finalize(output_path)
- implementer_finalize(output_path)
- reviewer_finalize(output_path)
- conflict_finalize(output_path)
- summarizer_finalize(output_path)
Creates parent directory if missing (so the controller doesn't
need to pre-create). Returns ``wrote_to`` in the ok dict so tests
can pin the path.
2. opencode.json registers the 5 controller MCPs
``.opencode/opencode.json`` adds:
- implementer-response-builder
- reviewer-response-builder
- estimator-response-builder
- conflict-resolver-response-builder
- summarizer-response-builder
Each spawned via ``python -m tools.controller.mcp.{role}_builder``
with PYTHONPATH=/repo-root so the controller imports resolve.
3. Controller prompt builder injects the EXACT tool sequence
``worker/prompts.py`` rewrites each role's "Output contract"
section. The old "PREFERRED MCP / FALLBACK file-write" instruction
becomes a single REQUIRED contract: numbered tool calls (``X_start``,
``X_set_*``, ``X_finalize(output_path=...)``) with the explicit
per-attempt path baked in. Explicit "DO NOT emit a JSON object in
your final chat message" instruction to override the legacy
contract baked into the agent system prompts.
4. Agent permission whitelists include the new MCPs
- ``.opencode/agents/task-implementor.md`` (source for tier-{0,1,2,min}
variants — regenerated via tools/sync_tier_models.py)
- ``.opencode/agents/pr-review-worker.md``
- ``.opencode/agents/estimator-implementation.md``
- ``.opencode/agents/conflict-resolver-worker.md``
Each adds the matching ``"{role}*": allow`` pattern.
Safety net preserved:
The legacy_adapter (commit
|
||
|
drew
|
bcc59d38af |
fix(controller): trial-path output channel — harvest inline JSON + legacy-shape adapter
Trial run-1 surfaced the real failure mode: the existing agent
prompts emit their result as a single JSON object in the FINAL
assistant message (the legacy dispatch_implementer/dispatch_review
contract). They DON'T call the controller's response-builder MCPs
(not registered in opencode.json) and they DON'T write a file. The
worker's canonical-output poller timed out every attempt.
Fix (two pieces):
1. INLINE JSON HARVEST
``_opencode_worker.run_session_blocking`` already extracts the
last JSON object from the agent's final response into
``SessionResult.parsed_json``. The controller's opencode_session
adapter was ignoring it.
- ``opencode_session.py:run_opencode_session`` now accepts an
``inline_output_callback`` kwarg. After the session completes,
if parsed_json is set, the callback fires.
- ``agent_runner.py`` provides the callback: writes the parsed
JSON to the canonical-output path so the existing poller picks
it up uniformly with the MCP + file-write channels.
2. LEGACY → V1 SHAPE ADAPTER (``worker/legacy_adapter.py``)
The legacy JSON shape doesn't match V1 contracts:
- estimator: legacy {recommended_tier, is_confident, reasoning}
vs V1 {output_version, recommended_tier, is_metadata_only,
confidence, reasoning, wallclock_seconds}
- implementer: legacy {outcome: resolved|unresolved, ...} vs V1
{outcome: resolved|rebase-failed|blocked|noop|competence-failure,
commit_shas, blockers, used_tier, ...}
- reviewer: legacy {verdict, ...} vs V1 {output_version, verdict,
blocking_issues, suggested_next_action, ...}
``adapt_to_v1(role, payload, tier, wallclock_seconds)`` normalizes
each role's legacy shape into the corresponding V1 dict:
- adds ``output_version="V1"``
- maps outcome enums (``unresolved`` → ``blocked``)
- synthesizes missing required fields with sensible defaults
- coerces singular commit_sha → list[commit_shas]
- converts legacy is_confident bool → confidence string
Already-V1 payloads pass through unchanged. The agent_runner
wraps the inline_output_callback to run the adapter before
writing to the canonical path.
Tests (+12 in test_legacy_adapter.py):
- Each role's adapt: legacy in, V1-validating-via-strict_parse out
- Already-V1 passthrough
- Outcome enum mappings + invalid-outcome fallback
- Singular commit_sha normalization
- Non-dict / unknown-role passthrough (lets strict_parse raise)
Total: 738 controller tests pass (+12 net), 0 regressions.
Long-term: update agent prompts to emit V1 directly, register the
MCPs in opencode.json. For the trial, this adapter lets the existing
agents flow through the new controller unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
f93f0d1c53 |
fix(controller): round-4 P2 — defensive input_payload normalization
If the DB returned NULL input_payload (legacy/seeded data, schema bug, hand-edited rows), agent_runner crashed on ``dict(None)`` → WorkerError → master re-pickups → STUCK after MAX_PICKUPS reaps. Silent infinite-loop until exhaustion. Fix: ``worker/runner.py:run_one_attempt`` checks isinstance(dict) on entry; substitutes empty dict + logs WARNING. Agent still runs normally with an empty payload. Test: ``test_none_input_payload_doesnt_crash`` — passes input_payload=None + verifies the attempt completes normally. Total: 712 controller tests pass (+1 net). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
f57d9f9478 |
fix(controller): batch L — round-4 trial-blockers (A1, P1, P3, P4, P5, T5)
Round-4 adversarial review found 5 trial-blockers + 1 silent-debt
item the post-round-3 deep pass missed. All fixed.
A1 — pre-clone the workspace so the agent has a worktree to operate on
``worker/__main__.py``: the agent_runner closure now constructs a
``PerPRWorkspace`` from input_payload.owner/repo/pr_number + the
FORGEJO_URL+FORGEJO_TOKEN env vars. Pre-flight:
- ``workspace.ensure_present()`` creates the dir skeleton.
- ``workspace.clone_if_absent()`` clones the repo into
``{workspace_dir}/worktree/`` if not already present (idempotent).
- ``workspace.fetch_and_validate(head_sha, head_ref)`` refreshes +
verifies the workspace is at the expected head. ``StaleInputError``
→ ``WorkerError(outcome='stale-input')`` so the master re-prefetches
without burning a pickup. ``RuntimeError`` → ``worker-internal-error``.
Previously the agent saw an empty workspace_dir + had no repo.
P1 — partial-write defense in the canonical-output poller
``worker/agent_runner.py:_wait_for_canonical_output`` now polls each
path with a two-pass quiescence check (size stable + content parses
as JSON) before returning. Partial writes (agent crashed mid-flush)
are skipped + the polling loop continues. The previous
``f.read().strip()`` returned partial JSON which then tripped
``ContractValidationError`` → ``worker-internal-error`` with no
record of WHICH path; now logs source path on every read.
P3 — TOCTOU defense in promote_discovered
``master/promote.py``: the UPDATE now filters
``current_state='DISCOVERED'``. If a concurrent reconciliation
moved the row off DISCOVERED between SELECT and UPDATE, rowcount=0
+ we skip the event-row write. No duplicate audit entry; no
overwriting a pause-by-label-removal.
P4 — explicit tuple-length validation in reconciliation_args + discovery_args
``master/loop.py``: previously a 6-tuple silently fell into the
``else`` 4-tuple unpack, raised ValueError("too many values"), got
swallowed by the per-iter ``except Exception``, and reconciliation
silently died forever. Now: ``elif n == 4`` + ``else: raise TypeError``.
The TypeError still hits the per-iter except (so the loop doesn't
crash) but ``logger.exception`` surfaces the actionable message in
journald. Operator sees "reconciliation_args must be a 4- or 5-tuple;
got length 6" instead of zero indication.
P5 — --tick-interval CLI flag preserves other config fields
``master/__main__.py``: replaced the manual ``MasterConfig(...)``
rebuild (which dropped reconciliation/ci_poll/discovery intervals)
with ``dataclasses.replace(cfg_loop, tick_interval_s=args.tick_interval)``.
Operators who pass --tick-interval no longer silently revert the
other intervals to defaults.
T5 — scheduler._commit_escalation uses safe_json_dumps
``master/scheduler.py``: the escalation event row's payload was the
only call site that bypassed safe_json_dumps. Now consistent — a
future contributor adding a datetime/Decimal field won't trip raw
json.dumps at runtime.
Tests (+4 net):
- ``test_worker_agent_runner.py::test_partial_write_not_read``: pins
P1 (truncated fallback file + valid MCP output → MCP wins).
- ``test_master_promote.py::test_toctou_state_change_between_select_and_update``:
pins P3 (steal state via monkey-patch → no double-promotion, no
extra event row).
- ``test_master_loop.py::test_reconciliation_args_wrong_length_logs_not_silent``:
pins P4 (6-tuple → logged error, not silent forever).
- ``test_entry_points.py::test_tick_interval_flag_preserves_other_cfg_fields``:
pins P5 (env-set non-default intervals survive --tick-interval).
Total: 711 controller tests pass (+4 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
7bf39a0b51 |
fix(controller): RB8 — head_sha bookkeeping in _write_outcome
The worker's _write_outcome never wrote head_sha_before / head_sha_after
on the workflow_attempts row. But tick.py reads both to compute
head_sha_advanced, which the outcome mapper REQUIRES to distinguish
``implementer_pushed`` (true push) from ``implementer_blocked``
(worker said resolved but git didn't move).
Without these columns set, every implementer attempt's "resolved"
outcome mapped to a no-op event, and workflows would stall
permanently after the agent ran successfully.
Fix:
- ``worker/runner.py:_write_outcome`` accepts ``head_sha_before`` +
``head_sha_after`` kwargs and writes both columns in the UPDATE.
- ``run_one_attempt`` computes:
- head_sha_before = input_payload["head_sha"] (what the agent
was told to start from)
- head_sha_after = output_payload["commit_shas"][-1] if any
commits were produced; falls back to head_sha_before otherwise
so tick.py correctly sees "no advance" when the agent didn't
commit.
Tests (+2):
- test_head_sha_before_after_recorded — pins the happy path where
the agent committed and head_sha advanced.
- test_no_commits_means_head_sha_unchanged — pins the no-advance
case where head_sha_after equals head_sha_before.
Total: 707 controller tests pass (+2 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
0c9e8fdd69 |
fix(controller): trial-path output channel — fallback file-write for unwired MCP layer
RB7 (newly discovered) — response-builder MCPs are NEVER REACHABLE
from the OpenCode agent:
The controller spawns the per-attempt response-builder MCP via
``python -m tools.controller.mcp.{role}_builder`` with stdin=PIPE
+ CONTROLLER_CANONICAL_OUTPUT_PATH set. But:
- ``.opencode/opencode.json`` does NOT register the controller MCPs.
- OpenCode only knows about MCPs it sees in opencode.json (graphify,
ci, forgejo, git, handoff).
- The OpenCode agent has NO transport to call
``implementer_finalize`` because OpenCode doesn't see the MCP.
- The controller's spawned MCP subprocess sits waiting for JSON-RPC
on stdin that no one sends.
- The agent does work, the session ends, the canonical-output-file
is empty, agent_runner times out with WorkerError.
This would have been the single biggest blocker to running an actual
Phase 2 trial. The MCP layer is structurally complete but not
plumbed into OpenCode — pivoting requires either registering them
in opencode.json (which doesn't support per-attempt env injection)
OR adding a fallback channel.
Fix (trial-path):
- ``worker/prompts.py``: each role's output-contract section now
documents BOTH the PREFERRED MCP call AND a FALLBACK direct
file-write to ``{workspace_dir}/{role}_output.json``. The agent
can pick either; once the MCP layer is wired (Phase 1m) it'll
naturally prefer the MCP.
- ``worker/agent_runner.py:_wait_for_canonical_output``: now polls
the MCP-canonical path AND a list of ``fallback_paths``. Call
site adds ``{workspace_dir}/{role}_output.json`` to fallback
paths. First non-empty wins.
Tests (+2):
- ``test_fallback_file_picked_up_when_mcp_silent``: agent writes
only the fallback file (no MCP drive); runner reads it.
- ``test_first_nonempty_output_wins``: documents the
poll-both-channels semantic.
Documented Phase-1m work:
- Register controller MCPs in ``.opencode/opencode.json``
- Solve per-attempt env injection (the canonical_output_path is
per-attempt; opencode.json env is static — needs a request_id
arg on finalize() OR a workspace-relative output convention)
Until that's done, the trial uses the fallback file-write path.
Total: 705 controller tests pass (+2 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
251eeb21ff |
fix(controller): more pipeline run-blockers — merging tick, periodic discovery, worker create_all, systemd ordering
Continuing the round-3 deep-pass cleanup. Three more run-blockers + one robustness fix. RB5 — MERGING handler never invoked from master loop: ``run_merging_tick`` was exported by the master package but no caller fired it. Workflows that transition to MERGING (via reviewer approval) would sit there indefinitely with no Forgejo merge call. Fix: - ``master/loop.py`` accepts a ``merging_args=(owner, repo, merge_callback)`` kwarg. When set, the tick fires every iteration (cheap if no workflows in MERGING). - ``MasterTickReport`` gains ``merging: MergingHandlerReport | None``. - ``master/__main__.py`` wires it from the Forgejo callback bundle. RB6 — periodic discovery never fires: ``run_discovery`` was only called at startup via ``run_startup_backfill`` + the ``--discovery-only-once`` smoke flag. PRs created after master startup would not be discovered until the master restarted. Fix: - ``master/loop.py`` accepts ``discovery_args=(owner, repo, list_prs, list_issues)`` or the 5-tuple with kwargs. Periodic tick on its own cadence (``CONTROLLER_DISCOVERY_INTERVAL_S``, default 30s). - ``MasterTickReport`` gains ``discovery: DiscoveryReport | None``. - ``master/__main__.py`` wires it + threads ``require_opt_in_label`` through. RB-robust — worker calls create_all defensively: Master is normally responsible for schema creation (workers run After= it via systemd ordering). But if the worker is started in isolation (test / local dev / unit ordering broken), it'd crash on the first query against missing tables. Fix: - ``worker/__main__.py`` calls ``create_all(engine)`` after ``build_engine``. ``create_all`` is idempotent (CREATE TABLE IF NOT EXISTS); safe to call from both master + worker. - ``cleveragents-controller-worker@.service`` adds ``After=cleveragents-controller-master.service`` + ``Wants=cleveragents-controller-master.service`` so systemd enforces the start ordering in production. Total: 703 controller tests pass (no test changes; all new wiring is exercised by master_main_loop tests via the new kwargs). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
febb352618 |
fix(controller): pipeline run-blockers — promoter, scheduler, owner/repo, workspace_dir patch
Round-3 deep pass identified four issues that would have prevented an
actual end-to-end pipeline run:
RB1 — DISCOVERED → ANALYZING never fired in production:
The state machine defines (DISCOVERED, discovery_picked_up) →
ANALYZING but NO production code fires the event. Workflows
created by discovery would sit in DISCOVERED forever.
Fix:
- New ``master/promote.py``: ``run_promote_discovered_tick`` scans
for DISCOVERED workflows + fires ``discovery_picked_up`` via
apply_event (state-machine invariants stay enforced) + emits a
``discovery-promoted`` controller_events row per transition.
- Composes with the master loop's other ticks; runs every iteration
(cheap — typically 0-1 row).
RB2 — scheduler.schedule_next_attempts never called from master loop:
The scheduler was exported by the master package but never invoked.
It creates the ``workflow_attempts`` rows that workers dequeue —
without it, workers would have nothing to pick up.
Fix:
- ``master/loop.py`` now accepts a ``prefetch: PrefetchCallback``
kwarg. When provided, the loop runs promote_discovered + scheduler
every iteration after tick/reaper/reconciliation.
- ``MasterTickReport`` gains ``promote_discovered`` and ``scheduler``
optional fields so on_iteration callbacks see both.
- ``master/__main__.py`` builds a ``PrefetchDataCallbacks`` from the
Forgejo callback bundle and constructs the production
``make_prefetch_callback(engine, callbacks)`` — wires through to
the loop's new prefetch kwarg.
RB3 — owner / repo missing from V1 input contracts:
The implementer / reviewer / estimator / conflict-resolver V1 inputs
had pr_number but not owner/repo. The OpenCode agent would have
had no way to know which Forgejo repo to clone — it would have had
to derive owner/repo from process env, coupling the worker to a
single repo.
Fix:
- ``contracts/v1.py``: added ``owner: str`` and ``repo: str``
(min_length=1) to ImplementerInputV1, ReviewerInputV1,
EstimatorInputV1, ConflictResolverInputV1.
- ``master/prefetch.py``: builders populate owner/repo from the
Workflow row (already known at prefetch time).
- Existing test fixtures in ``test_contracts_v1.py`` updated.
RB4 — input_payload.workspace_dir placeholder reached the agent:
Prefetch wrote ``workspace_dir = "<worker-injected>"`` as a
placeholder; the worker never patched it before invoking the
OpenCode session. The prompt builder rendered the literal
placeholder string into the agent's prompt — the agent had no idea
where to clone.
Fix:
- ``worker/agent_runner.py``: patches input_payload.workspace_dir
with the real path immediately before calling run_opencode_session.
Uses a shallow copy so the caller's dict isn't side-effected.
- ``worker/__main__.py``: workspace_dir naming convention is now
``pr-{owner}-{repo}-{pr_number}`` (matches workspace.py's
PerPRWorkspace convention) so the janitor's pr-* glob + the
agent's expected workspace location agree. Falls back to
``pr-attempt-{N}`` for legacy input_payloads missing owner/repo.
Tests:
- ``test_master_promote.py`` (NEW, +7 tests):
- empty DB no-op
- single workflow promoted
- multiple promoted in one tick
- only DISCOVERED targeted (non-DISCOVERED untouched)
- controller_events row emitted with correct shape
- idempotent after first promotion
- LoopIntegration end-to-end: DISCOVERED → ANALYZING → pending
estimator attempt visible in workflow_attempts (pins the entire
previously-broken pipeline from discovery to enqueue)
Total: 703 controller tests pass (+7 net), 0 regressions.
Without these four fixes, the pipeline would have looked alive in
unit tests but produced zero work in a real deployment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
c4a7f0027d |
fix(controller): batch H — round-2 important items (N5-N10)
Six items from the round-2 adversarial review's "important" list. N5 — restricted JSON encoder replaces ``default=str``: ``default=str`` silently stringified custom objects, sets, and bytes to ``"<MyObj at 0x...>"`` — masking worker output bugs. Now uses a restricted encoder (``tools/controller/_json_safe.safe_json_dumps``) with an allowlist: - datetime / date → ISO-8601 string - Decimal → str (preserves precision) - UUID → canonical string - Path → str - set / frozenset → sorted list (best-effort) - Everything else → TypeError (a worker output regression surfaces loudly instead of writing garbage to the DB) Replaces ``json.dumps(..., default=str)`` at: - ``worker/runner.py`` (terminal-state UPDATE write) - ``master/scheduler.py`` (input_payload INSERT + UPDATE) Tests: ``test_json_safe.py`` (+15 tests) — every allowlisted type + rejected types (custom class, bytes, complex) + nested structures + kwargs forwarding. N6 — strict parser check now runs BEFORE backfill + loop: Previously the strict-parser exit could run AFTER backfill (the test ``test_strict_parser_coverage_blocks_startup_with_stubs`` passed because backfill's ``fail_if_called`` AssertionError was swallowed by the bare ``except Exception``, then strict exited 2). The test asserted the right outcome via the wrong path. Fix: - ``master/__main__.py``: parser-coverage check moved to immediately after engine creation, BEFORE backfill + loop. Strict-mode failure exits 2 without wasting a Forgejo round-trip + without dependent code paths firing. - Test refactored: count-based assertions on backfill and loop call counts (0 each) instead of fail_if_called. Catches regressions where the strict check moves back below either. N7 — _to_aware_datetime unit-tested in isolation: Previously exercised only via end-to-end comment-filter test. New ``TestToAwareDatetime`` (+12 tests) covers: None, empty string, Z suffix, +00:00 offset, microseconds preserved, naive datetime → UTC, malformed string → None, partial string → None, unsupported types → None, timezone abbreviations → None, equality across Z + offset forms (the regression the helper exists to defend). N8 — runtime=None lazy-import branch tested: Previously all 24 HTTP factory tests injected a fake runtime; the production path (``build_callbacks(cfg=None)`` → sys.path injection + lazy import of ``tools._claim_runtime``) was untested. ``TestBuildCallbacksDefaultRuntime`` (+2 tests): verifies the import succeeds + every ForgejoCallbacks attribute is callable; verifies the import is idempotent (second call doesn't crash on sys.path re-insert). N9 — resume event-row emission asserted: Round-1's batch B added the PAUSE event-row test (``test_label_removed_emits_event_with_reason``) but not RESUME. ``test_label_restored_emits_event_with_reason`` pins the resume's controller_events shape (from_state=PAUSED, to_state=<prior>, reason="opt-in-label-restored", source="reconciliation") so operators auditing the timeline see both pause + resume. N10 — concurrent SQLite dequeue test: The dequeue docstring claims SQLite BEGIN-DEFERRED concurrent dequeues "retry via busy_timeout (5s)" — but no test verified. ``TestSQLiteConcurrentDequeue::test_two_threads_racing_one_wins`` spawns two threads, both attempt dequeue simultaneously via a threading.Barrier. Asserts: neither thread raises SQLITE_BUSY- without-retry, exactly one acquires the attempt, the loser sees no_pending_eligible (winner committed first). Total: 691 controller tests pass (+31 net), 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
147e3403c1 |
fix(controller): batch G — show-stoppers from round-2 review (N1–N4)
Four items the round-2 adversarial review flagged as ship-blockers.
N1 — PID-reuse defense is now WIRED in production:
Round 1's batch D shipped ``subprocess_starttime`` in the sidecar +
janitor checks against it, but NO production code wrote sidecars.
The defense was unwired; tests passed against a code path that
production never invoked.
Fix:
- ``worker/agent_runner.py`` accepts ``workspace_dir`` and
``opencode_server_url`` kwargs. When ``workspace_dir`` is set, it
writes a sidecar (``{workspace_dir}/worker.session``) immediately
after MCP spawn capturing the real PID + starttime from
``/proc/{pid}/stat`` field 22. Removes it on attempt completion.
- ``worker/__main__.py`` builds the per-attempt workspace dir
(``{workspace_root}/pr-attempt-{N}/``) and threads it through the
agent_runner closure with the OpenCode URL. The naming convention
is picked up by the janitor's ``pr-*`` glob; when per-PR shared
workspaces ship (Phase 1k++ follow-up), it changes to
``pr-{owner}-{repo}-{N}``.
- Test: ``TestSidecarWiring`` (+2 tests) verifies the sidecar appears
during the attempt, carries the right PID + starttime + instance,
and is cleaned up post-attempt.
N2 — AWAITING_CI escape event firing is now WIRED in production:
Round 1's batch D shipped ``ci_polling_exhausted`` /
``ci_flake_retries_exhausted`` in TRANSITIONS, but NO production code
emitted them. Workflows could still hang in AWAITING_CI forever.
Fix:
- New ``master/ci_poll.py``: ``run_ci_poll_exhaustion_tick`` scans
workflows whose ``entered_state_at`` is older than
``CONTROLLER_AWAITING_CI_TIMEOUT_S`` (default 7200s) and fires
``ci_polling_exhausted`` via ``apply_event`` → STUCK + emits a
``ci_poll_exhausted`` controller_events row with the threshold
payload.
- ``master/loop.py`` integrates the new tick on its own cadence
(``ci_poll_exhaustion_interval_s`` env, default 300s). Composes
with the existing master loop. ``MasterTickReport`` gains
``ci_poll_exhaustion: CIPollExhaustionReport | None``.
- Tests: ``test_master_ci_poll.py`` (+7 tests) — happy path, fresh
workflow stays untouched, only AWAITING_CI is targeted (other
long-lived non-terminal states ignored), event row shape pinned,
default threshold matches the documented 2h, end-to-end loop
integration (master_main_loop drives the exhaustion +
workflow → STUCK without operator intervention).
- Dialect-portable SQL (Postgres interval, SQLite julianday).
- Handles SQLite returning TIMESTAMP as str from text() queries
(no .isoformat() on str).
N3 — externally-merged/closed PRs now win over label removal:
Round-1's PAUSE-on-label-removed shipped, but reconciliation
checked the label gate BEFORE checking merged/closed. Operators
removing the opt-in label on an already-merged PR would PAUSE the
workflow forever — never transitioning to MERGED.
Fix:
- ``master/reconciliation.py:_reconcile_one`` re-ordered:
1. Check terminal-state mappings (merged/closed) FIRST — apply
immediately if they fire.
2. THEN the opt-in label gate (pause/resume).
3. Fall through to "consistent" otherwise.
- Tests: ``test_externally_merged_takes_priority_over_label_removal``
+ ``test_externally_closed_takes_priority_over_label_removal``
pin the contract. Both seed an IMPLEMENTING workflow + Forgejo
reporting "merged/closed AND no opt-in label" → workflow
transitions to MERGED/ABANDONED (not PAUSED) + pre_pause_state
stays None.
N4 — graceful handling of empty env vars:
``int(os.environ.get("CONTROLLER_FORGEJO_REQUEST_TIMEOUT_S", "30"))``
crashes with non-actionable ``int('') ValueError`` if the operator
sets the env to empty/whitespace (common when sourcing a partially-
edited /etc/cleveragents/master.env file).
Fix:
- ``master/forgejo_cfg.py:_env_int(name, default)`` — empty or
whitespace-only values fall back to the documented default; only
non-numeric values still raise (with a clear message naming the
variable).
- Tests: ``test_empty_env_value_falls_back_to_default`` +
``test_whitespace_only_env_falls_back`` + updated
``test_malformed_env_raises_value_error`` to match the new
"not a valid integer" wording.
Total: 660 controller tests pass (+13 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
d71046b9a0 |
fix(controller): batch D — PID-reuse, AWAITING_CI escape, flake bound, scheduler skip
Four safety items from the consolidated adversarial-review punch list.
ITEM 8 — PID-reuse hazard in janitor:
The janitor SIGKILL'd whatever process happened to live at the
sidecar's recorded subprocess_pid. Between sidecar write and janitor
sweep, the OS can reuse the PID for an unrelated process; the janitor
was killing innocents under fork-heavy workloads.
Fix:
- ``session_sidecar.py``: added ``subprocess_starttime`` field
(Optional[int]) + ``read_proc_starttime(pid)`` helper that reads
``/proc/{pid}/stat`` field 22 (clock ticks since boot — monotonic
for a (boot, pid) pair).
- ``WorkerSession.from_dict`` filters unknown keys so forward + back
compat with sidecars from earlier/later versions is preserved.
- ``janitor._pid_alive`` and ``_kill_with_grace`` accept
``expected_starttime``; on mismatch they short-circuit and DON'T
signal the impostor.
- ``_kill_with_grace`` return semantics tightened: True iff a signal
was actually delivered (False for "PID gone" / "PID reused"). The
``sessions_killed`` counter now reflects real kills.
ITEM 9 — AWAITING_CI escape from infinite poll:
Previously AWAITING_CI could only exit via ``ci_green`` /
``ci_red_*`` / ``ci_flake_retry`` — if CI hangs forever (runner
outage, broken integration, etc.) the workflow had no controller-
driven STUCK path; only operator_unstick could rescue it.
Fix: new ``ci_polling_exhausted`` event → STUCK. The master's
AWAITING_CI poll handler is the natural place to emit it once a
threshold passes (deferred to a follow-up — Phase 1k+ ships the
event in the table; the timer fires it).
ITEM 10 — ci_flake_retry was unbounded:
The ``ci_flake_retry`` self-loop on AWAITING_CI had no encoded
ceiling. Pathological flaky CI could loop forever (the docstring
said "retry once per gate" but nothing enforced it).
Fix:
- New ``workflows.ci_flake_retries_remaining`` column (server_default
'1', default 1 — operators tune via ``CONTROLLER_CI_FLAKE_RETRIES``
at startup or via direct UPDATE).
- New ``ci_flake_retries_exhausted`` event → ESCALATING. Master
decrements the column on each ci_flake_retry; at 0 the next CI
failure routes through ci_red_* (regular path) or this new
event (escalates if the operator wants a hard ceiling).
ITEM 11 — scheduler now skips PAUSED workflows:
Without this, the scheduler could enqueue a fresh attempt for a
PAUSED workflow between two reconciliation ticks (race: label
removed at T+0, reconciliation runs at T+300, scheduler ticks at
T+30 with stale DB state). The window is at most one attempt of
worker work.
Fix: ``schedule_next_attempts`` SQL now lists only
{ANALYZING, IMPLEMENTING, REVIEWING, CONFLICT_RESOLVING, ESCALATING}
explicitly; PAUSED is excluded by absence. Reconciliation owns the
PAUSED → resume transition; scheduler doesn't touch it.
Schema additions:
- ``workflows.ci_flake_retries_remaining`` (INTEGER NOT NULL DEFAULT 1)
- ``workflows.awaiting_ci_started_at`` (TIMESTAMP NULL) — for the
poll-exhaustion timer (timer impl deferred; column is staged).
Tests:
- TestReadProcStarttime — 3 tests (Linux skip-guard) for the
/proc/pid/stat parser (self-pid > 0, missing pid is None,
invalid pid is None).
- TestJanitor::test_pid_reuse_defended_via_starttime — pins the
contract end-to-end (real subprocess + fabricated wrong starttime
→ janitor doesn't signal).
- TestPhase1kPlusTransitions — 5 tests pinning the new events +
proving the load-bearing invariants still pass.
- test_scheduler_skips_paused_workflows — pins item 11.
Total: 603 controller tests pass (+10 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
6ba1926c52 |
fix(controller): batch A — datetime json, Forgejo state map, ISO compare, dequeue docs
Four narrow bug fixes flagged by adversarial code review (items 1, 5, 6, 7 from the consolidated critique). ITEM 1 — datetime → json.dumps crash (silent write-after-work failure): - ``worker/runner.py:274`` and ``master/scheduler.py:283`` now pass ``default=str`` to ``json.dumps`` so nested datetime fields (e.g. CISummary.observed_at) serialize without raising. - Before this fix: a worker would do its real work, then crash on the terminal-state UPDATE with TypeError, get recorded as ``worker-internal-error``, and the output payload would be lost. - Test: TestDatetimeSerializationSafety in test_master_ci_summarize + test_scheduler_handles_datetime_in_input_payload in test_master_prefetch (both pin the regression — the with-default test passes, the without-default test asserts the TypeError so future maintainers see the failure mode). ITEM 5 — Forgejo state mapping completeness: - Extended ``_FORGEJO_STATE_TO_GATE_STATUS`` in ``master/ci_summarize.py`` to cover ``cancelled``, ``timed_out``, ``action_required``, ``queued``, ``in_progress``, ``neutral``, ``skipped``, ``stale`` — states observed across Forgejo / Gitea / GH-mirror that previously collapsed to ``pending``, telling the implementer "CI is still running" when really a job was cancelled. - ``cancelled`` / ``timed_out`` / ``action_required`` / ``stale`` now map to ``error`` (the gate failed). - ``queued`` / ``in_progress`` stay ``pending`` (still running). - ``neutral`` / ``skipped`` → ``passed``/``skipped`` (informational). - Test: TestExtendedForgejoStates — 6 tests covering each new state. ITEM 6 — lexicographic ISO comparison drops/dupes comments: - ``master/prefetch.py:_comment_bodies_since`` and ``_iso`` replaced with ``_to_aware_datetime`` + datetime comparison. Forgejo emits ``2026-05-18T12:00:00Z``; Python's ``datetime.isoformat()`` emits ``2026-05-18T12:00:00+00:00`` — a string compare gives 'Z' (0x5A) vs '+' (0x2B) which silently misorders timestamps. - Now parses via ``datetime.fromisoformat`` (with Z → +00:00 rewrite), defaults naive timestamps to UTC, and compares as ``datetime``. - Test: test_comments_filter_handles_z_suffix_vs_offset_form pins the regression. ITEM 7 — false BEGIN IMMEDIATE claim in dequeue docstring: - The dequeue docstring claimed ``BEGIN IMMEDIATE`` was applied by session_scope; it wasn't. Attempted a global ``begin``-event listener that conflicted with StaticPool's shared-connection model (test_prefetch_callback_works_in_scheduler broke). - Reverted to a documentation fix: SQLite stays on default BEGIN DEFERRED (the SQLITE_BUSY retry via busy_timeout=5s is acceptable for single-host dev) + the docs make MULTI-MACHINE REQUIRES POSTGRES explicit at three call sites (db/session.py, db/dequeue.py, RUNBOOK.md was already updated in Phase 1l). Postgres has FOR UPDATE SKIP LOCKED which is what production actually uses. Tests: 586 controller tests pass (+10 new), 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
1e27039555 |
feat(controller): render CI raw_log_excerpt + target_url in worker prompts
Phase 1j shipped prefetch + parsers; Phase 1i shipped the prompt templates. But the prompt's CI-summary section only rendered each failing gate's name, status, and summary_line — the raw_log_excerpt (up to 16KB captured per gate) and target_url were sitting in the input_payload dict, never surfaced to the LLM. Impact: for stub-parser gates (7 of 10 tools — robot_framework, bandit, semgrep, vulture, radon, slipcover, build), the implementer saw "parser pending; see raw_log_excerpt" with no log in the prompt. It would have to shell out via OpenCode to re-fetch the log, which is wasteful and a regression vs the existing pipeline. This commit: - Renders ``target_url`` as ``log: <url>`` for failing gates so the LLM can curl it if needed. - Renders ``raw_log_excerpt`` inside a fenced code block, truncated to 3KB per gate (storage cap is 16KB; trim to prompt-friendly size with a marker preserving the original byte count). - Renders ``composite_findings`` as nested sub-findings (security_scan → bandit head + semgrep/vulture children, each with their own excerpt) so multi-tool gates are first-class in the prompt. - Skips the fence + URL for passing gates (already filtered) and for failures with no excerpt (defensive — keeps clean parsers from emitting empty log blocks). Conversion: the 20% of CI failures handled by stub parsers go from "broken — implementer doesn't see the log" to "less structured — implementer sees the raw log and can reason about it." Real parsers still ship in priority order as a follow-up. Tests (+7 in TestCIFailureRendering, 0 regressions): - target_url rendered for failing gate - target_url skipped for passing gate - raw_log_excerpt rendered in markdown code fence - Long log excerpt truncated with marker - Missing excerpt → no log block (no empty fence) - composite_findings render as sub-findings with their own excerpts - Stub-parser-pending failure now surfaces the actual log (the whole point of the commit) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
1b09a2054f |
feat(controller): Phase 1i — per-role prompt templates
Replace the JSON-dump prompt stub with structured per-role builders
that render the V1 input contract fields in human-friendly sections.
The agent's system prompt (.opencode/agents/{name}.md) provides the
HOW (tool calls, response shape); this module provides the WHAT (the
per-attempt context drawn from input_payload).
Per-role builders in tools/controller/worker/prompts.py:
- build_implementer_prompt — branch coords, CI summary (highlights
failing gates), active reviewer state, new comments, prior attempts,
optional allowed_files scope, output contract reminder
- build_reviewer_prompt — PR coords, CI summary, full diff (with
code fence), implementer's latest claim, prior reviewer outputs
- build_estimator_prompt — PR/issue context (title, body, diff
summary, head_sha), tier-pick guidance
- build_conflict_resolver_prompt — coords, conflicted files (or
porcelain fallback), prior implementer outputs
- build_summarizer_prompt — aged-out attempt + prior running
summary; instructs terse condensation
- build_prompt(role, tier, input_payload) — dispatcher
Truncation budgets (in chars):
- full_diff: 12,000
- pr_body / summaries: 4,000
- prior-attempt JSON: 2,000
- comment one-liner: 200
Wired as the default prompt builder in opencode_session via
``_default_prompt_for``. wire_opencode_session() callers that pass
prompt_builder=... override per-test.
Tests (+35 in test_worker_prompts.py, 1 updated adapter test, 0
regressions):
- Per-role: section headers present, required fields rendered,
missing fields fall back to (unavailable), oversize fields
truncated
- Dispatcher: routes by role; raises ValueError for missing tier on
tiered roles + unknown role
- Adapter integration: _default_prompt_for delegates to build_prompt
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
7b1b68b784 |
feat(controller): Phase 1e — master + worker __main__ entry points
Runnable as `python -m tools.controller.{master,worker}` for systemd
deployment. Wires the production callbacks (Forgejo HTTP +
OpenCode session adapter + agent_runner) into the previously-shipped
main loops.
tools/controller/master/__main__.py:
- Args: --owner, --repo (required); --opencode-url, --log-level,
--tick-interval, --discovery-only-once (smoke flag).
- Reads CLEVERAGENTS_DB_URL (returns 2 if unset — operator-actionable
error visible in systemd logs).
- build_cfg_stub reuses _mcp_common.ForgejoCfg → PAT rotation +
Forgejo URL env vars work without a new config module.
- SIGTERM/SIGINT → stop_event → master_main_loop drains + exits.
- --discovery-only-once: runs one discovery sweep + exits. Useful for
initial backfill or smoke testing the Forgejo callback wiring
without committing to the long-running loop.
tools/controller/worker/__main__.py:
- Args: --opencode-url, --roles (default all 5; comma-separated),
--max-concurrent, --poll-interval, --log-level, --no-startup-janitor.
- Empty --roles after parsing → exit 2.
- Startup sequence:
1. Run orphan-workspace sweep (unless --no-startup-janitor)
2. Build OpenCode session adapter via wire_opencode_session
3. Wrap production_agent_runner with the session injected
4. Wire SIGTERM/SIGINT → stop_event
5. Enter worker_main_loop
9 new tests in test_entry_points.py:
- Arg parsing: master --owner/--repo required; both DB-URL-missing
returns 2; worker empty-roles returns 2.
- --help works via subprocess for both (no SystemExit issue).
- Master --discovery-only-once smoke: stubs build_callbacks + cfg
builder; verifies exit 0 + the discovery call chain runs.
- Worker --no-startup-janitor: stubs sweep + loop; verifies sweep
is skipped when flag set, runs when not.
Total: 407 controller tests; full auto_agents suite 2769 pass.
Controller v1 is now end-to-end deployable as two systemd units
(one per machine for each instance). The remaining work for a real
production rollout is:
- Real prefetch callbacks for the scheduler (assembling V1 input
payloads from Forgejo data)
- Backfill at master startup (existing-PRs → DISCOVERED rows)
- Reconciliation tick (DB ↔ Forgejo sync)
- Per-role prompt templates (the V1 prompt stub works; specialised
per-role prompts will land as agents are migrated)
|
||
|
drew
|
bb8d872ba7 |
feat(controller): Phase 1d-4 — OpenCode session adapter
The production wrapper that wires the existing
_opencode_worker.run_session_blocking into the controller's
run_opencode_session protocol that production_agent_runner expects.
tools/controller/worker/opencode_session.py:
- agent_name_for(role, tier): role+tier → OpenCode agent name.
- implementer + tier {0,1,2} → task-implementor-tier-{0,1,2}
- reviewer → pr-review-worker
- estimator → estimator-implementation
- conflict_resolver → conflict-resolver-worker
- summarizer → controller-summarizer (new agent name)
- DEFAULT_TIER_TIMEOUT_S: per-tier wallclock budgets via env vars.
Defaults match plan v9 (tier 0: 600s, tier 1: 1200s, tier 2: 1800s).
Reviewer/estimator/conflict use the default-agent timeout (600s).
- wire_opencode_session(opencode_server_url, ...) → callable matching
the production_agent_runner's run_opencode_session contract.
- Resolves role+tier to agent name + per-tier timeout
- Builds the prompt (default stub or custom builder; per-role prompt
templates are a follow-up)
- Calls run_session_blocking with an on_poll callback that raises
WorkerLostLock if the controller's lost_lock_check returns True
mid-session (heartbeat thread reaper detected stolen lock)
- Routes SessionResult.status:
- 'completed' → return None (MCP's canonical output is in the
tempfile; production_agent_runner reads it after we return)
- 'timeout' → raise WorkerError(worker-internal-error)
- 'transport-error' → raise WorkerError(worker-internal-error,
including error_kind for forensics)
- unknown → raise WorkerError defensively
- Any unexpected exception from run_session_blocking wrapped as
WorkerError(worker-internal-error).
- Production wiring: dependency injection. wire_opencode_session()
lazy-imports the real run_session_blocking when not overridden.
Tests inject a stub.
22 new tests in test_worker_opencode_session.py:
- agent_name_for: parametrized per role (4 implementer-tier rows +
4 flat-role rows) + invalid tier + unknown role
- DEFAULT_TIER_TIMEOUT_S: each tier has a timeout + monotonic
- wire_opencode_session: completed/timeout/transport-error/unknown-
status routing; run_session_blocking raises wrapped as WorkerError;
on_poll propagates lost_lock; on_poll no-op when lock held;
custom prompt_builder used; default prompt includes role + tier
+ finalize() instruction + rendered input_payload; tag prefix
customizable.
Total: 398 controller tests; full auto_agents suite 2760 pass.
|
||
|
drew
|
23a014e402 |
feat(controller): Phase 1c-3 — production agent_runner (real MCP subprocess)
The production seam that connects the controller's runner to actual
LLM execution. Per-attempt MCP subprocess spawn + OpenCode session
drive + canonical JSON read + strict-parse → output dict.
tools/controller/worker/agent_runner.py:
- production_agent_runner(): the function tested production wires
into run_one_attempt's agent_runner param. Per-attempt lifecycle:
1. Map role → matching MCP module + V1 output contract.
2. mkstemp a canonical-output path; spawn `python -m {mcp_module}`
subprocess with CONTROLLER_CANONICAL_OUTPUT_PATH set in env.
3. Call injected run_opencode_session (tests use JSON-RPC stdin
driver; production wires _opencode_worker.run_session_blocking).
4. Wait for the MCP's finalize_and_emit to write the canonical
JSON to the tempfile.
5. Strict-parse against the role's V1 output model.
6. SIGTERM/grace/SIGKILL the MCP + unlink the tempfile in finally.
- ROLE_TO_MCP_MODULE / ROLE_TO_OUTPUT_MODEL: explicit per-role
dispatch tables. Covers all 5 worker roles.
- Error classification per the WorkerError outcome enum:
- unknown role → ValueError (master bug; not a worker outcome)
- session raises generic Exception → WorkerError(worker-internal-error)
- session raises WorkerLostLock → propagated unchanged
- lost_lock_check returns True after session → WorkerLostLock
- MCP didn't emit canonical output within timeout → WorkerError(
worker-internal-error)
- MCP emitted JSON that fails strict-parse → WorkerError(
contract-violation; per v9 status='failed' policy maps to
workflow STUCK)
tools/controller/mcp/_builder_base.py:
- finalize_and_emit() now writes to CONTROLLER_CANONICAL_OUTPUT_PATH
if set (production subprocess path) or stdout if not (direct-call
test path; capsys captures). Decouples canonical output from the
FastMCP JSON-RPC stdout transport — they would otherwise collide
in the subprocess (both writing to the same stream).
- Atomic file write: open + write + fsync + close.
- Existing test_mcp_builders.py (capsys-based) still passes with the
fallback path; new test_worker_agent_runner.py exercises the
file-based subprocess path with real MCPs.
10 new tests in test_worker_agent_runner.py:
- End-to-end with real MCP subprocesses (implementer-resolved,
reviewer-approve, estimator-tier-recommendation)
- Error paths: unknown role (ValueError), session raises (wrapped
as WorkerError), session raises WorkerLostLock (propagated),
lost_lock_check returns True post-session (raises WorkerLostLock),
no finalize → finalize_timeout → WorkerError
- Role-map sanity: every role has both an MCP module and output model
Total: 376 controller tests; full auto_agents suite 2738 pass.
This commit completes the v1 boundary — the controller now has the
complete stack from V1 contracts through MCP response builders, DB
schema, worker dequeue/heartbeat/runner/workspace, master state
machine + tick + reaper + pickup guard + scheduler + discovery +
Forgejo writes + MERGING + HTTP adapter, AND a production
agent_runner that wires it all to real OpenCode + MCP subprocess
spawning. Phase 1d-3+ enhancements (real OpenCode session wiring
in run_opencode_session) and Phase 2+ migration steps remain.
|
||
|
drew
|
30dfd92021 |
feat(controller): Phase 1c-2 — workspace + session sidecar + orphan janitor
Per-PR workspace umbrella manager (per plan v6) + the worker.session
sidecar that tracks live OpenCode session metadata for orphan cleanup
+ the startup janitor that sweeps orphaned workspaces from previous
worker crashes.
tools/controller/worker/:
- workspace.py: PerPRWorkspace + WorkspaceIdentity (frozen dataclass
yielding the canonical pr-{owner}-{repo}-{N} dir name). ensure_present
is idempotent. clone_if_absent runs `git clone --no-single-branch`
if .git/ missing; idempotent on re-call. fetch_and_validate runs
`git fetch origin --prune`, compares origin/<ref> to expected
head_sha, raises StaleInputError on mismatch (v6 stale-input fix),
then `git reset --hard <expected_sha>` + `git clean -fdx` to wipe
worktree residue from prior attempts. remove() rm -rf's the
workspace.
- session_sidecar.py: WorkerSession frozen dataclass +
write_sidecar (atomic via tmp+rename+fsync) + read_sidecar (tolerates
missing/empty/malformed/wrong-shape gracefully). Sidecar captures
opencode_server_url / session_id / subprocess_pid /
spawned_by_controller_pid / instance_id / spawned_at.
- janitor.py: sweep_orphans pre-queries the DB once for the set of
live instance_ids (workflow_attempts.status='in_progress'), then
scans workspace_root for pr-* dirs. For each:
- no sidecar → just delete (crashed pre-spawn)
- sidecar's instance_id is in live set → preserve (active worker)
- else → orphan. Try cancel_callback (production wires to OpenCode
cancel API); fall back to SIGTERM/grace/SIGKILL on the
subprocess_pid. Then delete sidecar + workspace. JanitorReport
summarises each sweep for structured logging.
Key v6 design points implemented:
- worker.session sidecar atomicity → orphan detection is robust
against partial writes (e.g., worker crashed mid-spawn).
- Cancel-callback-then-SIGKILL fallback → production prefers the
graceful OpenCode-side cancel; tests inject a fake.
- _kill_with_grace returns True after SIGKILL delivery; zombie
reaping is the parent's responsibility, not the janitor's.
27 new tests:
- WorkspaceIdentity (format + frozen)
- PerPRWorkspace paths + ensure_present idempotency
- clone_if_absent (creates worktree, idempotent, raises without URL)
- fetch_and_validate (matching sha resets clean; mismatched raises
StaleInputError; unknown ref raises)
- Sidecar I/O round-trip + atomicity + missing/empty/malformed handling
- Janitor: empty root / no-sidecar / active-lock / orphan-with-dead-pid
/ orphan-with-live-pid / cancel-callback (3 sub-paths: ok / fails /
raises) / mixed-workspaces / non-pr-skip
Total: 184 controller tests; full auto_agents suite 2546 pass.
|
||
|
drew
|
eab476e48e |
feat(controller): Phase 1c — worker controller skeleton
The dequeue+lock+heartbeat+runner+loop machinery. Production
OpenCode + MCP invocation slots in via the agent_runner callable
(Phase 1c-2). This commit is the structural foundation:
tools/controller/worker/:
- identity.py: build_instance_id() → "{hostname}/{pid}/{worker_uuid}"
per plan v9 (slash delimiter; IPv6-safe; uuid4 prefix for
per-instance uniqueness).
- heartbeat.py: Heartbeat thread that updates lock_heartbeat_at
every interval (default 30s). v9 simplified: TTL-only (no activity
tracking). UPDATE … WHERE locked_by_instance=us; rowcount=0 →
lost_lock_event.set() and thread exits, letting reaper handle it.
- runner.py: run_one_attempt() drives one attempt end-to-end.
Starts heartbeat → invokes agent_runner → on success writes
status='complete' + output_payload; on WorkerError writes
status='failed' with outcome label; on WorkerLostLock or detected
stolen-lock-at-write returns aborted (no DB write — reaper has
already re-pended). Defense-in-depth: even if agent returns
successfully, lost_lock_event.is_set() check skips the write.
- loop.py: worker_main_loop() polls the DB for pending attempts up
to MAX_CONCURRENT_WORKERS_PER_MACHINE, submits each to a
ThreadPoolExecutor. Honors stop_event for graceful shutdown
(drains in-flight before exit).
tools/controller/db/session.py: StaticPool for in-memory SQLite so
the heartbeat thread + runner write + dequeue all see the same DB
(without this, ":memory:" gives each connection an independent DB).
16 new tests in test_worker.py: instance ID format/uniqueness;
heartbeat tick (hold + steal); runner happy path; 5 error paths
(worker error / unexpected exception / WorkerLostLock raised /
stolen lock at write / lost_lock_event set defense-in-depth); 4
loop scenarios (single attempt, role filter skip, empty queue
exit-on-stop, explicit instance_id).
Total: 157 controller tests; full auto_agents suite 2519 pass.
|