751dbfed337bc18ec6dd96cc7b3f241bb3207acb
2482 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 drew
|
751dbfed33 |
fix(agents): batch Q — tier-variant model injection + Option B (drop dead .txt files)
Two related fixes from empirical testing of OpenCode's model
resolution against the controller's tier-escalation ladder.
ROOT-CAUSE FINDING (empirical, 2026-05-18)
==========================================
Spun up a probe agent (.opencode/agents/model-probe.md), asked the
model to self-identify, and tried three model-routing mechanisms:
1. **POST /session ``model`` in body**: OpenCode 0.x silently
IGNORES this — all probes returned ``openai/gpt-5.3-chat-latest``
(OpenCode's fallback default), not the requested haiku/sonnet/opus.
2. **.md frontmatter ``model:`` line** (after OpenCode restart):
HONORED — pinning to claude-haiku-4-5 yielded haiku, pinning to
sonnet yielded sonnet, etc.
3. **opencode.json ``agent.<name>.model``**: HONORED — same result
as .md frontmatter.
CONSEQUENCE: pre-batch-Q the tier variants had NO ``model:`` in
their .md frontmatter; the model lived only in
``.opencode/models/task-implementor-tier-<N>.txt`` files that the
dispatcher passed via POST /session body. Since OpenCode ignores
that pass-through, ALL FOUR tier variants ran on the SAME default
model (gpt-5.3 in this configuration) — the entire tier-escalation
ladder was cosmetic for model selection. The trial-2 sessions
logged ``model override -> claude-haiku-4-5`` but the actual
generation was on something else entirely.
WHAT THIS COMMIT DOES
=====================
1. **sync_tier_models.py rewrite** (already in batch P, refined here):
inject ``model: <providerID/modelID>`` line into each generated
tier variant's .md frontmatter. This is the mechanism OpenCode
actually reads at startup. The model values come from tiers.yaml
(source of truth).
2. **Drop the dead .opencode/models/task-implementor-tier-*.txt
files** (Option B): the dispatcher pass-through they fed was
empirically dead — OpenCode doesn't read the model from POST
/session. ``sync_tier_models.py`` now removes any stale .txt
files on each run (so a developer can't accidentally re-create
them).
3. **Test updates**:
- ``test_no_stale_variant_txt_files_remain``: pins that the .txt
files stay deleted (was ``test_every_tier_has_a_variant_txt``).
- ``test_each_variant_md_carries_correct_model_from_manifest``:
pins that the .md frontmatter model: matches tiers.yaml (was
``test_each_variant_txt_matches_manifest_model``).
- ``test_every_agent_file_reference_in_opencode_json_resolves``:
relaxed to skip when opencode.json has no agent block (which is
the Option B steady state). Still pins {file:...} resolution
for any future use.
- ``test_each_variant_md_matches_renderer_output_for_its_tier``:
updated to call the new renderer signature ``render_task_
implementer_variant(source, model)`` (was the byte-copy
identity test, retired because variants now differ by the
injected model: line).
- ``TestTaskImplementorVariantsAreByteIdentical`` → renamed
``test_variants_identical_except_for_model_line``: strips the
model: line via regex and asserts the rest is byte-identical.
OPERATOR WORKFLOW (unchanged surface)
=====================================
To swap a tier's model:
1. Edit ``.opencode/models/tiers.yaml`` (one line)
2. ``python3 tools/sync_tier_models.py`` (regenerates .md; removes
any stale .txt)
3. Commit both files
4. Restart OpenCode (it caches .md frontmatter at startup)
What's still ahead (deferred):
- ``_resolve_role_model()`` in tools/_opencode_worker.py is now
proven dead code (reads .txt files that don't exist; injects
model into POST /session body that OpenCode ignores). Should be
deleted in a follow-up — kept now to minimize blast radius.
- The misleading "if generation uses a different model, restart
OpenCode so opencode.json's {file:...} re-resolves" log line in
``_opencode_worker.py:1569`` is wrong post-Option-B; should be
retired with the dead code above.
- Whether opencode.json's agent block supports a ``permission`` field
is the gate for an even deeper simplification (Option D). Skipped
for now per operator direction; the controller path uses .md
frontmatter for permissions.
3148 tests pass, 4 skipped, 0 failures.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
c4c8f0af78 |
fix(agents): batch P hardening — 4 implementer permission tightenings
Follow-on review of the batch-P task-implementor.md surfaced 4 issues:
1. **Duplicate `nox *` allow** — removed (was listed once at the
bash block top and again with the uvx nox rules).
2. **Dropped `curl *` / `wget *`** — all HTTP access now goes
through typed MCPs (forgejo_* for Forgejo, ci_* for CI ops,
git_* for git with credential helper). The legacy curl allow
was there as a catchall for cases the MCPs didn't cover; with
the controller pipeline the MCPs cover everything the agent
should be doing. A debug allowlist can be added back per-op
when a session legitimately needs it.
3. **Tightened forgejo permissions** — controller-side master owns
ALL Forgejo writes for a controlled workflow (label transitions,
PR-state comments, claim/release, status comments) via
tools/controller/master/forgejo_writes.py. Previously the
implementer's `"forgejo*": allow` permitted writes that would
conflict with the controller's reconciliation. Now:
- `forgejo_fetch_*: allow` — agent can read PR/issue/comments/
reviews for verification
- `forgejo_post_comment/update_pr_body/add_label/remove_label/
claim_pr/release_pr/submit_review: deny` — controller's
territory
- `"forgejo*": deny` — default-deny for any new tool added later
4. **Denied public web by default** — webfetch / websearch /
codesearch now deny. The agent has graphify for cross-module
reasoning, the worktree for code context, and the prompt for
task definition; external lookups are unnecessary for a
controlled workflow. Defense-in-depth against data exfiltration
and prompt-injection via fetched content. Matches the reviewer's
read-only-by-role posture. Operators can re-enable per-task.
Tier variants regenerated via sync_tier_models.py.
3148 tests still pass, 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
8eb44990d7 |
refactor(agents): batch P — MCP-first rewrites for implementer, reviewer, conflict-resolver
Extends the estimator-implementation rewrite ( |
||
|
drew
|
c7e8c6b167 |
refactor(agents): rewrite estimator-implementation for MCP-first controller
Drops the legacy chat-JSON output contract and the references to ``tools/dispatch_implementer.py`` + ``_extract_last_json_object`` that no longer apply under the controller pipeline. The agent now has a single output channel — the ``estimator-response-builder`` MCP — and the system prompt explicitly tells it that chat-JSON is "silently discarded by the controller". Trial-2 surfaced the dual-contract trap: the system prompt's chat-JSON contract competed with the per-attempt prompt's MCP-call instructions, and when the MCP failed (the prompt-placeholder bug fixed in batch O) the agent had no fallback that the controller could read. Other changes: - Tightens the mission statement and consolidates the "don't implement" rules into one block (was scattered across 3 sections). - Removes the cross-cycle memory rule (§2a in the old prompt). The controller's pickup_guard + estimator cache invalidation handle this without per-agent label-reading. - Removes the empty-prompt fallback (controller always provides context; the rollback knob ``IMPLEMENTER_DISPATCHER_PREFETCH=0`` is a legacy-pipeline concept). - Drops the JSON-shape examples (the MCP enforces the shape now). - Adds the explicit ``estimator_set_is_metadata_only`` optional call so the agent knows the field exists. - Keeps the security lockdown verbatim — read-only permissions, no bash, no mutation, no network. Same denylist as before. - Preserves the trust boundary on embedded work-item content. Net: ~250 → ~200 lines, single output channel, no legacy noise. Model pinning to ``local-claude/claude-sonnet-4-6`` from the prior turn is preserved (sonnet's grading is the sweet spot for tier selection per the inline justification). 800 controller tests still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
2ad0958cb7 |
fix(controller): batch O — trial-2 fix: interpolate real attempt_id into prompts
Trial run-2 surfaced the actual blocker: the response-builder MCPs
shared their module-global ``_STATE`` across OpenCode sessions (they're
registered as ``type: local`` in opencode.json, so OpenCode spawns one
subprocess per OpenCode-server lifetime, not per session). The CA2/PD5
cross-session reset I added in the M-batch keyed on
``identity.attempt_id`` — but the prompt template literally read:
1. ``estimator_start(attempt_id=..., workflow_id=..., ...)``
The ``...`` were placeholder ellipses, not interpolated. So agents
guessed ``attempt_id=1`` every single time. The cross-session reset
compared ``1 == 1``, decided "same attempt — no reset", and the prior
attempt's ``finalized=True`` persisted forever. Every estimator
session after the first failed with ``"response already finalized;
no further mutations allowed"`` on every ``_set_*`` call → no
finalize → 30s worker timeout → workflow STUCK.
Observed in trial run-2: 2 successes (attempts 1, 2), then 15
consecutive failures (attempts 3–17 across all 6 workflows) before
the pickup_guard would have STUCK every workflow.
Two fixes (defense in depth):
1. **Unconditional reset on _start** in all 5 builders. We can't
distinguish "agent retry in same session" from "new session reusing
this MCP" reliably — the observable signature is identical. Just
reset whenever ``_STATE.started`` is True; ``reset_for_new_attempt``
logs a WARN if the prior state was in-flight so abandoned attempts
are still visible to ops. The agent's last ``_start`` always wins.
2. **Interpolate real attempt_id + workflow_id + pr_number + head_sha
into all 5 prompts' Output contract sections**. The agent_runner
now injects ``attempt_id`` into ``input_payload`` (matching the
existing ``workspace_dir`` injection pattern). Each ``build_*_prompt``
reads ``input_payload.get("attempt_id")`` and bakes the concrete
value into the MCP-call signature shown to the agent, plus a
``PASS THESE EXACT VALUES`` directive.
Also strengthened each prompt's ``_finalize`` line with
``**You MUST call this tool — without it the controller times out.**``
so the agent understands the contract is hard, not optional.
Updated tests:
- ``test_double_start_refused`` → ``test_double_start_resets_silently``:
pins the new permissive-reset behavior + asserts the WARN log fires.
- ``test_implementer_rejects_intra_session_double_start`` updated for
same reason; now asserts the second _start succeeds + last-wins
semantics (used_tier == new tier).
- NEW ``test_prompt_interpolates_real_attempt_id`` parametrized over
all 5 roles: asserts ``attempt_id=42`` + ``workflow_id=7`` appear
literally in the rendered prompt AND that ``attempt_id=...`` /
``workflow_id=...`` placeholders DO NOT.
Total: 795 → 800 tests, 0 regressions.
The model-override files (.md + .txt) are still in place from the
prior turn — that change is orthogonal to this bug fix; OpenCode
ignores the dispatcher's pass-through per the README, and the actual
generation has been on haiku the whole time. The .md frontmatter
change to sonnet will only take effect on the next OpenCode restart.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
a4892f8db9 |
fix(controller): batch N — second-round adversarial review fixes
Round-2 adversarial review on commit
|
||
|
drew
|
84da774212 |
fix(controller): batch M — post-Phase-1m adversarial review fixes
Three rounds of adversarial review (Chief Architect / Principal Dev /
Senior Test Engineer) on commits 3ca794be7..db12f45ac surfaced ~35
issues. This commit addresses 25+ across criticals, highs, and
mediums, and adds 40 new tests covering the changes plus key gaps the
review identified.
CRITICALS (M1):
- CA1: stale {role}_output.json from a prior attempt on the same
per-PR workspace was readable as "fresh" output of the new attempt.
agent_runner now unlinks the MCP-canonical path AND every fallback
path BEFORE the session runs.
- CA2/PD5: opencode.json-registered MCP subprocesses persist across
OpenCode sessions, but BuilderState was module-singleton. Added
reset_for_new_attempt() + cross-session detection (compare
identity.attempt_id) to every *_start; force-resets with WARN if
prior attempt was interrupted (timeout / lost lock).
- PD3: inline-JSON callback could overwrite an MCP-written canonical
V1 file with adapted-from-prose garbage. Callback now inspects
existing files and skips when V1 is already present.
- PD4: FORGEJO_URL = .rstrip("/api/v1") is a character-set strip —
catastrophic for hosts whose path contains /v1 in the middle.
Replaced with explicit endswith()-based suffix strip.
- CA10: clone URL embedded $FORGEJO_TOKEN, persisted into
.git/config where any agent could cat it. Token now sourced via
local credential.helper at clone-time, URL kept clean.
- CA12: state.finalized was set BEFORE the file write, so disk-full
/ OSError left the agent unable to retry finalize. Reordered.
HIGHS (M2):
- CA3/PD12: output_path validation (NUL-byte rejection, must be
absolute, parent-not-file check) in finalize_and_emit.
- CA6: ci_status_poll SELECT only considered implementer attempts;
conflict_resolver also pushes commits. SQL now unions both roles.
- PD9: ci_status_poll could advance on a stale "resolved" SHA from a
blocked attempt (whose head_sha_after == head_sha_before). Added
outcome='resolved' filter.
- CA8: cancelled/stale CI states mapped to ci_red_retry_same_tier,
burning pickup_count on healthy PRs. Both now wait (treated as
operator/system action, not failure). timed_out stays red.
- TE9: unknown Forgejo CI states now WARN-log instead of silently
being treated as pending — operators see new state strings.
- PD8: ci_status_poll event_type strings standardized to match the
state-machine event names (ci_green / ci_red_retry_same_tier)
instead of legacy ci-green / ci-red.
- CA7: inline-JSON callback now checks lost_lock_check BEFORE write
so a file isn't staged after lock loss.
- PD10: atomic .tmp + os.replace writes in both MCP finalize and
inline callback so the poller never sees a half-written file.
- PD16: inline_output_callback exceptions now re-raise as WorkerError
instead of being silently logged (root cause was buried 30s later
in a canonical-output timeout).
- CA9: WorkerConfig manual rebuild on --max-concurrent/--poll-interval
silently dropped new fields. Use dataclasses.replace, matching
round-4 P5 fix in master/__main__.py.
MEDIUMS (M3) — legacy_adapter quality upgrades:
- PD1: unrecognized confidence values now WARN instead of silently
defaulting to "medium" — surfaces agent prompt drift.
- PD2: estimator recommended_tier clamped to {0,1,2} so an out-of-
range int doesn't bypass the adapter's whole purpose.
- PD7: reviewer blocking_issues list-of-strings coerced into the
list-of-BlockingIssue-dict shape strict_parse requires.
- PD13: conflict_resolver prompt defaults tier=1 + warns instead of
raising; the scheduler always sets it but defends against drift.
- PD14: summarizer summary < 50 chars padded with a clear marker so
strict_parse accepts it (and the truncation is visible).
- PD15: implementer blockers capped at 4096 chars each so a buggy
agent can't blow up audit log / DB column.
- PD17: launch script accepts either FORGEJO_TOKEN or GITEA_TOKEN
with a clear error if both are unset.
- PD22: conflict_resolver adapter accepts singular commit_sha
fallback, matching implementer.
- CA4: every adapter invocation logs role + payload key fingerprint
so operators can measure agent-migration progress.
- estimator + summarizer now have explicit _start tools (the prompts
already referenced them; previously absent → first call would fail).
TESTS (M4) — added 40 tests in test_post_review_fixes.py:
- Cross-session MCP state reset (implementer + reviewer + estimator
+ summarizer; intra-session double-start still rejected).
- finalize_and_emit output_path precedence (arg > env > stdout),
parent-dir creation, rejection of relative/NUL paths, failed-write
leaves state retryable.
- legacy_adapter quality: tier clamping, blocker cap, non-string
commit warning, blocking_issues string coercion, conflict_resolver
full roundtrip + non-resolved head clearing, summarizer padding,
confidence warning, V1-passthrough no-log.
- opencode.json registration parity: every MCP the prompts name is
registered with the correct module path.
- Per-role prompts mention {role}_output.json (canonical poller path)
+ the "DO NOT emit chat-JSON" directive.
- FORGEJO_URL suffix-strip parametrized table.
- agent_runner stale-file cleanup: prior-attempt file is unlinked
before a new session can read it as phantom output.
Also updated 2 pre-existing tests for the CA8 / PD8 / PD13 behavior
changes (cancelled→wait, event_type renaming, conflict_resolver
default-tier warning).
Total: 741 → 781 tests, 0 regressions.
DEFERRED (M5 follow-up — non-trial-blocking):
- CA5: head_sha verification via git cat-file (requires subprocess).
- CA11: discovery_interval_s wall-time cadence (vs iteration count).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
db12f45acb |
feat(controller): Phase 1m — wire response-builder MCPs into OpenCode (Option A)
The trial-path expedient (legacy_adapter harvesting chat-JSON) is now
the SAFETY NET; the architecture-intended path (agents call the
controller's response-builder MCPs, which validate against V1 +
write canonical JSON) is now wired end-to-end.
Four pieces:
1. MCP `finalize` accepts `output_path` argument
``mcp/_builder_base.finalize_and_emit`` adds ``output_path: str | None``
kwarg. Precedence: explicit arg > ``CONTROLLER_CANONICAL_OUTPUT_PATH``
env > stdout. This solves the per-attempt path problem that
blocked opencode.json registration (the env var is static; the
per-attempt path comes from the prompt, the agent passes it as a
tool arg). Each role's finalize updated:
- estimator_finalize(output_path)
- implementer_finalize(output_path)
- reviewer_finalize(output_path)
- conflict_finalize(output_path)
- summarizer_finalize(output_path)
Creates parent directory if missing (so the controller doesn't
need to pre-create). Returns ``wrote_to`` in the ok dict so tests
can pin the path.
2. opencode.json registers the 5 controller MCPs
``.opencode/opencode.json`` adds:
- implementer-response-builder
- reviewer-response-builder
- estimator-response-builder
- conflict-resolver-response-builder
- summarizer-response-builder
Each spawned via ``python -m tools.controller.mcp.{role}_builder``
with PYTHONPATH=/repo-root so the controller imports resolve.
3. Controller prompt builder injects the EXACT tool sequence
``worker/prompts.py`` rewrites each role's "Output contract"
section. The old "PREFERRED MCP / FALLBACK file-write" instruction
becomes a single REQUIRED contract: numbered tool calls (``X_start``,
``X_set_*``, ``X_finalize(output_path=...)``) with the explicit
per-attempt path baked in. Explicit "DO NOT emit a JSON object in
your final chat message" instruction to override the legacy
contract baked into the agent system prompts.
4. Agent permission whitelists include the new MCPs
- ``.opencode/agents/task-implementor.md`` (source for tier-{0,1,2,min}
variants — regenerated via tools/sync_tier_models.py)
- ``.opencode/agents/pr-review-worker.md``
- ``.opencode/agents/estimator-implementation.md``
- ``.opencode/agents/conflict-resolver-worker.md``
Each adds the matching ``"{role}*": allow`` pattern.
Safety net preserved:
The legacy_adapter (commit
|
||
|
drew
|
bcc59d38af |
fix(controller): trial-path output channel — harvest inline JSON + legacy-shape adapter
Trial run-1 surfaced the real failure mode: the existing agent
prompts emit their result as a single JSON object in the FINAL
assistant message (the legacy dispatch_implementer/dispatch_review
contract). They DON'T call the controller's response-builder MCPs
(not registered in opencode.json) and they DON'T write a file. The
worker's canonical-output poller timed out every attempt.
Fix (two pieces):
1. INLINE JSON HARVEST
``_opencode_worker.run_session_blocking`` already extracts the
last JSON object from the agent's final response into
``SessionResult.parsed_json``. The controller's opencode_session
adapter was ignoring it.
- ``opencode_session.py:run_opencode_session`` now accepts an
``inline_output_callback`` kwarg. After the session completes,
if parsed_json is set, the callback fires.
- ``agent_runner.py`` provides the callback: writes the parsed
JSON to the canonical-output path so the existing poller picks
it up uniformly with the MCP + file-write channels.
2. LEGACY → V1 SHAPE ADAPTER (``worker/legacy_adapter.py``)
The legacy JSON shape doesn't match V1 contracts:
- estimator: legacy {recommended_tier, is_confident, reasoning}
vs V1 {output_version, recommended_tier, is_metadata_only,
confidence, reasoning, wallclock_seconds}
- implementer: legacy {outcome: resolved|unresolved, ...} vs V1
{outcome: resolved|rebase-failed|blocked|noop|competence-failure,
commit_shas, blockers, used_tier, ...}
- reviewer: legacy {verdict, ...} vs V1 {output_version, verdict,
blocking_issues, suggested_next_action, ...}
``adapt_to_v1(role, payload, tier, wallclock_seconds)`` normalizes
each role's legacy shape into the corresponding V1 dict:
- adds ``output_version="V1"``
- maps outcome enums (``unresolved`` → ``blocked``)
- synthesizes missing required fields with sensible defaults
- coerces singular commit_sha → list[commit_shas]
- converts legacy is_confident bool → confidence string
Already-V1 payloads pass through unchanged. The agent_runner
wraps the inline_output_callback to run the adapter before
writing to the canonical path.
Tests (+12 in test_legacy_adapter.py):
- Each role's adapt: legacy in, V1-validating-via-strict_parse out
- Already-V1 passthrough
- Outcome enum mappings + invalid-outcome fallback
- Singular commit_sha normalization
- Non-dict / unknown-role passthrough (lets strict_parse raise)
Total: 738 controller tests pass (+12 net), 0 regressions.
Long-term: update agent prompts to emit V1 directly, register the
MCPs in opencode.json. For the trial, this adapter lets the existing
agents flow through the new controller unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
3ca794be75 |
feat(controller): autonomous CI status polling — closes the last trial gap
The Phase 2 trial previously required operator-intervention SQL to
advance workflows from AWAITING_CI → REVIEWING (no automated CI
status polling). This commit wires the missing tick so the trial
runs end-to-end without manual help.
Components:
- ``master/forgejo_http.py``: new ``get_ci_status`` callback wraps
Forgejo's ``/commits/{sha}/status`` combined-status endpoint;
added to ``ForgejoCallbacks``.
- ``master/ci_status_poll.py`` (NEW): ``run_ci_status_poll_tick``
scans AWAITING_CI workflows, fetches CI status keyed on the
latest implementer attempt's ``head_sha_after``, and applies
state transitions via ``apply_event``. TOCTOU-defended UPDATE
(``WHERE current_state='AWAITING_CI'``) + per-row exception
isolation.
- ``master/loop.py``: new ``ci_status_poll_args=(owner, repo,
get_ci_status)`` kwarg + ``ci_status_poll_interval_s`` config
(default 60s) + ``MasterTickReport.ci_status_poll`` field.
- ``master/__main__.py``: threads ``callbacks.get_ci_status`` into
the loop.
State mapping (Forgejo combined-status state → event):
- success / neutral / skipped / warning → ci_green → REVIEWING
- failure / error / cancelled / timed_out / stale →
ci_red_retry_same_tier → IMPLEMENTING
- pending / queued / in_progress / action_required → no-op (wait)
- None / unknown / fetch failure → no-op (transient)
The ``ci_polling_exhausted`` timeout (default 2h) remains as the
safety net for CI that genuinely never reports.
Tests (+14 in test_master_ci_status_poll.py):
- Happy paths (success→green, failure→red, pending→wait)
- Error paths (callback raises; workflow without head_sha)
- Event row shape (event_type='ci-green'/'ci-red', reason payload)
- Extended state mapping (cancelled, neutral, in_progress)
- Other-repo isolation
- LoopIntegration end-to-end via master_main_loop with safety timer
RUNBOOK updated: removed the manual SQL workaround; added the
autonomous CI poll's tunables.
Total: 726 controller tests pass (+14 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
f93f0d1c53 |
fix(controller): round-4 P2 — defensive input_payload normalization
If the DB returned NULL input_payload (legacy/seeded data, schema bug, hand-edited rows), agent_runner crashed on ``dict(None)`` → WorkerError → master re-pickups → STUCK after MAX_PICKUPS reaps. Silent infinite-loop until exhaustion. Fix: ``worker/runner.py:run_one_attempt`` checks isinstance(dict) on entry; substitutes empty dict + logs WARNING. Agent still runs normally with an empty payload. Test: ``test_none_input_payload_doesnt_crash`` — passes input_payload=None + verifies the attempt completes normally. Total: 712 controller tests pass (+1 net). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
f57d9f9478 |
fix(controller): batch L — round-4 trial-blockers (A1, P1, P3, P4, P5, T5)
Round-4 adversarial review found 5 trial-blockers + 1 silent-debt
item the post-round-3 deep pass missed. All fixed.
A1 — pre-clone the workspace so the agent has a worktree to operate on
``worker/__main__.py``: the agent_runner closure now constructs a
``PerPRWorkspace`` from input_payload.owner/repo/pr_number + the
FORGEJO_URL+FORGEJO_TOKEN env vars. Pre-flight:
- ``workspace.ensure_present()`` creates the dir skeleton.
- ``workspace.clone_if_absent()`` clones the repo into
``{workspace_dir}/worktree/`` if not already present (idempotent).
- ``workspace.fetch_and_validate(head_sha, head_ref)`` refreshes +
verifies the workspace is at the expected head. ``StaleInputError``
→ ``WorkerError(outcome='stale-input')`` so the master re-prefetches
without burning a pickup. ``RuntimeError`` → ``worker-internal-error``.
Previously the agent saw an empty workspace_dir + had no repo.
P1 — partial-write defense in the canonical-output poller
``worker/agent_runner.py:_wait_for_canonical_output`` now polls each
path with a two-pass quiescence check (size stable + content parses
as JSON) before returning. Partial writes (agent crashed mid-flush)
are skipped + the polling loop continues. The previous
``f.read().strip()`` returned partial JSON which then tripped
``ContractValidationError`` → ``worker-internal-error`` with no
record of WHICH path; now logs source path on every read.
P3 — TOCTOU defense in promote_discovered
``master/promote.py``: the UPDATE now filters
``current_state='DISCOVERED'``. If a concurrent reconciliation
moved the row off DISCOVERED between SELECT and UPDATE, rowcount=0
+ we skip the event-row write. No duplicate audit entry; no
overwriting a pause-by-label-removal.
P4 — explicit tuple-length validation in reconciliation_args + discovery_args
``master/loop.py``: previously a 6-tuple silently fell into the
``else`` 4-tuple unpack, raised ValueError("too many values"), got
swallowed by the per-iter ``except Exception``, and reconciliation
silently died forever. Now: ``elif n == 4`` + ``else: raise TypeError``.
The TypeError still hits the per-iter except (so the loop doesn't
crash) but ``logger.exception`` surfaces the actionable message in
journald. Operator sees "reconciliation_args must be a 4- or 5-tuple;
got length 6" instead of zero indication.
P5 — --tick-interval CLI flag preserves other config fields
``master/__main__.py``: replaced the manual ``MasterConfig(...)``
rebuild (which dropped reconciliation/ci_poll/discovery intervals)
with ``dataclasses.replace(cfg_loop, tick_interval_s=args.tick_interval)``.
Operators who pass --tick-interval no longer silently revert the
other intervals to defaults.
T5 — scheduler._commit_escalation uses safe_json_dumps
``master/scheduler.py``: the escalation event row's payload was the
only call site that bypassed safe_json_dumps. Now consistent — a
future contributor adding a datetime/Decimal field won't trip raw
json.dumps at runtime.
Tests (+4 net):
- ``test_worker_agent_runner.py::test_partial_write_not_read``: pins
P1 (truncated fallback file + valid MCP output → MCP wins).
- ``test_master_promote.py::test_toctou_state_change_between_select_and_update``:
pins P3 (steal state via monkey-patch → no double-promotion, no
extra event row).
- ``test_master_loop.py::test_reconciliation_args_wrong_length_logs_not_silent``:
pins P4 (6-tuple → logged error, not silent forever).
- ``test_entry_points.py::test_tick_interval_flag_preserves_other_cfg_fields``:
pins P5 (env-set non-default intervals survive --tick-interval).
Total: 711 controller tests pass (+4 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
b4ebf6a2bd |
docs(controller): RUNBOOK — Phase 2 trial gaps + manual workarounds
Document the two known gaps that need operator intervention for an
end-to-end trial:
1. MCP-to-OpenCode transport: response-builder MCPs aren't yet
registered in opencode.json. Agents use the FALLBACK file-write
path (``{workspace_dir}/{role}_output.json``) per the prompt
instructions. The agent_runner polls both channels.
2. CI status polling not wired: AWAITING_CI exits only via
ci_polling_exhausted timeout (default 2h → STUCK). To advance
during the trial, the operator runs a SQL UPDATE to manually
transition AWAITING_CI → REVIEWING (snippet in the RUNBOOK).
Added a step-by-step Trial checklist covering: env vars, label
the PR, expected log timeline, the manual SQL to advance past
AWAITING_CI, and the verification query.
Both gaps have follow-up phases queued (1m for MCP wiring, 1n for
CI poller). The trial as described validates the full controller
state machine + most of the worker substrate; only the CI poll +
the MCP transport are operator-intervention paths.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
7bf39a0b51 |
fix(controller): RB8 — head_sha bookkeeping in _write_outcome
The worker's _write_outcome never wrote head_sha_before / head_sha_after
on the workflow_attempts row. But tick.py reads both to compute
head_sha_advanced, which the outcome mapper REQUIRES to distinguish
``implementer_pushed`` (true push) from ``implementer_blocked``
(worker said resolved but git didn't move).
Without these columns set, every implementer attempt's "resolved"
outcome mapped to a no-op event, and workflows would stall
permanently after the agent ran successfully.
Fix:
- ``worker/runner.py:_write_outcome`` accepts ``head_sha_before`` +
``head_sha_after`` kwargs and writes both columns in the UPDATE.
- ``run_one_attempt`` computes:
- head_sha_before = input_payload["head_sha"] (what the agent
was told to start from)
- head_sha_after = output_payload["commit_shas"][-1] if any
commits were produced; falls back to head_sha_before otherwise
so tick.py correctly sees "no advance" when the agent didn't
commit.
Tests (+2):
- test_head_sha_before_after_recorded — pins the happy path where
the agent committed and head_sha advanced.
- test_no_commits_means_head_sha_unchanged — pins the no-advance
case where head_sha_after equals head_sha_before.
Total: 707 controller tests pass (+2 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
0c9e8fdd69 |
fix(controller): trial-path output channel — fallback file-write for unwired MCP layer
RB7 (newly discovered) — response-builder MCPs are NEVER REACHABLE
from the OpenCode agent:
The controller spawns the per-attempt response-builder MCP via
``python -m tools.controller.mcp.{role}_builder`` with stdin=PIPE
+ CONTROLLER_CANONICAL_OUTPUT_PATH set. But:
- ``.opencode/opencode.json`` does NOT register the controller MCPs.
- OpenCode only knows about MCPs it sees in opencode.json (graphify,
ci, forgejo, git, handoff).
- The OpenCode agent has NO transport to call
``implementer_finalize`` because OpenCode doesn't see the MCP.
- The controller's spawned MCP subprocess sits waiting for JSON-RPC
on stdin that no one sends.
- The agent does work, the session ends, the canonical-output-file
is empty, agent_runner times out with WorkerError.
This would have been the single biggest blocker to running an actual
Phase 2 trial. The MCP layer is structurally complete but not
plumbed into OpenCode — pivoting requires either registering them
in opencode.json (which doesn't support per-attempt env injection)
OR adding a fallback channel.
Fix (trial-path):
- ``worker/prompts.py``: each role's output-contract section now
documents BOTH the PREFERRED MCP call AND a FALLBACK direct
file-write to ``{workspace_dir}/{role}_output.json``. The agent
can pick either; once the MCP layer is wired (Phase 1m) it'll
naturally prefer the MCP.
- ``worker/agent_runner.py:_wait_for_canonical_output``: now polls
the MCP-canonical path AND a list of ``fallback_paths``. Call
site adds ``{workspace_dir}/{role}_output.json`` to fallback
paths. First non-empty wins.
Tests (+2):
- ``test_fallback_file_picked_up_when_mcp_silent``: agent writes
only the fallback file (no MCP drive); runner reads it.
- ``test_first_nonempty_output_wins``: documents the
poll-both-channels semantic.
Documented Phase-1m work:
- Register controller MCPs in ``.opencode/opencode.json``
- Solve per-attempt env injection (the canonical_output_path is
per-attempt; opencode.json env is static — needs a request_id
arg on finalize() OR a workspace-relative output convention)
Until that's done, the trial uses the fallback file-write path.
Total: 705 controller tests pass (+2 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
251eeb21ff |
fix(controller): more pipeline run-blockers — merging tick, periodic discovery, worker create_all, systemd ordering
Continuing the round-3 deep-pass cleanup. Three more run-blockers + one robustness fix. RB5 — MERGING handler never invoked from master loop: ``run_merging_tick`` was exported by the master package but no caller fired it. Workflows that transition to MERGING (via reviewer approval) would sit there indefinitely with no Forgejo merge call. Fix: - ``master/loop.py`` accepts a ``merging_args=(owner, repo, merge_callback)`` kwarg. When set, the tick fires every iteration (cheap if no workflows in MERGING). - ``MasterTickReport`` gains ``merging: MergingHandlerReport | None``. - ``master/__main__.py`` wires it from the Forgejo callback bundle. RB6 — periodic discovery never fires: ``run_discovery`` was only called at startup via ``run_startup_backfill`` + the ``--discovery-only-once`` smoke flag. PRs created after master startup would not be discovered until the master restarted. Fix: - ``master/loop.py`` accepts ``discovery_args=(owner, repo, list_prs, list_issues)`` or the 5-tuple with kwargs. Periodic tick on its own cadence (``CONTROLLER_DISCOVERY_INTERVAL_S``, default 30s). - ``MasterTickReport`` gains ``discovery: DiscoveryReport | None``. - ``master/__main__.py`` wires it + threads ``require_opt_in_label`` through. RB-robust — worker calls create_all defensively: Master is normally responsible for schema creation (workers run After= it via systemd ordering). But if the worker is started in isolation (test / local dev / unit ordering broken), it'd crash on the first query against missing tables. Fix: - ``worker/__main__.py`` calls ``create_all(engine)`` after ``build_engine``. ``create_all`` is idempotent (CREATE TABLE IF NOT EXISTS); safe to call from both master + worker. - ``cleveragents-controller-worker@.service`` adds ``After=cleveragents-controller-master.service`` + ``Wants=cleveragents-controller-master.service`` so systemd enforces the start ordering in production. Total: 703 controller tests pass (no test changes; all new wiring is exercised by master_main_loop tests via the new kwargs). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
febb352618 |
fix(controller): pipeline run-blockers — promoter, scheduler, owner/repo, workspace_dir patch
Round-3 deep pass identified four issues that would have prevented an
actual end-to-end pipeline run:
RB1 — DISCOVERED → ANALYZING never fired in production:
The state machine defines (DISCOVERED, discovery_picked_up) →
ANALYZING but NO production code fires the event. Workflows
created by discovery would sit in DISCOVERED forever.
Fix:
- New ``master/promote.py``: ``run_promote_discovered_tick`` scans
for DISCOVERED workflows + fires ``discovery_picked_up`` via
apply_event (state-machine invariants stay enforced) + emits a
``discovery-promoted`` controller_events row per transition.
- Composes with the master loop's other ticks; runs every iteration
(cheap — typically 0-1 row).
RB2 — scheduler.schedule_next_attempts never called from master loop:
The scheduler was exported by the master package but never invoked.
It creates the ``workflow_attempts`` rows that workers dequeue —
without it, workers would have nothing to pick up.
Fix:
- ``master/loop.py`` now accepts a ``prefetch: PrefetchCallback``
kwarg. When provided, the loop runs promote_discovered + scheduler
every iteration after tick/reaper/reconciliation.
- ``MasterTickReport`` gains ``promote_discovered`` and ``scheduler``
optional fields so on_iteration callbacks see both.
- ``master/__main__.py`` builds a ``PrefetchDataCallbacks`` from the
Forgejo callback bundle and constructs the production
``make_prefetch_callback(engine, callbacks)`` — wires through to
the loop's new prefetch kwarg.
RB3 — owner / repo missing from V1 input contracts:
The implementer / reviewer / estimator / conflict-resolver V1 inputs
had pr_number but not owner/repo. The OpenCode agent would have
had no way to know which Forgejo repo to clone — it would have had
to derive owner/repo from process env, coupling the worker to a
single repo.
Fix:
- ``contracts/v1.py``: added ``owner: str`` and ``repo: str``
(min_length=1) to ImplementerInputV1, ReviewerInputV1,
EstimatorInputV1, ConflictResolverInputV1.
- ``master/prefetch.py``: builders populate owner/repo from the
Workflow row (already known at prefetch time).
- Existing test fixtures in ``test_contracts_v1.py`` updated.
RB4 — input_payload.workspace_dir placeholder reached the agent:
Prefetch wrote ``workspace_dir = "<worker-injected>"`` as a
placeholder; the worker never patched it before invoking the
OpenCode session. The prompt builder rendered the literal
placeholder string into the agent's prompt — the agent had no idea
where to clone.
Fix:
- ``worker/agent_runner.py``: patches input_payload.workspace_dir
with the real path immediately before calling run_opencode_session.
Uses a shallow copy so the caller's dict isn't side-effected.
- ``worker/__main__.py``: workspace_dir naming convention is now
``pr-{owner}-{repo}-{pr_number}`` (matches workspace.py's
PerPRWorkspace convention) so the janitor's pr-* glob + the
agent's expected workspace location agree. Falls back to
``pr-attempt-{N}`` for legacy input_payloads missing owner/repo.
Tests:
- ``test_master_promote.py`` (NEW, +7 tests):
- empty DB no-op
- single workflow promoted
- multiple promoted in one tick
- only DISCOVERED targeted (non-DISCOVERED untouched)
- controller_events row emitted with correct shape
- idempotent after first promotion
- LoopIntegration end-to-end: DISCOVERED → ANALYZING → pending
estimator attempt visible in workflow_attempts (pins the entire
previously-broken pipeline from discovery to enqueue)
Total: 703 controller tests pass (+7 net), 0 regressions.
Without these four fixes, the pipeline would have looked alive in
unit tests but produced zero work in a real deployment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
c8677d985d |
fix(controller): batch J — round-3 cleanup (R3, R4, R5, R6, R8, R10)
R3 — delete unused schema columns: ``workflows.ci_flake_retries_remaining`` and ``workflows.awaiting_ci_started_at`` shipped in round-1 batch D as "staged for future tick handlers" — but no producer or reader ever used them. ci_poll.py uses ``entered_state_at`` (already populated on every transition) as the AWAITING_CI start timestamp. Delete both columns; the flake-retries counter should ship with its producer, not as speculative schema. R4 — safe_json_dumps raises on sets: Previously sets/frozensets were silently coerced to sorted lists. This contradicted the "raise loudly" design intent (JSON has no native set; a reader doing ``parsed["tags"]`` would get a list, losing set algebra). Now raises with an actionable message pointing the producer at ``sorted(list(...))`` for explicit conversion. R5 — TestSQLiteConcurrentDequeue docstring honest: The test name implied it pinned SQLite's busy_timeout retry. It doesn't — :memory: + StaticPool means both threads share one connection (SQLAlchemy serializes per-connection). Updated docstring to say what the test ACTUALLY pins (Python-level serialization safety, exactly-one-winner, clean loser-reason) and explicitly what it doesn't (cross-process SQLITE_BUSY retry, which would need file-backed SQLite + QueuePool — not shipped because multi-machine requires Postgres). R6 — safety timer on test_loop_runs_ci_poll_exhaustion_on_cadence: Previously the test relied entirely on on_iter setting stop when workflows_exhausted>0. If the logic regressed (SQL schema drift, on_iter never seeing the count), the test wedged CI indefinitely. Now armed with threading.Timer(5.0, stop.set) safety net + an assertion that surfaces the failure mode if the safety timer fired first. R8 — _json_safe scope documented honestly: The docstring claimed "everywhere the controller serializes" but the encoder is only wired at runner.py and scheduler.py — the two sites that serialize WORKER-ORIGINATED payloads. Other json.dumps call sites (event-row payloads, discovery markers) serialize fixed-shape dicts of native types and don't need restriction. Updated docstring to scope the claim accurately. R10 — missing test assertions added: - test_strict_parser_coverage_blocks_startup_with_stubs: now captures logs + asserts the operator-facing error message is emitted (so journald shows the cause; a silent rc=2 would be confusing). - test_externally_merged_takes_priority_over_label_removal: now asserts event_type="external-merge" + reason="externally-merged" (a regression that transitioned correctly with the wrong reason in the audit trail would now be caught). - test_second_pause_captures_post_resume_state (NEW): pause/resume/ pause cycle. After RESUME + workflow advances to REVIEWING, a second PAUSE must capture REVIEWING as pre_pause_state (not the original IMPLEMENTING). Round-2 coverage only exercised first pause. Total: 696 controller tests pass (+3 net from new + updated tests), 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
9b6b64f0d3 |
fix(controller): batch I — round-3 correctness items (R2, R7, R9)
R2 — STUCK short-circuits the opt-in label gate:
Round-2's batch G shipped reconciliation-ordering with merged/closed
winning over label removal. But _decide_transition returns
("STUCK", "pr-not-found-on-forgejo") for 404s — that STUCK was
falling through to the label gate. Operator removing the label on a
deleted PR could PAUSE it forever.
Fix: extend the short-circuit set in master/reconciliation.py to
include STUCK alongside MERGED/ABANDONED. All Forgejo-terminal
transitions now bypass the label gate.
Test: test_pr_404_takes_priority_over_label_removal pins the
contract end-to-end (404 + no opt-in label → STUCK, not PAUSED).
R7 — defense-in-depth for corrupt workflow state in ci_poll:
ci_poll.py only caught IllegalTransitionError from apply_event, but
apply_event raises ValueError for states not in KNOWN_STATES (DB
row corruption, unknown-state guard miss). A single bad row would
have aborted the whole tick.
Fix: broaden the exception handler to (IllegalTransitionError,
ValueError). One bad row is skipped; valid rows still STUCK.
Test: test_unknown_state_skips_row_doesnt_crash_tick monkey-patches
apply_event to raise ValueError once + verifies the tick processes
the other workflow normally.
R9 — distinct event_types per reconciliation reason:
Pause / resume / external-merge / external-close / external-issue-
close / pr-not-found-on-forgejo all used event_type='reconciliation'.
Operators querying controller_events for "what happened" could
only distinguish via JSON-payload LIKE queries — dialect-specific
(SQLite LIKE vs Postgres ::jsonb->>).
Fix: master/reconciliation.py introduces _REASON_TO_EVENT_TYPE
mapping:
- opt-in-label-removed → 'label-pause'
- opt-in-label-restored → 'label-resume'
- externally-merged → 'external-merge'
- externally-closed-not-merged → 'external-close'
- issue-closed-externally → 'external-issue-close'
- pr-not-found-on-forgejo → 'external-pr-deleted'
Unknown reasons fall back to 'reconciliation' so future contributors
adding a new reason still emit a well-formed row.
Both _apply_transition and _apply_transition_with_pre_pause now
derive event_type via _event_type_for(reason).
Tests updated (filter by new event_type per case):
- test_master_reconciliation.py::TestEventRows refactored:
- test_transition_emits_external_merge_event
- test_closed_pr_emits_external_close_event (NEW)
- test_pr_404_emits_external_pr_deleted_event (NEW)
- test_consistent_workflow_no_event widened to all 7 event_types
- test_label_gate.py: pause/resume tests filter by 'label-pause' /
'label-resume' respectively + assert the event_type matches.
Total: 693 controller tests pass (+2 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
c4a7f0027d |
fix(controller): batch H — round-2 important items (N5-N10)
Six items from the round-2 adversarial review's "important" list. N5 — restricted JSON encoder replaces ``default=str``: ``default=str`` silently stringified custom objects, sets, and bytes to ``"<MyObj at 0x...>"`` — masking worker output bugs. Now uses a restricted encoder (``tools/controller/_json_safe.safe_json_dumps``) with an allowlist: - datetime / date → ISO-8601 string - Decimal → str (preserves precision) - UUID → canonical string - Path → str - set / frozenset → sorted list (best-effort) - Everything else → TypeError (a worker output regression surfaces loudly instead of writing garbage to the DB) Replaces ``json.dumps(..., default=str)`` at: - ``worker/runner.py`` (terminal-state UPDATE write) - ``master/scheduler.py`` (input_payload INSERT + UPDATE) Tests: ``test_json_safe.py`` (+15 tests) — every allowlisted type + rejected types (custom class, bytes, complex) + nested structures + kwargs forwarding. N6 — strict parser check now runs BEFORE backfill + loop: Previously the strict-parser exit could run AFTER backfill (the test ``test_strict_parser_coverage_blocks_startup_with_stubs`` passed because backfill's ``fail_if_called`` AssertionError was swallowed by the bare ``except Exception``, then strict exited 2). The test asserted the right outcome via the wrong path. Fix: - ``master/__main__.py``: parser-coverage check moved to immediately after engine creation, BEFORE backfill + loop. Strict-mode failure exits 2 without wasting a Forgejo round-trip + without dependent code paths firing. - Test refactored: count-based assertions on backfill and loop call counts (0 each) instead of fail_if_called. Catches regressions where the strict check moves back below either. N7 — _to_aware_datetime unit-tested in isolation: Previously exercised only via end-to-end comment-filter test. New ``TestToAwareDatetime`` (+12 tests) covers: None, empty string, Z suffix, +00:00 offset, microseconds preserved, naive datetime → UTC, malformed string → None, partial string → None, unsupported types → None, timezone abbreviations → None, equality across Z + offset forms (the regression the helper exists to defend). N8 — runtime=None lazy-import branch tested: Previously all 24 HTTP factory tests injected a fake runtime; the production path (``build_callbacks(cfg=None)`` → sys.path injection + lazy import of ``tools._claim_runtime``) was untested. ``TestBuildCallbacksDefaultRuntime`` (+2 tests): verifies the import succeeds + every ForgejoCallbacks attribute is callable; verifies the import is idempotent (second call doesn't crash on sys.path re-insert). N9 — resume event-row emission asserted: Round-1's batch B added the PAUSE event-row test (``test_label_removed_emits_event_with_reason``) but not RESUME. ``test_label_restored_emits_event_with_reason`` pins the resume's controller_events shape (from_state=PAUSED, to_state=<prior>, reason="opt-in-label-restored", source="reconciliation") so operators auditing the timeline see both pause + resume. N10 — concurrent SQLite dequeue test: The dequeue docstring claims SQLite BEGIN-DEFERRED concurrent dequeues "retry via busy_timeout (5s)" — but no test verified. ``TestSQLiteConcurrentDequeue::test_two_threads_racing_one_wins`` spawns two threads, both attempt dequeue simultaneously via a threading.Barrier. Asserts: neither thread raises SQLITE_BUSY- without-retry, exactly one acquires the attempt, the loser sees no_pending_eligible (winner committed first). Total: 691 controller tests pass (+31 net), 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
147e3403c1 |
fix(controller): batch G — show-stoppers from round-2 review (N1–N4)
Four items the round-2 adversarial review flagged as ship-blockers.
N1 — PID-reuse defense is now WIRED in production:
Round 1's batch D shipped ``subprocess_starttime`` in the sidecar +
janitor checks against it, but NO production code wrote sidecars.
The defense was unwired; tests passed against a code path that
production never invoked.
Fix:
- ``worker/agent_runner.py`` accepts ``workspace_dir`` and
``opencode_server_url`` kwargs. When ``workspace_dir`` is set, it
writes a sidecar (``{workspace_dir}/worker.session``) immediately
after MCP spawn capturing the real PID + starttime from
``/proc/{pid}/stat`` field 22. Removes it on attempt completion.
- ``worker/__main__.py`` builds the per-attempt workspace dir
(``{workspace_root}/pr-attempt-{N}/``) and threads it through the
agent_runner closure with the OpenCode URL. The naming convention
is picked up by the janitor's ``pr-*`` glob; when per-PR shared
workspaces ship (Phase 1k++ follow-up), it changes to
``pr-{owner}-{repo}-{N}``.
- Test: ``TestSidecarWiring`` (+2 tests) verifies the sidecar appears
during the attempt, carries the right PID + starttime + instance,
and is cleaned up post-attempt.
N2 — AWAITING_CI escape event firing is now WIRED in production:
Round 1's batch D shipped ``ci_polling_exhausted`` /
``ci_flake_retries_exhausted`` in TRANSITIONS, but NO production code
emitted them. Workflows could still hang in AWAITING_CI forever.
Fix:
- New ``master/ci_poll.py``: ``run_ci_poll_exhaustion_tick`` scans
workflows whose ``entered_state_at`` is older than
``CONTROLLER_AWAITING_CI_TIMEOUT_S`` (default 7200s) and fires
``ci_polling_exhausted`` via ``apply_event`` → STUCK + emits a
``ci_poll_exhausted`` controller_events row with the threshold
payload.
- ``master/loop.py`` integrates the new tick on its own cadence
(``ci_poll_exhaustion_interval_s`` env, default 300s). Composes
with the existing master loop. ``MasterTickReport`` gains
``ci_poll_exhaustion: CIPollExhaustionReport | None``.
- Tests: ``test_master_ci_poll.py`` (+7 tests) — happy path, fresh
workflow stays untouched, only AWAITING_CI is targeted (other
long-lived non-terminal states ignored), event row shape pinned,
default threshold matches the documented 2h, end-to-end loop
integration (master_main_loop drives the exhaustion +
workflow → STUCK without operator intervention).
- Dialect-portable SQL (Postgres interval, SQLite julianday).
- Handles SQLite returning TIMESTAMP as str from text() queries
(no .isoformat() on str).
N3 — externally-merged/closed PRs now win over label removal:
Round-1's PAUSE-on-label-removed shipped, but reconciliation
checked the label gate BEFORE checking merged/closed. Operators
removing the opt-in label on an already-merged PR would PAUSE the
workflow forever — never transitioning to MERGED.
Fix:
- ``master/reconciliation.py:_reconcile_one`` re-ordered:
1. Check terminal-state mappings (merged/closed) FIRST — apply
immediately if they fire.
2. THEN the opt-in label gate (pause/resume).
3. Fall through to "consistent" otherwise.
- Tests: ``test_externally_merged_takes_priority_over_label_removal``
+ ``test_externally_closed_takes_priority_over_label_removal``
pin the contract. Both seed an IMPLEMENTING workflow + Forgejo
reporting "merged/closed AND no opt-in label" → workflow
transitions to MERGED/ABANDONED (not PAUSED) + pre_pause_state
stays None.
N4 — graceful handling of empty env vars:
``int(os.environ.get("CONTROLLER_FORGEJO_REQUEST_TIMEOUT_S", "30"))``
crashes with non-actionable ``int('') ValueError`` if the operator
sets the env to empty/whitespace (common when sourcing a partially-
edited /etc/cleveragents/master.env file).
Fix:
- ``master/forgejo_cfg.py:_env_int(name, default)`` — empty or
whitespace-only values fall back to the documented default; only
non-numeric values still raise (with a clear message naming the
variable).
- Tests: ``test_empty_env_value_falls_back_to_default`` +
``test_whitespace_only_env_falls_back`` + updated
``test_malformed_env_raises_value_error`` to match the new
"not a valid integer" wording.
Total: 660 controller tests pass (+13 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
abff38a274 |
refactor(controller): batch F — ControllerForgejoConfig + parser strict mode (items 16, 17)
Final two items from the consolidated adversarial-review punch list. ITEM 16 — ControllerForgejoConfig: Previously ``master/__main__.py:build_cfg_stub`` reached into ``tools/_mcp_common.ForgejoCfg`` via sys.path injection and mutated ``cfg.owner`` / ``cfg.repo`` after construction. That inverted the dependency direction (the controller is the new system; it shouldn't reach into legacy pipeline modules) and tied controller deployments to whatever schema ForgejoCfg happened to have. Replaced with ``master/forgejo_cfg.py``: a dataclass owning exactly the fields ``_claim_runtime`` reads (token, request_timeout_s, api_retries, claim_ttl_seconds) plus the controller's own (owner, repo). ``from_environment(owner, repo)`` reads the same env vars the legacy ForgejoCfg used (FORGEJO_TOKEN, CONTROLLER_FORGEJO_* tunables) so operators don't have to relearn anything. ``build_cfg_stub`` is now a 1-line delegate to ``from_environment``; no sys.path mutation, no cross-package import. ITEM 17 — parser coverage validator + strict mode: Master startup now calls ``validate_parser_coverage()`` (already exposed by Phase 1j's parser registry) and logs the gap loudly: WARNING CI parser coverage: 3/10 real (7 stub: ['bandit', 'build', 'radon', 'robot_framework', 'semgrep', 'slipcover', 'vulture']). Stub-parser gates fall back to raw_log_excerpt; implementers see the log but not structured findings. Operators who want the strict plan-v10 "STUCK on unknown tool" behaviour set ``CONTROLLER_STRICT_PARSER_COVERAGE=1``; master then refuses to start (exit 2) until every EXPECTED_PARSER has a real implementation. Default off: ship-stubs is the migration-friendly path; strict is the after-everything-is-implemented gate. Tests: - test_forgejo_cfg.py (+13 tests): dataclass shape, env resolution (defaults, FORGEJO_TOKEN preferred over CONTROLLER_FORGEJO_TOKEN, malformed env raises), no-sys-path-injection AST audit + returns the typed dataclass. - test_entry_points.py (+2 tests): strict-mode blocks startup + default mode only warns. Total: 647 controller tests pass (+12 net), 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
99ad7d4df3 |
test(controller): batch E — fill gap inventory (items 12, 13, 14, 15, 18)
Five test-coverage items from the consolidated adversarial-review
punch list.
ITEM 12 — deterministic lock-stealing tests:
Replaced the race-tolerant ``test_stolen_lock_between_agent_return_and_write``
(which asserted ``outcome in {"lost-lock-at-write", "lost-lock"}``
and papered over a real race the test could no longer catch) with
TWO deterministic tests that monkeypatch ``Heartbeat._tick``:
- ``test_stolen_lock_detected_at_write_when_heartbeat_silent`` —
patches _tick to a no-op so the heartbeat thread can't fire;
forces the WRITE path to detect the theft. Pins
``outcome == "lost-lock-at-write"`` deterministically (verified
10/10 runs).
- ``test_stolen_lock_detected_by_heartbeat_when_write_delayed`` —
patches _tick to immediately set lost_lock_event; forces the
heartbeat path. Pins ``outcome == "lost-lock"`` deterministically.
ITEM 13 — Forgejo HTTP factory test gaps:
Added 6 new TestClasses covering the previously-untested factories:
- TestGetPRState (4 tests: 200, 404, 500-raises, non-dict-raises)
- TestGetIssueState (4 tests: same shape — reconciliation depends
on this for issue workflows)
- TestGetPRDetails (5 tests: 200, 404, non-200, transport exception
swallowed, non-dict body)
- TestGetPRDiff (6 tests: str body, bytes body, bytearray body,
non-str/non-bytes body, 404, transport exception)
- TestListPRReviews (3 tests: path correctness, non-200, non-list)
- TestListPRComments (2 tests: uses /issues/{n}/comments path,
non-200 returns empty)
Total: +24 tests over the previously-untested HTTP factory surface.
ITEM 14 — 5-tuple reconciliation_args at loop level:
Added ``test_reconciliation_args_5_tuple_threads_kwargs_to_recon_tick``
which uses the 5-element reconciliation_args form (the variant
production ``__main__.py`` uses) and verifies the extra kwargs
(``require_opt_in_label=True`` + ``opt_in_label="controller-managed"``)
actually reach ``run_reconciliation_tick``. Pins the PAUSE-on-
label-removed behaviour end-to-end through the loop.
ITEM 15 — __main__.py CLI coverage:
Added TestMasterOptInLabelFlag (3 tests):
- ``test_default_enables_opt_in_label_gate`` — without ``--no-opt-in-label``,
backfill and the loop both receive ``require_opt_in_label=True``.
- ``test_no_opt_in_label_flag_disables_gate`` — the flag wires
False through to both backfill + loop.
- ``test_backfill_exception_swallowed_loop_still_runs`` — backfill
ConnectionError doesn't abort the master; loop still starts.
- ``test_master_help_includes_no_opt_in_label_flag`` — operators
can discover the flag via --help.
ITEM 18 — canonical-output-file roundtrip:
Added TestCanonicalOutputFileChannel (2 tests):
- ``test_env_var_set_in_subprocess_and_file_read_back`` — reads
/proc/{pid}/environ to confirm the subprocess actually receives
``CONTROLLER_CANONICAL_OUTPUT_PATH``, then verifies the runner
picks up exactly what the subprocess wrote to that file.
- ``test_file_path_unique_per_attempt`` — two back-to-back attempts
get distinct temp file paths (otherwise concurrent attempts
would race on the same path).
Total: 635 controller tests pass (+32 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
d71046b9a0 |
fix(controller): batch D — PID-reuse, AWAITING_CI escape, flake bound, scheduler skip
Four safety items from the consolidated adversarial-review punch list.
ITEM 8 — PID-reuse hazard in janitor:
The janitor SIGKILL'd whatever process happened to live at the
sidecar's recorded subprocess_pid. Between sidecar write and janitor
sweep, the OS can reuse the PID for an unrelated process; the janitor
was killing innocents under fork-heavy workloads.
Fix:
- ``session_sidecar.py``: added ``subprocess_starttime`` field
(Optional[int]) + ``read_proc_starttime(pid)`` helper that reads
``/proc/{pid}/stat`` field 22 (clock ticks since boot — monotonic
for a (boot, pid) pair).
- ``WorkerSession.from_dict`` filters unknown keys so forward + back
compat with sidecars from earlier/later versions is preserved.
- ``janitor._pid_alive`` and ``_kill_with_grace`` accept
``expected_starttime``; on mismatch they short-circuit and DON'T
signal the impostor.
- ``_kill_with_grace`` return semantics tightened: True iff a signal
was actually delivered (False for "PID gone" / "PID reused"). The
``sessions_killed`` counter now reflects real kills.
ITEM 9 — AWAITING_CI escape from infinite poll:
Previously AWAITING_CI could only exit via ``ci_green`` /
``ci_red_*`` / ``ci_flake_retry`` — if CI hangs forever (runner
outage, broken integration, etc.) the workflow had no controller-
driven STUCK path; only operator_unstick could rescue it.
Fix: new ``ci_polling_exhausted`` event → STUCK. The master's
AWAITING_CI poll handler is the natural place to emit it once a
threshold passes (deferred to a follow-up — Phase 1k+ ships the
event in the table; the timer fires it).
ITEM 10 — ci_flake_retry was unbounded:
The ``ci_flake_retry`` self-loop on AWAITING_CI had no encoded
ceiling. Pathological flaky CI could loop forever (the docstring
said "retry once per gate" but nothing enforced it).
Fix:
- New ``workflows.ci_flake_retries_remaining`` column (server_default
'1', default 1 — operators tune via ``CONTROLLER_CI_FLAKE_RETRIES``
at startup or via direct UPDATE).
- New ``ci_flake_retries_exhausted`` event → ESCALATING. Master
decrements the column on each ci_flake_retry; at 0 the next CI
failure routes through ci_red_* (regular path) or this new
event (escalates if the operator wants a hard ceiling).
ITEM 11 — scheduler now skips PAUSED workflows:
Without this, the scheduler could enqueue a fresh attempt for a
PAUSED workflow between two reconciliation ticks (race: label
removed at T+0, reconciliation runs at T+300, scheduler ticks at
T+30 with stale DB state). The window is at most one attempt of
worker work.
Fix: ``schedule_next_attempts`` SQL now lists only
{ANALYZING, IMPLEMENTING, REVIEWING, CONFLICT_RESOLVING, ESCALATING}
explicitly; PAUSED is excluded by absence. Reconciliation owns the
PAUSED → resume transition; scheduler doesn't touch it.
Schema additions:
- ``workflows.ci_flake_retries_remaining`` (INTEGER NOT NULL DEFAULT 1)
- ``workflows.awaiting_ci_started_at`` (TIMESTAMP NULL) — for the
poll-exhaustion timer (timer impl deferred; column is staged).
Tests:
- TestReadProcStarttime — 3 tests (Linux skip-guard) for the
/proc/pid/stat parser (self-pid > 0, missing pid is None,
invalid pid is None).
- TestJanitor::test_pid_reuse_defended_via_starttime — pins the
contract end-to-end (real subprocess + fabricated wrong starttime
→ janitor doesn't signal).
- TestPhase1kPlusTransitions — 5 tests pinning the new events +
proving the load-bearing invariants still pass.
- test_scheduler_skips_paused_workflows — pins item 11.
Total: 603 controller tests pass (+10 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
a1c6646a64 |
fix(controller): batch C — placeholder patching + pickup_count semantics
Items 3 + 4 from the consolidated adversarial-review punch list. ITEM 3 — placeholders no longer poison the audit trail: The prefetch (master/prefetch.py) writes input_payload with ``attempt_id=0`` and ``attempt_number=1`` as placeholders because the autoincrement PK isn't known until after INSERT. Previously those values stayed in the DB forever — post-mortem queries against ``workflow_attempts.input_payload`` would show ``attempt_id=0`` and operators would chase ghosts. Fix: ``master/scheduler.py:_insert_pending_attempt`` now patches both fields with their real values: - attempt_number: patched BEFORE the INSERT (we compute it as MAX+1). - attempt_id: patched via a follow-up UPDATE after INSERT (we need the autoincrement first). One extra UPDATE per attempt; cheap compared to forever-incorrect audit trail. Test: ``test_scheduler_patches_attempt_id_and_number_into_payload`` asserts the stored payload carries the real values, not the placeholders. ITEM 4 — pickup_count tracks REAPS, not dequeues: Previously the dequeue path bumped ``pickup_count = pickup_count + 1`` on every successful pickup. With ``MAX_PICKUPS=3`` (default), 3 crashed-mid-attempt workers would STUCK the workflow — but that's the wrong semantic. A worker that successfully picks an attempt and runs it should NOT burn a pickup. Only failures (stale-heartbeat reset by the reaper) should count toward the exhaustion limit. Fix: - ``db/dequeue.py`` (both postgres + sqlite paths): removed the ``pickup_count = pickup_count + 1`` UPDATE. Dequeue is a healthy pickup; doesn't bump. - ``reaper.py``: added ``pickup_count = pickup_count + 1`` to the reset UPDATE. Each reap = one failed pickup. - Docstrings updated to reflect the new semantics in both files. Tests: - Updated existing assertions in ``test_db_dequeue.py`` and ``test_reaper_and_pickup_guard.py`` to reflect: dequeue keeps pickup_count; reaper bumps it. - ``TestPickupCountSemantics``: 2 new tests pin the contract end-to-end — N healthy dequeues stay at 0; alternating dequeue→reap→dequeue walks pickup_count up by 1 per reap. Impact: a worker pool that crashes 3 times mid-attempt now needs 3 REAPS (not 3 dequeues) to STUCK the workflow. With default TTL=600s + reaper_interval=60s, that's 30+ minutes of repeated mid-attempt failure before STUCK — appropriately conservative. Total: 593 controller tests pass (+3 new), 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
feeec3e9c7 |
fix(controller): batch B — PAUSED state for label-gate pause/resume (item 2)
Adversarial review flagged: removing the opt-in label transitions a live workflow to ABANDONED — but ABANDONED is TERMINAL with only ``operator_unstick`` re-entry → DISCOVERED, losing all prior controller_events continuity. Operators removing the label to "pause" a long-running PR will be surprised it restarted from scratch. Fix: introduce a non-terminal ``PAUSED`` state. State machine changes (``tools/controller/state_machine.py``): - ``PAUSED`` added to KNOWN_STATES (non-terminal — has exits via ``opt_in_label_restored`` and ``operator_unstick``). - ``opt_in_label_removed`` / ``opt_in_label_restored`` events documented in EVENTS but NOT listed per-state in TRANSITIONS — they're out-of-band master-driven events written directly by reconciliation. Listing them per-state breaks the per-state-event-set invariants (ESCALATING / CONFLICT_RESOLVING). - ``(PAUSED, operator_unstick) → DISCOVERED`` for the escape hatch. Schema change (``tools/controller/db/models.py``): - ``workflows.pre_pause_state: Mapped[str | None]`` column captures the resume target. Master writes it on pause; clears it on resume. Reconciliation logic (``tools/controller/master/reconciliation.py``): - New ``_apply_transition_with_pre_pause`` helper writes both ``current_state`` and ``pre_pause_state`` atomically + emits the ``reconciliation`` event row. - On label removal (current != PAUSED): captures pre_pause_state, transitions to PAUSED. - On label restoration (current == PAUSED): reads pre_pause_state (fallback DISCOVERED for legacy NULL data), transitions back, clears pre_pause_state. - PAUSED workflows are now SCANNED by reconciliation (not just non-terminals) so we can detect label-restored. Behaviour matrix: | current | label | result | |---------|----------|------------------------------------------| | any != | absent | → PAUSED, pre_pause_state = current | | PAUSED | present | → pre_pause_state (or DISCOVERED) | | PAUSED | absent | stays PAUSED (no transition) | | any != | present | regular state checks (no-op for label) | Tests (test_label_gate.py refactor + 3 new tests): - test_label_removed_pauses_workflow (was: abandons) - test_label_restored_resumes_from_pre_pause_state (new) - test_paused_workflow_without_label_stays_paused (new) - test_resume_fallback_when_pre_pause_state_missing (new — legacy data without the new column) - Existing event-reason test still passes (reason string unchanged). Total: 590 controller tests pass, 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
6ba1926c52 |
fix(controller): batch A — datetime json, Forgejo state map, ISO compare, dequeue docs
Four narrow bug fixes flagged by adversarial code review (items 1, 5, 6, 7 from the consolidated critique). ITEM 1 — datetime → json.dumps crash (silent write-after-work failure): - ``worker/runner.py:274`` and ``master/scheduler.py:283`` now pass ``default=str`` to ``json.dumps`` so nested datetime fields (e.g. CISummary.observed_at) serialize without raising. - Before this fix: a worker would do its real work, then crash on the terminal-state UPDATE with TypeError, get recorded as ``worker-internal-error``, and the output payload would be lost. - Test: TestDatetimeSerializationSafety in test_master_ci_summarize + test_scheduler_handles_datetime_in_input_payload in test_master_prefetch (both pin the regression — the with-default test passes, the without-default test asserts the TypeError so future maintainers see the failure mode). ITEM 5 — Forgejo state mapping completeness: - Extended ``_FORGEJO_STATE_TO_GATE_STATUS`` in ``master/ci_summarize.py`` to cover ``cancelled``, ``timed_out``, ``action_required``, ``queued``, ``in_progress``, ``neutral``, ``skipped``, ``stale`` — states observed across Forgejo / Gitea / GH-mirror that previously collapsed to ``pending``, telling the implementer "CI is still running" when really a job was cancelled. - ``cancelled`` / ``timed_out`` / ``action_required`` / ``stale`` now map to ``error`` (the gate failed). - ``queued`` / ``in_progress`` stay ``pending`` (still running). - ``neutral`` / ``skipped`` → ``passed``/``skipped`` (informational). - Test: TestExtendedForgejoStates — 6 tests covering each new state. ITEM 6 — lexicographic ISO comparison drops/dupes comments: - ``master/prefetch.py:_comment_bodies_since`` and ``_iso`` replaced with ``_to_aware_datetime`` + datetime comparison. Forgejo emits ``2026-05-18T12:00:00Z``; Python's ``datetime.isoformat()`` emits ``2026-05-18T12:00:00+00:00`` — a string compare gives 'Z' (0x5A) vs '+' (0x2B) which silently misorders timestamps. - Now parses via ``datetime.fromisoformat`` (with Z → +00:00 rewrite), defaults naive timestamps to UTC, and compares as ``datetime``. - Test: test_comments_filter_handles_z_suffix_vs_offset_form pins the regression. ITEM 7 — false BEGIN IMMEDIATE claim in dequeue docstring: - The dequeue docstring claimed ``BEGIN IMMEDIATE`` was applied by session_scope; it wasn't. Attempted a global ``begin``-event listener that conflicted with StaticPool's shared-connection model (test_prefetch_callback_works_in_scheduler broke). - Reverted to a documentation fix: SQLite stays on default BEGIN DEFERRED (the SQLITE_BUSY retry via busy_timeout=5s is acceptable for single-host dev) + the docs make MULTI-MACHINE REQUIRES POSTGRES explicit at three call sites (db/session.py, db/dequeue.py, RUNBOOK.md was already updated in Phase 1l). Postgres has FOR UPDATE SKIP LOCKED which is what production actually uses. Tests: 586 controller tests pass (+10 new), 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
1e27039555 |
feat(controller): render CI raw_log_excerpt + target_url in worker prompts
Phase 1j shipped prefetch + parsers; Phase 1i shipped the prompt templates. But the prompt's CI-summary section only rendered each failing gate's name, status, and summary_line — the raw_log_excerpt (up to 16KB captured per gate) and target_url were sitting in the input_payload dict, never surfaced to the LLM. Impact: for stub-parser gates (7 of 10 tools — robot_framework, bandit, semgrep, vulture, radon, slipcover, build), the implementer saw "parser pending; see raw_log_excerpt" with no log in the prompt. It would have to shell out via OpenCode to re-fetch the log, which is wasteful and a regression vs the existing pipeline. This commit: - Renders ``target_url`` as ``log: <url>`` for failing gates so the LLM can curl it if needed. - Renders ``raw_log_excerpt`` inside a fenced code block, truncated to 3KB per gate (storage cap is 16KB; trim to prompt-friendly size with a marker preserving the original byte count). - Renders ``composite_findings`` as nested sub-findings (security_scan → bandit head + semgrep/vulture children, each with their own excerpt) so multi-tool gates are first-class in the prompt. - Skips the fence + URL for passing gates (already filtered) and for failures with no excerpt (defensive — keeps clean parsers from emitting empty log blocks). Conversion: the 20% of CI failures handled by stub parsers go from "broken — implementer doesn't see the log" to "less structured — implementer sees the raw log and can reason about it." Real parsers still ship in priority order as a follow-up. Tests (+7 in TestCIFailureRendering, 0 regressions): - target_url rendered for failing gate - target_url skipped for passing gate - raw_log_excerpt rendered in markdown code fence - Long log excerpt truncated with marker - Missing excerpt → no log block (no empty fence) - composite_findings render as sub-findings with their own excerpts - Stub-parser-pending failure now surfaces the actual log (the whole point of the commit) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
a19b609554 |
feat(controller): Phase 1l — systemd units + ops runbook
Deployment surface for the controller. Operators copy the unit files
+ env examples to /etc/, customize, and `systemctl enable --now`.
Layout (under tools/controller/deploy/):
- systemd/cleveragents-controller-master.service — master singleton.
One unit per (owner, repo); plan v9 singleton constraint enforced
by deployment.
- systemd/cleveragents-controller-worker@.service — worker template.
`systemctl enable cleveragents-controller-worker@implementer-1`
spins up one instance; scale by adding instances. Per-instance
env override at /etc/cleveragents/worker.<inst>.env (optional)
specialises CLEVERAGENTS_WORKER_ROLES per instance.
- systemd/master.env.example — every env var the master reads,
documented. Copy to /etc/cleveragents/master.env, mode 0640.
- systemd/worker.env.example — every env var the workers read.
- RUNBOOK.md — operator guide:
* Prereqs (Linux + systemd 245+, Python 3.13 + uv, Postgres 14+,
OpenCode server, dedicated cleveragents user).
* First-time setup (7 ordered steps from useradd to first PR).
* Day-to-day ops (where logs live, healthy queries, role pool
sizing, label-based pause, clean restart).
* Incident response (6 named scenarios: STUCK workflows, MERGING
retry exhaustion, no-OpenCode, dequeue starvation, DB loss,
operator-unstick procedure).
* Migration playbook (Phase 2: opt one PR in via label,
monitor controller_events, escalate to full management).
* Tunables cheat sheet covering all 8 env vars + their tradeoffs.
- README.md — index linking the above.
Hardening on both units: NoNewPrivileges, PrivateTmp,
ProtectSystem=strict, ProtectHome, narrow ReadWritePaths, kernel
+ control-group protections. RestartPreventExitStatus=2 prevents
systemd loop-restart on misconfiguration (exit 2 = bad env).
No code changes; verified via `systemd-analyze verify` (unit syntax
parses; only "venv path doesn't exist on dev host" warnings, which
are expected). Test suite: 569/569 pass, 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
72c272504c |
feat(controller): Phase 1k — controller-managed opt-in label gate
Migration safety mechanism: the controller only manages PRs and
issues carrying a configurable opt-in label (default
``controller-managed``). Operators opt PRs in for parallel-run
trials, can pause management mid-flight by removing the label, and
gradually roll out without exposing the controller to PRs that
human reviewers are actively driving.
Components:
- tools/controller/master/label_gate.py — single source of truth for
the configured label name + pure predicates/filters over Forgejo
PR/issue dicts.
- ``opt_in_label_name()`` reads ``CONTROLLER_OPT_IN_LABEL`` env
(default 'controller-managed'); empty/whitespace falls back.
- ``has_opt_in_label(entity, name)`` defensively handles every
degenerate shape (non-dict entity, non-list labels, non-dict
label entries, missing name field).
- ``filter_by_opt_in_label`` / ``count_filtered`` for callers.
Wired through:
- discovery.run_discovery + backfill.run_startup_backfill +
reconciliation.run_reconciliation_tick each accept
``opt_in_label`` and ``require_opt_in_label`` kwargs.
- Function defaults are ``require_opt_in_label=False`` for API
back-compat (existing 30+ discovery/backfill/recon tests work
without changes).
- __main__.py defaults to ``--no-opt-in-label`` OFF (gate ENABLED in
production); add ``--no-opt-in-label`` to bypass.
- DiscoveryReport gains a ``label_filtered_out`` counter.
Reconciliation behavior:
- When opt_in_label is configured AND the Forgejo response carries a
``labels`` field AND the opt-in label is NOT present, the workflow
transitions to ABANDONED with reason ``opt-in-label-removed`` +
emits a controller_events 'reconciliation' row.
- Partial Forgejo responses (no ``labels`` field) skip the label
check — never ABANDON on incomplete data.
Master loop extension:
- ``reconciliation_args`` now accepts an optional 5th element — a
kwargs dict threaded through to ``run_reconciliation_tick``.
__main__.py uses this to pass ``require_opt_in_label`` per the CLI
flag. 4-tuple back-compat preserved.
Tests (+29 in test_label_gate.py, 0 regressions across 569 tests):
- Predicate edge cases (every degenerate shape returns False)
- Env-var resolution (default, override, empty, whitespace)
- filter/count helpers
- Discovery + backfill: kept/filtered counts, gate disabled,
explicit label overrides env
- Reconciliation: label removed → ABANDONED, label present →
no-op, partial response → no-op, gate disabled → bypass, event
row records reason
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
3e269ce011 |
feat(controller): Phase 1j — deterministic CI summarizer + priority parsers
Replaces "ci_summary=None / failing_gates=[]" placeholders from Phase
1h with a real summarizer that maps Forgejo combined-status →
CISummary V1 dict by running per-tool deterministic parsers on each
failing gate's log.
Priority parsers shipped (cover lint/format/typecheck/unit_tests, the
4 most-failed gates):
- ruff — F+E codes from `nox -s lint`; Would-reformat lines from
`nox -s format`. Aggregates to single error_class when
all findings share one code, else RuffMixed.
- pyright — error/warning/information diagnostics; rule name pulled
from trailing `(reportName)` parens. Abs-path
normalization strips container prefixes.
- behave — failing scenarios (file:line + name), AssertionError
extraction. Feature/scenario summary line aggregation.
Stub parsers for not-yet-shipped tools (robot_framework, slipcover,
bandit, semgrep, vulture, radon, build): return a structured
CIFailure with error_class="parser-pending-{name}" + the raw log
excerpt. Operators see the failure; implementer still has log
context. Phase 1j+ replaces stubs with real parsers without changing
the gate-→-session map.
Components:
- _base.py — ParserResult dataclass + select_log_excerpt()
(tail-N-lines smart selection within 16KB cap)
- _stub.py — make_stub(name) factory for pending tools
- _registry.py — resolve(parser_name) + resolve_for_nox_session()
+ validate_parser_coverage()
- master/ci_summarize.py — summarize_ci_status(head_sha, status,
log_fetcher) orchestrator. Handles:
- composite multi: gates → CIFailure.composite_findings
- log_fetcher returning None → log-fetch-failed
- log_fetcher raising → caught + log-fetch-failed
- Unknown gate context → NoParserAvailable
- Forgejo state=None → unknown summary
- Parameterized matrix gates ("unit_tests-3.13")
→ base session name resolution
Tests (+46 across 2 new files, 0 regressions):
- Per-parser canonical + empty + garbage input
- Registry resolution (real vs stub), coverage validator
- Summarizer V1 contract round-trip
- Composite security_scan composite_findings shape
- Error paths (None status, raising fetcher, unknown gate)
- Parser version aggregation across mixed gates
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
1b09a2054f |
feat(controller): Phase 1i — per-role prompt templates
Replace the JSON-dump prompt stub with structured per-role builders
that render the V1 input contract fields in human-friendly sections.
The agent's system prompt (.opencode/agents/{name}.md) provides the
HOW (tool calls, response shape); this module provides the WHAT (the
per-attempt context drawn from input_payload).
Per-role builders in tools/controller/worker/prompts.py:
- build_implementer_prompt — branch coords, CI summary (highlights
failing gates), active reviewer state, new comments, prior attempts,
optional allowed_files scope, output contract reminder
- build_reviewer_prompt — PR coords, CI summary, full diff (with
code fence), implementer's latest claim, prior reviewer outputs
- build_estimator_prompt — PR/issue context (title, body, diff
summary, head_sha), tier-pick guidance
- build_conflict_resolver_prompt — coords, conflicted files (or
porcelain fallback), prior implementer outputs
- build_summarizer_prompt — aged-out attempt + prior running
summary; instructs terse condensation
- build_prompt(role, tier, input_payload) — dispatcher
Truncation budgets (in chars):
- full_diff: 12,000
- pr_body / summaries: 4,000
- prior-attempt JSON: 2,000
- comment one-liner: 200
Wired as the default prompt builder in opencode_session via
``_default_prompt_for``. wire_opencode_session() callers that pass
prompt_builder=... override per-test.
Tests (+35 in test_worker_prompts.py, 1 updated adapter test, 0
regressions):
- Per-role: section headers present, required fields rendered,
missing fields fall back to (unavailable), oversize fields
truncated
- Dispatcher: routes by role; raises ValueError for missing tier on
tiered roles + unknown role
- Adapter integration: _default_prompt_for delegates to build_prompt
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
dfdfbf762b |
feat(controller): Phase 1h — prefetch callbacks (V1 input assembly)
Per plan v9, the master assembles the worker input_payload at attempt-enqueue time so the worker dequeues a ready-to-use payload with no extra Forgejo I/O of its own. This phase ships per-role V1-input builders + a factory matching the scheduler's PrefetchCallback protocol: - build_implementer_input → ImplementerInputV1 shape (head_sha, head_ref, base_branch, active_reviews, pr_comments_since_last_attempt, prior_attempts, diff_summary) - build_reviewer_input → ReviewerInputV1 shape (full_diff, prior_implementer_attempts, implementer_claim, prior_reviews) - build_estimator_input → EstimatorInputV1 shape (pr_title, pr_body, diff_summary) — works for both PR and issue kinds - build_conflict_resolver_input → ConflictResolverInputV1 shape with conflicted_files=[] stub (worker patches via git rebase) - make_prefetch_callback(engine, callbacks) → routes by role; returns (payload, "V1") matching the scheduler's PrefetchCallback signature Forgejo HTTP wiring adds four new callbacks (get_pr_details, get_pr_diff, list_pr_reviews, list_pr_comments) plumbed through ForgejoCallbacks. Worker-side patches (post-dequeue, pre-validation): - attempt_id, attempt_number (known from dequeue) - workspace_dir (worker filesystem path) - wallclock_budget_s (worker config) What this phase DOES NOT yet produce: - ci_summary / failing_gates — Phase 1j (deterministic CI summarizer) - Issue-kind estimator's title/body — needs list_issue_details callback (defer to future phase) - conflict_resolver's actual conflicted_files — needs worker-side git rebase + conflict-parse pass Tests (+26 in test_master_prefetch.py, 0 regressions): - Per-role shape validation + V1 contract parse after worker patches - Prior-attempts merge (verbatim cap=3, oldest-first, total count) - Active-reviews projection (filters invalid states/missing user) - pr_comments_since_last_attempt filtering by finished_at - Factory routes by role; unknown role raises - Scheduler integration end-to-end (real prefetch → real INSERT) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
b76c5f05e7 |
feat(controller): wire reconciliation into master main loop
Composes reconciliation as the 4th tick layer at its own cadence. tools/controller/master/loop.py: - MasterConfig gains reconciliation_interval_s (default 300s per plan v9). - MasterTickReport gains reconciliation: ReconciliationReport | None. - master_main_loop gains reconciliation_args parameter — tuple of (owner, repo, get_pr_state_cb, get_issue_state_cb). When provided, runs run_reconciliation_tick every reconciliation_interval_s. When None, reconciliation is disabled (useful for tests + one-shot modes). - Reconciliation exception is caught + logged; master keeps running. - Iteration log line now includes reconciled=N. tools/controller/master/__main__.py: - Passes reconciliation_args from ForgejoCallbacks (built earlier in the entry point) so the production master automatically runs reconciliation against the configured (owner, repo). 3 new tests in test_master_loop.py: - reconciliation_fires_when_configured: workflow with externally- merged state → reconciliation transitions to MERGED. - reconciliation_skipped_when_args_none: workflows untouched + no reconciliation reports. - reconciliation_exception_doesnt_break_loop: per-row fetch failures don't crash the master. Total: 433 controller tests; full auto_agents suite 2795 pass. |
||
|
drew
|
7d1dfb8635 |
feat(controller): Phase 1g — periodic reconciliation tick
Catches externally-merged / externally-closed PRs that the controller
didn't directly merge (operator clicked the merge button in Forgejo's
UI; collaborator closed a PR while controller was waiting). Inverse
of discovery: discovery ADDS new entities; reconciliation re-syncs
KNOWN ones.
tools/controller/master/reconciliation.py:
- run_reconciliation_tick(engine, owner, repo, get_pr_state,
get_issue_state=None):
- Scans all non-terminal workflows for (owner, repo).
- Calls get_pr_state / get_issue_state per workflow.
- Decision table:
- PR merged=True → MERGED (reason='externally-merged')
- PR state=closed not merged → ABANDONED ('externally-closed-
not-merged')
- PR state=open → consistent (no transition)
- PR not found (404) → STUCK ('pr-not-found-on-forgejo')
- Issue state=closed → ABANDONED ('issue-closed-externally')
- Per-row failures isolated: one Forgejo flake doesn't kill the
whole sweep. Failed fetches recorded as ReconciliationAction
with reason='fetch-failed' (workflow untouched).
- Pure decision function (_decide_transition) separated from SQL
writes (_apply_transition) for testability.
tools/controller/master/forgejo_http.py:
- Added get_pr_state + get_issue_state callbacks to ForgejoCallbacks.
- HTTP shape: 404 → None (workflow → STUCK); non-200/404 → raise
(workflow recorded as fetch-failed, not silently STUCK'd).
15 new tests in test_master_reconciliation.py:
- basics (empty DB, terminal workflows skipped, other-repo skipped)
- PR state mappings (open=consistent, merged → MERGED, closed →
ABANDONED, not-found → STUCK)
- issue state mappings (open=consistent, closed → ABANDONED,
no-callback → consistent)
- fetch failures (raised exception → fetch-failed, workflow untouched)
- event rows (reconciliation event emitted on transition; none on
consistent)
- ReconciliationAction dataclass shape
Total: 430 controller tests; full auto_agents suite 2792 pass.
|
||
|
drew
|
ba2e9472bc |
feat(controller): Phase 1f — master startup backfill
When the master starts (first deploy or after a long outage), it
needs to learn about existing open PRs/issues that weren't created
via discovery-tick-during-uptime. Backfill = discovery + a one-time
marker so subsequent restarts know "this isn't the first time."
tools/controller/master/backfill.py:
- run_startup_backfill(engine, owner, repo, list_prs, list_issues):
- Calls run_discovery (already idempotent — skips existing entities)
- Records a 'controller-backfill-complete' marker in controller_events
associated with the first new workflow OR an existing workflow OR
skipped if Forgejo is truly empty (no FK target)
- Returns BackfillReport{first_time, discovery}.
- has_backfill_run(engine, owner, repo): existence-check on the marker
by parsing controller_events.payload. Multi-tenant isolated — a
marker for (owner_a, repo_a) doesn't satisfy a check for
(owner_b, repo_b).
- The marker is informational; the dedup is provided by discovery's
unique-constraint skip. The marker exists so operators can answer
"has backfill ever run for this repo?" in one SQL query.
Wired into master __main__:
- Runs AFTER engine/create_all + Forgejo callback wiring, BEFORE
master_main_loop.
- Try/except wrapped so Forgejo flake at startup doesn't prevent
the main loop from running (discovery tick will retry).
- New --skip-backfill flag for tests + warm restarts.
8 new tests in test_master_backfill.py:
- has_backfill_run: no marker → False; multi-tenant isolation
(different owner OR different repo → False).
- First-time backfill creates workflows + marker; empty Forgejo
skips marker (no FK target).
- Second run reports first_time=False; picks up newly-appeared PRs
+ emits a second marker.
- Multi-tenant (owner-a, repo-a) and (owner-b, repo-b) both get
their own marker.
- New workflows are in DISCOVERED state.
Total: 415 controller tests; full auto_agents suite 2777 pass.
|
||
|
drew
|
7b1b68b784 |
feat(controller): Phase 1e — master + worker __main__ entry points
Runnable as `python -m tools.controller.{master,worker}` for systemd
deployment. Wires the production callbacks (Forgejo HTTP +
OpenCode session adapter + agent_runner) into the previously-shipped
main loops.
tools/controller/master/__main__.py:
- Args: --owner, --repo (required); --opencode-url, --log-level,
--tick-interval, --discovery-only-once (smoke flag).
- Reads CLEVERAGENTS_DB_URL (returns 2 if unset — operator-actionable
error visible in systemd logs).
- build_cfg_stub reuses _mcp_common.ForgejoCfg → PAT rotation +
Forgejo URL env vars work without a new config module.
- SIGTERM/SIGINT → stop_event → master_main_loop drains + exits.
- --discovery-only-once: runs one discovery sweep + exits. Useful for
initial backfill or smoke testing the Forgejo callback wiring
without committing to the long-running loop.
tools/controller/worker/__main__.py:
- Args: --opencode-url, --roles (default all 5; comma-separated),
--max-concurrent, --poll-interval, --log-level, --no-startup-janitor.
- Empty --roles after parsing → exit 2.
- Startup sequence:
1. Run orphan-workspace sweep (unless --no-startup-janitor)
2. Build OpenCode session adapter via wire_opencode_session
3. Wrap production_agent_runner with the session injected
4. Wire SIGTERM/SIGINT → stop_event
5. Enter worker_main_loop
9 new tests in test_entry_points.py:
- Arg parsing: master --owner/--repo required; both DB-URL-missing
returns 2; worker empty-roles returns 2.
- --help works via subprocess for both (no SystemExit issue).
- Master --discovery-only-once smoke: stubs build_callbacks + cfg
builder; verifies exit 0 + the discovery call chain runs.
- Worker --no-startup-janitor: stubs sweep + loop; verifies sweep
is skipped when flag set, runs when not.
Total: 407 controller tests; full auto_agents suite 2769 pass.
Controller v1 is now end-to-end deployable as two systemd units
(one per machine for each instance). The remaining work for a real
production rollout is:
- Real prefetch callbacks for the scheduler (assembling V1 input
payloads from Forgejo data)
- Backfill at master startup (existing-PRs → DISCOVERED rows)
- Reconciliation tick (DB ↔ Forgejo sync)
- Per-role prompt templates (the V1 prompt stub works; specialised
per-role prompts will land as agents are migrated)
|
||
|
drew
|
bb8d872ba7 |
feat(controller): Phase 1d-4 — OpenCode session adapter
The production wrapper that wires the existing
_opencode_worker.run_session_blocking into the controller's
run_opencode_session protocol that production_agent_runner expects.
tools/controller/worker/opencode_session.py:
- agent_name_for(role, tier): role+tier → OpenCode agent name.
- implementer + tier {0,1,2} → task-implementor-tier-{0,1,2}
- reviewer → pr-review-worker
- estimator → estimator-implementation
- conflict_resolver → conflict-resolver-worker
- summarizer → controller-summarizer (new agent name)
- DEFAULT_TIER_TIMEOUT_S: per-tier wallclock budgets via env vars.
Defaults match plan v9 (tier 0: 600s, tier 1: 1200s, tier 2: 1800s).
Reviewer/estimator/conflict use the default-agent timeout (600s).
- wire_opencode_session(opencode_server_url, ...) → callable matching
the production_agent_runner's run_opencode_session contract.
- Resolves role+tier to agent name + per-tier timeout
- Builds the prompt (default stub or custom builder; per-role prompt
templates are a follow-up)
- Calls run_session_blocking with an on_poll callback that raises
WorkerLostLock if the controller's lost_lock_check returns True
mid-session (heartbeat thread reaper detected stolen lock)
- Routes SessionResult.status:
- 'completed' → return None (MCP's canonical output is in the
tempfile; production_agent_runner reads it after we return)
- 'timeout' → raise WorkerError(worker-internal-error)
- 'transport-error' → raise WorkerError(worker-internal-error,
including error_kind for forensics)
- unknown → raise WorkerError defensively
- Any unexpected exception from run_session_blocking wrapped as
WorkerError(worker-internal-error).
- Production wiring: dependency injection. wire_opencode_session()
lazy-imports the real run_session_blocking when not overridden.
Tests inject a stub.
22 new tests in test_worker_opencode_session.py:
- agent_name_for: parametrized per role (4 implementer-tier rows +
4 flat-role rows) + invalid tier + unknown role
- DEFAULT_TIER_TIMEOUT_S: each tier has a timeout + monotonic
- wire_opencode_session: completed/timeout/transport-error/unknown-
status routing; run_session_blocking raises wrapped as WorkerError;
on_poll propagates lost_lock; on_poll no-op when lock held;
custom prompt_builder used; default prompt includes role + tier
+ finalize() instruction + rendered input_payload; tag prefix
customizable.
Total: 398 controller tests; full auto_agents suite 2760 pass.
|
||
|
drew
|
23a014e402 |
feat(controller): Phase 1c-3 — production agent_runner (real MCP subprocess)
The production seam that connects the controller's runner to actual
LLM execution. Per-attempt MCP subprocess spawn + OpenCode session
drive + canonical JSON read + strict-parse → output dict.
tools/controller/worker/agent_runner.py:
- production_agent_runner(): the function tested production wires
into run_one_attempt's agent_runner param. Per-attempt lifecycle:
1. Map role → matching MCP module + V1 output contract.
2. mkstemp a canonical-output path; spawn `python -m {mcp_module}`
subprocess with CONTROLLER_CANONICAL_OUTPUT_PATH set in env.
3. Call injected run_opencode_session (tests use JSON-RPC stdin
driver; production wires _opencode_worker.run_session_blocking).
4. Wait for the MCP's finalize_and_emit to write the canonical
JSON to the tempfile.
5. Strict-parse against the role's V1 output model.
6. SIGTERM/grace/SIGKILL the MCP + unlink the tempfile in finally.
- ROLE_TO_MCP_MODULE / ROLE_TO_OUTPUT_MODEL: explicit per-role
dispatch tables. Covers all 5 worker roles.
- Error classification per the WorkerError outcome enum:
- unknown role → ValueError (master bug; not a worker outcome)
- session raises generic Exception → WorkerError(worker-internal-error)
- session raises WorkerLostLock → propagated unchanged
- lost_lock_check returns True after session → WorkerLostLock
- MCP didn't emit canonical output within timeout → WorkerError(
worker-internal-error)
- MCP emitted JSON that fails strict-parse → WorkerError(
contract-violation; per v9 status='failed' policy maps to
workflow STUCK)
tools/controller/mcp/_builder_base.py:
- finalize_and_emit() now writes to CONTROLLER_CANONICAL_OUTPUT_PATH
if set (production subprocess path) or stdout if not (direct-call
test path; capsys captures). Decouples canonical output from the
FastMCP JSON-RPC stdout transport — they would otherwise collide
in the subprocess (both writing to the same stream).
- Atomic file write: open + write + fsync + close.
- Existing test_mcp_builders.py (capsys-based) still passes with the
fallback path; new test_worker_agent_runner.py exercises the
file-based subprocess path with real MCPs.
10 new tests in test_worker_agent_runner.py:
- End-to-end with real MCP subprocesses (implementer-resolved,
reviewer-approve, estimator-tier-recommendation)
- Error paths: unknown role (ValueError), session raises (wrapped
as WorkerError), session raises WorkerLostLock (propagated),
lost_lock_check returns True post-session (raises WorkerLostLock),
no finalize → finalize_timeout → WorkerError
- Role-map sanity: every role has both an MCP module and output model
Total: 376 controller tests; full auto_agents suite 2738 pass.
This commit completes the v1 boundary — the controller now has the
complete stack from V1 contracts through MCP response builders, DB
schema, worker dequeue/heartbeat/runner/workspace, master state
machine + tick + reaper + pickup guard + scheduler + discovery +
Forgejo writes + MERGING + HTTP adapter, AND a production
agent_runner that wires it all to real OpenCode + MCP subprocess
spawning. Phase 1d-3+ enhancements (real OpenCode session wiring
in run_opencode_session) and Phase 2+ migration steps remain.
|
||
|
drew
|
ae940f4564 |
feat(controller): Phase 1d-3c-4 — HTTP adapter wiring callbacks to _claim_runtime
The production glue between the controller's callback protocols
(discovery / forgejo_writes / merging) and the existing Forgejo HTTP
client in _claim_runtime. Single build_callbacks(cfg) factory; tests
use the same module with a fake runtime stub.
tools/controller/master/forgejo_http.py:
- ForgejoCallbacks dataclass bundling every callback the controller
needs: list_prs, list_issues, list_comments, post_comment,
get_labels, add_label, remove_label, merge_pr.
- build_callbacks(cfg, runtime=None): wires each callback as a thin
closure over runtime.get/post/delete. Production omits ``runtime``
to use the real _claim_runtime module; tests inject a fake.
- Forgejo path conventions match the API:
GET /repos/{o}/{r}/pulls?state=open
GET /repos/{o}/{r}/issues?state=open&type=issues (excludes PRs)
GET/POST /repos/{o}/{r}/issues/{n}/comments
GET/POST /repos/{o}/{r}/issues/{n}/labels
DELETE /repos/{o}/{r}/issues/{n}/labels/{name} (URL-encoded)
POST /repos/{o}/{r}/pulls/{n}/merge (body: {"Do": "merge"})
- Robust response handling:
- list endpoints: non-200 → empty list; non-list body → empty;
non-dict items filtered out.
- post_comment: 200/201 ok; other → RuntimeError.
- add_label / remove_label: 200/201/204 → True; remove-404 → True
(label already gone = goal achieved); else False.
- merge_pr: returns normalized MergeResponse. On 404, fetches the
PR's actual state (merged=True → pr_state='merged'; state='closed'
→ 'closed'; PR fetch failure or non-200 → leave pr_state=None so
the merging handler defaults to ABANDONED conservatively).
- Any callback exception → synthetic 503 so the merging handler's
retry logic kicks in cleanly.
23 new tests in test_master_forgejo_http.py:
- list_prs (path format, non-200 → empty, non-list body → empty,
filters non-dict items)
- list_issues (type=issues filter)
- list_comments (path format)
- post_comment (201, 200, non-2xx raises, non-dict body)
- labels (get / add 201/500 / remove 204/404/URL-encoded)
- merge (200, 409, 500, callback-raises-as-503, 404+merged,
404+closed, 404+pull-fetch-failure)
Plus a bug fix surfaced by the test_404_with_pull_fetch_failure test:
the PR-state-fetch branch was returning 'open' on a 500 response;
now correctly checks status==200 before inspecting the body.
Total: 366 controller tests; full auto_agents suite 2728 pass.
|
||
|
drew
|
94821d2702 |
feat(controller): Phase 1d-3c-3 — MERGING handler (6-response-shape table)
The master's per-tick handler for workflows in MERGING. Per plan v6+v9
non-blocking retry: state stays MERGING across ticks, retry counter +
backoff tracked in workflows.merging_retry_count and
merging_retry_next_attempt_at.
tools/controller/master/merging.py:
- MergeResponse: normalized {status_code, pr_state, error_message}
the callback returns. status_code drives the 6-response table:
200 → MERGED
409 → IMPLEMENTING(tier_last_succeeded) with reason=
'post-approval-base-conflict'
403 → STUCK ('branch-protection')
422 → AWAITING_CI ('ci-required-status-missing')
5xx (any) → stay MERGING; bump retry_count + schedule
next_attempt_at = now + 2^retry_count seconds
(capped at 60s); STUCK at retry_count >= 5
404 + pr_state='merged' → MERGED ('externally-merged')
404 + pr_state='closed' → ABANDONED ('externally-closed')
404 + no pr_state → ABANDONED (conservative default)
- run_merging_tick(engine, merge, owner, repo) — sweeps workflows
WHERE current_state='MERGING' AND kind='pr' AND owner+repo match
AND (next_attempt_at IS NULL OR next_attempt_at <= now). Per row:
call merge → map response → transition (or schedule retry) + emit
controller_events row.
- Callback failure (callback raises) wrapped as a synthetic 503 so
the retry logic kicks in cleanly.
- Counter resets on non-retry responses (409 / 422) — keeps backoff
fresh for future retries.
- MAX_MERGE_RETRIES = 5; MAX_BACKOFF_S = 60.0.
14 new tests across 7 classes:
- 200 happy path + event row
- 409 → IMPLEMENTING(tier_last_succeeded)
- 403 → STUCK
- 422 → AWAITING_CI + retry counter reset
- 404 paths (merged + closed + no-state default)
- 5xx retry (counter bumped + backoff scheduled +
in-window-skipped + max-retries-STUCK + callback-raises-as-5xx)
- Owner/repo filter (other repo's MERGING untouched)
- kind='pr' filter (issues never processed even if mis-seeded)
Total: 343 controller tests; full auto_agents suite 2705 pass.
|
||
|
drew
|
3e3796d918 |
feat(controller): Phase 1d-3c-2 — Forgejo write helpers (status + labels)
Idempotent status comment posting via fingerprint markers + per-label
no-op-aware adjust. Production wires the callbacks to existing
_status_comments + _claim_runtime helpers; tests use fakes.
tools/controller/master/forgejo_writes.py:
- compute_fingerprint(workflow_id, event_kind, content_key) →
16-char SHA256 prefix. Deterministic; same inputs → same fingerprint.
Different (workflow_id OR event_kind OR content_key) → different fp.
- build_marker(fp) → HTML comment "<!-- controller:fingerprint:abc -->".
Searchable + invisible in Forgejo's rendered UI.
- comment_has_fingerprint(body, fp) → bool. Used by post_status_comment
for the dedup check before posting.
- post_status_comment(): full idempotency protocol —
1. compute fingerprint
2. list_comments callback → check existing for marker
3. if found → return duplicate (no post)
4. else post via post_comment callback
Failure modes:
- list_comments raises → fall through to post (conservative;
fingerprint match on next sweep catches the duplicate)
- post_comment raises → return failed; caller retries
- adjust_labels(add, remove): one-shot get_labels + per-label
add/remove. add-when-already-present → no-op; remove-when-absent
→ no-op. Per-label failure isolated. get_labels failure marks
all requested actions failed (caller retries).
19 new tests:
- fingerprint helpers (5: deterministic, distinct inputs, marker
format, body match, empty body no-match)
- post_status_comment (5: new post, duplicate skip, list-failure
fall-through, post failure, distinct event_kinds get distinct fps)
- adjust_labels (9: empty no-op, add-when-absent, add no-op,
remove-when-present, remove no-op, get failure, per-label
isolation with mixed failures, returning-False failure)
Total: 329 controller tests; full auto_agents suite 2691 pass.
Phase 1d-3c-3 (MERGING handler) + Phase 1c-3 (real OpenCode + MCP
spawn) remain.
|
||
|
drew
|
e8045d4b9b |
feat(controller): Phase 1d-3c-1 — discovery sweep + worker test deflake
Discovery polls Forgejo for open PRs/issues and inserts a fresh DISCOVERED workflow for any entity the controller hasn't seen yet. Same callback-injection pattern as the scheduler's prefetch: tests provide synthetic ListPRsCallback / ListIssuesCallback; production wires them to the existing _review_fetch helpers in a follow-up. tools/controller/master/discovery.py: - run_discovery(engine, owner, repo, list_prs, list_issues=None) — idempotent. Skips existing entities via a one-shot SELECT(kind, entity_number) WHERE owner+repo membership check. Inserts a DISCOVERED workflow + a controller_events 'discovered' row per new entity. Per-callback try/except: a PR-list failure doesn't block issue discovery, and vice versa. - Defensive coerce_int: rejects bools (True is an int subclass — bug class to avoid). Accepts string digits. - Returns DiscoveryReport(prs_seen, issues_seen, new_workflows, existing_skipped, new_entities[]). 12 new tests: - basics (empty, PRs only, issues only, both PRs+issues) - idempotency (2nd sweep skips; PR #42 + issue #42 coexist; other (owner, repo) isolated) - malformed input (non-int + bool skipped; string digit accepted) - callback failures (PR raises → still process issues; issues callback optional) - event row creation per new workflow Plus a worker test deflake: test_stolen_lock_between_agent_return_and_write was asserting the specific 'lost-lock-at-write' outcome, but under full-suite CPU contention the heartbeat thread can fire between the agent's return and _write_outcome — taking the 'lost-lock' (via lost_lock_event) path instead. Both are valid for this scenario; loosened the assertion to accept either. Total: 310 controller tests; full auto_agents suite 2672 pass. |
||
|
drew
|
337af855f3 |
feat(controller): Phase 1d-3b — per-workflow scheduler + static escalation
When the state machine transitions a workflow into a state that needs a worker (ANALYZING / IMPLEMENTING / REVIEWING / CONFLICT_RESOLVING), the scheduler enqueues a fresh pending workflow_attempts row with the input_payload prefetched. tools/controller/master/scheduler.py: - schedule_next_attempts(engine, prefetch=...) — finds workflows in worker-needing states without a matching pending/in_progress attempt; enqueues one fresh attempt per. Skips: - workflows already with pending/in_progress attempt for the role - ESCALATING at MAX_TIER → resolved to ABANDONED (no enqueue) - prefetch raised → per-workflow isolated failure - Static escalation policy resolved inline: ESCALATING + current_tier → IMPLEMENTING(min(tier+1, MAX_TIER)) OR ABANDONED. Both produce controller_events transition rows with the appropriate v9 event name (escalate_next_tier_available / escalate_max_tier_exhausted). - State→role table: ANALYZING → estimator, IMPLEMENTING → implementer, REVIEWING → reviewer, CONFLICT_RESOLVING → conflict_resolver. - PrefetchCallback is parameterized; production wires it to the Forgejo prefetch (Phase 1d-3c), tests inject a fake. - attempt_number monotonically increments via COALESCE(MAX, 0) + 1. - DEFERRED to Phase 1d-3c: actual Forgejo prefetch implementation (this commit ships the scheduler skeleton + the prefetch callback contract). 20 new tests: - state→role parametrization (4 states × matching role) - no-double-enqueue (3 cases: pending blocks, in_progress blocks, different-role doesn't block) - escalation (4 cases: tier 0→1, 1→2, MAX→ABANDONED, event row content) - prefetch raises → per-workflow skip - non-schedulable states parametrized (7 cases: DISCOVERED, AWAITING_CI, MERGING, MERGED, ABANDONED, STUCK, CREATED_PR) - attempt_number monotonicity (uses MAX+1) Total: 298 controller tests; full auto_agents suite 2660 pass. |
||
|
drew
|
130bbbf3c5 |
feat(controller): Phase 1d-3a — master main loop (composite tick)
Long-running master orchestrator composing the deterministic per-iteration work shipped in Phase 1d-1/1d-2. One iteration = state-machine tick + reaper + pickup guard, in that order (reasoning in module docstring). tools/controller/master/loop.py: - MasterConfig: tick_interval_s (default 30s), reaper_interval_s (default 60s), pickup_guard_max_pickups (default 3 per v6). - run_master_iteration(engine): single synchronous iteration — composable for tests + master_main_loop. - master_main_loop(engine): runs run_master_iteration on a loop until stop_event fires. Reaper runs less often than tick (every reaper_interval_s, not every tick_interval_s). on_iteration callback gets each MasterTickReport (used by tests + future structured logging). 7 new tests: - run_master_iteration: empty DB no-op; tick advances state (blocked → STUCK); reaper + pickup guard compose (stale in_progress at pickup limit → reaped → STUCK in one iteration). - master_main_loop: runs until stop; empty loop exits cleanly; on_iteration exception doesn't break loop; reaper runs less frequently than tick. Phase 1d-3+ remains for: discovery (Forgejo poll), per-workflow scheduling (enqueue next workflow_attempts after transition), Forgejo writes, MERGING state's merge call, reconciliation, backfill, operator CLI. Total: 278 controller tests; full auto_agents suite 2640 pass. |
||
|
drew
|
b519ebd98f |
feat(controller): Phase 1d-2 — outcome mapper + master tick handler
The master-side state machine driver. When a worker writes a complete
(or failed) attempt, the tick handler picks it up next pass, maps the
outcome to a state machine event, and applies the transition.
tools/controller/master/:
- outcomes.py: map_outcome_to_event() — pure function. Inputs:
role, current_state, output_payload, status, head_sha_advanced,
attempts_remaining_at_tier, conflict_count_at_current_tier.
Returns EventMapResult(event_name|None, reason).
- Implementer: resolved+pushed → implementer_pushed;
resolved+NO push → implementer_competence_failure (worker lied);
rebase-failed / competence-failure / blocked / noop all routed.
- Reviewer: approve → reviewer_approve; request-changes → retry vs
escalate based on attempts_remaining_at_tier; abstain →
reviewer_abstain; comment → no transition (operator review needed).
- Estimator: is_metadata_only → estimator_metadata_only; else
estimator_done.
- Conflict resolver: counts at current tier — 1st → first; 2nd →
second_same_tier (escalate); 3rd+ → three_plus (STUCK).
- Summarizer: doesn't drive state transitions.
- status='failed' policy: contract-violation → pickup_exhausted;
other failed outcomes (worker-internal-error / stale-input /
ttl-insufficient-for-retry / git-clone-failed / lost-lock / etc.)
→ no event (master re-enqueues per the v9 table).
- status='reaped' → no event (pickup guard handled).
- tick.py: run_tick() — one master tick. Queries
workflow_attempts WHERE status IN ('complete', 'failed') AND
finished_at > workflow.last_transition_at (heuristic for "not yet
processed"). Per row:
1. Validate current_state ∈ KNOWN_STATES (per v6 unknown-state
guard); if not, transition workflow → STUCK with reason='unknown-state'.
2. Map outcome → event via outcomes.map_outcome_to_event.
3. If no event, bump workflow.last_transition_at so we don't
re-process forever.
4. Apply event via state_machine.apply_event; IllegalTransition →
STUCK with reason='illegal-event'.
5. Commit transition (workflows.current_state + entered_state_at +
last_transition_at) AND insert controller_events row.
Returns TickReport with attempts_processed + transitions_applied +
transitions_log.
42 new tests:
- outcomes (22): status='failed' policy (3 paths) + each role's
happy paths + edge cases (unknown outcome / missing field /
None payload / unknown role).
- tick (20): implementer transitions (resolved+push → AWAITING_CI,
resolved-no-push → ESCALATING, rebase → CONFLICT_RESOLVING,
blocked → STUCK), reviewer transitions (approve → MERGING,
request-changes → IMPLEMENTING), event row content, no
reprocessing on second tick, unmapped attempts bump
last_transition_at, terminal workflows skipped (4 parametrized),
unknown current_state → STUCK.
Total: 271 controller tests; full auto_agents suite 2633 pass.
|
||
|
drew
|
976817fa22 |
feat(controller): Phase 1d-1 — state machine + reaper + pickup guard
The deterministic spine of the master controller. State machine is
pure data with 6 load-bearing invariants enforced via property tests.
Reaper resets stale-heartbeat workflow_attempts to pending. Pickup
guard transitions workflows to STUCK when an attempt has been
re-pended too many times without success.
tools/controller/state_machine.py:
- KNOWN_STATES = 12; TERMINAL_STATES = {MERGED, ABANDONED, STUCK,
CREATED_PR}. STUCK's only allowed exit is the operator-driven
operator_unstick event (back to DISCOVERED).
- 32 TRANSITIONS entries covering DISCOVERED → ANALYZING →
IMPLEMENTING ↔ AWAITING_CI / CONFLICT_RESOLVING / ESCALATING →
REVIEWING → MERGING → MERGED. Plus pickup_exhausted exits from
IMPLEMENTING/CONFLICT_RESOLVING/REVIEWING.
- 27 named events with descriptions. apply_event() lookup raises
IllegalTransitionError (lists legal events from current state)
or ValueError on unknown state (per v6 unknown-state guard).
- 6 LOAD-BEARING invariants for v1 (per v9 simplification):
1. no_path_implementing_to_reviewing_skips_ci (Hard Rule #1
constructional fix for the no-mans-land race)
2. terminal_states_have_no_exits (only STUCK→operator_unstick OK)
3. tier_monotonic_non_decreasing
4. every_pr_workflow_includes_reviewing
5. conflict_resolving_bounded (1st→IMPLEMENTING, 2nd→ESCALATING,
3rd→STUCK; structurally encoded)
6. escalation_deterministic
- reachable_from() honors cycles (DISCOVERED ∈ reachable(DISCOVERED)
via STUCK→operator_unstick path; AWAITING_CI self-loops via
ci_flake_retry).
tools/controller/reaper.py:
- reap_stale_attempts(): SELECT in_progress attempts whose
lock_heartbeat_at + lock_ttl_seconds < NOW (per-row TTL respects
per-role differences — estimator 180s, reviewer 720s, tier-2
implementer 2160s). UPDATEs status='pending', clears lock columns,
preserves pickup_count (the pickup guard handles that). Inserts
controller_events row with reason='lock-ttl-expired' per reap.
- Dialect-portable: Postgres uses interval arithmetic; SQLite uses
julianday(). Same logic either way.
tools/controller/pickup_guard.py:
- transition_exhausted_to_stuck(): finds attempts with status='pending'
AND pickup_count >= MAX_PICKUPS (default 3 per v6 blocker fix)
AND workflow not already terminal. Transitions workflow → STUCK,
marks attempt as 'reaped', inserts controller_events with
reason='attempt-pickup-exhausted' + pickup_count + max_pickups.
45 new tests:
- state_machine: basic shape (states partition, every transition uses
known states + defined events), apply_event success/error paths,
events_from + reachable_from helpers (including cycle awareness),
per-invariant zero-violations against the live table, per-invariant
monkeypatch-violations to prove the checks catch the bug class they
claim to, parametrised sanity check "every non-terminal can reach
some terminal".
- reaper: empty DB / fresh heartbeat / stale heartbeat reaped /
per-row TTL respected / event row created / only-in-progress
reaped / multiple stale attempts.
- pickup guard: empty DB / below limit / at limit / in-progress not
checked / terminal workflow skipped / event payload content /
default max_pickups matches v6.
Total: 229 controller tests; full auto_agents suite 2591 pass.
|
||
|
drew
|
30dfd92021 |
feat(controller): Phase 1c-2 — workspace + session sidecar + orphan janitor
Per-PR workspace umbrella manager (per plan v6) + the worker.session
sidecar that tracks live OpenCode session metadata for orphan cleanup
+ the startup janitor that sweeps orphaned workspaces from previous
worker crashes.
tools/controller/worker/:
- workspace.py: PerPRWorkspace + WorkspaceIdentity (frozen dataclass
yielding the canonical pr-{owner}-{repo}-{N} dir name). ensure_present
is idempotent. clone_if_absent runs `git clone --no-single-branch`
if .git/ missing; idempotent on re-call. fetch_and_validate runs
`git fetch origin --prune`, compares origin/<ref> to expected
head_sha, raises StaleInputError on mismatch (v6 stale-input fix),
then `git reset --hard <expected_sha>` + `git clean -fdx` to wipe
worktree residue from prior attempts. remove() rm -rf's the
workspace.
- session_sidecar.py: WorkerSession frozen dataclass +
write_sidecar (atomic via tmp+rename+fsync) + read_sidecar (tolerates
missing/empty/malformed/wrong-shape gracefully). Sidecar captures
opencode_server_url / session_id / subprocess_pid /
spawned_by_controller_pid / instance_id / spawned_at.
- janitor.py: sweep_orphans pre-queries the DB once for the set of
live instance_ids (workflow_attempts.status='in_progress'), then
scans workspace_root for pr-* dirs. For each:
- no sidecar → just delete (crashed pre-spawn)
- sidecar's instance_id is in live set → preserve (active worker)
- else → orphan. Try cancel_callback (production wires to OpenCode
cancel API); fall back to SIGTERM/grace/SIGKILL on the
subprocess_pid. Then delete sidecar + workspace. JanitorReport
summarises each sweep for structured logging.
Key v6 design points implemented:
- worker.session sidecar atomicity → orphan detection is robust
against partial writes (e.g., worker crashed mid-spawn).
- Cancel-callback-then-SIGKILL fallback → production prefers the
graceful OpenCode-side cancel; tests inject a fake.
- _kill_with_grace returns True after SIGKILL delivery; zombie
reaping is the parent's responsibility, not the janitor's.
27 new tests:
- WorkspaceIdentity (format + frozen)
- PerPRWorkspace paths + ensure_present idempotency
- clone_if_absent (creates worktree, idempotent, raises without URL)
- fetch_and_validate (matching sha resets clean; mismatched raises
StaleInputError; unknown ref raises)
- Sidecar I/O round-trip + atomicity + missing/empty/malformed handling
- Janitor: empty root / no-sidecar / active-lock / orphan-with-dead-pid
/ orphan-with-live-pid / cancel-callback (3 sub-paths: ok / fails /
raises) / mixed-workspaces / non-pr-skip
Total: 184 controller tests; full auto_agents suite 2546 pass.
|
||
|
drew
|
eab476e48e |
feat(controller): Phase 1c — worker controller skeleton
The dequeue+lock+heartbeat+runner+loop machinery. Production
OpenCode + MCP invocation slots in via the agent_runner callable
(Phase 1c-2). This commit is the structural foundation:
tools/controller/worker/:
- identity.py: build_instance_id() → "{hostname}/{pid}/{worker_uuid}"
per plan v9 (slash delimiter; IPv6-safe; uuid4 prefix for
per-instance uniqueness).
- heartbeat.py: Heartbeat thread that updates lock_heartbeat_at
every interval (default 30s). v9 simplified: TTL-only (no activity
tracking). UPDATE … WHERE locked_by_instance=us; rowcount=0 →
lost_lock_event.set() and thread exits, letting reaper handle it.
- runner.py: run_one_attempt() drives one attempt end-to-end.
Starts heartbeat → invokes agent_runner → on success writes
status='complete' + output_payload; on WorkerError writes
status='failed' with outcome label; on WorkerLostLock or detected
stolen-lock-at-write returns aborted (no DB write — reaper has
already re-pended). Defense-in-depth: even if agent returns
successfully, lost_lock_event.is_set() check skips the write.
- loop.py: worker_main_loop() polls the DB for pending attempts up
to MAX_CONCURRENT_WORKERS_PER_MACHINE, submits each to a
ThreadPoolExecutor. Honors stop_event for graceful shutdown
(drains in-flight before exit).
tools/controller/db/session.py: StaticPool for in-memory SQLite so
the heartbeat thread + runner write + dequeue all see the same DB
(without this, ":memory:" gives each connection an independent DB).
16 new tests in test_worker.py: instance ID format/uniqueness;
heartbeat tick (hold + steal); runner happy path; 5 error paths
(worker error / unexpected exception / WorkerLostLock raised /
stolen lock at write / lost_lock_event set defense-in-depth); 4
loop scenarios (single attempt, role filter skip, empty queue
exit-on-stop, explicit instance_id).
Total: 157 controller tests; full auto_agents suite 2519 pass.
|
||
|
drew
|
36b133ec5e |
feat(controller): Phase 1b — DB schema + dequeue helper + payload guard
Five SQLAlchemy 2.0 declarative models implementing plan v6/v9's
unified workflow schema. Cross-dialect (SQLite for tests + local dev,
Postgres for multi-machine production). Lock columns on
workflow_attempts implement the multi-machine-safe dequeue protocol
(plan v5).
Modules:
- tools/controller/db/models.py:
- Workflow (kind discriminator pr/issue, unique on owner+repo+kind+
entity_number, parent_workflow_id FK for issue→PR linkage)
- WorkflowAttempt (status/locked_by_instance/locked_at/
lock_heartbeat_at/lock_ttl_seconds/pickup_count + CHECK
constraints on status enum and pickup_count≥0; partial indexes
on the pending/in_progress/complete hot paths)
- ControllerEvent (Forgejo-write replay support kept in-schema even
though v9 simplified to Forgejo-first protocol; allows v3-style
upgrade later without migration)
- FlakeHistory (composite PK; supports the v6 flake-learning
heuristic)
- CIObservation (raw CI state history; 90-day retention to be
enforced by a sweep task)
- AutoincrementPk variant (Integer on SQLite where it autoincrements
via rowid; BigInteger on Postgres for BIGSERIAL); JsonColumn
variant (JSON on SQLite, JSONB on Postgres)
- tools/controller/db/session.py: build_engine (per-dialect tuning —
SQLite WAL + foreign_keys + busy_timeout; Postgres pool_pre_ping);
create_all (idempotent); session_scope (transactional context).
- tools/controller/db/dequeue.py: dequeue_one (one row atomically;
Postgres path uses SELECT FOR UPDATE SKIP LOCKED, SQLite path uses
UPDATE-WHERE-id-IN-SELECT-LIMIT-1 with RETURNING). Bumps pickup_count
on dequeue; respects max_pickups guard (default 3 per v6 blocker fix).
Returns DequeueResult dataclass with role/tier/pickup_count and
reason on miss.
- tools/controller/db/payload_guard.py: enforce_input_payload_size
with 4MB cap and 5-step truncation priority (older_summary → oldest
verbatim → comments → full_diff → CI failure excerpts). Raises
PayloadTooLargeError after all steps exhausted; master maps to
workflow STUCK with reason='input-too-large'.
- pyproject.toml: new optional extras `controller-db` pinning
sqlalchemy + psycopg2-binary (latter installed only for prod
multi-machine deploy; tests use stdlib sqlite3 via SQLAlchemy's
SQLite dialect which is already pulled in transitively via alembic).
37 new tests across test_db_schema/dequeue/payload_guard; 141
controller tests total; full auto_agents suite 2503 pass (no
regressions).
|