a21466add232d59cdec1604e09d58ca05659a623
17 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 drew
|
a21466add2 |
feat(controller): Phase 4 metadata-hygiene + round-2 adversarial fixes
Five deterministic, idempotent Phase 4 checks: 1. completed_not_closed — close linked issues on MERGED 2. closing_keyword_fixup — add Closes #N to PR bodies 3. label_sync_from_issue — copy Priority/Type/MoSCoW labels 4. state_label_inference — sync State/* label to current_state 5. milestone_assignment — copy milestone from linked issue All default-off via CONTROLLER_METADATA_HYGIENE_ENABLED + per-check granular env flags. Dry-run mode shares the grooming CONTROLLER_GROOMING_DRY_RUN flag. Round-1 fixes (applied before this commit): - False-positive idempotency lock (executed=True on skip) - Unbounded MERGED scan → LEFT JOIN candidate query - Duplicate _classify_forgejo_status → import from forgejo_writes - Bare-ref regex too broad ([#42](url) misread) → add [ lookbehind - Wrong audit stage → 'metadata_hygiene' Round-2 adversarial fixes (3 architect, 4 principal, 7 test engineer): - completed_not_closed: executed=True only when ALL refs close; partial success writes executed=False so remaining issues retry - milestone_assignment: was calling get_pr_details (hits /pulls/, returns 404 for plain issues) → now uses get_issue_state (/issues/{n}) so milestone fetch works for all issue types - label_sync failure path: write executed=False audit row for observability; pre-fix left no audit trail for persistent failures - _BARE_REF_RE: add ( to lookbehind to exclude (#42) link destinations - state_label_inference: re-read current_state inside inner session to avoid stale-snapshot spurious label writes across session boundaries - _last_synced_state: add decision_id DESC tiebreaker for same-second wall-clock rows - dry-run completed_not_closed: separate early-return path to avoid inflating completed_not_closed_executed counter 71 tests (54 round-1 + 17 round-2): - TestCompletedNotClosedPartialSuccess (3) — partial/zero/full success - TestLabelSyncAdjustLabelsFailure (2) — failure audit + retry - TestStateLabelAdjustLabelsFailure (2) — no executed=1 on failure - TestStateLabelInferenceTerminalWorkflows (3) — MERGED/ABANDONED sync - TestLastSyncedStateDryRunThenReal (2) — dry-run → real-run - TestClosingKeywordFixupBareRefAlreadyCovered (2) — candidates subtraction - TestErrorPathHandlingRound2 (3) — label_sync + state_label errors Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
drew
|
4d969eaf2b |
feat(controller): Phase 3 — Gate 3 reviewer-abandon
When the reviewer finishes a review and judges the work fundamentally unworkable (implementation surfaced misdiagnosis, obsoleted-by-other-work, or irreducible complexity), it can now emit verdict='abstain' + suggested_next_action='abandon' with a Gate-3 abandon_reason_category. The controller routes REVIEWING → ABANDONED and (when the kill switch is on) performs the Forgejo close via the reviewer-abandon side-effect tick — no implementer/CI/merge cycles. Wired with the same defense-in-depth pattern Phase 2 established: MCP setter validation + outcomes mapper dispatch with confidence gating + Pydantic atomicity validator + side-effect tick with audit-trail attribution (cause=REVIEWER_ABANDON, event_type='reviewer_abandon'). Default-off CONTROLLER_GATE3_ABANDON_ENABLED kill switch so a fresh deploy is audit-only until the operator explicitly enables Forgejo writes. Bundled refactor: hoisted the 9 Gate-2 + 3 Gate-3-exclusive abandon categories into tools/controller/contracts/abandon_categories.py (triggered by Phase 3 per the plan's follow-up backlog). Both gates now consume the shared frozensets; doc-contract tests grep each agent prompt against the canonical list. Adversarial review (2 rounds): caught + fixed MCP cross-check ordering (atomicity FIRST so missing-setter shows actionable error), confidence=None symmetric downgrade across both gates, dead blocking-issues extraction in _run_close, idempotency clock-collision in the test, low-vs-missing reason-string conflation, and several test-quality gaps. 4064/4071 tests passing (7 pre-existing failures unrelated to Phase 3). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
a91df787d7 |
feat(controller): Phase 2 — Gate 2 estimator-abandon
Adds the second of three abandon gates: the estimator (Gate 2) can
mark a work item fundamentally unworkable, transitioning the
workflow ANALYZING -> ABANDONED and triggering a Forgejo close via
Phase 1's decomposed close_act orchestrator. Catches abandon cases
at the cheapest LLM stage, before implementer/reviewer tiers fire.
Substantive:
- EstimatorOutputV1: additive verdict + abandon_reason_category +
abandon_reason_detail fields (pre-Phase-2 outputs still parse).
@model_validator enforces abandon-requires-category atomicity at
parse time — third defense layer beyond MCP setter + outcomes
mapper
- state_machine: estimator_abandon event + (ANALYZING,
estimator_abandon) -> ABANDONED. 57 transitions; invariants clean
- mcp/estimator_builder: estimator_set_verdict setter validates
verdict enum + 9-category whitelist (scope_intractable,
intent_wrong, security_regression, deprecated_dependency,
breaks_protected_invariants, out_of_scope, low_value,
unmaintained_path, policy_violation) + cross-field rules
- outcomes._map_estimator_outcome: dispatch verdict='abandon'
-> estimator_abandon, with confidence-low downgrade to
estimator_done (honors the agent prompt's documented "high or
medium" requirement)
- estimator_abandon_side_effects.py: per-state side-effect tick
modeled on grooming_side_effects.py; invokes close_act with
cause=Cause.ESTIMATOR_ABANDON + event_type='estimator_abandon'
- _events.py: shared latest_transition_event +
workflows_with_latest_transition_in helpers; dialect-aware
payload['event'] extraction (SQLite json_extract +
PostgreSQL ->>); centralizes the event_type='transition' +
payload['event'] convention that side-effect ticks consume
- gate2_abandon_config.py: CONTROLLER_GATE2_ABANDON_ENABLED kill
switch (default false). Fresh Phase 2 deploys are audit-only
until operator explicitly enables; dry_run shared with grooming
for unified safe-rollout staging
- .opencode/agents/estimator-implementation.md: GATE 2 ABANDON
section with 9-category criteria + low_value disqualifier ("PR
cites an issue/ticket -> route to reviewer instead")
Round-2 adversarial-review fixes (all required pre-commit):
- forgejo_writes.close_issue / close_act: NEW cause + event_type
kwargs (defaults preserve Phase 1 grooming behavior; Phase 2
callsite overrides). Fixes audit-trail attribution: telemetry
queries SELECT WHERE cause='estimator_abandon' now return the
right rows. Phase 1 regression test pins the grooming defaults
- tick.py operator_unstick lookback: dialect-aware json_extract
fix (Phase 1 carry-over bug; would silently no-op on PostgreSQL)
- grooming_side_effects.py: idempotency filter now keys on
check_name set (grooming check_names only) so a Phase 1 close
and a Phase 2 close on the same workflow don't cross-cancel
Tests (+50): TestEstimatorOutputV1Phase2,
TestEstimatorAbandonStateMachine, TestMapEstimatorOutcomePhase2
(including confidence-low downgrade), TestEstimatorSetVerdict
(all 9 categories + cross-field rules), TestEventsHelper,
TestEstimatorAbandonSideEffectTick (including
test_close_writes_estimator_abandon_cause_and_event_type pinning
the audit-trail attribution, and Phase 1 regression guard).
Doc-contract test asserts all 9 categories appear in the agent
prompt. 1509/1509 passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
016b348117 |
feat(controller): grooming gate (Phase 0 + Phase 1 worker-shape dispatch)
Phase 0 (foundation):
- Cause enum (controller_events.cause) for action attribution
- Schema: grooming_decisions audit table; workflows gains
grooming_evaluated_at + deferred_reason + deferred_at +
deferred_target_workflow_id; pulls gains touched_files
- audit_comments: CLOSE / DEFER templates + render_comment_template
- forgejo_writes: close_issue + defer_issue 5-step crash-safe protocol
(fingerprint dedup, error matrix, dry-run)
- patch_pr_state callback in forgejo_http
- grooming_config: 22-env-var frozen-dataclass config + log_effective
- pulls.touched_files cache extension (_pipeline_cache.py schema v8)
- reaper.reap_grooming_decisions audit-retention sweep
- reconciliation RESUME guard (deferred_reason)
Phase 1 (worker-queue shape, 2026-05-25):
- New state: GROOMING. New events: grooming_started, groom_verdict_
{proceed,defer,close}. 5 new transitions; all invariants still clean
- GroomingInputV1 + GroomingOutputV1 Pydantic contracts
- outcomes._map_grooming_outcome routes verdicts to state-machine events
- prefetch.build_grooming_stage_b_input + list_open_prs callback
- scheduler GROOMING -> grooming_stage_b role
- promote: cfg-gated DISCOVERED -> GROOMING when CONTROLLER_GROOMING_
ENABLED=true; issues skip grooming
- forgejo_writes decomposed: close_act/defer_act (Forgejo writes only;
state-machine already transitioned) + close_decide_and_act/
defer_decide_and_act (Phase 0 callers); _apply_workflow_transition
is underscore-private
- grooming.py library: tokenization, suspicion scoring (Jaccard +
weighted overlap), deterministic checks, action -> verdict mapping
- mcp/grooming_builder.py: 14-tool FastMCP server emits GroomingOutputV1
- .opencode/agents/grooming-stage-b.md: duplicate-detection agent
prompt (claude-haiku-4-5)
- grooming_side_effects.run_grooming_side_effects_tick: per-state tick
performs Forgejo writes after groom_verdict_{defer,close} fires.
Filters on event_type='transition' + payload.event (centralizes the
convention pending Phase 2's latest_transition_event helper)
- GroomingCallbacks frozen dataclass; loop.py + __main__.py wired
Worker role registry (single source of truth):
- worker/roles.py: WORKER_ROLES + WorkerRoleSpec + default_roles_csv
+ output_filename_for. agent_runner.ROLE_TO_MCP_MODULE / ROLE_TO_
OUTPUT_MODEL derive from it; opencode_session.agent_name_for reads
it for flat cases; all 6 prompt builders use output_filename_for;
worker --roles default = default_roles_csv(); launcher script
derives --roles via shell substitution. Cross-site invariant test
enforces alignment across 5 sites + opencode.json MCP registry.
Phase 0 silent-bug fix:
- reconciliation.py RESUME guard SELECT now includes deferred_reason
(was missing since Phase 0; guard was a silent no-op). Tightened
from getattr to attribute access to fail fast on future omissions.
Tests (1456 total, +91 grooming-specific):
- test_grooming_phase0.py: 34 tests (orchestrator matrix, crash
recovery, idempotency, dry-run)
- test_grooming_phase1.py: 60 tests (library, contracts, state
machine, outcomes, scheduler, promote, prefetch, act-variants
with signature parity, side-effect tick incl. natural-idempotency
+ executed-flag-skip + verdict-mismatch + reconciliation RESUME)
- test_mcp_builders.py TestGroomingBuilder: 29 tests (happy paths
+ 22 validation rules + Pydantic round-trip + master-tick-read-
path companion)
- test_worker_agent_runner.py TestRoleMaps: cross-role wiring
alignment + agent-prompt-vs-worker-fallback filename contract +
inspect.signature equality (close_act/defer_act vs
close_issue/defer_issue)
- test_state_machine.py: transition count 51 -> 56 +
events_from_grooming
Live-validated end-to-end on 4 staged sentinel PRs (#55-#58) in
dry_run: agent emits verdicts via MCP, state-machine transitions
fire, side-effect tick writes audit row, deferred_reason gates
reconciliation RESUME correctly.
Deferred refinements + Phase 2 prerequisite (latest_transition_event
helper) tracked in .drew/regressions-plan.md "Phase 1 follow-up
backlog".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
14e592ddd5 |
feat(controller): zombie-CI detection — stop waiting on dead CI runs
A CI gate stuck `pending` is ambiguous: the job may genuinely be running, or the run may be dead (a crashed runner, an Actions job whose terminal commit-status was never posted — `CI / status-check` zombies routinely here). The "wait for the whole run to finish" fix then waited forever on the dead case (PR #36: a `status-check` gate pending for 8 h while the run had actually finished RED 8 h earlier). New `ci_run_status.classify_ci_run` resolves a still-pending run to `complete` / `running` / `stale` via two checks, authoritative-first: 1. ACTIVE-RUN — `get_action_tasks` asks Forgejo's Actions API directly whether a task for the commit is still running; catches a dead run immediately, regardless of age. 2. AGE — if no gate has updated in > CONTROLLER_CI_STALE_AFTER_MIN (default 90) the run has stopped; the fallback when the Actions API is unavailable. A `stale` run is no longer waited on: the verdict is taken from the gates that DID finish (`terminal_verdict`) — any failure → red, all pass → green, fully-dead → red. Applied in both `ci_status_poll` (the AWAITING_CI verdict) and `ci_summarize` (the implementer's summary — zombie pending gates drop out of `gates_pending`/`overall_state`) so the two agree and never ping-pong. New `get_action_tasks` Forgejo callback wired through forgejo_http → __main__ → the poll and the prefetch path. 23 new tests; full controller suite (1222) green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
9d20f865d3 |
feat(controller): pre-review mergeable gate — skip doomed reviews
When CI goes green, route to REVIEWING only if the PR still merges cleanly into base. If a cheap Forgejo mergeable check shows the base advanced while CI ran, route AWAITING_CI → CONFLICT_RESOLVING instead — handing the conflict to the LLM conflict_resolver BEFORE the expensive reviewer pass, since code that must be rebased gets re-CI'd and re-reviewed afterwards anyway. - new event `pre_review_base_conflict` + transition (AWAITING_CI → CONFLICT_RESOLVING) - ci_status_poll: `_decide_green_event` gate behind a new optional `get_pr_details` callback; conservative — only an explicit mergeable=false diverts, an unknown/uncomputed bit falls through to ci_green so a fresh PR is never false-routed - gate-fired event rows carry `mergeable` in the payload so the false-positive rate is observable from controller_events - wired through loop.py (5th ci_status_poll_args element) + __main__ One API call, no LLM — cheap+frequent detection gating the expensive+rare conflict_resolver/CI/reviewer stages. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
0db0a15dad |
feat(controller): RUN_CI_LOCAL verdict source, ci-not-ready outcome, escalation hardening
Adds RUN_CI_LOCAL — an on-demand local-CI verdict source for when the cluster's Forgejo CI is broken — plus robustness fixes, the telemetry Live-tab rewrite, and PR-level cost attribution. Controller: - RUN_CI_LOCAL: the master swaps its Forgejo CI callbacks for local `forgejo-runner exec` runs (tools/run-ci-full-local.sh + local_ci.py). Async per-(owner,repo,SHA) on-disk job cache; preflights the forgejo-runner binary + Docker daemon at startup (fail loud, not a red verdict on every PR); GCs finished run dirs + per-run actcache. - ci-not-ready implementer outcome + implementer_ci_not_ready event: an implementer that runs before the on-demand verdict exists parks in AWAITING_CI instead of dead-ending at STUCK; capped against ci_red ping-pong. - ci_poll_exhaustion skips its sweep while a local CI run is in flight, so AWAITING_CI workflows queued behind on-demand CI are not STUCK'd by the remote-CI-sized timeout. - Escalate the workflow after repeated worker-internal-error at a tier instead of retrying to pickup-exhaustion -> STUCK. - forgejo_http: normalise Forgejo's per-gate `status` key to `state` so failing gates are actually counted (they previously all read as pending). - Per-tier worker timeouts bumped +15 min; a timed-out attempt's dirty-worktree residue is preserved on auto-scratch/pr-<N> before the next attempt's reset. - Implementer agents now verify only the CI-flagged gate(s) via a targeted re-run rather than the full local battery before claiming resolved/noop. Re-running the whole suite CI will run anyway was the #1 cause of implementer timeouts; CI remains the real gate and re-dispatches the implementer on red. Telemetry: - Live tab rebuilt on /api/live (controller DB run state + the live OpenCode session forest) after the live_log_writer sidecar was retired with the legacy dispatchers. - Durable per-attempt input/output payloads surfaced in the Live drill-down, archived-session detail, and Workflows timeline. - PR-level cost attribution: worker session tags carry -pr-<n>; backfill_llm_activity_pr.py repairs rows written before the fix. Shared: - tools/controller/session_tag.py — one canonical controller-tag parser shared by the telemetry server and the backfill. Tests: new coverage for local_ci (state machine, log parsing, _summarize_run, GC, in-flight probe, preflight), the ci-not-ready path, the escalation/ci-not-ready SQL counters, the ci_poll in-flight skip, and CI-status payload parsing across both sources. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
0bc734c020 |
style: ruff format the controller-state-machine branch (288 files)
Applies `ruff format` to the accumulated formatting debt on this branch. Formatting-only — no behavioral changes. Required for CI/lint's format gate (`nox -s format -- --check`), which the branch was failing on 288 tracked files that drifted from ruff's canonical style. In-progress WIP files are intentionally excluded so this commit stays a clean formatting-only diff. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
a2fcecb28d |
feat(controller): CI-freshness gate — re-trigger stale/infra CI instead of dead-ending
A discovered PR with stale CI (every job failed at the checkout step —
a git-fetch connection reset, pure infra, no code ran) burned an
estimator + a tier-2 implementer and dead-ended at STUCK. The pipeline
had no notion of CI freshness and never triggered CI — only polled.
New early master tick (ci_gate) runs before DISCOVERED->ANALYZING
promotion. For each DISCOVERED pr-kind workflow it classifies the CI
via ci_freshness.classify_ci_result:
- infra_broken — failed; the failing jobs' LOG content carries a
checkout/setup signature (curl 56, expected 'packfile', ...).
Logs are fetched via _ci_logs (session-cookie auth).
- stale — failed; newest status older than CONTROLLER_CI_MAX_AGE_S
(default 6h). Timestamp-based, log-independent — catches an old
failure even when Forgejo has purged its logs.
- no_ci / pending / fresh_real — handled accordingly.
infra_broken/stale/no_ci -> push an empty commit to the PR branch
(Forgejo 15.0.2 has no rerun API), routing DISCOVERED -> AWAITING_CI
(new event discovery_ci_rerun_triggered). The existing AWAITING_CI
poller then gets a real verdict. A reran CI that is ALSO infra/stale
routes AWAITING_CI -> DISCOVERED (new event ci_infra_recheck) so the
gate re-handles it; bounded by a rerun budget of 3, then STUCK.
Also wires the existing CI summarizer into prefetch so workers stop
receiving ci_summary=null.
1095 controller tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
drew
|
3ca794be75 |
feat(controller): autonomous CI status polling — closes the last trial gap
The Phase 2 trial previously required operator-intervention SQL to
advance workflows from AWAITING_CI → REVIEWING (no automated CI
status polling). This commit wires the missing tick so the trial
runs end-to-end without manual help.
Components:
- ``master/forgejo_http.py``: new ``get_ci_status`` callback wraps
Forgejo's ``/commits/{sha}/status`` combined-status endpoint;
added to ``ForgejoCallbacks``.
- ``master/ci_status_poll.py`` (NEW): ``run_ci_status_poll_tick``
scans AWAITING_CI workflows, fetches CI status keyed on the
latest implementer attempt's ``head_sha_after``, and applies
state transitions via ``apply_event``. TOCTOU-defended UPDATE
(``WHERE current_state='AWAITING_CI'``) + per-row exception
isolation.
- ``master/loop.py``: new ``ci_status_poll_args=(owner, repo,
get_ci_status)`` kwarg + ``ci_status_poll_interval_s`` config
(default 60s) + ``MasterTickReport.ci_status_poll`` field.
- ``master/__main__.py``: threads ``callbacks.get_ci_status`` into
the loop.
State mapping (Forgejo combined-status state → event):
- success / neutral / skipped / warning → ci_green → REVIEWING
- failure / error / cancelled / timed_out / stale →
ci_red_retry_same_tier → IMPLEMENTING
- pending / queued / in_progress / action_required → no-op (wait)
- None / unknown / fetch failure → no-op (transient)
The ``ci_polling_exhausted`` timeout (default 2h) remains as the
safety net for CI that genuinely never reports.
Tests (+14 in test_master_ci_status_poll.py):
- Happy paths (success→green, failure→red, pending→wait)
- Error paths (callback raises; workflow without head_sha)
- Event row shape (event_type='ci-green'/'ci-red', reason payload)
- Extended state mapping (cancelled, neutral, in_progress)
- Other-repo isolation
- LoopIntegration end-to-end via master_main_loop with safety timer
RUNBOOK updated: removed the manual SQL workaround; added the
autonomous CI poll's tunables.
Total: 726 controller tests pass (+14 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
f57d9f9478 |
fix(controller): batch L — round-4 trial-blockers (A1, P1, P3, P4, P5, T5)
Round-4 adversarial review found 5 trial-blockers + 1 silent-debt
item the post-round-3 deep pass missed. All fixed.
A1 — pre-clone the workspace so the agent has a worktree to operate on
``worker/__main__.py``: the agent_runner closure now constructs a
``PerPRWorkspace`` from input_payload.owner/repo/pr_number + the
FORGEJO_URL+FORGEJO_TOKEN env vars. Pre-flight:
- ``workspace.ensure_present()`` creates the dir skeleton.
- ``workspace.clone_if_absent()`` clones the repo into
``{workspace_dir}/worktree/`` if not already present (idempotent).
- ``workspace.fetch_and_validate(head_sha, head_ref)`` refreshes +
verifies the workspace is at the expected head. ``StaleInputError``
→ ``WorkerError(outcome='stale-input')`` so the master re-prefetches
without burning a pickup. ``RuntimeError`` → ``worker-internal-error``.
Previously the agent saw an empty workspace_dir + had no repo.
P1 — partial-write defense in the canonical-output poller
``worker/agent_runner.py:_wait_for_canonical_output`` now polls each
path with a two-pass quiescence check (size stable + content parses
as JSON) before returning. Partial writes (agent crashed mid-flush)
are skipped + the polling loop continues. The previous
``f.read().strip()`` returned partial JSON which then tripped
``ContractValidationError`` → ``worker-internal-error`` with no
record of WHICH path; now logs source path on every read.
P3 — TOCTOU defense in promote_discovered
``master/promote.py``: the UPDATE now filters
``current_state='DISCOVERED'``. If a concurrent reconciliation
moved the row off DISCOVERED between SELECT and UPDATE, rowcount=0
+ we skip the event-row write. No duplicate audit entry; no
overwriting a pause-by-label-removal.
P4 — explicit tuple-length validation in reconciliation_args + discovery_args
``master/loop.py``: previously a 6-tuple silently fell into the
``else`` 4-tuple unpack, raised ValueError("too many values"), got
swallowed by the per-iter ``except Exception``, and reconciliation
silently died forever. Now: ``elif n == 4`` + ``else: raise TypeError``.
The TypeError still hits the per-iter except (so the loop doesn't
crash) but ``logger.exception`` surfaces the actionable message in
journald. Operator sees "reconciliation_args must be a 4- or 5-tuple;
got length 6" instead of zero indication.
P5 — --tick-interval CLI flag preserves other config fields
``master/__main__.py``: replaced the manual ``MasterConfig(...)``
rebuild (which dropped reconciliation/ci_poll/discovery intervals)
with ``dataclasses.replace(cfg_loop, tick_interval_s=args.tick_interval)``.
Operators who pass --tick-interval no longer silently revert the
other intervals to defaults.
T5 — scheduler._commit_escalation uses safe_json_dumps
``master/scheduler.py``: the escalation event row's payload was the
only call site that bypassed safe_json_dumps. Now consistent — a
future contributor adding a datetime/Decimal field won't trip raw
json.dumps at runtime.
Tests (+4 net):
- ``test_worker_agent_runner.py::test_partial_write_not_read``: pins
P1 (truncated fallback file + valid MCP output → MCP wins).
- ``test_master_promote.py::test_toctou_state_change_between_select_and_update``:
pins P3 (steal state via monkey-patch → no double-promotion, no
extra event row).
- ``test_master_loop.py::test_reconciliation_args_wrong_length_logs_not_silent``:
pins P4 (6-tuple → logged error, not silent forever).
- ``test_entry_points.py::test_tick_interval_flag_preserves_other_cfg_fields``:
pins P5 (env-set non-default intervals survive --tick-interval).
Total: 711 controller tests pass (+4 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
251eeb21ff |
fix(controller): more pipeline run-blockers — merging tick, periodic discovery, worker create_all, systemd ordering
Continuing the round-3 deep-pass cleanup. Three more run-blockers + one robustness fix. RB5 — MERGING handler never invoked from master loop: ``run_merging_tick`` was exported by the master package but no caller fired it. Workflows that transition to MERGING (via reviewer approval) would sit there indefinitely with no Forgejo merge call. Fix: - ``master/loop.py`` accepts a ``merging_args=(owner, repo, merge_callback)`` kwarg. When set, the tick fires every iteration (cheap if no workflows in MERGING). - ``MasterTickReport`` gains ``merging: MergingHandlerReport | None``. - ``master/__main__.py`` wires it from the Forgejo callback bundle. RB6 — periodic discovery never fires: ``run_discovery`` was only called at startup via ``run_startup_backfill`` + the ``--discovery-only-once`` smoke flag. PRs created after master startup would not be discovered until the master restarted. Fix: - ``master/loop.py`` accepts ``discovery_args=(owner, repo, list_prs, list_issues)`` or the 5-tuple with kwargs. Periodic tick on its own cadence (``CONTROLLER_DISCOVERY_INTERVAL_S``, default 30s). - ``MasterTickReport`` gains ``discovery: DiscoveryReport | None``. - ``master/__main__.py`` wires it + threads ``require_opt_in_label`` through. RB-robust — worker calls create_all defensively: Master is normally responsible for schema creation (workers run After= it via systemd ordering). But if the worker is started in isolation (test / local dev / unit ordering broken), it'd crash on the first query against missing tables. Fix: - ``worker/__main__.py`` calls ``create_all(engine)`` after ``build_engine``. ``create_all`` is idempotent (CREATE TABLE IF NOT EXISTS); safe to call from both master + worker. - ``cleveragents-controller-worker@.service`` adds ``After=cleveragents-controller-master.service`` + ``Wants=cleveragents-controller-master.service`` so systemd enforces the start ordering in production. Total: 703 controller tests pass (no test changes; all new wiring is exercised by master_main_loop tests via the new kwargs). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
febb352618 |
fix(controller): pipeline run-blockers — promoter, scheduler, owner/repo, workspace_dir patch
Round-3 deep pass identified four issues that would have prevented an
actual end-to-end pipeline run:
RB1 — DISCOVERED → ANALYZING never fired in production:
The state machine defines (DISCOVERED, discovery_picked_up) →
ANALYZING but NO production code fires the event. Workflows
created by discovery would sit in DISCOVERED forever.
Fix:
- New ``master/promote.py``: ``run_promote_discovered_tick`` scans
for DISCOVERED workflows + fires ``discovery_picked_up`` via
apply_event (state-machine invariants stay enforced) + emits a
``discovery-promoted`` controller_events row per transition.
- Composes with the master loop's other ticks; runs every iteration
(cheap — typically 0-1 row).
RB2 — scheduler.schedule_next_attempts never called from master loop:
The scheduler was exported by the master package but never invoked.
It creates the ``workflow_attempts`` rows that workers dequeue —
without it, workers would have nothing to pick up.
Fix:
- ``master/loop.py`` now accepts a ``prefetch: PrefetchCallback``
kwarg. When provided, the loop runs promote_discovered + scheduler
every iteration after tick/reaper/reconciliation.
- ``MasterTickReport`` gains ``promote_discovered`` and ``scheduler``
optional fields so on_iteration callbacks see both.
- ``master/__main__.py`` builds a ``PrefetchDataCallbacks`` from the
Forgejo callback bundle and constructs the production
``make_prefetch_callback(engine, callbacks)`` — wires through to
the loop's new prefetch kwarg.
RB3 — owner / repo missing from V1 input contracts:
The implementer / reviewer / estimator / conflict-resolver V1 inputs
had pr_number but not owner/repo. The OpenCode agent would have
had no way to know which Forgejo repo to clone — it would have had
to derive owner/repo from process env, coupling the worker to a
single repo.
Fix:
- ``contracts/v1.py``: added ``owner: str`` and ``repo: str``
(min_length=1) to ImplementerInputV1, ReviewerInputV1,
EstimatorInputV1, ConflictResolverInputV1.
- ``master/prefetch.py``: builders populate owner/repo from the
Workflow row (already known at prefetch time).
- Existing test fixtures in ``test_contracts_v1.py`` updated.
RB4 — input_payload.workspace_dir placeholder reached the agent:
Prefetch wrote ``workspace_dir = "<worker-injected>"`` as a
placeholder; the worker never patched it before invoking the
OpenCode session. The prompt builder rendered the literal
placeholder string into the agent's prompt — the agent had no idea
where to clone.
Fix:
- ``worker/agent_runner.py``: patches input_payload.workspace_dir
with the real path immediately before calling run_opencode_session.
Uses a shallow copy so the caller's dict isn't side-effected.
- ``worker/__main__.py``: workspace_dir naming convention is now
``pr-{owner}-{repo}-{pr_number}`` (matches workspace.py's
PerPRWorkspace convention) so the janitor's pr-* glob + the
agent's expected workspace location agree. Falls back to
``pr-attempt-{N}`` for legacy input_payloads missing owner/repo.
Tests:
- ``test_master_promote.py`` (NEW, +7 tests):
- empty DB no-op
- single workflow promoted
- multiple promoted in one tick
- only DISCOVERED targeted (non-DISCOVERED untouched)
- controller_events row emitted with correct shape
- idempotent after first promotion
- LoopIntegration end-to-end: DISCOVERED → ANALYZING → pending
estimator attempt visible in workflow_attempts (pins the entire
previously-broken pipeline from discovery to enqueue)
Total: 703 controller tests pass (+7 net), 0 regressions.
Without these four fixes, the pipeline would have looked alive in
unit tests but produced zero work in a real deployment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
147e3403c1 |
fix(controller): batch G — show-stoppers from round-2 review (N1–N4)
Four items the round-2 adversarial review flagged as ship-blockers.
N1 — PID-reuse defense is now WIRED in production:
Round 1's batch D shipped ``subprocess_starttime`` in the sidecar +
janitor checks against it, but NO production code wrote sidecars.
The defense was unwired; tests passed against a code path that
production never invoked.
Fix:
- ``worker/agent_runner.py`` accepts ``workspace_dir`` and
``opencode_server_url`` kwargs. When ``workspace_dir`` is set, it
writes a sidecar (``{workspace_dir}/worker.session``) immediately
after MCP spawn capturing the real PID + starttime from
``/proc/{pid}/stat`` field 22. Removes it on attempt completion.
- ``worker/__main__.py`` builds the per-attempt workspace dir
(``{workspace_root}/pr-attempt-{N}/``) and threads it through the
agent_runner closure with the OpenCode URL. The naming convention
is picked up by the janitor's ``pr-*`` glob; when per-PR shared
workspaces ship (Phase 1k++ follow-up), it changes to
``pr-{owner}-{repo}-{N}``.
- Test: ``TestSidecarWiring`` (+2 tests) verifies the sidecar appears
during the attempt, carries the right PID + starttime + instance,
and is cleaned up post-attempt.
N2 — AWAITING_CI escape event firing is now WIRED in production:
Round 1's batch D shipped ``ci_polling_exhausted`` /
``ci_flake_retries_exhausted`` in TRANSITIONS, but NO production code
emitted them. Workflows could still hang in AWAITING_CI forever.
Fix:
- New ``master/ci_poll.py``: ``run_ci_poll_exhaustion_tick`` scans
workflows whose ``entered_state_at`` is older than
``CONTROLLER_AWAITING_CI_TIMEOUT_S`` (default 7200s) and fires
``ci_polling_exhausted`` via ``apply_event`` → STUCK + emits a
``ci_poll_exhausted`` controller_events row with the threshold
payload.
- ``master/loop.py`` integrates the new tick on its own cadence
(``ci_poll_exhaustion_interval_s`` env, default 300s). Composes
with the existing master loop. ``MasterTickReport`` gains
``ci_poll_exhaustion: CIPollExhaustionReport | None``.
- Tests: ``test_master_ci_poll.py`` (+7 tests) — happy path, fresh
workflow stays untouched, only AWAITING_CI is targeted (other
long-lived non-terminal states ignored), event row shape pinned,
default threshold matches the documented 2h, end-to-end loop
integration (master_main_loop drives the exhaustion +
workflow → STUCK without operator intervention).
- Dialect-portable SQL (Postgres interval, SQLite julianday).
- Handles SQLite returning TIMESTAMP as str from text() queries
(no .isoformat() on str).
N3 — externally-merged/closed PRs now win over label removal:
Round-1's PAUSE-on-label-removed shipped, but reconciliation
checked the label gate BEFORE checking merged/closed. Operators
removing the opt-in label on an already-merged PR would PAUSE the
workflow forever — never transitioning to MERGED.
Fix:
- ``master/reconciliation.py:_reconcile_one`` re-ordered:
1. Check terminal-state mappings (merged/closed) FIRST — apply
immediately if they fire.
2. THEN the opt-in label gate (pause/resume).
3. Fall through to "consistent" otherwise.
- Tests: ``test_externally_merged_takes_priority_over_label_removal``
+ ``test_externally_closed_takes_priority_over_label_removal``
pin the contract. Both seed an IMPLEMENTING workflow + Forgejo
reporting "merged/closed AND no opt-in label" → workflow
transitions to MERGED/ABANDONED (not PAUSED) + pre_pause_state
stays None.
N4 — graceful handling of empty env vars:
``int(os.environ.get("CONTROLLER_FORGEJO_REQUEST_TIMEOUT_S", "30"))``
crashes with non-actionable ``int('') ValueError`` if the operator
sets the env to empty/whitespace (common when sourcing a partially-
edited /etc/cleveragents/master.env file).
Fix:
- ``master/forgejo_cfg.py:_env_int(name, default)`` — empty or
whitespace-only values fall back to the documented default; only
non-numeric values still raise (with a clear message naming the
variable).
- Tests: ``test_empty_env_value_falls_back_to_default`` +
``test_whitespace_only_env_falls_back`` + updated
``test_malformed_env_raises_value_error`` to match the new
"not a valid integer" wording.
Total: 660 controller tests pass (+13 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
72c272504c |
feat(controller): Phase 1k — controller-managed opt-in label gate
Migration safety mechanism: the controller only manages PRs and
issues carrying a configurable opt-in label (default
``controller-managed``). Operators opt PRs in for parallel-run
trials, can pause management mid-flight by removing the label, and
gradually roll out without exposing the controller to PRs that
human reviewers are actively driving.
Components:
- tools/controller/master/label_gate.py — single source of truth for
the configured label name + pure predicates/filters over Forgejo
PR/issue dicts.
- ``opt_in_label_name()`` reads ``CONTROLLER_OPT_IN_LABEL`` env
(default 'controller-managed'); empty/whitespace falls back.
- ``has_opt_in_label(entity, name)`` defensively handles every
degenerate shape (non-dict entity, non-list labels, non-dict
label entries, missing name field).
- ``filter_by_opt_in_label`` / ``count_filtered`` for callers.
Wired through:
- discovery.run_discovery + backfill.run_startup_backfill +
reconciliation.run_reconciliation_tick each accept
``opt_in_label`` and ``require_opt_in_label`` kwargs.
- Function defaults are ``require_opt_in_label=False`` for API
back-compat (existing 30+ discovery/backfill/recon tests work
without changes).
- __main__.py defaults to ``--no-opt-in-label`` OFF (gate ENABLED in
production); add ``--no-opt-in-label`` to bypass.
- DiscoveryReport gains a ``label_filtered_out`` counter.
Reconciliation behavior:
- When opt_in_label is configured AND the Forgejo response carries a
``labels`` field AND the opt-in label is NOT present, the workflow
transitions to ABANDONED with reason ``opt-in-label-removed`` +
emits a controller_events 'reconciliation' row.
- Partial Forgejo responses (no ``labels`` field) skip the label
check — never ABANDON on incomplete data.
Master loop extension:
- ``reconciliation_args`` now accepts an optional 5th element — a
kwargs dict threaded through to ``run_reconciliation_tick``.
__main__.py uses this to pass ``require_opt_in_label`` per the CLI
flag. 4-tuple back-compat preserved.
Tests (+29 in test_label_gate.py, 0 regressions across 569 tests):
- Predicate edge cases (every degenerate shape returns False)
- Env-var resolution (default, override, empty, whitespace)
- filter/count helpers
- Discovery + backfill: kept/filtered counts, gate disabled,
explicit label overrides env
- Reconciliation: label removed → ABANDONED, label present →
no-op, partial response → no-op, gate disabled → bypass, event
row records reason
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
b76c5f05e7 |
feat(controller): wire reconciliation into master main loop
Composes reconciliation as the 4th tick layer at its own cadence. tools/controller/master/loop.py: - MasterConfig gains reconciliation_interval_s (default 300s per plan v9). - MasterTickReport gains reconciliation: ReconciliationReport | None. - master_main_loop gains reconciliation_args parameter — tuple of (owner, repo, get_pr_state_cb, get_issue_state_cb). When provided, runs run_reconciliation_tick every reconciliation_interval_s. When None, reconciliation is disabled (useful for tests + one-shot modes). - Reconciliation exception is caught + logged; master keeps running. - Iteration log line now includes reconciled=N. tools/controller/master/__main__.py: - Passes reconciliation_args from ForgejoCallbacks (built earlier in the entry point) so the production master automatically runs reconciliation against the configured (owner, repo). 3 new tests in test_master_loop.py: - reconciliation_fires_when_configured: workflow with externally- merged state → reconciliation transitions to MERGED. - reconciliation_skipped_when_args_none: workflows untouched + no reconciliation reports. - reconciliation_exception_doesnt_break_loop: per-row fetch failures don't crash the master. Total: 433 controller tests; full auto_agents suite 2795 pass. |
||
|
drew
|
130bbbf3c5 |
feat(controller): Phase 1d-3a — master main loop (composite tick)
Long-running master orchestrator composing the deterministic per-iteration work shipped in Phase 1d-1/1d-2. One iteration = state-machine tick + reaper + pickup guard, in that order (reasoning in module docstring). tools/controller/master/loop.py: - MasterConfig: tick_interval_s (default 30s), reaper_interval_s (default 60s), pickup_guard_max_pickups (default 3 per v6). - run_master_iteration(engine): single synchronous iteration — composable for tests + master_main_loop. - master_main_loop(engine): runs run_master_iteration on a loop until stop_event fires. Reaper runs less often than tick (every reaper_interval_s, not every tick_interval_s). on_iteration callback gets each MasterTickReport (used by tests + future structured logging). 7 new tests: - run_master_iteration: empty DB no-op; tick advances state (blocked → STUCK); reaper + pickup guard compose (stale in_progress at pickup limit → reaped → STUCK in one iteration). - master_main_loop: runs until stop; empty loop exits cleanly; on_iteration exception doesn't break loop; reaper runs less frequently than tick. Phase 1d-3+ remains for: discovery (Forgejo poll), per-workflow scheduling (enqueue next workflow_attempts after transition), Forgejo writes, MERGING state's merge call, reconciliation, backfill, operator CLI. Total: 278 controller tests; full auto_agents suite 2640 pass. |