a21466add232d59cdec1604e09d58ca05659a623
15 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 drew
|
4d969eaf2b |
feat(controller): Phase 3 — Gate 3 reviewer-abandon
When the reviewer finishes a review and judges the work fundamentally unworkable (implementation surfaced misdiagnosis, obsoleted-by-other-work, or irreducible complexity), it can now emit verdict='abstain' + suggested_next_action='abandon' with a Gate-3 abandon_reason_category. The controller routes REVIEWING → ABANDONED and (when the kill switch is on) performs the Forgejo close via the reviewer-abandon side-effect tick — no implementer/CI/merge cycles. Wired with the same defense-in-depth pattern Phase 2 established: MCP setter validation + outcomes mapper dispatch with confidence gating + Pydantic atomicity validator + side-effect tick with audit-trail attribution (cause=REVIEWER_ABANDON, event_type='reviewer_abandon'). Default-off CONTROLLER_GATE3_ABANDON_ENABLED kill switch so a fresh deploy is audit-only until the operator explicitly enables Forgejo writes. Bundled refactor: hoisted the 9 Gate-2 + 3 Gate-3-exclusive abandon categories into tools/controller/contracts/abandon_categories.py (triggered by Phase 3 per the plan's follow-up backlog). Both gates now consume the shared frozensets; doc-contract tests grep each agent prompt against the canonical list. Adversarial review (2 rounds): caught + fixed MCP cross-check ordering (atomicity FIRST so missing-setter shows actionable error), confidence=None symmetric downgrade across both gates, dead blocking-issues extraction in _run_close, idempotency clock-collision in the test, low-vs-missing reason-string conflation, and several test-quality gaps. 4064/4071 tests passing (7 pre-existing failures unrelated to Phase 3). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
a91df787d7 |
feat(controller): Phase 2 — Gate 2 estimator-abandon
Adds the second of three abandon gates: the estimator (Gate 2) can
mark a work item fundamentally unworkable, transitioning the
workflow ANALYZING -> ABANDONED and triggering a Forgejo close via
Phase 1's decomposed close_act orchestrator. Catches abandon cases
at the cheapest LLM stage, before implementer/reviewer tiers fire.
Substantive:
- EstimatorOutputV1: additive verdict + abandon_reason_category +
abandon_reason_detail fields (pre-Phase-2 outputs still parse).
@model_validator enforces abandon-requires-category atomicity at
parse time — third defense layer beyond MCP setter + outcomes
mapper
- state_machine: estimator_abandon event + (ANALYZING,
estimator_abandon) -> ABANDONED. 57 transitions; invariants clean
- mcp/estimator_builder: estimator_set_verdict setter validates
verdict enum + 9-category whitelist (scope_intractable,
intent_wrong, security_regression, deprecated_dependency,
breaks_protected_invariants, out_of_scope, low_value,
unmaintained_path, policy_violation) + cross-field rules
- outcomes._map_estimator_outcome: dispatch verdict='abandon'
-> estimator_abandon, with confidence-low downgrade to
estimator_done (honors the agent prompt's documented "high or
medium" requirement)
- estimator_abandon_side_effects.py: per-state side-effect tick
modeled on grooming_side_effects.py; invokes close_act with
cause=Cause.ESTIMATOR_ABANDON + event_type='estimator_abandon'
- _events.py: shared latest_transition_event +
workflows_with_latest_transition_in helpers; dialect-aware
payload['event'] extraction (SQLite json_extract +
PostgreSQL ->>); centralizes the event_type='transition' +
payload['event'] convention that side-effect ticks consume
- gate2_abandon_config.py: CONTROLLER_GATE2_ABANDON_ENABLED kill
switch (default false). Fresh Phase 2 deploys are audit-only
until operator explicitly enables; dry_run shared with grooming
for unified safe-rollout staging
- .opencode/agents/estimator-implementation.md: GATE 2 ABANDON
section with 9-category criteria + low_value disqualifier ("PR
cites an issue/ticket -> route to reviewer instead")
Round-2 adversarial-review fixes (all required pre-commit):
- forgejo_writes.close_issue / close_act: NEW cause + event_type
kwargs (defaults preserve Phase 1 grooming behavior; Phase 2
callsite overrides). Fixes audit-trail attribution: telemetry
queries SELECT WHERE cause='estimator_abandon' now return the
right rows. Phase 1 regression test pins the grooming defaults
- tick.py operator_unstick lookback: dialect-aware json_extract
fix (Phase 1 carry-over bug; would silently no-op on PostgreSQL)
- grooming_side_effects.py: idempotency filter now keys on
check_name set (grooming check_names only) so a Phase 1 close
and a Phase 2 close on the same workflow don't cross-cancel
Tests (+50): TestEstimatorOutputV1Phase2,
TestEstimatorAbandonStateMachine, TestMapEstimatorOutcomePhase2
(including confidence-low downgrade), TestEstimatorSetVerdict
(all 9 categories + cross-field rules), TestEventsHelper,
TestEstimatorAbandonSideEffectTick (including
test_close_writes_estimator_abandon_cause_and_event_type pinning
the audit-trail attribution, and Phase 1 regression guard).
Doc-contract test asserts all 9 categories appear in the agent
prompt. 1509/1509 passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
016b348117 |
feat(controller): grooming gate (Phase 0 + Phase 1 worker-shape dispatch)
Phase 0 (foundation):
- Cause enum (controller_events.cause) for action attribution
- Schema: grooming_decisions audit table; workflows gains
grooming_evaluated_at + deferred_reason + deferred_at +
deferred_target_workflow_id; pulls gains touched_files
- audit_comments: CLOSE / DEFER templates + render_comment_template
- forgejo_writes: close_issue + defer_issue 5-step crash-safe protocol
(fingerprint dedup, error matrix, dry-run)
- patch_pr_state callback in forgejo_http
- grooming_config: 22-env-var frozen-dataclass config + log_effective
- pulls.touched_files cache extension (_pipeline_cache.py schema v8)
- reaper.reap_grooming_decisions audit-retention sweep
- reconciliation RESUME guard (deferred_reason)
Phase 1 (worker-queue shape, 2026-05-25):
- New state: GROOMING. New events: grooming_started, groom_verdict_
{proceed,defer,close}. 5 new transitions; all invariants still clean
- GroomingInputV1 + GroomingOutputV1 Pydantic contracts
- outcomes._map_grooming_outcome routes verdicts to state-machine events
- prefetch.build_grooming_stage_b_input + list_open_prs callback
- scheduler GROOMING -> grooming_stage_b role
- promote: cfg-gated DISCOVERED -> GROOMING when CONTROLLER_GROOMING_
ENABLED=true; issues skip grooming
- forgejo_writes decomposed: close_act/defer_act (Forgejo writes only;
state-machine already transitioned) + close_decide_and_act/
defer_decide_and_act (Phase 0 callers); _apply_workflow_transition
is underscore-private
- grooming.py library: tokenization, suspicion scoring (Jaccard +
weighted overlap), deterministic checks, action -> verdict mapping
- mcp/grooming_builder.py: 14-tool FastMCP server emits GroomingOutputV1
- .opencode/agents/grooming-stage-b.md: duplicate-detection agent
prompt (claude-haiku-4-5)
- grooming_side_effects.run_grooming_side_effects_tick: per-state tick
performs Forgejo writes after groom_verdict_{defer,close} fires.
Filters on event_type='transition' + payload.event (centralizes the
convention pending Phase 2's latest_transition_event helper)
- GroomingCallbacks frozen dataclass; loop.py + __main__.py wired
Worker role registry (single source of truth):
- worker/roles.py: WORKER_ROLES + WorkerRoleSpec + default_roles_csv
+ output_filename_for. agent_runner.ROLE_TO_MCP_MODULE / ROLE_TO_
OUTPUT_MODEL derive from it; opencode_session.agent_name_for reads
it for flat cases; all 6 prompt builders use output_filename_for;
worker --roles default = default_roles_csv(); launcher script
derives --roles via shell substitution. Cross-site invariant test
enforces alignment across 5 sites + opencode.json MCP registry.
Phase 0 silent-bug fix:
- reconciliation.py RESUME guard SELECT now includes deferred_reason
(was missing since Phase 0; guard was a silent no-op). Tightened
from getattr to attribute access to fail fast on future omissions.
Tests (1456 total, +91 grooming-specific):
- test_grooming_phase0.py: 34 tests (orchestrator matrix, crash
recovery, idempotency, dry-run)
- test_grooming_phase1.py: 60 tests (library, contracts, state
machine, outcomes, scheduler, promote, prefetch, act-variants
with signature parity, side-effect tick incl. natural-idempotency
+ executed-flag-skip + verdict-mismatch + reconciliation RESUME)
- test_mcp_builders.py TestGroomingBuilder: 29 tests (happy paths
+ 22 validation rules + Pydantic round-trip + master-tick-read-
path companion)
- test_worker_agent_runner.py TestRoleMaps: cross-role wiring
alignment + agent-prompt-vs-worker-fallback filename contract +
inspect.signature equality (close_act/defer_act vs
close_issue/defer_issue)
- test_state_machine.py: transition count 51 -> 56 +
events_from_grooming
Live-validated end-to-end on 4 staged sentinel PRs (#55-#58) in
dry_run: agent emits verdicts via MCP, state-machine transitions
fire, side-effect tick writes audit row, deferred_reason gates
reconciliation RESUME correctly.
Deferred refinements + Phase 2 prerequisite (latest_transition_event
helper) tracked in .drew/regressions-plan.md "Phase 1 follow-up
backlog".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
615a05b982 |
feat(controller): rebase-default conflict resolution with merge fallback
PR branches 177-180 commits ahead of base cannot be rebased commit-by-commit by a single-shot resolver agent (too many conflict stops for one session). Conflict-prep now defaults to rebase (linear history) and falls back to a single 3-way merge when the branch is too divergent (commit count over CONTROLLER_CONFLICT_REBASE_MAX_COMMITS, default 60). The merge pipeline derives the track from branch shape via a Do:rebase -> Do:merge ladder in _make_merge_pr — no stored flag. Adds a git_rebase_continue MCP tool plus status rebase/merge-in-progress fields so the conflict-resolver agent is fully MCP-driven and dual-mode (mid-rebase or mid-merge). Also routes a green-CI implementer noop straight to REVIEWING instead of a deadlock-prone AWAITING_CI round trip. No state-machine change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
9d20f865d3 |
feat(controller): pre-review mergeable gate — skip doomed reviews
When CI goes green, route to REVIEWING only if the PR still merges cleanly into base. If a cheap Forgejo mergeable check shows the base advanced while CI ran, route AWAITING_CI → CONFLICT_RESOLVING instead — handing the conflict to the LLM conflict_resolver BEFORE the expensive reviewer pass, since code that must be rebased gets re-CI'd and re-reviewed afterwards anyway. - new event `pre_review_base_conflict` + transition (AWAITING_CI → CONFLICT_RESOLVING) - ci_status_poll: `_decide_green_event` gate behind a new optional `get_pr_details` callback; conservative — only an explicit mergeable=false diverts, an unknown/uncomputed bit falls through to ci_green so a fresh PR is never false-routed - gate-fired event rows carry `mergeable` in the payload so the false-positive rate is observable from controller_events - wired through loop.py (5th ci_status_poll_args element) + __main__ One API call, no LLM — cheap+frequent detection gating the expensive+rare conflict_resolver/CI/reviewer stages. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
f76568a871 |
feat(controller): ci-infra-failure implementer outcome → bounded rerun
The implementer-side counterpart to the `indeterminate` verdict. When the implementer IS dispatched onto a CI failure and finds the log carries no verdict (a hard-kill / OOM — nothing in the diff to fix), it can now emit `outcome=ci-infra-failure` instead of being forced to `blocked` → STUCK. `ci-infra-failure` forbids commits/files/blockers (no-work invariant, like `noop`) and fires `implementer_ci_infra_failure`, routing IMPLEMENTING → DISCOVERED so the CI-freshness gate reruns CI under its bounded RERUN_BUDGET. Backstopped by `_MAX_CI_INFRA_FAILURE=4` so a mis-classification cannot loop the gate forever. Wired through: the V1 contract enum, the implementer MCP builder (outcome value + no-work invariant), the state machine (event + IMPLEMENTING→DISCOVERED transition), the outcome mapper (+ per-workflow cap), tick's `_count_prior_ci_infra_failure`, and the implementer prompt — which now surfaces the outcome whenever the CI summary shows a failing gate, with guidance to use it ONLY when the log genuinely shows no verdict. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
3106230ef8 |
fix(controller): cap estimator worker-internal-error retries
An estimator whose session ends without emitting canonical output fails with worker-internal-error. Unlike the implementer, the estimator has no escalation path (it runs pre-tier) and no salvage (it produces no git artifact) — so a flaky estimator session just re-enqueues, with nothing to stop it. Run-2 burned 174 consecutive estimator worker-internal-error attempts on one PR. After _ESTIMATOR_WORKER_ERROR_LIMIT (3) such failures the workflow now STUCKs for operator attention via estimator_failed_twice — symmetric with the implementer worker-error escalation cap. - outcomes.py: the cap + the prior_estimator_worker_errors param. - tick.py: _count_prior_estimator_worker_errors (per-workflow count; the estimator runs pre-tier, so tier is not a meaningful axis). - state_machine.py: estimator_failed_twice description corrected — the event had no emitter before this; strict-parse failures route via contract-violation -> pickup_exhausted, not this event. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
0db0a15dad |
feat(controller): RUN_CI_LOCAL verdict source, ci-not-ready outcome, escalation hardening
Adds RUN_CI_LOCAL — an on-demand local-CI verdict source for when the cluster's Forgejo CI is broken — plus robustness fixes, the telemetry Live-tab rewrite, and PR-level cost attribution. Controller: - RUN_CI_LOCAL: the master swaps its Forgejo CI callbacks for local `forgejo-runner exec` runs (tools/run-ci-full-local.sh + local_ci.py). Async per-(owner,repo,SHA) on-disk job cache; preflights the forgejo-runner binary + Docker daemon at startup (fail loud, not a red verdict on every PR); GCs finished run dirs + per-run actcache. - ci-not-ready implementer outcome + implementer_ci_not_ready event: an implementer that runs before the on-demand verdict exists parks in AWAITING_CI instead of dead-ending at STUCK; capped against ci_red ping-pong. - ci_poll_exhaustion skips its sweep while a local CI run is in flight, so AWAITING_CI workflows queued behind on-demand CI are not STUCK'd by the remote-CI-sized timeout. - Escalate the workflow after repeated worker-internal-error at a tier instead of retrying to pickup-exhaustion -> STUCK. - forgejo_http: normalise Forgejo's per-gate `status` key to `state` so failing gates are actually counted (they previously all read as pending). - Per-tier worker timeouts bumped +15 min; a timed-out attempt's dirty-worktree residue is preserved on auto-scratch/pr-<N> before the next attempt's reset. - Implementer agents now verify only the CI-flagged gate(s) via a targeted re-run rather than the full local battery before claiming resolved/noop. Re-running the whole suite CI will run anyway was the #1 cause of implementer timeouts; CI remains the real gate and re-dispatches the implementer on red. Telemetry: - Live tab rebuilt on /api/live (controller DB run state + the live OpenCode session forest) after the live_log_writer sidecar was retired with the legacy dispatchers. - Durable per-attempt input/output payloads surfaced in the Live drill-down, archived-session detail, and Workflows timeline. - PR-level cost attribution: worker session tags carry -pr-<n>; backfill_llm_activity_pr.py repairs rows written before the fix. Shared: - tools/controller/session_tag.py — one canonical controller-tag parser shared by the telemetry server and the backfill. Tests: new coverage for local_ci (state machine, log parsing, _summarize_run, GC, in-flight probe, preflight), the ci-not-ready path, the escalation/ci-not-ready SQL counters, the ci_poll in-flight skip, and CI-status payload parsing across both sources. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
0bc734c020 |
style: ruff format the controller-state-machine branch (288 files)
Applies `ruff format` to the accumulated formatting debt on this branch. Formatting-only — no behavioral changes. Required for CI/lint's format gate (`nox -s format -- --check`), which the branch was failing on 288 tracked files that drifted from ruff's canonical style. In-progress WIP files are intentionally excluded so this commit stays a clean formatting-only diff. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
a2fcecb28d |
feat(controller): CI-freshness gate — re-trigger stale/infra CI instead of dead-ending
A discovered PR with stale CI (every job failed at the checkout step —
a git-fetch connection reset, pure infra, no code ran) burned an
estimator + a tier-2 implementer and dead-ended at STUCK. The pipeline
had no notion of CI freshness and never triggered CI — only polled.
New early master tick (ci_gate) runs before DISCOVERED->ANALYZING
promotion. For each DISCOVERED pr-kind workflow it classifies the CI
via ci_freshness.classify_ci_result:
- infra_broken — failed; the failing jobs' LOG content carries a
checkout/setup signature (curl 56, expected 'packfile', ...).
Logs are fetched via _ci_logs (session-cookie auth).
- stale — failed; newest status older than CONTROLLER_CI_MAX_AGE_S
(default 6h). Timestamp-based, log-independent — catches an old
failure even when Forgejo has purged its logs.
- no_ci / pending / fresh_real — handled accordingly.
infra_broken/stale/no_ci -> push an empty commit to the PR branch
(Forgejo 15.0.2 has no rerun API), routing DISCOVERED -> AWAITING_CI
(new event discovery_ci_rerun_triggered). The existing AWAITING_CI
poller then gets a real verdict. A reran CI that is ALSO infra/stale
routes AWAITING_CI -> DISCOVERED (new event ci_infra_recheck) so the
gate re-handles it; bounded by a rerun budget of 3, then STUCK.
Also wires the existing CI summarizer into prefetch so workers stop
receiving ci_summary=null.
1095 controller tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
drew
|
17cd91d275 |
fix(controller): post-commit review — close 3 merge-bridge stranding bugs
The trial-5 merge-pipeline split could strand workflows in MERGING
forever on its main paths. The controller has no MERGING handler
(merge_drive owns it), so any terminal_state that maps to no event
leaves the workflow orphaned. Three such holes, all caught by the
post-commit multi-perspective review:
1. merge_train emits "merge-error-{403,5xx,...}" for any non-2xx/409
Forgejo merge POST — none were mapped. Added _resolve_bridge_event:
403 -> branch_protection_blocked, all else -> retry_exhausted (STUCK).
2. run_one_cycle applied one shared outcome.terminal_state to every
claimed PR. A bisected train returns "bisected" (unmapped) so all
PRs stranded; a mixed train could tell a merged PR merge_base_conflict.
CycleOutcome now carries pr_terminal_states and events emit per-PR.
3. Graceful shutdown mid-merge ("stopped") was unmapped. Added the
merge_interrupted event (MERGING -> APPROVED) so the workflow
returns to the handoff state for clean re-pickup.
Also fixes a stale comment in _controller_db_bridge.py (merge_base_conflict
routes to CONFLICT_RESOLVING, not IMPLEMENTING) and adds a drift-guard
test cross-checking the bridge's transition map against the canonical
state machine — that drift is what produced the stale comment.
3304 tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
drew
|
a6986008ee |
feat(controller): trial-5 batch — dispute path, merge pipeline split, conflict-resolver hardening
T5-1 reviewer feedback rendered full-body to the implementer
T5-4/9 implementer dispute path — dispute-at-any-tier with per-tier cap,
OPERATOR_ATTENTION state on stalemate, pr-review-worker-dispute agent
T5-5 reviewer BLOCKING ISSUE EVIDENCE RULE + 5-step validation
T5-7 merge step split into a singleton process — impl/review masters write
APPROVED and stop; merge_drive owns APPROVED -> MERGING -> MERGED
T5-10 merge process is fully deterministic; base conflicts bounce to the
controller's CONFLICT_RESOLVING (LLM); conflict_drive sidecar retired
T5-11 implementer fast success path — verified-clean outcome so a no-op
after conflict resolution doesn't force busywork
T5-12 conflict-resolver permissions fixed across all paths (/tmp/** glob)
T5-13 conflict-resolver PR-intent prehydration (title/body/comments)
Adds tools/_controller_db_bridge.py so merge_drive reads the controller DB
directly (Option B), plus APPROVED + OPERATOR_ATTENTION states, the
dispute/verified-clean events, and the V1 contract fields backing them.
Reviewer model: baseline -> sonnet, dispute -> opus.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
drew
|
d71046b9a0 |
fix(controller): batch D — PID-reuse, AWAITING_CI escape, flake bound, scheduler skip
Four safety items from the consolidated adversarial-review punch list.
ITEM 8 — PID-reuse hazard in janitor:
The janitor SIGKILL'd whatever process happened to live at the
sidecar's recorded subprocess_pid. Between sidecar write and janitor
sweep, the OS can reuse the PID for an unrelated process; the janitor
was killing innocents under fork-heavy workloads.
Fix:
- ``session_sidecar.py``: added ``subprocess_starttime`` field
(Optional[int]) + ``read_proc_starttime(pid)`` helper that reads
``/proc/{pid}/stat`` field 22 (clock ticks since boot — monotonic
for a (boot, pid) pair).
- ``WorkerSession.from_dict`` filters unknown keys so forward + back
compat with sidecars from earlier/later versions is preserved.
- ``janitor._pid_alive`` and ``_kill_with_grace`` accept
``expected_starttime``; on mismatch they short-circuit and DON'T
signal the impostor.
- ``_kill_with_grace`` return semantics tightened: True iff a signal
was actually delivered (False for "PID gone" / "PID reused"). The
``sessions_killed`` counter now reflects real kills.
ITEM 9 — AWAITING_CI escape from infinite poll:
Previously AWAITING_CI could only exit via ``ci_green`` /
``ci_red_*`` / ``ci_flake_retry`` — if CI hangs forever (runner
outage, broken integration, etc.) the workflow had no controller-
driven STUCK path; only operator_unstick could rescue it.
Fix: new ``ci_polling_exhausted`` event → STUCK. The master's
AWAITING_CI poll handler is the natural place to emit it once a
threshold passes (deferred to a follow-up — Phase 1k+ ships the
event in the table; the timer fires it).
ITEM 10 — ci_flake_retry was unbounded:
The ``ci_flake_retry`` self-loop on AWAITING_CI had no encoded
ceiling. Pathological flaky CI could loop forever (the docstring
said "retry once per gate" but nothing enforced it).
Fix:
- New ``workflows.ci_flake_retries_remaining`` column (server_default
'1', default 1 — operators tune via ``CONTROLLER_CI_FLAKE_RETRIES``
at startup or via direct UPDATE).
- New ``ci_flake_retries_exhausted`` event → ESCALATING. Master
decrements the column on each ci_flake_retry; at 0 the next CI
failure routes through ci_red_* (regular path) or this new
event (escalates if the operator wants a hard ceiling).
ITEM 11 — scheduler now skips PAUSED workflows:
Without this, the scheduler could enqueue a fresh attempt for a
PAUSED workflow between two reconciliation ticks (race: label
removed at T+0, reconciliation runs at T+300, scheduler ticks at
T+30 with stale DB state). The window is at most one attempt of
worker work.
Fix: ``schedule_next_attempts`` SQL now lists only
{ANALYZING, IMPLEMENTING, REVIEWING, CONFLICT_RESOLVING, ESCALATING}
explicitly; PAUSED is excluded by absence. Reconciliation owns the
PAUSED → resume transition; scheduler doesn't touch it.
Schema additions:
- ``workflows.ci_flake_retries_remaining`` (INTEGER NOT NULL DEFAULT 1)
- ``workflows.awaiting_ci_started_at`` (TIMESTAMP NULL) — for the
poll-exhaustion timer (timer impl deferred; column is staged).
Tests:
- TestReadProcStarttime — 3 tests (Linux skip-guard) for the
/proc/pid/stat parser (self-pid > 0, missing pid is None,
invalid pid is None).
- TestJanitor::test_pid_reuse_defended_via_starttime — pins the
contract end-to-end (real subprocess + fabricated wrong starttime
→ janitor doesn't signal).
- TestPhase1kPlusTransitions — 5 tests pinning the new events +
proving the load-bearing invariants still pass.
- test_scheduler_skips_paused_workflows — pins item 11.
Total: 603 controller tests pass (+10 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
feeec3e9c7 |
fix(controller): batch B — PAUSED state for label-gate pause/resume (item 2)
Adversarial review flagged: removing the opt-in label transitions a live workflow to ABANDONED — but ABANDONED is TERMINAL with only ``operator_unstick`` re-entry → DISCOVERED, losing all prior controller_events continuity. Operators removing the label to "pause" a long-running PR will be surprised it restarted from scratch. Fix: introduce a non-terminal ``PAUSED`` state. State machine changes (``tools/controller/state_machine.py``): - ``PAUSED`` added to KNOWN_STATES (non-terminal — has exits via ``opt_in_label_restored`` and ``operator_unstick``). - ``opt_in_label_removed`` / ``opt_in_label_restored`` events documented in EVENTS but NOT listed per-state in TRANSITIONS — they're out-of-band master-driven events written directly by reconciliation. Listing them per-state breaks the per-state-event-set invariants (ESCALATING / CONFLICT_RESOLVING). - ``(PAUSED, operator_unstick) → DISCOVERED`` for the escape hatch. Schema change (``tools/controller/db/models.py``): - ``workflows.pre_pause_state: Mapped[str | None]`` column captures the resume target. Master writes it on pause; clears it on resume. Reconciliation logic (``tools/controller/master/reconciliation.py``): - New ``_apply_transition_with_pre_pause`` helper writes both ``current_state`` and ``pre_pause_state`` atomically + emits the ``reconciliation`` event row. - On label removal (current != PAUSED): captures pre_pause_state, transitions to PAUSED. - On label restoration (current == PAUSED): reads pre_pause_state (fallback DISCOVERED for legacy NULL data), transitions back, clears pre_pause_state. - PAUSED workflows are now SCANNED by reconciliation (not just non-terminals) so we can detect label-restored. Behaviour matrix: | current | label | result | |---------|----------|------------------------------------------| | any != | absent | → PAUSED, pre_pause_state = current | | PAUSED | present | → pre_pause_state (or DISCOVERED) | | PAUSED | absent | stays PAUSED (no transition) | | any != | present | regular state checks (no-op for label) | Tests (test_label_gate.py refactor + 3 new tests): - test_label_removed_pauses_workflow (was: abandons) - test_label_restored_resumes_from_pre_pause_state (new) - test_paused_workflow_without_label_stays_paused (new) - test_resume_fallback_when_pre_pause_state_missing (new — legacy data without the new column) - Existing event-reason test still passes (reason string unchanged). Total: 590 controller tests pass, 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
976817fa22 |
feat(controller): Phase 1d-1 — state machine + reaper + pickup guard
The deterministic spine of the master controller. State machine is
pure data with 6 load-bearing invariants enforced via property tests.
Reaper resets stale-heartbeat workflow_attempts to pending. Pickup
guard transitions workflows to STUCK when an attempt has been
re-pended too many times without success.
tools/controller/state_machine.py:
- KNOWN_STATES = 12; TERMINAL_STATES = {MERGED, ABANDONED, STUCK,
CREATED_PR}. STUCK's only allowed exit is the operator-driven
operator_unstick event (back to DISCOVERED).
- 32 TRANSITIONS entries covering DISCOVERED → ANALYZING →
IMPLEMENTING ↔ AWAITING_CI / CONFLICT_RESOLVING / ESCALATING →
REVIEWING → MERGING → MERGED. Plus pickup_exhausted exits from
IMPLEMENTING/CONFLICT_RESOLVING/REVIEWING.
- 27 named events with descriptions. apply_event() lookup raises
IllegalTransitionError (lists legal events from current state)
or ValueError on unknown state (per v6 unknown-state guard).
- 6 LOAD-BEARING invariants for v1 (per v9 simplification):
1. no_path_implementing_to_reviewing_skips_ci (Hard Rule #1
constructional fix for the no-mans-land race)
2. terminal_states_have_no_exits (only STUCK→operator_unstick OK)
3. tier_monotonic_non_decreasing
4. every_pr_workflow_includes_reviewing
5. conflict_resolving_bounded (1st→IMPLEMENTING, 2nd→ESCALATING,
3rd→STUCK; structurally encoded)
6. escalation_deterministic
- reachable_from() honors cycles (DISCOVERED ∈ reachable(DISCOVERED)
via STUCK→operator_unstick path; AWAITING_CI self-loops via
ci_flake_retry).
tools/controller/reaper.py:
- reap_stale_attempts(): SELECT in_progress attempts whose
lock_heartbeat_at + lock_ttl_seconds < NOW (per-row TTL respects
per-role differences — estimator 180s, reviewer 720s, tier-2
implementer 2160s). UPDATEs status='pending', clears lock columns,
preserves pickup_count (the pickup guard handles that). Inserts
controller_events row with reason='lock-ttl-expired' per reap.
- Dialect-portable: Postgres uses interval arithmetic; SQLite uses
julianday(). Same logic either way.
tools/controller/pickup_guard.py:
- transition_exhausted_to_stuck(): finds attempts with status='pending'
AND pickup_count >= MAX_PICKUPS (default 3 per v6 blocker fix)
AND workflow not already terminal. Transitions workflow → STUCK,
marks attempt as 'reaped', inserts controller_events with
reason='attempt-pickup-exhausted' + pickup_count + max_pickups.
45 new tests:
- state_machine: basic shape (states partition, every transition uses
known states + defined events), apply_event success/error paths,
events_from + reachable_from helpers (including cycle awareness),
per-invariant zero-violations against the live table, per-invariant
monkeypatch-violations to prove the checks catch the bug class they
claim to, parametrised sanity check "every non-terminal can reach
some terminal".
- reaper: empty DB / fresh heartbeat / stale heartbeat reaped /
per-row TTL respected / event row created / only-in-progress
reaped / multiple stale attempts.
- pickup guard: empty DB / below limit / at limit / in-progress not
checked / terminal workflow skipped / event payload content /
default max_pickups matches v6.
Total: 229 controller tests; full auto_agents suite 2591 pass.
|