a21466add232d59cdec1604e09d58ca05659a623
17 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 drew
|
016b348117 |
feat(controller): grooming gate (Phase 0 + Phase 1 worker-shape dispatch)
Phase 0 (foundation):
- Cause enum (controller_events.cause) for action attribution
- Schema: grooming_decisions audit table; workflows gains
grooming_evaluated_at + deferred_reason + deferred_at +
deferred_target_workflow_id; pulls gains touched_files
- audit_comments: CLOSE / DEFER templates + render_comment_template
- forgejo_writes: close_issue + defer_issue 5-step crash-safe protocol
(fingerprint dedup, error matrix, dry-run)
- patch_pr_state callback in forgejo_http
- grooming_config: 22-env-var frozen-dataclass config + log_effective
- pulls.touched_files cache extension (_pipeline_cache.py schema v8)
- reaper.reap_grooming_decisions audit-retention sweep
- reconciliation RESUME guard (deferred_reason)
Phase 1 (worker-queue shape, 2026-05-25):
- New state: GROOMING. New events: grooming_started, groom_verdict_
{proceed,defer,close}. 5 new transitions; all invariants still clean
- GroomingInputV1 + GroomingOutputV1 Pydantic contracts
- outcomes._map_grooming_outcome routes verdicts to state-machine events
- prefetch.build_grooming_stage_b_input + list_open_prs callback
- scheduler GROOMING -> grooming_stage_b role
- promote: cfg-gated DISCOVERED -> GROOMING when CONTROLLER_GROOMING_
ENABLED=true; issues skip grooming
- forgejo_writes decomposed: close_act/defer_act (Forgejo writes only;
state-machine already transitioned) + close_decide_and_act/
defer_decide_and_act (Phase 0 callers); _apply_workflow_transition
is underscore-private
- grooming.py library: tokenization, suspicion scoring (Jaccard +
weighted overlap), deterministic checks, action -> verdict mapping
- mcp/grooming_builder.py: 14-tool FastMCP server emits GroomingOutputV1
- .opencode/agents/grooming-stage-b.md: duplicate-detection agent
prompt (claude-haiku-4-5)
- grooming_side_effects.run_grooming_side_effects_tick: per-state tick
performs Forgejo writes after groom_verdict_{defer,close} fires.
Filters on event_type='transition' + payload.event (centralizes the
convention pending Phase 2's latest_transition_event helper)
- GroomingCallbacks frozen dataclass; loop.py + __main__.py wired
Worker role registry (single source of truth):
- worker/roles.py: WORKER_ROLES + WorkerRoleSpec + default_roles_csv
+ output_filename_for. agent_runner.ROLE_TO_MCP_MODULE / ROLE_TO_
OUTPUT_MODEL derive from it; opencode_session.agent_name_for reads
it for flat cases; all 6 prompt builders use output_filename_for;
worker --roles default = default_roles_csv(); launcher script
derives --roles via shell substitution. Cross-site invariant test
enforces alignment across 5 sites + opencode.json MCP registry.
Phase 0 silent-bug fix:
- reconciliation.py RESUME guard SELECT now includes deferred_reason
(was missing since Phase 0; guard was a silent no-op). Tightened
from getattr to attribute access to fail fast on future omissions.
Tests (1456 total, +91 grooming-specific):
- test_grooming_phase0.py: 34 tests (orchestrator matrix, crash
recovery, idempotency, dry-run)
- test_grooming_phase1.py: 60 tests (library, contracts, state
machine, outcomes, scheduler, promote, prefetch, act-variants
with signature parity, side-effect tick incl. natural-idempotency
+ executed-flag-skip + verdict-mismatch + reconciliation RESUME)
- test_mcp_builders.py TestGroomingBuilder: 29 tests (happy paths
+ 22 validation rules + Pydantic round-trip + master-tick-read-
path companion)
- test_worker_agent_runner.py TestRoleMaps: cross-role wiring
alignment + agent-prompt-vs-worker-fallback filename contract +
inspect.signature equality (close_act/defer_act vs
close_issue/defer_issue)
- test_state_machine.py: transition count 51 -> 56 +
events_from_grooming
Live-validated end-to-end on 4 staged sentinel PRs (#55-#58) in
dry_run: agent emits verdicts via MCP, state-machine transitions
fire, side-effect tick writes audit row, deferred_reason gates
reconciliation RESUME correctly.
Deferred refinements + Phase 2 prerequisite (latest_transition_event
helper) tracked in .drew/regressions-plan.md "Phase 1 follow-up
backlog".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
0bc734c020 |
style: ruff format the controller-state-machine branch (288 files)
Applies `ruff format` to the accumulated formatting debt on this branch. Formatting-only — no behavioral changes. Required for CI/lint's format gate (`nox -s format -- --check`), which the branch was failing on 288 tracked files that drifted from ruff's canonical style. In-progress WIP files are intentionally excluded so this commit stays a clean formatting-only diff. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
3ca794be75 |
feat(controller): autonomous CI status polling — closes the last trial gap
The Phase 2 trial previously required operator-intervention SQL to
advance workflows from AWAITING_CI → REVIEWING (no automated CI
status polling). This commit wires the missing tick so the trial
runs end-to-end without manual help.
Components:
- ``master/forgejo_http.py``: new ``get_ci_status`` callback wraps
Forgejo's ``/commits/{sha}/status`` combined-status endpoint;
added to ``ForgejoCallbacks``.
- ``master/ci_status_poll.py`` (NEW): ``run_ci_status_poll_tick``
scans AWAITING_CI workflows, fetches CI status keyed on the
latest implementer attempt's ``head_sha_after``, and applies
state transitions via ``apply_event``. TOCTOU-defended UPDATE
(``WHERE current_state='AWAITING_CI'``) + per-row exception
isolation.
- ``master/loop.py``: new ``ci_status_poll_args=(owner, repo,
get_ci_status)`` kwarg + ``ci_status_poll_interval_s`` config
(default 60s) + ``MasterTickReport.ci_status_poll`` field.
- ``master/__main__.py``: threads ``callbacks.get_ci_status`` into
the loop.
State mapping (Forgejo combined-status state → event):
- success / neutral / skipped / warning → ci_green → REVIEWING
- failure / error / cancelled / timed_out / stale →
ci_red_retry_same_tier → IMPLEMENTING
- pending / queued / in_progress / action_required → no-op (wait)
- None / unknown / fetch failure → no-op (transient)
The ``ci_polling_exhausted`` timeout (default 2h) remains as the
safety net for CI that genuinely never reports.
Tests (+14 in test_master_ci_status_poll.py):
- Happy paths (success→green, failure→red, pending→wait)
- Error paths (callback raises; workflow without head_sha)
- Event row shape (event_type='ci-green'/'ci-red', reason payload)
- Extended state mapping (cancelled, neutral, in_progress)
- Other-repo isolation
- LoopIntegration end-to-end via master_main_loop with safety timer
RUNBOOK updated: removed the manual SQL workaround; added the
autonomous CI poll's tunables.
Total: 726 controller tests pass (+14 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
febb352618 |
fix(controller): pipeline run-blockers — promoter, scheduler, owner/repo, workspace_dir patch
Round-3 deep pass identified four issues that would have prevented an
actual end-to-end pipeline run:
RB1 — DISCOVERED → ANALYZING never fired in production:
The state machine defines (DISCOVERED, discovery_picked_up) →
ANALYZING but NO production code fires the event. Workflows
created by discovery would sit in DISCOVERED forever.
Fix:
- New ``master/promote.py``: ``run_promote_discovered_tick`` scans
for DISCOVERED workflows + fires ``discovery_picked_up`` via
apply_event (state-machine invariants stay enforced) + emits a
``discovery-promoted`` controller_events row per transition.
- Composes with the master loop's other ticks; runs every iteration
(cheap — typically 0-1 row).
RB2 — scheduler.schedule_next_attempts never called from master loop:
The scheduler was exported by the master package but never invoked.
It creates the ``workflow_attempts`` rows that workers dequeue —
without it, workers would have nothing to pick up.
Fix:
- ``master/loop.py`` now accepts a ``prefetch: PrefetchCallback``
kwarg. When provided, the loop runs promote_discovered + scheduler
every iteration after tick/reaper/reconciliation.
- ``MasterTickReport`` gains ``promote_discovered`` and ``scheduler``
optional fields so on_iteration callbacks see both.
- ``master/__main__.py`` builds a ``PrefetchDataCallbacks`` from the
Forgejo callback bundle and constructs the production
``make_prefetch_callback(engine, callbacks)`` — wires through to
the loop's new prefetch kwarg.
RB3 — owner / repo missing from V1 input contracts:
The implementer / reviewer / estimator / conflict-resolver V1 inputs
had pr_number but not owner/repo. The OpenCode agent would have
had no way to know which Forgejo repo to clone — it would have had
to derive owner/repo from process env, coupling the worker to a
single repo.
Fix:
- ``contracts/v1.py``: added ``owner: str`` and ``repo: str``
(min_length=1) to ImplementerInputV1, ReviewerInputV1,
EstimatorInputV1, ConflictResolverInputV1.
- ``master/prefetch.py``: builders populate owner/repo from the
Workflow row (already known at prefetch time).
- Existing test fixtures in ``test_contracts_v1.py`` updated.
RB4 — input_payload.workspace_dir placeholder reached the agent:
Prefetch wrote ``workspace_dir = "<worker-injected>"`` as a
placeholder; the worker never patched it before invoking the
OpenCode session. The prompt builder rendered the literal
placeholder string into the agent's prompt — the agent had no idea
where to clone.
Fix:
- ``worker/agent_runner.py``: patches input_payload.workspace_dir
with the real path immediately before calling run_opencode_session.
Uses a shallow copy so the caller's dict isn't side-effected.
- ``worker/__main__.py``: workspace_dir naming convention is now
``pr-{owner}-{repo}-{pr_number}`` (matches workspace.py's
PerPRWorkspace convention) so the janitor's pr-* glob + the
agent's expected workspace location agree. Falls back to
``pr-attempt-{N}`` for legacy input_payloads missing owner/repo.
Tests:
- ``test_master_promote.py`` (NEW, +7 tests):
- empty DB no-op
- single workflow promoted
- multiple promoted in one tick
- only DISCOVERED targeted (non-DISCOVERED untouched)
- controller_events row emitted with correct shape
- idempotent after first promotion
- LoopIntegration end-to-end: DISCOVERED → ANALYZING → pending
estimator attempt visible in workflow_attempts (pins the entire
previously-broken pipeline from discovery to enqueue)
Total: 703 controller tests pass (+7 net), 0 regressions.
Without these four fixes, the pipeline would have looked alive in
unit tests but produced zero work in a real deployment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
147e3403c1 |
fix(controller): batch G — show-stoppers from round-2 review (N1–N4)
Four items the round-2 adversarial review flagged as ship-blockers.
N1 — PID-reuse defense is now WIRED in production:
Round 1's batch D shipped ``subprocess_starttime`` in the sidecar +
janitor checks against it, but NO production code wrote sidecars.
The defense was unwired; tests passed against a code path that
production never invoked.
Fix:
- ``worker/agent_runner.py`` accepts ``workspace_dir`` and
``opencode_server_url`` kwargs. When ``workspace_dir`` is set, it
writes a sidecar (``{workspace_dir}/worker.session``) immediately
after MCP spawn capturing the real PID + starttime from
``/proc/{pid}/stat`` field 22. Removes it on attempt completion.
- ``worker/__main__.py`` builds the per-attempt workspace dir
(``{workspace_root}/pr-attempt-{N}/``) and threads it through the
agent_runner closure with the OpenCode URL. The naming convention
is picked up by the janitor's ``pr-*`` glob; when per-PR shared
workspaces ship (Phase 1k++ follow-up), it changes to
``pr-{owner}-{repo}-{N}``.
- Test: ``TestSidecarWiring`` (+2 tests) verifies the sidecar appears
during the attempt, carries the right PID + starttime + instance,
and is cleaned up post-attempt.
N2 — AWAITING_CI escape event firing is now WIRED in production:
Round 1's batch D shipped ``ci_polling_exhausted`` /
``ci_flake_retries_exhausted`` in TRANSITIONS, but NO production code
emitted them. Workflows could still hang in AWAITING_CI forever.
Fix:
- New ``master/ci_poll.py``: ``run_ci_poll_exhaustion_tick`` scans
workflows whose ``entered_state_at`` is older than
``CONTROLLER_AWAITING_CI_TIMEOUT_S`` (default 7200s) and fires
``ci_polling_exhausted`` via ``apply_event`` → STUCK + emits a
``ci_poll_exhausted`` controller_events row with the threshold
payload.
- ``master/loop.py`` integrates the new tick on its own cadence
(``ci_poll_exhaustion_interval_s`` env, default 300s). Composes
with the existing master loop. ``MasterTickReport`` gains
``ci_poll_exhaustion: CIPollExhaustionReport | None``.
- Tests: ``test_master_ci_poll.py`` (+7 tests) — happy path, fresh
workflow stays untouched, only AWAITING_CI is targeted (other
long-lived non-terminal states ignored), event row shape pinned,
default threshold matches the documented 2h, end-to-end loop
integration (master_main_loop drives the exhaustion +
workflow → STUCK without operator intervention).
- Dialect-portable SQL (Postgres interval, SQLite julianday).
- Handles SQLite returning TIMESTAMP as str from text() queries
(no .isoformat() on str).
N3 — externally-merged/closed PRs now win over label removal:
Round-1's PAUSE-on-label-removed shipped, but reconciliation
checked the label gate BEFORE checking merged/closed. Operators
removing the opt-in label on an already-merged PR would PAUSE the
workflow forever — never transitioning to MERGED.
Fix:
- ``master/reconciliation.py:_reconcile_one`` re-ordered:
1. Check terminal-state mappings (merged/closed) FIRST — apply
immediately if they fire.
2. THEN the opt-in label gate (pause/resume).
3. Fall through to "consistent" otherwise.
- Tests: ``test_externally_merged_takes_priority_over_label_removal``
+ ``test_externally_closed_takes_priority_over_label_removal``
pin the contract. Both seed an IMPLEMENTING workflow + Forgejo
reporting "merged/closed AND no opt-in label" → workflow
transitions to MERGED/ABANDONED (not PAUSED) + pre_pause_state
stays None.
N4 — graceful handling of empty env vars:
``int(os.environ.get("CONTROLLER_FORGEJO_REQUEST_TIMEOUT_S", "30"))``
crashes with non-actionable ``int('') ValueError`` if the operator
sets the env to empty/whitespace (common when sourcing a partially-
edited /etc/cleveragents/master.env file).
Fix:
- ``master/forgejo_cfg.py:_env_int(name, default)`` — empty or
whitespace-only values fall back to the documented default; only
non-numeric values still raise (with a clear message naming the
variable).
- Tests: ``test_empty_env_value_falls_back_to_default`` +
``test_whitespace_only_env_falls_back`` + updated
``test_malformed_env_raises_value_error`` to match the new
"not a valid integer" wording.
Total: 660 controller tests pass (+13 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
72c272504c |
feat(controller): Phase 1k — controller-managed opt-in label gate
Migration safety mechanism: the controller only manages PRs and
issues carrying a configurable opt-in label (default
``controller-managed``). Operators opt PRs in for parallel-run
trials, can pause management mid-flight by removing the label, and
gradually roll out without exposing the controller to PRs that
human reviewers are actively driving.
Components:
- tools/controller/master/label_gate.py — single source of truth for
the configured label name + pure predicates/filters over Forgejo
PR/issue dicts.
- ``opt_in_label_name()`` reads ``CONTROLLER_OPT_IN_LABEL`` env
(default 'controller-managed'); empty/whitespace falls back.
- ``has_opt_in_label(entity, name)`` defensively handles every
degenerate shape (non-dict entity, non-list labels, non-dict
label entries, missing name field).
- ``filter_by_opt_in_label`` / ``count_filtered`` for callers.
Wired through:
- discovery.run_discovery + backfill.run_startup_backfill +
reconciliation.run_reconciliation_tick each accept
``opt_in_label`` and ``require_opt_in_label`` kwargs.
- Function defaults are ``require_opt_in_label=False`` for API
back-compat (existing 30+ discovery/backfill/recon tests work
without changes).
- __main__.py defaults to ``--no-opt-in-label`` OFF (gate ENABLED in
production); add ``--no-opt-in-label`` to bypass.
- DiscoveryReport gains a ``label_filtered_out`` counter.
Reconciliation behavior:
- When opt_in_label is configured AND the Forgejo response carries a
``labels`` field AND the opt-in label is NOT present, the workflow
transitions to ABANDONED with reason ``opt-in-label-removed`` +
emits a controller_events 'reconciliation' row.
- Partial Forgejo responses (no ``labels`` field) skip the label
check — never ABANDON on incomplete data.
Master loop extension:
- ``reconciliation_args`` now accepts an optional 5th element — a
kwargs dict threaded through to ``run_reconciliation_tick``.
__main__.py uses this to pass ``require_opt_in_label`` per the CLI
flag. 4-tuple back-compat preserved.
Tests (+29 in test_label_gate.py, 0 regressions across 569 tests):
- Predicate edge cases (every degenerate shape returns False)
- Env-var resolution (default, override, empty, whitespace)
- filter/count helpers
- Discovery + backfill: kept/filtered counts, gate disabled,
explicit label overrides env
- Reconciliation: label removed → ABANDONED, label present →
no-op, partial response → no-op, gate disabled → bypass, event
row records reason
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
3e269ce011 |
feat(controller): Phase 1j — deterministic CI summarizer + priority parsers
Replaces "ci_summary=None / failing_gates=[]" placeholders from Phase
1h with a real summarizer that maps Forgejo combined-status →
CISummary V1 dict by running per-tool deterministic parsers on each
failing gate's log.
Priority parsers shipped (cover lint/format/typecheck/unit_tests, the
4 most-failed gates):
- ruff — F+E codes from `nox -s lint`; Would-reformat lines from
`nox -s format`. Aggregates to single error_class when
all findings share one code, else RuffMixed.
- pyright — error/warning/information diagnostics; rule name pulled
from trailing `(reportName)` parens. Abs-path
normalization strips container prefixes.
- behave — failing scenarios (file:line + name), AssertionError
extraction. Feature/scenario summary line aggregation.
Stub parsers for not-yet-shipped tools (robot_framework, slipcover,
bandit, semgrep, vulture, radon, build): return a structured
CIFailure with error_class="parser-pending-{name}" + the raw log
excerpt. Operators see the failure; implementer still has log
context. Phase 1j+ replaces stubs with real parsers without changing
the gate-→-session map.
Components:
- _base.py — ParserResult dataclass + select_log_excerpt()
(tail-N-lines smart selection within 16KB cap)
- _stub.py — make_stub(name) factory for pending tools
- _registry.py — resolve(parser_name) + resolve_for_nox_session()
+ validate_parser_coverage()
- master/ci_summarize.py — summarize_ci_status(head_sha, status,
log_fetcher) orchestrator. Handles:
- composite multi: gates → CIFailure.composite_findings
- log_fetcher returning None → log-fetch-failed
- log_fetcher raising → caught + log-fetch-failed
- Unknown gate context → NoParserAvailable
- Forgejo state=None → unknown summary
- Parameterized matrix gates ("unit_tests-3.13")
→ base session name resolution
Tests (+46 across 2 new files, 0 regressions):
- Per-parser canonical + empty + garbage input
- Registry resolution (real vs stub), coverage validator
- Summarizer V1 contract round-trip
- Composite security_scan composite_findings shape
- Error paths (None status, raising fetcher, unknown gate)
- Parser version aggregation across mixed gates
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
dfdfbf762b |
feat(controller): Phase 1h — prefetch callbacks (V1 input assembly)
Per plan v9, the master assembles the worker input_payload at attempt-enqueue time so the worker dequeues a ready-to-use payload with no extra Forgejo I/O of its own. This phase ships per-role V1-input builders + a factory matching the scheduler's PrefetchCallback protocol: - build_implementer_input → ImplementerInputV1 shape (head_sha, head_ref, base_branch, active_reviews, pr_comments_since_last_attempt, prior_attempts, diff_summary) - build_reviewer_input → ReviewerInputV1 shape (full_diff, prior_implementer_attempts, implementer_claim, prior_reviews) - build_estimator_input → EstimatorInputV1 shape (pr_title, pr_body, diff_summary) — works for both PR and issue kinds - build_conflict_resolver_input → ConflictResolverInputV1 shape with conflicted_files=[] stub (worker patches via git rebase) - make_prefetch_callback(engine, callbacks) → routes by role; returns (payload, "V1") matching the scheduler's PrefetchCallback signature Forgejo HTTP wiring adds four new callbacks (get_pr_details, get_pr_diff, list_pr_reviews, list_pr_comments) plumbed through ForgejoCallbacks. Worker-side patches (post-dequeue, pre-validation): - attempt_id, attempt_number (known from dequeue) - workspace_dir (worker filesystem path) - wallclock_budget_s (worker config) What this phase DOES NOT yet produce: - ci_summary / failing_gates — Phase 1j (deterministic CI summarizer) - Issue-kind estimator's title/body — needs list_issue_details callback (defer to future phase) - conflict_resolver's actual conflicted_files — needs worker-side git rebase + conflict-parse pass Tests (+26 in test_master_prefetch.py, 0 regressions): - Per-role shape validation + V1 contract parse after worker patches - Prior-attempts merge (verbatim cap=3, oldest-first, total count) - Active-reviews projection (filters invalid states/missing user) - pr_comments_since_last_attempt filtering by finished_at - Factory routes by role; unknown role raises - Scheduler integration end-to-end (real prefetch → real INSERT) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
7d1dfb8635 |
feat(controller): Phase 1g — periodic reconciliation tick
Catches externally-merged / externally-closed PRs that the controller
didn't directly merge (operator clicked the merge button in Forgejo's
UI; collaborator closed a PR while controller was waiting). Inverse
of discovery: discovery ADDS new entities; reconciliation re-syncs
KNOWN ones.
tools/controller/master/reconciliation.py:
- run_reconciliation_tick(engine, owner, repo, get_pr_state,
get_issue_state=None):
- Scans all non-terminal workflows for (owner, repo).
- Calls get_pr_state / get_issue_state per workflow.
- Decision table:
- PR merged=True → MERGED (reason='externally-merged')
- PR state=closed not merged → ABANDONED ('externally-closed-
not-merged')
- PR state=open → consistent (no transition)
- PR not found (404) → STUCK ('pr-not-found-on-forgejo')
- Issue state=closed → ABANDONED ('issue-closed-externally')
- Per-row failures isolated: one Forgejo flake doesn't kill the
whole sweep. Failed fetches recorded as ReconciliationAction
with reason='fetch-failed' (workflow untouched).
- Pure decision function (_decide_transition) separated from SQL
writes (_apply_transition) for testability.
tools/controller/master/forgejo_http.py:
- Added get_pr_state + get_issue_state callbacks to ForgejoCallbacks.
- HTTP shape: 404 → None (workflow → STUCK); non-200/404 → raise
(workflow recorded as fetch-failed, not silently STUCK'd).
15 new tests in test_master_reconciliation.py:
- basics (empty DB, terminal workflows skipped, other-repo skipped)
- PR state mappings (open=consistent, merged → MERGED, closed →
ABANDONED, not-found → STUCK)
- issue state mappings (open=consistent, closed → ABANDONED,
no-callback → consistent)
- fetch failures (raised exception → fetch-failed, workflow untouched)
- event rows (reconciliation event emitted on transition; none on
consistent)
- ReconciliationAction dataclass shape
Total: 430 controller tests; full auto_agents suite 2792 pass.
|
||
|
drew
|
ba2e9472bc |
feat(controller): Phase 1f — master startup backfill
When the master starts (first deploy or after a long outage), it
needs to learn about existing open PRs/issues that weren't created
via discovery-tick-during-uptime. Backfill = discovery + a one-time
marker so subsequent restarts know "this isn't the first time."
tools/controller/master/backfill.py:
- run_startup_backfill(engine, owner, repo, list_prs, list_issues):
- Calls run_discovery (already idempotent — skips existing entities)
- Records a 'controller-backfill-complete' marker in controller_events
associated with the first new workflow OR an existing workflow OR
skipped if Forgejo is truly empty (no FK target)
- Returns BackfillReport{first_time, discovery}.
- has_backfill_run(engine, owner, repo): existence-check on the marker
by parsing controller_events.payload. Multi-tenant isolated — a
marker for (owner_a, repo_a) doesn't satisfy a check for
(owner_b, repo_b).
- The marker is informational; the dedup is provided by discovery's
unique-constraint skip. The marker exists so operators can answer
"has backfill ever run for this repo?" in one SQL query.
Wired into master __main__:
- Runs AFTER engine/create_all + Forgejo callback wiring, BEFORE
master_main_loop.
- Try/except wrapped so Forgejo flake at startup doesn't prevent
the main loop from running (discovery tick will retry).
- New --skip-backfill flag for tests + warm restarts.
8 new tests in test_master_backfill.py:
- has_backfill_run: no marker → False; multi-tenant isolation
(different owner OR different repo → False).
- First-time backfill creates workflows + marker; empty Forgejo
skips marker (no FK target).
- Second run reports first_time=False; picks up newly-appeared PRs
+ emits a second marker.
- Multi-tenant (owner-a, repo-a) and (owner-b, repo-b) both get
their own marker.
- New workflows are in DISCOVERED state.
Total: 415 controller tests; full auto_agents suite 2777 pass.
|
||
|
drew
|
ae940f4564 |
feat(controller): Phase 1d-3c-4 — HTTP adapter wiring callbacks to _claim_runtime
The production glue between the controller's callback protocols
(discovery / forgejo_writes / merging) and the existing Forgejo HTTP
client in _claim_runtime. Single build_callbacks(cfg) factory; tests
use the same module with a fake runtime stub.
tools/controller/master/forgejo_http.py:
- ForgejoCallbacks dataclass bundling every callback the controller
needs: list_prs, list_issues, list_comments, post_comment,
get_labels, add_label, remove_label, merge_pr.
- build_callbacks(cfg, runtime=None): wires each callback as a thin
closure over runtime.get/post/delete. Production omits ``runtime``
to use the real _claim_runtime module; tests inject a fake.
- Forgejo path conventions match the API:
GET /repos/{o}/{r}/pulls?state=open
GET /repos/{o}/{r}/issues?state=open&type=issues (excludes PRs)
GET/POST /repos/{o}/{r}/issues/{n}/comments
GET/POST /repos/{o}/{r}/issues/{n}/labels
DELETE /repos/{o}/{r}/issues/{n}/labels/{name} (URL-encoded)
POST /repos/{o}/{r}/pulls/{n}/merge (body: {"Do": "merge"})
- Robust response handling:
- list endpoints: non-200 → empty list; non-list body → empty;
non-dict items filtered out.
- post_comment: 200/201 ok; other → RuntimeError.
- add_label / remove_label: 200/201/204 → True; remove-404 → True
(label already gone = goal achieved); else False.
- merge_pr: returns normalized MergeResponse. On 404, fetches the
PR's actual state (merged=True → pr_state='merged'; state='closed'
→ 'closed'; PR fetch failure or non-200 → leave pr_state=None so
the merging handler defaults to ABANDONED conservatively).
- Any callback exception → synthetic 503 so the merging handler's
retry logic kicks in cleanly.
23 new tests in test_master_forgejo_http.py:
- list_prs (path format, non-200 → empty, non-list body → empty,
filters non-dict items)
- list_issues (type=issues filter)
- list_comments (path format)
- post_comment (201, 200, non-2xx raises, non-dict body)
- labels (get / add 201/500 / remove 204/404/URL-encoded)
- merge (200, 409, 500, callback-raises-as-503, 404+merged,
404+closed, 404+pull-fetch-failure)
Plus a bug fix surfaced by the test_404_with_pull_fetch_failure test:
the PR-state-fetch branch was returning 'open' on a 500 response;
now correctly checks status==200 before inspecting the body.
Total: 366 controller tests; full auto_agents suite 2728 pass.
|
||
|
drew
|
94821d2702 |
feat(controller): Phase 1d-3c-3 — MERGING handler (6-response-shape table)
The master's per-tick handler for workflows in MERGING. Per plan v6+v9
non-blocking retry: state stays MERGING across ticks, retry counter +
backoff tracked in workflows.merging_retry_count and
merging_retry_next_attempt_at.
tools/controller/master/merging.py:
- MergeResponse: normalized {status_code, pr_state, error_message}
the callback returns. status_code drives the 6-response table:
200 → MERGED
409 → IMPLEMENTING(tier_last_succeeded) with reason=
'post-approval-base-conflict'
403 → STUCK ('branch-protection')
422 → AWAITING_CI ('ci-required-status-missing')
5xx (any) → stay MERGING; bump retry_count + schedule
next_attempt_at = now + 2^retry_count seconds
(capped at 60s); STUCK at retry_count >= 5
404 + pr_state='merged' → MERGED ('externally-merged')
404 + pr_state='closed' → ABANDONED ('externally-closed')
404 + no pr_state → ABANDONED (conservative default)
- run_merging_tick(engine, merge, owner, repo) — sweeps workflows
WHERE current_state='MERGING' AND kind='pr' AND owner+repo match
AND (next_attempt_at IS NULL OR next_attempt_at <= now). Per row:
call merge → map response → transition (or schedule retry) + emit
controller_events row.
- Callback failure (callback raises) wrapped as a synthetic 503 so
the retry logic kicks in cleanly.
- Counter resets on non-retry responses (409 / 422) — keeps backoff
fresh for future retries.
- MAX_MERGE_RETRIES = 5; MAX_BACKOFF_S = 60.0.
14 new tests across 7 classes:
- 200 happy path + event row
- 409 → IMPLEMENTING(tier_last_succeeded)
- 403 → STUCK
- 422 → AWAITING_CI + retry counter reset
- 404 paths (merged + closed + no-state default)
- 5xx retry (counter bumped + backoff scheduled +
in-window-skipped + max-retries-STUCK + callback-raises-as-5xx)
- Owner/repo filter (other repo's MERGING untouched)
- kind='pr' filter (issues never processed even if mis-seeded)
Total: 343 controller tests; full auto_agents suite 2705 pass.
|
||
|
drew
|
3e3796d918 |
feat(controller): Phase 1d-3c-2 — Forgejo write helpers (status + labels)
Idempotent status comment posting via fingerprint markers + per-label
no-op-aware adjust. Production wires the callbacks to existing
_status_comments + _claim_runtime helpers; tests use fakes.
tools/controller/master/forgejo_writes.py:
- compute_fingerprint(workflow_id, event_kind, content_key) →
16-char SHA256 prefix. Deterministic; same inputs → same fingerprint.
Different (workflow_id OR event_kind OR content_key) → different fp.
- build_marker(fp) → HTML comment "<!-- controller:fingerprint:abc -->".
Searchable + invisible in Forgejo's rendered UI.
- comment_has_fingerprint(body, fp) → bool. Used by post_status_comment
for the dedup check before posting.
- post_status_comment(): full idempotency protocol —
1. compute fingerprint
2. list_comments callback → check existing for marker
3. if found → return duplicate (no post)
4. else post via post_comment callback
Failure modes:
- list_comments raises → fall through to post (conservative;
fingerprint match on next sweep catches the duplicate)
- post_comment raises → return failed; caller retries
- adjust_labels(add, remove): one-shot get_labels + per-label
add/remove. add-when-already-present → no-op; remove-when-absent
→ no-op. Per-label failure isolated. get_labels failure marks
all requested actions failed (caller retries).
19 new tests:
- fingerprint helpers (5: deterministic, distinct inputs, marker
format, body match, empty body no-match)
- post_status_comment (5: new post, duplicate skip, list-failure
fall-through, post failure, distinct event_kinds get distinct fps)
- adjust_labels (9: empty no-op, add-when-absent, add no-op,
remove-when-present, remove no-op, get failure, per-label
isolation with mixed failures, returning-False failure)
Total: 329 controller tests; full auto_agents suite 2691 pass.
Phase 1d-3c-3 (MERGING handler) + Phase 1c-3 (real OpenCode + MCP
spawn) remain.
|
||
|
drew
|
e8045d4b9b |
feat(controller): Phase 1d-3c-1 — discovery sweep + worker test deflake
Discovery polls Forgejo for open PRs/issues and inserts a fresh DISCOVERED workflow for any entity the controller hasn't seen yet. Same callback-injection pattern as the scheduler's prefetch: tests provide synthetic ListPRsCallback / ListIssuesCallback; production wires them to the existing _review_fetch helpers in a follow-up. tools/controller/master/discovery.py: - run_discovery(engine, owner, repo, list_prs, list_issues=None) — idempotent. Skips existing entities via a one-shot SELECT(kind, entity_number) WHERE owner+repo membership check. Inserts a DISCOVERED workflow + a controller_events 'discovered' row per new entity. Per-callback try/except: a PR-list failure doesn't block issue discovery, and vice versa. - Defensive coerce_int: rejects bools (True is an int subclass — bug class to avoid). Accepts string digits. - Returns DiscoveryReport(prs_seen, issues_seen, new_workflows, existing_skipped, new_entities[]). 12 new tests: - basics (empty, PRs only, issues only, both PRs+issues) - idempotency (2nd sweep skips; PR #42 + issue #42 coexist; other (owner, repo) isolated) - malformed input (non-int + bool skipped; string digit accepted) - callback failures (PR raises → still process issues; issues callback optional) - event row creation per new workflow Plus a worker test deflake: test_stolen_lock_between_agent_return_and_write was asserting the specific 'lost-lock-at-write' outcome, but under full-suite CPU contention the heartbeat thread can fire between the agent's return and _write_outcome — taking the 'lost-lock' (via lost_lock_event) path instead. Both are valid for this scenario; loosened the assertion to accept either. Total: 310 controller tests; full auto_agents suite 2672 pass. |
||
|
drew
|
337af855f3 |
feat(controller): Phase 1d-3b — per-workflow scheduler + static escalation
When the state machine transitions a workflow into a state that needs a worker (ANALYZING / IMPLEMENTING / REVIEWING / CONFLICT_RESOLVING), the scheduler enqueues a fresh pending workflow_attempts row with the input_payload prefetched. tools/controller/master/scheduler.py: - schedule_next_attempts(engine, prefetch=...) — finds workflows in worker-needing states without a matching pending/in_progress attempt; enqueues one fresh attempt per. Skips: - workflows already with pending/in_progress attempt for the role - ESCALATING at MAX_TIER → resolved to ABANDONED (no enqueue) - prefetch raised → per-workflow isolated failure - Static escalation policy resolved inline: ESCALATING + current_tier → IMPLEMENTING(min(tier+1, MAX_TIER)) OR ABANDONED. Both produce controller_events transition rows with the appropriate v9 event name (escalate_next_tier_available / escalate_max_tier_exhausted). - State→role table: ANALYZING → estimator, IMPLEMENTING → implementer, REVIEWING → reviewer, CONFLICT_RESOLVING → conflict_resolver. - PrefetchCallback is parameterized; production wires it to the Forgejo prefetch (Phase 1d-3c), tests inject a fake. - attempt_number monotonically increments via COALESCE(MAX, 0) + 1. - DEFERRED to Phase 1d-3c: actual Forgejo prefetch implementation (this commit ships the scheduler skeleton + the prefetch callback contract). 20 new tests: - state→role parametrization (4 states × matching role) - no-double-enqueue (3 cases: pending blocks, in_progress blocks, different-role doesn't block) - escalation (4 cases: tier 0→1, 1→2, MAX→ABANDONED, event row content) - prefetch raises → per-workflow skip - non-schedulable states parametrized (7 cases: DISCOVERED, AWAITING_CI, MERGING, MERGED, ABANDONED, STUCK, CREATED_PR) - attempt_number monotonicity (uses MAX+1) Total: 298 controller tests; full auto_agents suite 2660 pass. |
||
|
drew
|
130bbbf3c5 |
feat(controller): Phase 1d-3a — master main loop (composite tick)
Long-running master orchestrator composing the deterministic per-iteration work shipped in Phase 1d-1/1d-2. One iteration = state-machine tick + reaper + pickup guard, in that order (reasoning in module docstring). tools/controller/master/loop.py: - MasterConfig: tick_interval_s (default 30s), reaper_interval_s (default 60s), pickup_guard_max_pickups (default 3 per v6). - run_master_iteration(engine): single synchronous iteration — composable for tests + master_main_loop. - master_main_loop(engine): runs run_master_iteration on a loop until stop_event fires. Reaper runs less often than tick (every reaper_interval_s, not every tick_interval_s). on_iteration callback gets each MasterTickReport (used by tests + future structured logging). 7 new tests: - run_master_iteration: empty DB no-op; tick advances state (blocked → STUCK); reaper + pickup guard compose (stale in_progress at pickup limit → reaped → STUCK in one iteration). - master_main_loop: runs until stop; empty loop exits cleanly; on_iteration exception doesn't break loop; reaper runs less frequently than tick. Phase 1d-3+ remains for: discovery (Forgejo poll), per-workflow scheduling (enqueue next workflow_attempts after transition), Forgejo writes, MERGING state's merge call, reconciliation, backfill, operator CLI. Total: 278 controller tests; full auto_agents suite 2640 pass. |
||
|
drew
|
b519ebd98f |
feat(controller): Phase 1d-2 — outcome mapper + master tick handler
The master-side state machine driver. When a worker writes a complete
(or failed) attempt, the tick handler picks it up next pass, maps the
outcome to a state machine event, and applies the transition.
tools/controller/master/:
- outcomes.py: map_outcome_to_event() — pure function. Inputs:
role, current_state, output_payload, status, head_sha_advanced,
attempts_remaining_at_tier, conflict_count_at_current_tier.
Returns EventMapResult(event_name|None, reason).
- Implementer: resolved+pushed → implementer_pushed;
resolved+NO push → implementer_competence_failure (worker lied);
rebase-failed / competence-failure / blocked / noop all routed.
- Reviewer: approve → reviewer_approve; request-changes → retry vs
escalate based on attempts_remaining_at_tier; abstain →
reviewer_abstain; comment → no transition (operator review needed).
- Estimator: is_metadata_only → estimator_metadata_only; else
estimator_done.
- Conflict resolver: counts at current tier — 1st → first; 2nd →
second_same_tier (escalate); 3rd+ → three_plus (STUCK).
- Summarizer: doesn't drive state transitions.
- status='failed' policy: contract-violation → pickup_exhausted;
other failed outcomes (worker-internal-error / stale-input /
ttl-insufficient-for-retry / git-clone-failed / lost-lock / etc.)
→ no event (master re-enqueues per the v9 table).
- status='reaped' → no event (pickup guard handled).
- tick.py: run_tick() — one master tick. Queries
workflow_attempts WHERE status IN ('complete', 'failed') AND
finished_at > workflow.last_transition_at (heuristic for "not yet
processed"). Per row:
1. Validate current_state ∈ KNOWN_STATES (per v6 unknown-state
guard); if not, transition workflow → STUCK with reason='unknown-state'.
2. Map outcome → event via outcomes.map_outcome_to_event.
3. If no event, bump workflow.last_transition_at so we don't
re-process forever.
4. Apply event via state_machine.apply_event; IllegalTransition →
STUCK with reason='illegal-event'.
5. Commit transition (workflows.current_state + entered_state_at +
last_transition_at) AND insert controller_events row.
Returns TickReport with attempts_processed + transitions_applied +
transitions_log.
42 new tests:
- outcomes (22): status='failed' policy (3 paths) + each role's
happy paths + edge cases (unknown outcome / missing field /
None payload / unknown role).
- tick (20): implementer transitions (resolved+push → AWAITING_CI,
resolved-no-push → ESCALATING, rebase → CONFLICT_RESOLVING,
blocked → STUCK), reviewer transitions (approve → MERGING,
request-changes → IMPLEMENTING), event row content, no
reprocessing on second tick, unmapped attempts bump
last_transition_at, terminal workflows skipped (4 parametrized),
unknown current_state → STUCK.
Total: 271 controller tests; full auto_agents suite 2633 pass.
|