ae940f45644971d5937b755d2459eceffeb6500a
7 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 drew
|
ae940f4564 |
feat(controller): Phase 1d-3c-4 — HTTP adapter wiring callbacks to _claim_runtime
The production glue between the controller's callback protocols
(discovery / forgejo_writes / merging) and the existing Forgejo HTTP
client in _claim_runtime. Single build_callbacks(cfg) factory; tests
use the same module with a fake runtime stub.
tools/controller/master/forgejo_http.py:
- ForgejoCallbacks dataclass bundling every callback the controller
needs: list_prs, list_issues, list_comments, post_comment,
get_labels, add_label, remove_label, merge_pr.
- build_callbacks(cfg, runtime=None): wires each callback as a thin
closure over runtime.get/post/delete. Production omits ``runtime``
to use the real _claim_runtime module; tests inject a fake.
- Forgejo path conventions match the API:
GET /repos/{o}/{r}/pulls?state=open
GET /repos/{o}/{r}/issues?state=open&type=issues (excludes PRs)
GET/POST /repos/{o}/{r}/issues/{n}/comments
GET/POST /repos/{o}/{r}/issues/{n}/labels
DELETE /repos/{o}/{r}/issues/{n}/labels/{name} (URL-encoded)
POST /repos/{o}/{r}/pulls/{n}/merge (body: {"Do": "merge"})
- Robust response handling:
- list endpoints: non-200 → empty list; non-list body → empty;
non-dict items filtered out.
- post_comment: 200/201 ok; other → RuntimeError.
- add_label / remove_label: 200/201/204 → True; remove-404 → True
(label already gone = goal achieved); else False.
- merge_pr: returns normalized MergeResponse. On 404, fetches the
PR's actual state (merged=True → pr_state='merged'; state='closed'
→ 'closed'; PR fetch failure or non-200 → leave pr_state=None so
the merging handler defaults to ABANDONED conservatively).
- Any callback exception → synthetic 503 so the merging handler's
retry logic kicks in cleanly.
23 new tests in test_master_forgejo_http.py:
- list_prs (path format, non-200 → empty, non-list body → empty,
filters non-dict items)
- list_issues (type=issues filter)
- list_comments (path format)
- post_comment (201, 200, non-2xx raises, non-dict body)
- labels (get / add 201/500 / remove 204/404/URL-encoded)
- merge (200, 409, 500, callback-raises-as-503, 404+merged,
404+closed, 404+pull-fetch-failure)
Plus a bug fix surfaced by the test_404_with_pull_fetch_failure test:
the PR-state-fetch branch was returning 'open' on a 500 response;
now correctly checks status==200 before inspecting the body.
Total: 366 controller tests; full auto_agents suite 2728 pass.
|
||
|
drew
|
94821d2702 |
feat(controller): Phase 1d-3c-3 — MERGING handler (6-response-shape table)
The master's per-tick handler for workflows in MERGING. Per plan v6+v9
non-blocking retry: state stays MERGING across ticks, retry counter +
backoff tracked in workflows.merging_retry_count and
merging_retry_next_attempt_at.
tools/controller/master/merging.py:
- MergeResponse: normalized {status_code, pr_state, error_message}
the callback returns. status_code drives the 6-response table:
200 → MERGED
409 → IMPLEMENTING(tier_last_succeeded) with reason=
'post-approval-base-conflict'
403 → STUCK ('branch-protection')
422 → AWAITING_CI ('ci-required-status-missing')
5xx (any) → stay MERGING; bump retry_count + schedule
next_attempt_at = now + 2^retry_count seconds
(capped at 60s); STUCK at retry_count >= 5
404 + pr_state='merged' → MERGED ('externally-merged')
404 + pr_state='closed' → ABANDONED ('externally-closed')
404 + no pr_state → ABANDONED (conservative default)
- run_merging_tick(engine, merge, owner, repo) — sweeps workflows
WHERE current_state='MERGING' AND kind='pr' AND owner+repo match
AND (next_attempt_at IS NULL OR next_attempt_at <= now). Per row:
call merge → map response → transition (or schedule retry) + emit
controller_events row.
- Callback failure (callback raises) wrapped as a synthetic 503 so
the retry logic kicks in cleanly.
- Counter resets on non-retry responses (409 / 422) — keeps backoff
fresh for future retries.
- MAX_MERGE_RETRIES = 5; MAX_BACKOFF_S = 60.0.
14 new tests across 7 classes:
- 200 happy path + event row
- 409 → IMPLEMENTING(tier_last_succeeded)
- 403 → STUCK
- 422 → AWAITING_CI + retry counter reset
- 404 paths (merged + closed + no-state default)
- 5xx retry (counter bumped + backoff scheduled +
in-window-skipped + max-retries-STUCK + callback-raises-as-5xx)
- Owner/repo filter (other repo's MERGING untouched)
- kind='pr' filter (issues never processed even if mis-seeded)
Total: 343 controller tests; full auto_agents suite 2705 pass.
|
||
|
drew
|
3e3796d918 |
feat(controller): Phase 1d-3c-2 — Forgejo write helpers (status + labels)
Idempotent status comment posting via fingerprint markers + per-label
no-op-aware adjust. Production wires the callbacks to existing
_status_comments + _claim_runtime helpers; tests use fakes.
tools/controller/master/forgejo_writes.py:
- compute_fingerprint(workflow_id, event_kind, content_key) →
16-char SHA256 prefix. Deterministic; same inputs → same fingerprint.
Different (workflow_id OR event_kind OR content_key) → different fp.
- build_marker(fp) → HTML comment "<!-- controller:fingerprint:abc -->".
Searchable + invisible in Forgejo's rendered UI.
- comment_has_fingerprint(body, fp) → bool. Used by post_status_comment
for the dedup check before posting.
- post_status_comment(): full idempotency protocol —
1. compute fingerprint
2. list_comments callback → check existing for marker
3. if found → return duplicate (no post)
4. else post via post_comment callback
Failure modes:
- list_comments raises → fall through to post (conservative;
fingerprint match on next sweep catches the duplicate)
- post_comment raises → return failed; caller retries
- adjust_labels(add, remove): one-shot get_labels + per-label
add/remove. add-when-already-present → no-op; remove-when-absent
→ no-op. Per-label failure isolated. get_labels failure marks
all requested actions failed (caller retries).
19 new tests:
- fingerprint helpers (5: deterministic, distinct inputs, marker
format, body match, empty body no-match)
- post_status_comment (5: new post, duplicate skip, list-failure
fall-through, post failure, distinct event_kinds get distinct fps)
- adjust_labels (9: empty no-op, add-when-absent, add no-op,
remove-when-present, remove no-op, get failure, per-label
isolation with mixed failures, returning-False failure)
Total: 329 controller tests; full auto_agents suite 2691 pass.
Phase 1d-3c-3 (MERGING handler) + Phase 1c-3 (real OpenCode + MCP
spawn) remain.
|
||
|
drew
|
e8045d4b9b |
feat(controller): Phase 1d-3c-1 — discovery sweep + worker test deflake
Discovery polls Forgejo for open PRs/issues and inserts a fresh DISCOVERED workflow for any entity the controller hasn't seen yet. Same callback-injection pattern as the scheduler's prefetch: tests provide synthetic ListPRsCallback / ListIssuesCallback; production wires them to the existing _review_fetch helpers in a follow-up. tools/controller/master/discovery.py: - run_discovery(engine, owner, repo, list_prs, list_issues=None) — idempotent. Skips existing entities via a one-shot SELECT(kind, entity_number) WHERE owner+repo membership check. Inserts a DISCOVERED workflow + a controller_events 'discovered' row per new entity. Per-callback try/except: a PR-list failure doesn't block issue discovery, and vice versa. - Defensive coerce_int: rejects bools (True is an int subclass — bug class to avoid). Accepts string digits. - Returns DiscoveryReport(prs_seen, issues_seen, new_workflows, existing_skipped, new_entities[]). 12 new tests: - basics (empty, PRs only, issues only, both PRs+issues) - idempotency (2nd sweep skips; PR #42 + issue #42 coexist; other (owner, repo) isolated) - malformed input (non-int + bool skipped; string digit accepted) - callback failures (PR raises → still process issues; issues callback optional) - event row creation per new workflow Plus a worker test deflake: test_stolen_lock_between_agent_return_and_write was asserting the specific 'lost-lock-at-write' outcome, but under full-suite CPU contention the heartbeat thread can fire between the agent's return and _write_outcome — taking the 'lost-lock' (via lost_lock_event) path instead. Both are valid for this scenario; loosened the assertion to accept either. Total: 310 controller tests; full auto_agents suite 2672 pass. |
||
|
drew
|
337af855f3 |
feat(controller): Phase 1d-3b — per-workflow scheduler + static escalation
When the state machine transitions a workflow into a state that needs a worker (ANALYZING / IMPLEMENTING / REVIEWING / CONFLICT_RESOLVING), the scheduler enqueues a fresh pending workflow_attempts row with the input_payload prefetched. tools/controller/master/scheduler.py: - schedule_next_attempts(engine, prefetch=...) — finds workflows in worker-needing states without a matching pending/in_progress attempt; enqueues one fresh attempt per. Skips: - workflows already with pending/in_progress attempt for the role - ESCALATING at MAX_TIER → resolved to ABANDONED (no enqueue) - prefetch raised → per-workflow isolated failure - Static escalation policy resolved inline: ESCALATING + current_tier → IMPLEMENTING(min(tier+1, MAX_TIER)) OR ABANDONED. Both produce controller_events transition rows with the appropriate v9 event name (escalate_next_tier_available / escalate_max_tier_exhausted). - State→role table: ANALYZING → estimator, IMPLEMENTING → implementer, REVIEWING → reviewer, CONFLICT_RESOLVING → conflict_resolver. - PrefetchCallback is parameterized; production wires it to the Forgejo prefetch (Phase 1d-3c), tests inject a fake. - attempt_number monotonically increments via COALESCE(MAX, 0) + 1. - DEFERRED to Phase 1d-3c: actual Forgejo prefetch implementation (this commit ships the scheduler skeleton + the prefetch callback contract). 20 new tests: - state→role parametrization (4 states × matching role) - no-double-enqueue (3 cases: pending blocks, in_progress blocks, different-role doesn't block) - escalation (4 cases: tier 0→1, 1→2, MAX→ABANDONED, event row content) - prefetch raises → per-workflow skip - non-schedulable states parametrized (7 cases: DISCOVERED, AWAITING_CI, MERGING, MERGED, ABANDONED, STUCK, CREATED_PR) - attempt_number monotonicity (uses MAX+1) Total: 298 controller tests; full auto_agents suite 2660 pass. |
||
|
drew
|
130bbbf3c5 |
feat(controller): Phase 1d-3a — master main loop (composite tick)
Long-running master orchestrator composing the deterministic per-iteration work shipped in Phase 1d-1/1d-2. One iteration = state-machine tick + reaper + pickup guard, in that order (reasoning in module docstring). tools/controller/master/loop.py: - MasterConfig: tick_interval_s (default 30s), reaper_interval_s (default 60s), pickup_guard_max_pickups (default 3 per v6). - run_master_iteration(engine): single synchronous iteration — composable for tests + master_main_loop. - master_main_loop(engine): runs run_master_iteration on a loop until stop_event fires. Reaper runs less often than tick (every reaper_interval_s, not every tick_interval_s). on_iteration callback gets each MasterTickReport (used by tests + future structured logging). 7 new tests: - run_master_iteration: empty DB no-op; tick advances state (blocked → STUCK); reaper + pickup guard compose (stale in_progress at pickup limit → reaped → STUCK in one iteration). - master_main_loop: runs until stop; empty loop exits cleanly; on_iteration exception doesn't break loop; reaper runs less frequently than tick. Phase 1d-3+ remains for: discovery (Forgejo poll), per-workflow scheduling (enqueue next workflow_attempts after transition), Forgejo writes, MERGING state's merge call, reconciliation, backfill, operator CLI. Total: 278 controller tests; full auto_agents suite 2640 pass. |
||
|
drew
|
b519ebd98f |
feat(controller): Phase 1d-2 — outcome mapper + master tick handler
The master-side state machine driver. When a worker writes a complete
(or failed) attempt, the tick handler picks it up next pass, maps the
outcome to a state machine event, and applies the transition.
tools/controller/master/:
- outcomes.py: map_outcome_to_event() — pure function. Inputs:
role, current_state, output_payload, status, head_sha_advanced,
attempts_remaining_at_tier, conflict_count_at_current_tier.
Returns EventMapResult(event_name|None, reason).
- Implementer: resolved+pushed → implementer_pushed;
resolved+NO push → implementer_competence_failure (worker lied);
rebase-failed / competence-failure / blocked / noop all routed.
- Reviewer: approve → reviewer_approve; request-changes → retry vs
escalate based on attempts_remaining_at_tier; abstain →
reviewer_abstain; comment → no transition (operator review needed).
- Estimator: is_metadata_only → estimator_metadata_only; else
estimator_done.
- Conflict resolver: counts at current tier — 1st → first; 2nd →
second_same_tier (escalate); 3rd+ → three_plus (STUCK).
- Summarizer: doesn't drive state transitions.
- status='failed' policy: contract-violation → pickup_exhausted;
other failed outcomes (worker-internal-error / stale-input /
ttl-insufficient-for-retry / git-clone-failed / lost-lock / etc.)
→ no event (master re-enqueues per the v9 table).
- status='reaped' → no event (pickup guard handled).
- tick.py: run_tick() — one master tick. Queries
workflow_attempts WHERE status IN ('complete', 'failed') AND
finished_at > workflow.last_transition_at (heuristic for "not yet
processed"). Per row:
1. Validate current_state ∈ KNOWN_STATES (per v6 unknown-state
guard); if not, transition workflow → STUCK with reason='unknown-state'.
2. Map outcome → event via outcomes.map_outcome_to_event.
3. If no event, bump workflow.last_transition_at so we don't
re-process forever.
4. Apply event via state_machine.apply_event; IllegalTransition →
STUCK with reason='illegal-event'.
5. Commit transition (workflows.current_state + entered_state_at +
last_transition_at) AND insert controller_events row.
Returns TickReport with attempts_processed + transitions_applied +
transitions_log.
42 new tests:
- outcomes (22): status='failed' policy (3 paths) + each role's
happy paths + edge cases (unknown outcome / missing field /
None payload / unknown role).
- tick (20): implementer transitions (resolved+push → AWAITING_CI,
resolved-no-push → ESCALATING, rebase → CONFLICT_RESOLVING,
blocked → STUCK), reviewer transitions (approve → MERGING,
request-changes → IMPLEMENTING), event row content, no
reprocessing on second tick, unmapped attempts bump
last_transition_at, terminal workflows skipped (4 parametrized),
unknown current_state → STUCK.
Total: 271 controller tests; full auto_agents suite 2633 pass.
|