ae940f45644971d5937b755d2459eceffeb6500a
2443 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 drew
|
ae940f4564 |
feat(controller): Phase 1d-3c-4 — HTTP adapter wiring callbacks to _claim_runtime
The production glue between the controller's callback protocols
(discovery / forgejo_writes / merging) and the existing Forgejo HTTP
client in _claim_runtime. Single build_callbacks(cfg) factory; tests
use the same module with a fake runtime stub.
tools/controller/master/forgejo_http.py:
- ForgejoCallbacks dataclass bundling every callback the controller
needs: list_prs, list_issues, list_comments, post_comment,
get_labels, add_label, remove_label, merge_pr.
- build_callbacks(cfg, runtime=None): wires each callback as a thin
closure over runtime.get/post/delete. Production omits ``runtime``
to use the real _claim_runtime module; tests inject a fake.
- Forgejo path conventions match the API:
GET /repos/{o}/{r}/pulls?state=open
GET /repos/{o}/{r}/issues?state=open&type=issues (excludes PRs)
GET/POST /repos/{o}/{r}/issues/{n}/comments
GET/POST /repos/{o}/{r}/issues/{n}/labels
DELETE /repos/{o}/{r}/issues/{n}/labels/{name} (URL-encoded)
POST /repos/{o}/{r}/pulls/{n}/merge (body: {"Do": "merge"})
- Robust response handling:
- list endpoints: non-200 → empty list; non-list body → empty;
non-dict items filtered out.
- post_comment: 200/201 ok; other → RuntimeError.
- add_label / remove_label: 200/201/204 → True; remove-404 → True
(label already gone = goal achieved); else False.
- merge_pr: returns normalized MergeResponse. On 404, fetches the
PR's actual state (merged=True → pr_state='merged'; state='closed'
→ 'closed'; PR fetch failure or non-200 → leave pr_state=None so
the merging handler defaults to ABANDONED conservatively).
- Any callback exception → synthetic 503 so the merging handler's
retry logic kicks in cleanly.
23 new tests in test_master_forgejo_http.py:
- list_prs (path format, non-200 → empty, non-list body → empty,
filters non-dict items)
- list_issues (type=issues filter)
- list_comments (path format)
- post_comment (201, 200, non-2xx raises, non-dict body)
- labels (get / add 201/500 / remove 204/404/URL-encoded)
- merge (200, 409, 500, callback-raises-as-503, 404+merged,
404+closed, 404+pull-fetch-failure)
Plus a bug fix surfaced by the test_404_with_pull_fetch_failure test:
the PR-state-fetch branch was returning 'open' on a 500 response;
now correctly checks status==200 before inspecting the body.
Total: 366 controller tests; full auto_agents suite 2728 pass.
|
||
|
drew
|
94821d2702 |
feat(controller): Phase 1d-3c-3 — MERGING handler (6-response-shape table)
The master's per-tick handler for workflows in MERGING. Per plan v6+v9
non-blocking retry: state stays MERGING across ticks, retry counter +
backoff tracked in workflows.merging_retry_count and
merging_retry_next_attempt_at.
tools/controller/master/merging.py:
- MergeResponse: normalized {status_code, pr_state, error_message}
the callback returns. status_code drives the 6-response table:
200 → MERGED
409 → IMPLEMENTING(tier_last_succeeded) with reason=
'post-approval-base-conflict'
403 → STUCK ('branch-protection')
422 → AWAITING_CI ('ci-required-status-missing')
5xx (any) → stay MERGING; bump retry_count + schedule
next_attempt_at = now + 2^retry_count seconds
(capped at 60s); STUCK at retry_count >= 5
404 + pr_state='merged' → MERGED ('externally-merged')
404 + pr_state='closed' → ABANDONED ('externally-closed')
404 + no pr_state → ABANDONED (conservative default)
- run_merging_tick(engine, merge, owner, repo) — sweeps workflows
WHERE current_state='MERGING' AND kind='pr' AND owner+repo match
AND (next_attempt_at IS NULL OR next_attempt_at <= now). Per row:
call merge → map response → transition (or schedule retry) + emit
controller_events row.
- Callback failure (callback raises) wrapped as a synthetic 503 so
the retry logic kicks in cleanly.
- Counter resets on non-retry responses (409 / 422) — keeps backoff
fresh for future retries.
- MAX_MERGE_RETRIES = 5; MAX_BACKOFF_S = 60.0.
14 new tests across 7 classes:
- 200 happy path + event row
- 409 → IMPLEMENTING(tier_last_succeeded)
- 403 → STUCK
- 422 → AWAITING_CI + retry counter reset
- 404 paths (merged + closed + no-state default)
- 5xx retry (counter bumped + backoff scheduled +
in-window-skipped + max-retries-STUCK + callback-raises-as-5xx)
- Owner/repo filter (other repo's MERGING untouched)
- kind='pr' filter (issues never processed even if mis-seeded)
Total: 343 controller tests; full auto_agents suite 2705 pass.
|
||
|
drew
|
3e3796d918 |
feat(controller): Phase 1d-3c-2 — Forgejo write helpers (status + labels)
Idempotent status comment posting via fingerprint markers + per-label
no-op-aware adjust. Production wires the callbacks to existing
_status_comments + _claim_runtime helpers; tests use fakes.
tools/controller/master/forgejo_writes.py:
- compute_fingerprint(workflow_id, event_kind, content_key) →
16-char SHA256 prefix. Deterministic; same inputs → same fingerprint.
Different (workflow_id OR event_kind OR content_key) → different fp.
- build_marker(fp) → HTML comment "<!-- controller:fingerprint:abc -->".
Searchable + invisible in Forgejo's rendered UI.
- comment_has_fingerprint(body, fp) → bool. Used by post_status_comment
for the dedup check before posting.
- post_status_comment(): full idempotency protocol —
1. compute fingerprint
2. list_comments callback → check existing for marker
3. if found → return duplicate (no post)
4. else post via post_comment callback
Failure modes:
- list_comments raises → fall through to post (conservative;
fingerprint match on next sweep catches the duplicate)
- post_comment raises → return failed; caller retries
- adjust_labels(add, remove): one-shot get_labels + per-label
add/remove. add-when-already-present → no-op; remove-when-absent
→ no-op. Per-label failure isolated. get_labels failure marks
all requested actions failed (caller retries).
19 new tests:
- fingerprint helpers (5: deterministic, distinct inputs, marker
format, body match, empty body no-match)
- post_status_comment (5: new post, duplicate skip, list-failure
fall-through, post failure, distinct event_kinds get distinct fps)
- adjust_labels (9: empty no-op, add-when-absent, add no-op,
remove-when-present, remove no-op, get failure, per-label
isolation with mixed failures, returning-False failure)
Total: 329 controller tests; full auto_agents suite 2691 pass.
Phase 1d-3c-3 (MERGING handler) + Phase 1c-3 (real OpenCode + MCP
spawn) remain.
|
||
|
drew
|
e8045d4b9b |
feat(controller): Phase 1d-3c-1 — discovery sweep + worker test deflake
Discovery polls Forgejo for open PRs/issues and inserts a fresh DISCOVERED workflow for any entity the controller hasn't seen yet. Same callback-injection pattern as the scheduler's prefetch: tests provide synthetic ListPRsCallback / ListIssuesCallback; production wires them to the existing _review_fetch helpers in a follow-up. tools/controller/master/discovery.py: - run_discovery(engine, owner, repo, list_prs, list_issues=None) — idempotent. Skips existing entities via a one-shot SELECT(kind, entity_number) WHERE owner+repo membership check. Inserts a DISCOVERED workflow + a controller_events 'discovered' row per new entity. Per-callback try/except: a PR-list failure doesn't block issue discovery, and vice versa. - Defensive coerce_int: rejects bools (True is an int subclass — bug class to avoid). Accepts string digits. - Returns DiscoveryReport(prs_seen, issues_seen, new_workflows, existing_skipped, new_entities[]). 12 new tests: - basics (empty, PRs only, issues only, both PRs+issues) - idempotency (2nd sweep skips; PR #42 + issue #42 coexist; other (owner, repo) isolated) - malformed input (non-int + bool skipped; string digit accepted) - callback failures (PR raises → still process issues; issues callback optional) - event row creation per new workflow Plus a worker test deflake: test_stolen_lock_between_agent_return_and_write was asserting the specific 'lost-lock-at-write' outcome, but under full-suite CPU contention the heartbeat thread can fire between the agent's return and _write_outcome — taking the 'lost-lock' (via lost_lock_event) path instead. Both are valid for this scenario; loosened the assertion to accept either. Total: 310 controller tests; full auto_agents suite 2672 pass. |
||
|
drew
|
337af855f3 |
feat(controller): Phase 1d-3b — per-workflow scheduler + static escalation
When the state machine transitions a workflow into a state that needs a worker (ANALYZING / IMPLEMENTING / REVIEWING / CONFLICT_RESOLVING), the scheduler enqueues a fresh pending workflow_attempts row with the input_payload prefetched. tools/controller/master/scheduler.py: - schedule_next_attempts(engine, prefetch=...) — finds workflows in worker-needing states without a matching pending/in_progress attempt; enqueues one fresh attempt per. Skips: - workflows already with pending/in_progress attempt for the role - ESCALATING at MAX_TIER → resolved to ABANDONED (no enqueue) - prefetch raised → per-workflow isolated failure - Static escalation policy resolved inline: ESCALATING + current_tier → IMPLEMENTING(min(tier+1, MAX_TIER)) OR ABANDONED. Both produce controller_events transition rows with the appropriate v9 event name (escalate_next_tier_available / escalate_max_tier_exhausted). - State→role table: ANALYZING → estimator, IMPLEMENTING → implementer, REVIEWING → reviewer, CONFLICT_RESOLVING → conflict_resolver. - PrefetchCallback is parameterized; production wires it to the Forgejo prefetch (Phase 1d-3c), tests inject a fake. - attempt_number monotonically increments via COALESCE(MAX, 0) + 1. - DEFERRED to Phase 1d-3c: actual Forgejo prefetch implementation (this commit ships the scheduler skeleton + the prefetch callback contract). 20 new tests: - state→role parametrization (4 states × matching role) - no-double-enqueue (3 cases: pending blocks, in_progress blocks, different-role doesn't block) - escalation (4 cases: tier 0→1, 1→2, MAX→ABANDONED, event row content) - prefetch raises → per-workflow skip - non-schedulable states parametrized (7 cases: DISCOVERED, AWAITING_CI, MERGING, MERGED, ABANDONED, STUCK, CREATED_PR) - attempt_number monotonicity (uses MAX+1) Total: 298 controller tests; full auto_agents suite 2660 pass. |
||
|
drew
|
130bbbf3c5 |
feat(controller): Phase 1d-3a — master main loop (composite tick)
Long-running master orchestrator composing the deterministic per-iteration work shipped in Phase 1d-1/1d-2. One iteration = state-machine tick + reaper + pickup guard, in that order (reasoning in module docstring). tools/controller/master/loop.py: - MasterConfig: tick_interval_s (default 30s), reaper_interval_s (default 60s), pickup_guard_max_pickups (default 3 per v6). - run_master_iteration(engine): single synchronous iteration — composable for tests + master_main_loop. - master_main_loop(engine): runs run_master_iteration on a loop until stop_event fires. Reaper runs less often than tick (every reaper_interval_s, not every tick_interval_s). on_iteration callback gets each MasterTickReport (used by tests + future structured logging). 7 new tests: - run_master_iteration: empty DB no-op; tick advances state (blocked → STUCK); reaper + pickup guard compose (stale in_progress at pickup limit → reaped → STUCK in one iteration). - master_main_loop: runs until stop; empty loop exits cleanly; on_iteration exception doesn't break loop; reaper runs less frequently than tick. Phase 1d-3+ remains for: discovery (Forgejo poll), per-workflow scheduling (enqueue next workflow_attempts after transition), Forgejo writes, MERGING state's merge call, reconciliation, backfill, operator CLI. Total: 278 controller tests; full auto_agents suite 2640 pass. |
||
|
drew
|
b519ebd98f |
feat(controller): Phase 1d-2 — outcome mapper + master tick handler
The master-side state machine driver. When a worker writes a complete
(or failed) attempt, the tick handler picks it up next pass, maps the
outcome to a state machine event, and applies the transition.
tools/controller/master/:
- outcomes.py: map_outcome_to_event() — pure function. Inputs:
role, current_state, output_payload, status, head_sha_advanced,
attempts_remaining_at_tier, conflict_count_at_current_tier.
Returns EventMapResult(event_name|None, reason).
- Implementer: resolved+pushed → implementer_pushed;
resolved+NO push → implementer_competence_failure (worker lied);
rebase-failed / competence-failure / blocked / noop all routed.
- Reviewer: approve → reviewer_approve; request-changes → retry vs
escalate based on attempts_remaining_at_tier; abstain →
reviewer_abstain; comment → no transition (operator review needed).
- Estimator: is_metadata_only → estimator_metadata_only; else
estimator_done.
- Conflict resolver: counts at current tier — 1st → first; 2nd →
second_same_tier (escalate); 3rd+ → three_plus (STUCK).
- Summarizer: doesn't drive state transitions.
- status='failed' policy: contract-violation → pickup_exhausted;
other failed outcomes (worker-internal-error / stale-input /
ttl-insufficient-for-retry / git-clone-failed / lost-lock / etc.)
→ no event (master re-enqueues per the v9 table).
- status='reaped' → no event (pickup guard handled).
- tick.py: run_tick() — one master tick. Queries
workflow_attempts WHERE status IN ('complete', 'failed') AND
finished_at > workflow.last_transition_at (heuristic for "not yet
processed"). Per row:
1. Validate current_state ∈ KNOWN_STATES (per v6 unknown-state
guard); if not, transition workflow → STUCK with reason='unknown-state'.
2. Map outcome → event via outcomes.map_outcome_to_event.
3. If no event, bump workflow.last_transition_at so we don't
re-process forever.
4. Apply event via state_machine.apply_event; IllegalTransition →
STUCK with reason='illegal-event'.
5. Commit transition (workflows.current_state + entered_state_at +
last_transition_at) AND insert controller_events row.
Returns TickReport with attempts_processed + transitions_applied +
transitions_log.
42 new tests:
- outcomes (22): status='failed' policy (3 paths) + each role's
happy paths + edge cases (unknown outcome / missing field /
None payload / unknown role).
- tick (20): implementer transitions (resolved+push → AWAITING_CI,
resolved-no-push → ESCALATING, rebase → CONFLICT_RESOLVING,
blocked → STUCK), reviewer transitions (approve → MERGING,
request-changes → IMPLEMENTING), event row content, no
reprocessing on second tick, unmapped attempts bump
last_transition_at, terminal workflows skipped (4 parametrized),
unknown current_state → STUCK.
Total: 271 controller tests; full auto_agents suite 2633 pass.
|
||
|
drew
|
976817fa22 |
feat(controller): Phase 1d-1 — state machine + reaper + pickup guard
The deterministic spine of the master controller. State machine is
pure data with 6 load-bearing invariants enforced via property tests.
Reaper resets stale-heartbeat workflow_attempts to pending. Pickup
guard transitions workflows to STUCK when an attempt has been
re-pended too many times without success.
tools/controller/state_machine.py:
- KNOWN_STATES = 12; TERMINAL_STATES = {MERGED, ABANDONED, STUCK,
CREATED_PR}. STUCK's only allowed exit is the operator-driven
operator_unstick event (back to DISCOVERED).
- 32 TRANSITIONS entries covering DISCOVERED → ANALYZING →
IMPLEMENTING ↔ AWAITING_CI / CONFLICT_RESOLVING / ESCALATING →
REVIEWING → MERGING → MERGED. Plus pickup_exhausted exits from
IMPLEMENTING/CONFLICT_RESOLVING/REVIEWING.
- 27 named events with descriptions. apply_event() lookup raises
IllegalTransitionError (lists legal events from current state)
or ValueError on unknown state (per v6 unknown-state guard).
- 6 LOAD-BEARING invariants for v1 (per v9 simplification):
1. no_path_implementing_to_reviewing_skips_ci (Hard Rule #1
constructional fix for the no-mans-land race)
2. terminal_states_have_no_exits (only STUCK→operator_unstick OK)
3. tier_monotonic_non_decreasing
4. every_pr_workflow_includes_reviewing
5. conflict_resolving_bounded (1st→IMPLEMENTING, 2nd→ESCALATING,
3rd→STUCK; structurally encoded)
6. escalation_deterministic
- reachable_from() honors cycles (DISCOVERED ∈ reachable(DISCOVERED)
via STUCK→operator_unstick path; AWAITING_CI self-loops via
ci_flake_retry).
tools/controller/reaper.py:
- reap_stale_attempts(): SELECT in_progress attempts whose
lock_heartbeat_at + lock_ttl_seconds < NOW (per-row TTL respects
per-role differences — estimator 180s, reviewer 720s, tier-2
implementer 2160s). UPDATEs status='pending', clears lock columns,
preserves pickup_count (the pickup guard handles that). Inserts
controller_events row with reason='lock-ttl-expired' per reap.
- Dialect-portable: Postgres uses interval arithmetic; SQLite uses
julianday(). Same logic either way.
tools/controller/pickup_guard.py:
- transition_exhausted_to_stuck(): finds attempts with status='pending'
AND pickup_count >= MAX_PICKUPS (default 3 per v6 blocker fix)
AND workflow not already terminal. Transitions workflow → STUCK,
marks attempt as 'reaped', inserts controller_events with
reason='attempt-pickup-exhausted' + pickup_count + max_pickups.
45 new tests:
- state_machine: basic shape (states partition, every transition uses
known states + defined events), apply_event success/error paths,
events_from + reachable_from helpers (including cycle awareness),
per-invariant zero-violations against the live table, per-invariant
monkeypatch-violations to prove the checks catch the bug class they
claim to, parametrised sanity check "every non-terminal can reach
some terminal".
- reaper: empty DB / fresh heartbeat / stale heartbeat reaped /
per-row TTL respected / event row created / only-in-progress
reaped / multiple stale attempts.
- pickup guard: empty DB / below limit / at limit / in-progress not
checked / terminal workflow skipped / event payload content /
default max_pickups matches v6.
Total: 229 controller tests; full auto_agents suite 2591 pass.
|
||
|
drew
|
30dfd92021 |
feat(controller): Phase 1c-2 — workspace + session sidecar + orphan janitor
Per-PR workspace umbrella manager (per plan v6) + the worker.session
sidecar that tracks live OpenCode session metadata for orphan cleanup
+ the startup janitor that sweeps orphaned workspaces from previous
worker crashes.
tools/controller/worker/:
- workspace.py: PerPRWorkspace + WorkspaceIdentity (frozen dataclass
yielding the canonical pr-{owner}-{repo}-{N} dir name). ensure_present
is idempotent. clone_if_absent runs `git clone --no-single-branch`
if .git/ missing; idempotent on re-call. fetch_and_validate runs
`git fetch origin --prune`, compares origin/<ref> to expected
head_sha, raises StaleInputError on mismatch (v6 stale-input fix),
then `git reset --hard <expected_sha>` + `git clean -fdx` to wipe
worktree residue from prior attempts. remove() rm -rf's the
workspace.
- session_sidecar.py: WorkerSession frozen dataclass +
write_sidecar (atomic via tmp+rename+fsync) + read_sidecar (tolerates
missing/empty/malformed/wrong-shape gracefully). Sidecar captures
opencode_server_url / session_id / subprocess_pid /
spawned_by_controller_pid / instance_id / spawned_at.
- janitor.py: sweep_orphans pre-queries the DB once for the set of
live instance_ids (workflow_attempts.status='in_progress'), then
scans workspace_root for pr-* dirs. For each:
- no sidecar → just delete (crashed pre-spawn)
- sidecar's instance_id is in live set → preserve (active worker)
- else → orphan. Try cancel_callback (production wires to OpenCode
cancel API); fall back to SIGTERM/grace/SIGKILL on the
subprocess_pid. Then delete sidecar + workspace. JanitorReport
summarises each sweep for structured logging.
Key v6 design points implemented:
- worker.session sidecar atomicity → orphan detection is robust
against partial writes (e.g., worker crashed mid-spawn).
- Cancel-callback-then-SIGKILL fallback → production prefers the
graceful OpenCode-side cancel; tests inject a fake.
- _kill_with_grace returns True after SIGKILL delivery; zombie
reaping is the parent's responsibility, not the janitor's.
27 new tests:
- WorkspaceIdentity (format + frozen)
- PerPRWorkspace paths + ensure_present idempotency
- clone_if_absent (creates worktree, idempotent, raises without URL)
- fetch_and_validate (matching sha resets clean; mismatched raises
StaleInputError; unknown ref raises)
- Sidecar I/O round-trip + atomicity + missing/empty/malformed handling
- Janitor: empty root / no-sidecar / active-lock / orphan-with-dead-pid
/ orphan-with-live-pid / cancel-callback (3 sub-paths: ok / fails /
raises) / mixed-workspaces / non-pr-skip
Total: 184 controller tests; full auto_agents suite 2546 pass.
|
||
|
drew
|
eab476e48e |
feat(controller): Phase 1c — worker controller skeleton
The dequeue+lock+heartbeat+runner+loop machinery. Production
OpenCode + MCP invocation slots in via the agent_runner callable
(Phase 1c-2). This commit is the structural foundation:
tools/controller/worker/:
- identity.py: build_instance_id() → "{hostname}/{pid}/{worker_uuid}"
per plan v9 (slash delimiter; IPv6-safe; uuid4 prefix for
per-instance uniqueness).
- heartbeat.py: Heartbeat thread that updates lock_heartbeat_at
every interval (default 30s). v9 simplified: TTL-only (no activity
tracking). UPDATE … WHERE locked_by_instance=us; rowcount=0 →
lost_lock_event.set() and thread exits, letting reaper handle it.
- runner.py: run_one_attempt() drives one attempt end-to-end.
Starts heartbeat → invokes agent_runner → on success writes
status='complete' + output_payload; on WorkerError writes
status='failed' with outcome label; on WorkerLostLock or detected
stolen-lock-at-write returns aborted (no DB write — reaper has
already re-pended). Defense-in-depth: even if agent returns
successfully, lost_lock_event.is_set() check skips the write.
- loop.py: worker_main_loop() polls the DB for pending attempts up
to MAX_CONCURRENT_WORKERS_PER_MACHINE, submits each to a
ThreadPoolExecutor. Honors stop_event for graceful shutdown
(drains in-flight before exit).
tools/controller/db/session.py: StaticPool for in-memory SQLite so
the heartbeat thread + runner write + dequeue all see the same DB
(without this, ":memory:" gives each connection an independent DB).
16 new tests in test_worker.py: instance ID format/uniqueness;
heartbeat tick (hold + steal); runner happy path; 5 error paths
(worker error / unexpected exception / WorkerLostLock raised /
stolen lock at write / lost_lock_event set defense-in-depth); 4
loop scenarios (single attempt, role filter skip, empty queue
exit-on-stop, explicit instance_id).
Total: 157 controller tests; full auto_agents suite 2519 pass.
|
||
|
drew
|
36b133ec5e |
feat(controller): Phase 1b — DB schema + dequeue helper + payload guard
Five SQLAlchemy 2.0 declarative models implementing plan v6/v9's
unified workflow schema. Cross-dialect (SQLite for tests + local dev,
Postgres for multi-machine production). Lock columns on
workflow_attempts implement the multi-machine-safe dequeue protocol
(plan v5).
Modules:
- tools/controller/db/models.py:
- Workflow (kind discriminator pr/issue, unique on owner+repo+kind+
entity_number, parent_workflow_id FK for issue→PR linkage)
- WorkflowAttempt (status/locked_by_instance/locked_at/
lock_heartbeat_at/lock_ttl_seconds/pickup_count + CHECK
constraints on status enum and pickup_count≥0; partial indexes
on the pending/in_progress/complete hot paths)
- ControllerEvent (Forgejo-write replay support kept in-schema even
though v9 simplified to Forgejo-first protocol; allows v3-style
upgrade later without migration)
- FlakeHistory (composite PK; supports the v6 flake-learning
heuristic)
- CIObservation (raw CI state history; 90-day retention to be
enforced by a sweep task)
- AutoincrementPk variant (Integer on SQLite where it autoincrements
via rowid; BigInteger on Postgres for BIGSERIAL); JsonColumn
variant (JSON on SQLite, JSONB on Postgres)
- tools/controller/db/session.py: build_engine (per-dialect tuning —
SQLite WAL + foreign_keys + busy_timeout; Postgres pool_pre_ping);
create_all (idempotent); session_scope (transactional context).
- tools/controller/db/dequeue.py: dequeue_one (one row atomically;
Postgres path uses SELECT FOR UPDATE SKIP LOCKED, SQLite path uses
UPDATE-WHERE-id-IN-SELECT-LIMIT-1 with RETURNING). Bumps pickup_count
on dequeue; respects max_pickups guard (default 3 per v6 blocker fix).
Returns DequeueResult dataclass with role/tier/pickup_count and
reason on miss.
- tools/controller/db/payload_guard.py: enforce_input_payload_size
with 4MB cap and 5-step truncation priority (older_summary → oldest
verbatim → comments → full_diff → CI failure excerpts). Raises
PayloadTooLargeError after all steps exhausted; master maps to
workflow STUCK with reason='input-too-large'.
- pyproject.toml: new optional extras `controller-db` pinning
sqlalchemy + psycopg2-binary (latter installed only for prod
multi-machine deploy; tests use stdlib sqlite3 via SQLAlchemy's
SQLite dialect which is already pulled in transitively via alembic).
37 new tests across test_db_schema/dequeue/payload_guard; 141
controller tests total; full auto_agents suite 2503 pass (no
regressions).
|
||
|
drew
|
eebb5718a8 |
feat(controller): Phase 1a — 5 response-builder MCP servers
Per-attempt MCP subprocesses that enforce V1 contract invariants at
construction time. Worker LLM calls builder tools incrementally; the
MCP validates each call against the schema + cross-field invariants;
`{role}_finalize()` emits canonical Pydantic-validated JSON to stdout
for the worker controller to read (Phase 1c). Defense-in-depth: the
controller strict-parses whatever finalize emits.
Builders shipped:
- reviewer_builder: 9 tools. Auto-acks all CISummary gates as passed
at start; reviewer only calls record_gate to discuss specifics.
reviewer_override_gate requires ≥20-char justification. Approve
with any failed gate is refused with an actionable error pointing
at the override path. Request-changes requires ≥1 blocking issue.
Verdict-vs-blocking-issues invariant checked at finalize.
- implementer_builder: 7 tools. Outcome-specific finalize invariants:
resolved → ≥1 commit + ≥1 file; blocked → ≥1 blocker; noop → no
commits/files/blockers.
- estimator_builder: 4 tools. Lightweight; requires
recommended_tier + confidence + reasoning at finalize. Reasoning
capped at 2048 chars.
- conflict_resolver_builder: 8 tools. outcome='resolved' requires
new_head_sha + ≥1 commit + ≥1 file. resolution_strategy enum-checked.
- summarizer_builder: 3 tools. Summary 50-2000 chars (enforced at
MCP layer and Pydantic).
Shared infrastructure:
- _builder_base.py: BuilderState dataclass + invariant guard helpers
(require_started / require_not_finalized) + audit-record-with-summary
+ finalize_and_emit (validates against Pydantic model class,
emits canonical JSON to stdout, marks finalized).
41 builder tests in test_mcp_builders.py (happy paths + every
invariant + outcome-specific paths + audit summarization + JSON
round-trip through Pydantic strict-parse). Plus the existing 62
Phase-0 tests. 103 controller tests total. Full auto_agents suite
(2465 tests) still passes.
|
||
|
drew
|
a10758177c |
feat(controller): Phase 0 foundation — V1 contracts + CI gate enumeration
Lays the foundation for the planned controller + DB-owned PR state machine rewrite (see .drew/controller_state_machine.md). Phase 0a — Pydantic V1 worker I/O contracts (tools/controller/contracts/): - v1.py: ImplementerInputV1/OutputV1, ReviewerInputV1/OutputV1, EstimatorOutputV1, IssueImplementerOutputV1, ConflictResolverInputV1/OutputV1, SummarizerInputV1/OutputV1. Plus CISummary / CIFailure / GateResult / FailedAssertion / FileLocation. All models use extra="forbid" + required output_version Literal discriminator (no defaults). - parse.py: strict_parse() + strict_parse_with_retry() helpers. Corrective prompt template quotes the Pydantic ValidationError + the JSON schema. TTL guard skips retry when remaining time too short. Defense-in-depth around the response-builder MCPs that arrive in Phase 1a. Phase 0b — Static CI parser-coverage discovery: - NOX_SESSION_TO_PARSER map in tools/controller/ci_summary_parsers/. Derived from .forgejo/workflows/*.yml + noxfile.py. Covers ruff, pyright, behave, robot_framework, slipcover, vulture, radon, bandit, semgrep, build. Cron-only sessions (benchmark, benchmark_regression) are in NOX_SESSIONS_SKIP_COVERAGE. - scripts/enumerate-ci-gates.py: reads the workflow YAML, joins against the map, exits non-zero on coverage gaps. Wired into tests so the actual .forgejo/ is regression-guarded. 63 new tests; full auto_agents suite still passes (2424). |
||
|
drew
|
e28b41f7ec |
fix(auto-agents): bump post-push CI verify per-call budget 90s → 180s
Observed healthy slow-runner days where 90 s was insufficient:
the verifier returned ``pending`` and let bad pushes through to
the next dispatcher cycle. 180 s catches the fast-failing checks
(lint/format/push-validation typically fail in 30-60 s) AND gives
headroom for a backlogged runner where those checks may queue for
a minute or two before they actually run.
- Per-call budget: 90 → 180 s (doubles polls-per-window from 9 to
18; 10 s poll interval unchanged).
- Worst-case cycle math updated in the cycle-budget block comment:
5 PRs × 180 s = 15 min (was 7.5 min) of polling per cycle in the
uncapped case. The per-cycle budget (commit
|
||
|
drew
|
26f6fe43e5 |
fix(auto-agents): fork-workflow — fetch fork master before force-with-lease
``tools/duplicate_prs_to_fork.py:reset_fork_master`` was using ``git push --force-with-lease`` to reset the fork's master to match origin/master, but on a fresh clone the lease check has no local ``refs/remotes/fork/master`` to compare against. Git refuses with ``! [rejected] origin/master -> master (stale info)``. Live-observed 2026-05-17 on a fresh clone of the auto-agents working directory. Fix is one extra ``git fetch fork master`` before the push so the lease ref is populated, then the ``--force-with-lease`` check has something to compare against and the push succeeds. Idempotent — re-running ``reset_fork_master`` is still safe; the fetch just re-affirms the remote ref. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
91073ef104 |
chore(auto-agents): rebind default + tier-0 to local-claude/claude-haiku-4-5
Both ``.opencode/models/default.txt`` (the agent default model when
no tier slot is specified) and the tier-0 entry in
``.opencode/models/tiers.yaml`` were pointing at OpenAI's GPT-5
family (gpt-5-mini and gpt-5-nano respectively). Rebinds to
``local-claude/claude-haiku-4-5`` to keep the default workload on
the local proxy at the per-million rate:
$1 in / $5 out / $0.10 cached (Haiku 4.5)
versus
$0.25 in / $2.00 out / $0.025 cached (gpt-5-mini)
$0.05 in / $0.40 out / $0.005 cached (gpt-5-nano)
Haiku-4-5 is more expensive per-token but Anthropic's prompt
caching (90% off cached input) typically wins for workloads with
large repeated system prompts — which the auto-agents pipeline
absolutely has. The actual cost/PR comparison will be visible
once a few cycles have run through the now-functional cost
dashboard (telemetry commit
|
||
|
drew
|
b80f5f7c3a |
fix(auto-agents): reviewer data-handling — five live-observed regressions
Five independent bug fixes around the reviewer's data pipeline, all live-observed on 2026-05-17 run-1 (PR #35 / PR-shaped traffic generally). Bundled because they share the same data-correctness intent and one fixture-update covers two of them. 1) **PR diff: distinguish "fetch failed" from "PR has no changes."** ``tools/_pr_diff.py``. The old code conflated both cases into ``unavailable=True``, which forced ``data_complete=False`` and blocked APPROVED verdicts. Live-observed on PR #35 created by the new_issue worker without any code changes — Forgejo returns HTTP 200 + empty body for a head==base PR, and the reviewer was incorrectly told the diff was unavailable. 2) **PR-state cache: refresh body + labels on the unchanged- ``updated_at`` branch.** ``tools/_pr_state_cache.py``. Forgejo label add/remove mutations do NOT bump ``updated_at``, so the warmer's cached PR object would carry stale labels for as long as the PR sat idle. Downstream consumers (cycle-cap, claim sweeps, filter exclusions) would never see them. Cheap fix — same row, two extra columns refreshed. 3) **Comments cache: URL-encode the ``since=`` cursor.** ``tools/_pr_comments_cache.py`` + matching test update. The ``+`` in ``+00:00`` decodes to a space on Forgejo's query- string parser, producing 422 errors. Live-observed on PR #35 run-1: 6 consecutive 422s on the same clean ``+00:00`` cursor before the cache backed off for 30 min. ``urllib.parse.quote`` with ``safe=''`` quotes every non-alphanumeric so ``+`` → ``%2B``, ``:`` → ``%3A``. Test updated to ``unquote`` the captured path before substring-matching. 4) **Reviewer prompt: surface the clone fallback.** ``tools/_review_prompt.py``. Adds an inline note in the pre-fetched diff section explaining the two diff sources (inline-truncated vs pre-cloned worktree) and the ``REVIEW_DISPATCHER_DIFF_MAX_BYTES`` cap. Closes a reviewer- side confusion where the model didn't know it could read source files from disk when the inline diff was truncated. 5) **Reviewer agent contract: truncated diff + clone IS data-complete.** ``.opencode/agents/pr-review-worker.md``. The prior wording said ``truncated=True`` forced ``data_complete= False`` and blocked APPROVED — but with the pre-cloned worktree available, the reviewer DOES have full code access and APPROVED should remain valid. Updated guidance now distinguishes "truncated but clone present" (APPROVED OK) from "truncated AND no clone" (COMMENT / REQUEST_CHANGES only). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
a6610a022c |
feat(auto-agents): launcher sidecars for conflict-drive + merge-drive
Two new long-lived sidecars folded into the dispatcher launcher, each gated by an env var for safe enable/disable: **Conflict driver** (``DISPATCHERS_DISABLE_CONFLICT_DRIVE``, default enabled). Watches Forgejo for ``auto/needs-conflict-resolution`` labels, dispatches ``conflict-resolver-worker`` (the new agent; bound to ``local-claude/claude-opus-4-6`` for high-quality rebases) to rebase + resolve conflicts, force-with-leases the result, and clears the label. Decoupled from approval — keeps PRs mergeable so the merge driver doesn't have to wait at approval time. **Merge driver** (``DISPATCHERS_DISABLE_MERGE_DRIVE``, default DISABLED in this test launcher to avoid accidental merges during validation runs). Watches Forgejo for APPROVED PRs without blocking labels, rebases against current master, runs the local CI gate, merges via squash with force-with-lease semantics. Terminal stage of the pipeline. Both sidecars follow the existing warmer pattern: best-effort respawn on failure, no contribution to the dispatcher crash-loop budget, clean shutdown on launcher exit. PRs that get ``auto/needs-conflict-resolution`` would sit forever without the conflict-driver sidecar — the implementer and reviewer don't act on that label. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
2c43179e70 |
feat(auto-agents): R3.7 post-push CI verification + per-cycle budget
Closes the local-vs-remote CI divergence the 2026-05-17 run-7
observation exposed: the implementer worker's ``ci_run_local_gate
--fast`` doesn't catch what remote CI does (test sharding,
parallel-job orchestration, remote-only timing). So the worker
could push a commit + claim ``outcome=resolved`` even when remote
CI would reject it; the dispatcher would treat the cycle as a
success and not re-dispatch.
With this gate, the dispatcher is the source of truth for "did
this PR actually pass CI."
How it works:
- Runs ONLY when the worker claims ``outcome=resolved`` AND
``head_sha_advanced is True`` (a real push happened).
- Polls Forgejo's combined CI status on the post-session head_sha
every 10 s for up to 90 s.
- If CI lands in {failure, error}: rewrites parsed_json's
``outcome`` to ``post-push-ci-failed`` so
``_implementer_escalation.decide`` routes it as a failure
(ESCALATE / EXHAUSTED). Stashes the original outcome +
failing-context list under ``_post_push_ci_verification`` for
telemetry.
- If CI is pending after the budget: outcome unchanged (don't
penalise the worker for slow CI; next dispatcher cycle
re-classifies).
- If fetch fails: outcome unchanged (Forgejo flake protection —
"I couldn't check" must not equal "the worker lied").
- Dry-run short-circuits to no-op so ``--dry-run`` cycles don't
burn 90 s polling.
Per-cycle polling budget:
Without a cycle-wide cap, a dispatcher cycle processing N PRs all
landing in ``outcome=resolved`` after a push could spend
``N * 90 s`` polling — at 5 PRs/cycle that's 7.5 min eating the
dispatcher cycle budget. Added a sliding-window budget:
- ``IMPLEMENTER_POST_PUSH_CI_CYCLE_BUDGET_S`` (default 300 s) caps
total polling time across all verify calls in one dispatcher
cycle.
- ``IMPLEMENTER_POST_PUSH_CI_CYCLE_RESET_AFTER_S`` (default 120 s)
auto-resets the budget after that much idle — naturally fires
on cycle boundaries without the dispatcher's outer loop having
to call a reset hook.
- Per-call budget is also capped by remaining cycle budget so a
near-exhausted cycle doesn't get blown through by one fat call.
- ``reset_post_push_ci_cycle_budget()`` exposed for tests and any
future dispatcher hook that wants to reset explicitly.
Tests:
- 22 tests in ``TestPostPushCIVerify`` (the pre-existing class)
cover happy path, rewrite-on-fail, head_sha_advanced gating,
feature-flag short-circuit, transport flake handling, failing-
contexts extraction, etc.
- 4 new tests in ``TestPostPushCIVerifyCycleBudget`` pin the
budget contract: exhausted budget skips verification, idle
auto-reset works, explicit reset zeroes state, per-call cap
respects remaining cycle budget.
Tuning notes:
- 90 s per-call: enough to catch fast-failing checks (lint/format
fail in 30-60 s typical) without dominating the cycle.
- 10 s poll interval: 9 polls per per-call budget. Forgejo's CI
status endpoint is fast (< 1 s typical).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
205bf56eec |
feat(auto-agents): cycle-cap safety mechanism + tests + hardening
R3.4 (commit
|
||
|
drew
|
6879c68f4e |
fix(auto-agents): R3.6 — CI failure-log cache re-fetches when new failures appear
Live-observed bug from PR #40 run-7 (2026-05-17): a SHA's CI is NOT terminally frozen at the first cache write — Forgejo runs checks asynchronously, and a check that's PENDING at fetch time can later transition to FAILURE. The original cache logic marked ``completed=True`` as soon as the currently-failing jobs' logs were fetched cleanly, then refused to ever re-fetch (per-SHA immutability claim). Result observed: PR #40 cache sealed at 20:38 with 1 failing job (push-validation, the only one transitioned by then). Lint and unit_tests transitioned pending→failure later in the same run. The cache was treated as frozen, never re-fetched, and BOTH downstream consumers (reviewer + implementer) saw only push-validation's log. The reviewer's RC review listed three failing checks by name but could only describe one in detail because the prefetch envelope literally didn't have the other two logs. The implementer, even with its R3.4 visibility into the reviewer's review, had the same gap. Fix: when the cached payload says ``completed=True`` AND the caller-provided ``ci_detail`` shows additional failing contexts not in the cached ``failing_jobs``, treat the cache as stale and live-fetch. Same-set comparison = cache still valid. The per-SHA immutability claim still holds in the steady state: once every check has reached a terminal state, the set of failing contexts stops changing, and the comparison becomes a no-op. The fix only triggers re-fetch during the window where checks are still transitioning. New helper: ``_cache_covers_all_current_failures(cached, ci_detail)`` returns True iff every distinct failing-context in ci_detail is already represented in cached.failing_jobs. When ci_detail is None (caller couldn't provide), conservatively assumes the cache is still valid (preserves prior behaviour for that path). Tests: - ``test_completed_cache_reinvalidated_when_new_failures_appear`` pins the regression: seeded cache with 1 failure, current ci_detail has 2, re-fetch must occur and capture both contexts. - ``test_completed_cache_still_served_when_ci_detail_unchanged`` counter-test: same failing set = cache preserved, no live call (the live-fetch path is stubbed to raise so any live call surfaces as a hard test failure). Full auto_agents suite: 2307 passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
f656f710db |
fix(auto-agents): R3.4 follow-up — wire review sections into pr_fix prompt
Live-observed regression from run-6 (2026-05-17): R3.4 added the data-side plumbing for the implementer to see reviewer feedback (``comment_reviews`` field, sentinel persistence, fetch in ``fetch_pr_fix_context``), but the PROMPT-side rendering was only added to ``build_request_changes_prompt`` — not to ``build_pr_fix_prompt``. So a failing-CI PR's sentinel had the reviewer's feedback, but the worker's prompt didn't render it. Verified live on PR #40 in run-6: sentinel comment_reviews: present + completed sentinel request_changes_reviews: 1 review worker prompt: zero review sections Fix: add ``_build_active_reviews_section`` + ``_build_comment_reviews_section`` to ``build_pr_fix_prompt``'s section list, mirroring the assembly in ``build_request_changes_prompt``. The worker on a failing-CI PR now sees both the blocking RC reviews (if any) and the advisory COMMENT/APPROVE reviews — closing the architectural silo R3.4 set out to fix. Test: - ``test_pr_fix_prompt_omits_active_reviews`` (which asserted the old pre-R3.4 behaviour) renamed + inverted to ``test_pr_fix_prompt_includes_review_sections``. Pins both section headers present in the rendered prompt. Regression-guard: a future edit that removes one but not the other (or removes both via a reverted helper call) would surface here. Full auto_agents suite: 2305 passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
8ef2f949c2 |
fix(auto-agents): R3.5 — cap parity + attempt-label conclusive-outcome gate
Two audit findings from the post-R3.4 walk-through, both addressing gaps where one pool's stop-signal didn't propagate to the other. P0 — Universal triage-label exclusion at the candidate collector ------------------------------------------------------------------ Pre-R3.5 only the reviewer-side Python filter honoured ``auto/needs-human-triage`` (via ``_evaluate_filter``'s ``is_excluded`` check). The implementer (and any other dispatcher that uses ``collect_candidates``) would still pick up triage-labeled PRs because their filter scripts (``list_prs_ci_failing`` etc) had no knowledge of the label. Result: the cycle-cap's "all automation pauses on this PR" signal was half-effective — the reviewer stopped iterating but the implementer kept trying, defeating the cap's purpose. Fix: filter triage-labeled items inline in ``collect_candidates`` so the exclusion applies to every dispatcher routed through that path, regardless of which filter (Python or legacy TS) produced the item. Uses the existing ``_cycle_cap.labels_carry_triage`` helper which handles both Forgejo label shapes (dict-of-name and flat string). P1.1 — apply_attempt_label gated on conclusive outcomes ------------------------------------------------------- ``_post_session_action_with_escalation`` wrote ``auto/last-attempt-tier-N`` unconditionally after the first worker attempt, including on ``timeout`` / ``transport-error`` sessions that never produced a verdict. The next cycle's ``_read_start_tier_from_labels`` then bumped the start tier to N+1 — wasting a tier on a fix the lower one might have handled, AND feeding the estimator's step 2a constraint a false "tier N failed" signal. Fix: gate the label write on ``terminal_state == "completed"`` AND a parsed_json with an outcome field. Environmental failures (network blip, OpenCode 5xx) now leave the prior cycle's tier label intact instead of bumping. An ``INFO`` log fires on the skip path so an operator grepping the journal can correlate "no label written this cycle" with the underlying terminal_state. The in-cycle escalation respawn (site 2 at line 3162) is unchanged — it only fires when ``_implementer_escalation.decide()`` returns ESCALATE, which already accounts for the retry budget and is only reached for conclusive failures. P1.2 — Cross-pool parity test ----------------------------- Two new tests in ``test_dispatch_runtime.py``: - ``test_collect_candidates_excludes_triage_labeled_items`` pins the cross-pool invariant with both label shapes (dict-of-name and flat string). Regression-guard: a future dispatcher that bypasses the collector or weakens the label check here would re-enable the doom-loop pattern. - ``test_collect_candidates_accepts_missing_labels_field`` covers the defensive case — items returned from older list scripts may omit ``labels`` entirely; the filter must treat that as "no labels" and pass the item through, not crash. Tests: 2305 auto_agents passing (+2 new from this commit). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
3f12e4140c |
fix(auto-agents): R3.4 — cycle-cap signature + implementer sees full reviewer record
Two related fixes surfaced by the 2026-05-17 run-5 live observation: 1. **Cycle-cap signature now includes total_reviews count** so the reviewer's COMMENT-only path actually moves the signature. Before: the cap signature was ``sha + (approvals + has_active_RC + has_unaddressed_RC)`` — all three boolean axes ignore COMMENT reviews entirely. After: ``+ total_reviews`` term bumps on EVERY review submission. The ``data_complete=False → COMMENT downgrade`` path the reviewer takes in low-context cycles now reflects in the signature, so the cap stops firing falsely. Without this, run-5 observed 4 fresh PRs hitting count=5 in ~10 minutes — the reviewer was successfully posting reviews every cycle but the cap saw "no change" because COMMENT-only reviews don't bump approvals_count, has_active_RC, or has_unaddressed_RC. 2. **Implementer now sees ALL reviewer feedback**, not just active REQUEST_CHANGES. Before: ``fetch_pr_fix_context`` passed ``include_active_reviews=False`` so failing-CI PRs reached the implementer with zero reviewer data. ``fetch_request_changes_pr_context`` only included active RC reviews — COMMENT-only feedback was invisible in both code paths. After: ``fetch_pr_fix_context`` also includes reviews, AND the fetcher now partitions reviews into TWO buckets — the existing ``request_changes_reviews`` (active blocking RC, unchanged semantic) and a new ``comment_reviews`` field carrying every non-dismissed non-RC review (COMMENT / APPROVE). Both are persisted in the PR-context sentinel and exposed via ``implementer_pr_context.py``'s ``comment_reviews`` field. New prompt section ``## Pre-fetched reviewer comments and approvals`` renders the comment_reviews bucket with author / event / commit / body / inline comments + a postscript marking them as ADVISORY (not blocking, unlike the existing RC section). This closes the architectural gap where the reviewer and implementer pools could work in silos on the same PR — the reviewer's substantive prose feedback now reaches the implementer regardless of which work-group routed it. Files touched: - tools/_pr_classification_cache.py — total_reviews in classify_pr + reactivity composite; docstring updated. - tools/_implementer_prefetch.py — new comment_reviews + comment_reviews_completed fields; fetcher partitions reviews once; fetch_pr_fix_context now includes reviews. - tools/_implementer_prompt.py — _build_comment_reviews_section; wired into prompt assembly between RC and PR-comments sections. - tools/_pr_context_sentinel.py — comment_reviews in _to_dict. - tools/implementer_pr_context.py — comment_reviews accessor for the worker's handoff read path. - tests/auto_agents/test_pr_context_sentinel.py — expected_value_keys + fixture + round-trip test updated. - .opencode/agents/estimator-implementation.md — restored canonical section header levels (####) for downstream test compatibility after R3.1 rewrite. Full auto_agents suite: 2303 passing (+12 from this and adjacent work, none broken). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
62d4e8f07d |
fix(telemetry): cost dashboard now computes real USD totals
Three intertwined bugs caused every Cost-tab row to display \$0 even after the scraper started writing real token data: 1. **Lookup key mismatch.** ``_cost_usd`` looked up bare ``model`` but ``_DEFAULT_PRICES`` was keyed by ``provider/model`` — every priced model silently missed. Fixed by adding ``_price_key`` and a fallback chain: ``provider/model`` → bare ``model`` → ``_unknown``. 2. **SQL grouped by model only.** Same modelID served by two providers (e.g. ``claude-opus-4-6`` via Anthropic direct vs a local proxy) at different rates was conflated into one row. Fixed: ``GROUP BY model, provider`` in ``_api_cost`` + ``provider`` returned in each row. 3. **Math convention mismatch.** ``_cost_usd`` did ``(tokens_in - cached) * in_rate`` assuming ``tokens_in`` was total input. But the scraper records ``tokens_in`` as OpenCode's ``info.tokens.input`` (fresh, non-cached), so ``tokens_in - cached`` went negative whenever cache reads exceeded fresh input — which is the common case with Anthropic prompt caching. Fixed: no subtraction; the three populations bill at their three rates. Pricing seeded for the 8 models the scraper has actually observed (``_DEFAULT_PRICES`` corrected from stale Opus-3 numbers + new entries for the Haiku 4.5 / GPT-5 family / CleverThis HF endpoints): | Provider | Model | in | out | cached_in | |--------------|---------------------------|-------|-------|-----------| | local-claude | claude-opus-4-6 | 5.00 | 25.00 | 0.50 | | local-claude | claude-sonnet-4-6 | 3.00 | 15.00 | 0.30 | | local-claude | claude-haiku-4-5 | 1.00 | 5.00 | 0.10 | | openai | gpt-5 / gpt-5-codex | 1.25 | 10.00 | 0.125 | | openai | gpt-5-mini | 0.25 | 2.00 | 0.025 | | openai | gpt-5-nano | 0.05 | 0.40 | 0.005 | | CleverThis-* | (HF endpoints, advisory) | 0.50 | 1.00 | — | Operators can override without touching code via ``.opencode/telemetry/prices.json`` (added; same keying convention). ``_load_prices`` now skips ``_comment`` / ``_last_updated`` / ``_sources`` metadata keys so docs in the JSON don't pollute the table. Smoke against current ``llm_activity`` (3743 turns, 450 archives): total USD over the lifetime window is now \$118.06, with all 8 models showing as priced. Tests pin all three regressions: provider-qualified lookup, bare-model fallback, no-subtraction math, GROUP BY (model, provider), and the metadata-key filter in ``_load_prices``. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
defea003a4 |
feat(auto-agents): llm_activity.prompt_hash for duplicate-prompt measurement
Adds a SHA-256 hash of each session's first user message to every ``llm_activity`` row so we can answer the question "would response caching for repeat prompts pay back?" with data instead of hypothesis. Schema (v7): - ``llm_activity`` grows a ``prompt_hash`` column (nullable, indexed, NOT unique — duplicates are the measurement signal) - Idempotent ALTER-gated migration; chains cleanly from v5/v6 - Migration test pinned for the v5→v6→v7 walk end-to-end Scraper: - ``_first_user_prompt_hash`` hashes the concatenated text parts of the session's first user message; that hash is applied to every assistant turn from the same session, so ``GROUP BY prompt_hash`` measures cross-session duplication, not within-session multi-turn Real-archive smoke (449 archives / 3740 turns): - 448 distinct sessions → 436 distinct prompt_hashes - 12 sessions share a prompt with another session (2.7% redundancy) - Confirms the hypothesis: workers have per-cycle entropy in their prompts; generic response caching wouldn't pay back. The estimator's existing ``(pr_number, head_sha)`` cache covers the only place exact-prompt repeats happen by design. Takes effect on the next pipeline run — existing rows stay NULL. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
88b9373fa9 |
feat(auto-agents): LLM activity scraper + cost dashboard wiring
Closes the cost-tracking instrumentation gap: the telemetry console's Cost tab read from an empty ``llm_activity`` table because nothing in production wrote to it. The scraper walks the OpenCode session archives that ``_opencode_worker`` already writes (including subagent trees via the BFS-walked ``parentID`` chain) and emits one row per assistant turn. Folded into the existing PR-State Warmer loop so it runs on the same 30s cadence without spinning a new sidecar. Schema (v6): - ``llm_activity`` grows ``session_id`` / ``message_id`` / ``provider`` / ``parent_session_id`` / ``subagent_depth`` columns - Partial UNIQUE INDEX on ``message_id`` makes re-scrapes idempotent - v5→v6 migration ALTER-gated on column existence (safe to re-run) Scraper (``tools/llm_activity_scraper.py``): - Reads ``.dispatcher-logs/sessions/*.json``, one row per assistant turn - Folds reasoning tokens into ``tokens_out`` and cache-write into ``tokens_in`` (preserves raw breakdown in ``raw`` JSON for future cost-calc refinements) - Normalises ``subagent_depth=0`` at top level so dashboards can filter ``WHERE subagent_depth > 0`` cleanly - Batch INSERT OR IGNORE via new ``PipelineCache.upsert_llm_activity_batch`` — one fsync per archive, not per turn Warmer integration: - First tick: full backfill of the archive directory - Subsequent ticks: 1h lookback via ``since=`` filter - Scraper failures are logged and swallowed — PR-state job stays load-bearing and unaffected - ``LLM_ACTIVITY_SCRAPER_DISABLE=1`` env kill switch Renames (mechanical, atomic): - ``tools/_forgejo_cache.py`` → ``tools/_pipeline_cache.py`` - ``ForgejoCache`` class → ``PipelineCache`` - Both reflect the module's broader scope (Forgejo data + pipeline telemetry tables); on-disk filename ``forgejo.sqlite`` and ``FORGEJO_*`` env vars are kept for compatibility Verified end-to-end on real archives: 435 archives → 3595 turns ingested (2873 from subagents) across 8 models / 5 providers / 9 PRs. Re-runs insert 0, dedup 3595. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
ee120011c8 |
feat(auto-agents): R3.2 — estimator cache invalidation on failure + polish
Addresses the P0/P1 + P2 items from the post-R3.1 review pass. What changed ------------ 1. **Cache invalidation on non-success worker outcome** (P2 #24, doom-loop defense). New ``_invalidate_estimator_cache_on_failure`` helper is called from ``_dispatch_post_session_action`` BEFORE the inner action runs. When the cycle's worker terminal_state isn't "completed" with ``outcome=resolved``, the cache entry for that PR is dropped — so the NEXT cycle re-asks the estimator with the freshly-updated attempt-history digest in the prompt. Without this, a stuck PR whose ``auto/last-attempt-tier-N`` label fails to land (the run-15 doom-spiral failure mode) would re-serve the same cached tier recommendation across cycles until the cache TTL (default 1 h) expired. The cache now ALSO closes the gap that amplified the failure mode the estimator's step 2a cross-cycle constraint exists to defend against. Success path preserved: ``outcome=resolved`` keeps the cache so the (rare) "same PR / same SHA next cycle" case still hits. 2. **``estimator-implementation.md`` prose updated** (P0): - Line 203 dropped the obsolete ``task_prompt`` reference (the retired tier-dispatcher's parameter). The body IS the prompt directly post-R3 — the agent reads sections inline at the top level. - Added a one-paragraph note documenting the cache + invalidation contract so operators / future readers know the agent's recommendation may be cached for up to 1 h and is invalidated on worker failure. Reinforces why step 2a (refuse-failed-tier constraint) is load-bearing — when the cache invalidates, the estimator MUST recommend a strictly-higher tier per the attempt-history digest. 3. **Code polish** (P1): - Hoisted ``import time`` to module level (was inline in two cache helpers — no circular-import reason for the local form). - Added explicit INFO log when the estimator returns ``is_confident: false`` so operators see the no-confidence path in the journal alongside the other estimator outcomes. - Expanded the ``_ESTIMATOR_CACHE`` comment to document all three invalidation modes (head_sha change, TTL, failure) AND the single-threaded-dispatch assumption (with a future-async warning for the lock requirement). - New ``_estimator_cache_drop_pr(pr_number)`` helper for single-PR invalidation (used by the failure path); the existing ``_estimator_cache_clear()`` still wipes everything. 4. **New test coverage** (+11 tests): - ``TestEstimatorCacheInvalidationOnFailure`` (7 tests): drops on unresolved / timeout / transport-error / synthesized failure outcomes; preserves on resolved; no-op for issue items (no head_sha); end-to-end via ``_dispatch_post_session_action``. - ``TestEstimatorResultCache::test_cache_does_not_bleed_across_pr_numbers`` — same head_sha + different PR must NOT share a cache slot. Full auto_agents suite: 2291 passing (+11 from new tests). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
12286b8a3f |
feat(auto-agents): R3.1 — estimator wrapper, cache, prose refinements
Three follow-ups to the R3 wrapper-chain retirement (
|
||
|
drew
|
80d61de942 |
feat(auto-agents): R3 wrapper-chain retirement — direct task-implementor variants
Eliminates the remaining LLM wrapper chain (``tier-dispatcher`` +
``tier-{min,0,1,2}`` selectors) between the Python dispatcher and the
``task-implementor`` worker. Follows the R2 implementation-worker
retirement (
|
||
|
drew
|
b8c1e49032 |
feat(auto-agents): R3 prep — generate per-tier task-implementor variants
Prep step for retiring the tier-dispatcher + tier-N wrapper chain (R3, follow-up to R2's implementation-worker retirement at |
||
|
drew
|
2658deee94 |
feat(auto-agents): PR State Warmer substrate + supporting infra
Adds a long-lived sidecar (pr_state_warmer.py) that polls Forgejo's /pulls endpoint every 30s and writes the full PR snapshot to a shared SQLite store, eliminating the dispatcher's per-cycle cold-cache stalls (24-30s rebuilds on flaky cycles) and the silent 50-PR pagination cap on the legacy single-page fetch. Substrate - tools/_pr_state_cache.py — SQLite store with (owner, repo) PK, WAL mode, additive v2→v3 migration (comments_refreshed_updated_at), bounded fcntl.flock migration lock, threading.Lock for per-process init, @_with_reheal decorator (catches OperationalError no-such- table + DatabaseError corruption with file quarantine), atomic TEMP-table chunking for >32k seen-set, _normalize_updated_at to canonicalize Forgejo tz-marker drift - tools/pr_state_warmer.py — poll/upsert/vanish/comments-refresh loop with fcntl.flock singleton (rejects second warmer), bounded comments-refresh cap, persistent deferral via SQL pending query, PermissionError-tolerant lock setup, cold-start log suppression - tools/_pr_classification_cache.py — three-layer fall-through (warmer cache → list cache → live fetch) with staleness gate (PR_STATE_WARMER_STALE_AFTER_S floored at 30s in prod) Comments cache hardening - Bot-filter at write time drops bot status/claim/release/sentinel while preserving **Implementation Attempt** markers (94.6% reduction on bot-heavy PRs like #30's 19k-comment thread) - _normalize_since_cursor strips microsecond precision before building ?since= query (fixes the live-observed Forgejo HTTP 422 bug on PRs #25 + #28); handles uppercase Z, lowercase z, ±HH:MM offsets (including non-zero like +05:30), naive ISO - Lazy migration of legacy null-key by_author entries on _read_cache - _newest_cursor walks tail-back skipping malformed entries Supporting infrastructure (cumulative dmpipeline-v2 work) - Telemetry server: SSE live tail, run-sessions enumeration, cost/token tracking, app.js UI rewrite with collapsible sections - MCP servers (mcp_ci_server, mcp_forgejo_server, mcp_git_server, mcp_handoff_server, mcp_graphify_server) for opencode worker context access - Live log writer (tools/live_log_writer.py) — SSE-streaming dispatcher event log - Tier-dispatcher escalation flow with prompts trimmed for budget - Shared bot-logins resolver (tools/_bot_logins.py) replacing two drift-prone copies - token_usage_audit.py for opencode cost analysis Tests - 2259 passing across 65 changed/new files - New suites: test_pr_state_cache, test_pr_state_warmer, test_pr_state_warmer_integration, test_pr_classification_cache, test_pr_list_cache_backoff, test_mcp_* (5 servers), test_live_log_writer_sse, test_telemetry_run_sessions, test_review_post_ready_label - Test_pr_comments_cache expanded with bot-filter coverage, cursor-normalization regression pins, format-drift, atomicity, failed-comments-not-stamped (silent-data-loss class) - Parametrized @_with_reheal coverage across 7 wrapped APIs - Real fault-inject atomicity test for chunked mark_vanished path via Connection wrapper class - Subprocess-based singleton flock test (cross-process contract) - Event-driven SIGTERM-mid-poll test (no fixed-sleep flake) Architecture notes - Schema v3 migration is additive (ALTER ADD COLUMN); v0/v1 still need destructive rebuild because pre-v2 column shape lacks owner/repo. Cross-process drop-table-ping-pong prevented by the fcntl migration lock + per-process _initialized flag. - Comments-refresh deferral is persistent via comments_refreshed_updated_at column — survives warmer restart, picks up next cycle even if PR didn't change again. Replaces in-memory changed_numbers list. - Rollback path: PR_STATE_WARMER_PREFER=0 bypasses the warmer cache and reverts to live-fetch behavior. PR_STATE_CACHE_DISABLE=1 short-circuits the warmer process at startup. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
dc4e9368f2 |
feat(auto-agents): block-store substrate + DRY refactors
Externalise large prompt sections (PR diff, comments, CI failure
logs, reviews, linked issues) into a cross-process SQLite-backed
block store so the worker can recover original content when an
intermediate tier-* agent's summarisation strips inline sections.
New substrate
-------------
- ``tools/_block_store.py`` — SQLite WAL, per-row 1MB cap, 1h TTL,
janitor (startup + opportunistic per-hour in register()),
threading.Lock around the periodic-sweep gate.
- ``tools/_block_prompt.py`` — registration glue + ``## Available
blocks`` Markdown table renderer.
- ``tools/_prefetch_section.py`` — single-source-of-truth section
registry; collapses the duplicate ``_register_*_blocks`` helpers
the reviewer and implementer previously kept in lockstep.
- ``tools/mcp_block_store_server.py`` — FastMCP wrapper exposing
``block_fetch`` / ``block_list`` / ``block_register`` /
``block_invalidate`` to agents. Uses the expanded ``_mcp_common``
helpers.
- ``tools/_implementer_escalation_helpers.py`` — pure helpers
extracted from ``dispatch_implementer.py`` (~180 lines off the
3284-line file); takes ``claim_runtime`` as a kwarg for clean DI.
DRY refactors
-------------
- ``tools/_backoff.py`` — shared ``Backoff`` dataclass collapses the
three near-identical exponential-backoff state machines in
``_pr_comments_cache``, ``_ci_logs``, ``_pr_classification_cache``.
- ``tools/_mcp_common.py`` — expanded with ``bootstrap_loader``,
``error_envelope``, ``make_main`` so each MCP server's prelude is
three lines.
- ``tools/_pr_diff.build_diff_section_full`` — returns a 4-tuple
including the raw diff body so the reviewer's block-store
registration reuses the bytes instead of doing a second HTTP fetch.
Wiring
------
- ``_review_prompt.build_review_prompt`` builds a ``PrefetchSection``
registry via ``_review_sections``, registers them, and renders the
``## Available blocks`` table at the end of the prompt.
- ``_implementer_prefetch._fetch_pr_context`` /
``fetch_new_issue_context`` build the equivalent registry via
``_implementer_sections`` and stamp ``result.block_refs`` for the
prompt builder to read.
- ``_implementer_prompt`` builders include
``_build_available_blocks_section(result)`` in all three flows.
- ``dispatch_review.main`` + ``dispatch_implementer.main`` call
``_block_store.janitor()`` at startup; the per-call opportunistic
janitor in ``register()`` keeps the file bounded between restarts.
Agent contract updates
----------------------
- ``.opencode/agents/task-implementor.md`` +
``.opencode/agents/pr-review-worker.md``:
- ``block_store*`` permission
- new "Block-store substrate" paragraph explaining
``block_fetch`` / ``block_list`` as the summarisation recovery
path.
Tests
-----
- ``test_block_store.py`` — 48 tests pinning every public contract
(register/fetch/list/invalidate/janitor, key whitelist, TTL,
size cap, WAL durability).
- ``test_block_prompt`` — covered transitively via the e2e test.
- ``test_block_store_recovery_e2e.py`` — builds a real prompt via
``build_pr_fix_prompt``, applies a heading-bounded summariser stub
(``_summarise_inline_sections``), asserts inline content is stripped
yet block keys survive and ``block_fetch`` recovers original content.
Plus a ``block_list`` fallback test for the worst case where the
table itself was summarised away.
- ``test_mcp_block_store_server.py`` — 27 wrapper-contract tests.
- ``test_mcp_block_store_transport.py`` — spawns the actual server
subprocess via ``mcp.client.stdio`` and exercises the JSON-RPC
transport round-trip in ~1.5s. Catches FastMCP schema /
serialisation bugs the in-process tests miss.
- ``test_backoff.py`` — 14 behaviour-focused tests of the shared
``Backoff`` curve.
- ``test_pr_comments_cache.py`` + ``test_ci_logs.py`` — deleted the
now-redundant ``TestComputeNextAttemptAfter`` / ``TestBackoffActive``
/ ``TestBackoffHelpers`` classes; ``test_backoff`` covers them.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
6e63073ad9 |
feat(auto-agents): retire implementation-worker wrapper (R2 wrapper-chain tax)
The depth-0 implementation-worker agent was a pure passthrough whose only LLM-side responsibility was translating the dispatcher-built prompt into tier-dispatcher's input format (top-level params + ``task_prompt:`` fenced body). Deterministic Python now does the same translation in microseconds, removing one full LLM hop (and its summarization-risk warning prose) from every implementer cycle. What changed: 1. **New ``_wrap_for_tier_dispatcher`` helper** in ``tools/dispatch_implementer.py`` — emits the exact format documented in ``.opencode/agents/tier-dispatcher.md``: outer-level ``task_agent: \`task-implementor\``` + ``estimator_agent: \`estimator-implementation\``` + optional ``escalation_tier_hint`` line, then the body verbatim inside a fenced ``task_prompt:`` block. 2. **``_implementation_prompt_dispatch`` refactored** to compute the escalation_tier_hint upfront (start_tier vs estimator-flag logic preserved exactly) and wrap+return once. The three prior return paths (early-return / explicit-hint / final-extras) all funnel through the same wrap helper. 3. **WORK_GROUP defs swapped**: three ``worker_agent="implementation- worker"`` → ``"tier-dispatcher"`` so the dispatcher invokes tier-dispatcher directly as the top-level OpenCode session. 4. **``release_claim_on_exit`` removed** from the emitted prompt. It was an implementation-worker-only directive telling the wrapper to skip its own session-end release; tier-dispatcher has never read it. The dispatcher's ``finally`` block has always owned the actual claim lifecycle and is unchanged. 5. **Test updates**: three test files updated to match the new contract — assert absence of ``release_claim_on_exit`` and ``worker_agent == "tier-dispatcher"``. All other tests pass without modification (the wrapper's .md file remains as historical doc; can be deleted in a follow-up cleanup). Audit confirmed no permission boundary lost (both agents are ``mode: all`` with equivalent bash/write allowlists), no agent- name conditionals in ``_opencode_worker``, no archive-filename grep filtering, tier-dispatcher input format is a 1:1 superset of what the wrapper emits. Live-cycle savings: removes the wrapper's ~1.5–3K tokens of reasoning + tool overhead, removes one full session-creation + session-archive cycle, removes the "verbatim forwarding" summarization risk (deterministic Python can't lose sections). Chain shrinks from 4 levels (impl-worker → tier-dispatcher → estimator/tier-N → task-implementor) to 3 (tier-dispatcher → estimator/tier-N → task-implementor). Full auto_agents suite: 2056 passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
ea4a96aad6 |
feat(auto-agents): worktree hygiene — startup janitor + retry-on-failure
Live evidence on PR #29 (runs 15-18): 4 implementer worktrees accumulated in ``/tmp/cleveragents-implementer-worktrees/``, one with a corrupted ``.git`` link (``fatal: not a git repository``). Root cause: every SIGTERM-induced dispatcher restart leaves the in-flight cycle's worktree orphaned, and the mirror's ``worktrees/<name>/`` bookkeeping survives without the directory. The next cycle's ``git worktree add`` against the same mirror can then fail with "already exists" or unhelpful path collisions. Two fixes that together close the loop: 1. **Startup janitor** (``_pr_clone.prune_orphan_worktrees``): scans the per-kind worktree base on dispatcher startup and removes any dir matching the canonical ``pr-{N}-{kind}-{hex-tag}`` shape that is EITHER older than the OpenCode worker ceiling (default 30 min — longer than any possible in-flight cycle) OR has a missing / zero-byte ``.git`` link (definitionally corrupted). Removes the dir AND the mirror's ``worktree`` bookkeeping. Idempotent. Skips operator scratch dirs that don't match the canonical name. Disable via ``DISPATCHER_WORKTREE_JANITOR_DISABLE=1``. Called once at the top of both ``dispatch_review.main`` and ``dispatch_implementer.main``. 2. **Retry-on-failure** in ``prepare_pr_worktree``: when ``git worktree add`` fails the first time, run ``git worktree prune`` to clear stale mirror bookkeeping, force- remove the target path if present, and retry exactly once. This rescues cycles whose janitor-min-age cushion missed a fresh orphan from a very-recent SIGTERM. Coverage: 13 new tests in ``test_pr_clone_janitor.py`` (recent vs stale removal, corruption detection regardless of age, non-canonical name safety, idempotency, disable env, ``git worktree remove`` call count). Full auto_agents suite: 2059 passing (+13 vs prior commit). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
3d30358333 |
feat(auto-agents): pre-fetch failing-CI job log tails (information-starvation fix)
Workers (reviewer + implementer) historically saw `CI / lint: failure` in the prompt but had no idea WHY. Three implementer cycles on PR #28 (runs 18-20, 2026-05-16) burned their full 30-min worker budget largely on: - ~14 file reads + ~14 bash calls inferring the failure from the diff - `bash curl … /actions/runs/N/jobs` attempts blocked by the bash allowlist - `webfetch` against the same Actions URL — also errored - one ~10-min `ci_run_local_gate` run (a full `nox -s coverage_report`) This adds a single source-of-truth cache that pre-fetches the LAST N chars of every failing job's raw CI log: - **`tools/_ci_logs.py`** — per-(head_sha) on-disk cache; parses Forgejo Actions `target_url` (both `/runs/N/jobs/M` and `/runs/N`-only shapes) to resolve job_ids; fetches `/repos/{owner}/{repo}/actions/jobs/{id}/logs` and keeps the tail. SHAs are immutable so `completed=True` cache entries are good forever; partial fetches stamp exponential backoff (same shape as `_pr_comments_cache`). - Dispatcher pre-fetch (reviewer + implementer): each dispatcher's prefetch coordinator (`_review_prompt.fetch_review_context` + `_implementer_prefetch.prefetch_for_pr_fix`) now calls `_ci_logs.fetch_pr_failure_logs` when CI overall != success. Cache exceptions WARN and serve None (prompt builds with empty section; worker still has `target_url` to follow). - Prompt sections: new `## Pre-fetched CI failure logs` section in both reviewer (`_review_views.build_ci_failure_logs_section` — JSON payload) and implementer (`_implementer_prompt._build_ci_failure_logs_section` — text blocks). The reviewer's `data_complete` aggregate gates on it. - Agent prompts updated: `pr-review-worker.md` line 414 (which already predicted this feature in prose) and `task-implementor.md` step 1 ("Read the CI failure picture FIRST") both now point at the new section and explicitly tell the worker NOT to use `curl` / `webfetch` / `ci_run_local_gate` for log content. Coverage: 26 new tests in `test_ci_logs.py` (parse_run_job_ids, the two URL shapes, cache miss/hit/backoff, log truncation, run-only URL job-id resolution, disabled-mode bypass) + 1 test update in `test_pr_context_sentinel.py` to include the new completion flag. Full auto_agents suite: 2046 passing (+36 vs prior commit, all green). NOTE: the matching `ci_fetch_pr_failure_logs(pr)` MCP wrapper for agents to call ad-hoc is implemented in `tools/mcp_ci_server.py` + covered by `tests/auto_agents/test_mcp_ci_fetch_pr_failure_logs.py`, but both files are currently untracked (Phase 1 work). They'll ride with Phase 1's commit. The dispatcher pre-fetch path here is self-contained and doesn't depend on the MCP wrapper. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
fedd99b9d4 |
fix(auto-agents): partial-delta branch persists merged view (not original cached)
On a partial live-delta the failing branch returned ``cached + delta`` merged to the caller, but persisted ONLY the original cached entries back to disk. The next cycle's backoff short-circuit then served the older, smaller view — a regression vs the failing branch's view. Live-test on PR #29 run-19 caught this: failing branch returned 144 comments, but the next short-circuit would have served 13. Persisting the merged view is safe because ``_merge_comments`` dedups by id — the write is strictly additive. ``any_partial_fetch`` stays True so the next clean delta still backfills toward completeness, and ``since_cursor`` advances to the newest merged entry so the next ``?since=`` picks up forward (not where we started). Coverage: 1 new test (``test_partial_delta_persists_merged_so_next_short_circuit_matches``) pins the contract. 28 cache tests passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
aa7c24fa18 |
feat(auto-agents): resilient comment cache — backoff + reviewer cutover
Two changes that together stop PR #29's per-cycle ``max_pages=20`` truncation WARN on its 2700+ comment history: 1. **Exponential backoff** in ``_pr_comments_cache.get_pr_comments``: on a failed live delta the cache stamps ``consecutive_failures`` and ``next_attempt_after`` (default base 60s × 2^(failures-1), capped at 30 min). The next cycle inside the backoff window short-circuits — serves the cached bulk stale with ``completed=False`` and does NOT hit the flaky endpoint again. First successful delta clears the counter, so a transient outage doesn't permanently throttle. Truncated cold seeds (page-cap hit on first walk) are also counted as failures so the every-cycle 30s pagination tax stops on PR-sized threads that genuinely exceed the cap. 2. **Reviewer prefetch cutover** in ``_review_prompt.py``: the reviewer's ``fetch_review_context`` now routes through ``_pr_comments_cache.get_pr_comments`` instead of the raw ``_review_pipeline.fetch_pr_comments``. Mirrors the Phase 2 feature-flag pattern: default ON, env off-switch ``REVIEW_DISPATCHER_USE_COMMENT_CACHE=0`` for rollback, WARN and fall back to the legacy paginator on any cache exception so a cache failure can never break a review cycle. Implementer dispatcher has used this cache directly since 2026-05-13 without incident. Coverage: 30 new tests (14 backoff + 16 cutover-wrapper). Touched-module suite: 43 passing. NOTE: the matching MCP wrapper (``forgejo_fetch_pr_comments_cached``) lives in ``tools/mcp_forgejo_server.py`` which is currently untracked; it'll ride with the Phase 1 commit that lands the MCP server file itself. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
cf13de9b5a |
feat(auto-agents): per-repo linked-issue policy (strict / informational-on-not-found / disabled)
Adds a per-repo policy knob for ``Closes #N`` / ``ISSUES CLOSED: #N``
resolution during dispatcher pre-fetch. Three modes:
- ``strict`` (default, unchanged) — every reference is resolved; a
``not-found`` (4xx) result is rendered as a quality concern the
reviewer is expected to surface; a ``fetch-error`` (5xx / runtime)
trips the dispatcher's defensive APPROVED→COMMENT downgrade.
- ``informational-on-not-found`` — a ``not-found`` result is shown
to the reviewer with an explicit policy note: "this signal is
informational, not blocking; MUST NOT factor into your verdict."
``fetch-error`` semantics are unchanged. Typical setting for forks
harvesting commits from an upstream with its own issue tracker.
- ``disabled`` — skip resolution entirely; section renders
"linked-issue resolution disabled by repo policy" preamble. The
worker is told it has no traceability check to perform.
Wired through ``DispatchConfig.linked_issue_policy`` (defaults to
``"strict"``) and ``{REVIEW,IMPLEMENTER}_DISPATCHER_LINKED_ISSUE_POLICY``
env vars. ``normalize_linked_issue_policy()`` validates the value at
``load_config`` time so a typo'd env var fails startup with a clear
error rather than silently defaulting.
Coverage: 15 new tests across ``test_review_fetch.py`` and
``test_review_views.py``. Touched-module suite: 250 passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
363bcfad61 |
refactor(auto-agents): extract _classify_metadata_only as the first C2 step
dispatch_implementer.py is 3225 lines — well over the ~500-line per- module budget the rest of tools/ honours. C2 is the long-overdue decomposition; this commit starts the work with the smallest / cleanest extraction available: the pure G1 metadata-only classifier (no module state, single public surface) moves to a new sibling _implementer_metadata_classifier.py, loaded through the existing _loader.load_sibling machinery just like every other helper in this file. The leading-underscore alias dispatch_implementer._classify_metadata_only re-exports the function so the G1 tests (and any other call sites) keep working unchanged. The extraction is a pure refactor — the full 1754-test auto-agents suite passes unchanged. Subsequent C2 steps (extract _post_session_action_with_escalation — the ~350-line nested loop the harvest plan singled out by name — plus prompt-assembly + short-circuit blocks) are larger pieces that warrant fresh-context attention. This commit establishes the pattern (new module, _load_sibling line, leading-underscore alias) for those follow-ups. Refs: docs/development/final-working-harvest-plan.md (C2). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
23ae44e432 |
feat(auto-agents): loud signal on degraded prompt assembly (W8, default-OFF)
Adds assess_prompt_completeness() to _implementer_prefetch.py — a
pure helper that translates the carrier's per-section completion
flags into a single {degraded, missing_sections, error_kinds} dict.
The aggregate data_complete flag already encoded the AND, but as a
single bool it lost the *which sections* signal an operator needs
to triage a degraded cycle. This helper keeps both together.
dispatch_implementer._prefetch_prompt now calls the helper after
prefetch and stashes the result on the per-item context as
prompt_completeness. When IMPLEMENTER_DEGRADED_PROMPT_LOG_ENABLED=1
(default OFF, per the dmpipeline safety contract) AND the cycle is
degraded, also emits a WARN log line naming the specific missing
sections so an operator tailing the dispatcher log sees the
silent-degradation signal without parsing telemetry JSONL.
The completeness signal is always stashed on the context regardless
of the flag, so future telemetry / status-comment paths can pick it
up without operators flipping any switch. The flag specifically
gates the LOG emission — the harvest plan's primary value
("converts a silent correctness risk into an observable one").
5 unit tests for the helper covering happy / single-fail / multi-
fail / diff-truncation / direct-data_complete-flip paths.
Refs: docs/development/final-working-harvest-plan.md (W8).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
561e7e86b9 |
docs(auto-agents): record A2/G10 verify-first findings (no-op)
A2 (open-dependency merge handling): verified that dmpipeline's _pr_has_open_dependencies already covers the case agents/final-working addresses with merge_pr.ts's --dep error|delete|reverse strategies. The two approaches differ: final-working detects the silent-no-op at MERGE TIME and recovers via strategy flags; dmpipeline detects the dependency BEFORE the merge attempt and excludes the PR from candidates with an auto/blocked-by-deps label. dmpipeline's is the more conservative path — operators see the state explicitly without the failed-merge dance. No code port required; annotated the helper so future readers see the deliberate divergence on record. G10 (git-isolator-util identifier line): verified that dmpipeline's git-isolator-util uses agent-name + timestamp for worktree naming and does NOT consume an identifier parameter, so the harvest plan's "port only if still consumed" condition is not met. No code change. Refs: docs/development/final-working-harvest-plan.md (A2, G10). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
c787c133df |
docs(auto-agents): document the WorkGroup wrapper seam (A7)
Adds an "Extensibility: the WorkGroup seam" section to the _dispatch_runtime.py module docstring naming the deliberate design choice: pipeline-specific behaviour is added by injecting WorkGroup instances (with prompt_factory + post_session_action) into the generic loop, rather than by replicating the cycle / claim / lock machinery in each driver. This is dmpipeline's Python-side expression of the same pattern agents/final-working put in markdown — three thin-wrapper supervisors (implementation-supervisor, pr-review-supervisor, pr-merge-supervisor) over a generic supervisor agent. The parallel was the cleanest idea from the LLM-supervised branch; this docstring makes it explicit so future pipelines reuse the seam instead of duplicating the boilerplate. Refs: docs/development/final-working-harvest-plan.md (A7). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
d52b85f063 |
docs(auto-agents): reviewer-identity separation policy (A5)
Documents the FORGEJO_REVIEWER_PAT / FORGEJO_REVIEWER_USERNAME convention as a load-bearing policy in the auto-agents-system skill. Pairs with the existing assert_reviewer_identity runtime enforcement in dispatch_review.py::load_config and the G5 startup PAT probe in _dispatch_runtime — the convention was always enforced at runtime, but the policy was undocumented and a fresh operator deploying the bot for the first time had no obvious place to learn WHY the reviewer pipeline needs a separate identity. The no-self-approval invariant matters because Forgejo branch protection rejects merges when the author is also the only approver. Misconfiguring the secrets so both pipelines share a PAT silently breaks the merge driver (HTTP 422 from the merge endpoint) in a way that surfaces only at human triage. Refs: docs/development/final-working-harvest-plan.md (A5). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
1635229828 |
docs(auto-agents): document model-override-needs-restart footgun (C3)
Replaces the README's optimistic "Changes to a .txt file take effect on the next dispatched session — no OpenCode restart needed" with the actual behaviour: BOTH paths require an OpenCode restart for a .txt edit to change what model OpenCode generates with. The dispatcher passes the resolved model on every POST /session (observability / consistency check), but OpenCode itself re-resolves agent.<name>.model from its startup-cached opencode.json on every generation. Without a restart, the dispatcher logs say one model ran and OpenCode actually ran another — a silent regression invisible from the dispatcher side. This is the same footgun the long comment in _opencode_worker.run_session_blocking documents inline; the README was the missing place where an operator naturally looks before editing a model file. Cross-links the consequence for G11 (estimator-driven adaptive tier selection) and the in-cycle escalation plan — both silently misbehave if a model swap lands without a restart (a Tier 1 escalation would run on the cached Tier 0 model). Refs: docs/development/final-working-harvest-plan.md (C3). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
094998dcfa |
test(auto-agents): drift-prevention sync test for compliance renderers (W10)
_implementer_compliance.render_prompt_stanza and dispatch_implementer._render_compliance_pointer_stanza are two parallel renderers (full markdown vs condensed pointer that survives tier-agent summarisation). Both must classify the SAME input identically on the three observable verdicts the worker keys off: - all clear, no masking → emit resolved - all clear, some masked → DO NOT emit resolved (hedge) - some gaps → fill the missing items This test pins that contract so hand-drift between the two renderers trips a clearly-named test instead of shipping a cycle archive with conflicting verdicts (the worker reading the sentinel JSON sees one verdict and the worker reading the prompt sees another). Option (b) from the harvest plan; the full collapse into a single renderer (option a) is out of scope because the two shapes are intentionally different — full markdown vs summarisation-resilient pointer. Refs: docs/development/final-working-harvest-plan.md (W10). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
afda1ab5f8 |
feat(auto-agents): post-claim TOCTOU verification (W5, default-OFF)
Closes the cross-driver claim-collision race. Forgejo's label-add
endpoint is idempotent — two concurrent dispatchers (impl ↔ rev,
or cross-host instances) can both come back HTTP 200 even though
only one was actually first. The single-instance fcntl lock
prevents same-driver races but cross-driver / cross-host races
remain. conflict_drive.py has had an opt-in mitigation for this
since plan § 3.3.1; this commit ports the simpler "different
auto/claimed-* label appeared in the race window" version into
_dispatch_runtime.dispatch_one so the implementer and reviewer
drivers benefit from the same protection.
Flag-gated via DISPATCHER_VERIFY_CLAIM_AFTER_APPLY=1, default OFF
— preserves today's accept-the-race behaviour byte-for-byte. When
ON:
1. claim_work_item succeeds → label attached.
2. Re-GET /issues/{N}/labels. If a different auto/claimed-* label
is present alongside ours, release our claim (with detail
"post-claim-verify") and return terminal_state="claim-collision".
3. Otherwise register the in-flight claim and proceed normally.
Transient GET failures don't trigger collision (defensive: rather
miss one race than spuriously abandon a healthy claim — the
cycle-failure-budget catches persistent fetch issues).
5 new tests: flag default OFF, the detector's three behaviours
(foreign label → collision, own only → no collision, fetch
failure → no collision), end-to-end dispatch_one releasing on
collision without spawning the worker.
Refs: docs/development/final-working-harvest-plan.md (W5).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
d758ca3529 |
feat(auto-agents): reviewer CI-failure triage procedure (G8)
Adds a deterministic CI-failure triage subsection to pr-review-worker.md. The reviewer's verdict outcome rules already distinguished "CI failing with issues introduced by this PR" (→ REQUEST_CHANGES) from "CI issues already known and not introduced by this PR" (→ may APPROVE), but the prompt gave the worker no procedure for making that determination. Net result: the call was inconsistent across reviewer cycles — different sessions on the same PR could land on different verdicts. The new section gives the worker a five-step method: 1. Identify the failing check's context. 2. Map the context to a code area (lint covers source diffs; e2e covers src/+features/; nightly/flaky covers nothing). 3. Cross-reference against the prefetched diff to determine whether the failure plausibly maps to changed files. 4. When in doubt, REQUEST_CHANGES — the same cost-asymmetry tie- breaker that applies elsewhere in the prompt. 5. Never APPROVE when overall CI is failing AND every per-check failure plausibly maps. The optional Python-side log-tail prefetch is documented as a future enhancement but not built in this commit (the deterministic mapping above is the immediate value). Contract test in test_agent_prompt_contracts.py pins the load- bearing section heading + the four signal phrases so a future prompt simplification cannot silently drop the procedure. Refs: docs/development/final-working-harvest-plan.md (G8). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
02f6f45f9e |
test(auto-agents): regression guard against PR body-deletion footgun (G9)
Verify-first audit (2026-05-15): no dispatcher hot-path module
under tools/ currently PATCHes /pulls/{N}. Implementer-side
compliance apply operates on the worktree only; reviewer-side POSTs
reviews and comments without PATCH; merge driver PATCHes /issues/
for state transitions, not /pulls/. So Forgejo's "PATCH /pulls/{N}
deletes body if omitted" footgun documented in agents/final-working's
pr-creator.md is unreachable today.
This test is a forward-looking guard: a future PR-edit path added
under tools/_implementer*.py, tools/_review*.py, tools/dispatch_*.py,
tools/merge_drive.py, tools/conflict_drive.py, or tools/_pr_*.py
will fail this test, prompting the contributor to either always
include the full body in the PATCH or add a dedicated body-
preservation test.
Net result: no production behaviour change; the footgun is locked
out so it cannot silently reappear.
Refs: docs/development/final-working-harvest-plan.md (G9).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
1e8ef793d2 |
feat(auto-agents): startup PAT validation hard-stop (G5)
Closes the dead-PAT-spins-idle-forever gap. Without this probe, a
rotated / revoked Forgejo PAT lets every dispatcher loop forever:
each work item's claim returns labels-fetch-failed (a "soft"
outcome that counts as a cycle success with zero dispatches), so
the consecutive_failures counter resets every iteration and the
cycle-failure-budget escape hatch never trips. Operators see
timed-out heartbeats but no SystemExit and no log alert.
_validate_pat_or_die() probes GET /user at run_outer_loop startup:
- 200: log INFO with the resolved login (auditable bot identity
at process start), return.
- 401 / 403: raise SystemExit(2) — matches the existing
cycle-failure-budget exit code so supervisor restart semantics
are uniform.
- Other (transient 5xx, network flap, malformed body): log
WARNING and return; the cycle-failure-budget covers persistent
cases.
Skippable via DISPATCHER_SKIP_PAT_VALIDATION=1 for tests / bisect.
The skip is logged loudly so a missing safety net in production
telemetry is obvious.
6 new tests cover the 5 response classes + the env-skip path.
Three existing run_outer_loop tests stub the validator (they test
loop semantics, not PAT validation, which has its own dedicated
tests).
Refs: docs/development/final-working-harvest-plan.md (G5).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|