a21466add232d59cdec1604e09d58ca05659a623
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 drew
|
a103a31bbf |
feat(controller): worker-owned gated push for the implementer
The implementer agent no longer pushes to git — the controller worker now owns the push: it gates the agent's commits on lint+typecheck and pushes via a single leased primitive. Closes two production defects: - Clobber: the pre-fix MCP --force-with-lease leased against a freshly-fetched tip, so the lease always passed — an in-flight implementer destroyed a commit pushed to the PR branch during its run (lost a hand-pushed skip_coverage fix on PR #46). - Gate-skip: the agent verified only the CI-flagged gate, so a fix for one gate shipped fresh violations in another (lint flapped pass->fail across CI runs 198->199). Step 1 — worker_push primitive: - New git_push.py: one leased push, pinned to the SHA the worker started from; classifies pushed / stale_input / diverged / infra_error; bounded infra-retry. - mcp_git_server.push gains expected_sha for a correctly-pinned lease. - finalize_conflict_resolution migrated onto worker_push. Step 2 — deterministic gate: - New gate.py: per-slot nox env-dirs (no venv races), manifest-hash staleness keying, lazy warm-up. - local_ci_gate.sh gains --envdir. Step 3 — worker-owned gated push: - New implementer_finalize.py: divergence pre-check -> lint+typecheck gate -> ruff auto-fix -> leased push. finalize's outcome is authoritative over the agent's emitted outcome. - agent_runner integrates finalize; salvage no longer pushes. - outcomes/tick: gate-failed + push-time stale-input caps, epoch-scoped; WorkerError carries an output_payload so the gate report reaches the next attempt's prompt; prefetch surfaces gate-failed attempts. - The 5 task-implementor prompts drop the agent push step. Reviewed across 4 adversarial rounds; full controller+MCP suite green (1379 passed). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
0bc734c020 |
style: ruff format the controller-state-machine branch (288 files)
Applies `ruff format` to the accumulated formatting debt on this branch. Formatting-only — no behavioral changes. Required for CI/lint's format gate (`nox -s format -- --check`), which the branch was failing on 288 tracked files that drifted from ruff's canonical style. In-progress WIP files are intentionally excluded so this commit stays a clean formatting-only diff. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
3e23853ffa |
fix(controller): batch R — wire 5 V1-contract fields the controller silently dropped
Trial run-3 (2026-05-19) surfaced the first instance of a broader bug class: V1 contract fields existed and agents emitted them, but no controller code wired them into state transitions. An adversarial "walk the happy path" code review found 4 more, all listed below. The class shape: a V1 field is "Required iff X" by contract docstring, the worker emits it correctly, but the master reads the wrong field (or doesn't read it at all), so a critical state transition silently no-ops or drops to the wrong default. FIX #0 — outcome-mapper early-return (committed earlier in this session) — moved role dispatch before the ``outcome is None`` guard so estimator+reviewer+summarizer (V1 contracts without an ``outcome`` field) are correctly handled. Without this fix, all estimator attempts in trial run-3 completed successfully then were silently discarded, stranding all 6 workflows in ANALYZING. FIX #1 — current_tier never written from estimator's recommended_tier File: tools/controller/master/tick.py The ANALYZING→IMPLEMENTING UPDATE wrote only current_state / last_transition_at / entered_state_at. recommended_tier from the estimator payload was never extracted, so every PR ran at the workflow's creation-time tier (typically 0) regardless of what the estimator recommended — the entire tier-escalation ladder was informational-only. Fix: per-event ``extra_set`` clauses; on ``estimator_done`` / ``estimator_metadata_only`` events, set ``current_tier = :rec_tier`` from the payload (with 0..2 validation). Tests: TestEstimatorRecommendedTierWritten (3 cases). FIX #2 — approved_at_sha never passed to merge callback File: tools/controller/master/merging.py, forgejo_http.py ReviewerOutputV1.approved_at_sha is the exact SHA the reviewer signed off on. Pre-fix the MergeCallback signature was ``(owner, repo, pr_number)`` — Forgejo merged whatever HEAD currently was. Race condition: a concurrent push (operator or another driver) between approval and merge would silently merge unapproved code. Fix: extended signature to ``(owner, repo, pr_number, approved_at_sha)``; SQL SELECT now pulls the latest reviewer attempt's output_payload as a subquery; merge_pr forwards it to Forgejo as ``head_commit_id`` (Forgejo refuses with 409 if HEAD has advanced). Defensive: still merges when approved_at_sha is None but logs a WARNING. Tests: TestApprovedAtShaPassedToMerge (2 cases). FIX #3 — tier_last_succeeded column had ZERO writers File: tools/controller/master/tick.py The schema column existed; the merging.py 409-conflict path read it to recover the last-known-good tier; but NOTHING ever wrote to it. Every workflow's tier_last_succeeded was permanently NULL → the 409-recovery path transitioned to IMPLEMENTING(tier=NULL) → scheduler silently coerced to tier 0. Fix: on ``implementer_pushed`` event, ``UPDATE workflows SET tier_last_succeeded = current_tier``. Tests: TestTierLastSucceededWritten. FIX #4 — outcome column NULL for estimator/reviewer/summarizer File: tools/controller/worker/runner.py ``workflow_attempts.outcome`` is the operator-facing audit column. Pre-fix the runner extracted ``output_payload.get("outcome")`` blindly — works for implementer/conflict_resolver but those three roles have no ``outcome`` field. Result: ``SELECT … WHERE outcome IS NOT NULL`` audit queries silently missed every estimator/reviewer/ summarizer attempt. Fix: new ``_derive_outcome_for_audit(role, payload)`` helper synthesizes meaningful per-role values: - implementer/conflict_resolver: payload['outcome'] (unchanged) - reviewer: payload['verdict'] - estimator: 'metadata-only' OR f'tier-{recommended_tier}' - summarizer: 'summarized' Tests: TestOutcomeAuditColumn (parametrized 8 cases). FIX #5 — conflict_resolver new_head_sha never preferred File: tools/controller/worker/runner.py ConflictResolverOutputV1.new_head_sha is "Required iff outcome='resolved'" (the canonical post-rebase branch tip). Pre-fix runner.py used ``commit_shas[-1]`` for head_sha_after — works for normal git rebase --continue but wrong for resolvers that did force-pushed merge commits where the last commit SHA ≠ the branch tip. CI status poll would then poll the wrong SHA. Fix: when role=='conflict_resolver', prefer ``new_head_sha`` over commits[-1]. Tests: TestConflictResolverNewHeadShaUsed (2 cases). ALSO updated existing tests that papered over the original bug: - test_master_outcomes.py: estimator tests used to inject a fake ``"outcome": "(implicit)"`` field; now use real V1 shape (no outcome). Reviewer tests now use ``verdict`` (the real V1 field) not ``outcome``. - test_master_tick.py reviewer tests: same `verdict` switch. - test_master_merging.py: updated all 13 ``lambda o, r, n: ...`` merge-callback stubs to the new 4-arg signature. CONFIRMED-CLEAN (no fix needed) by the same code review: - outcomes.py post-fix-#0 - prefetch.py field reads - prompts.py field accesses - ci_status_poll.py role+outcome filter The above were verified to handle all 5 V1 contract shapes correctly. Total: 802 → 819 controller tests, 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
f93f0d1c53 |
fix(controller): round-4 P2 — defensive input_payload normalization
If the DB returned NULL input_payload (legacy/seeded data, schema bug, hand-edited rows), agent_runner crashed on ``dict(None)`` → WorkerError → master re-pickups → STUCK after MAX_PICKUPS reaps. Silent infinite-loop until exhaustion. Fix: ``worker/runner.py:run_one_attempt`` checks isinstance(dict) on entry; substitutes empty dict + logs WARNING. Agent still runs normally with an empty payload. Test: ``test_none_input_payload_doesnt_crash`` — passes input_payload=None + verifies the attempt completes normally. Total: 712 controller tests pass (+1 net). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
7bf39a0b51 |
fix(controller): RB8 — head_sha bookkeeping in _write_outcome
The worker's _write_outcome never wrote head_sha_before / head_sha_after
on the workflow_attempts row. But tick.py reads both to compute
head_sha_advanced, which the outcome mapper REQUIRES to distinguish
``implementer_pushed`` (true push) from ``implementer_blocked``
(worker said resolved but git didn't move).
Without these columns set, every implementer attempt's "resolved"
outcome mapped to a no-op event, and workflows would stall
permanently after the agent ran successfully.
Fix:
- ``worker/runner.py:_write_outcome`` accepts ``head_sha_before`` +
``head_sha_after`` kwargs and writes both columns in the UPDATE.
- ``run_one_attempt`` computes:
- head_sha_before = input_payload["head_sha"] (what the agent
was told to start from)
- head_sha_after = output_payload["commit_shas"][-1] if any
commits were produced; falls back to head_sha_before otherwise
so tick.py correctly sees "no advance" when the agent didn't
commit.
Tests (+2):
- test_head_sha_before_after_recorded — pins the happy path where
the agent committed and head_sha advanced.
- test_no_commits_means_head_sha_unchanged — pins the no-advance
case where head_sha_after equals head_sha_before.
Total: 707 controller tests pass (+2 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
c4a7f0027d |
fix(controller): batch H — round-2 important items (N5-N10)
Six items from the round-2 adversarial review's "important" list. N5 — restricted JSON encoder replaces ``default=str``: ``default=str`` silently stringified custom objects, sets, and bytes to ``"<MyObj at 0x...>"`` — masking worker output bugs. Now uses a restricted encoder (``tools/controller/_json_safe.safe_json_dumps``) with an allowlist: - datetime / date → ISO-8601 string - Decimal → str (preserves precision) - UUID → canonical string - Path → str - set / frozenset → sorted list (best-effort) - Everything else → TypeError (a worker output regression surfaces loudly instead of writing garbage to the DB) Replaces ``json.dumps(..., default=str)`` at: - ``worker/runner.py`` (terminal-state UPDATE write) - ``master/scheduler.py`` (input_payload INSERT + UPDATE) Tests: ``test_json_safe.py`` (+15 tests) — every allowlisted type + rejected types (custom class, bytes, complex) + nested structures + kwargs forwarding. N6 — strict parser check now runs BEFORE backfill + loop: Previously the strict-parser exit could run AFTER backfill (the test ``test_strict_parser_coverage_blocks_startup_with_stubs`` passed because backfill's ``fail_if_called`` AssertionError was swallowed by the bare ``except Exception``, then strict exited 2). The test asserted the right outcome via the wrong path. Fix: - ``master/__main__.py``: parser-coverage check moved to immediately after engine creation, BEFORE backfill + loop. Strict-mode failure exits 2 without wasting a Forgejo round-trip + without dependent code paths firing. - Test refactored: count-based assertions on backfill and loop call counts (0 each) instead of fail_if_called. Catches regressions where the strict check moves back below either. N7 — _to_aware_datetime unit-tested in isolation: Previously exercised only via end-to-end comment-filter test. New ``TestToAwareDatetime`` (+12 tests) covers: None, empty string, Z suffix, +00:00 offset, microseconds preserved, naive datetime → UTC, malformed string → None, partial string → None, unsupported types → None, timezone abbreviations → None, equality across Z + offset forms (the regression the helper exists to defend). N8 — runtime=None lazy-import branch tested: Previously all 24 HTTP factory tests injected a fake runtime; the production path (``build_callbacks(cfg=None)`` → sys.path injection + lazy import of ``tools._claim_runtime``) was untested. ``TestBuildCallbacksDefaultRuntime`` (+2 tests): verifies the import succeeds + every ForgejoCallbacks attribute is callable; verifies the import is idempotent (second call doesn't crash on sys.path re-insert). N9 — resume event-row emission asserted: Round-1's batch B added the PAUSE event-row test (``test_label_removed_emits_event_with_reason``) but not RESUME. ``test_label_restored_emits_event_with_reason`` pins the resume's controller_events shape (from_state=PAUSED, to_state=<prior>, reason="opt-in-label-restored", source="reconciliation") so operators auditing the timeline see both pause + resume. N10 — concurrent SQLite dequeue test: The dequeue docstring claims SQLite BEGIN-DEFERRED concurrent dequeues "retry via busy_timeout (5s)" — but no test verified. ``TestSQLiteConcurrentDequeue::test_two_threads_racing_one_wins`` spawns two threads, both attempt dequeue simultaneously via a threading.Barrier. Asserts: neither thread raises SQLITE_BUSY- without-retry, exactly one acquires the attempt, the loser sees no_pending_eligible (winner committed first). Total: 691 controller tests pass (+31 net), 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
6ba1926c52 |
fix(controller): batch A — datetime json, Forgejo state map, ISO compare, dequeue docs
Four narrow bug fixes flagged by adversarial code review (items 1, 5, 6, 7 from the consolidated critique). ITEM 1 — datetime → json.dumps crash (silent write-after-work failure): - ``worker/runner.py:274`` and ``master/scheduler.py:283`` now pass ``default=str`` to ``json.dumps`` so nested datetime fields (e.g. CISummary.observed_at) serialize without raising. - Before this fix: a worker would do its real work, then crash on the terminal-state UPDATE with TypeError, get recorded as ``worker-internal-error``, and the output payload would be lost. - Test: TestDatetimeSerializationSafety in test_master_ci_summarize + test_scheduler_handles_datetime_in_input_payload in test_master_prefetch (both pin the regression — the with-default test passes, the without-default test asserts the TypeError so future maintainers see the failure mode). ITEM 5 — Forgejo state mapping completeness: - Extended ``_FORGEJO_STATE_TO_GATE_STATUS`` in ``master/ci_summarize.py`` to cover ``cancelled``, ``timed_out``, ``action_required``, ``queued``, ``in_progress``, ``neutral``, ``skipped``, ``stale`` — states observed across Forgejo / Gitea / GH-mirror that previously collapsed to ``pending``, telling the implementer "CI is still running" when really a job was cancelled. - ``cancelled`` / ``timed_out`` / ``action_required`` / ``stale`` now map to ``error`` (the gate failed). - ``queued`` / ``in_progress`` stay ``pending`` (still running). - ``neutral`` / ``skipped`` → ``passed``/``skipped`` (informational). - Test: TestExtendedForgejoStates — 6 tests covering each new state. ITEM 6 — lexicographic ISO comparison drops/dupes comments: - ``master/prefetch.py:_comment_bodies_since`` and ``_iso`` replaced with ``_to_aware_datetime`` + datetime comparison. Forgejo emits ``2026-05-18T12:00:00Z``; Python's ``datetime.isoformat()`` emits ``2026-05-18T12:00:00+00:00`` — a string compare gives 'Z' (0x5A) vs '+' (0x2B) which silently misorders timestamps. - Now parses via ``datetime.fromisoformat`` (with Z → +00:00 rewrite), defaults naive timestamps to UTC, and compares as ``datetime``. - Test: test_comments_filter_handles_z_suffix_vs_offset_form pins the regression. ITEM 7 — false BEGIN IMMEDIATE claim in dequeue docstring: - The dequeue docstring claimed ``BEGIN IMMEDIATE`` was applied by session_scope; it wasn't. Attempted a global ``begin``-event listener that conflicted with StaticPool's shared-connection model (test_prefetch_callback_works_in_scheduler broke). - Reverted to a documentation fix: SQLite stays on default BEGIN DEFERRED (the SQLITE_BUSY retry via busy_timeout=5s is acceptable for single-host dev) + the docs make MULTI-MACHINE REQUIRES POSTGRES explicit at three call sites (db/session.py, db/dequeue.py, RUNBOOK.md was already updated in Phase 1l). Postgres has FOR UPDATE SKIP LOCKED which is what production actually uses. Tests: 586 controller tests pass (+10 new), 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
eab476e48e |
feat(controller): Phase 1c — worker controller skeleton
The dequeue+lock+heartbeat+runner+loop machinery. Production
OpenCode + MCP invocation slots in via the agent_runner callable
(Phase 1c-2). This commit is the structural foundation:
tools/controller/worker/:
- identity.py: build_instance_id() → "{hostname}/{pid}/{worker_uuid}"
per plan v9 (slash delimiter; IPv6-safe; uuid4 prefix for
per-instance uniqueness).
- heartbeat.py: Heartbeat thread that updates lock_heartbeat_at
every interval (default 30s). v9 simplified: TTL-only (no activity
tracking). UPDATE … WHERE locked_by_instance=us; rowcount=0 →
lost_lock_event.set() and thread exits, letting reaper handle it.
- runner.py: run_one_attempt() drives one attempt end-to-end.
Starts heartbeat → invokes agent_runner → on success writes
status='complete' + output_payload; on WorkerError writes
status='failed' with outcome label; on WorkerLostLock or detected
stolen-lock-at-write returns aborted (no DB write — reaper has
already re-pended). Defense-in-depth: even if agent returns
successfully, lost_lock_event.is_set() check skips the write.
- loop.py: worker_main_loop() polls the DB for pending attempts up
to MAX_CONCURRENT_WORKERS_PER_MACHINE, submits each to a
ThreadPoolExecutor. Honors stop_event for graceful shutdown
(drains in-flight before exit).
tools/controller/db/session.py: StaticPool for in-memory SQLite so
the heartbeat thread + runner write + dequeue all see the same DB
(without this, ":memory:" gives each connection an independent DB).
16 new tests in test_worker.py: instance ID format/uniqueness;
heartbeat tick (hold + steal); runner happy path; 5 error paths
(worker error / unexpected exception / WorkerLostLock raised /
stolen lock at write / lost_lock_event set defense-in-depth); 4
loop scenarios (single attempt, role filter skip, empty queue
exit-on-stop, explicit instance_id).
Total: 157 controller tests; full auto_agents suite 2519 pass.
|