751dbfed337bc18ec6dd96cc7b3f241bb3207acb
7 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 drew
|
c8677d985d |
fix(controller): batch J — round-3 cleanup (R3, R4, R5, R6, R8, R10)
R3 — delete unused schema columns: ``workflows.ci_flake_retries_remaining`` and ``workflows.awaiting_ci_started_at`` shipped in round-1 batch D as "staged for future tick handlers" — but no producer or reader ever used them. ci_poll.py uses ``entered_state_at`` (already populated on every transition) as the AWAITING_CI start timestamp. Delete both columns; the flake-retries counter should ship with its producer, not as speculative schema. R4 — safe_json_dumps raises on sets: Previously sets/frozensets were silently coerced to sorted lists. This contradicted the "raise loudly" design intent (JSON has no native set; a reader doing ``parsed["tags"]`` would get a list, losing set algebra). Now raises with an actionable message pointing the producer at ``sorted(list(...))`` for explicit conversion. R5 — TestSQLiteConcurrentDequeue docstring honest: The test name implied it pinned SQLite's busy_timeout retry. It doesn't — :memory: + StaticPool means both threads share one connection (SQLAlchemy serializes per-connection). Updated docstring to say what the test ACTUALLY pins (Python-level serialization safety, exactly-one-winner, clean loser-reason) and explicitly what it doesn't (cross-process SQLITE_BUSY retry, which would need file-backed SQLite + QueuePool — not shipped because multi-machine requires Postgres). R6 — safety timer on test_loop_runs_ci_poll_exhaustion_on_cadence: Previously the test relied entirely on on_iter setting stop when workflows_exhausted>0. If the logic regressed (SQL schema drift, on_iter never seeing the count), the test wedged CI indefinitely. Now armed with threading.Timer(5.0, stop.set) safety net + an assertion that surfaces the failure mode if the safety timer fired first. R8 — _json_safe scope documented honestly: The docstring claimed "everywhere the controller serializes" but the encoder is only wired at runner.py and scheduler.py — the two sites that serialize WORKER-ORIGINATED payloads. Other json.dumps call sites (event-row payloads, discovery markers) serialize fixed-shape dicts of native types and don't need restriction. Updated docstring to scope the claim accurately. R10 — missing test assertions added: - test_strict_parser_coverage_blocks_startup_with_stubs: now captures logs + asserts the operator-facing error message is emitted (so journald shows the cause; a silent rc=2 would be confusing). - test_externally_merged_takes_priority_over_label_removal: now asserts event_type="external-merge" + reason="externally-merged" (a regression that transitioned correctly with the wrong reason in the audit trail would now be caught). - test_second_pause_captures_post_resume_state (NEW): pause/resume/ pause cycle. After RESUME + workflow advances to REVIEWING, a second PAUSE must capture REVIEWING as pre_pause_state (not the original IMPLEMENTING). Round-2 coverage only exercised first pause. Total: 696 controller tests pass (+3 net from new + updated tests), 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
d71046b9a0 |
fix(controller): batch D — PID-reuse, AWAITING_CI escape, flake bound, scheduler skip
Four safety items from the consolidated adversarial-review punch list.
ITEM 8 — PID-reuse hazard in janitor:
The janitor SIGKILL'd whatever process happened to live at the
sidecar's recorded subprocess_pid. Between sidecar write and janitor
sweep, the OS can reuse the PID for an unrelated process; the janitor
was killing innocents under fork-heavy workloads.
Fix:
- ``session_sidecar.py``: added ``subprocess_starttime`` field
(Optional[int]) + ``read_proc_starttime(pid)`` helper that reads
``/proc/{pid}/stat`` field 22 (clock ticks since boot — monotonic
for a (boot, pid) pair).
- ``WorkerSession.from_dict`` filters unknown keys so forward + back
compat with sidecars from earlier/later versions is preserved.
- ``janitor._pid_alive`` and ``_kill_with_grace`` accept
``expected_starttime``; on mismatch they short-circuit and DON'T
signal the impostor.
- ``_kill_with_grace`` return semantics tightened: True iff a signal
was actually delivered (False for "PID gone" / "PID reused"). The
``sessions_killed`` counter now reflects real kills.
ITEM 9 — AWAITING_CI escape from infinite poll:
Previously AWAITING_CI could only exit via ``ci_green`` /
``ci_red_*`` / ``ci_flake_retry`` — if CI hangs forever (runner
outage, broken integration, etc.) the workflow had no controller-
driven STUCK path; only operator_unstick could rescue it.
Fix: new ``ci_polling_exhausted`` event → STUCK. The master's
AWAITING_CI poll handler is the natural place to emit it once a
threshold passes (deferred to a follow-up — Phase 1k+ ships the
event in the table; the timer fires it).
ITEM 10 — ci_flake_retry was unbounded:
The ``ci_flake_retry`` self-loop on AWAITING_CI had no encoded
ceiling. Pathological flaky CI could loop forever (the docstring
said "retry once per gate" but nothing enforced it).
Fix:
- New ``workflows.ci_flake_retries_remaining`` column (server_default
'1', default 1 — operators tune via ``CONTROLLER_CI_FLAKE_RETRIES``
at startup or via direct UPDATE).
- New ``ci_flake_retries_exhausted`` event → ESCALATING. Master
decrements the column on each ci_flake_retry; at 0 the next CI
failure routes through ci_red_* (regular path) or this new
event (escalates if the operator wants a hard ceiling).
ITEM 11 — scheduler now skips PAUSED workflows:
Without this, the scheduler could enqueue a fresh attempt for a
PAUSED workflow between two reconciliation ticks (race: label
removed at T+0, reconciliation runs at T+300, scheduler ticks at
T+30 with stale DB state). The window is at most one attempt of
worker work.
Fix: ``schedule_next_attempts`` SQL now lists only
{ANALYZING, IMPLEMENTING, REVIEWING, CONFLICT_RESOLVING, ESCALATING}
explicitly; PAUSED is excluded by absence. Reconciliation owns the
PAUSED → resume transition; scheduler doesn't touch it.
Schema additions:
- ``workflows.ci_flake_retries_remaining`` (INTEGER NOT NULL DEFAULT 1)
- ``workflows.awaiting_ci_started_at`` (TIMESTAMP NULL) — for the
poll-exhaustion timer (timer impl deferred; column is staged).
Tests:
- TestReadProcStarttime — 3 tests (Linux skip-guard) for the
/proc/pid/stat parser (self-pid > 0, missing pid is None,
invalid pid is None).
- TestJanitor::test_pid_reuse_defended_via_starttime — pins the
contract end-to-end (real subprocess + fabricated wrong starttime
→ janitor doesn't signal).
- TestPhase1kPlusTransitions — 5 tests pinning the new events +
proving the load-bearing invariants still pass.
- test_scheduler_skips_paused_workflows — pins item 11.
Total: 603 controller tests pass (+10 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
a1c6646a64 |
fix(controller): batch C — placeholder patching + pickup_count semantics
Items 3 + 4 from the consolidated adversarial-review punch list. ITEM 3 — placeholders no longer poison the audit trail: The prefetch (master/prefetch.py) writes input_payload with ``attempt_id=0`` and ``attempt_number=1`` as placeholders because the autoincrement PK isn't known until after INSERT. Previously those values stayed in the DB forever — post-mortem queries against ``workflow_attempts.input_payload`` would show ``attempt_id=0`` and operators would chase ghosts. Fix: ``master/scheduler.py:_insert_pending_attempt`` now patches both fields with their real values: - attempt_number: patched BEFORE the INSERT (we compute it as MAX+1). - attempt_id: patched via a follow-up UPDATE after INSERT (we need the autoincrement first). One extra UPDATE per attempt; cheap compared to forever-incorrect audit trail. Test: ``test_scheduler_patches_attempt_id_and_number_into_payload`` asserts the stored payload carries the real values, not the placeholders. ITEM 4 — pickup_count tracks REAPS, not dequeues: Previously the dequeue path bumped ``pickup_count = pickup_count + 1`` on every successful pickup. With ``MAX_PICKUPS=3`` (default), 3 crashed-mid-attempt workers would STUCK the workflow — but that's the wrong semantic. A worker that successfully picks an attempt and runs it should NOT burn a pickup. Only failures (stale-heartbeat reset by the reaper) should count toward the exhaustion limit. Fix: - ``db/dequeue.py`` (both postgres + sqlite paths): removed the ``pickup_count = pickup_count + 1`` UPDATE. Dequeue is a healthy pickup; doesn't bump. - ``reaper.py``: added ``pickup_count = pickup_count + 1`` to the reset UPDATE. Each reap = one failed pickup. - Docstrings updated to reflect the new semantics in both files. Tests: - Updated existing assertions in ``test_db_dequeue.py`` and ``test_reaper_and_pickup_guard.py`` to reflect: dequeue keeps pickup_count; reaper bumps it. - ``TestPickupCountSemantics``: 2 new tests pin the contract end-to-end — N healthy dequeues stay at 0; alternating dequeue→reap→dequeue walks pickup_count up by 1 per reap. Impact: a worker pool that crashes 3 times mid-attempt now needs 3 REAPS (not 3 dequeues) to STUCK the workflow. With default TTL=600s + reaper_interval=60s, that's 30+ minutes of repeated mid-attempt failure before STUCK — appropriately conservative. Total: 593 controller tests pass (+3 new), 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
feeec3e9c7 |
fix(controller): batch B — PAUSED state for label-gate pause/resume (item 2)
Adversarial review flagged: removing the opt-in label transitions a live workflow to ABANDONED — but ABANDONED is TERMINAL with only ``operator_unstick`` re-entry → DISCOVERED, losing all prior controller_events continuity. Operators removing the label to "pause" a long-running PR will be surprised it restarted from scratch. Fix: introduce a non-terminal ``PAUSED`` state. State machine changes (``tools/controller/state_machine.py``): - ``PAUSED`` added to KNOWN_STATES (non-terminal — has exits via ``opt_in_label_restored`` and ``operator_unstick``). - ``opt_in_label_removed`` / ``opt_in_label_restored`` events documented in EVENTS but NOT listed per-state in TRANSITIONS — they're out-of-band master-driven events written directly by reconciliation. Listing them per-state breaks the per-state-event-set invariants (ESCALATING / CONFLICT_RESOLVING). - ``(PAUSED, operator_unstick) → DISCOVERED`` for the escape hatch. Schema change (``tools/controller/db/models.py``): - ``workflows.pre_pause_state: Mapped[str | None]`` column captures the resume target. Master writes it on pause; clears it on resume. Reconciliation logic (``tools/controller/master/reconciliation.py``): - New ``_apply_transition_with_pre_pause`` helper writes both ``current_state`` and ``pre_pause_state`` atomically + emits the ``reconciliation`` event row. - On label removal (current != PAUSED): captures pre_pause_state, transitions to PAUSED. - On label restoration (current == PAUSED): reads pre_pause_state (fallback DISCOVERED for legacy NULL data), transitions back, clears pre_pause_state. - PAUSED workflows are now SCANNED by reconciliation (not just non-terminals) so we can detect label-restored. Behaviour matrix: | current | label | result | |---------|----------|------------------------------------------| | any != | absent | → PAUSED, pre_pause_state = current | | PAUSED | present | → pre_pause_state (or DISCOVERED) | | PAUSED | absent | stays PAUSED (no transition) | | any != | present | regular state checks (no-op for label) | Tests (test_label_gate.py refactor + 3 new tests): - test_label_removed_pauses_workflow (was: abandons) - test_label_restored_resumes_from_pre_pause_state (new) - test_paused_workflow_without_label_stays_paused (new) - test_resume_fallback_when_pre_pause_state_missing (new — legacy data without the new column) - Existing event-reason test still passes (reason string unchanged). Total: 590 controller tests pass, 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
6ba1926c52 |
fix(controller): batch A — datetime json, Forgejo state map, ISO compare, dequeue docs
Four narrow bug fixes flagged by adversarial code review (items 1, 5, 6, 7 from the consolidated critique). ITEM 1 — datetime → json.dumps crash (silent write-after-work failure): - ``worker/runner.py:274`` and ``master/scheduler.py:283`` now pass ``default=str`` to ``json.dumps`` so nested datetime fields (e.g. CISummary.observed_at) serialize without raising. - Before this fix: a worker would do its real work, then crash on the terminal-state UPDATE with TypeError, get recorded as ``worker-internal-error``, and the output payload would be lost. - Test: TestDatetimeSerializationSafety in test_master_ci_summarize + test_scheduler_handles_datetime_in_input_payload in test_master_prefetch (both pin the regression — the with-default test passes, the without-default test asserts the TypeError so future maintainers see the failure mode). ITEM 5 — Forgejo state mapping completeness: - Extended ``_FORGEJO_STATE_TO_GATE_STATUS`` in ``master/ci_summarize.py`` to cover ``cancelled``, ``timed_out``, ``action_required``, ``queued``, ``in_progress``, ``neutral``, ``skipped``, ``stale`` — states observed across Forgejo / Gitea / GH-mirror that previously collapsed to ``pending``, telling the implementer "CI is still running" when really a job was cancelled. - ``cancelled`` / ``timed_out`` / ``action_required`` / ``stale`` now map to ``error`` (the gate failed). - ``queued`` / ``in_progress`` stay ``pending`` (still running). - ``neutral`` / ``skipped`` → ``passed``/``skipped`` (informational). - Test: TestExtendedForgejoStates — 6 tests covering each new state. ITEM 6 — lexicographic ISO comparison drops/dupes comments: - ``master/prefetch.py:_comment_bodies_since`` and ``_iso`` replaced with ``_to_aware_datetime`` + datetime comparison. Forgejo emits ``2026-05-18T12:00:00Z``; Python's ``datetime.isoformat()`` emits ``2026-05-18T12:00:00+00:00`` — a string compare gives 'Z' (0x5A) vs '+' (0x2B) which silently misorders timestamps. - Now parses via ``datetime.fromisoformat`` (with Z → +00:00 rewrite), defaults naive timestamps to UTC, and compares as ``datetime``. - Test: test_comments_filter_handles_z_suffix_vs_offset_form pins the regression. ITEM 7 — false BEGIN IMMEDIATE claim in dequeue docstring: - The dequeue docstring claimed ``BEGIN IMMEDIATE`` was applied by session_scope; it wasn't. Attempted a global ``begin``-event listener that conflicted with StaticPool's shared-connection model (test_prefetch_callback_works_in_scheduler broke). - Reverted to a documentation fix: SQLite stays on default BEGIN DEFERRED (the SQLITE_BUSY retry via busy_timeout=5s is acceptable for single-host dev) + the docs make MULTI-MACHINE REQUIRES POSTGRES explicit at three call sites (db/session.py, db/dequeue.py, RUNBOOK.md was already updated in Phase 1l). Postgres has FOR UPDATE SKIP LOCKED which is what production actually uses. Tests: 586 controller tests pass (+10 new), 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
eab476e48e |
feat(controller): Phase 1c — worker controller skeleton
The dequeue+lock+heartbeat+runner+loop machinery. Production
OpenCode + MCP invocation slots in via the agent_runner callable
(Phase 1c-2). This commit is the structural foundation:
tools/controller/worker/:
- identity.py: build_instance_id() → "{hostname}/{pid}/{worker_uuid}"
per plan v9 (slash delimiter; IPv6-safe; uuid4 prefix for
per-instance uniqueness).
- heartbeat.py: Heartbeat thread that updates lock_heartbeat_at
every interval (default 30s). v9 simplified: TTL-only (no activity
tracking). UPDATE … WHERE locked_by_instance=us; rowcount=0 →
lost_lock_event.set() and thread exits, letting reaper handle it.
- runner.py: run_one_attempt() drives one attempt end-to-end.
Starts heartbeat → invokes agent_runner → on success writes
status='complete' + output_payload; on WorkerError writes
status='failed' with outcome label; on WorkerLostLock or detected
stolen-lock-at-write returns aborted (no DB write — reaper has
already re-pended). Defense-in-depth: even if agent returns
successfully, lost_lock_event.is_set() check skips the write.
- loop.py: worker_main_loop() polls the DB for pending attempts up
to MAX_CONCURRENT_WORKERS_PER_MACHINE, submits each to a
ThreadPoolExecutor. Honors stop_event for graceful shutdown
(drains in-flight before exit).
tools/controller/db/session.py: StaticPool for in-memory SQLite so
the heartbeat thread + runner write + dequeue all see the same DB
(without this, ":memory:" gives each connection an independent DB).
16 new tests in test_worker.py: instance ID format/uniqueness;
heartbeat tick (hold + steal); runner happy path; 5 error paths
(worker error / unexpected exception / WorkerLostLock raised /
stolen lock at write / lost_lock_event set defense-in-depth); 4
loop scenarios (single attempt, role filter skip, empty queue
exit-on-stop, explicit instance_id).
Total: 157 controller tests; full auto_agents suite 2519 pass.
|
||
|
drew
|
36b133ec5e |
feat(controller): Phase 1b — DB schema + dequeue helper + payload guard
Five SQLAlchemy 2.0 declarative models implementing plan v6/v9's
unified workflow schema. Cross-dialect (SQLite for tests + local dev,
Postgres for multi-machine production). Lock columns on
workflow_attempts implement the multi-machine-safe dequeue protocol
(plan v5).
Modules:
- tools/controller/db/models.py:
- Workflow (kind discriminator pr/issue, unique on owner+repo+kind+
entity_number, parent_workflow_id FK for issue→PR linkage)
- WorkflowAttempt (status/locked_by_instance/locked_at/
lock_heartbeat_at/lock_ttl_seconds/pickup_count + CHECK
constraints on status enum and pickup_count≥0; partial indexes
on the pending/in_progress/complete hot paths)
- ControllerEvent (Forgejo-write replay support kept in-schema even
though v9 simplified to Forgejo-first protocol; allows v3-style
upgrade later without migration)
- FlakeHistory (composite PK; supports the v6 flake-learning
heuristic)
- CIObservation (raw CI state history; 90-day retention to be
enforced by a sweep task)
- AutoincrementPk variant (Integer on SQLite where it autoincrements
via rowid; BigInteger on Postgres for BIGSERIAL); JsonColumn
variant (JSON on SQLite, JSONB on Postgres)
- tools/controller/db/session.py: build_engine (per-dialect tuning —
SQLite WAL + foreign_keys + busy_timeout; Postgres pool_pre_ping);
create_all (idempotent); session_scope (transactional context).
- tools/controller/db/dequeue.py: dequeue_one (one row atomically;
Postgres path uses SELECT FOR UPDATE SKIP LOCKED, SQLite path uses
UPDATE-WHERE-id-IN-SELECT-LIMIT-1 with RETURNING). Bumps pickup_count
on dequeue; respects max_pickups guard (default 3 per v6 blocker fix).
Returns DequeueResult dataclass with role/tier/pickup_count and
reason on miss.
- tools/controller/db/payload_guard.py: enforce_input_payload_size
with 4MB cap and 5-step truncation priority (older_summary → oldest
verbatim → comments → full_diff → CI failure excerpts). Raises
PayloadTooLargeError after all steps exhausted; master maps to
workflow STUCK with reason='input-too-large'.
- pyproject.toml: new optional extras `controller-db` pinning
sqlalchemy + psycopg2-binary (latter installed only for prod
multi-machine deploy; tests use stdlib sqlite3 via SQLAlchemy's
SQLite dialect which is already pulled in transitively via alembic).
37 new tests across test_db_schema/dequeue/payload_guard; 141
controller tests total; full auto_agents suite 2503 pass (no
regressions).
|