a21466add232d59cdec1604e09d58ca05659a623
7 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 drew
|
016b348117 |
feat(controller): grooming gate (Phase 0 + Phase 1 worker-shape dispatch)
Phase 0 (foundation):
- Cause enum (controller_events.cause) for action attribution
- Schema: grooming_decisions audit table; workflows gains
grooming_evaluated_at + deferred_reason + deferred_at +
deferred_target_workflow_id; pulls gains touched_files
- audit_comments: CLOSE / DEFER templates + render_comment_template
- forgejo_writes: close_issue + defer_issue 5-step crash-safe protocol
(fingerprint dedup, error matrix, dry-run)
- patch_pr_state callback in forgejo_http
- grooming_config: 22-env-var frozen-dataclass config + log_effective
- pulls.touched_files cache extension (_pipeline_cache.py schema v8)
- reaper.reap_grooming_decisions audit-retention sweep
- reconciliation RESUME guard (deferred_reason)
Phase 1 (worker-queue shape, 2026-05-25):
- New state: GROOMING. New events: grooming_started, groom_verdict_
{proceed,defer,close}. 5 new transitions; all invariants still clean
- GroomingInputV1 + GroomingOutputV1 Pydantic contracts
- outcomes._map_grooming_outcome routes verdicts to state-machine events
- prefetch.build_grooming_stage_b_input + list_open_prs callback
- scheduler GROOMING -> grooming_stage_b role
- promote: cfg-gated DISCOVERED -> GROOMING when CONTROLLER_GROOMING_
ENABLED=true; issues skip grooming
- forgejo_writes decomposed: close_act/defer_act (Forgejo writes only;
state-machine already transitioned) + close_decide_and_act/
defer_decide_and_act (Phase 0 callers); _apply_workflow_transition
is underscore-private
- grooming.py library: tokenization, suspicion scoring (Jaccard +
weighted overlap), deterministic checks, action -> verdict mapping
- mcp/grooming_builder.py: 14-tool FastMCP server emits GroomingOutputV1
- .opencode/agents/grooming-stage-b.md: duplicate-detection agent
prompt (claude-haiku-4-5)
- grooming_side_effects.run_grooming_side_effects_tick: per-state tick
performs Forgejo writes after groom_verdict_{defer,close} fires.
Filters on event_type='transition' + payload.event (centralizes the
convention pending Phase 2's latest_transition_event helper)
- GroomingCallbacks frozen dataclass; loop.py + __main__.py wired
Worker role registry (single source of truth):
- worker/roles.py: WORKER_ROLES + WorkerRoleSpec + default_roles_csv
+ output_filename_for. agent_runner.ROLE_TO_MCP_MODULE / ROLE_TO_
OUTPUT_MODEL derive from it; opencode_session.agent_name_for reads
it for flat cases; all 6 prompt builders use output_filename_for;
worker --roles default = default_roles_csv(); launcher script
derives --roles via shell substitution. Cross-site invariant test
enforces alignment across 5 sites + opencode.json MCP registry.
Phase 0 silent-bug fix:
- reconciliation.py RESUME guard SELECT now includes deferred_reason
(was missing since Phase 0; guard was a silent no-op). Tightened
from getattr to attribute access to fail fast on future omissions.
Tests (1456 total, +91 grooming-specific):
- test_grooming_phase0.py: 34 tests (orchestrator matrix, crash
recovery, idempotency, dry-run)
- test_grooming_phase1.py: 60 tests (library, contracts, state
machine, outcomes, scheduler, promote, prefetch, act-variants
with signature parity, side-effect tick incl. natural-idempotency
+ executed-flag-skip + verdict-mismatch + reconciliation RESUME)
- test_mcp_builders.py TestGroomingBuilder: 29 tests (happy paths
+ 22 validation rules + Pydantic round-trip + master-tick-read-
path companion)
- test_worker_agent_runner.py TestRoleMaps: cross-role wiring
alignment + agent-prompt-vs-worker-fallback filename contract +
inspect.signature equality (close_act/defer_act vs
close_issue/defer_issue)
- test_state_machine.py: transition count 51 -> 56 +
events_from_grooming
Live-validated end-to-end on 4 staged sentinel PRs (#55-#58) in
dry_run: agent emits verdicts via MCP, state-machine transitions
fire, side-effect tick writes audit row, deferred_reason gates
reconciliation RESUME correctly.
Deferred refinements + Phase 2 prerequisite (latest_transition_event
helper) tracked in .drew/regressions-plan.md "Phase 1 follow-up
backlog".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
eb01eb0172 |
feat(controller): dual-mode launcher (fork/prod) + DB-mode validator
Adds the operator surface for switching the controller pipeline between
the personal fork (drew/cleveragents-core) and the canonical repo
(cleveragents/cleveragents-core) via a MODE env + --prod CLI flag,
backed by safety primitives that make a wrong-mode launch loud rather
than silent.
run-controller-state-machine-pipeline.sh: --prod flag and MODE env
(primary home: .devcontainer/.env) select fork vs prod. After resolving
MODE, the launcher auto-sources the matching overlay file
(.devcontainer/.env.{fork,prod}) and asserts MODE didn't drift during
the source step. The drift assertion uses a readonly snapshot under an
obscure variable name so a stray ``MODE=fork`` in .env.prod aborts the
launch with a clear bash error rather than silently demoting the run.
CONTROLLER_RUN_DIR_ROOT now overrides the trial /tmp path so prod can
use a persistent /var/lib/cleveragents/run dir.
tools/launch_prod.sh (new): sibling to launch_fork.sh with the opposite
safety primitive — affirmative GET /repos/{owner}/{repo} that asserts
the target is non-fork, exists, isn't archived, and the bot has push.
On any failure, no env is exported. Honors HAL_* aliases for parity
with launch_fork.sh and prints a hard-to-miss PROD-MODE banner.
tools/controller/deploy/validate_db_mode.py (new): stamps a _mode_marker
table on each SQLite db (controller DB + telemetry cache) on first use,
asserts a match on every subsequent launch, and moves mismatched files
aside as <name>.<prior-mode>.bak.<ts> — never deletes. The --adopt flag
lets an operator grandfather in already-good pre-marker data without
losing history. Wired into the launcher's startup sequence before
OpenCode and the master start.
tools/_cache_path.py (new): single source of truth for the per-(owner,
repo) Forgejo cache file convention. .opencode/telemetry/server.py and
the launcher both delegate here so the dual-source-truth drift risk is
eliminated. tools/_pipeline_cache.py and tools/controller/db/models.py
documented as not owning the _mode_marker table so future migrations
leave it alone.
.opencode/telemetry/server.py: hosts the llm_activity scraper as a
background subprocess thread (60s cadence, --since-hours 1 in steady
state, full backfill on first tick). Re-homes the cost-telemetry data
path after the pr_state_warmer was retired by the controller migration
— without this the Cost tab freezes when the warmer's loop is gone.
Subprocess (not in-process) for isolation; failures swallowed.
opencode.json: local-claude provider's baseURL now reads
{env:LOCAL_PROXY_URL} instead of the literal http://127.0.0.1:3456/v1,
matching the apiKey pattern already in use.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
0bc734c020 |
style: ruff format the controller-state-machine branch (288 files)
Applies `ruff format` to the accumulated formatting debt on this branch. Formatting-only — no behavioral changes. Required for CI/lint's format gate (`nox -s format -- --check`), which the branch was failing on 288 tracked files that drifted from ruff's canonical style. In-progress WIP files are intentionally excluded so this commit stays a clean formatting-only diff. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
c8677d985d |
fix(controller): batch J — round-3 cleanup (R3, R4, R5, R6, R8, R10)
R3 — delete unused schema columns: ``workflows.ci_flake_retries_remaining`` and ``workflows.awaiting_ci_started_at`` shipped in round-1 batch D as "staged for future tick handlers" — but no producer or reader ever used them. ci_poll.py uses ``entered_state_at`` (already populated on every transition) as the AWAITING_CI start timestamp. Delete both columns; the flake-retries counter should ship with its producer, not as speculative schema. R4 — safe_json_dumps raises on sets: Previously sets/frozensets were silently coerced to sorted lists. This contradicted the "raise loudly" design intent (JSON has no native set; a reader doing ``parsed["tags"]`` would get a list, losing set algebra). Now raises with an actionable message pointing the producer at ``sorted(list(...))`` for explicit conversion. R5 — TestSQLiteConcurrentDequeue docstring honest: The test name implied it pinned SQLite's busy_timeout retry. It doesn't — :memory: + StaticPool means both threads share one connection (SQLAlchemy serializes per-connection). Updated docstring to say what the test ACTUALLY pins (Python-level serialization safety, exactly-one-winner, clean loser-reason) and explicitly what it doesn't (cross-process SQLITE_BUSY retry, which would need file-backed SQLite + QueuePool — not shipped because multi-machine requires Postgres). R6 — safety timer on test_loop_runs_ci_poll_exhaustion_on_cadence: Previously the test relied entirely on on_iter setting stop when workflows_exhausted>0. If the logic regressed (SQL schema drift, on_iter never seeing the count), the test wedged CI indefinitely. Now armed with threading.Timer(5.0, stop.set) safety net + an assertion that surfaces the failure mode if the safety timer fired first. R8 — _json_safe scope documented honestly: The docstring claimed "everywhere the controller serializes" but the encoder is only wired at runner.py and scheduler.py — the two sites that serialize WORKER-ORIGINATED payloads. Other json.dumps call sites (event-row payloads, discovery markers) serialize fixed-shape dicts of native types and don't need restriction. Updated docstring to scope the claim accurately. R10 — missing test assertions added: - test_strict_parser_coverage_blocks_startup_with_stubs: now captures logs + asserts the operator-facing error message is emitted (so journald shows the cause; a silent rc=2 would be confusing). - test_externally_merged_takes_priority_over_label_removal: now asserts event_type="external-merge" + reason="externally-merged" (a regression that transitioned correctly with the wrong reason in the audit trail would now be caught). - test_second_pause_captures_post_resume_state (NEW): pause/resume/ pause cycle. After RESUME + workflow advances to REVIEWING, a second PAUSE must capture REVIEWING as pre_pause_state (not the original IMPLEMENTING). Round-2 coverage only exercised first pause. Total: 696 controller tests pass (+3 net from new + updated tests), 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
d71046b9a0 |
fix(controller): batch D — PID-reuse, AWAITING_CI escape, flake bound, scheduler skip
Four safety items from the consolidated adversarial-review punch list.
ITEM 8 — PID-reuse hazard in janitor:
The janitor SIGKILL'd whatever process happened to live at the
sidecar's recorded subprocess_pid. Between sidecar write and janitor
sweep, the OS can reuse the PID for an unrelated process; the janitor
was killing innocents under fork-heavy workloads.
Fix:
- ``session_sidecar.py``: added ``subprocess_starttime`` field
(Optional[int]) + ``read_proc_starttime(pid)`` helper that reads
``/proc/{pid}/stat`` field 22 (clock ticks since boot — monotonic
for a (boot, pid) pair).
- ``WorkerSession.from_dict`` filters unknown keys so forward + back
compat with sidecars from earlier/later versions is preserved.
- ``janitor._pid_alive`` and ``_kill_with_grace`` accept
``expected_starttime``; on mismatch they short-circuit and DON'T
signal the impostor.
- ``_kill_with_grace`` return semantics tightened: True iff a signal
was actually delivered (False for "PID gone" / "PID reused"). The
``sessions_killed`` counter now reflects real kills.
ITEM 9 — AWAITING_CI escape from infinite poll:
Previously AWAITING_CI could only exit via ``ci_green`` /
``ci_red_*`` / ``ci_flake_retry`` — if CI hangs forever (runner
outage, broken integration, etc.) the workflow had no controller-
driven STUCK path; only operator_unstick could rescue it.
Fix: new ``ci_polling_exhausted`` event → STUCK. The master's
AWAITING_CI poll handler is the natural place to emit it once a
threshold passes (deferred to a follow-up — Phase 1k+ ships the
event in the table; the timer fires it).
ITEM 10 — ci_flake_retry was unbounded:
The ``ci_flake_retry`` self-loop on AWAITING_CI had no encoded
ceiling. Pathological flaky CI could loop forever (the docstring
said "retry once per gate" but nothing enforced it).
Fix:
- New ``workflows.ci_flake_retries_remaining`` column (server_default
'1', default 1 — operators tune via ``CONTROLLER_CI_FLAKE_RETRIES``
at startup or via direct UPDATE).
- New ``ci_flake_retries_exhausted`` event → ESCALATING. Master
decrements the column on each ci_flake_retry; at 0 the next CI
failure routes through ci_red_* (regular path) or this new
event (escalates if the operator wants a hard ceiling).
ITEM 11 — scheduler now skips PAUSED workflows:
Without this, the scheduler could enqueue a fresh attempt for a
PAUSED workflow between two reconciliation ticks (race: label
removed at T+0, reconciliation runs at T+300, scheduler ticks at
T+30 with stale DB state). The window is at most one attempt of
worker work.
Fix: ``schedule_next_attempts`` SQL now lists only
{ANALYZING, IMPLEMENTING, REVIEWING, CONFLICT_RESOLVING, ESCALATING}
explicitly; PAUSED is excluded by absence. Reconciliation owns the
PAUSED → resume transition; scheduler doesn't touch it.
Schema additions:
- ``workflows.ci_flake_retries_remaining`` (INTEGER NOT NULL DEFAULT 1)
- ``workflows.awaiting_ci_started_at`` (TIMESTAMP NULL) — for the
poll-exhaustion timer (timer impl deferred; column is staged).
Tests:
- TestReadProcStarttime — 3 tests (Linux skip-guard) for the
/proc/pid/stat parser (self-pid > 0, missing pid is None,
invalid pid is None).
- TestJanitor::test_pid_reuse_defended_via_starttime — pins the
contract end-to-end (real subprocess + fabricated wrong starttime
→ janitor doesn't signal).
- TestPhase1kPlusTransitions — 5 tests pinning the new events +
proving the load-bearing invariants still pass.
- test_scheduler_skips_paused_workflows — pins item 11.
Total: 603 controller tests pass (+10 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
drew
|
feeec3e9c7 |
fix(controller): batch B — PAUSED state for label-gate pause/resume (item 2)
Adversarial review flagged: removing the opt-in label transitions a live workflow to ABANDONED — but ABANDONED is TERMINAL with only ``operator_unstick`` re-entry → DISCOVERED, losing all prior controller_events continuity. Operators removing the label to "pause" a long-running PR will be surprised it restarted from scratch. Fix: introduce a non-terminal ``PAUSED`` state. State machine changes (``tools/controller/state_machine.py``): - ``PAUSED`` added to KNOWN_STATES (non-terminal — has exits via ``opt_in_label_restored`` and ``operator_unstick``). - ``opt_in_label_removed`` / ``opt_in_label_restored`` events documented in EVENTS but NOT listed per-state in TRANSITIONS — they're out-of-band master-driven events written directly by reconciliation. Listing them per-state breaks the per-state-event-set invariants (ESCALATING / CONFLICT_RESOLVING). - ``(PAUSED, operator_unstick) → DISCOVERED`` for the escape hatch. Schema change (``tools/controller/db/models.py``): - ``workflows.pre_pause_state: Mapped[str | None]`` column captures the resume target. Master writes it on pause; clears it on resume. Reconciliation logic (``tools/controller/master/reconciliation.py``): - New ``_apply_transition_with_pre_pause`` helper writes both ``current_state`` and ``pre_pause_state`` atomically + emits the ``reconciliation`` event row. - On label removal (current != PAUSED): captures pre_pause_state, transitions to PAUSED. - On label restoration (current == PAUSED): reads pre_pause_state (fallback DISCOVERED for legacy NULL data), transitions back, clears pre_pause_state. - PAUSED workflows are now SCANNED by reconciliation (not just non-terminals) so we can detect label-restored. Behaviour matrix: | current | label | result | |---------|----------|------------------------------------------| | any != | absent | → PAUSED, pre_pause_state = current | | PAUSED | present | → pre_pause_state (or DISCOVERED) | | PAUSED | absent | stays PAUSED (no transition) | | any != | present | regular state checks (no-op for label) | Tests (test_label_gate.py refactor + 3 new tests): - test_label_removed_pauses_workflow (was: abandons) - test_label_restored_resumes_from_pre_pause_state (new) - test_paused_workflow_without_label_stays_paused (new) - test_resume_fallback_when_pre_pause_state_missing (new — legacy data without the new column) - Existing event-reason test still passes (reason string unchanged). Total: 590 controller tests pass, 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
36b133ec5e |
feat(controller): Phase 1b — DB schema + dequeue helper + payload guard
Five SQLAlchemy 2.0 declarative models implementing plan v6/v9's
unified workflow schema. Cross-dialect (SQLite for tests + local dev,
Postgres for multi-machine production). Lock columns on
workflow_attempts implement the multi-machine-safe dequeue protocol
(plan v5).
Modules:
- tools/controller/db/models.py:
- Workflow (kind discriminator pr/issue, unique on owner+repo+kind+
entity_number, parent_workflow_id FK for issue→PR linkage)
- WorkflowAttempt (status/locked_by_instance/locked_at/
lock_heartbeat_at/lock_ttl_seconds/pickup_count + CHECK
constraints on status enum and pickup_count≥0; partial indexes
on the pending/in_progress/complete hot paths)
- ControllerEvent (Forgejo-write replay support kept in-schema even
though v9 simplified to Forgejo-first protocol; allows v3-style
upgrade later without migration)
- FlakeHistory (composite PK; supports the v6 flake-learning
heuristic)
- CIObservation (raw CI state history; 90-day retention to be
enforced by a sweep task)
- AutoincrementPk variant (Integer on SQLite where it autoincrements
via rowid; BigInteger on Postgres for BIGSERIAL); JsonColumn
variant (JSON on SQLite, JSONB on Postgres)
- tools/controller/db/session.py: build_engine (per-dialect tuning —
SQLite WAL + foreign_keys + busy_timeout; Postgres pool_pre_ping);
create_all (idempotent); session_scope (transactional context).
- tools/controller/db/dequeue.py: dequeue_one (one row atomically;
Postgres path uses SELECT FOR UPDATE SKIP LOCKED, SQLite path uses
UPDATE-WHERE-id-IN-SELECT-LIMIT-1 with RETURNING). Bumps pickup_count
on dequeue; respects max_pickups guard (default 3 per v6 blocker fix).
Returns DequeueResult dataclass with role/tier/pickup_count and
reason on miss.
- tools/controller/db/payload_guard.py: enforce_input_payload_size
with 4MB cap and 5-step truncation priority (older_summary → oldest
verbatim → comments → full_diff → CI failure excerpts). Raises
PayloadTooLargeError after all steps exhausted; master maps to
workflow STUCK with reason='input-too-large'.
- pyproject.toml: new optional extras `controller-db` pinning
sqlalchemy + psycopg2-binary (latter installed only for prod
multi-machine deploy; tests use stdlib sqlite3 via SQLAlchemy's
SQLite dialect which is already pulled in transitively via alembic).
37 new tests across test_db_schema/dequeue/payload_guard; 141
controller tests total; full auto_agents suite 2503 pass (no
regressions).
|