355af84fb1740d74f2a48bf35768ff2fd6adbb98
7 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 drew
|
2f1be34d12 |
feat(auto-agents): implementer parity — verify-invariant verifiers, implementer-helpers skill, watchdog gate, _opencode_worker audit
Closes the four open items in `docs/development/auto-agents-tier-2-3-plan.md` § "Revised remaining scope (2026-05-08)" plus three rounds of fresh-eyes critique fold-in (rounds 3, 5, and post-round-5 polish). Highlights: - New continuous invariant verifiers on a shared `_verify_common.py` substrate: `verify_review_invariant.py` (R1: approval-without-CI) and `verify_implementer_invariant.py` (I1: head-commit fails commit-lint, I2: PR description missing Epic reference). Strictly additive cron-job- shaped scripts that open idempotent `auto/invariant-violation` issues; safe to run every 15 minutes in production. - New `implementer-helpers` skill at `.opencode/skills/implementer-helpers/SKILL.md` + CLI at `tools/implementer_validate.py` (4 subcommands: validate-commit-message, validate-pr-compliance, validate-file-budget, validate-changelog). Mirrors the reviewer side; `tools/_commit_lint.py` is shared so a future change to commit policy updates one place. - `auto-agents.md` watchdog gate: `DISPATCHERS_RUNNING=1` puts the primary orchestrator into watchdog-only mode. Heartbeat resolution + age computation factored into `tools/_watchdog_helpers.py` + the CLI `tools/watchdog_check.py` so the agent only needs `python3 tools/watchdog_check.py *` and `sleep *` bash permissions. The reader honours the env-var override first, then falls back to a freshest-mtime scan across `/var/run` / `$XDG_RUNTIME_DIR` / `/tmp` (deliberately diverging from the dispatcher's first-existing fallback to guard against stale heartbeats from previous root-owned sessions masking healthy user-mode heartbeats). - `_opencode_worker.py` audit: structured `error_kind` classification at every transport-error / timeout return site, plumbed through `_dispatch_runtime.py` into the cycle-log; new `_request_read` retry helper (3 × 0.5s linear backoff, transport-only) wrapping every idempotent read in a worker session so a single transient flap on a polling GET cannot trash a 10-minute worker session. - Static heredoc lint at `tests/auto_agents/test_prompt_heredoc_lint.py` glob-walks every agent prompt and skill recipe markdown, rejecting any heredoc bash recipe in a fenced code block (per `bash-commands.md` rule 2 — heredocs fail at OpenCode's permission-engine parse time). - `bash-commands.md` rule 2 + its fix-it advice both lead with apostrophe-safe `printf "%s" "<body>"` (double-quoted) form; single-quoted form documented as the fragile JSON-only fallback. - `CHANGELOG.md` carries the full multi-round narrative (round 3 CRITICAL/HIGH/MEDIUM/LOW fold-in, round 5 docstring drift + telemetry refactor + broader heredoc lint scope, post-round-5 doc-drift polish). Net delta: +911 passing tests / 3 skipped (was 825 / 3); ruff clean on every new file; pre-existing lint debt in `_dispatch_runtime.py`, `_opencode_worker.py`, `conftest.py`, `_commit_lint.py` unchanged and out of scope for this commit. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
drew
|
05d069cc3b |
fix(auto-agents): disambiguate transient vs terminal unknown after seen_busy
The previous polling-race fix made unknown && seen_busy loop forever, which hung every clean dispatcher cycle on the watchdog timeout because OpenCode permanently removes finished sessions from /session/status (status=unknown is the terminal signal there). Add _session_assistant_in_flight() that checks the latest assistant message's time.completed field on the unknown path: - in-flight => transient blip between turns, keep polling - complete => terminal, break out - transport error => conservative, keep polling The helper is off the happy path (busy sessions never trigger the extra HTTP call). Four new tests in test_opencode_worker_observability covering all three branches plus a direct unit test of the helper. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
drew
|
9bf44e174d |
fix(auto-agents): worker polling-race + session observability bundle
- Fix _opencode_worker.run_session_blocking treating ``unknown && seen_busy`` as transport-error after the warmup deadline. The session-status map briefly drops the entry between assistant turns; the dispatcher now keeps polling instead of killing a worker that is actively producing tokens. Regression that took out PR #30's reviewer cycle today. - Add per-poll state-transition INFO logging (init->busy, busy->unknown->busy, busy->idle), bounded per-poll history, and a per-assistant-turn metrics summary log so a 6-minute reviewer cycle is reconstructable from the dispatcher log. - Add session-message archival: every terminal return snapshots the full /session/{id}/message payload (plus per_turn metrics and state_history) to .dispatcher-logs/sessions/<file>.json before DELETE /session. Survives session deletion, so failed runs can be post-mortemed long after the cycle ends. Disable via OPENCODE_WORKER_ARCHIVE_DISABLED=1 (the test suite's autouse default). - Wire archives into the telemetry console: new "Sessions" tab, GET /api/sessions/archived index endpoint, GET /api/sessions/ archived/detail filename-validated detail endpoint with path-traversal guard. Operators can browse failed worker runs from the browser instead of grepping JSON files. - 24 new unit tests across two files covering the polling-race regression, archive paths on completed/timeout/transport-error, per-turn aggregation, env var resolution, write-failure swallow, archive index ordering + cap, and detail-endpoint path-traversal rejection. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
drew
|
2cbe62a70c |
fix(auto-agents): F1+F2+F3 telemetry liveness, reviewer perf (A+C), bash rules
Bundles a long-overdue set of fixes that surfaced while watching the single-PR pipeline test against PR #30 the morning of 2026-05-07. # F1 — long-worker liveness contract The dispatcher heartbeat was only refreshed *between* worker sessions. On a 30-minute review the heartbeat file went stale, and any heartbeat-watchdog (dispatchers-launcher.sh / cleveragents-dispatchers.service) would SIGTERM a perfectly-healthy worker mid-cycle, orphaning the OpenCode session and the auto/claimed-* lock. ``_opencode_worker.run_session_blocking`` now accepts an ``on_poll`` callback fired once per status-poll iteration; ``_dispatch_runtime.dispatch_one`` and ``conflict_drive.py`` wire it to ``write_heartbeat(cfg.heartbeat_path)``. Callback exceptions are logged and swallowed so a transient EROFS on the heartbeat path can never mask a successful worker completion. # F2 — in-flight cycle visibility (schema v5) The ``dispatch_*_cycles`` tables previously only recorded a row at cycle *end*. While a worker was running, the operator's only signal was the heartbeat file — and even that became stale (see F1). Schema bumped to v5: ``ended_at`` is now nullable and ``cycle_id`` carries a UNIQUE index. ``begin_cycle`` writes the in-flight row at start; ``finish_cycle`` updates it at end. ``run_one_cycle``'s try/finally guarantees ``finish_cycle`` runs even when ``collect_candidates`` / ``dispatch_one`` raises, so an orphan ``ended_at IS NULL`` can no longer be stuck forever after a crash. The v4→v5 migration is now defined in ONE place — a set of helpers in ``_forgejo_cache.py`` (``DISPATCH_CYCLE_TABLES``, ``_dispatch_cycle_create_sql``, ``_dispatch_cycle_index_sqls``, ``migrate_dispatch_cycle_table_to_v5``, ``ensure_dispatch_cycle_schema``). Both ``ForgejoCache._migrate_to_v5_in_flight_rows`` and ``_dispatch_runtime.ensure_cycle_table`` import from there, eliminating the drift risk of the previous duplicated DDL. Pre-existing rows are preserved verbatim across the migration. # F3 — telemetry surface for the new state ``/api/health`` now returns ``in_flight_cycle: {cycle_id, started_at, session_id, candidates_count, elapsed_s}`` per dispatcher daemon and a ``running_long_worker: bool`` flag (heartbeat older than 600s AND a matching pid alive — should never fire under healthy F1 operation, so when it does it points at a real bug). The Drivers and Overview tabs in ``.opencode/telemetry/{index.html,app.js,style.css}`` render in-flight rows with a tinted background + "in flight" pill, daemon tiles get a dashed border for the long-worker state, and each tile shows the running cycle's id + elapsed time inline. # Fix A — reasoningEffort high → medium for pr-review-worker On its own that change alone would not have been enough, but combined with Fix C below it dropped a representative cycle from "timed out at 30:00" to a target ~2-3min. Pure config change in ``.opencode/agents/pr-review-worker.md``; no code path touched. # Fix C — pre-fetch PR diff in dispatch_review and embed in prompt The reviewer used to spawn a ``git-isolator-util`` subagent, which shelled out to ``git clone``, ``git fetch``, and ``git diff master...HEAD``. That subagent burned 90+ seconds and several token budgets per cycle. ``dispatch_review.py`` now fetches the unified diff via the Forgejo ``/pulls/{n}.diff`` endpoint and embeds it into the worker prompt under an ``UNTRUSTED CONTENT`` fence with explicit BEGIN_PR_DIFF / END_PR_DIFF markers, head_sha pinning, character-count metadata, and END marker redaction to defeat patch-text injection. The worker is instructed to use the embedded diff and skip the isolator subagent entirely when it is present. Falls back to the old path on fetch failure or via the ``REVIEW_DISPATCHER_EMBED_DIFF=0`` env switch. # Cross-cutting bash rules The ``pr-review-worker``'s shell tool calls kept hitting ``permission denied`` because OpenCode's permission engine matches the *raw, unexpanded* command string against allow-globs. Chained commands (``&&``, ``||``, ``;``, ``|``), command substitution (``$(...)``), bare variable assignments, multi-line continuations (``\\\n``), heredocs, and inline ``python3 -c "..."`` strings all contain characters the permission glob cannot span, and were silently denied. Added ``.opencode/instructions/bash-commands.md`` (wired into ``opencode.json`` via the ``instructions`` array so it appends to EVERY agent's system prompt globally), with hard rules + recovery recipes (``printf > /tmp/file`` instead of heredocs; ``printf > /tmp/script.py`` + ``python3 /tmp/script.py`` instead of ``python3 -c``; ``curl -d @/tmp/body.json`` instead of multi-line ``-d '{...}'``). # Pre-commit polish (architect/dev/test review) Surfaced during a chief-architect / principal-developer / senior-test-engineer code review of the uncommitted change: - Schema DDL deduplication (described above under F2). - ``finish_cycle`` INSERT-fallback now preserves ``started_at`` / ``driver_name`` when caller provides them; otherwise stamps a ``synthetic_started_at: true`` flag in the raw blob so cycle-time analytics can exclude rows whose duration was synthesised. - ``bytes=`` → ``chars=`` in the embedded-diff header. The value is ``len(diff_text)`` after ``decode("utf-8")`` — a UTF-8 character count, not a byte count. Off-by-multibyte for non-ASCII patches. - ``scripts/opencode-builder.sh`` mode 644 → 755. - ``.gitignore`` entries for ``.parked-prs.json`` (runtime state for ``tools/park_other_prs.py --restore``) and ``.dispatcher-logs/`` (append-only local pipeline log directory). # Tests (381 passed, 1 skipped) - ``test_opencode_worker.py``: 3 new tests for ``on_poll`` cadence, error swallowing, and backwards-compatible default. - ``test_dispatch_runtime.py``: 7 new tests for ``begin_cycle`` / ``finish_cycle`` semantics, the v4→v5 migration with row preservation, the crash-safe try/finally path, the ``dispatch_one`` → ``run_session_blocking`` ``on_poll`` wiring, and the new ``started_at`` / ``driver_name`` plumbing through the INSERT-fallback branch. - ``test_telemetry_server.py``: 4 new tests for ``in_flight_cycle`` in ``/api/health``, the elapsed-seconds computation, and the ``running_long_worker`` flag. - ``test_telemetry_schema.py``: assertion bumped from v4 → v5 and a new test confirming the cycle tables now allow ``ended_at IS NULL``. - ``test_dispatch_review.py`` (new file): 14 tests for diff fetch (happy path, truncation, HTTP/URL errors, END_PR_DIFF redaction, Forgejo auth scheme), ``_build_diff_section`` (dry-run, env toggle, embedding, fallback), and end-to-end prompt embedding. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
drew
|
f89d275650 |
chore(dispatchers): apply pre-push critique fixes + ship production launcher
Layered polish on top of
|
||
|
drew
|
593d142f6f |
feat(auto-agents): Tier 2 deterministic review/implementer dispatchers
Replaces the long-running pr-review-supervisor / implementation-
supervisor LLM polling loops with host-level Python dispatchers that
own queueing, claim ownership, watchdogs, and SQLite telemetry. The
LLM workers retain sole responsibility for review judgment and code
generation; Python owns only orchestration. The hard merge invariant
is unaffected — these dispatchers do not touch master.
Driver surface
- _opencode_worker.run_session_blocking — outcome-agnostic OpenCode
session lifecycle (completed / timeout / transport-error). Used
directly by reviewer / implementer dispatchers whose workers do
not emit the conflict-driver JSON exit schema.
run_worker_blocking is now a thin wrapper that adds the
conflict-specific JSON-outcome classification.
- _dispatch_runtime — shared Python runtime (work-group polling,
claim helpers, dispatch loop, telemetry). Pre-checks issue labels
before claiming and refuses when any auto/claimed-* is already
present, distinguishing already-claimed vs labels-fetch-failed
vs claim-failed terminal states. Frozen DispatchConfig.
- dispatch_review.py / dispatch_implementer.py — per-pipeline work
groups, prompts, and CLIs. Each refuses startup when a competing
AUTO-REV-SUP / AUTO-IMP-SUP legacy supervisor is live on the
same OpenCode server (override:
{REVIEW,IMPLEMENTER}_DISPATCHER_ALLOW_SUPERVISOR_COEXIST=1).
- _loader.py — shared sibling-module loader; replaces the three
duplicated copies in the dispatcher entry points.
Operational hardening
- run_outer_loop tracks consecutive cycle exceptions against
cycle_failure_budget (default 5, env-tunable per driver) and
exits 2 on exhaustion for supervisor-driven restart.
- _sanitize_release_detail strips control bytes and neutralises
triple-backtick fences before quoting worker raw_response in
Forgejo claim-release comments.
- scripts/opencode-builder.sh: OPENCODE_BUILDER_SERVER_ONLY=1 keeps
only the OpenCode HTTP API up so the Python dispatchers own
queue orchestration without auto-agents running concurrently.
Telemetry
- _forgejo_cache.py schema v4: dispatch_review_cycles,
dispatch_implementer_cycles. One row per cycle with cycle_id,
driver, candidates_count, claims_acquired, swept_count,
processed_count, terminal_state, worker_outcome, session_id,
worker_wallclock_seconds, raw.
- .opencode/telemetry/server.py wires the new tables into
/api/cycles?driver=dispatch_review|dispatch_implementer and
surfaces a composite terminal_state/worker_outcome 24h breakdown
so dashboards can distinguish session-level vs work-level
outcomes.
Tests
- 31 new tests in tests/auto_agents/test_dispatch_runtime.py
covering: candidate priority/dedup, claim/release labels,
foreign-claim refusal, same-kind-claim refusal,
labels-fetch-failed terminal state, sanitization, supervisor
coexistence guard (pass/refuse/override/unreachable-server),
session timeout / transport-error propagation,
JSON-vs-no-JSON worker exits, cycle failure budget exit and
reset, heartbeat cadence, end-to-end --once --dry-run /
--status CLI smoke, and full prompt-snapshot tests for
_review_prompt and _implementation_prompt (PR-fix + issue-impl).
- test_telemetry_schema.py asserts schema v4 and the presence of
the two new dispatcher cycle tables.
338 auto_agents tests pass (was 322 before Tier 2). Conflict driver
regression suite unchanged. Dispatchers run cleanly under
--status / --once --dry-run with no Forgejo or OpenCode HTTP traffic.
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
drew
|
703a7c9090 |
feat(auto-agents): Phase A — conflict_drive.py deterministic conflict-resolution driver
Implement Tier 1.5 conflict resolution as a peer driver to merge_drive.py that honours the same hard invariant: every commit on master came from a SHA whose CI passed against the exact current master. New components - tools/conflict_drive.py: deterministic driver that picks PRs labelled auto/needs-conflict-resolution, claims them via the shared auto/claimed-merge label, attempts a deterministic rebase with deepen-on-demand fallback, dispatches conflict-resolver-worker on conflict, force-pushes with --force-with-lease, and clears the label for merge_drive to re-pick. Includes 24h retry budget, escalation to auto/needs-implementer, single-instance lock + heartbeat, opt-in TOCTOU mitigations (CONFLICT_DRIVER_CYCLE_JITTER_SECONDS, CONFLICT_DRIVER_VERIFY_CLAIM), startup TTL constraint and required- labels assertions, and full SQLite telemetry. - tools/_claim_runtime.py: shared HTTP/lock/heartbeat/claim primitives extracted from merge_drive.py (driver-aware claim/release markers, injectable op_label_map). merge_drive re-exports for back-compat. - tools/_opencode_worker.py: blocking Python client for the OpenCode HTTP API with O(N) string-aware bracket matcher for tolerant JSON extraction from worker output. - .opencode/agents/conflict-resolver-worker.md: subagent definition with tight permissions and "always finish, never abort, never --skip" doctrine. - tools/inject_synthetic_conflict.py: CLI that creates PRs on a test fork with guaranteed conflicts (trivial / multi-commit / unresolvable) for end-to-end testing. Telemetry & dashboard - tools/_forgejo_cache.py: new conflict_drive_cycles table, cycle-level escalated_count, indexed retry-budget query, mark_*_escalated helper. - tools/render-pr-velocity.py + pr-velocity.canvas.template.tsx: conflict-resolution activity section showing 7-day cycle counts, resolved/escalated/timeout/push-rejected breakdowns. Open-issue dependency check - tools/merge_drive.py: pr_is_eligible now consults Forgejo's blocks endpoint and applies auto/blocked-by-deps when any open dependency exists. Read-only predicate _pr_has_open_dependencies; label mutations live with the eligibility caller. - tools/setup_auto_labels.py: provisions auto/blocked-by-deps. Documentation - docs/development/conflict-drive-plan.md: full plan including TOCTOU race documentation (§3.3.1) with implementation/test pointers. - AGENTS.md: operator-facing section on conflict_drive.py and the expanded label registry. Tests - 300 unit tests pass / 1 skipped (opt-in fork integration test). - Coverage includes JSON extractor fuzz, push-stderr classification, PAT scrub, deepen-on-demand fallback, retry budget escalation, cycle-level escalated_count stamping, claim collision detection (latest-claim-only with marker-primary identity), jitter wiring, and verify_claim_after_apply plumbing. Quality invariant unchanged: conflict_drive.py only operates on PR head branches, never on master. CI gating on the train-merge SHA continues to enforce the exact-current-master rule for everything that lands. Co-authored-by: Cursor <cursoragent@cursor.com> |