3ca794be7589eb00bb2dd1a3b3a558cb402af45d
14 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 drew
|
e9f9573c63 |
feat(auto-agents): mid-flight worker doom-loop probe (G4, default-OFF)
Adds _session_pattern_checks.py — pure, network-free detectors over the OpenCode session message stream — and wires it into _opencode_worker.run_session_blocking as a soft-threshold probe past ~30% of the wallclock budget. A detector hit short-circuits the poll loop with a named error_kind (doom-loop, retry-cascade, permission-deadlock, empty-reasoning-loop) instead of the opaque watchdog-timeout that the hard timeout would otherwise produce — turning "the worker burnt 15 minutes for unclear reasons" into a specific diagnosis suitable for telemetry. Flag-gated via WORKER_DOOM_LOOP_ABORT_ENABLED; default OFF preserves the pre-G4 behaviour byte-for-byte. When OFF the probe block is a single boolean short-circuit. When ON, the probe is throttled by both a soft elapsed-time threshold and a 60s inter-probe interval, keeping per-cycle network cost bounded. A3 design constraints (carried from final-working's session-health-full-util.md) are pinned by tests: bash sleep invocations don't count as activity; turns whose only output is a tool call are activity (waiting, not stuck); healthy is the default when signals are mixed. 20 unit tests for the detectors + 3 integration tests for the worker wiring (flag-off must not /message-fetch; flag-on + doom pattern must abort with named error_kind; flag-on + healthy must complete normally). Refs: docs/development/final-working-harvest-plan.md (G4). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
1eac4ea233 |
refactor(auto-agents): manifest-driven tier-model registry, slot-based naming
Replace the scattered per-agent .txt-file mapping (whose names embedded model-family identity like tier-qwen-med and tier-kimi and went stale the moment a model was swapped) with a single source-of-truth manifest at .opencode/models/tiers.yaml. The four tier slots get model-agnostic slot-based names (tier-min, tier-0, tier-1, tier-2); the model in each slot is configured ONLY in the manifest. Derived artifacts (per-agent .txt files and the mapping table block in tier-dispatcher.md) are now generated by tools/sync_tier_models.py. A drift-detection test in tests/auto_agents/test_tier_model_registry.py fails CI if any derived file diverges from the manifest, if an agent referenced by the manifest lacks an agent file, if the manifest cites a provider not declared in opencode.json, or if opencode.json carries a stale tier-* entry. To swap a model in a slot: edit tiers.yaml -> run python3 tools/sync_tier_models.py -> commit. The runtime dispatcher re-reads the .txt files per cycle (no restart); the static OpenCode config path needs a server restart. Tier rename mapping (escalation_tier integers UNCHANGED): tier-qwen-small -> tier-min (slot -1) tier-qwen-med -> tier-0 (slot 0, default first attempt) tier-qwen-large -> tier-1 (slot 1) tier-kimi -> tier-2 (slot 2) Vestigial tier-* agents removed (declared but never in the active mapping): tier-haiku, tier-sonnet, tier-opus, tier-codex, tier-gpt5-mini, tier-gpt5-nano, tier-o4-mini. estimator-implementation.md now reasons in capability descriptors (cheapest / default / advanced / complex) instead of model-family labels (qwen-small / qwen-med / qwen-large / kimi), so the estimator stays correct across model swaps. The stale "default tier = gpt-5-mini" docstring claim (already drifted to claude-haiku-4-5) is removed. Companion prose updates across every consumer of tier names: - Agent prompts: tier-dispatcher.md, implementation-worker.md, estimator-implementation.md - Skills: implementer-pr-context, implementer-workspace - Python: dispatch_implementer.py, _opencode_worker.py, _pr_context_sentinel.py, implementer_workspace.py, setup_auto_labels.py, _attempt_history.py - Tests: test_worker_permissions.py (parametrize list + byte-identity test now covers 4 slot files instead of 3 family-named files), test_opencode_worker_models.py (synthetic-fixture names updated) - Docs: .opencode/models/README.md, docs/development/models.md, docs/development/agent-system-specification.md, docs/development/auto-agents-tier-2-3-plan.md, docs/development/implementer-in-cycle-escalation-plan.md, docs/development/final-working-harvest-plan.md Validation: 1625 tests pass (+6 net new from the tier-registry test file), 3 skipped. python3 tools/sync_tier_models.py --check exits 0. local_ci_gate.sh --gate lint PASS. Files: 9 added, 22 deleted, 18 modified. The drift-detection test ran green on every step of the refactor, catching one out-of-sync .txt file before commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
8ed4b96b1a |
feat(auto-agents): perf + observability + persistent comment cache
Folds B1-B4 + C2-C5 from the post-live-test plan into one commit:
B1 — npx tsx pre-warm in dispatchers-launcher.sh closes the cold-cache
30s AbortSignal timeout that killed both dispatchers' first cycle.
B2 — per-tier worker timeout
(IMPLEMENTER_DISPATCHER_WORKER_TIMEOUT_TIER_{N}_SECONDS) lets Tier 1
(qwen-large) and Tier 2 (kimi) get more wallclock than gpt-5-mini;
floor 60s.
B3 — _rebuild_prompt_from_cached_result skips the full prefetch on
tier transitions (worktree-reset puts everything back at the
prefetched head_sha, so the prefetch result + det_sections don't
change). Saves ~7 min per tier transition on comment-heavy PRs.
B4 — git-commit-util.md documents the FORBIDDEN naive recovery
pattern (git fetch && git reset --hard) that lost PR #30 attempt
3's real fix in the live test. Two correct paths now spelled out:
--force-with-lease=<branch>:<old-remote-sha> or stash+rebase+pop.
C2 — _pr_clone._refresh_mirror_with_retry adds one retry on git
fetch failure and force-reclones the bare mirror if both attempts
fail. Previously a single exit 128 logged WARN and continued with
stale data forever.
C3 — in-flight turn markers (asterisk suffix on input/output token
counts) in the per-turn log when completed=False. The archived
turn dict's completed field was already there; the log now surfaces
it. Sub-agent timeout archiving was already correct via
_archive_subagent_tree.
C4 — new module _recent_push_cache.py records per-PR push events
(head_sha + timestamp + cycle metadata). Prefetch surfaces in the
sentinel under recent_implementer_push (with --field accessor)
when the cached push matches the PR's current head_sha within
1h. Prevents the "dispatcher re-cycles right after pushing,
worker re-does the same compliance work" failure mode from
PR #28 cycle 2 in the live test.
C5 (replaces C1) — new module _pr_comments_cache.py wraps
_review_fetch.fetch_pr_comments with disk-backed delta-fetch
semantics. PR #30's 1340+ comment fetch (which previously took
~30s and hit the 20-page pagination cap) now becomes a 5-10 item
delta. Cache is per-PR, shared between reviewer + implementer
dispatchers, has 24h staleness bound, kill-switch via
IMPLEMENTER_DISPATCHER_COMMENT_CACHE_DISABLE=1.
Tests: 1484 passed, 3 skipped (+20 from
|
||
|
drew
|
0150c4fbc0 |
feat(auto-agents): Tier 1+2 follow-ups from 2026-05-11 post-mortem
R1: bake 20-min bash timeout into quality-gates skill recipes so the first cold-cache --fast call no longer trips OpenCode's 120 s default (recovered ~2 min that was lost on the 2026-05-11 PR #30 cycle to timeout-and-retry). R2: plumb subagent_max_depth from _archive_subagent_tree's BFS walk through SessionResult -> SessionContext -> extract_phase4_ telemetry so the field stops landing as null on real multi-tier cycles. Distinguishes None (walk failed / unknown) from 0 (measured-flat). +7 behavioural tests, 4 existing tests updated to consume the new (paths, max_depth) tuple. R3: throttled operator-visible "worker still in-flight" log line every 120 s (configurable via DISPATCHER_HEARTBEAT_LOG_INTERVAL_SECONDS) so a 20-min worker turn emits ~10 status lines instead of going completely silent between session-start and session-end. +5 tests covering env-var override, defaults, garbage-rejection, end-to-end emission, and short-cycle suppression. R4: triaged 25 errored steps in fork-local unit_tests -- conclusive finding that they're caused by Rich Console defaulting to 80-col width in non-TTY CliRunner mode, truncating asserted column headers. Pre-existing on every branch, unrelated to auto-agents. Documented in the Tier 2/3 plan so future operators don't re-spend the diagnostic time. R5: short-circuit tier-dispatcher's estimator call on first attempts via new optional escalation_tier_hint parameter; implementation-worker now hard-codes hint=0. Saves ~30-60 s wall-clock per cycle on the common case (estimator's recommendation converged on Tier 0 in every observed cycle to date; sample too small for a confidence interval). Future-proof: the hint becomes dynamic when the auto/last-attempt-tier-N label scheme lands. +2 static lint tests pin the contract. Pre-commit polish (P0/P1/P2 from the consolidated critique): - Renumber tier-dispatcher CRITICAL rules 6,9,7,8 -> 6,7,8,9 - End-to-end pin tests for the R2 closure-mutation chain (subagent_max_depth=1 and =None paths) - Deterministic time.monotonic mock in heartbeat tests so they no longer depend on real-clock timing - Reject bool from extract_phase4_telemetry's int check (bool is subclass of int in Python -- would slip True/False through as 1/0) - Replace d.get("_subagent_depth") or 0 footgun with int(d.get(..., 0)) - Document DISPATCHER_HEARTBEAT_LOG_INTERVAL_SECONDS as a startup-only knob - Calibrate the "~95% Tier 0" claim to "Tier 0 in every observed cycle, n=1" - Document the reviewer-side R2 asymmetry (SessionContext carries the field; reviewer telemetry sink doesn't emit yet) - Add R5 rollback procedure - Trim duplicated R5 rationale prose in implementation-worker.md - Add reviewer back-compat test for SessionContext with the new subagent_max_depth field Tests: 1238 passed (+3 net-new), 3 skipped. Lint + typecheck clean. Validated via the local_ci_gate.sh wrapper. |
||
|
drew
|
dc96848174 |
feat(auto-agents): archive entire task-tool subagent tree before DELETE root
Before this change the dispatcher archived only the top-level
wrapper session. The entire ``task``-tool subagent chain
(tier-dispatcher → estimator-implementation / tier-qwen-med →
task-implementor → git-isolator-util) was opaque the moment
the dispatcher's DELETE /session/{id} fired, so post-mortem
analysis of an implementer run was limited to whatever
live-API polling we'd done DURING the run. That's how the
recent optimization round had to work from two cherry-picked
live snapshots of task-implementor and git-isolator-util —
unreliable, only what happened to be active when polled.
Three changes:
1. tools/_opencode_worker.py
- New helpers: _walk_subagent_descendants (BFS over
GET /session keyed on parentID), _extract_subagent_agent_name
(parses OpenCode's "(@<agent> subagent)" title convention),
_ms_to_iso (epoch-ms to ISO-8601), and _archive_subagent_tree
(best-effort walk + fetch + write driver; never raises).
- _archive_session / _build_archive_payload gain optional
parent_session_id / subagent_title / subagent_depth kwargs.
When set, the filename includes a ``sub<depth>`` infix
(e.g. 2026-...__sub01__AUTO-IMP-PR-30__tier-dispatcher__ses_*.json)
so a directory listing groups every session from one
dispatcher cycle and reads top-down in BFS order.
- Archive schema bumped from v1 → v2. New fields are nullable;
v1 readers (the existing telemetry-console endpoints) treat
them as missing and remain forward-compatible.
- run_session_blocking's finally block calls
_archive_subagent_tree after the root archive write and
before the root DELETE. Both calls are wrapped in
try/except so a subagent-walk failure can never mask the
worker outcome or stop the dispatcher from cleaning up.
- The dispatcher's existing redact_values list (the Forgejo
PAT) propagates into every subagent archive too, so a
``git clone https://${PAT}@...`` in git-isolator-util's
bash history is masked the same way the wrapper's prompt is.
2. .opencode/telemetry/server.py
- _api_archived_sessions listing endpoint now surfaces the
three v2 fields (schema_version, parent_session_id,
subagent_title, subagent_depth) in each row payload so a
future UI render can nest subagents under their wrapper.
Additive — existing row keys are preserved.
3. tests/auto_agents/test_opencode_worker_observability.py
- 17 new tests across four classes:
- TestSubagentTitleExtraction (5): title parser edge cases
- TestWalkSubagentDescendants (6): BFS order, depth
annotation, transport-error / malformed-payload paths,
cycle safety
- TestArchiveSubagentTree (5): end-to-end orchestration
including a redaction-propagation test that asserts a
PAT inside a subagent's bash tool input is replaced
with <REDACTED>
- TestEndToEndSubagentArchive (1): drives the full
run_session_blocking lifecycle with a wired subagent
descendant and asserts BOTH archives land on disk
- Existing schema-version assertion updated to v2 + three
new ``None``-on-top-level field assertions.
- Two manually-wired archive tests (transport-error,
timeout) now wire GET /session so the walker doesn't emit
a spurious warning.
Total auto_agents suite: 1061 passed, 3 skipped (up from 1044).
This is the prerequisite for trustworthy quantification of the
upcoming default-flip of IMPLEMENTER_DISPATCHER_PREFETCH=1 and
IMPLEMENTER_DISPATCHER_PRECLONE=1. With the walker in place,
every cycle now leaves a complete trace on disk that a human
can read bottom-up months later.
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
drew
|
f27cf1e017 |
feat(auto-agents): prefetch worker credentials in prompt + redact PAT in archives + implementer-cycle skill (P0-1, P0-3)
P0-1: the implementer dispatcher now embeds forgejo_pat / git_user_name / git_user_email directly into the worker prompt under a new "## Worker credentials (use these instead of env vars)" block. Live-test post-mortem of the 2026-05-10 implementer run showed the worker burning 245 s across 3 turns probing for env vars (printf denied -> printenv ... || true denied -> printenv ... succeeded); with values inline those turns disappear entirely. The dispatcher additionally passes redact_values=[cfg.token] to run_session_blocking so every occurrence of the PAT is replaced with <REDACTED> in the on-disk session archive (prompt body, tool input.command, any nested error string). Minimum redact-length floor of 12 chars prevents accidental archive mangling when a caller passes too-short credentials. Both reviewer and implementer pipelines benefit. P0-3: new .opencode/skills/implementer-cycle/SKILL.md is a 130-line cheat sheet that replaces the heavier auto-agents-system skill load on the implementation-worker's claim/dispatch/release path. The worker .md inlines the full `npx --yes tsx ... claim_pr.ts ...` one-liners so the skill load is informational, not load-bearing. 15 new tests: 10 cover redaction unit/integration paths (multi-occurrence, multi-secret, short-value warning, negative control, prompt+tool-input end-to-end); 5 cover the credentials section (presence/absence/partial/empty/canonical-order). 1043 auto_agents tests pass / 3 skipped. Forward-looking expectation: ~245 s/4 min saved per implementer cycle + measurable input-token reduction. Will be re-measured against the next live dispatch_implementer --once run. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
drew
|
132a5a2269 |
feat(auto-agents): centralise model registry under .opencode/models with session-create model stamp
Every agent's model assignment now lives in a single-line text file at
.opencode/models/<name>.txt; default.txt is the 27-agent catch-all.
opencode.json's agent.<name>.model uses
{file:./.opencode/models/<name>.txt} interpolation, and
tools/_opencode_worker.py reads the same files at session-create to
stamp the resolved model on the session record (observability + drift
sentinel; OpenCode does NOT propagate session-level model to
prompt_async — schema for that is undocumented and deferred to Stage
2). 39 .md frontmatter `model:` lines stripped; the two intentional
inheritors (task-implementor, agent-evolution-pool-supervisor) keep
their model-less frontmatter.
Operator workflow for swapping a model is now: edit
.opencode/models/<role>.txt, restart OpenCode so opencode.json's
{file:...} re-resolves, run the dispatcher. Live-swap without restart
was attempted (override on prompt_async); OpenCode 0.x silently
dropped those requests (200 OK, no assistant message) and the
prompt_async override was reverted. The session-create override
remains for observability + drift detection.
End-to-end validation (2026-05-10): dispatch_review.py on PR #25 with
default.txt=openai/gpt-5-mini produced a clean REQUEST_CHANGES review
in 26 s for ~$0.016; dispatch_implementer.py on PR #30 with
tier-qwen-* files remapped to openai/{gpt-5-nano, gpt-5-mini,
gpt-5-codex} ran the full implementation-worker → tier-dispatcher →
estimator-implementation → tier-qwen-med → task-implementor →
git-isolator-util chain in 16 min with model=gpt-5-mini end-to-end.
Also documents the printenv VAR form as the only allowed env-read in
implementation-worker.md and task-implementor.md (live testing
showed the worker burning 2–4 turns on permission-denied
trial-and-error trying printf and echo variants).
Tests: 21 in tests/auto_agents/test_opencode_worker_models.py
(resolver semantics with caplog assertions on every malformed-input
path; session-create body shape; prompt_async body never carries
model; three repo-level invariants — every {file:...} reference
resolves, every .md is wired or in the inheritor allowlist, no .md
has a model: frontmatter). 1027 auto_agents tests pass / 3 skipped.
Note: .opencode/models/default.txt and the three tier-qwen-*.txt
files are committed with their OpenAI swaps in place (gpt-5-mini,
gpt-5-nano, gpt-5-mini, gpt-5-codex respectively) because the
CleverThis HuggingFace endpoints are paused. Revert with `git diff
HEAD~1 -- .opencode/models/*.txt | git apply -R` if/when they come
back.
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
drew
|
2f1be34d12 |
feat(auto-agents): implementer parity — verify-invariant verifiers, implementer-helpers skill, watchdog gate, _opencode_worker audit
Closes the four open items in `docs/development/auto-agents-tier-2-3-plan.md` § "Revised remaining scope (2026-05-08)" plus three rounds of fresh-eyes critique fold-in (rounds 3, 5, and post-round-5 polish). Highlights: - New continuous invariant verifiers on a shared `_verify_common.py` substrate: `verify_review_invariant.py` (R1: approval-without-CI) and `verify_implementer_invariant.py` (I1: head-commit fails commit-lint, I2: PR description missing Epic reference). Strictly additive cron-job- shaped scripts that open idempotent `auto/invariant-violation` issues; safe to run every 15 minutes in production. - New `implementer-helpers` skill at `.opencode/skills/implementer-helpers/SKILL.md` + CLI at `tools/implementer_validate.py` (4 subcommands: validate-commit-message, validate-pr-compliance, validate-file-budget, validate-changelog). Mirrors the reviewer side; `tools/_commit_lint.py` is shared so a future change to commit policy updates one place. - `auto-agents.md` watchdog gate: `DISPATCHERS_RUNNING=1` puts the primary orchestrator into watchdog-only mode. Heartbeat resolution + age computation factored into `tools/_watchdog_helpers.py` + the CLI `tools/watchdog_check.py` so the agent only needs `python3 tools/watchdog_check.py *` and `sleep *` bash permissions. The reader honours the env-var override first, then falls back to a freshest-mtime scan across `/var/run` / `$XDG_RUNTIME_DIR` / `/tmp` (deliberately diverging from the dispatcher's first-existing fallback to guard against stale heartbeats from previous root-owned sessions masking healthy user-mode heartbeats). - `_opencode_worker.py` audit: structured `error_kind` classification at every transport-error / timeout return site, plumbed through `_dispatch_runtime.py` into the cycle-log; new `_request_read` retry helper (3 × 0.5s linear backoff, transport-only) wrapping every idempotent read in a worker session so a single transient flap on a polling GET cannot trash a 10-minute worker session. - Static heredoc lint at `tests/auto_agents/test_prompt_heredoc_lint.py` glob-walks every agent prompt and skill recipe markdown, rejecting any heredoc bash recipe in a fenced code block (per `bash-commands.md` rule 2 — heredocs fail at OpenCode's permission-engine parse time). - `bash-commands.md` rule 2 + its fix-it advice both lead with apostrophe-safe `printf "%s" "<body>"` (double-quoted) form; single-quoted form documented as the fragile JSON-only fallback. - `CHANGELOG.md` carries the full multi-round narrative (round 3 CRITICAL/HIGH/MEDIUM/LOW fold-in, round 5 docstring drift + telemetry refactor + broader heredoc lint scope, post-round-5 doc-drift polish). Net delta: +911 passing tests / 3 skipped (was 825 / 3); ruff clean on every new file; pre-existing lint debt in `_dispatch_runtime.py`, `_opencode_worker.py`, `conftest.py`, `_commit_lint.py` unchanged and out of scope for this commit. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
drew
|
05d069cc3b |
fix(auto-agents): disambiguate transient vs terminal unknown after seen_busy
The previous polling-race fix made unknown && seen_busy loop forever, which hung every clean dispatcher cycle on the watchdog timeout because OpenCode permanently removes finished sessions from /session/status (status=unknown is the terminal signal there). Add _session_assistant_in_flight() that checks the latest assistant message's time.completed field on the unknown path: - in-flight => transient blip between turns, keep polling - complete => terminal, break out - transport error => conservative, keep polling The helper is off the happy path (busy sessions never trigger the extra HTTP call). Four new tests in test_opencode_worker_observability covering all three branches plus a direct unit test of the helper. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
drew
|
9bf44e174d |
fix(auto-agents): worker polling-race + session observability bundle
- Fix _opencode_worker.run_session_blocking treating ``unknown && seen_busy`` as transport-error after the warmup deadline. The session-status map briefly drops the entry between assistant turns; the dispatcher now keeps polling instead of killing a worker that is actively producing tokens. Regression that took out PR #30's reviewer cycle today. - Add per-poll state-transition INFO logging (init->busy, busy->unknown->busy, busy->idle), bounded per-poll history, and a per-assistant-turn metrics summary log so a 6-minute reviewer cycle is reconstructable from the dispatcher log. - Add session-message archival: every terminal return snapshots the full /session/{id}/message payload (plus per_turn metrics and state_history) to .dispatcher-logs/sessions/<file>.json before DELETE /session. Survives session deletion, so failed runs can be post-mortemed long after the cycle ends. Disable via OPENCODE_WORKER_ARCHIVE_DISABLED=1 (the test suite's autouse default). - Wire archives into the telemetry console: new "Sessions" tab, GET /api/sessions/archived index endpoint, GET /api/sessions/ archived/detail filename-validated detail endpoint with path-traversal guard. Operators can browse failed worker runs from the browser instead of grepping JSON files. - 24 new unit tests across two files covering the polling-race regression, archive paths on completed/timeout/transport-error, per-turn aggregation, env var resolution, write-failure swallow, archive index ordering + cap, and detail-endpoint path-traversal rejection. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
drew
|
2cbe62a70c |
fix(auto-agents): F1+F2+F3 telemetry liveness, reviewer perf (A+C), bash rules
Bundles a long-overdue set of fixes that surfaced while watching the single-PR pipeline test against PR #30 the morning of 2026-05-07. # F1 — long-worker liveness contract The dispatcher heartbeat was only refreshed *between* worker sessions. On a 30-minute review the heartbeat file went stale, and any heartbeat-watchdog (dispatchers-launcher.sh / cleveragents-dispatchers.service) would SIGTERM a perfectly-healthy worker mid-cycle, orphaning the OpenCode session and the auto/claimed-* lock. ``_opencode_worker.run_session_blocking`` now accepts an ``on_poll`` callback fired once per status-poll iteration; ``_dispatch_runtime.dispatch_one`` and ``conflict_drive.py`` wire it to ``write_heartbeat(cfg.heartbeat_path)``. Callback exceptions are logged and swallowed so a transient EROFS on the heartbeat path can never mask a successful worker completion. # F2 — in-flight cycle visibility (schema v5) The ``dispatch_*_cycles`` tables previously only recorded a row at cycle *end*. While a worker was running, the operator's only signal was the heartbeat file — and even that became stale (see F1). Schema bumped to v5: ``ended_at`` is now nullable and ``cycle_id`` carries a UNIQUE index. ``begin_cycle`` writes the in-flight row at start; ``finish_cycle`` updates it at end. ``run_one_cycle``'s try/finally guarantees ``finish_cycle`` runs even when ``collect_candidates`` / ``dispatch_one`` raises, so an orphan ``ended_at IS NULL`` can no longer be stuck forever after a crash. The v4→v5 migration is now defined in ONE place — a set of helpers in ``_forgejo_cache.py`` (``DISPATCH_CYCLE_TABLES``, ``_dispatch_cycle_create_sql``, ``_dispatch_cycle_index_sqls``, ``migrate_dispatch_cycle_table_to_v5``, ``ensure_dispatch_cycle_schema``). Both ``ForgejoCache._migrate_to_v5_in_flight_rows`` and ``_dispatch_runtime.ensure_cycle_table`` import from there, eliminating the drift risk of the previous duplicated DDL. Pre-existing rows are preserved verbatim across the migration. # F3 — telemetry surface for the new state ``/api/health`` now returns ``in_flight_cycle: {cycle_id, started_at, session_id, candidates_count, elapsed_s}`` per dispatcher daemon and a ``running_long_worker: bool`` flag (heartbeat older than 600s AND a matching pid alive — should never fire under healthy F1 operation, so when it does it points at a real bug). The Drivers and Overview tabs in ``.opencode/telemetry/{index.html,app.js,style.css}`` render in-flight rows with a tinted background + "in flight" pill, daemon tiles get a dashed border for the long-worker state, and each tile shows the running cycle's id + elapsed time inline. # Fix A — reasoningEffort high → medium for pr-review-worker On its own that change alone would not have been enough, but combined with Fix C below it dropped a representative cycle from "timed out at 30:00" to a target ~2-3min. Pure config change in ``.opencode/agents/pr-review-worker.md``; no code path touched. # Fix C — pre-fetch PR diff in dispatch_review and embed in prompt The reviewer used to spawn a ``git-isolator-util`` subagent, which shelled out to ``git clone``, ``git fetch``, and ``git diff master...HEAD``. That subagent burned 90+ seconds and several token budgets per cycle. ``dispatch_review.py`` now fetches the unified diff via the Forgejo ``/pulls/{n}.diff`` endpoint and embeds it into the worker prompt under an ``UNTRUSTED CONTENT`` fence with explicit BEGIN_PR_DIFF / END_PR_DIFF markers, head_sha pinning, character-count metadata, and END marker redaction to defeat patch-text injection. The worker is instructed to use the embedded diff and skip the isolator subagent entirely when it is present. Falls back to the old path on fetch failure or via the ``REVIEW_DISPATCHER_EMBED_DIFF=0`` env switch. # Cross-cutting bash rules The ``pr-review-worker``'s shell tool calls kept hitting ``permission denied`` because OpenCode's permission engine matches the *raw, unexpanded* command string against allow-globs. Chained commands (``&&``, ``||``, ``;``, ``|``), command substitution (``$(...)``), bare variable assignments, multi-line continuations (``\\\n``), heredocs, and inline ``python3 -c "..."`` strings all contain characters the permission glob cannot span, and were silently denied. Added ``.opencode/instructions/bash-commands.md`` (wired into ``opencode.json`` via the ``instructions`` array so it appends to EVERY agent's system prompt globally), with hard rules + recovery recipes (``printf > /tmp/file`` instead of heredocs; ``printf > /tmp/script.py`` + ``python3 /tmp/script.py`` instead of ``python3 -c``; ``curl -d @/tmp/body.json`` instead of multi-line ``-d '{...}'``). # Pre-commit polish (architect/dev/test review) Surfaced during a chief-architect / principal-developer / senior-test-engineer code review of the uncommitted change: - Schema DDL deduplication (described above under F2). - ``finish_cycle`` INSERT-fallback now preserves ``started_at`` / ``driver_name`` when caller provides them; otherwise stamps a ``synthetic_started_at: true`` flag in the raw blob so cycle-time analytics can exclude rows whose duration was synthesised. - ``bytes=`` → ``chars=`` in the embedded-diff header. The value is ``len(diff_text)`` after ``decode("utf-8")`` — a UTF-8 character count, not a byte count. Off-by-multibyte for non-ASCII patches. - ``scripts/opencode-builder.sh`` mode 644 → 755. - ``.gitignore`` entries for ``.parked-prs.json`` (runtime state for ``tools/park_other_prs.py --restore``) and ``.dispatcher-logs/`` (append-only local pipeline log directory). # Tests (381 passed, 1 skipped) - ``test_opencode_worker.py``: 3 new tests for ``on_poll`` cadence, error swallowing, and backwards-compatible default. - ``test_dispatch_runtime.py``: 7 new tests for ``begin_cycle`` / ``finish_cycle`` semantics, the v4→v5 migration with row preservation, the crash-safe try/finally path, the ``dispatch_one`` → ``run_session_blocking`` ``on_poll`` wiring, and the new ``started_at`` / ``driver_name`` plumbing through the INSERT-fallback branch. - ``test_telemetry_server.py``: 4 new tests for ``in_flight_cycle`` in ``/api/health``, the elapsed-seconds computation, and the ``running_long_worker`` flag. - ``test_telemetry_schema.py``: assertion bumped from v4 → v5 and a new test confirming the cycle tables now allow ``ended_at IS NULL``. - ``test_dispatch_review.py`` (new file): 14 tests for diff fetch (happy path, truncation, HTTP/URL errors, END_PR_DIFF redaction, Forgejo auth scheme), ``_build_diff_section`` (dry-run, env toggle, embedding, fallback), and end-to-end prompt embedding. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
drew
|
f89d275650 |
chore(dispatchers): apply pre-push critique fixes + ship production launcher
Layered polish on top of
|
||
|
drew
|
593d142f6f |
feat(auto-agents): Tier 2 deterministic review/implementer dispatchers
Replaces the long-running pr-review-supervisor / implementation-
supervisor LLM polling loops with host-level Python dispatchers that
own queueing, claim ownership, watchdogs, and SQLite telemetry. The
LLM workers retain sole responsibility for review judgment and code
generation; Python owns only orchestration. The hard merge invariant
is unaffected — these dispatchers do not touch master.
Driver surface
- _opencode_worker.run_session_blocking — outcome-agnostic OpenCode
session lifecycle (completed / timeout / transport-error). Used
directly by reviewer / implementer dispatchers whose workers do
not emit the conflict-driver JSON exit schema.
run_worker_blocking is now a thin wrapper that adds the
conflict-specific JSON-outcome classification.
- _dispatch_runtime — shared Python runtime (work-group polling,
claim helpers, dispatch loop, telemetry). Pre-checks issue labels
before claiming and refuses when any auto/claimed-* is already
present, distinguishing already-claimed vs labels-fetch-failed
vs claim-failed terminal states. Frozen DispatchConfig.
- dispatch_review.py / dispatch_implementer.py — per-pipeline work
groups, prompts, and CLIs. Each refuses startup when a competing
AUTO-REV-SUP / AUTO-IMP-SUP legacy supervisor is live on the
same OpenCode server (override:
{REVIEW,IMPLEMENTER}_DISPATCHER_ALLOW_SUPERVISOR_COEXIST=1).
- _loader.py — shared sibling-module loader; replaces the three
duplicated copies in the dispatcher entry points.
Operational hardening
- run_outer_loop tracks consecutive cycle exceptions against
cycle_failure_budget (default 5, env-tunable per driver) and
exits 2 on exhaustion for supervisor-driven restart.
- _sanitize_release_detail strips control bytes and neutralises
triple-backtick fences before quoting worker raw_response in
Forgejo claim-release comments.
- scripts/opencode-builder.sh: OPENCODE_BUILDER_SERVER_ONLY=1 keeps
only the OpenCode HTTP API up so the Python dispatchers own
queue orchestration without auto-agents running concurrently.
Telemetry
- _forgejo_cache.py schema v4: dispatch_review_cycles,
dispatch_implementer_cycles. One row per cycle with cycle_id,
driver, candidates_count, claims_acquired, swept_count,
processed_count, terminal_state, worker_outcome, session_id,
worker_wallclock_seconds, raw.
- .opencode/telemetry/server.py wires the new tables into
/api/cycles?driver=dispatch_review|dispatch_implementer and
surfaces a composite terminal_state/worker_outcome 24h breakdown
so dashboards can distinguish session-level vs work-level
outcomes.
Tests
- 31 new tests in tests/auto_agents/test_dispatch_runtime.py
covering: candidate priority/dedup, claim/release labels,
foreign-claim refusal, same-kind-claim refusal,
labels-fetch-failed terminal state, sanitization, supervisor
coexistence guard (pass/refuse/override/unreachable-server),
session timeout / transport-error propagation,
JSON-vs-no-JSON worker exits, cycle failure budget exit and
reset, heartbeat cadence, end-to-end --once --dry-run /
--status CLI smoke, and full prompt-snapshot tests for
_review_prompt and _implementation_prompt (PR-fix + issue-impl).
- test_telemetry_schema.py asserts schema v4 and the presence of
the two new dispatcher cycle tables.
338 auto_agents tests pass (was 322 before Tier 2). Conflict driver
regression suite unchanged. Dispatchers run cleanly under
--status / --once --dry-run with no Forgejo or OpenCode HTTP traffic.
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
drew
|
703a7c9090 |
feat(auto-agents): Phase A — conflict_drive.py deterministic conflict-resolution driver
Implement Tier 1.5 conflict resolution as a peer driver to merge_drive.py that honours the same hard invariant: every commit on master came from a SHA whose CI passed against the exact current master. New components - tools/conflict_drive.py: deterministic driver that picks PRs labelled auto/needs-conflict-resolution, claims them via the shared auto/claimed-merge label, attempts a deterministic rebase with deepen-on-demand fallback, dispatches conflict-resolver-worker on conflict, force-pushes with --force-with-lease, and clears the label for merge_drive to re-pick. Includes 24h retry budget, escalation to auto/needs-implementer, single-instance lock + heartbeat, opt-in TOCTOU mitigations (CONFLICT_DRIVER_CYCLE_JITTER_SECONDS, CONFLICT_DRIVER_VERIFY_CLAIM), startup TTL constraint and required- labels assertions, and full SQLite telemetry. - tools/_claim_runtime.py: shared HTTP/lock/heartbeat/claim primitives extracted from merge_drive.py (driver-aware claim/release markers, injectable op_label_map). merge_drive re-exports for back-compat. - tools/_opencode_worker.py: blocking Python client for the OpenCode HTTP API with O(N) string-aware bracket matcher for tolerant JSON extraction from worker output. - .opencode/agents/conflict-resolver-worker.md: subagent definition with tight permissions and "always finish, never abort, never --skip" doctrine. - tools/inject_synthetic_conflict.py: CLI that creates PRs on a test fork with guaranteed conflicts (trivial / multi-commit / unresolvable) for end-to-end testing. Telemetry & dashboard - tools/_forgejo_cache.py: new conflict_drive_cycles table, cycle-level escalated_count, indexed retry-budget query, mark_*_escalated helper. - tools/render-pr-velocity.py + pr-velocity.canvas.template.tsx: conflict-resolution activity section showing 7-day cycle counts, resolved/escalated/timeout/push-rejected breakdowns. Open-issue dependency check - tools/merge_drive.py: pr_is_eligible now consults Forgejo's blocks endpoint and applies auto/blocked-by-deps when any open dependency exists. Read-only predicate _pr_has_open_dependencies; label mutations live with the eligibility caller. - tools/setup_auto_labels.py: provisions auto/blocked-by-deps. Documentation - docs/development/conflict-drive-plan.md: full plan including TOCTOU race documentation (§3.3.1) with implementation/test pointers. - AGENTS.md: operator-facing section on conflict_drive.py and the expanded label registry. Tests - 300 unit tests pass / 1 skipped (opt-in fork integration test). - Coverage includes JSON extractor fuzz, push-stderr classification, PAT scrub, deepen-on-demand fallback, retry budget escalation, cycle-level escalated_count stamping, claim collision detection (latest-claim-only with marker-primary identity), jitter wiring, and verify_claim_after_apply plumbing. Quality invariant unchanged: conflict_drive.py only operates on PR head branches, never on master. CI gating on the train-merge SHA continues to enforce the exact-current-master rule for everything that lands. Co-authored-by: Cursor <cursoragent@cursor.com> |