Files
cleveragents-core/scripts/opencode-builder.sh
T
drew 2cbe62a70c fix(auto-agents): F1+F2+F3 telemetry liveness, reviewer perf (A+C), bash rules
Bundles a long-overdue set of fixes that surfaced while watching the
single-PR pipeline test against PR #30 the morning of 2026-05-07.

# F1 — long-worker liveness contract
The dispatcher heartbeat was only refreshed *between* worker sessions.
On a 30-minute review the heartbeat file went stale, and any
heartbeat-watchdog (dispatchers-launcher.sh /
cleveragents-dispatchers.service) would SIGTERM a perfectly-healthy
worker mid-cycle, orphaning the OpenCode session and the
auto/claimed-* lock. ``_opencode_worker.run_session_blocking`` now
accepts an ``on_poll`` callback fired once per status-poll iteration;
``_dispatch_runtime.dispatch_one`` and ``conflict_drive.py`` wire it to
``write_heartbeat(cfg.heartbeat_path)``. Callback exceptions are
logged and swallowed so a transient EROFS on the heartbeat path can
never mask a successful worker completion.

# F2 — in-flight cycle visibility (schema v5)
The ``dispatch_*_cycles`` tables previously only recorded a row at
cycle *end*. While a worker was running, the operator's only signal
was the heartbeat file — and even that became stale (see F1). Schema
bumped to v5: ``ended_at`` is now nullable and ``cycle_id`` carries a
UNIQUE index. ``begin_cycle`` writes the in-flight row at start;
``finish_cycle`` updates it at end. ``run_one_cycle``'s try/finally
guarantees ``finish_cycle`` runs even when ``collect_candidates`` /
``dispatch_one`` raises, so an orphan ``ended_at IS NULL`` can no
longer be stuck forever after a crash.

The v4→v5 migration is now defined in ONE place — a set of helpers in
``_forgejo_cache.py`` (``DISPATCH_CYCLE_TABLES``,
``_dispatch_cycle_create_sql``, ``_dispatch_cycle_index_sqls``,
``migrate_dispatch_cycle_table_to_v5``,
``ensure_dispatch_cycle_schema``). Both
``ForgejoCache._migrate_to_v5_in_flight_rows`` and
``_dispatch_runtime.ensure_cycle_table`` import from there, eliminating
the drift risk of the previous duplicated DDL. Pre-existing rows are
preserved verbatim across the migration.

# F3 — telemetry surface for the new state
``/api/health`` now returns ``in_flight_cycle: {cycle_id, started_at,
session_id, candidates_count, elapsed_s}`` per dispatcher daemon and a
``running_long_worker: bool`` flag (heartbeat older than 600s AND a
matching pid alive — should never fire under healthy F1 operation, so
when it does it points at a real bug). The Drivers and Overview tabs
in ``.opencode/telemetry/{index.html,app.js,style.css}`` render
in-flight rows with a tinted background + "in flight" pill, daemon
tiles get a dashed border for the long-worker state, and each tile
shows the running cycle's id + elapsed time inline.

# Fix A — reasoningEffort high → medium for pr-review-worker
On its own that change alone would not have been enough, but combined
with Fix C below it dropped a representative cycle from "timed out at
30:00" to a target ~2-3min. Pure config change in
``.opencode/agents/pr-review-worker.md``; no code path touched.

# Fix C — pre-fetch PR diff in dispatch_review and embed in prompt
The reviewer used to spawn a ``git-isolator-util`` subagent, which
shelled out to ``git clone``, ``git fetch``, and ``git diff
master...HEAD``. That subagent burned 90+ seconds and several token
budgets per cycle. ``dispatch_review.py`` now fetches the unified diff
via the Forgejo ``/pulls/{n}.diff`` endpoint and embeds it into the
worker prompt under an ``UNTRUSTED CONTENT`` fence with explicit
BEGIN_PR_DIFF / END_PR_DIFF markers, head_sha pinning, character-count
metadata, and END marker redaction to defeat patch-text injection. The
worker is instructed to use the embedded diff and skip the isolator
subagent entirely when it is present. Falls back to the old path on
fetch failure or via the ``REVIEW_DISPATCHER_EMBED_DIFF=0`` env switch.

# Cross-cutting bash rules
The ``pr-review-worker``'s shell tool calls kept hitting
``permission denied`` because OpenCode's permission engine matches
the *raw, unexpanded* command string against allow-globs. Chained
commands (``&&``, ``||``, ``;``, ``|``), command substitution
(``$(...)``), bare variable assignments, multi-line continuations
(``\\\n``), heredocs, and inline ``python3 -c "..."`` strings all
contain characters the permission glob cannot span, and were silently
denied. Added ``.opencode/instructions/bash-commands.md`` (wired into
``opencode.json`` via the ``instructions`` array so it appends to
EVERY agent's system prompt globally), with hard rules + recovery
recipes (``printf > /tmp/file`` instead of heredocs;
``printf > /tmp/script.py`` + ``python3 /tmp/script.py`` instead of
``python3 -c``; ``curl -d @/tmp/body.json`` instead of multi-line
``-d '{...}'``).

# Pre-commit polish (architect/dev/test review)
Surfaced during a chief-architect / principal-developer /
senior-test-engineer code review of the uncommitted change:

- Schema DDL deduplication (described above under F2).
- ``finish_cycle`` INSERT-fallback now preserves ``started_at`` /
  ``driver_name`` when caller provides them; otherwise stamps a
  ``synthetic_started_at: true`` flag in the raw blob so cycle-time
  analytics can exclude rows whose duration was synthesised.
- ``bytes=`` → ``chars=`` in the embedded-diff header. The value is
  ``len(diff_text)`` after ``decode("utf-8")`` — a UTF-8 character
  count, not a byte count. Off-by-multibyte for non-ASCII patches.
- ``scripts/opencode-builder.sh`` mode 644 → 755.
- ``.gitignore`` entries for ``.parked-prs.json`` (runtime state for
  ``tools/park_other_prs.py --restore``) and ``.dispatcher-logs/``
  (append-only local pipeline log directory).

# Tests (381 passed, 1 skipped)
- ``test_opencode_worker.py``: 3 new tests for ``on_poll`` cadence,
  error swallowing, and backwards-compatible default.
- ``test_dispatch_runtime.py``: 7 new tests for ``begin_cycle`` /
  ``finish_cycle`` semantics, the v4→v5 migration with row
  preservation, the crash-safe try/finally path, the
  ``dispatch_one`` → ``run_session_blocking`` ``on_poll`` wiring, and
  the new ``started_at`` / ``driver_name`` plumbing through the
  INSERT-fallback branch.
- ``test_telemetry_server.py``: 4 new tests for ``in_flight_cycle``
  in ``/api/health``, the elapsed-seconds computation, and the
  ``running_long_worker`` flag.
- ``test_telemetry_schema.py``: assertion bumped from v4 → v5 and a
  new test confirming the cycle tables now allow ``ended_at IS NULL``.
- ``test_dispatch_review.py`` (new file): 14 tests for diff fetch
  (happy path, truncation, HTTP/URL errors, END_PR_DIFF redaction,
  Forgejo auth scheme), ``_build_diff_section`` (dry-run, env
  toggle, embedding, fallback), and end-to-end prompt embedding.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-07 07:12:45 -04:00

278 lines
10 KiB
Bash
Executable File

#!/usr/bin/env bash
#
# opencode-builder.sh
#
# Launches an OpenCode server, creates a session with the auto-agents
# agent, and continuously sends "continue" messages in an infinite loop.
#
# Signal handling:
# Ctrl+C once — finishes the in-flight request, then stops the loop
# Ctrl+C twice — (within 2 seconds) kills the request and exits immediately
#
set -uo pipefail
# ── Configuration (override via environment) ──────────────────────────────
PORT="${OPENCODE_PORT:-4096}"
HOST="${OPENCODE_HOST:-127.0.0.1}"
BASE="http://${HOST}:${PORT}"
AGENT="auto-agents"
PROMPT="Complete the current project's milestones up to and including v3.7.0 to a production ready state"
SERVER_ONLY="${OPENCODE_BUILDER_SERVER_ONLY:-0}"
HEALTH_TIMEOUT=60 # seconds to wait for the server to become healthy
MAX_IDLE_PER_MINUTE=10
# ── Internal state ────────────────────────────────────────────────────────
SERVER_PID=""
CURL_PID=""
SESSION_ID=""
STOP_LOOP=false
LAST_SIGINT=0
OWN_SERVER=false
N=0
IDLE_EVENTS_FILE=""
# ── Helpers ───────────────────────────────────────────────────────────────
die() { printf "ERROR: %s\n" "$*" >&2; exit 1; }
log() { printf "[%s] %s\n" "$(date +%H:%M:%S)" "$*"; }
check_opencode_running() {
# Check if port is open and responding as OpenCode server
if curl -sf "${BASE}/global/health" >/dev/null 2>&1; then
return 0 # OpenCode server is running
else
return 1 # No OpenCode server detected
fi
}
cleanup() {
if [[ -n "$IDLE_EVENTS_FILE" ]]; then
rm -f "$IDLE_EVENTS_FILE" 2>/dev/null || true
fi
if $OWN_SERVER && [[ -n "$SERVER_PID" ]] && kill -0 "$SERVER_PID" 2>/dev/null; then
log "Stopping opencode server (PID $SERVER_PID)..."
kill "$SERVER_PID" 2>/dev/null
wait "$SERVER_PID" 2>/dev/null || true
elif ! $OWN_SERVER; then
log "Leaving existing OpenCode server running (not managed by this script)"
fi
}
handle_sigint() {
local now
now=$(date +%s)
# Second Ctrl+C within 2 seconds → force-quit
if $STOP_LOOP && (( now - LAST_SIGINT <= 2 )); then
printf "\nForce quit!\n"
[[ -n "$CURL_PID" ]] && kill "$CURL_PID" 2>/dev/null || true
cleanup
exit 130
fi
# First Ctrl+C → graceful stop after current request
STOP_LOOP=true
LAST_SIGINT=$now
printf "\nWill stop after the current request completes. Ctrl+C again within 2s to force-quit.\n"
}
trap handle_sigint INT
trap cleanup EXIT
# ── Idle-event tracking ───────────────────────────────────────────────────
init_idle_tracking() {
IDLE_EVENTS_FILE="/tmp/opencode-builder-idle-events-${SESSION_ID}.txt"
: > "$IDLE_EVENTS_FILE" 2>/dev/null || true
}
record_idle_event() {
local now
now=$(date +%s)
echo "$now" >> "$IDLE_EVENTS_FILE"
}
count_idle_events_last_minute() {
local now cutoff
now=$(date +%s)
cutoff=$((now - 60))
if [[ -f "$IDLE_EVENTS_FILE" ]]; then
awk -v c="$cutoff" '$1 > c {count++} END {print count+0}' "$IDLE_EVENTS_FILE" 2>/dev/null
else
echo 0
fi
}
# ── Session lifecycle ─────────────────────────────────────────────────────
create_session() {
log "Creating session..."
SESSION_ID=$(
curl -sf -X POST "${BASE}/session" \
-H "Content-Type: application/json" \
-d '{"title":"auto-agents"}' \
| jq -r '.id'
) || die "Failed to create session"
[[ -n "$SESSION_ID" && "$SESSION_ID" != "null" ]] || die "Session ID is empty or null"
log "Session ID: ${SESSION_ID}"
init_idle_tracking
}
delete_session() {
local sid="$1"
if [[ -n "$sid" ]]; then
curl -sf -X DELETE "${BASE}/session/${sid}" >/dev/null 2>&1 || true
fi
}
restart_session() {
log "Session appears broken (>${MAX_IDLE_PER_MINUTE} idle events/min). Restarting..."
local old_sid="$SESSION_ID"
SESSION_ID=""
N=0
delete_session "$old_sid"
create_session
send_message "$INIT_BODY"
log "Initial prompt complete after restart."
}
# ── Question handling ─────────────────────────────────────────────────────
# Queries the OpenCode /question endpoint and rejects any pending questions
# that belong to the supplied session. Returns the number of questions that
# were successfully dismissed.
dismiss_pending_questions() {
local sid="$1"
local questions_json dismissed qid
questions_json=$(curl -sf "${BASE}/question" 2>/dev/null) || return 1
dismissed=0
for qid in $(echo "$questions_json" | jq -r --arg sid "$sid" '.[] | select(.sessionID == $sid) | .id'); do
log "Dismissing question $qid"
if curl -sf -X POST "${BASE}/question/${qid}/reject" >/dev/null 2>&1; then
dismissed=$((dismissed + 1))
fi
done
echo "$dismissed"
}
# ── Preflight checks ─────────────────────────────────────────────────────
for cmd in curl jq opencode; do
command -v "$cmd" &>/dev/null || die "'$cmd' is required but not found in PATH"
done
# ── Start the opencode server ─────────────────────────────────────────────
# Check if OpenCode server is already running
if check_opencode_running; then
echo "┌─────────────────────────────────────────────────────────────────┐"
echo "│ ⚠️ WARNING: OpenCode server already running on ${HOST}:${PORT}"
echo "│ 🔗 Connecting to existing server instead of starting new one"
echo "│ 💡 The script will NOT manage this server's lifecycle"
echo "└─────────────────────────────────────────────────────────────────┘"
echo ""
log "Using existing OpenCode server at ${HOST}:${PORT}"
OWN_SERVER=false
else
# Start our own server as before
log "Starting opencode server on ${HOST}:${PORT}..."
opencode serve --port "$PORT" --hostname "$HOST" &
SERVER_PID=$!
OWN_SERVER=true
log "Waiting for server to become healthy (up to ${HEALTH_TIMEOUT}s)..."
healthy=false
for (( i = 1; i <= HEALTH_TIMEOUT; i++ )); do
if curl -sf "${BASE}/global/health" >/dev/null 2>&1; then
healthy=true
break
fi
kill -0 "$SERVER_PID" 2>/dev/null || die "Server process exited unexpectedly"
sleep 1
done
$healthy || die "Server did not become healthy within ${HEALTH_TIMEOUT}s"
log "Server is ready."
fi
if [[ "$SERVER_ONLY" == "1" || "$SERVER_ONLY" == "true" || "$SERVER_ONLY" == "yes" ]]; then
log "Server-only mode enabled; not launching auto-agents or LLM supervisors."
log "Use tools/dispatch_review.py and tools/dispatch_implementer.py as the deterministic dispatchers."
while ! $STOP_LOOP; do
sleep 5
done
exit 0
fi
# ── Create a session ──────────────────────────────────────────────────────
create_session
# ── send_message ──────────────────────────────────────────────────────────
# Sends a JSON body to POST /session/:id/message and blocks until the LLM
# responds. curl is run in a subprocess that ignores SIGINT (SIG_IGN is
# inherited across exec), so a single Ctrl+C cannot kill the in-flight
# request. The parent re-waits whenever its own trap handler interrupts
# the `wait` builtin.
send_message() {
local body=$1
( trap '' INT
exec curl -sf -X POST "${BASE}/session/${SESSION_ID}/message" \
-H "Content-Type: application/json" \
-d "$body" \
-o /dev/null
) &
CURL_PID=$!
# Keep waiting until curl actually exits — `wait` can return early when
# our SIGINT handler fires, so we loop on kill -0 to re-wait.
while kill -0 "$CURL_PID" 2>/dev/null; do
wait "$CURL_PID" 2>/dev/null || true
done
CURL_PID=""
}
# ── Send the initial prompt ──────────────────────────────────────────────
INIT_BODY=$(jq -nc --arg a "$AGENT" --arg t "$PROMPT" \
'{agent: $a, parts: [{type: "text", text: $t}]}')
log "Sending initial prompt (agent: ${AGENT})..."
log "Prompt: ${PROMPT}"
echo ""
send_message "$INIT_BODY"
log "Initial prompt complete."
# ── Continue loop ─────────────────────────────────────────────────────────
CONT_BODY=$(jq -nc --arg a "$AGENT" '{agent: $a, parts: [{type: "text", text: "continue with your mainloop, monitor the supervisors, keep them alive, and sleep in an infinite loop."}]}')
while ! $STOP_LOOP; do
N=$((N + 1))
# Check idle-event rate in the last minute
idle_count=$(count_idle_events_last_minute)
if (( idle_count > MAX_IDLE_PER_MINUTE )); then
restart_session
continue
fi
log "── iteration #${N} ──"
# Inspect messages: if a question is waiting for this session, dismiss it;
# otherwise issue the standard continue command.
dismissed=0
if dismissed_raw=$(dismiss_pending_questions "$SESSION_ID"); then
dismissed="$dismissed_raw"
fi
if (( dismissed > 0 )); then
log "Dismissed ${dismissed} pending question(s)."
record_idle_event
else
send_message "$CONT_BODY"
log "continue #${N} complete."
record_idle_event
fi
done
echo ""
log "Loop stopped. Shutting down."
exit 0