cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity
2,371 Commits 1,418 Branches 10 Tags
b154d48027fa001c862cc62adcb4eadef255461f
Commit Graph

5 Commits

Author SHA1 Message Date
drew 2cbe62a70c fix(auto-agents): F1+F2+F3 telemetry liveness, reviewer perf (A+C), bash rules 
Bundles a long-overdue set of fixes that surfaced while watching the
single-PR pipeline test against PR #30 the morning of 2026-05-07.

# F1 — long-worker liveness contract
The dispatcher heartbeat was only refreshed *between* worker sessions.
On a 30-minute review the heartbeat file went stale, and any
heartbeat-watchdog (dispatchers-launcher.sh /
cleveragents-dispatchers.service) would SIGTERM a perfectly-healthy
worker mid-cycle, orphaning the OpenCode session and the
auto/claimed-* lock. ``_opencode_worker.run_session_blocking`` now
accepts an ``on_poll`` callback fired once per status-poll iteration;
``_dispatch_runtime.dispatch_one`` and ``conflict_drive.py`` wire it to
``write_heartbeat(cfg.heartbeat_path)``. Callback exceptions are
logged and swallowed so a transient EROFS on the heartbeat path can
never mask a successful worker completion.

# F2 — in-flight cycle visibility (schema v5)
The ``dispatch_*_cycles`` tables previously only recorded a row at
cycle *end*. While a worker was running, the operator's only signal
was the heartbeat file — and even that became stale (see F1). Schema
bumped to v5: ``ended_at`` is now nullable and ``cycle_id`` carries a
UNIQUE index. ``begin_cycle`` writes the in-flight row at start;
``finish_cycle`` updates it at end. ``run_one_cycle``'s try/finally
guarantees ``finish_cycle`` runs even when ``collect_candidates`` /
``dispatch_one`` raises, so an orphan ``ended_at IS NULL`` can no
longer be stuck forever after a crash.

The v4→v5 migration is now defined in ONE place — a set of helpers in
``_forgejo_cache.py`` (``DISPATCH_CYCLE_TABLES``,
``_dispatch_cycle_create_sql``, ``_dispatch_cycle_index_sqls``,
``migrate_dispatch_cycle_table_to_v5``,
``ensure_dispatch_cycle_schema``). Both
``ForgejoCache._migrate_to_v5_in_flight_rows`` and
``_dispatch_runtime.ensure_cycle_table`` import from there, eliminating
the drift risk of the previous duplicated DDL. Pre-existing rows are
preserved verbatim across the migration.

# F3 — telemetry surface for the new state
``/api/health`` now returns ``in_flight_cycle: {cycle_id, started_at,
session_id, candidates_count, elapsed_s}`` per dispatcher daemon and a
``running_long_worker: bool`` flag (heartbeat older than 600s AND a
matching pid alive — should never fire under healthy F1 operation, so
when it does it points at a real bug). The Drivers and Overview tabs
in ``.opencode/telemetry/{index.html,app.js,style.css}`` render
in-flight rows with a tinted background + "in flight" pill, daemon
tiles get a dashed border for the long-worker state, and each tile
shows the running cycle's id + elapsed time inline.

# Fix A — reasoningEffort high → medium for pr-review-worker
On its own that change alone would not have been enough, but combined
with Fix C below it dropped a representative cycle from "timed out at
30:00" to a target ~2-3min. Pure config change in
``.opencode/agents/pr-review-worker.md``; no code path touched.

# Fix C — pre-fetch PR diff in dispatch_review and embed in prompt
The reviewer used to spawn a ``git-isolator-util`` subagent, which
shelled out to ``git clone``, ``git fetch``, and ``git diff
master...HEAD``. That subagent burned 90+ seconds and several token
budgets per cycle. ``dispatch_review.py`` now fetches the unified diff
via the Forgejo ``/pulls/{n}.diff`` endpoint and embeds it into the
worker prompt under an ``UNTRUSTED CONTENT`` fence with explicit
BEGIN_PR_DIFF / END_PR_DIFF markers, head_sha pinning, character-count
metadata, and END marker redaction to defeat patch-text injection. The
worker is instructed to use the embedded diff and skip the isolator
subagent entirely when it is present. Falls back to the old path on
fetch failure or via the ``REVIEW_DISPATCHER_EMBED_DIFF=0`` env switch.

# Cross-cutting bash rules
The ``pr-review-worker``'s shell tool calls kept hitting
``permission denied`` because OpenCode's permission engine matches
the *raw, unexpanded* command string against allow-globs. Chained
commands (``&&``, ``||``, ``;``, ``|``), command substitution
(``$(...)``), bare variable assignments, multi-line continuations
(``\\\n``), heredocs, and inline ``python3 -c "..."`` strings all
contain characters the permission glob cannot span, and were silently
denied. Added ``.opencode/instructions/bash-commands.md`` (wired into
``opencode.json`` via the ``instructions`` array so it appends to
EVERY agent's system prompt globally), with hard rules + recovery
recipes (``printf > /tmp/file`` instead of heredocs;
``printf > /tmp/script.py`` + ``python3 /tmp/script.py`` instead of
``python3 -c``; ``curl -d @/tmp/body.json`` instead of multi-line
``-d '{...}'``).

# Pre-commit polish (architect/dev/test review)
Surfaced during a chief-architect / principal-developer /
senior-test-engineer code review of the uncommitted change:

- Schema DDL deduplication (described above under F2).
- ``finish_cycle`` INSERT-fallback now preserves ``started_at`` /
  ``driver_name`` when caller provides them; otherwise stamps a
  ``synthetic_started_at: true`` flag in the raw blob so cycle-time
  analytics can exclude rows whose duration was synthesised.
- ``bytes=`` → ``chars=`` in the embedded-diff header. The value is
  ``len(diff_text)`` after ``decode("utf-8")`` — a UTF-8 character
  count, not a byte count. Off-by-multibyte for non-ASCII patches.
- ``scripts/opencode-builder.sh`` mode 644 → 755.
- ``.gitignore`` entries for ``.parked-prs.json`` (runtime state for
  ``tools/park_other_prs.py --restore``) and ``.dispatcher-logs/``
  (append-only local pipeline log directory).

# Tests (381 passed, 1 skipped)
- ``test_opencode_worker.py``: 3 new tests for ``on_poll`` cadence,
  error swallowing, and backwards-compatible default.
- ``test_dispatch_runtime.py``: 7 new tests for ``begin_cycle`` /
  ``finish_cycle`` semantics, the v4→v5 migration with row
  preservation, the crash-safe try/finally path, the
  ``dispatch_one`` → ``run_session_blocking`` ``on_poll`` wiring, and
  the new ``started_at`` / ``driver_name`` plumbing through the
  INSERT-fallback branch.
- ``test_telemetry_server.py``: 4 new tests for ``in_flight_cycle``
  in ``/api/health``, the elapsed-seconds computation, and the
  ``running_long_worker`` flag.
- ``test_telemetry_schema.py``: assertion bumped from v4 → v5 and a
  new test confirming the cycle tables now allow ``ended_at IS NULL``.
- ``test_dispatch_review.py`` (new file): 14 tests for diff fetch
  (happy path, truncation, HTTP/URL errors, END_PR_DIFF redaction,
  Forgejo auth scheme), ``_build_diff_section`` (dry-run, env
  toggle, embedding, fallback), and end-to-end prompt embedding.

Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-05-07 07:12:45 -04:00
drew 593d142f6f feat(auto-agents): Tier 2 deterministic review/implementer dispatchers 
Replaces the long-running pr-review-supervisor / implementation-
supervisor LLM polling loops with host-level Python dispatchers that
own queueing, claim ownership, watchdogs, and SQLite telemetry. The
LLM workers retain sole responsibility for review judgment and code
generation; Python owns only orchestration. The hard merge invariant
is unaffected — these dispatchers do not touch master.

Driver surface
- _opencode_worker.run_session_blocking — outcome-agnostic OpenCode
  session lifecycle (completed / timeout / transport-error). Used
  directly by reviewer / implementer dispatchers whose workers do
  not emit the conflict-driver JSON exit schema.
  run_worker_blocking is now a thin wrapper that adds the
  conflict-specific JSON-outcome classification.
- _dispatch_runtime — shared Python runtime (work-group polling,
  claim helpers, dispatch loop, telemetry). Pre-checks issue labels
  before claiming and refuses when any auto/claimed-* is already
  present, distinguishing already-claimed vs labels-fetch-failed
  vs claim-failed terminal states. Frozen DispatchConfig.
- dispatch_review.py / dispatch_implementer.py — per-pipeline work
  groups, prompts, and CLIs. Each refuses startup when a competing
  AUTO-REV-SUP / AUTO-IMP-SUP legacy supervisor is live on the
  same OpenCode server (override:
  {REVIEW,IMPLEMENTER}_DISPATCHER_ALLOW_SUPERVISOR_COEXIST=1).
- _loader.py — shared sibling-module loader; replaces the three
  duplicated copies in the dispatcher entry points.

Operational hardening
- run_outer_loop tracks consecutive cycle exceptions against
  cycle_failure_budget (default 5, env-tunable per driver) and
  exits 2 on exhaustion for supervisor-driven restart.
- _sanitize_release_detail strips control bytes and neutralises
  triple-backtick fences before quoting worker raw_response in
  Forgejo claim-release comments.
- scripts/opencode-builder.sh: OPENCODE_BUILDER_SERVER_ONLY=1 keeps
  only the OpenCode HTTP API up so the Python dispatchers own
  queue orchestration without auto-agents running concurrently.

Telemetry
- _forgejo_cache.py schema v4: dispatch_review_cycles,
  dispatch_implementer_cycles. One row per cycle with cycle_id,
  driver, candidates_count, claims_acquired, swept_count,
  processed_count, terminal_state, worker_outcome, session_id,
  worker_wallclock_seconds, raw.
- .opencode/telemetry/server.py wires the new tables into
  /api/cycles?driver=dispatch_review|dispatch_implementer and
  surfaces a composite terminal_state/worker_outcome 24h breakdown
  so dashboards can distinguish session-level vs work-level
  outcomes.

Tests
- 31 new tests in tests/auto_agents/test_dispatch_runtime.py
  covering: candidate priority/dedup, claim/release labels,
  foreign-claim refusal, same-kind-claim refusal,
  labels-fetch-failed terminal state, sanitization, supervisor
  coexistence guard (pass/refuse/override/unreachable-server),
  session timeout / transport-error propagation,
  JSON-vs-no-JSON worker exits, cycle failure budget exit and
  reset, heartbeat cadence, end-to-end --once --dry-run /
  --status CLI smoke, and full prompt-snapshot tests for
  _review_prompt and _implementation_prompt (PR-fix + issue-impl).
- test_telemetry_schema.py asserts schema v4 and the presence of
  the two new dispatcher cycle tables.

338 auto_agents tests pass (was 322 before Tier 2). Conflict driver
regression suite unchanged. Dispatchers run cleanly under
--status / --once --dry-run with no Forgejo or OpenCode HTTP traffic.

Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-05-06 23:12:09 -04:00
drew 703a7c9090 feat(auto-agents): Phase A — conflict_drive.py deterministic conflict-resolution driver 
Implement Tier 1.5 conflict resolution as a peer driver to merge_drive.py
that honours the same hard invariant: every commit on master came from a SHA
whose CI passed against the exact current master.

New components
- tools/conflict_drive.py: deterministic driver that picks PRs labelled
  auto/needs-conflict-resolution, claims them via the shared
  auto/claimed-merge label, attempts a deterministic rebase with
  deepen-on-demand fallback, dispatches conflict-resolver-worker on
  conflict, force-pushes with --force-with-lease, and clears the label
  for merge_drive to re-pick. Includes 24h retry budget, escalation to
  auto/needs-implementer, single-instance lock + heartbeat, opt-in
  TOCTOU mitigations (CONFLICT_DRIVER_CYCLE_JITTER_SECONDS,
  CONFLICT_DRIVER_VERIFY_CLAIM), startup TTL constraint and required-
  labels assertions, and full SQLite telemetry.
- tools/_claim_runtime.py: shared HTTP/lock/heartbeat/claim primitives
  extracted from merge_drive.py (driver-aware claim/release markers,
  injectable op_label_map). merge_drive re-exports for back-compat.
- tools/_opencode_worker.py: blocking Python client for the OpenCode
  HTTP API with O(N) string-aware bracket matcher for tolerant JSON
  extraction from worker output.
- .opencode/agents/conflict-resolver-worker.md: subagent definition
  with tight permissions and "always finish, never abort, never --skip"
  doctrine.
- tools/inject_synthetic_conflict.py: CLI that creates PRs on a test
  fork with guaranteed conflicts (trivial / multi-commit / unresolvable)
  for end-to-end testing.

Telemetry & dashboard
- tools/_forgejo_cache.py: new conflict_drive_cycles table, cycle-level
  escalated_count, indexed retry-budget query, mark_*_escalated helper.
- tools/render-pr-velocity.py + pr-velocity.canvas.template.tsx:
  conflict-resolution activity section showing 7-day cycle counts,
  resolved/escalated/timeout/push-rejected breakdowns.

Open-issue dependency check
- tools/merge_drive.py: pr_is_eligible now consults Forgejo's blocks
  endpoint and applies auto/blocked-by-deps when any open dependency
  exists. Read-only predicate _pr_has_open_dependencies; label
  mutations live with the eligibility caller.
- tools/setup_auto_labels.py: provisions auto/blocked-by-deps.

Documentation
- docs/development/conflict-drive-plan.md: full plan including TOCTOU
  race documentation (§3.3.1) with implementation/test pointers.
- AGENTS.md: operator-facing section on conflict_drive.py and the
  expanded label registry.

Tests
- 300 unit tests pass / 1 skipped (opt-in fork integration test).
- Coverage includes JSON extractor fuzz, push-stderr classification,
  PAT scrub, deepen-on-demand fallback, retry budget escalation,
  cycle-level escalated_count stamping, claim collision detection
  (latest-claim-only with marker-primary identity), jitter wiring,
  and verify_claim_after_apply plumbing.

Quality invariant unchanged: conflict_drive.py only operates on PR head
branches, never on master. CI gating on the train-merge SHA continues
to enforce the exact-current-master rule for everything that lands.

Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-05-06 01:28:41 -04:00
drew 816b93953d fix(auto-agents): pre-launch fork environment fixes from prelaunch audit 
Resolves all blockers identified in the upstream/fork prelaunch audit
so the deterministic merge driver can be exercised end-to-end against
drew/cleveragents-core sentinels without silently hitting upstream or
hanging on perpetually-failing CI.

Code changes
------------

- tools/launch_fork.sh (new): source-able preamble that pins
  FORGEJO_OWNER, FORGEJO_REPO, FORGEJO_URL, FORGEJO_API_BASE,
  FORGEJO_DEFAULT_BRANCH, FORGEJO_PAT, FORGEJO_ORG and validates the
  fork target via GET /repos/<owner>/<repo> (must exist, be a fork,
  grant push). Refuses to export anything on validation failure so
  the parent shell isn't half-configured. Closes the silent
  auto-detect risk where launching auto-agents.md from the canonical
  clone (whose origin points at upstream) would otherwise pick up
  upstream as the target.

- tools/flag_stale_prs.py: DEFAULT_ASSIGNEE is now env-driven via
  FORGEJO_DEFAULT_ASSIGNEE (default "freemo" preserves canonical
  behaviour). Fork-mode runs can set FORGEJO_DEFAULT_ASSIGNEE=drew
  once instead of remembering --assignee on every invocation.

- tools/{_forgejo_cache,count-master-merges,pr-stats,render-milestones}.py:
  read FORGEJO_OWNER / FORGEJO_REPO / FORGEJO_API_BASE with defaults
  matching the rest of the pipeline. _forgejo_cache.py partitions
  its SQLite path per-(owner, repo) so a fork-mode reporting pass
  never clobbers the canonical cache. The default
  cleveragents/cleveragents-core keeps the historical filename
  forgejo.sqlite for backward compatibility; non-default targets
  land at forgejo.<safe-owner>.<safe-repo>.sqlite.

- .opencode/skills/supervised-workers/scripts/{submit_review,
  submit_comment,fetch_pr_stale,fetch_pr_not_stale}.ts: deleted.
  These were orphaned LLM-session snapshot scripts referenced by no
  agent or skill, and they contained a hard-coded leaked Forgejo
  PAT. NOTE: file removal does not invalidate the token; the
  operator must rotate it on Forgejo separately.

Environment changes (applied via API; not in this commit)
---------------------------------------------------------

- Fork .forgejo/workflows/{benchmark-scheduled,ci,master,release}.yml
  synced to upstream HEAD via 4 API content-PUTs. The fork was 2
  days behind upstream and used a ${{vars.docker_prefix}} template
  that resolved to a non-pullable image, so every CI run failed in
  30-40 s.

- Fork ci.yml subsequently patched to drop the push-validation
  job: it requires secrets.FORGEJO_TOKEN, but Forgejo blocks
  creation of any secret whose name begins with FORGEJO_ via the
  API on personal forks (HTTP 400 "invalid secret name"); the
  merge driver pushes from outside CI in fork-mode anyway, so
  this validation is moot here. status-check.needs and conditional
  updated to drop the dependency. Net effect: required-status-check
  contexts (build, coverage, docker, integration_tests, lint,
  quality, security, typecheck, unit_tests, e2e_tests) now post
  real terminal states instead of being cancelled when
  push-validation aborted the whole workflow at 92 s.

- Fork branch protection PATCHed apply_to_admins=true. The fork
  was inheriting Forgejo's default of false, which would have let
  the merge driver bypass branch protection (it authenticates as
  drew, who is the fork admin). Production fidelity restored.

- Fork collaborators: HAL9000 added with write permission. The
  reviewer-identity slot (FORGEJO_REVIEWER_PAT) needs a PAT
  belonging to a different account than the merge driver since
  Forgejo blocks self-approval. Verified write permission via
  /repos/.../collaborators/HAL9000/permission.

Validation
----------

- 180/180 auto_agents unit tests pass.
- launch_fork.sh tested for: success path (sourced), failure path
  (sourced — no env vars leak when validation fails), and
  executed-not-sourced path (warns appropriately).
- Fork CI confirmed structurally healthy after the workflow patches:
  fresh run posts real statuses, 6/10 required gates already
  passing on commit 38fa7765 with the rest progressing normally
  (no fast failures).

Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-05-04 22:11:59 -04:00
drew 07013335b9 Auto-agents pipeline: enforce merge invariant with deterministic driver 
Implement the improve_auto-agents_pipeline plan end-to-end and apply post-implementation hardening fixes so master can only advance via SHAs that passed CI against the exact current master.

Key changes:
- Add deterministic merge driver in `tools/merge_drive.py`:
  - Single-instance lock (`fcntl.flock`) + heartbeat/status surfaces.
  - Train-merge with bisect-on-failure and independent bisect/restart budgets.
  - `head_commit_id` optimistic lock enforcement on merge endpoint.
  - Single-PR strategy switched to `Do=merge` (sha-stable) to avoid unverified rewritten commits.
  - Restart throttling/sleeps to prevent burning retry budget under master churn.
  - Persistent clone management (`ensure_repo`: fetch/reset/clean with fresh-clone fallback on corruption).
  - Cooperative SIGTERM/SIGINT stop propagation through long CI polling.
  - Explicit claim lifecycle for `auto/claimed-merge`:
    - claim/release comments with TTL,
    - expired-claim sweep,
    - operational labels on release (`auto/ci-timeout`, `auto/restart-throttled`,
      `auto/needs-implementer`, `auto/needs-conflict-resolution`).
  - Claim-marker interoperability: sweep recognizes both driver and `claim_pr.ts` markers.
  - Hardened API semantics: split idempotent GET retries vs state-change semantics.
  - Remove token-in-URL clone pattern; use git `http.extraheader` auth instead.
  - Adopt structured module logging + env-configurable levels.

- Add/expand invariant auditor in `tools/verify_invariant.py`:
  - Non-zero exit when violations exist (cron/alert correctness).
  - Robust auto-close routine using forward patch applicability on current `origin/master`.
  - Merge-bot commit validation now checks:
    - required CI contexts passed,
    - associated PR has non-dismissed APPROVED review.
  - Improve close-path wording/docs to match forward-apply algorithm.
  - Add logger-based output and verbosity controls.

- Add operational setup/audit tooling:
  - `tools/forgejo_audit.py` (preconditions/audit report).
  - `tools/setup_auto_labels.py` (idempotent `auto/*` label provisioning).
  - `tools/setup_branch_protection.py` (direct-push allow-list enforcement).
  - `tools/audit_branch_protection.py` (dismiss_stale_approvals audit/flip support).
  - `tools/migrate_to_new_driver.py` (claim/schedule/train cleanup migration).
  - `tools/flag_stale_prs.py` (idle PR triage flow).
  - `tools/local_ci_gate.sh` canonical local gate runner with `--continue-on-fail`.

- Add claim orchestration support in skills scripts:
  - New `claim_pr.ts` helper (claim/release + TTL comments).
  - `list_prs.ts` gains `--exclude-claimed` filter.
  - Update script reference docs accordingly.

- Telemetry/schema upgrades in `tools/_forgejo_cache.py`:
  - Add `merge_cycle`, `ci_gate_events`, `llm_activity`.
  - Add batched `ci_gate_events` insertion API with rollback semantics.
  - Ensure `bisect_depth` default handling is safe.
  - Surface merge-driver telemetry in velocity reporting pipeline.

- Agent prompt/behavior updates:
  - Review supervisor idle loop tuned (300s -> 60s).
  - Review worker cycle cap/escalation behavior refined.
  - Task implementor guidance updated to use local CI gate wrapper.

- Documentation and operational guidance:
  - Expand `AGENTS.md` with merge invariant runbook, label registry, tool links,
    and full merge-driver env var catalog (including logging/restart/claim TTL knobs).
  - Update `CHANGELOG.md` with implementation and hardening entries, plus deferred TS-test note.

- Repo hygiene:
  - Correct `.gitignore` to stop blanket ignoring `tools/*`; keep only generated artifacts ignored.

Testing/validation:
- Add comprehensive unit suite under `tests/auto_agents/` covering:
  - merge driver recursion/restarts/409 paths/signal handling/claim sweeps,
  - verifier auto-close logic with real local git fixtures,
  - schema migration + telemetry batch writes,
  - branch protection and setup/audit helpers.
- Current result: `100 passed` in `tests/auto_agents/`.
 2026-05-03 21:31:16 -04:00