Adds RUN_CI_LOCAL — an on-demand local-CI verdict source for when the
cluster's Forgejo CI is broken — plus robustness fixes, the telemetry
Live-tab rewrite, and PR-level cost attribution.
Controller:
- RUN_CI_LOCAL: the master swaps its Forgejo CI callbacks for local
`forgejo-runner exec` runs (tools/run-ci-full-local.sh + local_ci.py).
Async per-(owner,repo,SHA) on-disk job cache; preflights the
forgejo-runner binary + Docker daemon at startup (fail loud, not a
red verdict on every PR); GCs finished run dirs + per-run actcache.
- ci-not-ready implementer outcome + implementer_ci_not_ready event:
an implementer that runs before the on-demand verdict exists parks
in AWAITING_CI instead of dead-ending at STUCK; capped against
ci_red ping-pong.
- ci_poll_exhaustion skips its sweep while a local CI run is in
flight, so AWAITING_CI workflows queued behind on-demand CI are not
STUCK'd by the remote-CI-sized timeout.
- Escalate the workflow after repeated worker-internal-error at a
tier instead of retrying to pickup-exhaustion -> STUCK.
- forgejo_http: normalise Forgejo's per-gate `status` key to `state`
so failing gates are actually counted (they previously all read as
pending).
- Per-tier worker timeouts bumped +15 min; a timed-out attempt's
dirty-worktree residue is preserved on auto-scratch/pr-<N> before
the next attempt's reset.
- Implementer agents now verify only the CI-flagged gate(s) via a
targeted re-run rather than the full local battery before claiming
resolved/noop. Re-running the whole suite CI will run anyway was the
#1 cause of implementer timeouts; CI remains the real gate and
re-dispatches the implementer on red.
Telemetry:
- Live tab rebuilt on /api/live (controller DB run state + the live
OpenCode session forest) after the live_log_writer sidecar was
retired with the legacy dispatchers.
- Durable per-attempt input/output payloads surfaced in the Live
drill-down, archived-session detail, and Workflows timeline.
- PR-level cost attribution: worker session tags carry -pr-<n>;
backfill_llm_activity_pr.py repairs rows written before the fix.
Shared:
- tools/controller/session_tag.py — one canonical controller-tag
parser shared by the telemetry server and the backfill.
Tests: new coverage for local_ci (state machine, log parsing,
_summarize_run, GC, in-flight probe, preflight), the ci-not-ready
path, the escalation/ci-not-ready SQL counters, the ci_poll
in-flight skip, and CI-status payload parsing across both sources.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
drew
0db0a15dad