cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity
Files
ae940f45644971d5937b755d2459eceffeb6500a
cleveragents-core/.opencode/telemetry/index.html
T
drew 179b420270 feat(telemetry): live dispatcher-run observability subsystem 
A sidecar + web UI that turns a running 4-hour dispatcher test from
"tail three logs and guess" into a real-time status grid.

  tools/live_log_writer.py — sidecar. Tails the implementer + review
  dispatcher logs, watches OpenCode session archives + Phase-4
  telemetry JSONL, polls host process state, and emits two artifacts:
    <run_dir>/events.jsonl  — append-only event stream, closed event set
    <run_dir>/snapshot.json — rolling status grid, atomically replaced
  Restart-safe: replays events.jsonl to rebuild counters, resumes each
  tailer from a per-tailer offset file. Offsets are saved BEFORE the
  line is handled, so an unclean crash can drop <=1 event but never
  double-count.

  .opencode/telemetry/{server.py,app.js,index.html,style.css} — the
  "Live" tab. server.py serves /runs + /snapshot + /events (the last
  with a byte-offset cache so a 3s UI poll is O(new events), not
  O(file size)); app.js renders the process/service grids, the
  implementer/reviewer cards, and a bounded event feed.

  scripts/dispatchers-launcher.sh — wires the sidecar in as a
  best-effort child: respawned outside the dispatcher crash-loop
  budget, its exit never propagates to the launcher's, with a
  rapid-crash backoff floor so a writer that crashes on boot can't
  tight-loop.

Hardened through three review rounds (23 fixes) before landing:
partial-read race in the dir watchers, a shared offset-temp-file
race, replay/offset double-count, dead event types, the /events
O(n)-per-poll scan, the app.js↔snapshot field-contract drift, a
respawn tight-loop, and more.

Tests: tests/auto_agents/test_live_log_writer.py — 46 tests covering
every log-pattern handler, replay↔handler counter agreement, atomic
snapshot write, idempotent restart, partial-read resilience, the
/events offset cache, a cross-language app.js↔snapshot contract test,
a schema-doc consistency test, and two end-to-end smoke tests that
run the actual sidecar process. Full tests/auto_agents/ suite: 1548
passed, 3 skipped.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 08:34:43 -04:00

257 lines
13 KiB
HTML
Raw Blame History

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<title>Pipeline Telemetry Console</title>
<link rel="stylesheet" href="/static/style.css" />
</head>
<body>
<header class="topbar">
<div class="brand">
<span class="dot dot-pulse" id="conn-dot" title="connection status"></span>
<strong>Pipeline Telemetry</strong>
<span class="sep">·</span>
<span id="repo-target" class="muted">loading…</span>
</div>
<div class="meta-extra">
<span id="cache-info" class="muted small"></span>
<span class="sep">·</span>
<span id="server-time" class="muted small"></span>
</div>
</header>
<nav class="tabs" role="tablist">
<button class="tab" data-tab="overview" role="tab" aria-selected="true">Overview</button>
<button class="tab" data-tab="live" role="tab">Live</button>
<button class="tab" data-tab="drivers" role="tab">Drivers</button>
<button class="tab" data-tab="prs" role="tab">PRs</button>
<button class="tab" data-tab="velocity" role="tab">Velocity</button>
<button class="tab" data-tab="llm" role="tab">LLM</button>
<button class="tab" data-tab="sessions" role="tab">Sessions</button>
<button class="tab" data-tab="cost" role="tab">Cost</button>
<button class="tab" data-tab="about" role="tab">About</button>
</nav>
<main>
<!-- ── Overview ───────────────────────────────────────────── -->
<section class="pane active" data-pane="overview">
<h2>Pipeline health</h2>
<p class="muted small">Refresh: 5 s · daemon discovery via /proc · OpenCode probed at <code id="oc-url">…</code></p>
<div id="daemon-grid" class="grid grid-4">
<div class="card placeholder">loading…</div>
</div>
<h3 style="margin-top:24px">Last cycles (24 h)</h3>
<div class="grid grid-2">
<div class="card">
<div class="card-head">merge_drive — recent</div>
<div id="overview-merge" class="card-body">loading…</div>
</div>
<div class="card">
<div class="card-head">conflict_drive — recent</div>
<div id="overview-conflict" class="card-body">loading…</div>
</div>
<div class="card">
<div class="card-head">dispatch_review — recent</div>
<div id="overview-dispatch_review" class="card-body">loading…</div>
</div>
<div class="card">
<div class="card-head">dispatch_implementer — recent</div>
<div id="overview-dispatch_implementer" class="card-body">loading…</div>
</div>
</div>
</section>
<!-- ── Live (per-run status grid driven by /runs, /snapshot, /events) ── -->
<section class="pane" data-pane="live">
<div class="live-layout">
<aside class="live-runs">
<div class="live-runs-head">
<strong>Runs</strong>
<span class="muted small" id="live-runs-meta">…</span>
</div>
<div id="live-runs-list" class="live-runs-list">loading…</div>
<p class="muted small live-runs-foot">
Source: <code id="live-runs-root">…</code><br>
Click a row to inspect. Refresh: 5 s.
</p>
</aside>
<section class="live-panel" id="live-panel">
<div class="live-empty muted">Select a run from the left to see live status.</div>
</section>
</div>
</section>
<!-- ── Drivers ────────────────────────────────────────────── -->
<section class="pane" data-pane="drivers">
<h2>Driver telemetry</h2>
<p class="muted small">Refresh: 15 s · source: <code id="cache-path-drivers">…</code></p>
<div class="card">
<div class="card-head">
<span>merge_cycle</span>
<span class="muted small">last 24h breakdown: <span id="merge-breakdown">…</span></span>
</div>
<div id="merge-table" class="card-body table-wrap">loading…</div>
</div>
<div class="card">
<div class="card-head">
<span>conflict_drive_cycles</span>
<span class="muted small">last 24h breakdown: <span id="conflict-breakdown">…</span></span>
</div>
<div id="conflict-table" class="card-body table-wrap">loading…</div>
</div>
<div class="card">
<div class="card-head">
<span>dispatch_review_cycles</span>
<span class="muted small">last 24h breakdown (terminal_state/worker_outcome): <span id="dispatch_review-breakdown">…</span></span>
</div>
<div id="dispatch_review-table" class="card-body table-wrap">loading…</div>
</div>
<div class="card">
<div class="card-head">
<span>dispatch_implementer_cycles</span>
<span class="muted small">last 24h breakdown (terminal_state/worker_outcome): <span id="dispatch_implementer-breakdown">…</span></span>
</div>
<div id="dispatch_implementer-table" class="card-body table-wrap">loading…</div>
</div>
</section>
<!-- ── PRs by label ───────────────────────────────────────── -->
<section class="pane" data-pane="prs">
<h2>Open PRs by <code>auto/*</code> label</h2>
<p class="muted small">Refresh: 30 s · live Forgejo query (no cache)</p>
<div class="filter-row">
<label>Filter:
<select id="pr-label-select">
<option value="">— all open auto/* PRs —</option>
<option value="auto/claimed-merge">auto/claimed-merge</option>
<option value="auto/needs-conflict-resolution">auto/needs-conflict-resolution</option>
<option value="auto/needs-implementer">auto/needs-implementer</option>
<option value="auto/blocked-by-deps">auto/blocked-by-deps</option>
<option value="auto/restart-throttled">auto/restart-throttled</option>
<option value="auto/ci-timeout">auto/ci-timeout</option>
<option value="auto/stale-inactivity">auto/stale-inactivity</option>
<option value="auto/unstable">auto/unstable</option>
<option value="auto/sentinel">auto/sentinel</option>
</select>
</label>
<span id="pr-count" class="muted small"></span>
</div>
<div id="pr-table" class="card-body table-wrap">loading…</div>
</section>
<!-- ── Velocity ───────────────────────────────────────────── -->
<section class="pane" data-pane="velocity">
<h2>Velocity windows</h2>
<p class="muted small">Refresh: 5 min · cache-backed</p>
<div id="velocity-grid" class="grid grid-4">
<div class="card placeholder">loading…</div>
</div>
<p class="muted small" style="margin-top:24px">
For the full historical view (weekly throughput, top authors, monthly volume,
daily series, oldest open PRs), keep
<code>tools/reports.sh</code> rendering the canvas — the console focuses on
live/short-window pipeline state. Charts here mirror the canvas but are
cache-driven and refresh on demand.
</p>
</section>
<!-- ── LLM sessions ───────────────────────────────────────── -->
<section class="pane" data-pane="llm">
<h2>OpenCode sessions</h2>
<p class="muted small">Refresh: 10 s · OpenCode at <code id="oc-url-2">…</code></p>
<div id="llm-table" class="card-body table-wrap">loading…</div>
</section>
<!-- ── Archived sessions ──────────────────────────────────── -->
<section class="pane" data-pane="sessions">
<h2>Archived worker sessions</h2>
<p class="muted small">
Refresh: 30 s ·
snapshot of every <code>pr-review-worker</code> /
<code>implementation-worker</code> session before it was deleted ·
source: <code id="archive-dir-info">…</code>
</p>
<div class="card">
<div class="card-head">
<span>Sessions (newest first)</span>
<span class="muted small" id="archive-count">…</span>
</div>
<div id="archive-table" class="card-body table-wrap">loading…</div>
</div>
<div class="card hidden" id="archive-detail-card">
<div class="card-head">
<span id="archive-detail-title">Detail</span>
<button class="btn btn-link" id="archive-detail-close" type="button">close</button>
</div>
<div class="card-body" id="archive-detail-body">
<span class="muted">click a row to inspect</span>
</div>
</div>
</section>
<!-- ── Cost ───────────────────────────────────────────────── -->
<section class="pane" data-pane="cost">
<h2>Token usage &amp; cost</h2>
<p class="muted small">Refresh: 60 s · source: <code>llm_activity</code> table · prices: <code>.opencode/telemetry/prices.json</code> (override)</p>
<div class="filter-row">
<label>Window:
<select id="cost-days-select">
<option value="1">last 24 h</option>
<option value="7" selected>last 7 d</option>
<option value="30">last 30 d</option>
<option value="90">last 90 d</option>
</select>
</label>
<span id="cost-totals" class="muted small"></span>
</div>
<div id="cost-table" class="card-body table-wrap">loading…</div>
<div id="cost-instrumentation-note" class="callout hidden">
<strong>Instrumentation pending.</strong>
The <code>llm_activity</code> table has 0 rows for this window — drivers are
not yet writing per-dispatch token usage. The data shape is ready;
wiring the writers is a follow-up. This tab will populate automatically
once <code>conflict_drive.py</code> and the LLM supervisors record their
dispatches into <code>llm_activity</code>.
</div>
</section>
<!-- ── About ──────────────────────────────────────────────── -->
<section class="pane" data-pane="about">
<h2>About this console</h2>
<div class="card">
<div class="card-head">Environment audit</div>
<div class="card-body">
<dl id="about-dl"></dl>
</div>
</div>
<div class="card">
<div class="card-head">Where the data comes from</div>
<div class="card-body">
<ul class="tight">
<li><strong>Daemons + heartbeats:</strong> <code>/proc</code> walk for PIDs · file-mtime probes for heartbeats.</li>
<li><strong>Cycle telemetry:</strong> per-repo SQLite cache <code>tools/.cache/forgejo.&lt;owner&gt;.&lt;repo&gt;.sqlite</code> · tables <code>merge_cycle</code> and <code>conflict_drive_cycles</code>.</li>
<li><strong>PR label flow:</strong> live Forgejo API <code>GET /repos/&lt;owner&gt;/&lt;repo&gt;/issues</code>.</li>
<li><strong>OpenCode sessions:</strong> live <code>GET /session</code> (and per-session <code>/message</code>).</li>
<li><strong>Cost:</strong> aggregated from <code>llm_activity(model, tokens_in, tokens_out, cached_tokens)</code> · prices from <code>prices.json</code>.</li>
</ul>
</div>
</div>
<div class="card">
<div class="card-head">Operator notes</div>
<div class="card-body">
<ul class="tight">
<li>Console is <strong>read-only</strong>. It never writes to Forgejo, OpenCode, or the SQLite cache.</li>
<li>Default bind: <strong>127.0.0.1 (loopback only)</strong>. Set <code>TELEMETRY_HOST=0.0.0.0</code> only if you've added auth in a reverse proxy.</li>
<li>Source the same shell the drivers run from (<code>source tools/launch_fork.sh</code>) so the console reads the right cache and hits the right Forgejo target.</li>
<li>To switch repos without restarting, change <code>FORGEJO_OWNER</code>/<code>FORGEJO_REPO</code> in the launching shell and restart the console — the cache file is selected at startup.</li>
</ul>
</div>
</div>
</section>
</main>
<script src="/static/app.js"></script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink