Files
cleveragents-core/.opencode/telemetry/server.py
T
drew 62d4e8f07d fix(telemetry): cost dashboard now computes real USD totals
Three intertwined bugs caused every Cost-tab row to display \$0
even after the scraper started writing real token data:

1. **Lookup key mismatch.** ``_cost_usd`` looked up bare ``model``
   but ``_DEFAULT_PRICES`` was keyed by ``provider/model`` — every
   priced model silently missed. Fixed by adding ``_price_key`` and
   a fallback chain: ``provider/model`` → bare ``model`` → ``_unknown``.

2. **SQL grouped by model only.** Same modelID served by two providers
   (e.g. ``claude-opus-4-6`` via Anthropic direct vs a local proxy) at
   different rates was conflated into one row. Fixed: ``GROUP BY model,
   provider`` in ``_api_cost`` + ``provider`` returned in each row.

3. **Math convention mismatch.** ``_cost_usd`` did ``(tokens_in -
   cached) * in_rate`` assuming ``tokens_in`` was total input. But
   the scraper records ``tokens_in`` as OpenCode's ``info.tokens.input``
   (fresh, non-cached), so ``tokens_in - cached`` went negative
   whenever cache reads exceeded fresh input — which is the common
   case with Anthropic prompt caching. Fixed: no subtraction; the
   three populations bill at their three rates.

Pricing seeded for the 8 models the scraper has actually observed
(``_DEFAULT_PRICES`` corrected from stale Opus-3 numbers + new entries
for the Haiku 4.5 / GPT-5 family / CleverThis HF endpoints):

| Provider     | Model                     | in    | out   | cached_in |
|--------------|---------------------------|-------|-------|-----------|
| local-claude | claude-opus-4-6           | 5.00  | 25.00 | 0.50      |
| local-claude | claude-sonnet-4-6         | 3.00  | 15.00 | 0.30      |
| local-claude | claude-haiku-4-5          | 1.00  | 5.00  | 0.10      |
| openai       | gpt-5 / gpt-5-codex       | 1.25  | 10.00 | 0.125     |
| openai       | gpt-5-mini                | 0.25  | 2.00  | 0.025     |
| openai       | gpt-5-nano                | 0.05  | 0.40  | 0.005     |
| CleverThis-* | (HF endpoints, advisory)  | 0.50  | 1.00  | —         |

Operators can override without touching code via
``.opencode/telemetry/prices.json`` (added; same keying convention).
``_load_prices`` now skips ``_comment`` / ``_last_updated`` /
``_sources`` metadata keys so docs in the JSON don't pollute the table.

Smoke against current ``llm_activity`` (3743 turns, 450 archives):
total USD over the lifetime window is now \$118.06, with all 8 models
showing as priced.

Tests pin all three regressions: provider-qualified lookup, bare-model
fallback, no-subtraction math, GROUP BY (model, provider), and the
metadata-key filter in ``_load_prices``.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 16:47:54 -04:00

2060 lines
82 KiB
Python

#!/usr/bin/env python3
"""Pipeline telemetry console — a small, dependency-free HTTP server that
exposes everything you need to monitor the auto-agents pipeline in one
browser tab.
Why this exists
---------------
The auto-agents pipeline emits useful telemetry into several silos:
- ``tools/.cache/forgejo.<owner>.<repo>.sqlite`` — cached PR/commit data
+ ``merge_cycle`` / ``conflict_drive_cycles`` / ``llm_activity`` driver
telemetry rows.
- ``tools/<driver>.heartbeat`` (or ``/tmp/<driver>.heartbeat``) — per-driver
liveness probes touched once per cycle.
- ``ps``-discoverable Python daemons (``merge_drive``, ``conflict_drive``,
``verify_invariant``, ``opencode-builder.sh``).
- The OpenCode HTTP server (``http://127.0.0.1:4096`` by default) — agent
sessions, agent messages, idle/busy state.
- The Forgejo HTTP API — open PRs by ``auto/*`` label, branch protection,
approvals.
Tailing five terminals + running ``sqlite3`` ad-hoc + curl-ing two HTTP
APIs is fine for one operator; it does not scale to "let another dev
look at this without setting anything up". This console consolidates
all of the above into a single ``python3 .opencode/telemetry/server.py``
entry point with a tabbed web UI that auto-refreshes per pane.
Design constraints
------------------
- **Stdlib only.** No Flask / FastAPI / Jinja. Runs anywhere a driver
runs (the auto-agents pipeline already mandates Python 3.13+ stdlib;
matching that contract keeps the install footprint zero).
- **Repo-aware.** Reads ``FORGEJO_OWNER`` / ``FORGEJO_REPO`` and picks
the right cache file via the same partition logic as
``tools/_pipeline_cache.py``. The console banner spells out which
repo's data is on screen — never let an operator monitor canonical
while expecting fork data, or vice-versa.
- **Loopback-only by default.** Binds ``127.0.0.1``. Operators who
want LAN access set ``TELEMETRY_HOST=0.0.0.0`` deliberately AND
understand that v1 ships without auth (loopback is the security
model). README spells this out.
- **Per-tab polling.** Each tab fetches its own endpoints on its own
cadence; inactive tabs do not poll. Server load stays low even on
long-lived browser sessions.
Endpoints (all JSON unless noted)
---------------------------------
- ``GET /`` index.html (tabbed SPA)
- ``GET /static/<file>`` app.js / style.css
- ``GET /api/meta`` repo target, identity, paths in use
- ``GET /api/health`` daemon PIDs, heartbeats, OpenCode health
- ``GET /api/cycles?driver=...`` merge / conflict / dispatch_review /
dispatch_implementer cycle rows
- ``GET /api/prs?label=...`` open PRs by Forgejo label (live API)
- ``GET /api/velocity?window=...`` window stats from cache
- ``GET /api/sessions`` OpenCode sessions
- ``GET /api/cost?days=N`` token + estimated cost rollup
Run
---
::
source tools/launch_fork.sh # or canonical-mode env
python3 .opencode/telemetry/server.py # http://127.0.0.1:8765
Environment overrides
---------------------
- ``TELEMETRY_HOST`` (default ``127.0.0.1``)
- ``TELEMETRY_PORT`` (default ``8765``)
- ``OPENCODE_URL`` (default ``http://127.0.0.1:4096``)
- ``TELEMETRY_PRICES_JSON`` (default ``.opencode/telemetry/prices.json``;
used by /api/cost to convert tokens to dollars)
Exit codes
----------
::
0 — clean shutdown via Ctrl+C
2 — argument error / port-bind failure / missing cache file
"""
from __future__ import annotations
import argparse
import json
import logging
import os
import re
import signal
import sqlite3
import subprocess
import sys
import threading
import time
import urllib.error
import urllib.parse
import urllib.request
from datetime import datetime, timezone
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from typing import Any
logger = logging.getLogger("telemetry")
# ─── Paths + repo-aware cache resolution ─────────────────────────────────
REPO_ROOT = Path(__file__).resolve().parents[2]
STATIC_DIR = Path(__file__).resolve().parent
CACHE_DIR = REPO_ROOT / "tools" / ".cache"
# Mirrors the partition logic in tools/_pipeline_cache.py so the console
# always reads the SAME file the drivers write to. Computed at import.
REPO_OWNER = os.environ.get("FORGEJO_OWNER", "cleveragents")
REPO_NAME = os.environ.get("FORGEJO_REPO", "cleveragents-core")
API_BASE = os.environ.get(
"FORGEJO_API_BASE", "https://git.cleverthis.com/api/v1"
).rstrip("/")
FORGEJO_TOKEN = os.environ.get("GITEA_TOKEN") or os.environ.get(
"FORGEJO_PAT", ""
)
OPENCODE_URL = os.environ.get("OPENCODE_URL", "http://127.0.0.1:4096").rstrip(
"/"
)
def _cache_path() -> Path:
"""Resolve the per-(owner, repo) cache file the same way
``tools/_pipeline_cache.py:DEFAULT_CACHE_PATH`` does. Console always
reads the SAME file the drivers wrote to."""
if REPO_OWNER == "cleveragents" and REPO_NAME == "cleveragents-core":
return CACHE_DIR / "forgejo.sqlite"
safe_owner = re.sub(r"[^a-zA-Z0-9._-]+", "-", REPO_OWNER)
safe_repo = re.sub(r"[^a-zA-Z0-9._-]+", "-", REPO_NAME)
return CACHE_DIR / f"forgejo.{safe_owner}.{safe_repo}.sqlite"
CACHE_PATH = _cache_path()
# ─── SQLite helpers ──────────────────────────────────────────────────────
def _open_db() -> sqlite3.Connection | None:
"""Open the per-repo cache read-only. Returns None when the file
does not exist yet (cold-start before any sync) so endpoints can
return empty payloads instead of 500ing."""
if not CACHE_PATH.exists():
return None
# mode=ro keeps us from racing with driver writers; the SQLite
# locking story tolerates many readers + one writer regardless,
# but ro makes the intent unambiguous.
uri = f"file:{CACHE_PATH}?mode=ro"
conn = sqlite3.connect(uri, uri=True, timeout=2)
conn.row_factory = sqlite3.Row
return conn
def _rows_to_dicts(rows: list[sqlite3.Row]) -> list[dict[str, Any]]:
return [dict(r) for r in rows]
# ─── Forgejo API (read-only) ─────────────────────────────────────────────
def _forgejo_get(path: str, params: dict[str, str] | None = None) -> Any:
"""GET ``API_BASE + path``. Returns parsed JSON or None on error.
Console never POSTs — read-only by design."""
if not FORGEJO_TOKEN:
return None
url = API_BASE + path
if params:
url += "?" + urllib.parse.urlencode(params)
req = urllib.request.Request(
url,
headers={
"Authorization": f"token {FORGEJO_TOKEN}",
"Accept": "application/json",
},
)
try:
with urllib.request.urlopen(req, timeout=10) as resp:
return json.loads(resp.read())
except (urllib.error.HTTPError, urllib.error.URLError, OSError) as e:
logger.warning("forgejo GET %s failed: %r", path, e)
return None
# ─── OpenCode API (read-only) ────────────────────────────────────────────
def _opencode_get(path: str) -> Any:
"""GET against the local OpenCode server. Returns parsed JSON or
None when the server is unreachable. The console renders an
'OpenCode unreachable' badge in that case rather than failing
the whole page."""
url = OPENCODE_URL + path
try:
with urllib.request.urlopen(url, timeout=3) as resp:
return json.loads(resp.read())
except (urllib.error.HTTPError, urllib.error.URLError, OSError) as e:
logger.debug("opencode GET %s failed: %r", path, e)
return None
# ─── Daemon discovery (ps + heartbeat) ───────────────────────────────────
# Heartbeat probe order matches ``_dispatch_runtime.resolve_lock_or_heartbeat``
# / ``merge_drive.load_config`` / ``conflict_drive.load_config``: explicit env
# override → ``/var/run`` → ``$XDG_RUNTIME_DIR`` (typically
# ``/run/user/<uid>``) → ``/tmp``. Production hosts and dev containers usually
# land on ``XDG_RUNTIME_DIR``; the explicit env vars are the operator's escape
# hatch.
def _heartbeat_candidates(basename: str, env_var: str | None = None) -> list[str]:
paths: list[str] = []
if env_var:
explicit = os.environ.get(env_var)
if explicit:
paths.append(explicit)
paths.append(f"/var/run/{basename}")
runtime_dir = os.environ.get("XDG_RUNTIME_DIR")
if runtime_dir:
paths.append(f"{runtime_dir.rstrip('/')}/{basename}")
paths.append(f"/tmp/{basename}")
return paths
_DAEMON_SPECS: list[dict[str, Any]] = [
{
"name": "merge_drive",
"match": ["merge_drive.py"],
"heartbeat_candidates": _heartbeat_candidates(
"merge-driver.heartbeat", "MERGE_DRIVER_HEARTBEAT_PATH"
),
},
{
"name": "conflict_drive",
"match": ["conflict_drive.py"],
"heartbeat_candidates": _heartbeat_candidates(
"conflict-driver.heartbeat", "CONFLICT_DRIVER_HEARTBEAT_PATH"
),
},
{
"name": "verify_invariant",
"match": ["verify_invariant.py"],
"heartbeat_candidates": _heartbeat_candidates(
"verify-invariant.heartbeat", "VERIFY_INVARIANT_HEARTBEAT_PATH"
),
},
{
# Tier 2 deterministic dispatcher for pr-review-worker. Heartbeat
# is written by ``run_outer_loop`` after each cycle (see
# ``tools/_dispatch_runtime.py``).
"name": "dispatch_review",
"match": ["dispatch_review.py"],
"heartbeat_candidates": _heartbeat_candidates(
"review-dispatcher.heartbeat", "REVIEW_DISPATCHER_HEARTBEAT_PATH"
),
},
{
# Tier 2 deterministic dispatcher for implementation-worker.
"name": "dispatch_implementer",
"match": ["dispatch_implementer.py"],
"heartbeat_candidates": _heartbeat_candidates(
"implementer-dispatcher.heartbeat",
"IMPLEMENTER_DISPATCHER_HEARTBEAT_PATH",
),
},
{
"name": "opencode_builder",
"match": ["opencode-builder.sh", "opencode serve"],
"heartbeat_candidates": [],
},
]
def _ps_lookup(needles: list[str]) -> list[dict[str, Any]]:
"""Return [{pid, cmdline, cpu, mem, etime}] for processes whose
cmdline contains any of ``needles``. Cross-platform-ish via /proc;
falls back to ``ps -eo`` if /proc isn't available."""
found: list[dict[str, Any]] = []
proc_dir = Path("/proc")
if proc_dir.exists():
for entry in proc_dir.iterdir():
if not entry.name.isdigit():
continue
try:
cmdline_raw = (entry / "cmdline").read_bytes()
except (FileNotFoundError, PermissionError):
continue
cmd = cmdline_raw.replace(b"\x00", b" ").decode(
"utf-8", "replace"
).strip()
if not cmd:
continue
if any(n in cmd for n in needles):
pid = int(entry.name)
stat = _proc_stat_brief(pid)
found.append({"pid": pid, "cmdline": cmd, **stat})
return found
# ps -eo fallback (BSD-ish / macOS / non-/proc Linux)
try:
out = subprocess.check_output(
["ps", "-eo", "pid=,etime=,pcpu=,pmem=,args="],
text=True, timeout=5,
)
except (subprocess.SubprocessError, FileNotFoundError):
return []
for line in out.splitlines():
line = line.strip()
if not line:
continue
parts = line.split(None, 4)
if len(parts) < 5:
continue
pid, etime, cpu, mem, cmd = parts
if any(n in cmd for n in needles):
try:
pid_int = int(pid)
except ValueError:
continue
found.append(
{
"pid": pid_int, "cmdline": cmd,
"etime": etime, "cpu_pct": float(cpu),
"mem_pct": float(mem),
}
)
return found
def _proc_stat_brief(pid: int) -> dict[str, Any]:
"""Lightweight /proc-based stat. Returns etime/cpu/mem if available,
empty dict otherwise (process gone between iterdir and read)."""
try:
with open(f"/proc/{pid}/stat", encoding="utf-8") as f:
stat_line = f.read()
except (FileNotFoundError, PermissionError):
return {}
# comm field can contain spaces; use the rightmost ')' as boundary.
rb = stat_line.rfind(")")
if rb == -1:
return {}
rest = stat_line[rb + 2 :].split()
# Field positions per `man proc`: starttime is field 22 (0-indexed
# 21 in the post-comm slice).
try:
starttime_clk = int(rest[19])
except (IndexError, ValueError):
return {}
try:
clk_tck = os.sysconf("SC_CLK_TCK") or 100
except (ValueError, OSError):
clk_tck = 100
try:
with open("/proc/uptime", encoding="utf-8") as f:
uptime_s = float(f.read().split()[0])
except (FileNotFoundError, IndexError, ValueError):
return {}
elapsed_s = uptime_s - (starttime_clk / clk_tck)
return {"elapsed_s": round(elapsed_s, 1)}
def _heartbeat_age(candidates: list[str]) -> dict[str, Any] | None:
for path in candidates:
p = Path(path)
if p.exists():
mtime = p.stat().st_mtime
return {
"path": str(p),
"mtime_s_ago": round(time.time() - mtime, 1),
}
return None
# ─── Pricing (token → $) ─────────────────────────────────────────────────
# Defaults seeded from publicly-listed prices; override via
# .opencode/telemetry/prices.json. Costs are $/1,000,000 tokens.
_DEFAULT_PRICES: dict[str, dict[str, float]] = {
# Anthropic (public — May 2026 list prices). Cache reads are
# 10% of standard input across the family.
"local-claude/claude-opus-4-6": {"in": 5.0, "out": 25.0, "cached_in": 0.50},
"local-claude/claude-sonnet-4-6": {"in": 3.0, "out": 15.0, "cached_in": 0.30},
"local-claude/claude-haiku-4-5": {"in": 1.0, "out": 5.0, "cached_in": 0.10},
# OpenAI (public — May 2026 list prices). gpt-5 / gpt-5-codex
# share the same per-token rate; gpt-5-mini / gpt-5-nano are
# the budget tier.
"openai/gpt-5": {"in": 1.25, "out": 10.0, "cached_in": 0.125},
"openai/gpt-5-codex": {"in": 1.25, "out": 10.0, "cached_in": 0.125},
"openai/gpt-5-mini": {"in": 0.25, "out": 2.0, "cached_in": 0.025},
"openai/gpt-5-nano": {"in": 0.05, "out": 0.4, "cached_in": 0.005},
# CleverThis HF Inference Endpoints — billed per endpoint-hour,
# not per token. The per-token entries here are a best-effort
# imputation for "what would this have cost on a public API";
# treat as advisory not source-of-truth.
"CleverThis-15/Qwen3-6-35B-A3B-GGUF-UD-Q3-K-XL": {"in": 0.5, "out": 1.0},
"CleverThis/Qwen3-6-35B-A3B-GGUF-BF16": {"in": 0.5, "out": 1.0},
"CleverThis-2/MiniMax-M2-7-GGUF-BF16": {"in": 0.5, "out": 1.0},
"CleverThis-4/Kimi-K2-6-GGUF-Q2-K-XL": {"in": 0.5, "out": 1.0},
# Sentinel — when we don't know, we'd rather produce 0 than a wild
# number. The UI surfaces "model unpriced" rows separately.
"_unknown": {"in": 0.0, "out": 0.0},
}
def _load_prices() -> dict[str, dict[str, float]]:
override_path = Path(
os.environ.get(
"TELEMETRY_PRICES_JSON",
str(STATIC_DIR / "prices.json"),
)
)
if override_path.exists():
try:
data = json.loads(override_path.read_text())
if isinstance(data, dict):
merged = dict(_DEFAULT_PRICES)
# Skip metadata keys (``_comment``, ``_last_updated``,
# ``_sources``) so they don't pollute the price table.
# The ``_unknown`` sentinel is preserved because it IS
# a valid price-table entry the lookup falls back to.
for k, v in data.items():
if k.startswith("_") and k != "_unknown":
continue
if isinstance(v, dict):
merged[k] = v
return merged
except (OSError, json.JSONDecodeError) as e:
logger.warning("ignoring prices.json: %r", e)
return dict(_DEFAULT_PRICES)
def _price_key(provider: str | None, model: str) -> str:
"""Resolve the lookup key for the price table.
The price table is keyed by ``provider/model`` because the same
``modelID`` can be served by multiple providers at very different
economics (e.g. ``claude-opus-4-6`` via Anthropic direct vs via a
local proxy). Falls back to the bare model name for backward
compat with manually-inserted rows that never had a provider.
"""
if provider:
return f"{provider}/{model}"
return model
def _cost_usd(provider: str | None, model: str,
prices: dict[str, dict[str, float]],
tokens_in: int, tokens_out: int, cached: int) -> float:
"""Compute USD cost for a per-model token tally.
Lookup order: ``{provider}/{model}`` → bare ``model`` → ``_unknown``.
The fallback chain lets a price entry written without provider
namespacing still match, while still preferring the
provider-qualified entry when present.
Token-accounting convention: ``tokens_in`` and ``cached`` are
DISJOINT populations — the scraper writes ``tokens_in`` as
OpenCode's ``info.tokens.input`` (fresh, non-cached prompt
tokens) plus any cache-write tokens; ``cached`` is
``info.tokens.cache.read``. So total cost is the simple sum of
three line items at three rates, with no subtraction needed.
"""
p = prices.get(_price_key(provider, model)) or prices.get(model) \
or prices.get("_unknown", {})
in_cost = tokens_in * p.get("in", 0.0) / 1_000_000
cached_cost = cached * p.get("cached_in", p.get("in", 0.0)) / 1_000_000
out_cost = tokens_out * p.get("out", 0.0) / 1_000_000
return round(in_cost + cached_cost + out_cost, 4)
# ─── Endpoint handlers ───────────────────────────────────────────────────
def _api_meta() -> dict[str, Any]:
return {
"repo_owner": REPO_OWNER,
"repo_name": REPO_NAME,
"api_base": API_BASE,
"cache_path": str(CACHE_PATH),
"cache_present": CACHE_PATH.exists(),
"opencode_url": OPENCODE_URL,
"has_forgejo_token": bool(FORGEJO_TOKEN),
"server_time_utc": datetime.now(timezone.utc).isoformat(),
"console_version": "0.1.0",
"warn_canonical_target": (
REPO_OWNER == "cleveragents" and REPO_NAME == "cleveragents-core"
),
}
# Heartbeat staleness threshold (seconds) for the "running long worker"
# hint. Set well above the dispatcher's ``poll_interval_seconds`` (2s)
# so a healthy heartbeat refresh during run_session_blocking will keep
# the daemon out of this state. Triggered when the heartbeat is older
# than this AND a matching pid is alive — the combination indicates an
# in-flight worker session whose on_poll callback isn't writing the
# heartbeat (e.g. the ``on_poll`` wiring regressed, EROFS, or the
# dispatcher has wedged on a non-OpenCode codepath).
_LONG_WORKER_HEARTBEAT_THRESHOLD_S = 600.0
# Daemon name → cycle table. Used by _api_health to surface the most
# recent in-flight cycle (``ended_at IS NULL``) per dispatcher in the
# health response so the UI can render "running cycle X started Ys ago"
# instead of stale ``last_cycle_at`` from a previous completed cycle.
_DAEMON_CYCLE_TABLES: dict[str, str] = {
"dispatch_review": "dispatch_review_cycles",
"dispatch_implementer": "dispatch_implementer_cycles",
}
def _last_in_flight_cycle(table_name: str) -> dict[str, Any] | None:
"""Return the most recent in-flight cycle for ``table_name``, or
``None`` if the cache file is missing, the table is missing, or no
in-flight rows exist.
"In flight" means ``ended_at IS NULL`` per the v5 schema. The query
is read-only; on any SQLite error we return ``None`` so the health
probe degrades gracefully (a transient DB lock should not flip
daemon status to ``unhealthy``).
"""
conn = _open_db()
if conn is None:
return None
try:
try:
cur = conn.execute(
f"SELECT cycle_id, started_at, driver, candidates_count, "
f"session_id "
f"FROM {table_name} "
f"WHERE ended_at IS NULL "
f"ORDER BY started_at DESC LIMIT 1",
)
except sqlite3.Error:
return None
row = cur.fetchone()
if row is None:
return None
rec = _rows_to_dicts([row])[0]
try:
started = datetime.fromisoformat(rec["started_at"])
if started.tzinfo is None:
started = started.replace(tzinfo=timezone.utc)
elapsed = (
datetime.now(timezone.utc) - started
).total_seconds()
rec["elapsed_s"] = round(elapsed, 1)
except (TypeError, ValueError):
rec["elapsed_s"] = None
return rec
finally:
conn.close()
def _api_health() -> dict[str, Any]:
daemons: list[dict[str, Any]] = []
for spec in _DAEMON_SPECS:
procs = _ps_lookup(spec["match"])
hb = _heartbeat_age(spec["heartbeat_candidates"])
running = bool(procs)
in_flight = None
table = _DAEMON_CYCLE_TABLES.get(spec["name"])
if table is not None:
in_flight = _last_in_flight_cycle(table)
# "Running long worker" hint: the dispatcher is alive but its
# heartbeat is stale. With the F1 on_poll wiring landed this
# should never fire under healthy operation — every poll
# iteration writes the heartbeat. When it does fire, it points
# at a callback regression or a filesystem failure on the
# heartbeat path, which is operationally different from "the
# process is alive and freshly running" or "the process is
# dead". The threshold is large enough that it does not flap
# on normal cycle boundaries (cycle_interval_seconds ≤ 600).
running_long_worker = False
if running and hb is not None:
running_long_worker = (
hb.get("mtime_s_ago", 0.0)
>= _LONG_WORKER_HEARTBEAT_THRESHOLD_S
)
daemons.append(
{
"name": spec["name"],
"running": running,
"processes": procs,
"heartbeat": hb,
"in_flight_cycle": in_flight,
"running_long_worker": running_long_worker,
}
)
oc_health = _opencode_get("/global/health")
return {
"daemons": daemons,
"opencode": {
"url": OPENCODE_URL,
"reachable": oc_health is not None,
"details": oc_health,
},
}
# Whitelisted Tier 2 dispatcher driver names → SQLite table names.
# The ``/api/cycles`` handler interpolates the table name into a
# parametrized SQL query; gating the lookup through this dict keeps
# the f-string SQL safety explicit (the only path that picks a table
# name is membership in this set) and ensures the rows-query and the
# outcome-breakdown query agree on the same table.
_DISPATCH_TABLES: dict[str, str] = {
"dispatch_review": "dispatch_review_cycles",
"dispatch_implementer": "dispatch_implementer_cycles",
}
def _api_cycles(driver: str, limit: int) -> dict[str, Any]:
conn = _open_db()
if conn is None:
return {"rows": [], "note": f"cache file not present: {CACHE_PATH}"}
try:
if driver == "merge":
cur = conn.execute(
"SELECT id, started_at, ended_at, pr_numbers, train_id,"
" bisect_depth, total_seconds, terminal_state,"
" action_taken, action_detail"
" FROM merge_cycle ORDER BY started_at DESC LIMIT ?",
(limit,),
)
elif driver == "conflict":
cur = conn.execute(
"SELECT cycle_id, pr_number, started_at, ended_at,"
" outcome, failure_kind, candidates_count,"
" resolved_count, escalated_count, timeout_count,"
" push_rejected_count, details_json"
" FROM conflict_drive_cycles"
" ORDER BY started_at DESC LIMIT ?",
(limit,),
)
elif driver in _DISPATCH_TABLES:
# Tier 2 dispatcher telemetry. ``terminal_state`` describes
# the *session lifecycle* (completed / timeout /
# transport-error / already-claimed / labels-fetch-failed
# / claim-failed / dry-run) while ``worker_outcome`` is the
# worker's own JSON-emitted outcome when present (for
# implementer workers). Both are surfaced so an operator
# can distinguish "the OpenCode session ended cleanly" from
# "the worker reported success / failure for the actual
# work" — these are independent dimensions per the
# session-status / JSON-outcome separation introduced in
# the dispatcher hardening pass.
table = _DISPATCH_TABLES[driver]
cur = conn.execute(
f"SELECT cycle_id, started_at, ended_at, driver,"
f" candidates_count, claims_acquired, swept_count,"
f" processed_count, terminal_state, worker_outcome,"
f" session_id, worker_wallclock_seconds, raw"
f" FROM {table}"
f" ORDER BY started_at DESC LIMIT ?",
(limit,),
)
else:
return {
"rows": [],
"note": f"unknown driver: {driver!r} (expected "
f"merge|conflict|{'|'.join(_DISPATCH_TABLES)})",
}
rows = _rows_to_dicts(list(cur.fetchall()))
# outcome breakdown over the LAST 24 H, regardless of limit, so
# the UI's summary card is stable when the operator changes the
# row limit.
if driver == "merge":
counts = conn.execute(
"SELECT terminal_state AS k, COUNT(*) AS n FROM merge_cycle"
" WHERE started_at > datetime('now','-24 hours')"
" GROUP BY terminal_state ORDER BY n DESC"
).fetchall()
elif driver == "conflict":
counts = conn.execute(
"SELECT outcome AS k, COUNT(*) AS n FROM"
" conflict_drive_cycles"
" WHERE started_at > datetime('now','-24 hours')"
" GROUP BY outcome ORDER BY n DESC"
).fetchall()
else:
# Composite breakdown: dashboards must show both terminal
# state and worker outcome side by side, since a session
# that ``completed`` with ``worker_outcome="rebase-failed"``
# is operationally different from one that ``completed``
# with no JSON exit at all. ``table`` is the same lookup
# used for the rows query above; reusing the variable
# prevents the two queries from drifting onto different
# tables.
counts = conn.execute(
f"SELECT terminal_state || '/' ||"
f" COALESCE(worker_outcome, '-') AS k,"
f" COUNT(*) AS n FROM {table}"
f" WHERE started_at > datetime('now','-24 hours')"
f" GROUP BY k ORDER BY n DESC"
).fetchall()
return {
"rows": rows,
"outcome_breakdown_24h": _rows_to_dicts(list(counts)),
}
finally:
conn.close()
def _api_prs(label: str | None) -> dict[str, Any]:
"""Live Forgejo query — does not hit the cache. Cache is delta-synced
by the drivers; for label flow we want sub-second freshness."""
params = {
"type": "pulls",
"state": "open",
"limit": "50",
}
if label:
params["labels"] = label
body = _forgejo_get(
f"/repos/{REPO_OWNER}/{REPO_NAME}/issues", params,
)
if not isinstance(body, list):
return {"rows": [], "note": "forgejo unreachable or no token"}
rows: list[dict[str, Any]] = []
for issue in body:
if not isinstance(issue, dict):
continue
labels = [
lbl.get("name") for lbl in (issue.get("labels") or [])
if isinstance(lbl, dict)
]
rows.append(
{
"number": issue.get("number"),
"title": issue.get("title"),
"html_url": issue.get("html_url"),
"user": (issue.get("user") or {}).get("login"),
"created_at": issue.get("created_at"),
"updated_at": issue.get("updated_at"),
"labels": labels,
"auto_labels": [lb for lb in labels if (lb or "").startswith("auto/")],
}
)
return {"rows": rows}
def _api_velocity(window: str) -> dict[str, Any]:
conn = _open_db()
if conn is None:
return {"window": window, "stats": {}, "note": "cache absent"}
try:
# Mirror the windows tools/render-pr-velocity.py uses; convert
# the human label to a SQLite datetime modifier.
win_map = {
"24h": "-24 hours",
"48h": "-48 hours",
"7d": "-7 days",
"30d": "-30 days",
}
if window not in win_map:
return {
"window": window,
"stats": {},
"note": "unknown window",
}
modifier = win_map[window]
try:
commits = conn.execute(
"SELECT COUNT(*) FROM commits"
" WHERE author_date > datetime('now', ?)",
(modifier,),
).fetchone()[0]
except sqlite3.OperationalError:
# Schema variant — fall back to whatever the cache has.
commits = 0
try:
opened = conn.execute(
"SELECT COUNT(*) FROM pulls"
" WHERE created_at > datetime('now', ?)",
(modifier,),
).fetchone()[0]
except sqlite3.OperationalError:
opened = 0
try:
merged = conn.execute(
"SELECT COUNT(*) FROM pulls"
" WHERE merged = 1 AND merged_at > datetime('now', ?)",
(modifier,),
).fetchone()[0]
except sqlite3.OperationalError:
merged = 0
return {
"window": window,
"stats": {
"commits": commits,
"opened": opened,
"merged": merged,
},
}
finally:
conn.close()
def _api_sessions() -> dict[str, Any]:
sessions = _opencode_get("/session")
if not isinstance(sessions, list):
return {"rows": [], "note": "opencode unreachable"}
rows: list[dict[str, Any]] = []
now_ms = int(time.time() * 1000)
for s in sessions:
if not isinstance(s, dict):
continue
sid = s.get("id")
msgs = _opencode_get(f"/session/{sid}/message") if sid else None
msg_count = len(msgs) if isinstance(msgs, list) else 0
time_obj = s.get("time") or {}
created = time_obj.get("created")
updated = time_obj.get("updated")
rows.append(
{
"id": sid,
"title": s.get("title"),
"directory": s.get("directory"),
"created_s_ago": round((now_ms - created) / 1000, 1)
if created else None,
"updated_s_ago": round((now_ms - updated) / 1000, 1)
if updated else None,
"message_count": msg_count,
}
)
rows.sort(key=lambda r: r.get("updated_s_ago") or 1e18)
return {"rows": rows}
# ─── Archived sessions (post-mortem of past worker runs) ─────────────────
#
# These endpoints expose the on-disk archives written by
# ``tools/_opencode_worker.py:run_session_blocking`` after every
# terminal return (completed / timeout / transport-error). The archives
# survive session deletion in OpenCode, so an operator can reconstruct
# what a worker actually said even after the dispatcher cycle is over.
#
# The path resolution mirrors ``_opencode_worker._resolve_archive_dir``:
# explicit ``OPENCODE_WORKER_ARCHIVE_DIR`` env var wins over the repo
# default ``<repo_root>/.dispatcher-logs/sessions``. We DO NOT honour
# ``OPENCODE_WORKER_ARCHIVE_DISABLED`` here because that controls the
# writer side, not the read-only console.
_DEFAULT_ARCHIVE_DIR = REPO_ROOT / ".dispatcher-logs" / "sessions"
# Maximum number of archive entries to return from the listing endpoint.
# Keeps the page snappy even on long-running pipelines that have logged
# hundreds of sessions.
_ARCHIVE_LIST_LIMIT = 200
# Filename guard for the detail endpoint. Mirrors the writer's safe
# filename component regex (``[A-Za-z0-9_.-]``); the only legitimate
# extra character is ``__`` from the writer's section separator. Any
# filename that does not match this is treated as a path-traversal
# attempt and 404'd without touching the filesystem.
_SAFE_ARCHIVE_FILENAME_RE = re.compile(r"^[A-Za-z0-9_.-]+\.json$")
def _archive_dir_path() -> Path:
explicit = os.environ.get("OPENCODE_WORKER_ARCHIVE_DIR")
if explicit:
return Path(explicit)
return _DEFAULT_ARCHIVE_DIR
def _api_archived_sessions() -> dict[str, Any]:
"""Index of session archives.
Returns lightweight metadata only: filename, agent, tag, status,
started_at, archived_at, message_count, size_bytes. The detail
endpoint serves the full payload for a clicked-on row.
"""
archive_dir = _archive_dir_path()
if not archive_dir.is_dir():
return {
"archive_dir": str(archive_dir),
"rows": [],
"note": "archive directory does not exist yet",
}
entries: list[dict[str, Any]] = []
for path in archive_dir.glob("*.json"):
try:
stat = path.stat()
with path.open("r", encoding="utf-8") as f:
data = json.load(f)
except (OSError, ValueError) as e:
entries.append(
{
"filename": path.name,
"size_bytes": None,
"error": str(e),
}
)
continue
if not isinstance(data, dict):
continue
per_turn = data.get("per_turn") or []
entries.append(
{
"filename": path.name,
"size_bytes": stat.st_size,
"modified_at": datetime.fromtimestamp(
stat.st_mtime, tz=timezone.utc
).isoformat(),
"agent": data.get("agent"),
"tag": data.get("tag"),
"status": data.get("status"),
"session_id": data.get("session_id"),
"started_at": data.get("started_at"),
"archived_at": data.get("archived_at"),
"wallclock_seconds": data.get("wallclock_seconds"),
"message_count": len(data.get("messages") or []),
"turn_count": len(per_turn) if isinstance(per_turn, list) else 0,
# v2 (2026-05-10) fields: present for subagent archives,
# ``None`` for top-level wrapper archives. The UI can use
# ``parent_session_id`` to group every session from one
# dispatcher cycle into a single tree row, and
# ``subagent_depth`` to nest the table view. Old (v1)
# archives lack these keys; ``.get`` returns None so the
# row schema stays uniform.
"schema_version": data.get("schema_version"),
"parent_session_id": data.get("parent_session_id"),
"subagent_title": data.get("subagent_title"),
"subagent_depth": data.get("subagent_depth"),
}
)
# Newest archive first. Use ``modified_at`` since it is ISO-formatted
# in the entry already; sort uses the timestamp string lexicographically
# which is correct for ISO-8601.
entries.sort(key=lambda e: e.get("modified_at") or "", reverse=True)
return {
"archive_dir": str(archive_dir),
"rows": entries[:_ARCHIVE_LIST_LIMIT],
"total": len(entries),
"limit": _ARCHIVE_LIST_LIMIT,
}
def _api_archived_session_detail(filename: str | None) -> dict[str, Any]:
"""Full payload for a single archive file."""
if not filename:
return {"error": "missing filename query parameter"}
if not _SAFE_ARCHIVE_FILENAME_RE.match(filename):
# Reject anything containing ``/``, ``..``, NULL bytes, etc.
# Without this guard a request like
# ``?filename=../../../../etc/passwd`` would be served verbatim.
return {"error": "invalid filename"}
archive_dir = _archive_dir_path()
path = archive_dir / filename
# Defence in depth: even though _SAFE_ARCHIVE_FILENAME_RE rejects
# path separators, resolve the path and confirm it stays inside
# the archive directory before reading.
try:
resolved = path.resolve()
archive_resolved = archive_dir.resolve()
except OSError as e:
return {"error": f"path resolution failed: {e}"}
if archive_resolved not in resolved.parents and resolved != archive_resolved:
return {"error": "filename escapes archive directory"}
if not resolved.is_file():
return {"error": f"not found: {filename}"}
try:
with resolved.open("r", encoding="utf-8") as f:
return json.load(f)
except (OSError, ValueError) as e:
return {"error": str(e)}
# ─── Live-run artifacts (events.jsonl + snapshot.json) ──────────────────
#
# These endpoints are populated by ``tools/live_log_writer.py`` running as a
# sidecar of the dispatcher launcher. The writer is the only producer; this
# server is read-only.
#
# Runs are discovered under ``TELEMETRY_RUNS_ROOT`` (default ``/tmp``): any
# immediate subdirectory containing ``snapshot.json`` or ``events.jsonl`` is
# treated as a run. The run's basename (e.g.
# ``implementer-test-2026-05-13-run4``) is the ``run_id`` used in the
# ``?run=<id>`` query parameter on the snapshot/events endpoints.
#
# Runs are not assumed to be "live" — historical runs whose writer is no
# longer running stay browsable as long as their files exist. The runs list
# carries a ``is_live`` heuristic (snapshot.json mtime < 30s) so the UI can
# distinguish active runs from archived ones.
_RUNS_ROOT_ENV = "TELEMETRY_RUNS_ROOT"
_DEFAULT_RUNS_ROOT = "/tmp"
# A snapshot.json mtime newer than this many seconds means the writer is
# still alive. Matches the writer's default --snapshot-interval (5s) plus
# slack for clock skew + writer schedule jitter.
_LIVE_SNAPSHOT_FRESH_S = 30.0
# Filename guard for ``?run=`` values. The basename must match — no path
# separators, no ``..``, no NULL bytes. Anything else 400s.
_SAFE_RUN_ID_RE = re.compile(r"^[A-Za-z0-9._-]+$")
def _runs_root() -> Path:
return Path(os.environ.get(_RUNS_ROOT_ENV, _DEFAULT_RUNS_ROOT))
def _discover_runs() -> list[dict[str, Any]]:
"""Scan ``_runs_root()`` for subdirectories that look like writer runs.
A directory qualifies if it contains ``snapshot.json`` or
``events.jsonl``. The returned list is sorted with the most recently
updated run first so the UI lands on the active run by default.
"""
root = _runs_root()
if not root.is_dir():
return []
now = time.time()
rows: list[dict[str, Any]] = []
try:
children = list(root.iterdir())
except OSError:
return []
for child in children:
# ``is_dir`` / ``is_file`` can raise PermissionError on /tmp
# entries owned by other users (e.g. ``/tmp/.vnc-0``). We swallow
# every os-level error here so the runs list keeps working when
# the runs root happens to be a multi-tenant directory like
# ``/tmp``. The cost is silent skips for genuinely-corrupt
# entries, which is acceptable for a read-only discovery scan.
try:
if not child.is_dir():
continue
except OSError:
continue
snap = child / "snapshot.json"
events = child / "events.jsonl"
try:
has_snapshot = snap.is_file()
except OSError:
has_snapshot = False
try:
has_events = events.is_file()
except OSError:
has_events = False
if not (has_snapshot or has_events):
continue
snap_mtime: float | None = None
events_mtime: float | None = None
try:
if has_snapshot:
snap_mtime = snap.stat().st_mtime
except OSError:
has_snapshot = False
try:
if has_events:
events_mtime = events.stat().st_mtime
except OSError:
has_events = False
# Peek at snapshot for top-line fields so the runs list can render
# without an extra round-trip per row.
peek: dict[str, Any] = {}
if has_snapshot:
try:
snap_data = json.loads(snap.read_text(encoding="utf-8"))
if isinstance(snap_data, dict):
run_info = snap_data.get("run") or {}
impl = snap_data.get("implementer") or {}
rev = snap_data.get("reviewer") or {}
peek = {
"boot_ts": run_info.get("boot_ts"),
"uptime_seconds": run_info.get("uptime_seconds"),
"implementer_phase": impl.get("current_phase"),
"implementer_active_pr": impl.get("active_pr_number"),
"reviewer_phase": rev.get("current_phase"),
"telemetry_rows": (
(impl.get("totals") or {}).get("telemetry_rows_total")
),
"outcome_disputed_count": (
(impl.get("totals") or {}).get(
"outcome_disputed_count"
)
),
}
except (OSError, ValueError):
pass
# Cheap event count — line count of events.jsonl. SQLite-cheap on
# files of even tens of thousands of lines; we cap the read at 4MB
# by reading in chunks to bound worst-case latency on a runaway log.
event_count = None
if has_events:
event_count = _count_lines(events, max_bytes=4 * 1024 * 1024)
is_live = (
snap_mtime is not None and (now - snap_mtime) < _LIVE_SNAPSHOT_FRESH_S
)
rows.append(
{
"run_id": child.name,
"path": str(child),
"has_snapshot": has_snapshot,
"has_events": has_events,
"snapshot_mtime_s_ago": (
round(now - snap_mtime, 1) if snap_mtime is not None else None
),
"events_mtime_s_ago": (
round(now - events_mtime, 1)
if events_mtime is not None else None
),
"events_count_approx": event_count,
"is_live": is_live,
**peek,
}
)
rows.sort(
key=lambda r: r.get("snapshot_mtime_s_ago")
if r.get("snapshot_mtime_s_ago") is not None
else float("inf"),
)
return rows
def _count_lines(path: Path, max_bytes: int = 4 * 1024 * 1024) -> int | None:
"""Cheap line-counter that bails out after ``max_bytes`` to keep the
runs-list endpoint responsive even when an events.jsonl gets large."""
try:
with path.open("rb") as f:
count = 0
read = 0
while read < max_bytes:
chunk = f.read(64 * 1024)
if not chunk:
break
count += chunk.count(b"\n")
read += len(chunk)
return count
except OSError:
return None
def _resolve_run_dir(run_id: str | None) -> tuple[Path | None, str | None]:
"""Validate ``run_id`` against the discovered runs list. Returns
``(path, None)`` on success or ``(None, error_message)`` for any failure
mode (missing param, path-traversal attempt, unknown run)."""
if not run_id:
return None, (
"missing 'run' query parameter — call /runs to list available runs"
)
if not _SAFE_RUN_ID_RE.match(run_id):
return None, "invalid run id (path-traversal attempt or unsupported chars)"
candidate = _runs_root() / run_id
if not candidate.is_dir():
return None, f"unknown run: {run_id!r}"
# Defence in depth — confirm the resolved path is still inside the root.
try:
resolved = candidate.resolve()
root_resolved = _runs_root().resolve()
except OSError as e:
return None, f"path resolution failed: {e}"
if root_resolved not in resolved.parents:
return None, "run escapes runs root"
return candidate, None
def _api_runs() -> tuple[int, dict[str, Any]]:
runs = _discover_runs()
return 200, {
"runs_root": str(_runs_root()),
"rows": runs,
"count": len(runs),
}
def _api_snapshot(run_id: str | None) -> tuple[int, dict[str, Any]]:
run_dir, err = _resolve_run_dir(run_id)
if err is not None:
return 400 if "invalid" in err or "missing" in err else 404, {
"error": err
}
snap = run_dir / "snapshot.json" # type: ignore[union-attr]
if not snap.is_file():
return 503, {
"error": f"snapshot not yet written for run {run_id!r}",
"run_dir": str(run_dir),
}
try:
return 200, json.loads(snap.read_text(encoding="utf-8"))
except (OSError, ValueError) as e:
return 500, {"error": str(e), "snapshot_path": str(snap)}
def _parse_event_ts_ms(ts: str | None) -> int | None:
"""Parse the ISO-8601 ``ts`` field emitted by the live writer back to
a unix epoch in milliseconds. The writer's format is
``YYYY-MM-DDTHH:MM:SS.mmmZ`` (millisecond precision, trailing Z)."""
if not ts:
return None
try:
# Replace trailing Z with +00:00 for fromisoformat.
dt = datetime.fromisoformat(ts.replace("Z", "+00:00"))
except ValueError:
return None
return int(dt.timestamp() * 1000)
# ─── /events byte-offset cache ───────────────────────────────────────────
#
# events.jsonl is append-only and grows for the life of a run (no
# rotation). Re-scanning it from byte 0 on every ~3s UI poll is O(file
# size) per poll and degrades visibly over a multi-hour run.
#
# The cache exploits two facts: (1) the UI polls with a monotonically
# increasing ``since_ms`` (each poll passes back the previous response's
# ``next_since_ms``), and (2) events are appended in emission-time order.
#
# Invariant of a cache entry ``run_dir -> (since_ms, safe_offset)``:
# EVERY event whose line starts before ``safe_offset`` has
# ``ts_ms <= since_ms``. So a later request with ``since_ms' >= since_ms``
# can seek straight to ``safe_offset`` — everything before it would be
# filtered out anyway. Truncation is handled at use-time by checking the
# CURRENT file size against ``safe_offset``: if the file is now smaller
# than the cached offset, the entry is stale and we fall back to a full
# scan from byte 0 (no separate stored file-size needed).
_EVENTS_OFFSET_CACHE: dict[str, tuple[int, int]] = {}
_EVENTS_OFFSET_CACHE_LOCK = threading.Lock()
_EVENTS_OFFSET_CACHE_MAX = 32 # bounded — only a handful of runs ever exist
def _api_events(
run_id: str | None,
since_ms: int,
limit: int,
types_filter: set[str] | None,
) -> tuple[int, dict[str, Any]]:
run_dir, err = _resolve_run_dir(run_id)
if err is not None:
return 400 if "invalid" in err or "missing" in err else 404, {
"error": err
}
events_path = run_dir / "events.jsonl" # type: ignore[union-attr]
if not events_path.is_file():
return 200, {
"rows": [], "next_since_ms": since_ms,
"note": f"events.jsonl not yet present: {events_path}",
}
cache_key = str(run_dir)
try:
file_size = events_path.stat().st_size
except OSError as e:
return 500, {"error": str(e), "events_path": str(events_path)}
# Decide where to start scanning. Default: byte 0 (cold path).
start_offset = 0
with _EVENTS_OFFSET_CACHE_LOCK:
cached = _EVENTS_OFFSET_CACHE.get(cache_key)
if (
cached is not None
and since_ms >= cached[0] # request's since_ms >= cached since_ms
and file_size >= cached[1] # file did not shrink/truncate under us
):
start_offset = cached[1]
rows: list[dict[str, Any]] = []
latest_ts_ms = since_ms
# safe_offset: byte offset of the first event with ts_ms > since_ms.
# Until we see one, every event scanned is filterable, so the running
# safe_offset is "current position". Once we see an unfiltered event
# we freeze safe_offset at its start byte.
safe_offset = start_offset
safe_offset_frozen = False
try:
with events_path.open("r", encoding="utf-8") as f:
f.seek(start_offset)
while True:
line_start = f.tell()
line = f.readline()
if not line:
break # EOF
if not line.endswith("\n"):
break # partial trailing line — writer mid-append
stripped = line.strip()
if not stripped:
if not safe_offset_frozen:
safe_offset = f.tell()
continue
try:
evt = json.loads(stripped)
except ValueError:
if not safe_offset_frozen:
safe_offset = f.tell()
continue
ts_ms = _parse_event_ts_ms(evt.get("ts"))
if ts_ms is None:
if not safe_offset_frozen:
safe_offset = f.tell()
continue
if ts_ms <= since_ms:
# Filterable — advance the safe offset past it.
if not safe_offset_frozen:
safe_offset = f.tell()
continue
# First event with ts_ms > since_ms — freeze safe_offset
# at its START byte (everything before is <= since_ms).
if not safe_offset_frozen:
safe_offset = line_start
safe_offset_frozen = True
if ts_ms > latest_ts_ms:
latest_ts_ms = ts_ms
if types_filter and evt.get("type") not in types_filter:
continue
rows.append(evt)
if len(rows) >= limit:
break
except OSError as e:
return 500, {"error": str(e), "events_path": str(events_path)}
# Refresh the cache entry. safe_offset now satisfies the invariant
# for this request's since_ms. Bound the cache size.
with _EVENTS_OFFSET_CACHE_LOCK:
_EVENTS_OFFSET_CACHE[cache_key] = (since_ms, safe_offset)
if len(_EVENTS_OFFSET_CACHE) > _EVENTS_OFFSET_CACHE_MAX:
# Evict an arbitrary oldest-ish entry (dict preserves insertion
# order; the first key is the least-recently-inserted).
_EVENTS_OFFSET_CACHE.pop(next(iter(_EVENTS_OFFSET_CACHE)))
return 200, {
"rows": rows,
"next_since_ms": latest_ts_ms,
"count": len(rows),
"limit": limit,
"run_id": run_id,
}
# ─── Per-run session tree + detail ───────────────────────────────────────
#
# These two endpoints back the Live tab's "session tree → click-to-inspect"
# UX. They reuse two existing data sources:
#
# - events.jsonl (per run) — gives us every session_id the run ever
# touched, plus turn counts, state transitions, and the archive path
# once a session ends.
# - .dispatcher-logs/sessions/*.json — full transcripts, including
# parent_session_id, written by _opencode_worker.run_session_blocking
# after a session ends.
#
# Parent → child resolution: archive JSON carries ``parent_session_id``.
# For sessions whose archive has not landed yet (worker still running), we
# don't know the parent until either the archive lands or we ask OpenCode
# directly. The /session detail endpoint handles the live-OpenCode
# fall-through; the tree endpoint marks unresolved parents as None.
_ARCHIVE_FILENAME_SID_RE = re.compile(r"__(?P<sid>ses_[A-Za-z0-9]+)\.json$")
def _archive_index_by_session_id() -> dict[str, Path]:
"""Scan the archive directory once and return ``session_id -> Path``.
Filename-only scan (no JSON parse), so it's cheap even on a directory
with hundreds of archives. The caller reads JSON only for the
sessions it actually needs."""
archive_dir = _archive_dir_path()
out: dict[str, Path] = {}
if not archive_dir.is_dir():
return out
try:
for path in archive_dir.glob("*.json"):
m = _ARCHIVE_FILENAME_SID_RE.search(path.name)
if m:
out[m.group("sid")] = path
except OSError:
pass
return out
# Status precedence: a later event's status only overrides an earlier
# session's status if it's "more terminal". This lets a single scan over
# events build a stable status per session without needing to sort.
_STATUS_PRECEDENCE = {
"starting": 0,
"running": 1,
"idle": 2,
"terminated": 3,
"archived": 4,
"timeout": 5,
"transport_error": 5,
}
def _bump_status(current: str | None, candidate: str) -> str:
if current is None:
return candidate
if _STATUS_PRECEDENCE.get(candidate, 0) >= _STATUS_PRECEDENCE.get(current, 0):
return candidate
return current
def _scan_run_sessions(events_path: Path) -> dict[str, dict[str, Any]]:
"""Walk events.jsonl once, build a {session_id: row} dict. Each row
carries everything we can learn from events alone — no archive reads,
no OpenCode calls. The caller decorates with archive/OpenCode data."""
by_sid: dict[str, dict[str, Any]] = {}
try:
with events_path.open("r", encoding="utf-8") as f:
for line in f:
line = line.strip()
if not line:
continue
try:
evt = json.loads(line)
except ValueError:
continue
sid = evt.get("session_id")
if not sid:
continue
row = by_sid.setdefault(
sid,
{
"session_id": sid,
"agent": None,
"model": None,
"tag": None,
"pr_number": None,
"source": None,
"parent_session_id": None,
"depth": None,
"status": None,
"started_at": None,
"ended_at": None,
"turn_count": 0,
"input_tokens": 0,
"output_tokens": 0,
"tools_used": {},
"archive_path": None,
# ``tier_recommendation`` is populated only for
# estimator sessions (by subagent.tier_recommendation).
# None for everyone else — the UI checks for
# presence before badging.
"tier_recommendation": None,
},
)
ts = evt.get("ts")
data = evt.get("data") or {}
etype = evt.get("type")
if etype == "worker.session_created":
# Two emits per session: a "pre-id" with model only
# (session_id absent there, filtered above) and the
# real one with session_id. The data payload here is
# ``{"agent": ...}`` only.
row["agent"] = row["agent"] or data.get("agent")
row["tag"] = row["tag"] or evt.get("tag")
row["pr_number"] = row["pr_number"] or evt.get("pr_number")
row["source"] = row["source"] or evt.get("source")
if row["started_at"] is None:
row["started_at"] = ts
row["status"] = _bump_status(row["status"], "starting")
elif etype == "worker.state_change":
to_state = data.get("to") or ""
if to_state == "busy":
row["status"] = _bump_status(row["status"], "running")
elif to_state == "idle":
row["status"] = _bump_status(row["status"], "idle")
elif etype == "worker.turn_finished":
row["turn_count"] += 1
row["input_tokens"] += int(data.get("input_tok") or 0)
row["output_tokens"] += int(data.get("output_tok") or 0)
for tool in data.get("tools") or []:
row["tools_used"][tool] = (
row["tools_used"].get(tool, 0) + 1
)
elif etype == "worker.subagent_spawned":
# This event's session_id is the SUBAGENT's, not the
# parent's. We learn the agent + depth + archive_path
# here; parent is resolved from the archive JSON.
row["agent"] = row["agent"] or data.get("agent")
row["depth"] = (
row["depth"] if row["depth"] is not None
else data.get("depth")
)
row["archive_path"] = (
row["archive_path"] or data.get("archive_path")
)
row["source"] = row["source"] or evt.get("source")
if row["started_at"] is None:
row["started_at"] = ts
row["status"] = _bump_status(row["status"], "archived")
elif etype == "worker.terminated":
row["ended_at"] = ts
row["status"] = _bump_status(row["status"], "terminated")
elif etype == "worker.archive_written":
row["archive_path"] = (
row["archive_path"] or data.get("archive_path")
)
if row["ended_at"] is None:
row["ended_at"] = ts
row["status"] = _bump_status(row["status"], "archived")
elif etype == "worker.timeout":
row["ended_at"] = ts
row["status"] = _bump_status(row["status"], "timeout")
elif etype == "worker.transport_error":
row["ended_at"] = ts
row["status"] = _bump_status(row["status"], "transport_error")
# ── Live-subagent events (sourced from the OpenCode SSE
# ── subscriber in live_log_writer.py). These let the tree
# ── render BEFORE the archive lands, which is the whole
# ── point of the SSE wire — the archive only lands
# ── 10-25 min later. parent_session_id + depth come in
# ── via the event's ``data`` payload.
elif etype == "subagent.session_created":
row["agent"] = row["agent"] or data.get("agent")
row["depth"] = (
row["depth"] if row["depth"] is not None
else data.get("depth")
)
row["parent_session_id"] = (
row["parent_session_id"] or data.get("parent_session_id")
)
row["tag"] = row["tag"] or evt.get("tag")
row["pr_number"] = row["pr_number"] or evt.get("pr_number")
row["source"] = row["source"] or evt.get("source")
if row["started_at"] is None:
row["started_at"] = data.get("started_at") or ts
row["status"] = _bump_status(row["status"], "starting")
elif etype == "subagent.state_change":
to_state = data.get("to") or ""
if to_state == "running":
row["status"] = _bump_status(row["status"], "running")
elif to_state == "idle":
row["status"] = _bump_status(row["status"], "idle")
elif etype == "subagent.tool_call_start":
tool = data.get("tool") or "tool"
row["tools_used"][tool] = (
row["tools_used"].get(tool, 0) + 1
)
elif etype == "subagent.terminated":
row["ended_at"] = ts
row["status"] = _bump_status(row["status"], "terminated")
elif etype == "subagent.tier_recommendation":
# Surface the tier pick on the row so the UI can
# badge the estimator node without a separate fetch.
row["tier_recommendation"] = {
"tier": data.get("tier"),
"is_confident": data.get("is_confident"),
}
except OSError:
return {}
return by_sid
def _decorate_with_archive(
rows: dict[str, dict[str, Any]],
sid_to_archive: dict[str, Path],
) -> None:
"""For each session_id with an archive on disk, fill in parent_session_id,
depth, started_at, model, status corrections. Mutates ``rows`` in place."""
for sid, row in rows.items():
archive_path = row.get("archive_path")
if archive_path:
p = Path(archive_path)
else:
p = sid_to_archive.get(sid)
if p is None or not p.is_file():
continue
try:
data = json.loads(p.read_text(encoding="utf-8"))
except (OSError, ValueError):
continue
if not isinstance(data, dict):
continue
row["parent_session_id"] = (
row["parent_session_id"] or data.get("parent_session_id")
)
if row["depth"] is None:
row["depth"] = data.get("subagent_depth") or 0
row["agent"] = row["agent"] or data.get("agent")
row["tag"] = row["tag"] or data.get("tag")
row["started_at"] = row["started_at"] or data.get("started_at")
row["ended_at"] = row["ended_at"] or data.get("archived_at")
row["archive_path"] = str(p)
# Pull a representative modelID from the first assistant message.
# User messages carry ``info.model`` as a {providerID, modelID}
# dict; assistant messages carry ``info.modelID`` as the plain
# string. We want the plain string, so prefer assistant rows and
# unwrap dicts from user rows as a fallback.
if not row["model"]:
for msg in (data.get("messages") or []):
info = (msg or {}).get("info") or {}
if info.get("role") != "assistant":
continue
m = info.get("modelID") or info.get("model")
if isinstance(m, dict):
m = m.get("modelID")
if m:
row["model"] = m
break
if not row["model"]:
# No assistant message yet — fall back to the user-side dict.
for msg in (data.get("messages") or [])[:3]:
info = (msg or {}).get("info") or {}
m = info.get("model")
if isinstance(m, dict):
m = m.get("modelID")
if m:
row["model"] = m
break
# Subagents don't emit ``worker.turn_finished`` (that event only
# fires for the top-level worker session), so events-derived
# turn_count is 0 for them. The archive's ``per_turn`` array has
# the real number — use it when events didn't see any turns.
if row["turn_count"] == 0:
per_turn = data.get("per_turn") or []
if isinstance(per_turn, list):
row["turn_count"] = len(per_turn)
# Archive-derived status: prefer "completed" once the archive
# exists and shows a clean terminated status.
archive_status = data.get("status")
if archive_status == "completed":
row["status"] = _bump_status(row["status"], "terminated")
elif archive_status == "subagent":
row["status"] = _bump_status(row["status"], "archived")
def _api_run_sessions(run_id: str | None) -> tuple[int, dict[str, Any]]:
"""Per-run session inventory: every session_id seen in the run's
events.jsonl, decorated with archive-derived parent/depth/model so the
UI can render the parent→child tree without per-row fetches.
The endpoint is deliberately a single scan + a directory glob; no
OpenCode round-trip. The per-session detail endpoint does the
live-OpenCode fall-through when a transcript is requested."""
run_dir, err = _resolve_run_dir(run_id)
if err is not None:
return 400 if "invalid" in err or "missing" in err else 404, {
"error": err
}
events_path = run_dir / "events.jsonl" # type: ignore[union-attr]
if not events_path.is_file():
return 200, {
"run_id": run_id,
"rows": [],
"note": f"events.jsonl not yet present: {events_path}",
}
rows_by_sid = _scan_run_sessions(events_path)
sid_to_archive = _archive_index_by_session_id()
_decorate_with_archive(rows_by_sid, sid_to_archive)
rows = list(rows_by_sid.values())
# Stable ordering: parents before children when both are present, then
# by started_at. The UI does its own tree-building from
# parent_session_id, so order is purely cosmetic for unparented rows.
rows.sort(key=lambda r: (r.get("depth") or 0, r.get("started_at") or ""))
return 200, {
"run_id": run_id,
"rows": rows,
"count": len(rows),
"archive_dir": str(_archive_dir_path()),
}
# ─── Per-session transcript (archive OR live OpenCode) ───────────────────
def _normalize_message_parts(parts: list[Any]) -> list[dict[str, Any]]:
"""Collapse OpenCode's verbose message-parts into a smaller, UI-shaped
shape. Drops housekeeping parts (step-start/step-finish), preserves the
text/reasoning/tool parts the UI actually renders."""
out: list[dict[str, Any]] = []
for part in parts or []:
if not isinstance(part, dict):
continue
ptype = part.get("type")
if ptype == "text":
text = part.get("text") or ""
if text.strip():
out.append({"kind": "text", "text": text})
elif ptype == "reasoning":
text = part.get("text") or ""
if text.strip():
out.append({"kind": "reasoning", "text": text})
elif ptype == "tool":
state = part.get("state") or {}
entry: dict[str, Any] = {
"kind": "tool",
"tool": part.get("tool"),
"status": state.get("status"),
"title": state.get("title"),
"input": state.get("input"),
}
output = state.get("output")
if output is not None:
# Truncate at the transport boundary to keep the response
# cheap; the archive file is always available for full
# inspection via /api/sessions/archived/detail.
if isinstance(output, str) and len(output) > 8000:
entry["output"] = output[:8000]
entry["output_truncated"] = True
else:
entry["output"] = output
out.append(entry)
# step-start / step-finish / other housekeeping parts are dropped.
return out
def _normalize_messages(messages: list[Any]) -> list[dict[str, Any]]:
"""Turn OpenCode's ``[{info, parts}]`` shape into ``[{role, model, ts,
parts}]`` for the UI. Skips empty messages (no useful parts)."""
out: list[dict[str, Any]] = []
for msg in messages or []:
if not isinstance(msg, dict):
continue
info = msg.get("info") or {}
parts = _normalize_message_parts(msg.get("parts") or [])
if not parts:
continue
time_info = info.get("time") or {}
ts = None
if isinstance(time_info, dict):
ts = time_info.get("created") or time_info.get("completed")
# Same unwrap as _decorate_with_archive: user messages store
# ``info.model`` as a dict, assistant messages store
# ``info.modelID`` as a string. UI wants a string.
model = info.get("modelID") or info.get("model")
if isinstance(model, dict):
model = model.get("modelID")
out.append(
{
"role": info.get("role"),
"model": model,
"agent": info.get("agent"),
"ts": ts,
"parts": parts,
}
)
return out
def _api_run_session(
run_id: str | None, session_id: str | None
) -> tuple[int, dict[str, Any]]:
"""Per-session transcript. Uses the archive when present, falls back
to a live OpenCode fetch when the session is still running.
The shape is the same in both cases so the UI doesn't need to branch:
``{session_id, source, status, agent, model, parent_session_id,
turns: [{role, model, ts, parts}], summary: {...}}``."""
run_dir, err = _resolve_run_dir(run_id)
if err is not None:
return 400 if "invalid" in err or "missing" in err else 404, {
"error": err
}
if not session_id or not session_id.startswith("ses_") or "/" in session_id:
return 400, {"error": "missing or invalid 'session' query parameter"}
sid_to_archive = _archive_index_by_session_id()
archive_path = sid_to_archive.get(session_id)
if archive_path is not None and archive_path.is_file():
try:
data = json.loads(archive_path.read_text(encoding="utf-8"))
except (OSError, ValueError) as e:
return 500, {"error": str(e), "archive_path": str(archive_path)}
if not isinstance(data, dict):
return 500, {"error": "archive is not a JSON object"}
return 200, {
"session_id": session_id,
"source": "archive",
"archive_path": str(archive_path),
"status": data.get("status"),
"agent": data.get("agent"),
"tag": data.get("tag"),
"parent_session_id": data.get("parent_session_id"),
"subagent_depth": data.get("subagent_depth"),
"started_at": data.get("started_at"),
"ended_at": data.get("archived_at"),
"wallclock_seconds": data.get("wallclock_seconds"),
"turns": _normalize_messages(data.get("messages") or []),
}
# Live fall-through: ask OpenCode directly. This is the only path that
# makes a network call, and it only fires when an operator clicks a
# currently-running session — so a slow OpenCode does not affect the
# tree endpoint.
info = _opencode_get(f"/session/{session_id}")
messages = _opencode_get(f"/session/{session_id}/message")
if info is None and messages is None:
return 503, {
"error": (
f"no archive on disk for {session_id} AND OpenCode at "
f"{OPENCODE_URL} is unreachable — transcript not available"
),
"session_id": session_id,
}
if not isinstance(messages, list):
messages = []
info = info if isinstance(info, dict) else {}
return 200, {
"session_id": session_id,
"source": "live-opencode",
"status": "running",
"agent": info.get("agent"),
"parent_session_id": info.get("parentID"),
"started_at": (info.get("time") or {}).get("created")
if isinstance(info.get("time"), dict) else None,
"ended_at": None,
"turns": _normalize_messages(messages),
}
def _api_cost(days: int) -> dict[str, Any]:
conn = _open_db()
prices = _load_prices()
if conn is None:
return {
"days": days, "rows": [], "totals": {"usd": 0.0},
"note": "cache absent",
"instrumentation_pending": True,
}
try:
cur = conn.execute(
"SELECT model, provider, COUNT(*) AS n,"
" SUM(COALESCE(tokens_in, 0)) AS in_tok,"
" SUM(COALESCE(tokens_out, 0)) AS out_tok,"
" SUM(COALESCE(cached_tokens, 0)) AS cached_tok"
" FROM llm_activity"
" WHERE started_at > datetime('now', ?)"
" GROUP BY model, provider ORDER BY n DESC",
(f"-{int(days)} days",),
)
rows = []
total_usd = 0.0
total_in = total_out = total_cached = 0
for r in cur:
in_tok = int(r["in_tok"] or 0)
out_tok = int(r["out_tok"] or 0)
cached = int(r["cached_tok"] or 0)
model = r["model"] or "_unknown"
provider = r["provider"]
usd = _cost_usd(
provider, model, prices, in_tok, out_tok, cached,
)
key = _price_key(provider, model)
rows.append(
{
"model": model, "provider": provider, "n": r["n"],
"tokens_in": in_tok, "tokens_out": out_tok,
"cached": cached, "usd": usd,
"priced": (key in prices or model in prices),
}
)
total_usd += usd
total_in += in_tok
total_out += out_tok
total_cached += cached
return {
"days": days,
"rows": rows,
"totals": {
"usd": round(total_usd, 2),
"tokens_in": total_in,
"tokens_out": total_out,
"cached": total_cached,
},
"instrumentation_pending": (not rows),
}
except sqlite3.OperationalError as e:
return {
"days": days, "rows": [], "totals": {"usd": 0.0},
"note": f"llm_activity table missing or stale schema: {e}",
"instrumentation_pending": True,
}
finally:
conn.close()
# ─── HTTP server ─────────────────────────────────────────────────────────
_API_DISPATCH = {
"/api/meta": lambda q: _api_meta(),
"/api/health": lambda q: _api_health(),
"/api/cycles": lambda q: _api_cycles(
q.get("driver", ["merge"])[0],
int(q.get("limit", ["20"])[0]),
),
"/api/prs": lambda q: _api_prs(q.get("label", [None])[0]),
"/api/velocity": lambda q: _api_velocity(
q.get("window", ["48h"])[0],
),
"/api/sessions": lambda q: _api_sessions(),
"/api/sessions/archived": lambda q: _api_archived_sessions(),
"/api/sessions/archived/detail": lambda q: _api_archived_session_detail(
q.get("filename", [None])[0],
),
"/api/cost": lambda q: _api_cost(int(q.get("days", ["7"])[0])),
}
# Endpoints that return (status_code, payload) rather than a bare payload,
# so they can signal 503/500 to the UI without raising. Kept separate from
# ``_API_DISPATCH`` so the simple-case handlers stay simple.
_API_DISPATCH_WITH_STATUS = {
"/runs": lambda q: _api_runs(),
"/snapshot": lambda q: _api_snapshot(q.get("run", [None])[0]),
"/events": lambda q: _api_events(
q.get("run", [None])[0],
int(q.get("since", ["0"])[0] or 0),
int(q.get("limit", ["500"])[0] or 500),
(
{t.strip() for t in q["types"][0].split(",") if t.strip()}
if q.get("types")
else None
),
),
"/api/run/sessions": lambda q: _api_run_sessions(
q.get("run", [None])[0],
),
"/api/run/session": lambda q: _api_run_session(
q.get("run", [None])[0],
q.get("session", [None])[0],
),
}
_STATIC_FILES = {
"/": ("index.html", "text/html; charset=utf-8"),
"/index.html": ("index.html", "text/html; charset=utf-8"),
"/static/app.js": ("app.js", "application/javascript; charset=utf-8"),
"/static/style.css": ("style.css", "text/css; charset=utf-8"),
}
class _Handler(BaseHTTPRequestHandler):
server_version = "TelemetryConsole/0.1"
def log_message(self, fmt: str, *args: Any) -> None:
# Silence the default per-request stderr noise; only log warnings.
return
def _send_json(self, status: int, payload: Any) -> None:
body = json.dumps(payload, default=str).encode("utf-8")
self.send_response(status)
self.send_header("Content-Type", "application/json; charset=utf-8")
self.send_header("Content-Length", str(len(body)))
self.send_header("Cache-Control", "no-store")
self.end_headers()
self.wfile.write(body)
def _send_static(self, fname: str, content_type: str) -> None:
path = STATIC_DIR / fname
if not path.is_file():
self.send_error(404, f"missing static file: {fname}")
return
body = path.read_bytes()
self.send_response(200)
self.send_header("Content-Type", content_type)
self.send_header("Content-Length", str(len(body)))
self.send_header("Cache-Control", "no-store")
self.end_headers()
self.wfile.write(body)
def do_GET(self) -> None: # noqa: N802 (BaseHTTPRequestHandler API)
parsed = urllib.parse.urlparse(self.path)
route = parsed.path
query = urllib.parse.parse_qs(parsed.query)
if route in _STATIC_FILES:
fname, ctype = _STATIC_FILES[route]
self._send_static(fname, ctype)
return
handler = _API_DISPATCH.get(route)
if handler is not None:
try:
payload = handler(query)
self._send_json(200, payload)
except Exception as e: # noqa: BLE001 — keep server up
logger.exception("handler %s failed", route)
self._send_json(500, {"error": str(e), "route": route})
return
handler_with_status = _API_DISPATCH_WITH_STATUS.get(route)
if handler_with_status is not None:
try:
status, payload = handler_with_status(query)
self._send_json(status, payload)
except Exception as e: # noqa: BLE001
logger.exception("handler %s failed", route)
self._send_json(500, {"error": str(e), "route": route})
return
self.send_error(404, f"unknown route: {route}")
def _install_signal_handlers(httpd: ThreadingHTTPServer) -> None:
def _shutdown(signum: int, _frame: Any) -> None:
logger.info("received signal %d, shutting down", signum)
# shutdown() must not run in the request thread; spawn a thread
# via a separate signal-safe path.
import threading
threading.Thread(target=httpd.shutdown, daemon=True).start()
for sig in (signal.SIGINT, signal.SIGTERM):
try:
signal.signal(sig, _shutdown)
except (OSError, ValueError):
pass
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
description="auto-agents pipeline telemetry console",
)
parser.add_argument(
"--host",
default=os.environ.get("TELEMETRY_HOST", "127.0.0.1"),
help=(
"interface to bind (default 127.0.0.1 — loopback only). "
"Set 0.0.0.0 ONLY when you understand v1 ships without auth."
),
)
parser.add_argument(
"--port", type=int,
default=int(os.environ.get("TELEMETRY_PORT", "8765")),
help="TCP port to bind (default 8765)",
)
parser.add_argument(
"--log-level", default="INFO",
choices=("DEBUG", "INFO", "WARNING", "ERROR"),
)
args = parser.parse_args(argv)
logging.basicConfig(
level=args.log_level,
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
)
logger.info("repo target: %s/%s", REPO_OWNER, REPO_NAME)
logger.info("api base: %s", API_BASE)
logger.info("cache file: %s (%s)",
CACHE_PATH, "exists" if CACHE_PATH.exists() else "ABSENT")
logger.info("opencode url: %s", OPENCODE_URL)
logger.info("forgejo token: %s",
"present" if FORGEJO_TOKEN else "MISSING (PR tab disabled)")
try:
httpd = ThreadingHTTPServer((args.host, args.port), _Handler)
except OSError as e:
print(
f"ERROR: bind {args.host}:{args.port} failed: {e}",
file=sys.stderr,
)
return 2
_install_signal_handlers(httpd)
print(
f"Telemetry console running at http://{args.host}:{args.port}/ "
f"(Ctrl+C to stop)"
)
try:
httpd.serve_forever()
finally:
httpd.server_close()
return 0
if __name__ == "__main__":
raise SystemExit(main())