cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity
Files
a21466add232d59cdec1604e09d58ca05659a623
cleveragents-core/.opencode/telemetry/server.py
T
drew eb01eb0172 feat(controller): dual-mode launcher (fork/prod) + DB-mode validator 
Adds the operator surface for switching the controller pipeline between
the personal fork (drew/cleveragents-core) and the canonical repo
(cleveragents/cleveragents-core) via a MODE env + --prod CLI flag,
backed by safety primitives that make a wrong-mode launch loud rather
than silent.

run-controller-state-machine-pipeline.sh: --prod flag and MODE env
(primary home: .devcontainer/.env) select fork vs prod. After resolving
MODE, the launcher auto-sources the matching overlay file
(.devcontainer/.env.{fork,prod}) and asserts MODE didn't drift during
the source step. The drift assertion uses a readonly snapshot under an
obscure variable name so a stray ``MODE=fork`` in .env.prod aborts the
launch with a clear bash error rather than silently demoting the run.
CONTROLLER_RUN_DIR_ROOT now overrides the trial /tmp path so prod can
use a persistent /var/lib/cleveragents/run dir.

tools/launch_prod.sh (new): sibling to launch_fork.sh with the opposite
safety primitive — affirmative GET /repos/{owner}/{repo} that asserts
the target is non-fork, exists, isn't archived, and the bot has push.
On any failure, no env is exported. Honors HAL_* aliases for parity
with launch_fork.sh and prints a hard-to-miss PROD-MODE banner.

tools/controller/deploy/validate_db_mode.py (new): stamps a _mode_marker
table on each SQLite db (controller DB + telemetry cache) on first use,
asserts a match on every subsequent launch, and moves mismatched files
aside as <name>.<prior-mode>.bak.<ts> — never deletes. The --adopt flag
lets an operator grandfather in already-good pre-marker data without
losing history. Wired into the launcher's startup sequence before
OpenCode and the master start.

tools/_cache_path.py (new): single source of truth for the per-(owner,
repo) Forgejo cache file convention. .opencode/telemetry/server.py and
the launcher both delegate here so the dual-source-truth drift risk is
eliminated. tools/_pipeline_cache.py and tools/controller/db/models.py
documented as not owning the _mode_marker table so future migrations
leave it alone.

.opencode/telemetry/server.py: hosts the llm_activity scraper as a
background subprocess thread (60s cadence, --since-hours 1 in steady
state, full backfill on first tick). Re-homes the cost-telemetry data
path after the pr_state_warmer was retired by the controller migration
— without this the Cost tab freezes when the warmer's loop is gone.
Subprocess (not in-process) for isolation; failures swallowed.

opencode.json: local-claude provider's baseURL now reads
{env:LOCAL_PROXY_URL} instead of the literal http://127.0.0.1:3456/v1,
matching the apiKey pattern already in use.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 08:25:24 -04:00

3318 lines
133 KiB
Python
Raw Blame History

#!/usr/bin/env python3
"""Pipeline telemetry console — a small, dependency-free HTTP server that
exposes everything you need to monitor the auto-agents pipeline in one
browser tab.
Why this exists
---------------
The auto-agents pipeline emits useful telemetry into several silos:
- ``tools/.cache/forgejo.<owner>.<repo>.sqlite`` — cached PR/commit data
+ ``merge_cycle`` / ``conflict_drive_cycles`` / ``llm_activity`` driver
telemetry rows.
- ``tools/<driver>.heartbeat`` (or ``/tmp/<driver>.heartbeat``) — per-driver
liveness probes touched once per cycle.
- ``ps``-discoverable Python daemons (``merge_drive``, ``conflict_drive``,
``verify_invariant``, ``opencode-builder.sh``).
- The OpenCode HTTP server (``http://127.0.0.1:4096`` by default) — agent
sessions, agent messages, idle/busy state.
- The Forgejo HTTP API — open PRs by ``auto/*`` label, branch protection,
approvals.
Tailing five terminals + running ``sqlite3`` ad-hoc + curl-ing two HTTP
APIs is fine for one operator; it does not scale to "let another dev
look at this without setting anything up". This console consolidates
all of the above into a single ``python3 .opencode/telemetry/server.py``
entry point with a tabbed web UI that auto-refreshes per pane.
Design constraints
------------------
- **Stdlib only.** No Flask / FastAPI / Jinja. Runs anywhere a driver
runs (the auto-agents pipeline already mandates Python 3.13+ stdlib;
matching that contract keeps the install footprint zero).
- **Repo-aware.** Reads ``FORGEJO_OWNER`` / ``FORGEJO_REPO`` and picks
the right cache file via the same partition logic as
``tools/_pipeline_cache.py``. The console banner spells out which
repo's data is on screen — never let an operator monitor canonical
while expecting fork data, or vice-versa.
- **Loopback-only by default.** Binds ``127.0.0.1``. Operators who
want LAN access set ``TELEMETRY_HOST=0.0.0.0`` deliberately AND
understand that v1 ships without auth (loopback is the security
model). README spells this out.
- **Per-tab polling.** Each tab fetches its own endpoints on its own
cadence; inactive tabs do not poll. Server load stays low even on
long-lived browser sessions.
Endpoints (all JSON unless noted)
---------------------------------
- ``GET /`` index.html (tabbed SPA)
- ``GET /static/<file>`` app.js / style.css
- ``GET /api/meta`` repo target, identity, paths in use
- ``GET /api/health`` daemon PIDs, heartbeats, OpenCode health
- ``GET /api/cycles?driver=...`` merge / conflict / dispatch_review /
dispatch_implementer cycle rows
- ``GET /api/prs?label=...`` open PRs by Forgejo label (live API)
- ``GET /api/velocity?window=...`` window stats from cache
- ``GET /api/sessions`` OpenCode sessions
- ``GET /api/cost?days=N`` token + estimated cost rollup
Run
---
::
source tools/launch_fork.sh # or canonical-mode env
python3 .opencode/telemetry/server.py # http://127.0.0.1:8765
Environment overrides
---------------------
- ``TELEMETRY_HOST`` (default ``127.0.0.1``)
- ``TELEMETRY_PORT`` (default ``8765``)
- ``OPENCODE_URL`` (default ``http://127.0.0.1:4096``)
- ``TELEMETRY_PRICES_JSON`` (default ``.opencode/telemetry/prices.json``;
used by /api/cost to convert tokens to dollars)
Exit codes
----------
::
0 — clean shutdown via Ctrl+C
2 — argument error / port-bind failure / missing cache file
"""
from __future__ import annotations
import argparse
import json
import logging
import os
import re
import signal
import sqlite3
import subprocess
import sys
import threading
import time
import urllib.error
import urllib.parse
import urllib.request
from datetime import datetime, timedelta, timezone
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from typing import Any
# Optional: the controller's per-attempt journal lives in a SQLAlchemy
# DB selected by ``CLEVERAGENTS_DB_URL`` (SQLite locally, Postgres in
# prod). The phases endpoint reads from it when the env var is set;
# otherwise the panel shows the same "instrumentation pending" callout
# the cost tab uses. SQLAlchemy is an optional import so the telemetry
# server still starts in stripped-down envs.
try:
from sqlalchemy import bindparam as _sa_bindparam
from sqlalchemy import create_engine as _sa_create_engine
from sqlalchemy import text as _sa_text
from sqlalchemy.engine import Engine as _SAEngine
_SQLALCHEMY_AVAILABLE = True
except ImportError: # pragma: no cover — only hit in minimal envs
_SQLALCHEMY_AVAILABLE = False
_SAEngine = None # type: ignore[assignment,misc]
logger = logging.getLogger("telemetry")
# ─── Paths + repo-aware cache resolution ─────────────────────────────────
REPO_ROOT = Path(__file__).resolve().parents[2]
STATIC_DIR = Path(__file__).resolve().parent
CACHE_DIR = REPO_ROOT / "tools" / ".cache"
# Mirrors the partition logic in tools/_pipeline_cache.py so the console
# always reads the SAME file the drivers write to. Computed at import.
REPO_OWNER = os.environ.get("FORGEJO_OWNER", "cleveragents")
REPO_NAME = os.environ.get("FORGEJO_REPO", "cleveragents-core")
API_BASE = os.environ.get(
"FORGEJO_API_BASE", "https://git.cleverthis.com/api/v1"
).rstrip("/")
FORGEJO_TOKEN = os.environ.get("GITEA_TOKEN") or os.environ.get("FORGEJO_PAT", "")
OPENCODE_URL = os.environ.get("OPENCODE_URL", "http://127.0.0.1:4096").rstrip("/")
def _cache_path() -> Path:
"""Resolve the per-(owner, repo) cache file. Delegates to the shared
helper at ``tools/_cache_path.py`` so server.py, _pipeline_cache.py,
and the controller launcher all agree on the path."""
import importlib.util
helper = REPO_ROOT / "tools" / "_cache_path.py"
spec = importlib.util.spec_from_file_location("_cache_path", helper)
assert spec and spec.loader
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
return mod.cache_path_for(REPO_ROOT, REPO_OWNER, REPO_NAME)
CACHE_PATH = _cache_path()
# ─── SQLite helpers ──────────────────────────────────────────────────────
def _open_db() -> sqlite3.Connection | None:
"""Open the per-repo cache read-only. Returns None when the file
does not exist yet (cold-start before any sync) so endpoints can
return empty payloads instead of 500ing."""
if not CACHE_PATH.exists():
return None
# mode=ro keeps us from racing with driver writers; the SQLite
# locking story tolerates many readers + one writer regardless,
# but ro makes the intent unambiguous.
uri = f"file:{CACHE_PATH}?mode=ro"
conn = sqlite3.connect(uri, uri=True, timeout=2)
conn.row_factory = sqlite3.Row
return conn
def _rows_to_dicts(rows: list[sqlite3.Row]) -> list[dict[str, Any]]:
return [dict(r) for r in rows]
def _iso(value: Any) -> str | None:
"""Coerce a DB timestamp to an ISO string. Raw SQLAlchemy text()
queries hand back Python datetimes on Postgres but plain strings on
SQLite (no column type info) — normalise both, pass None through."""
if value is None:
return None
return value.isoformat() if hasattr(value, "isoformat") else str(value)
def _controller_db_note(exc: Exception) -> str:
"""Friendly note for a controller-DB query failure. A DB that exists
but has no schema yet (the controller master runs create_all on
boot) is the common pre-first-run state — distinguish it from a real
query error so the operator isn't shown a raw SQL dump."""
msg = str(exc)
if any(s in msg for s in ("no such table", "does not exist", "UndefinedTable")):
return (
"controller DB reachable but not initialized yet — "
"start the controller (its master creates the schema on boot)"
)
return f"controller DB query failed: {exc}"
# ─── Forgejo API (read-only) ─────────────────────────────────────────────
def _forgejo_get(path: str, params: dict[str, str] | None = None) -> Any:
"""GET ``API_BASE + path``. Returns parsed JSON or None on error.
Console never POSTs — read-only by design."""
if not FORGEJO_TOKEN:
return None
url = API_BASE + path
if params:
url += "?" + urllib.parse.urlencode(params)
req = urllib.request.Request(
url,
headers={
"Authorization": f"token {FORGEJO_TOKEN}",
"Accept": "application/json",
},
)
try:
with urllib.request.urlopen(req, timeout=10) as resp:
return json.loads(resp.read())
except (urllib.error.HTTPError, urllib.error.URLError, OSError) as e:
logger.warning("forgejo GET %s failed: %r", path, e)
return None
# ─── OpenCode API (read-only) ────────────────────────────────────────────
def _opencode_get(path: str) -> Any:
"""GET against the local OpenCode server. Returns parsed JSON or
None when the server is unreachable. The console renders an
'OpenCode unreachable' badge in that case rather than failing
the whole page."""
url = OPENCODE_URL + path
try:
with urllib.request.urlopen(url, timeout=3) as resp:
return json.loads(resp.read())
except (urllib.error.HTTPError, urllib.error.URLError, OSError) as e:
logger.debug("opencode GET %s failed: %r", path, e)
return None
# ─── Daemon discovery (ps + heartbeat) ───────────────────────────────────
# Heartbeat probe order matches ``_dispatch_runtime.resolve_lock_or_heartbeat``
# / ``merge_drive.load_config`` / ``conflict_drive.load_config``: explicit env
# override → ``/var/run`` → ``$XDG_RUNTIME_DIR`` (typically
# ``/run/user/<uid>``) → ``/tmp``. Production hosts and dev containers usually
# land on ``XDG_RUNTIME_DIR``; the explicit env vars are the operator's escape
# hatch.
def _heartbeat_candidates(basename: str, env_var: str | None = None) -> list[str]:
paths: list[str] = []
if env_var:
explicit = os.environ.get(env_var)
if explicit:
paths.append(explicit)
paths.append(f"/var/run/{basename}")
runtime_dir = os.environ.get("XDG_RUNTIME_DIR")
if runtime_dir:
paths.append(f"{runtime_dir.rstrip('/')}/{basename}")
paths.append(f"/tmp/{basename}")
return paths
_DAEMON_SPECS: list[dict[str, Any]] = [
{
# Controller master loop: scans workflows, drives state machine,
# writes controller_events. Invoked as `python -m tools.controller.master`;
# the resolved cmdline can show either the dotted module form or the
# __main__.py path, so match both.
"name": "controller_master",
"match": ["tools.controller.master", "controller/master/__main__"],
"heartbeat_candidates": _heartbeat_candidates(
"controller-master.heartbeat", "CONTROLLER_MASTER_HEARTBEAT_PATH"
),
},
{
# Controller worker: dequeues workflow_attempts by role and runs
# implementer / reviewer / conflict_resolver / estimator / summarizer.
"name": "controller_worker",
"match": ["tools.controller.worker", "controller/worker/__main__"],
"heartbeat_candidates": _heartbeat_candidates(
"controller-worker.heartbeat", "CONTROLLER_WORKER_HEARTBEAT_PATH"
),
},
{
# Merge singleton: picks up APPROVED workflows from the controller
# bridge and POSTs Forgejo merges. Still active alongside the
# controller (see tools/_controller_db_bridge.py).
"name": "merge_drive",
"match": ["merge_drive.py"],
"heartbeat_candidates": _heartbeat_candidates(
"merge-driver.heartbeat", "MERGE_DRIVER_HEARTBEAT_PATH"
),
},
{
"name": "opencode_builder",
"match": ["opencode-builder.sh", "opencode serve"],
"heartbeat_candidates": [],
},
]
def _ps_lookup(needles: list[str]) -> list[dict[str, Any]]:
"""Return [{pid, cmdline, cpu, mem, etime}] for processes whose
cmdline contains any of ``needles``. Cross-platform-ish via /proc;
falls back to ``ps -eo`` if /proc isn't available."""
found: list[dict[str, Any]] = []
proc_dir = Path("/proc")
if proc_dir.exists():
for entry in proc_dir.iterdir():
if not entry.name.isdigit():
continue
try:
cmdline_raw = (entry / "cmdline").read_bytes()
except (FileNotFoundError, PermissionError):
continue
cmd = cmdline_raw.replace(b"\x00", b" ").decode("utf-8", "replace").strip()
if not cmd:
continue
if any(n in cmd for n in needles):
pid = int(entry.name)
stat = _proc_stat_brief(pid)
found.append({"pid": pid, "cmdline": cmd, **stat})
return found
# ps -eo fallback (BSD-ish / macOS / non-/proc Linux)
try:
out = subprocess.check_output(
["ps", "-eo", "pid=,etime=,pcpu=,pmem=,args="],
text=True,
timeout=5,
)
except (subprocess.SubprocessError, FileNotFoundError):
return []
for line in out.splitlines():
line = line.strip()
if not line:
continue
parts = line.split(None, 4)
if len(parts) < 5:
continue
pid, etime, cpu, mem, cmd = parts
if any(n in cmd for n in needles):
try:
pid_int = int(pid)
except ValueError:
continue
found.append(
{
"pid": pid_int,
"cmdline": cmd,
"etime": etime,
"cpu_pct": float(cpu),
"mem_pct": float(mem),
}
)
return found
def _proc_stat_brief(pid: int) -> dict[str, Any]:
"""Lightweight /proc-based stat. Returns etime/cpu/mem if available,
empty dict otherwise (process gone between iterdir and read)."""
try:
with open(f"/proc/{pid}/stat", encoding="utf-8") as f:
stat_line = f.read()
except (FileNotFoundError, PermissionError):
return {}
# comm field can contain spaces; use the rightmost ')' as boundary.
rb = stat_line.rfind(")")
if rb == -1:
return {}
rest = stat_line[rb + 2 :].split()
# Field positions per `man proc`: starttime is field 22 (0-indexed
# 21 in the post-comm slice).
try:
starttime_clk = int(rest[19])
except (IndexError, ValueError):
return {}
try:
clk_tck = os.sysconf("SC_CLK_TCK") or 100
except (ValueError, OSError):
clk_tck = 100
try:
with open("/proc/uptime", encoding="utf-8") as f:
uptime_s = float(f.read().split()[0])
except (FileNotFoundError, IndexError, ValueError):
return {}
elapsed_s = uptime_s - (starttime_clk / clk_tck)
return {"elapsed_s": round(elapsed_s, 1)}
def _heartbeat_age(candidates: list[str]) -> dict[str, Any] | None:
for path in candidates:
p = Path(path)
if p.exists():
mtime = p.stat().st_mtime
return {
"path": str(p),
"mtime_s_ago": round(time.time() - mtime, 1),
}
return None
# ─── Pricing (token → $) ─────────────────────────────────────────────────
# Defaults seeded from publicly-listed prices; override via
# .opencode/telemetry/prices.json. Costs are $/1,000,000 tokens.
_DEFAULT_PRICES: dict[str, dict[str, float]] = {
# Anthropic (public — May 2026 list prices). Cache reads are
# 10% of standard input across the family.
"local-claude/claude-opus-4-6": {"in": 5.0, "out": 25.0, "cached_in": 0.50},
"local-claude/claude-sonnet-4-6": {"in": 3.0, "out": 15.0, "cached_in": 0.30},
"local-claude/claude-haiku-4-5": {"in": 1.0, "out": 5.0, "cached_in": 0.10},
# OpenAI (public — May 2026 list prices). gpt-5 / gpt-5-codex
# share the same per-token rate; gpt-5-mini / gpt-5-nano are
# the budget tier.
"openai/gpt-5": {"in": 1.25, "out": 10.0, "cached_in": 0.125},
"openai/gpt-5-codex": {"in": 1.25, "out": 10.0, "cached_in": 0.125},
"openai/gpt-5-mini": {"in": 0.25, "out": 2.0, "cached_in": 0.025},
"openai/gpt-5-nano": {"in": 0.05, "out": 0.4, "cached_in": 0.005},
# CleverThis HF Inference Endpoints — billed per endpoint-hour,
# not per token. The per-token entries here are a best-effort
# imputation for "what would this have cost on a public API";
# treat as advisory not source-of-truth.
"CleverThis-15/Qwen3-6-35B-A3B-GGUF-UD-Q3-K-XL": {"in": 0.5, "out": 1.0},
"CleverThis/Qwen3-6-35B-A3B-GGUF-BF16": {"in": 0.5, "out": 1.0},
"CleverThis-2/MiniMax-M2-7-GGUF-BF16": {"in": 0.5, "out": 1.0},
"CleverThis-4/Kimi-K2-6-GGUF-Q2-K-XL": {"in": 0.5, "out": 1.0},
# Sentinel — when we don't know, we'd rather produce 0 than a wild
# number. The UI surfaces "model unpriced" rows separately.
"_unknown": {"in": 0.0, "out": 0.0},
}
def _load_prices() -> dict[str, dict[str, float]]:
override_path = Path(
os.environ.get(
"TELEMETRY_PRICES_JSON",
str(STATIC_DIR / "prices.json"),
)
)
if override_path.exists():
try:
data = json.loads(override_path.read_text())
if isinstance(data, dict):
merged = dict(_DEFAULT_PRICES)
# Skip metadata keys (``_comment``, ``_last_updated``,
# ``_sources``) so they don't pollute the price table.
# The ``_unknown`` sentinel is preserved because it IS
# a valid price-table entry the lookup falls back to.
for k, v in data.items():
if k.startswith("_") and k != "_unknown":
continue
if isinstance(v, dict):
merged[k] = v
return merged
except (OSError, json.JSONDecodeError) as e:
logger.warning("ignoring prices.json: %r", e)
return dict(_DEFAULT_PRICES)
def _price_key(provider: str | None, model: str) -> str:
"""Resolve the lookup key for the price table.
The price table is keyed by ``provider/model`` because the same
``modelID`` can be served by multiple providers at very different
economics (e.g. ``claude-opus-4-6`` via Anthropic direct vs via a
local proxy). Falls back to the bare model name for backward
compat with manually-inserted rows that never had a provider.
"""
if provider:
return f"{provider}/{model}"
return model
def _cost_usd(
provider: str | None,
model: str,
prices: dict[str, dict[str, float]],
tokens_in: int,
tokens_out: int,
cached: int,
) -> float:
"""Compute USD cost for a per-model token tally.
Lookup order: ``{provider}/{model}`` → bare ``model`` → ``_unknown``.
The fallback chain lets a price entry written without provider
namespacing still match, while still preferring the
provider-qualified entry when present.
Token-accounting convention: ``tokens_in`` and ``cached`` are
DISJOINT populations — the scraper writes ``tokens_in`` as
OpenCode's ``info.tokens.input`` (fresh, non-cached prompt
tokens) plus any cache-write tokens; ``cached`` is
``info.tokens.cache.read``. So total cost is the simple sum of
three line items at three rates, with no subtraction needed.
"""
p = (
prices.get(_price_key(provider, model))
or prices.get(model)
or prices.get("_unknown", {})
)
in_cost = tokens_in * p.get("in", 0.0) / 1_000_000
cached_cost = cached * p.get("cached_in", p.get("in", 0.0)) / 1_000_000
out_cost = tokens_out * p.get("out", 0.0) / 1_000_000
return round(in_cost + cached_cost + out_cost, 4)
# ─── Endpoint handlers ───────────────────────────────────────────────────
def _api_meta() -> dict[str, Any]:
return {
"repo_owner": REPO_OWNER,
"repo_name": REPO_NAME,
"api_base": API_BASE,
"cache_path": str(CACHE_PATH),
"cache_present": CACHE_PATH.exists(),
"opencode_url": OPENCODE_URL,
"has_forgejo_token": bool(FORGEJO_TOKEN),
"server_time_utc": datetime.now(timezone.utc).isoformat(),
"console_version": "0.1.0",
"warn_canonical_target": (
REPO_OWNER == "cleveragents" and REPO_NAME == "cleveragents-core"
),
}
# Heartbeat staleness threshold (seconds) for the "running long worker"
# hint. Set well above the dispatcher's ``poll_interval_seconds`` (2s)
# so a healthy heartbeat refresh during run_session_blocking will keep
# the daemon out of this state. Triggered when the heartbeat is older
# than this AND a matching pid is alive — the combination indicates an
# in-flight worker session whose on_poll callback isn't writing the
# heartbeat (e.g. the ``on_poll`` wiring regressed, EROFS, or the
# dispatcher has wedged on a non-OpenCode codepath).
_LONG_WORKER_HEARTBEAT_THRESHOLD_S = 600.0
def _api_health() -> dict[str, Any]:
daemons: list[dict[str, Any]] = []
for spec in _DAEMON_SPECS:
procs = _ps_lookup(spec["match"])
hb = _heartbeat_age(spec["heartbeat_candidates"])
running = bool(procs)
# "Running long worker" hint: the daemon is alive but its
# heartbeat is stale. Under healthy operation every poll writes
# the heartbeat; when this fires it points at a callback
# regression or a filesystem failure on the heartbeat path. The
# threshold is large enough that it does not flap on normal
# cycle boundaries.
running_long_worker = False
if running and hb is not None:
running_long_worker = (
hb.get("mtime_s_ago", 0.0) >= _LONG_WORKER_HEARTBEAT_THRESHOLD_S
)
daemons.append(
{
"name": spec["name"],
"running": running,
"processes": procs,
"heartbeat": hb,
"running_long_worker": running_long_worker,
}
)
oc_health = _opencode_get("/global/health")
return {
"daemons": daemons,
"opencode": {
"url": OPENCODE_URL,
"reachable": oc_health is not None,
"details": oc_health,
},
}
def _api_cycles(driver: str, limit: int) -> dict[str, Any]:
# Only ``merge`` remains — the dispatcher + conflict drivers were
# subsumed by the controller state machine. Reviewer/implementer
# attempt history lives in workflow_attempts; query via /api/workflows.
if driver != "merge":
return {
"rows": [],
"note": f"driver {driver!r} retired; see /api/workflows for controller-era attempts",
}
conn = _open_db()
if conn is None:
return {"rows": [], "note": f"cache file not present: {CACHE_PATH}"}
try:
cur = conn.execute(
"SELECT id, started_at, ended_at, pr_numbers, train_id,"
" bisect_depth, total_seconds, terminal_state,"
" action_taken, action_detail"
" FROM merge_cycle ORDER BY started_at DESC LIMIT ?",
(limit,),
)
rows = _rows_to_dicts(list(cur.fetchall()))
counts = conn.execute(
"SELECT terminal_state AS k, COUNT(*) AS n FROM merge_cycle"
" WHERE started_at > datetime('now','-24 hours')"
" GROUP BY terminal_state ORDER BY n DESC"
).fetchall()
return {
"rows": rows,
"outcome_breakdown_24h": _rows_to_dicts(list(counts)),
}
finally:
conn.close()
def _api_prs(state: str | None) -> dict[str, Any]:
"""Live Forgejo PR list joined with controller state. ``state`` filters
to PRs whose controller-DB ``current_state`` matches (e.g. STUCK,
REVIEWING, MERGING). Unattached PRs (no controller workflow row) get
``current_state = None`` and are shown unless a state filter is set."""
body = _forgejo_get(
f"/repos/{REPO_OWNER}/{REPO_NAME}/issues",
{"type": "pulls", "state": "open", "limit": "50"},
)
if not isinstance(body, list):
return {"rows": [], "states": [], "note": "forgejo unreachable or no token"}
pr_to_state: dict[int, dict[str, Any]] = {}
states_summary: list[dict[str, Any]] = []
engine = _open_controller_db_engine()
if engine is not None:
try:
with engine.connect() as conn:
wf_rows = conn.execute(
_sa_text(
"SELECT entity_number, current_state, "
" last_transition_at, workflow_id "
" FROM workflows "
" WHERE owner = :owner AND repo = :repo "
" AND kind = 'pr'"
),
{"owner": REPO_OWNER, "repo": REPO_NAME},
).all()
for r in wf_rows:
if r.entity_number is None:
continue
pr_to_state[int(r.entity_number)] = {
"current_state": r.current_state,
"last_transition_at": _iso(r.last_transition_at),
"workflow_id": int(r.workflow_id),
}
state_counts = conn.execute(
_sa_text(
"SELECT current_state, COUNT(*) AS n "
" FROM workflows "
" WHERE owner = :owner AND repo = :repo "
" AND kind = 'pr' "
" GROUP BY current_state ORDER BY n DESC"
),
{"owner": REPO_OWNER, "repo": REPO_NAME},
).all()
states_summary = [
{"state": r.current_state, "n": int(r.n)} for r in state_counts
]
except Exception as exc: # noqa: BLE001
return {"rows": [], "states": [], "note": _controller_db_note(exc)}
rows: list[dict[str, Any]] = []
for issue in body:
if not isinstance(issue, dict):
continue
number = issue.get("number")
controller = pr_to_state.get(int(number)) if number is not None else None
cur_state = controller["current_state"] if controller else None
if state and cur_state != state:
continue
rows.append(
{
"number": number,
"title": issue.get("title"),
"html_url": issue.get("html_url"),
"user": (issue.get("user") or {}).get("login"),
"created_at": issue.get("created_at"),
"updated_at": issue.get("updated_at"),
"current_state": cur_state,
"last_transition_at": (
controller["last_transition_at"] if controller else None
),
"workflow_id": controller["workflow_id"] if controller else None,
}
)
return {"rows": rows, "states": states_summary, "filter": state}
def _api_velocity(window: str) -> dict[str, Any]:
conn = _open_db()
if conn is None:
return {"window": window, "stats": {}, "note": "cache absent"}
try:
# Mirror the windows tools/render-pr-velocity.py uses; convert
# the human label to a SQLite datetime modifier.
win_map = {
"24h": "-24 hours",
"48h": "-48 hours",
"7d": "-7 days",
"30d": "-30 days",
}
if window not in win_map:
return {
"window": window,
"stats": {},
"note": "unknown window",
}
modifier = win_map[window]
try:
commits = conn.execute(
"SELECT COUNT(*) FROM commits WHERE author_date > datetime('now', ?)",
(modifier,),
).fetchone()[0]
except sqlite3.OperationalError:
# Schema variant — fall back to whatever the cache has.
commits = 0
try:
opened = conn.execute(
"SELECT COUNT(*) FROM pulls WHERE created_at > datetime('now', ?)",
(modifier,),
).fetchone()[0]
except sqlite3.OperationalError:
opened = 0
try:
merged = conn.execute(
"SELECT COUNT(*) FROM pulls"
" WHERE merged = 1 AND merged_at > datetime('now', ?)",
(modifier,),
).fetchone()[0]
except sqlite3.OperationalError:
merged = 0
return {
"window": window,
"stats": {
"commits": commits,
"opened": opened,
"merged": merged,
},
}
finally:
conn.close()
def _api_sessions() -> dict[str, Any]:
sessions = _opencode_get("/session")
if not isinstance(sessions, list):
return {"rows": [], "note": "opencode unreachable"}
rows: list[dict[str, Any]] = []
now_ms = int(time.time() * 1000)
for s in sessions:
if not isinstance(s, dict):
continue
sid = s.get("id")
msgs = _opencode_get(f"/session/{sid}/message") if sid else None
msg_count = len(msgs) if isinstance(msgs, list) else 0
time_obj = s.get("time") or {}
created = time_obj.get("created")
updated = time_obj.get("updated")
rows.append(
{
"id": sid,
"title": s.get("title"),
"directory": s.get("directory"),
"created_s_ago": round((now_ms - created) / 1000, 1)
if created
else None,
"updated_s_ago": round((now_ms - updated) / 1000, 1)
if updated
else None,
"message_count": msg_count,
}
)
rows.sort(key=lambda r: r.get("updated_s_ago") or 1e18)
return {"rows": rows}
# ─── Archived sessions (post-mortem of past worker runs) ─────────────────
#
# These endpoints expose the on-disk archives written by
# ``tools/_opencode_worker.py:run_session_blocking`` after every
# terminal return (completed / timeout / transport-error). The archives
# survive session deletion in OpenCode, so an operator can reconstruct
# what a worker actually said even after the dispatcher cycle is over.
#
# The path resolution mirrors ``_opencode_worker._resolve_archive_dir``:
# explicit ``OPENCODE_WORKER_ARCHIVE_DIR`` env var wins over the repo
# default ``<repo_root>/.dispatcher-logs/sessions``. We DO NOT honour
# ``OPENCODE_WORKER_ARCHIVE_DISABLED`` here because that controls the
# writer side, not the read-only console.
_DEFAULT_ARCHIVE_DIR = REPO_ROOT / ".dispatcher-logs" / "sessions"
# Maximum number of archive entries to return from the listing endpoint.
# Keeps the page snappy even on long-running pipelines that have logged
# hundreds of sessions.
_ARCHIVE_LIST_LIMIT = 200
# Filename guard for the detail endpoint. Mirrors the writer's safe
# filename component regex (``[A-Za-z0-9_.-]``); the only legitimate
# extra character is ``__`` from the writer's section separator. Any
# filename that does not match this is treated as a path-traversal
# attempt and 404'd without touching the filesystem.
_SAFE_ARCHIVE_FILENAME_RE = re.compile(r"^[A-Za-z0-9_.-]+\.json$")
def _archive_dir_path() -> Path:
explicit = os.environ.get("OPENCODE_WORKER_ARCHIVE_DIR")
if explicit:
return Path(explicit)
return _DEFAULT_ARCHIVE_DIR
def _api_archived_sessions() -> dict[str, Any]:
"""Index of session archives.
Returns lightweight metadata only: filename, agent, tag, status,
started_at, archived_at, message_count, size_bytes. The detail
endpoint serves the full payload for a clicked-on row.
"""
archive_dir = _archive_dir_path()
if not archive_dir.is_dir():
return {
"archive_dir": str(archive_dir),
"rows": [],
"note": "archive directory does not exist yet",
}
entries: list[dict[str, Any]] = []
for path in archive_dir.glob("*.json"):
try:
stat = path.stat()
with path.open("r", encoding="utf-8") as f:
data = json.load(f)
except (OSError, ValueError) as e:
entries.append(
{
"filename": path.name,
"size_bytes": None,
"error": str(e),
}
)
continue
if not isinstance(data, dict):
continue
per_turn = data.get("per_turn") or []
entries.append(
{
"filename": path.name,
"size_bytes": stat.st_size,
"modified_at": datetime.fromtimestamp(
stat.st_mtime, tz=timezone.utc
).isoformat(),
"agent": data.get("agent"),
"tag": data.get("tag"),
"status": data.get("status"),
"session_id": data.get("session_id"),
"started_at": data.get("started_at"),
"archived_at": data.get("archived_at"),
"wallclock_seconds": data.get("wallclock_seconds"),
"message_count": len(data.get("messages") or []),
"turn_count": len(per_turn) if isinstance(per_turn, list) else 0,
# v2 (2026-05-10) fields: present for subagent archives,
# ``None`` for top-level wrapper archives. The UI can use
# ``parent_session_id`` to group every session from one
# dispatcher cycle into a single tree row, and
# ``subagent_depth`` to nest the table view. Old (v1)
# archives lack these keys; ``.get`` returns None so the
# row schema stays uniform.
"schema_version": data.get("schema_version"),
"parent_session_id": data.get("parent_session_id"),
"subagent_title": data.get("subagent_title"),
"subagent_depth": data.get("subagent_depth"),
}
)
# Newest archive first. Use ``modified_at`` since it is ISO-formatted
# in the entry already; sort uses the timestamp string lexicographically
# which is correct for ISO-8601.
entries.sort(key=lambda e: e.get("modified_at") or "", reverse=True)
return {
"archive_dir": str(archive_dir),
"rows": entries[:_ARCHIVE_LIST_LIMIT],
"total": len(entries),
"limit": _ARCHIVE_LIST_LIMIT,
}
def _api_archived_session_detail(filename: str | None) -> dict[str, Any]:
"""Full payload for a single archive file."""
if not filename:
return {"error": "missing filename query parameter"}
if not _SAFE_ARCHIVE_FILENAME_RE.match(filename):
# Reject anything containing ``/``, ``..``, NULL bytes, etc.
# Without this guard a request like
# ``?filename=../../../../etc/passwd`` would be served verbatim.
return {"error": "invalid filename"}
archive_dir = _archive_dir_path()
path = archive_dir / filename
# Defence in depth: even though _SAFE_ARCHIVE_FILENAME_RE rejects
# path separators, resolve the path and confirm it stays inside
# the archive directory before reading.
try:
resolved = path.resolve()
archive_resolved = archive_dir.resolve()
except OSError as e:
return {"error": f"path resolution failed: {e}"}
if archive_resolved not in resolved.parents and resolved != archive_resolved:
return {"error": "filename escapes archive directory"}
if not resolved.is_file():
return {"error": f"not found: {filename}"}
try:
with resolved.open("r", encoding="utf-8") as f:
data = json.load(f)
except (OSError, ValueError) as e:
return {"error": str(e)}
# Attach the durable controller-DB payloads for this attempt so the
# archive post-mortem shows the structured result the worker
# produced — not just the ephemeral workspace path in the transcript.
if isinstance(data, dict):
data["controller_attempt"] = _attempt_payloads(
_parse_controller_tag(data.get("tag"))["attempt_id"]
)
return data
# ─── Live-run artifacts (events.jsonl + snapshot.json) ──────────────────
#
# These endpoints are populated by ``tools/live_log_writer.py`` running as a
# sidecar of the dispatcher launcher. The writer is the only producer; this
# server is read-only.
#
# Runs are discovered under ``TELEMETRY_RUNS_ROOT`` (default ``/tmp``): any
# immediate subdirectory containing ``snapshot.json`` or ``events.jsonl`` is
# treated as a run. The run's basename (e.g.
# ``implementer-test-2026-05-13-run4``) is the ``run_id`` used in the
# ``?run=<id>`` query parameter on the snapshot/events endpoints.
#
# Runs are not assumed to be "live" — historical runs whose writer is no
# longer running stay browsable as long as their files exist. The runs list
# carries a ``is_live`` heuristic (snapshot.json mtime < 30s) so the UI can
# distinguish active runs from archived ones.
_RUNS_ROOT_ENV = "TELEMETRY_RUNS_ROOT"
_DEFAULT_RUNS_ROOT = "/tmp"
# A snapshot.json mtime newer than this many seconds means the writer is
# still alive. Matches the writer's default --snapshot-interval (5s) plus
# slack for clock skew + writer schedule jitter.
_LIVE_SNAPSHOT_FRESH_S = 30.0
# Filename guard for ``?run=`` values. The basename must match — no path
# separators, no ``..``, no NULL bytes. Anything else 400s.
_SAFE_RUN_ID_RE = re.compile(r"^[A-Za-z0-9._-]+$")
def _runs_root() -> Path:
return Path(os.environ.get(_RUNS_ROOT_ENV, _DEFAULT_RUNS_ROOT))
def _discover_runs() -> list[dict[str, Any]]:
"""Scan ``_runs_root()`` for subdirectories that look like writer runs.
A directory qualifies if it contains ``snapshot.json`` or
``events.jsonl``. The returned list is sorted with the most recently
updated run first so the UI lands on the active run by default.
"""
root = _runs_root()
if not root.is_dir():
return []
now = time.time()
rows: list[dict[str, Any]] = []
try:
children = list(root.iterdir())
except OSError:
return []
for child in children:
# ``is_dir`` / ``is_file`` can raise PermissionError on /tmp
# entries owned by other users (e.g. ``/tmp/.vnc-0``). We swallow
# every os-level error here so the runs list keeps working when
# the runs root happens to be a multi-tenant directory like
# ``/tmp``. The cost is silent skips for genuinely-corrupt
# entries, which is acceptable for a read-only discovery scan.
try:
if not child.is_dir():
continue
except OSError:
continue
snap = child / "snapshot.json"
events = child / "events.jsonl"
try:
has_snapshot = snap.is_file()
except OSError:
has_snapshot = False
try:
has_events = events.is_file()
except OSError:
has_events = False
if not (has_snapshot or has_events):
continue
snap_mtime: float | None = None
events_mtime: float | None = None
try:
if has_snapshot:
snap_mtime = snap.stat().st_mtime
except OSError:
has_snapshot = False
try:
if has_events:
events_mtime = events.stat().st_mtime
except OSError:
has_events = False
# Peek at snapshot for top-line fields so the runs list can render
# without an extra round-trip per row.
peek: dict[str, Any] = {}
if has_snapshot:
try:
snap_data = json.loads(snap.read_text(encoding="utf-8"))
if isinstance(snap_data, dict):
run_info = snap_data.get("run") or {}
impl = snap_data.get("implementer") or {}
rev = snap_data.get("reviewer") or {}
peek = {
"boot_ts": run_info.get("boot_ts"),
"uptime_seconds": run_info.get("uptime_seconds"),
"implementer_phase": impl.get("current_phase"),
"implementer_active_pr": impl.get("active_pr_number"),
"reviewer_phase": rev.get("current_phase"),
"telemetry_rows": (
(impl.get("totals") or {}).get("telemetry_rows_total")
),
"outcome_disputed_count": (
(impl.get("totals") or {}).get("outcome_disputed_count")
),
}
except (OSError, ValueError):
pass
# Cheap event count — line count of events.jsonl. SQLite-cheap on
# files of even tens of thousands of lines; we cap the read at 4MB
# by reading in chunks to bound worst-case latency on a runaway log.
event_count = None
if has_events:
event_count = _count_lines(events, max_bytes=4 * 1024 * 1024)
is_live = snap_mtime is not None and (now - snap_mtime) < _LIVE_SNAPSHOT_FRESH_S
rows.append(
{
"run_id": child.name,
"path": str(child),
"has_snapshot": has_snapshot,
"has_events": has_events,
"snapshot_mtime_s_ago": (
round(now - snap_mtime, 1) if snap_mtime is not None else None
),
"events_mtime_s_ago": (
round(now - events_mtime, 1) if events_mtime is not None else None
),
"events_count_approx": event_count,
"is_live": is_live,
**peek,
}
)
rows.sort(
key=lambda r: (
r.get("snapshot_mtime_s_ago")
if r.get("snapshot_mtime_s_ago") is not None
else float("inf")
),
)
return rows
def _count_lines(path: Path, max_bytes: int = 4 * 1024 * 1024) -> int | None:
"""Cheap line-counter that bails out after ``max_bytes`` to keep the
runs-list endpoint responsive even when an events.jsonl gets large."""
try:
with path.open("rb") as f:
count = 0
read = 0
while read < max_bytes:
chunk = f.read(64 * 1024)
if not chunk:
break
count += chunk.count(b"\n")
read += len(chunk)
return count
except OSError:
return None
def _resolve_run_dir(run_id: str | None) -> tuple[Path | None, str | None]:
"""Validate ``run_id`` against the discovered runs list. Returns
``(path, None)`` on success or ``(None, error_message)`` for any failure
mode (missing param, path-traversal attempt, unknown run)."""
if not run_id:
return None, (
"missing 'run' query parameter — call /runs to list available runs"
)
if not _SAFE_RUN_ID_RE.match(run_id):
return None, "invalid run id (path-traversal attempt or unsupported chars)"
candidate = _runs_root() / run_id
if not candidate.is_dir():
return None, f"unknown run: {run_id!r}"
# Defence in depth — confirm the resolved path is still inside the root.
try:
resolved = candidate.resolve()
root_resolved = _runs_root().resolve()
except OSError as e:
return None, f"path resolution failed: {e}"
if root_resolved not in resolved.parents:
return None, "run escapes runs root"
return candidate, None
def _api_runs() -> tuple[int, dict[str, Any]]:
runs = _discover_runs()
return 200, {
"runs_root": str(_runs_root()),
"rows": runs,
"count": len(runs),
}
def _api_snapshot(run_id: str | None) -> tuple[int, dict[str, Any]]:
run_dir, err = _resolve_run_dir(run_id)
if err is not None:
return 400 if "invalid" in err or "missing" in err else 404, {"error": err}
snap = run_dir / "snapshot.json" # type: ignore[union-attr]
if not snap.is_file():
return 503, {
"error": f"snapshot not yet written for run {run_id!r}",
"run_dir": str(run_dir),
}
try:
return 200, json.loads(snap.read_text(encoding="utf-8"))
except (OSError, ValueError) as e:
return 500, {"error": str(e), "snapshot_path": str(snap)}
def _parse_event_ts_ms(ts: str | None) -> int | None:
"""Parse the ISO-8601 ``ts`` field emitted by the live writer back to
a unix epoch in milliseconds. The writer's format is
``YYYY-MM-DDTHH:MM:SS.mmmZ`` (millisecond precision, trailing Z)."""
if not ts:
return None
try:
# Replace trailing Z with +00:00 for fromisoformat.
dt = datetime.fromisoformat(ts.replace("Z", "+00:00"))
except ValueError:
return None
return int(dt.timestamp() * 1000)
# ─── /events byte-offset cache ───────────────────────────────────────────
#
# events.jsonl is append-only and grows for the life of a run (no
# rotation). Re-scanning it from byte 0 on every ~3s UI poll is O(file
# size) per poll and degrades visibly over a multi-hour run.
#
# The cache exploits two facts: (1) the UI polls with a monotonically
# increasing ``since_ms`` (each poll passes back the previous response's
# ``next_since_ms``), and (2) events are appended in emission-time order.
#
# Invariant of a cache entry ``run_dir -> (since_ms, safe_offset)``:
# EVERY event whose line starts before ``safe_offset`` has
# ``ts_ms <= since_ms``. So a later request with ``since_ms' >= since_ms``
# can seek straight to ``safe_offset`` — everything before it would be
# filtered out anyway. Truncation is handled at use-time by checking the
# CURRENT file size against ``safe_offset``: if the file is now smaller
# than the cached offset, the entry is stale and we fall back to a full
# scan from byte 0 (no separate stored file-size needed).
_EVENTS_OFFSET_CACHE: dict[str, tuple[int, int]] = {}
_EVENTS_OFFSET_CACHE_LOCK = threading.Lock()
_EVENTS_OFFSET_CACHE_MAX = 32 # bounded — only a handful of runs ever exist
def _api_events(
run_id: str | None,
since_ms: int,
limit: int,
types_filter: set[str] | None,
) -> tuple[int, dict[str, Any]]:
run_dir, err = _resolve_run_dir(run_id)
if err is not None:
return 400 if "invalid" in err or "missing" in err else 404, {"error": err}
events_path = run_dir / "events.jsonl" # type: ignore[union-attr]
if not events_path.is_file():
return 200, {
"rows": [],
"next_since_ms": since_ms,
"note": f"events.jsonl not yet present: {events_path}",
}
cache_key = str(run_dir)
try:
file_size = events_path.stat().st_size
except OSError as e:
return 500, {"error": str(e), "events_path": str(events_path)}
# Decide where to start scanning. Default: byte 0 (cold path).
start_offset = 0
with _EVENTS_OFFSET_CACHE_LOCK:
cached = _EVENTS_OFFSET_CACHE.get(cache_key)
if (
cached is not None
and since_ms >= cached[0] # request's since_ms >= cached since_ms
and file_size >= cached[1] # file did not shrink/truncate under us
):
start_offset = cached[1]
rows: list[dict[str, Any]] = []
latest_ts_ms = since_ms
# safe_offset: byte offset of the first event with ts_ms > since_ms.
# Until we see one, every event scanned is filterable, so the running
# safe_offset is "current position". Once we see an unfiltered event
# we freeze safe_offset at its start byte.
safe_offset = start_offset
safe_offset_frozen = False
try:
with events_path.open("r", encoding="utf-8") as f:
f.seek(start_offset)
while True:
line_start = f.tell()
line = f.readline()
if not line:
break # EOF
if not line.endswith("\n"):
break # partial trailing line — writer mid-append
stripped = line.strip()
if not stripped:
if not safe_offset_frozen:
safe_offset = f.tell()
continue
try:
evt = json.loads(stripped)
except ValueError:
if not safe_offset_frozen:
safe_offset = f.tell()
continue
ts_ms = _parse_event_ts_ms(evt.get("ts"))
if ts_ms is None:
if not safe_offset_frozen:
safe_offset = f.tell()
continue
if ts_ms <= since_ms:
# Filterable — advance the safe offset past it.
if not safe_offset_frozen:
safe_offset = f.tell()
continue
# First event with ts_ms > since_ms — freeze safe_offset
# at its START byte (everything before is <= since_ms).
if not safe_offset_frozen:
safe_offset = line_start
safe_offset_frozen = True
if ts_ms > latest_ts_ms:
latest_ts_ms = ts_ms
if types_filter and evt.get("type") not in types_filter:
continue
rows.append(evt)
if len(rows) >= limit:
break
except OSError as e:
return 500, {"error": str(e), "events_path": str(events_path)}
# Refresh the cache entry. safe_offset now satisfies the invariant
# for this request's since_ms. Bound the cache size.
with _EVENTS_OFFSET_CACHE_LOCK:
_EVENTS_OFFSET_CACHE[cache_key] = (since_ms, safe_offset)
if len(_EVENTS_OFFSET_CACHE) > _EVENTS_OFFSET_CACHE_MAX:
# Evict an arbitrary oldest-ish entry (dict preserves insertion
# order; the first key is the least-recently-inserted).
_EVENTS_OFFSET_CACHE.pop(next(iter(_EVENTS_OFFSET_CACHE)))
return 200, {
"rows": rows,
"next_since_ms": latest_ts_ms,
"count": len(rows),
"limit": limit,
"run_id": run_id,
}
# ─── Per-run session tree + detail ───────────────────────────────────────
#
# These two endpoints back the Live tab's "session tree → click-to-inspect"
# UX. They reuse two existing data sources:
#
# - events.jsonl (per run) — gives us every session_id the run ever
# touched, plus turn counts, state transitions, and the archive path
# once a session ends.
# - .dispatcher-logs/sessions/*.json — full transcripts, including
# parent_session_id, written by _opencode_worker.run_session_blocking
# after a session ends.
#
# Parent → child resolution: archive JSON carries ``parent_session_id``.
# For sessions whose archive has not landed yet (worker still running), we
# don't know the parent until either the archive lands or we ask OpenCode
# directly. The /session detail endpoint handles the live-OpenCode
# fall-through; the tree endpoint marks unresolved parents as None.
_ARCHIVE_FILENAME_SID_RE = re.compile(r"__(?P<sid>ses_[A-Za-z0-9]+)\.json$")
def _archive_index_by_session_id() -> dict[str, Path]:
"""Scan the archive directory once and return ``session_id -> Path``.
Filename-only scan (no JSON parse), so it's cheap even on a directory
with hundreds of archives. The caller reads JSON only for the
sessions it actually needs."""
archive_dir = _archive_dir_path()
out: dict[str, Path] = {}
if not archive_dir.is_dir():
return out
try:
for path in archive_dir.glob("*.json"):
m = _ARCHIVE_FILENAME_SID_RE.search(path.name)
if m:
out[m.group("sid")] = path
except OSError:
pass
return out
# Status precedence: a later event's status only overrides an earlier
# session's status if it's "more terminal". This lets a single scan over
# events build a stable status per session without needing to sort.
_STATUS_PRECEDENCE = {
"starting": 0,
"running": 1,
"idle": 2,
"terminated": 3,
"archived": 4,
"timeout": 5,
"transport_error": 5,
}
def _bump_status(current: str | None, candidate: str) -> str:
if current is None:
return candidate
if _STATUS_PRECEDENCE.get(candidate, 0) >= _STATUS_PRECEDENCE.get(current, 0):
return candidate
return current
def _scan_run_sessions(events_path: Path) -> dict[str, dict[str, Any]]:
"""Walk events.jsonl once, build a {session_id: row} dict. Each row
carries everything we can learn from events alone — no archive reads,
no OpenCode calls. The caller decorates with archive/OpenCode data."""
by_sid: dict[str, dict[str, Any]] = {}
try:
with events_path.open("r", encoding="utf-8") as f:
for line in f:
line = line.strip()
if not line:
continue
try:
evt = json.loads(line)
except ValueError:
continue
sid = evt.get("session_id")
if not sid:
continue
row = by_sid.setdefault(
sid,
{
"session_id": sid,
"agent": None,
"model": None,
"tag": None,
"pr_number": None,
"source": None,
"parent_session_id": None,
"depth": None,
"status": None,
"started_at": None,
"ended_at": None,
"turn_count": 0,
"input_tokens": 0,
"output_tokens": 0,
"tools_used": {},
"archive_path": None,
# ``tier_recommendation`` is populated only for
# estimator sessions (by subagent.tier_recommendation).
# None for everyone else — the UI checks for
# presence before badging.
"tier_recommendation": None,
},
)
ts = evt.get("ts")
data = evt.get("data") or {}
etype = evt.get("type")
if etype == "worker.session_created":
# Two emits per session: a "pre-id" with model only
# (session_id absent there, filtered above) and the
# real one with session_id. The data payload here is
# ``{"agent": ...}`` only.
row["agent"] = row["agent"] or data.get("agent")
row["tag"] = row["tag"] or evt.get("tag")
row["pr_number"] = row["pr_number"] or evt.get("pr_number")
row["source"] = row["source"] or evt.get("source")
if row["started_at"] is None:
row["started_at"] = ts
row["status"] = _bump_status(row["status"], "starting")
elif etype == "worker.state_change":
to_state = data.get("to") or ""
if to_state == "busy":
row["status"] = _bump_status(row["status"], "running")
elif to_state == "idle":
row["status"] = _bump_status(row["status"], "idle")
elif etype == "worker.turn_finished":
row["turn_count"] += 1
row["input_tokens"] += int(data.get("input_tok") or 0)
row["output_tokens"] += int(data.get("output_tok") or 0)
for tool in data.get("tools") or []:
row["tools_used"][tool] = row["tools_used"].get(tool, 0) + 1
elif etype == "worker.subagent_spawned":
# This event's session_id is the SUBAGENT's, not the
# parent's. We learn the agent + depth + archive_path
# here; parent is resolved from the archive JSON.
row["agent"] = row["agent"] or data.get("agent")
row["depth"] = (
row["depth"] if row["depth"] is not None else data.get("depth")
)
row["archive_path"] = row["archive_path"] or data.get(
"archive_path"
)
row["source"] = row["source"] or evt.get("source")
if row["started_at"] is None:
row["started_at"] = ts
row["status"] = _bump_status(row["status"], "archived")
elif etype == "worker.terminated":
row["ended_at"] = ts
row["status"] = _bump_status(row["status"], "terminated")
elif etype == "worker.archive_written":
row["archive_path"] = row["archive_path"] or data.get(
"archive_path"
)
if row["ended_at"] is None:
row["ended_at"] = ts
row["status"] = _bump_status(row["status"], "archived")
elif etype == "worker.timeout":
row["ended_at"] = ts
row["status"] = _bump_status(row["status"], "timeout")
elif etype == "worker.transport_error":
row["ended_at"] = ts
row["status"] = _bump_status(row["status"], "transport_error")
# ── Live-subagent events (sourced from the OpenCode SSE
# ── subscriber in live_log_writer.py). These let the tree
# ── render BEFORE the archive lands, which is the whole
# ── point of the SSE wire — the archive only lands
# ── 10-25 min later. parent_session_id + depth come in
# ── via the event's ``data`` payload.
elif etype == "subagent.session_created":
row["agent"] = row["agent"] or data.get("agent")
row["depth"] = (
row["depth"] if row["depth"] is not None else data.get("depth")
)
row["parent_session_id"] = row["parent_session_id"] or data.get(
"parent_session_id"
)
row["tag"] = row["tag"] or evt.get("tag")
row["pr_number"] = row["pr_number"] or evt.get("pr_number")
row["source"] = row["source"] or evt.get("source")
if row["started_at"] is None:
row["started_at"] = data.get("started_at") or ts
row["status"] = _bump_status(row["status"], "starting")
elif etype == "subagent.state_change":
to_state = data.get("to") or ""
if to_state == "running":
row["status"] = _bump_status(row["status"], "running")
elif to_state == "idle":
row["status"] = _bump_status(row["status"], "idle")
elif etype == "subagent.tool_call_start":
tool = data.get("tool") or "tool"
row["tools_used"][tool] = row["tools_used"].get(tool, 0) + 1
elif etype == "subagent.terminated":
row["ended_at"] = ts
row["status"] = _bump_status(row["status"], "terminated")
elif etype == "subagent.tier_recommendation":
# Surface the tier pick on the row so the UI can
# badge the estimator node without a separate fetch.
row["tier_recommendation"] = {
"tier": data.get("tier"),
"is_confident": data.get("is_confident"),
}
except OSError:
return {}
return by_sid
def _decorate_with_archive(
rows: dict[str, dict[str, Any]],
sid_to_archive: dict[str, Path],
) -> None:
"""For each session_id with an archive on disk, fill in parent_session_id,
depth, started_at, model, status corrections. Mutates ``rows`` in place."""
for sid, row in rows.items():
archive_path = row.get("archive_path")
if archive_path:
p = Path(archive_path)
else:
p = sid_to_archive.get(sid)
if p is None or not p.is_file():
continue
try:
data = json.loads(p.read_text(encoding="utf-8"))
except (OSError, ValueError):
continue
if not isinstance(data, dict):
continue
row["parent_session_id"] = row["parent_session_id"] or data.get(
"parent_session_id"
)
if row["depth"] is None:
row["depth"] = data.get("subagent_depth") or 0
row["agent"] = row["agent"] or data.get("agent")
row["tag"] = row["tag"] or data.get("tag")
row["started_at"] = row["started_at"] or data.get("started_at")
row["ended_at"] = row["ended_at"] or data.get("archived_at")
row["archive_path"] = str(p)
# Pull a representative modelID from the first assistant message.
# User messages carry ``info.model`` as a {providerID, modelID}
# dict; assistant messages carry ``info.modelID`` as the plain
# string. We want the plain string, so prefer assistant rows and
# unwrap dicts from user rows as a fallback.
if not row["model"]:
for msg in data.get("messages") or []:
info = (msg or {}).get("info") or {}
if info.get("role") != "assistant":
continue
m = info.get("modelID") or info.get("model")
if isinstance(m, dict):
m = m.get("modelID")
if m:
row["model"] = m
break
if not row["model"]:
# No assistant message yet — fall back to the user-side dict.
for msg in (data.get("messages") or [])[:3]:
info = (msg or {}).get("info") or {}
m = info.get("model")
if isinstance(m, dict):
m = m.get("modelID")
if m:
row["model"] = m
break
# Subagents don't emit ``worker.turn_finished`` (that event only
# fires for the top-level worker session), so events-derived
# turn_count is 0 for them. The archive's ``per_turn`` array has
# the real number — use it when events didn't see any turns.
if row["turn_count"] == 0:
per_turn = data.get("per_turn") or []
if isinstance(per_turn, list):
row["turn_count"] = len(per_turn)
# Archive-derived status: prefer "completed" once the archive
# exists and shows a clean terminated status.
archive_status = data.get("status")
if archive_status == "completed":
row["status"] = _bump_status(row["status"], "terminated")
elif archive_status == "subagent":
row["status"] = _bump_status(row["status"], "archived")
def _api_run_sessions(run_id: str | None) -> tuple[int, dict[str, Any]]:
"""Per-run session inventory: every session_id seen in the run's
events.jsonl, decorated with archive-derived parent/depth/model so the
UI can render the parent→child tree without per-row fetches.
The endpoint is deliberately a single scan + a directory glob; no
OpenCode round-trip. The per-session detail endpoint does the
live-OpenCode fall-through when a transcript is requested."""
run_dir, err = _resolve_run_dir(run_id)
if err is not None:
return 400 if "invalid" in err or "missing" in err else 404, {"error": err}
events_path = run_dir / "events.jsonl" # type: ignore[union-attr]
if not events_path.is_file():
return 200, {
"run_id": run_id,
"rows": [],
"note": f"events.jsonl not yet present: {events_path}",
}
rows_by_sid = _scan_run_sessions(events_path)
sid_to_archive = _archive_index_by_session_id()
_decorate_with_archive(rows_by_sid, sid_to_archive)
rows = list(rows_by_sid.values())
# Stable ordering: parents before children when both are present, then
# by started_at. The UI does its own tree-building from
# parent_session_id, so order is purely cosmetic for unparented rows.
rows.sort(key=lambda r: (r.get("depth") or 0, r.get("started_at") or ""))
return 200, {
"run_id": run_id,
"rows": rows,
"count": len(rows),
"archive_dir": str(_archive_dir_path()),
}
# ─── Per-session transcript (archive OR live OpenCode) ───────────────────
def _normalize_message_parts(parts: list[Any]) -> list[dict[str, Any]]:
"""Collapse OpenCode's verbose message-parts into a smaller, UI-shaped
shape. Drops housekeeping parts (step-start/step-finish), preserves the
text/reasoning/tool parts the UI actually renders."""
out: list[dict[str, Any]] = []
for part in parts or []:
if not isinstance(part, dict):
continue
ptype = part.get("type")
if ptype == "text":
text = part.get("text") or ""
if text.strip():
out.append({"kind": "text", "text": text})
elif ptype == "reasoning":
text = part.get("text") or ""
if text.strip():
out.append({"kind": "reasoning", "text": text})
elif ptype == "tool":
state = part.get("state") or {}
entry: dict[str, Any] = {
"kind": "tool",
"tool": part.get("tool"),
"status": state.get("status"),
"title": state.get("title"),
"input": state.get("input"),
}
output = state.get("output")
if output is not None:
# Truncate at the transport boundary to keep the response
# cheap; the archive file is always available for full
# inspection via /api/sessions/archived/detail.
if isinstance(output, str) and len(output) > 8000:
entry["output"] = output[:8000]
entry["output_truncated"] = True
else:
entry["output"] = output
out.append(entry)
# step-start / step-finish / other housekeeping parts are dropped.
return out
def _normalize_messages(messages: list[Any]) -> list[dict[str, Any]]:
"""Turn OpenCode's ``[{info, parts}]`` shape into ``[{role, model, ts,
parts}]`` for the UI. Skips empty messages (no useful parts)."""
out: list[dict[str, Any]] = []
for msg in messages or []:
if not isinstance(msg, dict):
continue
info = msg.get("info") or {}
parts = _normalize_message_parts(msg.get("parts") or [])
if not parts:
continue
time_info = info.get("time") or {}
ts = None
if isinstance(time_info, dict):
ts = time_info.get("created") or time_info.get("completed")
# Same unwrap as _decorate_with_archive: user messages store
# ``info.model`` as a dict, assistant messages store
# ``info.modelID`` as a string. UI wants a string.
model = info.get("modelID") or info.get("model")
if isinstance(model, dict):
model = model.get("modelID")
out.append(
{
"role": info.get("role"),
"model": model,
"agent": info.get("agent"),
"ts": ts,
"parts": parts,
}
)
return out
def _api_run_session(
run_id: str | None, session_id: str | None
) -> tuple[int, dict[str, Any]]:
"""Per-session transcript. Uses the archive when present, falls back
to a live OpenCode fetch when the session is still running.
The shape is the same in both cases so the UI doesn't need to branch:
``{session_id, source, status, agent, model, parent_session_id,
turns: [{role, model, ts, parts}], summary: {...}}``."""
run_dir, err = _resolve_run_dir(run_id)
if err is not None:
return 400 if "invalid" in err or "missing" in err else 404, {"error": err}
if not session_id or not session_id.startswith("ses_") or "/" in session_id:
return 400, {"error": "missing or invalid 'session' query parameter"}
sid_to_archive = _archive_index_by_session_id()
archive_path = sid_to_archive.get(session_id)
if archive_path is not None and archive_path.is_file():
try:
data = json.loads(archive_path.read_text(encoding="utf-8"))
except (OSError, ValueError) as e:
return 500, {"error": str(e), "archive_path": str(archive_path)}
if not isinstance(data, dict):
return 500, {"error": "archive is not a JSON object"}
return 200, {
"session_id": session_id,
"source": "archive",
"archive_path": str(archive_path),
"status": data.get("status"),
"agent": data.get("agent"),
"tag": data.get("tag"),
"parent_session_id": data.get("parent_session_id"),
"subagent_depth": data.get("subagent_depth"),
"started_at": data.get("started_at"),
"ended_at": data.get("archived_at"),
"wallclock_seconds": data.get("wallclock_seconds"),
"turns": _normalize_messages(data.get("messages") or []),
}
# Live fall-through: ask OpenCode directly. This is the only path that
# makes a network call, and it only fires when an operator clicks a
# currently-running session — so a slow OpenCode does not affect the
# tree endpoint.
info = _opencode_get(f"/session/{session_id}")
messages = _opencode_get(f"/session/{session_id}/message")
if info is None and messages is None:
return 503, {
"error": (
f"no archive on disk for {session_id} AND OpenCode at "
f"{OPENCODE_URL} is unreachable — transcript not available"
),
"session_id": session_id,
}
if not isinstance(messages, list):
messages = []
info = info if isinstance(info, dict) else {}
return 200, {
"session_id": session_id,
"source": "live-opencode",
"status": "running",
"agent": info.get("agent"),
"parent_session_id": info.get("parentID"),
"started_at": (info.get("time") or {}).get("created")
if isinstance(info.get("time"), dict)
else None,
"ended_at": None,
"turns": _normalize_messages(messages),
}
# Legacy dispatcher tag abbreviation → controller role name, so the
# Cost tab's by-stage view unifies legacy + controller spend into the
# same buckets. AUTO-IMP / AUTO-REV / AUTO-CFL are the shapes observed
# in the cache; an unrecognised abbreviation falls through to "other".
_LEGACY_TAG_ROLE = {
"IMP": "implementer",
"REV": "reviewer",
"CFL": "conflict_resolver",
}
def _stage_from_tag(session_tag: str | None) -> str:
"""Map a session tag to a pipeline stage (controller role).
Controller tags are ``controller-<role>-<n>``; legacy dispatcher
tags are ``AUTO-<ABBR>-PR-<n>``. Crucially the scraper gives every
subagent turn its wrapper session's tag, so deriving the stage from
the tag (not the per-row ``agent``) attributes a subagent's tokens
to the stage that spawned it. Probe / overhead sessions whose tag
matches neither shape fall into ``other``."""
if isinstance(session_tag, str):
m = re.match(r"controller-([a-z_]+)-\d", session_tag)
if m:
return m.group(1)
m = re.match(r"AUTO-([A-Z]+)-", session_tag)
if m and m.group(1) in _LEGACY_TAG_ROLE:
return _LEGACY_TAG_ROLE[m.group(1)]
return "other"
def _api_cost(days: int, group_by: str = "model") -> dict[str, Any]:
# PR- and role-grouped paths SUM per (key, model, provider) and
# collapse in Python so per-model pricing stays exact when one key
# spans multiple models.
group_by = group_by if group_by in ("model", "pr", "role") else "model"
conn = _open_db()
prices = _load_prices()
if conn is None:
return {
"days": days,
"group_by": group_by,
"rows": [],
"totals": {"usd": 0.0},
"note": "cache absent",
"instrumentation_pending": True,
}
try:
if group_by == "pr":
# SUM tokens per (pr_number, model, provider) so per-model pricing
# stays exact, then collapse to one row per pr_number in Python.
cur = conn.execute(
"SELECT pr_number, model, provider, COUNT(*) AS n,"
" SUM(COALESCE(tokens_in, 0)) AS in_tok,"
" SUM(COALESCE(tokens_out, 0)) AS out_tok,"
" SUM(COALESCE(cached_tokens, 0)) AS cached_tok"
" FROM llm_activity"
" WHERE started_at > datetime('now', ?)"
" GROUP BY pr_number, model, provider",
(f"-{int(days)} days",),
)
by_pr: dict[Any, dict[str, Any]] = {}
total_usd = 0.0
total_in = total_out = total_cached = 0
for r in cur:
in_tok = int(r["in_tok"] or 0)
out_tok = int(r["out_tok"] or 0)
cached = int(r["cached_tok"] or 0)
model = r["model"] or "_unknown"
provider = r["provider"]
usd = _cost_usd(
provider,
model,
prices,
in_tok,
out_tok,
cached,
)
key = _price_key(provider, model)
priced = key in prices or model in prices
pr_key = r["pr_number"]
bucket = by_pr.setdefault(
pr_key,
{
"pr_number": pr_key,
"n": 0,
"tokens_in": 0,
"tokens_out": 0,
"cached": 0,
"usd": 0.0,
"models": set(),
"any_unpriced": False,
},
)
bucket["n"] += int(r["n"] or 0)
bucket["tokens_in"] += in_tok
bucket["tokens_out"] += out_tok
bucket["cached"] += cached
bucket["usd"] += usd
bucket["models"].add(model)
if not priced:
bucket["any_unpriced"] = True
total_usd += usd
total_in += in_tok
total_out += out_tok
total_cached += cached
rows = []
for bucket in by_pr.values():
models = sorted(bucket["models"])
rows.append(
{
"pr_number": bucket["pr_number"],
"n": bucket["n"],
"tokens_in": bucket["tokens_in"],
"tokens_out": bucket["tokens_out"],
"cached": bucket["cached"],
"usd": bucket["usd"],
"models": models,
"priced": not bucket["any_unpriced"],
}
)
# Sort by cost descending; NULL pr_number (unattributed) sinks
# to the bottom regardless of its USD so it doesn't crowd out
# real PRs at the top.
rows.sort(
key=lambda r: (
0 if r["pr_number"] is not None else 1,
-(r["usd"] or 0.0),
)
)
return {
"days": days,
"group_by": "pr",
"rows": rows,
"totals": {
"usd": round(total_usd, 2),
"tokens_in": total_in,
"tokens_out": total_out,
"cached": total_cached,
},
"instrumentation_pending": (not rows),
}
if group_by == "role":
# SUM per (session_tag, model, provider) — splitting on model
# + provider keeps pricing exact — then collapse to one row
# per pipeline stage in Python. Subagent rows carry the
# wrapper tag, so their tokens roll into the spawning stage.
cur = conn.execute(
"SELECT session_tag, model, provider, COUNT(*) AS n,"
" SUM(COALESCE(tokens_in, 0)) AS in_tok,"
" SUM(COALESCE(tokens_out, 0)) AS out_tok,"
" SUM(COALESCE(cached_tokens, 0)) AS cached_tok"
" FROM llm_activity"
" WHERE started_at > datetime('now', ?)"
" GROUP BY session_tag, model, provider",
(f"-{int(days)} days",),
)
by_role: dict[str, dict[str, Any]] = {}
total_usd = 0.0
total_in = total_out = total_cached = 0
for r in cur:
in_tok = int(r["in_tok"] or 0)
out_tok = int(r["out_tok"] or 0)
cached = int(r["cached_tok"] or 0)
model = r["model"] or "_unknown"
provider = r["provider"]
usd = _cost_usd(
provider, model, prices, in_tok, out_tok, cached,
)
key = _price_key(provider, model)
priced = key in prices or model in prices
role = _stage_from_tag(r["session_tag"])
bucket = by_role.setdefault(
role,
{
"role": role,
"n": 0,
"tokens_in": 0,
"tokens_out": 0,
"cached": 0,
"usd": 0.0,
"models": set(),
"any_unpriced": False,
},
)
bucket["n"] += int(r["n"] or 0)
bucket["tokens_in"] += in_tok
bucket["tokens_out"] += out_tok
bucket["cached"] += cached
bucket["usd"] += usd
bucket["models"].add(model)
if not priced:
bucket["any_unpriced"] = True
total_usd += usd
total_in += in_tok
total_out += out_tok
total_cached += cached
rows = []
for bucket in by_role.values():
rows.append(
{
"role": bucket["role"],
"n": bucket["n"],
"tokens_in": bucket["tokens_in"],
"tokens_out": bucket["tokens_out"],
"cached": bucket["cached"],
"usd": bucket["usd"],
"models": sorted(bucket["models"]),
"priced": not bucket["any_unpriced"],
}
)
# Cost descending; the catch-all "other" bucket sinks to the
# bottom so real pipeline stages lead the table.
rows.sort(
key=lambda r: (
1 if r["role"] == "other" else 0,
-(r["usd"] or 0.0),
)
)
return {
"days": days,
"group_by": "role",
"rows": rows,
"totals": {
"usd": round(total_usd, 2),
"tokens_in": total_in,
"tokens_out": total_out,
"cached": total_cached,
},
"instrumentation_pending": (not rows),
}
cur = conn.execute(
"SELECT model, provider, COUNT(*) AS n,"
" SUM(COALESCE(tokens_in, 0)) AS in_tok,"
" SUM(COALESCE(tokens_out, 0)) AS out_tok,"
" SUM(COALESCE(cached_tokens, 0)) AS cached_tok"
" FROM llm_activity"
" WHERE started_at > datetime('now', ?)"
" GROUP BY model, provider ORDER BY n DESC",
(f"-{int(days)} days",),
)
rows = []
total_usd = 0.0
total_in = total_out = total_cached = 0
for r in cur:
in_tok = int(r["in_tok"] or 0)
out_tok = int(r["out_tok"] or 0)
cached = int(r["cached_tok"] or 0)
model = r["model"] or "_unknown"
provider = r["provider"]
usd = _cost_usd(
provider,
model,
prices,
in_tok,
out_tok,
cached,
)
key = _price_key(provider, model)
# Resolved per-1M-token rates (same lookup chain _cost_usd
# uses) so the UI can show how each row's cost was derived.
p = prices.get(key) or prices.get(model) or prices.get("_unknown", {})
rows.append(
{
"model": model,
"provider": provider,
"n": r["n"],
"tokens_in": in_tok,
"tokens_out": out_tok,
"cached": cached,
"usd": usd,
"priced": (key in prices or model in prices),
"price": {
"in": p.get("in", 0.0),
"out": p.get("out", 0.0),
"cached_in": p.get("cached_in", p.get("in", 0.0)),
},
}
)
total_usd += usd
total_in += in_tok
total_out += out_tok
total_cached += cached
return {
"days": days,
"group_by": "model",
"rows": rows,
"totals": {
"usd": round(total_usd, 2),
"tokens_in": total_in,
"tokens_out": total_out,
"cached": total_cached,
},
"instrumentation_pending": (not rows),
}
except sqlite3.OperationalError as e:
return {
"days": days,
"group_by": group_by,
"rows": [],
"totals": {"usd": 0.0},
"note": f"llm_activity table missing or stale schema: {e}",
"instrumentation_pending": True,
}
finally:
conn.close()
# ─── Phases (controller DB + CI gate events) ────────────────────────────
_CONTROLLER_ENGINE: Any = None
_CONTROLLER_ENGINE_URL: str | None = None
_CONTROLLER_ENGINE_LOCK = threading.Lock()
def _open_controller_db_engine() -> Any:
"""Return a cached SQLAlchemy Engine for the controller DB, or None
when SQLAlchemy is absent, ``CLEVERAGENTS_DB_URL`` is unset, or the
handshake fails. Callers treat None as "instrumentation pending"."""
global _CONTROLLER_ENGINE, _CONTROLLER_ENGINE_URL
if not _SQLALCHEMY_AVAILABLE:
return None
url = os.environ.get("CLEVERAGENTS_DB_URL")
if not url:
return None
if _CONTROLLER_ENGINE is not None and _CONTROLLER_ENGINE_URL == url:
return _CONTROLLER_ENGINE
# Lock guards against ThreadingHTTPServer racing two threads into
# double-create on first miss (the loser would leak an engine).
with _CONTROLLER_ENGINE_LOCK:
if _CONTROLLER_ENGINE is not None and _CONTROLLER_ENGINE_URL == url:
return _CONTROLLER_ENGINE
try:
# pool_pre_ping recycles stale connections (Postgres idle-timeout
# is the usual culprit); without it the next call after a long
# idle period fails with OperationalError.
engine = _sa_create_engine(url, future=True, pool_pre_ping=True)
with engine.connect() as conn:
conn.execute(_sa_text("SELECT 1"))
_CONTROLLER_ENGINE = engine
_CONTROLLER_ENGINE_URL = url
return engine
except Exception as exc: # noqa: BLE001 — defensive: never 500
logger.warning("controller DB connect failed: %s", exc)
_CONTROLLER_ENGINE = None
_CONTROLLER_ENGINE_URL = None
return None
def _coerce_json(value: Any) -> Any:
"""Normalize a controller-DB JSON column to a Python object.
Raw ``text()`` queries hand JSON columns back as strings on SQLite
but as already-parsed objects on Postgres — callers want a uniform
object. A value that is not valid JSON is returned verbatim."""
if value is None or isinstance(value, (dict, list)):
return value
if isinstance(value, (str, bytes)):
try:
return json.loads(value)
except (ValueError, TypeError):
return value
return value
def _attempt_payloads(attempt_id: int | None) -> dict[str, Any] | None:
"""Durable input/output payloads for one ``workflow_attempts`` row.
The OpenCode session tag carries the controller ``attempt_id``; this
turns it into the structured result the controller actually acted on
(``output_payload``) — the same JSON the worker wrote to the
ephemeral ``{role}_output.json`` workspace file, but the persistent,
per-attempt copy. Returns None when the controller DB is
unavailable, the attempt is unknown, or it belongs to another repo.
"""
if attempt_id is None:
return None
engine = _open_controller_db_engine()
if engine is None:
return None
try:
with engine.connect() as conn:
row = conn.execute(
_sa_text(
"SELECT a.attempt_id, a.attempt_number, a.role, "
" a.status, a.outcome, "
" a.input_payload, a.output_payload, "
" a.output_version, a.input_payload_truncated, "
" w.owner, w.repo, w.entity_number "
" FROM workflow_attempts a "
" JOIN workflows w ON a.workflow_id = w.workflow_id "
" WHERE a.attempt_id = :aid"
),
{"aid": attempt_id},
).first()
except Exception as exc: # noqa: BLE001 — defensive: never 500
logger.debug("attempt payload lookup failed for %s: %s", attempt_id, exc)
return None
if row is None:
return None
# Scope guard: never hand back another repo's payloads.
if row.owner != REPO_OWNER or row.repo != REPO_NAME:
return None
return {
"attempt_id": int(row.attempt_id),
"attempt_number": (
int(row.attempt_number) if row.attempt_number is not None else None
),
"role": row.role,
"status": row.status,
"outcome": row.outcome,
"output_version": row.output_version,
"input_payload_truncated": bool(row.input_payload_truncated),
"input_payload": _coerce_json(row.input_payload),
"output_payload": _coerce_json(row.output_payload),
}
# Unknown roles pass through verbatim so a newly-added role surfaces
# immediately. "ci" is not here — it comes from ci_gate_events.
_ROLE_TO_PHASE = {
"implementer": "implementation",
"reviewer": "review",
"conflict_resolver": "conflict_resolution",
"estimator": "estimation",
"summarizer": "summarization",
}
# Phases sourced from ci_gate_events rather than workflow_attempts.
# Their "wallclock" is summed gate durations — a CPU/job-time figure,
# not the controller's wall-clock-in-state. Kept in its own set so
# share-of-time math can exclude it (mixing them misleads operators).
_CI_PHASES = frozenset({"ci"})
def _api_phases(days: int, group_by: str = "phase") -> dict[str, Any]:
"""Time spent per pipeline phase, scoped to this server's
``REPO_OWNER/REPO_NAME``. ``group_by`` is "phase" (totals across
all PRs in window) or "pr" (per-PR breakdown).
Worker phases come from controller DB ``workflow_attempts`` (filtered
to ``status IN ('complete','failed')`` so in-flight attempts don't
contribute partial / NULL wallclock). CI phase comes from legacy
``ci_gate_events.duration_s`` — that's per-gate job-time, not
wall-clock-in-state, so share-of-time is computed against worker
phases only.
"""
group_by = group_by if group_by in ("phase", "pr") else "phase"
engine = _open_controller_db_engine()
legacy = _open_db()
cutoff = datetime.now(timezone.utc) - timedelta(days=days)
per_pr: dict[tuple[int | None, str], dict[str, float]] = {}
note_parts: list[str] = []
if engine is not None:
try:
with engine.connect() as conn:
rows = conn.execute(
_sa_text(
"SELECT w.entity_number AS pr_number, "
" a.role AS role, "
" COUNT(*) AS attempts, "
" COALESCE(SUM(a.wallclock_seconds), 0) AS seconds "
" FROM workflow_attempts a "
" JOIN workflows w ON w.workflow_id = a.workflow_id "
" WHERE w.kind = 'pr' "
" AND w.owner = :owner "
" AND w.repo = :repo "
" AND a.status IN ('complete', 'failed') "
" AND a.finished_at IS NOT NULL "
" AND a.finished_at > :cutoff "
" GROUP BY w.entity_number, a.role"
),
{"owner": REPO_OWNER, "repo": REPO_NAME, "cutoff": cutoff},
).all()
for row in rows:
pr_number = (
int(row.pr_number) if row.pr_number is not None else None
)
phase = _ROLE_TO_PHASE.get(row.role, row.role)
bucket = per_pr.setdefault(
(pr_number, phase),
{"seconds": 0.0, "attempts": 0},
)
bucket["seconds"] += float(row.seconds or 0.0)
bucket["attempts"] += int(row.attempts or 0)
except Exception as exc: # noqa: BLE001 — surface as a note, not a 500
note_parts.append(f"controller DB query failed: {exc}")
else:
note_parts.append("CLEVERAGENTS_DB_URL not set (worker-phase data unavailable)")
if legacy is not None:
try:
# Same Python datetime cutoff as the controller query so the
# two source windows align exactly. ci_gate_events.observed_at
# is stored as tz-aware UTC ISO; lexical > works against an
# ISO string from the same producer.
cur = legacy.execute(
"SELECT pr_number, COUNT(*) AS attempts, "
" COALESCE(SUM(duration_s), 0) AS seconds "
" FROM ci_gate_events "
" WHERE observed_at > ? "
" GROUP BY pr_number",
(cutoff.isoformat(),),
)
for r in cur:
pr_number = int(r["pr_number"]) if r["pr_number"] is not None else None
bucket = per_pr.setdefault(
(pr_number, "ci"),
{"seconds": 0.0, "attempts": 0},
)
bucket["seconds"] += float(r["seconds"] or 0.0)
bucket["attempts"] += int(r["attempts"] or 0)
except sqlite3.OperationalError as exc:
note_parts.append(f"ci_gate_events query failed: {exc}")
finally:
legacy.close()
if not per_pr:
return {
"days": days,
"group_by": group_by,
"rows": [],
"totals": {"seconds": 0.0, "ci_seconds": 0.0, "attempts": 0},
"note": "; ".join(note_parts) or "no phase data in this window",
"instrumentation_pending": True,
}
worker_seconds = sum(
b["seconds"] for (_, ph), b in per_pr.items() if ph not in _CI_PHASES
)
ci_seconds = sum(b["seconds"] for (_, ph), b in per_pr.items() if ph in _CI_PHASES)
total_attempts = sum(b["attempts"] for b in per_pr.values())
if group_by == "phase":
by_phase: dict[str, dict[str, Any]] = {}
prs_seen_per_phase: dict[str, set[int | None]] = {}
for (pr, phase), b in per_pr.items():
agg = by_phase.setdefault(
phase,
{"phase": phase, "seconds": 0.0, "attempts": 0},
)
agg["seconds"] += b["seconds"]
agg["attempts"] += b["attempts"]
prs_seen_per_phase.setdefault(phase, set()).add(pr)
rows = []
for phase, agg in by_phase.items():
agg["pr_count"] = len(prs_seen_per_phase[phase])
agg["kind"] = "ci_job_time" if phase in _CI_PHASES else "wallclock"
# Share is wallclock-among-wallclock; CI is left as None because
# it's a CPU/job-time figure that isn't comparable to worker
# wallclock and would mislead at the same scale.
if phase in _CI_PHASES or not worker_seconds:
agg["share"] = None
else:
agg["share"] = round(100.0 * agg["seconds"] / worker_seconds, 1)
rows.append(agg)
rows.sort(key=lambda r: -r["seconds"])
return {
"days": days,
"group_by": "phase",
"rows": rows,
"totals": {
"seconds": round(worker_seconds, 1),
"ci_seconds": round(ci_seconds, 1),
"attempts": total_attempts,
},
"note": "; ".join(note_parts) or None,
"instrumentation_pending": False,
}
by_pr_num: dict[int | None, dict[str, Any]] = {}
for (pr, phase), b in per_pr.items():
entry = by_pr_num.setdefault(
pr,
{
"pr_number": pr,
"phases": {},
"seconds": 0.0,
"ci_seconds": 0.0,
"attempts": 0,
},
)
entry["phases"][phase] = round(b["seconds"], 1)
if phase in _CI_PHASES:
entry["ci_seconds"] += b["seconds"]
else:
entry["seconds"] += b["seconds"]
entry["attempts"] += b["attempts"]
rows = []
for entry in by_pr_num.values():
entry["seconds"] = round(entry["seconds"], 1)
entry["ci_seconds"] = round(entry["ci_seconds"], 1)
rows.append(entry)
# Unattributed (pr_number IS NULL) sinks to the bottom regardless
# of its cost — keeps real PRs at the top of the table.
rows.sort(
key=lambda r: (
0 if r["pr_number"] is not None else 1,
-(r["seconds"] or 0.0),
)
)
return {
"days": days,
"group_by": "pr",
"rows": rows,
"totals": {
"seconds": round(worker_seconds, 1),
"ci_seconds": round(ci_seconds, 1),
"attempts": total_attempts,
},
"note": "; ".join(note_parts) or None,
"instrumentation_pending": False,
}
# ─── Controller workflows (state machine inventory + timeline) ──────────
# States the operator needs to act on. STUCK is terminal-failure; the
# rest are non-terminal-but-blocked.
_ATTENTION_STATES = ("STUCK", "OPERATOR_ATTENTION", "PAUSED")
def _api_attention() -> dict[str, Any]:
"""Workflows that need operator eyes + recently reaped attempts.
Both signals come from the controller DB; legacy cache is not used."""
engine = _open_controller_db_engine()
if engine is None:
return {
"by_state": [],
"reaped_recent": [],
"note": "CLEVERAGENTS_DB_URL not set (controller backlog unavailable)",
"instrumentation_pending": True,
}
cutoff = datetime.now(timezone.utc) - timedelta(hours=24)
try:
with engine.connect() as conn:
state_rows = conn.execute(
_sa_text(
"SELECT current_state, COUNT(*) AS n "
" FROM workflows "
" WHERE owner = :owner AND repo = :repo "
" AND current_state IN :states "
" GROUP BY current_state"
).bindparams(_sa_bindparam("states", expanding=True)),
{
"owner": REPO_OWNER,
"repo": REPO_NAME,
"states": list(_ATTENTION_STATES),
},
).all()
entries = conn.execute(
_sa_text(
"SELECT entity_number AS pr, current_state AS state, "
" entered_state_at, last_transition_at "
" FROM workflows "
" WHERE owner = :owner AND repo = :repo "
" AND kind = 'pr' "
" AND current_state IN :states "
" ORDER BY entered_state_at ASC"
).bindparams(_sa_bindparam("states", expanding=True)),
{
"owner": REPO_OWNER,
"repo": REPO_NAME,
"states": list(_ATTENTION_STATES),
},
).all()
reaped = conn.execute(
_sa_text(
"SELECT a.attempt_id, a.workflow_id, a.role, "
" a.pickup_count, a.lock_heartbeat_at, "
" w.entity_number AS pr_number "
" FROM workflow_attempts a "
" JOIN workflows w ON w.workflow_id = a.workflow_id "
" WHERE w.owner = :owner AND w.repo = :repo "
" AND a.status = 'reaped' "
" AND (a.lock_heartbeat_at > :cutoff "
" OR a.locked_at > :cutoff) "
" ORDER BY COALESCE(a.lock_heartbeat_at, a.locked_at) DESC "
" LIMIT 50"
),
{"owner": REPO_OWNER, "repo": REPO_NAME, "cutoff": cutoff},
).all()
except Exception as exc: # noqa: BLE001
return {
"by_state": [],
"reaped_recent": [],
"note": _controller_db_note(exc),
"instrumentation_pending": True,
}
return {
"by_state": [{"state": r.current_state, "n": int(r.n)} for r in state_rows],
"entries": [
{
"pr_number": r.pr,
"state": r.state,
"entered_state_at": _iso(r.entered_state_at),
"last_transition_at": _iso(r.last_transition_at),
}
for r in entries
],
"reaped_recent": [
{
"attempt_id": int(r.attempt_id),
"workflow_id": int(r.workflow_id),
"pr_number": int(r.pr_number) if r.pr_number is not None else None,
"role": r.role,
"pickup_count": int(r.pickup_count or 0),
"lock_heartbeat_at": _iso(r.lock_heartbeat_at),
}
for r in reaped
],
}
def _api_workflows(state: str | None) -> dict[str, Any]:
"""All PR workflows for this repo, optionally filtered by ``state``.
Joined with an attempt summary so the table can show counts +
last activity per workflow."""
engine = _open_controller_db_engine()
if engine is None:
return {
"rows": [],
"states": [],
"note": "CLEVERAGENTS_DB_URL not set",
"instrumentation_pending": True,
}
where = ["w.owner = :owner", "w.repo = :repo", "w.kind = 'pr'"]
params: dict[str, Any] = {"owner": REPO_OWNER, "repo": REPO_NAME}
if state:
where.append("w.current_state = :state")
params["state"] = state
try:
with engine.connect() as conn:
rows = conn.execute(
_sa_text(
"SELECT w.workflow_id, w.entity_number AS pr_number, "
" w.current_state, w.current_tier, "
" w.started_at, w.last_transition_at, "
" w.entered_state_at, w.pre_pause_state, "
" COUNT(a.attempt_id) AS attempts_total, "
" SUM(CASE WHEN a.status = 'in_progress' THEN 1 ELSE 0 END) AS attempts_in_progress, "
" SUM(CASE WHEN a.status = 'reaped' THEN 1 ELSE 0 END) AS attempts_reaped "
" FROM workflows w "
" LEFT JOIN workflow_attempts a ON a.workflow_id = w.workflow_id "
f" WHERE {' AND '.join(where)} "
" GROUP BY w.workflow_id, w.entity_number, w.current_state, "
" w.current_tier, w.started_at, w.last_transition_at, "
" w.entered_state_at, w.pre_pause_state "
" ORDER BY w.last_transition_at DESC"
),
params,
).all()
state_counts = conn.execute(
_sa_text(
"SELECT current_state, COUNT(*) AS n "
" FROM workflows "
" WHERE owner = :owner AND repo = :repo AND kind = 'pr' "
" GROUP BY current_state"
),
{"owner": REPO_OWNER, "repo": REPO_NAME},
).all()
except Exception as exc: # noqa: BLE001
return {
"rows": [],
"states": [],
"note": _controller_db_note(exc),
"instrumentation_pending": True,
}
return {
"rows": [
{
"workflow_id": int(r.workflow_id),
"pr_number": int(r.pr_number) if r.pr_number is not None else None,
"current_state": r.current_state,
"current_tier": r.current_tier,
"started_at": _iso(r.started_at),
"last_transition_at": _iso(r.last_transition_at),
"entered_state_at": _iso(r.entered_state_at),
"pre_pause_state": r.pre_pause_state,
"attempts_total": int(r.attempts_total or 0),
"attempts_in_progress": int(r.attempts_in_progress or 0),
"attempts_reaped": int(r.attempts_reaped or 0),
}
for r in rows
],
"states": [{"state": r.current_state, "n": int(r.n)} for r in state_counts],
"filter": state,
}
def _api_workflow_events(workflow_id: int) -> dict[str, Any]:
"""State-transition timeline for a single workflow, oldest first.
Attempts are unioned in so the operator sees the work that happened
between transitions."""
engine = _open_controller_db_engine()
if engine is None:
return {
"workflow": None,
"events": [],
"attempts": [],
"note": "CLEVERAGENTS_DB_URL not set",
}
try:
with engine.connect() as conn:
wf = conn.execute(
_sa_text(
"SELECT workflow_id, kind, owner, repo, entity_number, "
" current_state, started_at "
" FROM workflows WHERE workflow_id = :wid"
),
{"wid": workflow_id},
).first()
if wf is None:
return {
"workflow": None,
"events": [],
"attempts": [],
"note": f"workflow_id={workflow_id} not found",
}
if wf.owner != REPO_OWNER or wf.repo != REPO_NAME:
return {
"workflow": None,
"events": [],
"attempts": [],
"note": "workflow belongs to a different repo",
}
events = conn.execute(
_sa_text(
"SELECT event_id, ts, event_type, from_state, to_state, "
" attempt_id "
" FROM controller_events "
" WHERE workflow_id = :wid "
" ORDER BY ts ASC"
),
{"wid": workflow_id},
).all()
attempts = conn.execute(
_sa_text(
"SELECT attempt_id, attempt_number, role, status, "
" outcome, started_at, finished_at, "
" wallclock_seconds, pickup_count, session_id, "
" input_payload, output_payload, output_version, "
" input_payload_truncated "
" FROM workflow_attempts "
" WHERE workflow_id = :wid "
" ORDER BY attempt_number ASC"
),
{"wid": workflow_id},
).all()
except Exception as exc: # noqa: BLE001
return {
"workflow": None,
"events": [],
"attempts": [],
"note": _controller_db_note(exc),
}
return {
"workflow": {
"workflow_id": int(wf.workflow_id),
"pr_number": int(wf.entity_number)
if wf.entity_number is not None
else None,
"kind": wf.kind,
"current_state": wf.current_state,
"started_at": _iso(wf.started_at),
},
"events": [
{
"event_id": int(e.event_id),
"ts": _iso(e.ts),
"event_type": e.event_type,
"from_state": e.from_state,
"to_state": e.to_state,
"attempt_id": int(e.attempt_id) if e.attempt_id is not None else None,
}
for e in events
],
"attempts": [
{
"attempt_id": int(a.attempt_id),
"attempt_number": int(a.attempt_number),
"role": a.role,
"status": a.status,
"outcome": a.outcome,
"started_at": _iso(a.started_at),
"finished_at": _iso(a.finished_at),
"wallclock_seconds": (
float(a.wallclock_seconds)
if a.wallclock_seconds is not None
else None
),
"pickup_count": int(a.pickup_count or 0),
"session_id": a.session_id,
"output_version": a.output_version,
"input_payload_truncated": bool(a.input_payload_truncated),
"input_payload": _coerce_json(a.input_payload),
"output_payload": _coerce_json(a.output_payload),
}
for a in attempts
],
}
# ─── Live (controller DB state + live OpenCode session tree) ────────────
#
# The Live tab is rebuilt on two LIVE sources rather than the retired
# ``live_log_writer.py`` sidecar (which tailed the legacy dispatchers and
# is no longer launched by the controller pipeline):
#
# - the controller DB (``workflows`` / ``workflow_attempts``) for
# run-level state — which PRs are active, which attempts are
# in-progress and on which worker;
# - OpenCode ``GET /session`` for the live session forest, so an
# operator can drill into a running worker session and walk down
# into the ``task``-tool subagents it spawned.
#
# Both sources are already reachable from this read-only server, so the
# Live tab needs no extra producer process.
# Controller workflow states that are terminal — mirrors the
# ``ix_workflows_active`` filtered index in the controller's models.py.
_TERMINAL_WORKFLOW_STATES = ("MERGED", "ABANDONED", "STUCK", "CREATED_PR")
# A live OpenCode session is one that is busy now OR was updated within
# this window. Keeps stale idle sessions from past runs out of the tree.
_LIVE_SESSION_WINDOW_S = float(
os.environ.get("TELEMETRY_LIVE_SESSION_WINDOW_S", "3600")
)
# OpenCode worker-session title shape: ``[<tag>] <agent-name>`` — the
# worker passes the dispatch tag, OpenCode brackets it and appends the
# agent name. ``task``-tool subagent sessions instead get an
# OpenCode-auto-generated free-text title (no brackets).
_TITLE_BRACKET_RE = re.compile(r"^\[([^\]]+)\]\s*(.*)$")
# Controller worker-session tag parsing (role / attempt_id / pr_number
# from ``controller-<role>-<attempt_id>[-pr-<n>]``) lives in the
# controller package so the telemetry server, the llm_activity backfill,
# and the worker that BUILDS the tag share one definition of the shape.
if str(REPO_ROOT) not in sys.path:
sys.path.insert(0, str(REPO_ROOT))
from tools.controller.session_tag import ( # noqa: E402
parse_controller_tag as _parse_controller_tag,
)
def _parse_session_title(title: str | None) -> dict[str, Any]:
"""Parse an OpenCode worker-session title ``[<tag>] <agent-name>``.
For a controller worker session the tag is
``controller-<role>-<attempt_id>[-pr-<n>]``, so role / attempt_id /
pr_number are filled too. A ``task``-tool subagent session that
OpenCode auto-titled (free-text, no brackets) parses to all-None."""
out: dict[str, Any] = {
"tag": None,
"agent": None,
"role": None,
"attempt_id": None,
"pr_number": None,
}
if not isinstance(title, str):
return out
m = _TITLE_BRACKET_RE.match(title.strip())
if not m:
return out
tag, agent = m.group(1).strip(), m.group(2).strip()
out["tag"] = tag
out["agent"] = agent or None
out.update(_parse_controller_tag(tag))
return out
def _age_seconds(value: Any) -> float | None:
"""Seconds between ``value`` (a DB datetime or ISO string) and now.
None for missing / unparseable values; naive datetimes read as UTC."""
if value is None:
return None
dt: datetime | None = None
if isinstance(value, datetime):
dt = value
elif isinstance(value, str):
try:
dt = datetime.fromisoformat(value.replace("Z", "+00:00"))
except ValueError:
return None
if dt is None:
return None
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
return round((datetime.now(timezone.utc) - dt).total_seconds(), 1)
def _live_controller_state() -> dict[str, Any]:
"""Run-level live state from the controller DB: active (non-terminal)
workflows + the attempts currently in progress."""
engine = _open_controller_db_engine()
empty = {
"available": False,
"note": "CLEVERAGENTS_DB_URL not set",
"active_workflows": [],
"in_progress_attempts": [],
"state_counts": [],
}
if engine is None:
return empty
params: dict[str, Any] = {"owner": REPO_OWNER, "repo": REPO_NAME}
# Bind the terminal-state list positionally so the query stays
# portable across SQLite + Postgres.
term = {f"t{i}": s for i, s in enumerate(_TERMINAL_WORKFLOW_STATES)}
term_ph = ", ".join(f":{k}" for k in term)
try:
with engine.connect() as conn:
workflows = conn.execute(
_sa_text(
"SELECT w.workflow_id, w.entity_number AS pr_number, "
" w.kind, w.current_state, w.current_tier, "
" w.entered_state_at, "
" SUM(CASE WHEN a.status='in_progress' THEN 1 ELSE 0 END) "
" AS in_prog "
" FROM workflows w "
" LEFT JOIN workflow_attempts a "
" ON a.workflow_id = w.workflow_id "
" WHERE w.owner=:owner AND w.repo=:repo "
f" AND w.current_state NOT IN ({term_ph}) "
" GROUP BY w.workflow_id, w.entity_number, w.kind, "
" w.current_state, w.current_tier, "
" w.entered_state_at "
" ORDER BY w.entered_state_at DESC"
),
{**params, **term},
).all()
attempts = conn.execute(
_sa_text(
"SELECT a.attempt_id, a.workflow_id, a.role, a.tier, "
" a.started_at, a.pickup_count, "
" a.locked_by_instance, a.lock_heartbeat_at, "
" w.entity_number AS pr_number, w.current_state "
" FROM workflow_attempts a "
" JOIN workflows w ON a.workflow_id = w.workflow_id "
" WHERE w.owner=:owner AND w.repo=:repo "
" AND a.status='in_progress' "
" ORDER BY a.started_at ASC"
),
params,
).all()
state_counts = conn.execute(
_sa_text(
"SELECT current_state, COUNT(*) AS n FROM workflows "
" WHERE owner=:owner AND repo=:repo AND kind='pr' "
" GROUP BY current_state"
),
params,
).all()
except Exception as exc: # noqa: BLE001 — defensive: never 500
return {**empty, "note": _controller_db_note(exc)}
return {
"available": True,
"note": None,
"active_workflows": [
{
"workflow_id": int(w.workflow_id),
"pr_number": (int(w.pr_number) if w.pr_number is not None else None),
"kind": w.kind,
"current_state": w.current_state,
"current_tier": w.current_tier,
"entered_state_at": _iso(w.entered_state_at),
"entered_state_s_ago": _age_seconds(w.entered_state_at),
"attempts_in_progress": int(w.in_prog or 0),
}
for w in workflows
],
"in_progress_attempts": [
{
"attempt_id": int(a.attempt_id),
"workflow_id": int(a.workflow_id),
"pr_number": (int(a.pr_number) if a.pr_number is not None else None),
"role": a.role,
"tier": a.tier,
"current_state": a.current_state,
"started_at": _iso(a.started_at),
"started_s_ago": _age_seconds(a.started_at),
"pickup_count": int(a.pickup_count or 0),
"locked_by_instance": a.locked_by_instance,
"heartbeat_s_ago": _age_seconds(a.lock_heartbeat_at),
}
for a in attempts
],
"state_counts": [
{"state": r.current_state, "n": int(r.n)} for r in state_counts
],
}
def _api_live() -> dict[str, Any]:
"""Live tab payload: controller run state + the live OpenCode session
forest (root worker sessions + their ``task``-tool subagents)."""
raw_sessions = _opencode_get("/session")
opencode_reachable = isinstance(raw_sessions, list)
raw_status = _opencode_get("/session/status") if opencode_reachable else None
status_map = raw_status if isinstance(raw_status, dict) else {}
sessions: list[dict[str, Any]] = []
if opencode_reachable:
by_id: dict[str, dict[str, Any]] = {
s["id"]: s for s in raw_sessions if isinstance(s, dict) and s.get("id")
}
def _depth(sid: str) -> int:
"""Distance to the root via the ``parentID`` chain; cycle- and
missing-parent-safe."""
seen: set[str] = set()
depth = 0
cur = by_id.get(sid)
while cur is not None:
pid = cur.get("parentID")
if not pid or pid in seen or pid not in by_id:
break
seen.add(pid)
depth += 1
cur = by_id.get(pid)
return depth
now_ms = int(time.time() * 1000)
for s in by_id.values():
sid = s["id"]
time_obj = s.get("time") if isinstance(s.get("time"), dict) else {}
created = time_obj.get("created")
updated = time_obj.get("updated")
updated_s_ago = round((now_ms - updated) / 1000, 1) if updated else None
entry = status_map.get(sid)
busy = isinstance(entry, dict) and entry.get("type") == "busy"
# Live = busy now, or touched within the window. Stale idle
# sessions from earlier runs are dropped so the tree only
# shows what is actually happening.
if not busy and (
updated_s_ago is None or updated_s_ago > _LIVE_SESSION_WINDOW_S
):
continue
parent = s.get("parentID") or None
title = s.get("title")
ti = _parse_session_title(title)
sessions.append(
{
"session_id": sid,
"parent_session_id": parent,
"title": title,
"tag": ti["tag"],
"agent": ti["agent"] or s.get("agent"),
"depth": _depth(sid),
"is_root": not parent or parent not in by_id,
"role": ti["role"],
"attempt_id": ti["attempt_id"],
"pr_number": ti["pr_number"],
"created_s_ago": (
round((now_ms - created) / 1000, 1) if created else None
),
"updated_s_ago": updated_s_ago,
"status": "busy" if busy else "idle",
}
)
# Oldest first so root worker sessions render top-to-bottom in
# spawn order (the UI re-sorts within each parent anyway).
sessions.sort(key=lambda r: -(r["created_s_ago"] or 0.0))
return {
"opencode_url": OPENCODE_URL,
"opencode_reachable": opencode_reachable,
"session_window_s": _LIVE_SESSION_WINDOW_S,
"sessions": sessions,
"controller": _live_controller_state(),
"server_time_utc": datetime.now(timezone.utc).isoformat(),
}
def _api_live_session(session_id: str | None) -> tuple[int, dict[str, Any]]:
"""Live transcript for one OpenCode session — the Live tab's
drill-down. OpenCode-first (this IS the live view); falls back to the
on-disk archive when OpenCode has already deleted a just-finished
session so it stays inspectable for a while after it ends."""
if not session_id or not session_id.startswith("ses_") or "/" in session_id:
return 400, {"error": "missing or invalid 'session' query parameter"}
info = _opencode_get(f"/session/{session_id}")
messages = _opencode_get(f"/session/{session_id}/message")
if isinstance(info, dict) or isinstance(messages, list):
info = info if isinstance(info, dict) else {}
msgs = messages if isinstance(messages, list) else []
title = info.get("title")
ti = _parse_session_title(title)
time_obj = info.get("time") if isinstance(info.get("time"), dict) else {}
entry = _opencode_get("/session/status") or {}
busy = (
isinstance(entry, dict)
and isinstance(entry.get(session_id), dict)
and entry[session_id].get("type") == "busy"
)
return 200, {
"session_id": session_id,
"source": "live-opencode",
"status": "busy" if busy else "idle",
"title": title,
"tag": ti["tag"],
"agent": info.get("agent") or ti["agent"],
"role": ti["role"],
"pr_number": ti["pr_number"],
"parent_session_id": info.get("parentID"),
"started_at": time_obj.get("created"),
"ended_at": None,
"attempt": _attempt_payloads(ti["attempt_id"]),
"turns": _normalize_messages(msgs),
}
# OpenCode has no live session by that id — try the on-disk archive
# so a session that finished moments ago is still inspectable.
archive_path = _archive_index_by_session_id().get(session_id)
if archive_path is not None and archive_path.is_file():
try:
data = json.loads(archive_path.read_text(encoding="utf-8"))
except (OSError, ValueError) as e:
return 500, {"error": str(e), "session_id": session_id}
if isinstance(data, dict):
tag_info = _parse_controller_tag(data.get("tag"))
return 200, {
"session_id": session_id,
"source": "archive",
"status": data.get("status"),
"title": data.get("tag"),
"tag": data.get("tag"),
"agent": data.get("agent"),
"role": tag_info["role"],
"pr_number": tag_info["pr_number"],
"parent_session_id": data.get("parent_session_id"),
"started_at": data.get("started_at"),
"ended_at": data.get("archived_at"),
"attempt": _attempt_payloads(tag_info["attempt_id"]),
"turns": _normalize_messages(data.get("messages") or []),
}
return 503, {
"error": (
f"OpenCode at {OPENCODE_URL} has no live session {session_id} "
f"and no archive for it is on disk — transcript unavailable"
),
"session_id": session_id,
}
# ─── HTTP server ─────────────────────────────────────────────────────────
_API_DISPATCH = {
"/api/meta": lambda q: _api_meta(),
"/api/health": lambda q: _api_health(),
"/api/cycles": lambda q: _api_cycles(
q.get("driver", ["merge"])[0],
int(q.get("limit", ["20"])[0]),
),
"/api/prs": lambda q: _api_prs(q.get("state", [None])[0]),
"/api/attention": lambda q: _api_attention(),
"/api/workflows": lambda q: _api_workflows(q.get("state", [None])[0]),
"/api/workflow_events": lambda q: _api_workflow_events(
int(q.get("workflow_id", ["0"])[0] or 0),
),
"/api/live": lambda q: _api_live(),
"/api/velocity": lambda q: _api_velocity(
q.get("window", ["48h"])[0],
),
"/api/sessions": lambda q: _api_sessions(),
"/api/sessions/archived": lambda q: _api_archived_sessions(),
"/api/sessions/archived/detail": lambda q: _api_archived_session_detail(
q.get("filename", [None])[0],
),
"/api/cost": lambda q: _api_cost(
int(q.get("days", ["7"])[0]),
group_by=q.get("group_by", ["model"])[0],
),
"/api/phases": lambda q: _api_phases(
int(q.get("days", ["7"])[0]),
group_by=q.get("group_by", ["phase"])[0],
),
}
# Endpoints that return (status_code, payload) rather than a bare payload,
# so they can signal 503/500 to the UI without raising. Kept separate from
# ``_API_DISPATCH`` so the simple-case handlers stay simple.
_API_DISPATCH_WITH_STATUS = {
"/runs": lambda q: _api_runs(),
"/snapshot": lambda q: _api_snapshot(q.get("run", [None])[0]),
"/events": lambda q: _api_events(
q.get("run", [None])[0],
int(q.get("since", ["0"])[0] or 0),
int(q.get("limit", ["500"])[0] or 500),
(
{t.strip() for t in q["types"][0].split(",") if t.strip()}
if q.get("types")
else None
),
),
"/api/run/sessions": lambda q: _api_run_sessions(
q.get("run", [None])[0],
),
"/api/run/session": lambda q: _api_run_session(
q.get("run", [None])[0],
q.get("session", [None])[0],
),
"/api/live/session": lambda q: _api_live_session(
q.get("session", [None])[0],
),
}
_STATIC_FILES = {
"/": ("index.html", "text/html; charset=utf-8"),
"/index.html": ("index.html", "text/html; charset=utf-8"),
"/static/app.js": ("app.js", "application/javascript; charset=utf-8"),
"/static/style.css": ("style.css", "text/css; charset=utf-8"),
}
class _Handler(BaseHTTPRequestHandler):
server_version = "TelemetryConsole/0.1"
def log_message(self, fmt: str, *args: Any) -> None:
# Silence the default per-request stderr noise; only log warnings.
return
def _send_json(self, status: int, payload: Any) -> None:
body = json.dumps(payload, default=str).encode("utf-8")
self.send_response(status)
self.send_header("Content-Type", "application/json; charset=utf-8")
self.send_header("Content-Length", str(len(body)))
self.send_header("Cache-Control", "no-store")
self.end_headers()
self.wfile.write(body)
def _send_static(self, fname: str, content_type: str) -> None:
path = STATIC_DIR / fname
if not path.is_file():
self.send_error(404, f"missing static file: {fname}")
return
body = path.read_bytes()
self.send_response(200)
self.send_header("Content-Type", content_type)
self.send_header("Content-Length", str(len(body)))
self.send_header("Cache-Control", "no-store")
self.end_headers()
self.wfile.write(body)
def do_GET(self) -> None: # noqa: N802 (BaseHTTPRequestHandler API)
parsed = urllib.parse.urlparse(self.path)
route = parsed.path
query = urllib.parse.parse_qs(parsed.query)
if route in _STATIC_FILES:
fname, ctype = _STATIC_FILES[route]
self._send_static(fname, ctype)
return
handler = _API_DISPATCH.get(route)
if handler is not None:
try:
payload = handler(query)
self._send_json(200, payload)
except Exception as e: # noqa: BLE001 — keep server up
logger.exception("handler %s failed", route)
self._send_json(500, {"error": str(e), "route": route})
return
handler_with_status = _API_DISPATCH_WITH_STATUS.get(route)
if handler_with_status is not None:
try:
status, payload = handler_with_status(query)
self._send_json(status, payload)
except Exception as e: # noqa: BLE001
logger.exception("handler %s failed", route)
self._send_json(500, {"error": str(e), "route": route})
return
self.send_error(404, f"unknown route: {route}")
# ─── Background: llm_activity archive scraper ────────────────────────────
# The Cost tab reads the ``llm_activity`` table, filled by
# tools/llm_activity_scraper.py from OpenCode session archives.
#
# The scraper used to ride inside tools/pr_state_warmer.py's poll loop.
# The controller-state-machine pipeline retired the warmer (the controller
# owns PR state now) and silently dropped the scraper with it — costs
# froze. The telemetry console is the table's ONLY consumer, so it now
# hosts the scraper itself: console up ⇒ Cost tab stays current.
#
# Run as a subprocess on a timer, not in-process, so a scraper fault is
# fully isolated from the HTTP server. Every failure is logged + swallowed.
_SCRAPER_SCRIPT = REPO_ROOT / "tools" / "llm_activity_scraper.py"
_SCRAPE_DISABLE_ENV = "TELEMETRY_SCRAPE_DISABLE"
_SCRAPE_TIMEOUT_S = 600
try:
_SCRAPE_INTERVAL_S = max(
10, int(os.environ.get("TELEMETRY_SCRAPE_INTERVAL_S", "60"))
)
except ValueError:
_SCRAPE_INTERVAL_S = 60
def _log_scrape_summary(stdout: str) -> None:
"""The scraper prints a JSON summary as its last stdout line; log a
one-liner only when it actually ingested turns (no steady-state spam)."""
last = next((ln for ln in reversed(stdout.splitlines()) if ln.strip()), "")
try:
summary = json.loads(last)
except (ValueError, TypeError):
return
if isinstance(summary, dict) and summary.get("turns_inserted"):
logger.info(
"llm_activity scrape: %s archives, +%s turns, %s dedup",
summary.get("archives_scanned", 0),
summary["turns_inserted"],
summary.get("turns_duplicate", 0),
)
def _run_archive_scraper() -> None:
"""Background loop: ingest OpenCode session archives into the
``llm_activity`` table so the Cost tab stays current.
The first pass scans every archive (back-fills anything missed while
the console was down); later passes pass ``--since-hours`` to bound
the file walk. The scraper is dedup-safe (UNIQUE on ``message_id``),
so a wide window only costs JSON re-parses, never duplicate rows. The
subprocess inherits this process's env — including ``FORGEJO_OWNER``/
``FORGEJO_REPO`` — so it writes the SAME per-repo cache file the
console reads.
"""
if os.environ.get(_SCRAPE_DISABLE_ENV) == "1":
logger.info("llm_activity scraper disabled (%s=1)", _SCRAPE_DISABLE_ENV)
return
if not _SCRAPER_SCRIPT.exists():
logger.warning(
"llm_activity scraper not found at %s — Cost tab will go stale",
_SCRAPER_SCRIPT,
)
return
backfilled = False
while True:
cmd = [sys.executable, str(_SCRAPER_SCRIPT)]
archive_dir = os.environ.get("OPENCODE_WORKER_ARCHIVE_DIR")
if archive_dir:
cmd += ["--archive-dir", archive_dir]
if backfilled:
# Steady state: only stat archives touched recently. The
# window is wider than the interval on purpose — dedup makes
# the overlap free and it tolerates clock skew.
cmd += ["--since-hours", "1"]
try:
proc = subprocess.run(
cmd, capture_output=True, text=True, timeout=_SCRAPE_TIMEOUT_S
)
if proc.returncode != 0:
logger.warning(
"llm_activity scrape exited %d: %s",
proc.returncode,
(proc.stderr or proc.stdout).strip()[-400:],
)
else:
backfilled = True
_log_scrape_summary(proc.stdout)
except subprocess.TimeoutExpired:
logger.warning(
"llm_activity scrape timed out after %ds", _SCRAPE_TIMEOUT_S
)
except Exception as exc: # never let the scraper kill the console
logger.warning(
"llm_activity scrape failed (continuing): %s: %s",
type(exc).__name__,
exc,
)
time.sleep(_SCRAPE_INTERVAL_S)
def _install_signal_handlers(httpd: ThreadingHTTPServer) -> None:
def _shutdown(signum: int, _frame: Any) -> None:
logger.info("received signal %d, shutting down", signum)
# shutdown() must not run in the request thread; spawn a thread
# via a separate signal-safe path.
import threading
threading.Thread(target=httpd.shutdown, daemon=True).start()
for sig in (signal.SIGINT, signal.SIGTERM):
try:
signal.signal(sig, _shutdown)
except (OSError, ValueError):
pass
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
description="auto-agents pipeline telemetry console",
)
parser.add_argument(
"--host",
default=os.environ.get("TELEMETRY_HOST", "127.0.0.1"),
help=(
"interface to bind (default 127.0.0.1 — loopback only). "
"Set 0.0.0.0 ONLY when you understand v1 ships without auth."
),
)
parser.add_argument(
"--port",
type=int,
default=int(os.environ.get("TELEMETRY_PORT", "8765")),
help="TCP port to bind (default 8765)",
)
parser.add_argument(
"--log-level",
default="INFO",
choices=("DEBUG", "INFO", "WARNING", "ERROR"),
)
args = parser.parse_args(argv)
logging.basicConfig(
level=args.log_level,
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
)
logger.info("repo target: %s/%s", REPO_OWNER, REPO_NAME)
logger.info("api base: %s", API_BASE)
logger.info(
"cache file: %s (%s)",
CACHE_PATH,
"exists" if CACHE_PATH.exists() else "ABSENT",
)
logger.info("opencode url: %s", OPENCODE_URL)
logger.info(
"forgejo token: %s",
"present" if FORGEJO_TOKEN else "MISSING (PR tab disabled)",
)
try:
httpd = ThreadingHTTPServer((args.host, args.port), _Handler)
except OSError as e:
print(
f"ERROR: bind {args.host}:{args.port} failed: {e}",
file=sys.stderr,
)
return 2
_install_signal_handlers(httpd)
scraper_thread = threading.Thread(
target=_run_archive_scraper, name="llm-activity-scraper", daemon=True
)
scraper_thread.start()
logger.info(
"llm_activity scraper: ingesting every %ss (set %s=1 to disable)",
_SCRAPE_INTERVAL_S,
_SCRAPE_DISABLE_ENV,
)
print(
f"Telemetry console running at http://{args.host}:{args.port}/ (Ctrl+C to stop)"
)
try:
httpd.serve_forever()
finally:
httpd.server_close()
return 0
if __name__ == "__main__":
raise SystemExit(main())
Reference in New Issue View Git Blame Copy Permalink