62d4e8f07d
Three intertwined bugs caused every Cost-tab row to display \$0 even after the scraper started writing real token data: 1. **Lookup key mismatch.** ``_cost_usd`` looked up bare ``model`` but ``_DEFAULT_PRICES`` was keyed by ``provider/model`` — every priced model silently missed. Fixed by adding ``_price_key`` and a fallback chain: ``provider/model`` → bare ``model`` → ``_unknown``. 2. **SQL grouped by model only.** Same modelID served by two providers (e.g. ``claude-opus-4-6`` via Anthropic direct vs a local proxy) at different rates was conflated into one row. Fixed: ``GROUP BY model, provider`` in ``_api_cost`` + ``provider`` returned in each row. 3. **Math convention mismatch.** ``_cost_usd`` did ``(tokens_in - cached) * in_rate`` assuming ``tokens_in`` was total input. But the scraper records ``tokens_in`` as OpenCode's ``info.tokens.input`` (fresh, non-cached), so ``tokens_in - cached`` went negative whenever cache reads exceeded fresh input — which is the common case with Anthropic prompt caching. Fixed: no subtraction; the three populations bill at their three rates. Pricing seeded for the 8 models the scraper has actually observed (``_DEFAULT_PRICES`` corrected from stale Opus-3 numbers + new entries for the Haiku 4.5 / GPT-5 family / CleverThis HF endpoints): | Provider | Model | in | out | cached_in | |--------------|---------------------------|-------|-------|-----------| | local-claude | claude-opus-4-6 | 5.00 | 25.00 | 0.50 | | local-claude | claude-sonnet-4-6 | 3.00 | 15.00 | 0.30 | | local-claude | claude-haiku-4-5 | 1.00 | 5.00 | 0.10 | | openai | gpt-5 / gpt-5-codex | 1.25 | 10.00 | 0.125 | | openai | gpt-5-mini | 0.25 | 2.00 | 0.025 | | openai | gpt-5-nano | 0.05 | 0.40 | 0.005 | | CleverThis-* | (HF endpoints, advisory) | 0.50 | 1.00 | — | Operators can override without touching code via ``.opencode/telemetry/prices.json`` (added; same keying convention). ``_load_prices`` now skips ``_comment`` / ``_last_updated`` / ``_sources`` metadata keys so docs in the JSON don't pollute the table. Smoke against current ``llm_activity`` (3743 turns, 450 archives): total USD over the lifetime window is now \$118.06, with all 8 models showing as priced. Tests pin all three regressions: provider-qualified lookup, bare-model fallback, no-subtraction math, GROUP BY (model, provider), and the metadata-key filter in ``_load_prices``. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2060 lines
82 KiB
Python
2060 lines
82 KiB
Python
#!/usr/bin/env python3
|
|
"""Pipeline telemetry console — a small, dependency-free HTTP server that
|
|
exposes everything you need to monitor the auto-agents pipeline in one
|
|
browser tab.
|
|
|
|
Why this exists
|
|
---------------
|
|
|
|
The auto-agents pipeline emits useful telemetry into several silos:
|
|
|
|
- ``tools/.cache/forgejo.<owner>.<repo>.sqlite`` — cached PR/commit data
|
|
+ ``merge_cycle`` / ``conflict_drive_cycles`` / ``llm_activity`` driver
|
|
telemetry rows.
|
|
- ``tools/<driver>.heartbeat`` (or ``/tmp/<driver>.heartbeat``) — per-driver
|
|
liveness probes touched once per cycle.
|
|
- ``ps``-discoverable Python daemons (``merge_drive``, ``conflict_drive``,
|
|
``verify_invariant``, ``opencode-builder.sh``).
|
|
- The OpenCode HTTP server (``http://127.0.0.1:4096`` by default) — agent
|
|
sessions, agent messages, idle/busy state.
|
|
- The Forgejo HTTP API — open PRs by ``auto/*`` label, branch protection,
|
|
approvals.
|
|
|
|
Tailing five terminals + running ``sqlite3`` ad-hoc + curl-ing two HTTP
|
|
APIs is fine for one operator; it does not scale to "let another dev
|
|
look at this without setting anything up". This console consolidates
|
|
all of the above into a single ``python3 .opencode/telemetry/server.py``
|
|
entry point with a tabbed web UI that auto-refreshes per pane.
|
|
|
|
Design constraints
|
|
------------------
|
|
|
|
- **Stdlib only.** No Flask / FastAPI / Jinja. Runs anywhere a driver
|
|
runs (the auto-agents pipeline already mandates Python 3.13+ stdlib;
|
|
matching that contract keeps the install footprint zero).
|
|
- **Repo-aware.** Reads ``FORGEJO_OWNER`` / ``FORGEJO_REPO`` and picks
|
|
the right cache file via the same partition logic as
|
|
``tools/_pipeline_cache.py``. The console banner spells out which
|
|
repo's data is on screen — never let an operator monitor canonical
|
|
while expecting fork data, or vice-versa.
|
|
- **Loopback-only by default.** Binds ``127.0.0.1``. Operators who
|
|
want LAN access set ``TELEMETRY_HOST=0.0.0.0`` deliberately AND
|
|
understand that v1 ships without auth (loopback is the security
|
|
model). README spells this out.
|
|
- **Per-tab polling.** Each tab fetches its own endpoints on its own
|
|
cadence; inactive tabs do not poll. Server load stays low even on
|
|
long-lived browser sessions.
|
|
|
|
Endpoints (all JSON unless noted)
|
|
---------------------------------
|
|
|
|
- ``GET /`` index.html (tabbed SPA)
|
|
- ``GET /static/<file>`` app.js / style.css
|
|
- ``GET /api/meta`` repo target, identity, paths in use
|
|
- ``GET /api/health`` daemon PIDs, heartbeats, OpenCode health
|
|
- ``GET /api/cycles?driver=...`` merge / conflict / dispatch_review /
|
|
dispatch_implementer cycle rows
|
|
- ``GET /api/prs?label=...`` open PRs by Forgejo label (live API)
|
|
- ``GET /api/velocity?window=...`` window stats from cache
|
|
- ``GET /api/sessions`` OpenCode sessions
|
|
- ``GET /api/cost?days=N`` token + estimated cost rollup
|
|
|
|
Run
|
|
---
|
|
|
|
::
|
|
|
|
source tools/launch_fork.sh # or canonical-mode env
|
|
python3 .opencode/telemetry/server.py # http://127.0.0.1:8765
|
|
|
|
Environment overrides
|
|
---------------------
|
|
|
|
- ``TELEMETRY_HOST`` (default ``127.0.0.1``)
|
|
- ``TELEMETRY_PORT`` (default ``8765``)
|
|
- ``OPENCODE_URL`` (default ``http://127.0.0.1:4096``)
|
|
- ``TELEMETRY_PRICES_JSON`` (default ``.opencode/telemetry/prices.json``;
|
|
used by /api/cost to convert tokens to dollars)
|
|
|
|
Exit codes
|
|
----------
|
|
|
|
::
|
|
|
|
0 — clean shutdown via Ctrl+C
|
|
2 — argument error / port-bind failure / missing cache file
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import json
|
|
import logging
|
|
import os
|
|
import re
|
|
import signal
|
|
import sqlite3
|
|
import subprocess
|
|
import sys
|
|
import threading
|
|
import time
|
|
import urllib.error
|
|
import urllib.parse
|
|
import urllib.request
|
|
from datetime import datetime, timezone
|
|
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
logger = logging.getLogger("telemetry")
|
|
|
|
# ─── Paths + repo-aware cache resolution ─────────────────────────────────
|
|
|
|
REPO_ROOT = Path(__file__).resolve().parents[2]
|
|
STATIC_DIR = Path(__file__).resolve().parent
|
|
CACHE_DIR = REPO_ROOT / "tools" / ".cache"
|
|
|
|
# Mirrors the partition logic in tools/_pipeline_cache.py so the console
|
|
# always reads the SAME file the drivers write to. Computed at import.
|
|
REPO_OWNER = os.environ.get("FORGEJO_OWNER", "cleveragents")
|
|
REPO_NAME = os.environ.get("FORGEJO_REPO", "cleveragents-core")
|
|
API_BASE = os.environ.get(
|
|
"FORGEJO_API_BASE", "https://git.cleverthis.com/api/v1"
|
|
).rstrip("/")
|
|
FORGEJO_TOKEN = os.environ.get("GITEA_TOKEN") or os.environ.get(
|
|
"FORGEJO_PAT", ""
|
|
)
|
|
OPENCODE_URL = os.environ.get("OPENCODE_URL", "http://127.0.0.1:4096").rstrip(
|
|
"/"
|
|
)
|
|
|
|
|
|
def _cache_path() -> Path:
|
|
"""Resolve the per-(owner, repo) cache file the same way
|
|
``tools/_pipeline_cache.py:DEFAULT_CACHE_PATH`` does. Console always
|
|
reads the SAME file the drivers wrote to."""
|
|
if REPO_OWNER == "cleveragents" and REPO_NAME == "cleveragents-core":
|
|
return CACHE_DIR / "forgejo.sqlite"
|
|
safe_owner = re.sub(r"[^a-zA-Z0-9._-]+", "-", REPO_OWNER)
|
|
safe_repo = re.sub(r"[^a-zA-Z0-9._-]+", "-", REPO_NAME)
|
|
return CACHE_DIR / f"forgejo.{safe_owner}.{safe_repo}.sqlite"
|
|
|
|
|
|
CACHE_PATH = _cache_path()
|
|
|
|
|
|
# ─── SQLite helpers ──────────────────────────────────────────────────────
|
|
|
|
|
|
def _open_db() -> sqlite3.Connection | None:
|
|
"""Open the per-repo cache read-only. Returns None when the file
|
|
does not exist yet (cold-start before any sync) so endpoints can
|
|
return empty payloads instead of 500ing."""
|
|
if not CACHE_PATH.exists():
|
|
return None
|
|
# mode=ro keeps us from racing with driver writers; the SQLite
|
|
# locking story tolerates many readers + one writer regardless,
|
|
# but ro makes the intent unambiguous.
|
|
uri = f"file:{CACHE_PATH}?mode=ro"
|
|
conn = sqlite3.connect(uri, uri=True, timeout=2)
|
|
conn.row_factory = sqlite3.Row
|
|
return conn
|
|
|
|
|
|
def _rows_to_dicts(rows: list[sqlite3.Row]) -> list[dict[str, Any]]:
|
|
return [dict(r) for r in rows]
|
|
|
|
|
|
# ─── Forgejo API (read-only) ─────────────────────────────────────────────
|
|
|
|
|
|
def _forgejo_get(path: str, params: dict[str, str] | None = None) -> Any:
|
|
"""GET ``API_BASE + path``. Returns parsed JSON or None on error.
|
|
Console never POSTs — read-only by design."""
|
|
if not FORGEJO_TOKEN:
|
|
return None
|
|
url = API_BASE + path
|
|
if params:
|
|
url += "?" + urllib.parse.urlencode(params)
|
|
req = urllib.request.Request(
|
|
url,
|
|
headers={
|
|
"Authorization": f"token {FORGEJO_TOKEN}",
|
|
"Accept": "application/json",
|
|
},
|
|
)
|
|
try:
|
|
with urllib.request.urlopen(req, timeout=10) as resp:
|
|
return json.loads(resp.read())
|
|
except (urllib.error.HTTPError, urllib.error.URLError, OSError) as e:
|
|
logger.warning("forgejo GET %s failed: %r", path, e)
|
|
return None
|
|
|
|
|
|
# ─── OpenCode API (read-only) ────────────────────────────────────────────
|
|
|
|
|
|
def _opencode_get(path: str) -> Any:
|
|
"""GET against the local OpenCode server. Returns parsed JSON or
|
|
None when the server is unreachable. The console renders an
|
|
'OpenCode unreachable' badge in that case rather than failing
|
|
the whole page."""
|
|
url = OPENCODE_URL + path
|
|
try:
|
|
with urllib.request.urlopen(url, timeout=3) as resp:
|
|
return json.loads(resp.read())
|
|
except (urllib.error.HTTPError, urllib.error.URLError, OSError) as e:
|
|
logger.debug("opencode GET %s failed: %r", path, e)
|
|
return None
|
|
|
|
|
|
# ─── Daemon discovery (ps + heartbeat) ───────────────────────────────────
|
|
|
|
# Heartbeat probe order matches ``_dispatch_runtime.resolve_lock_or_heartbeat``
|
|
# / ``merge_drive.load_config`` / ``conflict_drive.load_config``: explicit env
|
|
# override → ``/var/run`` → ``$XDG_RUNTIME_DIR`` (typically
|
|
# ``/run/user/<uid>``) → ``/tmp``. Production hosts and dev containers usually
|
|
# land on ``XDG_RUNTIME_DIR``; the explicit env vars are the operator's escape
|
|
# hatch.
|
|
def _heartbeat_candidates(basename: str, env_var: str | None = None) -> list[str]:
|
|
paths: list[str] = []
|
|
if env_var:
|
|
explicit = os.environ.get(env_var)
|
|
if explicit:
|
|
paths.append(explicit)
|
|
paths.append(f"/var/run/{basename}")
|
|
runtime_dir = os.environ.get("XDG_RUNTIME_DIR")
|
|
if runtime_dir:
|
|
paths.append(f"{runtime_dir.rstrip('/')}/{basename}")
|
|
paths.append(f"/tmp/{basename}")
|
|
return paths
|
|
|
|
|
|
_DAEMON_SPECS: list[dict[str, Any]] = [
|
|
{
|
|
"name": "merge_drive",
|
|
"match": ["merge_drive.py"],
|
|
"heartbeat_candidates": _heartbeat_candidates(
|
|
"merge-driver.heartbeat", "MERGE_DRIVER_HEARTBEAT_PATH"
|
|
),
|
|
},
|
|
{
|
|
"name": "conflict_drive",
|
|
"match": ["conflict_drive.py"],
|
|
"heartbeat_candidates": _heartbeat_candidates(
|
|
"conflict-driver.heartbeat", "CONFLICT_DRIVER_HEARTBEAT_PATH"
|
|
),
|
|
},
|
|
{
|
|
"name": "verify_invariant",
|
|
"match": ["verify_invariant.py"],
|
|
"heartbeat_candidates": _heartbeat_candidates(
|
|
"verify-invariant.heartbeat", "VERIFY_INVARIANT_HEARTBEAT_PATH"
|
|
),
|
|
},
|
|
{
|
|
# Tier 2 deterministic dispatcher for pr-review-worker. Heartbeat
|
|
# is written by ``run_outer_loop`` after each cycle (see
|
|
# ``tools/_dispatch_runtime.py``).
|
|
"name": "dispatch_review",
|
|
"match": ["dispatch_review.py"],
|
|
"heartbeat_candidates": _heartbeat_candidates(
|
|
"review-dispatcher.heartbeat", "REVIEW_DISPATCHER_HEARTBEAT_PATH"
|
|
),
|
|
},
|
|
{
|
|
# Tier 2 deterministic dispatcher for implementation-worker.
|
|
"name": "dispatch_implementer",
|
|
"match": ["dispatch_implementer.py"],
|
|
"heartbeat_candidates": _heartbeat_candidates(
|
|
"implementer-dispatcher.heartbeat",
|
|
"IMPLEMENTER_DISPATCHER_HEARTBEAT_PATH",
|
|
),
|
|
},
|
|
{
|
|
"name": "opencode_builder",
|
|
"match": ["opencode-builder.sh", "opencode serve"],
|
|
"heartbeat_candidates": [],
|
|
},
|
|
]
|
|
|
|
|
|
def _ps_lookup(needles: list[str]) -> list[dict[str, Any]]:
|
|
"""Return [{pid, cmdline, cpu, mem, etime}] for processes whose
|
|
cmdline contains any of ``needles``. Cross-platform-ish via /proc;
|
|
falls back to ``ps -eo`` if /proc isn't available."""
|
|
found: list[dict[str, Any]] = []
|
|
proc_dir = Path("/proc")
|
|
if proc_dir.exists():
|
|
for entry in proc_dir.iterdir():
|
|
if not entry.name.isdigit():
|
|
continue
|
|
try:
|
|
cmdline_raw = (entry / "cmdline").read_bytes()
|
|
except (FileNotFoundError, PermissionError):
|
|
continue
|
|
cmd = cmdline_raw.replace(b"\x00", b" ").decode(
|
|
"utf-8", "replace"
|
|
).strip()
|
|
if not cmd:
|
|
continue
|
|
if any(n in cmd for n in needles):
|
|
pid = int(entry.name)
|
|
stat = _proc_stat_brief(pid)
|
|
found.append({"pid": pid, "cmdline": cmd, **stat})
|
|
return found
|
|
# ps -eo fallback (BSD-ish / macOS / non-/proc Linux)
|
|
try:
|
|
out = subprocess.check_output(
|
|
["ps", "-eo", "pid=,etime=,pcpu=,pmem=,args="],
|
|
text=True, timeout=5,
|
|
)
|
|
except (subprocess.SubprocessError, FileNotFoundError):
|
|
return []
|
|
for line in out.splitlines():
|
|
line = line.strip()
|
|
if not line:
|
|
continue
|
|
parts = line.split(None, 4)
|
|
if len(parts) < 5:
|
|
continue
|
|
pid, etime, cpu, mem, cmd = parts
|
|
if any(n in cmd for n in needles):
|
|
try:
|
|
pid_int = int(pid)
|
|
except ValueError:
|
|
continue
|
|
found.append(
|
|
{
|
|
"pid": pid_int, "cmdline": cmd,
|
|
"etime": etime, "cpu_pct": float(cpu),
|
|
"mem_pct": float(mem),
|
|
}
|
|
)
|
|
return found
|
|
|
|
|
|
def _proc_stat_brief(pid: int) -> dict[str, Any]:
|
|
"""Lightweight /proc-based stat. Returns etime/cpu/mem if available,
|
|
empty dict otherwise (process gone between iterdir and read)."""
|
|
try:
|
|
with open(f"/proc/{pid}/stat", encoding="utf-8") as f:
|
|
stat_line = f.read()
|
|
except (FileNotFoundError, PermissionError):
|
|
return {}
|
|
# comm field can contain spaces; use the rightmost ')' as boundary.
|
|
rb = stat_line.rfind(")")
|
|
if rb == -1:
|
|
return {}
|
|
rest = stat_line[rb + 2 :].split()
|
|
# Field positions per `man proc`: starttime is field 22 (0-indexed
|
|
# 21 in the post-comm slice).
|
|
try:
|
|
starttime_clk = int(rest[19])
|
|
except (IndexError, ValueError):
|
|
return {}
|
|
try:
|
|
clk_tck = os.sysconf("SC_CLK_TCK") or 100
|
|
except (ValueError, OSError):
|
|
clk_tck = 100
|
|
try:
|
|
with open("/proc/uptime", encoding="utf-8") as f:
|
|
uptime_s = float(f.read().split()[0])
|
|
except (FileNotFoundError, IndexError, ValueError):
|
|
return {}
|
|
elapsed_s = uptime_s - (starttime_clk / clk_tck)
|
|
return {"elapsed_s": round(elapsed_s, 1)}
|
|
|
|
|
|
def _heartbeat_age(candidates: list[str]) -> dict[str, Any] | None:
|
|
for path in candidates:
|
|
p = Path(path)
|
|
if p.exists():
|
|
mtime = p.stat().st_mtime
|
|
return {
|
|
"path": str(p),
|
|
"mtime_s_ago": round(time.time() - mtime, 1),
|
|
}
|
|
return None
|
|
|
|
|
|
# ─── Pricing (token → $) ─────────────────────────────────────────────────
|
|
|
|
# Defaults seeded from publicly-listed prices; override via
|
|
# .opencode/telemetry/prices.json. Costs are $/1,000,000 tokens.
|
|
_DEFAULT_PRICES: dict[str, dict[str, float]] = {
|
|
# Anthropic (public — May 2026 list prices). Cache reads are
|
|
# 10% of standard input across the family.
|
|
"local-claude/claude-opus-4-6": {"in": 5.0, "out": 25.0, "cached_in": 0.50},
|
|
"local-claude/claude-sonnet-4-6": {"in": 3.0, "out": 15.0, "cached_in": 0.30},
|
|
"local-claude/claude-haiku-4-5": {"in": 1.0, "out": 5.0, "cached_in": 0.10},
|
|
# OpenAI (public — May 2026 list prices). gpt-5 / gpt-5-codex
|
|
# share the same per-token rate; gpt-5-mini / gpt-5-nano are
|
|
# the budget tier.
|
|
"openai/gpt-5": {"in": 1.25, "out": 10.0, "cached_in": 0.125},
|
|
"openai/gpt-5-codex": {"in": 1.25, "out": 10.0, "cached_in": 0.125},
|
|
"openai/gpt-5-mini": {"in": 0.25, "out": 2.0, "cached_in": 0.025},
|
|
"openai/gpt-5-nano": {"in": 0.05, "out": 0.4, "cached_in": 0.005},
|
|
# CleverThis HF Inference Endpoints — billed per endpoint-hour,
|
|
# not per token. The per-token entries here are a best-effort
|
|
# imputation for "what would this have cost on a public API";
|
|
# treat as advisory not source-of-truth.
|
|
"CleverThis-15/Qwen3-6-35B-A3B-GGUF-UD-Q3-K-XL": {"in": 0.5, "out": 1.0},
|
|
"CleverThis/Qwen3-6-35B-A3B-GGUF-BF16": {"in": 0.5, "out": 1.0},
|
|
"CleverThis-2/MiniMax-M2-7-GGUF-BF16": {"in": 0.5, "out": 1.0},
|
|
"CleverThis-4/Kimi-K2-6-GGUF-Q2-K-XL": {"in": 0.5, "out": 1.0},
|
|
# Sentinel — when we don't know, we'd rather produce 0 than a wild
|
|
# number. The UI surfaces "model unpriced" rows separately.
|
|
"_unknown": {"in": 0.0, "out": 0.0},
|
|
}
|
|
|
|
|
|
def _load_prices() -> dict[str, dict[str, float]]:
|
|
override_path = Path(
|
|
os.environ.get(
|
|
"TELEMETRY_PRICES_JSON",
|
|
str(STATIC_DIR / "prices.json"),
|
|
)
|
|
)
|
|
if override_path.exists():
|
|
try:
|
|
data = json.loads(override_path.read_text())
|
|
if isinstance(data, dict):
|
|
merged = dict(_DEFAULT_PRICES)
|
|
# Skip metadata keys (``_comment``, ``_last_updated``,
|
|
# ``_sources``) so they don't pollute the price table.
|
|
# The ``_unknown`` sentinel is preserved because it IS
|
|
# a valid price-table entry the lookup falls back to.
|
|
for k, v in data.items():
|
|
if k.startswith("_") and k != "_unknown":
|
|
continue
|
|
if isinstance(v, dict):
|
|
merged[k] = v
|
|
return merged
|
|
except (OSError, json.JSONDecodeError) as e:
|
|
logger.warning("ignoring prices.json: %r", e)
|
|
return dict(_DEFAULT_PRICES)
|
|
|
|
|
|
def _price_key(provider: str | None, model: str) -> str:
|
|
"""Resolve the lookup key for the price table.
|
|
|
|
The price table is keyed by ``provider/model`` because the same
|
|
``modelID`` can be served by multiple providers at very different
|
|
economics (e.g. ``claude-opus-4-6`` via Anthropic direct vs via a
|
|
local proxy). Falls back to the bare model name for backward
|
|
compat with manually-inserted rows that never had a provider.
|
|
"""
|
|
if provider:
|
|
return f"{provider}/{model}"
|
|
return model
|
|
|
|
|
|
def _cost_usd(provider: str | None, model: str,
|
|
prices: dict[str, dict[str, float]],
|
|
tokens_in: int, tokens_out: int, cached: int) -> float:
|
|
"""Compute USD cost for a per-model token tally.
|
|
|
|
Lookup order: ``{provider}/{model}`` → bare ``model`` → ``_unknown``.
|
|
The fallback chain lets a price entry written without provider
|
|
namespacing still match, while still preferring the
|
|
provider-qualified entry when present.
|
|
|
|
Token-accounting convention: ``tokens_in`` and ``cached`` are
|
|
DISJOINT populations — the scraper writes ``tokens_in`` as
|
|
OpenCode's ``info.tokens.input`` (fresh, non-cached prompt
|
|
tokens) plus any cache-write tokens; ``cached`` is
|
|
``info.tokens.cache.read``. So total cost is the simple sum of
|
|
three line items at three rates, with no subtraction needed.
|
|
"""
|
|
p = prices.get(_price_key(provider, model)) or prices.get(model) \
|
|
or prices.get("_unknown", {})
|
|
in_cost = tokens_in * p.get("in", 0.0) / 1_000_000
|
|
cached_cost = cached * p.get("cached_in", p.get("in", 0.0)) / 1_000_000
|
|
out_cost = tokens_out * p.get("out", 0.0) / 1_000_000
|
|
return round(in_cost + cached_cost + out_cost, 4)
|
|
|
|
|
|
# ─── Endpoint handlers ───────────────────────────────────────────────────
|
|
|
|
|
|
def _api_meta() -> dict[str, Any]:
|
|
return {
|
|
"repo_owner": REPO_OWNER,
|
|
"repo_name": REPO_NAME,
|
|
"api_base": API_BASE,
|
|
"cache_path": str(CACHE_PATH),
|
|
"cache_present": CACHE_PATH.exists(),
|
|
"opencode_url": OPENCODE_URL,
|
|
"has_forgejo_token": bool(FORGEJO_TOKEN),
|
|
"server_time_utc": datetime.now(timezone.utc).isoformat(),
|
|
"console_version": "0.1.0",
|
|
"warn_canonical_target": (
|
|
REPO_OWNER == "cleveragents" and REPO_NAME == "cleveragents-core"
|
|
),
|
|
}
|
|
|
|
|
|
# Heartbeat staleness threshold (seconds) for the "running long worker"
|
|
# hint. Set well above the dispatcher's ``poll_interval_seconds`` (2s)
|
|
# so a healthy heartbeat refresh during run_session_blocking will keep
|
|
# the daemon out of this state. Triggered when the heartbeat is older
|
|
# than this AND a matching pid is alive — the combination indicates an
|
|
# in-flight worker session whose on_poll callback isn't writing the
|
|
# heartbeat (e.g. the ``on_poll`` wiring regressed, EROFS, or the
|
|
# dispatcher has wedged on a non-OpenCode codepath).
|
|
_LONG_WORKER_HEARTBEAT_THRESHOLD_S = 600.0
|
|
|
|
|
|
# Daemon name → cycle table. Used by _api_health to surface the most
|
|
# recent in-flight cycle (``ended_at IS NULL``) per dispatcher in the
|
|
# health response so the UI can render "running cycle X started Ys ago"
|
|
# instead of stale ``last_cycle_at`` from a previous completed cycle.
|
|
_DAEMON_CYCLE_TABLES: dict[str, str] = {
|
|
"dispatch_review": "dispatch_review_cycles",
|
|
"dispatch_implementer": "dispatch_implementer_cycles",
|
|
}
|
|
|
|
|
|
def _last_in_flight_cycle(table_name: str) -> dict[str, Any] | None:
|
|
"""Return the most recent in-flight cycle for ``table_name``, or
|
|
``None`` if the cache file is missing, the table is missing, or no
|
|
in-flight rows exist.
|
|
|
|
"In flight" means ``ended_at IS NULL`` per the v5 schema. The query
|
|
is read-only; on any SQLite error we return ``None`` so the health
|
|
probe degrades gracefully (a transient DB lock should not flip
|
|
daemon status to ``unhealthy``).
|
|
"""
|
|
conn = _open_db()
|
|
if conn is None:
|
|
return None
|
|
try:
|
|
try:
|
|
cur = conn.execute(
|
|
f"SELECT cycle_id, started_at, driver, candidates_count, "
|
|
f"session_id "
|
|
f"FROM {table_name} "
|
|
f"WHERE ended_at IS NULL "
|
|
f"ORDER BY started_at DESC LIMIT 1",
|
|
)
|
|
except sqlite3.Error:
|
|
return None
|
|
row = cur.fetchone()
|
|
if row is None:
|
|
return None
|
|
rec = _rows_to_dicts([row])[0]
|
|
try:
|
|
started = datetime.fromisoformat(rec["started_at"])
|
|
if started.tzinfo is None:
|
|
started = started.replace(tzinfo=timezone.utc)
|
|
elapsed = (
|
|
datetime.now(timezone.utc) - started
|
|
).total_seconds()
|
|
rec["elapsed_s"] = round(elapsed, 1)
|
|
except (TypeError, ValueError):
|
|
rec["elapsed_s"] = None
|
|
return rec
|
|
finally:
|
|
conn.close()
|
|
|
|
|
|
def _api_health() -> dict[str, Any]:
|
|
daemons: list[dict[str, Any]] = []
|
|
for spec in _DAEMON_SPECS:
|
|
procs = _ps_lookup(spec["match"])
|
|
hb = _heartbeat_age(spec["heartbeat_candidates"])
|
|
running = bool(procs)
|
|
in_flight = None
|
|
table = _DAEMON_CYCLE_TABLES.get(spec["name"])
|
|
if table is not None:
|
|
in_flight = _last_in_flight_cycle(table)
|
|
# "Running long worker" hint: the dispatcher is alive but its
|
|
# heartbeat is stale. With the F1 on_poll wiring landed this
|
|
# should never fire under healthy operation — every poll
|
|
# iteration writes the heartbeat. When it does fire, it points
|
|
# at a callback regression or a filesystem failure on the
|
|
# heartbeat path, which is operationally different from "the
|
|
# process is alive and freshly running" or "the process is
|
|
# dead". The threshold is large enough that it does not flap
|
|
# on normal cycle boundaries (cycle_interval_seconds ≤ 600).
|
|
running_long_worker = False
|
|
if running and hb is not None:
|
|
running_long_worker = (
|
|
hb.get("mtime_s_ago", 0.0)
|
|
>= _LONG_WORKER_HEARTBEAT_THRESHOLD_S
|
|
)
|
|
daemons.append(
|
|
{
|
|
"name": spec["name"],
|
|
"running": running,
|
|
"processes": procs,
|
|
"heartbeat": hb,
|
|
"in_flight_cycle": in_flight,
|
|
"running_long_worker": running_long_worker,
|
|
}
|
|
)
|
|
oc_health = _opencode_get("/global/health")
|
|
return {
|
|
"daemons": daemons,
|
|
"opencode": {
|
|
"url": OPENCODE_URL,
|
|
"reachable": oc_health is not None,
|
|
"details": oc_health,
|
|
},
|
|
}
|
|
|
|
|
|
# Whitelisted Tier 2 dispatcher driver names → SQLite table names.
|
|
# The ``/api/cycles`` handler interpolates the table name into a
|
|
# parametrized SQL query; gating the lookup through this dict keeps
|
|
# the f-string SQL safety explicit (the only path that picks a table
|
|
# name is membership in this set) and ensures the rows-query and the
|
|
# outcome-breakdown query agree on the same table.
|
|
_DISPATCH_TABLES: dict[str, str] = {
|
|
"dispatch_review": "dispatch_review_cycles",
|
|
"dispatch_implementer": "dispatch_implementer_cycles",
|
|
}
|
|
|
|
|
|
def _api_cycles(driver: str, limit: int) -> dict[str, Any]:
|
|
conn = _open_db()
|
|
if conn is None:
|
|
return {"rows": [], "note": f"cache file not present: {CACHE_PATH}"}
|
|
try:
|
|
if driver == "merge":
|
|
cur = conn.execute(
|
|
"SELECT id, started_at, ended_at, pr_numbers, train_id,"
|
|
" bisect_depth, total_seconds, terminal_state,"
|
|
" action_taken, action_detail"
|
|
" FROM merge_cycle ORDER BY started_at DESC LIMIT ?",
|
|
(limit,),
|
|
)
|
|
elif driver == "conflict":
|
|
cur = conn.execute(
|
|
"SELECT cycle_id, pr_number, started_at, ended_at,"
|
|
" outcome, failure_kind, candidates_count,"
|
|
" resolved_count, escalated_count, timeout_count,"
|
|
" push_rejected_count, details_json"
|
|
" FROM conflict_drive_cycles"
|
|
" ORDER BY started_at DESC LIMIT ?",
|
|
(limit,),
|
|
)
|
|
elif driver in _DISPATCH_TABLES:
|
|
# Tier 2 dispatcher telemetry. ``terminal_state`` describes
|
|
# the *session lifecycle* (completed / timeout /
|
|
# transport-error / already-claimed / labels-fetch-failed
|
|
# / claim-failed / dry-run) while ``worker_outcome`` is the
|
|
# worker's own JSON-emitted outcome when present (for
|
|
# implementer workers). Both are surfaced so an operator
|
|
# can distinguish "the OpenCode session ended cleanly" from
|
|
# "the worker reported success / failure for the actual
|
|
# work" — these are independent dimensions per the
|
|
# session-status / JSON-outcome separation introduced in
|
|
# the dispatcher hardening pass.
|
|
table = _DISPATCH_TABLES[driver]
|
|
cur = conn.execute(
|
|
f"SELECT cycle_id, started_at, ended_at, driver,"
|
|
f" candidates_count, claims_acquired, swept_count,"
|
|
f" processed_count, terminal_state, worker_outcome,"
|
|
f" session_id, worker_wallclock_seconds, raw"
|
|
f" FROM {table}"
|
|
f" ORDER BY started_at DESC LIMIT ?",
|
|
(limit,),
|
|
)
|
|
else:
|
|
return {
|
|
"rows": [],
|
|
"note": f"unknown driver: {driver!r} (expected "
|
|
f"merge|conflict|{'|'.join(_DISPATCH_TABLES)})",
|
|
}
|
|
rows = _rows_to_dicts(list(cur.fetchall()))
|
|
# outcome breakdown over the LAST 24 H, regardless of limit, so
|
|
# the UI's summary card is stable when the operator changes the
|
|
# row limit.
|
|
if driver == "merge":
|
|
counts = conn.execute(
|
|
"SELECT terminal_state AS k, COUNT(*) AS n FROM merge_cycle"
|
|
" WHERE started_at > datetime('now','-24 hours')"
|
|
" GROUP BY terminal_state ORDER BY n DESC"
|
|
).fetchall()
|
|
elif driver == "conflict":
|
|
counts = conn.execute(
|
|
"SELECT outcome AS k, COUNT(*) AS n FROM"
|
|
" conflict_drive_cycles"
|
|
" WHERE started_at > datetime('now','-24 hours')"
|
|
" GROUP BY outcome ORDER BY n DESC"
|
|
).fetchall()
|
|
else:
|
|
# Composite breakdown: dashboards must show both terminal
|
|
# state and worker outcome side by side, since a session
|
|
# that ``completed`` with ``worker_outcome="rebase-failed"``
|
|
# is operationally different from one that ``completed``
|
|
# with no JSON exit at all. ``table`` is the same lookup
|
|
# used for the rows query above; reusing the variable
|
|
# prevents the two queries from drifting onto different
|
|
# tables.
|
|
counts = conn.execute(
|
|
f"SELECT terminal_state || '/' ||"
|
|
f" COALESCE(worker_outcome, '-') AS k,"
|
|
f" COUNT(*) AS n FROM {table}"
|
|
f" WHERE started_at > datetime('now','-24 hours')"
|
|
f" GROUP BY k ORDER BY n DESC"
|
|
).fetchall()
|
|
return {
|
|
"rows": rows,
|
|
"outcome_breakdown_24h": _rows_to_dicts(list(counts)),
|
|
}
|
|
finally:
|
|
conn.close()
|
|
|
|
|
|
def _api_prs(label: str | None) -> dict[str, Any]:
|
|
"""Live Forgejo query — does not hit the cache. Cache is delta-synced
|
|
by the drivers; for label flow we want sub-second freshness."""
|
|
params = {
|
|
"type": "pulls",
|
|
"state": "open",
|
|
"limit": "50",
|
|
}
|
|
if label:
|
|
params["labels"] = label
|
|
body = _forgejo_get(
|
|
f"/repos/{REPO_OWNER}/{REPO_NAME}/issues", params,
|
|
)
|
|
if not isinstance(body, list):
|
|
return {"rows": [], "note": "forgejo unreachable or no token"}
|
|
rows: list[dict[str, Any]] = []
|
|
for issue in body:
|
|
if not isinstance(issue, dict):
|
|
continue
|
|
labels = [
|
|
lbl.get("name") for lbl in (issue.get("labels") or [])
|
|
if isinstance(lbl, dict)
|
|
]
|
|
rows.append(
|
|
{
|
|
"number": issue.get("number"),
|
|
"title": issue.get("title"),
|
|
"html_url": issue.get("html_url"),
|
|
"user": (issue.get("user") or {}).get("login"),
|
|
"created_at": issue.get("created_at"),
|
|
"updated_at": issue.get("updated_at"),
|
|
"labels": labels,
|
|
"auto_labels": [lb for lb in labels if (lb or "").startswith("auto/")],
|
|
}
|
|
)
|
|
return {"rows": rows}
|
|
|
|
|
|
def _api_velocity(window: str) -> dict[str, Any]:
|
|
conn = _open_db()
|
|
if conn is None:
|
|
return {"window": window, "stats": {}, "note": "cache absent"}
|
|
try:
|
|
# Mirror the windows tools/render-pr-velocity.py uses; convert
|
|
# the human label to a SQLite datetime modifier.
|
|
win_map = {
|
|
"24h": "-24 hours",
|
|
"48h": "-48 hours",
|
|
"7d": "-7 days",
|
|
"30d": "-30 days",
|
|
}
|
|
if window not in win_map:
|
|
return {
|
|
"window": window,
|
|
"stats": {},
|
|
"note": "unknown window",
|
|
}
|
|
modifier = win_map[window]
|
|
try:
|
|
commits = conn.execute(
|
|
"SELECT COUNT(*) FROM commits"
|
|
" WHERE author_date > datetime('now', ?)",
|
|
(modifier,),
|
|
).fetchone()[0]
|
|
except sqlite3.OperationalError:
|
|
# Schema variant — fall back to whatever the cache has.
|
|
commits = 0
|
|
try:
|
|
opened = conn.execute(
|
|
"SELECT COUNT(*) FROM pulls"
|
|
" WHERE created_at > datetime('now', ?)",
|
|
(modifier,),
|
|
).fetchone()[0]
|
|
except sqlite3.OperationalError:
|
|
opened = 0
|
|
try:
|
|
merged = conn.execute(
|
|
"SELECT COUNT(*) FROM pulls"
|
|
" WHERE merged = 1 AND merged_at > datetime('now', ?)",
|
|
(modifier,),
|
|
).fetchone()[0]
|
|
except sqlite3.OperationalError:
|
|
merged = 0
|
|
return {
|
|
"window": window,
|
|
"stats": {
|
|
"commits": commits,
|
|
"opened": opened,
|
|
"merged": merged,
|
|
},
|
|
}
|
|
finally:
|
|
conn.close()
|
|
|
|
|
|
def _api_sessions() -> dict[str, Any]:
|
|
sessions = _opencode_get("/session")
|
|
if not isinstance(sessions, list):
|
|
return {"rows": [], "note": "opencode unreachable"}
|
|
rows: list[dict[str, Any]] = []
|
|
now_ms = int(time.time() * 1000)
|
|
for s in sessions:
|
|
if not isinstance(s, dict):
|
|
continue
|
|
sid = s.get("id")
|
|
msgs = _opencode_get(f"/session/{sid}/message") if sid else None
|
|
msg_count = len(msgs) if isinstance(msgs, list) else 0
|
|
time_obj = s.get("time") or {}
|
|
created = time_obj.get("created")
|
|
updated = time_obj.get("updated")
|
|
rows.append(
|
|
{
|
|
"id": sid,
|
|
"title": s.get("title"),
|
|
"directory": s.get("directory"),
|
|
"created_s_ago": round((now_ms - created) / 1000, 1)
|
|
if created else None,
|
|
"updated_s_ago": round((now_ms - updated) / 1000, 1)
|
|
if updated else None,
|
|
"message_count": msg_count,
|
|
}
|
|
)
|
|
rows.sort(key=lambda r: r.get("updated_s_ago") or 1e18)
|
|
return {"rows": rows}
|
|
|
|
|
|
# ─── Archived sessions (post-mortem of past worker runs) ─────────────────
|
|
#
|
|
# These endpoints expose the on-disk archives written by
|
|
# ``tools/_opencode_worker.py:run_session_blocking`` after every
|
|
# terminal return (completed / timeout / transport-error). The archives
|
|
# survive session deletion in OpenCode, so an operator can reconstruct
|
|
# what a worker actually said even after the dispatcher cycle is over.
|
|
#
|
|
# The path resolution mirrors ``_opencode_worker._resolve_archive_dir``:
|
|
# explicit ``OPENCODE_WORKER_ARCHIVE_DIR`` env var wins over the repo
|
|
# default ``<repo_root>/.dispatcher-logs/sessions``. We DO NOT honour
|
|
# ``OPENCODE_WORKER_ARCHIVE_DISABLED`` here because that controls the
|
|
# writer side, not the read-only console.
|
|
|
|
_DEFAULT_ARCHIVE_DIR = REPO_ROOT / ".dispatcher-logs" / "sessions"
|
|
|
|
# Maximum number of archive entries to return from the listing endpoint.
|
|
# Keeps the page snappy even on long-running pipelines that have logged
|
|
# hundreds of sessions.
|
|
_ARCHIVE_LIST_LIMIT = 200
|
|
|
|
# Filename guard for the detail endpoint. Mirrors the writer's safe
|
|
# filename component regex (``[A-Za-z0-9_.-]``); the only legitimate
|
|
# extra character is ``__`` from the writer's section separator. Any
|
|
# filename that does not match this is treated as a path-traversal
|
|
# attempt and 404'd without touching the filesystem.
|
|
_SAFE_ARCHIVE_FILENAME_RE = re.compile(r"^[A-Za-z0-9_.-]+\.json$")
|
|
|
|
|
|
def _archive_dir_path() -> Path:
|
|
explicit = os.environ.get("OPENCODE_WORKER_ARCHIVE_DIR")
|
|
if explicit:
|
|
return Path(explicit)
|
|
return _DEFAULT_ARCHIVE_DIR
|
|
|
|
|
|
def _api_archived_sessions() -> dict[str, Any]:
|
|
"""Index of session archives.
|
|
|
|
Returns lightweight metadata only: filename, agent, tag, status,
|
|
started_at, archived_at, message_count, size_bytes. The detail
|
|
endpoint serves the full payload for a clicked-on row.
|
|
"""
|
|
archive_dir = _archive_dir_path()
|
|
if not archive_dir.is_dir():
|
|
return {
|
|
"archive_dir": str(archive_dir),
|
|
"rows": [],
|
|
"note": "archive directory does not exist yet",
|
|
}
|
|
entries: list[dict[str, Any]] = []
|
|
for path in archive_dir.glob("*.json"):
|
|
try:
|
|
stat = path.stat()
|
|
with path.open("r", encoding="utf-8") as f:
|
|
data = json.load(f)
|
|
except (OSError, ValueError) as e:
|
|
entries.append(
|
|
{
|
|
"filename": path.name,
|
|
"size_bytes": None,
|
|
"error": str(e),
|
|
}
|
|
)
|
|
continue
|
|
if not isinstance(data, dict):
|
|
continue
|
|
per_turn = data.get("per_turn") or []
|
|
entries.append(
|
|
{
|
|
"filename": path.name,
|
|
"size_bytes": stat.st_size,
|
|
"modified_at": datetime.fromtimestamp(
|
|
stat.st_mtime, tz=timezone.utc
|
|
).isoformat(),
|
|
"agent": data.get("agent"),
|
|
"tag": data.get("tag"),
|
|
"status": data.get("status"),
|
|
"session_id": data.get("session_id"),
|
|
"started_at": data.get("started_at"),
|
|
"archived_at": data.get("archived_at"),
|
|
"wallclock_seconds": data.get("wallclock_seconds"),
|
|
"message_count": len(data.get("messages") or []),
|
|
"turn_count": len(per_turn) if isinstance(per_turn, list) else 0,
|
|
# v2 (2026-05-10) fields: present for subagent archives,
|
|
# ``None`` for top-level wrapper archives. The UI can use
|
|
# ``parent_session_id`` to group every session from one
|
|
# dispatcher cycle into a single tree row, and
|
|
# ``subagent_depth`` to nest the table view. Old (v1)
|
|
# archives lack these keys; ``.get`` returns None so the
|
|
# row schema stays uniform.
|
|
"schema_version": data.get("schema_version"),
|
|
"parent_session_id": data.get("parent_session_id"),
|
|
"subagent_title": data.get("subagent_title"),
|
|
"subagent_depth": data.get("subagent_depth"),
|
|
}
|
|
)
|
|
# Newest archive first. Use ``modified_at`` since it is ISO-formatted
|
|
# in the entry already; sort uses the timestamp string lexicographically
|
|
# which is correct for ISO-8601.
|
|
entries.sort(key=lambda e: e.get("modified_at") or "", reverse=True)
|
|
return {
|
|
"archive_dir": str(archive_dir),
|
|
"rows": entries[:_ARCHIVE_LIST_LIMIT],
|
|
"total": len(entries),
|
|
"limit": _ARCHIVE_LIST_LIMIT,
|
|
}
|
|
|
|
|
|
def _api_archived_session_detail(filename: str | None) -> dict[str, Any]:
|
|
"""Full payload for a single archive file."""
|
|
if not filename:
|
|
return {"error": "missing filename query parameter"}
|
|
if not _SAFE_ARCHIVE_FILENAME_RE.match(filename):
|
|
# Reject anything containing ``/``, ``..``, NULL bytes, etc.
|
|
# Without this guard a request like
|
|
# ``?filename=../../../../etc/passwd`` would be served verbatim.
|
|
return {"error": "invalid filename"}
|
|
archive_dir = _archive_dir_path()
|
|
path = archive_dir / filename
|
|
# Defence in depth: even though _SAFE_ARCHIVE_FILENAME_RE rejects
|
|
# path separators, resolve the path and confirm it stays inside
|
|
# the archive directory before reading.
|
|
try:
|
|
resolved = path.resolve()
|
|
archive_resolved = archive_dir.resolve()
|
|
except OSError as e:
|
|
return {"error": f"path resolution failed: {e}"}
|
|
if archive_resolved not in resolved.parents and resolved != archive_resolved:
|
|
return {"error": "filename escapes archive directory"}
|
|
if not resolved.is_file():
|
|
return {"error": f"not found: {filename}"}
|
|
try:
|
|
with resolved.open("r", encoding="utf-8") as f:
|
|
return json.load(f)
|
|
except (OSError, ValueError) as e:
|
|
return {"error": str(e)}
|
|
|
|
|
|
# ─── Live-run artifacts (events.jsonl + snapshot.json) ──────────────────
|
|
#
|
|
# These endpoints are populated by ``tools/live_log_writer.py`` running as a
|
|
# sidecar of the dispatcher launcher. The writer is the only producer; this
|
|
# server is read-only.
|
|
#
|
|
# Runs are discovered under ``TELEMETRY_RUNS_ROOT`` (default ``/tmp``): any
|
|
# immediate subdirectory containing ``snapshot.json`` or ``events.jsonl`` is
|
|
# treated as a run. The run's basename (e.g.
|
|
# ``implementer-test-2026-05-13-run4``) is the ``run_id`` used in the
|
|
# ``?run=<id>`` query parameter on the snapshot/events endpoints.
|
|
#
|
|
# Runs are not assumed to be "live" — historical runs whose writer is no
|
|
# longer running stay browsable as long as their files exist. The runs list
|
|
# carries a ``is_live`` heuristic (snapshot.json mtime < 30s) so the UI can
|
|
# distinguish active runs from archived ones.
|
|
|
|
_RUNS_ROOT_ENV = "TELEMETRY_RUNS_ROOT"
|
|
_DEFAULT_RUNS_ROOT = "/tmp"
|
|
# A snapshot.json mtime newer than this many seconds means the writer is
|
|
# still alive. Matches the writer's default --snapshot-interval (5s) plus
|
|
# slack for clock skew + writer schedule jitter.
|
|
_LIVE_SNAPSHOT_FRESH_S = 30.0
|
|
# Filename guard for ``?run=`` values. The basename must match — no path
|
|
# separators, no ``..``, no NULL bytes. Anything else 400s.
|
|
_SAFE_RUN_ID_RE = re.compile(r"^[A-Za-z0-9._-]+$")
|
|
|
|
|
|
def _runs_root() -> Path:
|
|
return Path(os.environ.get(_RUNS_ROOT_ENV, _DEFAULT_RUNS_ROOT))
|
|
|
|
|
|
def _discover_runs() -> list[dict[str, Any]]:
|
|
"""Scan ``_runs_root()`` for subdirectories that look like writer runs.
|
|
|
|
A directory qualifies if it contains ``snapshot.json`` or
|
|
``events.jsonl``. The returned list is sorted with the most recently
|
|
updated run first so the UI lands on the active run by default.
|
|
"""
|
|
root = _runs_root()
|
|
if not root.is_dir():
|
|
return []
|
|
now = time.time()
|
|
rows: list[dict[str, Any]] = []
|
|
try:
|
|
children = list(root.iterdir())
|
|
except OSError:
|
|
return []
|
|
for child in children:
|
|
# ``is_dir`` / ``is_file`` can raise PermissionError on /tmp
|
|
# entries owned by other users (e.g. ``/tmp/.vnc-0``). We swallow
|
|
# every os-level error here so the runs list keeps working when
|
|
# the runs root happens to be a multi-tenant directory like
|
|
# ``/tmp``. The cost is silent skips for genuinely-corrupt
|
|
# entries, which is acceptable for a read-only discovery scan.
|
|
try:
|
|
if not child.is_dir():
|
|
continue
|
|
except OSError:
|
|
continue
|
|
snap = child / "snapshot.json"
|
|
events = child / "events.jsonl"
|
|
try:
|
|
has_snapshot = snap.is_file()
|
|
except OSError:
|
|
has_snapshot = False
|
|
try:
|
|
has_events = events.is_file()
|
|
except OSError:
|
|
has_events = False
|
|
if not (has_snapshot or has_events):
|
|
continue
|
|
snap_mtime: float | None = None
|
|
events_mtime: float | None = None
|
|
try:
|
|
if has_snapshot:
|
|
snap_mtime = snap.stat().st_mtime
|
|
except OSError:
|
|
has_snapshot = False
|
|
try:
|
|
if has_events:
|
|
events_mtime = events.stat().st_mtime
|
|
except OSError:
|
|
has_events = False
|
|
# Peek at snapshot for top-line fields so the runs list can render
|
|
# without an extra round-trip per row.
|
|
peek: dict[str, Any] = {}
|
|
if has_snapshot:
|
|
try:
|
|
snap_data = json.loads(snap.read_text(encoding="utf-8"))
|
|
if isinstance(snap_data, dict):
|
|
run_info = snap_data.get("run") or {}
|
|
impl = snap_data.get("implementer") or {}
|
|
rev = snap_data.get("reviewer") or {}
|
|
peek = {
|
|
"boot_ts": run_info.get("boot_ts"),
|
|
"uptime_seconds": run_info.get("uptime_seconds"),
|
|
"implementer_phase": impl.get("current_phase"),
|
|
"implementer_active_pr": impl.get("active_pr_number"),
|
|
"reviewer_phase": rev.get("current_phase"),
|
|
"telemetry_rows": (
|
|
(impl.get("totals") or {}).get("telemetry_rows_total")
|
|
),
|
|
"outcome_disputed_count": (
|
|
(impl.get("totals") or {}).get(
|
|
"outcome_disputed_count"
|
|
)
|
|
),
|
|
}
|
|
except (OSError, ValueError):
|
|
pass
|
|
# Cheap event count — line count of events.jsonl. SQLite-cheap on
|
|
# files of even tens of thousands of lines; we cap the read at 4MB
|
|
# by reading in chunks to bound worst-case latency on a runaway log.
|
|
event_count = None
|
|
if has_events:
|
|
event_count = _count_lines(events, max_bytes=4 * 1024 * 1024)
|
|
is_live = (
|
|
snap_mtime is not None and (now - snap_mtime) < _LIVE_SNAPSHOT_FRESH_S
|
|
)
|
|
rows.append(
|
|
{
|
|
"run_id": child.name,
|
|
"path": str(child),
|
|
"has_snapshot": has_snapshot,
|
|
"has_events": has_events,
|
|
"snapshot_mtime_s_ago": (
|
|
round(now - snap_mtime, 1) if snap_mtime is not None else None
|
|
),
|
|
"events_mtime_s_ago": (
|
|
round(now - events_mtime, 1)
|
|
if events_mtime is not None else None
|
|
),
|
|
"events_count_approx": event_count,
|
|
"is_live": is_live,
|
|
**peek,
|
|
}
|
|
)
|
|
rows.sort(
|
|
key=lambda r: r.get("snapshot_mtime_s_ago")
|
|
if r.get("snapshot_mtime_s_ago") is not None
|
|
else float("inf"),
|
|
)
|
|
return rows
|
|
|
|
|
|
def _count_lines(path: Path, max_bytes: int = 4 * 1024 * 1024) -> int | None:
|
|
"""Cheap line-counter that bails out after ``max_bytes`` to keep the
|
|
runs-list endpoint responsive even when an events.jsonl gets large."""
|
|
try:
|
|
with path.open("rb") as f:
|
|
count = 0
|
|
read = 0
|
|
while read < max_bytes:
|
|
chunk = f.read(64 * 1024)
|
|
if not chunk:
|
|
break
|
|
count += chunk.count(b"\n")
|
|
read += len(chunk)
|
|
return count
|
|
except OSError:
|
|
return None
|
|
|
|
|
|
def _resolve_run_dir(run_id: str | None) -> tuple[Path | None, str | None]:
|
|
"""Validate ``run_id`` against the discovered runs list. Returns
|
|
``(path, None)`` on success or ``(None, error_message)`` for any failure
|
|
mode (missing param, path-traversal attempt, unknown run)."""
|
|
if not run_id:
|
|
return None, (
|
|
"missing 'run' query parameter — call /runs to list available runs"
|
|
)
|
|
if not _SAFE_RUN_ID_RE.match(run_id):
|
|
return None, "invalid run id (path-traversal attempt or unsupported chars)"
|
|
candidate = _runs_root() / run_id
|
|
if not candidate.is_dir():
|
|
return None, f"unknown run: {run_id!r}"
|
|
# Defence in depth — confirm the resolved path is still inside the root.
|
|
try:
|
|
resolved = candidate.resolve()
|
|
root_resolved = _runs_root().resolve()
|
|
except OSError as e:
|
|
return None, f"path resolution failed: {e}"
|
|
if root_resolved not in resolved.parents:
|
|
return None, "run escapes runs root"
|
|
return candidate, None
|
|
|
|
|
|
def _api_runs() -> tuple[int, dict[str, Any]]:
|
|
runs = _discover_runs()
|
|
return 200, {
|
|
"runs_root": str(_runs_root()),
|
|
"rows": runs,
|
|
"count": len(runs),
|
|
}
|
|
|
|
|
|
def _api_snapshot(run_id: str | None) -> tuple[int, dict[str, Any]]:
|
|
run_dir, err = _resolve_run_dir(run_id)
|
|
if err is not None:
|
|
return 400 if "invalid" in err or "missing" in err else 404, {
|
|
"error": err
|
|
}
|
|
snap = run_dir / "snapshot.json" # type: ignore[union-attr]
|
|
if not snap.is_file():
|
|
return 503, {
|
|
"error": f"snapshot not yet written for run {run_id!r}",
|
|
"run_dir": str(run_dir),
|
|
}
|
|
try:
|
|
return 200, json.loads(snap.read_text(encoding="utf-8"))
|
|
except (OSError, ValueError) as e:
|
|
return 500, {"error": str(e), "snapshot_path": str(snap)}
|
|
|
|
|
|
def _parse_event_ts_ms(ts: str | None) -> int | None:
|
|
"""Parse the ISO-8601 ``ts`` field emitted by the live writer back to
|
|
a unix epoch in milliseconds. The writer's format is
|
|
``YYYY-MM-DDTHH:MM:SS.mmmZ`` (millisecond precision, trailing Z)."""
|
|
if not ts:
|
|
return None
|
|
try:
|
|
# Replace trailing Z with +00:00 for fromisoformat.
|
|
dt = datetime.fromisoformat(ts.replace("Z", "+00:00"))
|
|
except ValueError:
|
|
return None
|
|
return int(dt.timestamp() * 1000)
|
|
|
|
|
|
# ─── /events byte-offset cache ───────────────────────────────────────────
|
|
#
|
|
# events.jsonl is append-only and grows for the life of a run (no
|
|
# rotation). Re-scanning it from byte 0 on every ~3s UI poll is O(file
|
|
# size) per poll and degrades visibly over a multi-hour run.
|
|
#
|
|
# The cache exploits two facts: (1) the UI polls with a monotonically
|
|
# increasing ``since_ms`` (each poll passes back the previous response's
|
|
# ``next_since_ms``), and (2) events are appended in emission-time order.
|
|
#
|
|
# Invariant of a cache entry ``run_dir -> (since_ms, safe_offset)``:
|
|
# EVERY event whose line starts before ``safe_offset`` has
|
|
# ``ts_ms <= since_ms``. So a later request with ``since_ms' >= since_ms``
|
|
# can seek straight to ``safe_offset`` — everything before it would be
|
|
# filtered out anyway. Truncation is handled at use-time by checking the
|
|
# CURRENT file size against ``safe_offset``: if the file is now smaller
|
|
# than the cached offset, the entry is stale and we fall back to a full
|
|
# scan from byte 0 (no separate stored file-size needed).
|
|
_EVENTS_OFFSET_CACHE: dict[str, tuple[int, int]] = {}
|
|
_EVENTS_OFFSET_CACHE_LOCK = threading.Lock()
|
|
_EVENTS_OFFSET_CACHE_MAX = 32 # bounded — only a handful of runs ever exist
|
|
|
|
|
|
def _api_events(
|
|
run_id: str | None,
|
|
since_ms: int,
|
|
limit: int,
|
|
types_filter: set[str] | None,
|
|
) -> tuple[int, dict[str, Any]]:
|
|
run_dir, err = _resolve_run_dir(run_id)
|
|
if err is not None:
|
|
return 400 if "invalid" in err or "missing" in err else 404, {
|
|
"error": err
|
|
}
|
|
events_path = run_dir / "events.jsonl" # type: ignore[union-attr]
|
|
if not events_path.is_file():
|
|
return 200, {
|
|
"rows": [], "next_since_ms": since_ms,
|
|
"note": f"events.jsonl not yet present: {events_path}",
|
|
}
|
|
cache_key = str(run_dir)
|
|
try:
|
|
file_size = events_path.stat().st_size
|
|
except OSError as e:
|
|
return 500, {"error": str(e), "events_path": str(events_path)}
|
|
|
|
# Decide where to start scanning. Default: byte 0 (cold path).
|
|
start_offset = 0
|
|
with _EVENTS_OFFSET_CACHE_LOCK:
|
|
cached = _EVENTS_OFFSET_CACHE.get(cache_key)
|
|
if (
|
|
cached is not None
|
|
and since_ms >= cached[0] # request's since_ms >= cached since_ms
|
|
and file_size >= cached[1] # file did not shrink/truncate under us
|
|
):
|
|
start_offset = cached[1]
|
|
|
|
rows: list[dict[str, Any]] = []
|
|
latest_ts_ms = since_ms
|
|
# safe_offset: byte offset of the first event with ts_ms > since_ms.
|
|
# Until we see one, every event scanned is filterable, so the running
|
|
# safe_offset is "current position". Once we see an unfiltered event
|
|
# we freeze safe_offset at its start byte.
|
|
safe_offset = start_offset
|
|
safe_offset_frozen = False
|
|
try:
|
|
with events_path.open("r", encoding="utf-8") as f:
|
|
f.seek(start_offset)
|
|
while True:
|
|
line_start = f.tell()
|
|
line = f.readline()
|
|
if not line:
|
|
break # EOF
|
|
if not line.endswith("\n"):
|
|
break # partial trailing line — writer mid-append
|
|
stripped = line.strip()
|
|
if not stripped:
|
|
if not safe_offset_frozen:
|
|
safe_offset = f.tell()
|
|
continue
|
|
try:
|
|
evt = json.loads(stripped)
|
|
except ValueError:
|
|
if not safe_offset_frozen:
|
|
safe_offset = f.tell()
|
|
continue
|
|
ts_ms = _parse_event_ts_ms(evt.get("ts"))
|
|
if ts_ms is None:
|
|
if not safe_offset_frozen:
|
|
safe_offset = f.tell()
|
|
continue
|
|
if ts_ms <= since_ms:
|
|
# Filterable — advance the safe offset past it.
|
|
if not safe_offset_frozen:
|
|
safe_offset = f.tell()
|
|
continue
|
|
# First event with ts_ms > since_ms — freeze safe_offset
|
|
# at its START byte (everything before is <= since_ms).
|
|
if not safe_offset_frozen:
|
|
safe_offset = line_start
|
|
safe_offset_frozen = True
|
|
if ts_ms > latest_ts_ms:
|
|
latest_ts_ms = ts_ms
|
|
if types_filter and evt.get("type") not in types_filter:
|
|
continue
|
|
rows.append(evt)
|
|
if len(rows) >= limit:
|
|
break
|
|
except OSError as e:
|
|
return 500, {"error": str(e), "events_path": str(events_path)}
|
|
|
|
# Refresh the cache entry. safe_offset now satisfies the invariant
|
|
# for this request's since_ms. Bound the cache size.
|
|
with _EVENTS_OFFSET_CACHE_LOCK:
|
|
_EVENTS_OFFSET_CACHE[cache_key] = (since_ms, safe_offset)
|
|
if len(_EVENTS_OFFSET_CACHE) > _EVENTS_OFFSET_CACHE_MAX:
|
|
# Evict an arbitrary oldest-ish entry (dict preserves insertion
|
|
# order; the first key is the least-recently-inserted).
|
|
_EVENTS_OFFSET_CACHE.pop(next(iter(_EVENTS_OFFSET_CACHE)))
|
|
|
|
return 200, {
|
|
"rows": rows,
|
|
"next_since_ms": latest_ts_ms,
|
|
"count": len(rows),
|
|
"limit": limit,
|
|
"run_id": run_id,
|
|
}
|
|
|
|
|
|
# ─── Per-run session tree + detail ───────────────────────────────────────
|
|
#
|
|
# These two endpoints back the Live tab's "session tree → click-to-inspect"
|
|
# UX. They reuse two existing data sources:
|
|
#
|
|
# - events.jsonl (per run) — gives us every session_id the run ever
|
|
# touched, plus turn counts, state transitions, and the archive path
|
|
# once a session ends.
|
|
# - .dispatcher-logs/sessions/*.json — full transcripts, including
|
|
# parent_session_id, written by _opencode_worker.run_session_blocking
|
|
# after a session ends.
|
|
#
|
|
# Parent → child resolution: archive JSON carries ``parent_session_id``.
|
|
# For sessions whose archive has not landed yet (worker still running), we
|
|
# don't know the parent until either the archive lands or we ask OpenCode
|
|
# directly. The /session detail endpoint handles the live-OpenCode
|
|
# fall-through; the tree endpoint marks unresolved parents as None.
|
|
|
|
_ARCHIVE_FILENAME_SID_RE = re.compile(r"__(?P<sid>ses_[A-Za-z0-9]+)\.json$")
|
|
|
|
|
|
def _archive_index_by_session_id() -> dict[str, Path]:
|
|
"""Scan the archive directory once and return ``session_id -> Path``.
|
|
Filename-only scan (no JSON parse), so it's cheap even on a directory
|
|
with hundreds of archives. The caller reads JSON only for the
|
|
sessions it actually needs."""
|
|
archive_dir = _archive_dir_path()
|
|
out: dict[str, Path] = {}
|
|
if not archive_dir.is_dir():
|
|
return out
|
|
try:
|
|
for path in archive_dir.glob("*.json"):
|
|
m = _ARCHIVE_FILENAME_SID_RE.search(path.name)
|
|
if m:
|
|
out[m.group("sid")] = path
|
|
except OSError:
|
|
pass
|
|
return out
|
|
|
|
|
|
# Status precedence: a later event's status only overrides an earlier
|
|
# session's status if it's "more terminal". This lets a single scan over
|
|
# events build a stable status per session without needing to sort.
|
|
_STATUS_PRECEDENCE = {
|
|
"starting": 0,
|
|
"running": 1,
|
|
"idle": 2,
|
|
"terminated": 3,
|
|
"archived": 4,
|
|
"timeout": 5,
|
|
"transport_error": 5,
|
|
}
|
|
|
|
|
|
def _bump_status(current: str | None, candidate: str) -> str:
|
|
if current is None:
|
|
return candidate
|
|
if _STATUS_PRECEDENCE.get(candidate, 0) >= _STATUS_PRECEDENCE.get(current, 0):
|
|
return candidate
|
|
return current
|
|
|
|
|
|
def _scan_run_sessions(events_path: Path) -> dict[str, dict[str, Any]]:
|
|
"""Walk events.jsonl once, build a {session_id: row} dict. Each row
|
|
carries everything we can learn from events alone — no archive reads,
|
|
no OpenCode calls. The caller decorates with archive/OpenCode data."""
|
|
by_sid: dict[str, dict[str, Any]] = {}
|
|
try:
|
|
with events_path.open("r", encoding="utf-8") as f:
|
|
for line in f:
|
|
line = line.strip()
|
|
if not line:
|
|
continue
|
|
try:
|
|
evt = json.loads(line)
|
|
except ValueError:
|
|
continue
|
|
sid = evt.get("session_id")
|
|
if not sid:
|
|
continue
|
|
row = by_sid.setdefault(
|
|
sid,
|
|
{
|
|
"session_id": sid,
|
|
"agent": None,
|
|
"model": None,
|
|
"tag": None,
|
|
"pr_number": None,
|
|
"source": None,
|
|
"parent_session_id": None,
|
|
"depth": None,
|
|
"status": None,
|
|
"started_at": None,
|
|
"ended_at": None,
|
|
"turn_count": 0,
|
|
"input_tokens": 0,
|
|
"output_tokens": 0,
|
|
"tools_used": {},
|
|
"archive_path": None,
|
|
# ``tier_recommendation`` is populated only for
|
|
# estimator sessions (by subagent.tier_recommendation).
|
|
# None for everyone else — the UI checks for
|
|
# presence before badging.
|
|
"tier_recommendation": None,
|
|
},
|
|
)
|
|
ts = evt.get("ts")
|
|
data = evt.get("data") or {}
|
|
etype = evt.get("type")
|
|
if etype == "worker.session_created":
|
|
# Two emits per session: a "pre-id" with model only
|
|
# (session_id absent there, filtered above) and the
|
|
# real one with session_id. The data payload here is
|
|
# ``{"agent": ...}`` only.
|
|
row["agent"] = row["agent"] or data.get("agent")
|
|
row["tag"] = row["tag"] or evt.get("tag")
|
|
row["pr_number"] = row["pr_number"] or evt.get("pr_number")
|
|
row["source"] = row["source"] or evt.get("source")
|
|
if row["started_at"] is None:
|
|
row["started_at"] = ts
|
|
row["status"] = _bump_status(row["status"], "starting")
|
|
elif etype == "worker.state_change":
|
|
to_state = data.get("to") or ""
|
|
if to_state == "busy":
|
|
row["status"] = _bump_status(row["status"], "running")
|
|
elif to_state == "idle":
|
|
row["status"] = _bump_status(row["status"], "idle")
|
|
elif etype == "worker.turn_finished":
|
|
row["turn_count"] += 1
|
|
row["input_tokens"] += int(data.get("input_tok") or 0)
|
|
row["output_tokens"] += int(data.get("output_tok") or 0)
|
|
for tool in data.get("tools") or []:
|
|
row["tools_used"][tool] = (
|
|
row["tools_used"].get(tool, 0) + 1
|
|
)
|
|
elif etype == "worker.subagent_spawned":
|
|
# This event's session_id is the SUBAGENT's, not the
|
|
# parent's. We learn the agent + depth + archive_path
|
|
# here; parent is resolved from the archive JSON.
|
|
row["agent"] = row["agent"] or data.get("agent")
|
|
row["depth"] = (
|
|
row["depth"] if row["depth"] is not None
|
|
else data.get("depth")
|
|
)
|
|
row["archive_path"] = (
|
|
row["archive_path"] or data.get("archive_path")
|
|
)
|
|
row["source"] = row["source"] or evt.get("source")
|
|
if row["started_at"] is None:
|
|
row["started_at"] = ts
|
|
row["status"] = _bump_status(row["status"], "archived")
|
|
elif etype == "worker.terminated":
|
|
row["ended_at"] = ts
|
|
row["status"] = _bump_status(row["status"], "terminated")
|
|
elif etype == "worker.archive_written":
|
|
row["archive_path"] = (
|
|
row["archive_path"] or data.get("archive_path")
|
|
)
|
|
if row["ended_at"] is None:
|
|
row["ended_at"] = ts
|
|
row["status"] = _bump_status(row["status"], "archived")
|
|
elif etype == "worker.timeout":
|
|
row["ended_at"] = ts
|
|
row["status"] = _bump_status(row["status"], "timeout")
|
|
elif etype == "worker.transport_error":
|
|
row["ended_at"] = ts
|
|
row["status"] = _bump_status(row["status"], "transport_error")
|
|
# ── Live-subagent events (sourced from the OpenCode SSE
|
|
# ── subscriber in live_log_writer.py). These let the tree
|
|
# ── render BEFORE the archive lands, which is the whole
|
|
# ── point of the SSE wire — the archive only lands
|
|
# ── 10-25 min later. parent_session_id + depth come in
|
|
# ── via the event's ``data`` payload.
|
|
elif etype == "subagent.session_created":
|
|
row["agent"] = row["agent"] or data.get("agent")
|
|
row["depth"] = (
|
|
row["depth"] if row["depth"] is not None
|
|
else data.get("depth")
|
|
)
|
|
row["parent_session_id"] = (
|
|
row["parent_session_id"] or data.get("parent_session_id")
|
|
)
|
|
row["tag"] = row["tag"] or evt.get("tag")
|
|
row["pr_number"] = row["pr_number"] or evt.get("pr_number")
|
|
row["source"] = row["source"] or evt.get("source")
|
|
if row["started_at"] is None:
|
|
row["started_at"] = data.get("started_at") or ts
|
|
row["status"] = _bump_status(row["status"], "starting")
|
|
elif etype == "subagent.state_change":
|
|
to_state = data.get("to") or ""
|
|
if to_state == "running":
|
|
row["status"] = _bump_status(row["status"], "running")
|
|
elif to_state == "idle":
|
|
row["status"] = _bump_status(row["status"], "idle")
|
|
elif etype == "subagent.tool_call_start":
|
|
tool = data.get("tool") or "tool"
|
|
row["tools_used"][tool] = (
|
|
row["tools_used"].get(tool, 0) + 1
|
|
)
|
|
elif etype == "subagent.terminated":
|
|
row["ended_at"] = ts
|
|
row["status"] = _bump_status(row["status"], "terminated")
|
|
elif etype == "subagent.tier_recommendation":
|
|
# Surface the tier pick on the row so the UI can
|
|
# badge the estimator node without a separate fetch.
|
|
row["tier_recommendation"] = {
|
|
"tier": data.get("tier"),
|
|
"is_confident": data.get("is_confident"),
|
|
}
|
|
except OSError:
|
|
return {}
|
|
return by_sid
|
|
|
|
|
|
def _decorate_with_archive(
|
|
rows: dict[str, dict[str, Any]],
|
|
sid_to_archive: dict[str, Path],
|
|
) -> None:
|
|
"""For each session_id with an archive on disk, fill in parent_session_id,
|
|
depth, started_at, model, status corrections. Mutates ``rows`` in place."""
|
|
for sid, row in rows.items():
|
|
archive_path = row.get("archive_path")
|
|
if archive_path:
|
|
p = Path(archive_path)
|
|
else:
|
|
p = sid_to_archive.get(sid)
|
|
if p is None or not p.is_file():
|
|
continue
|
|
try:
|
|
data = json.loads(p.read_text(encoding="utf-8"))
|
|
except (OSError, ValueError):
|
|
continue
|
|
if not isinstance(data, dict):
|
|
continue
|
|
row["parent_session_id"] = (
|
|
row["parent_session_id"] or data.get("parent_session_id")
|
|
)
|
|
if row["depth"] is None:
|
|
row["depth"] = data.get("subagent_depth") or 0
|
|
row["agent"] = row["agent"] or data.get("agent")
|
|
row["tag"] = row["tag"] or data.get("tag")
|
|
row["started_at"] = row["started_at"] or data.get("started_at")
|
|
row["ended_at"] = row["ended_at"] or data.get("archived_at")
|
|
row["archive_path"] = str(p)
|
|
# Pull a representative modelID from the first assistant message.
|
|
# User messages carry ``info.model`` as a {providerID, modelID}
|
|
# dict; assistant messages carry ``info.modelID`` as the plain
|
|
# string. We want the plain string, so prefer assistant rows and
|
|
# unwrap dicts from user rows as a fallback.
|
|
if not row["model"]:
|
|
for msg in (data.get("messages") or []):
|
|
info = (msg or {}).get("info") or {}
|
|
if info.get("role") != "assistant":
|
|
continue
|
|
m = info.get("modelID") or info.get("model")
|
|
if isinstance(m, dict):
|
|
m = m.get("modelID")
|
|
if m:
|
|
row["model"] = m
|
|
break
|
|
if not row["model"]:
|
|
# No assistant message yet — fall back to the user-side dict.
|
|
for msg in (data.get("messages") or [])[:3]:
|
|
info = (msg or {}).get("info") or {}
|
|
m = info.get("model")
|
|
if isinstance(m, dict):
|
|
m = m.get("modelID")
|
|
if m:
|
|
row["model"] = m
|
|
break
|
|
# Subagents don't emit ``worker.turn_finished`` (that event only
|
|
# fires for the top-level worker session), so events-derived
|
|
# turn_count is 0 for them. The archive's ``per_turn`` array has
|
|
# the real number — use it when events didn't see any turns.
|
|
if row["turn_count"] == 0:
|
|
per_turn = data.get("per_turn") or []
|
|
if isinstance(per_turn, list):
|
|
row["turn_count"] = len(per_turn)
|
|
# Archive-derived status: prefer "completed" once the archive
|
|
# exists and shows a clean terminated status.
|
|
archive_status = data.get("status")
|
|
if archive_status == "completed":
|
|
row["status"] = _bump_status(row["status"], "terminated")
|
|
elif archive_status == "subagent":
|
|
row["status"] = _bump_status(row["status"], "archived")
|
|
|
|
|
|
def _api_run_sessions(run_id: str | None) -> tuple[int, dict[str, Any]]:
|
|
"""Per-run session inventory: every session_id seen in the run's
|
|
events.jsonl, decorated with archive-derived parent/depth/model so the
|
|
UI can render the parent→child tree without per-row fetches.
|
|
|
|
The endpoint is deliberately a single scan + a directory glob; no
|
|
OpenCode round-trip. The per-session detail endpoint does the
|
|
live-OpenCode fall-through when a transcript is requested."""
|
|
run_dir, err = _resolve_run_dir(run_id)
|
|
if err is not None:
|
|
return 400 if "invalid" in err or "missing" in err else 404, {
|
|
"error": err
|
|
}
|
|
events_path = run_dir / "events.jsonl" # type: ignore[union-attr]
|
|
if not events_path.is_file():
|
|
return 200, {
|
|
"run_id": run_id,
|
|
"rows": [],
|
|
"note": f"events.jsonl not yet present: {events_path}",
|
|
}
|
|
rows_by_sid = _scan_run_sessions(events_path)
|
|
sid_to_archive = _archive_index_by_session_id()
|
|
_decorate_with_archive(rows_by_sid, sid_to_archive)
|
|
rows = list(rows_by_sid.values())
|
|
# Stable ordering: parents before children when both are present, then
|
|
# by started_at. The UI does its own tree-building from
|
|
# parent_session_id, so order is purely cosmetic for unparented rows.
|
|
rows.sort(key=lambda r: (r.get("depth") or 0, r.get("started_at") or ""))
|
|
return 200, {
|
|
"run_id": run_id,
|
|
"rows": rows,
|
|
"count": len(rows),
|
|
"archive_dir": str(_archive_dir_path()),
|
|
}
|
|
|
|
|
|
# ─── Per-session transcript (archive OR live OpenCode) ───────────────────
|
|
|
|
|
|
def _normalize_message_parts(parts: list[Any]) -> list[dict[str, Any]]:
|
|
"""Collapse OpenCode's verbose message-parts into a smaller, UI-shaped
|
|
shape. Drops housekeeping parts (step-start/step-finish), preserves the
|
|
text/reasoning/tool parts the UI actually renders."""
|
|
out: list[dict[str, Any]] = []
|
|
for part in parts or []:
|
|
if not isinstance(part, dict):
|
|
continue
|
|
ptype = part.get("type")
|
|
if ptype == "text":
|
|
text = part.get("text") or ""
|
|
if text.strip():
|
|
out.append({"kind": "text", "text": text})
|
|
elif ptype == "reasoning":
|
|
text = part.get("text") or ""
|
|
if text.strip():
|
|
out.append({"kind": "reasoning", "text": text})
|
|
elif ptype == "tool":
|
|
state = part.get("state") or {}
|
|
entry: dict[str, Any] = {
|
|
"kind": "tool",
|
|
"tool": part.get("tool"),
|
|
"status": state.get("status"),
|
|
"title": state.get("title"),
|
|
"input": state.get("input"),
|
|
}
|
|
output = state.get("output")
|
|
if output is not None:
|
|
# Truncate at the transport boundary to keep the response
|
|
# cheap; the archive file is always available for full
|
|
# inspection via /api/sessions/archived/detail.
|
|
if isinstance(output, str) and len(output) > 8000:
|
|
entry["output"] = output[:8000]
|
|
entry["output_truncated"] = True
|
|
else:
|
|
entry["output"] = output
|
|
out.append(entry)
|
|
# step-start / step-finish / other housekeeping parts are dropped.
|
|
return out
|
|
|
|
|
|
def _normalize_messages(messages: list[Any]) -> list[dict[str, Any]]:
|
|
"""Turn OpenCode's ``[{info, parts}]`` shape into ``[{role, model, ts,
|
|
parts}]`` for the UI. Skips empty messages (no useful parts)."""
|
|
out: list[dict[str, Any]] = []
|
|
for msg in messages or []:
|
|
if not isinstance(msg, dict):
|
|
continue
|
|
info = msg.get("info") or {}
|
|
parts = _normalize_message_parts(msg.get("parts") or [])
|
|
if not parts:
|
|
continue
|
|
time_info = info.get("time") or {}
|
|
ts = None
|
|
if isinstance(time_info, dict):
|
|
ts = time_info.get("created") or time_info.get("completed")
|
|
# Same unwrap as _decorate_with_archive: user messages store
|
|
# ``info.model`` as a dict, assistant messages store
|
|
# ``info.modelID`` as a string. UI wants a string.
|
|
model = info.get("modelID") or info.get("model")
|
|
if isinstance(model, dict):
|
|
model = model.get("modelID")
|
|
out.append(
|
|
{
|
|
"role": info.get("role"),
|
|
"model": model,
|
|
"agent": info.get("agent"),
|
|
"ts": ts,
|
|
"parts": parts,
|
|
}
|
|
)
|
|
return out
|
|
|
|
|
|
def _api_run_session(
|
|
run_id: str | None, session_id: str | None
|
|
) -> tuple[int, dict[str, Any]]:
|
|
"""Per-session transcript. Uses the archive when present, falls back
|
|
to a live OpenCode fetch when the session is still running.
|
|
|
|
The shape is the same in both cases so the UI doesn't need to branch:
|
|
``{session_id, source, status, agent, model, parent_session_id,
|
|
turns: [{role, model, ts, parts}], summary: {...}}``."""
|
|
run_dir, err = _resolve_run_dir(run_id)
|
|
if err is not None:
|
|
return 400 if "invalid" in err or "missing" in err else 404, {
|
|
"error": err
|
|
}
|
|
if not session_id or not session_id.startswith("ses_") or "/" in session_id:
|
|
return 400, {"error": "missing or invalid 'session' query parameter"}
|
|
|
|
sid_to_archive = _archive_index_by_session_id()
|
|
archive_path = sid_to_archive.get(session_id)
|
|
if archive_path is not None and archive_path.is_file():
|
|
try:
|
|
data = json.loads(archive_path.read_text(encoding="utf-8"))
|
|
except (OSError, ValueError) as e:
|
|
return 500, {"error": str(e), "archive_path": str(archive_path)}
|
|
if not isinstance(data, dict):
|
|
return 500, {"error": "archive is not a JSON object"}
|
|
return 200, {
|
|
"session_id": session_id,
|
|
"source": "archive",
|
|
"archive_path": str(archive_path),
|
|
"status": data.get("status"),
|
|
"agent": data.get("agent"),
|
|
"tag": data.get("tag"),
|
|
"parent_session_id": data.get("parent_session_id"),
|
|
"subagent_depth": data.get("subagent_depth"),
|
|
"started_at": data.get("started_at"),
|
|
"ended_at": data.get("archived_at"),
|
|
"wallclock_seconds": data.get("wallclock_seconds"),
|
|
"turns": _normalize_messages(data.get("messages") or []),
|
|
}
|
|
|
|
# Live fall-through: ask OpenCode directly. This is the only path that
|
|
# makes a network call, and it only fires when an operator clicks a
|
|
# currently-running session — so a slow OpenCode does not affect the
|
|
# tree endpoint.
|
|
info = _opencode_get(f"/session/{session_id}")
|
|
messages = _opencode_get(f"/session/{session_id}/message")
|
|
if info is None and messages is None:
|
|
return 503, {
|
|
"error": (
|
|
f"no archive on disk for {session_id} AND OpenCode at "
|
|
f"{OPENCODE_URL} is unreachable — transcript not available"
|
|
),
|
|
"session_id": session_id,
|
|
}
|
|
if not isinstance(messages, list):
|
|
messages = []
|
|
info = info if isinstance(info, dict) else {}
|
|
return 200, {
|
|
"session_id": session_id,
|
|
"source": "live-opencode",
|
|
"status": "running",
|
|
"agent": info.get("agent"),
|
|
"parent_session_id": info.get("parentID"),
|
|
"started_at": (info.get("time") or {}).get("created")
|
|
if isinstance(info.get("time"), dict) else None,
|
|
"ended_at": None,
|
|
"turns": _normalize_messages(messages),
|
|
}
|
|
|
|
|
|
def _api_cost(days: int) -> dict[str, Any]:
|
|
conn = _open_db()
|
|
prices = _load_prices()
|
|
if conn is None:
|
|
return {
|
|
"days": days, "rows": [], "totals": {"usd": 0.0},
|
|
"note": "cache absent",
|
|
"instrumentation_pending": True,
|
|
}
|
|
try:
|
|
cur = conn.execute(
|
|
"SELECT model, provider, COUNT(*) AS n,"
|
|
" SUM(COALESCE(tokens_in, 0)) AS in_tok,"
|
|
" SUM(COALESCE(tokens_out, 0)) AS out_tok,"
|
|
" SUM(COALESCE(cached_tokens, 0)) AS cached_tok"
|
|
" FROM llm_activity"
|
|
" WHERE started_at > datetime('now', ?)"
|
|
" GROUP BY model, provider ORDER BY n DESC",
|
|
(f"-{int(days)} days",),
|
|
)
|
|
rows = []
|
|
total_usd = 0.0
|
|
total_in = total_out = total_cached = 0
|
|
for r in cur:
|
|
in_tok = int(r["in_tok"] or 0)
|
|
out_tok = int(r["out_tok"] or 0)
|
|
cached = int(r["cached_tok"] or 0)
|
|
model = r["model"] or "_unknown"
|
|
provider = r["provider"]
|
|
usd = _cost_usd(
|
|
provider, model, prices, in_tok, out_tok, cached,
|
|
)
|
|
key = _price_key(provider, model)
|
|
rows.append(
|
|
{
|
|
"model": model, "provider": provider, "n": r["n"],
|
|
"tokens_in": in_tok, "tokens_out": out_tok,
|
|
"cached": cached, "usd": usd,
|
|
"priced": (key in prices or model in prices),
|
|
}
|
|
)
|
|
total_usd += usd
|
|
total_in += in_tok
|
|
total_out += out_tok
|
|
total_cached += cached
|
|
return {
|
|
"days": days,
|
|
"rows": rows,
|
|
"totals": {
|
|
"usd": round(total_usd, 2),
|
|
"tokens_in": total_in,
|
|
"tokens_out": total_out,
|
|
"cached": total_cached,
|
|
},
|
|
"instrumentation_pending": (not rows),
|
|
}
|
|
except sqlite3.OperationalError as e:
|
|
return {
|
|
"days": days, "rows": [], "totals": {"usd": 0.0},
|
|
"note": f"llm_activity table missing or stale schema: {e}",
|
|
"instrumentation_pending": True,
|
|
}
|
|
finally:
|
|
conn.close()
|
|
|
|
|
|
# ─── HTTP server ─────────────────────────────────────────────────────────
|
|
|
|
|
|
_API_DISPATCH = {
|
|
"/api/meta": lambda q: _api_meta(),
|
|
"/api/health": lambda q: _api_health(),
|
|
"/api/cycles": lambda q: _api_cycles(
|
|
q.get("driver", ["merge"])[0],
|
|
int(q.get("limit", ["20"])[0]),
|
|
),
|
|
"/api/prs": lambda q: _api_prs(q.get("label", [None])[0]),
|
|
"/api/velocity": lambda q: _api_velocity(
|
|
q.get("window", ["48h"])[0],
|
|
),
|
|
"/api/sessions": lambda q: _api_sessions(),
|
|
"/api/sessions/archived": lambda q: _api_archived_sessions(),
|
|
"/api/sessions/archived/detail": lambda q: _api_archived_session_detail(
|
|
q.get("filename", [None])[0],
|
|
),
|
|
"/api/cost": lambda q: _api_cost(int(q.get("days", ["7"])[0])),
|
|
}
|
|
|
|
|
|
# Endpoints that return (status_code, payload) rather than a bare payload,
|
|
# so they can signal 503/500 to the UI without raising. Kept separate from
|
|
# ``_API_DISPATCH`` so the simple-case handlers stay simple.
|
|
_API_DISPATCH_WITH_STATUS = {
|
|
"/runs": lambda q: _api_runs(),
|
|
"/snapshot": lambda q: _api_snapshot(q.get("run", [None])[0]),
|
|
"/events": lambda q: _api_events(
|
|
q.get("run", [None])[0],
|
|
int(q.get("since", ["0"])[0] or 0),
|
|
int(q.get("limit", ["500"])[0] or 500),
|
|
(
|
|
{t.strip() for t in q["types"][0].split(",") if t.strip()}
|
|
if q.get("types")
|
|
else None
|
|
),
|
|
),
|
|
"/api/run/sessions": lambda q: _api_run_sessions(
|
|
q.get("run", [None])[0],
|
|
),
|
|
"/api/run/session": lambda q: _api_run_session(
|
|
q.get("run", [None])[0],
|
|
q.get("session", [None])[0],
|
|
),
|
|
}
|
|
|
|
|
|
_STATIC_FILES = {
|
|
"/": ("index.html", "text/html; charset=utf-8"),
|
|
"/index.html": ("index.html", "text/html; charset=utf-8"),
|
|
"/static/app.js": ("app.js", "application/javascript; charset=utf-8"),
|
|
"/static/style.css": ("style.css", "text/css; charset=utf-8"),
|
|
}
|
|
|
|
|
|
class _Handler(BaseHTTPRequestHandler):
|
|
server_version = "TelemetryConsole/0.1"
|
|
|
|
def log_message(self, fmt: str, *args: Any) -> None:
|
|
# Silence the default per-request stderr noise; only log warnings.
|
|
return
|
|
|
|
def _send_json(self, status: int, payload: Any) -> None:
|
|
body = json.dumps(payload, default=str).encode("utf-8")
|
|
self.send_response(status)
|
|
self.send_header("Content-Type", "application/json; charset=utf-8")
|
|
self.send_header("Content-Length", str(len(body)))
|
|
self.send_header("Cache-Control", "no-store")
|
|
self.end_headers()
|
|
self.wfile.write(body)
|
|
|
|
def _send_static(self, fname: str, content_type: str) -> None:
|
|
path = STATIC_DIR / fname
|
|
if not path.is_file():
|
|
self.send_error(404, f"missing static file: {fname}")
|
|
return
|
|
body = path.read_bytes()
|
|
self.send_response(200)
|
|
self.send_header("Content-Type", content_type)
|
|
self.send_header("Content-Length", str(len(body)))
|
|
self.send_header("Cache-Control", "no-store")
|
|
self.end_headers()
|
|
self.wfile.write(body)
|
|
|
|
def do_GET(self) -> None: # noqa: N802 (BaseHTTPRequestHandler API)
|
|
parsed = urllib.parse.urlparse(self.path)
|
|
route = parsed.path
|
|
query = urllib.parse.parse_qs(parsed.query)
|
|
if route in _STATIC_FILES:
|
|
fname, ctype = _STATIC_FILES[route]
|
|
self._send_static(fname, ctype)
|
|
return
|
|
handler = _API_DISPATCH.get(route)
|
|
if handler is not None:
|
|
try:
|
|
payload = handler(query)
|
|
self._send_json(200, payload)
|
|
except Exception as e: # noqa: BLE001 — keep server up
|
|
logger.exception("handler %s failed", route)
|
|
self._send_json(500, {"error": str(e), "route": route})
|
|
return
|
|
handler_with_status = _API_DISPATCH_WITH_STATUS.get(route)
|
|
if handler_with_status is not None:
|
|
try:
|
|
status, payload = handler_with_status(query)
|
|
self._send_json(status, payload)
|
|
except Exception as e: # noqa: BLE001
|
|
logger.exception("handler %s failed", route)
|
|
self._send_json(500, {"error": str(e), "route": route})
|
|
return
|
|
self.send_error(404, f"unknown route: {route}")
|
|
|
|
|
|
def _install_signal_handlers(httpd: ThreadingHTTPServer) -> None:
|
|
def _shutdown(signum: int, _frame: Any) -> None:
|
|
logger.info("received signal %d, shutting down", signum)
|
|
# shutdown() must not run in the request thread; spawn a thread
|
|
# via a separate signal-safe path.
|
|
import threading
|
|
threading.Thread(target=httpd.shutdown, daemon=True).start()
|
|
|
|
for sig in (signal.SIGINT, signal.SIGTERM):
|
|
try:
|
|
signal.signal(sig, _shutdown)
|
|
except (OSError, ValueError):
|
|
pass
|
|
|
|
|
|
def main(argv: list[str] | None = None) -> int:
|
|
parser = argparse.ArgumentParser(
|
|
description="auto-agents pipeline telemetry console",
|
|
)
|
|
parser.add_argument(
|
|
"--host",
|
|
default=os.environ.get("TELEMETRY_HOST", "127.0.0.1"),
|
|
help=(
|
|
"interface to bind (default 127.0.0.1 — loopback only). "
|
|
"Set 0.0.0.0 ONLY when you understand v1 ships without auth."
|
|
),
|
|
)
|
|
parser.add_argument(
|
|
"--port", type=int,
|
|
default=int(os.environ.get("TELEMETRY_PORT", "8765")),
|
|
help="TCP port to bind (default 8765)",
|
|
)
|
|
parser.add_argument(
|
|
"--log-level", default="INFO",
|
|
choices=("DEBUG", "INFO", "WARNING", "ERROR"),
|
|
)
|
|
args = parser.parse_args(argv)
|
|
|
|
logging.basicConfig(
|
|
level=args.log_level,
|
|
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
|
|
)
|
|
|
|
logger.info("repo target: %s/%s", REPO_OWNER, REPO_NAME)
|
|
logger.info("api base: %s", API_BASE)
|
|
logger.info("cache file: %s (%s)",
|
|
CACHE_PATH, "exists" if CACHE_PATH.exists() else "ABSENT")
|
|
logger.info("opencode url: %s", OPENCODE_URL)
|
|
logger.info("forgejo token: %s",
|
|
"present" if FORGEJO_TOKEN else "MISSING (PR tab disabled)")
|
|
|
|
try:
|
|
httpd = ThreadingHTTPServer((args.host, args.port), _Handler)
|
|
except OSError as e:
|
|
print(
|
|
f"ERROR: bind {args.host}:{args.port} failed: {e}",
|
|
file=sys.stderr,
|
|
)
|
|
return 2
|
|
|
|
_install_signal_handlers(httpd)
|
|
print(
|
|
f"Telemetry console running at http://{args.host}:{args.port}/ "
|
|
f"(Ctrl+C to stop)"
|
|
)
|
|
try:
|
|
httpd.serve_forever()
|
|
finally:
|
|
httpd.server_close()
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
raise SystemExit(main())
|