d4e1219577e01f270aca90e2706cfdcd523100cd
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 drew
|
0bc734c020 |
style: ruff format the controller-state-machine branch (288 files)
Applies `ruff format` to the accumulated formatting debt on this branch. Formatting-only — no behavioral changes. Required for CI/lint's format gate (`nox -s format -- --check`), which the branch was failing on 288 tracked files that drifted from ruff's canonical style. In-progress WIP files are intentionally excluded so this commit stays a clean formatting-only diff. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
drew
|
3f12e4140c |
fix(auto-agents): R3.4 — cycle-cap signature + implementer sees full reviewer record
Two related fixes surfaced by the 2026-05-17 run-5 live observation: 1. **Cycle-cap signature now includes total_reviews count** so the reviewer's COMMENT-only path actually moves the signature. Before: the cap signature was ``sha + (approvals + has_active_RC + has_unaddressed_RC)`` — all three boolean axes ignore COMMENT reviews entirely. After: ``+ total_reviews`` term bumps on EVERY review submission. The ``data_complete=False → COMMENT downgrade`` path the reviewer takes in low-context cycles now reflects in the signature, so the cap stops firing falsely. Without this, run-5 observed 4 fresh PRs hitting count=5 in ~10 minutes — the reviewer was successfully posting reviews every cycle but the cap saw "no change" because COMMENT-only reviews don't bump approvals_count, has_active_RC, or has_unaddressed_RC. 2. **Implementer now sees ALL reviewer feedback**, not just active REQUEST_CHANGES. Before: ``fetch_pr_fix_context`` passed ``include_active_reviews=False`` so failing-CI PRs reached the implementer with zero reviewer data. ``fetch_request_changes_pr_context`` only included active RC reviews — COMMENT-only feedback was invisible in both code paths. After: ``fetch_pr_fix_context`` also includes reviews, AND the fetcher now partitions reviews into TWO buckets — the existing ``request_changes_reviews`` (active blocking RC, unchanged semantic) and a new ``comment_reviews`` field carrying every non-dismissed non-RC review (COMMENT / APPROVE). Both are persisted in the PR-context sentinel and exposed via ``implementer_pr_context.py``'s ``comment_reviews`` field. New prompt section ``## Pre-fetched reviewer comments and approvals`` renders the comment_reviews bucket with author / event / commit / body / inline comments + a postscript marking them as ADVISORY (not blocking, unlike the existing RC section). This closes the architectural gap where the reviewer and implementer pools could work in silos on the same PR — the reviewer's substantive prose feedback now reaches the implementer regardless of which work-group routed it. Files touched: - tools/_pr_classification_cache.py — total_reviews in classify_pr + reactivity composite; docstring updated. - tools/_implementer_prefetch.py — new comment_reviews + comment_reviews_completed fields; fetcher partitions reviews once; fetch_pr_fix_context now includes reviews. - tools/_implementer_prompt.py — _build_comment_reviews_section; wired into prompt assembly between RC and PR-comments sections. - tools/_pr_context_sentinel.py — comment_reviews in _to_dict. - tools/implementer_pr_context.py — comment_reviews accessor for the worker's handoff read path. - tests/auto_agents/test_pr_context_sentinel.py — expected_value_keys + fixture + round-trip test updated. - .opencode/agents/estimator-implementation.md — restored canonical section header levels (####) for downstream test compatibility after R3.1 rewrite. Full auto_agents suite: 2303 passing (+12 from this and adjacent work, none broken). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
88b9373fa9 |
feat(auto-agents): LLM activity scraper + cost dashboard wiring
Closes the cost-tracking instrumentation gap: the telemetry console's Cost tab read from an empty ``llm_activity`` table because nothing in production wrote to it. The scraper walks the OpenCode session archives that ``_opencode_worker`` already writes (including subagent trees via the BFS-walked ``parentID`` chain) and emits one row per assistant turn. Folded into the existing PR-State Warmer loop so it runs on the same 30s cadence without spinning a new sidecar. Schema (v6): - ``llm_activity`` grows ``session_id`` / ``message_id`` / ``provider`` / ``parent_session_id`` / ``subagent_depth`` columns - Partial UNIQUE INDEX on ``message_id`` makes re-scrapes idempotent - v5→v6 migration ALTER-gated on column existence (safe to re-run) Scraper (``tools/llm_activity_scraper.py``): - Reads ``.dispatcher-logs/sessions/*.json``, one row per assistant turn - Folds reasoning tokens into ``tokens_out`` and cache-write into ``tokens_in`` (preserves raw breakdown in ``raw`` JSON for future cost-calc refinements) - Normalises ``subagent_depth=0`` at top level so dashboards can filter ``WHERE subagent_depth > 0`` cleanly - Batch INSERT OR IGNORE via new ``PipelineCache.upsert_llm_activity_batch`` — one fsync per archive, not per turn Warmer integration: - First tick: full backfill of the archive directory - Subsequent ticks: 1h lookback via ``since=`` filter - Scraper failures are logged and swallowed — PR-state job stays load-bearing and unaffected - ``LLM_ACTIVITY_SCRAPER_DISABLE=1`` env kill switch Renames (mechanical, atomic): - ``tools/_forgejo_cache.py`` → ``tools/_pipeline_cache.py`` - ``ForgejoCache`` class → ``PipelineCache`` - Both reflect the module's broader scope (Forgejo data + pipeline telemetry tables); on-disk filename ``forgejo.sqlite`` and ``FORGEJO_*`` env vars are kept for compatibility Verified end-to-end on real archives: 435 archives → 3595 turns ingested (2873 from subagents) across 8 models / 5 providers / 9 PRs. Re-runs insert 0, dedup 3595. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
drew
|
2658deee94 |
feat(auto-agents): PR State Warmer substrate + supporting infra
Adds a long-lived sidecar (pr_state_warmer.py) that polls Forgejo's /pulls endpoint every 30s and writes the full PR snapshot to a shared SQLite store, eliminating the dispatcher's per-cycle cold-cache stalls (24-30s rebuilds on flaky cycles) and the silent 50-PR pagination cap on the legacy single-page fetch. Substrate - tools/_pr_state_cache.py — SQLite store with (owner, repo) PK, WAL mode, additive v2→v3 migration (comments_refreshed_updated_at), bounded fcntl.flock migration lock, threading.Lock for per-process init, @_with_reheal decorator (catches OperationalError no-such- table + DatabaseError corruption with file quarantine), atomic TEMP-table chunking for >32k seen-set, _normalize_updated_at to canonicalize Forgejo tz-marker drift - tools/pr_state_warmer.py — poll/upsert/vanish/comments-refresh loop with fcntl.flock singleton (rejects second warmer), bounded comments-refresh cap, persistent deferral via SQL pending query, PermissionError-tolerant lock setup, cold-start log suppression - tools/_pr_classification_cache.py — three-layer fall-through (warmer cache → list cache → live fetch) with staleness gate (PR_STATE_WARMER_STALE_AFTER_S floored at 30s in prod) Comments cache hardening - Bot-filter at write time drops bot status/claim/release/sentinel while preserving **Implementation Attempt** markers (94.6% reduction on bot-heavy PRs like #30's 19k-comment thread) - _normalize_since_cursor strips microsecond precision before building ?since= query (fixes the live-observed Forgejo HTTP 422 bug on PRs #25 + #28); handles uppercase Z, lowercase z, ±HH:MM offsets (including non-zero like +05:30), naive ISO - Lazy migration of legacy null-key by_author entries on _read_cache - _newest_cursor walks tail-back skipping malformed entries Supporting infrastructure (cumulative dmpipeline-v2 work) - Telemetry server: SSE live tail, run-sessions enumeration, cost/token tracking, app.js UI rewrite with collapsible sections - MCP servers (mcp_ci_server, mcp_forgejo_server, mcp_git_server, mcp_handoff_server, mcp_graphify_server) for opencode worker context access - Live log writer (tools/live_log_writer.py) — SSE-streaming dispatcher event log - Tier-dispatcher escalation flow with prompts trimmed for budget - Shared bot-logins resolver (tools/_bot_logins.py) replacing two drift-prone copies - token_usage_audit.py for opencode cost analysis Tests - 2259 passing across 65 changed/new files - New suites: test_pr_state_cache, test_pr_state_warmer, test_pr_state_warmer_integration, test_pr_classification_cache, test_pr_list_cache_backoff, test_mcp_* (5 servers), test_live_log_writer_sse, test_telemetry_run_sessions, test_review_post_ready_label - Test_pr_comments_cache expanded with bot-filter coverage, cursor-normalization regression pins, format-drift, atomicity, failed-comments-not-stamped (silent-data-loss class) - Parametrized @_with_reheal coverage across 7 wrapped APIs - Real fault-inject atomicity test for chunked mark_vanished path via Connection wrapper class - Subprocess-based singleton flock test (cross-process contract) - Event-driven SIGTERM-mid-poll test (no fixed-sleep flake) Architecture notes - Schema v3 migration is additive (ALTER ADD COLUMN); v0/v1 still need destructive rebuild because pre-v2 column shape lacks owner/repo. Cross-process drop-table-ping-pong prevented by the fcntl migration lock + per-process _initialized flag. - Comments-refresh deferral is persistent via comments_refreshed_updated_at column — survives warmer restart, picks up next cycle even if PR didn't change again. Replaces in-memory changed_numbers list. - Rollback path: PR_STATE_WARMER_PREFER=0 bypasses the warmer cache and reverts to live-fetch behavior. PR_STATE_CACHE_DISABLE=1 short-circuits the warmer process at startup. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |