From c14f2d0a54e80e1f0ee754d9b8ee3d9e3fa5e46a Mon Sep 17 00:00:00 2001 From: Jeffrey Phillips Freeman Date: Fri, 3 Apr 2026 18:24:03 +0000 Subject: [PATCH] docs(development): add system watchdog architecture and audit reference Document the ca-system-watchdog continuous monitoring agent: - All 12 audit descriptions with severity levels and detection patterns - Session introspection via OpenCode Server API (Audits 6, 11, 12) - Finding severity table and action dispatch table - Health reporting format and configuration parameters - Constraints and related documentation links Add System Watchdog entry to mkdocs.yml Development nav section. Covers commits 8c13e63c and 77427bd7 (system watchdog introduction and deep session introspection enhancement). --- docs/development/system-watchdog.md | 350 ++++++++++++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 351 insertions(+) create mode 100644 docs/development/system-watchdog.md diff --git a/docs/development/system-watchdog.md b/docs/development/system-watchdog.md new file mode 100644 index 000000000..03009e651 --- /dev/null +++ b/docs/development/system-watchdog.md @@ -0,0 +1,350 @@ +# System Watchdog + +The **System Watchdog** (`ca-system-watchdog`) is the 16th supervisor agent in the +CleverAgents autonomous build system. It runs continuously with a 5-minute polling +cycle and monitors the entire agent ecosystem for correctness, health, and policy +compliance. + +> **Audience:** Contributors and operators who need to understand how the autonomous +> agent system is monitored, what violations are detected, and how the watchdog +> responds to findings. + +--- + +## Overview + +The watchdog operates exclusively through the **Forgejo API** and the +**OpenCode Server HTTP API** — it does not clone the repository or access the +filesystem. It performs 12 audits per cycle, ranging from quality gate compliance +to deep session introspection of individual supervisor conversations. + +``` +cycle = 0 +LOOP FOREVER (5-minute intervals): + cycle += 1 + findings = [] + + findings += audit_quality_gates() # Audit 1 — MOST CRITICAL + findings += audit_branch_protection() # Audit 2 + findings += audit_ticket_states() # Audit 3 + findings += audit_priority_ordering() # Audit 4 + findings += audit_pr_pipeline() # Audit 5 + findings += audit_supervisor_health() # Audit 6 (session introspection) + findings += audit_labels_and_dependencies() # Audit 7 + findings += audit_ticket_hierarchy() # Audit 8 + findings += audit_test_health() # Audit 9 + findings += audit_improvement_generation() # Audit 10 + findings += audit_session_spot_check() # Audit 11 (every cycle) + + if cycle % 6 == 0: + findings += audit_deep_session_introspection() # Audit 12 (every 30 min) + + for finding in findings: + take_action(finding) + + if cycle % 6 == 0 and findings: + post_summary(cycle, findings) + + sleep(300) # 5 minutes +``` + +--- + +## Audit Descriptions + +### Audit 1 — Quality Gate Compliance (CRITICAL) + +Ensures **no code reaches `master` without passing all CI checks**. + +- Checks the last 10 commits on `master` for passing `status-check` CI status. +- Verifies recently merged PRs had passing CI at merge time. +- Detects direct pushes to `master` (all changes must go through PRs). + +**Severity:** CRITICAL for any violation. + +--- + +### Audit 2 — Branch Protection Verification + +Verifies that Forgejo branch protection is active and correctly configured for +`master`. + +Checks: +- Branch protection rules exist for `master`. +- `enable_status_check` is `true`. +- The `status-check` context is required. +- At least 2 approving reviews are required (per `CONTRIBUTING.md`). + +--- + +### Audit 3 — Ticket State Integrity + +Ensures all issues have correct `State/` labels matching their actual state. + +Checks: +- Closed issues have `State/Completed` (not `State/Unverified`). +- Issues with `State/In Review` have an open PR referencing them. +- No issue has multiple `State/` labels simultaneously. + +--- + +### Audit 4 — Priority and Milestone Ordering + +Ensures Critical bugs on lower milestones are addressed before feature work on +later milestones. + +- Finds the lowest milestone with `Priority/Critical` or `MoSCoW/Must Have` bugs. +- Flags any non-bug issue in `State/In Progress` on a later milestone. + +--- + +### Audit 5 — PR Pipeline Health + +Tracks PR aging, review coverage, and merge throughput. + +| Condition | Severity | +|-----------|----------| +| PR open >24h with no reviews | MEDIUM | +| PR approved but not merged >6h | HIGH | +| PR CI failing >2h | HIGH | + +--- + +### Audit 6 — Supervisor Health (Zombie Detection via Session Introspection) + +Uses the **OpenCode Server API** to read actual session messages and detect +zombie or stuck supervisors. + +**API endpoints used:** +- `GET /session` — list all sessions +- `GET /session/status` — get status of all sessions +- `GET /session/{id}/message?limit=5` — read last 5 messages +- `GET /session/{id}/todo` — read the agent's todo list + +**Detection patterns:** + +| Pattern | Signal | Severity | +|---------|--------|----------| +| Last 3+ tool calls are all `sleep` with zero productive calls | Zombie (likely context exhaustion) | HIGH | +| Last 3+ tool calls all returned errors | Stuck in error loop | HIGH | +| Same tool+args repeated 3+ times in last 5 messages | Circular loop | HIGH | +| Expected supervisor session not running | Missing supervisor | HIGH | + +**Expected supervisors (16 total):** +`implementor-pool`, `reviewer-pool`, `tester-pool`, `hunter-pool`, +`test-infra-pool`, `architect`, `epic-planner`, `human-liaison`, +`agent-evolver`, `arch-guard`, `spec-updater`, `backlog-groomer`, +`docs-writer`, `timeline-updater`, `project-owner`, `system-watchdog` + +--- + +### Audit 7 — Label and Dependency Compliance + +Checks all open issues for required labels and dependency links. + +| Check | Severity | +|-------|----------| +| Missing `State/` label | MEDIUM | +| Missing `Type/` label | MEDIUM | +| Missing `Priority/` label (non-Unverified) | LOW | +| Non-Epic/Legendary issue beyond Unverified with no milestone | MEDIUM | +| Issue with no parent Epic dependency link | LOW | + +--- + +### Audit 8 — Ticket Hierarchy Integrity + +Verifies the Issue → Epic → Legendary hierarchy is intact. + +- Epics must have a parent Legendary dependency link. +- Non-completed Epics should have at least 2 child issues. + +--- + +### Audit 9 — Test Infrastructure Health + +Monitors CI execution times and failure rates for signs of test suite bloat or +flaky tests. (Currently a lightweight check; expanded in future milestones.) + +--- + +### Audit 10 — Improvement Generation + +Verifies the system is generating `needs feedback` improvement tickets. + +- Checks for `needs feedback` issues created in the last 24 hours. +- Flags if none exist (the `ca-spec-updater` and `ca-agent-evolver` should be + generating proposals regularly). + +--- + +### Audit 11 — Quick Session Spot-Check (Every Cycle) + +Fast scan of the **3 most recently active** supervisor sessions for obvious +policy violations. Runs every cycle (~5 min) because catching `force_merge` or +direct master pushes quickly is critical. + +**API endpoints used:** +- `GET /session/status` — find active sessions +- `GET /session/{id}/message?limit=3` — read last 3 messages only + +**Checks:** + +| Violation | Severity | +|-----------|----------| +| `force_merge: true` in any tool call arguments | CRITICAL | +| `git push` directly to `master`/`main` without a feature branch | CRITICAL | +| `type: ignore` suppression written to a file | HIGH | + +--- + +### Audit 12 — Deep Session Introspection (Every 6th Cycle, ~30 min) + +Comprehensive analysis of **all 16 supervisor sessions**. Reads recent messages, +tool calls, and todo lists to understand what each agent is actually doing. + +**API endpoints used:** +- `GET /session/{id}/message?limit=10` — read last 10 messages +- `GET /session/{id}/todo` — read the agent's todo list +- `GET /session/{id}/children` — list child worker sessions + +**Analysis passes:** + +#### Analysis 1 — Misbehavior Detection + +Scans all tool calls in the last 10 messages for policy violations: +- `force_merge: true` usage (CRITICAL) +- Closing PRs as duplicates of their tracking issues (HIGH) +- Creating issues without `State/` or `Type/` labels (MEDIUM) + +#### Analysis 2 — Progress Assessment via Todo List + +Reads the agent's todo list to assess forward progress: +- Flags agents with 0 completed todos and 1+ in-progress todos out of 3+ total + (possible stall or spin). + +#### Analysis 3 — Conversation Health Metrics + +Computes per-session metrics from the last 10 messages: + +| Metric | Threshold | Severity | +|--------|-----------|----------| +| Error rate (errors / total tool calls) | >50% with ≥4 calls | HIGH | +| Sleep ratio (sleep calls / total tool calls) | >80% with ≥5 calls | MEDIUM | + +#### Analysis 4 — Context Exhaustion Signals + +Scans text outputs for phrases indicating the agent is hitting its context window: +`"context limit"`, `"context window"`, `"running out of"`, `"token limit"`, +`"maximum length"`, etc. + +#### Analysis 5 — Cross-Agent Conflict Detection + +Looks for multiple agents interacting with the same PR or issue: +- 3+ different agents touching the same PR → `cross_agent_pr_conflict` (MEDIUM) +- 3+ different agents modifying the same issue labels → `cross_agent_issue_conflict` (MEDIUM) + +--- + +## Finding Severity Levels + +| Severity | Response | +|----------|----------| +| **CRITICAL** | Immediate one-off agent dispatch + bug issue creation | +| **HIGH** | Issue creation or zombie alert posted to session state issue | +| **MEDIUM** | Tracked; issue created if finding persists for 3+ cycles | +| **LOW** | Tracked; no immediate action | + +--- + +## Action Dispatch + +When a CRITICAL finding is detected, the watchdog dispatches a one-off fix agent +via the OpenCode Server API: + +```bash +# Create a one-off session +SESSION_ID=$(curl -s -X POST "${SERVER}/session" \ + -H "Content-Type: application/json" \ + -d '{"title": "[CA-AUTO] one-off: ca-quality-enforcer — missing_ci"}' \ + | jq -r '.id') + +# Dispatch the agent +curl -s -X POST "${SERVER}/session/${SESSION_ID}/prompt_async" \ + -H "Content-Type: application/json" \ + -d '{"agent": "ca-quality-enforcer", + "parts": [{"type": "text", "text": "Fix this finding: ..."}]}' +``` + +**Dispatch table:** + +| Finding Type | Action | +|-------------|--------| +| `no_branch_protection`, `status_check_disabled` | Dispatch `ca-quality-enforcer` | +| `merged_without_ci`, `failing_ci_on_master` | Dispatch `ca-quality-enforcer` + create bug issue | +| `force_merge_detected` | Create bug issue + post CRITICAL alert | +| `direct_push_to_master` | Create bug issue + post CRITICAL alert | +| `closed_wrong_state`, `in_review_but_merged` | Dispatch `ca-state-reconciler` | +| `zombie_supervisor`, `stuck_supervisor`, `looping_supervisor` | Post zombie alert | +| `context_exhaustion` | Post alert (product-builder re-launches the supervisor) | +| `high_error_rate` | Post diagnostic comment | +| `type_ignore_suppression` | Create `Priority/High` bug issue | + +--- + +## Health Reporting + +Every 6 cycles (~30 min), the watchdog posts a health report to the session +state issue: + +``` +[WATCHDOG] Health report — cycle N: + - Quality gate violations: 0 + - State label mismatches: 2 + - Priority ordering issues: 0 + - PR pipeline issues: 1 + - Zombie/stuck/looping supervisors: 0 + - Missing labels/links: 5 + - Session introspection findings: 3 + - Misbehavior (force_merge, direct push): 0 + - Stuck/looping agents: 1 + - Context exhaustion signals: 0 + - Cross-agent conflicts: 2 + - One-off agents dispatched this period: 1 + - Issues created this period: 2 +``` + +--- + +## Configuration + +The watchdog is configured in `.opencode/agents/ca-system-watchdog.md`. + +Key parameters: +- **Model:** `anthropic/claude-opus-4-6` (highest capability for complex analysis) +- **Temperature:** `0.1` (deterministic auditing) +- **Cycle interval:** 5 minutes +- **Deep introspection interval:** Every 6th cycle (~30 min) +- **Context self-management:** Every 20 cycles, discard accumulated tool outputs + +--- + +## Constraints + +The watchdog operates under strict constraints to avoid interfering with the +build system: + +- **No filesystem access** — works exclusively through APIs +- **No merging PRs** — never calls `forgejo_merge_pull_request` +- **No modifying agent definitions** — creates `needs feedback` issues instead +- **No direct master pushes** — all corrections go through one-off agents +- **Corrections limited to:** state label fixes, dependency link fixes, and + issue creation + +--- + +## Related Documentation + +- [Quality Automation](quality-automation.md) — CI/CD quality gates +- [CI/CD Pipeline](ci-cd.md) — pipeline architecture +- [Review Playbook](review_playbook.md) — PR review process diff --git a/mkdocs.yml b/mkdocs.yml index de3458930..83244ee3c 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -29,6 +29,7 @@ nav: - Review Playbook: development/review_playbook.md - Scale Testing: development/scale_testing.md - Ops Runbook: development/ops-runbook.md + - System Watchdog: development/system-watchdog.md - Implementation Timeline: timeline.md - FAQ: faq.md - Reference: reference/ -- 2.52.0