cleveragents-core/docs/development/system-watchdog.md

# System Watchdog

The **System Watchdog** (`ca-system-watchdog`) is the 16th supervisor agent in the
CleverAgents autonomous build system. It runs continuously with a 5-minute polling
cycle and monitors the entire agent ecosystem for correctness, health, and policy
compliance.

> **Audience:** Contributors and operators who need to understand how the autonomous
> agent system is monitored, what violations are detected, and how the watchdog
> responds to findings.

---

## Overview

The watchdog operates exclusively through the **Forgejo API** and the
**OpenCode Server HTTP API** — it does not clone the repository or access the
filesystem. It performs 12 audits per cycle, ranging from quality gate compliance
to deep session introspection of individual supervisor conversations.

```
cycle = 0
LOOP FOREVER (5-minute intervals):
    cycle += 1
    findings = []

    findings += audit_quality_gates()           # Audit 1 — MOST CRITICAL
    findings += audit_branch_protection()       # Audit 2
    findings += audit_ticket_states()           # Audit 3
    findings += audit_priority_ordering()       # Audit 4
    findings += audit_pr_pipeline()             # Audit 5
    findings += audit_supervisor_health()       # Audit 6 (session introspection)
    findings += audit_labels_and_dependencies() # Audit 7
    findings += audit_ticket_hierarchy()        # Audit 8
    findings += audit_test_health()             # Audit 9
    findings += audit_improvement_generation()  # Audit 10
    findings += audit_session_spot_check()      # Audit 11 (every cycle)

    if cycle % 6 == 0:
        findings += audit_deep_session_introspection()  # Audit 12 (every 30 min)

    for finding in findings:
        take_action(finding)

    if cycle % 6 == 0 and findings:
        post_summary(cycle, findings)

    sleep(300)  # 5 minutes
```

---

## Audit Descriptions

### Audit 1 — Quality Gate Compliance (CRITICAL)

Ensures **no code reaches `master` without passing all CI checks**.

- Checks the last 10 commits on `master` for passing `status-check` CI status.
- Verifies recently merged PRs had passing CI at merge time.
- Detects direct pushes to `master` (all changes must go through PRs).
- When CI failures are detected on `master`, creates a `Priority/CI-Blocker`
  issue so the implementation orchestrator can work on it immediately, bypassing
  the normal PR-first rule.

**Severity:** CRITICAL for any violation.

---

### Audit 2 — Branch Protection Verification

Verifies that Forgejo branch protection is active and correctly configured for
`master`.

Checks:
- Branch protection rules exist for `master`.
- `enable_status_check` is `true`.
- The `status-check` context is required.
- At least 1 approving review is required (per `CONTRIBUTING.md`).

---

### Audit 3 — Ticket State Integrity

Ensures all issues have correct `State/` labels matching their actual state.

Checks:
- Closed issues have `State/Completed` (not `State/Unverified`).
- Issues with `State/In Review` have an open PR referencing them.
- No issue has multiple `State/` labels simultaneously.

---

### Audit 4 — Priority and Milestone Ordering

Ensures Critical bugs on lower milestones are addressed before feature work on
later milestones.

- Finds the lowest milestone with `Priority/Critical` or `MoSCoW/Must Have` bugs.
- Flags any non-bug issue in `State/In Progress` on a later milestone.
- `Priority/CI-Blocker` issues are treated as the absolute highest priority —
  higher than any milestone ordering.

---

### Audit 5 — PR Pipeline Health

Tracks PR aging, review coverage, and merge throughput.

| Condition | Severity |
|-----------|----------|
| PR open >24h with no reviews | MEDIUM |
| PR approved but not merged >6h | HIGH |
| PR CI failing >2h | HIGH |
| PR with 3+ failed fix attempts | HIGH — triggers human assistance request |

---

### Audit 6 — Supervisor Health (Zombie Detection via Session Introspection)

Uses the **OpenCode Server API** to read actual session messages and detect
zombie or stuck supervisors.

**API endpoints used:**
- `GET /session` — list all sessions
- `GET /session/status` — get status of all sessions
- `GET /session/{id}/message?limit=5` — read last 5 messages
- `GET /session/{id}/todo` — read the agent's todo list

**Detection patterns:**

| Pattern | Signal | Severity |
|---------|--------|----------|
| Last 3+ tool calls are all `sleep` with zero productive calls | Zombie (likely context exhaustion) | HIGH |
| Last 3+ tool calls all returned errors | Stuck in error loop | HIGH |
| Same tool+args repeated 3+ times in last 5 messages | Circular loop | HIGH |
| Expected supervisor session not running | Missing supervisor | HIGH |

**Expected supervisors (16 total):**
`implementor-pool`, `reviewer-pool`, `tester-pool`, `hunter-pool`,
`test-infra-pool`, `architect`, `epic-planner`, `human-liaison`,
`agent-evolver`, `arch-guard`, `spec-updater`, `backlog-groomer`,
`docs-writer`, `timeline-updater`, `project-owner`, `system-watchdog`

---

### Audit 7 — Label and Dependency Compliance

Checks all open issues for required labels and dependency links.

| Check | Severity |
|-------|----------|
| Missing `State/` label | MEDIUM |
| Missing `Type/` label | MEDIUM |
| Missing `Priority/` label (non-Unverified) | LOW |
| Non-Epic/Legendary issue beyond Unverified with no milestone | MEDIUM |
| Issue with no parent Epic dependency link | LOW |

---

### Audit 8 — Ticket Hierarchy Integrity

Verifies the Issue → Epic → Legendary hierarchy is intact.

- Epics must have a parent Legendary dependency link.
- Non-completed Epics should have at least 2 child issues.

---

### Audit 9 — Test Infrastructure Health

Monitors CI execution times and failure rates for signs of test suite bloat or
flaky tests.

The watchdog can trigger an **emergency test-skip response** when `master` branch
CI is failing due to flaky or broken tests:

1. Identifies the failing test(s) from CI logs via `ci-log-fetcher`.
2. Creates a `Priority/CI-Blocker` issue for the failing test.
3. Dispatches a one-off agent to add `@skip` / `skip` tags to the failing test.
4. The fix is submitted as a PR and merged to unblock `master`.

---

### Audit 10 — Improvement Generation

Verifies the system is generating `needs feedback` improvement tickets.

- Checks for `needs feedback` issues created in the last 24 hours.
- Flags if none exist (the `ca-spec-updater` and `ca-agent-evolver` should be
  generating proposals regularly).

---

### Audit 11 — Quick Session Spot-Check (Every Cycle)

Fast scan of the **3 most recently active** supervisor sessions for obvious
policy violations. Runs every cycle (~5 min) because catching `force_merge` or
direct master pushes quickly is critical.

**API endpoints used:**
- `GET /session/status` — find active sessions
- `GET /session/{id}/message?limit=3` — read last 3 messages only

**Checks:**

| Violation | Severity |
|-----------|----------|
| `force_merge: true` in any tool call arguments | CRITICAL |
| `git push` directly to `master`/`main` without a feature branch | CRITICAL |
| `type: ignore` suppression written to a file | HIGH |

---

### Audit 12 — Deep Session Introspection (Every 6th Cycle, ~30 min)

Comprehensive analysis of **all 16 supervisor sessions**. Reads recent messages,
tool calls, and todo lists to understand what each agent is actually doing.

**API endpoints used:**
- `GET /session/{id}/message?limit=10` — read last 10 messages
- `GET /session/{id}/todo` — read the agent's todo list
- `GET /session/{id}/children` — list child worker sessions

**Analysis passes:**

#### Analysis 1 — Misbehavior Detection

Scans all tool calls in the last 10 messages for policy violations:
- `force_merge: true` usage (CRITICAL)
- Closing PRs as duplicates of their tracking issues (HIGH)
- Creating issues without `State/` or `Type/` labels (MEDIUM)

#### Analysis 2 — Progress Assessment via Todo List

Reads the agent's todo list to assess forward progress:
- Flags agents with 0 completed todos and 1+ in-progress todos out of 3+ total
  (possible stall or spin).

#### Analysis 3 — Conversation Health Metrics

Computes per-session metrics from the last 10 messages:

| Metric | Threshold | Severity |
|--------|-----------|----------|
| Error rate (errors / total tool calls) | >50% with ≥4 calls | HIGH |
| Sleep ratio (sleep calls / total tool calls) | >80% with ≥5 calls | MEDIUM |

#### Analysis 4 — Context Exhaustion Signals

Scans text outputs for phrases indicating the agent is hitting its context window:
`"context limit"`, `"context window"`, `"running out of"`, `"token limit"`,
`"maximum length"`, etc.

#### Analysis 5 — Cross-Agent Conflict Detection

Looks for multiple agents interacting with the same PR or issue:
- 3+ different agents touching the same PR → `cross_agent_pr_conflict` (MEDIUM)
- 3+ different agents modifying the same issue labels → `cross_agent_issue_conflict` (MEDIUM)

#### Analysis 6 — Struggling PR Detection

Identifies PRs where the autonomous system is stuck:
- PRs with 3+ failed fix attempts are flagged as `struggling_pr` (HIGH).
- The watchdog posts a detailed analysis comment on the PR requesting human
  assistance, including: all previous fix attempts, CI failure patterns, and
  a summary of what has been tried.
- `implementation-worker` agents are notified to gather deep context before
  attempting further fixes on these PRs.

---

## Finding Severity Levels

| Severity | Response |
|----------|----------|
| **CRITICAL** | Immediate one-off agent dispatch + bug issue creation |
| **HIGH** | Issue creation or zombie alert posted to session state issue |
| **MEDIUM** | Tracked; issue created if finding persists for 3+ cycles |
| **LOW** | Tracked; no immediate action |

---

## Action Dispatch

When a CRITICAL finding is detected, the watchdog dispatches a one-off fix agent
via the OpenCode Server API:

```bash
# Create a one-off session
SESSION_ID=$(curl -s -X POST "${SERVER}/session" \
  -H "Content-Type: application/json" \
  -d '{"title": "[CA-AUTO] one-off: ca-quality-enforcer — missing_ci"}' \
  | jq -r '.id')

# Dispatch the agent
curl -s -X POST "${SERVER}/session/${SESSION_ID}/prompt_async" \
  -H "Content-Type: application/json" \
  -d '{"agent": "ca-quality-enforcer",
       "parts": [{"type": "text", "text": "Fix this finding: ..."}]}'
```

**Dispatch table:**

| Finding Type | Action |
|-------------|--------|
| `no_branch_protection`, `status_check_disabled` | Dispatch `ca-quality-enforcer` |
| `merged_without_ci`, `failing_ci_on_master` | Dispatch `ca-quality-enforcer` + create `Priority/CI-Blocker` issue |
| `force_merge_detected` | Create bug issue + post CRITICAL alert |
| `direct_push_to_master` | Create bug issue + post CRITICAL alert |
| `closed_wrong_state`, `in_review_but_merged` | Dispatch `ca-state-reconciler` |
| `zombie_supervisor`, `stuck_supervisor`, `looping_supervisor` | Post zombie alert |
| `context_exhaustion` | Post alert (product-builder re-launches the supervisor) |
| `high_error_rate` | Post diagnostic comment |
| `type_ignore_suppression` | Create `Priority/High` bug issue |
| `struggling_pr` | Post human assistance request on PR + notify `human-liaison` |
| `flaky_test_on_master` | Create `Priority/CI-Blocker` issue + dispatch test-skip agent |

---

## Health Reporting

Every 6 cycles (~30 min), the watchdog posts a health report to the session
state issue:

```
[WATCHDOG] Health report — cycle N:
 - Quality gate violations: 0
 - State label mismatches: 2
 - Priority ordering issues: 0
 - PR pipeline issues: 1
 - Zombie/stuck/looping supervisors: 0
 - Missing labels/links: 5
 - Session introspection findings: 3
   - Misbehavior (force_merge, direct push): 0
   - Stuck/looping agents: 1
   - Context exhaustion signals: 0
   - Cross-agent conflicts: 2
   - Struggling PRs (3+ failed attempts): 0
 - One-off agents dispatched this period: 1
 - Issues created this period: 2
```

---

## Configuration

The watchdog is configured in `.opencode/agents/ca-system-watchdog.md`.

Key parameters:
- **Model:** `anthropic/claude-sonnet-4-6` (default); escalates to `opus` for complex analysis
- **Temperature:** `0.1` (deterministic auditing)
- **Cycle interval:** 5 minutes
- **Deep introspection interval:** Every 6th cycle (~30 min)
- **Context self-management:** Every 20 cycles, discard accumulated tool outputs
- **Mode:** Monitoring and suggestions only — does **not** automatically throttle
  or pause agents; reports health issues with fix recommendations instead

---

## Constraints

The watchdog operates under strict constraints to avoid interfering with the
build system:

- **No filesystem access** — works exclusively through APIs
- **No merging PRs** — never calls `forgejo_merge_pull_request`
- **No modifying agent definitions** — creates `needs feedback` issues instead
- **No direct master pushes** — all corrections go through one-off agents
- **No automatic throttling** — reports issues with suggestions; humans or
  `Priority/CI-Blocker` issues drive remediation
- **Corrections limited to:** state label fixes, dependency link fixes, and
  issue creation

---

## Priority/CI-Blocker Label

The `Priority/CI-Blocker` label is a special priority label that breaks the
PR-first deadlock:

**The deadlock:** Broken CI blocks all PR merges. The PR-first rule blocks
CI-fixing issues from being worked. Result: the system cannot fix itself.

**The solution:** `Priority/CI-Blocker` issues are the **single exception** to
the absolute PR-first rule. The `implementation-orchestrator` and `issue-finder`
dispatch these issues immediately, ahead of any open PRs.

**Who creates them:**
- `system-watchdog` — when CI failures on `master` are detected (Audit 1, 9)
- `quality-enforcer` — when branch protection violations are found

**Label ID:** 1396

---

## Related Documentation

- [Quality Automation](quality-automation.md) — CI/CD quality gates
- [CI/CD Pipeline](ci-cd.md) — pipeline architecture
- [Review Playbook](review_playbook.md) — PR review process