docs: document 2026-04-06 to 2026-04-08 improvements

- add changelog entries summarizing watchdog supervision and documentation updates\n- add ci-log-fetcher module guide detailing interface contract and usage\n- refresh system-watchdog doc with CI-blocker escalation and struggling PR detection\n- normalize coverage threshold robot test by dropping temporary tdd_expected_fail tags\n\nISSUES CLOSED: #7987

docs/development/system-watchdog.md

@@ -59,6 +59,9 @@ Ensures **no code reaches `master` without passing all CI checks**.
 - Checks the last 10 commits on `master` for passing `status-check` CI status.
 - Verifies recently merged PRs had passing CI at merge time.
 - Detects direct pushes to `master` (all changes must go through PRs).
+- When CI failures are detected on `master`, creates a `Priority/CI-Blocker`
+  issue so the implementation orchestrator can work on it immediately, bypassing
+  the normal PR-first rule.
 
 **Severity:** CRITICAL for any violation.
 
@@ -95,6 +98,8 @@ later milestones.
 
 - Finds the lowest milestone with `Priority/Critical` or `MoSCoW/Must Have` bugs.
 - Flags any non-bug issue in `State/In Progress` on a later milestone.
+- `Priority/CI-Blocker` issues are treated as the absolute highest priority —
+  higher than any milestone ordering.
 
 ---
 
@@ -107,6 +112,7 @@ Tracks PR aging, review coverage, and merge throughput.
 | PR open >24h with no reviews | MEDIUM |
 | PR approved but not merged >6h | HIGH |
 | PR CI failing >2h | HIGH |
+| PR with 3+ failed fix attempts | HIGH — triggers human assistance request |
 
 ---
 
@@ -164,7 +170,15 @@ Verifies the Issue → Epic → Legendary hierarchy is intact.
 ### Audit 9 — Test Infrastructure Health
 
 Monitors CI execution times and failure rates for signs of test suite bloat or
-flaky tests. (Currently a lightweight check; expanded in future milestones.)
+flaky tests.
+
+The watchdog can trigger an **emergency test-skip response** when `master` branch
+CI is failing due to flaky or broken tests:
+
+1. Identifies the failing test(s) from CI logs via `ci-log-fetcher`.
+2. Creates a `Priority/CI-Blocker` issue for the failing test.
+3. Dispatches a one-off agent to add `@skip` / `skip` tags to the failing test.
+4. The fix is submitted as a PR and merged to unblock `master`.
 
 ---
 
@@ -244,6 +258,16 @@ Looks for multiple agents interacting with the same PR or issue:
 - 3+ different agents touching the same PR → `cross_agent_pr_conflict` (MEDIUM)
 - 3+ different agents modifying the same issue labels → `cross_agent_issue_conflict` (MEDIUM)
 
+#### Analysis 6 — Struggling PR Detection
+
+Identifies PRs where the autonomous system is stuck:
+- PRs with 3+ failed fix attempts are flagged as `struggling_pr` (HIGH).
+- The watchdog posts a detailed analysis comment on the PR requesting human
+  assistance, including: all previous fix attempts, CI failure patterns, and
+  a summary of what has been tried.
+- `implementation-worker` agents are notified to gather deep context before
+  attempting further fixes on these PRs.
+
 ---
 
 ## Finding Severity Levels
@@ -281,7 +305,7 @@ curl -s -X POST "${SERVER}/session/${SESSION_ID}/prompt_async" \
 | Finding Type | Action |
 |-------------|--------|
 | `no_branch_protection`, `status_check_disabled` | Dispatch `ca-quality-enforcer` |
-| `merged_without_ci`, `failing_ci_on_master` | Dispatch `ca-quality-enforcer` + create bug issue |
+| `merged_without_ci`, `failing_ci_on_master` | Dispatch `ca-quality-enforcer` + create `Priority/CI-Blocker` issue |
 | `force_merge_detected` | Create bug issue + post CRITICAL alert |
 | `direct_push_to_master` | Create bug issue + post CRITICAL alert |
 | `closed_wrong_state`, `in_review_but_merged` | Dispatch `ca-state-reconciler` |
@@ -289,6 +313,8 @@ curl -s -X POST "${SERVER}/session/${SESSION_ID}/prompt_async" \
 | `context_exhaustion` | Post alert (product-builder re-launches the supervisor) |
 | `high_error_rate` | Post diagnostic comment |
 | `type_ignore_suppression` | Create `Priority/High` bug issue |
+| `struggling_pr` | Post human assistance request on PR + notify `human-liaison` |
+| `flaky_test_on_master` | Create `Priority/CI-Blocker` issue + dispatch test-skip agent |
 
 ---
 
@@ -310,6 +336,7 @@ state issue:
    - Stuck/looping agents: 1
    - Context exhaustion signals: 0
    - Cross-agent conflicts: 2
+   - Struggling PRs (3+ failed attempts): 0
  - One-off agents dispatched this period: 1
  - Issues created this period: 2
 ```
@@ -321,11 +348,13 @@ state issue:
 The watchdog is configured in `.opencode/agents/ca-system-watchdog.md`.
 
 Key parameters:
-- **Model:** `anthropic/claude-opus-4-6` (highest capability for complex analysis)
+- **Model:** `anthropic/claude-sonnet-4-6` (default); escalates to `opus` for complex analysis
 - **Temperature:** `0.1` (deterministic auditing)
 - **Cycle interval:** 5 minutes
 - **Deep introspection interval:** Every 6th cycle (~30 min)
 - **Context self-management:** Every 20 cycles, discard accumulated tool outputs
+- **Mode:** Monitoring and suggestions only — does **not** automatically throttle
+  or pause agents; reports health issues with fix recommendations instead
 
 ---
 
@@ -338,11 +367,33 @@ build system:
 - **No merging PRs** — never calls `forgejo_merge_pull_request`
 - **No modifying agent definitions** — creates `needs feedback` issues instead
 - **No direct master pushes** — all corrections go through one-off agents
+- **No automatic throttling** — reports issues with suggestions; humans or
+  `Priority/CI-Blocker` issues drive remediation
 - **Corrections limited to:** state label fixes, dependency link fixes, and
   issue creation
 
 ---
 
+## Priority/CI-Blocker Label
+
+The `Priority/CI-Blocker` label is a special priority label that breaks the
+PR-first deadlock:
+
+**The deadlock:** Broken CI blocks all PR merges. The PR-first rule blocks
+CI-fixing issues from being worked. Result: the system cannot fix itself.
+
+**The solution:** `Priority/CI-Blocker` issues are the **single exception** to
+the absolute PR-first rule. The `implementation-orchestrator` and `issue-finder`
+dispatch these issues immediately, ahead of any open PRs.
+
+**Who creates them:**
+- `system-watchdog` — when CI failures on `master` are detected (Audit 1, 9)
+- `quality-enforcer` — when branch protection violations are found
+
+**Label ID:** 1396
+
+---
+
 ## Related Documentation
 
 - [Quality Automation](quality-automation.md) — CI/CD quality gates

docs/modules/ci-log-fetcher.md

@@ -0,0 +1,187 @@
+# CI Log Fetcher
+
+**Agent:** `ci-log-fetcher`
+**Location:** `.opencode/agents/ci-log-fetcher.md`
+
+The `ci-log-fetcher` agent retrieves CI job logs from Forgejo Actions using
+web authentication. It is a **read-only utility agent** used by all PR-related
+agents to access CI failure details without manual web scraping.
+
+> **Key rule:** Callers **must not** pass `forgejo_username` or
+> `forgejo_password` parameters. The agent reads `FORGEJO_USERNAME` and
+> `FORGEJO_PASSWORD` from environment variables automatically.
+
+---
+
+## Purpose
+
+When a PR has failing CI, agents need to read the raw log output to understand
+what went wrong. The Forgejo Actions API does not expose logs in a
+machine-readable way that is easy to parse, so `ci-log-fetcher` handles the
+authentication and log retrieval workflow, returning the raw log text to the
+caller.
+
+---
+
+## Usage
+
+```yaml
+# In an agent definition — grant permission to call ci-log-fetcher
+permissions:
+  - ci-log-fetcher
+```
+
+```
+# In agent prompt / instructions
+To get CI logs for PR #1234, job "unit-tests":
+  Use the ci-log-fetcher subagent with:
+    pr_number: 1234
+    job_name: unit-tests
+    repository: cleveragents/cleveragents-core
+  
+  DO NOT pass forgejo_username or forgejo_password — the agent
+  handles credentials automatically from environment variables.
+```
+
+### Parameters
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `pr_number` | Yes | The PR number to fetch logs for |
+| `job_name` | Yes | The CI job name (e.g. `unit-tests`, `integration-tests`) |
+| `repository` | Yes | `owner/repo` format (e.g. `cleveragents/cleveragents-core`) |
+
+### Common job names
+
+| Job | `job_name` value |
+|-----|-----------------|
+| Unit tests | `unit-tests` or `unit_tests` |
+| Integration tests | `integration-tests` or `integration_tests` |
+| Lint | `lint` |
+| Typecheck | `typecheck` |
+| Security scan | `security-scan` or `security_scan` |
+| Coverage | `coverage` |
+
+The agent normalises job names (converts `-` to `_` and vice versa) when
+looking up the job in the PR's check runs.
+
+---
+
+## Optimised Workflow
+
+The agent follows a tested, optimised workflow that minimises execution time
+(~5 seconds vs. ~30 seconds for naive approaches):
+
+```
+1. Load PR page: GET /cleveragents/cleveragents-core/pulls/{pr_number}
+   → Extracts all CI check run links directly from the page HTML.
+   → No separate API calls needed to find job IDs.
+
+2. Find the matching job link by normalised job name.
+
+3. Fetch logs: GET /actions/runs/{run_id}/jobs/{job_id}/attempt/{attempt}/logs
+   → Uses attempt=1 on first try (correct ~95% of the time).
+   → Falls back to attempt=2 if 404.
+
+4. Return raw log text to caller.
+```
+
+**What the agent does NOT do (saves time):**
+- Does not call `/api/v1/repos/.../actions/runs` (always returns 404 for logs)
+- Does not look up CSRF tokens (Forgejo login does not use them)
+- Does not make multiple API calls to discover job IDs
+
+---
+
+## Authentication
+
+The agent uses **web session authentication** (not the Forgejo API token):
+
+1. `POST https://git.cleverthis.com/user/login` with `FORGEJO_USERNAME` and
+   `FORGEJO_PASSWORD` to obtain a session cookie.
+2. All subsequent requests use the session cookie.
+
+This is necessary because the Actions log endpoint requires a web session,
+not a PAT token.
+
+**Environment variables used:**
+- `FORGEJO_USERNAME` — Forgejo username (e.g. `HAL9000`)
+- `FORGEJO_PASSWORD` — Forgejo password
+
+Both variables are pre-configured in the build environment. Callers do not
+need to provide them.
+
+---
+
+## Output Format
+
+The agent returns the **raw log text** directly — not a JSON wrapper. The
+caller receives the full stdout/stderr of the CI job, which can then be
+parsed for error messages, test failures, etc.
+
+Example output:
+```
+2026-04-08T10:23:45Z ##[group]Run nox -s unit_tests
+2026-04-08T10:23:45Z + nox -s unit_tests
+...
+2026-04-08T10:24:12Z FAILED features/session.feature:45 — AssertionError
+2026-04-08T10:24:12Z 1 feature passed, 0 failed, 1 error
+```
+
+---
+
+## Agents That Use ci-log-fetcher
+
+All PR-related agents call `ci-log-fetcher` instead of implementing their own
+CI log retrieval:
+
+| Agent | When it calls ci-log-fetcher |
+|-------|------------------------------|
+| `pr-checker` | After CI fails on a PR being monitored |
+| `implementation-worker` | In `pr-fix` mode, to understand CI failures |
+| `pr-fix-orchestrator` | To analyse failures before dispatching fix workers |
+| `pr-self-reviewer` | To check CI status before reviewing |
+| `human-liaison` | When responding to PR comments about CI failures |
+| `pr-status-checker` | When `include_logs=true` is requested |
+| `system-watchdog` | In Audit 9 (test health) for emergency test-skip response |
+
+---
+
+## Performance Notes
+
+| Metric | Value |
+|--------|-------|
+| Typical execution time | ~5 seconds |
+| Previous execution time (before optimisation) | ~30 seconds |
+| Improvement | ~85% reduction |
+
+The main savings come from:
+- Parsing job links from the PR page (1 request) instead of multi-step API
+  lookups (4–6 requests)
+- Skipping CSRF token lookup (saves ~2 s)
+- Using the correct log endpoint pattern on the first attempt (saves retries)
+
+---
+
+## Troubleshooting
+
+**"Job not found" error:**
+- Check that `job_name` matches one of the job names in the PR's check runs.
+- Try both hyphenated and underscored variants (e.g. `unit-tests` vs `unit_tests`).
+- The agent normalises names, but the PR page must have a matching check run.
+
+**"Authentication failed" error:**
+- Verify `FORGEJO_USERNAME` and `FORGEJO_PASSWORD` are set in the environment.
+- The agent will log which credential source it is using (env vars vs parameters).
+
+**Empty log output:**
+- The job may not have produced any output (e.g. it was skipped or cancelled).
+- Try fetching a different attempt number by checking the PR page manually.
+
+---
+
+## Related Documentation
+
+- [CI/CD Pipeline](../development/ci-cd.md) — pipeline architecture
+- [System Watchdog](../development/system-watchdog.md) — uses ci-log-fetcher for emergency test-skip
+- [Quality Automation](../development/quality-automation.md) — CI quality gates