diff --git a/docs/development/system-watchdog.md b/docs/development/system-watchdog.md index e9519726f..11eb959b1 100644 --- a/docs/development/system-watchdog.md +++ b/docs/development/system-watchdog.md @@ -59,6 +59,9 @@ Ensures **no code reaches `master` without passing all CI checks**. - Checks the last 10 commits on `master` for passing `status-check` CI status. - Verifies recently merged PRs had passing CI at merge time. - Detects direct pushes to `master` (all changes must go through PRs). +- When CI failures are detected on `master`, creates a `Priority/CI-Blocker` + issue so the implementation orchestrator can work on it immediately, bypassing + the normal PR-first rule. **Severity:** CRITICAL for any violation. @@ -95,6 +98,8 @@ later milestones. - Finds the lowest milestone with `Priority/Critical` or `MoSCoW/Must Have` bugs. - Flags any non-bug issue in `State/In Progress` on a later milestone. +- `Priority/CI-Blocker` issues are treated as the absolute highest priority — + higher than any milestone ordering. --- @@ -107,6 +112,7 @@ Tracks PR aging, review coverage, and merge throughput. | PR open >24h with no reviews | MEDIUM | | PR approved but not merged >6h | HIGH | | PR CI failing >2h | HIGH | +| PR with 3+ failed fix attempts | HIGH — triggers human assistance request | --- @@ -164,7 +170,15 @@ Verifies the Issue → Epic → Legendary hierarchy is intact. ### Audit 9 — Test Infrastructure Health Monitors CI execution times and failure rates for signs of test suite bloat or -flaky tests. (Currently a lightweight check; expanded in future milestones.) +flaky tests. + +The watchdog can trigger an **emergency test-skip response** when `master` branch +CI is failing due to flaky or broken tests: + +1. Identifies the failing test(s) from CI logs via `ci-log-fetcher`. +2. Creates a `Priority/CI-Blocker` issue for the failing test. +3. Dispatches a one-off agent to add `@skip` / `skip` tags to the failing test. +4. The fix is submitted as a PR and merged to unblock `master`. --- @@ -244,6 +258,16 @@ Looks for multiple agents interacting with the same PR or issue: - 3+ different agents touching the same PR → `cross_agent_pr_conflict` (MEDIUM) - 3+ different agents modifying the same issue labels → `cross_agent_issue_conflict` (MEDIUM) +#### Analysis 6 — Struggling PR Detection + +Identifies PRs where the autonomous system is stuck: +- PRs with 3+ failed fix attempts are flagged as `struggling_pr` (HIGH). +- The watchdog posts a detailed analysis comment on the PR requesting human + assistance, including: all previous fix attempts, CI failure patterns, and + a summary of what has been tried. +- `implementation-worker` agents are notified to gather deep context before + attempting further fixes on these PRs. + --- ## Finding Severity Levels @@ -281,7 +305,7 @@ curl -s -X POST "${SERVER}/session/${SESSION_ID}/prompt_async" \ | Finding Type | Action | |-------------|--------| | `no_branch_protection`, `status_check_disabled` | Dispatch `ca-quality-enforcer` | -| `merged_without_ci`, `failing_ci_on_master` | Dispatch `ca-quality-enforcer` + create bug issue | +| `merged_without_ci`, `failing_ci_on_master` | Dispatch `ca-quality-enforcer` + create `Priority/CI-Blocker` issue | | `force_merge_detected` | Create bug issue + post CRITICAL alert | | `direct_push_to_master` | Create bug issue + post CRITICAL alert | | `closed_wrong_state`, `in_review_but_merged` | Dispatch `ca-state-reconciler` | @@ -289,6 +313,8 @@ curl -s -X POST "${SERVER}/session/${SESSION_ID}/prompt_async" \ | `context_exhaustion` | Post alert (product-builder re-launches the supervisor) | | `high_error_rate` | Post diagnostic comment | | `type_ignore_suppression` | Create `Priority/High` bug issue | +| `struggling_pr` | Post human assistance request on PR + notify `human-liaison` | +| `flaky_test_on_master` | Create `Priority/CI-Blocker` issue + dispatch test-skip agent | --- @@ -310,6 +336,7 @@ state issue: - Stuck/looping agents: 1 - Context exhaustion signals: 0 - Cross-agent conflicts: 2 + - Struggling PRs (3+ failed attempts): 0 - One-off agents dispatched this period: 1 - Issues created this period: 2 ``` @@ -321,11 +348,13 @@ state issue: The watchdog is configured in `.opencode/agents/ca-system-watchdog.md`. Key parameters: -- **Model:** `anthropic/claude-opus-4-6` (highest capability for complex analysis) +- **Model:** `anthropic/claude-sonnet-4-6` (default); escalates to `opus` for complex analysis - **Temperature:** `0.1` (deterministic auditing) - **Cycle interval:** 5 minutes - **Deep introspection interval:** Every 6th cycle (~30 min) - **Context self-management:** Every 20 cycles, discard accumulated tool outputs +- **Mode:** Monitoring and suggestions only — does **not** automatically throttle + or pause agents; reports health issues with fix recommendations instead --- @@ -338,11 +367,33 @@ build system: - **No merging PRs** — never calls `forgejo_merge_pull_request` - **No modifying agent definitions** — creates `needs feedback` issues instead - **No direct master pushes** — all corrections go through one-off agents +- **No automatic throttling** — reports issues with suggestions; humans or + `Priority/CI-Blocker` issues drive remediation - **Corrections limited to:** state label fixes, dependency link fixes, and issue creation --- +## Priority/CI-Blocker Label + +The `Priority/CI-Blocker` label is a special priority label that breaks the +PR-first deadlock: + +**The deadlock:** Broken CI blocks all PR merges. The PR-first rule blocks +CI-fixing issues from being worked. Result: the system cannot fix itself. + +**The solution:** `Priority/CI-Blocker` issues are the **single exception** to +the absolute PR-first rule. The `implementation-orchestrator` and `issue-finder` +dispatch these issues immediately, ahead of any open PRs. + +**Who creates them:** +- `system-watchdog` — when CI failures on `master` are detected (Audit 1, 9) +- `quality-enforcer` — when branch protection violations are found + +**Label ID:** 1396 + +--- + ## Related Documentation - [Quality Automation](quality-automation.md) — CI/CD quality gates diff --git a/docs/modules/ci-log-fetcher.md b/docs/modules/ci-log-fetcher.md new file mode 100644 index 000000000..e75a8afba --- /dev/null +++ b/docs/modules/ci-log-fetcher.md @@ -0,0 +1,187 @@ +# CI Log Fetcher + +**Agent:** `ci-log-fetcher` +**Location:** `.opencode/agents/ci-log-fetcher.md` + +The `ci-log-fetcher` agent retrieves CI job logs from Forgejo Actions using +web authentication. It is a **read-only utility agent** used by all PR-related +agents to access CI failure details without manual web scraping. + +> **Key rule:** Callers **must not** pass `forgejo_username` or +> `forgejo_password` parameters. The agent reads `FORGEJO_USERNAME` and +> `FORGEJO_PASSWORD` from environment variables automatically. + +--- + +## Purpose + +When a PR has failing CI, agents need to read the raw log output to understand +what went wrong. The Forgejo Actions API does not expose logs in a +machine-readable way that is easy to parse, so `ci-log-fetcher` handles the +authentication and log retrieval workflow, returning the raw log text to the +caller. + +--- + +## Usage + +```yaml +# In an agent definition — grant permission to call ci-log-fetcher +permissions: + - ci-log-fetcher +``` + +``` +# In agent prompt / instructions +To get CI logs for PR #1234, job "unit-tests": + Use the ci-log-fetcher subagent with: + pr_number: 1234 + job_name: unit-tests + repository: cleveragents/cleveragents-core + + DO NOT pass forgejo_username or forgejo_password — the agent + handles credentials automatically from environment variables. +``` + +### Parameters + +| Parameter | Required | Description | +|-----------|----------|-------------| +| `pr_number` | Yes | The PR number to fetch logs for | +| `job_name` | Yes | The CI job name (e.g. `unit-tests`, `integration-tests`) | +| `repository` | Yes | `owner/repo` format (e.g. `cleveragents/cleveragents-core`) | + +### Common job names + +| Job | `job_name` value | +|-----|-----------------| +| Unit tests | `unit-tests` or `unit_tests` | +| Integration tests | `integration-tests` or `integration_tests` | +| Lint | `lint` | +| Typecheck | `typecheck` | +| Security scan | `security-scan` or `security_scan` | +| Coverage | `coverage` | + +The agent normalises job names (converts `-` to `_` and vice versa) when +looking up the job in the PR's check runs. + +--- + +## Optimised Workflow + +The agent follows a tested, optimised workflow that minimises execution time +(~5 seconds vs. ~30 seconds for naive approaches): + +``` +1. Load PR page: GET /cleveragents/cleveragents-core/pulls/{pr_number} + → Extracts all CI check run links directly from the page HTML. + → No separate API calls needed to find job IDs. + +2. Find the matching job link by normalised job name. + +3. Fetch logs: GET /actions/runs/{run_id}/jobs/{job_id}/attempt/{attempt}/logs + → Uses attempt=1 on first try (correct ~95% of the time). + → Falls back to attempt=2 if 404. + +4. Return raw log text to caller. +``` + +**What the agent does NOT do (saves time):** +- Does not call `/api/v1/repos/.../actions/runs` (always returns 404 for logs) +- Does not look up CSRF tokens (Forgejo login does not use them) +- Does not make multiple API calls to discover job IDs + +--- + +## Authentication + +The agent uses **web session authentication** (not the Forgejo API token): + +1. `POST https://git.cleverthis.com/user/login` with `FORGEJO_USERNAME` and + `FORGEJO_PASSWORD` to obtain a session cookie. +2. All subsequent requests use the session cookie. + +This is necessary because the Actions log endpoint requires a web session, +not a PAT token. + +**Environment variables used:** +- `FORGEJO_USERNAME` — Forgejo username (e.g. `HAL9000`) +- `FORGEJO_PASSWORD` — Forgejo password + +Both variables are pre-configured in the build environment. Callers do not +need to provide them. + +--- + +## Output Format + +The agent returns the **raw log text** directly — not a JSON wrapper. The +caller receives the full stdout/stderr of the CI job, which can then be +parsed for error messages, test failures, etc. + +Example output: +``` +2026-04-08T10:23:45Z ##[group]Run nox -s unit_tests +2026-04-08T10:23:45Z + nox -s unit_tests +... +2026-04-08T10:24:12Z FAILED features/session.feature:45 — AssertionError +2026-04-08T10:24:12Z 1 feature passed, 0 failed, 1 error +``` + +--- + +## Agents That Use ci-log-fetcher + +All PR-related agents call `ci-log-fetcher` instead of implementing their own +CI log retrieval: + +| Agent | When it calls ci-log-fetcher | +|-------|------------------------------| +| `pr-checker` | After CI fails on a PR being monitored | +| `implementation-worker` | In `pr-fix` mode, to understand CI failures | +| `pr-fix-orchestrator` | To analyse failures before dispatching fix workers | +| `pr-self-reviewer` | To check CI status before reviewing | +| `human-liaison` | When responding to PR comments about CI failures | +| `pr-status-checker` | When `include_logs=true` is requested | +| `system-watchdog` | In Audit 9 (test health) for emergency test-skip response | + +--- + +## Performance Notes + +| Metric | Value | +|--------|-------| +| Typical execution time | ~5 seconds | +| Previous execution time (before optimisation) | ~30 seconds | +| Improvement | ~85% reduction | + +The main savings come from: +- Parsing job links from the PR page (1 request) instead of multi-step API + lookups (4–6 requests) +- Skipping CSRF token lookup (saves ~2 s) +- Using the correct log endpoint pattern on the first attempt (saves retries) + +--- + +## Troubleshooting + +**"Job not found" error:** +- Check that `job_name` matches one of the job names in the PR's check runs. +- Try both hyphenated and underscored variants (e.g. `unit-tests` vs `unit_tests`). +- The agent normalises names, but the PR page must have a matching check run. + +**"Authentication failed" error:** +- Verify `FORGEJO_USERNAME` and `FORGEJO_PASSWORD` are set in the environment. +- The agent will log which credential source it is using (env vars vs parameters). + +**Empty log output:** +- The job may not have produced any output (e.g. it was skipped or cancelled). +- Try fetching a different attempt number by checking the PR page manually. + +--- + +## Related Documentation + +- [CI/CD Pipeline](../development/ci-cd.md) — pipeline architecture +- [System Watchdog](../development/system-watchdog.md) — uses ci-log-fetcher for emergency test-skip +- [Quality Automation](../development/quality-automation.md) — CI quality gates