cleveragents-core/docs/development/ci-incident-runbook.md

CI Incident Runbook: Master Branch Quality Gate Failures

This runbook documents how to diagnose, triage, and resolve CI quality gate failures on the master branch. It was written in response to issue #2597, which identified a critical pattern of direct pushes to master bypassing the PR process and introducing regressions.

Why a Broken Master Is a Critical Incident

When master CI is broken, all development work stops:

1. All open PRs are blocked. The status-check consolidation gate requires all 11 CI jobs to succeed. Any failure prevents any PR from merging.
2. New branches inherit failures. Any branch created from a broken master starts with broken code.
3. Branch protection enforces up-to-date branches. PRs must be rebased on master before they can merge — a broken master guarantees they will also fail CI.
4. The TDD workflow is broken. TDD issue-capture tests require merging to master before bug fixes can begin.
5. No releases can be cut. Release tags are pushed from master; a broken master means no releases.

Treat a broken master as the single highest-priority incident. All other work is blocked until master is green.

---

The 11-Job Status-Check Gate

The status-check consolidation job (defined in .forgejo/workflows/ci.yml) requires all of the following jobs to report success before any PR can merge:

#	CI Job	Nox Session(s)	What It Checks
1	lint	nox -s lint + nox -s format -- --check	Ruff lint rules and code formatting
2	typecheck	nox -s typecheck	Pyright strict type checking
3	security	nox -s security_scan + nox -s dead_code	Bandit HIGH gate, Semgrep custom rules, Vulture dead-code (≥80% confidence)
4	quality	nox -s complexity	Radon cyclomatic complexity (no grade-F methods)
5	unit_tests	nox -s unit_tests	All Behave BDD scenarios under features/
6	integration_tests	nox -s integration_tests	All Robot Framework tests under robot/ (excluding slow, discovery, code_blocks, wip, E2E, tdd_fixture)
7	e2e_tests	nox -s e2e_tests	End-to-end Robot tests with real LLM API keys under robot/e2e/
8	coverage	nox -s coverage_report	Slipcover test coverage ≥ 97% fail-under threshold
9	build	nox -s build	Python wheel build
10	docker	Docker CLI	Docker image build (Dockerfile + Dockerfile.server) and smoke test
11	helm	Helm CLI + kubeconform	Helm chart lint, template render, and Kubernetes manifest validation

The coverage job additionally depends on lint, typecheck, security, and quality passing first. The docker job depends on lint, typecheck, security, quality, and unit_tests.

---

How to Diagnose a Broken Master

Step 1: Identify the Failing Jobs

Navigate to the Forgejo Actions UI for the latest master commit and check which jobs are failing. The status-check job output lists each job's result:

lint: failure
typecheck: success
security: success
quality: success
unit_tests: failure
integration_tests: success
e2e_tests: failure
coverage: skipped
build: success
docker: skipped
helm: success

Download the log artifact for each failing job (e.g., ci-logs-lint, ci-logs-unit-tests, ci-logs-e2e-tests) from the Actions UI under Artifacts.

Step 2: Identify the Root Cause Commit

Check the recent commit history on master for any direct pushes (commits not from a PR merge):

git log --oneline -10 origin/master

Direct pushes bypass CI entirely. Look for commits that were not preceded by a PR merge commit. These are the most likely source of regressions.

Step 3: Reproduce Locally

Clone the master branch and run the failing nox sessions locally:

git clone https://git.cleverthis.com/cleveragents/cleveragents-core.git
cd cleveragents-core
git checkout master

# Run the specific failing sessions
nox -s lint
nox -s format -- --check
nox -s unit_tests
nox -s e2e_tests
nox -s coverage_report

---

Triage by Failure Type

Lint Failures (lint job)

Symptoms: Ruff reports lint violations or format differences.

Reproduce:

nox -s lint
nox -s format -- --check

Fix: Run nox -s format to auto-fix formatting, then fix any remaining lint violations reported by nox -s lint. Commit the result.

Prohibited fixes:

• Adding # noqa comments
• Modifying pyproject.toml [tool.ruff] to disable rules or add ignores

Unit Test Failures (unit_tests job)

Symptoms: One or more Behave BDD scenarios fail.

Reproduce:

nox -s unit_tests
# Or run a specific feature file:
nox -s unit_tests -- features/failing_feature.feature

Fix: Identify the failing scenario and fix the source code to match the expected behavior. Do not modify test expectations or skip scenarios.

Prohibited fixes:

• Adding @skip, @xfail, or @unittest.skip tags to scenarios
• Deleting or removing test scenarios or assertions
• Modifying test expectations to match broken behavior

Integration Test Failures (integration_tests job)

Symptoms: One or more Robot Framework tests fail.

Reproduce:

nox -s integration_tests
# Or run a specific suite:
nox -s integration_tests -- --suite robot/failing_suite.robot

Note: Integration tests require real LLM API keys (ANTHROPIC_API_KEY, OPENAI_API_KEY). Set these in your environment before running locally.

Fix: Identify the failing test and fix the underlying source code. Check robot/helpers_common.py for the shared reset_global_state() function if tests are failing due to state leakage between test runs.

E2E Test Failures (e2e_tests job)

Symptoms: End-to-end Robot Framework tests under robot/e2e/ fail.

Reproduce:

nox -s e2e_tests

Note: E2E tests require real LLM API keys (ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOGLE_API_KEY). These tests exercise full end-to-end workflows with live LLM providers and have a 45-minute timeout in CI.

Fix: Fix the underlying source code. E2E tests must not be skipped or mocked.

Coverage Failures (coverage job)

Symptoms: nox -s coverage_report reports coverage below 97%.

Reproduce:

nox -s coverage_report
# Open the HTML report to identify uncovered lines:
open build/htmlcov/index.html

Fix: Write new Behave scenarios targeting the uncovered code paths. Coverage must be ≥97% as measured by nox -s coverage_report.

Prohibited fixes:

• Lowering the --fail-under threshold in noxfile.py
• Adding # pragma: no cover comments
• Expanding the omit list in pyproject.toml [tool.coverage.run]

Type Check Failures (typecheck job)

Symptoms: Pyright reports type errors.

Reproduce:

nox -s typecheck

Fix: Fix the type errors in source code. Add explicit type annotations where missing.

Prohibited fixes:

• Adding # type: ignore comments
• Modifying pyrightconfig.json to relax strictness or add exclusions
• Adding paths to pyproject.toml [tool.pyright] exclusions

Security Failures (security job)

Symptoms: Bandit reports HIGH-severity findings, Semgrep detects custom rule violations, or Vulture detects dead code.

Reproduce:

nox -s security_scan
nox -s dead_code

Fix:

• Bandit: Fix the security issue in source code. If it is a genuine false positive, consider a safer alternative first. Only as a last resort, add to [tool.bandit] skips in pyproject.toml.
• Semgrep: Fix the code pattern flagged by the custom rule (see .semgrep.yml for rule descriptions).
• Vulture: If the symbol is intentionally unused (e.g., required by a protocol), add it to vulture_whitelist.py with a comment explaining why.

Build Failures (build job)

Symptoms: nox -s build fails to produce a wheel.

Reproduce:

nox -s build

Fix: Fix the build error. Common causes include missing __init__.py files, broken imports, or pyproject.toml misconfiguration.

Docker Failures (docker job)

Symptoms: Docker image build fails or the smoke test (--version) fails.

Reproduce:

docker build -t cleveragents:test .
docker run --rm cleveragents:test --version
docker build -f Dockerfile.server -t cleveragents-server:test .

Fix: Fix the Dockerfile or the application code causing the build or smoke test failure.

Helm Failures (helm job)

Symptoms: Helm lint, template render, or kubeconform validation fails.

Reproduce:

helm dependency build ./k8s
helm lint ./k8s --set database.url="postgresql+asyncpg://user:pass@db-host:5432/cleveragents"
helm template cleveragents ./k8s \
  --set database.url="postgresql+asyncpg://user:pass@db-host:5432/cleveragents" > /tmp/rendered.yaml
kubeconform -strict -ignore-missing-schemas -kubernetes-version 1.29.0 -summary /tmp/rendered.yaml

Fix: Fix the Helm chart templates or values in k8s/.

---

The Fix Branch Workflow

When master is broken, the fix must go through the PR process — never push directly to master again:

1. Create a fix branch from the current master HEAD:

   git checkout -b fix/master-ci-quality-gates origin/master
   

2. Fix the actual code (see triage sections above). Never suppress or bypass quality gates.

3. Verify locally that all nox sessions pass:

   nox  # runs all default sessions
   

4. Commit with the conventional commit format:

   fix(ci): restore all CI quality gates to passing on master
   
   - Fixed lint violations in src/...
   - Fixed failing Behave scenario in features/...
   - Fixed E2E test failure in robot/e2e/...
   - Coverage restored to 97.X%
   
   ISSUES CLOSED: #2597
   

5. Push the fix branch and open a PR to master.

6. Wait for CI to pass on the fix branch before merging.

7. After merge, verify the CI pipeline on the resulting master commit passes all 11 jobs.

---

Prohibited Actions

The following changes are strictly prohibited when fixing CI failures. They suppress or bypass quality enforcement rather than fixing the underlying code:

Prohibited Change	Why
Adding # type: ignore or Pyright suppression directives	Hides type errors instead of fixing them
Adding # noqa or Ruff suppression directives	Hides lint violations instead of fixing them
Adding @skip, @xfail, @unittest.skip to tests	Hides test failures instead of fixing the code
Modifying pyrightconfig.json to relax strictness	Weakens the type checking gate
Modifying pyproject.toml [tool.ruff] to disable rules	Weakens the lint gate
Modifying Bandit config to suppress findings	Weakens the security gate
Modifying .semgrep.yml to exclude patterns	Weakens the security gate
Modifying vulture_whitelist.py to suppress legitimate findings	Weakens the dead-code gate
Reducing the coverage threshold below 97%	Weakens the coverage gate
Modifying .forgejo/workflows/ci.yml to skip or make optional any required job	Weakens the CI gate
Deleting or removing test files, scenarios, or assertions	Hides test failures
Adding success_codes workarounds to nox sessions	Hides failures
Pushing directly to master without a PR	Bypasses CI entirely

---

Prevention: No Direct Pushes to Master

The root cause of issue #2597 was direct pushes to master that bypassed CI. The no-commit-to-branch pre-commit hook prevents this locally, but it can be bypassed with --no-verify.

Never push directly to master. All changes must go through a PR with CI passing. This is enforced by branch protection rules in Forgejo (see CI/CD Pipeline and Branch Protection).

If you find yourself needing to push directly to master for an emergency fix, stop and ask: can this wait for a PR? In almost all cases, the answer is yes. A broken master caused by a direct push is far more damaging than the delay of a PR review.

---

Related Documentation

• CI/CD Pipeline and Branch Protection — Full CI pipeline reference and branch protection rules
• Quality Automation Guide — Pre-commit hooks, nox sessions, and quality gate details
• Testing Guide — Unit tests (Behave), integration tests (Robot Framework), and coverage requirements

Related Issues

• #2597 — fix(ci): restore all CI quality gates to passing on master
• #2463 — Earlier automated report of CI failures on master (superseded by #2597)