cleveragents-core/docs/development/ci-cd.md

# CI/CD Pipeline and Branch Protection

This document describes the CI/CD pipeline, branch protection rules, and merge
requirements for the CleverAgents project. For quality tooling details (pre-commit
hooks, nox sessions, security scanning), see
[Quality Automation Guide](quality-automation.md).

## Branch Protection Rules

### Protected Branch: `master`

The `master` branch is the primary integration branch. All changes must go through
pull requests with the following protections enforced.

#### Required Status Checks

Every pull request targeting `master` must pass **all** of the following CI jobs
before merging is allowed. These are enforced by the `status-check` consolidation
gate — if any single job fails, the entire gate fails and the PR cannot merge.

| # | CI Job | Nox Session(s) | What It Checks | Failure Means |
|---|--------|----------------|----------------|---------------|
| 1 | `lint` | `nox -s lint` + `nox -s format -- --check` | Ruff lint rules and code formatting | Code style violations or lint errors |
| 2 | `typecheck` | `nox -s typecheck` | Pyright strict type checking | Type errors in `src/` |
| 3 | `security` | `nox -s security_scan` + `nox -s dead_code` | Bandit HIGH gate, Semgrep custom rules, Vulture dead-code | Security vulnerabilities or dead code |
| 4 | `quality` | `nox -s complexity` | Radon cyclomatic complexity | Extremely complex methods (31+ cyclomatic complexity) |
| 5 | `unit_tests` | `nox -s unit_tests` | All Behave BDD scenarios under `features/` | Failing unit test scenarios |
| 6 | `integration_tests` | `nox -s integration_tests` | Robot Framework tests under `robot/` (excl. slow/E2E) | Failing integration tests |
| 7 | `e2e_tests` | `nox -s e2e_tests` | End-to-end Robot tests under `robot/e2e/` with real LLM keys | Failing end-to-end tests |
| 8 | `coverage` | `nox -s coverage_report` | Slipcover test coverage ≥ 97% | Coverage dropped below 97% |
| 9 | `build` | `nox -s build` | Python wheel build | Build failure |
| 10 | `docker` | Docker CLI | Docker image build + smoke test (`--version`) | Image build or startup failure |
| 11 | `helm` | Helm CLI + kubeconform | Helm chart lint, template render, and Kubernetes manifest validation | Chart or manifest invalid |

**Job dependencies** (some jobs only run after others pass):

| CI Job | Depends On |
|--------|-----------|
| `coverage` | `lint`, `typecheck`, `security`, `quality` |
| `docker` | `lint`, `typecheck`, `security`, `quality`, `unit_tests` |

#### Required Reviews

- **Minimum 1 approving review** required before merge.
- Reviews are selective (see [Review Priority Matrix](#review-priority-matrix) below).
- Stale reviews are dismissed when new commits are pushed.
- Code owners can be configured per directory if needed.

#### Additional Protections

- **No direct pushes** to `master`. All changes must come via pull request.
  The `no-commit-to-branch` pre-commit hook enforces this locally. Direct
  pushes bypass CI entirely and are the primary cause of master CI breakage
  (see [CI Incident Runbook](ci-incident-runbook.md)).
- **No force pushes** to `master`. History must be preserved.
- **No deletions** of the `master` branch.
- **Require branches to be up-to-date** before merging (prevents merge skew).

### Setting Up Branch Protection in Forgejo

To configure these rules in Forgejo:

1. Navigate to **Repository Settings** > **Branches** > **Branch Protection**.
2. Add a rule for branch name pattern: `master`.
3. Enable the following settings:

```
[x] Enable branch protection
[x] Block push (no direct pushes)
[x] Block force push
[x] Block branch deletion
[x] Require pull request reviews before merging
    - Required approvals: 1
    - Dismiss stale reviews: Yes
[x] Require status checks to pass before merging
    - Required checks:
      - lint
      - typecheck
      - security
      - quality
      - behave
      - coverage
      - build
[x] Require branches to be up-to-date before merging
```

4. Save the branch protection rule.

## Review Process

### Review Priority Matrix

Not all code changes require the same level of review attention. Use this matrix
to determine review depth:

| Priority | Category | Review Depth | Examples |
|----------|----------|-------------|----------|
| **P0** | Architecture and Security | Deep review required | Service layer design, DB schema, API contracts, sandbox boundaries, authentication |
| **P1** | Complex Business Logic | Careful review | Decision tree traversal, merge conflict resolution, dependency closure, state machines |
| **P2** | Normal Features | Standard review | CLI commands, model definitions, straightforward service methods |
| **P3** | Tests, Docs, Refactoring | Trust automation | Behave scenarios, documentation updates, import reorganization, formatting |

### What Reviewers Should Focus On

**Always review** (P0 items):

- Service layer design choices and dependency injection patterns
- Database schema decisions and Alembic migrations
- API contract definitions and interface boundaries
- Sandbox security boundaries and isolation guarantees
- Input validation and sanitization logic
- Any code touching `eval()`, `exec()`, `compile()`, or `subprocess`

**Review carefully** (P1 items):

- Algorithm implementations (decision tree, merge, closure computation)
- State machine transitions and lifecycle management
- Error propagation across actor and plan boundaries
- Concurrent access patterns and thread safety

**Standard review** (P2 items):

- Verify tests exist for new functionality
- Check that argument validation follows `CONTRIBUTING.md` guidelines
- Confirm type annotations are complete (no bare `Any`)

**Trust automation** (P3 items):

- If `nox` passes (lint, typecheck, tests, coverage, security), formatting and
  style are covered.
- Behave test scenarios and Robot integration tests follow established patterns.
- Documentation changes are verified by `nox -s docs` (MkDocs build).

### Review Checklist

Every PR should pass this checklist (also available in the
PR template at `.forgejo/pull_request_template.md` in the repository root):

- [ ] Code follows `CONTRIBUTING.md` coding standards
- [ ] All public/protected methods have argument validation
- [ ] Static typing is complete (no bare `Any` unless justified)
- [ ] `nox -s typecheck` passes
- [ ] `nox -s lint` passes
- [ ] Unit tests written/updated (Behave scenarios in `features/`)
- [ ] Integration tests written/updated (Robot suites in `robot/`) if applicable
- [ ] Coverage remains above 97%
- [ ] No security issues introduced
- [ ] No dead code introduced
- [ ] Documentation updated if behavior changed

## CI Pipeline Reference

### Workflow Files

| Workflow | File | Trigger | Purpose |
|----------|------|---------|---------|
| CI | `.forgejo/workflows/ci.yml` | Push to `master`/`develop`, PRs to `master` | Full nox-based validation pipeline |
| Nightly Quality | `.forgejo/workflows/nightly-quality.yml` | Midnight UTC (cron), manual | Comprehensive quality monitoring |

### CI Job Dependency Graph

```
lint ──────────────────┐
typecheck ─────────────┤
                       ├── coverage (needs lint + typecheck + security + quality)
security ──────────────┤
quality ───────────────┘
                       └── docker (needs lint + typecheck + security + quality + unit_tests)
unit_tests ─────────────── (independent; also feeds docker)
integration_tests ──────── (independent)
e2e_tests ──────────────── (independent; requires LLM API keys)
build ──────────────────── (independent)
helm ───────────────────── (independent)
                       ┌── status-check (needs ALL 11 jobs above)
```

The `status-check` consolidation job is the single gate that branch protection
checks. It fails if **any** of the 11 dependent jobs fail, skipped, or are
cancelled. A broken `master` blocks all open PRs. See the
[CI Incident Runbook](ci-incident-runbook.md) for diagnosis and recovery
procedures.

### Nox-Based CI

All CI jobs now run their checks through nox sessions rather than invoking
tools directly. This ensures that the CI environment matches local development
exactly. Each job runs `pip install uv nox` and then delegates to the
appropriate nox session.

### Quality Gates Summary

All gates must pass for a PR to be mergeable. **No gate may be suppressed,
bypassed, or weakened** — fixes must address the actual code, not the
enforcement configuration. See [CI Incident Runbook](ci-incident-runbook.md)
for the complete list of prohibited suppression techniques.

| Gate | Threshold | Enforced By |
|------|-----------|-------------|
| Formatting | Zero violations | `lint` job (`nox -s format -- --check`) |
| Linting | Zero violations | `lint` job (`nox -s lint`) |
| Type Safety | Zero errors | `typecheck` job (pyright strict) |
| Security | Zero HIGH severity | `security` job (bandit) |
| Dead Code | Zero findings ≥80% confidence | `security` job (vulture) |
| Custom Rules | Zero violations | `security` job (semgrep) |
| Complexity | No grade-F methods (31+) | `quality` job (radon) |
| Unit Tests | All Behave scenarios pass | `unit_tests` job |
| Integration Tests | All Robot suites pass | `integration_tests` job |
| E2E Tests | All E2E Robot suites pass | `e2e_tests` job |
| Coverage | ≥ 97% | `coverage` job (`nox -s coverage_report`) |
| Build | Wheel builds | `build` job |
| Docker | Images build and smoke-test | `docker` job |
| Helm | Chart lints, renders, and validates | `helm` job |

### Nightly Quality Monitoring

The nightly workflow provides trend data beyond PR-level checks:

- Full lint, type check, and security scan (all severities, not just HIGH)
- Complete Behave test suite with coverage measurement
- Complexity and maintainability index analysis
- Quality trend JSON with timestamps (90-day artifact retention)

Reports are uploaded as artifacts under `nightly-quality-reports` and can be
downloaded from the Forgejo Actions UI.

## Local Development Workflow

### Before Pushing a PR

```bash
# Quick validation (runs default nox sessions: lint, format, typecheck,
# unit_tests, integration_tests, docs, build, benchmark, coverage_report)
nox

# Or run individual checks matching CI jobs
nox -s lint              # Formatting and linting (CI: lint job)
nox -s format -- --check # Format check only (CI: lint job)
nox -s typecheck         # Type checking (CI: typecheck job)
nox -s unit_tests        # Behave tests (CI: unit_tests job)
nox -s integration_tests # Robot tests (CI: integration_tests job)
nox -s coverage_report   # Coverage (must be >=97%) (CI: coverage job)
nox -s security_scan     # Bandit security scanning (CI: security job)
nox -s dead_code         # Vulture dead code detection (CI: security job)
nox -s complexity        # Radon complexity check (CI: quality job)
nox -s build             # Wheel build (CI: build job)
```

### Local Reproduction of CI Failures

To reproduce a CI failure locally, run the exact nox session that failed:

```bash
# Example: coverage job failed
nox -s coverage_report

# Example: typecheck job failed
nox -s typecheck

# Example: security job failed
nox -s security_scan
nox -s dead_code
```

### Caching Notes

CI jobs install `uv` (for fast dependency resolution) and `nox`. The nox
sessions use `uv` as the venv backend (`venv_backend="uv"`) and reuse
existing virtualenvs (`reuse_venv=True`) for speed. On CI, each job starts
fresh but `uv` provides fast installs via its built-in caching.

### CI Secrets

Some CI jobs require secrets configured in the Forgejo UI. Secrets are managed
under **Repository Settings > Actions > Secrets** and are automatically masked
in job logs.

| Secret Name | Required By | Purpose |
|-------------|-------------|---------|
| `ANTHROPIC_API_KEY` | `integration_tests` | Anthropic API key for Robot Framework integration tests |
| `OPENAI_API_KEY` | `integration_tests` | OpenAI API key for Robot Framework integration tests |
| `AWS_ACCESS_KEY_ID` | `benchmark-regression`, `benchmark-publish` | AWS credentials for ASV benchmark S3 storage |
| `AWS_SECRET_ACCESS_KEY` | `benchmark-regression`, `benchmark-publish` | AWS credentials for ASV benchmark S3 storage |
| `AWS_DEFAULT_REGION` | `benchmark-regression`, `benchmark-publish` | AWS region for ASV benchmark S3 storage |
| `ASV_S3_BUCKET` | `benchmark-regression`, `benchmark-publish` | S3 bucket name for ASV benchmark storage |

**Unit tests vs. integration tests:**

- **Unit tests** (Behave BDD, `nox -s unit_tests`) do **not** require any API
  keys. They test internal logic without calling external services.
- **Integration tests** (Robot Framework, `nox -s integration_tests`) **do**
  require real LLM API keys (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`). These
  tests exercise end-to-end workflows that interact with live LLM providers.

If the LLM secrets are not configured, integration tests will fail with
authentication errors at runtime.

### Pre-commit Hooks

Pre-commit hooks run automatically on `git commit` and catch most issues before
they reach CI. See [Quality Automation Guide](quality-automation.md) for the
full list of hooks.

If hooks are not installed:

```bash
bash scripts/setup-dev.sh
# Or manually:
pre-commit install
pre-commit install --hook-type commit-msg
```