cleveragents-core/docs/development/ci-cd.md

CI/CD Pipeline and Branch Protection

This document describes the CI/CD pipeline, branch protection rules, and merge requirements for the CleverAgents project. For quality tooling details (pre-commit hooks, nox sessions, security scanning), see Quality Automation Guide.

Branch Protection Rules

Protected Branch: master

The master branch is the primary integration branch. All changes must go through pull requests with the following protections enforced.

Required Status Checks

Every pull request targeting master must pass all of the following CI jobs before merging is allowed. These are enforced by the status-check consolidation gate — if any single job fails, the entire gate fails and the PR cannot merge.

#	CI Job	Nox Session(s)	What It Checks	Failure Means
1	lint	nox -s lint + nox -s format -- --check	Ruff lint rules and code formatting	Code style violations or lint errors
2	typecheck	nox -s typecheck	Pyright strict type checking	Type errors in src/
3	security	nox -s security_scan + nox -s dead_code	Bandit HIGH gate, Semgrep custom rules, Vulture dead-code	Security vulnerabilities or dead code
4	quality	nox -s complexity	Radon cyclomatic complexity	Extremely complex methods (31+ cyclomatic complexity)
5	unit_tests	nox -s unit_tests	All Behave BDD scenarios under features/	Failing unit test scenarios
6	integration_tests	nox -s integration_tests	Robot Framework tests under robot/ (excl. slow/E2E)	Failing integration tests
7	e2e_tests	nox -s e2e_tests	End-to-end Robot tests under robot/e2e/ with real LLM keys	Failing end-to-end tests
8	coverage	nox -s coverage_report	Slipcover test coverage ≥ 97%	Coverage dropped below 97%
9	build	nox -s build	Python wheel build	Build failure
10	docker	Docker CLI	Docker image build + smoke test (--version)	Image build or startup failure
11	helm	Helm CLI + kubeconform	Helm chart lint, template render, and Kubernetes manifest validation	Chart or manifest invalid

Job dependencies (some jobs only run after others pass):

CI Job	Depends On
coverage	lint, typecheck, security, quality
docker	lint, typecheck, security, quality, unit_tests

Required Reviews

• Minimum 1 approving review required before merge.
• Reviews are selective (see Review Priority Matrix below).
• Stale reviews are dismissed when new commits are pushed.
• Code owners can be configured per directory if needed.

Additional Protections

• No direct pushes to master. All changes must come via pull request. The no-commit-to-branch pre-commit hook enforces this locally. Direct pushes bypass CI entirely and are the primary cause of master CI breakage (see CI Incident Runbook).
• No force pushes to master. History must be preserved.
• No deletions of the master branch.
• Require branches to be up-to-date before merging (prevents merge skew).

Setting Up Branch Protection in Forgejo

To configure these rules in Forgejo:

1. Navigate to Repository Settings > Branches > Branch Protection.
2. Add a rule for branch name pattern: master.
3. Enable the following settings:

[x] Enable branch protection
[x] Block push (no direct pushes)
[x] Block force push
[x] Block branch deletion
[x] Require pull request reviews before merging
    - Required approvals: 1
    - Dismiss stale reviews: Yes
[x] Require status checks to pass before merging
    - Required checks:
      - lint
      - typecheck
      - security
      - quality
      - behave
      - coverage
      - build
[x] Require branches to be up-to-date before merging

4. Save the branch protection rule.

Review Process

Review Priority Matrix

Not all code changes require the same level of review attention. Use this matrix to determine review depth:

Priority	Category	Review Depth	Examples
P0	Architecture and Security	Deep review required	Service layer design, DB schema, API contracts, sandbox boundaries, authentication
P1	Complex Business Logic	Careful review	Decision tree traversal, merge conflict resolution, dependency closure, state machines
P2	Normal Features	Standard review	CLI commands, model definitions, straightforward service methods
P3	Tests, Docs, Refactoring	Trust automation	Behave scenarios, documentation updates, import reorganization, formatting

What Reviewers Should Focus On

Always review (P0 items):

• Service layer design choices and dependency injection patterns
• Database schema decisions and Alembic migrations
• API contract definitions and interface boundaries
• Sandbox security boundaries and isolation guarantees
• Input validation and sanitization logic
• Any code touching eval(), exec(), compile(), or subprocess

Review carefully (P1 items):

• Algorithm implementations (decision tree, merge, closure computation)
• State machine transitions and lifecycle management
• Error propagation across actor and plan boundaries
• Concurrent access patterns and thread safety

Standard review (P2 items):

• Verify tests exist for new functionality
• Check that argument validation follows CONTRIBUTING.md guidelines
• Confirm type annotations are complete (no bare Any)

Trust automation (P3 items):

• If nox passes (lint, typecheck, tests, coverage, security), formatting and style are covered.
• Behave test scenarios and Robot integration tests follow established patterns.
• Documentation changes are verified by nox -s docs (MkDocs build).

Review Checklist

Every PR should pass this checklist (also available in the PR template at .forgejo/pull_request_template.md in the repository root):

• Code follows CONTRIBUTING.md coding standards
• All public/protected methods have argument validation
• Static typing is complete (no bare Any unless justified)
• nox -s typecheck passes
• nox -s lint passes
• Unit tests written/updated (Behave scenarios in features/)
• Integration tests written/updated (Robot suites in robot/) if applicable
• Coverage remains above 97%
• No security issues introduced
• No dead code introduced
• Documentation updated if behavior changed

CI Pipeline Reference

Workflow Files

Workflow	File	Trigger	Purpose
CI	.forgejo/workflows/ci.yml	Push to master/develop, PRs to master	Full nox-based validation pipeline
Nightly Quality	.forgejo/workflows/nightly-quality.yml	Midnight UTC (cron), manual	Comprehensive quality monitoring

CI Job Dependency Graph

lint ──────────────────┐
typecheck ─────────────┤
                       ├── coverage (needs lint + typecheck + security + quality)
security ──────────────┤
quality ───────────────┘
                       └── docker (needs lint + typecheck + security + quality + unit_tests)
unit_tests ─────────────── (independent; also feeds docker)
integration_tests ──────── (independent)
e2e_tests ──────────────── (independent; requires LLM API keys)
build ──────────────────── (independent)
helm ───────────────────── (independent)
                       ┌── status-check (needs ALL 11 jobs above)

The status-check consolidation job is the single gate that branch protection checks. It fails if any of the 11 dependent jobs fail, skipped, or are cancelled. A broken master blocks all open PRs. See the CI Incident Runbook for diagnosis and recovery procedures.

Nox-Based CI

All CI jobs now run their checks through nox sessions rather than invoking tools directly. This ensures that the CI environment matches local development exactly. Each job runs pip install uv nox and then delegates to the appropriate nox session.

Quality Gates Summary

All gates must pass for a PR to be mergeable. No gate may be suppressed, bypassed, or weakened — fixes must address the actual code, not the enforcement configuration. See CI Incident Runbook for the complete list of prohibited suppression techniques.

Gate	Threshold	Enforced By
Formatting	Zero violations	lint job (nox -s format -- --check)
Linting	Zero violations	lint job (nox -s lint)
Type Safety	Zero errors	typecheck job (pyright strict)
Security	Zero HIGH severity	security job (bandit)
Dead Code	Zero findings ≥80% confidence	security job (vulture)
Custom Rules	Zero violations	security job (semgrep)
Complexity	No grade-F methods (31+)	quality job (radon)
Unit Tests	All Behave scenarios pass	unit_tests job
Integration Tests	All Robot suites pass	integration_tests job
E2E Tests	All E2E Robot suites pass	e2e_tests job
Coverage	≥ 97%	coverage job (nox -s coverage_report)
Build	Wheel builds	build job
Docker	Images build and smoke-test	docker job
Helm	Chart lints, renders, and validates	helm job

Nightly Quality Monitoring

The nightly workflow provides trend data beyond PR-level checks:

• Full lint, type check, and security scan (all severities, not just HIGH)
• Complete Behave test suite with coverage measurement
• Complexity and maintainability index analysis
• Quality trend JSON with timestamps (90-day artifact retention)

Reports are uploaded as artifacts under nightly-quality-reports and can be downloaded from the Forgejo Actions UI.

Local Development Workflow

Before Pushing a PR

# Quick validation (runs default nox sessions: lint, format, typecheck,
# unit_tests, integration_tests, docs, build, benchmark, coverage_report)
nox

# Or run individual checks matching CI jobs
nox -s lint              # Formatting and linting (CI: lint job)
nox -s format -- --check # Format check only (CI: lint job)
nox -s typecheck         # Type checking (CI: typecheck job)
nox -s unit_tests        # Behave tests (CI: unit_tests job)
nox -s integration_tests # Robot tests (CI: integration_tests job)
nox -s coverage_report   # Coverage (must be >=97%) (CI: coverage job)
nox -s security_scan     # Bandit security scanning (CI: security job)
nox -s dead_code         # Vulture dead code detection (CI: security job)
nox -s complexity        # Radon complexity check (CI: quality job)
nox -s build             # Wheel build (CI: build job)

Local Reproduction of CI Failures

To reproduce a CI failure locally, run the exact nox session that failed:

# Example: coverage job failed
nox -s coverage_report

# Example: typecheck job failed
nox -s typecheck

# Example: security job failed
nox -s security_scan
nox -s dead_code

Caching Notes

CI jobs install uv (for fast dependency resolution) and nox. The nox sessions use uv as the venv backend (venv_backend="uv") and reuse existing virtualenvs (reuse_venv=True) for speed. On CI, each job starts fresh but uv provides fast installs via its built-in caching.

CI Secrets

Some CI jobs require secrets configured in the Forgejo UI. Secrets are managed under Repository Settings > Actions > Secrets and are automatically masked in job logs.

Secret Name	Required By	Purpose
ANTHROPIC_API_KEY	integration_tests	Anthropic API key for Robot Framework integration tests
OPENAI_API_KEY	integration_tests	OpenAI API key for Robot Framework integration tests
AWS_ACCESS_KEY_ID	benchmark-regression, benchmark-publish	AWS credentials for ASV benchmark S3 storage
AWS_SECRET_ACCESS_KEY	benchmark-regression, benchmark-publish	AWS credentials for ASV benchmark S3 storage
AWS_DEFAULT_REGION	benchmark-regression, benchmark-publish	AWS region for ASV benchmark S3 storage
ASV_S3_BUCKET	benchmark-regression, benchmark-publish	S3 bucket name for ASV benchmark storage

Unit tests vs. integration tests:

• Unit tests (Behave BDD, nox -s unit_tests) do not require any API keys. They test internal logic without calling external services.
• Integration tests (Robot Framework, nox -s integration_tests) do require real LLM API keys (ANTHROPIC_API_KEY, OPENAI_API_KEY). These tests exercise end-to-end workflows that interact with live LLM providers.

If the LLM secrets are not configured, integration tests will fail with authentication errors at runtime.

Pre-commit Hooks

Pre-commit hooks run automatically on git commit and catch most issues before they reach CI. See Quality Automation Guide for the full list of hooks.

If hooks are not installed:

bash scripts/setup-dev.sh
# Or manually:
pre-commit install
pre-commit install --hook-type commit-msg