cleveragents-core/docs/development/ci-cd.md

CI/CD Pipeline and Branch Protection

This document describes the CI/CD pipeline, branch protection rules, and merge requirements for the CleverAgents project. For quality tooling details (pre-commit hooks, nox sessions, security scanning), see Quality Automation Guide.

Branch Protection Rules

Protected Branch: master

The master branch is the primary integration branch. All changes must go through pull requests with the following protections enforced.

Required Status Checks

Every pull request targeting master must pass all of the following CI jobs before merging is allowed:

CI Job	What It Checks	Failure Means
lint	Ruff format and lint (nox)	Code style violations or lint errors
typecheck	Pyright strict type checking (nox)	Type errors in src/
security	Security scan + dead code (nox)	Security vulnerabilities or dead code
quality	Radon complexity (nox)	Extremely complex methods (31+ cyclomatic complexity)
unit_tests	BDD unit tests via nox -s unit_tests	Failing unit test scenarios
integration_tests	Robot integration tests via nox -s integration_tests	Failing integration tests
coverage	Test coverage measurement (nox)	Coverage dropped below 97%
build	Wheel build	Build failure

Deployment-gating jobs (run after core checks pass):

CI Job	Depends On	What It Checks
docker	lint, typecheck, unit_tests, security	Docker image builds and runs

Required Reviews

• Minimum 1 approving review required before merge.
• Reviews are selective (see Review Priority Matrix below).
• Stale reviews are dismissed when new commits are pushed.
• Code owners can be configured per directory if needed.

Additional Protections

• No direct pushes to master. All changes must come via pull request. The no-commit-to-branch pre-commit hook enforces this locally.
• No force pushes to master. History must be preserved.
• No deletions of the master branch.
• Require branches to be up-to-date before merging (prevents merge skew).

Setting Up Branch Protection in Forgejo

To configure these rules in Forgejo:

1. Navigate to Repository Settings > Branches > Branch Protection.
2. Add a rule for branch name pattern: master.
3. Enable the following settings:

[x] Enable branch protection
[x] Block push (no direct pushes)
[x] Block force push
[x] Block branch deletion
[x] Require pull request reviews before merging
    - Required approvals: 1
    - Dismiss stale reviews: Yes
[x] Require status checks to pass before merging
    - Required checks:
      - lint
      - typecheck
      - security
      - quality
      - behave
      - coverage
      - build
[x] Require branches to be up-to-date before merging

4. Save the branch protection rule.

Review Process

Review Priority Matrix

Not all code changes require the same level of review attention. Use this matrix to determine review depth:

Priority	Category	Review Depth	Examples
P0	Architecture and Security	Deep review required	Service layer design, DB schema, API contracts, sandbox boundaries, authentication
P1	Complex Business Logic	Careful review	Decision tree traversal, merge conflict resolution, dependency closure, state machines
P2	Normal Features	Standard review	CLI commands, model definitions, straightforward service methods
P3	Tests, Docs, Refactoring	Trust automation	Behave scenarios, documentation updates, import reorganization, formatting

What Reviewers Should Focus On

Always review (P0 items):

• Service layer design choices and dependency injection patterns
• Database schema decisions and Alembic migrations
• API contract definitions and interface boundaries
• Sandbox security boundaries and isolation guarantees
• Input validation and sanitization logic
• Any code touching eval(), exec(), compile(), or subprocess

Review carefully (P1 items):

• Algorithm implementations (decision tree, merge, closure computation)
• State machine transitions and lifecycle management
• Error propagation across actor and plan boundaries
• Concurrent access patterns and thread safety

Standard review (P2 items):

• Verify tests exist for new functionality
• Check that argument validation follows CONTRIBUTING.md guidelines
• Confirm type annotations are complete (no bare Any)

Trust automation (P3 items):

• If nox passes (lint, typecheck, tests, coverage, security), formatting and style are covered.
• Behave test scenarios and Robot integration tests follow established patterns.
• Documentation changes are verified by nox -s docs (MkDocs build).

Review Checklist

Every PR should pass this checklist (also available in the PR template at .forgejo/pull_request_template.md in the repository root):

• Code follows CONTRIBUTING.md coding standards
• All public/protected methods have argument validation
• Static typing is complete (no bare Any unless justified)
• nox -s typecheck passes
• nox -s lint passes
• Unit tests written/updated (Behave scenarios in features/)
• Integration tests written/updated (Robot suites in robot/) if applicable
• Coverage remains above 97%
• No security issues introduced
• No dead code introduced
• Documentation updated if behavior changed

CI Pipeline Reference

Workflow Files

Workflow	File	Trigger	Purpose
CI	.forgejo/workflows/ci.yml	Push to master/develop, PRs to master	Full nox-based validation pipeline
Nightly Quality	.forgejo/workflows/nightly-quality.yml	Midnight UTC (cron), manual	Comprehensive quality monitoring

CI Job Dependency Graph

lint ──────────────────┐
typecheck ─────────────┤
                       ├── coverage (needs lint + typecheck)
security ──────────────┤
                       ├── docker (needs lint + typecheck + unit_tests + security)
unit_tests ────────────┘
integration_tests ──────── (independent)
quality ────────────────── (independent)
build ──────────────────── (independent)

Nox-Based CI

All CI jobs now run their checks through nox sessions rather than invoking tools directly. This ensures that the CI environment matches local development exactly. Each job runs pip install uv nox and then delegates to the appropriate nox session.

CI Artifact Capture

All 8 nox-running CI jobs capture their full stdout+stderr output as named Forgejo artifacts (PR #2782). This enables automated agents and developers to inspect CI failure output without cloning the repository or re-running jobs locally.

Artifact naming convention: <job-name>-output (e.g., lint-output, typecheck-output, unit_tests-output).

Retention: Artifacts are retained for 30 days by default.

Agent integration: Agent definition files reference these artifacts so that automated agents (e.g., ca-test-infra-improver) can read CI failure output via the Forgejo API and propose targeted fixes without needing repository access.

To download a CI artifact manually:

# Via Forgejo API
curl -H "Authorization: token <PAT>" \
  "https://git.cleverthis.com/api/v1/repos/cleveragents/cleveragents-core/actions/artifacts" \
  | jq '.[] | select(.name == "typecheck-output")'

Quality Gates Summary

All gates must pass for a PR to be mergeable:

Gate	Threshold	Enforced By
Formatting	Zero violations	lint job (ruff format)
Linting	Zero violations	lint job (ruff check)
Type Safety	Zero errors	typecheck job (pyright strict)
Security	Zero HIGH severity	security job (bandit)
Dead Code	Zero findings	security job (vulture)
Complexity	No grade-F methods	quality job (radon)
Unit Tests	All pass (3.11-3.13)	behave job
Coverage	>= 97%	coverage job (nox)
Build	Wheel builds	build job

Nightly Quality Monitoring

The nightly workflow provides trend data beyond PR-level checks:

• Full lint, type check, and security scan (all severities, not just HIGH)
• Complete Behave test suite with coverage measurement
• Complexity and maintainability index analysis
• Quality trend JSON with timestamps (90-day artifact retention)

Reports are uploaded as artifacts under nightly-quality-reports and can be downloaded from the Forgejo Actions UI.

Local Development Workflow

Before Pushing a PR

# Quick validation (runs default nox sessions: lint, format, typecheck,
# unit_tests, integration_tests, docs, build, benchmark, coverage_report)
nox

# Or run individual checks matching CI jobs
nox -s lint              # Formatting and linting (CI: lint job)
nox -s format -- --check # Format check only (CI: lint job)
nox -s typecheck         # Type checking (CI: typecheck job)
nox -s unit_tests        # Behave tests (CI: unit_tests job)
nox -s integration_tests # Robot tests (CI: integration_tests job)
nox -s coverage_report   # Coverage (must be >=97%) (CI: coverage job)
nox -s security_scan     # Bandit security scanning (CI: security job)
nox -s dead_code         # Vulture dead code detection (CI: security job)
nox -s complexity        # Radon complexity check (CI: quality job)
nox -s build             # Wheel build (CI: build job)

Local Reproduction of CI Failures

To reproduce a CI failure locally, run the exact nox session that failed:

# Example: coverage job failed
nox -s coverage_report

# Example: typecheck job failed
nox -s typecheck

# Example: security job failed
nox -s security_scan
nox -s dead_code

Caching Notes

CI jobs install uv (for fast dependency resolution) and nox. The nox sessions use uv as the venv backend (venv_backend="uv") and reuse existing virtualenvs (reuse_venv=True) for speed. On CI, each job starts fresh but uv provides fast installs via its built-in caching.

CI Secrets

Some CI jobs require secrets configured in the Forgejo UI. Secrets are managed under Repository Settings > Actions > Secrets and are automatically masked in job logs.

Secret Name	Required By	Purpose
ANTHROPIC_API_KEY	integration_tests	Anthropic API key for Robot Framework integration tests
OPENAI_API_KEY	integration_tests	OpenAI API key for Robot Framework integration tests
AWS_ACCESS_KEY_ID	benchmark-regression, benchmark-publish	AWS credentials for ASV benchmark S3 storage
AWS_SECRET_ACCESS_KEY	benchmark-regression, benchmark-publish	AWS credentials for ASV benchmark S3 storage
AWS_DEFAULT_REGION	benchmark-regression, benchmark-publish	AWS region for ASV benchmark S3 storage
ASV_S3_BUCKET	benchmark-regression, benchmark-publish	S3 bucket name for ASV benchmark storage

Unit tests vs. integration tests:

• Unit tests (Behave BDD, nox -s unit_tests) do not require any API keys. They test internal logic without calling external services.
• Integration tests (Robot Framework, nox -s integration_tests) do require real LLM API keys (ANTHROPIC_API_KEY, OPENAI_API_KEY). These tests exercise end-to-end workflows that interact with live LLM providers.

If the LLM secrets are not configured, integration tests will fail with authentication errors at runtime.

Pre-commit Hooks

Pre-commit hooks run automatically on git commit and catch most issues before they reach CI. See Quality Automation Guide for the full list of hooks.

If hooks are not installed:

bash scripts/setup-dev.sh
# Or manually:
pre-commit install
pre-commit install --hook-type commit-msg