cleveragents-core/docs/development/ci-cd.md

CI/CD Pipeline and Branch Protection

This document describes the CI/CD pipeline, branch protection rules, and merge requirements for the CleverAgents project. For quality tooling details (pre-commit hooks, nox sessions, security scanning), see Quality Automation Guide.

Branch Protection Rules

Protected Branch: master

The master branch is the primary integration branch. All changes must go through pull requests with the following protections enforced.

Required Status Checks

Every pull request targeting master must pass all of the following CI jobs before merging is allowed. These are enforced by the status-check consolidation gate — if any single job fails, the entire gate fails and the PR cannot merge.

#	CI Job	Nox Session(s)	What It Checks	Failure Means
1	lint	nox -s lint + nox -s format -- --check	Ruff lint rules and code formatting	Code style violations or lint errors
2	typecheck	nox -s typecheck	Pyright strict type checking	Type errors in src/
3	security	nox -s security_scan + nox -s dead_code	Bandit HIGH gate, Semgrep custom rules, Vulture dead-code	Security vulnerabilities or dead code
4	quality	nox -s complexity	Radon cyclomatic complexity	Extremely complex methods (31+ cyclomatic complexity)
5	unit_tests	nox -s unit_tests	All Behave BDD scenarios under features/	Failing unit test scenarios
6	integration_tests	nox -s integration_tests	Robot Framework tests under robot/ (excl. slow/E2E)	Failing integration tests
7	e2e_tests	nox -s e2e_tests	End-to-end Robot tests under robot/e2e/ with real LLM keys	Failing end-to-end tests
8	coverage	nox -s coverage_report	Slipcover test coverage ≥ 97%	Coverage dropped below 97%
9	build	nox -s build	Python wheel build	Build failure
10	docker	Docker CLI	Docker image build + smoke test (--version)	Image build or startup failure
11	helm	Helm CLI + kubeconform	Helm chart lint, template render, and Kubernetes manifest validation	Chart or manifest invalid

Job dependencies (some jobs only run after others pass):

CI Job	Depends On
coverage	lint, typecheck, security, quality
docker	lint, typecheck, security, quality, unit_tests

Required Reviews

• Minimum 1 approving review required before merge.
• Reviews are selective (see Review Priority Matrix below).
• Stale reviews are dismissed when new commits are pushed.
• Code owners can be configured per directory if needed.

Additional Protections

• No direct pushes to master. All changes must come via pull request. The no-commit-to-branch pre-commit hook enforces this locally. Direct pushes bypass CI entirely and are the primary cause of master CI breakage (see CI Incident Runbook).
• No force pushes to master. History must be preserved.
• No deletions of the master branch.
• Require branches to be up-to-date before merging (prevents merge skew).

Setting Up Branch Protection in Forgejo

To configure these rules in Forgejo:

1. Navigate to Repository Settings > Branches > Branch Protection.
2. Add a rule for branch name pattern: master.
3. Enable the following settings:

[x] Enable branch protection
[x] Block push (no direct pushes)
[x] Block force push
[x] Block branch deletion
[x] Require pull request reviews before merging
    - Required approvals: 1
    - Dismiss stale reviews: Yes
[x] Require status checks to pass before merging
    - Required checks:
      - lint
      - typecheck
      - security
      - quality
      - behave
      - coverage
      - build
[x] Require branches to be up-to-date before merging

4. Save the branch protection rule.

Review Process

Review Priority Matrix

Not all code changes require the same level of review attention. Use this matrix to determine review depth:

Priority	Category	Review Depth	Examples
P0	Architecture and Security	Deep review required	Service layer design, DB schema, API contracts, sandbox boundaries, authentication
P1	Complex Business Logic	Careful review	Decision tree traversal, merge conflict resolution, dependency closure, state machines
P2	Normal Features	Standard review	CLI commands, model definitions, straightforward service methods
P3	Tests, Docs, Refactoring	Trust automation	Behave scenarios, documentation updates, import reorganization, formatting

What Reviewers Should Focus On

Always review (P0 items):

• Service layer design choices and dependency injection patterns
• Database schema decisions and Alembic migrations
• API contract definitions and interface boundaries
• Sandbox security boundaries and isolation guarantees
• Input validation and sanitization logic
• Any code touching eval(), exec(), compile(), or subprocess

Review carefully (P1 items):

• Algorithm implementations (decision tree, merge, closure computation)
• State machine transitions and lifecycle management
• Error propagation across actor and plan boundaries
• Concurrent access patterns and thread safety

Standard review (P2 items):

• Verify tests exist for new functionality
• Check that argument validation follows CONTRIBUTING.md guidelines
• Confirm type annotations are complete (no bare Any)

Trust automation (P3 items):

• If nox passes (lint, typecheck, tests, coverage, security), formatting and style are covered.
• Behave test scenarios and Robot integration tests follow established patterns.
• Documentation changes are verified by nox -s docs (MkDocs build).

Review Checklist

Every PR should pass this checklist (also available in the PR template at .forgejo/pull_request_template.md in the repository root):

• Code follows CONTRIBUTING.md coding standards
• All public/protected methods have argument validation
• Static typing is complete (no bare Any unless justified)
• nox -s typecheck passes
• nox -s lint passes
• Unit tests written/updated (Behave scenarios in features/)
• Integration tests written/updated (Robot suites in robot/) if applicable
• Coverage remains above 97%
• No security issues introduced
• No dead code introduced
• Documentation updated if behavior changed

CI Pipeline Reference

Workflow Files

Workflow	File	Trigger	Purpose
CI	.forgejo/workflows/ci.yml	Push to master/develop, PRs to master	Full nox-based validation pipeline
Nightly Quality	.forgejo/workflows/nightly-quality.yml	Midnight UTC (cron), manual	Comprehensive quality monitoring

CI Job Dependency Graph

lint ──────────────────┐
typecheck ─────────────┤
                       ├── coverage (needs lint + typecheck + security + quality)
security ──────────────┤
quality ───────────────┘
                       └── docker (needs lint + typecheck + security + quality + unit_tests)
unit_tests ─────────────── (independent; also feeds docker)
integration_tests ──────── (independent)
e2e_tests ──────────────── (independent; requires LLM API keys)
build ──────────────────── (independent)
helm ───────────────────── (independent)
                       ┌── status-check (needs ALL 11 jobs above)

The status-check consolidation job is the single gate that branch protection checks. It fails if any of the 11 dependent jobs fail, skipped, or are cancelled. A broken master blocks all open PRs. See the CI Incident Runbook for diagnosis and recovery procedures.

Nox-Based CI

All CI jobs now run their checks through nox sessions rather than invoking tools directly. This ensures that the CI environment matches local development exactly. Each job runs pip install uv nox and then delegates to the appropriate nox session.

CI Artifact Capture

All 8 nox-running CI jobs capture their full stdout+stderr output as named Forgejo artifacts (PR #2782). This enables automated agents and developers to inspect CI failure output without cloning the repository or re-running jobs locally.

Artifact naming convention: <job-name>-output (e.g., lint-output, typecheck-output, unit_tests-output).

Retention: Artifacts are retained for 30 days by default.

Agent integration: Agent definition files reference these artifacts so that automated agents (e.g., ca-test-infra-improver) can read CI failure output via the Forgejo API and propose targeted fixes without needing repository access.

To download a CI artifact manually:

# Via Forgejo API
curl -H "Authorization: token <PAT>" \
  "https://git.cleverthis.com/api/v1/repos/cleveragents/cleveragents-core/actions/artifacts" \
  | jq '.[] | select(.name == "typecheck-output")'

Quality Gates Summary

All gates must pass for a PR to be mergeable. No gate may be suppressed, bypassed, or weakened — fixes must address the actual code, not the enforcement configuration. See CI Incident Runbook for the complete list of prohibited suppression techniques.

Gate	Threshold	Enforced By
Formatting	Zero violations	lint job (nox -s format -- --check)
Linting	Zero violations	lint job (nox -s lint)
Type Safety	Zero errors	typecheck job (pyright strict)
Security	Zero HIGH severity	security job (bandit)
Dead Code	Zero findings ≥80% confidence	security job (vulture)
Custom Rules	Zero violations	security job (semgrep)
Complexity	No grade-F methods (31+)	quality job (radon)
Unit Tests	All Behave scenarios pass	unit_tests job
Integration Tests	All Robot suites pass	integration_tests job
E2E Tests	All E2E Robot suites pass	e2e_tests job
Coverage	≥ 97%	coverage job (nox -s coverage_report)
Build	Wheel builds	build job
Docker	Images build and smoke-test	docker job
Helm	Chart lints, renders, and validates	helm job

Nightly Quality Monitoring

The nightly workflow provides trend data beyond PR-level checks:

• Full lint, type check, and security scan (all severities, not just HIGH)
• Complete Behave test suite with coverage measurement
• Complexity and maintainability index analysis
• Quality trend JSON with timestamps (90-day artifact retention)

Reports are uploaded as artifacts under nightly-quality-reports and can be downloaded from the Forgejo Actions UI.

Local Development Workflow

Before Pushing a PR

# Quick validation (runs default nox sessions: lint, format, typecheck,
# unit_tests, integration_tests, docs, build, benchmark, coverage_report)
nox

# Or run individual checks matching CI jobs
nox -s lint              # Formatting and linting (CI: lint job)
nox -s format -- --check # Format check only (CI: lint job)
nox -s typecheck         # Type checking (CI: typecheck job)
nox -s unit_tests        # Behave tests (CI: unit_tests job)
nox -s integration_tests # Robot tests (CI: integration_tests job)
nox -s coverage_report   # Coverage (must be >=97%) (CI: coverage job)
nox -s security_scan     # Bandit security scanning (CI: security job)
nox -s dead_code         # Vulture dead code detection (CI: security job)
nox -s complexity        # Radon complexity check (CI: quality job)
nox -s build             # Wheel build (CI: build job)

Local Reproduction of CI Failures

To reproduce a CI failure locally, run the exact nox session that failed:

# Example: coverage job failed
nox -s coverage_report

# Example: typecheck job failed
nox -s typecheck

# Example: security job failed
nox -s security_scan
nox -s dead_code

Caching Notes

CI jobs install uv (for fast dependency resolution) and nox. The nox sessions use uv as the venv backend (venv_backend="uv") and reuse existing virtualenvs (reuse_venv=True) for speed. On CI, each job starts fresh but uv provides fast installs via its built-in caching.

CI Secrets

Some CI jobs require secrets configured in the Forgejo UI. Secrets are managed under Repository Settings > Actions > Secrets and are automatically masked in job logs.

Secret Name	Required By	Purpose
ANTHROPIC_API_KEY	integration_tests	Anthropic API key for Robot Framework integration tests
OPENAI_API_KEY	integration_tests	OpenAI API key for Robot Framework integration tests
AWS_ACCESS_KEY_ID	benchmark-regression, benchmark-publish	AWS credentials for ASV benchmark S3 storage
AWS_SECRET_ACCESS_KEY	benchmark-regression, benchmark-publish	AWS credentials for ASV benchmark S3 storage
AWS_DEFAULT_REGION	benchmark-regression, benchmark-publish	AWS region for ASV benchmark S3 storage
ASV_S3_BUCKET	benchmark-regression, benchmark-publish	S3 bucket name for ASV benchmark storage
FORGEJO_TOKEN	release (create-release job)	Forgejo API token with repository write scope — used to authenticate git push and create releases
FORGEJO_URL	release (create-release job)	Base URL of the Forgejo instance (e.g., https://git.cleverthis.com)
CONTAINER_REGISTRY	release (build-docker job)	Container registry URL for Docker image pushes
CONTAINER_REGISTRY_USER	release (build-docker job)	Username for container registry authentication
CONTAINER_REGISTRY_PASSWORD	release (build-docker job)	Password/token for container registry authentication

Unit tests vs. integration tests:

• Unit tests (Behave BDD, nox -s unit_tests) do not require any API keys. They test internal logic without calling external services.
• Integration tests (Robot Framework, nox -s integration_tests) do require real LLM API keys (ANTHROPIC_API_KEY, OPENAI_API_KEY). These tests exercise end-to-end workflows that interact with live LLM providers.

If the LLM secrets are not configured, integration tests will fail with authentication errors at runtime.

Repository Push Authentication

Any CI workflow step that writes back to the repository (e.g., pushing tags, committing auto-generated files, or creating changelog commits) must use explicit token authentication. The push-validation job in ci.yml validates that push credentials are correctly configured on every CI run.

Fix applied (issue #1541): The actions/checkout@v4 action was not configured with token: ${{ forgejo.token }} and persist-credentials: true, and no git user config (user.name / user.email) was set. Both are required for push operations. The push-validation job now validates these on every run.

Root Cause of "Unable to Push" Failures

The most common cause of CI push failures is missing or misconfigured credentials. Symptoms include:

remote: Permission denied
fatal: unable to access 'https://...': The requested URL returned error: 403

or:

ERROR: Repository not found.
fatal: Could not read from remote repository.

Fix: HTTPS Token Authentication

The canonical fix is to configure git to use the FORGEJO_TOKEN secret via HTTPS credential store. This is done in the create-release job of release.yml:

- uses: actions/checkout@v4
  with:
    fetch-depth: 0
    token: ${{ secrets.FORGEJO_TOKEN }}  # Use write-scoped token

- name: Configure git identity for push operations
  env:
    FORGEJO_URL: ${{ secrets.FORGEJO_URL }}
    FORGEJO_TOKEN: ${{ secrets.FORGEJO_TOKEN }}
  run: |
    git config user.name "CleverAgents CI"
    git config user.email "ci@cleverthis.com"
    FORGEJO_HOST=$(echo "${FORGEJO_URL}" | sed 's|https\?://||' | cut -d/ -f1)
    git config credential.helper store
    echo "https://ci:${FORGEJO_TOKEN}@${FORGEJO_HOST}" > ~/.git-credentials
    chmod 600 ~/.git-credentials

Smoke-Test Step

The create-release job includes a smoke-test step that validates push access before attempting the real push. This catches credential issues early with a clear error message:

- name: Smoke-test push access
  env:
    FORGEJO_URL: ${{ secrets.FORGEJO_URL }}
    FORGEJO_TOKEN: ${{ secrets.FORGEJO_TOKEN }}
  run: |
    REPO="${{ forgejo.repository }}"
    API_URL="${FORGEJO_URL}/api/v1/repos/${REPO}"
    HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
      -H "Authorization: token ${FORGEJO_TOKEN}" \
      "${API_URL}")
    if [ "${HTTP_STATUS}" != "200" ]; then
      echo "ERROR: FORGEJO_TOKEN cannot access repository API (HTTP ${HTTP_STATUS})."
      exit 1
    fi
    PUSH_ALLOWED=$(curl -s \
      -H "Authorization: token ${FORGEJO_TOKEN}" \
      "${API_URL}" | python3 -c "import sys,json; d=json.load(sys.stdin); print(str(d.get('permissions',{}).get('push',False)).lower())")
    if [ "${PUSH_ALLOWED}" != "true" ]; then
      echo "ERROR: FORGEJO_TOKEN does not have push (write) permission."
      exit 1
    fi
    echo "Push access verified."

Setting Up the FORGEJO_TOKEN Secret

1. Create a Forgejo personal access token (PAT) with repository write scope:

   • Navigate to User Settings > Applications > Access Tokens
   • Create a token with repository scope (read + write)
   • Copy the token value (shown only once)

2. Add the token as a repository secret:

   • Navigate to Repository Settings > Actions > Secrets
   • Add secret FORGEJO_TOKEN with the PAT value
   • Add secret FORGEJO_URL with the Forgejo base URL (e.g., https://git.cleverthis.com)

3. Never hardcode tokens in workflow files. Always use ${{ secrets.SECRET_NAME }}.

Security Notes

• The FORGEJO_TOKEN secret is automatically masked in job logs by Forgejo Actions.
• The ~/.git-credentials file is created with chmod 600 (owner-read-only).
• The credential file is ephemeral — it exists only for the duration of the job in the container's filesystem and is destroyed when the container exits.
• No SSH deploy keys are required; HTTPS token authentication is sufficient and simpler to manage.

Pre-commit Hooks

Pre-commit hooks run automatically on git commit and catch most issues before they reach CI. See Quality Automation Guide for the full list of hooks.

If hooks are not installed:

bash scripts/setup-dev.sh
# Or manually:
pre-commit install
pre-commit install --hook-type commit-msg

CI Log Artifacts

All 8 nox-running CI jobs capture their stdout/stderr output as named Forgejo artifacts. This makes CI logs immediately downloadable without scraping the UI.

Artifact Name	CI Job	Contents
ci-logs-lint	lint	Ruff format + lint output
ci-logs-typecheck	typecheck	Pyright output
ci-logs-security	security	Bandit + Vulture output (combined)
ci-logs-quality	quality	Radon complexity output
ci-logs-unit-tests	unit_tests	Behave BDD test output
ci-logs-integration-tests	integration_tests	Robot Framework output
ci-logs-e2e-tests	e2e_tests	End-to-end test output
ci-logs-coverage	coverage	Coverage measurement output

Key properties:

• Artifacts are uploaded with if: always() so they are available even when the job fails — which is precisely when they are most needed.
• Multi-session jobs (e.g., lint runs both lint and format) append to a single log file using tee -a, keeping one artifact per CI job.
• Retention: 30 days, consistent with the existing coverage-reports policy.

Downloading Artifacts via API

# List artifacts for a workflow run
curl -H 'Authorization: token <PAT>' \
  'https://git.cleverthis.com/api/v1/repos/cleveragents/cleveragents-core/actions/artifacts'

# Download a specific artifact
curl -H 'Authorization: token <PAT>' \
  'https://git.cleverthis.com/api/v1/repos/cleveragents/cleveragents-core/actions/artifacts/<ID>/zip' \
  -o ci-logs-lint.zip

Agent Usage

Agent definition files (ca-pr-checker.md, ca-lint-fixer.md, etc.) are configured to download the relevant artifact first, then fall back to running nox locally if the artifact is unavailable (e.g., on a first-run PR).