cleveragents-core/docs/development/ci-cd.md

CI/CD Pipeline and Branch Protection

This document describes the CI/CD pipeline, branch protection rules, and merge requirements for the CleverAgents project. For quality tooling details (pre-commit hooks, nox sessions, security scanning), see Quality Automation Guide.

Branch Protection Rules

Protected Branch: master

The master branch is the primary integration branch. All changes must go through pull requests with the following protections enforced.

Required Status Checks

Every pull request targeting master must pass all of the following CI jobs before merging is allowed:

CI Job	What It Checks	Failure Means
lint	Ruff format and lint (nox)	Code style violations or lint errors
typecheck	Pyright strict type checking (nox)	Type errors in src/
security	Security scan + dead code (nox)	Security vulnerabilities or dead code
quality	Radon complexity (nox)	Extremely complex methods (31+ cyclomatic complexity)
unit_tests	BDD unit tests via nox -s unit_tests	Failing unit test scenarios
integration_tests	Robot integration tests via nox -s integration_tests	Failing integration tests
coverage	Test coverage measurement (nox)	Coverage dropped below 97%
build	Wheel build	Build failure

Deployment-gating jobs (run after core checks pass):

CI Job	Depends On	What It Checks
docker	lint, typecheck, unit_tests, security	Docker image builds and runs

Required Reviews

• Minimum 1 approving review required before merge.
• Reviews are selective (see Review Priority Matrix below).
• Stale reviews are dismissed when new commits are pushed.
• Code owners can be configured per directory if needed.

Additional Protections

• No direct pushes to master. All changes must come via pull request. The no-commit-to-branch pre-commit hook enforces this locally.
• No force pushes to master. History must be preserved.
• No deletions of the master branch.
• Require branches to be up-to-date before merging (prevents merge skew).

Setting Up Branch Protection in Forgejo

To configure these rules in Forgejo:

1. Navigate to Repository Settings > Branches > Branch Protection.
2. Add a rule for branch name pattern: master.
3. Enable the following settings:

[x] Enable branch protection
[x] Block push (no direct pushes)
[x] Block force push
[x] Block branch deletion
[x] Require pull request reviews before merging
    - Required approvals: 1
    - Dismiss stale reviews: Yes
[x] Require status checks to pass before merging
    - Required checks:
      - lint
      - typecheck
      - security
      - quality
      - behave
      - coverage
      - build
[x] Require branches to be up-to-date before merging

4. Save the branch protection rule.

Review Process

Review Priority Matrix

Not all code changes require the same level of review attention. Use this matrix to determine review depth:

Priority	Category	Review Depth	Examples
P0	Architecture and Security	Deep review required	Service layer design, DB schema, API contracts, sandbox boundaries, authentication
P1	Complex Business Logic	Careful review	Decision tree traversal, merge conflict resolution, dependency closure, state machines
P2	Normal Features	Standard review	CLI commands, model definitions, straightforward service methods
P3	Tests, Docs, Refactoring	Trust automation	Behave scenarios, documentation updates, import reorganization, formatting

What Reviewers Should Focus On

Always review (P0 items):

• Service layer design choices and dependency injection patterns
• Database schema decisions and Alembic migrations
• API contract definitions and interface boundaries
• Sandbox security boundaries and isolation guarantees
• Input validation and sanitization logic
• Any code touching eval(), exec(), compile(), or subprocess

Review carefully (P1 items):

• Algorithm implementations (decision tree, merge, closure computation)
• State machine transitions and lifecycle management
• Error propagation across actor and plan boundaries
• Concurrent access patterns and thread safety

Standard review (P2 items):

• Verify tests exist for new functionality
• Check that argument validation follows CONTRIBUTING.md guidelines
• Confirm type annotations are complete (no bare Any)

Trust automation (P3 items):

• If nox passes (lint, typecheck, tests, coverage, security), formatting and style are covered.
• Behave test scenarios and Robot integration tests follow established patterns.
• Documentation changes are verified by nox -s docs (MkDocs build).

Review Checklist

Every PR should pass this checklist (also available in the PR template at .forgejo/pull_request_template.md in the repository root):

• Code follows CONTRIBUTING.md coding standards
• All public/protected methods have argument validation
• Static typing is complete (no bare Any unless justified)
• nox -s typecheck passes
• nox -s lint passes
• Unit tests written/updated (Behave scenarios in features/)
• Integration tests written/updated (Robot suites in robot/) if applicable
• Coverage remains above 97%
• No security issues introduced
• No dead code introduced
• Documentation updated if behavior changed

CI Pipeline Reference

Workflow Files

Workflow	File	Trigger	Purpose
CI	.forgejo/workflows/ci.yml	Push to master/develop, PRs to master	Full nox-based validation pipeline
Nightly Quality	.forgejo/workflows/nightly-quality.yml	Midnight UTC (cron), manual	Comprehensive quality monitoring

CI Job Dependency Graph

lint ──────────────────┐
typecheck ─────────────┤
                       ├── coverage (needs lint + typecheck)
security ──────────────┤
                       ├── docker (needs lint + typecheck + unit_tests + security)
unit_tests ────────────┘
integration_tests ──────── (independent)
quality ────────────────── (independent)
build ──────────────────── (independent)
push-validation ────────── (independent — validates CI push credentials)

Nox-Based CI

All CI jobs now run their checks through nox sessions rather than invoking tools directly. This ensures that the CI environment matches local development exactly. Each job runs pip install uv nox and then delegates to the appropriate nox session.

Quality Gates Summary

All gates must pass for a PR to be mergeable:

Gate	Threshold	Enforced By
Formatting	Zero violations	lint job (ruff format)
Linting	Zero violations	lint job (ruff check)
Type Safety	Zero errors	typecheck job (pyright strict)
Security	Zero HIGH severity	security job (bandit)
Dead Code	Zero findings	security job (vulture)
Complexity	No grade-F methods	quality job (radon)
Unit Tests	All pass (3.11-3.13)	behave job
Coverage	>= 97%	coverage job (nox)
Build	Wheel builds	build job
Push Access	Credentials valid	push-validation job

Nightly Quality Monitoring

The nightly workflow provides trend data beyond PR-level checks:

• Full lint, type check, and security scan (all severities, not just HIGH)
• Complete Behave test suite with coverage measurement
• Complexity and maintainability index analysis
• Quality trend JSON with timestamps (90-day artifact retention)

Reports are uploaded as artifacts under nightly-quality-reports and can be downloaded from the Forgejo Actions UI.

Local Development Workflow

Before Pushing a PR

# Quick validation (runs default nox sessions: lint, format, typecheck,
# unit_tests, integration_tests, docs, build, benchmark, coverage_report)
nox

# Or run individual checks matching CI jobs
nox -s lint              # Formatting and linting (CI: lint job)
nox -s format -- --check # Format check only (CI: lint job)
nox -s typecheck         # Type checking (CI: typecheck job)
nox -s unit_tests        # Behave tests (CI: unit_tests job)
nox -s integration_tests # Robot tests (CI: integration_tests job)
nox -s coverage_report   # Coverage (must be >=97%) (CI: coverage job)
nox -s security_scan     # Bandit security scanning (CI: security job)
nox -s dead_code         # Vulture dead code detection (CI: security job)
nox -s complexity        # Radon complexity check (CI: quality job)
nox -s build             # Wheel build (CI: build job)

Local Reproduction of CI Failures

To reproduce a CI failure locally, run the exact nox session that failed:

# Example: coverage job failed
nox -s coverage_report

# Example: typecheck job failed
nox -s typecheck

# Example: security job failed
nox -s security_scan
nox -s dead_code

Caching Notes

CI jobs install uv (for fast dependency resolution) and nox. The nox sessions use uv as the venv backend (venv_backend="uv") and reuse existing virtualenvs (reuse_venv=True) for speed. On CI, each job starts fresh but uv provides fast installs via its built-in caching.

CI Secrets

Some CI jobs require secrets configured in the Forgejo UI. Secrets are managed under Repository Settings > Actions > Secrets and are automatically masked in job logs.

Secret Name	Required By	Purpose
ANTHROPIC_API_KEY	integration_tests	Anthropic API key for Robot Framework integration tests
OPENAI_API_KEY	integration_tests	OpenAI API key for Robot Framework integration tests
AWS_ACCESS_KEY_ID	benchmark-regression, benchmark-publish	AWS credentials for ASV benchmark S3 storage
AWS_SECRET_ACCESS_KEY	benchmark-regression, benchmark-publish	AWS credentials for ASV benchmark S3 storage
AWS_DEFAULT_REGION	benchmark-regression, benchmark-publish	AWS region for ASV benchmark S3 storage
ASV_S3_BUCKET	benchmark-regression, benchmark-publish	S3 bucket name for ASV benchmark storage
FORGEJO_TOKEN	release (create-release job)	Forgejo API token with repository write scope — used to authenticate git push and create releases
FORGEJO_URL	release (create-release job)	Base URL of the Forgejo instance (e.g., https://git.cleverthis.com)
CONTAINER_REGISTRY	release (build-docker job)	Container registry URL for Docker image pushes
CONTAINER_REGISTRY_USER	release (build-docker job)	Username for container registry authentication
CONTAINER_REGISTRY_PASSWORD	release (build-docker job)	Password/token for container registry authentication

Unit tests vs. integration tests:

• Unit tests (Behave BDD, nox -s unit_tests) do not require any API keys. They test internal logic without calling external services.
• Integration tests (Robot Framework, nox -s integration_tests) do require real LLM API keys (ANTHROPIC_API_KEY, OPENAI_API_KEY). These tests exercise end-to-end workflows that interact with live LLM providers.

If the LLM secrets are not configured, integration tests will fail with authentication errors at runtime.

Repository Push Authentication

Any CI workflow step that writes back to the repository (e.g., pushing tags, committing auto-generated files, or creating changelog commits) must use explicit token authentication. The push-validation job in ci.yml validates that push credentials are correctly configured on every CI run.

Fix applied (issue #1541): The actions/checkout@v4 action was not configured with token: ${{ forgejo.token }} and persist-credentials: true, and no git user config (user.name / user.email) was set. Both are required for push operations. The push-validation job now validates these on every run.

Root Cause of "Unable to Push" Failures

The most common cause of CI push failures is missing or misconfigured credentials. Symptoms include:

remote: Permission denied
fatal: unable to access 'https://...': The requested URL returned error: 403

or:

ERROR: Repository not found.
fatal: Could not read from remote repository.

Fix: HTTPS Token Authentication

The canonical fix is to configure git to use the FORGEJO_TOKEN secret via HTTPS credential store. This is done in the create-release job of release.yml:

- uses: actions/checkout@v4
  with:
    fetch-depth: 0
    token: ${{ secrets.FORGEJO_TOKEN }}  # Use write-scoped token

- name: Configure git identity for push operations
  env:
    FORGEJO_URL: ${{ secrets.FORGEJO_URL }}
    FORGEJO_TOKEN: ${{ secrets.FORGEJO_TOKEN }}
  run: |
    git config user.name "CleverAgents CI"
    git config user.email "ci@cleverthis.com"
    FORGEJO_HOST=$(echo "${FORGEJO_URL}" | sed 's|https\?://||' | cut -d/ -f1)
    git config credential.helper store
    echo "https://ci:${FORGEJO_TOKEN}@${FORGEJO_HOST}" > ~/.git-credentials
    chmod 600 ~/.git-credentials

Smoke-Test Step

The create-release job includes a smoke-test step that validates push access before attempting the real push. This catches credential issues early with a clear error message:

- name: Smoke-test push access
  env:
    FORGEJO_URL: ${{ secrets.FORGEJO_URL }}
    FORGEJO_TOKEN: ${{ secrets.FORGEJO_TOKEN }}
  run: |
    REPO="${{ forgejo.repository }}"
    API_URL="${FORGEJO_URL}/api/v1/repos/${REPO}"
    HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
      -H "Authorization: token ${FORGEJO_TOKEN}" \
      "${API_URL}")
    if [ "${HTTP_STATUS}" != "200" ]; then
      echo "ERROR: FORGEJO_TOKEN cannot access repository API (HTTP ${HTTP_STATUS})."
      exit 1
    fi
    PUSH_ALLOWED=$(curl -s \
      -H "Authorization: token ${FORGEJO_TOKEN}" \
      "${API_URL}" | python3 -c "import sys,json; d=json.load(sys.stdin); print(str(d.get('permissions',{}).get('push',False)).lower())")
    if [ "${PUSH_ALLOWED}" != "true" ]; then
      echo "ERROR: FORGEJO_TOKEN does not have push (write) permission."
      exit 1
    fi
    echo "Push access verified."

Setting Up the FORGEJO_TOKEN Secret

1. Create a Forgejo personal access token (PAT) with repository write scope:

   • Navigate to User Settings > Applications > Access Tokens
   • Create a token with repository scope (read + write)
   • Copy the token value (shown only once)

2. Add the token as a repository secret:

   • Navigate to Repository Settings > Actions > Secrets
   • Add secret FORGEJO_TOKEN with the PAT value
   • Add secret FORGEJO_URL with the Forgejo base URL (e.g., https://git.cleverthis.com)

3. Never hardcode tokens in workflow files. Always use ${{ secrets.SECRET_NAME }}.

Security Notes

• The FORGEJO_TOKEN secret is automatically masked in job logs by Forgejo Actions.
• The ~/.git-credentials file is created with chmod 600 (owner-read-only).
• The credential file is ephemeral — it exists only for the duration of the job in the container's filesystem and is destroyed when the container exits.
• No SSH deploy keys are required; HTTPS token authentication is sufficient and simpler to manage.

Pre-commit Hooks

Pre-commit hooks run automatically on git commit and catch most issues before they reach CI. See Quality Automation Guide for the full list of hooks.

If hooks are not installed:

bash scripts/setup-dev.sh
# Or manually:
pre-commit install
pre-commit install --hook-type commit-msg

Benchmark Workflow

Why Benchmarks Are Separate from PR Validation

The benchmark-regression job was removed from the default PR workflow because:

• Long runtime: ASV benchmark runs take 99-132 minutes, blocking PR feedback even when lint/typecheck/tests pass in under 20 minutes.
• Runner contention: Long-running ASV tasks occupy docker runners, exacerbating queue times for other PRs.
• Low signal for most changes: The vast majority of code changes do not affect performance; running benchmarks on every PR is wasteful.

The benchmark workflow now runs on a schedule and can be triggered manually for performance-sensitive changes.

Benchmark Workflow Triggers

The benchmark workflow (.forgejo/workflows/benchmark-scheduled.yml) runs:

Trigger	Schedule	Job	Purpose
Nightly cron	2 AM UTC daily	benchmark-regression	Compare HEAD against master for regressions
Weekly cron	3 AM UTC Sundays	benchmark-full	Full benchmark suite for trend analysis
Manual dispatch	On demand	Either	Ad-hoc validation for performance-sensitive PRs

How to Trigger Benchmarks Manually

To run benchmarks for a specific PR or branch:

1. Navigate to Actions > Benchmark Regression in the Forgejo UI.
2. Click Run workflow.
3. Select the branch to benchmark.
4. Set base_sha to the SHA or branch to compare against (default: master).
5. Set run_full_suite to true for the full suite, or false (default) for regression-only comparison.

Alternatively, use the Forgejo API:

curl -X POST \
  -H "Authorization: token ${FORGEJO_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"ref": "your-branch", "inputs": {"base_sha": "master", "run_full_suite": "false"}}' \
  "${FORGEJO_URL}/api/v1/repos/cleveragents/cleveragents-core/actions/workflows/benchmark-scheduled.yml/dispatches"

AWS Credentials Required for S3 Benchmark Storage

Benchmark results are stored in S3 for historical trend analysis. The following secrets must be configured in Repository Settings > Actions > Secrets:

Secret	Purpose
AWS_ACCESS_KEY_ID	AWS credentials for S3 benchmark storage
AWS_SECRET_ACCESS_KEY	AWS credentials for S3 benchmark storage
AWS_DEFAULT_REGION	AWS region for S3 benchmark storage
ASV_S3_BUCKET	S3 bucket name for ASV benchmark storage

If these secrets are not configured, benchmark runs will still execute locally but results will not be published to S3 (a warning is emitted instead of failing).

Benchmark Results

Benchmark results are available as:

• Forgejo Actions artifacts: Uploaded as benchmark-regression-results or benchmark-full-results with 90-day retention.
• S3 storage: Published to s3://${ASV_S3_BUCKET}/asv/ when AWS credentials are configured. The HTML report is at s3://${ASV_S3_BUCKET}/asv/html/.