cleveragents-core/docs/development/quality-automation.md

Quality Automation Guide

This document describes the automated quality gates, CI pipeline, and developer tooling for the CleverAgents project.

Quick Start

# Set up development environment (installs dependencies + pre-commit hooks)
bash scripts/setup-dev.sh

# Run all quality checks
nox

# Run specific checks
nox -s lint              # Ruff linting
nox -s format            # Ruff formatting
nox -s typecheck         # Pyright type checking
nox -s unit_tests        # Behave unit tests
nox -s coverage_report   # Coverage (must be >=97%)
nox -s security_scan     # Bandit + Semgrep + Vulture security scanning
nox -s dead_code         # Vulture dead code detection
nox -s complexity        # Radon complexity analysis
nox -s pre_commit        # Run all pre-commit hooks

Pre-commit Hooks

Pre-commit hooks run automatically on every git commit. They are configured in .pre-commit-config.yaml.

Installed Hooks

Hook	Tool	Purpose	Auto-fix
no-commit-to-branch	pre-commit-hooks	Prevents commits to main	No
check-yaml	pre-commit-hooks	Validates YAML syntax	No
check-toml	pre-commit-hooks	Validates TOML syntax	No
check-json	pre-commit-hooks	Validates JSON syntax	No
check-merge-conflict	pre-commit-hooks	Detects merge conflict markers	No
check-added-large-files	pre-commit-hooks	Prevents files >500KB	No
end-of-file-fixer	pre-commit-hooks	Ensures files end with newline	Yes
trailing-whitespace	pre-commit-hooks	Removes trailing whitespace	Yes
debug-statements	pre-commit-hooks	Detects pdb/breakpoint	No
ruff-format	ruff	Code formatting	Yes
ruff	ruff	Linting with safe auto-fixes	Yes
bandit	bandit	Security scanning	No
vulture	vulture	Dead code detection	No
pyright	pyright	Type checking (src/ only)	No
semgrep-eval-exec	semgrep	eval/exec detection (optional)	No
commitizen	commitizen	Commit message format (commit-msg stage)	No

Running Hooks Manually

# Run all hooks on all files
pre-commit run --all-files

# Run a specific hook
pre-commit run ruff-format --all-files
pre-commit run bandit --all-files
pre-commit run pyright --all-files

# Skip hooks for emergency (not recommended)
git commit --no-verify -m "emergency: fix critical bug"

CI Pipeline

The CI pipeline runs on Forgejo Actions (.forgejo/workflows/ci.yml).

CI Jobs

Job	Trigger	Purpose	Failure Impact
lint	Push/PR	Ruff format + lint check	Blocks merge
typecheck	Push/PR	Pyright type checking	Blocks merge
security	Push/PR	Bandit + Semgrep + Vulture	Blocks merge
quality	Push/PR	Radon complexity check	Blocks merge (grade F)
unit_tests	Push/PR	Behave BDD tests (Python 3.13)	Blocks merge
coverage	Push/PR	Coverage measurement	Blocks merge (<97%)
build	Push/PR	Wheel build	Blocks release
docker	Push/PR	Docker image build + test	Blocks deployment
helm	Push/PR	Helm chart lint + template	Blocks deployment

Nightly Quality

A nightly workflow (.forgejo/workflows/nightly-quality.yml) runs on all branches at midnight UTC:

• Full lint, type check, security scan
• Complete Behave test suite with coverage
• Complexity and maintainability analysis
• Quality trend data generation
• Reports uploaded as artifacts (90-day retention)

Security Scanning

Bandit Configuration

Configured in pyproject.toml under [tool.bandit]:

• Targets: src/cleveragents only
• Severity threshold: MEDIUM (pre-commit), HIGH (CI fail gate)
• Excludes: tests, features, build directories

Semgrep Rules

Custom rules in .semgrep.yml:

• no-eval: Blocks eval() usage
• no-exec: Blocks exec() usage
• no-compile-exec: Blocks compile() with exec mode
• no-os-system: Warns on os.system() (use subprocess.run)
• no-pickle-loads: Warns on pickle.loads() (use JSON)

Note: Semgrep is a project dependency (included in [project.optional-dependencies.dev]). It runs automatically in the nox -s security_scan session and as a pre-commit hook.

Dead Code Detection

Vulture checks src/cleveragents/ for unused code with 80% confidence threshold.

False Positive Whitelist

Known false positives are listed in vulture_whitelist.py:

• exc_tb: Required by __exit__ protocol but not used in body
• build_data: Required by mapping interface in legacy migrator

Add new entries when vulture flags intentionally unused code.

Complexity Monitoring

Radon measures cyclomatic complexity:

Grade	Complexity	Status
A	1-5	Excellent
B	6-10	Good
C	11-15	Moderate - review recommended
D	16-20	Complex - refactoring recommended
E	21-30	Very complex - must refactor
F	31+	Extremely complex - CI fails

Quality Gate Script

scripts/check-quality-gates.py aggregates all quality checks:

python scripts/check-quality-gates.py --coverage-min 97 --complexity-max F

Gates checked:

1. Coverage >= 97%
2. Pyright: 0 type errors
3. Bandit: 0 high-severity security issues
4. Vulture: 0 dead code findings
5. Radon: No grade-F complexity blocks

Troubleshooting

Pre-commit hooks not running

pre-commit install
pre-commit install --hook-type commit-msg

Pyright errors on clean code

Check pyrightconfig.json for configuration. Pyright uses strict mode.

Bandit false positive

If bandit flags intentionally safe code, consider if there's a safer alternative first. Only as a last resort, add to [tool.bandit] skips in pyproject.toml.

Vulture false positive

Add the symbol name to vulture_whitelist.py with a comment explaining why.

Coverage below 97%

Run nox -s coverage_report to see which lines are uncovered, then add Behave scenarios. Coverage must be >=97% for CI to pass. The threshold is defined by COVERAGE_THRESHOLD in noxfile.py.