186 lines
7.1 KiB
Markdown
186 lines
7.1 KiB
Markdown
# Quality Automation Guide
|
|
|
|
This document describes the automated quality gates, CI pipeline, and developer tooling
|
|
for the CleverAgents project.
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# Set up development environment (installs dependencies + pre-commit hooks)
|
|
bash scripts/setup-dev.sh
|
|
|
|
# Run all quality checks
|
|
nox
|
|
|
|
# Run specific checks
|
|
nox -s lint # Ruff linting
|
|
nox -s format # Ruff formatting
|
|
nox -s typecheck # Pyright type checking
|
|
nox -s unit_tests # Behave unit tests
|
|
nox -s coverage_report # Coverage (must be >=97%)
|
|
nox -s security_scan # Bandit + Semgrep + Vulture security scanning
|
|
nox -s dead_code # Vulture dead code detection
|
|
nox -s complexity # Radon complexity analysis
|
|
nox -s pre_commit # Run all pre-commit hooks
|
|
```
|
|
|
|
## Pre-commit Hooks
|
|
|
|
Pre-commit hooks run automatically on every `git commit`. They are configured in
|
|
`.pre-commit-config.yaml`.
|
|
|
|
### Installed Hooks
|
|
|
|
| Hook | Tool | Purpose | Auto-fix |
|
|
| ------------------------- | ---------------- | ---------------------------------------- | -------- |
|
|
| `no-commit-to-branch` | pre-commit-hooks | Prevents commits to `main` | No |
|
|
| `check-yaml` | pre-commit-hooks | Validates YAML syntax | No |
|
|
| `check-toml` | pre-commit-hooks | Validates TOML syntax | No |
|
|
| `check-json` | pre-commit-hooks | Validates JSON syntax | No |
|
|
| `check-merge-conflict` | pre-commit-hooks | Detects merge conflict markers | No |
|
|
| `check-added-large-files` | pre-commit-hooks | Prevents files >500KB | No |
|
|
| `end-of-file-fixer` | pre-commit-hooks | Ensures files end with newline | Yes |
|
|
| `trailing-whitespace` | pre-commit-hooks | Removes trailing whitespace | Yes |
|
|
| `debug-statements` | pre-commit-hooks | Detects `pdb`/`breakpoint` | No |
|
|
| `ruff-format` | ruff | Code formatting | Yes |
|
|
| `ruff` | ruff | Linting with safe auto-fixes | Yes |
|
|
| `bandit` | bandit | Security scanning | No |
|
|
| `vulture` | vulture | Dead code detection | No |
|
|
| `pyright` | pyright | Type checking (src/ only) | No |
|
|
| `semgrep-eval-exec` | semgrep | eval/exec detection (optional) | No |
|
|
| `commitizen` | commitizen | Commit message format (commit-msg stage) | No |
|
|
|
|
### Running Hooks Manually
|
|
|
|
```bash
|
|
# Run all hooks on all files
|
|
pre-commit run --all-files
|
|
|
|
# Run a specific hook
|
|
pre-commit run ruff-format --all-files
|
|
pre-commit run bandit --all-files
|
|
pre-commit run pyright --all-files
|
|
|
|
# Skip hooks for emergency (not recommended)
|
|
git commit --no-verify -m "emergency: fix critical bug"
|
|
```
|
|
|
|
## CI Pipeline
|
|
|
|
The CI pipeline runs on **Forgejo Actions** (`.forgejo/workflows/ci.yml`).
|
|
|
|
### CI Jobs
|
|
|
|
| Job | Trigger | Purpose | Failure Impact |
|
|
| ----------- | ------- | ----------------------------------- | ---------------------- |
|
|
| `lint` | Push/PR | Ruff format + lint check | Blocks merge |
|
|
| `typecheck` | Push/PR | Pyright type checking | Blocks merge |
|
|
| `security` | Push/PR | Bandit + Semgrep + Vulture | Blocks merge |
|
|
| `quality` | Push/PR | Radon complexity check | Blocks merge (grade F) |
|
|
| `unit_tests`| Push/PR | Behave BDD tests (Python 3.13) | Blocks merge |
|
|
| `coverage` | Push/PR | Coverage measurement | Blocks merge (<97%) |
|
|
| `build` | Push/PR | Wheel build | Blocks release |
|
|
| `docker` | Push/PR | Docker image build + test | Blocks deployment |
|
|
| `helm` | Push/PR | Helm chart lint + template | Blocks deployment |
|
|
|
|
### Nightly Quality
|
|
|
|
A nightly workflow (`.forgejo/workflows/nightly-quality.yml`) runs on all branches at midnight UTC:
|
|
|
|
- Full lint, type check, security scan
|
|
- Complete Behave test suite with coverage
|
|
- Complexity and maintainability analysis
|
|
- Quality trend data generation
|
|
- Reports uploaded as artifacts (90-day retention)
|
|
|
|
## Security Scanning
|
|
|
|
### Bandit Configuration
|
|
|
|
Configured in `pyproject.toml` under `[tool.bandit]`:
|
|
|
|
- Targets: `src/cleveragents` only
|
|
- Severity threshold: MEDIUM (pre-commit), HIGH (CI fail gate)
|
|
- Excludes: tests, features, build directories
|
|
|
|
### Semgrep Rules
|
|
|
|
Custom rules in `.semgrep.yml`:
|
|
|
|
- `no-eval`: Blocks `eval()` usage
|
|
- `no-exec`: Blocks `exec()` usage
|
|
- `no-compile-exec`: Blocks `compile()` with exec mode
|
|
- `no-os-system`: Warns on `os.system()` (use `subprocess.run`)
|
|
- `no-pickle-loads`: Warns on `pickle.loads()` (use JSON)
|
|
|
|
**Note:** Semgrep is a project dependency (included in `[project.optional-dependencies.dev]`). It runs automatically in the `nox -s security_scan` session and as a pre-commit hook.
|
|
|
|
## Dead Code Detection
|
|
|
|
Vulture checks `src/cleveragents/` for unused code with 80% confidence threshold.
|
|
|
|
### False Positive Whitelist
|
|
|
|
Known false positives are listed in `vulture_whitelist.py`:
|
|
|
|
- `exc_tb`: Required by `__exit__` protocol but not used in body
|
|
- `build_data`: Required by mapping interface in legacy migrator
|
|
|
|
Add new entries when vulture flags intentionally unused code.
|
|
|
|
## Complexity Monitoring
|
|
|
|
Radon measures cyclomatic complexity:
|
|
|
|
| Grade | Complexity | Status |
|
|
| ----- | ---------- | --------------------------------- |
|
|
| A | 1-5 | Excellent |
|
|
| B | 6-10 | Good |
|
|
| C | 11-15 | Moderate - review recommended |
|
|
| D | 16-20 | Complex - refactoring recommended |
|
|
| E | 21-30 | Very complex - must refactor |
|
|
| F | 31+ | Extremely complex - CI fails |
|
|
|
|
## Quality Gate Script
|
|
|
|
`scripts/check-quality-gates.py` aggregates all quality checks:
|
|
|
|
```bash
|
|
python scripts/check-quality-gates.py --coverage-min 97 --complexity-max F
|
|
```
|
|
|
|
Gates checked:
|
|
|
|
1. Coverage >= 97%
|
|
2. Pyright: 0 type errors
|
|
3. Bandit: 0 high-severity security issues
|
|
4. Vulture: 0 dead code findings
|
|
5. Radon: No grade-F complexity blocks
|
|
|
|
## Troubleshooting
|
|
|
|
### Pre-commit hooks not running
|
|
|
|
```bash
|
|
pre-commit install
|
|
pre-commit install --hook-type commit-msg
|
|
```
|
|
|
|
### Pyright errors on clean code
|
|
|
|
Check `pyrightconfig.json` for configuration. Pyright uses `strict` mode.
|
|
|
|
### Bandit false positive
|
|
|
|
If bandit flags intentionally safe code, consider if there's a safer alternative first.
|
|
Only as a last resort, add to `[tool.bandit]` `skips` in `pyproject.toml`.
|
|
|
|
### Vulture false positive
|
|
|
|
Add the symbol name to `vulture_whitelist.py` with a comment explaining why.
|
|
|
|
### Coverage below 97%
|
|
|
|
Run `nox -s coverage_report` to see which lines are uncovered, then add Behave scenarios.
|
|
Coverage must be >=97% for CI to pass. The threshold is defined by `COVERAGE_THRESHOLD` in `noxfile.py`.
|