cleveragents-core/docs/development/testing.md

Testing Guide

Overview

CleverAgents uses a layered testing strategy:

• Unit Tests: Behave (Cucumber/Gherkin BDD) feature suites under features/
• Integration Tests: Robot Framework suites under robot/
• Performance Benchmarks: ASV (airspeed velocity) benchmarks under benchmarks/
• Coverage: coverage.py with branch coverage, enforced at >=97%

All tests run exclusively through nox sessions. Do not invoke behave, robot, or coverage directly.

Running Tests

# All default sessions (lint, typecheck, unit, integration, benchmarks)
nox

# Unit tests only (Behave)
nox -s unit_tests

# Integration tests only (Robot Framework)
nox -s integration_tests

# Coverage report with 97% enforcement
nox -s coverage_report

# Benchmarks (ASV)
nox -s benchmark

# Type checking (pyright)
nox -s typecheck

# Linting (ruff)
nox -s lint

# Formatting check
nox -s format -- --check

Coverage Requirement: >=97%

The project enforces a minimum 97% code coverage threshold. This is a hard gate:

• The nox -s coverage_report session fails if coverage drops below 97%.
• The Forgejo CI coverage job runs nox -s coverage_report and blocks merges on failure.
• Branch coverage is enabled (branch = true in pyproject.toml).

How Coverage Is Measured

Coverage is measured by running Behave tests under coverage.py in serial mode:

coverage run --source=src -m behave -q --tags=-discovery --no-capture features/
coverage report --show-missing --fail-under=97

Coverage Output

On success, the nox session emits:

COVERAGE OK: 97.2% (threshold: 97%)

On failure, the nox session emits and exits non-zero:

COVERAGE FAILED: 95.3% < 97% threshold

Both messages are single-line and CI-parseable. The CI pipeline greps for these lines to surface them in job summaries.

Coverage Reports

Three report formats are generated under build/:

Report	Path	Description
Terminal	stdout	Per-file summary with --show-missing
HTML	build/htmlcov/index.html	Interactive browser report
XML	build/coverage.xml	Cobertura-format for CI tools
JSON	build/coverage.json	Machine-readable totals and per-file data

Coverage Configuration

Coverage settings live in pyproject.toml:

[tool.coverage.run]
source = ["src", "scripts"]
branch = true
parallel = false
omit = [
    "*/tests/*",
    "*/test_*",
    "features/*",
    "*/features/*",
    "*/__pycache__/*",
    "*/site-packages/*",
    "*/dependency_injector/*",
    "*/venv/*",
    "*/.venv/*",
    "*/.nox/*",
    "src/cleveragents/discovery/*",
]
data_file = "build/.coverage"

What to Do When Coverage Drops

1. Run nox -s coverage_report locally to see the per-file report with --show-missing.
2. Open build/htmlcov/index.html for an interactive view of uncovered lines.
3. Identify the files with the most uncovered lines.
4. Write new Behave scenarios in features/ that exercise the uncovered code paths.
5. Re-run nox -s coverage_report and verify the threshold passes.

Do not lower the threshold, add # pragma: no cover comments, or expand the omit list without explicit team agreement.

Unit Tests (Behave)

Unit tests use the Behave BDD framework with Gherkin-syntax .feature files.

Structure

features/
  *.feature          # Feature files (Gherkin scenarios)
  steps/             # Step definitions (Python)
  mocks/             # Mock implementations (test doubles)
  fixtures/          # Test fixture data
  environment.py     # Behave hooks (before/after scenario, etc.)

Guidelines

• No pytest: All unit tests must be Behave-based. There is intentionally no tests/ directory.
• Step naming: Use unique, descriptive step names. Behave loads all step files globally; ambiguous steps cause AmbiguousStep errors.
• Feature naming: Name feature-specific step files after their feature (e.g., foo.feature -> foo_steps.py).
• Mocks in features/ only: All mock implementations live in features/mocks/. Production code (src/) must never contain test doubles.

Running Specific Features

# Run a single feature file
nox -s unit_tests -- features/plan_model.feature

# Run scenarios with a specific tag
nox -s unit_tests -- --tags=@smoke

Integration Tests (Robot Framework)

Integration tests use Robot Framework for end-to-end and system-level testing.

Structure

robot/
  *.robot            # Test suites
  *.resource         # Shared keywords and variables
  common.resource    # Common setup/teardown keywords
  v2_paths.resource  # Path configuration

Guidelines

• Resource paths: Always use ${CURDIR}/ prefix for Resource imports (e.g., Resource ${CURDIR}/common.resource).
• Python injection: Use ${PYTHON} variable (injected by nox) instead of bare python in Run Process calls.
• Timeouts: Always add timeout= to Run Process calls that invoke long-running commands.
• Slow tests: Tag tests that require external services (API keys, running servers) with slow. These are excluded in CI via --exclude slow.

Running Specific Suites

# Run a single Robot suite
nox -s integration_tests -- --suite robot/plan_generation_graph.robot

# Include slow tests
nox -s slow_integration_tests

Performance Benchmarks (ASV)

ASV (airspeed velocity) tracks performance over time.

Structure

benchmarks/
  *.py               # Benchmark suites (Time*/Mem*/Track* classes)
asv.conf.json        # ASV configuration

Running Benchmarks

nox -s benchmark

CI Pipeline

The Forgejo CI pipeline runs these jobs in order:

1. lint — nox -s lint (ruff check)
2. typecheck — nox -s typecheck (pyright)
3. security — nox -s security_scan (bandit + vulture)
4. quality — nox -s complexity (radon)
5. unit_tests — nox -s unit_tests (behave)
6. integration_tests — nox -s integration_tests (robot)
7. coverage — nox -s coverage_report (fail-under 97%)
8. build — nox -s build (wheel)

All jobs use Python 3.13 and route commands through nox. See docs/development/ci-cd.md for full CI/CD documentation.

Troubleshooting

Coverage report shows 0% or very low coverage

Ensure you're running in serial mode (not parallel). The coverage_report nox session handles this correctly. Do not run behave directly outside of coverage run.

AmbiguousStep errors in Behave

Two step files define the same @given/@when/@then pattern. Behave loads all step files globally. Make step names unique or use more specific patterns.

Robot Resource file does not exist

Use Resource ${CURDIR}/filename.resource instead of bare Resource filename.resource. Robot resolves bare paths relative to CWD, not the test file directory.

Robot No module named cleveragents

Ensure Robot tests use ${PYTHON} (injected by nox from the venv) instead of bare python in Run Process calls.