cleveragents-core/docs/development/testing.md

Testing Guide

Overview

CleverAgents uses a comprehensive testing strategy with three complementary frameworks:

• Behave (BDD/Gherkin) for unit-level and scenario tests under features/
• Robot Framework for integration and end-to-end tests under robot/
• ASV (airspeed velocity) for performance benchmarks under benchmarks/ and asv/benchmarks/

All tests are executed exclusively through nox sessions. Never invoke behave, robot, or other test runners directly.

Running Tests

# Run all default sessions (lint, typecheck, unit tests, integration tests, coverage)
nox

# Unit tests only (Behave)
nox -s unit_tests

# Integration tests only (Robot Framework)
nox -s integration_tests

# Coverage report with 97% threshold enforcement
nox -s coverage_report

# Run a specific feature file
nox -s unit_tests -- features/plan_model.feature

# Run benchmarks
nox -s benchmark

Coverage Requirements

Threshold: 97%

The project enforces a minimum 97% code coverage at all times. This is a hard gate — both CI and local nox runs will fail if coverage drops below this threshold.

The coverage check is performed by nox -s coverage_report, which:

1. Runs all Behave tests under coverage run
2. Generates HTML (build/htmlcov/), XML (build/coverage.xml), and terminal reports
3. Emits a single-line verdict: COVERAGE PASS: XX% (threshold 97%) or COVERAGE FAIL: XX% (threshold 97%)
4. Exits with a non-zero code if coverage is below 97%

Configuration

Coverage is configured in pyproject.toml:

[tool.coverage.run]
source = ["src", "scripts"]
branch = true
data_file = "build/.coverage"

[tool.coverage.html]
directory = "build/htmlcov"

[tool.coverage.xml]
output = "build/coverage.xml"

The --fail-under=97 threshold is enforced in the coverage_report nox session (noxfile.py).

Sample Failure Output

When coverage drops below 97%, the output looks like:

Name                                              Stmts   Miss Branch BrMiss  Cover   Missing
----------------------------------------------------------------------------------------------
src/cleveragents/domain/models/core/plan.py         150      8     40      5    94%   45-52, 89
src/cleveragents/application/services/foo.py         80      5     20      3    93%   12, 34-38
...
----------------------------------------------------------------------------------------------
TOTAL                                              9860    400   2852    280    95%

nox > COVERAGE FAIL: 95% (threshold 97%)
nox > error: Coverage 95% is below the required 97% threshold.

How to Improve Coverage

1. Run nox -s coverage_report to generate the HTML report
2. Open build/htmlcov/index.html in a browser to identify uncovered files
3. Sort by "Missing" column to find files with the most uncovered lines
4. Write Behave scenarios targeting the uncovered code paths
5. Re-run nox -s coverage_report to verify improvement

All test files go under features/ (Behave) or robot/ (Robot Framework). Never create a tests/ directory.

Test Organization

Behave Tests (features/)

• Feature files: features/<name>.feature
• Step definitions: features/steps/<name>_steps.py
• Mock implementations: features/mocks/
• Environment setup: features/environment.py

Naming conventions:

• Feature-specific steps go in <feature_name>_steps.py
• Shared steps go in clearly named reusable modules
• All step files must be complete — no placeholder steps

Robot Framework Tests (robot/)

• Test suites: robot/<name>.robot
• Resource files: robot/<name>.resource
• Common resources: robot/common.resource

Benchmarks

• ASV benchmarks: asv/benchmarks/<name>.py
• Legacy benchmarks: benchmarks/<name>.py

CI Integration

The coverage CI job runs nox -s coverage_report and:

• Fails the pipeline if coverage is below 97%
• Uploads build/coverage.xml and build/htmlcov/ as artifacts (retained 30 days)
• Surfaces the coverage verdict in the job summary

See CI/CD documentation for the full CI pipeline description.

Writing Effective Tests

Behave Test Guidelines

Feature: Plan lifecycle transitions

  Scenario: Execute plan transitions phase from strategize to execute
    Given an action "local/build-app" exists
    And a plan is created using "local/build-app" on project "local/my-project"
    And strategize is completed
    When I execute the plan
    Then the plan phase should be "execute"
    And the processing state should be "queued"

Robot Framework Guidelines

*** Settings ***
Documentation     Plan lifecycle integration tests
Library           Process
Library           OperatingSystem
Resource          ${CURDIR}/common.resource

*** Test Cases ***
Plan Execute Transitions Correctly
    [Documentation]    Verify execute phase transition via CLI
    ${result}=    Run Process    ${PYTHON}    -m    cleveragents    plan    execute    ${PLAN_ID}
    Should Be Equal As Integers    ${result.rc}    0
    Should Contain    ${result.stdout}    execute

Troubleshooting

Coverage report shows 0% or very low coverage

This usually means behave tests are running in a subprocess that coverage cannot instrument. Ensure:

• nox -s coverage_report is used (not direct behave invocation)
• parallel = false is set in [tool.coverage.run]
• COVERAGE_FILE and COVERAGE_RCFILE env vars are set correctly

Robot tests fail with "resource not found"

Use ${CURDIR}/ prefix for all Resource imports in robot files:

Resource    ${CURDIR}/common.resource

Tests hang indefinitely

Add timeout parameters to Run Process calls in Robot tests:

${result}=    Run Process    ${PYTHON}    -m    cleveragents    ...    timeout=30s