cleveragents-core/docs/development/testing.md

# Testing Guide

## Overview

CleverAgents uses a comprehensive testing strategy with three complementary frameworks:

- **Behave** (BDD/Gherkin) for unit-level and scenario tests under `features/`
- **Robot Framework** for integration and end-to-end tests under `robot/`
- **ASV (airspeed velocity)** for performance benchmarks under `benchmarks/` and `asv/benchmarks/`

All tests are executed exclusively through `nox` sessions. Never invoke `behave`, `robot`, or other test runners directly.

## Running Tests

```bash
# Run all default sessions (lint, typecheck, unit tests, integration tests, coverage)
nox

# Unit tests only (Behave)
nox -s unit_tests

# Integration tests only (Robot Framework)
nox -s integration_tests

# Coverage report with 97% threshold enforcement
nox -s coverage_report

# Run a specific feature file
nox -s unit_tests -- features/plan_model.feature

# Run benchmarks
nox -s benchmark
```

## Coverage Requirements

### Threshold: 97%

The project enforces a **minimum 97% code coverage** at all times. This is a hard gate — both CI and local `nox` runs will fail if coverage drops below this threshold.

The coverage check is performed by `nox -s coverage_report`, which:

1. Runs all Behave tests under `coverage run`
2. Generates HTML (`build/htmlcov/`), XML (`build/coverage.xml`), and terminal reports
3. Emits a single-line verdict: `COVERAGE PASS: XX% (threshold 97%)` or `COVERAGE FAIL: XX% (threshold 97%)`
4. Exits with a non-zero code if coverage is below 97%

### Configuration

Coverage is configured in `pyproject.toml`:

```toml
[tool.coverage.run]
source = ["src", "scripts"]
branch = true
data_file = "build/.coverage"

[tool.coverage.html]
directory = "build/htmlcov"

[tool.coverage.xml]
output = "build/coverage.xml"
```

The `--fail-under=97` threshold is enforced in the `coverage_report` nox session (`noxfile.py`).

### Sample Failure Output

When coverage drops below 97%, the output looks like:

```
Name                                              Stmts   Miss Branch BrMiss  Cover   Missing
----------------------------------------------------------------------------------------------
src/cleveragents/domain/models/core/plan.py         150      8     40      5    94%   45-52, 89
src/cleveragents/application/services/foo.py         80      5     20      3    93%   12, 34-38
...
----------------------------------------------------------------------------------------------
TOTAL                                              9860    400   2852    280    95%

nox > COVERAGE FAIL: 95% (threshold 97%)
nox > error: Coverage 95% is below the required 97% threshold.
```

### How to Improve Coverage

1. Run `nox -s coverage_report` to generate the HTML report
2. Open `build/htmlcov/index.html` in a browser to identify uncovered files
3. Sort by "Missing" column to find files with the most uncovered lines
4. Write Behave scenarios targeting the uncovered code paths
5. Re-run `nox -s coverage_report` to verify improvement

All test files go under `features/` (Behave) or `robot/` (Robot Framework). Never create a `tests/` directory.

## Test Organization

### Behave Tests (`features/`)

- Feature files: `features/<name>.feature`
- Step definitions: `features/steps/<name>_steps.py`
- Mock implementations: `features/mocks/`
- Environment setup: `features/environment.py`

**Naming conventions:**
- Feature-specific steps go in `<feature_name>_steps.py`
- Shared steps go in clearly named reusable modules
- All step files must be complete — no placeholder steps

### Robot Framework Tests (`robot/`)

- Test suites: `robot/<name>.robot`
- Resource files: `robot/<name>.resource`
- Common resources: `robot/common.resource`

### Benchmarks

- ASV benchmarks: `asv/benchmarks/<name>.py`
- Legacy benchmarks: `benchmarks/<name>.py`

## CI Integration

The `coverage` CI job runs `nox -s coverage_report` and:

- Fails the pipeline if coverage is below 97%
- Uploads `build/coverage.xml` and `build/htmlcov/` as artifacts (retained 30 days)
- Surfaces the coverage verdict in the job summary

See [CI/CD documentation](ci-cd.md) for the full CI pipeline description.

## Writing Effective Tests

### Behave Test Guidelines

```gherkin
Feature: Plan lifecycle transitions

  Scenario: Execute plan transitions phase from strategize to execute
    Given an action "local/build-app" exists
    And a plan is created using "local/build-app" on project "local/my-project"
    And strategize is completed
    When I execute the plan
    Then the plan phase should be "execute"
    And the processing state should be "queued"
```

### Robot Framework Guidelines

```robot
*** Settings ***
Documentation     Plan lifecycle integration tests
Library           Process
Library           OperatingSystem
Resource          ${CURDIR}/common.resource

*** Test Cases ***
Plan Execute Transitions Correctly
    [Documentation]    Verify execute phase transition via CLI
    ${result}=    Run Process    ${PYTHON}    -m    cleveragents    plan    execute    ${PLAN_ID}
    Should Be Equal As Integers    ${result.rc}    0
    Should Contain    ${result.stdout}    execute
```

## Troubleshooting

### Coverage report shows 0% or very low coverage

This usually means behave tests are running in a subprocess that coverage cannot instrument. Ensure:

- `nox -s coverage_report` is used (not direct `behave` invocation)
- `parallel = false` is set in `[tool.coverage.run]`
- `COVERAGE_FILE` and `COVERAGE_RCFILE` env vars are set correctly

### Robot tests fail with "resource not found"

Use `${CURDIR}/` prefix for all `Resource` imports in robot files:

```robot
Resource    ${CURDIR}/common.resource
```

### Tests hang indefinitely

Add `timeout` parameters to `Run Process` calls in Robot tests:

```robot
${result}=    Run Process    ${PYTHON}    -m    cleveragents    ...    timeout=30s
```