cleveragents-core/docs/development/testing.md

# Testing Guide

## Overview

CleverAgents uses a comprehensive testing strategy with three complementary frameworks:

- **Behave** (BDD/Gherkin) for unit-level and scenario tests under `features/`
- **Robot Framework** for integration and end-to-end tests under `robot/`
- **ASV (airspeed velocity)** for performance benchmarks under `benchmarks/`
- **Coverage**: `coverage.py` with branch coverage, enforced at **>=97%**

All tests are executed exclusively through `nox` sessions. Never invoke `behave`, `robot`, or other test runners directly.

## Running Tests

```bash
# Run all default sessions (lint, typecheck, unit tests, integration tests, coverage)
nox

# Unit tests only (Behave)
nox -s unit_tests

# Integration tests only (Robot Framework)
nox -s integration_tests

# Coverage report with 97% enforcement
nox -s coverage_report

# Run a specific feature file
nox -s unit_tests -- features/plan_model.feature

# Benchmarks (ASV)
nox -s benchmark

# Type checking (pyright)
nox -s typecheck

# Linting (ruff)
nox -s lint

# Formatting check
nox -s format -- --check
```

## Coverage Requirements

### Threshold: 97%

The project enforces a **minimum 97% code coverage** at all times. This is a hard gate:

- The `nox -s coverage_report` session fails if coverage drops below 97%.
- The Forgejo CI `coverage` job runs `nox -s coverage_report` and blocks merges on failure.
- Branch coverage is enabled (`branch = true` in `pyproject.toml`).

### How Coverage Is Measured

Coverage is measured by running Behave tests under `coverage.py` in serial mode:

```
coverage run --source=src -m behave -q --tags=-discovery --no-capture features/
coverage report --show-missing --fail-under=97
```

### Coverage Output

On **success**, the nox session emits:

```
COVERAGE OK: 97.2% (threshold: 97%)
```

On **failure**, the nox session emits and exits non-zero:

```
COVERAGE FAILED: 95.3% < 97% threshold
```

Both messages are single-line and CI-parseable. The CI pipeline greps for these lines to surface them in job summaries.

### Coverage Reports

Three report formats are generated under `build/`:

| Report | Path | Description |
|--------|------|-------------|
| Terminal | stdout | Per-file summary with `--show-missing` |
| HTML | `build/htmlcov/index.html` | Interactive browser report |
| XML | `build/coverage.xml` | Cobertura-format for CI tools |
| JSON | `build/coverage.json` | Machine-readable totals and per-file data |

### Coverage Configuration

Coverage settings live in `pyproject.toml`:

```toml
[tool.coverage.run]
source = ["src", "scripts"]
branch = true
parallel = false
omit = [
    "*/tests/*",
    "*/test_*",
    "features/*",
    "*/features/*",
    "*/__pycache__/*",
    "*/site-packages/*",
    "*/dependency_injector/*",
    "*/venv/*",
    "*/.venv/*",
    "*/.nox/*",
    "src/cleveragents/discovery/*",
]
data_file = "build/.coverage"

[tool.coverage.html]
directory = "build/htmlcov"

[tool.coverage.xml]
output = "build/coverage.xml"
```

The `--fail-under=97` threshold is enforced in the `coverage_report` nox session (`noxfile.py`).

### How to Improve Coverage

1. Run `nox -s coverage_report` to generate the HTML report.
2. Open `build/htmlcov/index.html` in a browser to identify uncovered files.
3. Sort by "Missing" column to find files with the most uncovered lines.
4. Write Behave scenarios targeting the uncovered code paths.
5. Re-run `nox -s coverage_report` to verify improvement.

**Do not** lower the threshold, add `# pragma: no cover` comments, or expand the `omit` list without explicit team agreement.

All test files go under `features/` (Behave) or `robot/` (Robot Framework). Never create a `tests/` directory.

## Unit Tests (Behave)

Unit tests use the [Behave](https://behave.readthedocs.io/) BDD framework with Gherkin-syntax `.feature` files.

### Structure

```
features/
  *.feature          # Feature files (Gherkin scenarios)
  steps/             # Step definitions (Python)
  mocks/             # Mock implementations (test doubles)
  fixtures/          # Test fixture data
  environment.py     # Behave hooks (before/after scenario, etc.)
```

### Guidelines

- **No pytest**: All unit tests must be Behave-based. There is intentionally no `tests/` directory.
- **Step naming**: Use unique, descriptive step names. Behave loads all step files globally; ambiguous steps cause `AmbiguousStep` errors.
- **Feature naming**: Name feature-specific step files after their feature (e.g., `foo.feature` -> `foo_steps.py`).
- **Mocks in features/ only**: All mock implementations live in `features/mocks/`. Production code (`src/`) must never contain test doubles.

### Running Specific Features

```bash
# Run a single feature file
nox -s unit_tests -- features/plan_model.feature

# Run scenarios with a specific tag
nox -s unit_tests -- --tags=@smoke
```

## Integration Tests (Robot Framework)

Integration tests use [Robot Framework](https://robotframework.org/) for end-to-end and system-level testing.

### Structure

```
robot/
  *.robot            # Test suites
  *.resource         # Shared keywords and variables
  common.resource    # Common setup/teardown keywords
  v2_paths.resource  # Path configuration
```

### Guidelines

- **Resource paths**: Always use `${CURDIR}/` prefix for `Resource` imports (e.g., `Resource ${CURDIR}/common.resource`).
- **Python injection**: Use `${PYTHON}` variable (injected by nox) instead of bare `python` in `Run Process` calls.
- **Timeouts**: Always add `timeout=` to `Run Process` calls that invoke long-running commands.
- **Slow tests**: Tag tests that require external services (API keys, running servers) with `slow`. These are excluded in CI via `--exclude slow`.

### Running Specific Suites

```bash
# Run a single Robot suite
nox -s integration_tests -- --suite robot/plan_generation_graph.robot

# Include slow tests
nox -s slow_integration_tests
```

## Performance Benchmarks (ASV)

[ASV (airspeed velocity)](https://asv.readthedocs.io/) tracks performance over time.

### Structure

```
benchmarks/
  *.py               # Benchmark suites (Time*/Mem*/Track* classes)
asv.conf.json        # ASV configuration
```

### Running Benchmarks

```bash
nox -s benchmark
```

### Writing Effective Tests

#### Behave Test Guidelines

```gherkin
Feature: Plan lifecycle transitions

  Scenario: Execute plan transitions phase from strategize to execute
    Given an action "local/build-app" exists
    And a plan is created using "local/build-app" on project "local/my-project"
    And strategize is completed
    When I execute the plan
    Then the plan phase should be "execute"
    And the processing state should be "queued"
```

#### Robot Framework Guidelines

```robot
*** Settings ***
Documentation     Plan lifecycle integration tests
Library           Process
Library           OperatingSystem
Resource          ${CURDIR}/common.resource

*** Test Cases ***
Plan Execute Transitions Correctly
    [Documentation]    Verify execute phase transition via CLI
    ${result}=    Run Process    ${PYTHON}    -m    cleveragents    plan    execute    ${PLAN_ID}
    Should Be Equal As Integers    ${result.rc}    0
    Should Contain    ${result.stdout}    execute
```

## CI Pipeline

The Forgejo CI pipeline runs these jobs in order:

1. **lint** — `nox -s lint` (ruff check)
2. **typecheck** — `nox -s typecheck` (pyright)
3. **security** — `nox -s security_scan` (bandit + vulture)
4. **quality** — `nox -s complexity` (radon)
5. **unit_tests** — `nox -s unit_tests` (behave)
6. **integration_tests** — `nox -s integration_tests` (robot)
7. **coverage** — `nox -s coverage_report` (fail-under 97%)
8. **build** — `nox -s build` (wheel)

All jobs use Python 3.13 and route commands through `nox`. See `docs/development/ci-cd.md` for full CI/CD documentation.

## Troubleshooting

### Coverage report shows 0% or very low coverage

Ensure you're running in serial mode (not parallel). The `coverage_report` nox session handles this correctly. Do not run `behave` directly outside of `coverage run`.

Also ensure:
- `parallel = false` is set in `[tool.coverage.run]`
- `COVERAGE_FILE` and `COVERAGE_RCFILE` env vars are set correctly by the nox session

### AmbiguousStep errors in Behave

Two step files define the same `@given`/`@when`/`@then` pattern. Behave loads all step files globally. Make step names unique or use more specific patterns.

### Robot `Resource file does not exist`

Use `Resource ${CURDIR}/filename.resource` instead of bare `Resource filename.resource`. Robot resolves bare paths relative to CWD, not the test file directory.

### Robot `No module named cleveragents`

Ensure Robot tests use `${PYTHON}` (injected by nox from the venv) instead of bare `python` in `Run Process` calls.

### Tests hang indefinitely

Add `timeout` parameters to `Run Process` calls in Robot tests:

```robot
${result}=    Run Process    ${PYTHON}    -m    cleveragents    ...    timeout=30s
```