cleveragents-core/docs/development/testing.md

# Testing Guide

## Overview

CleverAgents uses a comprehensive testing strategy with three complementary frameworks:

- **Behave** (BDD/Gherkin) for unit-level and scenario tests under `features/`
- **Robot Framework** for integration and end-to-end tests under `robot/`
- **ASV (airspeed velocity)** for performance benchmarks under `benchmarks/`
- **Coverage**: `coverage.py` with branch coverage, enforced at **>=97%**

All tests are executed exclusively through `nox` sessions. Never invoke `behave`, `robot`, or other test runners directly.

## Running Tests

```bash
# Run all default sessions (lint, typecheck, unit tests, integration tests, coverage)
nox

# Unit tests only (Behave)
nox -s unit_tests

# Integration tests only (Robot Framework)
nox -s integration_tests

# Coverage report with 97% enforcement
nox -s coverage_report

# Run a specific feature file
nox -s unit_tests -- features/plan_model.feature

# Benchmarks (ASV)
nox -s benchmark

# Type checking (pyright)
nox -s typecheck

# Linting (ruff)
nox -s lint

# Formatting check
nox -s format -- --check
```

## Coverage Requirements

### Threshold: 97%

The project enforces a **minimum 97% code coverage** at all times. This is a hard gate:

- The `nox -s coverage_report` session fails if coverage drops below 97%.
- The Forgejo CI `coverage` job runs `nox -s coverage_report` and blocks merges on failure.
- Branch coverage is enabled (`branch = true` in `pyproject.toml`).

### How Coverage Is Measured

Coverage is measured by running Behave tests under `coverage.py` in serial mode:

```
coverage run --source=src -m behave -q --tags=-discovery --no-capture features/
coverage report --show-missing --fail-under=97
```

### Coverage Output

On **success**, the nox session emits:

```
COVERAGE OK: 97.2% (threshold: 97%)
```

On **failure**, the nox session emits and exits non-zero:

```
COVERAGE FAILED: 95.3% < 97% threshold
```

Both messages are single-line and CI-parseable. The CI pipeline greps for these lines to surface them in job summaries.

### Coverage Reports

Three report formats are generated under `build/`:

| Report | Path | Description |
|--------|------|-------------|
| Terminal | stdout | Per-file summary with `--show-missing` |
| HTML | `build/htmlcov/index.html` | Interactive browser report |
| XML | `build/coverage.xml` | Cobertura-format for CI tools |
| JSON | `build/coverage.json` | Machine-readable totals and per-file data |

### Coverage Configuration

Coverage settings live in `pyproject.toml`:

```toml
[tool.coverage.run]
source = ["src", "scripts"]
branch = true
parallel = false
omit = [
    "*/tests/*",
    "*/test_*",
    "features/*",
    "*/features/*",
    "*/__pycache__/*",
    "*/site-packages/*",
    "*/dependency_injector/*",
    "*/venv/*",
    "*/.venv/*",
    "*/.nox/*",
    "src/cleveragents/discovery/*",
]
data_file = "build/.coverage"

[tool.coverage.html]
directory = "build/htmlcov"

[tool.coverage.xml]
output = "build/coverage.xml"
```

The `--fail-under=97` threshold is enforced in the `coverage_report` nox session (`noxfile.py`).

### How to Improve Coverage

1. Run `nox -s coverage_report` to generate the HTML report.
2. Open `build/htmlcov/index.html` in a browser to identify uncovered files.
3. Sort by "Missing" column to find files with the most uncovered lines.
4. Write Behave scenarios targeting the uncovered code paths.
5. Re-run `nox -s coverage_report` to verify improvement.

**Do not** lower the threshold, add `# pragma: no cover` comments, or expand the `omit` list without explicit team agreement.

All test files go under `features/` (Behave) or `robot/` (Robot Framework). Never create a `tests/` directory.

## Unit Tests (Behave)

Unit tests use the [Behave](https://behave.readthedocs.io/) BDD framework with Gherkin-syntax `.feature` files.

### Structure

```
features/
  *.feature          # Feature files (Gherkin scenarios)
  steps/             # Step definitions (Python)
  mocks/             # Mock implementations (test doubles)
  fixtures/          # Test fixture data
  environment.py     # Behave hooks (before/after scenario, etc.)
```

### Guidelines

- **No pytest**: All unit tests must be Behave-based. There is intentionally no `tests/` directory.
- **Step naming**: Use unique, descriptive step names. Behave loads all step files globally; ambiguous steps cause `AmbiguousStep` errors.
- **Feature naming**: Name feature-specific step files after their feature (e.g., `foo.feature` -> `foo_steps.py`).
- **Mocks in features/ only**: All mock implementations live in `features/mocks/`. Production code (`src/`) must never contain test doubles.

### Running Specific Features

```bash
# Run a single feature file
nox -s unit_tests -- features/plan_model.feature

# Run scenarios with a specific tag
nox -s unit_tests -- --tags=@smoke
```

## Integration Tests (Robot Framework)

Integration tests use [Robot Framework](https://robotframework.org/) for end-to-end and system-level testing.

### Structure

```
robot/
  *.robot            # Test suites
  *.resource         # Shared keywords and variables
  common.resource    # Common setup/teardown keywords
  v2_paths.resource  # Path configuration
```

### Guidelines

- **Resource paths**: Always use `${CURDIR}/` prefix for `Resource` imports (e.g., `Resource ${CURDIR}/common.resource`).
- **Python injection**: Use `${PYTHON}` variable (injected by nox) instead of bare `python` in `Run Process` calls.
- **Timeouts**: Always add `timeout=` to `Run Process` calls that invoke long-running commands.
- **Slow tests**: Tag tests that require external services (API keys, running servers) with `slow`. These are excluded in CI via `--exclude slow`.

### Running Specific Suites

```bash
# Run a single Robot suite
nox -s integration_tests -- --suite robot/plan_generation_graph.robot

# Include slow tests
nox -s slow_integration_tests
```

## Performance Benchmarks (ASV)

[ASV (airspeed velocity)](https://asv.readthedocs.io/) tracks performance over time.

### Structure

```
benchmarks/
  *.py               # Benchmark suites (Time*/Mem*/Track* classes)
asv.conf.json        # ASV configuration
```

### Running Benchmarks

```bash
nox -s benchmark
```

### Writing Effective Tests

#### Behave Test Guidelines

```gherkin
Feature: Plan lifecycle transitions

  Scenario: Execute plan transitions phase from strategize to execute
    Given an action "local/build-app" exists
    And a plan is created using "local/build-app" on project "local/my-project"
    And strategize is completed
    When I execute the plan
    Then the plan phase should be "execute"
    And the processing state should be "queued"
```

#### Robot Framework Guidelines

```robot
*** Settings ***
Documentation     Plan lifecycle integration tests
Library           Process
Library           OperatingSystem
Resource          ${CURDIR}/common.resource

*** Test Cases ***
Plan Execute Transitions Correctly
    [Documentation]    Verify execute phase transition via CLI
    ${result}=    Run Process    ${PYTHON}    -m    cleveragents    plan    execute    ${PLAN_ID}
    Should Be Equal As Integers    ${result.rc}    0
    Should Contain    ${result.stdout}    execute
```

## CLI Lifecycle Test Suites

The CLI lifecycle suites provide comprehensive coverage for action and plan lifecycle
commands, including success paths, error paths, multi-project arguments, automation
profile overrides, invariants, actor overrides, phase visibility, and terminal
outcomes.

### Behave Suite: `features/cli_lifecycle_coverage.feature`

Covers 68 scenarios across these areas:

| Area | Scenarios | Description |
|------|-----------|-------------|
| Action CRUD | 13 | create, list, show, archive with success + error paths |
| Plan Use | 10 | basic use, multi-project, automation profile, invariants, actor overrides |
| Plan Status | 9 | phase visibility (strategize, execute, apply) and terminal outcomes (applied, constrained, errored, cancelled) |
| Plan Execute | 5 | execute success, phase transition, error paths |
| Plan Apply | 5 | lifecycle-apply success, terminal outcomes, error paths |
| Plan Cancel | 5 | cancel with reason, already-terminal guard, error paths |
| Plan List | 4 | lifecycle-list with and without filters |
| Negative cases | 17 | missing config, invalid args, invalid project names, unknown actions/resources |

Step definitions: `features/steps/cli_lifecycle_coverage_steps.py`

All step names are prefixed with `lifecycle coverage` to avoid `AmbiguousStep`
conflicts with existing steps.

### Robot Suite: `robot/cli_lifecycle_e2e.robot`

End-to-end integration tests exercising the full action-to-apply lifecycle through
the CLI entry points. Uses `robot/helper_cli_lifecycle_e2e.py` as a Python helper
that mocks the service layer while exercising real CLI argument parsing and output
formatting.

| Test Case | Description |
|-----------|-------------|
| Action Create From Config Via CLI | Creates an action from YAML config |
| Plan Use Creates Plan In Strategize Phase | Uses an action to create a plan |
| Plan Execute Transitions To Execute Phase | Executes a plan, checks phase |
| Plan Lifecycle Apply Transitions To Apply Phase | Applies a plan, checks phase |
| Plan Status Shows Plan Details | Verifies status output rendering |
| Plan Cancel Cancels Non-Terminal Plan | Cancels with a reason string |
| Full Lifecycle Action To Apply | End-to-end: create -> use -> execute -> apply |
| Plan Lifecycle List Shows Plans | Verifies lifecycle-list output |

### ASV Benchmark: `benchmarks/plan_cli_smoke_bench.py`

Three benchmark suites measuring CLI argument parsing overhead (service layer is
mocked so only parsing + routing cost is measured):

- **`ActionCLIParsingSuite`** -- `time_action_create`, `time_action_list`,
  `time_action_list_with_filters`, `time_action_show`, `time_action_archive`
- **`PlanUseArgParsingSuite`** -- `time_use_minimal`, `time_use_multi_project`,
  `time_use_with_all_flags`, `time_use_with_args_only`
- **`PlanLifecycleCmdParsingSuite`** -- `time_execute`, `time_lifecycle_apply`,
  `time_status_by_id`, `time_status_list_all`, `time_cancel_with_reason`,
  `time_lifecycle_list`, `time_lifecycle_list_filtered`

### Running the CLI Lifecycle Suites

```bash
# Behave only (our feature)
nox -s unit_tests -- features/cli_lifecycle_coverage.feature

# Robot only (our suite)
nox -s integration_tests -- --suite robot/cli_lifecycle_e2e.robot

# Benchmarks
nox -s benchmark
```

## Persistence Test Suites

The persistence layer (SQLAlchemy repositories for plans and actions) has dedicated test suites at both unit and integration levels.

### Behave: Plan Persistence (`features/plan_persistence.feature`)

19 scenarios covering `LifecyclePlanRepository` operations:

- **Create**: Persist plans via repository, retrieve by ULID, verify all fields round-trip.
- **Phase/state transitions**: Update `phase` and `processing_state`, verify persistence after each transition.
- **List filters**: Filter plans by phase, processing state, and action name.
- **Plan tree links**: Parent/child/root hierarchy with `parent_plan_id` and `root_plan_id`.
- **Cross-restart**: Close the SQLite session, reopen from disk, verify plans survive reconnection (including project links, arguments, and invariants).

Step definitions: `features/steps/plan_persistence_steps.py`

**Fixtures**: Each scenario gets a fresh temp SQLite database via `_setup_db()` / `_teardown_db()` helper functions. The database is file-backed (not `:memory:`) so cross-restart scenarios can close and reopen the connection.

### Behave: Action Persistence (`features/action_persistence.feature`)

14 scenarios covering `ActionRepository` operations:

- **CRUD**: Create, retrieve by `namespaced_name`, list available, filter by namespace, archive, delete.
- **Argument ordering**: Persist 3 `ActionArgument` entries and verify positional order is preserved.
- **Invariant ordering**: Persist 3 invariant strings and verify order is preserved.
- **Terminal state storage**: Create plans from actions in Apply phase with each terminal `ProcessingState` (`applied`, `constrained`, `errored`, `cancelled`) and verify persistence.

Step definitions: `features/steps/action_persistence_steps.py`

### Robot: Plan Persistence E2E (`robot/plan_persistence_e2e.robot`)

5 end-to-end tests exercising the persistence layer through Python helper scripts:

| Test Case | Description |
|-----------|-------------|
| Plan Full Lifecycle Persistence | Create action + plan, transition through all phases to `applied` terminal state |
| Plan Restart Persistence | Create plan, close DB, reopen, verify all fields survive |
| Plan Concurrent Session Access | Two independent sessions against the same DB file |
| Action CRUD Persistence E2E | Create, read, update, delete an action with arguments/invariants |
| Plan Tree Hierarchy Persistence E2E | Parent/child plan hierarchy with `root_plan_id` verification |

Helper script: `robot/helper_plan_persistence_e2e.py`

### ASV Benchmarks (`benchmarks/persistence_suites_bench.py`)

Performance baselines for persistence test operations:

- `TimePlanPersistenceSuites.time_create_and_retrieve_plan` -- Plan round-trip latency
- `TimePlanPersistenceSuites.time_phase_state_transition` -- Phase/state update cycle
- `TimePlanPersistenceSuites.time_list_plans_filtered` -- List with filters
- `TimePlanPersistenceSuites.time_plan_tree_operations` -- Parent/child hierarchy
- `TimeActionPersistenceSuites.time_create_and_retrieve_action` -- Action round-trip
- `TimeActionPersistenceSuites.time_action_arguments_ordering` -- Ordered arguments persistence
- `TimeActionPersistenceSuites.time_cross_restart_persistence` -- Close/reopen cycle

Run with: `nox -s benchmark`

### Running Persistence Tests

```bash
# Unit tests (Behave) -- runs all features including persistence
nox -s unit_tests

# Run only plan persistence scenarios
nox -s unit_tests -- features/plan_persistence.feature

# Run only action persistence scenarios
nox -s unit_tests -- features/action_persistence.feature

# Integration tests (Robot) -- runs all robot suites including persistence E2E
nox -s integration_tests

# Run only plan persistence E2E
python -m robot robot/plan_persistence_e2e.robot
```

## CI Pipeline

The Forgejo CI pipeline runs these jobs in order:

1. **lint** — `nox -s lint` (ruff check)
2. **typecheck** — `nox -s typecheck` (pyright)
3. **security** — `nox -s security_scan` (bandit + vulture)
4. **quality** — `nox -s complexity` (radon)
5. **unit_tests** — `nox -s unit_tests` (behave)
6. **integration_tests** — `nox -s integration_tests` (robot)
7. **coverage** — `nox -s coverage_report` (fail-under 97%)
8. **build** — `nox -s build` (wheel)

All jobs use Python 3.13 and route commands through `nox`. See `docs/development/ci-cd.md` for full CI/CD documentation.

## Troubleshooting

### Coverage report shows 0% or very low coverage

Ensure you're running in serial mode (not parallel). The `coverage_report` nox session handles this correctly. Do not run `behave` directly outside of `coverage run`.

Also ensure:
- `parallel = false` is set in `[tool.coverage.run]`
- `COVERAGE_FILE` and `COVERAGE_RCFILE` env vars are set correctly by the nox session

### AmbiguousStep errors in Behave

Two step files define the same `@given`/`@when`/`@then` pattern. Behave loads all step files globally. Make step names unique or use more specific patterns.

### Robot `Resource file does not exist`

Use `Resource ${CURDIR}/filename.resource` instead of bare `Resource filename.resource`. Robot resolves bare paths relative to CWD, not the test file directory.

### Robot `No module named cleveragents`

Ensure Robot tests use `${PYTHON}` (injected by nox from the venv) instead of bare `python` in `Run Process` calls.

### Tests hang indefinitely

Add `timeout` parameters to `Run Process` calls in Robot tests:

```robot
${result}=    Run Process    ${PYTHON}    -m    cleveragents    ...    timeout=30s
```

## Scale Testing

Scale tests validate performance baselines for repository indexing and
decomposition across different repository sizes (1K, 5K, and 10K files).

### Fixtures

Scale test fixtures live in `features/fixtures/scale/`:

- `scale_metadata.json` — Scale profile definitions (file count, language mix, expected timings)
- `baseline_thresholds.json` — Performance threshold matrices (p50/p95/p99 per operation per file count)
- `generator_instructions.md` — Manual instructions for generating synthetic repos

### Running Scale Tests

```bash
# Behave scale test scenarios
nox -s unit_tests -- features/scale_test.feature

# Robot scale test integration
nox -s integration_tests -- --suite robot/scale_test.robot

# ASV scale fixture benchmarks
nox -s benchmark
```

### Scale Test Suites

| Suite | Framework | Scenarios | Description |
|-------|-----------|-----------|-------------|
| `features/scale_test.feature` | Behave | 20 | Fixture loading, metadata validation, threshold checks, distribution simulation |
| `robot/scale_test.robot` | Robot | 6 | Integration-level fixture validation via helper script |
| `benchmarks/scale_fixture_bench.py` | ASV | 6 | Performance benchmarks for fixture processing |

For full scale testing documentation, see `docs/development/scale_testing.md`.