cleveragents-core/docs/development/testing.md

Testing Guide

Overview

CleverAgents uses a comprehensive testing strategy with three complementary frameworks:

• Behave (BDD/Gherkin) for unit-level and scenario tests under features/
• Robot Framework for integration and end-to-end tests under robot/
• ASV (airspeed velocity) for performance benchmarks under benchmarks/
• Coverage: coverage.py with branch coverage, enforced at >=97%

All tests are executed exclusively through nox sessions. Never invoke behave, robot, or other test runners directly.

Running Tests

# Run all default sessions (lint, typecheck, unit tests, integration tests, coverage)
nox

# Unit tests only (Behave)
nox -s unit_tests

# Integration tests only (Robot Framework)
nox -s integration_tests

# Coverage report with 97% enforcement
nox -s coverage_report

# Run a specific feature file
nox -s unit_tests -- features/plan_model.feature

# Benchmarks (ASV)
nox -s benchmark

# Type checking (pyright)
nox -s typecheck

# Linting (ruff)
nox -s lint

# Formatting check
nox -s format -- --check

Coverage Requirements

Threshold: 97%

The project enforces a minimum 97% code coverage at all times. This is a hard gate:

• The nox -s coverage_report session fails if coverage drops below 97%.
• The Forgejo CI coverage job runs nox -s coverage_report and blocks merges on failure.
• Branch coverage is enabled (branch = true in pyproject.toml).

How Coverage Is Measured

Coverage is measured by running Behave tests under coverage.py in serial mode:

coverage run --source=src -m behave -q --tags=-discovery --no-capture features/
coverage report --show-missing --fail-under=97

Coverage Output

On success, the nox session emits:

COVERAGE OK: 97.2% (threshold: 97%)

On failure, the nox session emits and exits non-zero:

COVERAGE FAILED: 95.3% < 97% threshold

Both messages are single-line and CI-parseable. The CI pipeline greps for these lines to surface them in job summaries.

Coverage Reports

Three report formats are generated under build/:

Report	Path	Description
Terminal	stdout	Per-file summary with --show-missing
HTML	build/htmlcov/index.html	Interactive browser report
XML	build/coverage.xml	Cobertura-format for CI tools
JSON	build/coverage.json	Machine-readable totals and per-file data

Coverage Configuration

Coverage settings live in pyproject.toml:

[tool.coverage.run]
source = ["src", "scripts"]
branch = true
parallel = false
omit = [
    "*/tests/*",
    "*/test_*",
    "features/*",
    "*/features/*",
    "*/__pycache__/*",
    "*/site-packages/*",
    "*/dependency_injector/*",
    "*/venv/*",
    "*/.venv/*",
    "*/.nox/*",
    "src/cleveragents/discovery/*",
]
data_file = "build/.coverage"

[tool.coverage.html]
directory = "build/htmlcov"

[tool.coverage.xml]
output = "build/coverage.xml"

The --fail-under=97 threshold is enforced in the coverage_report nox session (noxfile.py).

How to Improve Coverage

1. Run nox -s coverage_report to generate the HTML report.
2. Open build/htmlcov/index.html in a browser to identify uncovered files.
3. Sort by "Missing" column to find files with the most uncovered lines.
4. Write Behave scenarios targeting the uncovered code paths.
5. Re-run nox -s coverage_report to verify improvement.

Do not lower the threshold, add # pragma: no cover comments, or expand the omit list without explicit team agreement.

All test files go under features/ (Behave) or robot/ (Robot Framework). Never create a tests/ directory.

Unit Tests (Behave)

Unit tests use the Behave BDD framework with Gherkin-syntax .feature files.

Structure

features/
  *.feature          # Feature files (Gherkin scenarios)
  steps/             # Step definitions (Python)
  mocks/             # Mock implementations (test doubles)
  fixtures/          # Test fixture data
  environment.py     # Behave hooks (before/after scenario, etc.)

Guidelines

• No pytest: All unit tests must be Behave-based. There is intentionally no tests/ directory.
• Step naming: Use unique, descriptive step names. Behave loads all step files globally; ambiguous steps cause AmbiguousStep errors.
• Feature naming: Name feature-specific step files after their feature (e.g., foo.feature -> foo_steps.py).
• Mocks in features/ only: All mock implementations live in features/mocks/. Production code (src/) must never contain test doubles.

Running Specific Features

# Run a single feature file
nox -s unit_tests -- features/plan_model.feature

# Run scenarios with a specific tag
nox -s unit_tests -- --tags=@smoke

Integration Tests (Robot Framework)

Integration tests use Robot Framework for end-to-end and system-level testing.

Structure

robot/
  *.robot            # Test suites
  *.resource         # Shared keywords and variables
  common.resource    # Common setup/teardown keywords
  v2_paths.resource  # Path configuration

Guidelines

• Resource paths: Always use ${CURDIR}/ prefix for Resource imports (e.g., Resource ${CURDIR}/common.resource).
• Python injection: Use ${PYTHON} variable (injected by nox) instead of bare python in Run Process calls.
• Timeouts: Always add timeout= to Run Process calls that invoke long-running commands.
• Slow tests: Tag tests that require external services (API keys, running servers) with slow. These are excluded in CI via --exclude slow.

Running Specific Suites

# Run a single Robot suite
nox -s integration_tests -- --suite robot/plan_generation_graph.robot

# Include slow tests
nox -s slow_integration_tests

Performance Benchmarks (ASV)

ASV (airspeed velocity) tracks performance over time.

Structure

benchmarks/
  *.py               # Benchmark suites (Time*/Mem*/Track* classes)
asv.conf.json        # ASV configuration

Running Benchmarks

nox -s benchmark

Writing Effective Tests

Behave Test Guidelines

Feature: Plan lifecycle transitions

  Scenario: Execute plan transitions phase from strategize to execute
    Given an action "local/build-app" exists
    And a plan is created using "local/build-app" on project "local/my-project"
    And strategize is completed
    When I execute the plan
    Then the plan phase should be "execute"
    And the processing state should be "queued"

Robot Framework Guidelines

*** Settings ***
Documentation     Plan lifecycle integration tests
Library           Process
Library           OperatingSystem
Resource          ${CURDIR}/common.resource

*** Test Cases ***
Plan Execute Transitions Correctly
    [Documentation]    Verify execute phase transition via CLI
    ${result}=    Run Process    ${PYTHON}    -m    cleveragents    plan    execute    ${PLAN_ID}
    Should Be Equal As Integers    ${result.rc}    0
    Should Contain    ${result.stdout}    execute

CLI Lifecycle Test Suites

The CLI lifecycle suites provide comprehensive coverage for action and plan lifecycle commands, including success paths, error paths, multi-project arguments, automation profile overrides, invariants, actor overrides, phase visibility, and terminal outcomes.

Behave Suite: features/cli_lifecycle_coverage.feature

Covers 68 scenarios across these areas:

Area	Scenarios	Description
Action CRUD	13	create, list, show, archive with success + error paths
Plan Use	10	basic use, multi-project, automation profile, invariants, actor overrides
Plan Status	9	phase visibility (strategize, execute, apply) and terminal outcomes (applied, constrained, errored, cancelled)
Plan Execute	5	execute success, phase transition, error paths
Plan Apply	5	lifecycle-apply success, terminal outcomes, error paths
Plan Cancel	5	cancel with reason, already-terminal guard, error paths
Plan List	4	lifecycle-list with and without filters
Negative cases	17	missing config, invalid args, invalid project names, unknown actions/resources

Step definitions: features/steps/cli_lifecycle_coverage_steps.py

All step names are prefixed with lifecycle coverage to avoid AmbiguousStep conflicts with existing steps.

Robot Suite: robot/cli_lifecycle_e2e.robot

End-to-end integration tests exercising the full action-to-apply lifecycle through the CLI entry points. Uses robot/helper_cli_lifecycle_e2e.py as a Python helper that mocks the service layer while exercising real CLI argument parsing and output formatting.

Test Case	Description
Action Create From Config Via CLI	Creates an action from YAML config
Plan Use Creates Plan In Strategize Phase	Uses an action to create a plan
Plan Execute Transitions To Execute Phase	Executes a plan, checks phase
Plan Lifecycle Apply Transitions To Apply Phase	Applies a plan, checks phase
Plan Status Shows Plan Details	Verifies status output rendering
Plan Cancel Cancels Non-Terminal Plan	Cancels with a reason string
Full Lifecycle Action To Apply	End-to-end: create -> use -> execute -> apply
Plan Lifecycle List Shows Plans	Verifies lifecycle-list output

ASV Benchmark: benchmarks/plan_cli_smoke_bench.py

Three benchmark suites measuring CLI argument parsing overhead (service layer is mocked so only parsing + routing cost is measured):

• ActionCLIParsingSuite -- time_action_create, time_action_list, time_action_list_with_filters, time_action_show, time_action_archive
• PlanUseArgParsingSuite -- time_use_minimal, time_use_multi_project, time_use_with_all_flags, time_use_with_args_only
• PlanLifecycleCmdParsingSuite -- time_execute, time_lifecycle_apply, time_status_by_id, time_status_list_all, time_cancel_with_reason, time_lifecycle_list, time_lifecycle_list_filtered

Running the CLI Lifecycle Suites

# Behave only (our feature)
nox -s unit_tests -- features/cli_lifecycle_coverage.feature

# Robot only (our suite)
nox -s integration_tests -- --suite robot/cli_lifecycle_e2e.robot

# Benchmarks
nox -s benchmark

Persistence Test Suites

The persistence layer (SQLAlchemy repositories for plans and actions) has dedicated test suites at both unit and integration levels.

Behave: Plan Persistence (features/plan_persistence.feature)

19 scenarios covering LifecyclePlanRepository operations:

• Create: Persist plans via repository, retrieve by ULID, verify all fields round-trip.
• Phase/state transitions: Update phase and processing_state, verify persistence after each transition.
• List filters: Filter plans by phase, processing state, and action name.
• Plan tree links: Parent/child/root hierarchy with parent_plan_id and root_plan_id.
• Cross-restart: Close the SQLite session, reopen from disk, verify plans survive reconnection (including project links, arguments, and invariants).

Step definitions: features/steps/plan_persistence_steps.py

Fixtures: Each scenario gets a fresh temp SQLite database via _setup_db() / _teardown_db() helper functions. The database is file-backed (not :memory:) so cross-restart scenarios can close and reopen the connection.

Behave: Action Persistence (features/action_persistence.feature)

14 scenarios covering ActionRepository operations:

• CRUD: Create, retrieve by namespaced_name, list available, filter by namespace, archive, delete.
• Argument ordering: Persist 3 ActionArgument entries and verify positional order is preserved.
• Invariant ordering: Persist 3 invariant strings and verify order is preserved.
• Terminal state storage: Create plans from actions in Apply phase with each terminal ProcessingState (applied, constrained, errored, cancelled) and verify persistence.

Step definitions: features/steps/action_persistence_steps.py

Robot: Plan Persistence E2E (robot/plan_persistence_e2e.robot)

5 end-to-end tests exercising the persistence layer through Python helper scripts:

Test Case	Description
Plan Full Lifecycle Persistence	Create action + plan, transition through all phases to applied terminal state
Plan Restart Persistence	Create plan, close DB, reopen, verify all fields survive
Plan Concurrent Session Access	Two independent sessions against the same DB file
Action CRUD Persistence E2E	Create, read, update, delete an action with arguments/invariants
Plan Tree Hierarchy Persistence E2E	Parent/child plan hierarchy with root_plan_id verification

Helper script: robot/helper_plan_persistence_e2e.py

ASV Benchmarks (benchmarks/persistence_suites_bench.py)

Performance baselines for persistence test operations:

• TimePlanPersistenceSuites.time_create_and_retrieve_plan -- Plan round-trip latency
• TimePlanPersistenceSuites.time_phase_state_transition -- Phase/state update cycle
• TimePlanPersistenceSuites.time_list_plans_filtered -- List with filters
• TimePlanPersistenceSuites.time_plan_tree_operations -- Parent/child hierarchy
• TimeActionPersistenceSuites.time_create_and_retrieve_action -- Action round-trip
• TimeActionPersistenceSuites.time_action_arguments_ordering -- Ordered arguments persistence
• TimeActionPersistenceSuites.time_cross_restart_persistence -- Close/reopen cycle

Run with: nox -s benchmark

Running Persistence Tests

# Unit tests (Behave) -- runs all features including persistence
nox -s unit_tests

# Run only plan persistence scenarios
nox -s unit_tests -- features/plan_persistence.feature

# Run only action persistence scenarios
nox -s unit_tests -- features/action_persistence.feature

# Integration tests (Robot) -- runs all robot suites including persistence E2E
nox -s integration_tests

# Run only plan persistence E2E
python -m robot robot/plan_persistence_e2e.robot

CI Pipeline

The Forgejo CI pipeline runs these jobs in order:

1. lint — nox -s lint (ruff check)
2. typecheck — nox -s typecheck (pyright)
3. security — nox -s security_scan (bandit + vulture)
4. quality — nox -s complexity (radon)
5. unit_tests — nox -s unit_tests (behave)
6. integration_tests — nox -s integration_tests (robot)
7. coverage — nox -s coverage_report (fail-under 97%)
8. build — nox -s build (wheel)

All jobs use Python 3.13 and route commands through nox. See docs/development/ci-cd.md for full CI/CD documentation.

Troubleshooting

Coverage report shows 0% or very low coverage

Ensure you're running in serial mode (not parallel). The coverage_report nox session handles this correctly. Do not run behave directly outside of coverage run.

Also ensure:

• parallel = false is set in [tool.coverage.run]
• COVERAGE_FILE and COVERAGE_RCFILE env vars are set correctly by the nox session

AmbiguousStep errors in Behave

Two step files define the same @given/@when/@then pattern. Behave loads all step files globally. Make step names unique or use more specific patterns.

Robot Resource file does not exist

Use Resource ${CURDIR}/filename.resource instead of bare Resource filename.resource. Robot resolves bare paths relative to CWD, not the test file directory.

Robot No module named cleveragents

Ensure Robot tests use ${PYTHON} (injected by nox from the venv) instead of bare python in Run Process calls.

Tests hang indefinitely

Add timeout parameters to Run Process calls in Robot tests:

${result}=    Run Process    ${PYTHON}    -m    cleveragents    ...    timeout=30s

Scale Testing

Scale tests validate performance baselines for repository indexing and decomposition across different repository sizes (1K, 5K, and 10K files).

Fixtures

Scale test fixtures live in features/fixtures/scale/:

• scale_metadata.json — Scale profile definitions (file count, language mix, expected timings)
• baseline_thresholds.json — Performance threshold matrices (p50/p95/p99 per operation per file count)
• generator_instructions.md — Manual instructions for generating synthetic repos

Running Scale Tests

# Behave scale test scenarios
nox -s unit_tests -- features/scale_test.feature

# Robot scale test integration
nox -s integration_tests -- --suite robot/scale_test.robot

# ASV scale fixture benchmarks
nox -s benchmark

Scale Test Suites

Suite	Framework	Scenarios	Description
features/scale_test.feature	Behave	20	Fixture loading, metadata validation, threshold checks, distribution simulation
robot/scale_test.robot	Robot	6	Integration-level fixture validation via helper script
benchmarks/scale_fixture_bench.py	ASV	6	Performance benchmarks for fixture processing

For full scale testing documentation, see docs/development/scale_testing.md.