cleveragents-core/docs/development/testing.md

Testing Guide

Overview

CleverAgents uses a comprehensive testing strategy with three complementary frameworks:

• Behave (BDD/Gherkin) for unit-level and scenario tests under features/
• Robot Framework for integration and end-to-end tests under robot/
• ASV (airspeed velocity) for performance benchmarks under benchmarks/
• Coverage: coverage.py with branch coverage, enforced at >=97%

All tests are executed exclusively through nox sessions. Never invoke behave, robot, or other test runners directly.

Running Tests

# Run all default sessions (lint, typecheck, unit tests, integration tests, coverage)
nox

# Unit tests only (Behave)
nox -s unit_tests

# Integration tests only (Robot Framework)
nox -s integration_tests

# Coverage report with 97% enforcement
nox -s coverage_report

# Run a specific feature file
nox -s unit_tests -- features/plan_model.feature

# Benchmarks (ASV)
nox -s benchmark

# Type checking (pyright)
nox -s typecheck

# Linting (ruff)
nox -s lint

# Formatting check
nox -s format -- --check

Coverage Requirements

Threshold: 97%

The project enforces a minimum 97% code coverage at all times. This is a hard gate:

• The nox -s coverage_report session fails if coverage drops below 97%.
• The Forgejo CI coverage job runs nox -s coverage_report and blocks merges on failure.
• Branch coverage is enabled (branch = true in pyproject.toml).

How Coverage Is Measured

Coverage is measured by running Behave tests under coverage.py in serial mode:

coverage run --source=src -m behave -q --tags=-discovery --no-capture features/
coverage report --show-missing --fail-under=97

Coverage Output

On success, the nox session emits:

COVERAGE OK: 97.2% (threshold: 97%)

On failure, the nox session emits and exits non-zero:

COVERAGE FAILED: 95.3% < 97% threshold

Both messages are single-line and CI-parseable. The CI pipeline greps for these lines to surface them in job summaries.

Coverage Reports

Three report formats are generated under build/:

Report	Path	Description
Terminal	stdout	Per-file summary with --show-missing
HTML	build/htmlcov/index.html	Interactive browser report
XML	build/coverage.xml	Cobertura-format for CI tools
JSON	build/coverage.json	Machine-readable totals and per-file data

Coverage Configuration

Coverage settings live in pyproject.toml:

[tool.coverage.run]
source = ["src", "scripts"]
branch = true
parallel = false
omit = [
    "*/tests/*",
    "*/test_*",
    "features/*",
    "*/features/*",
    "*/__pycache__/*",
    "*/site-packages/*",
    "*/dependency_injector/*",
    "*/venv/*",
    "*/.venv/*",
    "*/.nox/*",
    "src/cleveragents/discovery/*",
]
data_file = "build/.coverage"

[tool.coverage.html]
directory = "build/htmlcov"

[tool.coverage.xml]
output = "build/coverage.xml"

The --fail-under=97 threshold is enforced in the coverage_report nox session (noxfile.py).

How to Improve Coverage

1. Run nox -s coverage_report to generate the HTML report.
2. Open build/htmlcov/index.html in a browser to identify uncovered files.
3. Sort by "Missing" column to find files with the most uncovered lines.
4. Write Behave scenarios targeting the uncovered code paths.
5. Re-run nox -s coverage_report to verify improvement.

Do not lower the threshold, add # pragma: no cover comments, or expand the omit list without explicit team agreement.

All test files go under features/ (Behave) or robot/ (Robot Framework). Never create a tests/ directory.

Unit Tests (Behave)

Unit tests use the Behave BDD framework with Gherkin-syntax .feature files.

Structure

features/
  *.feature          # Feature files (Gherkin scenarios)
  steps/             # Step definitions (Python)
  mocks/             # Mock implementations (test doubles)
  fixtures/          # Test fixture data
  environment.py     # Behave hooks (before/after scenario, etc.)

Guidelines

• No pytest: All unit tests must be Behave-based. There is intentionally no tests/ directory.
• Step naming: Use unique, descriptive step names. Behave loads all step files globally; ambiguous steps cause AmbiguousStep errors.
• Feature naming: Name feature-specific step files after their feature (e.g., foo.feature -> foo_steps.py).
• Mocks in features/ only: All mock implementations live in features/mocks/. Production code (src/) must never contain test doubles.

Running Specific Features

# Run a single feature file
nox -s unit_tests -- features/plan_model.feature

# Run scenarios with a specific tag
nox -s unit_tests -- --tags=@smoke

Integration Tests (Robot Framework)

Integration tests use Robot Framework for end-to-end and system-level testing.

Structure

robot/
  *.robot            # Test suites
  *.resource         # Shared keywords and variables
  common.resource    # Common setup/teardown keywords
  v2_paths.resource  # Path configuration

Guidelines

• Resource paths: Always use ${CURDIR}/ prefix for Resource imports (e.g., Resource ${CURDIR}/common.resource).
• Python injection: Use ${PYTHON} variable (injected by nox) instead of bare python in Run Process calls.
• Timeouts: Always add timeout= to Run Process calls that invoke long-running commands.
• Slow tests: Tag tests that require external services (API keys, running servers) with slow. These are excluded in CI via --exclude slow.

Running Specific Suites

# Run a single Robot suite
nox -s integration_tests -- --suite robot/plan_generation_graph.robot

# Include slow tests
nox -s slow_integration_tests

Performance Benchmarks (ASV)

ASV (airspeed velocity) tracks performance over time.

Structure

benchmarks/
  *.py               # Benchmark suites (Time*/Mem*/Track* classes)
asv.conf.json        # ASV configuration

Running Benchmarks

nox -s benchmark

Writing Effective Tests

Behave Test Guidelines

Feature: Plan lifecycle transitions

  Scenario: Execute plan transitions phase from strategize to execute
    Given an action "local/build-app" exists
    And a plan is created using "local/build-app" on project "local/my-project"
    And strategize is completed
    When I execute the plan
    Then the plan phase should be "execute"
    And the processing state should be "queued"

Robot Framework Guidelines

*** Settings ***
Documentation     Plan lifecycle integration tests
Library           Process
Library           OperatingSystem
Resource          ${CURDIR}/common.resource

*** Test Cases ***
Plan Execute Transitions Correctly
    [Documentation]    Verify execute phase transition via CLI
    ${result}=    Run Process    ${PYTHON}    -m    cleveragents    plan    execute    ${PLAN_ID}
    Should Be Equal As Integers    ${result.rc}    0
    Should Contain    ${result.stdout}    execute

CLI Lifecycle Test Suites

The CLI lifecycle suites provide comprehensive coverage for action and plan lifecycle commands, including success paths, error paths, multi-project arguments, automation profile overrides, invariants, actor overrides, phase visibility, and terminal outcomes.

Behave Suite: features/cli_lifecycle_coverage.feature

Covers 68 scenarios across these areas:

Area	Scenarios	Description
Action CRUD	13	create, list, show, archive with success + error paths
Plan Use	10	basic use, multi-project, automation profile, invariants, actor overrides
Plan Status	9	phase visibility (strategize, execute, apply) and terminal outcomes (applied, constrained, errored, cancelled)
Plan Execute	5	execute success, phase transition, error paths
Plan Apply	5	lifecycle-apply success, terminal outcomes, error paths
Plan Cancel	5	cancel with reason, already-terminal guard, error paths
Plan List	4	lifecycle-list with and without filters
Negative cases	17	missing config, invalid args, invalid project names, unknown actions/resources

Step definitions: features/steps/cli_lifecycle_coverage_steps.py

All step names are prefixed with lifecycle coverage to avoid AmbiguousStep conflicts with existing steps.

Robot Suite: robot/cli_lifecycle_e2e.robot

End-to-end integration tests exercising the full action-to-apply lifecycle through the CLI entry points. Uses robot/helper_cli_lifecycle_e2e.py as a Python helper that mocks the service layer while exercising real CLI argument parsing and output formatting.

Test Case	Description
Action Create From Config Via CLI	Creates an action from YAML config
Plan Use Creates Plan In Strategize Phase	Uses an action to create a plan
Plan Execute Transitions To Execute Phase	Executes a plan, checks phase
Plan Lifecycle Apply Transitions To Apply Phase	Applies a plan, checks phase
Plan Status Shows Plan Details	Verifies status output rendering
Plan Cancel Cancels Non-Terminal Plan	Cancels with a reason string
Full Lifecycle Action To Apply	End-to-end: create -> use -> execute -> apply
Plan Lifecycle List Shows Plans	Verifies lifecycle-list output

ASV Benchmark: benchmarks/plan_cli_smoke_bench.py

Three benchmark suites measuring CLI argument parsing overhead (service layer is mocked so only parsing + routing cost is measured):

• ActionCLIParsingSuite -- time_action_create, time_action_list, time_action_list_with_filters, time_action_show, time_action_archive
• PlanUseArgParsingSuite -- time_use_minimal, time_use_multi_project, time_use_with_all_flags, time_use_with_args_only
• PlanLifecycleCmdParsingSuite -- time_execute, time_lifecycle_apply, time_status_by_id, time_status_list_all, time_cancel_with_reason, time_lifecycle_list, time_lifecycle_list_filtered

Running the CLI Lifecycle Suites

# Behave only (our feature)
nox -s unit_tests -- features/cli_lifecycle_coverage.feature

# Robot only (our suite)
nox -s integration_tests -- --suite robot/cli_lifecycle_e2e.robot

# Benchmarks
nox -s benchmark

Persistence Test Suites

The persistence layer (SQLAlchemy repositories for plans and actions) has dedicated test suites at both unit and integration levels.

Behave: Plan Persistence (features/plan_persistence.feature)

19 scenarios covering LifecyclePlanRepository operations:

• Create: Persist plans via repository, retrieve by ULID, verify all fields round-trip.
• Phase/state transitions: Update phase and processing_state, verify persistence after each transition.
• List filters: Filter plans by phase, processing state, and action name.
• Plan tree links: Parent/child/root hierarchy with parent_plan_id and root_plan_id.
• Cross-restart: Close the SQLite session, reopen from disk, verify plans survive reconnection (including project links, arguments, and invariants).

Step definitions: features/steps/plan_persistence_steps.py

Fixtures: Each scenario gets a fresh temp SQLite database via _setup_db() / _teardown_db() helper functions. The database is file-backed (not :memory:) so cross-restart scenarios can close and reopen the connection.

Behave: Action Persistence (features/action_persistence.feature)

14 scenarios covering ActionRepository operations:

• CRUD: Create, retrieve by namespaced_name, list available, filter by namespace, archive, delete.
• Argument ordering: Persist 3 ActionArgument entries and verify positional order is preserved.
• Invariant ordering: Persist 3 invariant strings and verify order is preserved.
• Terminal state storage: Create plans from actions in Apply phase with each terminal ProcessingState (applied, constrained, errored, cancelled) and verify persistence.

Step definitions: features/steps/action_persistence_steps.py

Robot: Plan Persistence E2E (robot/plan_persistence_e2e.robot)

5 end-to-end tests exercising the persistence layer through Python helper scripts:

Test Case	Description
Plan Full Lifecycle Persistence	Create action + plan, transition through all phases to applied terminal state
Plan Restart Persistence	Create plan, close DB, reopen, verify all fields survive
Plan Concurrent Session Access	Two independent sessions against the same DB file
Action CRUD Persistence E2E	Create, read, update, delete an action with arguments/invariants
Plan Tree Hierarchy Persistence E2E	Parent/child plan hierarchy with root_plan_id verification

Helper script: robot/helper_plan_persistence_e2e.py

ASV Benchmarks (benchmarks/persistence_suites_bench.py)

Performance baselines for persistence test operations:

• TimePlanPersistenceSuites.time_create_and_retrieve_plan -- Plan round-trip latency
• TimePlanPersistenceSuites.time_phase_state_transition -- Phase/state update cycle
• TimePlanPersistenceSuites.time_list_plans_filtered -- List with filters
• TimePlanPersistenceSuites.time_plan_tree_operations -- Parent/child hierarchy
• TimeActionPersistenceSuites.time_create_and_retrieve_action -- Action round-trip
• TimeActionPersistenceSuites.time_action_arguments_ordering -- Ordered arguments persistence
• TimeActionPersistenceSuites.time_cross_restart_persistence -- Close/reopen cycle

Run with: nox -s benchmark

Running Persistence Tests

# Unit tests (Behave) -- runs all features including persistence
nox -s unit_tests

# Run only plan persistence scenarios
nox -s unit_tests -- features/plan_persistence.feature

# Run only action persistence scenarios
nox -s unit_tests -- features/action_persistence.feature

# Integration tests (Robot) -- runs all robot suites including persistence E2E
nox -s integration_tests

# Run only plan persistence E2E
python -m robot robot/plan_persistence_e2e.robot

Subplan Model Test Suites

The subplan model (hierarchy, configuration, failure handling) has dedicated test suites at unit, integration, and benchmark levels.

Behave: Subplan Model (features/subplan_model.feature)

33 scenarios covering Plan hierarchy properties, SubplanConfig defaults, SubplanStatus tracking, SubplanFailureHandler decisions, and CLI dict rendering:

Area	Scenarios	Description
Root plan identity	2	is_root_plan, is_subplan, depth for root plans
Child plan identity	3	Parent reference, attempt counter, root propagation
Default values	4	Phase, state, automation level, subplan_config defaults
SubplanConfig storage	3	Sequential/parallel config, standalone defaults
Hierarchy helpers	3	has_subplans, depth for non-root plans
Lifecycle transitions	2	Child plans follow same phase transition rules
Dependency guardrails	5	FailureHandler stop/retry decisions, phase independence
CLI dict rendering	3	parent_plan_id, subplan_count in output
Validation guardrails	4	Invalid ULIDs, out-of-range config values

Step definitions: features/steps/subplan_model_steps.py

All step names are prefixed with subplan-test to avoid AmbiguousStep conflicts with existing plan_model_steps.py steps.

Robot: Subplan Model (robot/subplan_model.robot)

11 smoke tests exercising subplan model properties via Python helper scripts:

Test Case	Description
Root Plan Identity Flags	is_root_plan=True, is_subplan=False, depth=0
Child Plan Identity Flags	is_subplan=True, is_root_plan=False, depth=-1
Three Level Hierarchy Root Propagation	root_plan_id propagates through 3 levels
SubplanConfig Defaults	All default values are sensible
SubplanConfig Custom Settings	Parallel mode with custom retry settings
Failure Handler Stop Others On Fail Fast	should_stop_others with fail_fast=True
Failure Handler Retry Retriable Error	should_retry for TimeoutError
Failure Handler Skip Non Retriable Error	should_retry skips ConfigurationError
CLI Dict Renders Parent Plan Id For Child	parent_plan_id in child plan CLI dict
CLI Dict Renders Subplan Count	subplan_count in parent plan CLI dict
Plan With Subplan Config And Statuses	Full plan model with config + statuses

Helper script: robot/helper_subplan_model.py

ASV Benchmarks (benchmarks/subplan_model_validation_bench.py)

Performance baselines for subplan model operations:

• SubplanConfigValidationSuite -- time_default_config_creation, time_custom_config_creation, time_config_model_dump, time_config_model_dump_json
• SubplanStatusValidationSuite -- time_minimal_status_creation, time_full_status_creation, time_status_model_dump
• SubplanFailureHandlerSuite -- time_should_stop_others_fail_fast, time_should_stop_others_no_fail_fast, time_should_retry_retriable, time_should_retry_non_retriable
• SubplanHierarchySuite -- time_is_root_plan, time_is_subplan, time_depth_root, time_depth_child, time_has_subplans, time_cli_dict_with_subplans, time_child_plan_creation, time_plan_with_subplans_creation

Running Subplan Model Tests

# Behave only (subplan model feature)
nox -s unit_tests -- features/subplan_model.feature

# Robot only (subplan model suite)
nox -s integration_tests -- --suite robot/subplan_model.robot

# Benchmarks
nox -s benchmark

CI Pipeline

The Forgejo CI pipeline runs these jobs in order:

1. lint — nox -s lint (ruff check)
2. typecheck — nox -s typecheck (pyright)
3. security — nox -s security_scan (bandit + vulture)
4. quality — nox -s complexity (radon)
5. unit_tests — nox -s unit_tests (behave)
6. integration_tests — nox -s integration_tests (robot)
7. coverage — nox -s coverage_report (fail-under 97%)
8. build — nox -s build (wheel)

All jobs use Python 3.13 and route commands through nox. See docs/development/ci-cd.md for full CI/CD documentation.

Troubleshooting

Coverage report shows 0% or very low coverage

Ensure you're running in serial mode (not parallel). The coverage_report nox session handles this correctly. Do not run behave directly outside of coverage run.

Also ensure:

• parallel = false is set in [tool.coverage.run]
• COVERAGE_FILE and COVERAGE_RCFILE env vars are set correctly by the nox session

AmbiguousStep errors in Behave

Two step files define the same @given/@when/@then pattern. Behave loads all step files globally. Make step names unique or use more specific patterns.

Robot Resource file does not exist

Use Resource ${CURDIR}/filename.resource instead of bare Resource filename.resource. Robot resolves bare paths relative to CWD, not the test file directory.

Robot No module named cleveragents

Ensure Robot tests use ${PYTHON} (injected by nox from the venv) instead of bare python in Run Process calls.

Tests hang indefinitely

Add timeout parameters to Run Process calls in Robot tests:

${result}=    Run Process    ${PYTHON}    -m    cleveragents    ...    timeout=30s