cleveragents-core/docs/development/testing.md

# Testing Guide

## Overview

CleverAgents uses a comprehensive testing strategy with three complementary frameworks:

- **Behave** (BDD/Gherkin) for unit-level and scenario tests under `features/`
- **Robot Framework** for integration and end-to-end tests under `robot/`
- **ASV (airspeed velocity)** for performance benchmarks under `benchmarks/`
- **Coverage**: `coverage.py` with branch coverage, enforced at **>=97%**

All tests are executed exclusively through `nox` sessions. Never invoke `behave`, `robot`, or other test runners directly.

## Running Tests

```bash
# Run all default sessions (lint, typecheck, unit tests, integration tests, coverage)
nox

# Unit tests only (Behave)
nox -s unit_tests

# Integration tests only (Robot Framework)
nox -s integration_tests

# Coverage report with 97% enforcement
nox -s coverage_report

# Run a specific feature file
nox -s unit_tests -- features/plan_model.feature

# Benchmarks (ASV)
nox -s benchmark

# Type checking (pyright)
nox -s typecheck

# Linting (ruff)
nox -s lint

# Formatting check
nox -s format -- --check
```

## Coverage Requirements

### Threshold: 97%

The project enforces a **minimum 97% code coverage** at all times. This is a hard gate:

- The `nox -s coverage_report` session fails if coverage drops below 97%.
- The Forgejo CI `coverage` job runs `nox -s coverage_report` and blocks merges on failure.
- Branch coverage is enabled (`branch = true` in `pyproject.toml`).

### How Coverage Is Measured

Coverage is measured by running Behave tests under `coverage.py` in serial mode:

```
coverage run --source=src -m behave -q --tags=-discovery --no-capture features/
coverage report --show-missing --fail-under=97
```

### Coverage Output

On **success**, the nox session emits:

```
COVERAGE OK: 97.2% (threshold: 97%)
```

On **failure**, the nox session emits and exits non-zero:

```
COVERAGE FAILED: 95.3% < 97% threshold
```

Both messages are single-line and CI-parseable. The CI pipeline greps for these lines to surface them in job summaries.

### Coverage Reports

Three report formats are generated under `build/`:

| Report | Path | Description |
|--------|------|-------------|
| Terminal | stdout | Per-file summary with `--show-missing` |
| HTML | `build/htmlcov/index.html` | Interactive browser report |
| XML | `build/coverage.xml` | Cobertura-format for CI tools |
| JSON | `build/coverage.json` | Machine-readable totals and per-file data |

### Coverage Configuration

Coverage settings live in `pyproject.toml`:

```toml
[tool.coverage.run]
source = ["src", "scripts"]
branch = true
parallel = false
omit = [
    "*/tests/*",
    "*/test_*",
    "features/*",
    "*/features/*",
    "*/__pycache__/*",
    "*/site-packages/*",
    "*/dependency_injector/*",
    "*/venv/*",
    "*/.venv/*",
    "*/.nox/*",
    "src/cleveragents/discovery/*",
]
data_file = "build/.coverage"

[tool.coverage.html]
directory = "build/htmlcov"

[tool.coverage.xml]
output = "build/coverage.xml"
```

The `--fail-under=97` threshold is enforced in the `coverage_report` nox session (`noxfile.py`).

### How to Improve Coverage

1. Run `nox -s coverage_report` to generate the HTML report.
2. Open `build/htmlcov/index.html` in a browser to identify uncovered files.
3. Sort by "Missing" column to find files with the most uncovered lines.
4. Write Behave scenarios targeting the uncovered code paths.
5. Re-run `nox -s coverage_report` to verify improvement.

**Do not** lower the threshold, add `# pragma: no cover` comments, or expand the `omit` list without explicit team agreement.

All test files go under `features/` (Behave) or `robot/` (Robot Framework). Never create a `tests/` directory.

## Unit Tests (Behave)

Unit tests use the [Behave](https://behave.readthedocs.io/) BDD framework with Gherkin-syntax `.feature` files.

### Structure

```
features/
  *.feature          # Feature files (Gherkin scenarios)
  steps/             # Step definitions (Python)
  mocks/             # Mock implementations (test doubles)
  fixtures/          # Test fixture data
  environment.py     # Behave hooks (before/after scenario, etc.)
```

### Guidelines

- **No pytest**: All unit tests must be Behave-based. There is intentionally no `tests/` directory.
- **Step naming**: Use unique, descriptive step names. Behave loads all step files globally; ambiguous steps cause `AmbiguousStep` errors.
- **Feature naming**: Name feature-specific step files after their feature (e.g., `foo.feature` -> `foo_steps.py`).
- **Mocks in features/ only**: All mock implementations live in `features/mocks/`. Production code (`src/`) must never contain test doubles.

### Running Specific Features

```bash
# Run a single feature file
nox -s unit_tests -- features/plan_model.feature

# Run scenarios with a specific tag
nox -s unit_tests -- --tags=@smoke
```

## Integration Tests (Robot Framework)

Integration tests use [Robot Framework](https://robotframework.org/) for end-to-end and system-level testing.

### Structure

```
robot/
  *.robot            # Test suites
  *.resource         # Shared keywords and variables
  common.resource    # Common setup/teardown keywords
  v2_paths.resource  # Path configuration
```

### Guidelines

- **Resource paths**: Always use `${CURDIR}/` prefix for `Resource` imports (e.g., `Resource ${CURDIR}/common.resource`).
- **Python injection**: Use `${PYTHON}` variable (injected by nox) instead of bare `python` in `Run Process` calls.
- **Timeouts**: Always add `timeout=` to `Run Process` calls that invoke long-running commands.
- **Slow tests**: Tag tests that require external services (API keys, running servers) with `slow`. These are excluded in CI via `--exclude slow`.

### Running Specific Suites

```bash
# Run a single Robot suite
nox -s integration_tests -- --suite robot/plan_generation_graph.robot

# Include slow tests
nox -s slow_integration_tests
```

## Performance Benchmarks (ASV)

[ASV (airspeed velocity)](https://asv.readthedocs.io/) tracks performance over time.

### Structure

```
benchmarks/
  *.py               # Benchmark suites (Time*/Mem*/Track* classes)
asv.conf.json        # ASV configuration
```

### Running Benchmarks

```bash
nox -s benchmark
```

### Writing Effective Tests

#### Behave Test Guidelines

```gherkin
Feature: Plan lifecycle transitions

  Scenario: Execute plan transitions phase from strategize to execute
    Given an action "local/build-app" exists
    And a plan is created using "local/build-app" on project "local/my-project"
    And strategize is completed
    When I execute the plan
    Then the plan phase should be "execute"
    And the processing state should be "queued"
```

#### Robot Framework Guidelines

```robot
*** Settings ***
Documentation     Plan lifecycle integration tests
Library           Process
Library           OperatingSystem
Resource          ${CURDIR}/common.resource

*** Test Cases ***
Plan Execute Transitions Correctly
    [Documentation]    Verify execute phase transition via CLI
    ${result}=    Run Process    ${PYTHON}    -m    cleveragents    plan    execute    ${PLAN_ID}
    Should Be Equal As Integers    ${result.rc}    0
    Should Contain    ${result.stdout}    execute
```

## CLI Lifecycle Test Suites

The CLI lifecycle suites provide comprehensive coverage for action and plan lifecycle
commands, including success paths, error paths, multi-project arguments, automation
profile overrides, invariants, actor overrides, phase visibility, and terminal
outcomes.

### Behave Suite: `features/cli_lifecycle_coverage.feature`

Covers 68 scenarios across these areas:

| Area | Scenarios | Description |
|------|-----------|-------------|
| Action CRUD | 13 | create, list, show, archive with success + error paths |
| Plan Use | 10 | basic use, multi-project, automation profile, invariants, actor overrides |
| Plan Status | 9 | phase visibility (strategize, execute, apply) and terminal outcomes (applied, constrained, errored, cancelled) |
| Plan Execute | 5 | execute success, phase transition, error paths |
| Plan Apply | 5 | lifecycle-apply success, terminal outcomes, error paths |
| Plan Cancel | 5 | cancel with reason, already-terminal guard, error paths |
| Plan List | 4 | lifecycle-list with and without filters |
| Negative cases | 17 | missing config, invalid args, invalid project names, unknown actions/resources |

Step definitions: `features/steps/cli_lifecycle_coverage_steps.py`

All step names are prefixed with `lifecycle coverage` to avoid `AmbiguousStep`
conflicts with existing steps.

### Robot Suite: `robot/cli_lifecycle_e2e.robot`

End-to-end integration tests exercising the full action-to-apply lifecycle through
the CLI entry points. Uses `robot/helper_cli_lifecycle_e2e.py` as a Python helper
that mocks the service layer while exercising real CLI argument parsing and output
formatting.

| Test Case | Description |
|-----------|-------------|
| Action Create From Config Via CLI | Creates an action from YAML config |
| Plan Use Creates Plan In Strategize Phase | Uses an action to create a plan |
| Plan Execute Transitions To Execute Phase | Executes a plan, checks phase |
| Plan Lifecycle Apply Transitions To Apply Phase | Applies a plan, checks phase |
| Plan Status Shows Plan Details | Verifies status output rendering |
| Plan Cancel Cancels Non-Terminal Plan | Cancels with a reason string |
| Full Lifecycle Action To Apply | End-to-end: create -> use -> execute -> apply |
| Plan Lifecycle List Shows Plans | Verifies lifecycle-list output |

### ASV Benchmark: `benchmarks/plan_cli_smoke_bench.py`

Three benchmark suites measuring CLI argument parsing overhead (service layer is
mocked so only parsing + routing cost is measured):

- **`ActionCLIParsingSuite`** -- `time_action_create`, `time_action_list`,
  `time_action_list_with_filters`, `time_action_show`, `time_action_archive`
- **`PlanUseArgParsingSuite`** -- `time_use_minimal`, `time_use_multi_project`,
  `time_use_with_all_flags`, `time_use_with_args_only`
- **`PlanLifecycleCmdParsingSuite`** -- `time_execute`, `time_lifecycle_apply`,
  `time_status_by_id`, `time_status_list_all`, `time_cancel_with_reason`,
  `time_lifecycle_list`, `time_lifecycle_list_filtered`

### Running the CLI Lifecycle Suites

```bash
# Behave only (our feature)
nox -s unit_tests -- features/cli_lifecycle_coverage.feature

# Robot only (our suite)
nox -s integration_tests -- --suite robot/cli_lifecycle_e2e.robot

# Benchmarks
nox -s benchmark
```

### Robot Suite: `robot/cli_lifecycle.robot`

End-to-end Robot suite for the CLI lifecycle with git_worktree sandbox simulation
and ChangeSet capture verification. Uses `robot/helper_cli_lifecycle.py` as a
Python helper that exercises the full action → plan → execute → apply flow with
mocked service layer, simulated sandbox workspace, and `InMemoryChangeSetStore`
for change tracking.

**Positive tests:**

| Test Case | Description |
|-----------|-------------|
| Action Create Via Config CLI | Creates an action from YAML config |
| Plan Use Creates Strategize Plan With Sandbox | Uses action with sandbox workspace mock |
| Plan Execute With ChangeSet Capture | Executes plan, verifies changeset entry recorded |
| Plan Apply With ChangeSet Verification | Applies plan, verifies changeset summary |
| Full Lifecycle Action To Apply With Sandbox | End-to-end with sandbox + changeset |

**Negative tests:**

| Test Case | Description |
|-----------|-------------|
| Invalid Project Name Rejected | Invalid project name format handled gracefully |
| Missing Resource Returns Error | Non-existent action returns error |
| Invalid Arg Format Rejected | `--arg` without `=` separator rejected |

**Fixtures:**

- `helper_cli_lifecycle.py` creates temporary YAML config files and sandbox directories
- Uses `InMemoryChangeSetStore` for ChangeSet capture without database
- Each subcommand is self-contained; cleanup runs in `finally` blocks

**Aligned Behave feature:** `features/cli_lifecycle_robot_alignment.feature`
mirrors the Robot E2E flow to keep unit/integration expectations aligned.

**ASV benchmark:** `benchmarks/cli_robot_flow_bench.py` measures fixture setup
overhead (`CLIRobotFixtureSetupSuite`) and full lifecycle CLI throughput
(`CLIRobotLifecycleSuite`).

### Running the Robot CLI Lifecycle Suite

```bash
# Robot suite
nox -s integration_tests -- --suite robot/cli_lifecycle.robot

# Aligned Behave feature
nox -s unit_tests -- features/cli_lifecycle_robot_alignment.feature

# Benchmarks
nox -s benchmark
```

## Persistence Test Suites

The persistence layer (SQLAlchemy repositories for plans and actions) has dedicated test suites at both unit and integration levels.

### Behave: Plan Persistence (`features/plan_persistence.feature`)

19 scenarios covering `LifecyclePlanRepository` operations:

- **Create**: Persist plans via repository, retrieve by ULID, verify all fields round-trip.
- **Phase/state transitions**: Update `phase` and `processing_state`, verify persistence after each transition.
- **List filters**: Filter plans by phase, processing state, and action name.
- **Plan tree links**: Parent/child/root hierarchy with `parent_plan_id` and `root_plan_id`.
- **Cross-restart**: Close the SQLite session, reopen from disk, verify plans survive reconnection (including project links, arguments, and invariants).

Step definitions: `features/steps/plan_persistence_steps.py`

**Fixtures**: Each scenario gets a fresh temp SQLite database via `_setup_db()` / `_teardown_db()` helper functions. The database is file-backed (not `:memory:`) so cross-restart scenarios can close and reopen the connection.

### Behave: Action Persistence (`features/action_persistence.feature`)

14 scenarios covering `ActionRepository` operations:

- **CRUD**: Create, retrieve by `namespaced_name`, list available, filter by namespace, archive, delete.
- **Argument ordering**: Persist 3 `ActionArgument` entries and verify positional order is preserved.
- **Invariant ordering**: Persist 3 invariant strings and verify order is preserved.
- **Terminal state storage**: Create plans from actions in Apply phase with each terminal `ProcessingState` (`applied`, `constrained`, `errored`, `cancelled`) and verify persistence.

Step definitions: `features/steps/action_persistence_steps.py`

### Robot: Plan Persistence E2E (`robot/plan_persistence_e2e.robot`)

5 end-to-end tests exercising the persistence layer through Python helper scripts:

| Test Case | Description |
|-----------|-------------|
| Plan Full Lifecycle Persistence | Create action + plan, transition through all phases to `applied` terminal state |
| Plan Restart Persistence | Create plan, close DB, reopen, verify all fields survive |
| Plan Concurrent Session Access | Two independent sessions against the same DB file |
| Action CRUD Persistence E2E | Create, read, update, delete an action with arguments/invariants |
| Plan Tree Hierarchy Persistence E2E | Parent/child plan hierarchy with `root_plan_id` verification |

Helper script: `robot/helper_plan_persistence_e2e.py`

### Robot: Persistence Lifecycle (`robot/persistence_lifecycle.robot`)

7 end-to-end tests exercising persistence lifecycle patterns through a Python helper:

| Test Case | Description |
|-----------|-------------|
| Plan Full Lifecycle Phase Transitions | Create action + plan, transition through all phases to `applied` terminal state |
| Process Restart Simulation | Write plan, close DB, reopen from disk, verify all fields survive |
| Reopen Plan Status After Restart | Close/reopen, verify phase, state, automation level, and tags unchanged |
| Concurrent CLI Access Safeguards | Two independent sessions see each other's committed data |
| Stored Arguments Ordering Persistence | 4 ordered `ActionArgument` entries survive roundtrip |
| Stored Invariants Ordering Persistence | 3 ordered invariant strings survive roundtrip |
| Project Links Persistence Through Restart | 2 `ProjectLink` entries survive close/reopen cycle |

Helper script: `robot/helper_persistence_lifecycle.py`

### Behave: Persistence Robot Alignment (`features/persistence_robot_alignment.feature`)

1 scenario mirroring the Robot restart-simulation flow to keep unit and integration persistence expectations aligned. Step definitions in `features/steps/persistence_robot_alignment_steps.py`.

### ASV Benchmarks: Robot Fixture Overhead (`benchmarks/persistence_robot_bench.py`)

Performance baselines for Robot fixture setup operations:

- `TimeRobotFixtureSetupSuite.time_file_db_init` -- File-backed SQLite init + schema creation
- `TimeRobotFixtureSetupSuite.time_seed_action` -- Action seeding into a fresh file DB
- `TimeRobotFixtureSetupSuite.time_seed_plan` -- Plan creation including FK action seeding
- `TimeRobotRestartCycleSuite.time_close_reopen_verify` -- Close + reopen + get cycle latency

### ASV Benchmarks (`benchmarks/persistence_suites_bench.py`)

Performance baselines for persistence test operations:

- `TimePlanPersistenceSuites.time_create_and_retrieve_plan` -- Plan round-trip latency
- `TimePlanPersistenceSuites.time_phase_state_transition` -- Phase/state update cycle
- `TimePlanPersistenceSuites.time_list_plans_filtered` -- List with filters
- `TimePlanPersistenceSuites.time_plan_tree_operations` -- Parent/child hierarchy
- `TimeActionPersistenceSuites.time_create_and_retrieve_action` -- Action round-trip
- `TimeActionPersistenceSuites.time_action_arguments_ordering` -- Ordered arguments persistence
- `TimeActionPersistenceSuites.time_cross_restart_persistence` -- Close/reopen cycle

Run with: `nox -s benchmark`

### ASV Benchmarks (`benchmarks/decision_persistence_serialization_bench.py`)

Performance baselines for decision persistence (serialization round-trip) operations:

- `DecisionDumpRoundTripSuite.time_root_dump` -- Root decision model_dump
- `DecisionDumpRoundTripSuite.time_root_validate` -- Root decision model_validate
- `DecisionDumpRoundTripSuite.time_root_roundtrip` -- Root decision full round-trip
- `DecisionDumpRoundTripSuite.time_child_roundtrip` -- Child decision full round-trip
- `DecisionDumpRoundTripSuite.time_full_roundtrip` -- Fully-populated decision round-trip
- `DecisionJsonRoundTripSuite.time_root_dump_json` -- Root decision model_dump_json
- `DecisionJsonRoundTripSuite.time_root_json_roundtrip` -- Root decision JSON round-trip
- `DecisionJsonRoundTripSuite.time_full_json_roundtrip` -- Full decision JSON round-trip
- `DecisionJsonRoundTripSuite.time_root_json_validate` -- Root decision model_validate from JSON
- `DecisionJsonRoundTripSuite.time_full_json_validate` -- Full decision model_validate from JSON
- `DecisionSnapshotPersistenceSuite.time_empty_snapshot_roundtrip` -- Empty snapshot round-trip
- `DecisionSnapshotPersistenceSuite.time_small_snapshot_roundtrip` -- Small snapshot round-trip
- `DecisionSnapshotPersistenceSuite.time_large_snapshot_roundtrip` -- Large snapshot (50 resources) round-trip
- `DecisionTreeSerializationSuite.time_small_tree_roundtrip` -- 3-node tree round-trip
- `DecisionTreeSerializationSuite.time_medium_tree_roundtrip` -- 7-node tree round-trip
- `DecisionTreeSerializationSuite.time_large_tree_roundtrip` -- 15-node tree round-trip
- `DecisionTreeSerializationSuite.time_large_tree_json_roundtrip` -- 15-node tree JSON round-trip

Run with: `nox -s benchmark`

### Behave: Decision Persistence Serialization (`features/decision_persistence_serialization.feature`)

21 scenarios covering Decision serialization round-trips, context-snapshot
persistence, correction-chain reconstruction, and decision-tree
reconstruction from serialized data:

- **CRUD round-trips**: Root and child decisions survive `model_dump` / `model_validate`.
- **JSON round-trips**: Decisions with artifacts survive `model_dump_json` / `model_validate`.
- **Snapshot persistence**: Empty snapshots, snapshots with resources, and actor state refs.
- **Correction chains**: Correction metadata, superseded_by, and 3-decision chain reconstruction.
- **Tree reconstruction**: 3-level tree (7 nodes) with parent-link integrity verification.
- **Edge cases**: Empty alternatives, confidence boundaries (0.0, 1.0, None), all types, long rationale.
- **Downstream IDs**: `downstream_decision_ids` and `downstream_plan_ids` survive round-trip.

Step definitions: `features/steps/decision_persistence_serialization_steps.py`

### Robot Framework: Decision Persistence Serialization (`robot/decision_persistence_serialization.robot`)

7 integration smoke tests exercising decision persistence round-trips via
a helper script (`robot/helper_decision_persistence_serialization.py`):

- Root decision round-trip
- Child decision round-trip
- Context snapshot persistence
- JSON serialization round-trip
- Correction chain persistence
- Decision tree reconstruction
- All decision types round-trip

### Behave: Skill Registry Persistence (`features/skill_registry.feature`)

23 scenarios covering `SkillRepository` and `SkillRegistryService` operations:

- **Register**: Persist skills with each item type (`tool_ref`, `include`, `inline_tool`, `mcp_source`, `agent_source`), verify round-trip.
- **Duplicate rejection**: Second registration with the same name raises `DuplicateSkillError`.
- **Retrieve / list**: Get by name, list all, list with namespace filter.
- **Update / remove**: Change description, remove with cascade, non-existent skill error paths.
- **Overrides round-trip**: Override metadata survives persistence.
- **Ordering stability**: Mixed item types maintain stable `item_order` across round-trips.
- **Name validation**: Empty, no-slash, and special-character names rejected at persistence layer.
- **Large payload**: 50 tool refs persist correctly.
- **Domain objects**: Listed results are `Skill` domain instances.

Step definitions: `features/steps/skill_registry_steps.py`

**Fixtures**: Each scenario gets a fresh in-memory SQLite database. A single
shared `Session` is reused across all repository calls within each scenario via a
lambda session factory (`lambda: shared_session`). This mirrors the production
`UnitOfWork` pattern (ADR-019) and avoids transaction-scoping issues that arise
when multiple ephemeral sessions share one in-memory SQLite connection
(`StaticPool`): abandoned sessions can be garbage-collected at unpredictable
times, rolling back previously flushed but uncommitted data.

### Robot: Skill Registry (`robot/skill_registry.robot`)

5 smoke tests exercising the registry through a Python helper script:

| Test Case | Description |
|-----------|-------------|
| Register And Retrieve A Skill | Round-trip: register then get by name |
| List Skills With Namespace Filter | Register multiple skills, filter by namespace |
| Update A Skill | Change description after registration |
| Reject Duplicate Skill Name | Duplicate name produces error |
| Remove A Skill | Remove and verify absence |

Helper script: `robot/helper_skill_registry.py`

### ASV Benchmarks: Skill Registry (`benchmarks/skill_registry_bench.py`)

Performance baselines for skill registry persistence operations:

- `SkillModelConstruction.time_from_domain` -- Domain-to-ORM mapping
- `SkillModelReconstruction.time_to_domain` -- ORM-to-domain reconstruction
- `SkillRepositoryCRUD.time_create_skill` -- Single skill insert + commit
- `SkillRepositoryCRUD.time_get_skill` -- Lookup by name
- `SkillRepositoryCRUD.time_list_all` -- List all skills
- `SkillRepositoryCRUD.time_list_namespace` -- List with namespace filter
- `SkillRegistryListPerformance.time_list_100_skills` -- List with 100 seeded skills
- `SkillRegistryListPerformance.time_list_100_skills_filtered` -- Filtered list with 100 skills

### Running Persistence Tests

```bash
# Unit tests (Behave) -- runs all features including persistence
nox -s unit_tests

# Run only plan persistence scenarios
nox -s unit_tests -- features/plan_persistence.feature

# Run only action persistence scenarios
nox -s unit_tests -- features/action_persistence.feature

# Run only skill registry persistence scenarios
nox -s unit_tests -- features/skill_registry.feature

# Integration tests (Robot) -- runs all robot suites including persistence E2E
nox -s integration_tests

# Run only plan persistence E2E
python -m robot robot/plan_persistence_e2e.robot

# Run only persistence lifecycle E2E
python -m robot robot/persistence_lifecycle.robot

# Run only skill registry smoke tests
nox -s integration_tests -- robot/skill_registry.robot

# Run only persistence robot alignment (Behave)
nox -s unit_tests -- features/persistence_robot_alignment.feature
```

## Subplan Model Test Suites

The subplan model (hierarchy, configuration, failure handling) has dedicated test
suites at unit, integration, and benchmark levels.

### Behave: Subplan Model (`features/subplan_model.feature`)

33 scenarios covering Plan hierarchy properties, SubplanConfig defaults, SubplanStatus
tracking, SubplanFailureHandler decisions, and CLI dict rendering:

| Area | Scenarios | Description |
|------|-----------|-------------|
| Root plan identity | 2 | `is_root_plan`, `is_subplan`, `depth` for root plans |
| Child plan identity | 3 | Parent reference, attempt counter, root propagation |
| Default values | 4 | Phase, state, automation level, subplan_config defaults |
| SubplanConfig storage | 3 | Sequential/parallel config, standalone defaults |
| Hierarchy helpers | 3 | `has_subplans`, `depth` for non-root plans |
| Lifecycle transitions | 2 | Child plans follow same phase transition rules |
| Dependency guardrails | 5 | FailureHandler stop/retry decisions, phase independence |
| CLI dict rendering | 3 | `parent_plan_id`, `subplan_count` in output |
| Validation guardrails | 4 | Invalid ULIDs, out-of-range config values |

Step definitions: `features/steps/subplan_model_steps.py`

All step names are prefixed with `subplan-test` to avoid `AmbiguousStep` conflicts
with existing `plan_model_steps.py` steps.

### Robot: Subplan Model (`robot/subplan_model.robot`)

11 smoke tests exercising subplan model properties via Python helper scripts:

| Test Case | Description |
|-----------|-------------|
| Root Plan Identity Flags | `is_root_plan=True`, `is_subplan=False`, `depth=0` |
| Child Plan Identity Flags | `is_subplan=True`, `is_root_plan=False`, `depth=-1` |
| Three Level Hierarchy Root Propagation | `root_plan_id` propagates through 3 levels |
| SubplanConfig Defaults | All default values are sensible |
| SubplanConfig Custom Settings | Parallel mode with custom retry settings |
| Failure Handler Stop Others On Fail Fast | `should_stop_others` with `fail_fast=True` |
| Failure Handler Retry Retriable Error | `should_retry` for `TimeoutError` |
| Failure Handler Skip Non Retriable Error | `should_retry` skips `ConfigurationError` |
| CLI Dict Renders Parent Plan Id For Child | `parent_plan_id` in child plan CLI dict |
| CLI Dict Renders Subplan Count | `subplan_count` in parent plan CLI dict |
| Plan With Subplan Config And Statuses | Full plan model with config + statuses |

Helper script: `robot/helper_subplan_model.py`

### ASV Benchmarks (`benchmarks/subplan_model_validation_bench.py`)

Performance baselines for subplan model operations:

- **`SubplanConfigValidationSuite`** -- `time_default_config_creation`,
  `time_custom_config_creation`, `time_config_model_dump`, `time_config_model_dump_json`
- **`SubplanStatusValidationSuite`** -- `time_minimal_status_creation`,
  `time_full_status_creation`, `time_status_model_dump`
- **`SubplanFailureHandlerSuite`** -- `time_should_stop_others_fail_fast`,
  `time_should_stop_others_no_fail_fast`, `time_should_retry_retriable`,
  `time_should_retry_non_retriable`
- **`SubplanHierarchySuite`** -- `time_is_root_plan`, `time_is_subplan`,
  `time_depth_root`, `time_depth_child`, `time_has_subplans`,
  `time_cli_dict_with_subplans`, `time_child_plan_creation`,
  `time_plan_with_subplans_creation`

### Running Subplan Model Tests

```bash
# Behave only (subplan model feature)
nox -s unit_tests -- features/subplan_model.feature

# Robot only (subplan model suite)
nox -s integration_tests -- --suite robot/subplan_model.robot

# Benchmarks
nox -s benchmark
```

## CI Pipeline

The Forgejo CI pipeline runs these jobs in order:

1. **lint** — `nox -s lint` (ruff check)
2. **typecheck** — `nox -s typecheck` (pyright)
3. **security** — `nox -s security_scan` (bandit + vulture)
4. **quality** — `nox -s complexity` (radon)
5. **unit_tests** — `nox -s unit_tests` (behave)
6. **integration_tests** — `nox -s integration_tests` (robot)
7. **coverage** — `nox -s coverage_report` (fail-under 97%)
8. **build** — `nox -s build` (wheel)

All jobs use Python 3.13 and route commands through `nox`. See `docs/development/ci-cd.md` for full CI/CD documentation.

## Troubleshooting

### Coverage report shows 0% or very low coverage

Ensure you're running in serial mode (not parallel). The `coverage_report` nox session handles this correctly. Do not run `behave` directly outside of `coverage run`.

Also ensure:
- `parallel = false` is set in `[tool.coverage.run]`
- `COVERAGE_FILE` and `COVERAGE_RCFILE` env vars are set correctly by the nox session

### AmbiguousStep errors in Behave

Two step files define the same `@given`/`@when`/`@then` pattern. Behave loads all step files globally. Make step names unique or use more specific patterns.

### Robot `Resource file does not exist`

Use `Resource ${CURDIR}/filename.resource` instead of bare `Resource filename.resource`. Robot resolves bare paths relative to CWD, not the test file directory.

### Robot `No module named cleveragents`

Ensure Robot tests use `${PYTHON}` (injected by nox from the venv) instead of bare `python` in `Run Process` calls.

### In-memory SQLite tests lose data between repository calls

When writing Behave tests that use `sqlite:///:memory:` with a `sessionmaker`,
do **not** create a new `Session` per repository call. SQLite in-memory databases
use `StaticPool` (a single shared connection). If each repository method creates
its own session, the abandoned sessions may be garbage-collected at unpredictable
times, triggering a `ROLLBACK` that discards previously flushed data.

**Fix:** Create a single shared `Session` at scenario setup and pass a factory
that always returns it:

```python
real_factory = sessionmaker(bind=engine)
shared_session = real_factory()
repo = MyRepository(session_factory=lambda: shared_session)
```

This mirrors the production `UnitOfWork` pattern (ADR-019) where all
repositories share one session within a transaction. See
`features/steps/skill_registry_steps.py` for a working example.

### Tests hang indefinitely

Add `timeout` parameters to `Run Process` calls in Robot tests:

```robot
${result}=    Run Process    ${PYTHON}    -m    cleveragents    ...    timeout=30s
```

## Validation Edge Case Test Catalog

The validation edge case suites exercise boundary conditions, error paths, and
conflict detection across the validation subsystem.

### Shared Fixtures (`features/fixtures/validation/`)

| Fixture File | Description |
|---|---|
| `malformed_tool_output.json` | Missing required fields, wrong types, extra nulls, non-object outputs |
| `missing_resources.json` | Dangling references, unresolvable paths, circular dependencies |
| `validation_timeouts.json` | Slow validators, zero/negative timeouts, partial results |
| `invalid_schema_transforms.json` | Transforms returning non-dict, missing type, null, circular refs |
| `mixed_ordering.json` | Required vs informational ordering, duplicate attachment IDs |

Each fixture file is a JSON document with a `"fixtures"` array. Each entry has
`"name"`, `"description"`, `"input"`, and `"expected_error"` fields.

### Behave Suite: `features/validation_edge_cases.feature`

25 scenarios covering:

| Area | Scenarios | Description |
|---|---|---|
| Malformed tool output | 6 | Missing fields, wrong types, nulls, empty/string outputs |
| Missing resources | 5 | Dangling actions, unresolvable paths, missing actors, circular deps |
| Validation timeouts | 4 | Slow validators, zero timeout, negative timeout, partial results |
| Invalid schema transforms | 3 | Non-dict returns, missing type field, null schema |
| Mixed ordering | 4 | Required-before-informational, duplicate attachment IDs, unknown levels |
| Concurrent conflicts | 1 | Two validators on same resource detect conflict |
| Rollback on failure | 2 | Required failure triggers rollback of pending changes |

Step definitions: `features/steps/validation_edge_case_steps.py`

All step names are prefixed with `validation edge` to avoid `AmbiguousStep`
conflicts with existing steps.

### Robot Suite: `robot/validation_edge.robot`

6 integration tests exercising the validation edge case helper:

| Test Case | Description |
|---|---|
| Validation Edge Load All Fixtures | Loads all fixture files and verifies structure |
| Validation Edge Malformed Output Detection | Detects malformed tool output patterns |
| Validation Edge Missing Resource Error Paths | Validates missing resource reference detection |
| Validation Edge Timeout Simulation | Validates timeout simulation fixture data |
| Validation Edge Schema Validation Errors | Validates schema transform error detection |
| Validation Edge Unknown Command Returns Error | Verifies unknown command exits with code 1 |

Helper script: `robot/helper_validation_edge.py`

### ASV Benchmarks: `benchmarks/validation_edge_bench.py`

Two benchmark suites:

- **`ValidationEdgeFixtureLoadSuite`** -- `time_load_malformed_tool_output`,
  `time_load_missing_resources`, `time_load_validation_timeouts`,
  `time_load_invalid_schema_transforms`, `time_load_mixed_ordering`,
  `time_load_all_fixtures`
- **`ValidationEdgeDetectionSuite`** -- `time_malformed_output_detection`,
  `time_missing_resource_detection`, `time_validation_ordering`,
  `time_combined_edge_detection`

### Running the Validation Edge Case Suites

```bash
# Behave only (validation edge cases)
nox -s unit_tests -- features/validation_edge_cases.feature

# Robot only (validation edge suite)
nox -s integration_tests -- --suite robot/validation_edge.robot

# Benchmarks
nox -s benchmark
```

## Scale Testing

Scale tests validate performance baselines for repository indexing and
decomposition across different repository sizes (1K, 5K, and 10K files).

### Fixtures

Scale test fixtures live in `features/fixtures/scale/`:

- `scale_metadata.json` — Scale profile definitions (file count, language mix, expected timings)
- `baseline_thresholds.json` — Performance threshold matrices (p50/p95/p99 per operation per file count)
- `generator_instructions.md` — Manual instructions for generating synthetic repos

### Running Scale Tests

```bash
# Behave scale test scenarios
nox -s unit_tests -- features/scale_test.feature

# Robot scale test integration
nox -s integration_tests -- --suite robot/scale_test.robot

# ASV scale fixture benchmarks
nox -s benchmark
```

### Scale Test Suites

| Suite | Framework | Scenarios | Description |
|-------|-----------|-----------|-------------|
| `features/scale_test.feature` | Behave | 20 | Fixture loading, metadata validation, threshold checks, distribution simulation |
| `robot/scale_test.robot` | Robot | 6 | Integration-level fixture validation via helper script |
| `benchmarks/scale_fixture_bench.py` | ASV | 6 | Performance benchmarks for fixture processing |

For full scale testing documentation, see `docs/development/scale_testing.md`.

## CLI Extension Test Fixtures

The CLI extension tests cover automation profile, invariant, and actor override
functionality added in #325. Deeper coverage added in #326 validates resolution
logic, ordering semantics, error cases, and output format snapshots.

### Test Fixture Patterns

The CLI extension test suites use a **mocked lifecycle service** pattern. The
`_get_lifecycle_service` function in both `plan.py` and `action.py` is patched
so that tests exercise real CLI argument parsing, validation, and output
rendering without requiring a database or running server.

#### Common Fixtures

| Fixture | Module | Description |
|---------|--------|-------------|
| `_make_plan()` | `cli_extensions_steps.py` | Creates a `Plan` instance with configurable automation profile, invariants, and actor overrides |
| `_make_action()` | `cli_extensions_steps.py` | Creates an `Action` with configurable optional fields (estimation_actor, invariant_actor, inputs_schema, automation_profile) |
| `CliRunner` | `typer.testing` | Invokes CLI commands in-process without subprocess overhead |
| `MagicMock` service | `unittest.mock` | Mocks `PlanLifecycleService` for plan/action operations |

#### Automation Profile Resolution Fixtures

Tests validate that each builtin profile name (`manual`, `review`, `supervised`,
`cautious`, `trusted`, `auto`) resolves correctly through the CLI and that
invalid profile names are rejected with a helpful error listing available
profiles. The service mock returns a `Plan` with an `AutomationProfileRef` to
verify end-to-end profile assignment.

#### Invariant Ordering Fixtures

Tests pass multiple `--invariant` flags and verify the service receives them in
exact insertion order. This tests the list-accumulation behaviour of Typer's
repeated option handling.

#### Actor Override Error Fixtures

Tests exercise the `validate_namespaced_actor` regex
(`^[a-z][a-z0-9-]*/[a-z][a-z0-9._-]*$`) with various invalid inputs:
- Empty string
- Missing slash (`no-slash`)
- Uppercase characters (`UPPER/case`)
- Leading slash (`/leading-slash`)
- Trailing slash (`trail/ing/`)
- Double slashes (`open//ai`)
- Special characters (`open@ai/gpt-4`)
- Numeric-leading namespace (`123/model`)
- Whitespace (`open ai/gpt`)

#### Output Snapshot Assertions

Tests render plan status, lifecycle-list, and action show output in JSON, YAML,
and table formats, then assert that:
- JSON output is valid (parses without error)
- Required keys are present (`namespaced_name`, `phase`, `automation_profile`, etc.)
- Invariant text values survive serialization round-trips
- YAML output contains expected field names
- Table output contains human-readable field values

### Behave Suite: `features/cli_extensions.feature`

Covers scenarios across these areas:

| Area | Scenarios | Description |
|------|-----------|-------------|
| Automation profile flags | 2 | Valid and invalid profile names |
| Profile resolution (deep) | 9 | All builtin profiles + error cases with special chars, spaces, empty |
| Invariant flags | 2 | Single and dual invariant flags |
| Invariant ordering (deep) | 2 | Three and five invariant insertion-order preservation |
| Actor overrides (valid) | 4 | Strategy, execution, estimation, invariant actors |
| Actor overrides (invalid) | 4 | Malformed namespace/name formats |
| Actor error cases (deep) | 11 | Empty, double-slash, special chars, numeric-leading, whitespace |
| Plan status display | 4 | Table, JSON, single plan with invariants/profile |
| Plan lifecycle-list | 2 | Table and JSON with invariant count |
| Action show | 8 | Optional actors, invariants, inputs_schema, automation profile |
| Combined flags | 1 | Profile + invariant together |
| validate_namespaced_actor | 2 | Unit-level accept/reject assertions |
| Output snapshots (deep) | 9 | JSON/YAML/table format validation for plan and action |

Step definitions: `features/steps/cli_extensions_steps.py`

### Robot Suite: `robot/cli_extensions.robot`

Integration tests exercising CLI extensions through Python helper scripts:

| Test Case | Description |
|-----------|-------------|
| Plan Use With Invariants | Verifies `--invariant` flags pass through correctly |
| Plan Use With Automation Profile | Verifies `--automation-profile` works |
| Plan Use Actor Validation Valid | Valid namespaced actor format accepted |
| Plan Use Actor Validation Invalid | Invalid actor format rejected |
| Plan Use Combined Profile And Invariants | Both profile and invariants together |
| Action Show With Optional Actors And Invariants | `action show` renders estimation_actor, invariant_actor, invariants, inputs_schema |
| Action Show JSON With Optional Fields | JSON output includes all optional fields |
| Invariant Ordering Preserved | Multiple invariant flags preserve insertion order |
| Profile Resolution All Builtins | All builtin profile names resolve correctly |

Helper script: `robot/helper_cli_extensions.py`

### ASV Benchmarks

Two benchmark files cover CLI extension performance:

**`benchmarks/cli_extensions_bench.py`** (original):
- `PlanUseWithProfileSuite` — plan use with automation profile
- `PlanUseWithInvariantsSuite` — plan use with invariant flags
- `PlanUseActorValidationSuite` — plan use with actor overrides
- `PlanStatusWithExtendedFieldsSuite` — status/lifecycle-list rendering
- `ActionShowExtendedSuite` — action show in rich and JSON formats

**`benchmarks/cli_extension_tests_bench.py`** (extended #326):
- `ProfileResolutionSuite` — resolve all builtin profiles, single profile, invalid profile
- `InvariantOrderingSuite` — three and five invariant flag parsing
- `ActorValidationErrorSuite` — valid/invalid actor regex, CLI rejection
- `OutputFormatRenderingSuite` — JSON/YAML/table rendering for action show and plan status

### Running CLI Extension Tests

```bash
# Behave only (CLI extensions feature)
nox -s unit_tests -- features/cli_extensions.feature

# Robot only (CLI extensions suite)
nox -s integration_tests -- --suite robot/cli_extensions.robot

# Benchmarks
nox -s benchmark
```

## M2 Actor + Tool Source Smoke Suite

The M2 smoke suite validates the foundation for the Actor Graphs + Tool Sources epic (#356). It covers actor YAML loading, skill registry, tool lifecycle, and MCP stub tool discovery.

### Fixtures

M2 fixtures live in `features/fixtures/m2/`:

| Fixture File | Description |
|---|---|
| `m2_hierarchical_actor.yaml` | Graph-type actor with planner → executor → reviewer nodes |
| `m2_skill_pack.yaml` | Skill pack with tool references and an inline tool |

### MCP Stub Server

The MCP stub server (`features/mocks/mcp_stub_server.py`) simulates MCP tool discovery and invocation in-process without requiring the real MCP adapter (#159). It exposes three deterministic tools: `mcp/search`, `mcp/fetch`, and `mcp/transform`.

Usage in tests:

```python
from features.mocks.mcp_stub_server import McpStubServer

server = McpStubServer()
server.start()
tools = server.discover()       # Returns 3 stub tools
result = server.invoke("mcp/search", {"query": "hello"})
server.stop()
```

### Test Suites

| Suite | Framework | Scenarios | Description |
|-------|-----------|-----------|-------------|
| `features/m2_actor_tool_smoke.feature` | Behave | 10 | Actor loading, skill registry, tool lifecycle, MCP stub |
| `robot/m2_actor_tool_smoke.robot` | Robot | 6 | CLI-level smoke for actor/skill/tool/MCP operations |
| `benchmarks/m2_actor_tool_smoke_bench.py` | ASV | 12 | Baseline runtime for actor, skill, tool, and MCP operations |

### Running the M2 Smoke Suite

```bash
# Behave only (M2 feature)
nox -s unit_tests -- features/m2_actor_tool_smoke.feature

# Robot only (M2 suite)
nox -s integration_tests -- --suite robot/m2_actor_tool_smoke.robot

# Benchmarks
nox -s benchmark
```

### What Is NOT Tested (deferred to dependent issues)

- Real MCP adapter integration (#159, assigned to Aditya)
- Agent skills loader (#160, assigned to Aditya)
- Hierarchical actor compiler (#158, assigned to Jeff)

The MCP stub server provides a test seam that can be replaced with the real adapter once #159 lands.

## M1 Source-Code Plan Lifecycle Smoke Tests

The M1 source-code smoke suites verify the minimal end-to-end source-code
workflow: a git repo fixture, a `git-checkout` resource config, and a minimal
action YAML with strategy/execution actors. Helper steps create temp projects,
link resources, and capture plan IDs for subsequent CLI verification.

### Fixtures (`features/fixtures/m1/`)

| Fixture File | Description |
|---|---|
| `git_repo.json` | Minimal git repo definitions with file listings and branch info |
| `git_checkout_resource.json` | Git-checkout resource configs with path and branch properties |
| `action_sourcecode.yaml` | Minimal action YAML with strategy/execution actors and arguments |

### Behave Suite: `features/m1_sourcecode_smoke.feature`

16 scenarios covering:

| Area | Scenarios | Description |
|------|-----------|-------------|
| Fixture loading | 3 | Load and validate git repo, checkout resource, action YAML |
| Action create | 1 | Create action from YAML config via CLI |
| Project/resource | 2 | Create temp project, link resource |
| Plan use | 3 | Use action to create plan, with project and args |
| Plan execute | 1 | Execute transitions to execute phase |
| Plan diff | 1 | Show changeset via diff |
| Plan apply | 1 | Apply transitions to terminal state |
| Negative cases | 3 | Unknown action, wrong phase, invalid actor |

Step definitions: `features/steps/m1_sourcecode_smoke_steps.py`

All step names are prefixed with `m1 smoke` to avoid `AmbiguousStep`
conflicts with existing steps.

### Robot Suite: `robot/m1_sourcecode_smoke.robot`

8 integration tests exercising the M1 lifecycle through CLI:

| Test Case | Description |
|-----------|-------------|
| M1 Action Create From Config | Creates action from fixture YAML |
| M1 Plan Use Creates Strategize Plan | Uses action to create plan |
| M1 Plan Use With Project Link | Plan use with project argument |
| M1 Plan Execute Transitions Phase | Execute phase transition |
| M1 Plan Diff Shows Changeset | Diff command for changeset |
| M1 Plan Apply Reaches Terminal | Apply to terminal state |
| M1 Full Lifecycle Action To Apply | End-to-end flow |
| M1 Plan Use With Plain Format | `--format plain` for stable assertions |

Helper script: `robot/helper_m1_sourcecode_smoke.py`

### ASV Benchmarks: `benchmarks/m1_sourcecode_smoke_bench.py`

Six benchmark suites measuring M1 CLI overhead:

- **`M1ActionCreateSuite`** -- `time_action_create_from_yaml`
- **`M1PlanUseSuite`** -- `time_plan_use_minimal`, `time_plan_use_with_project`,
  `time_plan_use_with_args`, `time_plan_use_plain_format`
- **`M1PlanExecuteSuite`** -- `time_plan_execute`
- **`M1PlanApplySuite`** -- `time_plan_lifecycle_apply`
- **`M1FixtureLoadSuite`** -- `time_load_git_repo_fixture`,
  `time_load_git_checkout_fixture`

### Running the M1 Source-Code Smoke Suites

```bash
# Behave only (M1 smoke feature)
nox -s unit_tests -- features/m1_sourcecode_smoke.feature

# Robot only (M1 smoke suite)
nox -s integration_tests -- --suite robot/m1_sourcecode_smoke.robot

# Benchmarks
nox -s benchmark
```

### M1 Smoke Run Instructions

To perform a quick M1 source-code smoke run:

1. Run the Behave feature:
   ```bash
   nox -s unit_tests -- features/m1_sourcecode_smoke.feature
   ```
2. Run the Robot suite:
   ```bash
   nox -s integration_tests -- --suite robot/m1_sourcecode_smoke.robot
   ```
3. Run benchmarks:
   ```bash
   nox -s benchmark
   ```

### Failure Triage Tips

- **`AmbiguousStep` errors**: All M1 smoke steps are prefixed with `m1 smoke`.
  If ambiguous, check that no other step file defines a conflicting pattern.
- **Fixture file not found**: Verify `features/fixtures/m1/` contains all three
  fixture files (`git_repo.json`, `git_checkout_resource.json`,
  `action_sourcecode.yaml`).
- **Robot `FAIL` sentinel**: Each Robot helper subcommand prints a detailed
  `FAIL:` line with exit code and output when something goes wrong. Check the
  Robot log for these messages.
- **`InvalidPhaseTransitionError`**: Verify the plan is in the correct phase
  before attempting a transition. The M1 smoke tests mock phase state carefully.
- **Coverage drops**: The M1 smoke tests cover fixture loading, CLI argument
  parsing, and mock service integration. If coverage drops, look at
  `build/htmlcov/index.html` for uncovered lines in plan/action CLI commands.

## M3 Decision Tree, Validation Gating, and Invariant Enforcement Smoke Tests

### Overview

The M3 smoke suite validates the decision tree, correction workflow,
validation gating (add / attach / detach), and invariant constraint
(add / list / remove) subsystems via CLI integration tests.

Issue: [#179](https://git.cleverthis.com/cleveragents/cleveragents-core/issues/179)

### Files

| File | Purpose |
|------|------|
| `features/m3_decision_validation_smoke.feature` | Behave BDD scenarios (25 scenarios) |
| `features/steps/m3_decision_validation_smoke_steps.py` | Step implementations |
| `features/fixtures/m3/decision_tree_outputs.json` | Decision tree fixture data |
| `features/fixtures/m3/validation_attachments.json` | Validation attachment fixture data |
| `features/fixtures/m3/invariant_configs.json` | Invariant config fixture data |
| `robot/m3_decision_validation_smoke.robot` | Robot Framework integration tests (8 cases) |
| `robot/helper_m3_decision_validation_smoke.py` | Robot helper with 8 subcommands |
| `benchmarks/m3_smoke_bench.py` | ASV benchmark suites (5 classes) |

### Scenario Categories

1. **Decision tree fixture loading** -- Loads JSON fixtures and verifies
   decision tree structure, invariant nodes, and correction metadata.
2. **Validation fixture loading** -- Verifies validation attachment data
   including required and informational modes.
3. **Invariant fixture loading** -- Verifies invariant config data
   including merge set precedence.
4. **Invariant CLI** -- Tests `agents invariant add`, `list`, and `remove`
   commands with global and project scopes.
5. **Validation CLI** -- Tests `agents validation add`, `attach`, and
   `detach` commands.
6. **Plan correct CLI** -- Tests `agents plan correct` with `--dry-run`,
   `--mode revert`, and `--mode append`.
7. **Negative cases** -- Empty guidance, invalid correction mode.

### Mock Strategy

- **Invariant CLI** patches `cleveragents.cli.commands.invariant._get_service`
- **Validation CLI** patches `cleveragents.cli.commands.validation._get_tool_registry_service`
- **Plan CLI** patches `cleveragents.cli.commands.plan._get_lifecycle_service`
- **Correction** patches `CorrectionService` constructor at module level

### Failure Triage Tips

- **`AmbiguousStep` errors**: All M3 smoke steps are prefixed with `m3 smoke`.
  If ambiguous, check that no other step file defines a conflicting pattern.
- **Fixture file not found**: Verify `features/fixtures/m3/` contains all
  three JSON fixture files.
- **Robot `FAIL` sentinel**: Each Robot helper subcommand prints a detailed
  `FAIL:` line with exit code and output. Check the Robot log.
- **Coverage drops**: The M3 smoke tests cover invariant/validation CLI
  commands, plan correct workflow, and fixture loading. Check
  `build/htmlcov/index.html` for uncovered lines.

## M4 Correction + Subplan Smoke Suite

The M4 smoke suite validates correction flows, subplan execution, and conflict
simulation for the M4 milestone. It covers decision tree corrections (revert and
append modes, dry-run impact analysis), subplan lifecycle (sequential, parallel,
dependency-ordered execution), failure handling (stop-others, retry logic), and
merge conflict simulation (git-three-way, fail-on-conflict, last-wins,
sequential-apply strategies).

### Fixtures (`features/fixtures/m4/`)

| Fixture File | Description |
|---|---|
| `correction_flows.json` | Revert, append, dry-run, and high-risk correction scenarios |
| `subplan_execution.json` | Sequential, parallel, dependency-ordered, and retry subplan configs |
| `conflict_simulations.json` | Merge conflict scenarios for all SubplanMergeStrategy variants |

### Behave Suite: `features/m4_correction_subplan_smoke.feature`

20 scenarios covering:

| Area | Scenarios | Description |
|------|-----------|-------------|
| Fixture loading | 3 | Load and validate correction, subplan, and conflict fixtures |
| Correction flows | 4 | Revert, append, dry-run, and invalid mode via CLI |
| Subplan execution | 2 | Sequential and parallel subplan status rendering |
| Failure handling | 3 | Stop-others, retry retriable, skip non-retriable |
| Conflict simulation | 3 | No-conflict, fail-on-conflict, last-wins merge strategies |
| Negative cases | 2 | Non-existent plan, empty decision ID |

Step definitions: `features/steps/m4_correction_subplan_smoke_steps.py`

All step names are prefixed with `m4 smoke` to avoid `AmbiguousStep`
conflicts with existing steps.

### Robot Suite: `robot/m4_correction_subplan_smoke.robot`

8 integration tests exercising M4 correction and subplan flows through CLI:

| Test Case | Description |
|-----------|-------------|
| M4 Correction Revert Via CLI | Invoke plan correct --mode revert |
| M4 Correction Append Via CLI | Invoke plan correct --mode append |
| M4 Correction Dry Run Via CLI | Invoke plan correct --dry-run |
| M4 Subplan Status Sequential | Plan status with sequential subplan config |
| M4 Subplan Status Parallel | Plan status with parallel subplan config |
| M4 Failure Handler Evaluation | SubplanFailureHandler decision logic |
| M4 Fixture Loading | Load all M4 fixture files |
| M4 Full Correction And Subplan Flow | End-to-end correction + subplan + failure handler |

Helper script: `robot/helper_m4_correction_subplan_smoke.py`

### ASV Benchmarks: `benchmarks/m4_smoke_bench.py`

Four benchmark suites measuring M4 suite runtime:

- **`M4CorrectionCLISuite`** -- `time_correction_revert`, `time_correction_append`,
  `time_correction_dry_run`
- **`M4SubplanStatusSuite`** -- `time_status_with_subplans_json`,
  `time_status_with_subplans_rich`
- **`M4FailureHandlerSuite`** -- `time_should_stop_others_fail_fast`,
  `time_should_retry_retriable`, `time_should_retry_non_retriable`
- **`M4FixtureLoadSuite`** -- `time_load_correction_flows`,
  `time_load_subplan_execution`, `time_load_conflict_simulations`,
  `time_load_all_fixtures`

### Running the M4 Correction + Subplan Smoke Suites

```bash
# Behave only (M4 smoke feature)
nox -s unit_tests -- features/m4_correction_subplan_smoke.feature

# Robot only (M4 smoke suite)
nox -s integration_tests -- --suite robot/m4_correction_subplan_smoke.robot

# Benchmarks
nox -s benchmark
```

### Failure Triage Tips

- **`AmbiguousStep` errors**: All M4 smoke steps are prefixed with `m4 smoke`.
  If ambiguous, check that no other step file defines a conflicting pattern.
- **Fixture file not found**: Verify `features/fixtures/m4/` contains all three
  fixture files (`correction_flows.json`, `subplan_execution.json`,
  `conflict_simulations.json`).
- **Robot `FAIL` sentinel**: Each Robot helper subcommand prints a detailed
  `FAIL:` line with exit code and output when something goes wrong.
- **Correction service mock**: The M4 steps patch `CorrectionService` directly
  in the plan CLI module. If the import path changes, update the patch target.
- **Coverage drops**: The M4 smoke tests cover correction CLI commands,
  SubplanFailureHandler logic, and fixture loading. Check
  `build/htmlcov/index.html` for uncovered lines in correction/plan CLI code.

---

## M5 ACMS Pipeline + Large-Project Context Smoke Tests

The M5 smoke suites verify the ACMS context pipeline foundation: context
assembly, budget enforcement, multi-project context, context analysis,
and project context policy resolution.

### Behave: `features/m5_acms_smoke.feature`

27 scenarios covering:

| Area | Scenarios |
|------|-----------|
| Fixture loading | Load ACMS context policy, large project context, analysis results |
| Context policy resolution | Default view, inheritance chain, strategize/execute/apply overrides |
| Budget enforcement | max_file_size, max_total_size, zero/negative validation, unlimited |
| Context assembly (CLI) | list, add, show, clear via mocked ContextService |
| Project context policy CLI | show policy, inspect/simulate stubs (NotImplementedError) |
| Context analysis agent | Summary, dependencies, relevance scores (mocked) |
| Multi-project context | Independent context per project |
| Exclusion patterns | Glob-based path filtering |

Step definitions: `features/steps/m5_acms_smoke_steps.py`
Fixtures: `features/fixtures/m5/`

- `acms_context_policy.json` -- Phase-specific context policy with size limits
- `large_project_context.json` -- Simulated 500-file project with tier metadata
- `context_analysis_results.json` -- Pre-computed analysis with dependencies and
  relevance scores

### Robot: `robot/m5_acms_smoke.robot`

| Test Case | Description |
|-----------|-------------|
| Load ACMS Context Policy Fixture | Validates fixture structure |
| Load Large Project Context Fixture | Validates file count and entries |
| Resolve Default View From Empty Policy | Empty policy includes all paths |
| Resolve Strategize Inherits From Default | Phase inheritance works |
| Resolve Strategize With Override | Override takes precedence |
| Budget Max File Size Enforcement | File-level budget check |
| Budget Max Total Size Enforcement | Aggregate budget check |
| Invalid Phase Raises Error | Error handling for invalid phases |
| Context Analysis Fixture Has Required Fields | Analysis result structure |
| Multi Project Independent Context | Independent context per project |

Helper script: `robot/helper_m5_acms_smoke.py`

### ASV Benchmarks: `benchmarks/m5_smoke_bench.py`

Four benchmark suites measuring M5 suite runtime:

- **`PolicyResolutionSuite`** -- `time_resolve_view` across all 4 phases
- **`BudgetEnforcementSuite`** -- `time_check_file_budget`,
  `time_check_total_budget` with 100/1000/10000 files
- **`FixtureLoadSuite`** -- `time_load_policy_fixture`,
  `time_load_large_project_fixture`, `time_load_analysis_fixture`,
  `time_load_all_fixtures`
- **`ContextViewConstructionSuite`** -- `time_construct_view`,
  `time_construct_full_policy` with 1/10/50 path patterns

### Running the M5 ACMS Smoke Suites

```bash
# Behave only (M5 smoke feature)
nox -s unit_tests -- features/m5_acms_smoke.feature

# Robot only (M5 smoke suite)
nox -s integration_tests -- --suite robot/m5_acms_smoke.robot

# Benchmarks
nox -s benchmark
```

### Failure Triage Tips

- **`AmbiguousStep` errors**: All M5 smoke steps are prefixed with `m5 smoke`.
  If ambiguous, check that no other step file defines a conflicting pattern.
- **Fixture file not found**: Verify `features/fixtures/m5/` contains all three
  fixture files (`acms_context_policy.json`, `large_project_context.json`,
  `context_analysis_results.json`).
- **`context_inspect`/`context_simulate` stubs**: These commands raise
  `NotImplementedError` by design until ACMS wiring is complete.
- **Coverage drops**: The M5 smoke tests cover context policy resolution, budget
  enforcement, and CLI context commands. Check `build/htmlcov/index.html` for
  uncovered lines in context-related modules.