# Testing Guide

## Overview

CleverAgents uses a comprehensive testing strategy with three complementary frameworks:

- **Behave** (BDD/Gherkin) for unit-level and scenario tests under `features/`
- **Robot Framework** for integration and end-to-end tests under `robot/`
- **ASV (airspeed velocity)** for performance benchmarks under `benchmarks/`
- **Coverage**: `coverage.py` with branch coverage, enforced at **>=97%**

All tests are executed exclusively through `nox` sessions. Never invoke `behave`, `robot`, or other test runners directly.

## Running Tests

```bash
# Run all default sessions (lint, typecheck, unit tests, integration tests, coverage)
nox

# Unit tests only (Behave)
nox -s unit_tests

# Integration tests only (Robot Framework)
nox -s integration_tests

# Coverage report with 97% enforcement
nox -s coverage_report

# Run a specific feature file
nox -s unit_tests -- features/plan_model.feature

# Benchmarks (ASV)
nox -s benchmark

# Type checking (pyright)
nox -s typecheck

# Linting (ruff)
nox -s lint

# Formatting check
nox -s format -- --check
```

## Coverage Requirements

### Threshold: 97%

The project enforces a **minimum 97% code coverage** at all times. This is a hard gate:

- The `nox -s coverage_report` session fails if coverage drops below 97%.
- The Forgejo CI `coverage` job runs `nox -s coverage_report` and blocks merges on failure.
- Branch coverage is enabled (`branch = true` in `pyproject.toml`).

### How Coverage Is Measured

Coverage is measured by running Behave tests under `slipcover` (bytecode-based instrumentation) in parallel via behave-parallel. Each worker produces a per-feature JSON coverage file, then slipcover merges them:

```
# Each behave-parallel worker runs:
python -m slipcover --json --out build/.slipcover.<uuid>.json --source src -m behave <feature>
# After all workers finish:
python -m slipcover --merge build/.slipcover.*.json --json --out build/coverage.json
python -m slipcover --merge build/coverage.json --fail-under=97
```

### Coverage Output

On **success**, the nox session emits:

```
COVERAGE OK: 97.2% (threshold: 97%)
```

On **failure**, the nox session emits and exits non-zero:

```
COVERAGE FAILED: 95.3% < 97% threshold
```

Both messages are single-line and CI-parseable. The CI pipeline greps for these lines to surface them in job summaries.

### Coverage Reports

Three report formats are generated under `build/`:

| Report   | Path                       | Description                               |
| -------- | -------------------------- | ----------------------------------------- |
| Terminal | stdout                     | Per-file summary with `--show-missing`    |
| HTML     | `build/htmlcov/index.html` | Interactive browser report                |
| XML      | `build/coverage.xml`       | Cobertura-format for CI tools             |
| JSON     | `build/coverage.json`      | Machine-readable totals and per-file data |

### Coverage Configuration

Coverage is collected by `slipcover` (bytecode-based, faster than `coverage.py`'s `sys.settrace`).
Source and omit patterns are passed directly to slipcover via the `coverage_report` nox session.
The `pyproject.toml` `[tool.coverage.run]` section is retained for any tools that still read it:

```toml
[tool.coverage.run]
source = ["src", "scripts"]
branch = true
parallel = false
omit = [
    "*/tests/*",
    "*/test_*",
    "features/*",
    "*/features/*",
    "*/__pycache__/*",
    "*/site-packages/*",
    "*/dependency_injector/*",
    "*/venv/*",
    "*/.venv/*",
    "*/.nox/*",
    "src/cleveragents/discovery/*",
]
data_file = "build/.coverage"

[tool.coverage.html]
directory = "build/htmlcov"

[tool.coverage.xml]
output = "build/coverage.xml"
```

The `--fail-under=97` threshold is enforced in the `coverage_report` nox session (`noxfile.py`) via `slipcover --fail-under=97`.

### How to Improve Coverage

1. Run `nox -s coverage_report` to generate the HTML report.
2. Open `build/htmlcov/index.html` in a browser to identify uncovered files.
3. Sort by "Missing" column to find files with the most uncovered lines.
4. Write Behave scenarios targeting the uncovered code paths.
5. Re-run `nox -s coverage_report` to verify improvement.

**Do not** lower the threshold, add `# pragma: no cover` comments, or expand the `omit` list without explicit team agreement.

All test files go under `features/` (Behave) or `robot/` (Robot Framework). Never create a `tests/` directory.

## Unit Tests (Behave)

Unit tests use the [Behave](https://behave.readthedocs.io/) BDD framework with Gherkin-syntax `.feature` files.

### Structure

```
features/
  *.feature          # Feature files (Gherkin scenarios)
  steps/             # Step definitions (Python)
  mocks/             # Mock implementations (test doubles)
  fixtures/          # Test fixture data
  environment.py     # Behave hooks (before/after scenario, etc.)
```

### Guidelines

- **No pytest**: All unit tests must be Behave-based. There is intentionally no `tests/` directory.
- **Step naming**: Use unique, descriptive step names. Behave loads all step files globally; ambiguous steps cause `AmbiguousStep` errors.
- **Feature naming**: Name feature-specific step files after their feature (e.g., `foo.feature` -> `foo_steps.py`).
- **Mocks in features/ only**: All mock implementations live in `features/mocks/`. Production code (`src/`) must never contain test doubles.

### Running Specific Features

```bash
# Run a single feature file
nox -s unit_tests -- features/plan_model.feature

# Run scenarios with a specific tag
nox -s unit_tests -- --tags=@smoke
```

## Integration Tests (Robot Framework)

Integration tests use [Robot Framework](https://robotframework.org/) for end-to-end and system-level testing.

### Structure

```
robot/
  *.robot            # Test suites
  *.resource         # Shared keywords and variables
  common.resource    # Common setup/teardown keywords
  v2_paths.resource  # Path configuration
```

### Guidelines

- **Resource paths**: Always use `${CURDIR}/` prefix for `Resource` imports (e.g., `Resource ${CURDIR}/common.resource`).
- **Python injection**: Use `${PYTHON}` variable (injected by nox) instead of bare `python` in `Run Process` calls.
- **Timeouts**: Always add `timeout=` to `Run Process` calls that invoke long-running commands.
- **Slow tests**: Tag tests that require external services (API keys, running servers) with `slow`. These are excluded in CI via `--exclude slow`.

### Parallel Execution Isolation (pabot)

Robot integration tests run in parallel via `pabot` (CPU-count workers by default).
To prevent race conditions, `common.resource` provides per-suite isolation:

1. **Per-suite CLEVERAGENTS_HOME** — Each suite gets a unique temp directory
   (`${TEMPDIR}/.cleveragents_<SuiteName>`) so configuration and data files
   never collide.

2. **Per-suite database URL** — `CLEVERAGENTS_DATABASE_URL` and
   `CLEVERAGENTS_TEST_DATABASE_URL` point to SQLite files inside the
   per-suite home directory.  Without this, all pabot workers would fall
   back to the default `sqlite:///cleveragents.db` (relative to CWD),
   causing write contention.

3. **Singleton reset in helpers** — Robot helpers that chain multiple CLI
   invocations within a single process (e.g. `full_flow()`) must call
   `reset_global_state()` between invocations.  The reset logic is
   centralised in `robot/helpers_common.py`; each helper imports and
   delegates to `helpers_common.reset_global_state()`.  This function
   resets:
   - `Settings._instance` (singleton config)
   - `reset_container()` (DI container)
   - `reset_provider_registry()` (provider registry)
   - `MEMORY_ENGINES` (SQLAlchemy engine cache)

4. **Environment variable cleanup** — `Cleanup Test Environment` removes
   all environment variables set during setup (`CLEVERAGENTS_HOME`,
   `CLEVERAGENTS_AUTO_APPLY_MIGRATIONS`, `CLEVERAGENTS_TESTING_USE_MOCK_AI`,
   `CLEVERAGENTS_DATABASE_URL`, and `CLEVERAGENTS_TEST_DATABASE_URL`) after
   each suite to prevent leakage into subsequent suites.

#### Adding a new Robot helper

When writing a new Robot helper that invokes `CliRunner` or otherwise
exercises the application layer, import the shared reset function from
`helpers_common.py`:

```python
from helpers_common import reset_global_state

# In the __main__ dispatcher:
if __name__ == "__main__":
    reset_global_state()
    fn = _COMMANDS[sys.argv[1]]
    fn()
```

Call `reset_global_state()` once at the start of the `__main__` block and
again between chained CLI invocations within the same process.  Do **not**
redefine the reset logic locally — always delegate to `helpers_common`.

### Running Specific Suites

```bash
# Run a single Robot suite
nox -s integration_tests -- --suite robot/plan_generation_graph.robot

# Include slow tests
nox -s slow_integration_tests
```

## Performance Benchmarks (ASV)

[ASV (airspeed velocity)](https://asv.readthedocs.io/) tracks performance over time.

### Structure

```
benchmarks/
  *.py               # Benchmark suites (Time*/Mem*/Track* classes)
asv.conf.json        # ASV configuration
```

### Running Benchmarks

```bash
nox -s benchmark
```

### Writing Effective Tests

#### Behave Test Guidelines

```gherkin
Feature: Plan lifecycle transitions

  Scenario: Execute plan transitions phase from strategize to execute
    Given an action "local/build-app" exists
    And a plan is created using "local/build-app" on project "local/my-project"
    And strategize is completed
    When I execute the plan
    Then the plan phase should be "execute"
    And the processing state should be "queued"
```

#### Robot Framework Guidelines

```robot
*** Settings ***
Documentation     Plan lifecycle integration tests
Library           Process
Library           OperatingSystem
Resource          ${CURDIR}/common.resource

*** Test Cases ***
Plan Execute Transitions Correctly
    [Documentation]    Verify execute phase transition via CLI
    ${result}=    Run Process    ${PYTHON}    -m    cleveragents    plan    execute    ${PLAN_ID}
    Should Be Equal As Integers    ${result.rc}    0
    Should Contain    ${result.stdout}    execute
```

## CLI Lifecycle Test Suites

The CLI lifecycle suites provide comprehensive coverage for action and plan lifecycle
commands, including success paths, error paths, multi-project arguments, automation
profile overrides, invariants, actor overrides, phase visibility, and terminal
outcomes.

### Behave Suite: `features/cli_lifecycle_coverage.feature`

Covers 68 scenarios across these areas:

| Area           | Scenarios | Description                                                                                                    |
| -------------- | --------- | -------------------------------------------------------------------------------------------------------------- |
| Action CRUD    | 13        | create, list, show, archive with success + error paths                                                         |
| Plan Use       | 10        | basic use, multi-project, automation profile, invariants, actor overrides                                      |
| Plan Status    | 9         | phase visibility (strategize, execute, apply) and terminal outcomes (applied, constrained, errored, cancelled) |
| Plan Execute   | 5         | execute success, phase transition, error paths                                                                 |
| Plan Apply     | 5         | apply success, terminal outcomes, error paths                                                                  |
| Plan Cancel    | 5         | cancel with reason, already-terminal guard, error paths                                                        |
| Plan List      | 4         | list with and without filters                                                                                  |
| Negative cases | 17        | missing config, invalid args, invalid project names, unknown actions/resources                                 |

Step definitions: `features/steps/cli_lifecycle_coverage_steps.py`

All step names are prefixed with `lifecycle coverage` to avoid `AmbiguousStep`
conflicts with existing steps.

### Robot Suite: `robot/cli_lifecycle_e2e.robot`

End-to-end integration tests exercising the full action-to-apply lifecycle through
the CLI entry points. Uses `robot/helper_cli_lifecycle_e2e.py` as a Python helper
that mocks the service layer while exercising real CLI argument parsing and output
formatting.

| Test Case                                       | Description                                   |
| ----------------------------------------------- | --------------------------------------------- |
| Action Create From Config Via CLI               | Creates an action from YAML config            |
| Plan Use Creates Plan In Strategize Phase       | Uses an action to create a plan               |
| Plan Execute Transitions To Execute Phase       | Executes a plan, checks phase                 |
| Plan Lifecycle Apply Transitions To Apply Phase | Applies a plan, checks phase                  |
| Plan Status Shows Plan Details                  | Verifies status output rendering              |
| Plan Cancel Cancels Non-Terminal Plan           | Cancels with a reason string                  |
| Full Lifecycle Action To Apply                  | End-to-end: create -> use -> execute -> apply |
| Plan List Shows Plans                           | Verifies list output                          |

### ASV Benchmark: `benchmarks/plan_cli_smoke_bench.py`

Three benchmark suites measuring CLI argument parsing overhead (service layer is
mocked so only parsing + routing cost is measured):

- **`ActionCLIParsingSuite`** -- `time_action_create`, `time_action_list`,
  `time_action_list_with_filters`, `time_action_show`, `time_action_archive`
- **`PlanUseArgParsingSuite`** -- `time_use_minimal`, `time_use_multi_project`,
  `time_use_with_all_flags`, `time_use_with_args_only`
- **`PlanLifecycleCmdParsingSuite`** -- `time_execute`, `time_lifecycle_apply`,
  `time_status_by_id`, `time_status_list_all`, `time_cancel_with_reason`,
  `time_lifecycle_list`, `time_lifecycle_list_filtered`

### Running the CLI Lifecycle Suites

```bash
# Behave only (our feature)
nox -s unit_tests -- features/cli_lifecycle_coverage.feature

# Robot only (our suite)
nox -s integration_tests -- --suite robot/cli_lifecycle_e2e.robot

# Benchmarks
nox -s benchmark
```

### Robot Suite: `robot/cli_lifecycle.robot`

End-to-end Robot suite for the CLI lifecycle with git_worktree sandbox simulation
and ChangeSet capture verification. Uses `robot/helper_cli_lifecycle.py` as a
Python helper that exercises the full action → plan → execute → apply flow with
mocked service layer, simulated sandbox workspace, and `InMemoryChangeSetStore`
for change tracking.

**Positive tests:**

| Test Case                                     | Description                                      |
| --------------------------------------------- | ------------------------------------------------ |
| Action Create Via Config CLI                  | Creates an action from YAML config               |
| Plan Use Creates Strategize Plan With Sandbox | Uses action with sandbox workspace mock          |
| Plan Execute With ChangeSet Capture           | Executes plan, verifies changeset entry recorded |
| Plan Apply With ChangeSet Verification        | Applies plan, verifies changeset summary         |
| Full Lifecycle Action To Apply With Sandbox   | End-to-end with sandbox + changeset              |

**Negative tests:**

| Test Case                      | Description                                    |
| ------------------------------ | ---------------------------------------------- |
| Invalid Project Name Rejected  | Invalid project name format handled gracefully |
| Missing Resource Returns Error | Non-existent action returns error              |
| Invalid Arg Format Rejected    | `--arg` without `=` separator rejected         |

**Fixtures:**

- `helper_cli_lifecycle.py` creates temporary YAML config files and sandbox directories
- Uses `InMemoryChangeSetStore` for ChangeSet capture without database
- Each subcommand is self-contained; cleanup runs in `finally` blocks

**Aligned Behave feature:** `features/cli_lifecycle_robot_alignment.feature`
mirrors the Robot E2E flow to keep unit/integration expectations aligned.

**ASV benchmark:** `benchmarks/cli_robot_flow_bench.py` measures fixture setup
overhead (`CLIRobotFixtureSetupSuite`) and full lifecycle CLI throughput
(`CLIRobotLifecycleSuite`).

### Running the Robot CLI Lifecycle Suite

```bash
# Robot suite
nox -s integration_tests -- --suite robot/cli_lifecycle.robot

# Aligned Behave feature
nox -s unit_tests -- features/cli_lifecycle_robot_alignment.feature

# Benchmarks
nox -s benchmark
```

## Persistence Test Suites

The persistence layer (SQLAlchemy repositories for plans and actions) has dedicated test suites at both unit and integration levels.

### Behave: Plan Persistence (`features/plan_persistence.feature`)

19 scenarios covering `LifecyclePlanRepository` operations:

- **Create**: Persist plans via repository, retrieve by ULID, verify all fields round-trip.
- **Phase/state transitions**: Update `phase` and `processing_state`, verify persistence after each transition.
- **List filters**: Filter plans by phase, processing state, and action name.
- **Plan tree links**: Parent/child/root hierarchy with `parent_plan_id` and `root_plan_id`.
- **Cross-restart**: Close the SQLite session, reopen from disk, verify plans survive reconnection (including project links, arguments, and invariants).

Step definitions: `features/steps/plan_persistence_steps.py`

**Fixtures**: Each scenario gets a fresh temp SQLite database via `_setup_db()` / `_teardown_db()` helper functions. The database is file-backed (not `:memory:`) so cross-restart scenarios can close and reopen the connection.

### Behave: Action Persistence (`features/action_persistence.feature`)

14 scenarios covering `ActionRepository` operations:

- **CRUD**: Create, retrieve by `namespaced_name`, list available, filter by namespace, archive, delete.
- **Argument ordering**: Persist 3 `ActionArgument` entries and verify positional order is preserved.
- **Invariant ordering**: Persist 3 invariant strings and verify order is preserved.
- **Terminal state storage**: Create plans from actions in Apply phase with each terminal `ProcessingState` (`applied`, `constrained`, `errored`, `cancelled`) and verify persistence.

Step definitions: `features/steps/action_persistence_steps.py`

### Robot: Plan Persistence E2E (`robot/plan_persistence_e2e.robot`)

5 end-to-end tests exercising the persistence layer through Python helper scripts:

| Test Case                           | Description                                                                     |
| ----------------------------------- | ------------------------------------------------------------------------------- |
| Plan Full Lifecycle Persistence     | Create action + plan, transition through all phases to `applied` terminal state |
| Plan Restart Persistence            | Create plan, close DB, reopen, verify all fields survive                        |
| Plan Concurrent Session Access      | Two independent sessions against the same DB file                               |
| Action CRUD Persistence E2E         | Create, read, update, delete an action with arguments/invariants                |
| Plan Tree Hierarchy Persistence E2E | Parent/child plan hierarchy with `root_plan_id` verification                    |

Helper script: `robot/helper_plan_persistence_e2e.py`

### Robot: Persistence Lifecycle (`robot/persistence_lifecycle.robot`)

7 end-to-end tests exercising persistence lifecycle patterns through a Python helper:

| Test Case                                 | Description                                                                     |
| ----------------------------------------- | ------------------------------------------------------------------------------- |
| Plan Full Lifecycle Phase Transitions     | Create action + plan, transition through all phases to `applied` terminal state |
| Process Restart Simulation                | Write plan, close DB, reopen from disk, verify all fields survive               |
| Reopen Plan Status After Restart          | Close/reopen, verify phase, state, automation level, and tags unchanged         |
| Concurrent CLI Access Safeguards          | Two independent sessions see each other's committed data                        |
| Stored Arguments Ordering Persistence     | 4 ordered `ActionArgument` entries survive roundtrip                            |
| Stored Invariants Ordering Persistence    | 3 ordered invariant strings survive roundtrip                                   |
| Project Links Persistence Through Restart | 2 `ProjectLink` entries survive close/reopen cycle                              |

Helper script: `robot/helper_persistence_lifecycle.py`

### Behave: Persistence Robot Alignment (`features/persistence_robot_alignment.feature`)

1 scenario mirroring the Robot restart-simulation flow to keep unit and integration persistence expectations aligned. Step definitions in `features/steps/persistence_robot_alignment_steps.py`.

### ASV Benchmarks: Robot Fixture Overhead (`benchmarks/persistence_robot_bench.py`)

Performance baselines for Robot fixture setup operations:

- `TimeRobotFixtureSetupSuite.time_file_db_init` -- File-backed SQLite init + schema creation
- `TimeRobotFixtureSetupSuite.time_seed_action` -- Action seeding into a fresh file DB
- `TimeRobotFixtureSetupSuite.time_seed_plan` -- Plan creation including FK action seeding
- `TimeRobotRestartCycleSuite.time_close_reopen_verify` -- Close + reopen + get cycle latency

### ASV Benchmarks (`benchmarks/persistence_suites_bench.py`)

Performance baselines for persistence test operations:

- `TimePlanPersistenceSuites.time_create_and_retrieve_plan` -- Plan round-trip latency
- `TimePlanPersistenceSuites.time_phase_state_transition` -- Phase/state update cycle
- `TimePlanPersistenceSuites.time_list_plans_filtered` -- List with filters
- `TimePlanPersistenceSuites.time_plan_tree_operations` -- Parent/child hierarchy
- `TimeActionPersistenceSuites.time_create_and_retrieve_action` -- Action round-trip
- `TimeActionPersistenceSuites.time_action_arguments_ordering` -- Ordered arguments persistence
- `TimeActionPersistenceSuites.time_cross_restart_persistence` -- Close/reopen cycle

Run with: `nox -s benchmark`

### ASV Benchmarks (`benchmarks/decision_persistence_serialization_bench.py`)

Performance baselines for decision persistence (serialization round-trip) operations:

- `DecisionDumpRoundTripSuite.time_root_dump` -- Root decision model_dump
- `DecisionDumpRoundTripSuite.time_root_validate` -- Root decision model_validate
- `DecisionDumpRoundTripSuite.time_root_roundtrip` -- Root decision full round-trip
- `DecisionDumpRoundTripSuite.time_child_roundtrip` -- Child decision full round-trip
- `DecisionDumpRoundTripSuite.time_full_roundtrip` -- Fully-populated decision round-trip
- `DecisionJsonRoundTripSuite.time_root_dump_json` -- Root decision model_dump_json
- `DecisionJsonRoundTripSuite.time_root_json_roundtrip` -- Root decision JSON round-trip
- `DecisionJsonRoundTripSuite.time_full_json_roundtrip` -- Full decision JSON round-trip
- `DecisionJsonRoundTripSuite.time_root_json_validate` -- Root decision model_validate from JSON
- `DecisionJsonRoundTripSuite.time_full_json_validate` -- Full decision model_validate from JSON
- `DecisionSnapshotPersistenceSuite.time_empty_snapshot_roundtrip` -- Empty snapshot round-trip
- `DecisionSnapshotPersistenceSuite.time_small_snapshot_roundtrip` -- Small snapshot round-trip
- `DecisionSnapshotPersistenceSuite.time_large_snapshot_roundtrip` -- Large snapshot (50 resources) round-trip
- `DecisionTreeSerializationSuite.time_small_tree_roundtrip` -- 3-node tree round-trip
- `DecisionTreeSerializationSuite.time_medium_tree_roundtrip` -- 7-node tree round-trip
- `DecisionTreeSerializationSuite.time_large_tree_roundtrip` -- 15-node tree round-trip
- `DecisionTreeSerializationSuite.time_large_tree_json_roundtrip` -- 15-node tree JSON round-trip

Run with: `nox -s benchmark`

### Behave: Decision Persistence Serialization (`features/decision_persistence_serialization.feature`)

21 scenarios covering Decision serialization round-trips, context-snapshot
persistence, correction-chain reconstruction, and decision-tree
reconstruction from serialized data:

- **CRUD round-trips**: Root and child decisions survive `model_dump` / `model_validate`.
- **JSON round-trips**: Decisions with artifacts survive `model_dump_json` / `model_validate`.
- **Snapshot persistence**: Empty snapshots, snapshots with resources, and actor state refs.
- **Correction chains**: Correction metadata, superseded_by, and 3-decision chain reconstruction.
- **Tree reconstruction**: 3-level tree (7 nodes) with parent-link integrity verification.
- **Edge cases**: Empty alternatives, confidence boundaries (0.0, 1.0, None), all types, long rationale.
- **Downstream IDs**: `downstream_decision_ids` and `downstream_plan_ids` survive round-trip.

Step definitions: `features/steps/decision_persistence_serialization_steps.py`

### Robot Framework: Decision Persistence Serialization (`robot/decision_persistence_serialization.robot`)

7 integration smoke tests exercising decision persistence round-trips via
a helper script (`robot/helper_decision_persistence_serialization.py`):

- Root decision round-trip
- Child decision round-trip
- Context snapshot persistence
- JSON serialization round-trip
- Correction chain persistence
- Decision tree reconstruction
- All decision types round-trip

### Behave: Skill Registry Persistence (`features/skill_registry.feature`)

23 scenarios covering `SkillRepository` and `SkillRegistryService` operations:

- **Register**: Persist skills with each item type (`tool_ref`, `include`, `inline_tool`, `mcp_source`, `agent_source`), verify round-trip.
- **Duplicate rejection**: Second registration with the same name raises `DuplicateSkillError`.
- **Retrieve / list**: Get by name, list all, list with namespace filter.
- **Update / remove**: Change description, remove with cascade, non-existent skill error paths.
- **Overrides round-trip**: Override metadata survives persistence.
- **Ordering stability**: Mixed item types maintain stable `item_order` across round-trips.
- **Name validation**: Empty, no-slash, and special-character names rejected at persistence layer.
- **Large payload**: 50 tool refs persist correctly.
- **Domain objects**: Listed results are `Skill` domain instances.

Step definitions: `features/steps/skill_registry_steps.py`

**Fixtures**: Each scenario gets a fresh in-memory SQLite database. A single
shared `Session` is reused across all repository calls within each scenario via a
lambda session factory (`lambda: shared_session`). This mirrors the production
`UnitOfWork` pattern (ADR-019) and avoids transaction-scoping issues that arise
when multiple ephemeral sessions share one in-memory SQLite connection
(`StaticPool`): abandoned sessions can be garbage-collected at unpredictable
times, rolling back previously flushed but uncommitted data.

### Robot: Skill Registry (`robot/skill_registry.robot`)

5 smoke tests exercising the registry through a Python helper script:

| Test Case                         | Description                                   |
| --------------------------------- | --------------------------------------------- |
| Register And Retrieve A Skill     | Round-trip: register then get by name         |
| List Skills With Namespace Filter | Register multiple skills, filter by namespace |
| Update A Skill                    | Change description after registration         |
| Reject Duplicate Skill Name       | Duplicate name produces error                 |
| Remove A Skill                    | Remove and verify absence                     |

Helper script: `robot/helper_skill_registry.py`

### ASV Benchmarks: Skill Registry (`benchmarks/skill_registry_bench.py`)

Performance baselines for skill registry persistence operations:

- `SkillModelConstruction.time_from_domain` -- Domain-to-ORM mapping
- `SkillModelReconstruction.time_to_domain` -- ORM-to-domain reconstruction
- `SkillRepositoryCRUD.time_create_skill` -- Single skill insert + commit
- `SkillRepositoryCRUD.time_get_skill` -- Lookup by name
- `SkillRepositoryCRUD.time_list_all` -- List all skills
- `SkillRepositoryCRUD.time_list_namespace` -- List with namespace filter
- `SkillRegistryListPerformance.time_list_100_skills` -- List with 100 seeded skills
- `SkillRegistryListPerformance.time_list_100_skills_filtered` -- Filtered list with 100 skills

### Running Persistence Tests

```bash
# Unit tests (Behave) -- runs all features including persistence
nox -s unit_tests

# Run only plan persistence scenarios
nox -s unit_tests -- features/plan_persistence.feature

# Run only action persistence scenarios
nox -s unit_tests -- features/action_persistence.feature

# Run only skill registry persistence scenarios
nox -s unit_tests -- features/skill_registry.feature

# Integration tests (Robot) -- runs all robot suites including persistence E2E
nox -s integration_tests

# Run only plan persistence E2E
python -m robot robot/plan_persistence_e2e.robot

# Run only persistence lifecycle E2E
python -m robot robot/persistence_lifecycle.robot

# Run only skill registry smoke tests
nox -s integration_tests -- robot/skill_registry.robot

# Run only persistence robot alignment (Behave)
nox -s unit_tests -- features/persistence_robot_alignment.feature
```

## Subplan Model Test Suites

The subplan model (hierarchy, configuration, failure handling) has dedicated test
suites at unit, integration, and benchmark levels.

### Behave: Subplan Model (`features/subplan_model.feature`)

33 scenarios covering Plan hierarchy properties, SubplanConfig defaults, SubplanStatus
tracking, SubplanFailureHandler decisions, and CLI dict rendering:

| Area                  | Scenarios | Description                                             |
| --------------------- | --------- | ------------------------------------------------------- |
| Root plan identity    | 2         | `is_root_plan`, `is_subplan`, `depth` for root plans    |
| Child plan identity   | 3         | Parent reference, attempt counter, root propagation     |
| Default values        | 4         | Phase, state, automation level, subplan_config defaults |
| SubplanConfig storage | 3         | Sequential/parallel config, standalone defaults         |
| Hierarchy helpers     | 3         | `has_subplans`, `depth` for non-root plans              |
| Lifecycle transitions | 2         | Child plans follow same phase transition rules          |
| Dependency guardrails | 5         | FailureHandler stop/retry decisions, phase independence |
| CLI dict rendering    | 3         | `parent_plan_id`, `subplan_count` in output             |
| Validation guardrails | 4         | Invalid ULIDs, out-of-range config values               |

Step definitions: `features/steps/subplan_model_steps.py`

All step names are prefixed with `subplan-test` to avoid `AmbiguousStep` conflicts
with existing `plan_model_steps.py` steps.

### Robot: Subplan Model (`robot/subplan_model.robot`)

11 smoke tests exercising subplan model properties via Python helper scripts:

| Test Case                                 | Description                                         |
| ----------------------------------------- | --------------------------------------------------- |
| Root Plan Identity Flags                  | `is_root_plan=True`, `is_subplan=False`, `depth=0`  |
| Child Plan Identity Flags                 | `is_subplan=True`, `is_root_plan=False`, `depth=-1` |
| Three Level Hierarchy Root Propagation    | `root_plan_id` propagates through 3 levels          |
| SubplanConfig Defaults                    | All default values are sensible                     |
| SubplanConfig Custom Settings             | Parallel mode with custom retry settings            |
| Failure Handler Stop Others On Fail Fast  | `should_stop_others` with `fail_fast=True`          |
| Failure Handler Retry Retriable Error     | `should_retry` for `TimeoutError`                   |
| Failure Handler Skip Non Retriable Error  | `should_retry` skips `ConfigurationError`           |
| CLI Dict Renders Parent Plan Id For Child | `parent_plan_id` in child plan CLI dict             |
| CLI Dict Renders Subplan Count            | `subplan_count` in parent plan CLI dict             |
| Plan With Subplan Config And Statuses     | Full plan model with config + statuses              |

Helper script: `robot/helper_subplan_model.py`

### ASV Benchmarks (`benchmarks/subplan_model_validation_bench.py`)

Performance baselines for subplan model operations:

- **`SubplanConfigValidationSuite`** -- `time_default_config_creation`,
  `time_custom_config_creation`, `time_config_model_dump`, `time_config_model_dump_json`
- **`SubplanStatusValidationSuite`** -- `time_minimal_status_creation`,
  `time_full_status_creation`, `time_status_model_dump`
- **`SubplanFailureHandlerSuite`** -- `time_should_stop_others_fail_fast`,
  `time_should_stop_others_no_fail_fast`, `time_should_retry_retriable`,
  `time_should_retry_non_retriable`
- **`SubplanHierarchySuite`** -- `time_is_root_plan`, `time_is_subplan`,
  `time_depth_root`, `time_depth_child`, `time_has_subplans`,
  `time_cli_dict_with_subplans`, `time_child_plan_creation`,
  `time_plan_with_subplans_creation`

### Running Subplan Model Tests

```bash
# Behave only (subplan model feature)
nox -s unit_tests -- features/subplan_model.feature

# Robot only (subplan model suite)
nox -s integration_tests -- --suite robot/subplan_model.robot

# Benchmarks
nox -s benchmark
```

## CI Pipeline

The Forgejo CI pipeline runs these jobs in order:

1. **lint** — `nox -s lint` (ruff check)
2. **typecheck** — `nox -s typecheck` (pyright)
3. **security** — `nox -s security_scan` (bandit + vulture)
4. **quality** — `nox -s complexity` (radon)
5. **unit_tests** — `nox -s unit_tests` (behave)
6. **integration_tests** — `nox -s integration_tests` (robot)
7. **coverage** — `nox -s coverage_report` (fail-under 97%)
8. **build** — `nox -s build` (wheel)

All jobs use Python 3.13 and route commands through `nox`. See `docs/development/ci-cd.md` for full CI/CD documentation.

## Troubleshooting

### Coverage report shows 0% or very low coverage

Ensure you're running in serial mode (not parallel). The `coverage_report` nox session handles this correctly. Do not run `behave` directly outside of `coverage run`.

Also ensure:

- `parallel = false` is set in `[tool.coverage.run]`
- `COVERAGE_FILE` and `COVERAGE_RCFILE` env vars are set correctly by the nox session

### AmbiguousStep errors in Behave

Two step files define the same `@given`/`@when`/`@then` pattern. Behave loads all step files globally. Make step names unique or use more specific patterns.

### Robot `Resource file does not exist`

Use `Resource ${CURDIR}/filename.resource` instead of bare `Resource filename.resource`. Robot resolves bare paths relative to CWD, not the test file directory.

### Robot `No module named cleveragents`

Ensure Robot tests use `${PYTHON}` (injected by nox from the venv) instead of bare `python` in `Run Process` calls.

### In-memory SQLite tests lose data between repository calls

When writing Behave tests that use `sqlite:///:memory:` with a `sessionmaker`,
do **not** create a new `Session` per repository call. SQLite in-memory databases
use `StaticPool` (a single shared connection). If each repository method creates
its own session, the abandoned sessions may be garbage-collected at unpredictable
times, triggering a `ROLLBACK` that discards previously flushed data.

**Fix:** Create a single shared `Session` at scenario setup and pass a factory
that always returns it:

```python
real_factory = sessionmaker(bind=engine)
shared_session = real_factory()
repo = MyRepository(session_factory=lambda: shared_session)
```

This mirrors the production `UnitOfWork` pattern (ADR-019) where all
repositories share one session within a transaction. See
`features/steps/skill_registry_steps.py` for a working example.

### Tests hang indefinitely

Add `timeout` parameters to `Run Process` calls in Robot tests:

```robot
${result}=    Run Process    ${PYTHON}    -m    cleveragents    ...    timeout=30s
```

## Validation Edge Case Test Catalog

The validation edge case suites exercise boundary conditions, error paths, and
conflict detection across the validation subsystem.

### Shared Fixtures (`features/fixtures/validation/`)

| Fixture File                     | Description                                                           |
| -------------------------------- | --------------------------------------------------------------------- |
| `malformed_tool_output.json`     | Missing required fields, wrong types, extra nulls, non-object outputs |
| `missing_resources.json`         | Dangling references, unresolvable paths, circular dependencies        |
| `validation_timeouts.json`       | Slow validators, zero/negative timeouts, partial results              |
| `invalid_schema_transforms.json` | Transforms returning non-dict, missing type, null, circular refs      |
| `mixed_ordering.json`            | Required vs informational ordering, duplicate attachment IDs          |

Each fixture file is a JSON document with a `"fixtures"` array. Each entry has
`"name"`, `"description"`, `"input"`, and `"expected_error"` fields.

### Behave Suite: `features/validation_edge_cases.feature`

25 scenarios covering:

| Area                      | Scenarios | Description                                                             |
| ------------------------- | --------- | ----------------------------------------------------------------------- |
| Malformed tool output     | 6         | Missing fields, wrong types, nulls, empty/string outputs                |
| Missing resources         | 5         | Dangling actions, unresolvable paths, missing actors, circular deps     |
| Validation timeouts       | 4         | Slow validators, zero timeout, negative timeout, partial results        |
| Invalid schema transforms | 3         | Non-dict returns, missing type field, null schema                       |
| Mixed ordering            | 4         | Required-before-informational, duplicate attachment IDs, unknown levels |
| Concurrent conflicts      | 1         | Two validators on same resource detect conflict                         |
| Rollback on failure       | 2         | Required failure triggers rollback of pending changes                   |

Step definitions: `features/steps/validation_edge_case_steps.py`

All step names are prefixed with `validation edge` to avoid `AmbiguousStep`
conflicts with existing steps.

### Robot Suite: `robot/validation_edge.robot`

6 integration tests exercising the validation edge case helper:

| Test Case                                     | Description                                    |
| --------------------------------------------- | ---------------------------------------------- |
| Validation Edge Load All Fixtures             | Loads all fixture files and verifies structure |
| Validation Edge Malformed Output Detection    | Detects malformed tool output patterns         |
| Validation Edge Missing Resource Error Paths  | Validates missing resource reference detection |
| Validation Edge Timeout Simulation            | Validates timeout simulation fixture data      |
| Validation Edge Schema Validation Errors      | Validates schema transform error detection     |
| Validation Edge Unknown Command Returns Error | Verifies unknown command exits with code 1     |

Helper script: `robot/helper_validation_edge.py`

### ASV Benchmarks: `benchmarks/validation_edge_bench.py`

Two benchmark suites:

- **`ValidationEdgeFixtureLoadSuite`** -- `time_load_malformed_tool_output`,
  `time_load_missing_resources`, `time_load_validation_timeouts`,
  `time_load_invalid_schema_transforms`, `time_load_mixed_ordering`,
  `time_load_all_fixtures`
- **`ValidationEdgeDetectionSuite`** -- `time_malformed_output_detection`,
  `time_missing_resource_detection`, `time_validation_ordering`,
  `time_combined_edge_detection`

### Running the Validation Edge Case Suites

```bash
# Behave only (validation edge cases)
nox -s unit_tests -- features/validation_edge_cases.feature

# Robot only (validation edge suite)
nox -s integration_tests -- --suite robot/validation_edge.robot

# Benchmarks
nox -s benchmark
```

## Scale Testing

Scale tests validate performance baselines for repository indexing and
decomposition across different repository sizes (1K, 5K, and 10K files).

### Fixtures

Scale test fixtures live in `features/fixtures/scale/`:

- `scale_metadata.json` — Scale profile definitions (file count, language mix, expected timings)
- `baseline_thresholds.json` — Performance threshold matrices (p50/p95/p99 per operation per file count)
- `generator_instructions.md` — Manual instructions for generating synthetic repos

### Running Scale Tests

```bash
# Behave scale test scenarios
nox -s unit_tests -- features/scale_test.feature

# Robot scale test integration
nox -s integration_tests -- --suite robot/scale_test.robot

# ASV scale fixture benchmarks
nox -s benchmark
```

### Scale Test Suites

| Suite                               | Framework | Scenarios | Description                                                                     |
| ----------------------------------- | --------- | --------- | ------------------------------------------------------------------------------- |
| `features/scale_test.feature`       | Behave    | 20        | Fixture loading, metadata validation, threshold checks, distribution simulation |
| `robot/scale_test.robot`            | Robot     | 6         | Integration-level fixture validation via helper script                          |
| `benchmarks/scale_fixture_bench.py` | ASV       | 6         | Performance benchmarks for fixture processing                                   |

For full scale testing documentation, see `docs/development/scale_testing.md`.

## CLI Extension Test Fixtures

The CLI extension tests cover automation profile, invariant, and actor override
functionality added in #325. Deeper coverage added in #326 validates resolution
logic, ordering semantics, error cases, and output format snapshots.

### Test Fixture Patterns

The CLI extension test suites use a **mocked lifecycle service** pattern. The
`_get_lifecycle_service` function in both `plan.py` and `action.py` is patched
so that tests exercise real CLI argument parsing, validation, and output
rendering without requiring a database or running server.

#### Common Fixtures

| Fixture             | Module                    | Description                                                                                                                  |
| ------------------- | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `_make_plan()`      | `cli_extensions_steps.py` | Creates a `Plan` instance with configurable automation profile, invariants, and actor overrides                              |
| `_make_action()`    | `cli_extensions_steps.py` | Creates an `Action` with configurable optional fields (estimation_actor, invariant_actor, inputs_schema, automation_profile) |
| `CliRunner`         | `typer.testing`           | Invokes CLI commands in-process without subprocess overhead                                                                  |
| `MagicMock` service | `unittest.mock`           | Mocks `PlanLifecycleService` for plan/action operations                                                                      |

#### Automation Profile Resolution Fixtures

Tests validate that each builtin profile name (`manual`, `review`, `supervised`,
`cautious`, `trusted`, `auto`) resolves correctly through the CLI and that
invalid profile names are rejected with a helpful error listing available
profiles. The service mock returns a `Plan` with an `AutomationProfileRef` to
verify end-to-end profile assignment.

#### Invariant Ordering Fixtures

Tests pass multiple `--invariant` flags and verify the service receives them in
exact insertion order. This tests the list-accumulation behaviour of Typer's
repeated option handling.

#### Actor Override Error Fixtures

Tests exercise the `validate_namespaced_actor` regex
(`^[a-z][a-z0-9-]*/[a-z][a-z0-9._-]*$`) with various invalid inputs:

- Empty string
- Missing slash (`no-slash`)
- Uppercase characters (`UPPER/case`)
- Leading slash (`/leading-slash`)
- Trailing slash (`trail/ing/`)
- Double slashes (`open//ai`)
- Special characters (`open@ai/gpt-4`)
- Numeric-leading namespace (`123/model`)
- Whitespace (`open ai/gpt`)

#### Output Snapshot Assertions

Tests render plan status, list, and action show output in JSON, YAML,
and table formats, then assert that:

- JSON output is valid (parses without error)
- Required keys are present (`namespaced_name`, `phase`, `automation_profile`, etc.)
- Invariant text values survive serialization round-trips
- YAML output contains expected field names
- Table output contains human-readable field values

### Behave Suite: `features/cli_extensions.feature`

Covers scenarios across these areas:

| Area                      | Scenarios | Description                                                          |
| ------------------------- | --------- | -------------------------------------------------------------------- |
| Automation profile flags  | 2         | Valid and invalid profile names                                      |
| Profile resolution (deep) | 9         | All builtin profiles + error cases with special chars, spaces, empty |
| Invariant flags           | 2         | Single and dual invariant flags                                      |
| Invariant ordering (deep) | 2         | Three and five invariant insertion-order preservation                |
| Actor overrides (valid)   | 4         | Strategy, execution, estimation, invariant actors                    |
| Actor overrides (invalid) | 4         | Malformed namespace/name formats                                     |
| Actor error cases (deep)  | 11        | Empty, double-slash, special chars, numeric-leading, whitespace      |
| Plan status display       | 4         | Table, JSON, single plan with invariants/profile                     |
| Plan list                 | 2         | Table and JSON with invariant count                                  |
| Action show               | 8         | Optional actors, invariants, inputs_schema, automation profile       |
| Combined flags            | 1         | Profile + invariant together                                         |
| validate_namespaced_actor | 2         | Unit-level accept/reject assertions                                  |
| Output snapshots (deep)   | 9         | JSON/YAML/table format validation for plan and action                |

Step definitions: `features/steps/cli_extensions_steps.py`

### Robot Suite: `robot/cli_extensions.robot`

Integration tests exercising CLI extensions through Python helper scripts:

| Test Case                                       | Description                                                                        |
| ----------------------------------------------- | ---------------------------------------------------------------------------------- |
| Plan Use With Invariants                        | Verifies `--invariant` flags pass through correctly                                |
| Plan Use With Automation Profile                | Verifies `--automation-profile` works                                              |
| Plan Use Actor Validation Valid                 | Valid namespaced actor format accepted                                             |
| Plan Use Actor Validation Invalid               | Invalid actor format rejected                                                      |
| Plan Use Combined Profile And Invariants        | Both profile and invariants together                                               |
| Action Show With Optional Actors And Invariants | `action show` renders estimation_actor, invariant_actor, invariants, inputs_schema |
| Action Show JSON With Optional Fields           | JSON output includes all optional fields                                           |
| Invariant Ordering Preserved                    | Multiple invariant flags preserve insertion order                                  |
| Profile Resolution All Builtins                 | All builtin profile names resolve correctly                                        |

Helper script: `robot/helper_cli_extensions.py`

### ASV Benchmarks

Two benchmark files cover CLI extension performance:

**`benchmarks/cli_extensions_bench.py`** (original):

- `PlanUseWithProfileSuite` — plan use with automation profile
- `PlanUseWithInvariantsSuite` — plan use with invariant flags
- `PlanUseActorValidationSuite` — plan use with actor overrides
- `PlanStatusWithExtendedFieldsSuite` — status/list rendering
- `ActionShowExtendedSuite` — action show in rich and JSON formats

**`benchmarks/cli_extension_tests_bench.py`** (extended #326):

- `ProfileResolutionSuite` — resolve all builtin profiles, single profile, invalid profile
- `InvariantOrderingSuite` — three and five invariant flag parsing
- `ActorValidationErrorSuite` — valid/invalid actor regex, CLI rejection
- `OutputFormatRenderingSuite` — JSON/YAML/table rendering for action show and plan status

### Running CLI Extension Tests

```bash
# Behave only (CLI extensions feature)
nox -s unit_tests -- features/cli_extensions.feature

# Robot only (CLI extensions suite)
nox -s integration_tests -- --suite robot/cli_extensions.robot

# Benchmarks
nox -s benchmark
```

## M2 Actor + Tool Source Smoke Suite

The M2 smoke suite validates the foundation for the Actor Graphs + Tool Sources epic (#356). It covers actor YAML loading, skill registry, tool lifecycle, and MCP stub tool discovery.

### Fixtures

M2 fixtures live in `features/fixtures/m2/`:

| Fixture File                 | Description                                               |
| ---------------------------- | --------------------------------------------------------- |
| `m2_hierarchical_actor.yaml` | Graph-type actor with planner → executor → reviewer nodes |
| `m2_skill_pack.yaml`         | Skill pack with tool references and an inline tool        |

### MCP Stub Server

The MCP stub server (`features/mocks/mcp_stub_server.py`) simulates MCP tool discovery and invocation in-process without requiring the real MCP adapter (#159). It exposes three deterministic tools: `mcp/search`, `mcp/fetch`, and `mcp/transform`.

Usage in tests:

```python
from features.mocks.mcp_stub_server import McpStubServer

server = McpStubServer()
server.start()
tools = server.discover()       # Returns 3 stub tools
result = server.invoke("mcp/search", {"query": "hello"})
server.stop()
```

### Test Suites

| Suite                                     | Framework | Scenarios | Description                                                 |
| ----------------------------------------- | --------- | --------- | ----------------------------------------------------------- |
| `features/m2_actor_tool_smoke.feature`    | Behave    | 10        | Actor loading, skill registry, tool lifecycle, MCP stub     |
| `robot/m2_actor_tool_smoke.robot`         | Robot     | 6         | CLI-level smoke for actor/skill/tool/MCP operations         |
| `benchmarks/m2_actor_tool_smoke_bench.py` | ASV       | 12        | Baseline runtime for actor, skill, tool, and MCP operations |

### Running the M2 Smoke Suite

```bash
# Behave only (M2 feature)
nox -s unit_tests -- features/m2_actor_tool_smoke.feature

# Robot only (M2 suite)
nox -s integration_tests -- --suite robot/m2_actor_tool_smoke.robot

# Benchmarks
nox -s benchmark
```

### What Is NOT Tested (deferred to dependent issues)

- Real MCP adapter integration (#159, assigned to Aditya)
- Agent skills loader (#160, assigned to Aditya)
- Hierarchical actor compiler (#158, assigned to Jeff)

The MCP stub server provides a test seam that can be replaced with the real adapter once #159 lands.

## M1 Source-Code Plan Lifecycle Smoke Tests

The M1 source-code smoke suites verify the minimal end-to-end source-code
workflow: a git repo fixture, a `git-checkout` resource config, and a minimal
action YAML with strategy/execution actors. Helper steps create temp projects,
link resources, and capture plan IDs for subsequent CLI verification.

### Fixtures (`features/fixtures/m1/`)

| Fixture File                 | Description                                                      |
| ---------------------------- | ---------------------------------------------------------------- |
| `git_repo.json`              | Minimal git repo definitions with file listings and branch info  |
| `git_checkout_resource.json` | Git-checkout resource configs with path and branch properties    |
| `action_sourcecode.yaml`     | Minimal action YAML with strategy/execution actors and arguments |

### Behave Suite: `features/m1_sourcecode_smoke.feature`

16 scenarios covering:

| Area             | Scenarios | Description                                                |
| ---------------- | --------- | ---------------------------------------------------------- |
| Fixture loading  | 3         | Load and validate git repo, checkout resource, action YAML |
| Action create    | 1         | Create action from YAML config via CLI                     |
| Project/resource | 2         | Create temp project, link resource                         |
| Plan use         | 3         | Use action to create plan, with project and args           |
| Plan execute     | 1         | Execute transitions to execute phase                       |
| Plan diff        | 1         | Show changeset via diff                                    |
| Plan apply       | 1         | Apply transitions to terminal state                        |
| Negative cases   | 3         | Unknown action, wrong phase, invalid actor                 |

Step definitions: `features/steps/m1_sourcecode_smoke_steps.py`

All step names are prefixed with `m1 smoke` to avoid `AmbiguousStep`
conflicts with existing steps.

### Robot Suite: `robot/m1_sourcecode_smoke.robot`

8 integration tests exercising the M1 lifecycle through CLI:

| Test Case                           | Description                            |
| ----------------------------------- | -------------------------------------- |
| M1 Action Create From Config        | Creates action from fixture YAML       |
| M1 Plan Use Creates Strategize Plan | Uses action to create plan             |
| M1 Plan Use With Project Link       | Plan use with project argument         |
| M1 Plan Execute Transitions Phase   | Execute phase transition               |
| M1 Plan Diff Shows Changeset        | Diff command for changeset             |
| M1 Plan Apply Reaches Terminal      | Apply to terminal state                |
| M1 Full Lifecycle Action To Apply   | End-to-end flow                        |
| M1 Plan Use With Plain Format       | `--format plain` for stable assertions |

Helper script: `robot/helper_m1_sourcecode_smoke.py`

### ASV Benchmarks: `benchmarks/m1_sourcecode_smoke_bench.py`

Six benchmark suites measuring M1 CLI overhead:

- **`M1ActionCreateSuite`** -- `time_action_create_from_yaml`
- **`M1PlanUseSuite`** -- `time_plan_use_minimal`, `time_plan_use_with_project`,
  `time_plan_use_with_args`, `time_plan_use_plain_format`
- **`M1PlanExecuteSuite`** -- `time_plan_execute`
- **`M1PlanApplySuite`** -- `time_plan_lifecycle_apply`
- **`M1FixtureLoadSuite`** -- `time_load_git_repo_fixture`,
  `time_load_git_checkout_fixture`

### Running the M1 Source-Code Smoke Suites

```bash
# Behave only (M1 smoke feature)
nox -s unit_tests -- features/m1_sourcecode_smoke.feature

# Robot only (M1 smoke suite)
nox -s integration_tests -- --suite robot/m1_sourcecode_smoke.robot

# Benchmarks
nox -s benchmark
```

### M1 Smoke Run Instructions

To perform a quick M1 source-code smoke run:

1. Run the Behave feature:
    ```bash
    nox -s unit_tests -- features/m1_sourcecode_smoke.feature
    ```
2. Run the Robot suite:
    ```bash
    nox -s integration_tests -- --suite robot/m1_sourcecode_smoke.robot
    ```
3. Run benchmarks:
    ```bash
    nox -s benchmark
    ```

### Failure Triage Tips

- **`AmbiguousStep` errors**: All M1 smoke steps are prefixed with `m1 smoke`.
  If ambiguous, check that no other step file defines a conflicting pattern.
- **Fixture file not found**: Verify `features/fixtures/m1/` contains all three
  fixture files (`git_repo.json`, `git_checkout_resource.json`,
  `action_sourcecode.yaml`).
- **Robot `FAIL` sentinel**: Each Robot helper subcommand prints a detailed
  `FAIL:` line with exit code and output when something goes wrong. Check the
  Robot log for these messages.
- **`InvalidPhaseTransitionError`**: Verify the plan is in the correct phase
  before attempting a transition. The M1 smoke tests mock phase state carefully.
- **Coverage drops**: The M1 smoke tests cover fixture loading, CLI argument
  parsing, and mock service integration. If coverage drops, look at
  `build/htmlcov/index.html` for uncovered lines in plan/action CLI commands.

## M3 Decision Tree, Validation Gating, and Invariant Enforcement Smoke Tests

### Overview

The M3 smoke suite validates the decision tree, correction workflow,
validation gating (add / attach / detach), and invariant constraint
(add / list / remove) subsystems via CLI integration tests.

Issue: [#179](https://git.cleverthis.com/cleveragents/cleveragents-core/issues/179)

### Files

| File                                                   | Purpose                                     |
| ------------------------------------------------------ | ------------------------------------------- |
| `features/m3_decision_validation_smoke.feature`        | Behave BDD scenarios (25 scenarios)         |
| `features/steps/m3_decision_validation_smoke_steps.py` | Step implementations                        |
| `features/fixtures/m3/decision_tree_outputs.json`      | Decision tree fixture data                  |
| `features/fixtures/m3/validation_attachments.json`     | Validation attachment fixture data          |
| `features/fixtures/m3/invariant_configs.json`          | Invariant config fixture data               |
| `robot/m3_decision_validation_smoke.robot`             | Robot Framework integration tests (8 cases) |
| `robot/helper_m3_decision_validation_smoke.py`         | Robot helper with 8 subcommands             |
| `benchmarks/m3_smoke_bench.py`                         | ASV benchmark suites (5 classes)            |

### Scenario Categories

1. **Decision tree fixture loading** -- Loads JSON fixtures and verifies
   decision tree structure, invariant nodes, and correction metadata.
2. **Validation fixture loading** -- Verifies validation attachment data
   including required and informational modes.
3. **Invariant fixture loading** -- Verifies invariant config data
   including merge set precedence.
4. **Invariant CLI** -- Tests `agents invariant add`, `list`, and `remove`
   commands with global and project scopes.
5. **Validation CLI** -- Tests `agents validation add`, `attach`, and
   `detach` commands.
6. **Plan correct CLI** -- Tests `agents plan correct` with `--dry-run`,
   `--mode revert`, and `--mode append`.
7. **Negative cases** -- Empty guidance, invalid correction mode.

### Mock Strategy

- **Invariant CLI** patches `cleveragents.cli.commands.invariant._get_service`
- **Validation CLI** patches `cleveragents.cli.commands.validation._get_tool_registry_service`
- **Plan CLI** patches `cleveragents.cli.commands.plan._get_lifecycle_service`
- **Correction** patches `CorrectionService` constructor at module level

### Failure Triage Tips

- **`AmbiguousStep` errors**: All M3 smoke steps are prefixed with `m3 smoke`.
  If ambiguous, check that no other step file defines a conflicting pattern.
- **Fixture file not found**: Verify `features/fixtures/m3/` contains all
  three JSON fixture files.
- **Robot `FAIL` sentinel**: Each Robot helper subcommand prints a detailed
  `FAIL:` line with exit code and output. Check the Robot log.
- **Coverage drops**: The M3 smoke tests cover invariant/validation CLI
  commands, plan correct workflow, and fixture loading. Check
  `build/htmlcov/index.html` for uncovered lines.

## M4 Correction + Subplan Smoke Suite

The M4 smoke suite validates correction flows, subplan execution, and conflict
simulation for the M4 milestone. It covers decision tree corrections (revert and
append modes, dry-run impact analysis), subplan lifecycle (sequential, parallel,
dependency-ordered execution), failure handling (stop-others, retry logic), and
merge conflict simulation (git-three-way, fail-on-conflict, last-wins,
sequential-apply strategies).

### Fixtures (`features/fixtures/m4/`)

| Fixture File                | Description                                                         |
| --------------------------- | ------------------------------------------------------------------- |
| `correction_flows.json`     | Revert, append, dry-run, and high-risk correction scenarios         |
| `subplan_execution.json`    | Sequential, parallel, dependency-ordered, and retry subplan configs |
| `conflict_simulations.json` | Merge conflict scenarios for all SubplanMergeStrategy variants      |

### Behave Suite: `features/m4_correction_subplan_smoke.feature`

20 scenarios covering:

| Area                | Scenarios | Description                                                  |
| ------------------- | --------- | ------------------------------------------------------------ |
| Fixture loading     | 3         | Load and validate correction, subplan, and conflict fixtures |
| Correction flows    | 4         | Revert, append, dry-run, and invalid mode via CLI            |
| Subplan execution   | 2         | Sequential and parallel subplan status rendering             |
| Failure handling    | 3         | Stop-others, retry retriable, skip non-retriable             |
| Conflict simulation | 3         | No-conflict, fail-on-conflict, last-wins merge strategies    |
| Negative cases      | 2         | Non-existent plan, empty decision ID                         |

Step definitions: `features/steps/m4_correction_subplan_smoke_steps.py`

All step names are prefixed with `m4 smoke` to avoid `AmbiguousStep`
conflicts with existing steps.

### Robot Suite: `robot/m4_correction_subplan_smoke.robot`

8 integration tests exercising M4 correction and subplan flows through CLI:

| Test Case                           | Description                                       |
| ----------------------------------- | ------------------------------------------------- |
| M4 Correction Revert Via CLI        | Invoke plan correct --mode revert                 |
| M4 Correction Append Via CLI        | Invoke plan correct --mode append                 |
| M4 Correction Dry Run Via CLI       | Invoke plan correct --dry-run                     |
| M4 Subplan Status Sequential        | Plan status with sequential subplan config        |
| M4 Subplan Status Parallel          | Plan status with parallel subplan config          |
| M4 Failure Handler Evaluation       | SubplanFailureHandler decision logic              |
| M4 Fixture Loading                  | Load all M4 fixture files                         |
| M4 Full Correction And Subplan Flow | End-to-end correction + subplan + failure handler |

Helper script: `robot/helper_m4_correction_subplan_smoke.py`

### ASV Benchmarks: `benchmarks/m4_smoke_bench.py`

Four benchmark suites measuring M4 suite runtime:

- **`M4CorrectionCLISuite`** -- `time_correction_revert`, `time_correction_append`,
  `time_correction_dry_run`
- **`M4SubplanStatusSuite`** -- `time_status_with_subplans_json`,
  `time_status_with_subplans_rich`
- **`M4FailureHandlerSuite`** -- `time_should_stop_others_fail_fast`,
  `time_should_retry_retriable`, `time_should_retry_non_retriable`
- **`M4FixtureLoadSuite`** -- `time_load_correction_flows`,
  `time_load_subplan_execution`, `time_load_conflict_simulations`,
  `time_load_all_fixtures`

### Running the M4 Correction + Subplan Smoke Suites

```bash
# Behave only (M4 smoke feature)
nox -s unit_tests -- features/m4_correction_subplan_smoke.feature

# Robot only (M4 smoke suite)
nox -s integration_tests -- --suite robot/m4_correction_subplan_smoke.robot

# Benchmarks
nox -s benchmark
```

### Failure Triage Tips

- **`AmbiguousStep` errors**: All M4 smoke steps are prefixed with `m4 smoke`.
  If ambiguous, check that no other step file defines a conflicting pattern.
- **Fixture file not found**: Verify `features/fixtures/m4/` contains all three
  fixture files (`correction_flows.json`, `subplan_execution.json`,
  `conflict_simulations.json`).
- **Robot `FAIL` sentinel**: Each Robot helper subcommand prints a detailed
  `FAIL:` line with exit code and output when something goes wrong.
- **Correction service mock**: The M4 steps patch `CorrectionService` directly
  in the plan CLI module. If the import path changes, update the patch target.
- **Coverage drops**: The M4 smoke tests cover correction CLI commands,
  SubplanFailureHandler logic, and fixture loading. Check
  `build/htmlcov/index.html` for uncovered lines in correction/plan CLI code.

## M5 ACMS Pipeline + Large-Project Context Smoke Tests

The M5 smoke suites verify the ACMS context pipeline foundation: context
assembly, budget enforcement, multi-project context, context analysis,
and project context policy resolution.

### Behave: `features/m5_acms_smoke.feature`

27 scenarios covering:

| Area                       | Scenarios                                                           |
| -------------------------- | ------------------------------------------------------------------- |
| Fixture loading            | Load ACMS context policy, large project context, analysis results   |
| Context policy resolution  | Default view, inheritance chain, strategize/execute/apply overrides |
| Budget enforcement         | max_file_size, max_total_size, zero/negative validation, unlimited  |
| Context assembly (CLI)     | list, add, show, clear via mocked ContextService                    |
| Project context policy CLI | show policy, inspect/simulate stubs (NotImplementedError)           |
| Context analysis agent     | Summary, dependencies, relevance scores (mocked)                    |
| Multi-project context      | Independent context per project                                     |
| Exclusion patterns         | Glob-based path filtering                                           |

Step definitions: `features/steps/m5_acms_smoke_steps.py`
Fixtures: `features/fixtures/m5/`

- `acms_context_policy.json` -- Phase-specific context policy with size limits
- `large_project_context.json` -- Simulated 500-file project with tier metadata
- `context_analysis_results.json` -- Pre-computed analysis with dependencies and
  relevance scores

### Robot: `robot/m5_acms_smoke.robot`

| Test Case                                    | Description                       |
| -------------------------------------------- | --------------------------------- |
| Load ACMS Context Policy Fixture             | Validates fixture structure       |
| Load Large Project Context Fixture           | Validates file count and entries  |
| Resolve Default View From Empty Policy       | Empty policy includes all paths   |
| Resolve Strategize Inherits From Default     | Phase inheritance works           |
| Resolve Strategize With Override             | Override takes precedence         |
| Budget Max File Size Enforcement             | File-level budget check           |
| Budget Max Total Size Enforcement            | Aggregate budget check            |
| Invalid Phase Raises Error                   | Error handling for invalid phases |
| Context Analysis Fixture Has Required Fields | Analysis result structure         |
| Multi Project Independent Context            | Independent context per project   |

Helper script: `robot/helper_m5_acms_smoke.py`

### ASV Benchmarks: `benchmarks/m5_smoke_bench.py`

Four benchmark suites measuring M5 suite runtime:

- **`PolicyResolutionSuite`** -- `time_resolve_view` across all 4 phases
- **`BudgetEnforcementSuite`** -- `time_check_file_budget`,
  `time_check_total_budget` with 100/1000/10000 files
- **`FixtureLoadSuite`** -- `time_load_policy_fixture`,
  `time_load_large_project_fixture`, `time_load_analysis_fixture`,
  `time_load_all_fixtures`
- **`ContextViewConstructionSuite`** -- `time_construct_view`,
  `time_construct_full_policy` with 1/10/50 path patterns

### Running the M5 ACMS Smoke Suites

```bash
# Behave only (M5 smoke feature)
nox -s unit_tests -- features/m5_acms_smoke.feature

# Robot only (M5 smoke suite)
nox -s integration_tests -- --suite robot/m5_acms_smoke.robot

# Benchmarks
nox -s benchmark
```

### Failure Triage Tips

- **`AmbiguousStep` errors**: All M5 smoke steps are prefixed with `m5 smoke`.
  If ambiguous, check that no other step file defines a conflicting pattern.
- **Fixture file not found**: Verify `features/fixtures/m5/` contains all three
  fixture files (`acms_context_policy.json`, `large_project_context.json`,
  `context_analysis_results.json`).
- **`context_inspect`/`context_simulate` stubs**: These commands raise
  `NotImplementedError` by design until ACMS wiring is complete.
- **Coverage drops**: The M5 smoke tests cover context policy resolution, budget
  enforcement, and CLI context commands. Check `build/htmlcov/index.html` for
  uncovered lines in context-related modules.

## M6 Autonomy Acceptance Suite

E2E fixtures covering A2A facade flows and autonomy guardrails. The suite
exercises guardrail enforcement (max steps, tool budget, confirmations) and
audit trail persistence in integrated scenarios.

### Overview

The M6 acceptance suite validates the autonomy hardening layer:

- **A2A Local Facade** — local-mode dispatch of all 11 supported operations
  (session, plan, registry, context, event).
- **Autonomy Guardrails** — denylist/allowlist enforcement, cost budget caps,
  tool call limits, write approval gates, apply approval gates.
- **Automation Profiles** — 8 built-in profiles, custom namespaced profiles,
  YAML loading, threshold validation, and four-level resolution precedence
  (plan > action > project > global).
- **A2A Event Queue** — publish/subscribe, local callbacks, unsubscribe, close
  semantics, remote stub rejection.
- **A2A HTTP Transport Stub** — all mutating methods raise
  `A2aNotAvailableError` in local mode.
- **A2A Version Negotiation** — accept supported versions, reject unsupported.
- **A2A Model Validation** — non-empty operation, valid status values,
  non-empty event types, non-empty error codes.

### Fixtures (`features/fixtures/m6/`)

| File                       | Purpose                                                                         |
| -------------------------- | ------------------------------------------------------------------------------- |
| `a2a_facade_flows.json`    | A2A operation flows (session, plan, registry, context, event)                   |
| `autonomy_guardrails.json` | Guard configurations (denylist, allowlist, budget, write/apply approval)        |
| `automation_profiles.json` | Built-in profile assertions, custom profile config, resolution precedence cases |

### Behave Suite: `features/m6_autonomy_acceptance.feature`

- **Fixture loading** — 3 scenarios verifying JSON structure
- **A2A facade dispatch** — 10 scenarios covering all 42 operations
- **A2A error handling** — 2 scenarios (unknown operation, invalid request type)
- **Service registration** — 2 scenarios (register + list operations)
- **Event queue** — 5 scenarios (publish, subscribe, unsubscribe, close, remote reject)
- **HTTP transport stub** — 4 scenarios (send, connect, disconnect, is_connected)
- **Version negotiation** — 3 scenarios (accept, reject, is_supported)
- **Built-in profiles** — 3 scenarios (list, manual thresholds, full-auto thresholds)
- **Profile creation** — 4 scenarios (custom, invalid name, bad threshold, YAML load)
- **Guard enforcement** — 8 scenarios (denylist, allowlist, call limit, budget, write, apply, no guards)
- **Profile resolution** — 3 scenarios (plan precedence, action precedence, global default)
- **Service guard evaluation** — 1 scenario
- **Model validation** — 4 scenarios (A2aRequest, A2aResponse, A2aEvent, A2aErrorDetail)

Step implementations: `features/steps/m6_autonomy_acceptance_steps.py`

### Robot Suite: `robot/m6_autonomy_acceptance.robot`

| Test Case                             | What it checks                                     |
| ------------------------------------- | -------------------------------------------------- |
| M6 A2A Facade Session Lifecycle       | session.create + session.close dispatch            |
| M6 A2A Facade Plan Lifecycle          | plan create/execute/status/diff/apply              |
| M6 A2A Facade Unknown Operation Error | A2aOperationNotFoundError on bad op                |
| M6 A2A Event Queue Publish Subscribe  | publish, subscribe, unsubscribe, close             |
| M6 A2A Transport Stub Rejects All     | send/connect/disconnect raise A2aNotAvailableError |
| M6 A2A Version Negotiation            | accept 1.0, reject 2.0                             |
| M6 Guard Denylist Enforcement         | denylist blocks denied tools                       |
| M6 Guard Budget Enforcement           | cost budget and call limit guards                  |
| M6 Profile Resolution Precedence      | plan > action > project > global                   |
| M6 Fixture Loading                    | all 3 fixture files load and validate              |
| M6 Full Autonomy Acceptance Flow      | end-to-end facade + guard + resolution             |

Helper: `robot/helper_m6_autonomy_acceptance.py`

### ASV Benchmarks: `benchmarks/m6_acceptance_bench.py`

| Suite                      | Methods                                                                                                                                                                                    |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `M6FacadeDispatchSuite`    | `time_session_create`, `time_plan_create`, `time_plan_execute`, `time_list_operations`                                                                                                     |
| `M6GuardEvaluationSuite`   | `time_denylist_check_allowed`, `time_denylist_check_denied`, `time_allowlist_check_allowed`, `time_allowlist_check_denied`, `time_budget_check_under`, `time_no_guards_check`              |
| `M6ProfileResolutionSuite` | `time_resolve_plan_level`, `time_resolve_action_level`, `time_resolve_global_default`, `time_get_builtin_profile`, `time_list_profiles`, `time_evaluate_guard`, `time_version_negotiation` |
| `M6EventQueueSuite`        | `time_publish_event`, `time_subscribe_and_publish`, `time_get_events`                                                                                                                      |
| `M6FixtureLoadSuite`       | `time_load_a2a_facade_flows`, `time_load_autonomy_guardrails`, `time_load_automation_profiles`, `time_load_all_fixtures`                                                                   |

### Running the M6 Autonomy Acceptance Suites

```bash
# Behave only
nox -s behave -- features/m6_autonomy_acceptance.feature

# Robot only
nox -s robot -- robot/m6_autonomy_acceptance.robot

# ASV benchmarks
nox -s benchmark

# Everything (default nox sessions)
nox
```

### Failure Triage Tips

- **`AmbiguousStep` errors**: All M6 smoke steps are prefixed with `m6 smoke`.
  If ambiguous, check that no other step file defines a conflicting pattern.
- **Fixture file not found**: Verify `features/fixtures/m6/` contains all three
  fixture files (`a2a_facade_flows.json`, `autonomy_guardrails.json`,
  `automation_profiles.json`).
- **Robot `FAIL` sentinel**: Each Robot helper subcommand prints a detailed
  `FAIL:` line with exit code and output when something goes wrong.
- **A2aNotAvailableError expected**: The transport stub and remote event
  subscribe are _supposed_ to raise this error. If a test fails here, it
  likely means the stub was changed to not raise.
- **Guard evaluation failures**: Check that the `AutomationGuard` model fields
  match the fixture data. Denylist/allowlist/budget fields are all optional.
- **Coverage drops**: The M6 acceptance tests cover the `a2a/` package facade,
  events, transport, versioning, errors, and models, plus
  `automation_profile.py` and `automation_profile_service.py`. Check
  `build/htmlcov/index.html` for uncovered lines.

---

##