# Testing — CleverAgents Project

> ⚠️ Rules here override `cleverthis-guidelines`. Apply these exactly.

---

## Framework Decisions (non-negotiable)

| Test type | Framework | Directory | nox session |
|-----------|-----------|-----------|-------------|
| Unit / BDD | Behave (Cucumber/Gherkin) | `features/` | `nox -s unit_tests` |
| Integration | Robot Framework via pabot | `robot/` | `nox -s integration_tests` |
| End-to-end | Robot Framework (real LLM keys) | `robot/` | `nox -s e2e_tests` |
| Performance | ASV (Airspeed Velocity) | `benchmarks/` | `nox -s benchmark` |

**Prohibited:**
- No pytest-style unit tests — ever.
- No `tests/` directory — it intentionally does not exist in this project.
- No xUnit-style tests (JUnit, NUnit, etc.) at the unit level.

---

## Behave (BDD Unit Tests)

### Location rules

- Feature files: `features/` or `features/<topic>/`
- Step definitions: `features/steps/`
- Mocks/fakes/stubs: `features/mocks/` **ONLY**

### Step file naming (strictly enforced)

1. **Before creating a new step file** — check whether an existing file already
   covers the same behavior. Extend it rather than creating a duplicate.
2. Steps used **only** by `foo.feature` → `features/steps/foo_steps.py`
3. Steps used by **multiple features** → clearly named, reusable file under
   `features/steps/` (update existing shared files before creating new ones)
4. Every `.feature` file must be committed with **all its steps fully implemented**
   — never placeholder step definitions

### Running Behave

```bash
nox -s unit_tests        # always use nox — never run behave directly
```

---

## Robot Framework (Integration + E2E)

### Integration tests (`robot/`)

- Exercise real services, real endpoints, real dependencies
- **NO mocking of any kind** — strictly prohibited
- Mocks acceptable ONLY for truly impractical external dependencies (document why)
- Must be updated whenever component interfaces change

```bash
nox -s integration_tests  # Robot Framework via pabot (parallel)
```

### End-to-end tests (`robot/` — e2e tagged)

- Require real LLM API keys — separate from integration tests
- Test the complete system with real providers end-to-end

```bash
nox -s e2e_tests
```

---

## Coverage Requirement (hard merge gate)

- **Threshold: ≥ 97%** — this is a hard project requirement
- Measured by: Slipcover via `nox -s coverage_report`
- PR is **blocked automatically** if coverage drops below 97%
- Exception: project owner can grant a documented exception only
- Any code excluded from coverage: requires explicit project owner approval

```bash
nox -s coverage_report    # look for "COVERAGE OK" or "COVERAGE FAILED"
```

---

## Mock Placement (absolute rule)

| Location | Permitted? |
|----------|-----------|
| `features/mocks/` | ✅ YES — the ONLY permitted location |
| `src/cleveragents/` | ❌ NEVER |
| `scripts/` | ❌ NEVER |
| Anywhere else | ❌ NEVER |

- Production code must **not** contain `if testing:` guards or test-only paths
- Use **dependency injection** to substitute test doubles — not conditional logic

---

## Performance Benchmarks

- Write ASV benchmarks for all performance-sensitive code
- Benchmarks live in `benchmarks/`
- Run benchmarks: `nox -s benchmark`
- PR regression check: `nox -s benchmark_regression` (results uploaded as artifact)

---

## TDD Bug Fix Tag System

### The three tags

| Tag | Lifecycle | Purpose |
|-----|-----------|---------|
| `@tdd_issue` | **PERMANENT** | Generic filter — on ALL TDD issue tests |
| `@tdd_issue_N` | **PERMANENT** | Issue link (N = bug number) — regression reference |
| `@tdd_expected_fail` | **TEMPORARY** | Inverts pass/fail while bug unfixed |

### How @tdd_expected_fail works

- When **present**: test PASSES CI if the underlying assertion FAILS (bug still exists)
- When **present**: test FAILS CI if the underlying assertion PASSES (bug fixed but tag not removed)
- When **absent**: test runs normally

### CI-enforced validation rules (all checked on every PR)

1. `@tdd_issue_N` present → `@tdd_issue` **MUST** also be present
2. `@tdd_expected_fail` present → `@tdd_issue` AND `@tdd_issue_N` **MUST** both be present
3. Bug fix PR closing #N → `@tdd_expected_fail` **MUST** be removed from all `@tdd_issue_N` scenarios
   - PR is **BLOCKED** if the tag is still present
4. Bug fix PR closing #N → a `@tdd_issue_N` test **MUST** exist in the codebase
   - PR is **BLOCKED** if no TDD test was ever written

### Assertion type rule (critical)

Expected-fail steps **must** use `AssertionError`:

```python
# CORRECT — inverted by the TDD hook:
assert some_condition, "descriptive failure message"
raise AssertionError("bug still present: description of what should work")

# WRONG — these are NOT inverted, they break CI regardless of the tag:
raise ValueError(...)
raise RuntimeError(...)
raise OSError(...)
raise TypeError(...)
```

### Complete example

```gherkin
# Before fix — all three tags:
@tdd_issue @tdd_issue_123 @tdd_expected_fail
Scenario: Bug #123 — SHACL validation rejects valid graph
  Given a valid resource graph
  When SHACL validation is applied
  Then the validation should succeed

# After fix — dev removes @tdd_expected_fail, leaves the other two:
@tdd_issue @tdd_issue_123
Scenario: Bug #123 — SHACL validation rejects valid graph
  Given a valid resource graph
  When SHACL validation is applied
  Then the validation should succeed
```

---

## Gherkin Quality Guidelines

Good Gherkin scenarios are readable by non-developers and serve as living documentation.

### Given / When / Then semantics

| Keyword | Purpose | What it tests |
|---------|---------|---------------|
| `Given` | Pre-conditions — establish the state of the world before the action | Inputs, setup |
| `When` | The action under test — what the user or system does | The triggering event |
| `Then` | The observable outcome — what must be true after the action | Outputs, side effects |
| `And` | Continuation — extends the previous Given/When/Then | More of the same type |
| `But` | Contrasting continuation — typically for "does NOT happen" assertions | Negative outcomes |

### Scenario Outline (data-driven scenarios)

Use `Scenario Outline` when the same behavior must be verified for multiple data combinations:

```gherkin
Scenario Outline: Invalid confidence scores are rejected
  Given an automation profile with <flag> set to <value>
  When the profile is validated
  Then a validation error is raised

  Examples:
    | flag          | value |
    | decompose_task | -0.1  |
    | decompose_task |  1.1  |
    | edit_code      | -1.0  |
```

### Naming rules

```gherkin
# GOOD — describes the behavior:
Scenario: Plan enters errored state when execution actor raises uncaught exception
Scenario: SHACL validation rejects a graph with a missing required property

# BAD — describes the test mechanics:
Scenario: Test plan execution with error
Scenario: Test case 1 for validator
```

### Common anti-patterns to avoid

- **Implementation details in steps**: `When PlanService._execute_phase() is called` → `When the plan executes`
- **Multiple behaviors per scenario**: split into separate scenarios
- **Over-specific data**: use meaningful values, not arbitrary UUIDs in step text
- **Missing Then**: every scenario must have at least one assertion

---

## Property-Based Testing (Hypothesis)

Use Hypothesis for verifying invariants and boundary conditions.

### When to use Hypothesis

| Use case | Example |
|----------|---------|
| Invariants that must hold for any valid input | Decision tree structural invariants |
| Boundary conditions | Confidence score clamp to 0.0–1.0 |
| Merge correctness | No data loss for any combination of file edits |
| Ordering/sorting | ULID lexicographic monotonicity |
| Precedence rules | Invariant resolution always follows plan > action > project > global |

### Recommended strategies to build

```python
from hypothesis import given, strategies as st
from hypothesis.strategies import composite

@composite
def ulid_strategy(draw):
    """Strategy that generates valid ULID strings."""
    ...

@composite
def confidence_score_strategy(draw):
    """Strategy for valid confidence scores (0.0–1.0 inclusive)."""
    return draw(st.floats(min_value=0.0, max_value=1.0, allow_nan=False))

@composite
def invariant_set_strategy(draw):
    """Strategy for sets of invariants at multiple scopes."""
    ...
```

### Integration with Behave

Hypothesis tests run inside Behave step definitions:

```python
from hypothesis import given, settings
import hypothesis.strategies as st

@then("the confidence score is always clamped to the valid range")
def step_confidence_clamped(context):
    @given(raw=st.floats(allow_nan=True, allow_infinity=True))
    @settings(max_examples=200)
    def check_clamp(raw):
        clamped = clamp_confidence(raw)
        assert 0.0 <= clamped <= 1.0

    check_clamp()
```

---

## LangChain/LangGraph Testing

- Use `FakeListLLM` or a custom mock provider — **never** real LLM API in unit tests
- Test each graph node's state transformation in isolation
- Verify complete workflow execution with expected state transitions
- Streaming: test both event emission AND final results
- Memory: verify conversation history and entity tracking

---

## Test-First Development (mandatory)

1. Write the test **before** the implementation — no exceptions
2. Tests go in the **same commit** as the implementation they cover
3. Never commit implementation without its tests
4. Test failure during development → immediately becomes a blocking task
   - Exception: `@tdd_expected_fail` tests are tracked failures, not unaddressed ones
