cleveragents-core/.opencode/skills/cleveragents-contributing/references/testing

Testing — CleverAgents Project

⚠️ Rules here override cleverthis-guidelines. Apply these exactly.

---

Framework Decisions (non-negotiable)

Test type	Framework	Directory	nox session
Unit / BDD	Behave (Cucumber/Gherkin)	features/	nox -s unit_tests
Integration	Robot Framework via pabot	robot/	nox -s integration_tests
End-to-end	Robot Framework (real LLM keys)	robot/	nox -s e2e_tests
Performance	ASV (Airspeed Velocity)	benchmarks/	nox -s benchmark

Prohibited:

• No pytest-style unit tests — ever.
• No tests/ directory — it intentionally does not exist in this project.
• No xUnit-style tests (JUnit, NUnit, etc.) at the unit level.

---

Behave (BDD Unit Tests)

Location rules

• Feature files: features/ or features/<topic>/
• Step definitions: features/steps/
• Mocks/fakes/stubs: features/mocks/ ONLY

Step file naming (strictly enforced)

1. Before creating a new step file — check whether an existing file already covers the same behavior. Extend it rather than creating a duplicate.
2. Steps used only by foo.feature → features/steps/foo_steps.py
3. Steps used by multiple features → clearly named, reusable file under features/steps/ (update existing shared files before creating new ones)
4. Every .feature file must be committed with all its steps fully implemented — never placeholder step definitions

Running Behave

nox -s unit_tests        # always use nox — never run behave directly

---

Robot Framework (Integration + E2E)

Integration tests (robot/)

• Exercise real services, real endpoints, real dependencies
• NO mocking of any kind — strictly prohibited
• Mocks acceptable ONLY for truly impractical external dependencies (document why)
• Must be updated whenever component interfaces change

nox -s integration_tests  # Robot Framework via pabot (parallel)

End-to-end tests (robot/ — e2e tagged)

• Require real LLM API keys — separate from integration tests
• Test the complete system with real providers end-to-end

nox -s e2e_tests

---

Coverage Requirement (hard merge gate)

• Threshold: ≥ 97% — this is a hard project requirement
• Measured by: Slipcover via nox -s coverage_report
• PR is blocked automatically if coverage drops below 97%
• Exception: project owner can grant a documented exception only
• Any code excluded from coverage: requires explicit project owner approval

nox -s coverage_report    # look for "COVERAGE OK" or "COVERAGE FAILED"

---

Mock Placement (absolute rule)

Location	Permitted?
features/mocks/	✅ YES — the ONLY permitted location
src/cleveragents/	❌ NEVER
scripts/	❌ NEVER
Anywhere else	❌ NEVER

• Production code must not contain if testing: guards or test-only paths
• Use dependency injection to substitute test doubles — not conditional logic

---

Performance Benchmarks

• Write ASV benchmarks for all performance-sensitive code
• Benchmarks live in benchmarks/
• Run benchmarks: nox -s benchmark
• PR regression check: nox -s benchmark_regression (results uploaded as artifact)

---

TDD Bug Fix Tag System

The three tags

Tag	Lifecycle	Purpose
@tdd_issue	PERMANENT	Generic filter — on ALL TDD issue tests
@tdd_issue_N	PERMANENT	Issue link (N = bug number) — regression reference
@tdd_expected_fail	TEMPORARY	Inverts pass/fail while bug unfixed

How @tdd_expected_fail works

• When present: test PASSES CI if the underlying assertion FAILS (bug still exists)
• When present: test FAILS CI if the underlying assertion PASSES (bug fixed but tag not removed)
• When absent: test runs normally

CI-enforced validation rules (all checked on every PR)

1. @tdd_issue_N present → @tdd_issue MUST also be present
2. @tdd_expected_fail present → @tdd_issue AND @tdd_issue_N MUST both be present
3. Bug fix PR closing #N → @tdd_expected_fail MUST be removed from all @tdd_issue_N scenarios
   • PR is BLOCKED if the tag is still present
4. Bug fix PR closing #N → a @tdd_issue_N test MUST exist in the codebase
   • PR is BLOCKED if no TDD test was ever written

Assertion type rule (critical)

Expected-fail steps must use AssertionError:

# CORRECT — inverted by the TDD hook:
assert some_condition, "descriptive failure message"
raise AssertionError("bug still present: description of what should work")

# WRONG — these are NOT inverted, they break CI regardless of the tag:
raise ValueError(...)
raise RuntimeError(...)
raise OSError(...)
raise TypeError(...)

Complete example

# Before fix — all three tags:
@tdd_issue @tdd_issue_123 @tdd_expected_fail
Scenario: Bug #123 — SHACL validation rejects valid graph
  Given a valid resource graph
  When SHACL validation is applied
  Then the validation should succeed

# After fix — dev removes @tdd_expected_fail, leaves the other two:
@tdd_issue @tdd_issue_123
Scenario: Bug #123 — SHACL validation rejects valid graph
  Given a valid resource graph
  When SHACL validation is applied
  Then the validation should succeed

---

Gherkin Quality Guidelines

Good Gherkin scenarios are readable by non-developers and serve as living documentation.

Given / When / Then semantics

Keyword	Purpose	What it tests
Given	Pre-conditions — establish the state of the world before the action	Inputs, setup
When	The action under test — what the user or system does	The triggering event
Then	The observable outcome — what must be true after the action	Outputs, side effects
And	Continuation — extends the previous Given/When/Then	More of the same type
But	Contrasting continuation — typically for "does NOT happen" assertions	Negative outcomes

Scenario Outline (data-driven scenarios)

Use Scenario Outline when the same behavior must be verified for multiple data combinations:

Scenario Outline: Invalid confidence scores are rejected
  Given an automation profile with <flag> set to <value>
  When the profile is validated
  Then a validation error is raised

  Examples:
    | flag          | value |
    | decompose_task | -0.1  |
    | decompose_task |  1.1  |
    | edit_code      | -1.0  |

Naming rules

# GOOD — describes the behavior:
Scenario: Plan enters errored state when execution actor raises uncaught exception
Scenario: SHACL validation rejects a graph with a missing required property

# BAD — describes the test mechanics:
Scenario: Test plan execution with error
Scenario: Test case 1 for validator

Common anti-patterns to avoid

• Implementation details in steps: When PlanService._execute_phase() is called → When the plan executes
• Multiple behaviors per scenario: split into separate scenarios
• Over-specific data: use meaningful values, not arbitrary UUIDs in step text
• Missing Then: every scenario must have at least one assertion

---

Property-Based Testing (Hypothesis)

Use Hypothesis for verifying invariants and boundary conditions.

When to use Hypothesis

Use case	Example
Invariants that must hold for any valid input	Decision tree structural invariants
Boundary conditions	Confidence score clamp to 0.0–1.0
Merge correctness	No data loss for any combination of file edits
Ordering/sorting	ULID lexicographic monotonicity
Precedence rules	Invariant resolution always follows plan > action > project > global

Recommended strategies to build

from hypothesis import given, strategies as st
from hypothesis.strategies import composite

@composite
def ulid_strategy(draw):
    """Strategy that generates valid ULID strings."""
    ...

@composite
def confidence_score_strategy(draw):
    """Strategy for valid confidence scores (0.0–1.0 inclusive)."""
    return draw(st.floats(min_value=0.0, max_value=1.0, allow_nan=False))

@composite
def invariant_set_strategy(draw):
    """Strategy for sets of invariants at multiple scopes."""
    ...

Integration with Behave

Hypothesis tests run inside Behave step definitions:

from hypothesis import given, settings
import hypothesis.strategies as st

@then("the confidence score is always clamped to the valid range")
def step_confidence_clamped(context):
    @given(raw=st.floats(allow_nan=True, allow_infinity=True))
    @settings(max_examples=200)
    def check_clamp(raw):
        clamped = clamp_confidence(raw)
        assert 0.0 <= clamped <= 1.0

    check_clamp()

---

LangChain/LangGraph Testing

• Use FakeListLLM or a custom mock provider — never real LLM API in unit tests
• Test each graph node's state transformation in isolation
• Verify complete workflow execution with expected state transitions
• Streaming: test both event emission AND final results
• Memory: verify conversation history and entity tracking

---

Test-First Development (mandatory)

1. Write the test before the implementation — no exceptions
2. Tests go in the same commit as the implementation they cover
3. Never commit implementation without its tests
4. Test failure during development → immediately becomes a blocking task
   • Exception: @tdd_expected_fail tests are tracked failures, not unaddressed ones