diff --git a/docs/development/TEST_ARCHITECTURE.md b/docs/development/TEST_ARCHITECTURE.md new file mode 100644 index 000000000..bd642e40a --- /dev/null +++ b/docs/development/TEST_ARCHITECTURE.md @@ -0,0 +1,141 @@ +# Test Architecture: Behave vs Robot Framework + +## Overview + +CleverAgents uses two complementary testing frameworks with distinct, non-overlapping roles: + +- **Behave (BDD/Gherkin)** for unit-level and scenario tests +- **Robot Framework** for integration and end-to-end tests + +## Behave (BDD/Gherkin) — Unit-Level Tests + +**Purpose:** Express business-facing behavior and unit-level logic in human-readable Gherkin syntax. + +**Location:** `features/` directory + +**When to use:** +- Testing individual components, services, or domain models in isolation +- Expressing acceptance criteria as executable scenarios +- Validating business logic and state transitions +- Unit-level coverage of algorithms, calculations, and data transformations +- Any test that can be expressed as "Given/When/Then" behavior + +**Examples:** +- Testing plan lifecycle transitions +- Validating action persistence and retrieval +- Testing validation logic and error handling +- Verifying business rule enforcement + +**Coverage:** Behave tests are measured for code coverage and must maintain ≥97% coverage. + +**Key Characteristics:** +- Tests run in-process without spawning subprocesses +- Mock internal services and dependencies +- Focus on component behavior, not system integration +- Gherkin syntax makes tests self-documenting +- Fast execution (no subprocess overhead) + +## Robot Framework — Integration and End-to-End Tests + +**Purpose:** Exercise the system as a whole through its public interfaces (CLI, APIs, services). + +**Location:** `robot/` directory + +**When to use:** +- Testing full workflows across multiple components +- Validating CLI command behavior and output formatting +- Integration testing with real services and databases +- End-to-end scenarios that require subprocess execution +- Testing system-level invariants and cross-component interactions +- Any test that requires spawning processes or exercising the CLI + +**Examples:** +- Testing CLI lifecycle workflows (action create → plan use → execute → apply) +- Validating database persistence across process restarts +- Testing multi-step user journeys through the system +- Verifying CLI output formatting and error messages +- Testing integration with external services + +**Coverage:** Robot tests are not measured for code coverage; they validate system behavior and integration. + +**Key Characteristics:** +- Tests spawn subprocesses and invoke the CLI +- Exercise real implementations, not mocks +- Validate end-to-end workflows +- Test system-level behavior and invariants +- Slower execution (subprocess overhead) + +## Key Distinction: Do Not Duplicate Tests + +**Critical Rule:** If a scenario is already tested in Behave, do not write the same test in Robot Framework. + +Choose the appropriate framework based on the nature of the test: +- **If it tests a component or service in isolation** → Use **Behave** +- **If it tests the system end-to-end through public interfaces** → Use **Robot Framework** + +### Avoid Mixing Concerns + +- **Behave tests should NOT:** + - Spawn subprocesses or invoke the CLI directly + - Test CLI output formatting + - Test system-level integration + +- **Robot tests should NOT:** + - Mock internal services (use real implementations) + - Test individual component logic + - Test business logic that should be in Behave + +## Audit and Consolidation + +When auditing the test suite, look for: + +1. **Duplicate test scenarios** across frameworks (same test in both `.feature` and `.robot` files) +2. **Misplaced tests** (component tests in Robot, CLI tests in Behave) +3. **Unnecessary mocking** in Robot tests (should use real implementations) +4. **Subprocess invocations** in Behave tests (should be in Robot) + +If you find duplicates, consolidate them into the appropriate framework and remove the duplicate. + +### Known Duplications + +The following test pairs have been identified as potential duplicates and should be audited: + +- `features/a2a_facade_coverage.feature` vs `robot/a2a_facade.robot` + - **Status:** Needs consolidation + - **Recommendation:** Keep Robot Framework version (integration test), remove Behave version (or refactor to test specific components) + +## Implementation Guidelines + +### For New Tests + +1. **Determine the scope:** Is this testing a component in isolation or the system end-to-end? +2. **Choose the framework:** + - Component/unit test → Behave + - End-to-end/integration test → Robot Framework +3. **Check for duplicates:** Before writing a test, verify it doesn't already exist in the other framework +4. **Follow the guidelines:** Ensure your test follows the framework-specific best practices + +### For Existing Tests + +1. **Audit regularly:** Review test suite for duplicates and misplaced tests +2. **Consolidate:** Move tests to the appropriate framework +3. **Remove duplicates:** Delete the duplicate test after consolidation +4. **Maintain coverage:** Ensure coverage remains ≥97% after consolidation + +## References + +- `docs/development/testing.md` - Comprehensive testing guide +- `CONTRIBUTING.md` - Contribution guidelines including testing philosophy +- `noxfile.py` - Test execution configuration + +## Conclusion + +By maintaining clear separation between Behave (unit/component tests) and Robot Framework (integration/E2E tests), we: + +- Reduce test duplication and maintenance burden +- Improve test readability and clarity +- Maintain fast unit test execution +- Ensure comprehensive system validation +- Make the test suite easier to understand and navigate + +Contributors should always ask: "Is this testing a component in isolation, or the system end-to-end?" The answer determines which framework to use.