diff --git a/CHANGELOG.md b/CHANGELOG.md index 08308763b..02dee0385 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -164,6 +164,10 @@ ensuring data is stored with proper parameter values. LLM streaming output. A `SessionActorNotConfiguredError` is raised with exit code 1 when no actor is configured. +### Added + +- **Test Architecture documentation** (#9049): Created `docs/development/TEST_ARCHITECTURE.md` with clear guidelines defining the roles and responsibilities of Behave (business-facing BDD/Gherkin tests) versus Robot Framework (technical integration/E2E tests). Includes decision criteria for framework selection, known test duplication areas to investigate for consolidation, and audit recommendations. + ### Fixed - **fileConfig error handling in alembic env.py** (#7874): Wrapped the `fileConfig()` diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index ea5b2a3b1..870b7a7be 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -64,3 +64,4 @@ Below are some of the specific details of various contributions. * HAL 9000 has contributed the Definition-of-Done gating feature for the Apply phase (PR #8299 / issue #7927): `PlanLifecycleService.apply_plan` now evaluates DoD criteria before transitioning to Apply, raising `DoDGatingError` when required criteria fail and storing evaluation results in `plan.validation_summary`. * HAL 9000 has contributed the engine cache TOCTOU race condition fix (PR #8265 / issue #7566): added `MEMORY_ENGINES_LOCK` to `engine_cache.py` and wrapped the check-and-set operation in `UnitOfWork.engine` with `with MEMORY_ENGINES_LOCK:` to prevent concurrent threads from creating duplicate in-memory SQLite engine instances; also fixed a cache-hit bug where `self._engine` was never assigned on a cache hit. * HAL 9000 has contributed the plan correct JSON output envelope fix (PR #8662 / issue #8584): restructured `agents plan correct --format json` output to nest correction fields under `data.correction` and pass `command="plan correct"` to `format_output`, producing the spec-required CLI envelope. Added three BDD scenarios validating `data.correction.mode` (revert and append modes) and the `command` field. +* HAL 9000 has contributed the test architecture documentation (PR #9219 / issue #9049): created `docs/development/TEST_ARCHITECTURE.md` defining clear roles and responsibilities for Behave vs Robot Framework in the test suite, with decision criteria for framework selection and known consolidation opportunities. diff --git a/docs/development/TEST_ARCHITECTURE.md b/docs/development/TEST_ARCHITECTURE.md new file mode 100644 index 000000000..73b54af24 --- /dev/null +++ b/docs/development/TEST_ARCHITECTURE.md @@ -0,0 +1,149 @@ +# Test Architecture + +**Issue:** #9049 + +This document defines the roles and responsibilities of each testing framework in the CleverAgents test suite, +establishes clear guidelines for selecting the appropriate framework when writing new tests, and documents +known areas where test logic is duplicated across frameworks. + +## Frameworks Overview + +CleverAgents uses two complementary BDD/testing frameworks: + +| Framework | Purpose | Typical Test Level | Audience | Language | +|-----------|---------|-------------------|----------|----------| +| **Behave** | Business-facing behavior verification (BDD/Gherkin) | Unit and integration | Technical + non-technical | Gherkin `.feature` files with Python step definitions | +| **Robot Framework** | Technical integration and end-to-end testing | Integration and E2E | Primarily technical | Robot `.robot` files with Python/Robot keywords | + +## Behave: Business-Facing BDD Tests + +Use Behave when writing tests that describe the system from a user or customer perspective. + +### When to Use Behave + +- The test scenario can be expressed as **Given / When / Then** steps readable by non-developers +- You're testing **business logic rules**, validation, or domain constraints +- Stakeholders need to review or modify test scenarios without understanding implementation details +- You're testing CLI command behavior through a step-mapping abstration +- Coverage goals require high-scenario-count verification of edge cases and boundary conditions + +### Behave Convention + +```gherkin +Feature: Example feature name + As a [role] + I want [behavior/goal] + So that [benefit] + + @example_tag + Scenario: Specific behavior description + Given the system is in state X + When the user performs action Y + Then the result should be Z +``` + +**Location:** `features/*.feature` +**Step definitions:** `features/steps/*_steps.py` +**Run command:** `nox -s unit_tests -- features/.feature` + +### Common Behave Testing Patterns + +- Fixture loading and validation (JSON/YAML fixtures in `features/fixtures/`) +- Repository persistence tests with in-memory SQLite databases +- Domain model lifecycle and transition tests +- CLI command argument parsing, validation, and output rendering +- Mocked service integration scenarios +- Validation edge cases and boundary conditions + +## Robot Framework: Technical Integration and E2E Tests + +Use Robot Framework when writing tests that verify technical behavior, system interactions, and integrated workflows. + +### When to Use Robot Framework + +- The test requires **process execution**, subprocess commands, or shell operations +- You're testing **end-to-end workflows** across multiple services or components +- The test involves actual file system operations, database connections, or network calls +- You need structured logging for debugging complex multi-step procedures +- You're testing CLI tools as they run in production-like conditions + +### Robot Framework Convention + +```robot +*** Test Cases *** +Example Test Case Name + [Documentation] Brief description of what the test verifies + ${result}= ${PYTHON} -m cleveragents + Should Be Equal As Strings ${result.returncode} 0 + Log ${result.stdout} +``` + +**Location:** `robot/*.robot` +**Helper scripts:** `robot/helper_*.py` +**Run command:** `nox -s integration_tests -- robot/.robot` + +### Common Robot Framework Testing Patterns + +- CLI end-to-end execution with actual subprocess invocation +- Tool and skill lifecycle smoke tests +- Integration-level fixture validation via helper Python scripts +- Performance benchmarks (ASV) often paired with Robot execution +- Database persistence and migration testing in real environments +- File system operations, sandboxing, and resource management +- Concurrency and threading behavior under production-like conditions + +## Decision Criteria: Behe vs. Robot Framework + +When deciding which framework to use for a new test, follow this decision tree: + +1. **Can the scenario be described in natural language (Given/When/That) without implementation details?** + - Yes → Use **Behave** + - No → Continue to question 2 + +2. **Does the test require subprocess execution, process spawning, or shell commands?** + - Yes → Use **Robot Framework** + - No → Continue to question 3 + +3. **Are you testing domain logic, validation rules, or business constraints?** + - Yes → Use **Behave** + - No → Continue to question 4 + +4. **Is the test an end-to-end workflow involving multiple system components?** + - Yes → Use **Robot Framework** + - No → Use **Robot Framework** for infrastructure/service tests, **Behave** for pure unit tests + +5. **Are you writing performance benchmarks?** + - Both can be used: **ASV (via Python)** for baseline measurements, paired with **Robot** for integration validation + +### Summary Rule of Thumb + +- **Business-facing, behavior-driven, stakeholder-readable** → Behave +- **Technical, integration-focused, subprocess-heavy, E2E workflows** → Robot Framework + +## Known Test Duplications and Consolidation Candidates + +The following areas have overlapping test coverage across both frameworks and should be considered for consolidation: + +| Area | Behave Location | Robot Location | Recommendation | +|------|-----------------|---------------|----------------| +| Actor Compiler | `features/actor_cli_yaml.feature` / `*_steps.py` | `robot/actor_compiler.robot` / `helper_actor_compiler.py` | Consolidate into Behave for unit, keep Robot for E2E validation only | +| Provider Registry | `features/provider_registry_coverage.feature` | Not applicable | No duplication — Behave only | +| Session CLI | `features/session_cli.feature` / `*_steps.py` | `robot/session_commands.robot` | Consolidate session commands into one framework per feature area | +| Actor Subgraph Cycle Detection | `features/actor_subgraph_cycle_detection.feature` / `*_steps.py` | `robot/actor_compiler.robot` cycle detection tests | Deduplicate — Robot should not repeat compiler unit tests | +| Git Tools Concurrency | `features/git_tools.feature` / `git_tools_thread_safety_steps.py` | None | No duplication | +| Provider Registry Coverage Boost | `features/provider_registry_coverage_boost.feature` | None | No duplication | +| Actor Remove CLI | Not applicable | `robot/actor_remove_cli.robot` / `helper_actor_remove_cli.py` | Robot only — this is an E2E CLI smoke test | + +## Audit Recommendations + +1. **Quarterly Duplication Review**: Run both frameworks side by side and compare scenario lists to identify newly overlapping coverage +2. **New Test Checklist**: Before creating a new feature file or robot suite, check for existing coverage in the alternative framework +3. **Framework Rationale Documentation**: Each new test suite should include a comment at the top explaining why its chosen framework was selected over the alternative +4. **Coverage Monitoring**: Ensure that neither framework's deletion causes the total project coverage to fall below the 97% threshold + +## Related Documents + +- [CI/CD Pipeline](ci-cd.md) — Test execution in CI +- [Quality Automation](quality-automation.md) — Linting, type-checking, and security scanning +- [Test Documentation](testing.md) — Detailed per-module test suite documentation +- [CI/CD Guide](../development/ci-cd.md) — Full CI/CD pipeline configuration