refactor(tests): clarify roles of behave and robot framework in test architecture
CI / lint (pull_request) Successful in 35s
CI / typecheck (pull_request) Successful in 50s
CI / quality (pull_request) Successful in 45s
CI / security (pull_request) Successful in 1m4s
CI / build (pull_request) Successful in 32s
CI / helm (pull_request) Successful in 30s
CI / push-validation (pull_request) Successful in 27s
CI / e2e_tests (pull_request) Successful in 3m43s
CI / integration_tests (pull_request) Successful in 6m42s
CI / unit_tests (pull_request) Successful in 7m57s
CI / docker (pull_request) Successful in 17s
CI / coverage (pull_request) Successful in 14m8s
CI / status-check (pull_request) Successful in 1s
CI / lint (pull_request) Successful in 35s
CI / typecheck (pull_request) Successful in 50s
CI / quality (pull_request) Successful in 45s
CI / security (pull_request) Successful in 1m4s
CI / build (pull_request) Successful in 32s
CI / helm (pull_request) Successful in 30s
CI / push-validation (pull_request) Successful in 27s
CI / e2e_tests (pull_request) Successful in 3m43s
CI / integration_tests (pull_request) Successful in 6m42s
CI / unit_tests (pull_request) Successful in 7m57s
CI / docker (pull_request) Successful in 17s
CI / coverage (pull_request) Successful in 14m8s
CI / status-check (pull_request) Successful in 1s
Added comprehensive documentation in docs/development/TEST_ARCHITECTURE.md that clarifies: - Behave (BDD/Gherkin) is for unit-level and scenario tests - Robot Framework is for integration and end-to-end tests - Clear guidelines on when to use each framework - Audit and consolidation recommendations - Known duplications to address (e.g., a2a_facade tests) This documentation serves as the authoritative source for test framework roles and helps contributors choose the appropriate framework for new tests, reducing duplication and improving test suite maintainability. ISSUES CLOSED: #9049
This commit is contained in:
@@ -0,0 +1,141 @@
|
||||
# Test Architecture: Behave vs Robot Framework
|
||||
|
||||
## Overview
|
||||
|
||||
CleverAgents uses two complementary testing frameworks with distinct, non-overlapping roles:
|
||||
|
||||
- **Behave (BDD/Gherkin)** for unit-level and scenario tests
|
||||
- **Robot Framework** for integration and end-to-end tests
|
||||
|
||||
## Behave (BDD/Gherkin) — Unit-Level Tests
|
||||
|
||||
**Purpose:** Express business-facing behavior and unit-level logic in human-readable Gherkin syntax.
|
||||
|
||||
**Location:** `features/` directory
|
||||
|
||||
**When to use:**
|
||||
- Testing individual components, services, or domain models in isolation
|
||||
- Expressing acceptance criteria as executable scenarios
|
||||
- Validating business logic and state transitions
|
||||
- Unit-level coverage of algorithms, calculations, and data transformations
|
||||
- Any test that can be expressed as "Given/When/Then" behavior
|
||||
|
||||
**Examples:**
|
||||
- Testing plan lifecycle transitions
|
||||
- Validating action persistence and retrieval
|
||||
- Testing validation logic and error handling
|
||||
- Verifying business rule enforcement
|
||||
|
||||
**Coverage:** Behave tests are measured for code coverage and must maintain ≥97% coverage.
|
||||
|
||||
**Key Characteristics:**
|
||||
- Tests run in-process without spawning subprocesses
|
||||
- Mock internal services and dependencies
|
||||
- Focus on component behavior, not system integration
|
||||
- Gherkin syntax makes tests self-documenting
|
||||
- Fast execution (no subprocess overhead)
|
||||
|
||||
## Robot Framework — Integration and End-to-End Tests
|
||||
|
||||
**Purpose:** Exercise the system as a whole through its public interfaces (CLI, APIs, services).
|
||||
|
||||
**Location:** `robot/` directory
|
||||
|
||||
**When to use:**
|
||||
- Testing full workflows across multiple components
|
||||
- Validating CLI command behavior and output formatting
|
||||
- Integration testing with real services and databases
|
||||
- End-to-end scenarios that require subprocess execution
|
||||
- Testing system-level invariants and cross-component interactions
|
||||
- Any test that requires spawning processes or exercising the CLI
|
||||
|
||||
**Examples:**
|
||||
- Testing CLI lifecycle workflows (action create → plan use → execute → apply)
|
||||
- Validating database persistence across process restarts
|
||||
- Testing multi-step user journeys through the system
|
||||
- Verifying CLI output formatting and error messages
|
||||
- Testing integration with external services
|
||||
|
||||
**Coverage:** Robot tests are not measured for code coverage; they validate system behavior and integration.
|
||||
|
||||
**Key Characteristics:**
|
||||
- Tests spawn subprocesses and invoke the CLI
|
||||
- Exercise real implementations, not mocks
|
||||
- Validate end-to-end workflows
|
||||
- Test system-level behavior and invariants
|
||||
- Slower execution (subprocess overhead)
|
||||
|
||||
## Key Distinction: Do Not Duplicate Tests
|
||||
|
||||
**Critical Rule:** If a scenario is already tested in Behave, do not write the same test in Robot Framework.
|
||||
|
||||
Choose the appropriate framework based on the nature of the test:
|
||||
- **If it tests a component or service in isolation** → Use **Behave**
|
||||
- **If it tests the system end-to-end through public interfaces** → Use **Robot Framework**
|
||||
|
||||
### Avoid Mixing Concerns
|
||||
|
||||
- **Behave tests should NOT:**
|
||||
- Spawn subprocesses or invoke the CLI directly
|
||||
- Test CLI output formatting
|
||||
- Test system-level integration
|
||||
|
||||
- **Robot tests should NOT:**
|
||||
- Mock internal services (use real implementations)
|
||||
- Test individual component logic
|
||||
- Test business logic that should be in Behave
|
||||
|
||||
## Audit and Consolidation
|
||||
|
||||
When auditing the test suite, look for:
|
||||
|
||||
1. **Duplicate test scenarios** across frameworks (same test in both `.feature` and `.robot` files)
|
||||
2. **Misplaced tests** (component tests in Robot, CLI tests in Behave)
|
||||
3. **Unnecessary mocking** in Robot tests (should use real implementations)
|
||||
4. **Subprocess invocations** in Behave tests (should be in Robot)
|
||||
|
||||
If you find duplicates, consolidate them into the appropriate framework and remove the duplicate.
|
||||
|
||||
### Known Duplications
|
||||
|
||||
The following test pairs have been identified as potential duplicates and should be audited:
|
||||
|
||||
- `features/a2a_facade_coverage.feature` vs `robot/a2a_facade.robot`
|
||||
- **Status:** Needs consolidation
|
||||
- **Recommendation:** Keep Robot Framework version (integration test), remove Behave version (or refactor to test specific components)
|
||||
|
||||
## Implementation Guidelines
|
||||
|
||||
### For New Tests
|
||||
|
||||
1. **Determine the scope:** Is this testing a component in isolation or the system end-to-end?
|
||||
2. **Choose the framework:**
|
||||
- Component/unit test → Behave
|
||||
- End-to-end/integration test → Robot Framework
|
||||
3. **Check for duplicates:** Before writing a test, verify it doesn't already exist in the other framework
|
||||
4. **Follow the guidelines:** Ensure your test follows the framework-specific best practices
|
||||
|
||||
### For Existing Tests
|
||||
|
||||
1. **Audit regularly:** Review test suite for duplicates and misplaced tests
|
||||
2. **Consolidate:** Move tests to the appropriate framework
|
||||
3. **Remove duplicates:** Delete the duplicate test after consolidation
|
||||
4. **Maintain coverage:** Ensure coverage remains ≥97% after consolidation
|
||||
|
||||
## References
|
||||
|
||||
- `docs/development/testing.md` - Comprehensive testing guide
|
||||
- `CONTRIBUTING.md` - Contribution guidelines including testing philosophy
|
||||
- `noxfile.py` - Test execution configuration
|
||||
|
||||
## Conclusion
|
||||
|
||||
By maintaining clear separation between Behave (unit/component tests) and Robot Framework (integration/E2E tests), we:
|
||||
|
||||
- Reduce test duplication and maintenance burden
|
||||
- Improve test readability and clarity
|
||||
- Maintain fast unit test execution
|
||||
- Ensure comprehensive system validation
|
||||
- Make the test suite easier to understand and navigate
|
||||
|
||||
Contributors should always ask: "Is this testing a component in isolation, or the system end-to-end?" The answer determines which framework to use.
|
||||
Reference in New Issue
Block a user