refactor(tests): clarify roles of behave and robot framework in test architecture
CI / benchmark-publish (pull_request) Has been skipped
CI / push-validation (pull_request) Successful in 30s
CI / helm (pull_request) Successful in 32s
CI / build (pull_request) Successful in 54s
CI / lint (pull_request) Successful in 1m23s
CI / quality (pull_request) Successful in 1m24s
CI / typecheck (pull_request) Successful in 1m48s
CI / security (pull_request) Successful in 1m49s
CI / integration_tests (pull_request) Successful in 4m28s
CI / e2e_tests (pull_request) Successful in 5m6s
CI / unit_tests (pull_request) Successful in 6m39s
CI / docker (pull_request) Successful in 1m38s
CI / coverage (pull_request) Successful in 12m34s
CI / status-check (pull_request) Successful in 6s
CI / benchmark-publish (pull_request) Has been skipped
CI / push-validation (pull_request) Successful in 30s
CI / helm (pull_request) Successful in 32s
CI / build (pull_request) Successful in 54s
CI / lint (pull_request) Successful in 1m23s
CI / quality (pull_request) Successful in 1m24s
CI / typecheck (pull_request) Successful in 1m48s
CI / security (pull_request) Successful in 1m49s
CI / integration_tests (pull_request) Successful in 4m28s
CI / e2e_tests (pull_request) Successful in 5m6s
CI / unit_tests (pull_request) Successful in 6m39s
CI / docker (pull_request) Successful in 1m38s
CI / coverage (pull_request) Successful in 12m34s
CI / status-check (pull_request) Successful in 6s
Added comprehensive documentation in docs/development/TEST_ARCHITECTURE.md that clarifies: - Behave (BDD/Gherkin) is for unit-level and scenario tests - Robot Framework is for integration and end-to-end tests - Clear guidelines on when to use each framework - Audit and consolidation recommendations - Known duplications to address (e.g., a2a_facade tests) This documentation serves as the authoritative source for test framework roles and helps contributors choose the appropriate framework for new tests, reducing duplication and improving test suite maintainability. Refs: #9049
This commit is contained in:
@@ -23,6 +23,8 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
|
||||
### Added
|
||||
|
||||
- **Test Architecture documentation** (#9049): Added `docs/development/TEST_ARCHITECTURE.md` providing clear guidelines distinguishing Behave (BDD/Gherkin) unit-level tests from Robot Framework integration and end-to-end tests. Includes decision criteria for framework selection, known duplication pairs (`a2a_facade`, etc.), and audit recommendations. Reduces developer confusion about which framework to use for new tests.
|
||||
|
||||
- **TDD: MCPToolAdapter.infer_resource_slots() TypeError with null properties** (#10470):
|
||||
Added a TDD issue-capture Behave scenario that reproduces the bug where
|
||||
`MCPToolAdapter.infer_resource_slots()` raises `TypeError` when the input schema
|
||||
|
||||
@@ -14,6 +14,7 @@ Below are some of the specific details of various contributions.
|
||||
|
||||
* Jeffrey Phillips Freeman has acted as Lead Developer, daily contributor, and Project Owner.
|
||||
* Brent E. Edwards has contributed quality assurance, test coverage, and CI pipeline improvements.
|
||||
* HAL 9000 has contributed test architecture documentation (PR #9219), clarifying the distinction between Behave BDD unit tests and Robot Framework integration/E2E tests in docs/development/TEST_ARCHITECTURE.md.
|
||||
* HAL 9000 has contributed automated implementation, bug fixes, and feature development as part of the CleverAgents automation pool.
|
||||
* HAL 9000 has contributed concurrency safety improvements, including thread-safe context tier management (issue #7547) for parallel plan execution.
|
||||
* HAL 9000 has contributed the plan concurrency race-condition fix (#7989): wired `LockService` into the plan lifecycle, guarding `execute_plan()` and `apply_plan()` with plan-level advisory locks and unique per-invocation owner identities to prevent silent concurrent state corruption.
|
||||
|
||||
@@ -0,0 +1,141 @@
|
||||
# Test Architecture: Behave vs Robot Framework
|
||||
|
||||
## Overview
|
||||
|
||||
CleverAgents uses two complementary testing frameworks with distinct, non-overlapping roles:
|
||||
|
||||
- **Behave (BDD/Gherkin)** for unit-level and scenario tests
|
||||
- **Robot Framework** for integration and end-to-end tests
|
||||
|
||||
## Behave (BDD/Gherkin) — Unit-Level Tests
|
||||
|
||||
**Purpose:** Express business-facing behavior and unit-level logic in human-readable Gherkin syntax.
|
||||
|
||||
**Location:** `features/` directory
|
||||
|
||||
**When to use:**
|
||||
- Testing individual components, services, or domain models in isolation
|
||||
- Expressing acceptance criteria as executable scenarios
|
||||
- Validating business logic and state transitions
|
||||
- Unit-level coverage of algorithms, calculations, and data transformations
|
||||
- Any test that can be expressed as "Given/When/Then" behavior
|
||||
|
||||
**Examples:**
|
||||
- Testing plan lifecycle transitions
|
||||
- Validating action persistence and retrieval
|
||||
- Testing validation logic and error handling
|
||||
- Verifying business rule enforcement
|
||||
|
||||
**Coverage:** Behave tests are measured for code coverage and must maintain ≥97% coverage.
|
||||
|
||||
**Key Characteristics:**
|
||||
- Tests run in-process without spawning subprocesses
|
||||
- Mock internal services and dependencies
|
||||
- Focus on component behavior, not system integration
|
||||
- Gherkin syntax makes tests self-documenting
|
||||
- Fast execution (no subprocess overhead)
|
||||
|
||||
## Robot Framework — Integration and End-to-End Tests
|
||||
|
||||
**Purpose:** Exercise the system as a whole through its public interfaces (CLI, APIs, services).
|
||||
|
||||
**Location:** `robot/` directory
|
||||
|
||||
**When to use:**
|
||||
- Testing full workflows across multiple components
|
||||
- Validating CLI command behavior and output formatting
|
||||
- Integration testing with real services and databases
|
||||
- End-to-end scenarios that require subprocess execution
|
||||
- Testing system-level invariants and cross-component interactions
|
||||
- Any test that requires spawning processes or exercising the CLI
|
||||
|
||||
**Examples:**
|
||||
- Testing CLI lifecycle workflows (action create → plan use → execute → apply)
|
||||
- Validating database persistence across process restarts
|
||||
- Testing multi-step user journeys through the system
|
||||
- Verifying CLI output formatting and error messages
|
||||
- Testing integration with external services
|
||||
|
||||
**Coverage:** Robot tests are not measured for code coverage; they validate system behavior and integration.
|
||||
|
||||
**Key Characteristics:**
|
||||
- Tests spawn subprocesses and invoke the CLI
|
||||
- Exercise real implementations, not mocks
|
||||
- Validate end-to-end workflows
|
||||
- Test system-level behavior and invariants
|
||||
- Slower execution (subprocess overhead)
|
||||
|
||||
## Key Distinction: Do Not Duplicate Tests
|
||||
|
||||
**Critical Rule:** If a scenario is already tested in Behave, do not write the same test in Robot Framework.
|
||||
|
||||
Choose the appropriate framework based on the nature of the test:
|
||||
- **If it tests a component or service in isolation** → Use **Behave**
|
||||
- **If it tests the system end-to-end through public interfaces** → Use **Robot Framework**
|
||||
|
||||
### Avoid Mixing Concerns
|
||||
|
||||
- **Behave tests should NOT:**
|
||||
- Spawn subprocesses or invoke the CLI directly
|
||||
- Test CLI output formatting
|
||||
- Test system-level integration
|
||||
|
||||
- **Robot tests should NOT:**
|
||||
- Mock internal services (use real implementations)
|
||||
- Test individual component logic
|
||||
- Test business logic that should be in Behave
|
||||
|
||||
## Audit and Consolidation
|
||||
|
||||
When auditing the test suite, look for:
|
||||
|
||||
1. **Duplicate test scenarios** across frameworks (same test in both `.feature` and `.robot` files)
|
||||
2. **Misplaced tests** (component tests in Robot, CLI tests in Behave)
|
||||
3. **Unnecessary mocking** in Robot tests (should use real implementations)
|
||||
4. **Subprocess invocations** in Behave tests (should be in Robot)
|
||||
|
||||
If you find duplicates, consolidate them into the appropriate framework and remove the duplicate.
|
||||
|
||||
### Known Duplications
|
||||
|
||||
The following test pairs have been identified as potential duplicates and should be audited:
|
||||
|
||||
- `features/a2a_facade_coverage.feature` vs `robot/a2a_facade.robot`
|
||||
- **Status:** Needs consolidation
|
||||
- **Recommendation:** Keep Robot Framework version (integration test), remove Behave version (or refactor to test specific components)
|
||||
|
||||
## Implementation Guidelines
|
||||
|
||||
### For New Tests
|
||||
|
||||
1. **Determine the scope:** Is this testing a component in isolation or the system end-to-end?
|
||||
2. **Choose the framework:**
|
||||
- Component/unit test → Behave
|
||||
- End-to-end/integration test → Robot Framework
|
||||
3. **Check for duplicates:** Before writing a test, verify it doesn't already exist in the other framework
|
||||
4. **Follow the guidelines:** Ensure your test follows the framework-specific best practices
|
||||
|
||||
### For Existing Tests
|
||||
|
||||
1. **Audit regularly:** Review test suite for duplicates and misplaced tests
|
||||
2. **Consolidate:** Move tests to the appropriate framework
|
||||
3. **Remove duplicates:** Delete the duplicate test after consolidation
|
||||
4. **Maintain coverage:** Ensure coverage remains ≥97% after consolidation
|
||||
|
||||
## References
|
||||
|
||||
- `docs/development/testing.md` - Comprehensive testing guide
|
||||
- `CONTRIBUTING.md` - Contribution guidelines including testing philosophy
|
||||
- `noxfile.py` - Test execution configuration
|
||||
|
||||
## Conclusion
|
||||
|
||||
By maintaining clear separation between Behave (unit/component tests) and Robot Framework (integration/E2E tests), we:
|
||||
|
||||
- Reduce test duplication and maintenance burden
|
||||
- Improve test readability and clarity
|
||||
- Maintain fast unit test execution
|
||||
- Ensure comprehensive system validation
|
||||
- Make the test suite easier to understand and navigate
|
||||
|
||||
Contributors should always ask: "Is this testing a component in isolation, or the system end-to-end?" The answer determines which framework to use.
|
||||
Reference in New Issue
Block a user