Files
cleveragents-core/features/context_strategy_registry.feature
freemo af9db5ea28
CI / lint (pull_request) Successful in 32s
CI / quality (pull_request) Successful in 32s
CI / typecheck (pull_request) Successful in 46s
CI / build (pull_request) Successful in 31s
CI / helm (pull_request) Successful in 35s
CI / security (pull_request) Successful in 1m19s
CI / unit_tests (pull_request) Failing after 6m45s
CI / docker (pull_request) Has been skipped
CI / e2e_tests (pull_request) Successful in 15m36s
CI / coverage (pull_request) Successful in 10m43s
CI / integration_tests (pull_request) Failing after 23m29s
CI / status-check (pull_request) Failing after 3s
CI / benchmark-publish (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Successful in 56m58s
fix(acms): implement real retrieval logic in all 6 spec-required context strategies
2026-04-05 20:56:32 +00:00

590 lines
25 KiB
Gherkin

@phase2 @acms @strategy
Feature: Context Strategy Registry
As a CleverAgents developer
I want to register and configure context strategies
So that the ACMS pipeline can select and execute them
# ---------------------------------------------------------------------------
# Strategy protocol conformance
# ---------------------------------------------------------------------------
@strategy_protocol
Scenario: All built-in strategies satisfy the ContextStrategy protocol
Given all built-in strategies are instantiated
Then every strategy should satisfy the ContextStrategy protocol
And every strategy should have a non-empty name
And every strategy should have a capabilities object
@strategy_protocol
Scenario: Built-in strategy quality scores match specification
Given all built-in strategies are instantiated
Then the strategy "simple-keyword" should have quality score 0.3
And the strategy "semantic-embedding" should have quality score 0.6
And the strategy "breadth-depth-navigator" should have quality score 0.85
And the strategy "arce" should have quality score 0.95
And the strategy "temporal-archaeology" should have quality score 0.5
And the strategy "plan-decision-context" should have quality score 0.7
# ---------------------------------------------------------------------------
# Registry registration
# ---------------------------------------------------------------------------
@strategy_registry
Scenario: Register a single strategy
Given an empty strategy registry
When I register the "simple-keyword" built-in strategy
Then the strategy registry should contain "simple-keyword"
And the registry size should be 1
@strategy_registry
Scenario: Register all built-in strategies
Given an empty strategy registry
When I register all 6 built-in strategies
Then the registry should contain 6 strategies
And the strategy registry should contain "simple-keyword"
And the strategy registry should contain "semantic-embedding"
And the strategy registry should contain "breadth-depth-navigator"
And the strategy registry should contain "arce"
And the strategy registry should contain "temporal-archaeology"
And the strategy registry should contain "plan-decision-context"
@strategy_registry @error_handling
Scenario: Reject duplicate strategy registration
Given an empty strategy registry
And I register the "simple-keyword" built-in strategy
When I attempt to register "simple-keyword" again
Then a StrategyRegistrationError should be raised
@strategy_registry @error_handling
Scenario: Reject strategy with empty name
Given an empty strategy registry
When I attempt to register a strategy with an empty name
Then a StrategyRegistrationError should be raised
# ---------------------------------------------------------------------------
# Registry query
# ---------------------------------------------------------------------------
@strategy_query
Scenario: Get a registered strategy by name
Given a registry with all built-in strategies
When I get the strategy "simple-keyword"
Then the strategy name should be "simple-keyword"
@strategy_query @error_handling
Scenario: Get a non-existent strategy raises error
Given a registry with all built-in strategies
When I attempt to get the strategy "nonexistent"
Then a StrategyNotFoundError should be raised
@strategy_query
Scenario: List all registered strategy names
Given a registry with all built-in strategies
Then the registry should list 6 strategies
And the list should be sorted alphabetically
# ---------------------------------------------------------------------------
# Enable / disable
# ---------------------------------------------------------------------------
@strategy_config
Scenario: Default enabled list matches spec
Given a registry with all built-in strategies using default enabled list
Then the enabled strategies should be "simple-keyword", "semantic-embedding", "breadth-depth-navigator"
@strategy_config
Scenario: Override enabled list per project
Given a registry with all built-in strategies using default enabled list
When I set the enabled list to "simple-keyword", "arce"
Then the enabled strategies should be "simple-keyword", "arce"
@strategy_config
Scenario: Set enabled list deduplicates duplicate names
Given a registry with all built-in strategies using default enabled list
When I set the enabled list to "simple-keyword", "arce", "simple-keyword"
Then the enabled strategies should be "simple-keyword", "arce"
@strategy_config @error_handling
Scenario: Set enabled list with unknown strategy raises error
Given a registry with all built-in strategies using default enabled list
When I attempt to set the enabled list to "nonexistent"
Then a StrategyNotFoundError should be raised
@strategy_config
Scenario: Disabled strategy excluded from enabled list
Given a registry with all built-in strategies using default enabled list
When I disable the strategy "semantic-embedding"
Then the enabled strategies should not include "semantic-embedding"
# ---------------------------------------------------------------------------
# Per-strategy configuration
# ---------------------------------------------------------------------------
@strategy_config
Scenario: Default strategy config values
Given a registry with all built-in strategies
When I get the config for "simple-keyword"
Then the timeout_seconds should be 30
And the max_fragments should be 100
And the circuit_breaker_threshold should be 3
@strategy_config
Scenario: Update per-strategy timeout
Given a registry with all built-in strategies
When I update config for "simple-keyword" with timeout_seconds 10
Then the config for "simple-keyword" should have timeout_seconds 10
# ---------------------------------------------------------------------------
# can_handle with backends
# ---------------------------------------------------------------------------
@strategy_can_handle
Scenario: simple-keyword returns quality score with text backend
Given a BackendSet with text backend only
And a default ContextRequest
Then "simple-keyword" can_handle should return 0.3
@strategy_can_handle
Scenario: simple-keyword returns 0 without text backend
Given a BackendSet with no backends
And a default ContextRequest
Then "simple-keyword" can_handle should return 0.0
@strategy_can_handle
Scenario: semantic-embedding returns quality score with vector backend
Given a BackendSet with vector backend only
And a default ContextRequest
Then "semantic-embedding" can_handle should return 0.6
@strategy_can_handle
Scenario: arce requires all backends
Given a BackendSet with all backends
And a default ContextRequest
Then "arce" can_handle should return 0.95
@strategy_can_handle
Scenario: arce returns 0 when missing a backend
Given a BackendSet with text backend only
And a default ContextRequest
Then "arce" can_handle should return 0.0
@strategy_can_handle
Scenario: plan-decision-context returns quality score with temporal backend
Given a BackendSet with temporal backend only
And a default ContextRequest
Then "plan-decision-context" can_handle should return 0.7
# ---------------------------------------------------------------------------
# Strategy assemble — empty backends return empty fragment lists
# ---------------------------------------------------------------------------
@strategy_assemble
Scenario: Strategies return empty fragment lists when backends return no data
Given all built-in strategies are instantiated
And a BackendSet with all backends
And a default ContextRequest
And a default PlanContext
When each strategy assembles with budget 1000
Then every strategy should return an empty fragment list
# ---------------------------------------------------------------------------
# Strategy assemble — populated backends return non-empty fragment lists
# ---------------------------------------------------------------------------
@strategy_assemble @real_retrieval
Scenario: SimpleKeywordStrategy returns fragments from text backend
Given all built-in strategies are instantiated
And a BackendSet with a populated text backend
And a ContextRequest with query "authentication"
And a default PlanContext
When the "simple-keyword" strategy assembles with budget 10000
Then the "simple-keyword" strategy should return at least 1 fragment
@strategy_assemble @real_retrieval
Scenario: SemanticEmbeddingStrategy returns fragments from vector backend
Given all built-in strategies are instantiated
And a BackendSet with a populated vector backend
And a ContextRequest with query "authentication"
And a default PlanContext
When the "semantic-embedding" strategy assembles with budget 10000
Then the "semantic-embedding" strategy should return at least 1 fragment
@strategy_assemble @real_retrieval
Scenario: BreadthDepthNavigatorStrategy returns fragments from graph backend
Given all built-in strategies are instantiated
And a BackendSet with a populated graph backend
And a ContextRequest with focus "uko:class/AuthManager"
And a default PlanContext
When the "breadth-depth-navigator" strategy assembles with budget 10000
Then the "breadth-depth-navigator" strategy should return at least 1 fragment
@strategy_assemble @real_retrieval
Scenario: ARCEStrategy returns fragments from all backends
Given all built-in strategies are instantiated
And a BackendSet with all populated backends
And a ContextRequest with query "authentication"
And a default PlanContext
When the "arce" strategy assembles with budget 10000
Then the "arce" strategy should return at least 1 fragment
@strategy_assemble @real_retrieval
Scenario: TemporalArchaeologyStrategy returns fragments from temporal backend
Given all built-in strategies are instantiated
And a BackendSet with populated graph and temporal backends
And a default ContextRequest
And a default PlanContext
When the "temporal-archaeology" strategy assembles with budget 10000
Then the "temporal-archaeology" strategy should return at least 1 fragment
@strategy_assemble @real_retrieval
Scenario: PlanDecisionContextStrategy returns fragments from temporal backend
Given all built-in strategies are instantiated
And a BackendSet with a populated temporal backend
And a default ContextRequest
And a default PlanContext
When the "plan-decision-context" strategy assembles with budget 10000
Then the "plan-decision-context" strategy should return at least 1 fragment
# ---------------------------------------------------------------------------
# ContextStrategyResult deterministic ordering
# ---------------------------------------------------------------------------
@strategy_result
Scenario: ContextStrategyResult sorts fragments deterministically
Given a ContextStrategyResult with unordered fragments
Then the fragments should be sorted by relevance_score DESC then uko_node ASC
@strategy_result
Scenario: ContextStrategyResult with no fragments
Given a ContextStrategyResult with no fragments
Then the total_fragments should be 0
And the tokens_used should be 0
# ---------------------------------------------------------------------------
# Registry validation
# ---------------------------------------------------------------------------
@strategy_validation
Scenario: Validate registry warns about missing resource types
Given a registry with a strategy that declares no resource types
When I validate the registry
Then validation warnings should mention "resource types"
@strategy_validation
Scenario: Validate registry warns about missing capabilities
Given a registry with a strategy that declares no backend capabilities
When I validate the registry
Then validation warnings should mention "backend capabilities"
@strategy_validation
Scenario: Valid registry returns no warnings for built-in strategies
Given a registry with all built-in strategies
When I validate the registry
Then there should be no validation warnings
# ---------------------------------------------------------------------------
# Unregister
# ---------------------------------------------------------------------------
# ---------------------------------------------------------------------------
# Plugin discovery (register_from_module)
# ---------------------------------------------------------------------------
@strategy_registry @plugin
Scenario: Register a strategy from a module path
Given an empty strategy registry
When I register a strategy from module "cleveragents.domain.models.acms.strategy_stubs:SimpleKeywordStrategy"
Then the strategy registry should contain "simple-keyword"
And the registry size should be 1
@strategy_registry @plugin @error_handling
Scenario: Reject module path without colon separator
Given an empty strategy registry
When I attempt to register a strategy from module "no_colon_here"
Then a StrategyRegistrationError should be raised
@strategy_registry @plugin @error_handling
Scenario: Reject module path with missing module
Given an empty strategy registry
When I attempt to register a strategy from module "nonexistent.module:FakeClass"
Then a StrategyRegistrationError should be raised
@strategy_registry @plugin @error_handling
Scenario: Reject module path with missing class
Given an empty strategy registry
When I attempt to register a strategy from module "cleveragents.domain.models.acms.strategy_stubs:NonExistentClass"
Then a StrategyRegistrationError should be raised
@strategy_registry @plugin @error_handling
Scenario: Reject module path when instantiation fails
Given an empty strategy registry
When I attempt to register a strategy from module "cleveragents.domain.models.acms.strategy_stubs:BUILTIN_STRATEGY_CLASSES"
Then a StrategyRegistrationError should be raised
# ---------------------------------------------------------------------------
# Additional query coverage
# ---------------------------------------------------------------------------
@strategy_query
Scenario: Get entry for a registered strategy
Given a registry with all built-in strategies
When I get the entry for "simple-keyword"
Then the entry name should be "simple-keyword"
And the entry should be marked as builtin
@strategy_query @error_handling
Scenario: Get entry for non-existent strategy raises error
Given a registry with all built-in strategies
When I attempt to get the entry for "nonexistent"
Then a StrategyNotFoundError should be raised
@strategy_query
Scenario: List built-in strategies
Given a registry with all built-in strategies
Then the builtin list should contain 6 strategies
@strategy_query
Scenario: Registry supports the in operator
Given a registry with all built-in strategies
Then "simple-keyword" should be in the registry via contains
And "nonexistent" should not be in the registry via contains
# ---------------------------------------------------------------------------
# Validation edge case: enabled-but-unregistered
# ---------------------------------------------------------------------------
@strategy_validation
Scenario: Validate warns when enabled list references unregistered strategy
Given a registry with a stale enabled list entry
When I validate the registry
Then validation warnings should mention "not registered"
# ---------------------------------------------------------------------------
# Unregister error path
# ---------------------------------------------------------------------------
@strategy_registry @error_handling
Scenario: Unregister non-existent strategy raises error
Given a registry with all built-in strategies
When I attempt to unregister "nonexistent"
Then a StrategyNotFoundError should be raised
# ---------------------------------------------------------------------------
# Unregister
# ---------------------------------------------------------------------------
@strategy_registry
Scenario: Unregister a strategy
Given a registry with all built-in strategies
When I unregister "temporal-archaeology"
Then the registry should contain 5 strategies
And the strategy registry should not contain "temporal-archaeology"
@strategy_registry
Scenario: Clear the registry
Given a registry with all built-in strategies
When I clear the registry
Then the registry should contain 0 strategies
# ---------------------------------------------------------------------------
# Boundary tests for Pydantic-validated config fields
# ---------------------------------------------------------------------------
@strategy_config @boundary
Scenario Outline: Reject invalid StrategyConfig numeric values
When I attempt to create a StrategyConfig with <field> set to <value>
Then a Pydantic ValidationError should be raised
Examples:
| field | value |
| timeout_seconds | 0 |
| timeout_seconds | -1 |
| max_fragments | 0 |
| max_fragments | -1 |
| max_workers | 0 |
| circuit_breaker_threshold | 0 |
@strategy_config @boundary
Scenario Outline: Reject invalid quality_score values
When I attempt to create a StrategyCapabilities with quality_score <value>
Then a Pydantic ValidationError should be raised
Examples:
| value |
| -0.1 |
| 1.1 |
@strategy_config @boundary
Scenario: Accept quality_score at lower bound 0.0
When I create a StrategyCapabilities with quality_score 0.0
Then the quality_score should be 0.0
@strategy_config @boundary
Scenario: Accept quality_score at upper bound 1.0
When I create a StrategyCapabilities with quality_score 1.0
Then the quality_score should be 1.0
# ---------------------------------------------------------------------------
# Bug regression tests
# ---------------------------------------------------------------------------
@strategy_config @regression
Scenario: update_config enabled=True adds strategy to enabled list
Given an empty strategy registry
When I register the "simple-keyword" built-in strategy with enabled False
Then the enabled strategies should be ""
When I update config for "simple-keyword" with enabled True
Then the enabled strategies should include "simple-keyword"
@strategy_config @regression
Scenario: update_config enabled=False removes strategy from enabled list
Given a registry with all built-in strategies using default enabled list
When I update config for "simple-keyword" with enabled False
Then the enabled strategies should not include "simple-keyword"
@strategy_registry @plugin @regression
Scenario: register_from_module uses the provided name parameter
Given an empty strategy registry
When I register a strategy named "custom-search" from module "cleveragents.domain.models.acms.strategy_stubs:SimpleKeywordStrategy"
Then the strategy registry should contain "custom-search"
And the registry size should be 1
@strategy_config @regression
Scenario: update_config with max_workers and circuit_breaker_threshold
Given a registry with all built-in strategies
When I update config for "simple-keyword" with max_workers 8 and circuit_breaker_threshold 5
Then the config for "simple-keyword" should have max_workers 8
And the config for "simple-keyword" should have circuit_breaker_threshold 5
@strategy_registry @error_handling
Scenario: Reject object that does not satisfy ContextStrategy protocol
Given an empty strategy registry
When I attempt to register a non-protocol object
Then a StrategyRegistrationError should be raised
@strategy_registry @error_handling
Scenario: Reject non-protocol object without explicit name
Given an empty strategy registry
When I attempt to register a non-protocol object without a name
Then a StrategyRegistrationError should be raised
And the error message should contain "does not satisfy"
# ---------------------------------------------------------------------------
# explain() coverage
# ---------------------------------------------------------------------------
@strategy_protocol
Scenario: Every built-in strategy returns a non-empty explain string
Given all built-in strategies are instantiated
Then every strategy should return a non-empty explain string
# ---------------------------------------------------------------------------
# can_handle coverage for strategies not tested elsewhere
# ---------------------------------------------------------------------------
@strategy_can_handle
Scenario: breadth-depth-navigator returns quality score with graph backend
Given a BackendSet with graph backend only
And a default ContextRequest
Then "breadth-depth-navigator" can_handle should return 0.85
@strategy_can_handle
Scenario: breadth-depth-navigator returns 0 without graph backend
Given a BackendSet with no backends
And a default ContextRequest
Then "breadth-depth-navigator" can_handle should return 0.0
@strategy_can_handle
Scenario: temporal-archaeology returns quality score with graph and temporal backends
Given a BackendSet with graph and temporal backends
And a default ContextRequest
Then "temporal-archaeology" can_handle should return 0.5
@strategy_can_handle
Scenario: temporal-archaeology returns 0 without graph backend
Given a BackendSet with no backends
And a default ContextRequest
Then "temporal-archaeology" can_handle should return 0.0
@strategy_can_handle
Scenario: semantic-embedding returns 0 without vector backend
Given a BackendSet with no backends
And a default ContextRequest
Then "semantic-embedding" can_handle should return 0.0
@strategy_config @regression
Scenario: update_config rejects invalid timeout via Pydantic validation
Given a registry with all built-in strategies
When I attempt to update config for "simple-keyword" with timeout_seconds 0
Then a Pydantic ValidationError should be raised
@strategy_config @regression
Scenario: update_config rejects negative max_fragments via Pydantic validation
Given a registry with all built-in strategies
When I attempt to update config for "simple-keyword" with max_fragments -1
Then a Pydantic ValidationError should be raised
# ---------------------------------------------------------------------------
# update_config: resource_types and extra parameters
# ---------------------------------------------------------------------------
@strategy_config @regression
Scenario: update_config with resource_types
Given a registry with all built-in strategies
When I update config for "simple-keyword" with resource_types "code", "doc"
Then the config for "simple-keyword" should have resource_types "code", "doc"
@strategy_config @regression
Scenario: update_config with extra dict
Given a registry with all built-in strategies
When I update config for "simple-keyword" with extra key "boost" value "2"
Then the config for "simple-keyword" extra should contain key "boost"
And the config for "simple-keyword" extra should be immutable
# ---------------------------------------------------------------------------
# MappingProxyType coercion validators
# ---------------------------------------------------------------------------
@strategy_config @boundary
Scenario: StrategyConfig accepts plain dict for extra field
When I create a StrategyConfig with extra as a plain dict
Then the extra field should be a MappingProxyType
@strategy_config @boundary
Scenario: StrategyConfig rejects non-dict non-MappingProxyType extra
When I attempt to create a StrategyConfig with extra as an integer
Then a Pydantic ValidationError should be raised
@strategy_result @boundary
Scenario: ContextStrategyResult accepts plain dict for stats field
When I create a ContextStrategyResult with stats as a plain dict
Then the stats field should be a MappingProxyType
# ---------------------------------------------------------------------------
# Thread safety
# ---------------------------------------------------------------------------
@strategy_registry @thread_safety
Scenario: Concurrent register and list_enabled do not raise
Given a registry with all built-in strategies
When 10 threads concurrently register and query the registry
Then no thread should have raised an exception
And the registry should contain at least 10 strategies
And every thread-registered strategy should be present
# ---------------------------------------------------------------------------
# temporal/cold-tier can_handle regression
# ---------------------------------------------------------------------------
@strategy_can_handle @regression
Scenario: temporal-archaeology returns 0 without temporal backend
Given a BackendSet with graph backend only
And a default ContextRequest
Then "temporal-archaeology" can_handle should return 0.0
@strategy_can_handle @regression
Scenario: plan-decision-context returns 0 without temporal backend
Given a BackendSet with no backends
And a default ContextRequest
Then "plan-decision-context" can_handle should return 0.0