khyari hamza
1521c4ae8c
Implement the ACMS context strategy registry per spec §25162-25233, §28682-28708, §42628-42653, and §43167-43199. - Define ContextStrategy protocol, StrategyCapabilities, BackendSet, PlanContext, StrategyConfig, ContextStrategyResult models - Add 6 built-in stub strategies (simple-keyword, semantic-embedding, breadth-depth-navigator, arce, temporal-archaeology, plan-decision-context) with spec quality scores and feature flags - Add StrategyRegistry with register, register_from_module (plugin discovery), enable/disable, per-strategy config, and validation - Add ContextStrategyResult with deterministic fragment ordering (-relevance_score, uko_node) - Add configuration-driven enabled list with per-project overrides - Add per-strategy timeout, max-fragment, circuit-breaker config - Add registry validation for resource types and backend capabilities - Fix update_config to sync _enabled_order when toggling enabled flag - Fix register_from_module to honour the name parameter as registry key - Fix update_config to re-run Pydantic validators via model_validate - Add docs/reference/context_strategies.md - Add 55 BDD scenarios (Behave), 4 Robot integration tests, ASV benchmarks ISSUES CLOSED: #191
455 lines
19 KiB
Gherkin
455 lines
19 KiB
Gherkin
@phase2 @acms @strategy
|
|
Feature: Context Strategy Registry
|
|
As a CleverAgents developer
|
|
I want to register and configure context strategies
|
|
So that the ACMS pipeline can select and execute them
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Strategy protocol conformance
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@strategy_protocol
|
|
Scenario: All built-in strategies satisfy the ContextStrategy protocol
|
|
Given all built-in strategies are instantiated
|
|
Then every strategy should satisfy the ContextStrategy protocol
|
|
And every strategy should have a non-empty name
|
|
And every strategy should have a capabilities object
|
|
|
|
@strategy_protocol
|
|
Scenario: Built-in strategy quality scores match specification
|
|
Given all built-in strategies are instantiated
|
|
Then the strategy "simple-keyword" should have quality score 0.3
|
|
And the strategy "semantic-embedding" should have quality score 0.6
|
|
And the strategy "breadth-depth-navigator" should have quality score 0.85
|
|
And the strategy "arce" should have quality score 0.95
|
|
And the strategy "temporal-archaeology" should have quality score 0.5
|
|
And the strategy "plan-decision-context" should have quality score 0.7
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Registry registration
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@strategy_registry
|
|
Scenario: Register a single strategy
|
|
Given an empty strategy registry
|
|
When I register the "simple-keyword" built-in strategy
|
|
Then the registry should contain "simple-keyword"
|
|
And the registry size should be 1
|
|
|
|
@strategy_registry
|
|
Scenario: Register all built-in strategies
|
|
Given an empty strategy registry
|
|
When I register all 6 built-in strategies
|
|
Then the registry should contain 6 strategies
|
|
And the registry should contain "simple-keyword"
|
|
And the registry should contain "semantic-embedding"
|
|
And the registry should contain "breadth-depth-navigator"
|
|
And the registry should contain "arce"
|
|
And the registry should contain "temporal-archaeology"
|
|
And the registry should contain "plan-decision-context"
|
|
|
|
@strategy_registry @error_handling
|
|
Scenario: Reject duplicate strategy registration
|
|
Given an empty strategy registry
|
|
And I register the "simple-keyword" built-in strategy
|
|
When I attempt to register "simple-keyword" again
|
|
Then a StrategyRegistrationError should be raised
|
|
|
|
@strategy_registry @error_handling
|
|
Scenario: Reject strategy with empty name
|
|
Given an empty strategy registry
|
|
When I attempt to register a strategy with an empty name
|
|
Then a StrategyRegistrationError should be raised
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Registry query
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@strategy_query
|
|
Scenario: Get a registered strategy by name
|
|
Given a registry with all built-in strategies
|
|
When I get the strategy "simple-keyword"
|
|
Then the strategy name should be "simple-keyword"
|
|
|
|
@strategy_query @error_handling
|
|
Scenario: Get a non-existent strategy raises error
|
|
Given a registry with all built-in strategies
|
|
When I attempt to get the strategy "nonexistent"
|
|
Then a StrategyNotFoundError should be raised
|
|
|
|
@strategy_query
|
|
Scenario: List all registered strategy names
|
|
Given a registry with all built-in strategies
|
|
Then the registry should list 6 strategies
|
|
And the list should be sorted alphabetically
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Enable / disable
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@strategy_config
|
|
Scenario: Default enabled list matches spec
|
|
Given a registry with all built-in strategies using default enabled list
|
|
Then the enabled strategies should be "simple-keyword", "semantic-embedding", "breadth-depth-navigator"
|
|
|
|
@strategy_config
|
|
Scenario: Override enabled list per project
|
|
Given a registry with all built-in strategies using default enabled list
|
|
When I set the enabled list to "simple-keyword", "arce"
|
|
Then the enabled strategies should be "simple-keyword", "arce"
|
|
|
|
@strategy_config @error_handling
|
|
Scenario: Set enabled list with unknown strategy raises error
|
|
Given a registry with all built-in strategies using default enabled list
|
|
When I attempt to set the enabled list to "nonexistent"
|
|
Then a StrategyNotFoundError should be raised
|
|
|
|
@strategy_config
|
|
Scenario: Disabled strategy excluded from enabled list
|
|
Given a registry with all built-in strategies using default enabled list
|
|
When I disable the strategy "semantic-embedding"
|
|
Then the enabled strategies should not include "semantic-embedding"
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Per-strategy configuration
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@strategy_config
|
|
Scenario: Default strategy config values
|
|
Given a registry with all built-in strategies
|
|
When I get the config for "simple-keyword"
|
|
Then the timeout_seconds should be 30
|
|
And the max_fragments should be 100
|
|
And the circuit_breaker_threshold should be 3
|
|
|
|
@strategy_config
|
|
Scenario: Update per-strategy timeout
|
|
Given a registry with all built-in strategies
|
|
When I update config for "simple-keyword" with timeout_seconds 10
|
|
Then the config for "simple-keyword" should have timeout_seconds 10
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# can_handle with backends
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@strategy_can_handle
|
|
Scenario: simple-keyword returns quality score with text backend
|
|
Given a BackendSet with text backend only
|
|
And a default ContextRequest
|
|
Then "simple-keyword" can_handle should return 0.3
|
|
|
|
@strategy_can_handle
|
|
Scenario: simple-keyword returns 0 without text backend
|
|
Given a BackendSet with no backends
|
|
And a default ContextRequest
|
|
Then "simple-keyword" can_handle should return 0.0
|
|
|
|
@strategy_can_handle
|
|
Scenario: semantic-embedding returns quality score with vector backend
|
|
Given a BackendSet with vector backend only
|
|
And a default ContextRequest
|
|
Then "semantic-embedding" can_handle should return 0.6
|
|
|
|
@strategy_can_handle
|
|
Scenario: arce requires all backends
|
|
Given a BackendSet with all backends
|
|
And a default ContextRequest
|
|
Then "arce" can_handle should return 0.95
|
|
|
|
@strategy_can_handle
|
|
Scenario: arce returns 0 when missing a backend
|
|
Given a BackendSet with text backend only
|
|
And a default ContextRequest
|
|
Then "arce" can_handle should return 0.0
|
|
|
|
@strategy_can_handle
|
|
Scenario: plan-decision-context always returns quality score
|
|
Given a BackendSet with no backends
|
|
And a default ContextRequest
|
|
Then "plan-decision-context" can_handle should return 0.7
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Stub assemble (no-op)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@strategy_assemble
|
|
Scenario: Stub strategies return empty fragment lists
|
|
Given all built-in strategies are instantiated
|
|
And a BackendSet with all backends
|
|
And a default ContextRequest
|
|
And a default PlanContext
|
|
When each strategy assembles with budget 1000
|
|
Then every strategy should return an empty fragment list
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ContextStrategyResult deterministic ordering
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@strategy_result
|
|
Scenario: ContextStrategyResult sorts fragments deterministically
|
|
Given a ContextStrategyResult with unordered fragments
|
|
Then the fragments should be sorted by relevance_score DESC then uko_node ASC
|
|
|
|
@strategy_result
|
|
Scenario: ContextStrategyResult with no fragments
|
|
Given a ContextStrategyResult with no fragments
|
|
Then the total_fragments should be 0
|
|
And the tokens_used should be 0
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Registry validation
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@strategy_validation
|
|
Scenario: Validate registry warns about missing resource types
|
|
Given a registry with a strategy that declares no resource types
|
|
When I validate the registry
|
|
Then validation warnings should mention "resource types"
|
|
|
|
@strategy_validation
|
|
Scenario: Validate registry warns about missing capabilities
|
|
Given a registry with a strategy that declares no backend capabilities
|
|
When I validate the registry
|
|
Then validation warnings should mention "backend capabilities"
|
|
|
|
@strategy_validation
|
|
Scenario: Valid registry returns no warnings for built-in strategies
|
|
Given a registry with all built-in strategies
|
|
When I validate the registry
|
|
Then there should be no validation warnings
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Unregister
|
|
# ---------------------------------------------------------------------------
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Plugin discovery (register_from_module)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@strategy_registry @plugin
|
|
Scenario: Register a strategy from a module path
|
|
Given an empty strategy registry
|
|
When I register a strategy from module "cleveragents.domain.models.acms.strategy_stubs:SimpleKeywordStrategy"
|
|
Then the registry should contain "simple-keyword"
|
|
And the registry size should be 1
|
|
|
|
@strategy_registry @plugin @error_handling
|
|
Scenario: Reject module path without colon separator
|
|
Given an empty strategy registry
|
|
When I attempt to register a strategy from module "no_colon_here"
|
|
Then a StrategyRegistrationError should be raised
|
|
|
|
@strategy_registry @plugin @error_handling
|
|
Scenario: Reject module path with missing module
|
|
Given an empty strategy registry
|
|
When I attempt to register a strategy from module "nonexistent.module:FakeClass"
|
|
Then a StrategyRegistrationError should be raised
|
|
|
|
@strategy_registry @plugin @error_handling
|
|
Scenario: Reject module path with missing class
|
|
Given an empty strategy registry
|
|
When I attempt to register a strategy from module "cleveragents.domain.models.acms.strategy_stubs:NonExistentClass"
|
|
Then a StrategyRegistrationError should be raised
|
|
|
|
@strategy_registry @plugin @error_handling
|
|
Scenario: Reject module path when instantiation fails
|
|
Given an empty strategy registry
|
|
When I attempt to register a strategy from module "cleveragents.domain.models.acms.strategy_stubs:BUILTIN_STRATEGY_CLASSES"
|
|
Then a StrategyRegistrationError should be raised
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Additional query coverage
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@strategy_query
|
|
Scenario: Get entry for a registered strategy
|
|
Given a registry with all built-in strategies
|
|
When I get the entry for "simple-keyword"
|
|
Then the entry name should be "simple-keyword"
|
|
And the entry should be marked as builtin
|
|
|
|
@strategy_query @error_handling
|
|
Scenario: Get entry for non-existent strategy raises error
|
|
Given a registry with all built-in strategies
|
|
When I attempt to get the entry for "nonexistent"
|
|
Then a StrategyNotFoundError should be raised
|
|
|
|
@strategy_query
|
|
Scenario: List built-in strategies
|
|
Given a registry with all built-in strategies
|
|
Then the builtin list should contain 6 strategies
|
|
|
|
@strategy_query
|
|
Scenario: Registry supports the in operator
|
|
Given a registry with all built-in strategies
|
|
Then "simple-keyword" should be in the registry via contains
|
|
And "nonexistent" should not be in the registry via contains
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Validation edge case: enabled-but-unregistered
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@strategy_validation
|
|
Scenario: Validate warns when enabled list references unregistered strategy
|
|
Given a registry with a stale enabled list entry
|
|
When I validate the registry
|
|
Then validation warnings should mention "not registered"
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Unregister error path
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@strategy_registry @error_handling
|
|
Scenario: Unregister non-existent strategy raises error
|
|
Given a registry with all built-in strategies
|
|
When I attempt to unregister "nonexistent"
|
|
Then a StrategyNotFoundError should be raised
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Unregister
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@strategy_registry
|
|
Scenario: Unregister a strategy
|
|
Given a registry with all built-in strategies
|
|
When I unregister "temporal-archaeology"
|
|
Then the registry should contain 5 strategies
|
|
And the registry should not contain "temporal-archaeology"
|
|
|
|
@strategy_registry
|
|
Scenario: Clear the registry
|
|
Given a registry with all built-in strategies
|
|
When I clear the registry
|
|
Then the registry should contain 0 strategies
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Boundary tests for Pydantic-validated config fields
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@strategy_config @boundary
|
|
Scenario Outline: Reject invalid StrategyConfig numeric values
|
|
When I attempt to create a StrategyConfig with <field> set to <value>
|
|
Then a Pydantic ValidationError should be raised
|
|
|
|
Examples:
|
|
| field | value |
|
|
| timeout_seconds | 0 |
|
|
| timeout_seconds | -1 |
|
|
| max_fragments | 0 |
|
|
| max_fragments | -1 |
|
|
| max_workers | 0 |
|
|
| circuit_breaker_threshold | 0 |
|
|
|
|
@strategy_config @boundary
|
|
Scenario Outline: Reject invalid quality_score values
|
|
When I attempt to create a StrategyCapabilities with quality_score <value>
|
|
Then a Pydantic ValidationError should be raised
|
|
|
|
Examples:
|
|
| value |
|
|
| -0.1 |
|
|
| 1.1 |
|
|
|
|
@strategy_config @boundary
|
|
Scenario: Accept quality_score at lower bound 0.0
|
|
When I create a StrategyCapabilities with quality_score 0.0
|
|
Then the quality_score should be 0.0
|
|
|
|
@strategy_config @boundary
|
|
Scenario: Accept quality_score at upper bound 1.0
|
|
When I create a StrategyCapabilities with quality_score 1.0
|
|
Then the quality_score should be 1.0
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Bug regression tests
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@strategy_config @regression
|
|
Scenario: update_config enabled=True adds strategy to enabled list
|
|
Given an empty strategy registry
|
|
When I register the "simple-keyword" built-in strategy with enabled False
|
|
Then the enabled strategies should be ""
|
|
When I update config for "simple-keyword" with enabled True
|
|
Then the enabled strategies should include "simple-keyword"
|
|
|
|
@strategy_config @regression
|
|
Scenario: update_config enabled=False removes strategy from enabled list
|
|
Given a registry with all built-in strategies using default enabled list
|
|
When I update config for "simple-keyword" with enabled False
|
|
Then the enabled strategies should not include "simple-keyword"
|
|
|
|
@strategy_registry @plugin @regression
|
|
Scenario: register_from_module uses the provided name parameter
|
|
Given an empty strategy registry
|
|
When I register a strategy named "custom-search" from module "cleveragents.domain.models.acms.strategy_stubs:SimpleKeywordStrategy"
|
|
Then the registry should contain "custom-search"
|
|
And the registry size should be 1
|
|
|
|
@strategy_config @regression
|
|
Scenario: update_config with max_workers and circuit_breaker_threshold
|
|
Given a registry with all built-in strategies
|
|
When I update config for "simple-keyword" with max_workers 8 and circuit_breaker_threshold 5
|
|
Then the config for "simple-keyword" should have max_workers 8
|
|
And the config for "simple-keyword" should have circuit_breaker_threshold 5
|
|
|
|
@strategy_registry @error_handling
|
|
Scenario: Reject object that does not satisfy ContextStrategy protocol
|
|
Given an empty strategy registry
|
|
When I attempt to register a non-protocol object
|
|
Then a StrategyRegistrationError should be raised
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# explain() coverage
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@strategy_protocol
|
|
Scenario: Every built-in strategy returns a non-empty explain string
|
|
Given all built-in strategies are instantiated
|
|
Then every strategy should return a non-empty explain string
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# can_handle coverage for strategies not tested elsewhere
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@strategy_can_handle
|
|
Scenario: breadth-depth-navigator returns quality score with graph backend
|
|
Given a BackendSet with graph backend only
|
|
And a default ContextRequest
|
|
Then "breadth-depth-navigator" can_handle should return 0.85
|
|
|
|
@strategy_can_handle
|
|
Scenario: breadth-depth-navigator returns 0 without graph backend
|
|
Given a BackendSet with no backends
|
|
And a default ContextRequest
|
|
Then "breadth-depth-navigator" can_handle should return 0.0
|
|
|
|
@strategy_can_handle
|
|
Scenario: temporal-archaeology returns quality score with graph backend
|
|
Given a BackendSet with graph backend only
|
|
And a default ContextRequest
|
|
Then "temporal-archaeology" can_handle should return 0.5
|
|
|
|
@strategy_can_handle
|
|
Scenario: temporal-archaeology returns 0 without graph backend
|
|
Given a BackendSet with no backends
|
|
And a default ContextRequest
|
|
Then "temporal-archaeology" can_handle should return 0.0
|
|
|
|
@strategy_can_handle
|
|
Scenario: semantic-embedding returns 0 without vector backend
|
|
Given a BackendSet with no backends
|
|
And a default ContextRequest
|
|
Then "semantic-embedding" can_handle should return 0.0
|
|
|
|
@strategy_config @regression
|
|
Scenario: update_config rejects invalid timeout via Pydantic validation
|
|
Given a registry with all built-in strategies
|
|
When I attempt to update config for "simple-keyword" with timeout_seconds 0
|
|
Then a Pydantic ValidationError should be raised
|
|
|
|
@strategy_config @regression
|
|
Scenario: update_config rejects negative max_fragments via Pydantic validation
|
|
Given a registry with all built-in strategies
|
|
When I attempt to update config for "simple-keyword" with max_fragments -1
|
|
Then a Pydantic ValidationError should be raised
|