freemo
487e16a9f0
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 13s
CI / build (pull_request) Successful in 16s
CI / quality (pull_request) Successful in 17s
CI / security (pull_request) Successful in 32s
CI / typecheck (pull_request) Successful in 33s
CI / unit_tests (pull_request) Successful in 2m7s
CI / docker (pull_request) Successful in 37s
CI / integration_tests (pull_request) Successful in 2m56s
CI / coverage (pull_request) Successful in 4m26s
CI / benchmark-regression (pull_request) Successful in 28m12s
CI / lint (push) Successful in 12s
CI / build (push) Successful in 15s
CI / quality (push) Successful in 16s
CI / security (push) Successful in 31s
CI / typecheck (push) Successful in 34s
CI / benchmark-regression (push) Has been skipped
CI / unit_tests (push) Successful in 2m4s
CI / docker (push) Successful in 38s
CI / integration_tests (push) Successful in 3m7s
CI / coverage (push) Successful in 5m4s
CI / benchmark-publish (push) Has been cancelled
Implement the first three built-in context strategies for the ACMS v1 context assembly pipeline: 1. SimpleKeywordStrategy (quality 0.3) - Keyword matching on fragment content with word-density fallback. Universal fallback strategy. 2. SemanticEmbeddingStrategy (quality 0.6) - Jaccard word-overlap similarity scoring between query and fragment content. 3. BreadthDepthNavigatorStrategy (quality 0.85) - UKO node hierarchy navigation prioritising fragments near focus nodes with higher detail depths. Primary strategy for code projects. All strategies implement the v1 ContextStrategy Protocol from acms_service.py and can be registered with ACMSPipeline via register_strategy(). Includes: - 28 Behave BDD scenarios covering ranking, budget, capabilities, can_handle confidence, explain, empty input, and pipeline registration - 9 Robot Framework integration tests - ASV benchmarks at 10/100/1000 fragment scales for all 3 strategies - Vulture whitelist entries for public API symbols - 100% coverage on context_strategies.py ISSUES CLOSED: #541
258 lines
12 KiB
Gherkin
258 lines
12 KiB
Gherkin
@phase2 @acms @context_strategies
|
|
Feature: Built-in Context Strategies Batch 1
|
|
As a CleverAgents developer
|
|
I want built-in context strategies
|
|
So that the ACMS pipeline can rank fragments using different approaches
|
|
|
|
# ===========================================================================
|
|
# SimpleKeywordStrategy
|
|
# ===========================================================================
|
|
|
|
@simple_keyword
|
|
Scenario: SimpleKeyword ranks by keyword match count
|
|
Given a SimpleKeywordStrategy with query "async IO"
|
|
And the following strategy fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/io.py | Use async IO pattern | 0.5 | 20 | 3 |
|
|
| project://app/main.py | Main entry point | 0.8 | 15 | 3 |
|
|
| project://app/net.py | Async network handler | 0.6 | 25 | 3 |
|
|
And a strategy budget with max_tokens 1000 and reserved_tokens 0
|
|
When I assemble with the SimpleKeywordStrategy
|
|
Then the first result fragment should have uko_node "project://app/io.py"
|
|
|
|
@simple_keyword
|
|
Scenario: SimpleKeyword without query falls back to word density
|
|
Given a SimpleKeywordStrategy without query
|
|
And the following strategy fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/io.py | io | 0.5 | 10 | 3 |
|
|
| project://app/main.py | Main entry point with many words here | 0.5 | 10 | 3 |
|
|
And a strategy budget with max_tokens 1000 and reserved_tokens 0
|
|
When I assemble with the SimpleKeywordStrategy
|
|
Then 2 fragments should be returned by strategy
|
|
|
|
@simple_keyword
|
|
Scenario: SimpleKeyword returns empty for empty input
|
|
Given a SimpleKeywordStrategy with query "test"
|
|
And an empty strategy fragment list
|
|
And a strategy budget with max_tokens 1000 and reserved_tokens 0
|
|
When I assemble with the SimpleKeywordStrategy
|
|
Then 0 fragments should be returned by strategy
|
|
|
|
@simple_keyword
|
|
Scenario: SimpleKeyword respects budget
|
|
Given a SimpleKeywordStrategy with query "hello"
|
|
And the following strategy fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/a.py | hello world | 0.9 | 100 | 3 |
|
|
| project://app/b.py | hello there | 0.7 | 100 | 3 |
|
|
| project://app/c.py | hello again | 0.5 | 100 | 3 |
|
|
And a strategy budget with max_tokens 250 and reserved_tokens 0
|
|
When I assemble with the SimpleKeywordStrategy
|
|
Then 2 fragments should be returned by strategy
|
|
|
|
@simple_keyword
|
|
Scenario: SimpleKeyword can_handle returns 0.3
|
|
Given a SimpleKeywordStrategy without query
|
|
When I check can_handle on SimpleKeywordStrategy with query "test"
|
|
Then the strategy confidence should be 0.3
|
|
|
|
@simple_keyword
|
|
Scenario: SimpleKeyword reports capabilities
|
|
Given a SimpleKeywordStrategy without query
|
|
Then the SimpleKeywordStrategy should not support semantic search
|
|
And the SimpleKeywordStrategy name should be "simple-keyword"
|
|
|
|
# ===========================================================================
|
|
# SemanticEmbeddingStrategy
|
|
# ===========================================================================
|
|
|
|
@semantic_embedding
|
|
Scenario: SemanticEmbedding ranks by word similarity
|
|
Given a SemanticEmbeddingStrategy with query "database connection pool"
|
|
And the following strategy fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/db.py | Database connection pool manager | 0.5 | 20 | 3 |
|
|
| project://app/io.py | File input output handler | 0.8 | 15 | 3 |
|
|
| project://app/sql.py | SQL database query executor | 0.6 | 25 | 3 |
|
|
And a strategy budget with max_tokens 1000 and reserved_tokens 0
|
|
When I assemble with the SemanticEmbeddingStrategy
|
|
Then the first result fragment should have uko_node "project://app/db.py"
|
|
|
|
@semantic_embedding
|
|
Scenario: SemanticEmbedding filters below similarity threshold
|
|
Given a SemanticEmbeddingStrategy with query "quantum computing"
|
|
And the following strategy fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/db.py | database handler | 0.9 | 20 | 3 |
|
|
| project://app/io.py | file io module | 0.8 | 15 | 3 |
|
|
And a strategy budget with max_tokens 1000 and reserved_tokens 0
|
|
When I assemble with the SemanticEmbeddingStrategy
|
|
Then 0 fragments should be returned by strategy
|
|
|
|
@semantic_embedding
|
|
Scenario: SemanticEmbedding without query falls back to relevance
|
|
Given a SemanticEmbeddingStrategy without query
|
|
And the following strategy fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/a.py | alpha | 0.3 | 10 | 3 |
|
|
| project://app/b.py | beta | 0.9 | 10 | 3 |
|
|
And a strategy budget with max_tokens 1000 and reserved_tokens 0
|
|
When I assemble with the SemanticEmbeddingStrategy
|
|
Then the first result fragment should have uko_node "project://app/b.py"
|
|
|
|
@semantic_embedding
|
|
Scenario: SemanticEmbedding can_handle returns 0.6 with query
|
|
Given a SemanticEmbeddingStrategy without query
|
|
When I check can_handle on SemanticEmbeddingStrategy with query "test"
|
|
Then the strategy confidence should be 0.6
|
|
|
|
@semantic_embedding
|
|
Scenario: SemanticEmbedding can_handle returns 0.1 without query
|
|
Given a SemanticEmbeddingStrategy without query
|
|
When I check can_handle on SemanticEmbeddingStrategy without query
|
|
Then the strategy confidence should be 0.1
|
|
|
|
@semantic_embedding
|
|
Scenario: SemanticEmbedding reports capabilities
|
|
Given a SemanticEmbeddingStrategy without query
|
|
Then the SemanticEmbeddingStrategy should support semantic search
|
|
And the SemanticEmbeddingStrategy name should be "semantic-embedding"
|
|
|
|
# ===========================================================================
|
|
# BreadthDepthNavigatorStrategy
|
|
# ===========================================================================
|
|
|
|
@breadth_depth
|
|
Scenario: BreadthDepthNavigator prioritises near focus nodes
|
|
Given a BreadthDepthNavigatorStrategy with focus "project://app/io.py"
|
|
And the following strategy fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/io.py | io module | 0.5 | 20 | 5 |
|
|
| project://app/main.py | main entry | 0.9 | 15 | 3 |
|
|
| project://other/lib.py | library | 0.7 | 25 | 9 |
|
|
And a strategy budget with max_tokens 1000 and reserved_tokens 0
|
|
When I assemble with the BreadthDepthNavigatorStrategy
|
|
Then the first result fragment should have uko_node "project://app/io.py"
|
|
|
|
@breadth_depth
|
|
Scenario: BreadthDepthNavigator prefers higher depth near focus
|
|
Given a BreadthDepthNavigatorStrategy with focus "project://app"
|
|
And the following strategy fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/a.py | alpha | 0.5 | 10 | 9 |
|
|
| project://app/b.py | beta | 0.5 | 10 | 1 |
|
|
And a strategy budget with max_tokens 1000 and reserved_tokens 0
|
|
When I assemble with the BreadthDepthNavigatorStrategy
|
|
Then the first result fragment should have uko_node "project://app/a.py"
|
|
|
|
@breadth_depth
|
|
Scenario: BreadthDepthNavigator without focus falls back to depth+relevance
|
|
Given a BreadthDepthNavigatorStrategy without focus
|
|
And the following strategy fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/a.py | alpha | 0.5 | 10 | 9 |
|
|
| project://app/b.py | beta | 0.9 | 10 | 2 |
|
|
And a strategy budget with max_tokens 1000 and reserved_tokens 0
|
|
When I assemble with the BreadthDepthNavigatorStrategy
|
|
Then the first result fragment should have uko_node "project://app/a.py"
|
|
|
|
@breadth_depth
|
|
Scenario: BreadthDepthNavigator can_handle returns 0.85 with focus
|
|
Given a BreadthDepthNavigatorStrategy without focus
|
|
When I check can_handle on BreadthDepthNavigator with focus
|
|
Then the strategy confidence should be 0.85
|
|
|
|
@breadth_depth
|
|
Scenario: BreadthDepthNavigator can_handle returns 0.2 without focus
|
|
Given a BreadthDepthNavigatorStrategy without focus
|
|
When I check can_handle on BreadthDepthNavigator without focus
|
|
Then the strategy confidence should be 0.2
|
|
|
|
@breadth_depth
|
|
Scenario: BreadthDepthNavigator reports capabilities
|
|
Given a BreadthDepthNavigatorStrategy without focus
|
|
Then the BreadthDepthNavigatorStrategy should support graph navigation
|
|
And the BreadthDepthNavigatorStrategy name should be "breadth-depth-navigator"
|
|
|
|
@breadth_depth
|
|
Scenario: BreadthDepthNavigator respects budget
|
|
Given a BreadthDepthNavigatorStrategy with focus "project://app"
|
|
And the following strategy fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/a.py | alpha | 0.9 | 100 | 5 |
|
|
| project://app/b.py | beta | 0.7 | 100 | 5 |
|
|
| project://app/c.py | gamma | 0.5 | 100 | 5 |
|
|
And a strategy budget with max_tokens 250 and reserved_tokens 0
|
|
When I assemble with the BreadthDepthNavigatorStrategy
|
|
Then 2 fragments should be returned by strategy
|
|
|
|
@breadth_depth
|
|
Scenario: BreadthDepthNavigator returns empty for empty input
|
|
Given a BreadthDepthNavigatorStrategy with focus "project://app"
|
|
And an empty strategy fragment list
|
|
And a strategy budget with max_tokens 1000 and reserved_tokens 0
|
|
When I assemble with the BreadthDepthNavigatorStrategy
|
|
Then 0 fragments should be returned by strategy
|
|
|
|
@breadth_depth
|
|
Scenario: BreadthDepthNavigator can_handle accepts focus as string
|
|
Given a BreadthDepthNavigatorStrategy without focus
|
|
When I check can_handle on BreadthDepthNavigator with string focus
|
|
Then the strategy confidence should be 0.85
|
|
|
|
@semantic_embedding
|
|
Scenario: SemanticEmbedding returns empty for empty input
|
|
Given a SemanticEmbeddingStrategy with query "test"
|
|
And an empty strategy fragment list
|
|
And a strategy budget with max_tokens 1000 and reserved_tokens 0
|
|
When I assemble with the SemanticEmbeddingStrategy
|
|
Then 0 fragments should be returned by strategy
|
|
|
|
@semantic_embedding
|
|
Scenario: SemanticEmbedding handles fragment with empty content
|
|
Given a SemanticEmbeddingStrategy with query "test"
|
|
And the following strategy fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/a.py | | 0.5 | 10 | 3 |
|
|
And a strategy budget with max_tokens 1000 and reserved_tokens 0
|
|
When I assemble with the SemanticEmbeddingStrategy
|
|
Then 0 fragments should be returned by strategy
|
|
|
|
# ===========================================================================
|
|
# explain() coverage
|
|
# ===========================================================================
|
|
|
|
@simple_keyword
|
|
Scenario: SimpleKeyword explain returns description
|
|
Given a SimpleKeywordStrategy without query
|
|
Then the SimpleKeywordStrategy explain should contain "keyword"
|
|
|
|
@semantic_embedding
|
|
Scenario: SemanticEmbedding explain returns description
|
|
Given a SemanticEmbeddingStrategy without query
|
|
Then the SemanticEmbeddingStrategy explain should contain "semantic"
|
|
|
|
@breadth_depth
|
|
Scenario: BreadthDepthNavigator explain returns description
|
|
Given a BreadthDepthNavigatorStrategy without focus
|
|
Then the BreadthDepthNavigatorStrategy explain should contain "hierarchy"
|
|
|
|
# ===========================================================================
|
|
# Pipeline Registration
|
|
# ===========================================================================
|
|
|
|
@registration
|
|
Scenario: Register SimpleKeywordStrategy with pipeline
|
|
Given an ACMS pipeline for strategy tests
|
|
When I register SimpleKeywordStrategy with the pipeline
|
|
Then the pipeline should have strategy "simple-keyword"
|
|
|
|
@registration
|
|
Scenario: Register all three strategies with pipeline
|
|
Given an ACMS pipeline for strategy tests
|
|
When I register all batch 1 strategies with the pipeline
|
|
Then the pipeline should have strategy "simple-keyword"
|
|
And the pipeline should have strategy "semantic-embedding"
|
|
And the pipeline should have strategy "breadth-depth-navigator"
|