UAT: `SemanticEmbeddingStrategy` in `context_strategies.py` uses word-overlap (Jaccard) instead of vector similarity — semantic search is not semantic #4006

New issue

Open

opened 2026-04-06 08:31:34 +00:00 by freemo · 0 comments

freemo commented

2026-04-06 08:31:34 +00:00

Owner

Metadata

Branch: fix/semantic-embedding-strategy-vector-similarity
Commit Message: fix(acms): replace Jaccard word-overlap in SemanticEmbeddingStrategy with real vector similarity search
Milestone: None (backlog)
Parent Epic: #396

Background and Context

The SemanticEmbeddingStrategy class in src/cleveragents/application/services/context_strategies.py (lines 139–224) is supposed to implement vector similarity search per the specification. However, the implementation uses word-level Jaccard similarity (word overlap scoring) instead of actual vector embeddings.

Per docs/specification.md §25551:

semantic-embedding | 0.6 | Vector | Vector similarity search for semantically related content.

The spec explicitly states this strategy requires a Vector backend and performs vector similarity search.

Current Behavior

# src/cleveragents/application/services/context_strategies.py lines 139-224
class SemanticEmbeddingStrategy:
    """Approximate semantic similarity via word-overlap scoring.

    In the v1 pipeline, strategies receive pre-fetched fragments rather
    than querying vector backends directly.  This strategy approximates
    embedding-based similarity by computing word-level Jaccard similarity
    between a query (set via ``set_query()`` or ``can_handle()``) and
    each fragment's content.
    ...
    """
    def assemble(self, fragments, budget):
        # Computes word-level Jaccard similarity — NOT vector similarity
        sim = _jaccard_similarity(query_words, _tokenize(frag.content))

The implementation:

Does not use any vector backend
Does not generate or use embeddings
Uses naive word-overlap (Jaccard) scoring instead of semantic vector similarity
Cannot distinguish semantically similar but lexically different content (e.g., "car" vs "automobile")

Note: There is a separate SemanticEmbeddingStrategy stub in src/cleveragents/domain/models/acms/strategy_stubs.py that correctly declares uses_vector=True and requires a vector backend, but its assemble() method returns an empty list (stub). The context_strategies.py version is the one registered in BUILTIN_STRATEGIES and actually used.

Expected Behavior

Per docs/specification.md §45340-45342:

Strategy: semantic-embedding
Vector similarity search. Finds semantically related content even without exact keyword matches. Requires a vector backend. Quality score: 0.6.

The SemanticEmbeddingStrategy should:

Accept a query string and generate a query embedding using the configured embedding model
Use the vector backend to perform similarity search against indexed embeddings
Return fragments ranked by cosine/vector similarity to the query embedding
Correctly declare uses_vector=True in its capabilities

Steps to Reproduce

Open src/cleveragents/application/services/context_strategies.py
Navigate to SemanticEmbeddingStrategy.assemble() (lines 180–216)
Observe that it calls _jaccard_similarity() (word overlap) instead of any vector backend
Compare with docs/specification.md §45340-45342 which requires vector similarity search

Impact

The semantic-embedding ACMS context strategy does not perform semantic search — it performs keyword overlap scoring identical in nature to simple-keyword but with a different scoring formula. The strategy's claimed quality score of 0.6 (higher than simple-keyword's 0.3) is misleading because it does not provide the semantic differentiation that justifies the higher score.

Code locations affected:

src/cleveragents/application/services/context_strategies.py lines 139–224 (SemanticEmbeddingStrategy)
src/cleveragents/domain/models/acms/strategy_stubs.py lines 110–156 (correct stub with empty assemble())

Backlog note: This issue was discovered during autonomous UAT operation on milestone (active). It does not block milestone completion and has been placed in the backlog for human review and future milestone assignment.

Subtasks

Replace word-overlap scoring in SemanticEmbeddingStrategy.assemble() with actual vector similarity search using the configured vector backend
Update SemanticEmbeddingStrategy.capabilities to declare uses_vector=True
Integrate with the VectorBackend protocol for query-time embedding generation and similarity search
Write BDD tests verifying semantic similarity returns different results than keyword overlap for semantically similar but lexically different queries
Verify nox -e coverage_report reports >= 97% coverage

Definition of Done

SemanticEmbeddingStrategy.assemble() uses vector backend for similarity search
Strategy correctly declares uses_vector=True in capabilities
Semantic similarity search returns meaningful results for semantically related but lexically different queries
All BDD unit tests pass
All nox stages pass
Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-new-issue-creator

## Metadata - **Branch**: `fix/semantic-embedding-strategy-vector-similarity` - **Commit Message**: `fix(acms): replace Jaccard word-overlap in SemanticEmbeddingStrategy with real vector similarity search` - **Milestone**: None (backlog) - **Parent Epic**: #396 ## Background and Context The `SemanticEmbeddingStrategy` class in `src/cleveragents/application/services/context_strategies.py` (lines 139–224) is supposed to implement vector similarity search per the specification. However, the implementation uses **word-level Jaccard similarity** (word overlap scoring) instead of actual vector embeddings. Per `docs/specification.md` §25551: > `semantic-embedding` | 0.6 | Vector | Vector similarity search for semantically related content. The spec explicitly states this strategy requires a **Vector** backend and performs **vector similarity search**. ## Current Behavior ```python # src/cleveragents/application/services/context_strategies.py lines 139-224 class SemanticEmbeddingStrategy: """Approximate semantic similarity via word-overlap scoring. In the v1 pipeline, strategies receive pre-fetched fragments rather than querying vector backends directly. This strategy approximates embedding-based similarity by computing word-level Jaccard similarity between a query (set via ``set_query()`` or ``can_handle()``) and each fragment's content. ... """ def assemble(self, fragments, budget): # Computes word-level Jaccard similarity — NOT vector similarity sim = _jaccard_similarity(query_words, _tokenize(frag.content)) ``` The implementation: 1. Does not use any vector backend 2. Does not generate or use embeddings 3. Uses naive word-overlap (Jaccard) scoring instead of semantic vector similarity 4. Cannot distinguish semantically similar but lexically different content (e.g., "car" vs "automobile") Note: There is a separate `SemanticEmbeddingStrategy` stub in `src/cleveragents/domain/models/acms/strategy_stubs.py` that correctly declares `uses_vector=True` and requires a vector backend, but its `assemble()` method returns an empty list (stub). The `context_strategies.py` version is the one registered in `BUILTIN_STRATEGIES` and actually used. ## Expected Behavior Per `docs/specification.md` §45340-45342: > **Strategy: `semantic-embedding`** > Vector similarity search. Finds semantically related content even without exact keyword matches. Requires a vector backend. Quality score: 0.6. The `SemanticEmbeddingStrategy` should: 1. Accept a query string and generate a query embedding using the configured embedding model 2. Use the vector backend to perform similarity search against indexed embeddings 3. Return fragments ranked by cosine/vector similarity to the query embedding 4. Correctly declare `uses_vector=True` in its capabilities ## Steps to Reproduce 1. Open `src/cleveragents/application/services/context_strategies.py` 2. Navigate to `SemanticEmbeddingStrategy.assemble()` (lines 180–216) 3. Observe that it calls `_jaccard_similarity()` (word overlap) instead of any vector backend 4. Compare with `docs/specification.md` §45340-45342 which requires vector similarity search ## Impact The `semantic-embedding` ACMS context strategy does not perform semantic search — it performs keyword overlap scoring identical in nature to `simple-keyword` but with a different scoring formula. The strategy's claimed quality score of 0.6 (higher than `simple-keyword`'s 0.3) is misleading because it does not provide the semantic differentiation that justifies the higher score. **Code locations affected:** - `src/cleveragents/application/services/context_strategies.py` lines 139–224 (`SemanticEmbeddingStrategy`) - `src/cleveragents/domain/models/acms/strategy_stubs.py` lines 110–156 (correct stub with empty `assemble()`) > **Backlog note:** This issue was discovered during autonomous UAT operation on milestone (active). It does not block milestone completion and has been placed in the backlog for human review and future milestone assignment. ## Subtasks - [ ] Replace word-overlap scoring in `SemanticEmbeddingStrategy.assemble()` with actual vector similarity search using the configured vector backend - [ ] Update `SemanticEmbeddingStrategy.capabilities` to declare `uses_vector=True` - [ ] Integrate with the `VectorBackend` protocol for query-time embedding generation and similarity search - [ ] Write BDD tests verifying semantic similarity returns different results than keyword overlap for semantically similar but lexically different queries - [ ] Verify `nox -e coverage_report` reports >= 97% coverage ## Definition of Done - [ ] `SemanticEmbeddingStrategy.assemble()` uses vector backend for similarity search - [ ] Strategy correctly declares `uses_vector=True` in capabilities - [ ] Semantic similarity search returns meaningful results for semantically related but lexically different queries - [ ] All BDD unit tests pass - [ ] All nox stages pass - [ ] Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-new-issue-creator

freemo added a new dependency

2026-04-06 08:32:05 +00:00

#396 Epic: ACMS Context Pipeline

freemo referenced this issue

2026-04-06 08:35:23 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775

freemo referenced this issue

2026-04-06 08:38:21 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775

freemo referenced this issue

2026-04-06 08:50:45 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775