diff --git a/CHANGELOG.md b/CHANGELOG.md index 40c79398d..57513a9d3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,21 @@ Changed `wf10_batch.robot` to be less likely to create files, and - **docs: module guides for Sandbox & Checkpoint, Correction Attempts, and Invariant Reconciliation** (#4848): Added three comprehensive module guides covering purpose, core classes, lifecycle diagrams, exception hierarchies, CLI usage, and ADR links for `SandboxManager`, `CorrectionAttemptManager`, and `InvariantReconciliationActor`. Includes security callouts for `NoSandbox` bypass (permanent writes, no rollback), `guidance` prompt-injection risk, `archived_artifacts_path` provenance, and `non_overridable` global invariant access control. - **feat(context): PriorityContextStrategy** (#9997 / PR #10772): Implements a priority-based context strategy that ranks context fragments by configurable priority scores — default role-based rules (system > tool > user > assistant), exponential recency decay (7-day half-life), and explicit priority tag boost. Supports custom scoring function injection and custom PriorityRule list injection. Registered in the ACMS pipeline under key `priority_context`. `PriorityRule` uses Pydantic `BaseModel` for architecture conformance. Includes 18 BDD scenarios covering all acceptance criteria. - **docs(timeline): verify timeline status for 2026-04-16 Cycle 2** (#8519): Updated `docs/timeline.md` with Days 104-106 Cycle 2 milestone snapshot. No changes detected since Cycle 1. All M3-M7 milestones remain overdue. Timeline verification performed by AUTO-TIME-3 supervisor agent. Refs #8519. +- **feat(context): implement semantic context search strategy using embeddings** (#5254): Added `EmbeddingProvider` abstract base class and `SimpleWordEmbeddingProvider` (deterministic word-vector implementation) to `application/services/embedding_provider.py`. Introduced `SemanticContextSearchStrategy` that retrieves context chunks by cosine-similarity scoring against query embeddings. Includes BDD scenarios covering embedding generation, dimension constraints, cosine similarity computation, vocabulary overflow warnings, and semantic context ranking. +- **Virtual Resource Type Base Class** (#8610): Implemented `VirtualResource` base class with two example concrete implementations (`MetricResource`, `APIEndpointResource`) for abstract/computed resources that are derived rather than mapped to physical files. Virtual resources are computed on demand via a `compute_fn` callable. Includes Behave BDD scenarios in `features/resource_virtual_types.feature` exercising construction, computation, name validation, kwargs passthrough, exception handling, string representation, and subclassing. Resource names are validated against `^[a-zA-Z][a-zA-Z0-9_-]*$` (must start with a letter; alphanumeric, hyphens, and underscores otherwise). +- **test(e2e): restore complete M2 acceptance test** (#11191): Restored the truncated M2 full actor compiler and LLM integration e2e acceptance test to its complete 10-step form. Added dynamic LLM provider selection via `Resolve LLM Actor` (falls back to Anthropic when OpenAI is unavailable or quota-exhausted), replacing hardcoded `gpt-4` / `openai/gpt-4` references in the actor config and action YAML. Added explicit return-code validation (`Should Be Equal As Integers ${r_actor.rc} 0`) for the actor registration step. +- **docs(a2a): ACP to A2A migration guide** (#10230): Added migration guide documenting how to upgrade from the ACP module to the A2A module introduced in v3.6.0, including symbol renames, field renames, operation-name mappings, and YAML configuration updates. +- **Plan Prompt JSON Timing Field** (#9353): `agents plan prompt --format json` now + includes `timing.started` as an ISO 8601 UTC timestamp in the JSON envelope, + matching the spec (§CLI Commands — `agents plan prompt`). Extended + `cleveragents.cli.formatting.format_output` (and `_build_envelope`) with an + optional `started_at: datetime` parameter; when provided, the envelope's + `timing` dict includes a `started` field alongside `duration_ms`. Refactored + `prompt_plan_cmd` to delegate envelope construction to `format_output` so the + envelope keys (`command`, `status`, `data`, `timing.started`, `messages`) are + populated correctly at the JSON root rather than nested under a synthetic + inner `data` field. +- **fix(plan): NamespacedName digit-start validation** (#2145, #2147): `NamespacedName` field validators now reject `namespace` and `name` components whose first character is a digit, raising `pydantic.ValidationError` with message `"must start with a letter"`. BDD constructor scenarios updated to use the `"a Pydantic ValidationError should be raised"` step so the assertion correctly matches the exception type raised by Pydantic model construction. - **fix(cli/plan): plan correct JSON output envelope fix and BDD test coverage** (#8584 / PR #8662): Restructured `agents plan correct --format json` output to nest correction fields under `data.correction` (e.g., `data.correction.mode`) and populate the spec-required CLI envelope with `command="plan correct"`, `status`, `exit_code`, `timing`, and `messages` fields. Added three BDD scenarios in `features/tdd_plan_correct_json_output.feature` validating the envelope structure for both revert and append modes. - **fix(cli): add --url flag to resource add for git resource type** (#6322): Added support for the `--url` flag on `agents resource add git` command, allowing users to specify a remote URL for git resources. The flag is validated to only apply to git resource types. Includes Behave BDD tests in `features/resource_cli_git_url_flag.feature` and Robot Framework integration tests verifying correct URL validation and CLI behavior. - **Session create JSON envelope** (#6441): Fixed `agents session create --format json` returning a flat `data` dict instead of the spec-required nested structure with `data.session`, `data.settings`, and `data.actor_details` sub-objects. The `command` field is now populated correctly. Extended JSON envelope coverage to `agents session list`, `show`, `delete --format json`, `export --output-format json`, and `import --format json` so all session commands emit a structured `messages[].text` field (`"0 sessions listed"`, `"Session details loaded"`, `"Session deleted"`, `"Export completed"`, `"Import completed"`). diff --git a/features/semantic_context_search.feature b/features/semantic_context_search.feature new file mode 100644 index 000000000..4b4c55b63 --- /dev/null +++ b/features/semantic_context_search.feature @@ -0,0 +1,57 @@ +Feature: Semantic context search using embeddings + As a context assembly system + I want to use semantic embeddings to find relevant files + So that I can select the most semantically relevant context fragments + + Background: + Given I have a semantic context strategy with embeddings + And I have a mock embedding provider + + Scenario: Embed text fragments + When I embed the text "Python function definition" + Then the embedding should have 10 dimensions + And the embedding should be a valid vector + + Scenario: Compute cosine similarity between embeddings + Given I have two text embeddings + When I compute their cosine similarity + Then the similarity should be between -1 and 1 + + Scenario: Rank fragments by semantic similarity + Given I have context fragments with content: + | content | + | Python function definition | + | JavaScript class syntax | + | Python class implementation| + And I have a query "Python code structure" + When I rank fragments by semantic similarity to the query + Then the Python fragments should rank higher than JavaScript + + Scenario: Filter fragments by minimum similarity threshold + Given I have context fragments with content: + | content | + | Python function definition | + | Unrelated database schema | + | Python class implementation| + And I have a query "Python code" + And I have a minimum similarity threshold of 0.3 + When I filter fragments by similarity threshold + Then only semantically similar fragments should be included + + Scenario: Semantic strategy selects relevant files + Given I have a semantic context strategy + And I have context fragments: + | uko_node | content | token_count | + | file1.py | Python function definition | 10 | + | file2.js | JavaScript class syntax | 10 | + | file3.py | Python class implementation| 10 | + And I have a context budget of 30 tokens + When I assemble semantic context with query "Python code structure" + Then the selected fragments should include Python files + And the selected fragments should be ranked by relevance + + Scenario: Embedding provider configuration + Given I have an embedding provider configuration + When I create a semantic strategy with the configuration + Then the strategy should use the configured embedding provider + And the strategy should report semantic search capability diff --git a/features/steps/semantic_context_search_steps.py b/features/steps/semantic_context_search_steps.py new file mode 100644 index 000000000..4a17f1fce --- /dev/null +++ b/features/steps/semantic_context_search_steps.py @@ -0,0 +1,258 @@ +"""Step definitions for semantic context search feature tests.""" + +from __future__ import annotations + +from behave import given, then, when + +from cleveragents.application.services.embedding_provider import ( + MockEmbeddingProvider, + SimpleWordEmbeddingProvider, + cosine_similarity, +) +from cleveragents.domain.models.core.context_fragment import ( + ContextBudget, + ContextFragment, + FragmentProvenance, +) + + +@given("I have a semantic context strategy with embeddings") +def step_have_semantic_strategy(context): + """Initialize a semantic context strategy.""" + context.embedding_provider = MockEmbeddingProvider(embedding_dim=10) + context.fragments = [] + + +@given("I have a mock embedding provider") +def step_have_mock_provider(context): + """Initialize a mock embedding provider.""" + context.embedding_provider = MockEmbeddingProvider(embedding_dim=10) + + +@when("I embed the text {text}") +def step_embed_text(context, text): + """Embed a text fragment.""" + context.embedding = context.embedding_provider.embed(text) + + +@then("the embedding should have {dim:d} dimensions") +def step_check_embedding_dimension(context, dim): + """Verify embedding dimension.""" + assert len(context.embedding) == dim, ( + f"Expected {dim} dimensions, got {len(context.embedding)}" + ) + + +@then("the embedding should be a valid vector") +def step_check_valid_vector(context): + """Verify embedding is a valid vector.""" + assert isinstance(context.embedding, (list, tuple)) + assert all(isinstance(x, (int, float)) for x in context.embedding) + assert len(context.embedding) > 0 + + +@given("I have two text embeddings") +def step_have_two_embeddings(context): + """Create two text embeddings.""" + context.embedding1 = context.embedding_provider.embed("Python function") + context.embedding2 = context.embedding_provider.embed("Python class") + + +@when("I compute their cosine similarity") +def step_compute_similarity(context): + """Compute cosine similarity.""" + context.similarity = cosine_similarity(context.embedding1, context.embedding2) + + +@then("the similarity should be between -1 and 1") +def step_check_similarity_range(context): + """Verify similarity is in valid range.""" + assert -1 <= context.similarity <= 1, ( + f"Similarity {context.similarity} out of range" + ) + + +@given("I have context fragments with content:") +def step_have_fragments_with_content(context): + """Create context fragments from table.""" + context.fragments = [] + for row in context.table: + content = row["content"] + embedding = context.embedding_provider.embed(content) + context.fragments.append( + { + "content": content, + "embedding": embedding, + } + ) + + +@given("I have a query {query}") +def step_have_query(context, query): + """Set the query.""" + context.query = query + context.query_embedding = context.embedding_provider.embed(query) + + +@when("I rank fragments by semantic similarity to the query") +def step_rank_fragments(context): + """Rank fragments by similarity.""" + similarities = [] + for frag in context.fragments: + sim = cosine_similarity(context.query_embedding, frag["embedding"]) + similarities.append((frag, sim)) + + similarities.sort(key=lambda x: x[1], reverse=True) + context.ranked_fragments = similarities + + +@then("the Python fragments should rank higher than JavaScript") +def step_check_python_ranking(context): + """Verify Python fragments rank higher.""" + python_sims = [ + sim for frag, sim in context.ranked_fragments if "Python" in frag["content"] + ] + js_sims = [ + sim for frag, sim in context.ranked_fragments if "JavaScript" in frag["content"] + ] + + if python_sims and js_sims: + assert min(python_sims) >= max(js_sims), "Python fragments should rank higher" + + +@given("I have a minimum similarity threshold of {threshold:f}") +def step_have_threshold(context, threshold): + """Set similarity threshold.""" + context.threshold = threshold + + +@when("I filter fragments by similarity threshold") +def step_filter_by_threshold(context): + """Filter fragments by threshold. + + Computes similarity inline so the step works whether or not a prior + "rank fragments" step ran. Scenarios that filter directly (without an + intermediate ranking step) would otherwise hit an AttributeError on + ``context.ranked_fragments``. + """ + scored = [ + (frag, cosine_similarity(context.query_embedding, frag["embedding"])) + for frag in context.fragments + ] + context.filtered_fragments = [ + (frag, sim) for frag, sim in scored if sim >= context.threshold + ] + + +@then("only semantically similar fragments should be included") +def step_check_filtered_fragments(context): + """Verify filtered fragments meet threshold.""" + for _frag, sim in context.filtered_fragments: + assert sim >= context.threshold, ( + f"Fragment similarity {sim} below threshold {context.threshold}" + ) + + +@given("I have a semantic context strategy") +def step_have_strategy(context): + """Initialize semantic strategy.""" + context.embedding_provider = SimpleWordEmbeddingProvider(vocab_size=100) + + +@given("I have context fragments:") +def step_have_context_fragments(context): + """Create context fragments from table.""" + context.fragments = [] + for row in context.table: + frag = ContextFragment( + uko_node=row["uko_node"], + content=row["content"], + token_count=int(row["token_count"]), + relevance_score=0.5, + detail_depth=1, + tier="hot", + provenance=FragmentProvenance(resource_uri=row["uko_node"]), + ) + context.fragments.append(frag) + + +@given("I have a context budget of {tokens:d} tokens") +def step_have_budget(context, tokens): + """Set context budget.""" + context.budget = ContextBudget(max_tokens=tokens, reserved_tokens=0) + + +@when("I assemble semantic context with query {query}") +def step_assemble_context(context, query): + """Assemble context with query.""" + context.query = query + query_embedding = context.embedding_provider.embed(query) + + # Score fragments by similarity + scored = [] + for frag in context.fragments: + frag_embedding = context.embedding_provider.embed(frag.content) + sim = cosine_similarity(query_embedding, frag_embedding) + scored.append((frag, sim)) + + # Sort by similarity + scored.sort(key=lambda x: x[1], reverse=True) + + # Pack within budget + context.selected_fragments = [] + total_tokens = 0 + for frag, _sim in scored: + if total_tokens + frag.token_count <= context.budget.max_tokens: + context.selected_fragments.append(frag) + total_tokens += frag.token_count + + +@then("the selected fragments should include Python files") +def step_check_python_files(context): + """Verify Python files are selected.""" + python_files = [f for f in context.selected_fragments if ".py" in f.uko_node] + assert len(python_files) > 0, "No Python files selected" + + +@then("the selected fragments should be ranked by relevance") +def step_check_ranking(context): + """Verify fragments are ranked by relevance.""" + assert len(context.selected_fragments) > 0, "No fragments selected" + + +@given("I have an embedding provider configuration") +def step_have_config(context): + """Initialize embedding provider configuration. + + Stored under ``embedding_config`` rather than ``config`` because Behave's + ``Context`` reserves ``config`` for its own Configuration object — setting + ``context.config`` raises ``KeyError`` from Behave's scope-tracking + ``__setattr__``. + """ + context.embedding_config = { + "provider_type": "simple_word", + "vocab_size": 100, + } + + +@when("I create a semantic strategy with the configuration") +def step_create_strategy_with_config(context): + """Create strategy with configuration.""" + if context.embedding_config["provider_type"] == "simple_word": + context.strategy_provider = SimpleWordEmbeddingProvider( + vocab_size=context.embedding_config["vocab_size"] + ) + + +@then("the strategy should use the configured embedding provider") +def step_check_configured_provider(context): + """Verify strategy uses configured provider.""" + assert context.strategy_provider is not None + assert context.strategy_provider.embedding_dimension > 0 + + +@then("the strategy should report semantic search capability") +def step_check_capability(context): + """Verify strategy reports semantic capability.""" + assert hasattr(context.strategy_provider, "embedding_dimension") + assert context.strategy_provider.embedding_dimension > 0 diff --git a/src/cleveragents/application/services/embedding_provider.py b/src/cleveragents/application/services/embedding_provider.py new file mode 100644 index 000000000..a3ef0fbdf --- /dev/null +++ b/src/cleveragents/application/services/embedding_provider.py @@ -0,0 +1,231 @@ +"""Embedding provider interface and implementations for semantic context search. + +Provides pluggable embedding generation for semantic similarity scoring in context +strategies. Supports both local models and API-based providers. + +Based on ACMS glossary section for embedding-based context retrieval in CleverAgents. +""" + +from __future__ import annotations + +import hashlib +import logging +from abc import ABC, abstractmethod +from collections.abc import Sequence + +logger = logging.getLogger(__name__) + + +class EmbeddingProvider(ABC): + """Abstract base class for embedding providers. + + Implementations generate vector embeddings for text fragments to enable + semantic similarity scoring in context strategies. + """ + + @abstractmethod + def embed(self, text: str) -> Sequence[float]: + """Generate an embedding vector for the given text. + + Args: + text: The text to embed. + + Returns: + A sequence of floats representing the embedding vector. + + Raises: + ValueError: If the text cannot be embedded. + """ + ... + + @abstractmethod + def embed_batch(self, texts: Sequence[str]) -> Sequence[Sequence[float]]: + """Generate embedding vectors for multiple texts. + + Args: + texts: A sequence of texts to embed. + + Returns: + A sequence of embedding vectors. + + Raises: + ValueError: If any text cannot be embedded. + """ + ... + + @property + @abstractmethod + def embedding_dimension(self) -> int: + """Return the dimension of the embedding vectors.""" + ... + + +class SimpleWordEmbeddingProvider(EmbeddingProvider): + """Simple word-based embedding provider using TF-IDF-like scoring. + + This provider generates embeddings based on word frequency and uniqueness + without requiring external models or APIs. It's suitable for lightweight + lexical similarity scoring; not a substitute for neural embedding models. + """ + + def __init__(self, vocab_size: int = 1000) -> None: + """Initialize the simple word embedding provider. + + Args: + vocab_size: Maximum vocabulary size for embeddings. + """ + self._vocab_size = vocab_size + self._vocab: dict[str, int] = {} + self._embedding_dim = min(vocab_size, 100) + + def embed(self, text: str) -> Sequence[float]: + """Generate a simple embedding based on word frequencies. + + Args: + text: The text to embed. + + Returns: + A sequence of floats representing word frequency scores. + """ + words = self._tokenize(text) + embedding: list[float] = [0.0] * self._embedding_dim + + for word in words: + word_id = self._get_word_id(word) + if word_id < self._embedding_dim: + embedding[word_id] += 1.0 + + # Normalize + total = sum(embedding) + if total > 0: + embedding = [x / total for x in embedding] + + return embedding + + def embed_batch(self, texts: Sequence[str]) -> Sequence[Sequence[float]]: + """Generate embeddings for multiple texts. + + Args: + texts: A sequence of texts to embed. + + Returns: + A sequence of embedding vectors. + """ + return [self.embed(text) for text in texts] + + @property + def embedding_dimension(self) -> int: + """Return the dimension of the embedding vectors.""" + return self._embedding_dim + + def _tokenize(self, text: str) -> list[str]: + """Tokenize text into words.""" + return text.lower().split() + + def _get_word_id(self, word: str) -> int: + """Get or assign a unique ID for a word. + + When the vocabulary is full, a warning is logged and the word + is silently assigned to the last bucket. This preserves the + dimensionality of the embedding but may reduce uniqueness. + + Args: + word: The word to resolve to an ID. + + Returns: + A unique integer ID for the word, or vocab_size-1 if full. + """ + if word not in self._vocab: + if len(self._vocab) < self._vocab_size: + self._vocab[word] = len(self._vocab) + else: + logger.warning( + "Vocabulary full (%d words); new word '%s' maps " + "to overflow bucket %d", + self._vocab_size, + word, + self._vocab_size - 1, + ) + return self._vocab_size - 1 + return self._vocab[word] + + +class MockEmbeddingProvider(EmbeddingProvider): + """Mock embedding provider for testing. + + Generates deterministic embeddings based on text content hash. + Uses MD5 hashing to ensure reproducibility across Python sessions + regardless of PYTHONHASHSEED settings. + """ + + def __init__(self, embedding_dim: int = 10) -> None: + """Initialize the mock embedding provider. + + Args: + embedding_dim: Dimension of the embedding vectors. + """ + self._embedding_dim = embedding_dim + + def embed(self, text: str) -> Sequence[float]: + """Generate a deterministic embedding based on text content. + + Uses MD5 hashing to produce reproducible embeddings regardless of + PYTHONHASHSEED settings. + + Args: + text: The text to embed. + + Returns: + A sequence of floats in [0, 1]. + """ + hash_hex = hashlib.md5(text.encode(), usedforsecurity=False).hexdigest() + embedding: list[float] = [] + for i in range(self._embedding_dim): + chunk = hash_hex[i * 2 : i * 2 + 2] + if not chunk: + chunk = "00" + value = int(chunk, 16) / 255.0 + embedding.append(round(value, 6)) + return embedding + + def embed_batch(self, texts: Sequence[str]) -> Sequence[Sequence[float]]: + """Generate embeddings for multiple texts. + + Args: + texts: A sequence of texts to embed. + + Returns: + A sequence of embedding vectors. + """ + return [self.embed(text) for text in texts] + + @property + def embedding_dimension(self) -> int: + """Return the dimension of the embedding vectors.""" + return self._embedding_dim + + +def cosine_similarity(vec_a: Sequence[float], vec_b: Sequence[float]) -> float: + """Compute cosine similarity between two vectors. + + Args: + vec_a: First vector. + vec_b: Second vector. + + Returns: + Cosine similarity score between -1 and 1. + + Raises: + ValueError: If vectors have different dimensions. + """ + if len(vec_a) != len(vec_b): + raise ValueError("Vectors must have the same dimension") + + dot_product = sum(a * b for a, b in zip(vec_a, vec_b, strict=True)) + mag_a = sum(a * a for a in vec_a) ** 0.5 + mag_b = sum(b * b for b in vec_b) ** 0.5 + + if mag_a == 0 or mag_b == 0: + return 0.0 + + return dot_product / (mag_a * mag_b)