212 lines
8.6 KiB
Gherkin
212 lines
8.6 KiB
Gherkin
Feature: Context Analysis Agent Coverage
|
|
As a developer
|
|
I want confidence the ContextAnalysisAgent behaves correctly
|
|
So that context generation remains reliable
|
|
|
|
Background:
|
|
Given the context analysis agent module is importable
|
|
And I have a mock LLM provider configured for context analysis
|
|
|
|
Scenario: Agent initializes with defaults
|
|
When I create a ContextAnalysisAgent with default parameters
|
|
Then the context analysis agent should be initialized successfully
|
|
And the agent should have a chunk_size attribute set to 2000
|
|
And the agent should have a chunk_overlap attribute set to 200
|
|
And the context analysis agent should have an llm provider configured
|
|
|
|
Scenario: Agent respects custom chunk configuration
|
|
When I create a ContextAnalysisAgent with chunk_size 800 and chunk_overlap 80
|
|
Then the agent should have a chunk_size attribute set to 800
|
|
And the agent should have a chunk_overlap attribute set to 80
|
|
|
|
Scenario: Workflow graph contains expected nodes
|
|
Given I have a ContextAnalysisAgent instance
|
|
When I inspect the workflow graph
|
|
Then the graph should contain node "load_files"
|
|
And the graph should contain node "analyze_dependencies"
|
|
And the graph should contain node "chunk_documents"
|
|
And the graph should contain node "score_relevance"
|
|
And the graph should contain node "summarize_context"
|
|
|
|
Scenario: Load files node reads real files
|
|
Given I have a ContextAnalysisAgent instance
|
|
And I have a temporary test file named "example.py" with content:
|
|
"""
|
|
import os
|
|
print("hi")
|
|
"""
|
|
When I execute the load_files node with file paths:
|
|
"""
|
|
["example.py"]
|
|
"""
|
|
Then the state should contain documents
|
|
And the documents list should have 1 documents
|
|
And the first document should contain "hi"
|
|
And there should be no error
|
|
|
|
Scenario: Load files node reports missing and invalid paths
|
|
Given I have a ContextAnalysisAgent instance
|
|
And I have a temporary directory named "invalid"
|
|
When I execute the load_files node with file paths:
|
|
"""
|
|
["missing.py", "invalid"]
|
|
"""
|
|
Then the state should contain documents
|
|
And the documents list should have 0 documents
|
|
And the state error should contain "File not found"
|
|
And the state error should contain "Not a file"
|
|
|
|
Scenario: Dependency analysis returns structured data
|
|
Given I have a ContextAnalysisAgent instance
|
|
And I have a state with loaded documents containing:
|
|
"""
|
|
import os
|
|
import sys
|
|
"""
|
|
When I execute the analyze_dependencies node
|
|
Then the state should contain dependencies
|
|
And the dependencies should be a dictionary
|
|
|
|
Scenario: Chunking splits large documents
|
|
Given I have a ContextAnalysisAgent instance with chunk_size 50 and chunk_overlap 10
|
|
And I have a state with a document of 140 characters
|
|
When I execute the chunk_documents node
|
|
Then the chunks list should have at least 2 chunks
|
|
|
|
Scenario: Relevance scoring produces values per file
|
|
Given I have a ContextAnalysisAgent instance
|
|
And I have a state with chunks from 2 different files
|
|
When I execute the score_relevance node
|
|
Then the relevance_scores should be a dictionary
|
|
And the relevance_scores should contain 2 entries
|
|
And all scores should be between 0.0 and 1.0
|
|
|
|
Scenario: Summarization generates a summary
|
|
Given I have a ContextAnalysisAgent instance
|
|
And I have a complete analysis state with:
|
|
| field | value |
|
|
| documents | 3 |
|
|
| dependencies | 5 |
|
|
| relevance_scores | 3 |
|
|
When I execute the summarize_context node
|
|
Then the state should contain a summary
|
|
|
|
Scenario: ContextAnalysisAgent retries transient summary failures
|
|
Given I have a ContextAnalysisAgent instance
|
|
And I have a complete analysis state with:
|
|
| field | value |
|
|
| documents | 2 |
|
|
| dependencies | 2 |
|
|
| relevance_scores | 2 |
|
|
When I execute the summarize_context node with a flaky LLM that fails once
|
|
Then the summarize_context node should succeed after retry
|
|
|
|
Scenario: Complete workflow processes files end to end
|
|
Given I have a ContextAnalysisAgent instance
|
|
And I have temporary test files:
|
|
| filename | content |
|
|
| a.py | import os\nprint("A") |
|
|
| b.py | import sys\nprint("B") |
|
|
When I run the complete workflow with file paths:
|
|
"""
|
|
["a.py", "b.py"]
|
|
"""
|
|
Then the workflow should complete successfully
|
|
And the final state should contain documents
|
|
And the final state should contain dependencies
|
|
And the final state should contain relevance_scores
|
|
And the final state should contain a summary
|
|
|
|
Scenario: Async execution matches sync behavior
|
|
Given I have a ContextAnalysisAgent instance
|
|
And I have a temporary test file named "async.py" with content "print('x')"
|
|
When I run the workflow asynchronously with file paths:
|
|
"""
|
|
["async.py"]
|
|
"""
|
|
Then the async workflow should complete successfully
|
|
And the final state should contain all expected fields
|
|
|
|
Scenario: Streaming produces intermediate updates
|
|
Given I have a ContextAnalysisAgent instance
|
|
And I have a temporary test file named "stream.py" with content "print('x')"
|
|
When I stream the workflow with file paths:
|
|
"""
|
|
["stream.py"]
|
|
"""
|
|
Then I should receive multiple state updates
|
|
And each update should correspond to a node execution
|
|
|
|
Scenario: Helper parsing handles structured dependency output
|
|
Given I have a ContextAnalysisAgent instance
|
|
When I parse dependencies from:
|
|
"""
|
|
Dependencies: ['os', 'sys', 'pathlib']
|
|
"""
|
|
Then the parsed dependencies should include "os"
|
|
And the parsed dependencies should include "sys"
|
|
|
|
Scenario: Dependency analysis reports chained errors
|
|
Given I have a ContextAnalysisAgent instance with an LLM that raises "dependency failure"
|
|
And I have a state with loaded documents containing:
|
|
"""
|
|
import pathlib
|
|
"""
|
|
And the state error is "load failure"
|
|
When I execute the analyze_dependencies node
|
|
Then the state should contain dependencies
|
|
And the dependencies for "test.py" should be empty
|
|
And the state error should contain "load failure"
|
|
And the state error should contain "dependency failure"
|
|
|
|
Scenario: Relevance scoring skips duplicate chunks per file
|
|
Given I have a ContextAnalysisAgent instance
|
|
And I have a state with duplicate chunks from "dup.py"
|
|
When I execute the score_relevance node
|
|
Then the relevance_scores should be a dictionary
|
|
And the relevance_scores should contain 1 entries
|
|
And all scores should be between 0.0 and 1.0
|
|
|
|
Scenario: Relevance scoring reports LLM errors and preserves prior issues
|
|
Given I have a ContextAnalysisAgent instance with an LLM that raises "relevance failure"
|
|
And I have a state with chunks from 1 different files
|
|
And the state error is "previous issue"
|
|
When I execute the score_relevance node
|
|
Then the relevance_scores should be a dictionary
|
|
And the relevance_scores should contain 1 entries
|
|
And the state error should contain "previous issue"
|
|
And the state error should contain "relevance failure"
|
|
|
|
Scenario: Relevance parser handles qualitative hints
|
|
Given I have a ContextAnalysisAgent instance
|
|
When I parse relevance scores from hints:
|
|
| hint | expected |
|
|
| High likelihood of use | 0.8 |
|
|
| low confidence in result | 0.3 |
|
|
| outcome undecided | 0.5 |
|
|
Then the parsed scores should match expected values
|
|
|
|
Scenario: Summarization merges prior errors when LLM fails
|
|
Given I have a ContextAnalysisAgent instance with an LLM that raises "summary failure"
|
|
And I have a complete analysis state with:
|
|
| field | value |
|
|
| documents | 2 |
|
|
| dependencies | 2 |
|
|
| relevance_scores | 2 |
|
|
And the state error is "previous pipeline error"
|
|
When I execute the summarize_context node
|
|
Then the state should contain a summary
|
|
And the summary should equal "Context analysis failed"
|
|
And the state error should contain "summary failure"
|
|
And the state error should contain "previous pipeline error"
|
|
|
|
Scenario: Async streaming produces node updates
|
|
Given I have a ContextAnalysisAgent instance
|
|
And I have a temporary test file named "astream.py" with content "print('stream')"
|
|
When I stream the workflow asynchronously with file paths:
|
|
"""
|
|
["astream.py"]
|
|
"""
|
|
Then I should receive multiple state updates
|
|
And each update should correspond to a node execution
|