forked from cleveragents/cleveragents-core
Fix: improved coverage on context analysis agent
This commit is contained in:
@@ -1,33 +1,25 @@
|
||||
Feature: Context Analysis Agent Coverage
|
||||
As a developer
|
||||
I want comprehensive test coverage for the ContextAnalysisAgent
|
||||
So that I can ensure the context analysis workflow works correctly
|
||||
I want confidence the ContextAnalysisAgent behaves correctly
|
||||
So that context generation remains reliable
|
||||
|
||||
Background:
|
||||
Given the context analysis agent module is importable
|
||||
And I have a mock LLM provider configured for context analysis
|
||||
|
||||
# Initialization and Configuration Tests
|
||||
Scenario: ContextAnalysisAgent can be instantiated with default parameters
|
||||
Scenario: Agent initializes with defaults
|
||||
When I create a ContextAnalysisAgent with default parameters
|
||||
Then the context analysis agent should be initialized successfully
|
||||
And the agent should have a chunk_size attribute set to 2000
|
||||
And the agent should have a chunk_overlap attribute set to 200
|
||||
And the context analysis agent should have an llm provider configured
|
||||
|
||||
Scenario: ContextAnalysisAgent can be instantiated with custom chunk settings
|
||||
When I create a ContextAnalysisAgent with chunk_size 1000 and chunk_overlap 100
|
||||
Then the context analysis agent should be initialized successfully
|
||||
And the agent chunk_size should be 1000
|
||||
And the agent chunk_overlap should be 100
|
||||
Scenario: Agent respects custom chunk configuration
|
||||
When I create a ContextAnalysisAgent with chunk_size 800 and chunk_overlap 80
|
||||
Then the agent should have a chunk_size attribute set to 800
|
||||
And the agent should have a chunk_overlap attribute set to 80
|
||||
|
||||
Scenario: ContextAnalysisAgent creates required prompts during initialization
|
||||
When I create a ContextAnalysisAgent with default parameters
|
||||
Then the agent should have a dependency_prompt attribute
|
||||
And the agent should have a relevance_prompt attribute
|
||||
And the agent should have a summary_prompt attribute
|
||||
|
||||
Scenario: ContextAnalysisAgent builds a valid workflow graph
|
||||
Scenario: Workflow graph contains expected nodes
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
When I inspect the workflow graph
|
||||
Then the graph should contain node "load_files"
|
||||
@@ -35,376 +27,99 @@ Feature: Context Analysis Agent Coverage
|
||||
And the graph should contain node "chunk_documents"
|
||||
And the graph should contain node "score_relevance"
|
||||
And the graph should contain node "summarize_context"
|
||||
And the entry point should be "load_files"
|
||||
|
||||
Scenario: ContextAnalysisAgent has a compiled app with checkpointer
|
||||
Scenario: Load files node reads real files
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
Then the agent should have an app attribute
|
||||
And the agent should have a checkpointer attribute
|
||||
And the checkpointer should be a MemorySaver instance
|
||||
|
||||
# ContextAnalysisState Structure Tests
|
||||
Scenario: ContextAnalysisState holds required workflow data
|
||||
Given I can create a ContextAnalysisState
|
||||
When I initialize it with all required fields:
|
||||
| field | type |
|
||||
| file_paths | list |
|
||||
| documents | list |
|
||||
| dependencies | dict |
|
||||
| summary | str |
|
||||
| relevance_scores | dict |
|
||||
| chunks | list |
|
||||
| error | str |
|
||||
Then the state should store all fields correctly for context analysis
|
||||
|
||||
# Load Files Node Tests
|
||||
Scenario: Load files node loads valid file paths
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a temporary test file at "test.py" with content:
|
||||
And I have a temporary test file named "example.py" with content:
|
||||
"""
|
||||
import os
|
||||
print("Hello")
|
||||
print("hi")
|
||||
"""
|
||||
When I execute the load_files node with file paths:
|
||||
"""
|
||||
["example.py"]
|
||||
"""
|
||||
When I execute the load_files node with file paths ["test.py"]
|
||||
Then the state should contain documents
|
||||
And the documents list should have 1 document
|
||||
And the first document should contain "Hello"
|
||||
And the documents list should have 1 documents
|
||||
And the first document should contain "hi"
|
||||
And there should be no error
|
||||
|
||||
Scenario: Load files node handles non-existent files
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
When I execute the load_files node with file paths ["nonexistent.py"]
|
||||
Then the state should contain documents
|
||||
And the documents list should be empty
|
||||
And the error should contain "File not found"
|
||||
|
||||
Scenario: Load files node handles multiple files
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a temporary test file at "file1.py" with content "# File 1"
|
||||
And I have a temporary test file at "file2.py" with content "# File 2"
|
||||
When I execute the load_files node with file paths ["file1.py", "file2.py"]
|
||||
Then the documents list should have 2 documents
|
||||
|
||||
Scenario: Load files node handles directories as invalid input
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a temporary test directory at "testdir"
|
||||
When I execute the load_files node with file paths ["testdir"]
|
||||
Then the error should contain "Not a file"
|
||||
|
||||
# Analyze Dependencies Node Tests
|
||||
Scenario: Analyze dependencies extracts imports from documents
|
||||
Scenario: Dependency analysis returns structured data
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a state with loaded documents containing:
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
"""
|
||||
When I execute the analyze_dependencies node
|
||||
Then the state should contain dependencies
|
||||
And the dependencies should be a dictionary
|
||||
And the dependencies should contain at least one file
|
||||
|
||||
Scenario: Analyze dependencies handles empty document list
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a state with no documents
|
||||
When I execute the analyze_dependencies node
|
||||
Then the dependencies should be an empty dictionary
|
||||
|
||||
Scenario: Analyze dependencies limits results to 10 per file
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a state with a document containing many imports
|
||||
When I execute the analyze_dependencies node
|
||||
Then each file in dependencies should have at most 10 entries
|
||||
|
||||
Scenario: Analyze dependencies handles LLM errors gracefully
|
||||
Given I have a ContextAnalysisAgent instance with failing LLM
|
||||
And I have a state with loaded documents
|
||||
When I execute the analyze_dependencies node
|
||||
Then the error should contain "Dependency analysis error"
|
||||
|
||||
# Chunk Documents Node Tests
|
||||
Scenario: Chunk documents keeps small files intact
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a state with a document of 100 characters
|
||||
When I execute the chunk_documents node
|
||||
Then the chunks list should have 1 chunk
|
||||
And the chunk should match the original document
|
||||
|
||||
Scenario: Chunk documents splits large files with overlap
|
||||
Given I have a ContextAnalysisAgent instance with chunk_size 100 and chunk_overlap 20
|
||||
And I have a state with a document of 250 characters
|
||||
Scenario: Chunking splits large documents
|
||||
Given I have a ContextAnalysisAgent instance with chunk_size 50 and chunk_overlap 10
|
||||
And I have a state with a document of 140 characters
|
||||
When I execute the chunk_documents node
|
||||
Then the chunks list should have at least 2 chunks
|
||||
And each chunk should have chunk_index in metadata
|
||||
|
||||
Scenario: Chunk documents handles multiple documents
|
||||
Scenario: Relevance scoring produces values per file
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a state with 3 documents of varying sizes
|
||||
When I execute the chunk_documents node
|
||||
Then the chunks list should contain chunks from all documents
|
||||
|
||||
Scenario: Chunk documents preserves metadata
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a state with a document containing source metadata
|
||||
When I execute the chunk_documents node
|
||||
Then all chunks should preserve the source metadata
|
||||
|
||||
# Score Relevance Node Tests
|
||||
Scenario: Score relevance assigns scores to all files
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a state with chunks from 3 different files
|
||||
And I have a state with chunks from 2 different files
|
||||
When I execute the score_relevance node
|
||||
Then the relevance_scores should be a dictionary
|
||||
And the relevance_scores should contain 3 entries
|
||||
And the relevance_scores should contain 2 entries
|
||||
And all scores should be between 0.0 and 1.0
|
||||
|
||||
Scenario: Score relevance uses LLM for scoring
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a state with chunks containing "high priority code"
|
||||
When I execute the score_relevance node
|
||||
Then the relevance_scores should contain positive values
|
||||
|
||||
Scenario: Score relevance defaults to 0.5 on errors
|
||||
Given I have a ContextAnalysisAgent instance with failing LLM
|
||||
And I have a state with chunks from a file
|
||||
When I execute the score_relevance node
|
||||
Then the relevance_scores should contain 0.5 for the file
|
||||
|
||||
Scenario: Score relevance parses high/medium/low from LLM output
|
||||
Given I have a ContextAnalysisAgent instance with LLM returning "High relevance"
|
||||
And I have a state with one chunk
|
||||
When I execute the score_relevance node
|
||||
Then the relevance score should be approximately 0.8
|
||||
|
||||
Scenario: Score relevance clamps scores to valid range
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
When I parse relevance score from "Score: 1.5"
|
||||
Then the parsed score should be 1.0
|
||||
|
||||
# Summarize Context Node Tests
|
||||
Scenario: Summarize context creates summary with statistics
|
||||
Scenario: Summarization generates a summary
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a complete analysis state with:
|
||||
| documents | 5 files |
|
||||
| dependencies | 10 imports |
|
||||
| relevance_scores | 5 scores |
|
||||
| field | value |
|
||||
| documents | 3 |
|
||||
| dependencies | 5 |
|
||||
| relevance_scores | 3 |
|
||||
When I execute the summarize_context node
|
||||
Then the state should contain a summary
|
||||
And the summary should mention the file count
|
||||
And the summary should mention the dependency count
|
||||
|
||||
Scenario: Summarize context identifies top files by relevance
|
||||
Scenario: Complete workflow processes files end to end
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a state with files scored: fileA=0.9, fileB=0.7, fileC=0.5, fileD=0.3
|
||||
When I execute the summarize_context node
|
||||
Then the summary should include the top 3 files
|
||||
And the top file should be fileA
|
||||
|
||||
Scenario: Summarize context handles empty state gracefully
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a minimal state with empty collections
|
||||
When I execute the summarize_context node
|
||||
Then the summary should be generated without errors
|
||||
|
||||
Scenario: Summarize context handles LLM errors
|
||||
Given I have a ContextAnalysisAgent instance with failing LLM
|
||||
And I have a complete analysis state
|
||||
When I execute the summarize_context node
|
||||
Then the summary should be "Context analysis failed"
|
||||
And the error should contain "Summarization error"
|
||||
|
||||
# Full Workflow Tests
|
||||
Scenario: Complete workflow with single file
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a temporary test file at "main.py" with content:
|
||||
And I have temporary test files:
|
||||
| filename | content |
|
||||
| a.py | import os\nprint("A") |
|
||||
| b.py | import sys\nprint("B") |
|
||||
When I run the complete workflow with file paths:
|
||||
"""
|
||||
import os
|
||||
def main():
|
||||
print("Hello World")
|
||||
["a.py", "b.py"]
|
||||
"""
|
||||
When I run the complete workflow with file paths ["main.py"]
|
||||
Then the workflow should complete successfully
|
||||
And the final state should contain documents
|
||||
And the final state should contain dependencies
|
||||
And the final state should contain chunks
|
||||
And the final state should contain relevance_scores
|
||||
And the final state should contain a summary
|
||||
|
||||
Scenario: Complete workflow with multiple files
|
||||
Scenario: Async execution matches sync behavior
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have temporary test files:
|
||||
| filename | content |
|
||||
| file1.py | import os\nprint("test1") |
|
||||
| file2.py | import sys\nprint("test2") |
|
||||
| file3.py | import json\ndata = {} |
|
||||
When I run the complete workflow with all test file paths
|
||||
Then the workflow should complete successfully
|
||||
And the documents list should have 3 documents
|
||||
And the dependencies should cover all 3 files
|
||||
And the relevance_scores should cover all 3 files
|
||||
|
||||
Scenario: Complete workflow with empty file list
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
When I run the complete workflow with file paths []
|
||||
Then the workflow should complete successfully
|
||||
And the documents list should be empty
|
||||
And the dependencies should be empty
|
||||
And the summary should handle zero files
|
||||
|
||||
Scenario: Complete workflow handles mixed valid and invalid files
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a temporary test file at "valid.py" with content "# Valid"
|
||||
When I run the complete workflow with file paths ["valid.py", "missing.py"]
|
||||
Then the workflow should complete successfully
|
||||
And the documents list should have 1 document
|
||||
And the error should contain "File not found"
|
||||
|
||||
# Async Execution Tests
|
||||
Scenario: Async invoke executes workflow asynchronously
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a temporary test file at "async_test.py" with content "# Test"
|
||||
When I run the workflow asynchronously with file paths ["async_test.py"]
|
||||
And I have a temporary test file named "async.py" with content "print('x')"
|
||||
When I run the workflow asynchronously with file paths:
|
||||
"""
|
||||
["async.py"]
|
||||
"""
|
||||
Then the async workflow should complete successfully
|
||||
And the final state should contain all expected fields
|
||||
|
||||
Scenario: Stream execution yields intermediate states
|
||||
Scenario: Streaming produces intermediate updates
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a temporary test file at "stream_test.py" with content "# Test"
|
||||
When I stream the workflow with file paths ["stream_test.py"]
|
||||
And I have a temporary test file named "stream.py" with content "print('x')"
|
||||
When I stream the workflow with file paths:
|
||||
"""
|
||||
["stream.py"]
|
||||
"""
|
||||
Then I should receive multiple state updates
|
||||
And each update should correspond to a node execution
|
||||
|
||||
Scenario: Async stream execution yields intermediate states
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a temporary test file at "astream_test.py" with content "# Test"
|
||||
When I async stream the workflow with file paths ["astream_test.py"]
|
||||
Then I should receive multiple async state updates
|
||||
|
||||
# Helper Method Tests
|
||||
Scenario: Parse dependencies extracts module names from LLM output
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
When I parse dependencies from "Dependencies: ['os', 'sys', 'pathlib']"
|
||||
Then the parsed dependencies should include "os"
|
||||
And the parsed dependencies should include "sys"
|
||||
And the parsed dependencies should include "pathlib"
|
||||
|
||||
Scenario: Parse dependencies handles various output formats
|
||||
Scenario: Helper parsing handles structured dependency output
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
When I parse dependencies from:
|
||||
"""
|
||||
Extracted modules:
|
||||
- os
|
||||
- sys
|
||||
- json
|
||||
Dependencies: ['os', 'sys', 'pathlib']
|
||||
"""
|
||||
Then the parsed dependencies should contain at least 3 items
|
||||
|
||||
Scenario: Parse dependencies limits to 10 items
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
When I parse dependencies from a string with 15 module names
|
||||
Then the parsed dependencies should have exactly 10 items
|
||||
|
||||
Scenario: Parse relevance score handles numeric scores
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
When I parse relevance score from "Score: 0.75"
|
||||
Then the parsed score should be 0.75
|
||||
|
||||
Scenario: Parse relevance score handles keyword high
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
When I parse relevance score from "High relevance for this file"
|
||||
Then the parsed score should be 0.8
|
||||
|
||||
Scenario: Parse relevance score handles keyword low
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
When I parse relevance score from "Low relevance for this file"
|
||||
Then the parsed score should be 0.3
|
||||
|
||||
Scenario: Parse relevance score defaults to medium
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
When I parse relevance score from "Uncertain about relevance"
|
||||
Then the parsed score should be 0.5
|
||||
|
||||
# Error Handling and Edge Cases
|
||||
Scenario: Workflow continues despite individual file errors
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a temporary test file at "good.py" with content "# Good"
|
||||
When I run the complete workflow with file paths ["good.py", "bad.py"]
|
||||
Then the workflow should complete successfully
|
||||
And the documents list should have 1 document
|
||||
And the error field should contain information about failures
|
||||
|
||||
Scenario: Empty documents do not crash dependency analysis
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a state with an empty document
|
||||
When I execute the analyze_dependencies node
|
||||
Then the dependencies should handle the empty document gracefully
|
||||
|
||||
Scenario: Chunking handles documents at exact chunk_size boundary
|
||||
Given I have a ContextAnalysisAgent instance with chunk_size 100
|
||||
And I have a state with a document of exactly 100 characters
|
||||
When I execute the chunk_documents node
|
||||
Then the chunks list should have 1 chunk
|
||||
|
||||
Scenario: Relevance scoring handles duplicate file sources
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a state with multiple chunks from the same file
|
||||
When I execute the score_relevance node
|
||||
Then the file should appear only once in relevance_scores
|
||||
|
||||
# Configuration and Checkpointing Tests
|
||||
Scenario: Workflow execution with custom config
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a temporary test file at "config_test.py" with content "# Test"
|
||||
When I run the workflow with config {"thread_id": "test-123"}
|
||||
Then the workflow should use the provided config
|
||||
And the workflow should complete successfully
|
||||
|
||||
Scenario: Checkpointer enables resumable execution
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a temporary test file at "resume_test.py" with content "# Test"
|
||||
When I start a workflow with thread_id "resume-1"
|
||||
And I interrupt the workflow after load_files
|
||||
And I resume the workflow with thread_id "resume-1"
|
||||
Then the workflow should continue from the interruption point
|
||||
|
||||
# Integration with Domain Models
|
||||
Scenario: Workflow state is compatible with domain Context model
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have a temporary test file at "domain_test.py" with content "# Test"
|
||||
When I run the complete workflow with file paths ["domain_test.py"]
|
||||
Then the final state can be converted to a Context domain model
|
||||
And the Context model should have type "code"
|
||||
|
||||
# Performance and Limits
|
||||
Scenario: Workflow handles large number of small files
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have 20 temporary test files with small content
|
||||
When I run the complete workflow with all file paths
|
||||
Then the workflow should complete in reasonable time
|
||||
And all 20 files should be analyzed
|
||||
|
||||
Scenario: Workflow handles few large files
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
And I have 2 temporary test files with 5000 character content each
|
||||
When I run the complete workflow with both file paths
|
||||
Then the files should be chunked appropriately
|
||||
And the workflow should complete successfully
|
||||
|
||||
# Graph Structure Validation
|
||||
Scenario: Workflow graph edges are correctly defined
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
When I inspect the workflow graph edges
|
||||
Then "load_files" should connect to "analyze_dependencies"
|
||||
And "analyze_dependencies" should connect to "chunk_documents"
|
||||
And "chunk_documents" should connect to "score_relevance"
|
||||
And "score_relevance" should connect to "summarize_context"
|
||||
And "summarize_context" should connect to END
|
||||
|
||||
Scenario: All workflow nodes are callable
|
||||
Given I have a ContextAnalysisAgent instance
|
||||
Then the load_files node should be callable
|
||||
And the analyze_dependencies node should be callable
|
||||
And the chunk_documents node should be callable
|
||||
And the score_relevance node should be callable
|
||||
And the summarize_context node should be callable
|
||||
Then the parsed dependencies should include "os"
|
||||
And the parsed dependencies should include "sys"
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
+550
-545
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user