forked from cleveragents/cleveragents-core
326 lines
13 KiB
Gherkin
326 lines
13 KiB
Gherkin
Feature: Plan Generation Agent Coverage
|
|
As a developer
|
|
I want comprehensive test coverage for the PlanGenerationGraph agent
|
|
So that I can ensure the plan generation workflow works correctly
|
|
|
|
Background:
|
|
Given the plan generation agent module is importable
|
|
And I have a mock LLM provider configured
|
|
|
|
Scenario: PlanGenerationGraph can be instantiated with default parameters
|
|
When I create a PlanGenerationGraph with default parameters
|
|
Then the agent should be initialized successfully
|
|
And the agent should have a max_refinements attribute set to 2
|
|
And the agent should have an llm provider configured
|
|
|
|
Scenario: PlanGenerationGraph can be instantiated with custom parameters
|
|
When I create a PlanGenerationGraph with parameters:
|
|
| parameter | value |
|
|
| provider | openai |
|
|
| model | gpt-4 |
|
|
| temperature | 0.5 |
|
|
| max_refinements | 3 |
|
|
Then the agent should be initialized successfully
|
|
And the agent max_refinements should be 3
|
|
And the agent temperature should be 0.5 for plan generation
|
|
|
|
Scenario: PlanGenerationGraph builds a valid workflow graph
|
|
Given I have a PlanGenerationGraph instance
|
|
When I build the workflow graph for plan generation
|
|
Then the graph should contain node "analyze_context" for plan generation
|
|
And the graph should contain node "generate_changes" for plan generation
|
|
And the graph should contain node "validate_changes" for plan generation
|
|
And the graph should contain node "refine_changes" for plan generation
|
|
And the graph should contain node "finalize_results" for plan generation
|
|
And the entry point should be "analyze_context" for plan generation
|
|
|
|
Scenario: Analyze context step processes project context and instructions
|
|
Given I have a PlanGenerationGraph instance
|
|
And I have a state with project context:
|
|
"""
|
|
{
|
|
"project_name": "test_project",
|
|
"tech_stack": ["python", "fastapi"],
|
|
"structure": {"src": "application code"}
|
|
}
|
|
"""
|
|
And I have plan instructions "Add a new API endpoint for user management"
|
|
When I execute the analyze_context step
|
|
Then the state messages should contain a context_analysis message
|
|
And the context_analysis should mention "project structure"
|
|
And the LLM should have been invoked with a context analysis prompt
|
|
|
|
Scenario: Generate changes step creates code changes based on analysis
|
|
Given I have a PlanGenerationGraph instance
|
|
And I have a state with context analysis completed
|
|
And I have plan instructions "Create a new user service class"
|
|
When I execute the generate_changes step
|
|
Then the state should contain generated_changes
|
|
And the generated_changes should be a non-empty list
|
|
And the state messages should contain a code_generation message
|
|
And the LLM should have been invoked with a generation prompt
|
|
|
|
Scenario: Generate changes includes previous validation results on refinement
|
|
Given I have a PlanGenerationGraph instance
|
|
And I have a state with context analysis completed
|
|
And I have validation results indicating issues:
|
|
"""
|
|
{
|
|
"is_valid": false,
|
|
"issues": ["Missing error handling", "Incomplete implementation"]
|
|
}
|
|
"""
|
|
And the refinement_count is 1
|
|
When I execute the generate_changes step
|
|
Then the generation prompt should include validation results
|
|
And the state should contain updated generated_changes
|
|
|
|
Scenario: Validate changes step validates generated code
|
|
Given I have a PlanGenerationGraph instance
|
|
And I have a state with generated changes:
|
|
"""
|
|
[
|
|
{
|
|
"file_path": "src/services/user_service.py",
|
|
"operation": "create",
|
|
"content": "class UserService:\n pass",
|
|
"description": "Create user service"
|
|
}
|
|
]
|
|
"""
|
|
When I execute the validate_changes step
|
|
Then the state should contain validation_results
|
|
And the validation_results should have an is_valid field
|
|
And the state messages should contain a validation message
|
|
And the LLM should have been invoked with a validation prompt
|
|
|
|
Scenario: Validate changes checks for syntax, logic, completeness, and best practices
|
|
Given I have a PlanGenerationGraph instance
|
|
And I have a state with generated changes containing code
|
|
When I execute the validate_changes step
|
|
Then the validation prompt should mention "syntax correctness"
|
|
And the validation prompt should mention "logic errors"
|
|
And the validation prompt should mention "completeness"
|
|
And the validation prompt should mention "best practices"
|
|
|
|
Scenario: Refine changes step increments refinement count
|
|
Given I have a PlanGenerationGraph instance
|
|
And I have a state with refinement_count of 0
|
|
When I execute the refine_changes step
|
|
Then the state refinement_count should be 1
|
|
|
|
Scenario: Refine changes step updates state for regeneration
|
|
Given I have a PlanGenerationGraph instance
|
|
And I have a state with validation failures
|
|
And the refinement_count is 0
|
|
When I execute the refine_changes step
|
|
Then the state should be prepared for regeneration
|
|
And the validation results should still be available
|
|
|
|
Scenario: Finalize results step creates final result structure
|
|
Given I have a PlanGenerationGraph instance
|
|
And I have a state with successful validation:
|
|
"""
|
|
{
|
|
"generated_changes": [{"file_path": "test.py", "operation": "create"}],
|
|
"validation_results": {"is_valid": true, "issues": []},
|
|
"refinement_count": 1
|
|
}
|
|
"""
|
|
When I execute the finalize_results step
|
|
Then the state should contain a result field for plan generation
|
|
And the result should have a changes field
|
|
And the result should have a validation field
|
|
And the result should have a refinement_count field
|
|
And the result should have a success field set to true
|
|
|
|
Scenario: Finalize results with failed validation marks success as false
|
|
Given I have a PlanGenerationGraph instance
|
|
And I have a state with failed validation after max refinements
|
|
When I execute the finalize_results step
|
|
Then the result success field should be false
|
|
|
|
Scenario: Should refine returns "refine" when validation fails and under max refinements
|
|
Given I have a PlanGenerationGraph instance with max_refinements of 2
|
|
And I have a state with validation results:
|
|
"""
|
|
{
|
|
"is_valid": false,
|
|
"issues": ["Error found"]
|
|
}
|
|
"""
|
|
And the refinement_count is 0
|
|
When I check if refinement is needed
|
|
Then the decision should be "refine"
|
|
|
|
Scenario: Should refine returns "refine" on second refinement attempt
|
|
Given I have a PlanGenerationGraph instance with max_refinements of 2
|
|
And I have a state with validation results:
|
|
"""
|
|
{
|
|
"is_valid": false,
|
|
"issues": ["Error found"]
|
|
}
|
|
"""
|
|
And the refinement_count is 1
|
|
When I check if refinement is needed
|
|
Then the decision should be "refine"
|
|
|
|
Scenario: Should refine returns "finalize" when validation succeeds
|
|
Given I have a PlanGenerationGraph instance with max_refinements of 2
|
|
And I have a state with validation results:
|
|
"""
|
|
{
|
|
"is_valid": true,
|
|
"issues": []
|
|
}
|
|
"""
|
|
And the refinement_count is 0
|
|
When I check if refinement is needed
|
|
Then the decision should be "finalize"
|
|
|
|
Scenario: Should refine returns "finalize" when max refinements reached
|
|
Given I have a PlanGenerationGraph instance with max_refinements of 2
|
|
And I have a state with validation results:
|
|
"""
|
|
{
|
|
"is_valid": false,
|
|
"issues": ["Error found"]
|
|
}
|
|
"""
|
|
And the refinement_count is 2
|
|
When I check if refinement is needed
|
|
Then the decision should be "finalize"
|
|
|
|
Scenario: Parse changes extracts changes from LLM response
|
|
Given I have a PlanGenerationGraph instance
|
|
When I parse changes from content:
|
|
"""
|
|
Here are the changes:
|
|
[
|
|
{
|
|
"file_path": "src/api/users.py",
|
|
"operation": "create",
|
|
"content": "def get_users(): pass",
|
|
"description": "Add users endpoint"
|
|
}
|
|
]
|
|
"""
|
|
Then the parsed changes should be a list
|
|
And the parsed changes should contain at least 1 change
|
|
|
|
Scenario: Parse validation extracts validation results from LLM response
|
|
Given I have a PlanGenerationGraph instance
|
|
When I parse validation from content:
|
|
"""
|
|
Validation results:
|
|
{
|
|
"is_valid": true,
|
|
"issues": [],
|
|
"suggestions": ["Consider adding docstrings"]
|
|
}
|
|
"""
|
|
Then the parsed validation should be a dictionary
|
|
And the parsed validation should have an is_valid field
|
|
|
|
Scenario: Full workflow with successful first attempt
|
|
Given I have a PlanGenerationGraph instance
|
|
And I have initial state with:
|
|
"""
|
|
{
|
|
"project_context": {"name": "test"},
|
|
"plan_instructions": "Create API endpoint",
|
|
"messages": [],
|
|
"refinement_count": 0
|
|
}
|
|
"""
|
|
And the mock LLM returns valid responses
|
|
When I run the complete workflow for plan generation
|
|
Then the workflow should complete successfully for plan generation
|
|
And the final result should have success true
|
|
And the refinement_count should be 0
|
|
|
|
Scenario: Full workflow with one refinement cycle
|
|
Given I have a PlanGenerationGraph instance
|
|
And I have initial state with plan instructions
|
|
And the mock LLM returns invalid validation on first attempt
|
|
And the mock LLM returns valid validation on second attempt
|
|
When I run the complete workflow for plan generation
|
|
Then the workflow should complete successfully for plan generation
|
|
And the refinement_count should be 1
|
|
|
|
Scenario: Full workflow reaching max refinements
|
|
Given I have a PlanGenerationGraph instance with max_refinements of 2
|
|
And I have initial state with plan instructions
|
|
And the mock LLM always returns invalid validation
|
|
When I run the complete workflow for plan generation
|
|
Then the workflow should complete
|
|
And the refinement_count should be 2
|
|
And the final result success should be false
|
|
|
|
Scenario: Context analysis logs appropriate messages
|
|
Given I have a PlanGenerationGraph instance
|
|
And logging is enabled at INFO level
|
|
And I have a state with project context and instructions
|
|
When I execute the analyze_context step
|
|
Then the log should contain "Analyzing project context and requirements"
|
|
And the log should contain "Context analysis completed"
|
|
|
|
Scenario: Generate changes logs refinement attempt number
|
|
Given I have a PlanGenerationGraph instance
|
|
And logging is enabled at INFO level
|
|
And I have a state with refinement_count of 2
|
|
When I execute the generate_changes step
|
|
Then the log should contain "attempt 3"
|
|
|
|
Scenario: Validate changes logs validation completion with result
|
|
Given I have a PlanGenerationGraph instance
|
|
And logging is enabled at INFO level
|
|
And I have a state with generated changes
|
|
When I execute the validate_changes step
|
|
Then the log should contain "Validation completed"
|
|
And the log should contain "valid="
|
|
|
|
Scenario: Refine changes logs refinement intention
|
|
Given I have a PlanGenerationGraph instance
|
|
And logging is enabled at INFO level
|
|
And I have a state for refinement
|
|
When I execute the refine_changes step
|
|
Then the log should contain "Refining changes based on validation feedback"
|
|
|
|
Scenario: Finalize results logs completion with change count
|
|
Given I have a PlanGenerationGraph instance
|
|
And logging is enabled at INFO level
|
|
And I have a state with 3 generated changes
|
|
When I execute the finalize_results step
|
|
Then the log should contain "Finalizing plan generation results"
|
|
And the log should contain "3 changes"
|
|
|
|
Scenario: Should refine logs refinement decision with attempt details
|
|
Given I have a PlanGenerationGraph instance with max_refinements of 3
|
|
And logging is enabled at INFO level
|
|
And I have a state requiring refinement with count 1
|
|
When I check if refinement is needed
|
|
Then the log should contain "Refinement needed"
|
|
And the log should contain "attempt 2/3"
|
|
|
|
Scenario: PlanGenerationState holds required workflow data
|
|
Given I can create a PlanGenerationState
|
|
When I initialize it with all required fields for plan generation:
|
|
| field | type |
|
|
| project_context | dict |
|
|
| plan_instructions | str |
|
|
| generated_changes | list |
|
|
| validation_results | dict |
|
|
| refinement_count | int |
|
|
Then the state should store all fields correctly
|
|
|
|
Scenario: Workflow graph edges connect nodes correctly
|
|
Given I have a PlanGenerationGraph instance
|
|
When I build the workflow graph for plan generation
|
|
Then "analyze_context" should connect to "generate_changes" for plan generation
|
|
And "generate_changes" should connect to "validate_changes" for plan generation
|
|
And "validate_changes" should have conditional edges to "refine_changes" and "finalize_results"
|
|
And "refine_changes" should connect back to "generate_changes"
|
|
And "finalize_results" should connect to END for plan generation
|