Files
cleveragents-core/robot/plan_generation_graph.robot
freemo a074b4846f fix(provider): remove FakeListLLM defaults
Remove FakeListLLM as a silent fallback in agent graph constructors
(plan_generation.py, context_analysis.py, auto_debug.py). All three now
raise ValueError when llm=None, making missing-provider errors explicit.

Add Settings.mock_providers flag and validate_provider_availability()
method. Update container.get_ai_provider() to check Settings.mock_providers
first, with env-var fallback for backward compatibility.

Add resolve_provider_by_name() helper to the provider registry and export
it from cleveragents.providers. Add structlog trace logging to
ProviderRegistry.get_default_provider_type() to record selection reasoning.

Update all existing behave step files, robot tests, and benchmarks that
relied on the implicit FakeListLLM default to pass an explicit LLM
instance instead.

Add new BDD tests (features/provider_fixes.feature with 17 scenarios),
Robot Framework integration tests (robot/provider_detection_smoke.robot),
and ASV benchmarks (benchmarks/provider_selection_bench.py).

ISSUES CLOSED: #323
2026-02-27 09:47:10 -05:00

443 lines
22 KiB
Plaintext

*** Settings ***
Documentation Integration tests for PlanGenerationGraph workflow
Resource ${CURDIR}/common.resource
Library Process
Library OperatingSystem
Library Collections
Library String
Suite Setup Setup Test Environment
Suite Teardown Cleanup Test Environment
*** Variables ***
${PYTHON} python
${SRC_DIR} ${CURDIR}/../src
${TEST_PROJECT} test_plan_generation_project
*** Test Cases ***
Plan Generation Graph Module Can Be Imported
[Documentation] Verify the plan generation module can be imported
${result}= Run Process ${PYTHON} -c
... import sys; sys.path.insert(0, '${SRC_DIR}'); from cleveragents.agents.plan_generation import PlanGenerationGraph; print('SUCCESS')
... shell=True
Should Contain ${result.stdout} SUCCESS
Should Be Equal As Integers ${result.rc} 0
Plan Generation Graph Can Be Instantiated With Default Parameters
[Documentation] Create PlanGenerationGraph with defaults
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... assert graph is not None
... assert graph.max_retries == 3
... assert graph.llm is not None
... print('Graph initialized successfully')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} initialized successfully
Should Be Equal As Integers ${result.rc} 0
Plan Generation Graph Can Be Instantiated With Custom Max Retries
[Documentation] Create PlanGenerationGraph with custom max_retries
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3), max_retries=5)
... assert graph.max_retries == 5
... print('Max retries: ' + str(graph.max_retries))
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Max retries: 5
Should Be Equal As Integers ${result.rc} 0
Plan Generation Graph Creates Prompt Templates
[Documentation] Verify prompt templates are created
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... assert hasattr(graph, 'analyze_prompt')
... assert hasattr(graph, 'generate_prompt')
... assert hasattr(graph, 'validate_prompt')
... assert 'prompt' in graph.analyze_prompt.input_variables
... assert 'context_summary' in graph.analyze_prompt.input_variables
... print('All prompts configured')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} All prompts configured
Should Be Equal As Integers ${result.rc} 0
Plan Generation Graph Builds Workflow With Correct Nodes
[Documentation] Verify workflow graph has correct nodes
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... nodes = graph.graph.nodes
... assert 'load_context' in nodes
... assert 'analyze_requirements' in nodes
... assert 'generate_plan' in nodes
... assert 'validate' in nodes
... print(f'Graph has {len(nodes)} nodes')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Graph has 4 nodes
Should Be Equal As Integers ${result.rc} 0
LangGraph Graphs Package Exports Workflow Classes
[Documentation] Ensure cleveragents.agents.graphs exports LangGraph workflows
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.graphs import PlanGenerationGraph, ContextAnalysisAgent
... from cleveragents.agents.graphs import __all__
... from cleveragents.agents import PlanGenerationGraph as ExportedPlanGraph
... from cleveragents.agents import ContextAnalysisAgent as ExportedContextAgent
... assert 'PlanGenerationGraph' in __all__
... assert 'ContextAnalysisAgent' in __all__
... assert PlanGenerationGraph is ExportedPlanGraph
... assert ContextAnalysisAgent is ExportedContextAgent
... print('LangGraph graphs exports verified')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} LangGraph graphs exports verified
Should Be Equal As Integers ${result.rc} 0
Format Context Summary With No Files Returns Appropriate Message
[Documentation] Test _format_context_summary with empty list
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... summary = graph._format_context_summary([])
... assert summary == 'No context files provided'
... print('Empty context handled correctly')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Empty context handled correctly
Should Be Equal As Integers ${result.rc} 0
Format Context Summary With Multiple Files
[Documentation] Test _format_context_summary with multiple Context objects
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... from cleveragents.domain.models.core import Context
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... contexts = [
... Context(plan_id=1, path='file1.py', content='# File 1 content'),
... Context(plan_id=1, path='file2.py', content='# File 2 content'),
... ]
... summary = graph._format_context_summary(contexts)
... assert 'file1.py' in summary
... assert 'file2.py' in summary
... assert 'File:' in summary
... print('Multiple files formatted correctly')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Multiple files formatted correctly
Should Be Equal As Integers ${result.rc} 0
Format Context Summary Limits To Five Files
[Documentation] Test that _format_context_summary limits to first 5 files
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... from cleveragents.domain.models.core import Context
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... contexts = [Context(plan_id=1, path=f'file{i}.py', content='content') for i in range(8)]
... summary = graph._format_context_summary(contexts)
... assert 'file0.py' in summary
... assert 'file4.py' in summary
... assert 'and 3 more files' in summary
... print('File limiting works correctly')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} File limiting works correctly
Should Be Equal As Integers ${result.rc} 0
Load Context Node Initializes State
[Documentation] Test _load_context node execution
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... from cleveragents.domain.models.core import Project, Plan
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... state = {'project': None, 'plan': None, 'contexts': []}
... result = graph._load_context(state)
... assert result['retry_count'] == 0
... assert result['error'] is None
... assert result['context_summary'] == 'No context files provided'
... assert result['context_dependencies'] == {}
... assert result['context_relevance'] == {}
... assert result['context_analysis_error'] is None
... print('Load context node works')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Load context node works
Should Be Equal As Integers ${result.rc} 0
Load Context Node Generates Summary With Sample Contexts
[Documentation] Ensure context analysis metadata is populated
${result}= Run Process ${PYTHON} ${CURDIR}/helper_plan_generation.py
Should Contain ${result.stdout} Context analysis summary ready
Should Be Equal As Integers ${result.rc} 0
Should Retry Returns Retry When Validation Fails And Retries Available
[Documentation] Test _should_retry returns "retry" appropriately
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3), max_retries=3)
... state = {
... 'validation_result': {'status': 'FAIL'},
... 'retry_count': 0
... }
... decision = graph._should_retry(state)
... assert decision == 'retry'
... assert state['retry_count'] == 1
... print('Should retry: retry decision correct')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Should retry: retry decision correct
Should Be Equal As Integers ${result.rc} 0
Should Retry Returns End When Validation Passes
[Documentation] Test _should_retry returns "end" when validation succeeds
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3), max_retries=3)
... state = {
... 'validation_result': {'status': 'PASS'},
... 'retry_count': 0
... }
... decision = graph._should_retry(state)
... assert decision == 'end'
... print('Should retry: end decision correct')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Should retry: end decision correct
Should Be Equal As Integers ${result.rc} 0
Should Retry Returns End When Max Retries Reached
[Documentation] Test _should_retry returns "end" when max retries reached
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3), max_retries=3)
... state = {
... 'validation_result': {'status': 'FAIL'},
... 'retry_count': 3
... }
... decision = graph._should_retry(state)
... assert decision == 'end'
... print('Should retry: max retries respected')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Should retry: max retries respected
Should Be Equal As Integers ${result.rc} 0
Plan Generation State TypedDict Has Correct Structure
[Documentation] Verify PlanGenerationState TypedDict structure
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationState
... annotations = PlanGenerationState.__annotations__
... assert 'project' in annotations
... assert 'plan' in annotations
... assert 'contexts' in annotations
... assert 'context_summary' in annotations
... assert 'context_dependencies' in annotations
... assert 'context_relevance' in annotations
... assert 'context_analysis_error' in annotations
... assert 'prompt' in annotations
... assert 'analyzed_requirements' in annotations
... assert 'generated_changes' in annotations
... assert 'validation_result' in annotations
... assert 'retry_count' in annotations
... assert 'error' in annotations
... print('PlanGenerationState structure verified')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} PlanGenerationState structure verified
Should Be Equal As Integers ${result.rc} 0
Validate Node Fails When No Changes Provided
[Documentation] Test _validate with no changes
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... state = {'generated_changes': []}
... result = graph._validate(state)
... assert result['validation_result']['status'] == 'FAIL'
... assert 'No changes to validate' in result['validation_result']['message']
... print('Validate correctly handles empty changes')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Validate correctly handles empty changes
Should Be Equal As Integers ${result.rc} 0
Generate Plan Handles Missing Requirements
[Documentation] Test _generate_plan with no requirements
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... state = {'analyzed_requirements': {}}
... result = graph._generate_plan(state)
... assert result['generated_changes'] == []
... assert 'No requirements to generate from' in result['error']
... print('Generate plan handles missing requirements')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Generate plan handles missing requirements
Should Be Equal As Integers ${result.rc} 0
Generate Plan Infers Test File Name From Prompt
[Documentation] Test file name inference for test-related prompts
${script}= Catenate SEPARATOR=\n
... import sys
... from pathlib import Path
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... from cleveragents.domain.models.core import Project, Plan
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... state = {
... 'project': Project(id=1, name='test', path=Path('/tmp/test_project')),
... 'plan': Plan(id=1, project_id=1, name='Unit Test Plan', prompt='Create unit tests'),
... 'contexts': [],
... 'prompt': 'Create unit tests',
... 'analyzed_requirements': {
... 'description': 'tests',
... 'files_to_modify': [],
... 'operation': 'create'
... }
... }
... result = graph._generate_plan(state)
... assert len(result['generated_changes']) > 0
... assert result['generated_changes'][0].file_path == 'test_generated.py'
... print('Test file name inferred correctly')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Test file name inferred correctly
Should Be Equal As Integers ${result.rc} 0
Generate Plan Infers Error Handler File Name From Prompt
[Documentation] Test file name inference for error/exception prompts
${script}= Catenate SEPARATOR=\n
... import sys
... from pathlib import Path
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... from cleveragents.domain.models.core import Project, Plan
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... state = {
... 'project': Project(id=1, name='test', path=Path('/tmp/test_project')),
... 'plan': Plan(id=1, project_id=1, name='Error Handling Plan', prompt='Add error handling'),
... 'contexts': [],
... 'prompt': 'Add error handling',
... 'analyzed_requirements': {
... 'description': 'error handling',
... 'files_to_modify': [],
... 'operation': 'create'
... }
... }
... result = graph._generate_plan(state)
... assert result['generated_changes'][0].file_path == 'error_handler.py'
... print('Error handler file name inferred')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Error handler file name inferred
Should Be Equal As Integers ${result.rc} 0
Workflow Invoke Method Returns Complete State
[Documentation] Test that invoke() returns complete workflow state
${script}= Catenate SEPARATOR=\n
... import sys
... from pathlib import Path
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... from cleveragents.domain.models.core import Project, Plan, Context
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... project = Project(id=1, name='test_project', path=Path('/tmp/test_project'))
... plan = Plan(id=1, project_id=1, name='Logging Plan', prompt='Add logging')
... contexts = [Context(plan_id=plan.id, path='app.py', content='def main(): pass')]
... result = graph.invoke(project, plan, contexts, thread_id='test-123')
... assert result['plan'].name == plan.name
... assert 'project' in result
... assert 'plan' in result
... assert 'contexts' in result
... assert 'context_summary' in result
... assert 'context_dependencies' in result
... assert 'context_relevance' in result
... assert 'context_analysis_error' in result
... assert 'generated_changes' in result
... assert 'validation_result' in result
... assert 'retry_count' in result
... print('Workflow invoke completes successfully')
${result}= Run Process ${PYTHON} -c ${script} shell=True timeout=30s
Should Contain ${result.stdout} Workflow invoke completes successfully
Should Be Equal As Integers ${result.rc} 0
Workflow Stream Method Yields Events
[Documentation] Test that stream() yields workflow events
${script}= Catenate SEPARATOR=\n
... import sys
... from pathlib import Path
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... from cleveragents.domain.models.core import Project, Plan, Context
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... project = Project(id=1, name='test_project', path=Path('/tmp/test_project'))
... plan = Plan(id=1, project_id=1, name='Feature Plan', prompt='Add feature')
... contexts = [Context(plan_id=plan.id, path='app.py', content='# app')]
... events = list(graph.stream(project, plan, contexts))
... assert len(events) > 0
... assert all(isinstance(e, dict) for e in events)
... print(f'Stream yielded {len(events)} events')
${result}= Run Process ${PYTHON} -c ${script} shell=True timeout=30s
Should Contain ${result.stdout} Stream yielded
Should Contain ${result.stdout} events
Should Be Equal As Integers ${result.rc} 0
Graph Has Checkpointer For State Persistence
[Documentation] Verify checkpointer is configured
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... from langgraph.checkpoint.memory import MemorySaver
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... assert graph.checkpointer is not None
... assert isinstance(graph.checkpointer, MemorySaver)
... assert graph.app is not None
... print('Checkpointer configured correctly')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Checkpointer configured correctly
Should Be Equal As Integers ${result.rc} 0
*** Keywords ***
Create Test Project
[Arguments] ${project_name}
Create Directory /tmp/${project_name}
Set Suite Variable ${TEST_PROJECT} /tmp/${project_name}