Files
cleveragents-core/robot/plan_generation_graph.robot
T
brent.edwards 3eecb79003
CI / lint (pull_request) Successful in 16s
CI / quality (pull_request) Successful in 46s
CI / typecheck (pull_request) Successful in 56s
CI / security (pull_request) Successful in 56s
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 29s
CI / e2e_tests (pull_request) Successful in 2m7s
CI / unit_tests (pull_request) Successful in 3m8s
CI / docker (pull_request) Successful in 9s
CI / integration_tests (pull_request) Successful in 3m57s
CI / coverage (pull_request) Successful in 7m48s
CI / benchmark-regression (pull_request) Successful in 40m48s
test(plan): TDD failing tests for checkpoint real rollback (bug #822)
TDD expected-fail tests proving bug #822 exists:
CheckpointService.rollback_to_checkpoint() returns a successful
RollbackResult but does not execute git reset --hard. Files modified
after the checkpoint remain unchanged after rollback.

Also fixes Robot Framework timeout robustness across the entire test
suite: all Run Process calls now use on_timeout=kill (prevents
SIGTERM-induced -15 exit codes under CI load) and timeouts increased
to 120s (prevents premature kills during heavy parallel execution).

ISSUES CLOSED: #839
2026-03-16 01:45:50 +00:00

443 lines
22 KiB
Plaintext

*** Settings ***
Documentation Integration tests for PlanGenerationGraph workflow
Resource ${CURDIR}/common.resource
Library Process
Library OperatingSystem
Library Collections
Library String
Suite Setup Setup Test Environment
Suite Teardown Cleanup Test Environment
*** Variables ***
${PYTHON} python
${SRC_DIR} ${CURDIR}/../src
${TEST_PROJECT} test_plan_generation_project
*** Test Cases ***
Plan Generation Graph Module Can Be Imported
[Documentation] Verify the plan generation module can be imported
${result}= Run Process ${PYTHON} -c
... import sys; sys.path.insert(0, '${SRC_DIR}'); from cleveragents.agents.plan_generation import PlanGenerationGraph; print('SUCCESS')
... shell=True
Should Contain ${result.stdout} SUCCESS
Should Be Equal As Integers ${result.rc} 0
Plan Generation Graph Can Be Instantiated With Default Parameters
[Documentation] Create PlanGenerationGraph with defaults
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... assert graph is not None
... assert graph.max_retries == 3
... assert graph.llm is not None
... print('Graph initialized successfully')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} initialized successfully
Should Be Equal As Integers ${result.rc} 0
Plan Generation Graph Can Be Instantiated With Custom Max Retries
[Documentation] Create PlanGenerationGraph with custom max_retries
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3), max_retries=5)
... assert graph.max_retries == 5
... print('Max retries: ' + str(graph.max_retries))
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Max retries: 5
Should Be Equal As Integers ${result.rc} 0
Plan Generation Graph Creates Prompt Templates
[Documentation] Verify prompt templates are created
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... assert hasattr(graph, 'analyze_prompt')
... assert hasattr(graph, 'generate_prompt')
... assert hasattr(graph, 'validate_prompt')
... assert 'prompt' in graph.analyze_prompt.input_variables
... assert 'context_summary' in graph.analyze_prompt.input_variables
... print('All prompts configured')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} All prompts configured
Should Be Equal As Integers ${result.rc} 0
Plan Generation Graph Builds Workflow With Correct Nodes
[Documentation] Verify workflow graph has correct nodes
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... nodes = graph.graph.nodes
... assert 'load_context' in nodes
... assert 'analyze_requirements' in nodes
... assert 'generate_plan' in nodes
... assert 'validate' in nodes
... print(f'Graph has {len(nodes)} nodes')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Graph has 4 nodes
Should Be Equal As Integers ${result.rc} 0
LangGraph Graphs Package Exports Workflow Classes
[Documentation] Ensure cleveragents.agents.graphs exports LangGraph workflows
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.graphs import PlanGenerationGraph, ContextAnalysisAgent
... from cleveragents.agents.graphs import __all__
... from cleveragents.agents import PlanGenerationGraph as ExportedPlanGraph
... from cleveragents.agents import ContextAnalysisAgent as ExportedContextAgent
... assert 'PlanGenerationGraph' in __all__
... assert 'ContextAnalysisAgent' in __all__
... assert PlanGenerationGraph is ExportedPlanGraph
... assert ContextAnalysisAgent is ExportedContextAgent
... print('LangGraph graphs exports verified')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} LangGraph graphs exports verified
Should Be Equal As Integers ${result.rc} 0
Format Context Summary With No Files Returns Appropriate Message
[Documentation] Test _format_context_summary with empty list
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... summary = graph._format_context_summary([])
... assert summary == 'No context files provided'
... print('Empty context handled correctly')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Empty context handled correctly
Should Be Equal As Integers ${result.rc} 0
Format Context Summary With Multiple Files
[Documentation] Test _format_context_summary with multiple Context objects
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... from cleveragents.domain.models.core import Context
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... contexts = [
... Context(plan_id=1, path='file1.py', content='# File 1 content'),
... Context(plan_id=1, path='file2.py', content='# File 2 content'),
... ]
... summary = graph._format_context_summary(contexts)
... assert 'file1.py' in summary
... assert 'file2.py' in summary
... assert 'File:' in summary
... print('Multiple files formatted correctly')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Multiple files formatted correctly
Should Be Equal As Integers ${result.rc} 0
Format Context Summary Limits To Five Files
[Documentation] Test that _format_context_summary limits to first 5 files
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... from cleveragents.domain.models.core import Context
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... contexts = [Context(plan_id=1, path=f'file{i}.py', content='content') for i in range(8)]
... summary = graph._format_context_summary(contexts)
... assert 'file0.py' in summary
... assert 'file4.py' in summary
... assert 'and 3 more files' in summary
... print('File limiting works correctly')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} File limiting works correctly
Should Be Equal As Integers ${result.rc} 0
Load Context Node Initializes State
[Documentation] Test _load_context node execution
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... from cleveragents.domain.models.core import Project, Plan
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... state = {'project': None, 'plan': None, 'contexts': []}
... result = graph._load_context(state)
... assert result['retry_count'] == 0
... assert result['error'] is None
... assert result['context_summary'] == 'No context files provided'
... assert result['context_dependencies'] == {}
... assert result['context_relevance'] == {}
... assert result['context_analysis_error'] is None
... print('Load context node works')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Load context node works
Should Be Equal As Integers ${result.rc} 0
Load Context Node Generates Summary With Sample Contexts
[Documentation] Ensure context analysis metadata is populated
${result}= Run Process ${PYTHON} ${CURDIR}/helper_plan_generation.py
Should Contain ${result.stdout} Context analysis summary ready
Should Be Equal As Integers ${result.rc} 0
Should Retry Returns Retry When Validation Fails And Retries Available
[Documentation] Test _should_retry returns "retry" appropriately
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3), max_retries=3)
... state = {
... 'validation_result': {'status': 'FAIL'},
... 'retry_count': 0
... }
... decision = graph._should_retry(state)
... assert decision == 'retry'
... assert state['retry_count'] == 1
... print('Should retry: retry decision correct')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Should retry: retry decision correct
Should Be Equal As Integers ${result.rc} 0
Should Retry Returns End When Validation Passes
[Documentation] Test _should_retry returns "end" when validation succeeds
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3), max_retries=3)
... state = {
... 'validation_result': {'status': 'PASS'},
... 'retry_count': 0
... }
... decision = graph._should_retry(state)
... assert decision == 'end'
... print('Should retry: end decision correct')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Should retry: end decision correct
Should Be Equal As Integers ${result.rc} 0
Should Retry Returns End When Max Retries Reached
[Documentation] Test _should_retry returns "end" when max retries reached
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3), max_retries=3)
... state = {
... 'validation_result': {'status': 'FAIL'},
... 'retry_count': 3
... }
... decision = graph._should_retry(state)
... assert decision == 'end'
... print('Should retry: max retries respected')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Should retry: max retries respected
Should Be Equal As Integers ${result.rc} 0
Plan Generation State TypedDict Has Correct Structure
[Documentation] Verify PlanGenerationState TypedDict structure
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationState
... annotations = PlanGenerationState.__annotations__
... assert 'project' in annotations
... assert 'plan' in annotations
... assert 'contexts' in annotations
... assert 'context_summary' in annotations
... assert 'context_dependencies' in annotations
... assert 'context_relevance' in annotations
... assert 'context_analysis_error' in annotations
... assert 'prompt' in annotations
... assert 'analyzed_requirements' in annotations
... assert 'generated_changes' in annotations
... assert 'validation_result' in annotations
... assert 'retry_count' in annotations
... assert 'error' in annotations
... print('PlanGenerationState structure verified')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} PlanGenerationState structure verified
Should Be Equal As Integers ${result.rc} 0
Validate Node Fails When No Changes Provided
[Documentation] Test _validate with no changes
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... state = {'generated_changes': []}
... result = graph._validate(state)
... assert result['validation_result']['status'] == 'FAIL'
... assert 'No changes to validate' in result['validation_result']['message']
... print('Validate correctly handles empty changes')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Validate correctly handles empty changes
Should Be Equal As Integers ${result.rc} 0
Generate Plan Handles Missing Requirements
[Documentation] Test _generate_plan with no requirements
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... state = {'analyzed_requirements': {}}
... result = graph._generate_plan(state)
... assert result['generated_changes'] == []
... assert 'No requirements to generate from' in result['error']
... print('Generate plan handles missing requirements')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Generate plan handles missing requirements
Should Be Equal As Integers ${result.rc} 0
Generate Plan Infers Test File Name From Prompt
[Documentation] Test file name inference for test-related prompts
${script}= Catenate SEPARATOR=\n
... import sys
... from pathlib import Path
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... from cleveragents.domain.models.core import Project, Plan
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... state = {
... 'project': Project(id=1, name='test', path=Path('/tmp/test_project')),
... 'plan': Plan(id=1, project_id=1, name='Unit Test Plan', prompt='Create unit tests'),
... 'contexts': [],
... 'prompt': 'Create unit tests',
... 'analyzed_requirements': {
... 'description': 'tests',
... 'files_to_modify': [],
... 'operation': 'create'
... }
... }
... result = graph._generate_plan(state)
... assert len(result['generated_changes']) > 0
... assert result['generated_changes'][0].file_path == 'test_generated.py'
... print('Test file name inferred correctly')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Test file name inferred correctly
Should Be Equal As Integers ${result.rc} 0
Generate Plan Infers Error Handler File Name From Prompt
[Documentation] Test file name inference for error/exception prompts
${script}= Catenate SEPARATOR=\n
... import sys
... from pathlib import Path
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... from cleveragents.domain.models.core import Project, Plan
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... state = {
... 'project': Project(id=1, name='test', path=Path('/tmp/test_project')),
... 'plan': Plan(id=1, project_id=1, name='Error Handling Plan', prompt='Add error handling'),
... 'contexts': [],
... 'prompt': 'Add error handling',
... 'analyzed_requirements': {
... 'description': 'error handling',
... 'files_to_modify': [],
... 'operation': 'create'
... }
... }
... result = graph._generate_plan(state)
... assert result['generated_changes'][0].file_path == 'error_handler.py'
... print('Error handler file name inferred')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Error handler file name inferred
Should Be Equal As Integers ${result.rc} 0
Workflow Invoke Method Returns Complete State
[Documentation] Test that invoke() returns complete workflow state
${script}= Catenate SEPARATOR=\n
... import sys
... from pathlib import Path
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... from cleveragents.domain.models.core import Project, Plan, Context
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... project = Project(id=1, name='test_project', path=Path('/tmp/test_project'))
... plan = Plan(id=1, project_id=1, name='Logging Plan', prompt='Add logging')
... contexts = [Context(plan_id=plan.id, path='app.py', content='def main(): pass')]
... result = graph.invoke(project, plan, contexts, thread_id='test-123')
... assert result['plan'].name == plan.name
... assert 'project' in result
... assert 'plan' in result
... assert 'contexts' in result
... assert 'context_summary' in result
... assert 'context_dependencies' in result
... assert 'context_relevance' in result
... assert 'context_analysis_error' in result
... assert 'generated_changes' in result
... assert 'validation_result' in result
... assert 'retry_count' in result
... print('Workflow invoke completes successfully')
${result}= Run Process ${PYTHON} -c ${script} shell=True timeout=120s on_timeout=kill
Should Contain ${result.stdout} Workflow invoke completes successfully
Should Be Equal As Integers ${result.rc} 0
Workflow Stream Method Yields Events
[Documentation] Test that stream() yields workflow events
${script}= Catenate SEPARATOR=\n
... import sys
... from pathlib import Path
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... from cleveragents.domain.models.core import Project, Plan, Context
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... project = Project(id=1, name='test_project', path=Path('/tmp/test_project'))
... plan = Plan(id=1, project_id=1, name='Feature Plan', prompt='Add feature')
... contexts = [Context(plan_id=plan.id, path='app.py', content='# app')]
... events = list(graph.stream(project, plan, contexts))
... assert len(events) > 0
... assert all(isinstance(e, dict) for e in events)
... print(f'Stream yielded {len(events)} events')
${result}= Run Process ${PYTHON} -c ${script} shell=True timeout=120s on_timeout=kill
Should Contain ${result.stdout} Stream yielded
Should Contain ${result.stdout} events
Should Be Equal As Integers ${result.rc} 0
Graph Has Checkpointer For State Persistence
[Documentation] Verify checkpointer is configured
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... from langgraph.checkpoint.memory import MemorySaver
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... assert graph.checkpointer is not None
... assert isinstance(graph.checkpointer, MemorySaver)
... assert graph.app is not None
... print('Checkpointer configured correctly')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Checkpointer configured correctly
Should Be Equal As Integers ${result.rc} 0
*** Keywords ***
Create Test Project
[Arguments] ${project_name}
Create Directory /tmp/${project_name}
Set Suite Variable ${TEST_PROJECT} /tmp/${project_name}