Files
cleveragents-core/robot/plan_generation_graph.robot
T
HAL9000 6141d2a36d
CI / lint (pull_request) Successful in 42s
CI / quality (pull_request) Successful in 1m2s
CI / security (pull_request) Successful in 1m12s
CI / typecheck (pull_request) Successful in 1m19s
CI / build (pull_request) Successful in 47s
CI / helm (pull_request) Successful in 51s
CI / push-validation (pull_request) Successful in 25s
CI / unit_tests (pull_request) Successful in 5m57s
CI / docker (pull_request) Successful in 1m40s
CI / integration_tests (pull_request) Successful in 9m35s
CI / coverage (pull_request) Successful in 12m42s
CI / status-check (pull_request) Successful in 4s
fix(agents/graphs): unblock TDD validate tests + read-only _should_retry + max_context_files
- features/steps/tdd_plan_generation_validate_steps.py: drop duplicate
  @given/@then registrations (live in plan_generation_langgraph_coverage_
  steps.py; redefinition caused AmbiguousStep errors crashing all 8
  behave-parallel workers); have the @when step feed the docstring as the
  FakeListLLM response so each scenario tests _validate()'s parsing of a
  specific PASS/FAIL signal rather than the input code.
- features/tdd_plan_generation_validate_logic.feature: add the required
  @tdd_issue tag alongside @tdd_issue_10746 (enforced by
  features/environment.py); tighten scenario 4 wording to remove the
  reference to the obsolete length guard.
- robot/plan_generation_graph.robot: drop assertion for handle_retry node
  (retry is a conditional edge, not a fifth node) and update node-count
  check from 5 to 4; update Should Retry test to assert _should_retry does
  NOT mutate state (read-only contract for LangGraph conditional edges).
- src/cleveragents/agents/graphs/plan_generation.py:
  * _validate: persist retry_count increment in return dict (FAIL path and
    exception path) so LangGraph propagates it through the state graph.
  * _should_retry: remove state mutation (conditional-edge functions are
    read-only in LangGraph; mutations were silently dropped, causing
    retry_count to remain 0 forever and the graph to loop infinitely).
    Adjust comparison to retry_count <= max_retries because _validate has
    already incremented before _should_retry runs.
  * __init__: add max_context_files parameter (default 5, validated > 0)
    and wire it into _format_context_summary in place of the hardcoded 5,
    implementing the configurable-limits contract tested by
    features/agent_configurable_limits.feature.

ISSUES CLOSED: #10746
2026-06-14 16:34:44 -04:00

446 lines
23 KiB
Plaintext

*** Settings ***
Documentation Integration tests for PlanGenerationGraph workflow
Resource ${CURDIR}/common.resource
Library Process
Library OperatingSystem
Library Collections
Library String
Suite Setup Setup Test Environment
Suite Teardown Cleanup Test Environment
*** Variables ***
${SRC_DIR} ${CURDIR}/../src
${TEST_PROJECT} test_plan_generation_project
*** Test Cases ***
Plan Generation Graph Module Can Be Imported
[Documentation] Verify the plan generation module can be imported
${result}= Run Process ${PYTHON} -c
... import sys; sys.path.insert(0, '${SRC_DIR}'); from cleveragents.agents.plan_generation import PlanGenerationGraph; print('SUCCESS')
... shell=True
Should Contain ${result.stdout} SUCCESS
Should Be Equal As Integers ${result.rc} 0
Plan Generation Graph Can Be Instantiated With Default Parameters
[Documentation] Create PlanGenerationGraph with defaults
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... assert graph is not None
... assert graph.max_retries == 3
... assert graph.llm is not None
... print('Graph initialized successfully')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} initialized successfully
Should Be Equal As Integers ${result.rc} 0
Plan Generation Graph Can Be Instantiated With Custom Max Retries
[Documentation] Create PlanGenerationGraph with custom max_retries
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3), max_retries=5)
... assert graph.max_retries == 5
... print('Max retries: ' + str(graph.max_retries))
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Max retries: 5
Should Be Equal As Integers ${result.rc} 0
Plan Generation Graph Creates Prompt Templates
[Documentation] Verify prompt templates are created
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... assert hasattr(graph, 'analyze_prompt')
... assert hasattr(graph, 'generate_prompt')
... assert hasattr(graph, 'validate_prompt')
... assert 'prompt' in graph.analyze_prompt.input_variables
... assert 'context_summary' in graph.analyze_prompt.input_variables
... print('All prompts configured')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} All prompts configured
Should Be Equal As Integers ${result.rc} 0
Plan Generation Graph Builds Workflow With Correct Nodes
[Documentation] Verify workflow graph has correct nodes
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... nodes = graph.graph.nodes
... assert 'load_context' in nodes
... assert 'analyze_requirements' in nodes
... assert 'generate_plan' in nodes
... assert 'validate' in nodes
... print(f'Graph has {len(nodes)} nodes')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Graph has 4 nodes
Should Be Equal As Integers ${result.rc} 0
LangGraph Graphs Package Exports Workflow Classes
[Documentation] Ensure cleveragents.agents.graphs exports LangGraph workflows
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.graphs import PlanGenerationGraph, ContextAnalysisAgent
... from cleveragents.agents.graphs import __all__
... from cleveragents.agents import PlanGenerationGraph as ExportedPlanGraph
... from cleveragents.agents import ContextAnalysisAgent as ExportedContextAgent
... assert 'PlanGenerationGraph' in __all__
... assert 'ContextAnalysisAgent' in __all__
... assert PlanGenerationGraph is ExportedPlanGraph
... assert ContextAnalysisAgent is ExportedContextAgent
... print('LangGraph graphs exports verified')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} LangGraph graphs exports verified
Should Be Equal As Integers ${result.rc} 0
Format Context Summary With No Files Returns Appropriate Message
[Documentation] Test _format_context_summary with empty list
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... summary = graph._format_context_summary([])
... assert summary == 'No context files provided'
... print('Empty context handled correctly')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Empty context handled correctly
Should Be Equal As Integers ${result.rc} 0
Format Context Summary With Multiple Files
[Documentation] Test _format_context_summary with multiple Context objects
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... from cleveragents.domain.models.core import Context
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... contexts = [
... Context(plan_id=1, path='file1.py', content='# File 1 content'),
... Context(plan_id=1, path='file2.py', content='# File 2 content'),
... ]
... summary = graph._format_context_summary(contexts)
... assert 'file1.py' in summary
... assert 'file2.py' in summary
... assert 'File:' in summary
... print('Multiple files formatted correctly')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Multiple files formatted correctly
Should Be Equal As Integers ${result.rc} 0
Format Context Summary Limits To Five Files
[Documentation] Test that _format_context_summary limits to first 5 files
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... from cleveragents.domain.models.core import Context
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... contexts = [Context(plan_id=1, path=f'file{i}.py', content='content') for i in range(8)]
... summary = graph._format_context_summary(contexts)
... assert 'file0.py' in summary
... assert 'file4.py' in summary
... assert 'and 3 more files' in summary
... print('File limiting works correctly')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} File limiting works correctly
Should Be Equal As Integers ${result.rc} 0
Load Context Node Initializes State
[Documentation] Test _load_context node execution
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... from cleveragents.domain.models.core import Project, Plan
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... state = {'project': None, 'plan': None, 'contexts': []}
... result = graph._load_context(state)
... assert result['retry_count'] == 0
... assert result['error'] is None
... assert result['context_summary'] == 'No context files provided'
... assert result['context_dependencies'] == {}
... assert result['context_relevance'] == {}
... assert result['context_analysis_error'] is None
... print('Load context node works')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Load context node works
Should Be Equal As Integers ${result.rc} 0
Load Context Node Generates Summary With Sample Contexts
[Documentation] Ensure context analysis metadata is populated
${result}= Run Process ${PYTHON} ${CURDIR}/helper_plan_generation.py
Should Contain ${result.stdout} Context analysis summary ready
Should Be Equal As Integers ${result.rc} 0
Should Retry Returns Retry When Validation Fails And Retries Available
[Documentation] Test _should_retry returns "retry" appropriately.
... Note: _should_retry is a read-only conditional-edge function in LangGraph.
... It does NOT mutate state; retry_count is incremented inside _validate.
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3), max_retries=3)
... state = {
... 'validation_result': {'status': 'FAIL'},
... 'retry_count': 0
... }
... decision = graph._should_retry(state)
... assert decision == 'retry'
... assert state['retry_count'] == 0, f'_should_retry must not mutate state, got {state["retry_count"]}'
... print('Should retry: retry decision correct')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Should retry: retry decision correct
Should Be Equal As Integers ${result.rc} 0
Should Retry Returns End When Validation Passes
[Documentation] Test _should_retry returns "end" when validation succeeds
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3), max_retries=3)
... state = {
... 'validation_result': {'status': 'PASS'},
... 'retry_count': 0
... }
... decision = graph._should_retry(state)
... assert decision == 'end'
... print('Should retry: end decision correct')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Should retry: end decision correct
Should Be Equal As Integers ${result.rc} 0
Should Retry Returns End When Max Retries Reached
[Documentation] Test _should_retry returns "end" when max retries exceeded.
... Because _validate increments retry_count before _should_retry is called,
... the boundary is retry_count > max_retries (i.e. retry_count=4 for max_retries=3).
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3), max_retries=3)
... state = {
... 'validation_result': {'status': 'FAIL'},
... 'retry_count': 4
... }
... decision = graph._should_retry(state)
... assert decision == 'end'
... print('Should retry: max retries respected')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Should retry: max retries respected
Should Be Equal As Integers ${result.rc} 0
Plan Generation State TypedDict Has Correct Structure
[Documentation] Verify PlanGenerationState TypedDict structure
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationState
... annotations = PlanGenerationState.__annotations__
... assert 'project' in annotations
... assert 'plan' in annotations
... assert 'contexts' in annotations
... assert 'context_summary' in annotations
... assert 'context_dependencies' in annotations
... assert 'context_relevance' in annotations
... assert 'context_analysis_error' in annotations
... assert 'prompt' in annotations
... assert 'analyzed_requirements' in annotations
... assert 'generated_changes' in annotations
... assert 'validation_result' in annotations
... assert 'retry_count' in annotations
... assert 'error' in annotations
... print('PlanGenerationState structure verified')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} PlanGenerationState structure verified
Should Be Equal As Integers ${result.rc} 0
Validate Node Fails When No Changes Provided
[Documentation] Test _validate with no changes
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... state = {'generated_changes': []}
... result = graph._validate(state)
... assert result['validation_result']['status'] == 'FAIL'
... assert 'No changes to validate' in result['validation_result']['message']
... print('Validate correctly handles empty changes')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Validate correctly handles empty changes
Should Be Equal As Integers ${result.rc} 0
Generate Plan Handles Missing Requirements
[Documentation] Test _generate_plan with no requirements
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... state = {'analyzed_requirements': {}}
... result = graph._generate_plan(state)
... assert result['generated_changes'] == []
... assert 'No requirements to generate from' in result['error']
... print('Generate plan handles missing requirements')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Generate plan handles missing requirements
Should Be Equal As Integers ${result.rc} 0
Generate Plan Infers Test File Name From Prompt
[Documentation] Test file name inference for test-related prompts
${script}= Catenate SEPARATOR=\n
... import sys
... from pathlib import Path
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... from cleveragents.domain.models.core import Project, Plan
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... state = {
... 'project': Project(id=1, name='test', path=Path('/tmp/test_project')),
... 'plan': Plan(id=1, project_id=1, name='Unit Test Plan', prompt='Create unit tests'),
... 'contexts': [],
... 'prompt': 'Create unit tests',
... 'analyzed_requirements': {
... 'description': 'tests',
... 'files_to_modify': [],
... 'operation': 'create'
... }
... }
... result = graph._generate_plan(state)
... assert len(result['generated_changes']) > 0
... assert result['generated_changes'][0].file_path == 'test_generated.py'
... print('Test file name inferred correctly')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Test file name inferred correctly
Should Be Equal As Integers ${result.rc} 0
Generate Plan Infers Error Handler File Name From Prompt
[Documentation] Test file name inference for error/exception prompts
${script}= Catenate SEPARATOR=\n
... import sys
... from pathlib import Path
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... from cleveragents.domain.models.core import Project, Plan
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... state = {
... 'project': Project(id=1, name='test', path=Path('/tmp/test_project')),
... 'plan': Plan(id=1, project_id=1, name='Error Handling Plan', prompt='Add error handling'),
... 'contexts': [],
... 'prompt': 'Add error handling',
... 'analyzed_requirements': {
... 'description': 'error handling',
... 'files_to_modify': [],
... 'operation': 'create'
... }
... }
... result = graph._generate_plan(state)
... assert result['generated_changes'][0].file_path == 'error_handler.py'
... print('Error handler file name inferred')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Error handler file name inferred
Should Be Equal As Integers ${result.rc} 0
Workflow Invoke Method Returns Complete State
[Documentation] Test that invoke() returns complete workflow state
${script}= Catenate SEPARATOR=\n
... import sys
... from pathlib import Path
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... from cleveragents.domain.models.core import Project, Plan, Context
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*12))
... project = Project(id=1, name='test_project', path=Path('/tmp/test_project'))
... plan = Plan(id=1, project_id=1, name='Logging Plan', prompt='Add logging')
... contexts = [Context(plan_id=plan.id, path='app.py', content='def main(): pass')]
... result = graph.invoke(project, plan, contexts, thread_id='test-123')
... assert result['plan'].name == plan.name
... assert 'project' in result
... assert 'plan' in result
... assert 'contexts' in result
... assert 'context_summary' in result
... assert 'context_dependencies' in result
... assert 'context_relevance' in result
... assert 'context_analysis_error' in result
... assert 'generated_changes' in result
... assert 'validation_result' in result
... assert 'retry_count' in result
... print('Workflow invoke completes successfully')
${result}= Run Process ${PYTHON} -c ${script} shell=True timeout=120s on_timeout=kill
Should Contain ${result.stdout} Workflow invoke completes successfully
Should Be Equal As Integers ${result.rc} 0
Workflow Stream Method Yields Events
[Documentation] Test that stream() yields workflow events
${script}= Catenate SEPARATOR=\n
... import sys
... from pathlib import Path
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... from cleveragents.domain.models.core import Project, Plan, Context
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*12))
... project = Project(id=1, name='test_project', path=Path('/tmp/test_project'))
... plan = Plan(id=1, project_id=1, name='Feature Plan', prompt='Add feature')
... contexts = [Context(plan_id=plan.id, path='app.py', content='# app')]
... events = list(graph.stream(project, plan, contexts))
... assert len(events) > 0
... assert all(isinstance(e, dict) for e in events)
... print(f'Stream yielded {len(events)} events')
${result}= Run Process ${PYTHON} -c ${script} shell=True timeout=120s on_timeout=kill
Should Contain ${result.stdout} Stream yielded
Should Contain ${result.stdout} events
Should Be Equal As Integers ${result.rc} 0
Graph Has Checkpointer For State Persistence
[Documentation] Verify checkpointer is configured
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.plan_generation import PlanGenerationGraph
... from langchain_community.llms import FakeListLLM
... from langgraph.checkpoint.memory import MemorySaver
... graph = PlanGenerationGraph(llm=FakeListLLM(responses=['test']*3))
... assert graph.checkpointer is not None
... assert isinstance(graph.checkpointer, MemorySaver)
... assert graph.app is not None
... print('Checkpointer configured correctly')
${result}= Run Process ${PYTHON} -c ${script} shell=True
Should Contain ${result.stdout} Checkpointer configured correctly
Should Be Equal As Integers ${result.rc} 0
*** Keywords ***
Create Test Project
[Arguments] ${project_name}
Create Directory /tmp/${project_name}
Set Suite Variable ${TEST_PROJECT} /tmp/${project_name}