forked from cleveragents/cleveragents-core
freemo
a074b4846f
Remove FakeListLLM as a silent fallback in agent graph constructors (plan_generation.py, context_analysis.py, auto_debug.py). All three now raise ValueError when llm=None, making missing-provider errors explicit. Add Settings.mock_providers flag and validate_provider_availability() method. Update container.get_ai_provider() to check Settings.mock_providers first, with env-var fallback for backward compatibility. Add resolve_provider_by_name() helper to the provider registry and export it from cleveragents.providers. Add structlog trace logging to ProviderRegistry.get_default_provider_type() to record selection reasoning. Update all existing behave step files, robot tests, and benchmarks that relied on the implicit FakeListLLM default to pass an explicit LLM instance instead. Add new BDD tests (features/provider_fixes.feature with 17 scenarios), Robot Framework integration tests (robot/provider_detection_smoke.robot), and ASV benchmarks (benchmarks/provider_selection_bench.py). ISSUES CLOSED: #323
163 lines
7.1 KiB
Plaintext
163 lines
7.1 KiB
Plaintext
*** Settings ***
|
|
Documentation Integration tests for ContextAnalysisAgent workflow
|
|
Resource ${CURDIR}/common.resource
|
|
Library Process
|
|
Library OperatingSystem
|
|
Library Collections
|
|
Library String
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
|
|
*** Variables ***
|
|
${PYTHON} python
|
|
${SRC_DIR} ${CURDIR}/../src
|
|
${HELPER} ${CURDIR}/helper_context_analysis.py
|
|
|
|
*** Test Cases ***
|
|
|
|
Context Analysis Agent Module Can Be Imported
|
|
[Documentation] Verify the context analysis module can be imported
|
|
${result}= Run Process ${PYTHON} -c
|
|
... import sys; sys.path.insert(0, '${SRC_DIR}'); from cleveragents.agents.context_analysis import ContextAnalysisAgent; print('SUCCESS')
|
|
... shell=True
|
|
Should Contain ${result.stdout} SUCCESS
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
Context Analysis Agent Can Be Instantiated With Default Parameters
|
|
[Documentation] Create ContextAnalysisAgent with defaults
|
|
${script}= Catenate SEPARATOR=\n
|
|
... import sys
|
|
... sys.path.insert(0, '${SRC_DIR}')
|
|
... from cleveragents.agents.context_analysis import ContextAnalysisAgent
|
|
... from langchain_community.llms import FakeListLLM
|
|
... agent = ContextAnalysisAgent(llm=FakeListLLM(responses=['test']*3))
|
|
... assert agent is not None
|
|
... assert agent.chunk_size == 2000
|
|
... assert agent.chunk_overlap == 200
|
|
... assert agent.llm is not None
|
|
... print('Agent initialized successfully')
|
|
${result}= Run Process ${PYTHON} -c ${script}
|
|
Should Contain ${result.stdout} initialized successfully
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
Context Analysis Agent Can Be Instantiated With Custom Chunk Settings
|
|
[Documentation] Create ContextAnalysisAgent with custom chunk_size and overlap
|
|
${script}= Catenate SEPARATOR=\n
|
|
... import sys
|
|
... sys.path.insert(0, '${SRC_DIR}')
|
|
... from cleveragents.agents.context_analysis import ContextAnalysisAgent
|
|
... from langchain_community.llms import FakeListLLM
|
|
... agent = ContextAnalysisAgent(llm=FakeListLLM(responses=['test']*3), chunk_size=1000, chunk_overlap=100)
|
|
... assert agent.chunk_size == 1000
|
|
... assert agent.chunk_overlap == 100
|
|
... print('Chunk size: ' + str(agent.chunk_size) + ', overlap: ' + str(agent.chunk_overlap))
|
|
${result}= Run Process ${PYTHON} -c ${script}
|
|
Should Contain ${result.stdout} Chunk size: 1000
|
|
Should Contain ${result.stdout} overlap: 100
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
Context Analysis Agent Workflow Contains Expected Nodes
|
|
[Documentation] Verify all workflow nodes are present using helper script
|
|
${result}= Run Process ${PYTHON} ${HELPER} nodes
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Contain ${result.stdout} SUCCESS
|
|
Should Contain ${result.stdout} All nodes present
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
LangGraph Graphs Package Exports Context Analysis Agent
|
|
[Documentation] Ensure cleveragents.agents.graphs exports ContextAnalysisAgent
|
|
${script}= Catenate SEPARATOR=\n
|
|
... import sys
|
|
... sys.path.insert(0, '${SRC_DIR}')
|
|
... from cleveragents.agents.graphs import ContextAnalysisAgent, __all__
|
|
... from cleveragents.agents import ContextAnalysisAgent as ExportedContextAgent
|
|
... assert 'ContextAnalysisAgent' in __all__
|
|
... assert ContextAnalysisAgent is ExportedContextAgent
|
|
... print('ContextAnalysisAgent export verified')
|
|
${result}= Run Process ${PYTHON} -c ${script}
|
|
Should Contain ${result.stdout} ContextAnalysisAgent export verified
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
Context Analysis Agent Can Load Files
|
|
[Documentation] Test file loading functionality using helper script
|
|
${result}= Run Process ${PYTHON} ${HELPER} load_files
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Contain ${result.stdout} SUCCESS
|
|
Should Contain ${result.stdout} Loaded
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
Context Analysis Agent Handles Missing Files
|
|
[Documentation] Test error handling for missing files using helper script
|
|
${result}= Run Process ${PYTHON} ${HELPER} missing_file
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Contain ${result.stdout} SUCCESS
|
|
Should Contain ${result.stdout} Error handled
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
Context Analysis Agent Invoke Returns Complete Result
|
|
[Documentation] Test complete workflow execution via invoke using helper script
|
|
${result}= Run Process ${PYTHON} ${HELPER} invoke
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Contain ${result.stdout} SUCCESS
|
|
Should Contain ${result.stdout} Workflow completed
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
Context Analysis Agent Streaming Produces Updates
|
|
[Documentation] Test streaming workflow execution using helper script
|
|
${result}= Run Process ${PYTHON} ${HELPER} streaming
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Contain ${result.stdout} SUCCESS
|
|
Should Contain ${result.stdout} stream updates
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
Context Update Summary Models Validate Incoming Data
|
|
[Documentation] Ensure ContextUpdateResult and SummaryForUpdateContextParams accept camelCase payloads
|
|
${script}= Catenate SEPARATOR=\n
|
|
... import sys
|
|
... sys.path.insert(0, '${SRC_DIR}')
|
|
... from cleveragents.domain.models.core.context import (
|
|
... ContextType,
|
|
... ContextUpdateResult,
|
|
... SummaryForUpdateContextParams,
|
|
... )
|
|
... result = ContextUpdateResult(
|
|
... updatedContexts=[
|
|
... {
|
|
... "plan_id": 1,
|
|
... "type": ContextType.FILE,
|
|
... "path": "/tmp/main.py",
|
|
... "size": 256,
|
|
... }
|
|
... ],
|
|
... tokenDiffsById={"ctx-1": {"added": 10, "removed": 0}},
|
|
... tokensDiff=10,
|
|
... totalTokens=128,
|
|
... numFiles=1,
|
|
... numUrls=0,
|
|
... numImages=0,
|
|
... numTrees=0,
|
|
... numMaps=0,
|
|
... maxTokens=2048,
|
|
... )
|
|
... summary = SummaryForUpdateContextParams(
|
|
... numFiles=2,
|
|
... numTrees=1,
|
|
... numUrls=1,
|
|
... numMaps=0,
|
|
... tokensDiff=15,
|
|
... totalTokens=320,
|
|
... )
|
|
... assert result.num_files == 1
|
|
... assert result.updated_contexts[0].path.endswith("main.py")
|
|
... assert summary.num_urls == 1
|
|
... assert summary.tokens_diff == 15
|
|
... print('Context update models validated')
|
|
${result}= Run Process ${PYTHON} -c ${script}
|
|
Should Contain ${result.stdout} Context update models validated
|
|
Should Be Equal As Integers ${result.rc} 0
|