Files
cleveragents-core/robot/context_analysis_agent.robot
freemo a074b4846f
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 15s
CI / build (pull_request) Successful in 19s
CI / quality (pull_request) Successful in 28s
CI / security (pull_request) Successful in 50s
CI / typecheck (pull_request) Successful in 56s
CI / integration_tests (pull_request) Successful in 4m51s
CI / unit_tests (pull_request) Successful in 19m29s
CI / docker (pull_request) Successful in 39s
CI / benchmark-regression (pull_request) Successful in 26m10s
CI / coverage (pull_request) Successful in 47m42s
CI / lint (push) Successful in 13s
CI / quality (push) Successful in 17s
CI / build (push) Successful in 23s
CI / typecheck (push) Successful in 30s
CI / security (push) Successful in 30s
CI / benchmark-regression (push) Has been skipped
CI / integration_tests (push) Successful in 2m52s
CI / unit_tests (push) Successful in 10m8s
CI / docker (push) Successful in 1m19s
CI / benchmark-publish (push) Successful in 11m54s
CI / coverage (push) Failing after 39m51s
fix(provider): remove FakeListLLM defaults
Remove FakeListLLM as a silent fallback in agent graph constructors
(plan_generation.py, context_analysis.py, auto_debug.py). All three now
raise ValueError when llm=None, making missing-provider errors explicit.

Add Settings.mock_providers flag and validate_provider_availability()
method. Update container.get_ai_provider() to check Settings.mock_providers
first, with env-var fallback for backward compatibility.

Add resolve_provider_by_name() helper to the provider registry and export
it from cleveragents.providers. Add structlog trace logging to
ProviderRegistry.get_default_provider_type() to record selection reasoning.

Update all existing behave step files, robot tests, and benchmarks that
relied on the implicit FakeListLLM default to pass an explicit LLM
instance instead.

Add new BDD tests (features/provider_fixes.feature with 17 scenarios),
Robot Framework integration tests (robot/provider_detection_smoke.robot),
and ASV benchmarks (benchmarks/provider_selection_bench.py).

ISSUES CLOSED: #323
2026-02-27 09:47:10 -05:00

163 lines
7.1 KiB
Plaintext

*** Settings ***
Documentation Integration tests for ContextAnalysisAgent workflow
Resource ${CURDIR}/common.resource
Library Process
Library OperatingSystem
Library Collections
Library String
Suite Setup Setup Test Environment
Suite Teardown Cleanup Test Environment
*** Variables ***
${PYTHON} python
${SRC_DIR} ${CURDIR}/../src
${HELPER} ${CURDIR}/helper_context_analysis.py
*** Test Cases ***
Context Analysis Agent Module Can Be Imported
[Documentation] Verify the context analysis module can be imported
${result}= Run Process ${PYTHON} -c
... import sys; sys.path.insert(0, '${SRC_DIR}'); from cleveragents.agents.context_analysis import ContextAnalysisAgent; print('SUCCESS')
... shell=True
Should Contain ${result.stdout} SUCCESS
Should Be Equal As Integers ${result.rc} 0
Context Analysis Agent Can Be Instantiated With Default Parameters
[Documentation] Create ContextAnalysisAgent with defaults
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.context_analysis import ContextAnalysisAgent
... from langchain_community.llms import FakeListLLM
... agent = ContextAnalysisAgent(llm=FakeListLLM(responses=['test']*3))
... assert agent is not None
... assert agent.chunk_size == 2000
... assert agent.chunk_overlap == 200
... assert agent.llm is not None
... print('Agent initialized successfully')
${result}= Run Process ${PYTHON} -c ${script}
Should Contain ${result.stdout} initialized successfully
Should Be Equal As Integers ${result.rc} 0
Context Analysis Agent Can Be Instantiated With Custom Chunk Settings
[Documentation] Create ContextAnalysisAgent with custom chunk_size and overlap
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.context_analysis import ContextAnalysisAgent
... from langchain_community.llms import FakeListLLM
... agent = ContextAnalysisAgent(llm=FakeListLLM(responses=['test']*3), chunk_size=1000, chunk_overlap=100)
... assert agent.chunk_size == 1000
... assert agent.chunk_overlap == 100
... print('Chunk size: ' + str(agent.chunk_size) + ', overlap: ' + str(agent.chunk_overlap))
${result}= Run Process ${PYTHON} -c ${script}
Should Contain ${result.stdout} Chunk size: 1000
Should Contain ${result.stdout} overlap: 100
Should Be Equal As Integers ${result.rc} 0
Context Analysis Agent Workflow Contains Expected Nodes
[Documentation] Verify all workflow nodes are present using helper script
${result}= Run Process ${PYTHON} ${HELPER} nodes
Log ${result.stdout}
Log ${result.stderr}
Should Contain ${result.stdout} SUCCESS
Should Contain ${result.stdout} All nodes present
Should Be Equal As Integers ${result.rc} 0
LangGraph Graphs Package Exports Context Analysis Agent
[Documentation] Ensure cleveragents.agents.graphs exports ContextAnalysisAgent
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.agents.graphs import ContextAnalysisAgent, __all__
... from cleveragents.agents import ContextAnalysisAgent as ExportedContextAgent
... assert 'ContextAnalysisAgent' in __all__
... assert ContextAnalysisAgent is ExportedContextAgent
... print('ContextAnalysisAgent export verified')
${result}= Run Process ${PYTHON} -c ${script}
Should Contain ${result.stdout} ContextAnalysisAgent export verified
Should Be Equal As Integers ${result.rc} 0
Context Analysis Agent Can Load Files
[Documentation] Test file loading functionality using helper script
${result}= Run Process ${PYTHON} ${HELPER} load_files
Log ${result.stdout}
Log ${result.stderr}
Should Contain ${result.stdout} SUCCESS
Should Contain ${result.stdout} Loaded
Should Be Equal As Integers ${result.rc} 0
Context Analysis Agent Handles Missing Files
[Documentation] Test error handling for missing files using helper script
${result}= Run Process ${PYTHON} ${HELPER} missing_file
Log ${result.stdout}
Log ${result.stderr}
Should Contain ${result.stdout} SUCCESS
Should Contain ${result.stdout} Error handled
Should Be Equal As Integers ${result.rc} 0
Context Analysis Agent Invoke Returns Complete Result
[Documentation] Test complete workflow execution via invoke using helper script
${result}= Run Process ${PYTHON} ${HELPER} invoke
Log ${result.stdout}
Log ${result.stderr}
Should Contain ${result.stdout} SUCCESS
Should Contain ${result.stdout} Workflow completed
Should Be Equal As Integers ${result.rc} 0
Context Analysis Agent Streaming Produces Updates
[Documentation] Test streaming workflow execution using helper script
${result}= Run Process ${PYTHON} ${HELPER} streaming
Log ${result.stdout}
Log ${result.stderr}
Should Contain ${result.stdout} SUCCESS
Should Contain ${result.stdout} stream updates
Should Be Equal As Integers ${result.rc} 0
Context Update Summary Models Validate Incoming Data
[Documentation] Ensure ContextUpdateResult and SummaryForUpdateContextParams accept camelCase payloads
${script}= Catenate SEPARATOR=\n
... import sys
... sys.path.insert(0, '${SRC_DIR}')
... from cleveragents.domain.models.core.context import (
... ContextType,
... ContextUpdateResult,
... SummaryForUpdateContextParams,
... )
... result = ContextUpdateResult(
... updatedContexts=[
... {
... "plan_id": 1,
... "type": ContextType.FILE,
... "path": "/tmp/main.py",
... "size": 256,
... }
... ],
... tokenDiffsById={"ctx-1": {"added": 10, "removed": 0}},
... tokensDiff=10,
... totalTokens=128,
... numFiles=1,
... numUrls=0,
... numImages=0,
... numTrees=0,
... numMaps=0,
... maxTokens=2048,
... )
... summary = SummaryForUpdateContextParams(
... numFiles=2,
... numTrees=1,
... numUrls=1,
... numMaps=0,
... tokensDiff=15,
... totalTokens=320,
... )
... assert result.num_files == 1
... assert result.updated_contexts[0].path.endswith("main.py")
... assert summary.num_urls == 1
... assert summary.tokens_diff == 15
... print('Context update models validated')
${result}= Run Process ${PYTHON} -c ${script}
Should Contain ${result.stdout} Context update models validated
Should Be Equal As Integers ${result.rc} 0