brent.edwards
e8aa5ac268
CI / lint (pull_request) Successful in 15s
CI / typecheck (pull_request) Successful in 28s
CI / security (pull_request) Successful in 21s
CI / quality (pull_request) Successful in 15s
CI / integration_tests (pull_request) Failing after 6m25s
CI / build (pull_request) Successful in 15s
CI / unit_tests (pull_request) Successful in 6m52s
CI / docker (pull_request) Successful in 9s
CI / coverage (pull_request) Successful in 5m5s
- Restore session.env["PATH"] in integration_tests nox session to ensure
Run Process uses venv Python instead of system Python
- Convert bare Resource references to ${CURDIR}/ absolute paths across
30 robot files to fix CI resolution failures
- Add timeout=30s to all Run Process calls in rxpy_route_validation.robot
to prevent hanging tests
- Tag 2 rxpy tests as slow (require running actors unavailable on CI)
- Fix LangGraph test to use correct config file (LANGGRAPH_CONFIG)
- All 204 tests pass (4 excluded: 2 slow + 2 discovery)
91 lines
4.2 KiB
Plaintext
91 lines
4.2 KiB
Plaintext
*** Settings ***
|
|
Documentation CLI Benchmark and Integration Tests for CleverAgents
|
|
Resource ${CURDIR}/common.resource
|
|
Library Process
|
|
Library OperatingSystem
|
|
Library String
|
|
Library Collections
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
|
|
*** Variables ***
|
|
# PYTHON variable is passed from nox via --variable PYTHON:/path/to/python
|
|
# Default to 'python' if not passed (for standalone runs)
|
|
${PYTHON} python
|
|
|
|
*** Test Cases ***
|
|
CLI Help Command Performance
|
|
[Documentation] Benchmark help command execution time
|
|
${start_time}= Get Time epoch
|
|
${result}= Run Process ${PYTHON} -m cleveragents --help
|
|
${end_time}= Get Time epoch
|
|
${duration}= Evaluate ${end_time} - ${start_time}
|
|
Should Be True ${duration} < 30 Help command took too long: ${duration}s
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} AI-powered development assistant
|
|
Should Contain ${result.stdout} Usage:
|
|
|
|
CLI Version Command Performance
|
|
[Documentation] Benchmark version command execution time
|
|
${start_time}= Get Time epoch
|
|
${result}= Run Process ${PYTHON} -m cleveragents --version
|
|
${end_time}= Get Time epoch
|
|
${duration}= Evaluate ${end_time} - ${start_time}
|
|
Should Be True ${duration} < 30 Version command took too long: ${duration}s
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} 1.0.0
|
|
|
|
Python Module Entry Works
|
|
[Documentation] Verify Python module entry point works correctly
|
|
${result}= Run Process ${PYTHON} -m cleveragents --help
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} AI-powered development assistant
|
|
Should Contain ${result.stdout} Usage:
|
|
|
|
CLI Diagnostics Command
|
|
[Documentation] Test diagnostics command functionality
|
|
${result}= Run Process ${PYTHON} -m cleveragents diagnostics
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} CleverAgents Diagnostics
|
|
Should Contain ${result.stdout} Version:
|
|
Should Contain ${result.stdout} Python:
|
|
Should Contain ${result.stdout} Platform:
|
|
|
|
CLI Info Command
|
|
[Documentation] Test info command functionality
|
|
${result}= Run Process ${PYTHON} -m cleveragents info
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} CleverAgents Information
|
|
Should Contain ${result.stdout} Version:
|
|
|
|
Invalid Command Error Handling
|
|
[Documentation] Verify proper error handling for invalid commands
|
|
${result}= Run Process ${PYTHON} -m cleveragents invalid-command-xyz
|
|
Should Not Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stderr} Error: Invalid command
|
|
|
|
Multiple Commands Benchmark
|
|
[Documentation] Run multiple commands and measure overall performance
|
|
${commands}= Create List --help --version diagnostics info
|
|
${total_start}= Get Time epoch
|
|
FOR ${cmd} IN @{commands}
|
|
${result}= Run Process ${PYTHON} -m cleveragents ${cmd}
|
|
Should Be Equal As Integers ${result.rc} 0 Command failed: ${cmd}
|
|
END
|
|
${total_end}= Get Time epoch
|
|
${total_duration}= Evaluate ${total_end} - ${total_start}
|
|
Should Be True ${total_duration} < 60 Multiple commands took too long: ${total_duration}s
|
|
|
|
CLI Response Time Consistency
|
|
[Documentation] Verify consistent response times across multiple runs
|
|
@{durations}= Create List
|
|
FOR ${i} IN RANGE 5
|
|
${start}= Get Time epoch
|
|
${result}= Run Process ${PYTHON} -m cleveragents --help
|
|
${end}= Get Time epoch
|
|
${duration}= Evaluate ${end} - ${start}
|
|
Append To List ${durations} ${duration}
|
|
END
|
|
${avg_duration}= Evaluate sum(${durations}) / len(${durations})
|
|
Should Be True ${avg_duration} < 10 Average response time too high: ${avg_duration}s
|