87 lines
4.0 KiB
Plaintext
87 lines
4.0 KiB
Plaintext
*** Settings ***
|
|
Documentation CLI Benchmark and Integration Tests for CleverAgents
|
|
Library Process
|
|
Library OperatingSystem
|
|
Library String
|
|
Library Collections
|
|
|
|
*** Variables ***
|
|
# PYTHON variable is passed from nox via --variable PYTHON:/path/to/python
|
|
# Default to 'python' if not passed (for standalone runs)
|
|
${PYTHON} python
|
|
|
|
*** Test Cases ***
|
|
CLI Help Command Performance
|
|
[Documentation] Benchmark help command execution time
|
|
${start_time}= Get Time epoch
|
|
${result}= Run Process ${PYTHON} -m cleveragents --help
|
|
${end_time}= Get Time epoch
|
|
${duration}= Evaluate ${end_time} - ${start_time}
|
|
Should Be True ${duration} < 2 Help command took too long: ${duration}s
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} AI-powered development assistant
|
|
Should Contain ${result.stdout} Usage:
|
|
|
|
CLI Version Command Performance
|
|
[Documentation] Benchmark version command execution time
|
|
${start_time}= Get Time epoch
|
|
${result}= Run Process ${PYTHON} -m cleveragents --version
|
|
${end_time}= Get Time epoch
|
|
${duration}= Evaluate ${end_time} - ${start_time}
|
|
Should Be True ${duration} < 2 Version command took too long: ${duration}s
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} 1.0.0
|
|
|
|
Python Module Entry Works
|
|
[Documentation] Verify Python module entry point works correctly
|
|
${result}= Run Process ${PYTHON} -m cleveragents --help
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} AI-powered development assistant
|
|
Should Contain ${result.stdout} Usage:
|
|
|
|
CLI Diagnostics Command
|
|
[Documentation] Test diagnostics command functionality
|
|
${result}= Run Process ${PYTHON} -m cleveragents diagnostics
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} CleverAgents Diagnostics
|
|
Should Contain ${result.stdout} Version:
|
|
Should Contain ${result.stdout} Python:
|
|
Should Contain ${result.stdout} Platform:
|
|
|
|
CLI Info Command
|
|
[Documentation] Test info command functionality
|
|
${result}= Run Process ${PYTHON} -m cleveragents info
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} CleverAgents Information
|
|
Should Contain ${result.stdout} Version:
|
|
|
|
Invalid Command Error Handling
|
|
[Documentation] Verify proper error handling for invalid commands
|
|
${result}= Run Process ${PYTHON} -m cleveragents invalid-command-xyz
|
|
Should Not Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stderr} Error: Invalid command
|
|
|
|
Multiple Commands Benchmark
|
|
[Documentation] Run multiple commands and measure overall performance
|
|
${commands}= Create List --help --version diagnostics info
|
|
${total_start}= Get Time epoch
|
|
FOR ${cmd} IN @{commands}
|
|
${result}= Run Process ${PYTHON} -m cleveragents ${cmd}
|
|
Should Be Equal As Integers ${result.rc} 0 Command failed: ${cmd}
|
|
END
|
|
${total_end}= Get Time epoch
|
|
${total_duration}= Evaluate ${total_end} - ${total_start}
|
|
Should Be True ${total_duration} < 10 Multiple commands took too long: ${total_duration}s
|
|
|
|
CLI Response Time Consistency
|
|
[Documentation] Verify consistent response times across multiple runs
|
|
@{durations}= Create List
|
|
FOR ${i} IN RANGE 5
|
|
${start}= Get Time epoch
|
|
${result}= Run Process ${PYTHON} -m cleveragents --help
|
|
${end}= Get Time epoch
|
|
${duration}= Evaluate ${end} - ${start}
|
|
Append To List ${durations} ${duration}
|
|
END
|
|
${avg_duration}= Evaluate sum(${durations}) / len(${durations})
|
|
Should Be True ${avg_duration} < 1.5 Average response time too high: ${avg_duration}s |