brent.edwards
e4705822b1
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 16s
CI / build (pull_request) Successful in 16s
CI / quality (pull_request) Successful in 19s
CI / typecheck (pull_request) Successful in 28s
CI / security (pull_request) Successful in 31s
CI / integration_tests (pull_request) Successful in 2m24s
CI / unit_tests (pull_request) Successful in 6m13s
CI / docker (pull_request) Successful in 14s
CI / benchmark-regression (pull_request) Successful in 8m40s
CI / coverage (pull_request) Successful in 13m4s
- Add timeout=60s to all Run Process calls in cli.robot (previously had no timeout, causing hangs under CI parallel load) - Increase cli_core.robot timeouts from 30s to 60s to match other robot files - Relax performance assertions from <30s to <60s to account for Python cold-start overhead under pabot parallel execution
89 lines
4.1 KiB
Plaintext
89 lines
4.1 KiB
Plaintext
*** Settings ***
|
|
Documentation CLI Benchmark and Integration Tests for CleverAgents
|
|
Resource ${CURDIR}/common.resource
|
|
Library Process
|
|
Library OperatingSystem
|
|
Library String
|
|
Library Collections
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
|
|
*** Variables ***
|
|
# PYTHON variable is passed from nox via --variable PYTHON:/path/to/python
|
|
# Default to 'python' if not passed (for standalone runs)
|
|
${PYTHON} python
|
|
|
|
*** Test Cases ***
|
|
CLI Help Command Performance
|
|
[Documentation] Benchmark help command execution time
|
|
${start_time}= Get Time epoch
|
|
${result}= Run Process ${PYTHON} -m cleveragents --help timeout=60s
|
|
${end_time}= Get Time epoch
|
|
${duration}= Evaluate ${end_time} - ${start_time}
|
|
Should Be True ${duration} < 60 Help command took too long: ${duration}s
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} AI-powered development assistant
|
|
Should Contain ${result.stdout} Usage:
|
|
|
|
CLI Version Command Performance
|
|
[Documentation] Benchmark version command execution time
|
|
${start_time}= Get Time epoch
|
|
${result}= Run Process ${PYTHON} -m cleveragents --version timeout=60s
|
|
${end_time}= Get Time epoch
|
|
${duration}= Evaluate ${end_time} - ${start_time}
|
|
Should Be True ${duration} < 60 Version command took too long: ${duration}s
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} 1.0.0
|
|
|
|
Python Module Entry Works
|
|
[Documentation] Verify Python module entry point works correctly
|
|
${result}= Run Process ${PYTHON} -m cleveragents --help timeout=60s
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} AI-powered development assistant
|
|
Should Contain ${result.stdout} Usage:
|
|
|
|
CLI Diagnostics Command
|
|
[Documentation] Test diagnostics command functionality
|
|
${result}= Run Process ${PYTHON} -m cleveragents diagnostics timeout=60s
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} Checks
|
|
Should Contain ${result.stdout} Summary
|
|
|
|
CLI Info Command
|
|
[Documentation] Test info command functionality
|
|
${result}= Run Process ${PYTHON} -m cleveragents info timeout=60s
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} Environment
|
|
Should Contain ${result.stdout} Runtime
|
|
|
|
Invalid Command Error Handling
|
|
[Documentation] Verify proper error handling for invalid commands
|
|
${result}= Run Process ${PYTHON} -m cleveragents invalid-command-xyz timeout=60s
|
|
Should Not Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stderr} Error: Invalid command
|
|
|
|
Multiple Commands Benchmark
|
|
[Documentation] Run multiple commands and measure overall performance
|
|
${commands}= Create List --help --version diagnostics info
|
|
${total_start}= Get Time epoch
|
|
FOR ${cmd} IN @{commands}
|
|
${result}= Run Process ${PYTHON} -m cleveragents ${cmd} timeout=60s
|
|
Should Be Equal As Integers ${result.rc} 0 Command failed: ${cmd}
|
|
END
|
|
${total_end}= Get Time epoch
|
|
${total_duration}= Evaluate ${total_end} - ${total_start}
|
|
Should Be True ${total_duration} < 240 Multiple commands took too long: ${total_duration}s
|
|
|
|
CLI Response Time Consistency
|
|
[Documentation] Verify consistent response times across multiple runs
|
|
@{durations}= Create List
|
|
FOR ${i} IN RANGE 5
|
|
${start}= Get Time epoch
|
|
${result}= Run Process ${PYTHON} -m cleveragents --help timeout=60s
|
|
${end}= Get Time epoch
|
|
${duration}= Evaluate ${end} - ${start}
|
|
Append To List ${durations} ${duration}
|
|
END
|
|
${avg_duration}= Evaluate sum(${durations}) / len(${durations})
|
|
Should Be True ${avg_duration} < 60 Average response time too high: ${avg_duration}s
|