freemo
b122ec7ed5
CI / lint (pull_request) Successful in 23s
CI / quality (pull_request) Successful in 51s
CI / typecheck (pull_request) Successful in 1m0s
CI / security (pull_request) Successful in 55s
CI / build (pull_request) Successful in 25s
CI / helm (pull_request) Successful in 32s
CI / push-validation (pull_request) Successful in 26s
CI / e2e_tests (pull_request) Successful in 3m38s
CI / integration_tests (pull_request) Successful in 6m42s
CI / unit_tests (pull_request) Successful in 8m19s
CI / docker (pull_request) Successful in 13s
CI / coverage (pull_request) Successful in 15m23s
CI / status-check (pull_request) Successful in 2s
Remove the local ${PYTHON} python (and python3) variable definitions from
the *** Variables *** sections of all affected robot files. These local
definitions were overriding the pabot-injected venv Python path passed via
--variable PYTHON:/path/to/venv/python, causing tests to use the system
Python (which lacks required packages like structlog, sqlalchemy, etc.)
instead of the nox venv Python.
The correct ${PYTHON} value is already set by Setup Test Environment in
common.resource via sys.executable, and pabot passes it via --variable.
The local fallback definitions are redundant and harmful in parallel runs.
Audit found 56 robot files with the pattern (more than the 9 originally
identified in the issue). All occurrences have been removed.
ISSUES CLOSED: #1309
84 lines
4.1 KiB
Plaintext
84 lines
4.1 KiB
Plaintext
*** Settings ***
|
|
Documentation CLI Benchmark and Integration Tests for CleverAgents
|
|
Resource ${CURDIR}/common.resource
|
|
Library Process
|
|
Library OperatingSystem
|
|
Library String
|
|
Library Collections
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
|
|
*** Test Cases ***
|
|
CLI Help Command Performance
|
|
[Documentation] Benchmark help command execution time
|
|
${start_time}= Get Time epoch
|
|
${result}= Run Process ${PYTHON} -m cleveragents --help timeout=120s on_timeout=kill
|
|
${end_time}= Get Time epoch
|
|
${duration}= Evaluate ${end_time} - ${start_time}
|
|
Should Be True ${duration} < 90 Help command took too long: ${duration}s
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} AI-powered development assistant
|
|
Should Contain ${result.stdout} Usage:
|
|
|
|
CLI Version Command Performance
|
|
[Documentation] Benchmark version command execution time
|
|
${start_time}= Get Time epoch
|
|
${result}= Run Process ${PYTHON} -m cleveragents --version timeout=120s on_timeout=kill
|
|
${end_time}= Get Time epoch
|
|
${duration}= Evaluate ${end_time} - ${start_time}
|
|
Should Be True ${duration} < 90 Version command took too long: ${duration}s
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} 1.0.0
|
|
|
|
Python Module Entry Works
|
|
[Documentation] Verify Python module entry point works correctly
|
|
${result}= Run Process ${PYTHON} -m cleveragents --help timeout=120s on_timeout=kill
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} AI-powered development assistant
|
|
Should Contain ${result.stdout} Usage:
|
|
|
|
CLI Diagnostics Command
|
|
[Documentation] Test diagnostics command functionality
|
|
${result}= Run Process ${PYTHON} -m cleveragents diagnostics timeout=120s on_timeout=kill
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} Checks
|
|
Should Contain ${result.stdout} Summary
|
|
|
|
CLI Info Command
|
|
[Documentation] Test info command functionality
|
|
${result}= Run Process ${PYTHON} -m cleveragents info timeout=120s on_timeout=kill
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} Environment
|
|
Should Contain ${result.stdout} Runtime
|
|
|
|
Invalid Command Error Handling
|
|
[Documentation] Verify proper error handling for invalid commands
|
|
${result}= Run Process ${PYTHON} -m cleveragents invalid-command-xyz timeout=120s on_timeout=kill
|
|
Should Not Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stderr} Error: Invalid command
|
|
|
|
Multiple Commands Benchmark
|
|
[Documentation] Run multiple commands and measure overall performance
|
|
${commands}= Create List --help --version diagnostics info
|
|
${total_start}= Get Time epoch
|
|
FOR ${cmd} IN @{commands}
|
|
${result}= Run Process ${PYTHON} -m cleveragents ${cmd} timeout=120s on_timeout=kill
|
|
Should Be Equal As Integers ${result.rc} 0 Command failed: ${cmd}
|
|
END
|
|
${total_end}= Get Time epoch
|
|
${total_duration}= Evaluate ${total_end} - ${total_start}
|
|
Should Be True ${total_duration} < 360 Multiple commands took too long: ${total_duration}s
|
|
|
|
CLI Response Time Consistency
|
|
[Documentation] Verify consistent response times across multiple runs
|
|
@{durations}= Create List
|
|
FOR ${i} IN RANGE 5
|
|
${start}= Get Time epoch
|
|
${result}= Run Process ${PYTHON} -m cleveragents --help timeout=120s on_timeout=kill
|
|
${end}= Get Time epoch
|
|
${duration}= Evaluate ${end} - ${start}
|
|
Append To List ${durations} ${duration}
|
|
END
|
|
${avg_duration}= Evaluate sum(${durations}) / len(${durations})
|
|
Should Be True ${avg_duration} < 90 Average response time too high: ${avg_duration}s
|