brent.edwards
4c8cfdcc60
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 19s
CI / helm (pull_request) Successful in 21s
CI / lint (pull_request) Successful in 3m18s
CI / quality (pull_request) Successful in 3m42s
CI / typecheck (pull_request) Successful in 3m56s
CI / security (pull_request) Successful in 4m6s
CI / unit_tests (pull_request) Successful in 6m34s
CI / docker (pull_request) Successful in 1m31s
CI / coverage (pull_request) Successful in 12m43s
CI / e2e_tests (pull_request) Successful in 19m39s
CI / integration_tests (pull_request) Successful in 24m43s
CI / status-check (pull_request) Successful in 1s
CI / benchmark-regression (pull_request) Successful in 55m19s
Add M6 parallel-scaling coverage for 10+ concurrent subplans by introducing a 15-subplan Behave scenario with explicit peak-concurrency bound checks (max_parallel=10) and thread-safe concurrency tracking via _build_executor(). Add deep hierarchical decomposition coverage (4+ levels) and adjust the decomposition leaf condition to only stop early when hitting max_depth or when the workset is trivially small (min_files_per_subplan). A non-progress guard prevents pathological recursion when clustering cannot meaningfully split the file set. Add a small-project regression test (< 50 files) verifying that the relaxed leaf condition does not cause unexpected decomposition depth for small worksets. Add ASV benchmark coverage for 15-subplan parallel execution to track scaling behavior under higher concurrency. Note: the _build_hierarchy child-linkage correctness fix (returning node_id from recursive calls instead of using nodes[-1].node_id) has been removed from this PR per review feedback — it is a separate bug fix and will be submitted as an independent issue/PR per CONTRIBUTING.md atomic commit policy. ISSUES CLOSED: #855
89 lines
4.3 KiB
Plaintext
89 lines
4.3 KiB
Plaintext
*** Settings ***
|
|
Documentation CLI Benchmark and Integration Tests for CleverAgents
|
|
Resource ${CURDIR}/common.resource
|
|
Library Process
|
|
Library OperatingSystem
|
|
Library String
|
|
Library Collections
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
|
|
*** Variables ***
|
|
# PYTHON variable is passed from nox via --variable PYTHON:/path/to/python
|
|
# Default to 'python' if not passed (for standalone runs)
|
|
${PYTHON} python
|
|
|
|
*** Test Cases ***
|
|
CLI Help Command Performance
|
|
[Documentation] Benchmark help command execution time
|
|
${start_time}= Get Time epoch
|
|
${result}= Run Process ${PYTHON} -m cleveragents --help timeout=120s on_timeout=kill
|
|
${end_time}= Get Time epoch
|
|
${duration}= Evaluate ${end_time} - ${start_time}
|
|
Should Be True ${duration} < 90 Help command took too long: ${duration}s
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} AI-powered development assistant
|
|
Should Contain ${result.stdout} Usage:
|
|
|
|
CLI Version Command Performance
|
|
[Documentation] Benchmark version command execution time
|
|
${start_time}= Get Time epoch
|
|
${result}= Run Process ${PYTHON} -m cleveragents --version timeout=120s on_timeout=kill
|
|
${end_time}= Get Time epoch
|
|
${duration}= Evaluate ${end_time} - ${start_time}
|
|
Should Be True ${duration} < 90 Version command took too long: ${duration}s
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} 1.0.0
|
|
|
|
Python Module Entry Works
|
|
[Documentation] Verify Python module entry point works correctly
|
|
${result}= Run Process ${PYTHON} -m cleveragents --help timeout=120s on_timeout=kill
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} AI-powered development assistant
|
|
Should Contain ${result.stdout} Usage:
|
|
|
|
CLI Diagnostics Command
|
|
[Documentation] Test diagnostics command functionality
|
|
${result}= Run Process ${PYTHON} -m cleveragents diagnostics timeout=120s on_timeout=kill
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} Checks
|
|
Should Contain ${result.stdout} Summary
|
|
|
|
CLI Info Command
|
|
[Documentation] Test info command functionality
|
|
${result}= Run Process ${PYTHON} -m cleveragents info timeout=120s on_timeout=kill
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} Environment
|
|
Should Contain ${result.stdout} Runtime
|
|
|
|
Invalid Command Error Handling
|
|
[Documentation] Verify proper error handling for invalid commands
|
|
${result}= Run Process ${PYTHON} -m cleveragents invalid-command-xyz timeout=120s on_timeout=kill
|
|
Should Not Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stderr} Error: Invalid command
|
|
|
|
Multiple Commands Benchmark
|
|
[Documentation] Run multiple commands and measure overall performance
|
|
${commands}= Create List --help --version diagnostics info
|
|
${total_start}= Get Time epoch
|
|
FOR ${cmd} IN @{commands}
|
|
${result}= Run Process ${PYTHON} -m cleveragents ${cmd} timeout=120s on_timeout=kill
|
|
Should Be Equal As Integers ${result.rc} 0 Command failed: ${cmd}
|
|
END
|
|
${total_end}= Get Time epoch
|
|
${total_duration}= Evaluate ${total_end} - ${total_start}
|
|
Should Be True ${total_duration} < 360 Multiple commands took too long: ${total_duration}s
|
|
|
|
CLI Response Time Consistency
|
|
[Documentation] Verify consistent response times across multiple runs
|
|
@{durations}= Create List
|
|
FOR ${i} IN RANGE 5
|
|
${start}= Get Time epoch
|
|
${result}= Run Process ${PYTHON} -m cleveragents --help timeout=120s on_timeout=kill
|
|
${end}= Get Time epoch
|
|
${duration}= Evaluate ${end} - ${start}
|
|
Append To List ${durations} ${duration}
|
|
END
|
|
${avg_duration}= Evaluate sum(${durations}) / len(${durations})
|
|
Should Be True ${avg_duration} < 90 Average response time too high: ${avg_duration}s
|