forked from cleveragents/cleveragents-core
brent.edwards
01b6eb1804
## Summary Add M6 parallel-scaling coverage for 10+ concurrent subplans: - **15-subplan parallel scenario** with explicit peak-concurrency bound checks (`max_parallel=10`) and thread-safe concurrency tracking via `_build_executor()`. - **Deep hierarchical decomposition** coverage (4+ levels) with adjusted leaf condition that only stops early when hitting `max_depth` or when the workset is trivially small (`min_files_per_subplan`). - **Non-progress guard** in `_build_hierarchy` to prevent pathological recursion when clustering cannot meaningfully split the file set. - **Small-project regression test** (< 50 files) verifying decomposition depth does not increase unexpectedly with the relaxed leaf condition. - **ASV benchmark** for 15-subplan parallel execution with `max_parallel=10` to track scaling behavior. ### Removed from this PR The `_build_hierarchy` child-linkage correctness fix (returning `node_id` from recursive calls instead of using `nodes[-1].node_id`) has been **removed** per review feedback — it is a separate bug fix and will be submitted as an independent issue/PR per CONTRIBUTING.md §Atomic Commits. ## Approach - **Concurrency tracking:** The `_build_executor()` closure in step definitions detects `context.concurrency_counter` / `context.concurrency_lock` and performs thread-safe peak tracking in a try/finally block. - **Leaf condition:** Replaced the `max_files_per_subplan` / `max_tokens_per_subplan` leaf check with a `min_files_per_subplan` check to allow deeper decomposition for large projects. Added a non-progress guard so clustering that cannot split the file set terminates immediately rather than recursing to `max_depth`. - **Deterministic IDs:** `_ids_for_count()` preserves legacy fixed IDs for the first 5 subplans and generates additional deterministic IDs for scale scenarios. ## Validation ### Passing - `nox -s lint` — all checks passed - `nox -s typecheck` — 0 errors, 0 warnings - `nox -s unit_tests` — 12,988 scenarios passed, 0 failed - `nox -s coverage_report` — 97% (passes `--fail-under=97`) Closes #855 Reviewed-on: cleveragents/cleveragents-core#1201 Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com> Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
123 lines
6.4 KiB
Plaintext
123 lines
6.4 KiB
Plaintext
*** Settings ***
|
|
Documentation Integration tests for enhanced version, info, diagnostics commands
|
|
Resource ${CURDIR}/common.resource
|
|
Library Process
|
|
Library OperatingSystem
|
|
Library String
|
|
Library Collections
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
|
|
*** Variables ***
|
|
${PYTHON} python
|
|
|
|
*** Keywords ***
|
|
Run CLI With Clean Home
|
|
[Documentation] Run a CLI command with a temp HOME to avoid stale config
|
|
[Arguments] @{cmd}
|
|
${tmpdir}= Evaluate __import__('tempfile').mkdtemp(prefix='cli_core_')
|
|
${result}= Run Process @{cmd} timeout=120s on_timeout=kill env:HOME=${tmpdir}
|
|
RETURN ${result}
|
|
|
|
*** Test Cases ***
|
|
Version Command Default Rich Format
|
|
[Documentation] Version command with default (rich) format shows version string
|
|
${result}= Run Process ${PYTHON} -m cleveragents version timeout=120s on_timeout=kill
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} 1.0.0
|
|
Should Contain ${result.stdout} CleverAgents
|
|
|
|
Version Command JSON Format
|
|
[Documentation] Version command with --format json returns valid JSON
|
|
${result}= Run Process ${PYTHON} -m cleveragents version --format json timeout=120s on_timeout=kill
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} "version": "1.0.0"
|
|
Should Contain ${result.stdout} "schema": "v3"
|
|
Should Contain ${result.stdout} "channel": "stable"
|
|
Should Contain ${result.stdout} "python"
|
|
Should Contain ${result.stdout} "dependencies"
|
|
Should Contain ${result.stdout} "build_date"
|
|
Should Contain ${result.stdout} "commit"
|
|
|
|
Version Command Plain Format
|
|
[Documentation] Version command with --format plain returns key-value pairs
|
|
${result}= Run Process ${PYTHON} -m cleveragents version --format plain timeout=120s on_timeout=kill
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} version: 1.0.0
|
|
Should Contain ${result.stdout} schema: v3
|
|
|
|
Version Command YAML Format
|
|
[Documentation] Version command with --format yaml returns YAML
|
|
${result}= Run Process ${PYTHON} -m cleveragents version --format yaml timeout=120s on_timeout=kill
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} version: 1.0.0
|
|
Should Contain ${result.stdout} schema: v3
|
|
|
|
Info Command Default Rich Format
|
|
[Documentation] Info command with default (rich) format shows environment details
|
|
${result}= Run Process ${PYTHON} -m cleveragents info timeout=120s on_timeout=kill
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} Environment
|
|
Should Contain ${result.stdout} Runtime
|
|
|
|
Info Command JSON Format
|
|
[Documentation] Info command with --format json returns structured data
|
|
${result}= Run CLI With Clean Home ${PYTHON} -m cleveragents info --format json
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} "version": "1.0.0"
|
|
Should Contain ${result.stdout} "data_dir"
|
|
Should Contain ${result.stdout} "database"
|
|
Should Contain ${result.stdout} "server_mode"
|
|
|
|
Info Command Plain Format
|
|
[Documentation] Info command with --format plain returns key-value pairs
|
|
${result}= Run CLI With Clean Home ${PYTHON} -m cleveragents info --format plain
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} version: 1.0.0
|
|
Should Contain ${result.stdout} server_mode:
|
|
|
|
Diagnostics Command Default Rich Format
|
|
[Documentation] Diagnostics command with default (rich) format runs checks
|
|
${result}= Run Process ${PYTHON} -m cleveragents diagnostics timeout=120s on_timeout=kill
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} Checks
|
|
Should Contain ${result.stdout} Summary
|
|
|
|
Diagnostics Command JSON Format
|
|
[Documentation] Diagnostics command with --format json returns structured data
|
|
${result}= Run Process ${PYTHON} -m cleveragents diagnostics --format json timeout=120s on_timeout=kill
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} "checks"
|
|
Should Contain ${result.stdout} "summary"
|
|
Should Contain ${result.stdout} "recommendations"
|
|
Should Contain ${result.stdout} "total"
|
|
Should Contain ${result.stdout} "warnings"
|
|
Should Contain ${result.stdout} "errors"
|
|
|
|
Diagnostics Command Plain Format
|
|
[Documentation] Diagnostics command with --format plain returns key-value pairs
|
|
${result}= Run Process ${PYTHON} -m cleveragents diagnostics --format plain timeout=120s on_timeout=kill
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} checks:
|
|
Should Contain ${result.stdout} summary:
|
|
|
|
Diagnostics Command Check Flag Returns Valid Exit Code
|
|
[Documentation] Diagnostics --check exits 0 when no critical errors are found.
|
|
... Disk-space errors are tolerated since CI runners may have low disk.
|
|
${result}= Run Process ${PYTHON} -m cleveragents diagnostics --check --format json timeout=120s on_timeout=kill
|
|
Should Contain ${result.stdout} "checks"
|
|
Should Contain ${result.stdout} "has_errors"
|
|
# Accept exit code 0 (clean) or 1 if the only error is disk space (CI environment)
|
|
IF ${result.rc} != 0
|
|
Should Contain ${result.stdout} Disk space Diagnostics --check failed with non-disk error: ${result.stdout}
|
|
END
|
|
|
|
Diagnostics Command Performance
|
|
[Documentation] Diagnostics command completes within acceptable time
|
|
${start}= Get Time epoch
|
|
${result}= Run Process ${PYTHON} -m cleveragents diagnostics --format json timeout=120s on_timeout=kill
|
|
${end}= Get Time epoch
|
|
${duration}= Evaluate ${end} - ${start}
|
|
Should Be True ${duration} < 90 Diagnostics took too long: ${duration}s
|
|
Should Be Equal As Integers ${result.rc} 0
|