brent.edwards
01b6eb1804
CI / build (push) Successful in 17s
CI / helm (push) Successful in 22s
CI / lint (push) Successful in 28s
CI / typecheck (push) Successful in 47s
CI / benchmark-regression (push) Has been skipped
CI / quality (push) Successful in 3m49s
CI / security (push) Successful in 4m11s
CI / unit_tests (push) Successful in 9m19s
CI / docker (push) Successful in 1m22s
CI / coverage (push) Successful in 12m35s
CI / e2e_tests (push) Successful in 16m17s
CI / integration_tests (push) Successful in 25m5s
CI / status-check (push) Successful in 2s
CI / benchmark-publish (push) Successful in 28m31s
## Summary Add M6 parallel-scaling coverage for 10+ concurrent subplans: - **15-subplan parallel scenario** with explicit peak-concurrency bound checks (`max_parallel=10`) and thread-safe concurrency tracking via `_build_executor()`. - **Deep hierarchical decomposition** coverage (4+ levels) with adjusted leaf condition that only stops early when hitting `max_depth` or when the workset is trivially small (`min_files_per_subplan`). - **Non-progress guard** in `_build_hierarchy` to prevent pathological recursion when clustering cannot meaningfully split the file set. - **Small-project regression test** (< 50 files) verifying decomposition depth does not increase unexpectedly with the relaxed leaf condition. - **ASV benchmark** for 15-subplan parallel execution with `max_parallel=10` to track scaling behavior. ### Removed from this PR The `_build_hierarchy` child-linkage correctness fix (returning `node_id` from recursive calls instead of using `nodes[-1].node_id`) has been **removed** per review feedback — it is a separate bug fix and will be submitted as an independent issue/PR per CONTRIBUTING.md §Atomic Commits. ## Approach - **Concurrency tracking:** The `_build_executor()` closure in step definitions detects `context.concurrency_counter` / `context.concurrency_lock` and performs thread-safe peak tracking in a try/finally block. - **Leaf condition:** Replaced the `max_files_per_subplan` / `max_tokens_per_subplan` leaf check with a `min_files_per_subplan` check to allow deeper decomposition for large projects. Added a non-progress guard so clustering that cannot split the file set terminates immediately rather than recursing to `max_depth`. - **Deterministic IDs:** `_ids_for_count()` preserves legacy fixed IDs for the first 5 subplans and generates additional deterministic IDs for scale scenarios. ## Validation ### Passing - `nox -s lint` — all checks passed - `nox -s typecheck` — 0 errors, 0 warnings - `nox -s unit_tests` — 12,988 scenarios passed, 0 failed - `nox -s coverage_report` — 97% (passes `--fail-under=97`) Closes #855 Reviewed-on: #1201 Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com> Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
248 lines
14 KiB
Plaintext
248 lines
14 KiB
Plaintext
*** Settings ***
|
|
Documentation Core CLI Commands Integration Tests for Project, Context and Plan Management
|
|
Library OperatingSystem
|
|
Library Process
|
|
Library String
|
|
Library Collections
|
|
Resource discovery_common.resource
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
Test Timeout 900 seconds
|
|
|
|
*** Variables ***
|
|
${PYTHON} python
|
|
${TEST_DIR} ${TEMPDIR}${/}cleveragents_core_cli_test
|
|
${PROJECT_NAME} test-project
|
|
|
|
*** Test Cases ***
|
|
Test CLI Help Shows All Commands
|
|
[Documentation] Verify that CLI help displays all expected command groups
|
|
${result} = Run Process ${PYTHON} -m cleveragents --help timeout=120s
|
|
Should Be Equal As Numbers ${result.rc} 0
|
|
Should Contain ${result.stdout} AI-powered development assistant
|
|
Should Contain ${result.stdout} project
|
|
Should Contain ${result.stdout} context
|
|
Should Contain ${result.stdout} plan
|
|
Should Contain ${result.stdout} init
|
|
Should Contain ${result.stdout} tell
|
|
Should Contain ${result.stdout} build
|
|
Should Contain ${result.stdout} apply
|
|
|
|
Test Project Initialization
|
|
[Documentation] Test initializing a new CleverAgents project
|
|
Create Directory ${TEST_DIR}/project1
|
|
${result} = Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME}
|
|
... cwd=${TEST_DIR}/project1 timeout=120s
|
|
Should Be Equal As Numbers ${result.rc} 0
|
|
Should Contain ${result.stdout} Project '${PROJECT_NAME}'
|
|
Should Contain ${result.stdout} initialized
|
|
Directory Should Exist ${TEST_DIR}/project1/.cleveragents
|
|
File Should Exist ${TEST_DIR}/project1/.cleveragents/db.sqlite
|
|
File Should Exist ${TEST_DIR}/project1/.cleveragents/config.yaml
|
|
File Should Exist ${TEST_DIR}/project1/.cleveragents/current
|
|
${current} = Get File ${TEST_DIR}/project1/.cleveragents/current
|
|
Should Be Equal ${current} main
|
|
|
|
Test Project Cannot Initialize Twice
|
|
[Documentation] Verify project cannot be initialized twice without force
|
|
Create Directory ${TEST_DIR}/project2
|
|
${first_init}= Run Process ${PYTHON} -m cleveragents init first-project cwd=${TEST_DIR}/project2 timeout=120s
|
|
Should Be Equal As Numbers ${first_init.rc} 0
|
|
${result} = Run Process ${PYTHON} -m cleveragents init second-project
|
|
... cwd=${TEST_DIR}/project2 timeout=120s
|
|
Should Not Be Equal As Numbers ${result.rc} 0
|
|
Should Contain ${result.stderr} Project already initialized
|
|
Should Contain ${result.stderr} --force
|
|
|
|
Test Force Reinitialize Project
|
|
[Documentation] Test force reinitialization of a project
|
|
Create Directory ${TEST_DIR}/project3
|
|
${first_init}= Run Process ${PYTHON} -m cleveragents init first-project cwd=${TEST_DIR}/project3 timeout=120s
|
|
Should Be Equal As Numbers ${first_init.rc} 0
|
|
${result} = Run Process ${PYTHON} -m cleveragents init second-project --force
|
|
... cwd=${TEST_DIR}/project3 timeout=120s
|
|
Should Be Equal As Numbers ${result.rc} 0
|
|
Should Contain ${result.stdout} Project 'second-project' initialized successfully
|
|
|
|
Test Context Add Files
|
|
[Documentation] Test adding files to context
|
|
Create Directory ${TEST_DIR}/project4
|
|
${init_result}= Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME} cwd=${TEST_DIR}/project4 timeout=120s
|
|
Should Be Equal As Numbers ${init_result.rc} 0
|
|
Create File ${TEST_DIR}/project4/test.py print('hello world')
|
|
${result} = Run Process ${PYTHON} -m cleveragents context add test.py
|
|
... cwd=${TEST_DIR}/project4 timeout=120s
|
|
Should Be Equal As Numbers ${result.rc} 0
|
|
Should Contain ${result.stdout} Added 1 file(s) to context
|
|
|
|
Test Context List Files
|
|
[Documentation] Test listing context files
|
|
${unique_dir} = Set Variable ${TEST_DIR}/project5_${TEST NAME.replace(' ', '_')}
|
|
Create Directory ${unique_dir}
|
|
${init_result} = Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME}
|
|
... cwd=${unique_dir} timeout=120s
|
|
Should Be Equal As Numbers ${init_result.rc} 0
|
|
Create File ${unique_dir}/file1.py # File 1
|
|
Create File ${unique_dir}/file2.py # File 2
|
|
${add_result} = Run Process ${PYTHON} -m cleveragents context add file1.py file2.py
|
|
... cwd=${unique_dir} timeout=120s
|
|
Should Be Equal As Numbers ${add_result.rc} 0
|
|
Directory Should Exist ${unique_dir}
|
|
${result} = Run Process ${PYTHON} -m cleveragents context list
|
|
... cwd=${unique_dir} timeout=120s
|
|
Should Be Equal As Numbers ${result.rc} 0
|
|
Should Contain ${result.stdout} file1.py
|
|
Should Contain ${result.stdout} file2.py
|
|
|
|
Test Context Clear
|
|
[Documentation] Test clearing context files
|
|
Create Directory ${TEST_DIR}/project6
|
|
${init_result}= Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME} cwd=${TEST_DIR}/project6 timeout=120s
|
|
Should Be Equal As Numbers ${init_result.rc} 0
|
|
Create File ${TEST_DIR}/project6/test.py # Test file
|
|
${add_result}= Run Process ${PYTHON} -m cleveragents context add test.py cwd=${TEST_DIR}/project6 timeout=120s
|
|
Should Be Equal As Numbers ${add_result.rc} 0
|
|
${result} = Run Process ${PYTHON} -m cleveragents context clear --yes
|
|
... cwd=${TEST_DIR}/project6 timeout=120s
|
|
Should Be Equal As Numbers ${result.rc} 0
|
|
Should Contain ${result.stdout} Cleared all files from context
|
|
|
|
Test Plan Creation With Tell
|
|
[Documentation] Test creating a plan using tell command
|
|
Create Directory ${TEST_DIR}/project7
|
|
${init_result}= Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME} cwd=${TEST_DIR}/project7 timeout=120s
|
|
Should Be Equal As Numbers ${init_result.rc} 0
|
|
${result} = Run Process ${PYTHON} -m cleveragents tell Add error handling to main function
|
|
... cwd=${TEST_DIR}/project7 timeout=120s
|
|
Should Be Equal As Numbers ${result.rc} 0
|
|
Should Contain ${result.stdout} Plan created
|
|
Should Contain ${result.stdout} error handling
|
|
|
|
Test Plan Build
|
|
[Documentation] Test building a plan
|
|
Create Directory ${TEST_DIR}/project8
|
|
${init_result}= Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME} cwd=${TEST_DIR}/project8 timeout=120s
|
|
Should Be Equal As Numbers ${init_result.rc} 0
|
|
${tell_result}= Run Process ${PYTHON} -m cleveragents tell Create example code cwd=${TEST_DIR}/project8 timeout=120s
|
|
Should Be Equal As Numbers ${tell_result.rc} 0
|
|
${result} = Run Process ${PYTHON} -m cleveragents build cwd=${TEST_DIR}/project8
|
|
... env:CLEVERAGENTS_TESTING_USE_MOCK_AI=true timeout=120s
|
|
Should Be Equal As Numbers ${result.rc} 0
|
|
Should Contain ${result.stdout} Plan built successfully
|
|
Should Contain ${result.stdout} Generated
|
|
Should Contain ${result.stdout} change(s)
|
|
|
|
Test Plan Apply
|
|
[Documentation] Test applying plan changes
|
|
Create Directory ${TEST_DIR}/project9
|
|
${init_result}= Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME} cwd=${TEST_DIR}/project9 timeout=120s
|
|
Should Be Equal As Numbers ${init_result.rc} 0
|
|
${tell_result}= Run Process ${PYTHON} -m cleveragents tell Create example file cwd=${TEST_DIR}/project9 timeout=120s
|
|
Should Be Equal As Numbers ${tell_result.rc} 0
|
|
${build_result}= Run Process ${PYTHON} -m cleveragents build cwd=${TEST_DIR}/project9
|
|
... env:CLEVERAGENTS_TESTING_USE_MOCK_AI=true timeout=120s
|
|
Should Be Equal As Numbers ${build_result.rc} 0
|
|
${result} = Run Process ${PYTHON} -m cleveragents apply --yes cwd=${TEST_DIR}/project9 timeout=120s
|
|
Should Be Equal As Numbers ${result.rc} 0
|
|
Should Contain ${result.stdout} Successfully applied
|
|
File Should Exist ${TEST_DIR}/project9/example.py
|
|
|
|
Test Plan List
|
|
[Documentation] Test listing plans
|
|
Create Directory ${TEST_DIR}/project10
|
|
${init_result}= Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME} cwd=${TEST_DIR}/project10 timeout=120s
|
|
Should Be Equal As Numbers ${init_result.rc} 0
|
|
${tell_result}= Run Process ${PYTHON} -m cleveragents tell First plan cwd=${TEST_DIR}/project10 timeout=120s
|
|
Should Be Equal As Numbers ${tell_result.rc} 0
|
|
${plan_result}= Run Process ${PYTHON} -m cleveragents plan new second-plan cwd=${TEST_DIR}/project10 timeout=120s
|
|
Should Be Equal As Numbers ${plan_result.rc} 0
|
|
${result} = Run Process ${PYTHON} -m cleveragents plan list cwd=${TEST_DIR}/project10 timeout=120s
|
|
Should Be Equal As Numbers ${result.rc} 0
|
|
Should Contain ${result.stdout} Plans (3 total)
|
|
|
|
Test Shortcut Commands Work
|
|
[Documentation] Test that shortcut commands work properly
|
|
Create Directory ${TEST_DIR}/project11
|
|
${init_result}= Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME} cwd=${TEST_DIR}/project11 timeout=120s
|
|
Should Be Equal As Numbers ${init_result.rc} 0
|
|
Create File ${TEST_DIR}/project11/test.txt Test content
|
|
# Test context-load shortcut
|
|
${result} = Run Process ${PYTHON} -m cleveragents context-load test.txt
|
|
... cwd=${TEST_DIR}/project11 timeout=120s
|
|
Should Be Equal As Numbers ${result.rc} 0
|
|
Should Contain ${result.stdout} Added 1 file(s) to context
|
|
|
|
Test End To End Workflow
|
|
[Documentation] Test complete workflow from init to apply
|
|
Create Directory ${TEST_DIR}/project12
|
|
# Initialize project
|
|
${init} = Run Process ${PYTHON} -m cleveragents init workflow-project
|
|
... cwd=${TEST_DIR}/project12 timeout=120s
|
|
Should Be Equal As Numbers ${init.rc} 0
|
|
# Add context
|
|
Create File ${TEST_DIR}/project12/input.py def main(): pass
|
|
${add} = Run Process ${PYTHON} -m cleveragents context add input.py
|
|
... cwd=${TEST_DIR}/project12 timeout=120s
|
|
Should Be Equal As Numbers ${add.rc} 0
|
|
# Create plan
|
|
${tell} = Run Process ${PYTHON} -m cleveragents tell Add logging to main function
|
|
... cwd=${TEST_DIR}/project12 timeout=120s
|
|
Should Be Equal As Numbers ${tell.rc} 0
|
|
# Build plan
|
|
${build} = Run Process ${PYTHON} -m cleveragents build cwd=${TEST_DIR}/project12
|
|
... env:CLEVERAGENTS_TESTING_USE_MOCK_AI=true timeout=120s
|
|
Should Be Equal As Numbers ${build.rc} 0
|
|
# Apply changes
|
|
${apply} = Run Process ${PYTHON} -m cleveragents apply --yes cwd=${TEST_DIR}/project12 timeout=120s
|
|
Should Be Equal As Numbers ${apply.rc} 0
|
|
# Verify file created
|
|
File Should Exist ${TEST_DIR}/project12/example.py
|
|
|
|
Test Command Error Handling
|
|
[Documentation] Test error handling for invalid commands
|
|
${result} = Run Process ${PYTHON} -m cleveragents invalid-command timeout=120s
|
|
Should Not Be Equal As Numbers ${result.rc} 0
|
|
Should Contain ${result.stderr} Invalid command
|
|
|
|
Test Project Status Without Project
|
|
[Documentation] Test project status command without initialized project
|
|
Create Directory ${TEST_DIR}/no_project
|
|
${result} = Run Process ${PYTHON} -m cleveragents project status
|
|
... cwd=${TEST_DIR}/no_project timeout=120s
|
|
Should Not Be Equal As Numbers ${result.rc} 0
|
|
Should Contain ${result.stderr} No project found
|
|
|
|
Test Project Status With Project
|
|
[Documentation] Test project status command with initialized project
|
|
Create Directory ${TEST_DIR}/with_project
|
|
${init_result}= Run Process ${PYTHON} -m cleveragents init status-test cwd=${TEST_DIR}/with_project timeout=120s
|
|
Should Be Equal As Numbers ${init_result.rc} 0
|
|
${result} = Run Process ${PYTHON} -m cleveragents project status
|
|
... cwd=${TEST_DIR}/with_project timeout=120s
|
|
Should Be Equal As Numbers ${result.rc} 0
|
|
Should Contain ${result.stdout} Project: status-test
|
|
Should Contain ${result.stdout} Plans: 1
|
|
Should Contain ${result.stdout} Context Files: 0
|
|
|
|
*** Keywords ***
|
|
Setup Test Environment
|
|
[Documentation] Setup the test environment
|
|
${run_id}= Evaluate __import__('uuid').uuid4().hex
|
|
Set Suite Variable ${TEST_DIR} ${TEMPDIR}${/}cleveragents_core_cli_test_${run_id}
|
|
Set Suite Variable ${PROJECT_NAME} test-project-${run_id}
|
|
Set Environment Variable CLEVERAGENTS_TESTING_USE_MOCK_AI true
|
|
Create Directory ${TEST_DIR}
|
|
${home}= Set Variable ${TEST_DIR}${/}.cleveragents_home
|
|
Run Keyword And Ignore Error Remove Directory ${home} recursive=True
|
|
Create Directory ${home}
|
|
Set Environment Variable CLEVERAGENTS_HOME ${home}
|
|
|
|
Clean Core CLI Temp Dir
|
|
Run Keyword And Ignore Error Remove Directory ${TEST_DIR} recursive=True
|
|
Create Directory ${TEST_DIR}
|
|
|
|
Cleanup Test Environment
|
|
[Documentation] Clean up after tests
|
|
# Clean up test directory
|
|
Run Keyword And Ignore Error Remove Directory ${TEST_DIR} recursive=true
|