forked from cleveragents/cleveragents-core
brent.edwards
01b6eb1804
## Summary Add M6 parallel-scaling coverage for 10+ concurrent subplans: - **15-subplan parallel scenario** with explicit peak-concurrency bound checks (`max_parallel=10`) and thread-safe concurrency tracking via `_build_executor()`. - **Deep hierarchical decomposition** coverage (4+ levels) with adjusted leaf condition that only stops early when hitting `max_depth` or when the workset is trivially small (`min_files_per_subplan`). - **Non-progress guard** in `_build_hierarchy` to prevent pathological recursion when clustering cannot meaningfully split the file set. - **Small-project regression test** (< 50 files) verifying decomposition depth does not increase unexpectedly with the relaxed leaf condition. - **ASV benchmark** for 15-subplan parallel execution with `max_parallel=10` to track scaling behavior. ### Removed from this PR The `_build_hierarchy` child-linkage correctness fix (returning `node_id` from recursive calls instead of using `nodes[-1].node_id`) has been **removed** per review feedback — it is a separate bug fix and will be submitted as an independent issue/PR per CONTRIBUTING.md §Atomic Commits. ## Approach - **Concurrency tracking:** The `_build_executor()` closure in step definitions detects `context.concurrency_counter` / `context.concurrency_lock` and performs thread-safe peak tracking in a try/finally block. - **Leaf condition:** Replaced the `max_files_per_subplan` / `max_tokens_per_subplan` leaf check with a `min_files_per_subplan` check to allow deeper decomposition for large projects. Added a non-progress guard so clustering that cannot split the file set terminates immediately rather than recursing to `max_depth`. - **Deterministic IDs:** `_ids_for_count()` preserves legacy fixed IDs for the first 5 subplans and generates additional deterministic IDs for scale scenarios. ## Validation ### Passing - `nox -s lint` — all checks passed - `nox -s typecheck` — 0 errors, 0 warnings - `nox -s unit_tests` — 12,988 scenarios passed, 0 failed - `nox -s coverage_report` — 97% (passes `--fail-under=97`) Closes #855 Reviewed-on: cleveragents/cleveragents-core#1201 Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com> Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
303 lines
12 KiB
Plaintext
303 lines
12 KiB
Plaintext
*** Settings ***
|
|
Documentation CLI Plan and Context Commands - Robot Framework Tests
|
|
... Comprehensive testing of plan and context workflows
|
|
Resource ${CURDIR}/common.resource
|
|
Library OperatingSystem
|
|
Library Process
|
|
Library String
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
Test Timeout 900 seconds
|
|
|
|
*** Variables ***
|
|
# ${PYTHON} will be set in Setup Test Environment from common.resource
|
|
# Make sure to call Setup Test Environment before using ${PYTHON}
|
|
${TEST_DIR} ${TEMPDIR}${/}cleveragents_plan_ctx_test
|
|
${PROJECT_NAME} test-project
|
|
|
|
*** Test Cases ***
|
|
Initialize New Project
|
|
[Documentation] Test cleveragents init command
|
|
[Setup] Setup Test Directory
|
|
|
|
${result}= Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME}
|
|
... cwd=${TEST_DIR} env:CLEVERAGENTS_AUTO_APPLY_MIGRATIONS=true stderr=STDOUT timeout=300s on_timeout=kill
|
|
|
|
Should Be Equal As Integers ${result.rc} 0 Init failed: ${result.stdout}
|
|
Should Exist ${TEST_DIR}${/}.cleveragents
|
|
Should Exist ${TEST_DIR}${/}.cleveragents${/}db.sqlite
|
|
|
|
[Teardown] Cleanup Test Directory
|
|
|
|
Add Files To Context
|
|
[Documentation] Test context-load command
|
|
[Setup] Initialize Test Project
|
|
|
|
# Create test files
|
|
Create File ${TEST_DIR}${/}test.py print('hello')
|
|
Create File ${TEST_DIR}${/}main.py def main(): pass
|
|
|
|
# Add files to context
|
|
${result}= Run Process ${PYTHON} -m cleveragents context-load test.py main.py
|
|
... cwd=${TEST_DIR} timeout=300s on_timeout=kill
|
|
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
# Verify files are in context
|
|
${result}= Run Process ${PYTHON} -m cleveragents context list
|
|
... cwd=${TEST_DIR} timeout=120s on_timeout=kill
|
|
|
|
Should Contain ${result.stdout} test.py
|
|
Should Contain ${result.stdout} main.py
|
|
|
|
[Teardown] Cleanup Test Directory
|
|
|
|
Create Plan With Tell
|
|
[Documentation] Test tell command
|
|
[Setup] Initialize Test Project
|
|
|
|
${result}= Run Process ${PYTHON} -m cleveragents tell Add error handling
|
|
... cwd=${TEST_DIR} timeout=120s on_timeout=kill
|
|
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
# Verify plan was created with correct name derived from prompt
|
|
${result}= Run Process ${PYTHON} -m cleveragents plan current
|
|
... cwd=${TEST_DIR} timeout=120s on_timeout=kill
|
|
|
|
Should Contain ${result.stdout} add_error_handling
|
|
|
|
[Teardown] Cleanup Test Directory
|
|
|
|
Build Plan
|
|
[Documentation] Test build command
|
|
[Setup] Initialize Test Project With Plan
|
|
|
|
${result}= Run Process ${PYTHON} -m cleveragents build
|
|
... cwd=${TEST_DIR} env:CLEVERAGENTS_TESTING_USE_MOCK_AI=true timeout=120s on_timeout=kill
|
|
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
[Teardown] Cleanup Test Directory
|
|
|
|
Apply Plan Changes
|
|
[Documentation] Test apply command
|
|
[Setup] Initialize Test Project With Built Plan
|
|
|
|
${result}= Run Process ${PYTHON} -m cleveragents apply --yes
|
|
... cwd=${TEST_DIR} timeout=120s on_timeout=kill
|
|
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
[Teardown] Cleanup Test Directory
|
|
|
|
Create New Empty Plan
|
|
[Documentation] Test plan new command
|
|
[Setup] Initialize Test Project
|
|
|
|
${result}= Run Process ${PYTHON} -m cleveragents plan new feature-plan
|
|
... cwd=${TEST_DIR} timeout=120s on_timeout=kill
|
|
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
# Verify new plan is current
|
|
${result}= Run Process ${PYTHON} -m cleveragents plan current
|
|
... cwd=${TEST_DIR} timeout=120s on_timeout=kill
|
|
|
|
Should Contain ${result.stdout} feature-plan
|
|
|
|
[Teardown] Cleanup Test Directory
|
|
|
|
Show Current Plan
|
|
[Documentation] Test plan current command
|
|
[Setup] Initialize Test Project
|
|
|
|
${result}= Run Process ${PYTHON} -m cleveragents plan current
|
|
... cwd=${TEST_DIR} timeout=120s on_timeout=kill
|
|
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} main
|
|
|
|
[Teardown] Cleanup Test Directory
|
|
|
|
List All Plans
|
|
[Documentation] Test plan list command
|
|
[Setup] Initialize Test Project With Multiple Plans
|
|
|
|
${result}= Run Process ${PYTHON} -m cleveragents plan list
|
|
... cwd=${TEST_DIR} timeout=120s on_timeout=kill
|
|
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} main
|
|
Should Contain ${result.stdout} feature-1
|
|
Should Contain ${result.stdout} feature-2
|
|
|
|
[Teardown] Cleanup Test Directory
|
|
|
|
Switch Between Plans
|
|
[Documentation] Test plan cd command
|
|
[Setup] Initialize Test Project With Multiple Plans
|
|
|
|
# Switch to feature-1
|
|
${result}= Run Process ${PYTHON} -m cleveragents plan cd feature-1
|
|
... cwd=${TEST_DIR} timeout=120s on_timeout=kill
|
|
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
# Verify switch
|
|
${result}= Run Process ${PYTHON} -m cleveragents plan current
|
|
... cwd=${TEST_DIR} timeout=120s on_timeout=kill
|
|
|
|
Should Contain ${result.stdout} feature-1
|
|
|
|
[Teardown] Cleanup Test Directory
|
|
|
|
Continue Working On Plan
|
|
[Documentation] Test plan continue command
|
|
[Setup] Initialize Test Project With Plan
|
|
|
|
${result}= Run Process ${PYTHON} -m cleveragents plan continue Also add logging
|
|
... cwd=${TEST_DIR} env:CLEVERAGENTS_TESTING_USE_MOCK_AI=true timeout=120s on_timeout=kill
|
|
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
[Teardown] Cleanup Test Directory
|
|
|
|
List Context Files
|
|
[Documentation] Test context list command
|
|
[Setup] Initialize Test Project With Context
|
|
|
|
${result}= Run Process ${PYTHON} -m cleveragents context list
|
|
... cwd=${TEST_DIR} timeout=120s on_timeout=kill
|
|
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} test.py
|
|
Should Contain ${result.stdout} utils.py
|
|
|
|
[Teardown] Cleanup Test Directory
|
|
|
|
Show Context Content
|
|
[Documentation] Test context show command
|
|
[Setup] Initialize Test Project With Context
|
|
|
|
# Show specific file content
|
|
${result}= Run Process ${PYTHON} -m cleveragents context show test.py
|
|
... cwd=${TEST_DIR} timeout=120s on_timeout=kill
|
|
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} def hello
|
|
|
|
# Also test showing summary (without file argument)
|
|
${result}= Run Process ${PYTHON} -m cleveragents context show
|
|
... cwd=${TEST_DIR} timeout=120s on_timeout=kill
|
|
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} Context Summary
|
|
Should Contain ${result.stdout} Total files: 2
|
|
|
|
[Teardown] Cleanup Test Directory
|
|
|
|
Remove File From Context
|
|
[Documentation] Test context rm command
|
|
[Setup] Initialize Test Project With Context
|
|
|
|
# Remove test.py
|
|
${result}= Run Process ${PYTHON} -m cleveragents context rm test.py
|
|
... cwd=${TEST_DIR} timeout=120s on_timeout=kill
|
|
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
# Verify removal
|
|
${result}= Run Process ${PYTHON} -m cleveragents context list
|
|
... cwd=${TEST_DIR} timeout=120s on_timeout=kill
|
|
|
|
Should Not Contain ${result.stdout} test.py
|
|
Should Contain ${result.stdout} utils.py
|
|
|
|
[Teardown] Cleanup Test Directory
|
|
|
|
Clear All Context
|
|
[Documentation] Test context clear command
|
|
[Setup] Initialize Test Project With Context
|
|
|
|
${result}= Run Process ${PYTHON} -m cleveragents context clear --yes
|
|
... cwd=${TEST_DIR} env:CLEVERAGENTS_AUTO_APPLY_MIGRATIONS=true stderr=STDOUT timeout=120s on_timeout=kill
|
|
|
|
Should Be Equal As Integers ${result.rc} 0 Context clear failed: ${result.stdout}
|
|
|
|
# Verify context is empty
|
|
${result}= Run Process ${PYTHON} -m cleveragents context list
|
|
... cwd=${TEST_DIR} env:CLEVERAGENTS_AUTO_APPLY_MIGRATIONS=true stderr=STDOUT timeout=120s on_timeout=kill
|
|
|
|
Should Not Contain ${result.stdout} test.py
|
|
Should Not Contain ${result.stdout} utils.py
|
|
|
|
[Teardown] Cleanup Test Directory
|
|
|
|
Command Error Handling
|
|
[Documentation] Test error cases for commands
|
|
[Setup] Setup Test Directory
|
|
|
|
# Try to use command without project
|
|
${result}= Run Process ${PYTHON} -m cleveragents plan tell test
|
|
... cwd=${TEST_DIR} stderr=STDOUT timeout=120s on_timeout=kill
|
|
|
|
Should Not Be Equal As Integers ${result.rc} 0
|
|
|
|
[Teardown] Cleanup Test Directory
|
|
|
|
*** Keywords ***
|
|
Setup Test Directory
|
|
[Documentation] Create clean test directory
|
|
${run_id}= Evaluate __import__('uuid').uuid4().hex
|
|
Set Test Variable ${TEST_DIR} ${TEMPDIR}${/}cleveragents_plan_ctx_test_${run_id}
|
|
Set Test Variable ${PROJECT_NAME} test-project-${run_id}
|
|
Set Environment Variable CLEVERAGENTS_TESTING_USE_MOCK_AI true
|
|
Run Keyword And Ignore Error Remove Directory ${TEST_DIR} recursive=True
|
|
Create Directory ${TEST_DIR}
|
|
|
|
Cleanup Test Directory
|
|
[Documentation] Remove test directory and cleanup processes
|
|
# Clean up test directory
|
|
Run Keyword And Ignore Error Remove Directory ${TEST_DIR} recursive=True
|
|
|
|
Initialize Test Project
|
|
[Documentation] Initialize a test project
|
|
Setup Test Directory
|
|
${result}= Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME}
|
|
... cwd=${TEST_DIR} timeout=300s on_timeout=kill
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
Initialize Test Project With Plan
|
|
[Documentation] Initialize project and create a plan
|
|
Initialize Test Project
|
|
${result}= Run Process ${PYTHON} -m cleveragents tell Test instruction
|
|
... cwd=${TEST_DIR} env:CLEVERAGENTS_TESTING_USE_MOCK_AI=true timeout=120s on_timeout=kill
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
Initialize Test Project With Built Plan
|
|
[Documentation] Initialize project with a built plan
|
|
Initialize Test Project With Plan
|
|
${result}= Run Process ${PYTHON} -m cleveragents build
|
|
... cwd=${TEST_DIR} env:CLEVERAGENTS_TESTING_USE_MOCK_AI=true timeout=120s on_timeout=kill
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
Initialize Test Project With Multiple Plans
|
|
[Documentation] Initialize project with multiple plans
|
|
Initialize Test Project
|
|
${result}= Run Process ${PYTHON} -m cleveragents plan new feature-1
|
|
... cwd=${TEST_DIR} timeout=120s on_timeout=kill
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
${result}= Run Process ${PYTHON} -m cleveragents plan new feature-2
|
|
... cwd=${TEST_DIR} timeout=120s on_timeout=kill
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
Initialize Test Project With Context
|
|
[Documentation] Initialize project and add files to context
|
|
Initialize Test Project
|
|
Create File ${TEST_DIR}${/}test.py def hello(): pass
|
|
Create File ${TEST_DIR}${/}utils.py import os
|
|
${result}= Run Process ${PYTHON} -m cleveragents context-load test.py utils.py
|
|
... cwd=${TEST_DIR} env:CLEVERAGENTS_AUTO_APPLY_MIGRATIONS=true stderr=STDOUT timeout=120s on_timeout=kill
|
|
Should Be Equal As Integers ${result.rc} 0 context-load failed: ${result.stdout}
|