forked from cleveragents/cleveragents-core
cb1403c24d
## Summary
Validates all 10 M3 (v3.2.0) acceptance criteria via the E2E verification suite. This is the final gate before closing milestone v3.2.0.
### Single Commit (Squashed per CONTRIBUTING.md)
Branch history was squashed from 2 commits into 1 (`e1665c6e`) to comply with the one-commit-per-issue rule. The single commit includes:
1. **Original work**: Suite-level Force Tags, per-test acceptance criteria tags, enhanced documentation with milestone cross-references, CHANGELOG and CONTRIBUTORS entries.
2. **Review feedback fixes** (addressing Luis's CRITICAL/HIGH findings): Replaced fixture-self-assertion patterns with real CLI-path validation for all 10 acceptance criteria, added persistence-backed decision tree checks, robust JSON output parsing, explicit Robot process timeouts, and clearer correction tags.
3. **Rebase onto current master** (`fff6bb38`): Clean rebase resolving CHANGELOG conflict, as requested by PM (freemo).
4. **Bug fix — `async_enabled` attribute**: `PlanLifecycleService.execute_plan()` accesses `self.settings.async_enabled` (`plan_lifecycle_service.py:283`). The test helper's `_make_settings()` used `create_autospec(Settings)` but didn't set this attribute, causing `plan execute` CLI tests to fail with `AttributeError`. Fixed by adding `settings.async_enabled = False`.
5. **Timeout fixes for parallel execution**: When `pabot` runs 16 parallel Robot processes, cold-start import times + Alembic migrations cause 30s timeouts to be insufficient. Increased timeouts from 30s to 120s in 3 robot files: `actor_context_management.robot`, `decision_di_wiring_smoke.robot`, `changeset_persistence.robot`.
### Files Changed (7)
| File | Change |
|------|--------|
| `robot/helper_m3_e2e_verification.py` | Rewritten for real CLI-path validation; `_make_settings()` fixed with `async_enabled = False` |
| `robot/m3_e2e_verification.robot` | Force Tags, per-test tags, timeouts, enhanced docs |
| `robot/actor_context_management.robot` | Timeout 30s → 120s (line 92) |
| `robot/decision_di_wiring_smoke.robot` | Timeout 30s → 120s (lines 14, 21) |
| `robot/changeset_persistence.robot` | `${TIMEOUT}` 30s → 120s (line 12) |
| `CHANGELOG.md` | Entry for #494 |
| `CONTRIBUTORS.md` | Added Rui Hu |
### Acceptance Criteria Verified (10/10)
**Success Criteria:**
- `plan use` + `plan execute` generates decisions during Strategize
- `plan tree` displays decision tree correctly (via CLI invocation)
- `plan explain` shows full decision context (via CLI invocation)
- `invariant add --project` / `invariant list --project` work with project scope
- `plan correct --dry-run` performs impact analysis (via CLI invocation)
- `plan correct --mode=revert` executes live correction (via CLI invocation)
**Technical Criteria:**
- Decisions recorded with full context snapshot
- Decision tree persists to database and renders correctly
- Correction in revert mode re-executes from decision point
- Invariants enforced during strategize
### Quality Gates (Full Nox Suite)
| Session | Result | Details |
|---------|--------|---------|
| `lint` | PASS | ruff clean |
| `format` | PASS | ruff format clean |
| `typecheck` | PASS | pyright 0 errors, 0 warnings |
| `security_scan` | PASS | bandit clean |
| `dead_code` | PASS | vulture clean |
| `unit_tests` | PASS | 8500 BDD scenarios, 0 failures |
| `integration_tests` | PASS | 1194/1194 tests, 0 failures |
| `docs` | PASS | mkdocs build clean |
| `build` | PASS | wheel built |
| `benchmark` | PASS | performance benchmarks pass |
| `coverage_report` | PASS | 96.96% → rounds to 97% (meets `--fail-under=97`) |
Closes #494
Reviewed-on: cleveragents/cleveragents-core#559
Reviewed-by: Luis Mendes <luis.mendes@cleverthis.com>
Co-authored-by: Rui Hu <rui.hu@cleverthis.com>
Co-committed-by: Rui Hu <rui.hu@cleverthis.com>
198 lines
8.4 KiB
Plaintext
198 lines
8.4 KiB
Plaintext
*** Settings ***
|
|
Documentation Actor-first context management tests for CleverAgents CLI
|
|
Library Process
|
|
Library OperatingSystem
|
|
Library String
|
|
Library DateTime
|
|
Resource ${CURDIR}/common.resource
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
|
|
*** Variables ***
|
|
${TEST_PROJECT_DIR} ${TEMPDIR}/test_project_${EMPTY}
|
|
${UNIQUE_ID} ${EMPTY}
|
|
|
|
*** Test Cases ***
|
|
Test Context Commands With Actor
|
|
[Documentation] Verify context commands work with actor-first approach
|
|
|
|
# Initialize project first
|
|
Create Directory ${TEST_PROJECT_DIR}
|
|
${result} = Run Process ${PYTHON} -m cleveragents init test-project
|
|
... cwd=${TEST_PROJECT_DIR}
|
|
Log Init stdout: ${result.stdout}
|
|
Log Init stderr: ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
# Create test files
|
|
Create Directory ${TEST_PROJECT_DIR}/src
|
|
Create File ${TEST_PROJECT_DIR}/src/main.py print("Hello World")
|
|
|
|
# Load context with actor (simulating with environment variable)
|
|
Set Environment Variable CLEVERAGENTS_TESTING_USE_MOCK_AI true
|
|
Set Environment Variable CLEVERAGENTS_DEFAULT_ACTOR openai/gpt-4
|
|
|
|
${result} = Run Process ${PYTHON} -m cleveragents context-load src/
|
|
... cwd=${TEST_PROJECT_DIR}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
# List contexts
|
|
${result} = Run Process ${PYTHON} -m cleveragents context list
|
|
... cwd=${TEST_PROJECT_DIR}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} main.py
|
|
|
|
Test Plan Creation With Actor
|
|
[Documentation] Test plan creation using actor instead of provider/model
|
|
|
|
# Initialize project
|
|
Create Directory ${TEST_PROJECT_DIR}_plan
|
|
${result} = Run Process ${PYTHON} -m cleveragents init test-plan-project
|
|
... cwd=${TEST_PROJECT_DIR}_plan
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
# Create plan with actor
|
|
Set Environment Variable CLEVERAGENTS_TESTING_USE_MOCK_AI true
|
|
Set Environment Variable CLEVERAGENTS_DEFAULT_ACTOR anthropic/claude-3
|
|
|
|
${result} = Run Process ${PYTHON} -m cleveragents tell Create a hello world function
|
|
... cwd=${TEST_PROJECT_DIR}_plan env:CLEVERAGENTS_TESTING_USE_MOCK_AI=true
|
|
Log Tell stdout: ${result.stdout}
|
|
Log Tell stderr: ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} Plan
|
|
|
|
Test Actor-Based Workflow
|
|
[Documentation] Test complete workflow with actor configuration
|
|
|
|
# Initialize
|
|
${project_dir} = Set Variable ${TEST_PROJECT_DIR}_workflow
|
|
Create Directory ${project_dir}
|
|
${result} = Run Process ${PYTHON} -m cleveragents init workflow-project
|
|
... cwd=${project_dir}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
# Set up actor environment
|
|
Set Environment Variable CLEVERAGENTS_TESTING_USE_MOCK_AI true
|
|
Set Environment Variable CLEVERAGENTS_DEFAULT_ACTOR openai/gpt-4
|
|
|
|
# Add context
|
|
Create File ${project_dir}/test.py def hello():\n${SPACE*4}pass
|
|
${result} = Run Process ${PYTHON} -m cleveragents context-load test.py
|
|
... cwd=${project_dir}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
# Create plan
|
|
${result} = Run Process ${PYTHON} -m cleveragents tell Add docstring to hello function
|
|
... cwd=${project_dir} env:CLEVERAGENTS_TESTING_USE_MOCK_AI=true
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
# Build plan
|
|
${result} = Run Process ${PYTHON} -m cleveragents build
|
|
# Normal duration: ~10-15s. Timeout raised from 30s to 120s for pabot
|
|
# cold-start (16 parallel processes) + Alembic migration overhead.
|
|
... cwd=${project_dir} env:CLEVERAGENTS_TESTING_USE_MOCK_AI=true timeout=120s
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
# Apply changes
|
|
${result} = Run Process ${PYTHON} -m cleveragents apply
|
|
... cwd=${project_dir} env:CLEVERAGENTS_TESTING_USE_MOCK_AI=true
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
Test Multiple Actors In Project
|
|
[Documentation] Test switching between actors in a project
|
|
|
|
${project_dir} = Set Variable ${TEST_PROJECT_DIR}_multi_actor
|
|
|
|
# Initialize
|
|
Create Directory ${project_dir}
|
|
${result} = Run Process ${PYTHON} -m cleveragents init multi-actor-project
|
|
... cwd=${project_dir}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
Set Environment Variable CLEVERAGENTS_TESTING_USE_MOCK_AI true
|
|
|
|
# Create plan with first actor
|
|
Set Environment Variable CLEVERAGENTS_DEFAULT_ACTOR openai/gpt-3.5-turbo
|
|
${result} = Run Process ${PYTHON} -m cleveragents tell Create function A --name plan1
|
|
... cwd=${project_dir} env:CLEVERAGENTS_TESTING_USE_MOCK_AI=true
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
# Create plan with second actor
|
|
Set Environment Variable CLEVERAGENTS_DEFAULT_ACTOR anthropic/claude-3
|
|
${result} = Run Process ${PYTHON} -m cleveragents tell Create function B --name plan2
|
|
... cwd=${project_dir} env:CLEVERAGENTS_TESTING_USE_MOCK_AI=true
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
# List plans
|
|
${result} = Run Process ${PYTHON} -m cleveragents plan list
|
|
... cwd=${project_dir}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} plan1
|
|
Should Contain ${result.stdout} plan2
|
|
|
|
Test Context Clear Command
|
|
[Documentation] Test clearing all contexts
|
|
|
|
${project_dir} = Set Variable ${TEST_PROJECT_DIR}_clear
|
|
|
|
# Initialize and add contexts
|
|
Create Directory ${project_dir}
|
|
${result} = Run Process ${PYTHON} -m cleveragents init clear-project
|
|
... cwd=${project_dir}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
Set Environment Variable CLEVERAGENTS_TESTING_USE_MOCK_AI true
|
|
|
|
# Create a plan first
|
|
${result} = Run Process ${PYTHON} -m cleveragents tell Test clearing contexts
|
|
... cwd=${project_dir} env:CLEVERAGENTS_TESTING_USE_MOCK_AI=true
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
Create File ${project_dir}/file1.py # test file 1
|
|
Create File ${project_dir}/file2.py # test file 2
|
|
|
|
${result} = Run Process ${PYTHON} -m cleveragents context-load file1.py file2.py
|
|
... cwd=${project_dir} env:CLEVERAGENTS_TESTING_USE_MOCK_AI=true
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
# Clear contexts
|
|
${result} = Run Process ${PYTHON} -m cleveragents context clear --yes
|
|
... cwd=${project_dir} env:CLEVERAGENTS_TESTING_USE_MOCK_AI=true
|
|
Log Clear stdout: ${result.stdout}
|
|
Log Clear stderr: ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
# Verify contexts are cleared
|
|
${result} = Run Process ${PYTHON} -m cleveragents context list
|
|
... cwd=${project_dir}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Not Contain ${result.stdout} file1.py
|
|
Should Not Contain ${result.stdout} file2.py
|
|
|
|
*** Keywords ***
|
|
Setup Test Environment
|
|
[Documentation] Create test environment
|
|
# First run the common setup
|
|
common.Setup Test Environment
|
|
|
|
${timestamp} = Get Current Date result_format=%Y%m%d%H%M%S
|
|
${random} = Generate Random String 6 [NUMBERS]
|
|
Set Suite Variable ${UNIQUE_ID} ${timestamp}_${random}
|
|
|
|
# Create temp directory
|
|
${temp} = Evaluate tempfile.mkdtemp() modules=tempfile
|
|
Set Suite Variable ${TEMP} ${temp}
|
|
Create Directory ${TEMP}
|
|
|
|
# Update TEST_PROJECT_DIR with unique ID
|
|
Set Suite Variable ${TEST_PROJECT_DIR} ${TEMP}/test_project_${UNIQUE_ID}
|
|
|
|
Log Test environment created with ID: ${UNIQUE_ID}
|
|
|
|
Cleanup Test Environment
|
|
[Documentation] Clean up test environment
|
|
Run Keyword If '${TEMP}' != '${EMPTY}' Remove Directory ${TEMP} recursive=True
|
|
Remove Environment Variable CLEVERAGENTS_TESTING_USE_MOCK_AI
|
|
Remove Environment Variable CLEVERAGENTS_DEFAULT_ACTOR
|