forked from cleveragents/cleveragents-core
brent.edwards
81c141716b
## Summary
- Refactors WF02 integration helper into focused modules to satisfy maintainability guidelines and keep command responsibilities isolated.
- Moves WF02 artifact assertions onto lifecycle-produced change artifacts and verifies user-facing `plan artifacts` flow with mocked providers.
- Reworks WF02 coverage-gate assertions to run through validation registration/attachment/execution lifecycle and apply-gate outcomes.
## What changed in this fix round (cycle-6)
- **CI must-fix — trusted profile lifecycle integration failure:**
- Fixed failing integration test `WF02 Trusted Profile Auto Runs Strategize And Execute` caused by legacy AutomationProfile fields that no longer exist.
- Updated WF02 trusted-profile assertions from removed legacy fields (`auto_strategize`, `auto_execute`, `auto_apply`) to current spec-aligned fields (`decompose_task`, `create_tool`, `select_tool`).
- Key module: `robot/wf02_test_generation_commands.py` (`wf02_trusted_lifecycle`).
## Prior fix rounds retained
- **P1 must-fix — validation payload assertions strengthened:**
- Added explicit assertions for validation payload correctness in both failing and passing branches.
- Assertions now verify:
- validation identity (`local/auth-coverage` on `local/api-service-repo`)
- required-mode semantics (`mode=required`, required pass/fail counters, `all_required_passed`)
- `passed` boolean for each branch
- measured-vs-target semantics (`measured_coverage` compared against `target`).
- Key module: `robot/wf02_test_generation_validation.py`.
- **P2 should-fix — artifact membership assertions tightened:**
- Artifact assertions now enforce exact expected membership and count instead of permissive subset checks.
- Persisted changeset assertions were aligned to the same strict set/count checks.
- Key module: `robot/wf02_test_generation_artifacts.py`.
- **P2 should-fix — removed unused helper:**
- Deleted unused `_bind_changeset_to_plan` helper to eliminate dead code.
- Key module: `robot/wf02_test_generation_artifacts.py`.
- **P2 should-fix — concrete typing:**
- Replaced `_setup_validation_db` return type from `tuple[Any, Any]` to concrete typed return (`Callable[[], _NoCloseSession]`, `Session`).
- Key module: `robot/wf02_test_generation_validation.py`.
## Quality gates
- `nox -e lint` ✅
- `nox -e typecheck` ✅
- `nox -e unit_tests` ✅
- `nox -e integration_tests` ✅
- `nox -e e2e_tests` ✅
- `nox -e coverage_report` ✅ (coverage: **98.74%**)
## Scope / limitations
- No deferred items identified in cycle-6.
Closes #766
Reviewed-on: cleveragents/cleveragents-core#1176
Reviewed-by: Jeffrey Phillips Freeman <jeffrey.freeman@cleverthis.com>
Co-authored-by: Brent Edwards <brent.edwards@cleverthis.com>
Co-committed-by: Brent Edwards <brent.edwards@cleverthis.com>
57 lines
3.0 KiB
Plaintext
57 lines
3.0 KiB
Plaintext
*** Settings ***
|
|
Documentation Integration test for Workflow Example 2: automated test generation
|
|
... for a module using the trusted automation profile.
|
|
...
|
|
... Validates trusted-profile lifecycle behavior, mocked LLM-based
|
|
... deterministic test generation, coverage validation enforcement,
|
|
... and invariant-safe artifacts (test-only file changes).
|
|
Resource ${CURDIR}/common.resource
|
|
Suite Setup Setup Test Environment With Database Isolation
|
|
Suite Teardown Cleanup Test Environment
|
|
Force Tags wf02 integration trusted workflow2 test-generation
|
|
|
|
*** Variables ***
|
|
${HELPER} ${CURDIR}/helper_wf02_test_generation.py
|
|
|
|
*** Test Cases ***
|
|
WF02 Trusted Profile Auto Runs Strategize And Execute
|
|
[Documentation] Verify trusted profile auto-runs strategize and execute
|
|
... while not auto-applying (human approval gate retained).
|
|
[Tags] lifecycle automation-profile trusted
|
|
${result}= Run Process ${PYTHON} ${HELPER} trusted-lifecycle cwd=${WORKSPACE} timeout=120s on_timeout=kill
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} wf02-trusted-lifecycle-ok
|
|
|
|
WF02 Mocked Generation Produces Test Artifacts Only
|
|
[Documentation] Verify mocked LLM responses generate deterministic test
|
|
... files, assert user-facing plan artifacts command path,
|
|
... and artifacts list only test paths (no prod edits).
|
|
[Tags] mocking artifacts invariants
|
|
${result}= Run Process ${PYTHON} ${HELPER} mocked-generation-artifacts cwd=${WORKSPACE} timeout=120s on_timeout=kill
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} wf02-mocked-generation-artifacts-ok
|
|
|
|
WF02 Coverage Validation Enforced For Target Threshold
|
|
[Documentation] Verify required coverage validation blocks below-threshold
|
|
... runs and passes after deterministic generated tests.
|
|
[Tags] validation coverage
|
|
${result}= Run Process ${PYTHON} ${HELPER} coverage-validation cwd=${WORKSPACE} timeout=120s on_timeout=kill
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} wf02-coverage-validation-ok
|
|
|
|
WF02 Path Safety Guardrails Reject Unsafe Destinations
|
|
[Documentation] Verify generated-file path guardrails reject absolute,
|
|
... traversal, and non-tests destinations.
|
|
[Tags] guardrails path-safety
|
|
${result}= Run Process ${PYTHON} ${HELPER} path-safety-guardrails cwd=${WORKSPACE} timeout=120s on_timeout=kill
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} wf02-path-safety-guardrails-ok
|