brent.edwards
81c141716b
CI / build (push) Successful in 22s
CI / helm (push) Successful in 22s
CI / lint (push) Successful in 3m20s
CI / quality (push) Successful in 3m46s
CI / typecheck (push) Successful in 3m56s
CI / security (push) Successful in 4m11s
CI / unit_tests (push) Successful in 8m55s
CI / docker (push) Successful in 1m35s
CI / coverage (push) Successful in 12m29s
CI / e2e_tests (push) Successful in 18m50s
CI / integration_tests (push) Successful in 24m10s
CI / status-check (push) Successful in 1s
CI / benchmark-publish (push) Has been cancelled
CI / benchmark-regression (push) Has been cancelled
## Summary
- Refactors WF02 integration helper into focused modules to satisfy maintainability guidelines and keep command responsibilities isolated.
- Moves WF02 artifact assertions onto lifecycle-produced change artifacts and verifies user-facing `plan artifacts` flow with mocked providers.
- Reworks WF02 coverage-gate assertions to run through validation registration/attachment/execution lifecycle and apply-gate outcomes.
## What changed in this fix round (cycle-6)
- **CI must-fix — trusted profile lifecycle integration failure:**
- Fixed failing integration test `WF02 Trusted Profile Auto Runs Strategize And Execute` caused by legacy AutomationProfile fields that no longer exist.
- Updated WF02 trusted-profile assertions from removed legacy fields (`auto_strategize`, `auto_execute`, `auto_apply`) to current spec-aligned fields (`decompose_task`, `create_tool`, `select_tool`).
- Key module: `robot/wf02_test_generation_commands.py` (`wf02_trusted_lifecycle`).
## Prior fix rounds retained
- **P1 must-fix — validation payload assertions strengthened:**
- Added explicit assertions for validation payload correctness in both failing and passing branches.
- Assertions now verify:
- validation identity (`local/auth-coverage` on `local/api-service-repo`)
- required-mode semantics (`mode=required`, required pass/fail counters, `all_required_passed`)
- `passed` boolean for each branch
- measured-vs-target semantics (`measured_coverage` compared against `target`).
- Key module: `robot/wf02_test_generation_validation.py`.
- **P2 should-fix — artifact membership assertions tightened:**
- Artifact assertions now enforce exact expected membership and count instead of permissive subset checks.
- Persisted changeset assertions were aligned to the same strict set/count checks.
- Key module: `robot/wf02_test_generation_artifacts.py`.
- **P2 should-fix — removed unused helper:**
- Deleted unused `_bind_changeset_to_plan` helper to eliminate dead code.
- Key module: `robot/wf02_test_generation_artifacts.py`.
- **P2 should-fix — concrete typing:**
- Replaced `_setup_validation_db` return type from `tuple[Any, Any]` to concrete typed return (`Callable[[], _NoCloseSession]`, `Session`).
- Key module: `robot/wf02_test_generation_validation.py`.
## Quality gates
- `nox -e lint` ✅
- `nox -e typecheck` ✅
- `nox -e unit_tests` ✅
- `nox -e integration_tests` ✅
- `nox -e e2e_tests` ✅
- `nox -e coverage_report` ✅ (coverage: **98.74%**)
## Scope / limitations
- No deferred items identified in cycle-6.
Closes #766
Reviewed-on: #1176
Reviewed-by: Jeffrey Phillips Freeman <jeffrey.freeman@cleverthis.com>
Co-authored-by: Brent Edwards <brent.edwards@cleverthis.com>
Co-committed-by: Brent Edwards <brent.edwards@cleverthis.com>
70 lines
3.6 KiB
Gherkin
70 lines
3.6 KiB
Gherkin
@wf02 @test_generation @trusted
|
|
Feature: WF02 automated test generation for a module (trusted profile)
|
|
As a developer using the trusted automation profile
|
|
I want the system to generate tests for an under-covered module
|
|
So that coverage improves without modifying production source code
|
|
|
|
# --------------------------------------------------------------------------
|
|
# AC1: Trusted profile lifecycle gating (auto-run strategize + execute,
|
|
# human gate on apply)
|
|
# --------------------------------------------------------------------------
|
|
|
|
Scenario: Trusted profile auto-runs strategize and execute phases
|
|
Given a WF02 lifecycle service with mock providers
|
|
And the WF02 action "local/generate-tests" is registered
|
|
And a trusted-profile plan for module "src/auth" with coverage target 80
|
|
When I auto-run the WF02 plan
|
|
Then the plan should be in the execute phase
|
|
And the plan processing state should be complete
|
|
And the plan automation profile should be "trusted"
|
|
|
|
# --------------------------------------------------------------------------
|
|
# AC2: Action definition carries correct invariants
|
|
# --------------------------------------------------------------------------
|
|
|
|
Scenario: WF02 action carries production-code and naming invariants
|
|
Given a WF02 lifecycle service with mock providers
|
|
And the WF02 action "local/generate-tests" is registered
|
|
And a trusted-profile plan for module "src/auth" with coverage target 80
|
|
Then the plan should carry an invariant containing "Do not modify any production code"
|
|
And the plan should carry an invariant containing "test file naming convention"
|
|
And the plan should carry an invariant containing "test fixtures and conftest.py patterns"
|
|
|
|
# --------------------------------------------------------------------------
|
|
# AC3: Path safety guardrails reject unsafe output destinations
|
|
# --------------------------------------------------------------------------
|
|
|
|
Scenario: Path guardrail rejects absolute paths
|
|
Given a temporary WF02 fixture repository
|
|
When I validate the generated path "/tmp/test_abs.py"
|
|
Then the path should be rejected with reason "must be relative"
|
|
|
|
Scenario: Path guardrail rejects traversal paths
|
|
Given a temporary WF02 fixture repository
|
|
When I validate the generated path "../tests/test_escape.py"
|
|
Then the path should be rejected with reason "escapes fixture root"
|
|
|
|
Scenario: Path guardrail rejects non-test destinations
|
|
Given a temporary WF02 fixture repository
|
|
When I validate the generated path "src/test_not_allowed.py"
|
|
Then the path should be rejected with reason "must stay in tests/"
|
|
|
|
Scenario: Path guardrail accepts valid test paths
|
|
Given a temporary WF02 fixture repository
|
|
When I validate the generated path "tests/test_auth_new.py"
|
|
Then the path should be accepted
|
|
|
|
# --------------------------------------------------------------------------
|
|
# AC4: Generated artifacts follow naming and scope conventions
|
|
# --------------------------------------------------------------------------
|
|
|
|
Scenario: Generated test files follow project conventions
|
|
Given a WF02 lifecycle service with mock providers
|
|
And the WF02 action "local/generate-tests" is registered
|
|
And a trusted-profile plan for module "src/auth" with coverage target 80
|
|
When I generate tests from the mock provider with coverage suite
|
|
Then all generated file paths should start with "tests/"
|
|
And all generated file paths should end with ".py"
|
|
And all generated file names should start with "test_"
|
|
And no generated file path should start with "src/"
|