Files
cleveragents-core/robot/wf02_test_generation_commands.py
brent.edwards 81c141716b
CI / build (push) Successful in 22s
CI / helm (push) Successful in 22s
CI / lint (push) Successful in 3m20s
CI / quality (push) Successful in 3m46s
CI / typecheck (push) Successful in 3m56s
CI / security (push) Successful in 4m11s
CI / unit_tests (push) Successful in 8m55s
CI / docker (push) Successful in 1m35s
CI / coverage (push) Successful in 12m29s
CI / e2e_tests (push) Successful in 18m50s
CI / integration_tests (push) Successful in 24m10s
CI / status-check (push) Successful in 1s
CI / benchmark-publish (push) Has been cancelled
CI / benchmark-regression (push) Has been cancelled
test(integration): workflow example 2 — automated test generation for a module (trusted profile) (#1176)
## Summary
- Refactors WF02 integration helper into focused modules to satisfy maintainability guidelines and keep command responsibilities isolated.
- Moves WF02 artifact assertions onto lifecycle-produced change artifacts and verifies user-facing `plan artifacts` flow with mocked providers.
- Reworks WF02 coverage-gate assertions to run through validation registration/attachment/execution lifecycle and apply-gate outcomes.

## What changed in this fix round (cycle-6)
- **CI must-fix — trusted profile lifecycle integration failure:**
  - Fixed failing integration test `WF02 Trusted Profile Auto Runs Strategize And Execute` caused by legacy AutomationProfile fields that no longer exist.
  - Updated WF02 trusted-profile assertions from removed legacy fields (`auto_strategize`, `auto_execute`, `auto_apply`) to current spec-aligned fields (`decompose_task`, `create_tool`, `select_tool`).
  - Key module: `robot/wf02_test_generation_commands.py` (`wf02_trusted_lifecycle`).

## Prior fix rounds retained
- **P1 must-fix — validation payload assertions strengthened:**
  - Added explicit assertions for validation payload correctness in both failing and passing branches.
  - Assertions now verify:
    - validation identity (`local/auth-coverage` on `local/api-service-repo`)
    - required-mode semantics (`mode=required`, required pass/fail counters, `all_required_passed`)
    - `passed` boolean for each branch
    - measured-vs-target semantics (`measured_coverage` compared against `target`).
  - Key module: `robot/wf02_test_generation_validation.py`.

- **P2 should-fix — artifact membership assertions tightened:**
  - Artifact assertions now enforce exact expected membership and count instead of permissive subset checks.
  - Persisted changeset assertions were aligned to the same strict set/count checks.
  - Key module: `robot/wf02_test_generation_artifacts.py`.

- **P2 should-fix — removed unused helper:**
  - Deleted unused `_bind_changeset_to_plan` helper to eliminate dead code.
  - Key module: `robot/wf02_test_generation_artifacts.py`.

- **P2 should-fix — concrete typing:**
  - Replaced `_setup_validation_db` return type from `tuple[Any, Any]` to concrete typed return (`Callable[[], _NoCloseSession]`, `Session`).
  - Key module: `robot/wf02_test_generation_validation.py`.

## Quality gates
- `nox -e lint` 
- `nox -e typecheck` 
- `nox -e unit_tests` 
- `nox -e integration_tests` 
- `nox -e e2e_tests` 
- `nox -e coverage_report`  (coverage: **98.74%**)

## Scope / limitations
- No deferred items identified in cycle-6.

Closes #766

Reviewed-on: #1176
Reviewed-by: Jeffrey Phillips Freeman <jeffrey.freeman@cleverthis.com>
Co-authored-by: Brent Edwards <brent.edwards@cleverthis.com>
Co-committed-by: Brent Edwards <brent.edwards@cleverthis.com>
2026-04-01 03:30:00 +00:00

78 lines
2.7 KiB
Python

"""WF02 command entry points used by helper dispatcher."""
from __future__ import annotations
from collections.abc import Callable
from wf02_test_generation_common import (
auth_module_fixture,
create_trusted_plan,
create_wf02_action,
setup_lifecycle,
validated_relative_test_path,
)
from wf02_test_generation_lifecycle import assert_rejected_generated_path
from cleveragents.domain.models.core.automation_profile import BUILTIN_PROFILES
from cleveragents.domain.models.core.plan import PlanPhase, ProcessingState
def wf02_trusted_lifecycle() -> None:
"""Verify trusted profile auto-runs strategize+execute only."""
lifecycle = setup_lifecycle()
create_wf02_action(lifecycle)
plan_id = create_trusted_plan(lifecycle)
trusted_profile = BUILTIN_PROFILES["trusted"]
# Spec-aligned task-threshold fields (legacy phase names were removed).
assert trusted_profile.decompose_task == 0.0
assert trusted_profile.create_tool == 0.0
assert trusted_profile.select_tool == 1.0
plan = lifecycle.try_auto_run(plan_id)
assert plan.phase == PlanPhase.EXECUTE, (
f"Expected execute phase for trusted profile, got {plan.phase.value}"
)
assert plan.state == ProcessingState.COMPLETE, (
"Expected execute/complete after trusted auto-run"
)
assert plan.automation_profile is not None
assert plan.automation_profile.profile_name == "trusted"
print("wf02-trusted-lifecycle-ok")
def wf02_path_safety_guardrails() -> None:
"""Verify generated file guardrails reject unsafe output paths."""
with auth_module_fixture() as (fixture_root, _):
assert_rejected_generated_path(
lambda rel: validated_relative_test_path(fixture_root, rel),
"/tmp/test_abs.py",
"must be relative",
)
assert_rejected_generated_path(
lambda rel: validated_relative_test_path(fixture_root, rel),
"../tests/test_escape.py",
"escapes fixture root",
)
assert_rejected_generated_path(
lambda rel: validated_relative_test_path(fixture_root, rel),
"src/test_not_allowed.py",
"must stay in tests/",
)
print("wf02-path-safety-guardrails-ok")
def command_map(
mocked_generation_artifacts: Callable[[], None],
coverage_validation: Callable[[], None],
) -> dict[str, Callable[[], None]]:
"""Build command map for helper dispatcher."""
return {
"trusted-lifecycle": wf02_trusted_lifecycle,
"mocked-generation-artifacts": mocked_generation_artifacts,
"coverage-validation": coverage_validation,
"path-safety-guardrails": wf02_path_safety_guardrails,
}