007af498b8
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 17s
CI / helm (pull_request) Successful in 33s
CI / lint (pull_request) Successful in 3m42s
CI / security (pull_request) Successful in 4m8s
CI / quality (pull_request) Successful in 4m9s
CI / typecheck (pull_request) Successful in 4m20s
CI / integration_tests (pull_request) Successful in 7m2s
CI / unit_tests (pull_request) Successful in 7m55s
CI / docker (pull_request) Successful in 1m19s
CI / coverage (pull_request) Successful in 8m46s
CI / e2e_tests (pull_request) Successful in 16m1s
CI / status-check (pull_request) Successful in 1s
CI / build (push) Successful in 17s
CI / helm (push) Successful in 22s
CI / quality (push) Successful in 31s
CI / lint (push) Successful in 3m28s
CI / typecheck (push) Successful in 3m54s
CI / benchmark-regression (push) Has been skipped
CI / security (push) Successful in 4m5s
CI / integration_tests (push) Successful in 6m13s
CI / unit_tests (push) Successful in 6m28s
CI / docker (push) Successful in 1m34s
CI / coverage (push) Successful in 12m5s
CI / e2e_tests (push) Successful in 18m47s
CI / status-check (push) Successful in 1s
CI / benchmark-publish (push) Successful in 28m0s
CI / benchmark-regression (pull_request) Successful in 59m48s
Renamed all 11 task-type confidence threshold fields in AutomationProfile from phase-transition semantics to spec-defined task-type semantics. Updated all 8 built-in profiles, CLI formatting, YAML schema, services, and all Behave/Robot tests referencing the old field names. Post-review fixes: - Fixed 24 stale old field names in M6 fixture files (automation_profiles.json, autonomy_guardrails.json) - Added model_validator(mode='before') to detect legacy field names and raise actionable ValueError with rename mapping - Added semantic bridge comments in PlanLifecycleService mapping task-type thresholds to phase-transition gates - Added threshold_field to structured log messages for observability - Restored categorised CLI automation-profile show output to match spec (Phase Transitions / Decision Automation / Self-Repair / Execution Controls) instead of flat list - Added missing access_network field to spec show output examples (Rich, Plain, JSON, YAML variants) - Aligned ADR-017 profile fields table to all 11 fields with descriptions matching spec Automatable Tasks table - Aligned automation_profiles.md threshold descriptions with spec - Added spec section references in phase_reversion.md, error_recovery.md, and plan_execute.md for field naming context - Extended repository roundtrip test to assert all 11 threshold fields - Fixed benchmark _make_profile() passing safety fields as top-level kwargs instead of via SafetyProfile sub-model (incompatible with extra="forbid") - Aligned CLI JSON/YAML output structure for automation-profile show with the specification grouped format (phase_transitions, decision_automation, self_repair, execution_controls) - Moved safety boolean fields into the Execution Controls section of Rich output per spec examples - Reverted auto profile description to "Fully automatic except apply" per specification (line 16703, line 28406) - Improved bridge comments in test steps with semantic context for threshold-to-gate mappings ISSUES CLOSED: #902
231 lines
8.1 KiB
Python
231 lines
8.1 KiB
Python
"""Step definitions for AutomationProfileService tests."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
|
|
from behave import given, then, when
|
|
from behave.runner import Context
|
|
|
|
from cleveragents.application.services.automation_profile_service import (
|
|
AutomationProfileService,
|
|
)
|
|
from cleveragents.core.exceptions import (
|
|
NotFoundError,
|
|
)
|
|
|
|
# -------------------------------------------------------------------
|
|
# Helpers
|
|
# -------------------------------------------------------------------
|
|
|
|
_ENV_VAR = "CLEVERAGENTS_AUTOMATION_PROFILE"
|
|
|
|
|
|
def _cleanup_env(context: Context) -> None:
|
|
"""Remove the env var if it was set during a scenario."""
|
|
os.environ.pop(_ENV_VAR, None)
|
|
|
|
|
|
# -------------------------------------------------------------------
|
|
# Given steps
|
|
# -------------------------------------------------------------------
|
|
|
|
|
|
@given('an automation profile service with global default "{default}"')
|
|
def step_given_service_with_default(context: Context, default: str) -> None:
|
|
"""Create a service with an explicit global default."""
|
|
context.profile_service = AutomationProfileService(
|
|
repo=None,
|
|
global_default=default,
|
|
)
|
|
context.env_vars_to_clean = getattr(context, "env_vars_to_clean", [])
|
|
|
|
|
|
@given("an automation profile service with no configuration")
|
|
def step_given_service_no_config(context: Context) -> None:
|
|
"""Create a service with no explicit config."""
|
|
os.environ.pop(_ENV_VAR, None)
|
|
context.profile_service = AutomationProfileService(
|
|
repo=None,
|
|
global_default=None,
|
|
)
|
|
context.env_vars_to_clean = getattr(context, "env_vars_to_clean", [])
|
|
|
|
|
|
@given('the env var CLEVERAGENTS_AUTOMATION_PROFILE is set to "{value}"')
|
|
def step_given_env_var_set(context: Context, value: str) -> None:
|
|
"""Set the CLEVERAGENTS_AUTOMATION_PROFILE env var."""
|
|
os.environ[_ENV_VAR] = value
|
|
context.env_vars_to_clean = getattr(context, "env_vars_to_clean", [])
|
|
context.env_vars_to_clean.append(_ENV_VAR)
|
|
|
|
|
|
# -------------------------------------------------------------------
|
|
# When steps: resolve
|
|
# -------------------------------------------------------------------
|
|
|
|
|
|
def _none_or_str(val: str) -> str | None:
|
|
"""Convert 'None' string to actual None."""
|
|
if val == "None":
|
|
return None
|
|
return val
|
|
|
|
|
|
@when("I resolve profile with plan {plan_p} action {action_p} project {project_p}")
|
|
def step_when_resolve_profile(
|
|
context: Context,
|
|
plan_p: str,
|
|
action_p: str,
|
|
project_p: str,
|
|
) -> None:
|
|
"""Resolve a profile with given precedence values."""
|
|
plan_val = _none_or_str(plan_p.strip('"'))
|
|
action_val = _none_or_str(action_p.strip('"'))
|
|
project_val = _none_or_str(project_p.strip('"'))
|
|
context.resolved_profile = context.profile_service.resolve_profile(
|
|
plan_profile=plan_val,
|
|
action_profile=action_val,
|
|
project_profile=project_val,
|
|
)
|
|
|
|
|
|
@when("I try to resolve profile with all None")
|
|
def step_when_try_resolve_all_none(
|
|
context: Context,
|
|
) -> None:
|
|
"""Try to resolve with all None (may fail)."""
|
|
context.profile_not_found_error = None
|
|
try:
|
|
context.resolved_profile = context.profile_service.resolve_profile()
|
|
except NotFoundError as exc:
|
|
context.profile_not_found_error = exc
|
|
|
|
|
|
# -------------------------------------------------------------------
|
|
# When steps: get_profile
|
|
# -------------------------------------------------------------------
|
|
|
|
|
|
@when('I get profile "{name}"')
|
|
def step_when_get_profile(context: Context, name: str) -> None:
|
|
"""Get a profile by name."""
|
|
context.retrieved_profile = context.profile_service.get_profile(name)
|
|
|
|
|
|
@when('I try to get profile "{name}"')
|
|
def step_when_try_get_profile(context: Context, name: str) -> None:
|
|
"""Try to get a profile that might not exist."""
|
|
context.profile_not_found_error = None
|
|
try:
|
|
context.profile_service.get_profile(name)
|
|
except NotFoundError as exc:
|
|
context.profile_not_found_error = exc
|
|
|
|
|
|
# -------------------------------------------------------------------
|
|
# When steps: list
|
|
# -------------------------------------------------------------------
|
|
|
|
|
|
@when("I list all profiles")
|
|
def step_when_list_profiles(context: Context) -> None:
|
|
"""List all profiles."""
|
|
context.profile_list = context.profile_service.list_profiles()
|
|
|
|
|
|
# -------------------------------------------------------------------
|
|
# Then steps
|
|
# -------------------------------------------------------------------
|
|
|
|
|
|
@then('the resolved profile name should be "{expected}"')
|
|
def step_then_resolved_name(context: Context, expected: str) -> None:
|
|
"""Check resolved profile name."""
|
|
actual = context.resolved_profile.name
|
|
assert actual == expected, f"Expected '{expected}', got '{actual}'"
|
|
|
|
|
|
@then("a profile not found error should be raised")
|
|
def step_then_not_found_error(context: Context) -> None:
|
|
"""Verify NotFoundError was raised."""
|
|
assert context.profile_not_found_error is not None, "Expected NotFoundError"
|
|
|
|
|
|
@then('the profile service error should mention "{text}"')
|
|
def step_then_service_error_mentions(context: Context, text: str) -> None:
|
|
"""Check error message contains text."""
|
|
err = str(context.profile_not_found_error)
|
|
assert text in err, f"Expected '{text}' in error, got: {err}"
|
|
|
|
|
|
@then('the mapped profile name should be "{expected}"')
|
|
def step_then_mapped_name(context: Context, expected: str) -> None:
|
|
"""Check mapped profile name."""
|
|
actual = context.mapped_profile_name
|
|
assert actual == expected, f"Expected '{expected}', got '{actual}'"
|
|
|
|
|
|
@then("a legacy mapping validation error should be raised")
|
|
def step_then_legacy_validation_error(
|
|
context: Context,
|
|
) -> None:
|
|
"""Verify ValidationError was raised."""
|
|
assert context.legacy_validation_error is not None, "Expected ValidationError"
|
|
|
|
|
|
@then('the validation error should mention "{text}"')
|
|
def step_then_validation_error_mentions(context: Context, text: str) -> None:
|
|
"""Check validation error message."""
|
|
err = str(context.legacy_validation_error)
|
|
assert text in err, f"Expected '{text}' in error, got: {err}"
|
|
|
|
|
|
@then("the resolved profile should have select_tool {expected:g}")
|
|
def step_then_resolved_select_tool(context: Context, expected: float) -> None:
|
|
"""Check resolved profile select_tool."""
|
|
actual = context.resolved_profile.select_tool
|
|
assert actual == expected, f"Expected select_tool {expected}, got {actual}"
|
|
|
|
|
|
@then('the profile list should include "{name}"')
|
|
def step_then_list_includes(context: Context, name: str) -> None:
|
|
"""Check profile list contains a profile name."""
|
|
names = [p.name for p in context.profile_list]
|
|
assert name in names, f"Expected '{name}' in list, got: {names}"
|
|
|
|
|
|
@then("the profile list should have at least {count:d} entries")
|
|
def step_then_list_min_count(context: Context, count: int) -> None:
|
|
"""Check profile list has minimum entries."""
|
|
actual = len(context.profile_list)
|
|
assert actual >= count, f"Expected >= {count}, got {actual}"
|
|
|
|
|
|
@then('the retrieved profile name should be "{expected}"')
|
|
def step_then_retrieved_name(context: Context, expected: str) -> None:
|
|
"""Check retrieved profile name."""
|
|
actual = context.retrieved_profile.name
|
|
assert actual == expected, f"Expected '{expected}', got '{actual}'"
|
|
|
|
|
|
@then("the retrieved profile select_tool should be {expected:g}")
|
|
def step_then_retrieved_select_tool(context: Context, expected: float) -> None:
|
|
"""Check retrieved profile select_tool."""
|
|
actual = context.retrieved_profile.select_tool
|
|
assert actual == expected, f"Expected {expected}, got {actual}"
|
|
|
|
|
|
@then("the retrieved profile decompose_task should be {expected:g}")
|
|
def step_then_retrieved_decompose_task(context: Context, expected: float) -> None:
|
|
"""Check retrieved profile decompose_task."""
|
|
actual = context.retrieved_profile.decompose_task
|
|
assert actual == expected, f"Expected {expected}, got {actual}"
|
|
|
|
|
|
@then("the retrieved profile create_tool should be {expected:g}")
|
|
def step_then_retrieved_create_tool(context: Context, expected: float) -> None:
|
|
"""Check retrieved profile create_tool."""
|
|
actual = context.retrieved_profile.create_tool
|
|
assert actual == expected, f"Expected {expected}, got {actual}"
|