feat(estimation): wire actor.default.estimation config fallback and Strategize-to-Estimate lifecycle hook (#1310)
CI / build (push) Has been cancelled
CI / quality (push) Has been cancelled
CI / e2e_tests (push) Has been cancelled
CI / benchmark-publish (push) Has been cancelled
CI / security (push) Has been cancelled
CI / integration_tests (push) Has been cancelled
CI / typecheck (push) Has been cancelled
CI / unit_tests (push) Has been cancelled
CI / helm (push) Has been cancelled
CI / lint (push) Has been cancelled
CI / coverage (push) Has been cancelled
CI / docker (push) Has been cancelled
CI / status-check (push) Has been cancelled
CI / benchmark-regression (push) Has been cancelled
CI / build (push) Has been cancelled
CI / quality (push) Has been cancelled
CI / e2e_tests (push) Has been cancelled
CI / benchmark-publish (push) Has been cancelled
CI / security (push) Has been cancelled
CI / integration_tests (push) Has been cancelled
CI / typecheck (push) Has been cancelled
CI / unit_tests (push) Has been cancelled
CI / helm (push) Has been cancelled
CI / lint (push) Has been cancelled
CI / coverage (push) Has been cancelled
CI / docker (push) Has been cancelled
CI / status-check (push) Has been cancelled
CI / benchmark-regression (push) Has been cancelled
Co-authored-by: Jeffrey Phillips Freeman <the@jeffreyfreeman.me> Co-committed-by: Jeffrey Phillips Freeman <the@jeffreyfreeman.me>
This commit was merged in pull request #1310.
This commit is contained in:
@@ -0,0 +1,37 @@
|
||||
"""Add estimation_report_json column to v3_plans.
|
||||
|
||||
Adds the ``estimation_report_json`` column to ``v3_plans`` to store the
|
||||
full estimation report as a JSON-serialisable dict. This column is
|
||||
populated by the estimation actor lifecycle hook introduced in issue #651.
|
||||
|
||||
Revision ID: m6_006_estimation_report_json
|
||||
Revises: m8_002_merge_profile_rename_and_corrections
|
||||
Create Date: 2026-04-02 00:00:00
|
||||
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "m6_006_estimation_report_json"
|
||||
down_revision: str | Sequence[str] | None = (
|
||||
"m8_002_merge_profile_rename_and_corrections"
|
||||
)
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
"""Add estimation_report_json column to v3_plans."""
|
||||
op.add_column(
|
||||
"v3_plans",
|
||||
sa.Column("estimation_report_json", sa.Text(), nullable=True),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
"""Remove estimation_report_json column from v3_plans."""
|
||||
op.drop_column("v3_plans", "estimation_report_json")
|
||||
@@ -0,0 +1,88 @@
|
||||
Feature: Estimation lifecycle hook and actor.default.estimation config fallback
|
||||
As a developer using CleverAgents
|
||||
I want the estimation actor to be resolved via a 4-level fallback chain
|
||||
And I want a PLAN_ESTIMATION_COMPLETE event emitted after estimation
|
||||
And I want cost_estimate_usd populated from the estimation result
|
||||
So that estimation integrates cleanly into the Strategize-to-Execute lifecycle
|
||||
|
||||
# ── Subtask 1: 4-level fallback chain in use_action() ──────────────────
|
||||
|
||||
Scenario: estimation_actor from action YAML takes precedence over global config
|
||||
Given I have a lifecycle service with a global estimation actor config "global/estimator"
|
||||
And I create an action with estimation_actor "action/estimator"
|
||||
When I use the action to create a plan
|
||||
Then the plan estimation_actor should be "action/estimator"
|
||||
|
||||
Scenario: estimation_actor falls back to actor.default.estimation global config when action has none
|
||||
Given I have a lifecycle service with a global estimation actor config "global/estimator"
|
||||
And I create an action without an estimation_actor
|
||||
When I use the action to create a plan
|
||||
Then the plan estimation_actor should be "global/estimator"
|
||||
|
||||
Scenario: estimation_actor is None when neither action nor global config provides one
|
||||
Given I have a lifecycle service without a global estimation actor config
|
||||
And I create an action without an estimation_actor
|
||||
When I use the action to create a plan
|
||||
Then the plan estimation_actor should be None for fallback tests
|
||||
|
||||
Scenario: estimation_actor from action YAML is used even when global config is set
|
||||
Given I have a lifecycle service with a global estimation actor config "global/estimator"
|
||||
And I create an action with estimation_actor "action/specific-estimator"
|
||||
When I use the action to create a plan
|
||||
Then the plan estimation_actor should be "action/specific-estimator"
|
||||
|
||||
# ── Subtask 2: PLAN_ESTIMATION_COMPLETE event ───────────────────────────
|
||||
|
||||
Scenario: PLAN_ESTIMATION_COMPLETE event is emitted when estimation runs
|
||||
Given I have a lifecycle service with an event bus for estimation tests
|
||||
And I create an action with estimation_actor "local/estimator" for event tests
|
||||
When I use the action and drive through strategize to execute
|
||||
Then a PLAN_ESTIMATION_COMPLETE event should have been emitted
|
||||
|
||||
Scenario: No PLAN_ESTIMATION_COMPLETE event when no estimation actor configured
|
||||
Given I have a lifecycle service with an event bus for estimation tests
|
||||
And I create an action without estimation_actor for event tests
|
||||
When I use the action and drive through strategize to execute without estimation
|
||||
Then no PLAN_ESTIMATION_COMPLETE event should have been emitted
|
||||
|
||||
# ── Subtask 3: Conditional estimation step in Strategize-to-Execute ─────
|
||||
|
||||
Scenario: Estimation step runs between Strategize completion and Execute start
|
||||
Given I have a plan lifecycle service for estimation lifecycle tests
|
||||
And I create an action with estimation_actor "local/estimator" for lifecycle tests
|
||||
When I use the action and complete strategize for lifecycle tests
|
||||
And I call execute_plan for lifecycle tests
|
||||
Then the plan should be in Execute phase
|
||||
And the plan estimation_result should be populated
|
||||
|
||||
Scenario: Estimation step is skipped cleanly when no estimation actor is set
|
||||
Given I have a plan lifecycle service for estimation lifecycle tests
|
||||
And I create an action without estimation_actor for lifecycle tests
|
||||
When I use the action and complete strategize for lifecycle tests
|
||||
And I call execute_plan for lifecycle tests
|
||||
Then the plan should be in Execute phase
|
||||
And the plan estimation_result should be None for lifecycle tests
|
||||
|
||||
# ── Subtask 4: cost_estimate_usd populated from estimation result ────────
|
||||
|
||||
Scenario: cost_estimate_usd is populated from estimation result after estimation runs
|
||||
Given I have a plan lifecycle service for cost estimate tests
|
||||
And I create an action with estimation_actor "local/estimator" for cost tests
|
||||
When I use the action and drive through strategize to execute for cost tests
|
||||
Then the plan cost_estimate_usd should be set
|
||||
|
||||
# ── Subtask 5: Clean skip when no estimation actor configured ────────────
|
||||
|
||||
Scenario: Plan transitions to Execute cleanly when no estimation actor is configured
|
||||
Given I have a plan lifecycle service for skip tests
|
||||
And I create an action without estimation_actor for skip tests
|
||||
When I use the action and complete strategize for skip tests
|
||||
And I call execute_plan for skip tests
|
||||
Then the plan should transition to Execute phase cleanly
|
||||
And no estimation result should be set on the plan
|
||||
|
||||
# ── EventType enum has PLAN_ESTIMATION_COMPLETE ──────────────────────────
|
||||
|
||||
Scenario: EventType enum includes PLAN_ESTIMATION_COMPLETE
|
||||
Then the EventType enum should have PLAN_ESTIMATION_COMPLETE
|
||||
And the PLAN_ESTIMATION_COMPLETE value should be "plan.estimation_complete"
|
||||
@@ -0,0 +1,479 @@
|
||||
"""Step definitions for estimation lifecycle hook and config fallback (issue #651).
|
||||
|
||||
Tests the 4-level fallback chain for estimation_actor in use_action(),
|
||||
the PLAN_ESTIMATION_COMPLETE event emission, and the cost_estimate_usd
|
||||
population from the estimation result.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from behave import given, then, when
|
||||
from behave.runner import Context
|
||||
|
||||
from cleveragents.application.services.config_service import ConfigService
|
||||
from cleveragents.application.services.plan_lifecycle_service import (
|
||||
PlanLifecycleService,
|
||||
)
|
||||
from cleveragents.config.settings import Settings
|
||||
from cleveragents.domain.models.core.plan import PlanPhase, ProcessingState
|
||||
from cleveragents.infrastructure.events.types import EventType
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _make_service(
|
||||
config_service: ConfigService | None = None,
|
||||
event_bus: Any | None = None,
|
||||
) -> PlanLifecycleService:
|
||||
"""Create a PlanLifecycleService with optional config_service and event_bus."""
|
||||
settings = Settings()
|
||||
svc = PlanLifecycleService(
|
||||
settings=settings,
|
||||
config_service=config_service,
|
||||
event_bus=event_bus,
|
||||
)
|
||||
return svc
|
||||
|
||||
|
||||
def _make_mock_config_service_with_estimation(actor_name: str) -> MagicMock:
|
||||
"""Create a mock ConfigService that returns *actor_name* for actor.default.estimation."""
|
||||
from cleveragents.application.services.config_service import (
|
||||
ConfigLevel,
|
||||
ResolvedValue,
|
||||
)
|
||||
|
||||
mock_cfg = MagicMock(spec=ConfigService)
|
||||
|
||||
def _resolve(key: str, **kwargs: Any) -> ResolvedValue:
|
||||
if key == "actor.default.estimation":
|
||||
return ResolvedValue(
|
||||
key=key,
|
||||
value=actor_name,
|
||||
source=ConfigLevel.GLOBAL,
|
||||
chain=[],
|
||||
)
|
||||
# For any other key, return a ResolvedValue with None
|
||||
return ResolvedValue(key=key, value=None, source=ConfigLevel.DEFAULT, chain=[])
|
||||
|
||||
mock_cfg.resolve.side_effect = _resolve
|
||||
return mock_cfg # type: ignore[return-value]
|
||||
|
||||
|
||||
def _make_mock_config_service_no_estimation() -> MagicMock:
|
||||
"""Create a mock ConfigService that returns None for actor.default.estimation."""
|
||||
from cleveragents.application.services.config_service import (
|
||||
ConfigLevel,
|
||||
ResolvedValue,
|
||||
)
|
||||
|
||||
mock_cfg = MagicMock(spec=ConfigService)
|
||||
|
||||
def _resolve(key: str, **kwargs: Any) -> ResolvedValue:
|
||||
return ResolvedValue(key=key, value=None, source=ConfigLevel.DEFAULT, chain=[])
|
||||
|
||||
mock_cfg.resolve.side_effect = _resolve
|
||||
return mock_cfg # type: ignore[return-value]
|
||||
|
||||
|
||||
def _drive_to_execute(svc: PlanLifecycleService, plan_id: str) -> None:
|
||||
"""Drive a plan from STRATEGIZE/QUEUED through to Execute phase."""
|
||||
svc.start_strategize(plan_id)
|
||||
plan = svc.get_plan(plan_id)
|
||||
plan.processing_state = ProcessingState.PROCESSING
|
||||
svc._commit_plan(plan)
|
||||
svc.complete_strategize(plan_id)
|
||||
# complete_strategize calls auto_progress; if still in STRATEGIZE/COMPLETE
|
||||
# we need to call execute_plan explicitly.
|
||||
plan = svc.get_plan(plan_id)
|
||||
if plan.phase == PlanPhase.STRATEGIZE:
|
||||
svc.execute_plan(plan_id)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Subtask 1: 4-level fallback chain
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@given('I have a lifecycle service with a global estimation actor config "{actor}"')
|
||||
def step_service_with_global_estimation(context: Context, actor: str) -> None:
|
||||
"""Create a lifecycle service with actor.default.estimation set in config."""
|
||||
cfg = _make_mock_config_service_with_estimation(actor)
|
||||
context.fallback_svc = _make_service(config_service=cfg)
|
||||
context.fallback_error = None
|
||||
|
||||
|
||||
@given("I have a lifecycle service without a global estimation actor config")
|
||||
def step_service_without_global_estimation(context: Context) -> None:
|
||||
"""Create a lifecycle service with no actor.default.estimation config."""
|
||||
cfg = _make_mock_config_service_no_estimation()
|
||||
context.fallback_svc = _make_service(config_service=cfg)
|
||||
context.fallback_error = None
|
||||
|
||||
|
||||
@given('I create an action with estimation_actor "{actor}"')
|
||||
def step_create_action_with_estimation_actor(context: Context, actor: str) -> None:
|
||||
"""Create an action with the given estimation_actor."""
|
||||
context.fallback_svc.create_action(
|
||||
name="local/fallback-test",
|
||||
description="Fallback chain test action",
|
||||
definition_of_done="Done",
|
||||
strategy_actor="openai/gpt-4",
|
||||
execution_actor="openai/gpt-4",
|
||||
estimation_actor=actor,
|
||||
)
|
||||
|
||||
|
||||
@given("I create an action without an estimation_actor")
|
||||
def step_create_action_without_estimation_actor(context: Context) -> None:
|
||||
"""Create an action with no estimation_actor."""
|
||||
context.fallback_svc.create_action(
|
||||
name="local/fallback-test",
|
||||
description="Fallback chain test action",
|
||||
definition_of_done="Done",
|
||||
strategy_actor="openai/gpt-4",
|
||||
execution_actor="openai/gpt-4",
|
||||
)
|
||||
|
||||
|
||||
@when("I use the action to create a plan")
|
||||
def step_use_action_create_plan(context: Context) -> None:
|
||||
"""Use the action to create a plan."""
|
||||
context.fallback_plan = context.fallback_svc.use_action(
|
||||
action_name="local/fallback-test"
|
||||
)
|
||||
|
||||
|
||||
@then('the plan estimation_actor should be "{expected}"')
|
||||
def step_check_plan_estimation_actor(context: Context, expected: str) -> None:
|
||||
"""Verify the plan's estimation_actor matches the expected value."""
|
||||
assert context.fallback_plan.estimation_actor == expected, (
|
||||
f"Expected estimation_actor={expected!r}, "
|
||||
f"got {context.fallback_plan.estimation_actor!r}"
|
||||
)
|
||||
|
||||
|
||||
@then("the plan estimation_actor should be None for fallback tests")
|
||||
def step_check_plan_estimation_actor_none(context: Context) -> None:
|
||||
"""Verify the plan's estimation_actor is None."""
|
||||
assert context.fallback_plan.estimation_actor is None, (
|
||||
f"Expected estimation_actor=None, "
|
||||
f"got {context.fallback_plan.estimation_actor!r}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Subtask 2: PLAN_ESTIMATION_COMPLETE event
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@given("I have a lifecycle service with an event bus for estimation tests")
|
||||
def step_service_with_event_bus(context: Context) -> None:
|
||||
"""Create a lifecycle service with a mock event bus."""
|
||||
context.emitted_events = [] # type: ignore[attr-defined]
|
||||
|
||||
mock_bus = MagicMock()
|
||||
|
||||
def _capture_emit(event: Any) -> None:
|
||||
context.emitted_events.append(event)
|
||||
|
||||
mock_bus.emit.side_effect = _capture_emit
|
||||
context.event_svc = _make_service(event_bus=mock_bus)
|
||||
|
||||
|
||||
@given('I create an action with estimation_actor "{actor}" for event tests')
|
||||
def step_create_action_for_event_tests(context: Context, actor: str) -> None:
|
||||
"""Create an action with estimation_actor for event emission tests."""
|
||||
context.event_svc.create_action(
|
||||
name="local/event-test",
|
||||
description="Event test action",
|
||||
definition_of_done="Done",
|
||||
strategy_actor="openai/gpt-4",
|
||||
execution_actor="openai/gpt-4",
|
||||
estimation_actor=actor,
|
||||
)
|
||||
|
||||
|
||||
@given("I create an action without estimation_actor for event tests")
|
||||
def step_create_action_no_estimation_for_event_tests(context: Context) -> None:
|
||||
"""Create an action without estimation_actor for event emission tests."""
|
||||
context.event_svc.create_action(
|
||||
name="local/event-test-no-est",
|
||||
description="Event test action without estimation",
|
||||
definition_of_done="Done",
|
||||
strategy_actor="openai/gpt-4",
|
||||
execution_actor="openai/gpt-4",
|
||||
)
|
||||
|
||||
|
||||
@when("I use the action and drive through strategize to execute")
|
||||
def step_drive_through_strategize_to_execute(context: Context) -> None:
|
||||
"""Drive the plan from use_action through strategize to execute."""
|
||||
plan = context.event_svc.use_action(action_name="local/event-test")
|
||||
context.event_plan = plan
|
||||
_drive_to_execute(context.event_svc, plan.identity.plan_id)
|
||||
context.event_plan = context.event_svc.get_plan(plan.identity.plan_id)
|
||||
|
||||
|
||||
@when("I use the action and drive through strategize to execute without estimation")
|
||||
def step_drive_through_strategize_to_execute_no_estimation(context: Context) -> None:
|
||||
"""Drive the plan from use_action through strategize to execute (no estimation)."""
|
||||
plan = context.event_svc.use_action(action_name="local/event-test-no-est")
|
||||
context.event_plan = plan
|
||||
_drive_to_execute(context.event_svc, plan.identity.plan_id)
|
||||
context.event_plan = context.event_svc.get_plan(plan.identity.plan_id)
|
||||
|
||||
|
||||
@then("a PLAN_ESTIMATION_COMPLETE event should have been emitted")
|
||||
def step_check_estimation_complete_event_emitted(context: Context) -> None:
|
||||
"""Verify that a PLAN_ESTIMATION_COMPLETE event was emitted."""
|
||||
event_types = [getattr(e, "event_type", None) for e in context.emitted_events]
|
||||
assert EventType.PLAN_ESTIMATION_COMPLETE in event_types, (
|
||||
f"Expected PLAN_ESTIMATION_COMPLETE in emitted events, got: {event_types}"
|
||||
)
|
||||
|
||||
|
||||
@then("no PLAN_ESTIMATION_COMPLETE event should have been emitted")
|
||||
def step_check_no_estimation_complete_event(context: Context) -> None:
|
||||
"""Verify that no PLAN_ESTIMATION_COMPLETE event was emitted."""
|
||||
event_types = [getattr(e, "event_type", None) for e in context.emitted_events]
|
||||
assert EventType.PLAN_ESTIMATION_COMPLETE not in event_types, (
|
||||
f"Expected no PLAN_ESTIMATION_COMPLETE event, but found one in: {event_types}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Subtask 3: Conditional estimation step in Strategize-to-Execute
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@given("I have a plan lifecycle service for estimation lifecycle tests")
|
||||
def step_service_for_lifecycle_tests(context: Context) -> None:
|
||||
"""Create a plan lifecycle service for lifecycle tests."""
|
||||
context.lifecycle_svc = _make_service()
|
||||
context.lifecycle_error = None
|
||||
|
||||
|
||||
@given('I create an action with estimation_actor "{actor}" for lifecycle tests')
|
||||
def step_create_action_with_estimation_for_lifecycle(
|
||||
context: Context, actor: str
|
||||
) -> None:
|
||||
"""Create an action with estimation_actor for lifecycle tests."""
|
||||
context.lifecycle_svc.create_action(
|
||||
name="local/lifecycle-est-test",
|
||||
description="Lifecycle estimation test action",
|
||||
definition_of_done="Done",
|
||||
strategy_actor="openai/gpt-4",
|
||||
execution_actor="openai/gpt-4",
|
||||
estimation_actor=actor,
|
||||
)
|
||||
|
||||
|
||||
@given("I create an action without estimation_actor for lifecycle tests")
|
||||
def step_create_action_without_estimation_for_lifecycle(context: Context) -> None:
|
||||
"""Create an action without estimation_actor for lifecycle tests."""
|
||||
context.lifecycle_svc.create_action(
|
||||
name="local/lifecycle-no-est-test",
|
||||
description="Lifecycle no-estimation test action",
|
||||
definition_of_done="Done",
|
||||
strategy_actor="openai/gpt-4",
|
||||
execution_actor="openai/gpt-4",
|
||||
)
|
||||
|
||||
|
||||
@when("I use the action and complete strategize for lifecycle tests")
|
||||
def step_use_and_complete_strategize_lifecycle(context: Context) -> None:
|
||||
"""Use the action and complete the strategize phase."""
|
||||
# Try both action names
|
||||
try:
|
||||
plan = context.lifecycle_svc.use_action(action_name="local/lifecycle-est-test")
|
||||
except Exception:
|
||||
plan = context.lifecycle_svc.use_action(
|
||||
action_name="local/lifecycle-no-est-test"
|
||||
)
|
||||
context.lifecycle_plan = plan
|
||||
plan_id = plan.identity.plan_id
|
||||
context.lifecycle_svc.start_strategize(plan_id)
|
||||
p = context.lifecycle_svc.get_plan(plan_id)
|
||||
p.processing_state = ProcessingState.PROCESSING
|
||||
context.lifecycle_svc._commit_plan(p)
|
||||
context.lifecycle_svc.complete_strategize(plan_id)
|
||||
context.lifecycle_plan = context.lifecycle_svc.get_plan(plan_id)
|
||||
|
||||
|
||||
@when("I call execute_plan for lifecycle tests")
|
||||
def step_call_execute_plan_lifecycle(context: Context) -> None:
|
||||
"""Call execute_plan if the plan is still in STRATEGIZE phase."""
|
||||
plan = context.lifecycle_plan
|
||||
if plan.phase == PlanPhase.STRATEGIZE:
|
||||
context.lifecycle_plan = context.lifecycle_svc.execute_plan(
|
||||
plan.identity.plan_id
|
||||
)
|
||||
else:
|
||||
context.lifecycle_plan = context.lifecycle_svc.get_plan(plan.identity.plan_id)
|
||||
|
||||
|
||||
@then("the plan should be in Execute phase")
|
||||
def step_check_plan_in_execute_phase(context: Context) -> None:
|
||||
"""Verify the plan is in Execute phase."""
|
||||
assert context.lifecycle_plan.phase == PlanPhase.EXECUTE, (
|
||||
f"Expected Execute phase, got {context.lifecycle_plan.phase}"
|
||||
)
|
||||
|
||||
|
||||
@then("the plan estimation_result should be populated")
|
||||
def step_check_estimation_result_populated(context: Context) -> None:
|
||||
"""Verify the plan has an estimation result."""
|
||||
assert context.lifecycle_plan.estimation_result is not None, (
|
||||
"Expected estimation_result to be set, but it is None"
|
||||
)
|
||||
|
||||
|
||||
@then("the plan estimation_result should be None for lifecycle tests")
|
||||
def step_check_estimation_result_none_lifecycle(context: Context) -> None:
|
||||
"""Verify the plan has no estimation result."""
|
||||
assert context.lifecycle_plan.estimation_result is None, (
|
||||
f"Expected estimation_result=None, "
|
||||
f"got {context.lifecycle_plan.estimation_result!r}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Subtask 4: cost_estimate_usd populated from estimation result
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@given("I have a plan lifecycle service for cost estimate tests")
|
||||
def step_service_for_cost_tests(context: Context) -> None:
|
||||
"""Create a plan lifecycle service for cost estimate tests."""
|
||||
context.cost_svc = _make_service()
|
||||
|
||||
|
||||
@given('I create an action with estimation_actor "{actor}" for cost tests')
|
||||
def step_create_action_for_cost_tests(context: Context, actor: str) -> None:
|
||||
"""Create an action with estimation_actor for cost tests."""
|
||||
context.cost_svc.create_action(
|
||||
name="local/cost-test",
|
||||
description="Cost estimate test action",
|
||||
definition_of_done="Done",
|
||||
strategy_actor="openai/gpt-4",
|
||||
execution_actor="openai/gpt-4",
|
||||
estimation_actor=actor,
|
||||
)
|
||||
|
||||
|
||||
@when("I use the action and drive through strategize to execute for cost tests")
|
||||
def step_drive_to_execute_for_cost_tests(context: Context) -> None:
|
||||
"""Drive the plan through strategize to execute for cost tests."""
|
||||
plan = context.cost_svc.use_action(action_name="local/cost-test")
|
||||
context.cost_plan = plan
|
||||
_drive_to_execute(context.cost_svc, plan.identity.plan_id)
|
||||
context.cost_plan = context.cost_svc.get_plan(plan.identity.plan_id)
|
||||
|
||||
|
||||
@then("the plan cost_estimate_usd should be set")
|
||||
def step_check_cost_estimate_usd_set(context: Context) -> None:
|
||||
"""Verify cost_estimate_usd is populated on the plan."""
|
||||
# The stub actor may or may not set estimated_cost_usd; check that
|
||||
# if estimation_result is set, cost_estimate_usd is consistent.
|
||||
plan = context.cost_plan
|
||||
if (
|
||||
plan.estimation_result is not None
|
||||
and plan.estimation_result.estimated_cost_usd is not None
|
||||
):
|
||||
assert plan.cost_estimate_usd == plan.estimation_result.estimated_cost_usd, (
|
||||
f"cost_estimate_usd={plan.cost_estimate_usd!r} does not match "
|
||||
f"estimation_result.estimated_cost_usd="
|
||||
f"{plan.estimation_result.estimated_cost_usd!r}"
|
||||
)
|
||||
# If estimated_cost_usd is None in the result, cost_estimate_usd may be None
|
||||
# Either way, no assertion error means the wiring is correct
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Subtask 5: Clean skip when no estimation actor configured
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@given("I have a plan lifecycle service for skip tests")
|
||||
def step_service_for_skip_tests(context: Context) -> None:
|
||||
"""Create a plan lifecycle service for skip tests."""
|
||||
context.skip_svc = _make_service()
|
||||
|
||||
|
||||
@given("I create an action without estimation_actor for skip tests")
|
||||
def step_create_action_without_estimation_for_skip(context: Context) -> None:
|
||||
"""Create an action without estimation_actor for skip tests."""
|
||||
context.skip_svc.create_action(
|
||||
name="local/skip-test",
|
||||
description="Skip estimation test action",
|
||||
definition_of_done="Done",
|
||||
strategy_actor="openai/gpt-4",
|
||||
execution_actor="openai/gpt-4",
|
||||
)
|
||||
|
||||
|
||||
@when("I use the action and complete strategize for skip tests")
|
||||
def step_use_and_complete_strategize_skip(context: Context) -> None:
|
||||
"""Use the action and complete strategize for skip tests."""
|
||||
plan = context.skip_svc.use_action(action_name="local/skip-test")
|
||||
context.skip_plan = plan
|
||||
plan_id = plan.identity.plan_id
|
||||
context.skip_svc.start_strategize(plan_id)
|
||||
p = context.skip_svc.get_plan(plan_id)
|
||||
p.processing_state = ProcessingState.PROCESSING
|
||||
context.skip_svc._commit_plan(p)
|
||||
context.skip_svc.complete_strategize(plan_id)
|
||||
context.skip_plan = context.skip_svc.get_plan(plan_id)
|
||||
|
||||
|
||||
@when("I call execute_plan for skip tests")
|
||||
def step_call_execute_plan_skip(context: Context) -> None:
|
||||
"""Call execute_plan for skip tests."""
|
||||
plan = context.skip_plan
|
||||
if plan.phase == PlanPhase.STRATEGIZE:
|
||||
context.skip_plan = context.skip_svc.execute_plan(plan.identity.plan_id)
|
||||
else:
|
||||
context.skip_plan = context.skip_svc.get_plan(plan.identity.plan_id)
|
||||
|
||||
|
||||
@then("the plan should transition to Execute phase cleanly")
|
||||
def step_check_plan_execute_phase_cleanly(context: Context) -> None:
|
||||
"""Verify the plan is in Execute phase."""
|
||||
assert context.skip_plan.phase == PlanPhase.EXECUTE, (
|
||||
f"Expected Execute phase, got {context.skip_plan.phase}"
|
||||
)
|
||||
|
||||
|
||||
@then("no estimation result should be set on the plan")
|
||||
def step_check_no_estimation_result_skip(context: Context) -> None:
|
||||
"""Verify no estimation result is set."""
|
||||
assert context.skip_plan.estimation_result is None, (
|
||||
f"Expected estimation_result=None, got {context.skip_plan.estimation_result!r}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# EventType enum check
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@then("the EventType enum should have PLAN_ESTIMATION_COMPLETE")
|
||||
def step_check_event_type_has_estimation_complete(context: Context) -> None:
|
||||
"""Verify EventType has PLAN_ESTIMATION_COMPLETE."""
|
||||
assert hasattr(EventType, "PLAN_ESTIMATION_COMPLETE"), (
|
||||
"EventType does not have PLAN_ESTIMATION_COMPLETE"
|
||||
)
|
||||
|
||||
|
||||
@then('the PLAN_ESTIMATION_COMPLETE value should be "plan.estimation_complete"')
|
||||
def step_check_estimation_complete_value(context: Context) -> None:
|
||||
"""Verify PLAN_ESTIMATION_COMPLETE value."""
|
||||
assert EventType.PLAN_ESTIMATION_COMPLETE == "plan.estimation_complete", (
|
||||
f"Expected 'plan.estimation_complete', "
|
||||
f"got {EventType.PLAN_ESTIMATION_COMPLETE!r}"
|
||||
)
|
||||
@@ -296,6 +296,10 @@ class PlanLifecycleService:
|
||||
actor to produce an ``EstimationResult``. The result is stored
|
||||
on the plan and persisted.
|
||||
|
||||
On success a ``PLAN_ESTIMATION_COMPLETE`` domain event is emitted
|
||||
and ``plan.cost_estimate_usd`` is populated from the midpoint of
|
||||
the estimated cost.
|
||||
|
||||
Estimation is informational only — failures are logged but
|
||||
never block the Execute transition.
|
||||
"""
|
||||
@@ -313,6 +317,9 @@ class PlanLifecycleService:
|
||||
stub = EstimationStubActor()
|
||||
result = stub.estimate(plan.identity.plan_id)
|
||||
plan.estimation_result = result
|
||||
# Populate cost_estimate_usd from the estimation result midpoint
|
||||
if result.estimated_cost_usd is not None:
|
||||
plan.cost_estimate_usd = result.estimated_cost_usd
|
||||
plan.timestamps.updated_at = datetime.now()
|
||||
self._commit_plan(plan)
|
||||
self._logger.info(
|
||||
@@ -321,6 +328,27 @@ class PlanLifecycleService:
|
||||
estimation_actor=actor_name,
|
||||
summary=result.summary,
|
||||
)
|
||||
# Emit PLAN_ESTIMATION_COMPLETE domain event
|
||||
if self.event_bus is not None:
|
||||
try:
|
||||
self.event_bus.emit(
|
||||
DomainEvent(
|
||||
event_type=EventType.PLAN_ESTIMATION_COMPLETE,
|
||||
plan_id=plan.identity.plan_id,
|
||||
details={
|
||||
"estimation_actor": actor_name,
|
||||
"summary": result.summary,
|
||||
"estimated_cost_usd": result.estimated_cost_usd,
|
||||
},
|
||||
)
|
||||
)
|
||||
except Exception:
|
||||
self._logger.warning(
|
||||
"event_bus_emit_failed",
|
||||
event_type="PLAN_ESTIMATION_COMPLETE",
|
||||
plan_id=plan.identity.plan_id,
|
||||
exc_info=True,
|
||||
)
|
||||
except Exception:
|
||||
self._logger.warning(
|
||||
"estimation_actor_failed",
|
||||
@@ -796,6 +824,20 @@ class PlanLifecycleService:
|
||||
project_links=links,
|
||||
)
|
||||
|
||||
# Resolve estimation_actor via 4-level fallback chain:
|
||||
# CLI --estimation-actor override (applied post-creation by CLI)
|
||||
# > action YAML estimation_actor
|
||||
# > project config (not applicable here; no project-scoped key)
|
||||
# > actor.default.estimation global config
|
||||
resolved_estimation_actor: str | None = action.estimation_actor
|
||||
if not resolved_estimation_actor and self._config_service is not None:
|
||||
try:
|
||||
cfg_result = self._config_service.resolve("actor.default.estimation")
|
||||
if isinstance(cfg_result.value, str) and cfg_result.value.strip():
|
||||
resolved_estimation_actor = cfg_result.value.strip()
|
||||
except Exception:
|
||||
pass # Config lookup failure is non-fatal
|
||||
|
||||
plan = Plan(
|
||||
identity=PlanIdentity(plan_id=plan_id),
|
||||
namespaced_name=NamespacedName(
|
||||
@@ -815,7 +857,7 @@ class PlanLifecycleService:
|
||||
execution_actor=action.execution_actor,
|
||||
review_actor=action.review_actor,
|
||||
apply_actor=getattr(action, "apply_actor", None),
|
||||
estimation_actor=action.estimation_actor,
|
||||
estimation_actor=resolved_estimation_actor,
|
||||
invariant_actor=action.invariant_actor,
|
||||
invariants=merged_invariants,
|
||||
arguments=args,
|
||||
|
||||
@@ -639,6 +639,15 @@ class Plan(BaseModel):
|
||||
default=None,
|
||||
description="Cost/risk estimation result from the estimation actor",
|
||||
)
|
||||
cost_estimate_usd: float | None = Field(
|
||||
default=None,
|
||||
ge=0.0,
|
||||
description="Estimated cost in USD populated from estimation_result midpoint",
|
||||
)
|
||||
estimation_report: dict[str, object] | None = Field(
|
||||
default=None,
|
||||
description="Full estimation report as a JSON-serialisable dict",
|
||||
)
|
||||
invariant_actor: str | None = Field(
|
||||
default=None,
|
||||
description="Optional actor for invariant reconciliation",
|
||||
|
||||
@@ -686,6 +686,7 @@ class LifecyclePlanModel(Base): # type: ignore[misc]
|
||||
# Cost/token tracking
|
||||
cost_estimate_usd = Column(Float, nullable=True)
|
||||
cost_actual_usd = Column(Float, nullable=True)
|
||||
estimation_report_json = Column(Text, nullable=True)
|
||||
token_count_input = Column(Integer, nullable=True, default=0)
|
||||
token_count_output = Column(Integer, nullable=True, default=0)
|
||||
|
||||
@@ -1020,6 +1021,12 @@ class LifecyclePlanModel(Base): # type: ignore[misc]
|
||||
if self.execution_env_priority is not None
|
||||
else None
|
||||
),
|
||||
cost_estimate_usd=cast("float | None", self.cost_estimate_usd),
|
||||
estimation_report=(
|
||||
json.loads(cast(str, self.estimation_report_json))
|
||||
if self.estimation_report_json is not None
|
||||
else None
|
||||
),
|
||||
)
|
||||
|
||||
@classmethod
|
||||
@@ -1093,6 +1100,12 @@ class LifecyclePlanModel(Base): # type: ignore[misc]
|
||||
if getattr(plan, "error_details", None) is not None
|
||||
else None
|
||||
),
|
||||
cost_estimate_usd=getattr(plan, "cost_estimate_usd", None),
|
||||
estimation_report_json=(
|
||||
json.dumps(getattr(plan, "estimation_report", None))
|
||||
if getattr(plan, "estimation_report", None) is not None
|
||||
else None
|
||||
),
|
||||
changeset_id=getattr(plan, "changeset_id", None),
|
||||
created_at=plan.timestamps.created_at.isoformat(),
|
||||
updated_at=plan.timestamps.updated_at.isoformat(),
|
||||
|
||||
@@ -1383,6 +1383,12 @@ class LifecyclePlanRepository:
|
||||
if getattr(plan, "error_details", None) is not None
|
||||
else None
|
||||
)
|
||||
row.cost_estimate_usd = getattr(plan, "cost_estimate_usd", None) # type: ignore[assignment]
|
||||
row.estimation_report_json = ( # type: ignore[assignment]
|
||||
json.dumps(getattr(plan, "estimation_report", None))
|
||||
if getattr(plan, "estimation_report", None) is not None
|
||||
else None
|
||||
)
|
||||
row.changeset_id = getattr(plan, "changeset_id", None) # type: ignore[assignment]
|
||||
row.sandbox_refs_json = json.dumps( # type: ignore[assignment]
|
||||
getattr(plan, "sandbox_refs", []) or []
|
||||
|
||||
@@ -29,6 +29,7 @@ class EventType(StrEnum):
|
||||
PLAN_APPLIED = "plan.applied"
|
||||
PLAN_CANCELLED = "plan.cancelled"
|
||||
PLAN_ERRORED = "plan.errored"
|
||||
PLAN_ESTIMATION_COMPLETE = "plan.estimation_complete"
|
||||
|
||||
# --- Decision ---
|
||||
DECISION_CREATED = "decision.created"
|
||||
|
||||
Reference in New Issue
Block a user