forked from cleveragents/cleveragents-core
aditya
3f6b255747
Add --no-estimate opt-out flag for plan use command and persist EstimationSkipped reason when estimation is skipped or fails. Add EstimationReport Pydantic domain model (identical to #649) with spec-aligned multi-dimensional output: cost range (min/max USD), expected steps, expected child plans, rollback risk (0.0-1.0), estimated duration in minutes, confidence (0.0-1.0), rationale, and optional historical basis. Add EstimationSkipped domain model with reason, timestamp, and optional actor_name fields for tracking when estimation is skipped via --no-estimate or when the estimation actor fails. Add estimation_produced to DecisionType enum and STRATEGIZE_TYPES. Add Alembic migration m6_006 adding estimation_report_json and estimation_skipped_json columns to v3_plans and updating the ck_decisions_type CHECK constraint. Wire --no-estimate through plan use CLI: when set, creates EstimationSkipped with reason and clears the estimation actor. Update _run_estimation() to persist EstimationSkipped on actor failure instead of only logging. Add docs/reference/estimation.md documenting the estimation feature, EstimationReport schema, EstimationSkipped schema, and CLI examples. Add benchmarks/estimation_actor_bench.py with ASV benchmarks for EstimationReport and EstimationSkipped operations. Add 13 Behave scenarios (estimation_skip.feature) and 6 Robot integration tests (estimation_skip.robot) covering EstimationSkipped model, EstimationReport model, plan field mutual exclusion, and --no-estimate CLI behavior. ISSUES CLOSED: #209
84 lines
3.8 KiB
Gherkin
84 lines
3.8 KiB
Gherkin
Feature: Estimation skip behavior
|
|
Plans can skip estimation via --no-estimate or when the estimation actor fails.
|
|
|
|
# ------------------------------------------------------------------
|
|
# EstimationSkipped model
|
|
# ------------------------------------------------------------------
|
|
|
|
Scenario: Create an EstimationSkipped with required fields
|
|
When I create an EstimationSkipped with only reason "User opted out"
|
|
Then the EstimationSkipped should be created successfully
|
|
And the EstimationSkipped reason should be "User opted out"
|
|
And the EstimationSkipped actor_name should be None
|
|
|
|
Scenario: Create an EstimationSkipped with actor name
|
|
When I create an EstimationSkipped with reason "Actor failed" and actor "local/estimator"
|
|
Then the EstimationSkipped should be created successfully
|
|
And the EstimationSkipped actor_name should be "local/estimator"
|
|
|
|
Scenario: EstimationSkipped is frozen
|
|
When I create an EstimationSkipped with only reason "Frozen test"
|
|
Then modifying the EstimationSkipped reason should raise an error
|
|
|
|
Scenario: EstimationSkipped round-trips through serialization
|
|
When I create an EstimationSkipped with reason "Round-trip test" and actor "local/est"
|
|
Then the EstimationSkipped should round-trip through model_dump and model_validate
|
|
|
|
Scenario: EstimationSkipped requires non-empty reason
|
|
When I try to create an EstimationSkipped with empty reason
|
|
Then an estimation skip validation error should be raised
|
|
|
|
# ------------------------------------------------------------------
|
|
# EstimationReport model
|
|
# ------------------------------------------------------------------
|
|
|
|
Scenario: Create a valid EstimationReport
|
|
When I create a valid EstimationReport
|
|
Then the EstimationReport should be created successfully
|
|
And the EstimationReport confidence should be 0.85
|
|
|
|
Scenario: EstimationReport rejects max < min cost
|
|
When I try to create an EstimationReport with max cost less than min cost
|
|
Then an estimation report validation error should be raised
|
|
|
|
Scenario: EstimationReport rejects confidence above 1.0
|
|
When I try to create an EstimationReport with confidence 1.5
|
|
Then an estimation report validation error should be raised
|
|
|
|
Scenario: EstimationReport rejects negative rollback risk
|
|
When I try to create an EstimationReport with rollback risk -0.1
|
|
Then an estimation report validation error should be raised
|
|
|
|
Scenario: EstimationReport caps historical basis at 100
|
|
When I try to create an EstimationReport with 101 historical basis entries
|
|
Then an estimation report validation error should be raised
|
|
|
|
Scenario: EstimationReport is frozen
|
|
When I create a valid EstimationReport
|
|
Then modifying the EstimationReport rationale should raise an error
|
|
|
|
# ------------------------------------------------------------------
|
|
# Mutual exclusion on Plan
|
|
# ------------------------------------------------------------------
|
|
|
|
Scenario: Plan with estimation_skipped but no estimation_report
|
|
When I create a plan with estimation_skipped set
|
|
Then the plan estimation_skipped should not be None
|
|
And the plan estimation_report should be None
|
|
|
|
Scenario: Plan with estimation_report but no estimation_skipped
|
|
When I create a plan with estimation_report set
|
|
Then the plan estimation_report should not be None
|
|
And the plan estimation_skipped should be None
|
|
|
|
# ------------------------------------------------------------------
|
|
# --no-estimate CLI flag behavior
|
|
# ------------------------------------------------------------------
|
|
|
|
Scenario: --no-estimate sets estimation_skipped on plan
|
|
Given a lifecycle service with a test action
|
|
When I use the action with --no-estimate
|
|
Then the plan should have estimation_skipped set
|
|
And the estimation_skipped reason should mention "opted out"
|
|
And the plan estimation_actor should be None
|