aditya
50680612d5
Implemented optional estimation_actor role for cost, risk, and duration estimation during plan lifecycle. Estimates are persisted to plan metadata and surfaced in CLI output. Key implementation details: - EstimationOutput (Pydantic models): CostEstimate with currency, token estimates, and confidence ranges; RiskScore with 0-100 scale and factors; DurationEstimate with min/expected/max seconds. All include confidence levels and validation. EstimationSkipped records when estimation is opted out. - EstimationService: stateless async service that invokes estimation actor (stub implementation for M6). Handles actor output parsing, error recovery, and fallback to EstimationSkipped on failure. - Integration: Plan model gains estimation_output and estimation_skipped fields. LifecyclePlanModel adds JSON columns for persistence. PlanLifecycleService invokes estimation during use_action unless skip_estimation is true. - CLI: --no-estimate flag added to 'agents plan use'. plan status displays cost (USD with token estimates), risk (score/100 with confidence), and duration (seconds with ranges) in rich format. - Database: Alembic migration m6_003_estimation_metadata adds estimation_output_json and estimation_skipped_json columns to v3_plans. - Tests: 44 BDD scenarios (features/estimation.feature) covering validation, lifecycle integration, persistence, CLI display, and edge cases. 18 Robot Framework smoke tests. 19 ASV benchmark suites for schema, serialization, validation, and plan integration performance. - Documentation: docs/reference/estimation.md with schema, configuration, and examples. plan_cli.md updated with --no-estimate flag usage. ISSUES CLOSED: #209
125 lines
5.9 KiB
Gherkin
125 lines
5.9 KiB
Gherkin
@m6 @cli
|
|
Feature: Plan CLI uncovered lines and branches coverage
|
|
As a developer
|
|
I want to exercise the uncovered lines and branches in plan.py
|
|
So that code coverage gaps at lines 80, 818, 827-828, 999, 1017-1035,
|
|
1314-1315, 1391-1392, 1573-1578, 1594-1596, 1700-1701, 1739 are closed
|
|
|
|
# ===================================================================
|
|
# Legacy apply command — Progress block and PlanError handler
|
|
# ===================================================================
|
|
|
|
Scenario: Legacy apply succeeds through Progress spinner block
|
|
Given a plan-cov CLI runner
|
|
And a mocked legacy container for plan-cov apply that succeeds
|
|
When I invoke plan-cov legacy apply with --yes
|
|
Then the plan-cov command should exit normally
|
|
And the plan-cov output should contain "Successfully applied"
|
|
|
|
Scenario: Legacy apply raises PlanError
|
|
Given a plan-cov CLI runner
|
|
And a mocked legacy container for plan-cov apply that raises PlanError
|
|
When I invoke plan-cov legacy apply with --yes
|
|
Then the plan-cov command should abort
|
|
And the plan-cov output should contain "Apply Error"
|
|
|
|
# ===================================================================
|
|
# Legacy list command — plan.current=False branch
|
|
# ===================================================================
|
|
|
|
Scenario: Legacy list shows plans with current=False
|
|
Given a plan-cov CLI runner
|
|
And a mocked legacy container for plan-cov list with non-current plans
|
|
When I invoke plan-cov legacy list
|
|
Then the plan-cov command should exit normally
|
|
And the plan-cov output should contain "Plans"
|
|
|
|
# ===================================================================
|
|
# Legacy cd command — full body coverage
|
|
# ===================================================================
|
|
|
|
Scenario: Legacy cd switches to an existing plan
|
|
Given a plan-cov CLI runner
|
|
And a mocked legacy container for plan-cov cd that succeeds
|
|
When I invoke plan-cov legacy cd "my-plan"
|
|
Then the plan-cov command should exit normally
|
|
And the plan-cov output should contain "Switched to plan"
|
|
|
|
Scenario: Legacy cd raises CleverAgentsError
|
|
Given a plan-cov CLI runner
|
|
And a mocked legacy container for plan-cov cd that raises CleverAgentsError
|
|
When I invoke plan-cov legacy cd "bad-plan"
|
|
Then the plan-cov command should abort
|
|
And the plan-cov output should contain "Error"
|
|
|
|
# ===================================================================
|
|
# use command — invariant_actor and invariants
|
|
# ===================================================================
|
|
|
|
Scenario: Use action with estimation-actor and invariant-actor overrides
|
|
Given a plan-cov CLI runner
|
|
And a mocked lifecycle service for plan-cov use action
|
|
When I invoke plan-cov use with estimation-actor "openai/gpt-4" and invariant-actor "openai/gpt-4"
|
|
Then the plan-cov command should exit normally
|
|
And the plan-cov created plan should have estimation_actor set
|
|
|
|
Scenario: Use action with --invariant flags builds plan invariants
|
|
Given a plan-cov CLI runner
|
|
And a mocked lifecycle service for plan-cov use action
|
|
When I invoke plan-cov use with invariant "No new warnings"
|
|
Then the plan-cov command should exit normally
|
|
|
|
# ===================================================================
|
|
# lifecycle-apply — no plan_id auto-discovery branches
|
|
# ===================================================================
|
|
|
|
Scenario: lifecycle-apply without plan_id when no plans are ready
|
|
Given a plan-cov CLI runner
|
|
And a mocked lifecycle service for plan-cov that returns no execute-complete plans
|
|
When I invoke plan-cov lifecycle-apply without plan_id
|
|
Then the plan-cov command should abort
|
|
And the plan-cov output should contain "No plans ready for apply"
|
|
|
|
Scenario: lifecycle-apply without plan_id when multiple plans are ready
|
|
Given a plan-cov CLI runner
|
|
And a mocked lifecycle service for plan-cov that returns multiple execute-complete plans
|
|
When I invoke plan-cov lifecycle-apply without plan_id
|
|
Then the plan-cov command should abort
|
|
And the plan-cov output should contain "Multiple plans ready"
|
|
|
|
Scenario: lifecycle-apply rejects a read-only plan
|
|
Given a plan-cov CLI runner
|
|
And a mocked lifecycle service for plan-cov that returns a read-only plan
|
|
When I invoke plan-cov lifecycle-apply with plan_id "01ARZ3NDEKTSV4RRFFQ69G5FAA"
|
|
Then the plan-cov command should abort
|
|
And the plan-cov output should contain "read-only"
|
|
|
|
# ===================================================================
|
|
# status command — CleverAgentsError handler
|
|
# ===================================================================
|
|
|
|
Scenario: status command raises CleverAgentsError
|
|
Given a plan-cov CLI runner
|
|
And a mocked lifecycle service for plan-cov status that raises CleverAgentsError
|
|
When I invoke plan-cov status "01ARZ3NDEKTSV4RRFFQ69G5FAA"
|
|
Then the plan-cov command should abort
|
|
And the plan-cov output should contain "Error"
|
|
|
|
# ===================================================================
|
|
# errors command — no error_category branch
|
|
# ===================================================================
|
|
|
|
Scenario: errors command on plan with no error recovery data
|
|
Given a plan-cov CLI runner
|
|
And a mocked lifecycle service for plan-cov errors with no error_category
|
|
When I invoke plan-cov errors "01ARZ3NDEKTSV4RRFFQ69G5FAA" with format "json"
|
|
Then the plan-cov command should exit normally
|
|
And the plan-cov output should contain "plan_id"
|
|
|
|
Scenario: errors command on plan with empty error_details in rich format
|
|
Given a plan-cov CLI runner
|
|
And a mocked lifecycle service for plan-cov errors with empty details
|
|
When I invoke plan-cov errors "01ARZ3NDEKTSV4RRFFQ69G5FAA"
|
|
Then the plan-cov command should exit normally
|
|
And the plan-cov output should contain "No error recovery records"
|