aditya
97920678b9
Fix unit test and integration test failures on Forgejo CI caused by Rich/Click text wrapping in narrow terminal environments. The assertion for "--no-estimate cannot be used with --estimation-actor" was split across lines when COLUMNS was small, breaking exact substring checks. - Normalize whitespace before assertion in cli_extensions_steps.py so the check is resilient to line wrapping regardless of terminal width. - Set COLUMNS=240 in robot/helper_plan_cli_spec.py to give CLI subprocesses a consistent wide terminal, preventing Rich from reflowing output mid-assertion. - Add 16 new BDD scenarios to estimation_coverage.feature exercising previously-uncovered estimation branches: validation constraints (max_length, min_length, frozen model), EstimationActorProtocol runtime check, all three exception handler paths in estimate_plan, parse_estimation_report generic error path, serialization of non-None values, EstimationError attribute preservation, and CLI dict / status detail rendering of estimation data. - estimation_service.py coverage: 75% → 99% (only TYPE_CHECKING and Protocol stub lines remain uncovered). - Overall coverage: 96.89% → 97.0%, meeting the project threshold. All changes are scoped to estimation feature tests and the estimation CLI assertion helpers introduced by this branch. Refs: #209
229 lines
10 KiB
Gherkin
229 lines
10 KiB
Gherkin
@m6 @estimation
|
|
Feature: Estimation Coverage - Edge Cases
|
|
Additional scenarios that exercise estimation parsing/serialization branches.
|
|
|
|
Background:
|
|
Given a clean test environment for estimation
|
|
|
|
@estimation_coverage @validation
|
|
Scenario: Actor name without namespace separator
|
|
When I create an estimation report with invalid actor "badactor"
|
|
Then a validation error should occur
|
|
And the coverage validation error should mention "must be namespaced"
|
|
|
|
@estimation_coverage @serialization
|
|
Scenario: Serialize None estimation report returns None
|
|
Given an estimation service
|
|
When I serialize a None estimation report
|
|
Then the serialization result should be None
|
|
|
|
@estimation_coverage @serialization
|
|
Scenario: Serialize None estimation skipped returns None
|
|
Given an estimation service
|
|
When I serialize a None estimation skipped
|
|
Then the serialization result should be None
|
|
|
|
@estimation_coverage @parsing
|
|
Scenario: Parse estimation report with invalid JSON
|
|
Given an estimation service
|
|
When I parse estimation report with invalid JSON "{not valid"
|
|
Then an estimation error should be raised
|
|
And the coverage estimation error should mention "Invalid JSON"
|
|
|
|
@estimation_coverage @parsing
|
|
Scenario: Parse estimation report with generic parse error
|
|
Given an estimation service
|
|
When I parse estimation report that causes a parse error
|
|
Then an estimation error should be raised
|
|
And the coverage estimation error should mention "Failed to parse"
|
|
|
|
@estimation_coverage @parsing
|
|
Scenario: Parse estimation report without actor_used sets default
|
|
Given an estimation service
|
|
When I parse estimation report without actor_used field for actor "local/test"
|
|
Then the parsed estimation should have actor "local/test"
|
|
|
|
@estimation_coverage @parsing
|
|
Scenario: Parse estimation report does not mutate caller dict input
|
|
Given an estimation service
|
|
When I parse estimation report from dict input without actor_used for actor "local/test"
|
|
Then the parsed estimation should have actor "local/test"
|
|
And the original estimation dict should not contain key "actor_used"
|
|
|
|
@estimation_coverage @stub_actor
|
|
Scenario: Stub actor with invariants increases cost
|
|
Given an action with estimation actor "local/estimator"
|
|
And the action has invariants
|
|
When I use the action to create a plan
|
|
And I transition the created plan to execute
|
|
Then the plan should have estimation report
|
|
And the cost should reflect invariant overhead
|
|
|
|
@estimation_coverage @stub_actor
|
|
Scenario: Stub actor with multiple projects increases risk
|
|
Given an action with estimation actor "local/estimator"
|
|
And the action has 3 project links
|
|
When I use the action to create a plan
|
|
And I transition the created plan to execute
|
|
Then the plan should have estimation report
|
|
And the risk factors should mention "Multiple projects"
|
|
|
|
@estimation_coverage @stub_actor
|
|
Scenario: Stub actor with invariants increases risk
|
|
Given an action with estimation actor "local/estimator"
|
|
And the action has invariants
|
|
When I use the action to create a plan
|
|
And I transition the created plan to execute
|
|
Then the plan should have estimation report
|
|
And the risk factors should mention "Invariant constraints"
|
|
|
|
@estimation_coverage @stub_actor
|
|
Scenario: Stub actor with custom arguments increases risk
|
|
Given an action with estimation actor "local/estimator"
|
|
And the action has custom arguments
|
|
When I use the action to create a plan
|
|
And I transition the created plan to execute
|
|
Then the plan should have estimation report
|
|
And the risk factors should mention "Custom arguments"
|
|
|
|
@estimation_coverage @stub_actor
|
|
Scenario: Stub actor with invariants increases duration
|
|
Given an action with estimation actor "local/estimator"
|
|
And the action has invariants
|
|
When I use the action to create a plan
|
|
And I transition the created plan to execute
|
|
Then the plan should have estimation report
|
|
And the duration estimate should be increased for invariants
|
|
|
|
@estimation_coverage @lifecycle
|
|
Scenario: update_plan_overrides atomically sets multiple fields
|
|
Given an action with estimation actor "local/estimator"
|
|
When I use the action to create a plan
|
|
Then the plan should be created successfully
|
|
When I update plan overrides with strategy_actor "local/new-strategy" and execution_actor "local/new-exec"
|
|
Then the overridden plan should have strategy_actor "local/new-strategy"
|
|
And the overridden plan should have execution_actor "local/new-exec"
|
|
And the overridden plan updated_at should be later than creation time
|
|
|
|
@estimation_coverage @validation
|
|
Scenario: EstimationReport rejects rationale exceeding max length
|
|
When I create an estimation report with rationale exceeding 10000 characters
|
|
Then a validation error should occur
|
|
And the coverage validation error should mention "rationale"
|
|
|
|
@estimation_coverage @validation
|
|
Scenario: EstimationSkipped rejects empty reason
|
|
When I create an estimation skipped with empty reason
|
|
Then a validation error should occur
|
|
|
|
@estimation_coverage @validation
|
|
Scenario: EstimationReport is frozen and rejects mutation
|
|
Given a valid estimation report
|
|
When I attempt to mutate the estimation report confidence
|
|
Then a validation error should occur
|
|
|
|
@estimation_coverage @validation
|
|
Scenario: EstimationReport historical_basis is an immutable tuple
|
|
When I create an estimation report with historical basis entries
|
|
Then the historical basis should be a tuple
|
|
And the historical basis should contain the expected entries
|
|
|
|
@estimation_coverage @protocol
|
|
Scenario: EstimationActorProtocol is runtime checkable
|
|
When I check a conforming class against EstimationActorProtocol
|
|
Then the conforming class should satisfy the protocol
|
|
And the estimation service should not satisfy the protocol
|
|
|
|
@estimation_coverage @error
|
|
Scenario: EstimationError preserves actor name and reason
|
|
When I create an estimation error with actor "local/broken" and reason "timeout"
|
|
Then the estimation error actor_name should be "local/broken"
|
|
And the estimation error reason should be "timeout"
|
|
And the estimation error message should contain "local/broken"
|
|
|
|
@estimation_coverage @service
|
|
Scenario: estimate_plan catches EstimationError from actor
|
|
Given a plan with estimation actor configured
|
|
And the estimation service has a patched actor raising EstimationError
|
|
When the estimation service estimates the plan
|
|
Then an EstimationSkipped record should be returned
|
|
And the estimation result reason should include "Estimation actor failed"
|
|
|
|
@estimation_coverage @service
|
|
Scenario: estimate_plan catches TypeError from actor
|
|
Given a plan with estimation actor configured
|
|
And the estimation service has a patched actor raising TypeError
|
|
When the estimation service estimates the plan
|
|
Then an EstimationSkipped record should be returned
|
|
And the estimation result reason should include "TypeError"
|
|
|
|
@estimation_coverage @service
|
|
Scenario: estimate_plan catches unexpected Exception from actor
|
|
Given a plan with estimation actor configured
|
|
And the estimation service has a patched actor raising RuntimeError
|
|
When the estimation service estimates the plan
|
|
Then an EstimationSkipped record should be returned
|
|
And the estimation result reason should include "RuntimeError"
|
|
|
|
@estimation_coverage @serialization
|
|
Scenario: Serialize a valid estimation report returns JSON
|
|
Given an estimation service
|
|
When I serialize a valid estimation report
|
|
Then the serialization result should be a JSON string
|
|
And the serialized JSON should contain key "cost_range_usd_min"
|
|
|
|
@estimation_coverage @serialization
|
|
Scenario: Serialize a valid estimation skipped returns JSON
|
|
Given an estimation service
|
|
When I serialize a valid estimation skipped
|
|
Then the serialization result should be a JSON string
|
|
And the serialized JSON should contain key "reason"
|
|
|
|
@estimation_coverage @parsing
|
|
Scenario: Parse estimation report with unexpected non-validation error
|
|
Given an estimation service
|
|
When I parse estimation report with input causing a generic error
|
|
Then an estimation error should be raised
|
|
And the coverage estimation error should mention "Failed to parse"
|
|
|
|
@estimation_coverage @cli
|
|
Scenario: Plan with estimation report renders in CLI dict format
|
|
Given an action with estimation actor "local/estimator"
|
|
When I use the action to create a plan
|
|
And I transition the created plan to execute
|
|
Then the plan should have estimation report
|
|
When I render the created plan as a CLI dict
|
|
Then the CLI dict should contain key "estimation_report"
|
|
And the CLI dict should contain key "estimation_actor"
|
|
And the CLI dict estimation_report should contain "cost_range_usd_min"
|
|
|
|
@estimation_coverage @cli
|
|
Scenario: Plan with estimation skipped renders in CLI dict format
|
|
Given an action with estimation actor "local/estimator"
|
|
And skip_estimation is True
|
|
When I use the action to create a plan
|
|
Then the plan should be created successfully
|
|
When I render the created plan as a CLI dict
|
|
Then the CLI dict should contain key "estimation_skipped"
|
|
And the CLI dict estimation_skipped should contain "reason"
|
|
|
|
@estimation_coverage @cli
|
|
Scenario: Plan status detail shows estimation report data
|
|
Given an action with estimation actor "local/estimator"
|
|
When I use the action to create a plan
|
|
And I transition the created plan to execute
|
|
Then the plan should have estimation report
|
|
When I render the plan status detail
|
|
Then the status detail should contain "Cost Range"
|
|
And the status detail should contain "Rollback Risk"
|
|
And the status detail should contain "Execution Estimate"
|
|
|
|
@estimation_coverage @cli
|
|
Scenario: Plan status detail shows estimation skipped reason
|
|
Given an action with estimation actor "local/estimator"
|
|
And skip_estimation is True
|
|
When I use the action to create a plan
|
|
Then the plan should be created successfully
|
|
When I render the plan status detail
|
|
Then the status detail should contain "Estimation skipped"
|