d70a5d84ae
ISSUES CLOSED: #186
176 lines
8.8 KiB
Gherkin
176 lines
8.8 KiB
Gherkin
Feature: Error recovery coverage boost
|
|
As a developer
|
|
I want to cover uncovered lines in plan CLI errors/diff/artifacts commands
|
|
and plan executor edge cases introduced by the error recovery feature
|
|
So that overall code coverage exceeds the 97% threshold
|
|
|
|
Background:
|
|
Given an errcov test environment
|
|
|
|
# -- plan errors CLI (plan.py:1621-1720) ---------------------------------
|
|
|
|
Scenario: Plan errors command rich output with full error details
|
|
Given an errcov plan with error details category "transient" phase "execute"
|
|
And the errcov plan has error message "Connection timed out after 30s"
|
|
And the errcov plan error details include actor "test-actor" and tool "test-tool"
|
|
And the errcov plan error details include stack summary "File test.py line 10"
|
|
When I invoke errcov plan errors in "rich" format
|
|
Then the errcov errors output should contain "Plan Errors"
|
|
And the errcov errors output should contain "transient"
|
|
And the errcov errors output should contain "execute"
|
|
And the errcov errors output should contain "test-actor"
|
|
And the errcov errors output should contain "test-tool"
|
|
And the errcov errors output should contain "File test.py"
|
|
And the errcov errors output should contain "Recovery Suggestions"
|
|
|
|
Scenario: Plan errors command plain output with error details
|
|
Given an errcov plan with error details category "validation" phase "apply"
|
|
And the errcov plan has error message "Schema validation failed"
|
|
When I invoke errcov plan errors in "plain" format
|
|
Then the errcov errors output should contain "validation"
|
|
And the errcov errors output should contain "apply"
|
|
And the errcov errors output should contain "Schema validation failed"
|
|
|
|
Scenario: Plan errors command JSON output
|
|
Given an errcov plan with error details category "transient" phase "execute"
|
|
And the errcov plan has error message "Timeout"
|
|
When I invoke errcov plan errors in "json" format
|
|
Then the errcov errors output should contain "transient"
|
|
And the errcov errors output should contain "plan_id"
|
|
|
|
Scenario: Plan errors command rich output with no errors and no message
|
|
Given an errcov plan with no error details and no error message
|
|
When I invoke errcov plan errors in "rich" format
|
|
Then the errcov errors output should contain "No error recovery records"
|
|
|
|
Scenario: Plan errors command rich output with error message but no recovery
|
|
Given an errcov plan with no error details but has error message "Something broke"
|
|
When I invoke errcov plan errors in "rich" format
|
|
Then the errcov errors output should contain "Something broke"
|
|
|
|
Scenario: Plan errors command handles CleverAgentsError
|
|
Given an errcov lifecycle service that raises CleverAgentsError on get_plan
|
|
When I invoke errcov plan errors expecting abort
|
|
Then the errcov errors output should contain "Error:"
|
|
|
|
Scenario: Plan errors command plain output with actor and tool and stack
|
|
Given an errcov plan with error details category "provider" phase "execute"
|
|
And the errcov plan error details include actor "gpt4-actor" and tool "code-gen"
|
|
And the errcov plan error details include stack summary "traceback info"
|
|
And the errcov plan error details include retry count "2" max "3" retriable "true"
|
|
When I invoke errcov plan errors in "plain" format
|
|
Then the errcov errors output should contain "gpt4-actor"
|
|
And the errcov errors output should contain "code-gen"
|
|
And the errcov errors output should contain "traceback info"
|
|
And the errcov errors output should contain "recovery_hints"
|
|
|
|
# -- plan diff CLI (plan.py:1969-1979) -----------------------------------
|
|
|
|
Scenario: Plan diff command calls apply service diff
|
|
Given an errcov apply service returning diff text "--- a/file.py"
|
|
When I invoke errcov plan diff in "rich" format
|
|
Then the errcov output should contain "--- a/file.py"
|
|
|
|
Scenario: Plan diff command handles PlanError
|
|
Given an errcov apply service that raises PlanError on diff
|
|
When I invoke errcov plan diff expecting abort
|
|
Then the errcov output should contain "Diff Error:"
|
|
|
|
Scenario: Plan diff command handles CleverAgentsError
|
|
Given an errcov apply service that raises CleverAgentsError on diff
|
|
When I invoke errcov plan diff expecting abort
|
|
Then the errcov output should contain "Error:"
|
|
|
|
# -- plan artifacts CLI (plan.py:2002-2012) ------------------------------
|
|
|
|
Scenario: Plan artifacts command calls apply service artifacts
|
|
Given an errcov apply service returning artifacts text "ChangeSet: cs_001"
|
|
When I invoke errcov plan artifacts in "rich" format
|
|
Then the errcov output should contain "ChangeSet: cs_001"
|
|
|
|
Scenario: Plan artifacts command handles PlanError
|
|
Given an errcov apply service that raises PlanError on artifacts
|
|
When I invoke errcov plan artifacts expecting abort
|
|
Then the errcov output should contain "Artifacts Error:"
|
|
|
|
Scenario: Plan artifacts command handles CleverAgentsError
|
|
Given an errcov apply service that raises CleverAgentsError on artifacts
|
|
When I invoke errcov plan artifacts expecting abort
|
|
Then the errcov output should contain "Error:"
|
|
|
|
# -- _get_apply_service (plan.py:1941-1946) ------------------------------
|
|
|
|
Scenario: Get apply service constructs PlanApplyService
|
|
When I invoke errcov _get_apply_service
|
|
Then the errcov apply service should be a PlanApplyService instance
|
|
|
|
# -- PlanExecutor edge cases (plan_executor.py) --------------------------
|
|
|
|
Scenario: Strategize with empty plan_id raises ValidationError
|
|
When I invoke errcov strategize with empty plan_id
|
|
Then an errcov ValidationError should be raised with "plan_id must not be empty"
|
|
|
|
Scenario: Execute stub actor with empty plan_id raises ValidationError
|
|
When I invoke errcov execute stub with empty plan_id
|
|
Then an errcov ValidationError should be raised with "plan_id must not be empty"
|
|
|
|
Scenario: Run execute with wrong phase raises PlanError
|
|
Given an errcov plan in strategize phase
|
|
When I invoke errcov run_execute on wrong-phase plan
|
|
Then an errcov PlanError should be raised containing "not in Execute phase"
|
|
|
|
Scenario: Run execute with wrong state raises PlanError
|
|
Given an errcov plan in execute phase but processing state
|
|
When I invoke errcov run_execute on wrong-state plan
|
|
Then an errcov PlanError should be raised containing "not queued"
|
|
|
|
Scenario: Parse definition of done skips blank lines
|
|
When I parse errcov definition of done with blank lines
|
|
Then the parsed steps should not contain empty entries
|
|
|
|
Scenario: Parse definition of done handles numbered prefixes
|
|
When I parse errcov definition of done with numbered items "1. First\n2) Second\n3. Third"
|
|
Then the errcov parsed steps should be "First" and "Second" and "Third"
|
|
|
|
Scenario: Strategize error recording with recovery service
|
|
Given an errcov executor with error recovery service
|
|
And the errcov strategize actor raises RuntimeError "strategize failed"
|
|
When I invoke errcov run_strategize expecting failure
|
|
Then the errcov error recovery should have recorded a strategize error
|
|
|
|
Scenario: Execute retry loop exhaustion without recovery service
|
|
Given an errcov executor without error recovery service
|
|
And the errcov execute actor always fails with "total failure"
|
|
When I invoke errcov run_execute expecting failure
|
|
Then the errcov lifecycle should have called fail_execute
|
|
|
|
# -- error_recovery.py policy edge cases ---------------------------------
|
|
|
|
Scenario: Policy should_retry returns false for exhausted retriable error
|
|
Given an errcov policy with threshold 0.0 and max retries 3
|
|
And an errcov retriable error record with retry count 3 max 3
|
|
Then the errcov policy should not recommend retry
|
|
|
|
Scenario: Policy should_escalate returns true for exhausted retriable error
|
|
Given an errcov policy with threshold 0.0 and max retries 3
|
|
And an errcov retriable error record with retry count 3 max 3
|
|
Then the errcov policy should recommend escalation
|
|
|
|
Scenario: Policy format_recovery_output shows hint cli commands
|
|
Given an errcov policy with threshold 0.0 and max retries 3
|
|
And an errcov retriable error record with hints retry count 0 max 3
|
|
When I format errcov recovery output
|
|
Then the errcov recovery output should contain "agents plan"
|
|
|
|
# -- plan.py branch: actor_registry and testing_mode (647->649, 649->653)
|
|
|
|
Scenario: Plan use command with actor registry present
|
|
Given an errcov plan creation environment with actor registry
|
|
When I invoke errcov plan use with actor registry
|
|
Then the errcov actor registry ensure_built_in_actors should have been called
|
|
|
|
Scenario: Plan use command with testing mode enabled
|
|
Given an errcov plan creation environment with testing mode
|
|
When I invoke errcov plan use with testing mode
|
|
Then the errcov container actor_service should have been called
|