forked from cleveragents/cleveragents-core
224 lines
11 KiB
Gherkin
224 lines
11 KiB
Gherkin
Feature: Semantic Escalation with Confidence Scoring
|
|
As an automation framework
|
|
I want to compute confidence scores and compare them against profile thresholds
|
|
So that the system can decide whether to proceed automatically or escalate to the user
|
|
|
|
Background:
|
|
Given a default autonomy controller
|
|
|
|
# ---- Confidence computation ----
|
|
|
|
Scenario: Confidence with all factors at maximum
|
|
When I compute confidence with past_success_rate 1.0 codebase_familiarity 1.0 risk_assessment 0.0 invariant_complexity 0.0
|
|
Then the confidence score should be 1.0
|
|
|
|
Scenario: Confidence with all factors at minimum
|
|
When I compute confidence with past_success_rate 0.0 codebase_familiarity 0.0 risk_assessment 1.0 invariant_complexity 1.0
|
|
Then the confidence score should be 0.0
|
|
|
|
Scenario: Confidence with neutral factors
|
|
When I compute confidence with past_success_rate 0.5 codebase_familiarity 0.5 risk_assessment 0.5 invariant_complexity 0.5
|
|
Then the confidence score should be 0.5
|
|
|
|
Scenario: Confidence with mixed high and low factors
|
|
When I compute confidence with past_success_rate 0.9 codebase_familiarity 0.8 risk_assessment 0.2 invariant_complexity 0.1
|
|
Then the confidence score should be approximately 0.84
|
|
|
|
Scenario: Risk inversion in confidence computation
|
|
When I compute confidence with past_success_rate 0.5 codebase_familiarity 0.5 risk_assessment 1.0 invariant_complexity 0.5
|
|
Then the confidence score should be less than 0.5
|
|
|
|
Scenario: Invariant complexity inversion in confidence computation
|
|
When I compute confidence with past_success_rate 0.5 codebase_familiarity 0.5 risk_assessment 0.5 invariant_complexity 1.0
|
|
Then the confidence score should be less than 0.5
|
|
|
|
# ---- Threshold comparison for each of the 8 profiles ----
|
|
|
|
Scenario Outline: Threshold comparison for built-in profile <profile_name>
|
|
Given the automation profile "<profile_name>"
|
|
When I evaluate escalation for operation "auto_execute" with confidence factors past_success_rate 0.85 codebase_familiarity 0.9 risk_assessment 0.1 invariant_complexity 0.1
|
|
Then the escalation decision should match the "<profile_name>" profile threshold for "auto_execute"
|
|
|
|
Examples:
|
|
| profile_name |
|
|
| manual |
|
|
| review |
|
|
| supervised |
|
|
| cautious |
|
|
| trusted |
|
|
| auto |
|
|
| ci |
|
|
| full-auto |
|
|
|
|
Scenario: Threshold 0.0 always proceeds automatically
|
|
Given a profile with auto_execute threshold 0.0
|
|
When I evaluate escalation for operation "auto_execute" with confidence factors past_success_rate 0.0 codebase_familiarity 0.0 risk_assessment 1.0 invariant_complexity 1.0
|
|
Then the escalation decision proceed should be true
|
|
|
|
Scenario: Threshold 1.0 escalates with realistic confidence
|
|
Given a profile with auto_execute threshold 1.0
|
|
When I evaluate escalation for operation "auto_execute" with confidence factors past_success_rate 0.95 codebase_familiarity 0.95 risk_assessment 0.05 invariant_complexity 0.05
|
|
Then the escalation decision proceed should be false
|
|
|
|
Scenario: Exactly at threshold proceeds
|
|
Given a profile with auto_execute threshold 0.5
|
|
When I evaluate escalation for operation "auto_execute" with confidence factors past_success_rate 0.5 codebase_familiarity 0.5 risk_assessment 0.5 invariant_complexity 0.5
|
|
Then the escalation decision proceed should be true
|
|
|
|
Scenario: Just below threshold escalates
|
|
Given a profile with auto_execute threshold 0.51
|
|
When I evaluate escalation for operation "auto_execute" with confidence factors past_success_rate 0.5 codebase_familiarity 0.5 risk_assessment 0.5 invariant_complexity 0.5
|
|
Then the escalation decision proceed should be false
|
|
|
|
# ---- Historical success tracking ----
|
|
|
|
Scenario: Historical success rate starts at neutral
|
|
Then the historical success rate for "auto_execute" should be 0.5
|
|
|
|
Scenario: Recording successes increases success rate
|
|
When I record 8 successes and 2 failures for "auto_execute"
|
|
Then the historical success rate for "auto_execute" should be 0.8
|
|
|
|
Scenario: Recording all failures gives zero success rate
|
|
When I record 0 successes and 5 failures for "auto_execute"
|
|
Then the historical success rate for "auto_execute" should be 0.0
|
|
|
|
Scenario: Recording all successes gives perfect success rate
|
|
When I record 10 successes and 0 failures for "auto_execute"
|
|
Then the historical success rate for "auto_execute" should be 1.0
|
|
|
|
Scenario: Success rate evolves with new outcomes
|
|
When I record 5 successes and 5 failures for "auto_execute"
|
|
Then the historical success rate for "auto_execute" should be 0.5
|
|
When I record 5 additional successes for "auto_execute"
|
|
Then the historical success rate for "auto_execute" should be approximately 0.667
|
|
|
|
Scenario: Clearing history resets to neutral
|
|
When I record 10 successes and 0 failures for "auto_execute"
|
|
And I clear history for "auto_execute"
|
|
Then the historical success rate for "auto_execute" should be 0.5
|
|
|
|
Scenario: History count tracks recorded outcomes
|
|
When I record 3 successes and 2 failures for "auto_execute"
|
|
Then the history count for "auto_execute" should be 5
|
|
|
|
# ---- Escalation explanation generation ----
|
|
|
|
Scenario: Proceeding decision includes explanation
|
|
Given a profile with auto_execute threshold 0.3
|
|
When I evaluate escalation for operation "auto_execute" with confidence factors past_success_rate 0.9 codebase_familiarity 0.9 risk_assessment 0.1 invariant_complexity 0.1
|
|
Then the escalation explanation should contain "Proceeding automatically"
|
|
And the escalation explanation should contain "auto_execute"
|
|
|
|
Scenario: Escalating decision includes explanation
|
|
Given a profile with auto_execute threshold 0.99
|
|
When I evaluate escalation for operation "auto_execute" with confidence factors past_success_rate 0.5 codebase_familiarity 0.5 risk_assessment 0.5 invariant_complexity 0.5
|
|
Then the escalation explanation should contain "Escalating to user"
|
|
And the escalation explanation should contain "auto_execute"
|
|
|
|
# ---- Edge cases ----
|
|
|
|
Scenario: Zero confidence with zero threshold proceeds
|
|
Given a profile with auto_execute threshold 0.0
|
|
When I evaluate escalation for operation "auto_execute" with confidence factors past_success_rate 0.0 codebase_familiarity 0.0 risk_assessment 1.0 invariant_complexity 1.0
|
|
Then the escalation decision proceed should be true
|
|
And the confidence should be 0.0
|
|
|
|
Scenario: Perfect confidence with threshold below 1.0 proceeds
|
|
Given a profile with auto_execute threshold 0.99
|
|
When I evaluate escalation for operation "auto_execute" with confidence factors past_success_rate 1.0 codebase_familiarity 1.0 risk_assessment 0.0 invariant_complexity 0.0
|
|
Then the escalation decision proceed should be true
|
|
And the confidence should be 1.0
|
|
|
|
Scenario: Unknown operation type defaults to manual threshold
|
|
Given the automation profile "full-auto"
|
|
When I evaluate escalation for operation "nonexistent_operation" with confidence factors past_success_rate 0.95 codebase_familiarity 0.95 risk_assessment 0.05 invariant_complexity 0.05
|
|
Then the escalation threshold should be 1.0
|
|
And the escalation decision proceed should be false
|
|
|
|
# ---- Custom weights ----
|
|
|
|
Scenario: Custom weights are validated
|
|
When I try to create a controller with invalid weights
|
|
Then a weight validation error should be raised
|
|
|
|
Scenario: Custom equal weights produce different score
|
|
Given a controller with equal weights
|
|
When I compute confidence with past_success_rate 1.0 codebase_familiarity 0.0 risk_assessment 0.0 invariant_complexity 0.0
|
|
Then the confidence score should be 0.75
|
|
|
|
# ---- Factor model validation ----
|
|
|
|
Scenario: ConfidenceFactors rejects out-of-range values
|
|
When I try to create confidence factors with past_success_rate 1.5
|
|
Then a factor validation error should be raised
|
|
|
|
Scenario: OperationContext requires non-empty operation type
|
|
When I try to create an operation context with empty operation type
|
|
Then an operation context validation error should be raised
|
|
|
|
# ---- EscalationDecision model ----
|
|
|
|
Scenario: EscalationDecision captures all fields
|
|
Given a profile with auto_execute threshold 0.5
|
|
When I evaluate escalation for operation "auto_execute" with confidence factors past_success_rate 0.8 codebase_familiarity 0.7 risk_assessment 0.2 invariant_complexity 0.3
|
|
Then the escalation decision should have operation_type "auto_execute"
|
|
And the escalation decision should have 4 factors
|
|
And the escalation decision threshold should be 0.5
|
|
|
|
# ---- Argument validation coverage ----
|
|
|
|
Scenario: should_proceed_automatically rejects None operation
|
|
When I call should_proceed_automatically with None operation
|
|
Then a weight validation error should be raised
|
|
|
|
Scenario: should_proceed_automatically rejects None factors
|
|
When I call should_proceed_automatically with None factors
|
|
Then a weight validation error should be raised
|
|
|
|
Scenario: should_proceed_automatically rejects None profile
|
|
When I call should_proceed_automatically with None profile
|
|
Then a weight validation error should be raised
|
|
|
|
Scenario: compute_confidence rejects None factors
|
|
When I call compute_confidence with None factors
|
|
Then a weight validation error should be raised
|
|
|
|
Scenario: record_outcome rejects None outcome
|
|
When I call record_outcome with None outcome
|
|
Then a weight validation error should be raised
|
|
|
|
Scenario: get_historical_success_rate rejects empty operation type
|
|
When I call get_historical_success_rate with empty string
|
|
Then a weight validation error should be raised
|
|
|
|
Scenario: get_history_count rejects empty operation type
|
|
When I call get_history_count with empty string
|
|
Then a weight validation error should be raised
|
|
|
|
Scenario: Clearing all history resets everything
|
|
When I record 5 successes and 0 failures for "auto_execute"
|
|
And I record 5 successes and 0 failures for "auto_apply"
|
|
And I clear all history
|
|
Then the historical success rate for "auto_execute" should be 0.5
|
|
And the historical success rate for "auto_apply" should be 0.5
|
|
|
|
Scenario: Weights property returns current weights
|
|
Then the controller weights should have 4 entries
|
|
|
|
Scenario: History eviction at max capacity
|
|
When I record max history outcomes for "auto_execute"
|
|
Then the history count for "auto_execute" should be 10000
|
|
|
|
Scenario: Controller rejects extra weight keys
|
|
When I try to create a controller with extra weight keys
|
|
Then a weight validation error should be raised
|
|
|
|
Scenario: Controller rejects negative weight
|
|
When I try to create a controller with negative weight
|
|
Then a weight validation error should be raised
|
|
|
|
Scenario: Controller rejects weights not summing to 1
|
|
When I try to create a controller with weights not summing to 1
|
|
Then a weight validation error should be raised
|