Feature: Semantic Escalation with Confidence Scoring As an automation framework I want to compute confidence scores and compare them against profile thresholds So that the system can decide whether to proceed automatically or escalate to the user Background: Given a default autonomy controller # ---- Confidence computation ---- Scenario: Confidence with all factors at maximum When I compute confidence with past_success_rate 1.0 codebase_familiarity 1.0 risk_assessment 0.0 invariant_complexity 0.0 Then the confidence score should be 1.0 Scenario: Confidence with all factors at minimum When I compute confidence with past_success_rate 0.0 codebase_familiarity 0.0 risk_assessment 1.0 invariant_complexity 1.0 Then the confidence score should be 0.0 Scenario: Confidence with neutral factors When I compute confidence with past_success_rate 0.5 codebase_familiarity 0.5 risk_assessment 0.5 invariant_complexity 0.5 Then the confidence score should be 0.5 Scenario: Confidence with mixed high and low factors When I compute confidence with past_success_rate 0.9 codebase_familiarity 0.8 risk_assessment 0.2 invariant_complexity 0.1 Then the confidence score should be approximately 0.84 Scenario: Risk inversion in confidence computation When I compute confidence with past_success_rate 0.5 codebase_familiarity 0.5 risk_assessment 1.0 invariant_complexity 0.5 Then the confidence score should be less than 0.5 Scenario: Invariant complexity inversion in confidence computation When I compute confidence with past_success_rate 0.5 codebase_familiarity 0.5 risk_assessment 0.5 invariant_complexity 1.0 Then the confidence score should be less than 0.5 # ---- Threshold comparison for each of the 8 profiles ---- Scenario Outline: Threshold comparison for built-in profile Given the automation profile "" When I evaluate escalation for operation "create_tool" with confidence factors past_success_rate 0.85 codebase_familiarity 0.9 risk_assessment 0.1 invariant_complexity 0.1 Then the escalation decision should match the "" profile threshold for "create_tool" Examples: | profile_name | | manual | | review | | supervised | | cautious | | trusted | | auto | | ci | | full-auto | Scenario: Threshold 0.0 always proceeds automatically Given a profile with create_tool threshold 0.0 When I evaluate escalation for operation "create_tool" with confidence factors past_success_rate 0.0 codebase_familiarity 0.0 risk_assessment 1.0 invariant_complexity 1.0 Then the escalation decision proceed should be true Scenario: Threshold 1.0 escalates with realistic confidence Given a profile with create_tool threshold 1.0 When I evaluate escalation for operation "create_tool" with confidence factors past_success_rate 0.95 codebase_familiarity 0.95 risk_assessment 0.05 invariant_complexity 0.05 Then the escalation decision proceed should be false Scenario: Exactly at threshold proceeds Given a profile with create_tool threshold 0.5 When I evaluate escalation for operation "create_tool" with confidence factors past_success_rate 0.5 codebase_familiarity 0.5 risk_assessment 0.5 invariant_complexity 0.5 Then the escalation decision proceed should be true Scenario: Just below threshold escalates Given a profile with create_tool threshold 0.51 When I evaluate escalation for operation "create_tool" with confidence factors past_success_rate 0.5 codebase_familiarity 0.5 risk_assessment 0.5 invariant_complexity 0.5 Then the escalation decision proceed should be false # ---- Historical success tracking ---- Scenario: Historical success rate starts at neutral Then the historical success rate for "create_tool" should be 0.5 Scenario: Recording successes increases success rate When I record 8 successes and 2 failures for "create_tool" Then the historical success rate for "create_tool" should be 0.8 Scenario: Recording all failures gives zero success rate When I record 0 successes and 5 failures for "create_tool" Then the historical success rate for "create_tool" should be 0.0 Scenario: Recording all successes gives perfect success rate When I record 10 successes and 0 failures for "create_tool" Then the historical success rate for "create_tool" should be 1.0 Scenario: Success rate evolves with new outcomes When I record 5 successes and 5 failures for "create_tool" Then the historical success rate for "create_tool" should be 0.5 When I record 5 additional successes for "create_tool" Then the historical success rate for "create_tool" should be approximately 0.667 Scenario: Clearing history resets to neutral When I record 10 successes and 0 failures for "create_tool" And I clear history for "create_tool" Then the historical success rate for "create_tool" should be 0.5 Scenario: History count tracks recorded outcomes When I record 3 successes and 2 failures for "create_tool" Then the history count for "create_tool" should be 5 # ---- Escalation explanation generation ---- Scenario: Proceeding decision includes explanation Given a profile with create_tool threshold 0.3 When I evaluate escalation for operation "create_tool" with confidence factors past_success_rate 0.9 codebase_familiarity 0.9 risk_assessment 0.1 invariant_complexity 0.1 Then the escalation explanation should contain "Proceeding automatically" And the escalation explanation should contain "create_tool" Scenario: Escalating decision includes explanation Given a profile with create_tool threshold 0.99 When I evaluate escalation for operation "create_tool" with confidence factors past_success_rate 0.5 codebase_familiarity 0.5 risk_assessment 0.5 invariant_complexity 0.5 Then the escalation explanation should contain "Escalating to user" And the escalation explanation should contain "create_tool" # ---- Edge cases ---- Scenario: Zero confidence with zero threshold proceeds Given a profile with create_tool threshold 0.0 When I evaluate escalation for operation "create_tool" with confidence factors past_success_rate 0.0 codebase_familiarity 0.0 risk_assessment 1.0 invariant_complexity 1.0 Then the escalation decision proceed should be true And the confidence should be 0.0 Scenario: Perfect confidence with threshold below 1.0 proceeds Given a profile with create_tool threshold 0.99 When I evaluate escalation for operation "create_tool" with confidence factors past_success_rate 1.0 codebase_familiarity 1.0 risk_assessment 0.0 invariant_complexity 0.0 Then the escalation decision proceed should be true And the confidence should be 1.0 Scenario: Unknown operation type defaults to manual threshold Given the automation profile "full-auto" When I evaluate escalation for operation "nonexistent_operation" with confidence factors past_success_rate 0.95 codebase_familiarity 0.95 risk_assessment 0.05 invariant_complexity 0.05 Then the escalation threshold should be 1.0 And the escalation decision proceed should be false # ---- Custom weights ---- Scenario: Custom weights are validated When I try to create a controller with invalid weights Then a weight validation error should be raised Scenario: Custom equal weights produce different score Given a controller with equal weights When I compute confidence with past_success_rate 1.0 codebase_familiarity 0.0 risk_assessment 0.0 invariant_complexity 0.0 Then the confidence score should be 0.75 # ---- Factor model validation ---- Scenario: ConfidenceFactors rejects out-of-range values When I try to create confidence factors with past_success_rate 1.5 Then a factor validation error should be raised Scenario: OperationContext requires non-empty operation type When I try to create an operation context with empty operation type Then an operation context validation error should be raised # ---- EscalationDecision model ---- Scenario: EscalationDecision captures all fields Given a profile with create_tool threshold 0.5 When I evaluate escalation for operation "create_tool" with confidence factors past_success_rate 0.8 codebase_familiarity 0.7 risk_assessment 0.2 invariant_complexity 0.3 Then the escalation decision should have operation_type "create_tool" And the escalation decision should have 4 factors And the escalation decision threshold should be 0.5 # ---- Argument validation coverage ---- Scenario: should_proceed_automatically rejects None operation When I call should_proceed_automatically with None operation Then a weight validation error should be raised Scenario: should_proceed_automatically rejects None factors When I call should_proceed_automatically with None factors Then a weight validation error should be raised Scenario: should_proceed_automatically rejects None profile When I call should_proceed_automatically with None profile Then a weight validation error should be raised Scenario: compute_confidence rejects None factors When I call compute_confidence with None factors Then a weight validation error should be raised Scenario: record_outcome rejects None outcome When I call record_outcome with None outcome Then a weight validation error should be raised Scenario: get_historical_success_rate rejects empty operation type When I call get_historical_success_rate with empty string Then a weight validation error should be raised Scenario: get_history_count rejects empty operation type When I call get_history_count with empty string Then a weight validation error should be raised Scenario: Clearing all history resets everything When I record 5 successes and 0 failures for "create_tool" And I record 5 successes and 0 failures for "select_tool" And I clear all history Then the historical success rate for "create_tool" should be 0.5 And the historical success rate for "select_tool" should be 0.5 Scenario: Weights property returns current weights Then the controller weights should have 4 entries Scenario: History eviction at max capacity When I record max history outcomes for "create_tool" Then the history count for "create_tool" should be 10000 Scenario: Controller rejects extra weight keys When I try to create a controller with extra weight keys Then a weight validation error should be raised Scenario: Controller rejects negative weight When I try to create a controller with negative weight Then a weight validation error should be raised Scenario: Controller rejects weights not summing to 1 When I try to create a controller with weights not summing to 1 Then a weight validation error should be raised