cleveragents-core/features/semantic_escalation.feature

Feature: Semantic Escalation with Confidence Scoring
  As an automation framework
  I want to compute confidence scores and compare them against profile thresholds
  So that the system can decide whether to proceed automatically or escalate to the user

  Background:
    Given a default autonomy controller

  # ---- Confidence computation ----

  Scenario: Confidence with all factors at maximum
    When I compute confidence with past_success_rate 1.0 codebase_familiarity 1.0 risk_assessment 0.0 invariant_complexity 0.0
    Then the confidence score should be 1.0

  Scenario: Confidence with all factors at minimum
    When I compute confidence with past_success_rate 0.0 codebase_familiarity 0.0 risk_assessment 1.0 invariant_complexity 1.0
    Then the confidence score should be 0.0

  Scenario: Confidence with neutral factors
    When I compute confidence with past_success_rate 0.5 codebase_familiarity 0.5 risk_assessment 0.5 invariant_complexity 0.5
    Then the confidence score should be 0.5

  Scenario: Confidence with mixed high and low factors
    When I compute confidence with past_success_rate 0.9 codebase_familiarity 0.8 risk_assessment 0.2 invariant_complexity 0.1
    Then the confidence score should be approximately 0.84

  Scenario: Risk inversion in confidence computation
    When I compute confidence with past_success_rate 0.5 codebase_familiarity 0.5 risk_assessment 1.0 invariant_complexity 0.5
    Then the confidence score should be less than 0.5

  Scenario: Invariant complexity inversion in confidence computation
    When I compute confidence with past_success_rate 0.5 codebase_familiarity 0.5 risk_assessment 0.5 invariant_complexity 1.0
    Then the confidence score should be less than 0.5

  # ---- Threshold comparison for each of the 8 profiles ----

  Scenario Outline: Threshold comparison for built-in profile <profile_name>
    Given the automation profile "<profile_name>"
    When I evaluate escalation for operation "create_tool" with confidence factors past_success_rate 0.85 codebase_familiarity 0.9 risk_assessment 0.1 invariant_complexity 0.1
    Then the escalation decision should match the "<profile_name>" profile threshold for "create_tool"

    Examples:
      | profile_name |
      | manual       |
      | review       |
      | supervised   |
      | cautious     |
      | trusted      |
      | auto         |
      | ci           |
      | full-auto    |

  Scenario: Threshold 0.0 always proceeds automatically
    Given a profile with create_tool threshold 0.0
    When I evaluate escalation for operation "create_tool" with confidence factors past_success_rate 0.0 codebase_familiarity 0.0 risk_assessment 1.0 invariant_complexity 1.0
    Then the escalation decision proceed should be true

  Scenario: Threshold 1.0 escalates with realistic confidence
    Given a profile with create_tool threshold 1.0
    When I evaluate escalation for operation "create_tool" with confidence factors past_success_rate 0.95 codebase_familiarity 0.95 risk_assessment 0.05 invariant_complexity 0.05
    Then the escalation decision proceed should be false

  Scenario: Exactly at threshold proceeds
    Given a profile with create_tool threshold 0.5
    When I evaluate escalation for operation "create_tool" with confidence factors past_success_rate 0.5 codebase_familiarity 0.5 risk_assessment 0.5 invariant_complexity 0.5
    Then the escalation decision proceed should be true

  Scenario: Just below threshold escalates
    Given a profile with create_tool threshold 0.51
    When I evaluate escalation for operation "create_tool" with confidence factors past_success_rate 0.5 codebase_familiarity 0.5 risk_assessment 0.5 invariant_complexity 0.5
    Then the escalation decision proceed should be false

  # ---- Historical success tracking ----

  Scenario: Historical success rate starts at neutral
    Then the historical success rate for "create_tool" should be 0.5

  Scenario: Recording successes increases success rate
    When I record 8 successes and 2 failures for "create_tool"
    Then the historical success rate for "create_tool" should be 0.8

  Scenario: Recording all failures gives zero success rate
    When I record 0 successes and 5 failures for "create_tool"
    Then the historical success rate for "create_tool" should be 0.0

  Scenario: Recording all successes gives perfect success rate
    When I record 10 successes and 0 failures for "create_tool"
    Then the historical success rate for "create_tool" should be 1.0

  Scenario: Success rate evolves with new outcomes
    When I record 5 successes and 5 failures for "create_tool"
    Then the historical success rate for "create_tool" should be 0.5
    When I record 5 additional successes for "create_tool"
    Then the historical success rate for "create_tool" should be approximately 0.667

  Scenario: Clearing history resets to neutral
    When I record 10 successes and 0 failures for "create_tool"
    And I clear history for "create_tool"
    Then the historical success rate for "create_tool" should be 0.5

  Scenario: History count tracks recorded outcomes
    When I record 3 successes and 2 failures for "create_tool"
    Then the history count for "create_tool" should be 5

  # ---- Escalation explanation generation ----

  Scenario: Proceeding decision includes explanation
    Given a profile with create_tool threshold 0.3
    When I evaluate escalation for operation "create_tool" with confidence factors past_success_rate 0.9 codebase_familiarity 0.9 risk_assessment 0.1 invariant_complexity 0.1
    Then the escalation explanation should contain "Proceeding automatically"
    And the escalation explanation should contain "create_tool"

  Scenario: Escalating decision includes explanation
    Given a profile with create_tool threshold 0.99
    When I evaluate escalation for operation "create_tool" with confidence factors past_success_rate 0.5 codebase_familiarity 0.5 risk_assessment 0.5 invariant_complexity 0.5
    Then the escalation explanation should contain "Escalating to user"
    And the escalation explanation should contain "create_tool"

  # ---- Edge cases ----

  Scenario: Zero confidence with zero threshold proceeds
    Given a profile with create_tool threshold 0.0
    When I evaluate escalation for operation "create_tool" with confidence factors past_success_rate 0.0 codebase_familiarity 0.0 risk_assessment 1.0 invariant_complexity 1.0
    Then the escalation decision proceed should be true
    And the confidence should be 0.0

  Scenario: Perfect confidence with threshold below 1.0 proceeds
    Given a profile with create_tool threshold 0.99
    When I evaluate escalation for operation "create_tool" with confidence factors past_success_rate 1.0 codebase_familiarity 1.0 risk_assessment 0.0 invariant_complexity 0.0
    Then the escalation decision proceed should be true
    And the confidence should be 1.0

  Scenario: Unknown operation type defaults to manual threshold
    Given the automation profile "full-auto"
    When I evaluate escalation for operation "nonexistent_operation" with confidence factors past_success_rate 0.95 codebase_familiarity 0.95 risk_assessment 0.05 invariant_complexity 0.05
    Then the escalation threshold should be 1.0
    And the escalation decision proceed should be false

  # ---- Custom weights ----

  Scenario: Custom weights are validated
    When I try to create a controller with invalid weights
    Then a weight validation error should be raised

  Scenario: Custom equal weights produce different score
    Given a controller with equal weights
    When I compute confidence with past_success_rate 1.0 codebase_familiarity 0.0 risk_assessment 0.0 invariant_complexity 0.0
    Then the confidence score should be 0.75

  # ---- Factor model validation ----

  Scenario: ConfidenceFactors rejects out-of-range values
    When I try to create confidence factors with past_success_rate 1.5
    Then a factor validation error should be raised

  Scenario: OperationContext requires non-empty operation type
    When I try to create an operation context with empty operation type
    Then an operation context validation error should be raised

  # ---- EscalationDecision model ----

  Scenario: EscalationDecision captures all fields
    Given a profile with create_tool threshold 0.5
    When I evaluate escalation for operation "create_tool" with confidence factors past_success_rate 0.8 codebase_familiarity 0.7 risk_assessment 0.2 invariant_complexity 0.3
    Then the escalation decision should have operation_type "create_tool"
    And the escalation decision should have 4 factors
    And the escalation decision threshold should be 0.5

  # ---- Argument validation coverage ----

  Scenario: should_proceed_automatically rejects None operation
    When I call should_proceed_automatically with None operation
    Then a weight validation error should be raised

  Scenario: should_proceed_automatically rejects None factors
    When I call should_proceed_automatically with None factors
    Then a weight validation error should be raised

  Scenario: should_proceed_automatically rejects None profile
    When I call should_proceed_automatically with None profile
    Then a weight validation error should be raised

  Scenario: compute_confidence rejects None factors
    When I call compute_confidence with None factors
    Then a weight validation error should be raised

  Scenario: record_outcome rejects None outcome
    When I call record_outcome with None outcome
    Then a weight validation error should be raised

  Scenario: get_historical_success_rate rejects empty operation type
    When I call get_historical_success_rate with empty string
    Then a weight validation error should be raised

  Scenario: get_history_count rejects empty operation type
    When I call get_history_count with empty string
    Then a weight validation error should be raised

  Scenario: Clearing all history resets everything
    When I record 5 successes and 0 failures for "create_tool"
    And I record 5 successes and 0 failures for "select_tool"
    And I clear all history
    Then the historical success rate for "create_tool" should be 0.5
    And the historical success rate for "select_tool" should be 0.5

  Scenario: Weights property returns current weights
    Then the controller weights should have 4 entries

  Scenario: History eviction at max capacity
    When I record max history outcomes for "create_tool"
    Then the history count for "create_tool" should be 10000

  Scenario: Controller rejects extra weight keys
    When I try to create a controller with extra weight keys
    Then a weight validation error should be raised

  Scenario: Controller rejects negative weight
    When I try to create a controller with negative weight
    Then a weight validation error should be raised

  Scenario: Controller rejects weights not summing to 1
    When I try to create a controller with weights not summing to 1
    Then a weight validation error should be raised