CoreRasurae
b41f536da6
CI / helm (push) Successful in 47s
CI / build (push) Successful in 1m11s
CI / push-validation (push) Successful in 40s
CI / benchmark-publish (push) Has started running
CI / lint (push) Successful in 1m26s
CI / typecheck (push) Successful in 1m55s
CI / e2e_tests (push) Successful in 1m12s
CI / quality (push) Successful in 2m6s
CI / security (push) Successful in 2m9s
CI / benchmark-regression (push) Failing after 1m28s
CI / integration_tests (push) Successful in 3m48s
CI / unit_tests (push) Successful in 7m31s
CI / coverage (push) Has started running
CI / docker (push) Successful in 1m35s
CI / helm (pull_request) Successful in 39s
CI / push-validation (pull_request) Successful in 39s
CI / build (pull_request) Successful in 1m13s
CI / lint (pull_request) Successful in 1m32s
CI / quality (pull_request) Successful in 1m49s
CI / typecheck (pull_request) Successful in 1m48s
CI / security (pull_request) Successful in 1m58s
CI / status-check (push) Has been cancelled
CI / integration_tests (pull_request) Successful in 4m48s
CI / unit_tests (pull_request) Successful in 5m11s
CI / coverage (pull_request) Has started running
CI / docker (pull_request) Successful in 1m44s
CI / status-check (pull_request) Has been cancelled
Add BDD regression tests for issue #4328 automation profile gates Tests verify that complete_strategize() and complete_execute() respect automation profile thresholds and do NOT unconditionally call auto_progress(). Covers 8 built-in profiles: manual, full-auto, supervised, auto, review_before_apply, ci, trusted ISSUES CLOSED: #4328
128 lines
7.2 KiB
Gherkin
128 lines
7.2 KiB
Gherkin
@tdd_issue @tdd_issue_4328
|
|
Feature: Automation Profile Gates - Issue #4328 Regression Tests
|
|
Regression tests for manual automation profile being disrespected during phase
|
|
completion. Verifies that complete_strategize() and complete_execute() respect
|
|
automation profile gates and do NOT auto-progress when thresholds are 1.0.
|
|
|
|
Background:
|
|
Given I have a plan lifecycle service with automation level support
|
|
|
|
# ========================================================================
|
|
# Issue #4328: Manual profile (all thresholds = 1.0) should NOT auto-progress
|
|
# These scenarios verify the fix - that complete_strategize/execute do NOT
|
|
# unconditionally call auto_progress()
|
|
# ========================================================================
|
|
|
|
Scenario: Manual profile create_tool=1.0 blocks strategize to execute transition
|
|
Given I have a plan with automation level "manual" in strategize phase with processing state
|
|
When I complete strategize on the automated plan
|
|
Then the automated plan phase should be "strategize"
|
|
And the automated plan processing state should be "complete"
|
|
And should_auto_progress should return false
|
|
|
|
Scenario: Manual profile select_tool=1.0 blocks execute to apply transition
|
|
Given I have a plan with automation level "manual" in execute phase with processing state
|
|
When I complete execute on the automated plan
|
|
Then the automated plan phase should be "execute"
|
|
And the automated plan processing state should be "complete"
|
|
And should_auto_progress should return false
|
|
|
|
# ========================================================================
|
|
# Verify auto_progress() IS called when should_auto_progress() returns true
|
|
# ========================================================================
|
|
|
|
Scenario: Full-auto profile create_tool=0.0 allows strategize to execute auto-progress
|
|
Given I have a plan with automation level "full_automation" in strategize phase with processing state
|
|
When I complete strategize on the automated plan
|
|
Then the automated plan phase should be "execute"
|
|
And the automated plan processing state should be "queued"
|
|
|
|
Scenario: Full-auto profile select_tool=0.0 allows execute to apply auto-progress
|
|
Given I have a plan with automation level "full_automation" in execute phase with processing state
|
|
When I complete execute on the automated plan
|
|
Then the automated plan phase should be "apply"
|
|
And the automated plan processing state should be "applied"
|
|
|
|
# ========================================================================
|
|
# Supervised profile: create_tool=1.0 (blocks execute), select_tool=1.0 (blocks apply)
|
|
# ========================================================================
|
|
|
|
Scenario: Supervised profile create_tool=1.0 blocks strategize to execute auto-progress
|
|
Given I have a plan with automation level "supervised" in strategize phase with processing state
|
|
When I complete strategize on the automated plan
|
|
Then the automated plan phase should be "strategize"
|
|
And the automated plan processing state should be "complete"
|
|
And should_auto_progress should return false
|
|
|
|
Scenario: Supervised profile select_tool=1.0 blocks execute to apply transition
|
|
Given I have a plan with automation level "supervised" in execute phase with processing state
|
|
When I complete execute on the automated plan
|
|
Then the automated plan phase should be "execute"
|
|
And the automated plan processing state should be "complete"
|
|
And should_auto_progress should return false
|
|
|
|
# ========================================================================
|
|
# Review-before-apply (auto) profile: create_tool=0.0, select_tool=1.0
|
|
# ========================================================================
|
|
|
|
Scenario: Auto profile create_tool=0.0 allows strategize to execute auto-progress
|
|
Given I have a plan with automation level "review_before_apply" in strategize phase with processing state
|
|
When I complete strategize on the automated plan
|
|
Then the automated plan phase should be "execute"
|
|
And the automated plan processing state should be "queued"
|
|
|
|
Scenario: Auto profile select_tool=1.0 blocks execute to apply transition
|
|
Given I have a plan with automation level "review_before_apply" in execute phase with processing state
|
|
When I complete execute on the automated plan
|
|
Then the automated plan phase should be "execute"
|
|
And the automated plan processing state should be "complete"
|
|
And should_auto_progress should return false
|
|
|
|
# ========================================================================
|
|
# CI profile: all thresholds=0.0 (auto-progresses everything)
|
|
# ========================================================================
|
|
|
|
Scenario: CI profile blocks neither strategize nor execute transitions
|
|
Given I have a plan with automation level "ci" in strategize phase with processing state
|
|
When I complete strategize on the automated plan
|
|
Then the automated plan phase should be "execute"
|
|
And the automated plan processing state should be "queued"
|
|
|
|
Scenario: CI profile execute phase auto-progresses to apply
|
|
Given I have a plan with automation level "ci" in execute phase with processing state
|
|
When I complete execute on the automated plan
|
|
Then the automated plan phase should be "apply"
|
|
And the automated plan processing state should be "applied"
|
|
|
|
# ========================================================================
|
|
# Trusted profile: create_tool=0.0, select_tool=1.0 (same as auto for strategize/execute)
|
|
# ========================================================================
|
|
|
|
Scenario: Trusted profile allows strategize to execute auto-progress
|
|
Given I have a plan with automation level "trusted" in strategize phase with processing state
|
|
When I complete strategize on the automated plan
|
|
Then the automated plan phase should be "execute"
|
|
And the automated plan processing state should be "queued"
|
|
|
|
Scenario: Trusted profile select_tool=1.0 blocks execute to apply transition
|
|
Given I have a plan with automation level "trusted" in execute phase with processing state
|
|
When I complete execute on the automated plan
|
|
Then the automated plan phase should be "execute"
|
|
And the automated plan processing state should be "complete"
|
|
And should_auto_progress should return false
|
|
|
|
# ========================================================================
|
|
# Cautious profile: intermediate thresholds (0.0 < v < 1.0)
|
|
# Tests the spec's Threshold Semantics for probabilistic gates:
|
|
# "0.0 < v < 1.0 → system may proceed if confidence >= threshold"
|
|
# With default confidence=0.5, thresholds > 0.5 should block.
|
|
# ========================================================================
|
|
|
|
Scenario: Cautious profile create_tool=0.7 blocks strategize to execute with default confidence
|
|
Given I have a plan with automation level "cautious" in strategize phase with complete state for query
|
|
Then should_auto_progress should return false
|
|
|
|
Scenario: Cautious profile select_tool=1.0 blocks execute to apply transition
|
|
Given I have a plan with automation level "cautious" in execute phase with complete state for query
|
|
Then should_auto_progress should return false
|