diff --git a/features/m6_autonomy_acceptance.feature b/features/m6_autonomy_acceptance.feature new file mode 100644 index 000000000..ef7d225a0 --- /dev/null +++ b/features/m6_autonomy_acceptance.feature @@ -0,0 +1,310 @@ +Feature: M6 autonomy acceptance smoke tests + As a developer working with the CleverAgents M6 milestone (v3.5.0) + I want to verify the A2A facade, event queue, guard enforcement, and + automation profile resolution end-to-end + So that all v3.5.0 acceptance criteria are satisfied before milestone closure + + Background: + Given a m6 smoke test runner + And a m6 smoke A2A local facade + + # --- Fixture loading --- + + Scenario: M6 smoke load A2A facade flows fixture + When I m6 smoke load the A2A facade flows fixture + Then the m6 smoke facade fixture should have a session lifecycle entry + And the m6 smoke facade fixture should have a plan lifecycle entry + + Scenario: M6 smoke load autonomy guardrails fixture + When I m6 smoke load the autonomy guardrails fixture + Then the m6 smoke guardrails fixture should have a denylist entry + And the m6 smoke guardrails fixture should have an allowlist entry + And the m6 smoke guardrails fixture should have a cost budget entry + + Scenario: M6 smoke load automation profiles fixture + When I m6 smoke load the automation profiles fixture + Then the m6 smoke profiles fixture should list all 8 built-in names + And the m6 smoke profiles fixture should have a custom profile entry + + # --- A2A facade session operations (AC-1: session lifecycle) --- + + Scenario: M6 smoke A2A session create returns session id + When I m6 smoke dispatch "session.create" with params {} + Then the m6 smoke response status should be "ok" + And the m6 smoke response data should contain key "session_id" + And the m6 smoke response data should contain key "status" + + Scenario: M6 smoke A2A session close returns closed status + When I m6 smoke dispatch "session.close" with params {} + Then the m6 smoke response status should be "ok" + And the m6 smoke response data "status" should equal "closed" + + # --- A2A facade plan operations (AC-1: plan lifecycle) --- + + Scenario: M6 smoke A2A plan create returns plan id + When I m6 smoke dispatch "plan.create" with params {} + Then the m6 smoke response status should be "ok" + And the m6 smoke response data should contain key "plan_id" + + Scenario: M6 smoke A2A plan execute queues plan + When I m6 smoke dispatch "plan.execute" with params {"plan_id": "01M6SM0KE00000000000000001"} + Then the m6 smoke response status should be "ok" + And the m6 smoke response data "status" should equal "queued" + + Scenario: M6 smoke A2A plan status returns phase + When I m6 smoke dispatch "plan.status" with params {"plan_id": "01M6SM0KE00000000000000001"} + Then the m6 smoke response status should be "ok" + And the m6 smoke response data should contain key "phase" + + Scenario: M6 smoke A2A plan diff returns changes list + When I m6 smoke dispatch "plan.diff" with params {"plan_id": "01M6SM0KE00000000000000001"} + Then the m6 smoke response status should be "ok" + And the m6 smoke response data should contain key "changes" + + Scenario: M6 smoke A2A plan apply returns applied status + When I m6 smoke dispatch "plan.apply" with params {"plan_id": "01M6SM0KE00000000000000001"} + Then the m6 smoke response status should be "ok" + And the m6 smoke response data "status" should equal "applied" + + # --- A2A facade registry and context operations --- + + Scenario: M6 smoke A2A registry list tools returns empty list + When I m6 smoke dispatch "registry.list_tools" with params {} + Then the m6 smoke response status should be "ok" + And the m6 smoke response data should contain key "tools" + + Scenario: M6 smoke A2A registry list resources returns empty list + When I m6 smoke dispatch "registry.list_resources" with params {} + Then the m6 smoke response status should be "ok" + And the m6 smoke response data should contain key "resources" + + Scenario: M6 smoke A2A context get returns context dict + When I m6 smoke dispatch "context.get" with params {} + Then the m6 smoke response status should be "ok" + And the m6 smoke response data should contain key "context" + + Scenario: M6 smoke A2A event subscribe returns subscription id + When I m6 smoke dispatch "event.subscribe" with params {} + Then the m6 smoke response status should be "ok" + And the m6 smoke response data should contain key "subscription_id" + + # --- A2A facade error handling --- + + Scenario: M6 smoke A2A unknown operation raises error + When I m6 smoke dispatch unknown operation "nonexistent.op" + Then the m6 smoke facade should raise A2aOperationNotFoundError + + Scenario: M6 smoke A2A dispatch with invalid request type raises TypeError + When I m6 smoke dispatch with a non-A2aRequest object + Then the m6 smoke facade should raise TypeError + + # --- A2A facade service registration --- + + Scenario: M6 smoke A2A register service stores service + When I m6 smoke register service "plan_service" on the facade + Then the m6 smoke facade should have service "plan_service" + + Scenario: M6 smoke A2A list operations returns all supported + When I m6 smoke list facade operations + Then the m6 smoke operations should include "session.create" + And the m6 smoke operations should include "plan.execute" + And the m6 smoke operations should include "event.subscribe" + And the m6 smoke operations count should be 42 + + # --- A2A event queue (AC-2: event queue publish/subscribe) --- + + Scenario: M6 smoke A2A event queue publish and retrieve + Given a m6 smoke A2A event queue + When I m6 smoke publish an event with type "plan.progress" + Then the m6 smoke event queue should have 1 event + And the m6 smoke last event type should be "plan.progress" + + Scenario: M6 smoke A2A event queue subscribe local callback + Given a m6 smoke A2A event queue + When I m6 smoke subscribe a local callback + And I m6 smoke publish an event with type "plan.complete" + Then the m6 smoke callback should have been called once + + Scenario: M6 smoke A2A event queue unsubscribe + Given a m6 smoke A2A event queue + When I m6 smoke subscribe a local callback + And I m6 smoke unsubscribe the callback + And I m6 smoke publish an event with type "plan.complete" + Then the m6 smoke callback should not have been called + + Scenario: M6 smoke A2A event queue close prevents publish + Given a m6 smoke A2A event queue + When I m6 smoke close the event queue + Then the m6 smoke publishing should raise RuntimeError + + Scenario: M6 smoke A2A event queue remote subscribe raises error + Given a m6 smoke A2A event queue + When I m6 smoke attempt remote subscribe to "https://example.com/events" + Then the m6 smoke facade should raise A2aNotAvailableError + + # --- A2A HTTP transport stub --- + + Scenario: M6 smoke A2A transport send raises not available + When I m6 smoke attempt transport send + Then the m6 smoke facade should raise A2aNotAvailableError + + Scenario: M6 smoke A2A transport connect raises not available + When I m6 smoke attempt transport connect to "https://example.com/a2a" + Then the m6 smoke facade should raise A2aNotAvailableError + + Scenario: M6 smoke A2A transport disconnect raises not available + When I m6 smoke attempt transport disconnect + Then the m6 smoke facade should raise A2aNotAvailableError + + Scenario: M6 smoke A2A transport is_connected returns false + When I m6 smoke check transport is_connected + Then the m6 smoke transport should not be connected + + # --- A2A version negotiation --- + + Scenario: M6 smoke A2A version negotiation accepts 1.0 + When I m6 smoke negotiate A2A version "1.0" + Then the m6 smoke negotiated version should be "1.0" + + Scenario: M6 smoke A2A version negotiation rejects unsupported + When I m6 smoke negotiate A2A version "2.0" + Then the m6 smoke facade should raise A2aVersionMismatchError + + Scenario: M6 smoke A2A version is_supported returns correct result + When I m6 smoke check if version "1.0" is supported + Then the m6 smoke version support should be true + When I m6 smoke check if version "99.0" is supported + Then the m6 smoke version support should be false + + # --- Automation profiles built-in (AC-4: profile resolution) --- + + Scenario: M6 smoke all 8 built-in profiles exist + When I m6 smoke list all built-in profiles + Then the m6 smoke profile count should be 8 + And the m6 smoke profiles should include "manual" + And the m6 smoke profiles should include "full-auto" + + Scenario: M6 smoke manual profile has all thresholds at 1.0 + When I m6 smoke load built-in profile "manual" + Then the m6 smoke profile decompose_task should be 1.0 + And the m6 smoke profile create_tool should be 1.0 + And the m6 smoke profile select_tool should be 1.0 + And the m6 smoke profile require_sandbox should be true + + Scenario: M6 smoke full-auto profile has no gates + When I m6 smoke load built-in profile "full-auto" + Then the m6 smoke profile decompose_task should be 0.0 + And the m6 smoke profile create_tool should be 0.0 + And the m6 smoke profile select_tool should be 0.0 + And the m6 smoke profile require_sandbox should be false + And the m6 smoke profile allow_unsafe_tools should be true + + # --- Automation profile creation and validation --- + + Scenario: M6 smoke create custom namespaced profile + When I m6 smoke create a profile named "acme/strict" with select_tool 1.0 + Then the m6 smoke created profile name should be "acme/strict" + And the m6 smoke created profile select_tool should be 1.0 + + Scenario: M6 smoke profile name validation rejects invalid + When I m6 smoke create a profile with invalid name "has spaces" + Then the m6 smoke creation should raise ValueError + + Scenario: M6 smoke profile threshold validation rejects out of range + When I m6 smoke create a profile with decompose_task 1.5 + Then the m6 smoke creation should raise ValueError + + Scenario: M6 smoke profile from_yaml loads correctly + Given a m6 smoke temporary profile YAML file + When I m6 smoke load profile from the temp YAML + Then the m6 smoke loaded profile name should be "test-yaml-profile" + + # --- Guard enforcement (AC-3: denylist, budget caps, tool call limits) --- + + Scenario: M6 smoke guard denylist blocks denied tool + Given a m6 smoke profile with denylist guard for "rm_rf" + When I m6 smoke check guard for tool "rm_rf" + Then the m6 smoke guard result should not be allowed + And the m6 smoke guard reason should contain "denylist" + + Scenario: M6 smoke guard denylist allows non-denied tool + Given a m6 smoke profile with denylist guard for "rm_rf" + When I m6 smoke check guard for tool "read_file" + Then the m6 smoke guard result should be allowed + + Scenario: M6 smoke guard allowlist blocks unlisted tool + Given a m6 smoke profile with allowlist guard for "read_file" and "search" + When I m6 smoke check guard for tool "write_file" + Then the m6 smoke guard result should not be allowed + And the m6 smoke guard reason should contain "allowlist" + + Scenario: M6 smoke guard max tool calls blocks at limit + Given a m6 smoke profile with max 5 tool calls per step + When I m6 smoke check guard for tool "llm_call" with 5 calls so far + Then the m6 smoke guard result should not be allowed + And the m6 smoke guard reason should contain "limit" + + Scenario: M6 smoke guard cost budget blocks at cap + Given a m6 smoke profile with max cost 10.0 + When I m6 smoke check guard for tool "llm_call" with cost 10.0 + Then the m6 smoke guard result should not be allowed + And the m6 smoke guard reason should contain "budget" + + Scenario: M6 smoke guard write approval blocks write operations + Given a m6 smoke profile with write approval required + When I m6 smoke check guard for tool "write_file" as a write operation + Then the m6 smoke guard result should not be allowed + And the m6 smoke guard reason should contain "Write operations" + + Scenario: M6 smoke guard apply approval blocks apply phase + Given a m6 smoke profile with apply approval required + When I m6 smoke check guard for tool "__apply__" + Then the m6 smoke guard result should not be allowed + And the m6 smoke guard reason should contain "Apply phase" + + Scenario: M6 smoke guard with no guards allows everything + Given a m6 smoke profile with no guards + When I m6 smoke check guard for tool "anything" + Then the m6 smoke guard result should be allowed + + # --- Profile service resolution precedence (AC-4: plan > action > global) --- + + Scenario: M6 smoke profile resolution plan takes precedence + Given a m6 smoke automation profile service + When I m6 smoke resolve profile with plan "ci" action "auto" project "manual" + Then the m6 smoke resolved profile name should be "ci" + + Scenario: M6 smoke profile resolution action takes precedence over project + Given a m6 smoke automation profile service + When I m6 smoke resolve profile with plan null action "auto" project "manual" + Then the m6 smoke resolved profile name should be "auto" + + Scenario: M6 smoke profile resolution falls back to global default + Given a m6 smoke automation profile service + When I m6 smoke resolve profile with plan null action null project null + Then the m6 smoke resolved profile name should be "manual" + + # --- Profile service guard evaluation --- + + Scenario: M6 smoke service evaluate guard delegates to profile + Given a m6 smoke automation profile service + When I m6 smoke evaluate guard for profile "manual" and tool "read_file" + Then the m6 smoke guard result should be allowed + + # --- A2A model validation --- + + Scenario: M6 smoke A2aRequest validates non-empty operation + When I m6 smoke create A2aRequest with empty operation + Then the m6 smoke creation should raise ValueError + + Scenario: M6 smoke A2aResponse validates status values + When I m6 smoke create A2aResponse with invalid status "maybe" + Then the m6 smoke creation should raise ValueError + + Scenario: M6 smoke A2aEvent validates non-empty event_type + When I m6 smoke create A2aEvent with empty event_type + Then the m6 smoke creation should raise ValueError + + Scenario: M6 smoke A2aErrorDetail validates non-empty fields + When I m6 smoke create A2aErrorDetail with empty code + Then the m6 smoke creation should raise ValueError