hurui200320
87a7ce35d7
CI / benchmark-regression (push) Has been skipped
CI / helm (push) Successful in 38s
CI / lint (push) Successful in 1m14s
CI / build (push) Successful in 1m10s
CI / push-validation (push) Successful in 49s
CI / quality (push) Successful in 1m35s
CI / typecheck (push) Successful in 1m39s
CI / security (push) Successful in 1m45s
CI / integration_tests (push) Successful in 3m44s
CI / e2e_tests (push) Successful in 4m29s
CI / unit_tests (push) Successful in 5m10s
CI / docker (push) Successful in 1m55s
CI / coverage (push) Successful in 11m8s
CI / status-check (push) Successful in 3s
CI / benchmark-publish (push) Successful in 1h20m21s
CI / benchmark-publish (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Failing after 1m35s
CI / docker (pull_request) Successful in 1m26s
CI / unit_tests (pull_request) Successful in 6m40s
CI / push-validation (pull_request) Successful in 1m24s
CI / quality (pull_request) Successful in 4m4s
CI / integration_tests (pull_request) Successful in 5m44s
CI / e2e_tests (pull_request) Failing after 6m5s
CI / helm (pull_request) Successful in 1m10s
CI / build (pull_request) Successful in 2m39s
CI / lint (pull_request) Successful in 2m58s
CI / typecheck (pull_request) Successful in 3m56s
CI / security (pull_request) Successful in 3m56s
CI / coverage (pull_request) Successful in 10m52s
CI / status-check (pull_request) Failing after 3s
Replace the M3 stub in `agents session tell` with real orchestrator actor
invocation via `SessionWorkflow.tell()`. The stub echoed a canned
`Acknowledged: ...` response without calling any LLM or actor; this
commit wires up the full pipeline:
Architecture:
- New `SessionWorkflow` (Application layer) orchestrates LLM invocation
for session tell. Accepts a `ProviderRegistry` and `ToolRegistry`;
falls back to `FakeListLLM` when no provider is configured.
- `LangChainSessionCaller` implements the `LLMCaller` protocol, building
history-aware LangChain message lists from the session conversation and
invoking the LLM via `ToolCallingRuntime.run_tool_loop()`.
- `TellResult` (Pydantic BaseModel) carries the assistant response plus
token usage (input_tokens, output_tokens, cost_usd, duration_ms).
Domain / service layer:
- `SessionActorNotConfiguredError` added to the session domain model;
raised when tell is invoked with no actor on the session and no
`--actor` override. Clear message; CLI exits with code 1.
- `SessionService.get_messages()` abstract method added (+ implementation
in `PersistentSessionService`) to load ordered message history.
A2A facade:
- `A2aLocalFacade` gains `message/send` and `message/stream` standard
A2A operation handlers that route to `SessionWorkflow.tell()`.
Total supported operations count: 42 → 44.
CLI:
- `session tell` command delegates to `_build_session_workflow()`
(patchable factory) instead of directly calling `SessionService`.
- Non-streaming: `SessionWorkflow.tell()` returns `TellResult`; output
includes a Usage panel (Rich/Plain) or a `usage` object (JSON/YAML).
- Streaming: `SessionWorkflow.tell_stream()` yields tokens; CLI prints
them via `console.print` (not raw `sys.stdout.write`).
- Token usage recorded via `SessionService.update_token_usage()` in
both paths.
Tests:
- New Behave feature `session_tell_llm.feature` (4 scenarios): real LLM
response persisted; streaming yields tokens; no-actor exits code 1;
`--actor` override resolves correctly.
- New Robot suite `session_tell_llm.robot` (4 tests): end-to-end with
stub LLM injected via monkey-patching `_resolve_llm`.
- Updated all existing `session tell` test steps to patch
`_build_session_workflow` returning a mock `TellResult` so tests
remain isolated from the LLM layer.
- Updated operation-count assertions (42 → 44) in
`a2a_cli_facade_integration`, `consolidated_misc`,
`m6_autonomy_acceptance` feature files and steps.
Cycle 7 (Review ID 8088) fixes:
- Blocker 4: Split `session_workflow.py` (was 801 lines) into two files:
`session_workflow.py` (467 lines) and `session_caller.py` (318 lines).
Extracted: `LangChainSessionCaller`, `extract_content`,
`extract_token_usage`, `estimate_cost`, `estimate_tokens`,
`history_to_langchain_messages`, and stub classes.
- Blocker 5: Route CLI streaming path through
`_facade_dispatch("message/stream", ...)` instead of directly
calling `workflow.tell_stream()`. The facade's
`_handle_message_stream` falls back to non-streaming with
`streamed: false` (acceptable for this milestone per the spec).
- Updated streaming test assertion to validate full response presence
(no longer checks token-by-token word positions, since the facade
fallback returns a complete message).
ISSUES CLOSED: #5784
314 lines
14 KiB
Gherkin
314 lines
14 KiB
Gherkin
Feature: M6 autonomy acceptance smoke tests
|
|
As a developer working with the CleverAgents M6 milestone (v3.5.0)
|
|
I want to verify the A2A facade, event queue, guard enforcement, and
|
|
automation profile resolution end-to-end
|
|
So that all v3.5.0 acceptance criteria are satisfied before milestone closure
|
|
|
|
Background:
|
|
Given a m6 smoke test runner
|
|
And a m6 smoke A2A local facade
|
|
|
|
# --- Fixture loading ---
|
|
|
|
Scenario: M6 smoke load A2A facade flows fixture
|
|
When I m6 smoke load the A2A facade flows fixture
|
|
Then the m6 smoke facade fixture should have a session lifecycle entry
|
|
And the m6 smoke facade fixture should have a plan lifecycle entry
|
|
|
|
Scenario: M6 smoke load autonomy guardrails fixture
|
|
When I m6 smoke load the autonomy guardrails fixture
|
|
Then the m6 smoke guardrails fixture should have a denylist entry
|
|
And the m6 smoke guardrails fixture should have an allowlist entry
|
|
And the m6 smoke guardrails fixture should have a cost budget entry
|
|
|
|
Scenario: M6 smoke load automation profiles fixture
|
|
When I m6 smoke load the automation profiles fixture
|
|
Then the m6 smoke profiles fixture should list all 8 built-in names
|
|
And the m6 smoke profiles fixture should have a custom profile entry
|
|
|
|
# --- A2A facade session operations (AC-1: session lifecycle) ---
|
|
|
|
Scenario: M6 smoke A2A session create returns session id
|
|
When I m6 smoke dispatch "session.create" with params {}
|
|
Then the m6 smoke response status should be "ok"
|
|
And the m6 smoke response data should contain key "session_id"
|
|
And the m6 smoke response data should contain key "status"
|
|
|
|
Scenario: M6 smoke A2A session close returns closed status
|
|
When I m6 smoke dispatch "session.close" with params {}
|
|
Then the m6 smoke response status should be "ok"
|
|
And the m6 smoke response data "status" should equal "closed"
|
|
|
|
# --- A2A facade plan operations (AC-1: plan lifecycle) ---
|
|
|
|
Scenario: M6 smoke A2A plan create returns plan id
|
|
When I m6 smoke dispatch "plan.create" with params {}
|
|
Then the m6 smoke response status should be "ok"
|
|
And the m6 smoke response data should contain key "plan_id"
|
|
|
|
Scenario: M6 smoke A2A plan execute queues plan
|
|
When I m6 smoke dispatch "plan.execute" with params {"plan_id": "01M6SM0KE00000000000000001"}
|
|
Then the m6 smoke response status should be "ok"
|
|
And the m6 smoke response data "status" should equal "queued"
|
|
|
|
Scenario: M6 smoke A2A plan status returns phase
|
|
When I m6 smoke dispatch "plan.status" with params {"plan_id": "01M6SM0KE00000000000000001"}
|
|
Then the m6 smoke response status should be "ok"
|
|
And the m6 smoke response data should contain key "phase"
|
|
|
|
Scenario: M6 smoke A2A plan diff returns changes list
|
|
When I m6 smoke dispatch "plan.diff" with params {"plan_id": "01M6SM0KE00000000000000001"}
|
|
Then the m6 smoke response status should be "ok"
|
|
And the m6 smoke response data should contain key "changes"
|
|
|
|
Scenario: M6 smoke A2A plan apply returns applied status
|
|
When I m6 smoke dispatch "plan.apply" with params {"plan_id": "01M6SM0KE00000000000000001"}
|
|
Then the m6 smoke response status should be "ok"
|
|
And the m6 smoke response data "status" should equal "applied"
|
|
|
|
# --- A2A facade registry and context operations ---
|
|
|
|
Scenario: M6 smoke A2A registry list tools returns empty list
|
|
When I m6 smoke dispatch "registry.list_tools" with params {}
|
|
Then the m6 smoke response status should be "ok"
|
|
And the m6 smoke response data should contain key "tools"
|
|
|
|
Scenario: M6 smoke A2A registry list resources returns empty list
|
|
When I m6 smoke dispatch "registry.list_resources" with params {}
|
|
Then the m6 smoke response status should be "ok"
|
|
And the m6 smoke response data should contain key "resources"
|
|
|
|
Scenario: M6 smoke A2A context get returns context dict
|
|
When I m6 smoke dispatch "context.get" with params {}
|
|
Then the m6 smoke response status should be "ok"
|
|
And the m6 smoke response data should contain key "context"
|
|
|
|
Scenario: M6 smoke A2A event subscribe returns subscription id
|
|
When I m6 smoke dispatch "event.subscribe" with params {}
|
|
Then the m6 smoke response status should be "ok"
|
|
And the m6 smoke response data should contain key "subscription_id"
|
|
|
|
# --- A2A facade error handling ---
|
|
|
|
Scenario: M6 smoke A2A unknown operation raises error
|
|
When I m6 smoke dispatch unknown operation "nonexistent.op"
|
|
Then the m6 smoke facade should raise A2aOperationNotFoundError
|
|
|
|
Scenario: M6 smoke A2A dispatch with invalid request type raises TypeError
|
|
When I m6 smoke dispatch with a non-A2aRequest object
|
|
Then the m6 smoke facade should raise TypeError
|
|
|
|
# --- A2A facade service registration ---
|
|
|
|
Scenario: M6 smoke A2A register service stores service
|
|
When I m6 smoke register service "plan_service" on the facade
|
|
Then the m6 smoke facade should have service "plan_service"
|
|
|
|
Scenario: M6 smoke A2A list operations returns all supported
|
|
When I m6 smoke list facade operations
|
|
Then the m6 smoke operations should include "session.create"
|
|
And the m6 smoke operations should include "plan.execute"
|
|
And the m6 smoke operations should include "event.subscribe"
|
|
And the m6 smoke operations count should be 44
|
|
|
|
# --- A2A event queue (AC-2: event queue publish/subscribe) ---
|
|
|
|
Scenario: M6 smoke A2A event queue publish and retrieve
|
|
Given a m6 smoke A2A event queue
|
|
When I m6 smoke publish an event with type "plan.progress"
|
|
Then the m6 smoke event queue should have 1 event
|
|
And the m6 smoke last event type should be "plan.progress"
|
|
|
|
Scenario: M6 smoke A2A event queue subscribe local callback
|
|
Given a m6 smoke A2A event queue
|
|
When I m6 smoke subscribe a local callback
|
|
And I m6 smoke publish an event with type "plan.complete"
|
|
Then the m6 smoke callback should have been called once
|
|
|
|
Scenario: M6 smoke A2A event queue unsubscribe
|
|
Given a m6 smoke A2A event queue
|
|
When I m6 smoke subscribe a local callback
|
|
And I m6 smoke unsubscribe the callback
|
|
And I m6 smoke publish an event with type "plan.complete"
|
|
Then the m6 smoke callback should not have been called
|
|
|
|
Scenario: M6 smoke A2A event queue close prevents publish
|
|
Given a m6 smoke A2A event queue
|
|
When I m6 smoke close the event queue
|
|
Then the m6 smoke publishing should raise RuntimeError
|
|
|
|
Scenario: M6 smoke A2A event queue remote subscribe raises error
|
|
Given a m6 smoke A2A event queue
|
|
When I m6 smoke attempt remote subscribe to "https://example.com/events"
|
|
Then the m6 smoke facade should raise A2aNotAvailableError
|
|
|
|
# --- A2A HTTP transport stub ---
|
|
|
|
Scenario: M6 smoke A2A transport send raises not available
|
|
When I m6 smoke attempt transport send
|
|
Then the m6 smoke facade should raise A2aNotAvailableError
|
|
|
|
Scenario: M6 smoke A2A transport connect raises not available
|
|
When I m6 smoke attempt transport connect to "https://example.com/a2a"
|
|
Then the m6 smoke facade should raise A2aNotAvailableError
|
|
|
|
Scenario: M6 smoke A2A transport disconnect raises not available
|
|
When I m6 smoke attempt transport disconnect
|
|
Then the m6 smoke facade should raise A2aNotAvailableError
|
|
|
|
Scenario: M6 smoke A2A transport is_connected returns false
|
|
When I m6 smoke check transport is_connected
|
|
Then the m6 smoke transport should not be connected
|
|
|
|
# --- A2A version negotiation ---
|
|
|
|
@tdd_issue @tdd_issue_4249 @tdd_expected_fail
|
|
Scenario: M6 smoke A2A version negotiation accepts 1.0
|
|
When I m6 smoke negotiate A2A version "1.0"
|
|
Then the m6 smoke negotiated version should be "1.0"
|
|
|
|
@tdd_issue @tdd_issue_4365 @tdd_expected_fail
|
|
Scenario: M6 smoke A2A version negotiation rejects unsupported
|
|
When I m6 smoke negotiate A2A version "2.0"
|
|
Then the m6 smoke facade should raise A2aVersionMismatchError
|
|
|
|
@tdd_issue @tdd_issue_4249 @tdd_expected_fail
|
|
Scenario: M6 smoke A2A version is_supported returns correct result
|
|
When I m6 smoke check if version "1.0" is supported
|
|
Then the m6 smoke version support should be true
|
|
When I m6 smoke check if version "99.0" is supported
|
|
Then the m6 smoke version support should be false
|
|
|
|
# --- Automation profiles built-in (AC-4: profile resolution) ---
|
|
|
|
Scenario: M6 smoke all 8 built-in profiles exist
|
|
When I m6 smoke list all built-in profiles
|
|
Then the m6 smoke profile count should be 8
|
|
And the m6 smoke profiles should include "manual"
|
|
And the m6 smoke profiles should include "full-auto"
|
|
|
|
Scenario: M6 smoke manual profile has all thresholds at 1.0
|
|
When I m6 smoke load built-in profile "manual"
|
|
Then the m6 smoke profile decompose_task should be 1.0
|
|
And the m6 smoke profile create_tool should be 1.0
|
|
And the m6 smoke profile select_tool should be 1.0
|
|
And the m6 smoke profile require_sandbox should be true
|
|
|
|
Scenario: M6 smoke full-auto profile has no gates
|
|
When I m6 smoke load built-in profile "full-auto"
|
|
Then the m6 smoke profile decompose_task should be 0.0
|
|
And the m6 smoke profile create_tool should be 0.0
|
|
And the m6 smoke profile select_tool should be 0.0
|
|
And the m6 smoke profile require_sandbox should be false
|
|
And the m6 smoke profile allow_unsafe_tools should be true
|
|
|
|
# --- Automation profile creation and validation ---
|
|
|
|
Scenario: M6 smoke create custom namespaced profile
|
|
When I m6 smoke create a profile named "acme/strict" with select_tool 1.0
|
|
Then the m6 smoke created profile name should be "acme/strict"
|
|
And the m6 smoke created profile select_tool should be 1.0
|
|
|
|
Scenario: M6 smoke profile name validation rejects invalid
|
|
When I m6 smoke create a profile with invalid name "has spaces"
|
|
Then the m6 smoke creation should raise ValueError
|
|
|
|
Scenario: M6 smoke profile threshold validation rejects out of range
|
|
When I m6 smoke create a profile with decompose_task 1.5
|
|
Then the m6 smoke creation should raise ValueError
|
|
|
|
Scenario: M6 smoke profile from_yaml loads correctly
|
|
Given a m6 smoke temporary profile YAML file
|
|
When I m6 smoke load profile from the temp YAML
|
|
Then the m6 smoke loaded profile name should be "test-yaml-profile"
|
|
|
|
# --- Guard enforcement (AC-3: denylist, budget caps, tool call limits) ---
|
|
|
|
Scenario: M6 smoke guard denylist blocks denied tool
|
|
Given a m6 smoke profile with denylist guard for "rm_rf"
|
|
When I m6 smoke check guard for tool "rm_rf"
|
|
Then the m6 smoke guard result should not be allowed
|
|
And the m6 smoke guard reason should contain "denylist"
|
|
|
|
Scenario: M6 smoke guard denylist allows non-denied tool
|
|
Given a m6 smoke profile with denylist guard for "rm_rf"
|
|
When I m6 smoke check guard for tool "read_file"
|
|
Then the m6 smoke guard result should be allowed
|
|
|
|
Scenario: M6 smoke guard allowlist blocks unlisted tool
|
|
Given a m6 smoke profile with allowlist guard for "read_file" and "search"
|
|
When I m6 smoke check guard for tool "write_file"
|
|
Then the m6 smoke guard result should not be allowed
|
|
And the m6 smoke guard reason should contain "allowlist"
|
|
|
|
Scenario: M6 smoke guard max tool calls blocks at limit
|
|
Given a m6 smoke profile with max 5 tool calls per step
|
|
When I m6 smoke check guard for tool "llm_call" with 5 calls so far
|
|
Then the m6 smoke guard result should not be allowed
|
|
And the m6 smoke guard reason should contain "limit"
|
|
|
|
Scenario: M6 smoke guard cost budget blocks at cap
|
|
Given a m6 smoke profile with max cost 10.0
|
|
When I m6 smoke check guard for tool "llm_call" with cost 10.0
|
|
Then the m6 smoke guard result should not be allowed
|
|
And the m6 smoke guard reason should contain "budget"
|
|
|
|
Scenario: M6 smoke guard write approval blocks write operations
|
|
Given a m6 smoke profile with write approval required
|
|
When I m6 smoke check guard for tool "write_file" as a write operation
|
|
Then the m6 smoke guard result should not be allowed
|
|
And the m6 smoke guard reason should contain "Write operations"
|
|
|
|
Scenario: M6 smoke guard apply approval blocks apply phase
|
|
Given a m6 smoke profile with apply approval required
|
|
When I m6 smoke check guard for tool "__apply__"
|
|
Then the m6 smoke guard result should not be allowed
|
|
And the m6 smoke guard reason should contain "Apply phase"
|
|
|
|
Scenario: M6 smoke guard with no guards allows everything
|
|
Given a m6 smoke profile with no guards
|
|
When I m6 smoke check guard for tool "anything"
|
|
Then the m6 smoke guard result should be allowed
|
|
|
|
# --- Profile service resolution precedence (AC-4: plan > action > global) ---
|
|
|
|
Scenario: M6 smoke profile resolution plan takes precedence
|
|
Given a m6 smoke automation profile service
|
|
When I m6 smoke resolve profile with plan "ci" action "auto" project "manual"
|
|
Then the m6 smoke resolved profile name should be "ci"
|
|
|
|
Scenario: M6 smoke profile resolution action takes precedence over project
|
|
Given a m6 smoke automation profile service
|
|
When I m6 smoke resolve profile with plan null action "auto" project "manual"
|
|
Then the m6 smoke resolved profile name should be "auto"
|
|
|
|
Scenario: M6 smoke profile resolution falls back to global default
|
|
Given a m6 smoke automation profile service
|
|
When I m6 smoke resolve profile with plan null action null project null
|
|
Then the m6 smoke resolved profile name should be "manual"
|
|
|
|
# --- Profile service guard evaluation ---
|
|
|
|
Scenario: M6 smoke service evaluate guard delegates to profile
|
|
Given a m6 smoke automation profile service
|
|
When I m6 smoke evaluate guard for profile "manual" and tool "read_file"
|
|
Then the m6 smoke guard result should be allowed
|
|
|
|
# --- A2A model validation ---
|
|
|
|
Scenario: M6 smoke A2aRequest validates non-empty operation
|
|
When I m6 smoke create A2aRequest with empty operation
|
|
Then the m6 smoke creation should raise ValueError
|
|
|
|
Scenario: M6 smoke A2aResponse validates status values
|
|
When I m6 smoke create A2aResponse with invalid status "maybe"
|
|
Then the m6 smoke creation should raise ValueError
|
|
|
|
Scenario: M6 smoke A2aEvent validates non-empty event_type
|
|
When I m6 smoke create A2aEvent with empty event_type
|
|
Then the m6 smoke creation should raise ValueError
|
|
|
|
Scenario: M6 smoke A2aErrorDetail validates non-empty fields
|
|
When I m6 smoke create A2aErrorDetail with empty code
|
|
Then the m6 smoke creation should raise ValueError
|