Compare commits
6 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| b692894c88 | |||
| 2f01e7d625 | |||
| 63746b4d30 | |||
| 05acc26c40 | |||
| a1ea40d2e7 | |||
| 87a7ce35d7 |
@@ -5,6 +5,17 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
- **`agents session tell` invokes real LLM orchestrator actor** (#5784): Replaced the
|
||||
M3 echo-stub with real actor invocation via `SessionWorkflow`, routing through
|
||||
`LangChainSessionCaller` → `ToolCallingRuntime.run_tool_loop()`. The user prompt
|
||||
is sent to the session's bound orchestrator actor (or `--actor` override), the
|
||||
assistant's response is persisted via `SessionService.append_message()`, and token
|
||||
usage is tracked via `SessionService.update_token_usage()`. Output includes a
|
||||
**Usage** panel (Rich/Plain) or `usage` object (JSON/YAML) with input tokens,
|
||||
output tokens, estimated cost, and duration. The `--stream` flag produces real
|
||||
LLM streaming output. A `SessionActorNotConfiguredError` is raised with exit
|
||||
code 1 when no actor is configured.
|
||||
|
||||
- Fixed `ReactiveEventBus.emit()` exception handler to log the full exception
|
||||
message (`str(exc)`) and enable traceback forwarding (`exc_info=True`).
|
||||
Previously the handler logged only the exception type name (e.g.
|
||||
@@ -13,6 +24,10 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
traceback in the structlog warning entry. Removed `@tdd_expected_fail` tag
|
||||
from the TDD test so both scenarios run as normal regression guards. (#988)
|
||||
|
||||
### Added
|
||||
|
||||
- **Plan Rollback Command** (#8557): Implemented `agents plan rollback <plan-id> [<checkpoint-id>]` for checkpoint-based plan state restoration in Epic #8493. The command restores a plan's sandbox to the state captured at a given checkpoint, discarding all decisions made after that checkpoint. The checkpoint can be specified as an optional positional second argument or via the `--to-checkpoint` named option. Supports `--yes/-y` flag to skip confirmation prompts and `--format/-f` for output format selection (rich/plain/json/yaml). Included with comprehensive BDD test coverage (>= 97%) and spec-aligned output formatting showing rollback summary, changes reverted, impact analysis, and post-rollback state panels.
|
||||
|
||||
### Fixed
|
||||
- **Actor configuration validation incorrectly requires top-level provider field** (#4300):
|
||||
Actor configuration in V3 is now obtained from the nested configuration
|
||||
@@ -129,6 +144,11 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
|
||||
### Security
|
||||
|
||||
- **PyYAML upgraded to >=6.0.3 to address known security vulnerability** (#9055):
|
||||
Added an explicit `pyyaml>=6.0.3` dependency constraint to `pyproject.toml` to
|
||||
prevent installation of vulnerable older versions with known YAML parsing security
|
||||
issues.
|
||||
|
||||
- **aiohttp upgraded to >=3.13.4 to remediate CVE-2026-34513 and CVE-2026-34515** (#1549, #1544):
|
||||
Added an explicit `aiohttp>=3.13.4` dependency constraint to `pyproject.toml` to remediate
|
||||
two high-severity open redirect vulnerabilities. Both CVEs affect the CleverAgents platform's
|
||||
|
||||
@@ -39,3 +39,5 @@ Below are some of the specific details of various contributions.
|
||||
* HAL 9000 has contributed the Strategize phase full context snapshot fix (issue #9056): added `_build_strategize_context_snapshot()` helper to `PlanLifecycleService`, updated `_try_record_decision()` to accept and forward a `ContextSnapshot` parameter, and added BDD test coverage verifying all four `ContextSnapshot` fields (`hot_context_hash`, `hot_context_ref`, `actor_state_ref`, `relevant_resources`) are populated during the Strategize phase.
|
||||
* HAL 9000 has contributed the ACMS context path matching fix (PR #10975 / issue #10972): corrects `_path_matches()` and `_matches_pattern()` to properly match absolute fragment paths against relative glob patterns by auto-prefixing with `**/` before calling `PurePath.full_match()`, preventing silent inefficacy of include/exclude filters for absolute paths in fragment metadata.
|
||||
* HAL 9000 has contributed database resource types (PostgreSQL, SQLite) with transaction-based sandbox strategy: implemented ``DatabaseResourceHandler`` providing full CRUD operations (`read`, `write`, `delete`, `list_children`) and connection validation with automatic credential masking for PostgreSQL and SQLite backends. Includes ``TransactionSandbox`` infrastructure wired into ``SandboxFactory``, BDD test coverage in ``features/database_resources.feature``, and Robot Framework integration tests in ``robot/database_resources.robot`` (PR #10591 / issue #8608, Epic #8568).
|
||||
* HAL 9000 has contributed the agents plan rollback command (PR #8674 / issue #8557): implemented checkpoint-based plan state restoration with the `agents plan rollback <plan-id> [<checkpoint-id>]` CLI command as part of Epic #8493, enabling plans to be restored to previous checkpoints, discarding post-checkpoint decisions, and resuming execution from the rolled-back state. Supported by `--yes/-y`, `--to-checkpoint`, and `--format/-f` flags. Includes comprehensive BDD test coverage (>= 97%) for rollback, decision discarding, and plan resume functionality.
|
||||
* HAL 9000 has contributed the PyYAML security upgrade (PR #11012 / issue #9055): added `pyyaml>=6.0.3` dependency constraint to address known YAML parsing vulnerabilities.
|
||||
|
||||
@@ -140,7 +140,7 @@ Feature: Consolidated Misc
|
||||
And the operations list should contain "registry.list_tools"
|
||||
And the operations list should contain "context.get"
|
||||
And the operations list should contain "event.subscribe"
|
||||
And the operations list should have 42 items
|
||||
And the operations list should have 44 items
|
||||
|
||||
# -----------------------------------------------------------------------
|
||||
# A2aHttpTransport — all stubs raise A2aNotAvailableError
|
||||
@@ -639,7 +639,7 @@ Feature: Consolidated Misc
|
||||
Then the m6 smoke operations should include "session.create"
|
||||
And the m6 smoke operations should include "plan.execute"
|
||||
And the m6 smoke operations should include "event.subscribe"
|
||||
And the m6 smoke operations count should be 42
|
||||
And the m6 smoke operations count should be 44
|
||||
|
||||
# --- A2A event queue ---
|
||||
|
||||
|
||||
@@ -109,7 +109,7 @@ Feature: M6 autonomy acceptance smoke tests
|
||||
Then the m6 smoke operations should include "session.create"
|
||||
And the m6 smoke operations should include "plan.execute"
|
||||
And the m6 smoke operations should include "event.subscribe"
|
||||
And the m6 smoke operations count should be 42
|
||||
And the m6 smoke operations count should be 44
|
||||
|
||||
# --- A2A event queue (AC-2: event queue publish/subscribe) ---
|
||||
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
@tdd_issue @tdd_issue_9055
|
||||
Feature: PyYAML is a declared project dependency with minimum secure version
|
||||
As a CleverAgents developer
|
||||
I want the PyYAML dependency to be listed in pyproject.toml with a minimum secure version
|
||||
So that vulnerable older versions of PyYAML cannot be installed
|
||||
|
||||
Background:
|
||||
Given the pyproject.toml file exists at "pyproject.toml"
|
||||
|
||||
@tdd_issue @tdd_issue_9055
|
||||
Scenario: PyYAML is listed in project dependencies with minimum version constraint
|
||||
When I read the project dependencies from pyproject.toml
|
||||
Then the dependency list should include a package matching "pyyaml>="
|
||||
|
||||
@tdd_issue @tdd_issue_9055
|
||||
Scenario: PyYAML minimum version is >=6.0.3
|
||||
When I read the PyYAML dependency specification from pyproject.toml
|
||||
Then the minimum version should be at least "6.0.3"
|
||||
|
||||
@tdd_issue @tdd_issue_9055
|
||||
Scenario: yaml module is importable as a project dependency
|
||||
When I attempt to import the "yaml" module
|
||||
Then the import should succeed without errors
|
||||
@@ -0,0 +1,58 @@
|
||||
Feature: Real LLM actor invocation in session tell
|
||||
As a user of CleverAgents
|
||||
I want `agents session tell` to invoke the real orchestrator actor
|
||||
So that I get meaningful AI responses instead of stub acknowledgements
|
||||
|
||||
Background:
|
||||
Given a session tell LLM mock environment
|
||||
|
||||
@session_tell_llm
|
||||
Scenario: Real LLM response is returned and persisted
|
||||
Given a session with actor "openai/gpt-4" exists
|
||||
When I invoke session tell with prompt "What can you do?"
|
||||
Then the tell command returns the LLM response
|
||||
And the assistant message is persisted to the session
|
||||
And token usage is recorded
|
||||
|
||||
@session_tell_llm
|
||||
Scenario: --stream flag yields incremental tokens
|
||||
Given a session with actor "openai/gpt-4" exists
|
||||
When I invoke session tell with --stream and prompt "Hello"
|
||||
Then the streamed output contains the LLM response tokens
|
||||
And the assistant message is persisted after streaming
|
||||
|
||||
@session_tell_llm
|
||||
Scenario: No actor configured raises clear error with exit code 1
|
||||
Given a session with no actor exists
|
||||
When I invoke session tell with prompt "Hello"
|
||||
Then the tell command exits with code 1
|
||||
And the error output mentions actor configuration
|
||||
|
||||
@session_tell_llm
|
||||
Scenario: --actor flag overrides the session's bound actor
|
||||
Given a session with actor "anthropic/claude-3-haiku" exists
|
||||
When I invoke session tell with --actor "openai/gpt-4" and prompt "Hello"
|
||||
Then the tell command uses the override actor "openai/gpt-4"
|
||||
And the tell command returns the LLM response
|
||||
|
||||
@session_tell_llm
|
||||
Scenario: --format json output includes usage object
|
||||
Given a session with actor "openai/gpt-4" exists
|
||||
When I invoke session tell with --format json and prompt "What can you do?"
|
||||
Then the tell output is valid JSON with a data envelope
|
||||
And the data section contains the session_id
|
||||
And the data section contains a usage object with expected keys
|
||||
|
||||
@session_tell_llm
|
||||
Scenario: --stream --format json produces valid JSON with usage
|
||||
Given a session with actor "openai/gpt-4" exists
|
||||
When I invoke session tell with --stream --format json and prompt "Hello"
|
||||
Then the tell output is valid JSON with a data envelope
|
||||
And the data section contains a usage object with expected keys
|
||||
|
||||
@session_tell_llm
|
||||
Scenario: Usage panel appears in Rich output
|
||||
Given a session with actor "openai/gpt-4" exists
|
||||
When I invoke session tell with prompt "What can you do?"
|
||||
Then the tell command returns the LLM response
|
||||
And the output contains a Usage panel
|
||||
@@ -0,0 +1,105 @@
|
||||
Feature: Session workflow coverage boost
|
||||
Coverage boost for session_workflow.py helper functions (M1, n4).
|
||||
|
||||
Background:
|
||||
Given the session workflow coverage environment is set up
|
||||
|
||||
# _extract_content
|
||||
Scenario: _extract_content extracts text from content attribute
|
||||
Given coverage boost a mock response with content "hello world"
|
||||
When coverage boost _extract_content is called
|
||||
Then coverage boost the extracted result should be "hello world"
|
||||
|
||||
Scenario: _extract_content falls back to text attribute
|
||||
Given coverage boost a mock response with text "from text attr" and no content
|
||||
When coverage boost _extract_content is called
|
||||
Then coverage boost the extracted result should be "from text attr"
|
||||
|
||||
Scenario: _extract_content handles list content
|
||||
Given coverage boost a mock response with list content containing text dicts and plain strings
|
||||
When coverage boost _extract_content is called
|
||||
Then coverage boost the result should contain the concatenated texts
|
||||
|
||||
Scenario: _extract_content falls back to str for unknown types
|
||||
Given coverage boost a mock response with no content or text attribute
|
||||
When coverage boost _extract_content is called
|
||||
Then coverage boost the result should be a string
|
||||
|
||||
# _extract_token_usage
|
||||
Scenario: _extract_token_usage reads from response_metadata.usage
|
||||
Given coverage boost a mock response with response_metadata usage input_tokens=100 output_tokens=50
|
||||
When coverage boost _extract_token_usage is called
|
||||
Then coverage boost input tokens should be 100 and output tokens should be 50
|
||||
|
||||
Scenario: _extract_token_usage reads from response_metadata.token_usage
|
||||
Given coverage boost a mock response with response_metadata token_usage input_tokens=200 output_tokens=100
|
||||
When coverage boost _extract_token_usage is called
|
||||
Then coverage boost input tokens should be 200 and output tokens should be 100
|
||||
|
||||
Scenario: _extract_token_usage reads prompt_tokens and completion_tokens
|
||||
Given coverage boost a mock response with response_metadata usage prompt_tokens=300 completion_tokens=150
|
||||
When coverage boost _extract_token_usage is called
|
||||
Then coverage boost input tokens should be 300 and output tokens should be 150
|
||||
|
||||
Scenario: _extract_token_usage reads from usage_metadata
|
||||
Given coverage boost a mock response with usage_metadata input_tokens=400 output_tokens=200
|
||||
When coverage boost _extract_token_usage is called
|
||||
Then coverage boost input tokens should be 400 and output tokens should be 200
|
||||
|
||||
Scenario: _extract_token_usage returns zeros for no metadata
|
||||
Given coverage boost a mock response with no usage metadata
|
||||
When coverage boost _extract_token_usage is called
|
||||
Then coverage boost input tokens should be 0 and output tokens should be 0
|
||||
|
||||
# _estimate_cost
|
||||
Scenario: _estimate_cost computes cost from token counts
|
||||
Given coverage boost input tokens 1000 and output tokens 500
|
||||
When coverage boost _estimate_cost is called
|
||||
Then coverage boost the estimated cost should be positive
|
||||
|
||||
# _history_to_langchain_messages
|
||||
Scenario: _history_to_langchain_messages converts all roles
|
||||
Given coverage boost session messages with roles SYSTEM, USER, ASSISTANT, and TOOL
|
||||
When coverage boost _history_to_langchain_messages is called
|
||||
Then coverage boost the result should contain SystemMessage, HumanMessage, AIMessage, and ToolMessage
|
||||
|
||||
Scenario: _history_to_langchain_messages treats unknown role as human
|
||||
Given coverage boost a session message with an unknown role
|
||||
When coverage boost _history_to_langchain_messages is called
|
||||
Then coverage boost the result should contain a HumanMessage
|
||||
|
||||
Scenario: _history_to_langchain_messages handles empty list
|
||||
Given coverage boost an empty list of session messages
|
||||
When coverage boost _history_to_langchain_messages is called
|
||||
Then coverage boost the result should be an empty list
|
||||
|
||||
# LangChainSessionCaller.invoke() tool_results branch
|
||||
Scenario: LangChainSessionCaller.invoke appends tool results
|
||||
Given coverage boost a LangChainSessionCaller with a stub LLM and empty history
|
||||
When coverage boost invoke is called with tool_results containing one success and one failure
|
||||
Then coverage boost the accumulated messages should include tool result messages
|
||||
|
||||
# LangChainSessionCaller.invoke() with tool_calls in response
|
||||
Scenario: LangChainSessionCaller.invoke extracts tool calls from response
|
||||
Given coverage boost a LangChainSessionCaller with a stub LLM that returns tool calls
|
||||
When coverage boost invoke is called for the first time
|
||||
Then coverage boost the LLMResponse should contain the extracted tool calls
|
||||
|
||||
# _build_lc_messages_from_history
|
||||
Scenario: _build_lc_messages_from_history adds system prompt when absent
|
||||
Given coverage boost a SessionWorkflow with a stub service and no registry
|
||||
And coverage boost session history without a system message
|
||||
When coverage boost _build_lc_messages_from_history is called with a prompt
|
||||
Then coverage boost the first message should be a SystemMessage with the session system prompt
|
||||
|
||||
# _MinimalStubLLM
|
||||
Scenario: _MinimalStubLLM.invoke returns stub response
|
||||
Given coverage boost a _MinimalStubLLM instance
|
||||
When coverage boost invoke on the stub is called
|
||||
Then coverage boost the stub response content should be "(no LLM configured)"
|
||||
And coverage boost the stub response should have empty tool_calls
|
||||
|
||||
Scenario: _MinimalStubLLM.stream yields stub chunk
|
||||
Given coverage boost a _MinimalStubLLM instance
|
||||
When coverage boost stream on the stub is called
|
||||
Then coverage boost it should yield a chunk with content "(no LLM configured)"
|
||||
@@ -127,7 +127,8 @@ def step_call_notify_facade(context: Any, operation: str) -> None:
|
||||
@then("the facade should support all 42 operations")
|
||||
def step_check_42_operations(context: Any) -> None:
|
||||
ops = context.facade.list_operations()
|
||||
assert len(ops) == 42, f"Expected 42 operations, got {len(ops)}: {ops}"
|
||||
# 42 original ops + 2 standard A2A ops (message/send, message/stream) = 44
|
||||
assert len(ops) == 44, f"Expected 44 operations, got {len(ops)}: {ops}"
|
||||
|
||||
|
||||
@then("the facade should be cached on subsequent calls")
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
"""Step definitions for PyYAML security dependency verification (#9055).
|
||||
|
||||
Note: the shared steps for reading pyproject.toml and verifying importability
|
||||
are defined in ``tdd_a2a_sdk_dependency_steps.py`` to avoid AmbiguousStep
|
||||
conflicts. This file contains only the PyYAML-specific step definitions.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from behave import then, when
|
||||
from behave.runner import Context
|
||||
from packaging.version import parse as parse_version
|
||||
|
||||
|
||||
@then('the dependency list should include a package matching "{pattern}"')
|
||||
def step_dependency_matches_pattern(context: Context, pattern: str) -> None:
|
||||
"""Assert that a dependency in the project dependencies matches the given regex pattern."""
|
||||
deps: list[str] = context.project_dependencies
|
||||
found = any(re.search(pattern, dep) for dep in deps)
|
||||
assert found, (
|
||||
f"No dependency matching '{pattern}' found in "
|
||||
f"[project.dependencies]. Current deps: {deps}"
|
||||
)
|
||||
|
||||
|
||||
@when("I read the PyYAML dependency specification from pyproject.toml")
|
||||
def step_read_pyyaml_dependency(context: Context) -> None:
|
||||
"""Read and parse the PyYAML dependency from pyproject.toml."""
|
||||
content = context.pyproject_path.read_bytes()
|
||||
data: dict[str, Any] = _load_toml(content)
|
||||
deps: list[str] = data.get("project", {}).get("dependencies", [])
|
||||
|
||||
# Find the pyyaml dependency entry
|
||||
pyyaml_dep = None
|
||||
for dep in deps:
|
||||
if dep.strip().startswith("pyyaml"):
|
||||
pyyaml_dep = dep.strip()
|
||||
break
|
||||
|
||||
assert pyyaml_dep is not None, (
|
||||
f"PyYAML dependency not found in [project.dependencies]. Current deps: {deps}"
|
||||
)
|
||||
|
||||
# Parse the minimum version from the constraint
|
||||
# Examples: "pyyaml>=6.0.3", "pyyaml>=6.0", "pyyaml==6.0.2"
|
||||
match = re.search(r"(?:>=|==|~>)([\d]+(?:\.[\d]+)*(?:\.\w+)*)", pyyaml_dep)
|
||||
assert match is not None, (
|
||||
f"Could not parse version from PyYAML dependency: {pyyaml_dep}"
|
||||
)
|
||||
|
||||
context.pyyaml_version_spec = pyyaml_dep
|
||||
context.pyyaml_min_version = match.group(1)
|
||||
|
||||
|
||||
@then('the minimum version should be at least "{expected}"')
|
||||
def step_pyyaml_minimum_version(context: Context, expected: str) -> None:
|
||||
"""Assert that the PyYAML minimum version is >= expected version."""
|
||||
min_version = context.pyyaml_min_version
|
||||
assert parse_version(min_version) >= parse_version(expected), (
|
||||
f"PyYAML minimum version {min_version} is less than required {expected}. "
|
||||
f"Dependency spec: {context.pyyaml_version_spec}"
|
||||
)
|
||||
|
||||
|
||||
def _flatten_toml_dict(d: dict[Any, Any]) -> dict[str, Any]:
|
||||
"""Recursively convert a tomlkit dict to a plain Python dict.
|
||||
|
||||
tomlkit wraps every value in a proxy object that behaves like the
|
||||
underlying Python type but is not identical to it. This helper
|
||||
strips those wrappers so the result is a plain ``dict[str, Any]``
|
||||
compatible with ``tomllib`` output.
|
||||
"""
|
||||
result: dict[str, Any] = {}
|
||||
for key, value in d.items():
|
||||
if isinstance(value, dict):
|
||||
result[key] = _flatten_toml_dict(value)
|
||||
else:
|
||||
result[key] = value
|
||||
return result
|
||||
|
||||
|
||||
def _toml_to_python(data: object) -> dict[str, Any]:
|
||||
"""Convert a tomlkit document to a plain Python dict.
|
||||
|
||||
Delegates recursive flattening to the module-level
|
||||
``_flatten_toml_dict`` helper so the logic stays testable in
|
||||
isolation and ruff does not flag nested function definitions.
|
||||
"""
|
||||
if isinstance(data, dict):
|
||||
return _flatten_toml_dict(data)
|
||||
raise TypeError(f"Expected dict-like TOML table, got {type(data).__name__}")
|
||||
|
||||
|
||||
def _load_toml(content: bytes) -> dict[str, Any]:
|
||||
"""Load a TOML file from bytes. Uses tomllib if available, else fallback."""
|
||||
try:
|
||||
import tomllib
|
||||
|
||||
return tomllib.loads(content.decode("utf-8"))
|
||||
except ImportError: # pragma: no cover — Python <3.11
|
||||
# Fallback for older Python versions without tomllib
|
||||
import tomlkit
|
||||
|
||||
parsed_doc = tomlkit.parse(content.decode("utf-8"))
|
||||
return _toml_to_python(parsed_doc)
|
||||
@@ -27,6 +27,7 @@ from unittest.mock import MagicMock, patch
|
||||
from behave import given, then, when
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from cleveragents.application.services.session_workflow import TellResult
|
||||
from cleveragents.cli.commands.session import app as session_app
|
||||
from cleveragents.core.exceptions import DatabaseError
|
||||
from cleveragents.domain.models.core.cost_budget import SessionCostBudget
|
||||
@@ -101,15 +102,13 @@ def _mock_service() -> MagicMock:
|
||||
|
||||
|
||||
def _patch_service(context, svc):
|
||||
"""Patch _get_session_service to return *svc* deterministically.
|
||||
"""Patch ``_get_session_service`` to return *svc* deterministically.
|
||||
|
||||
Patching the function directly (rather than the module-level ``_service``
|
||||
attribute) prevents a race condition in parallel Behave workers: if a
|
||||
concurrent worker's cleanup resets ``_service`` to ``None`` while this
|
||||
scenario's CLI command is executing, ``_get_session_service()`` would fall
|
||||
through to the real DI container and raise, producing a spurious exit
|
||||
code 1. Patching the function itself short-circuits the cache check
|
||||
entirely and is immune to cross-worker ``_service`` mutations.
|
||||
Patches the function rather than mutating the module-level ``_service``
|
||||
singleton to prevent a race condition in parallel Behave workers.
|
||||
Mutating ``_service`` could cause concurrent cleanup in one worker to
|
||||
reset it to ``None`` while another worker was still using it, leading
|
||||
to intermittent test failures.
|
||||
"""
|
||||
patcher = patch(
|
||||
"cleveragents.cli.commands.session._get_session_service",
|
||||
@@ -119,6 +118,33 @@ def _patch_service(context, svc):
|
||||
context._scvbst_cleanups.append(patcher.stop)
|
||||
|
||||
|
||||
def _patch_workflow(context, wf):
|
||||
"""Patch _build_session_workflow to return *wf* deterministically."""
|
||||
patcher = patch(
|
||||
"cleveragents.cli.commands.session._build_session_workflow",
|
||||
return_value=wf,
|
||||
)
|
||||
patcher.start()
|
||||
context._scvbst_cleanups.append(patcher.stop)
|
||||
|
||||
|
||||
def _make_tell_result(
|
||||
session_id: str = _ULID1,
|
||||
prompt: str = "Hello world",
|
||||
response: str = "Acknowledged: Hello world",
|
||||
) -> TellResult:
|
||||
return TellResult(
|
||||
session_id=session_id,
|
||||
user_message=prompt,
|
||||
assistant_message=response,
|
||||
input_tokens=5,
|
||||
output_tokens=5,
|
||||
cost=0.0,
|
||||
duration_ms=0.0,
|
||||
tool_calls_count=0,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Background
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -601,7 +627,14 @@ def step_invoke_import_db_error(context):
|
||||
def step_tell_service(context):
|
||||
svc = _mock_service()
|
||||
svc.append_message.return_value = None
|
||||
svc.get_messages.return_value = []
|
||||
_patch_service(context, svc)
|
||||
# Patch workflow so tell uses a canned response without LLM/DB.
|
||||
wf = MagicMock()
|
||||
wf.tell.return_value = _make_tell_result()
|
||||
wf.tell_stream.return_value = iter(["Acknowledged: Hello stream"])
|
||||
_patch_workflow(context, wf)
|
||||
context._scvbst_mock_wf = wf
|
||||
|
||||
|
||||
@when("session coverage boost I invoke the tell command without stream")
|
||||
@@ -622,6 +655,11 @@ def step_invoke_tell_stream(context):
|
||||
|
||||
@when("session coverage boost I invoke the tell command with actor override")
|
||||
def step_invoke_tell_actor(context):
|
||||
# Override the mock workflow response to include actor name
|
||||
context._scvbst_mock_wf.tell.return_value = _make_tell_result(
|
||||
prompt="Plan a feature",
|
||||
response="[openai/gpt-4] Acknowledged: Plan a feature",
|
||||
)
|
||||
context.result = _runner.invoke(
|
||||
session_app,
|
||||
[
|
||||
@@ -647,6 +685,11 @@ def step_tell_not_found(context):
|
||||
svc = _mock_service()
|
||||
svc.append_message.side_effect = SessionNotFoundError("no session")
|
||||
_patch_service(context, svc)
|
||||
# Workflow raises SessionNotFoundError so the CLI handler catches it.
|
||||
wf = MagicMock()
|
||||
wf.tell.side_effect = SessionNotFoundError("no session")
|
||||
wf.tell_stream.side_effect = SessionNotFoundError("no session")
|
||||
_patch_workflow(context, wf)
|
||||
|
||||
|
||||
@given("session coverage boost a mock service that raises DatabaseError on append")
|
||||
@@ -654,6 +697,11 @@ def step_tell_db_error(context):
|
||||
svc = _mock_service()
|
||||
svc.append_message.side_effect = DatabaseError("tell db fail")
|
||||
_patch_service(context, svc)
|
||||
# Workflow raises DatabaseError so the CLI handler catches it.
|
||||
wf = MagicMock()
|
||||
wf.tell.side_effect = DatabaseError("tell db fail")
|
||||
wf.tell_stream.side_effect = DatabaseError("tell db fail")
|
||||
_patch_workflow(context, wf)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -8,14 +8,14 @@ import re
|
||||
import tempfile
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from behave import given, then, when
|
||||
from behave.runner import Context
|
||||
from typer.testing import CliRunner
|
||||
from ulid import ULID
|
||||
|
||||
from cleveragents.cli.commands import session as session_mod
|
||||
from cleveragents.application.services.session_workflow import TellResult
|
||||
from cleveragents.cli.commands.session import app as session_app
|
||||
from cleveragents.domain.models.core.session import (
|
||||
MessageRole,
|
||||
@@ -98,12 +98,43 @@ def step_session_cli_runner(context: Context) -> None:
|
||||
# Default: create returns a session
|
||||
default_session = _make_session(session_id=_SESSION_ID)
|
||||
context.mock_service.create.return_value = default_session
|
||||
# get_messages returns empty list by default (used by SessionWorkflow)
|
||||
context.mock_service.get_messages.return_value = []
|
||||
|
||||
# Patch the module-level service
|
||||
session_mod._service = context.mock_service
|
||||
# Patch _get_session_service so tests are safe for parallel workers
|
||||
# and do not mutate the module-level _service cache (M9).
|
||||
svc_patcher = patch(
|
||||
"cleveragents.cli.commands.session._get_session_service",
|
||||
return_value=context.mock_service,
|
||||
)
|
||||
svc_patcher.start()
|
||||
|
||||
# Patch _build_session_workflow to return a controllable mock workflow.
|
||||
# This lets tell-specific tests configure exact return values without
|
||||
# needing a real LLM or provider registry in the test environment.
|
||||
context.mock_workflow = MagicMock()
|
||||
_default_tell_result = TellResult(
|
||||
session_id=_SESSION_ID,
|
||||
user_message="",
|
||||
assistant_message="Acknowledged",
|
||||
input_tokens=10,
|
||||
output_tokens=5,
|
||||
cost=0.0,
|
||||
duration_ms=0.0,
|
||||
tool_calls_count=0,
|
||||
)
|
||||
context.mock_workflow.tell.return_value = _default_tell_result
|
||||
context.mock_workflow.tell_stream.return_value = iter(["Acknowledged"])
|
||||
|
||||
workflow_patcher = patch(
|
||||
"cleveragents.cli.commands.session._build_session_workflow",
|
||||
return_value=context.mock_workflow,
|
||||
)
|
||||
workflow_patcher.start()
|
||||
|
||||
def cleanup() -> None:
|
||||
session_mod._service = None
|
||||
svc_patcher.stop()
|
||||
workflow_patcher.stop()
|
||||
_cleanup(context)
|
||||
|
||||
context.add_cleanup(cleanup)
|
||||
@@ -306,10 +337,17 @@ def step_capture_first_ulid(context: Context) -> None:
|
||||
def step_tell_with_stored_ulid(context: Context, prompt: str) -> None:
|
||||
"""Run session tell using the full ULID stored from session list."""
|
||||
session_id = context.full_session_id
|
||||
context.mock_service.append_message.side_effect = [
|
||||
_make_message(MessageRole.USER, prompt, 0),
|
||||
_make_message(MessageRole.ASSISTANT, f"Acknowledged: {prompt}", 1),
|
||||
]
|
||||
# Configure mock workflow for this prompt.
|
||||
context.mock_workflow.tell.return_value = TellResult(
|
||||
session_id=session_id,
|
||||
user_message=prompt,
|
||||
assistant_message=f"Acknowledged: {prompt}",
|
||||
input_tokens=len(prompt) // 4,
|
||||
output_tokens=5,
|
||||
cost=0.0,
|
||||
duration_ms=0.0,
|
||||
tool_calls_count=0,
|
||||
)
|
||||
context.result = context.runner.invoke(
|
||||
session_app,
|
||||
["tell", "--session", session_id, prompt],
|
||||
@@ -563,21 +601,37 @@ def step_import_succeeds(context: Context) -> None:
|
||||
|
||||
@given("there is a mocked session for tell")
|
||||
def step_session_for_tell(context: Context) -> None:
|
||||
session = _make_session(session_id=_SESSION_ID)
|
||||
# Session with actor so tell can proceed without --actor flag.
|
||||
session = _make_session(session_id=_SESSION_ID, actor_name="openai/gpt-4")
|
||||
context.mock_service.get.return_value = session
|
||||
context.mock_service.append_message.side_effect = [
|
||||
_make_message(MessageRole.USER, "Hello, world", 0),
|
||||
_make_message(MessageRole.ASSISTANT, "Acknowledged: Hello, world", 1),
|
||||
]
|
||||
context.mock_service.get_messages.return_value = []
|
||||
context.session_id = _SESSION_ID
|
||||
# Configure the mock workflow to return a canned TellResult.
|
||||
context.mock_workflow.tell.return_value = TellResult(
|
||||
session_id=_SESSION_ID,
|
||||
user_message="Hello, world",
|
||||
assistant_message="Acknowledged: Hello, world",
|
||||
input_tokens=10,
|
||||
output_tokens=5,
|
||||
cost=0.0,
|
||||
duration_ms=0.0,
|
||||
tool_calls_count=0,
|
||||
)
|
||||
|
||||
|
||||
@when('I run session CLI tell with a prompt "{prompt}"')
|
||||
def step_tell_prompt(context: Context, prompt: str) -> None:
|
||||
context.mock_service.append_message.side_effect = [
|
||||
_make_message(MessageRole.USER, prompt, 0),
|
||||
_make_message(MessageRole.ASSISTANT, f"Acknowledged: {prompt}", 1),
|
||||
]
|
||||
# Update the mock workflow result to reflect the current prompt.
|
||||
context.mock_workflow.tell.return_value = TellResult(
|
||||
session_id=context.session_id,
|
||||
user_message=prompt,
|
||||
assistant_message=f"Acknowledged: {prompt}",
|
||||
input_tokens=len(prompt) // 4,
|
||||
output_tokens=5,
|
||||
cost=0.0,
|
||||
duration_ms=0.0,
|
||||
tool_calls_count=0,
|
||||
)
|
||||
context.result = context.runner.invoke(
|
||||
session_app,
|
||||
["tell", "--session", context.session_id, prompt],
|
||||
@@ -586,14 +640,17 @@ def step_tell_prompt(context: Context, prompt: str) -> None:
|
||||
|
||||
@when('I run session CLI tell with --actor "{actor}" and prompt "{prompt}"')
|
||||
def step_tell_with_actor(context: Context, actor: str, prompt: str) -> None:
|
||||
context.mock_service.append_message.side_effect = [
|
||||
_make_message(MessageRole.USER, prompt, 0),
|
||||
_make_message(
|
||||
MessageRole.ASSISTANT,
|
||||
f"[{actor}] Acknowledged: {prompt}",
|
||||
1,
|
||||
),
|
||||
]
|
||||
# Update the mock workflow result to reflect actor and prompt.
|
||||
context.mock_workflow.tell.return_value = TellResult(
|
||||
session_id=context.session_id,
|
||||
user_message=prompt,
|
||||
assistant_message=f"[{actor}] Acknowledged: {prompt}",
|
||||
input_tokens=len(prompt) // 4,
|
||||
output_tokens=5,
|
||||
cost=0.0,
|
||||
duration_ms=0.0,
|
||||
tool_calls_count=0,
|
||||
)
|
||||
context.result = context.runner.invoke(
|
||||
session_app,
|
||||
["tell", "--session", context.session_id, "--actor", actor, prompt],
|
||||
@@ -602,9 +659,8 @@ def step_tell_with_actor(context: Context, actor: str, prompt: str) -> None:
|
||||
|
||||
@when("I run session CLI tell to a non-existent session")
|
||||
def step_tell_nonexistent(context: Context) -> None:
|
||||
context.mock_service.append_message.side_effect = SessionNotFoundError(
|
||||
"Session not found"
|
||||
)
|
||||
# Configure the workflow to raise SessionNotFoundError.
|
||||
context.mock_workflow.tell.side_effect = SessionNotFoundError("Session not found")
|
||||
context.result = context.runner.invoke(
|
||||
session_app,
|
||||
["tell", "--session", "NONEXISTENT", "Hello"],
|
||||
|
||||
@@ -9,6 +9,7 @@ from unittest.mock import MagicMock, patch
|
||||
from behave import given, then, when
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from cleveragents.application.services.session_workflow import TellResult
|
||||
from cleveragents.cli.commands.session import app as session_app
|
||||
from cleveragents.domain.models.core.session import (
|
||||
MessageRole,
|
||||
@@ -358,10 +359,28 @@ def step_export_output_contains(context, text):
|
||||
@given("session cli branch a mock session service for tell")
|
||||
def step_service_for_tell(context):
|
||||
svc = _mock_service()
|
||||
# append_message doesn't need to return anything meaningful for the
|
||||
# code path under test — the assistant content is computed inline.
|
||||
svc.append_message.return_value = None
|
||||
svc.get_messages.return_value = []
|
||||
_patch_service(context, svc)
|
||||
# Patch _build_session_workflow to return a controllable mock.
|
||||
mock_wf = MagicMock()
|
||||
mock_wf.tell_stream.return_value = iter(["Acknowledged: Hello world"])
|
||||
mock_wf.tell.return_value = TellResult(
|
||||
session_id=_ULID,
|
||||
user_message="Hello world",
|
||||
assistant_message="Acknowledged: Hello world",
|
||||
input_tokens=5,
|
||||
output_tokens=5,
|
||||
cost=0.0,
|
||||
duration_ms=0.0,
|
||||
tool_calls_count=0,
|
||||
)
|
||||
wf_patcher = patch(
|
||||
"cleveragents.cli.commands.session._build_session_workflow",
|
||||
return_value=mock_wf,
|
||||
)
|
||||
wf_patcher.start()
|
||||
context._cleanup_handlers.append(wf_patcher.stop)
|
||||
|
||||
|
||||
@when("session cli branch I invoke the tell command with --stream")
|
||||
@@ -374,7 +393,8 @@ def step_invoke_tell_stream(context):
|
||||
|
||||
@then("session cli branch the streamed output contains the assistant response")
|
||||
def step_streamed_output(context):
|
||||
# The assistant content for no actor is: "Acknowledged: Hello world"
|
||||
# After the stub LLM replacement the assistant response comes from the
|
||||
# mock workflow's tell_stream. Check that ANY assistant output arrived.
|
||||
assert "Acknowledged" in context.result.output, (
|
||||
f"Expected 'Acknowledged' in output. Got:\n{context.result.output}"
|
||||
)
|
||||
|
||||
@@ -0,0 +1,565 @@
|
||||
"""Step definitions for session tell real LLM invocation tests (issue #5784).
|
||||
|
||||
Tests verify that ``agents session tell`` routes through
|
||||
:class:`~cleveragents.application.services.session_workflow.SessionWorkflow`
|
||||
with a real (mock) :class:`LLMCaller`, persists the response, and records
|
||||
token usage.
|
||||
|
||||
All LLM interactions are handled via a mock ``LLMCaller`` so no real API
|
||||
keys or network access is required.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import json
|
||||
import re
|
||||
from collections.abc import Iterator
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from behave import given, then, when
|
||||
from behave.runner import Context
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from cleveragents.application.services.session_workflow import SessionWorkflow
|
||||
from cleveragents.cli.commands.session import app as session_app
|
||||
from cleveragents.domain.models.core.session import (
|
||||
MessageRole,
|
||||
Session,
|
||||
SessionMessage,
|
||||
SessionService,
|
||||
SessionTokenUsage,
|
||||
)
|
||||
|
||||
_ULID = "01KJ1N8YDN05N29P5RZV4SPDVZ"
|
||||
_STUB_RESPONSE = "Here is what I can help you with."
|
||||
|
||||
|
||||
class _StubResp:
|
||||
"""Minimal LLM response stub for _StubLLM.invoke()."""
|
||||
|
||||
def __init__(self, content: str) -> None:
|
||||
self.content = content
|
||||
self.tool_calls: list[Any] = []
|
||||
self.response_metadata: dict[str, Any] = {
|
||||
"usage": {"input_tokens": 10, "output_tokens": 5}
|
||||
}
|
||||
|
||||
|
||||
class _StubChunk:
|
||||
"""Minimal streaming chunk for _StubLLM.stream()."""
|
||||
|
||||
def __init__(self, content: str) -> None:
|
||||
self.content = content
|
||||
|
||||
|
||||
runner = CliRunner()
|
||||
|
||||
|
||||
class _StubLLM:
|
||||
"""Minimal LangChain-compatible LLM stub for LangChainSessionCaller tests."""
|
||||
|
||||
def __init__(self, response: str = _STUB_RESPONSE) -> None:
|
||||
self._response = response
|
||||
self.invoked_messages: list[Any] = []
|
||||
|
||||
def invoke(self, messages: Any, **kwargs: Any) -> _StubResp:
|
||||
self.invoked_messages = list(messages)
|
||||
return _StubResp(self._response)
|
||||
|
||||
def stream(self, messages: Any, **kwargs: Any) -> Iterator[_StubChunk]:
|
||||
self.invoked_messages = list(messages)
|
||||
words = self._response.split()
|
||||
for i, word in enumerate(words):
|
||||
# Reconstruct spaces between words so the concatenated
|
||||
# output matches the original response string verbatim.
|
||||
token = word if i == 0 else " " + word
|
||||
yield _StubChunk(token)
|
||||
|
||||
|
||||
def _make_session(
|
||||
session_id: str = _ULID,
|
||||
actor_name: str | None = None,
|
||||
) -> Session:
|
||||
return Session(
|
||||
session_id=session_id,
|
||||
actor_name=actor_name,
|
||||
namespace="local",
|
||||
messages=[],
|
||||
token_usage=SessionTokenUsage(),
|
||||
)
|
||||
|
||||
|
||||
def _make_mock_service(session: Session) -> MagicMock:
|
||||
"""Build a MagicMock SessionService pre-configured for session tell."""
|
||||
svc = MagicMock(spec=SessionService)
|
||||
svc.get.return_value = session
|
||||
svc.get_messages.return_value = []
|
||||
svc.append_message.return_value = MagicMock(spec=SessionMessage)
|
||||
svc.update_token_usage.return_value = None
|
||||
return svc
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Background
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@given("a session tell LLM mock environment")
|
||||
def step_setup_llm_mock_env(context: Context) -> None:
|
||||
"""Set up runner, stub LLM, and cleanup handlers."""
|
||||
context.runner = runner
|
||||
context.stub_llm = _StubLLM()
|
||||
context.captured_actor_name: str | None = None
|
||||
context._cleanup_handlers: list[Any] = []
|
||||
|
||||
def cleanup() -> None:
|
||||
for stop in reversed(context._cleanup_handlers):
|
||||
with contextlib.suppress(Exception):
|
||||
stop()
|
||||
|
||||
context.add_cleanup(cleanup)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Given steps
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@given('a session with actor "{actor}" exists')
|
||||
def step_session_with_actor(context: Context, actor: str) -> None:
|
||||
session = _make_session(actor_name=actor)
|
||||
context.session = session
|
||||
context.svc = _make_mock_service(session)
|
||||
# Patch _get_session_service instead of module-level _service so tests
|
||||
# are resilient to changes in the internal caching strategy (m4).
|
||||
svc_patcher = patch(
|
||||
"cleveragents.cli.commands.session._get_session_service",
|
||||
return_value=context.svc,
|
||||
)
|
||||
svc_patcher.start()
|
||||
context._cleanup_handlers.append(svc_patcher.stop)
|
||||
|
||||
|
||||
@given("a session with no actor exists")
|
||||
def step_session_no_actor(context: Context) -> None:
|
||||
session = _make_session(actor_name=None)
|
||||
context.session = session
|
||||
context.svc = _make_mock_service(session)
|
||||
svc_patcher = patch(
|
||||
"cleveragents.cli.commands.session._get_session_service",
|
||||
return_value=context.svc,
|
||||
)
|
||||
svc_patcher.start()
|
||||
context._cleanup_handlers.append(svc_patcher.stop)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# When steps
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@when('I invoke session tell with prompt "{prompt}"')
|
||||
def step_invoke_tell(context: Context, prompt: str) -> None:
|
||||
"""Invoke tell routing through _facade_dispatch.
|
||||
|
||||
The non-streaming CLI path calls :func:`_facade_dispatch`, which in
|
||||
production delegates to ``A2aLocalFacade``. The test intercepts
|
||||
``_facade_dispatch`` with a thin adapter that runs the real
|
||||
``SessionWorkflow.tell()`` (backed by the test's stub LLM) and
|
||||
returns a dict matching the facade's response envelope. This
|
||||
keeps the workflow assertions (append_message, token usage) working
|
||||
while also testing the CLI's facade-routing code path.
|
||||
"""
|
||||
context.prompt = prompt
|
||||
context.invoked_actor_name = None
|
||||
stub_llm = context.stub_llm
|
||||
svc = context.svc
|
||||
|
||||
def _llm_factory(actor: str) -> Any:
|
||||
context.invoked_actor_name = actor
|
||||
return stub_llm
|
||||
|
||||
wf = SessionWorkflow(session_service=svc, llm_factory=_llm_factory)
|
||||
|
||||
def _mock_dispatch(operation: str, params: dict[str, Any]) -> dict[str, Any]:
|
||||
if operation == "message/send":
|
||||
result = wf.tell(
|
||||
session_id=params["session_id"],
|
||||
prompt=params["message"],
|
||||
actor_override=params.get("actor"),
|
||||
)
|
||||
return {
|
||||
"session_id": result.session_id,
|
||||
"assistant_message": result.assistant_message,
|
||||
"usage": result.usage_dict(),
|
||||
}
|
||||
raise RuntimeError(f"Unknown operation: {operation}")
|
||||
|
||||
dispatch_patcher = patch(
|
||||
"cleveragents.cli.commands.session._facade_dispatch",
|
||||
side_effect=_mock_dispatch,
|
||||
)
|
||||
dispatch_patcher.start()
|
||||
context._cleanup_handlers.append(dispatch_patcher.stop)
|
||||
|
||||
context.result = runner.invoke(
|
||||
session_app,
|
||||
["tell", "--session", _ULID, prompt],
|
||||
)
|
||||
|
||||
|
||||
@when('I invoke session tell with --stream and prompt "{prompt}"')
|
||||
def step_invoke_tell_stream(context: Context, prompt: str) -> None:
|
||||
"""Invoke tell --stream with a real SessionWorkflow backed by _StubLLM."""
|
||||
context.prompt = prompt
|
||||
stub_llm = context.stub_llm
|
||||
svc = context.svc
|
||||
|
||||
def _workflow_factory() -> SessionWorkflow:
|
||||
return SessionWorkflow(
|
||||
session_service=svc,
|
||||
llm_factory=lambda actor: stub_llm,
|
||||
)
|
||||
|
||||
wf_patcher = patch(
|
||||
"cleveragents.cli.commands.session._build_session_workflow",
|
||||
side_effect=_workflow_factory,
|
||||
)
|
||||
wf_patcher.start()
|
||||
context._cleanup_handlers.append(wf_patcher.stop)
|
||||
|
||||
context.result = runner.invoke(
|
||||
session_app,
|
||||
["tell", "--session", _ULID, "--stream", prompt],
|
||||
)
|
||||
|
||||
|
||||
@when('I invoke session tell with --actor "{actor}" and prompt "{prompt}"')
|
||||
def step_invoke_tell_with_actor(context: Context, actor: str, prompt: str) -> None:
|
||||
"""Invoke tell with --actor override, routing through _facade_dispatch."""
|
||||
context.prompt = prompt
|
||||
context.override_actor = actor
|
||||
stub_llm = context.stub_llm
|
||||
svc = context.svc
|
||||
|
||||
def _llm_factory(actor_name: str) -> Any:
|
||||
context.invoked_actor_name = actor_name
|
||||
return stub_llm
|
||||
|
||||
wf = SessionWorkflow(session_service=svc, llm_factory=_llm_factory)
|
||||
|
||||
def _mock_dispatch(operation: str, params: dict[str, Any]) -> dict[str, Any]:
|
||||
if operation == "message/send":
|
||||
result = wf.tell(
|
||||
session_id=params["session_id"],
|
||||
prompt=params["message"],
|
||||
actor_override=params.get("actor"),
|
||||
)
|
||||
return {
|
||||
"session_id": result.session_id,
|
||||
"assistant_message": result.assistant_message,
|
||||
"usage": result.usage_dict(),
|
||||
}
|
||||
raise RuntimeError(f"Unknown operation: {operation}")
|
||||
|
||||
dispatch_patcher = patch(
|
||||
"cleveragents.cli.commands.session._facade_dispatch",
|
||||
side_effect=_mock_dispatch,
|
||||
)
|
||||
dispatch_patcher.start()
|
||||
context._cleanup_handlers.append(dispatch_patcher.stop)
|
||||
|
||||
context.result = runner.invoke(
|
||||
session_app,
|
||||
["tell", "--session", _ULID, "--actor", actor, prompt],
|
||||
)
|
||||
|
||||
|
||||
@when('I invoke session tell with --format json and prompt "{prompt}"')
|
||||
def step_invoke_tell_json(context: Context, prompt: str) -> None:
|
||||
"""Invoke tell --format json, routing through _facade_dispatch."""
|
||||
context.prompt = prompt
|
||||
stub_llm = context.stub_llm
|
||||
svc = context.svc
|
||||
|
||||
wf = SessionWorkflow(
|
||||
session_service=svc,
|
||||
llm_factory=lambda actor: stub_llm,
|
||||
)
|
||||
|
||||
def _mock_dispatch(operation: str, params: dict[str, Any]) -> dict[str, Any]:
|
||||
if operation == "message/send":
|
||||
result = wf.tell(
|
||||
session_id=params["session_id"],
|
||||
prompt=params["message"],
|
||||
actor_override=params.get("actor"),
|
||||
)
|
||||
return {
|
||||
"session_id": result.session_id,
|
||||
"assistant_message": result.assistant_message,
|
||||
"usage": result.usage_dict(),
|
||||
}
|
||||
raise RuntimeError(f"Unknown operation: {operation}")
|
||||
|
||||
dispatch_patcher = patch(
|
||||
"cleveragents.cli.commands.session._facade_dispatch",
|
||||
side_effect=_mock_dispatch,
|
||||
)
|
||||
dispatch_patcher.start()
|
||||
context._cleanup_handlers.append(dispatch_patcher.stop)
|
||||
|
||||
context.result = runner.invoke(
|
||||
session_app,
|
||||
["tell", "--session", _ULID, "--format", "json", prompt],
|
||||
)
|
||||
|
||||
def _mock_dispatch(operation: str, params: dict[str, Any]) -> dict[str, Any]:
|
||||
if operation == "message/send":
|
||||
result = wf.tell(
|
||||
session_id=params["session_id"],
|
||||
prompt=params["message"],
|
||||
actor_override=params.get("actor"),
|
||||
)
|
||||
return {
|
||||
"session_id": result.session_id,
|
||||
"assistant_message": result.assistant_message,
|
||||
"usage": result.usage_dict(),
|
||||
}
|
||||
raise RuntimeError(f"Unknown operation: {operation}")
|
||||
|
||||
dispatch_patcher = patch(
|
||||
"cleveragents.cli.commands.session._facade_dispatch",
|
||||
side_effect=_mock_dispatch,
|
||||
)
|
||||
dispatch_patcher.start()
|
||||
context._cleanup_handlers.append(dispatch_patcher.stop)
|
||||
|
||||
context.result = runner.invoke(
|
||||
session_app,
|
||||
["tell", "--session", _ULID, "--format", "json", prompt],
|
||||
)
|
||||
|
||||
|
||||
@when('I invoke session tell with --stream --format json and prompt "{prompt}"')
|
||||
def step_invoke_tell_stream_json(context: Context, prompt: str) -> None:
|
||||
"""Invoke tell --stream --format json with a real SessionWorkflow backed by _StubLLM."""
|
||||
context.prompt = prompt
|
||||
stub_llm = context.stub_llm
|
||||
svc = context.svc
|
||||
|
||||
def _workflow_factory() -> SessionWorkflow:
|
||||
return SessionWorkflow(
|
||||
session_service=svc,
|
||||
llm_factory=lambda actor: stub_llm,
|
||||
)
|
||||
|
||||
wf_patcher = patch(
|
||||
"cleveragents.cli.commands.session._build_session_workflow",
|
||||
side_effect=_workflow_factory,
|
||||
)
|
||||
wf_patcher.start()
|
||||
context._cleanup_handlers.append(wf_patcher.stop)
|
||||
|
||||
context.result = runner.invoke(
|
||||
session_app,
|
||||
["tell", "--session", _ULID, "--stream", "--format", "json", prompt],
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Then steps
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@then("the tell command returns the LLM response")
|
||||
def step_tell_returns_llm_response(context: Context) -> None:
|
||||
assert context.result.exit_code == 0, (
|
||||
f"Expected exit code 0, got {context.result.exit_code}.\n"
|
||||
f"Output:\n{context.result.output}"
|
||||
)
|
||||
# The stub LLM response should appear in the output
|
||||
assert _STUB_RESPONSE in context.result.output, (
|
||||
f"LLM response not found in output:\n{context.result.output}"
|
||||
)
|
||||
|
||||
|
||||
@then("the assistant message is persisted to the session")
|
||||
def step_assistant_message_persisted(context: Context) -> None:
|
||||
# append_message should be called exactly twice:
|
||||
# once for the user message and once for the assistant response.
|
||||
call_count = context.svc.append_message.call_count
|
||||
assert call_count == 2, (
|
||||
f"Expected append_message to be called exactly 2 times (user + assistant), "
|
||||
f"got {call_count}"
|
||||
)
|
||||
# The second call should be for the ASSISTANT role
|
||||
calls = context.svc.append_message.call_args_list
|
||||
assistant_calls = [
|
||||
c
|
||||
for c in calls
|
||||
if c.kwargs.get("role") == MessageRole.ASSISTANT
|
||||
or (len(c.args) >= 2 and c.args[1] == MessageRole.ASSISTANT)
|
||||
]
|
||||
assert len(assistant_calls) == 1, (
|
||||
f"Expected exactly 1 ASSISTANT persist, got {len(assistant_calls)}. "
|
||||
f"Calls: {calls}"
|
||||
)
|
||||
# The second call should be for the ASSISTANT role
|
||||
calls = context.svc.append_message.call_args_list
|
||||
assistant_calls = [
|
||||
c
|
||||
for c in calls
|
||||
if c.kwargs.get("role") == MessageRole.ASSISTANT
|
||||
or (len(c.args) >= 2 and c.args[1] == MessageRole.ASSISTANT)
|
||||
]
|
||||
assert len(assistant_calls) == 1, (
|
||||
f"Expected exactly 1 ASSISTANT persist, got {len(assistant_calls)}. "
|
||||
f"Calls: {calls}"
|
||||
)
|
||||
|
||||
|
||||
@then("token usage is recorded")
|
||||
def step_token_usage_recorded(context: Context) -> None:
|
||||
assert context.svc.update_token_usage.called, (
|
||||
"update_token_usage was not called — token usage was not recorded"
|
||||
)
|
||||
assert context.svc.update_token_usage.call_count == 1, (
|
||||
f"Expected update_token_usage to be called exactly once, "
|
||||
f"got {context.svc.update_token_usage.call_count}"
|
||||
)
|
||||
# Verify exact token counts match the metadata from the stub LLM (m5).
|
||||
call_kwargs = context.svc.update_token_usage.call_args.kwargs
|
||||
assert call_kwargs.get("input_tokens") == 10, (
|
||||
f"Expected input_tokens=10, got {call_kwargs.get('input_tokens')}"
|
||||
)
|
||||
assert call_kwargs.get("output_tokens") == 5, (
|
||||
f"Expected output_tokens=5, got {call_kwargs.get('output_tokens')}"
|
||||
)
|
||||
assert context.svc.update_token_usage.call_count == 1, (
|
||||
f"Expected update_token_usage to be called exactly once, "
|
||||
f"got {context.svc.update_token_usage.call_count}"
|
||||
)
|
||||
# Verify exact token counts match the metadata from the stub LLM (m5).
|
||||
call_kwargs = context.svc.update_token_usage.call_args.kwargs
|
||||
assert call_kwargs.get("input_tokens") == 10, (
|
||||
f"Expected input_tokens=10, got {call_kwargs.get('input_tokens')}"
|
||||
)
|
||||
assert call_kwargs.get("output_tokens") == 5, (
|
||||
f"Expected output_tokens=5, got {call_kwargs.get('output_tokens')}"
|
||||
)
|
||||
|
||||
|
||||
@then("the streamed output contains the LLM response tokens")
|
||||
def step_streamed_output_contains_tokens(context: Context) -> None:
|
||||
assert context.result.exit_code == 0, (
|
||||
f"Expected exit code 0, got {context.result.exit_code}.\n"
|
||||
f"Output:\n{context.result.output}"
|
||||
)
|
||||
# Strip ANSI escape sequences for text matching
|
||||
output_clean = re.sub(r"\x1b\[[0-9;]*m", "", context.result.output)
|
||||
# When streamed through the A2A facade (Option B: fallback to
|
||||
# non-streaming), verify the full response is present in the output
|
||||
# rather than checking token-by-token word positions (C5).
|
||||
assert _STUB_RESPONSE in output_clean, (
|
||||
f"Expected full response '{_STUB_RESPONSE}' in streamed output, "
|
||||
f"but it was not found.\nOutput:\n{context.result.output}"
|
||||
)
|
||||
|
||||
|
||||
@then("the assistant message is persisted after streaming")
|
||||
def step_assistant_persisted_after_stream(context: Context) -> None:
|
||||
call_count = context.svc.append_message.call_count
|
||||
assert call_count == 2, (
|
||||
f"Expected append_message to be called exactly 2 times (user + assistant), "
|
||||
f"got {call_count}"
|
||||
)
|
||||
# Verify that the last persisted message is an ASSISTANT role.
|
||||
calls = context.svc.append_message.call_args_list
|
||||
assert calls[-1].kwargs.get("role") == MessageRole.ASSISTANT, (
|
||||
f"Last persisted message role is not ASSISTANT. Calls: {calls}"
|
||||
)
|
||||
# Verify that the last persisted message is an ASSISTANT role.
|
||||
calls = context.svc.append_message.call_args_list
|
||||
assert calls[-1].kwargs.get("role") == MessageRole.ASSISTANT, (
|
||||
f"Last persisted message role is not ASSISTANT. Calls: {calls}"
|
||||
)
|
||||
|
||||
|
||||
@then("the tell command exits with code 1")
|
||||
def step_tell_exits_code_1(context: Context) -> None:
|
||||
assert context.result.exit_code == 1, (
|
||||
f"Expected exit code 1, got {context.result.exit_code}.\n"
|
||||
f"Output:\n{context.result.output}"
|
||||
)
|
||||
|
||||
|
||||
@then("the error output mentions actor configuration")
|
||||
def step_error_mentions_actor(context: Context) -> None:
|
||||
output = context.result.output.lower()
|
||||
assert "actor" in output, (
|
||||
f"Expected 'actor' in error output.\nGot:\n{context.result.output}"
|
||||
)
|
||||
|
||||
|
||||
@then('the tell command uses the override actor "{actor}"')
|
||||
def step_tell_uses_override_actor(context: Context, actor: str) -> None:
|
||||
invoked = getattr(context, "invoked_actor_name", None)
|
||||
assert invoked is not None, "Actor name was never captured from _resolve_llm"
|
||||
assert invoked == actor, f"Expected actor '{actor}' to be used, but got '{invoked}'"
|
||||
|
||||
|
||||
@then("the tell output is valid JSON with a data envelope")
|
||||
def step_tell_output_is_valid_json(context: Context) -> None:
|
||||
assert context.result.exit_code == 0, (
|
||||
f"Expected exit code 0, got {context.result.exit_code}.\n"
|
||||
f"Output:\n{context.result.output}"
|
||||
)
|
||||
try:
|
||||
raw = json.loads(context.result.output.strip())
|
||||
except json.JSONDecodeError as exc:
|
||||
raise AssertionError(
|
||||
f"Output is not valid JSON: {exc}\nOutput:\n{context.result.output}"
|
||||
) from exc
|
||||
# format_output wraps data in a spec-required envelope
|
||||
context.json_output = raw
|
||||
context.json_data = raw.get("data", {})
|
||||
assert isinstance(context.json_data, dict), (
|
||||
f"Expected data envelope to be a dict, got {type(context.json_data)}"
|
||||
)
|
||||
|
||||
|
||||
@then("the data section contains the session_id")
|
||||
def step_json_data_contains_session_id(context: Context) -> None:
|
||||
assert context.json_data.get("session_id") == _ULID, (
|
||||
f"Expected session_id={_ULID}, got {context.json_data.get('session_id')}"
|
||||
)
|
||||
|
||||
|
||||
@then("the data section contains a usage object with expected keys")
|
||||
def step_json_data_contains_usage_object(context: Context) -> None:
|
||||
usage = context.json_data.get("usage")
|
||||
assert isinstance(usage, dict), f"Expected usage to be a dict, got {type(usage)}"
|
||||
expected_keys = {
|
||||
"input_tokens",
|
||||
"output_tokens",
|
||||
"cost_usd",
|
||||
"duration_ms",
|
||||
"tool_calls",
|
||||
}
|
||||
missing = expected_keys - set(usage.keys())
|
||||
assert not missing, f"Usage object missing expected keys: {missing}\nUsage: {usage}"
|
||||
|
||||
|
||||
@then("the output contains a Usage panel")
|
||||
def step_output_contains_usage_panel(context: Context) -> None:
|
||||
output = context.result.output
|
||||
# Rich Usage panel is rendered with title="Usage"
|
||||
assert "Usage" in output, f"Expected Usage panel in output.\nOutput:\n{output}"
|
||||
assert "Input tokens" in output, (
|
||||
f"Expected input tokens in Usage panel.\nOutput:\n{output}"
|
||||
)
|
||||
assert "Output tokens" in output, (
|
||||
f"Expected output tokens in Usage panel.\nOutput:\n{output}"
|
||||
)
|
||||
@@ -0,0 +1,473 @@
|
||||
"""Step definitions for session_workflow.py coverage boost tests."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from behave import given, then, when
|
||||
from behave.runner import Context
|
||||
|
||||
from cleveragents.application.services.session_caller import (
|
||||
LangChainSessionCaller,
|
||||
_MinimalStubLLM,
|
||||
estimate_cost,
|
||||
extract_content,
|
||||
extract_token_usage,
|
||||
history_to_langchain_messages,
|
||||
)
|
||||
from cleveragents.application.services.session_workflow import (
|
||||
SessionWorkflow,
|
||||
)
|
||||
from cleveragents.domain.models.core.session import (
|
||||
MessageRole,
|
||||
SessionMessage,
|
||||
)
|
||||
from cleveragents.tool.actor_runtime import LLMResponse
|
||||
|
||||
|
||||
@given("the session workflow coverage environment is set up")
|
||||
def step_sw_coverage_setup(context: Context) -> None:
|
||||
"""No special setup needed."""
|
||||
pass
|
||||
|
||||
|
||||
# ====================================================================
|
||||
# _extract_content
|
||||
# ====================================================================
|
||||
|
||||
|
||||
@given('coverage boost a mock response with content "hello world"')
|
||||
def step_cb_mock_content(context: Context) -> None:
|
||||
resp = MagicMock()
|
||||
resp.content = "hello world"
|
||||
resp.text = "ignored"
|
||||
context.cb_mock_response = resp
|
||||
|
||||
|
||||
@given('coverage boost a mock response with text "from text attr" and no content')
|
||||
def step_cb_text_fallback(context: Context) -> None:
|
||||
class _TextOnly:
|
||||
def __init__(self) -> None:
|
||||
self.text = "from text attr"
|
||||
|
||||
context.cb_mock_response = _TextOnly()
|
||||
|
||||
|
||||
@given(
|
||||
"coverage boost a mock response with list content containing text dicts and plain strings"
|
||||
)
|
||||
def step_cb_list_content(context: Context) -> None:
|
||||
resp = MagicMock()
|
||||
resp.content = [
|
||||
{"text": "hello"},
|
||||
"world",
|
||||
{"content": "test"},
|
||||
]
|
||||
context.cb_mock_response = resp
|
||||
|
||||
|
||||
@given("coverage boost a mock response with no content or text attribute")
|
||||
def step_cb_no_attrs(context: Context) -> None:
|
||||
class _Unknown:
|
||||
pass
|
||||
|
||||
context.cb_mock_response = _Unknown()
|
||||
|
||||
|
||||
@when("coverage boost _extract_content is called")
|
||||
def step_cb_call_extract_content(context: Context) -> None:
|
||||
context.cb_extract_result = extract_content(context.cb_mock_response)
|
||||
|
||||
|
||||
@then('coverage boost the extracted result should be "{expected}"')
|
||||
def step_cb_extract_result_is(context: Context, expected: str) -> None:
|
||||
assert context.cb_extract_result == expected, (
|
||||
f"Expected {expected!r}, got {context.cb_extract_result!r}"
|
||||
)
|
||||
|
||||
|
||||
@then("coverage boost the result should contain the concatenated texts")
|
||||
def step_cb_result_concat(context: Context) -> None:
|
||||
assert "hello" in context.cb_extract_result
|
||||
assert "world" in context.cb_extract_result
|
||||
assert "test" in context.cb_extract_result
|
||||
|
||||
|
||||
@then("coverage boost the result should be a string")
|
||||
def step_cb_result_str(context: Context) -> None:
|
||||
assert isinstance(context.cb_extract_result, str)
|
||||
|
||||
|
||||
# ====================================================================
|
||||
# _extract_token_usage
|
||||
# ====================================================================
|
||||
|
||||
|
||||
@given(
|
||||
"coverage boost a mock response with response_metadata usage input_tokens=100 output_tokens=50"
|
||||
)
|
||||
def step_cb_usage_metadata(context: Context) -> None:
|
||||
resp = MagicMock()
|
||||
resp.response_metadata = {"usage": {"input_tokens": 100, "output_tokens": 50}}
|
||||
resp.usage_metadata = {}
|
||||
context.cb_mock_response = resp
|
||||
|
||||
|
||||
@given(
|
||||
"coverage boost a mock response with response_metadata token_usage input_tokens=200 output_tokens=100"
|
||||
)
|
||||
def step_cb_token_usage_metadata(context: Context) -> None:
|
||||
resp = MagicMock()
|
||||
resp.response_metadata = {
|
||||
"token_usage": {"input_tokens": 200, "output_tokens": 100}
|
||||
}
|
||||
resp.usage_metadata = {}
|
||||
context.cb_mock_response = resp
|
||||
|
||||
|
||||
@given(
|
||||
"coverage boost a mock response with response_metadata usage prompt_tokens=300 completion_tokens=150"
|
||||
)
|
||||
def step_cb_prompt_completion(context: Context) -> None:
|
||||
resp = MagicMock()
|
||||
resp.response_metadata = {"usage": {"prompt_tokens": 300, "completion_tokens": 150}}
|
||||
resp.usage_metadata = {}
|
||||
context.cb_mock_response = resp
|
||||
|
||||
|
||||
@given(
|
||||
"coverage boost a mock response with usage_metadata input_tokens=400 output_tokens=200"
|
||||
)
|
||||
def step_cb_usage_meta_attr(context: Context) -> None:
|
||||
resp = MagicMock()
|
||||
resp.response_metadata = {}
|
||||
resp.usage_metadata = {"input_tokens": 400, "output_tokens": 200}
|
||||
context.cb_mock_response = resp
|
||||
|
||||
|
||||
@given("coverage boost a mock response with no usage metadata")
|
||||
def step_cb_no_metadata(context: Context) -> None:
|
||||
resp = MagicMock()
|
||||
resp.response_metadata = {}
|
||||
resp.usage_metadata = {}
|
||||
context.cb_mock_response = resp
|
||||
|
||||
|
||||
@when("coverage boost _extract_token_usage is called")
|
||||
def step_cb_call_extract_token_usage(context: Context) -> None:
|
||||
context.cb_token_input, context.cb_token_output = extract_token_usage(
|
||||
context.cb_mock_response
|
||||
)
|
||||
|
||||
|
||||
@then(
|
||||
"coverage boost input tokens should be {input_tokens:d} and output tokens should be {output_tokens:d}"
|
||||
)
|
||||
def step_cb_assert_token_counts(
|
||||
context: Context, input_tokens: int, output_tokens: int
|
||||
) -> None:
|
||||
assert context.cb_token_input == input_tokens, (
|
||||
f"Expected {input_tokens} input tokens, got {context.cb_token_input}"
|
||||
)
|
||||
assert context.cb_token_output == output_tokens, (
|
||||
f"Expected {output_tokens} output tokens, got {context.cb_token_output}"
|
||||
)
|
||||
|
||||
|
||||
# ====================================================================
|
||||
# _estimate_cost
|
||||
# ====================================================================
|
||||
|
||||
|
||||
@given(
|
||||
"coverage boost input tokens {input_tokens:d} and output tokens {output_tokens:d}"
|
||||
)
|
||||
def step_cb_given_tokens(
|
||||
context: Context, input_tokens: int, output_tokens: int
|
||||
) -> None:
|
||||
context.cb_input_tokens = input_tokens
|
||||
context.cb_output_tokens = output_tokens
|
||||
|
||||
|
||||
@when("coverage boost _estimate_cost is called")
|
||||
def step_cb_call_estimate_cost(context: Context) -> None:
|
||||
context.cb_estimated_cost = estimate_cost(
|
||||
context.cb_input_tokens, context.cb_output_tokens
|
||||
)
|
||||
|
||||
|
||||
@then("coverage boost the estimated cost should be positive")
|
||||
def step_cb_cost_positive(context: Context) -> None:
|
||||
assert context.cb_estimated_cost > 0.0, (
|
||||
f"Expected positive cost, got {context.cb_estimated_cost}"
|
||||
)
|
||||
|
||||
|
||||
# ====================================================================
|
||||
# _history_to_langchain_messages
|
||||
# ====================================================================
|
||||
|
||||
|
||||
@given("coverage boost session messages with roles SYSTEM, USER, ASSISTANT, and TOOL")
|
||||
def step_cb_messages_all_roles(context: Context) -> None:
|
||||
context.cb_history_messages = [
|
||||
SessionMessage(
|
||||
message_id="01KJ1N8YDN05N29P5RZV4SPDVZ",
|
||||
role=MessageRole.SYSTEM,
|
||||
content="You are helpful.",
|
||||
sequence=1,
|
||||
timestamp="2026-01-01T00:00:00Z",
|
||||
),
|
||||
SessionMessage(
|
||||
message_id="01KJ1N8YDN05N29P5RZV4SPDW0",
|
||||
role=MessageRole.USER,
|
||||
content="Hello",
|
||||
sequence=2,
|
||||
timestamp="2026-01-01T00:00:00Z",
|
||||
),
|
||||
SessionMessage(
|
||||
message_id="01KJ1N8YDN05N29P5RZV4SPDW1",
|
||||
role=MessageRole.ASSISTANT,
|
||||
content="Hi!",
|
||||
sequence=3,
|
||||
timestamp="2026-01-01T00:00:00Z",
|
||||
),
|
||||
SessionMessage(
|
||||
message_id="01KJ1N8YDN05N29P5RZV4SPDW2",
|
||||
role=MessageRole.TOOL,
|
||||
content="result",
|
||||
sequence=4,
|
||||
timestamp="2026-01-01T00:00:00Z",
|
||||
tool_call_id="tc1",
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
@when("coverage boost _history_to_langchain_messages is called")
|
||||
def step_cb_call_history_to_lc(context: Context) -> None:
|
||||
context.cb_lc_messages = history_to_langchain_messages(context.cb_history_messages)
|
||||
|
||||
|
||||
@then(
|
||||
"coverage boost the result should contain SystemMessage, HumanMessage, AIMessage, and ToolMessage"
|
||||
)
|
||||
def step_cb_assert_lc_types(context: Context) -> None:
|
||||
from langchain_core.messages import AIMessage, HumanMessage, SystemMessage
|
||||
from langchain_core.messages.tool import ToolMessage
|
||||
|
||||
types = [type(m) for m in context.cb_lc_messages]
|
||||
assert SystemMessage in types, f"Missing SystemMessage in {types}"
|
||||
assert HumanMessage in types, f"Missing HumanMessage in {types}"
|
||||
assert AIMessage in types, f"Missing AIMessage in {types}"
|
||||
assert ToolMessage in types, f"Missing ToolMessage in {types}"
|
||||
|
||||
|
||||
@given("coverage boost a session message with an unknown role")
|
||||
def step_cb_message_unknown_role(context: Context) -> None:
|
||||
msg = MagicMock(spec=SessionMessage)
|
||||
msg.role = "NOT_A_REAL_ROLE"
|
||||
msg.content = "test content"
|
||||
msg.tool_call_id = None
|
||||
context.cb_history_messages = [msg]
|
||||
|
||||
|
||||
@then("coverage boost the result should contain a HumanMessage")
|
||||
def step_cb_result_contains_human(context: Context) -> None:
|
||||
from langchain_core.messages import HumanMessage
|
||||
|
||||
assert len(context.cb_lc_messages) > 0, "Expected at least one message"
|
||||
assert isinstance(context.cb_lc_messages[0], HumanMessage), (
|
||||
f"Expected HumanMessage, got {type(context.cb_lc_messages[0])}"
|
||||
)
|
||||
|
||||
|
||||
@given("coverage boost an empty list of session messages")
|
||||
def step_cb_empty_history(context: Context) -> None:
|
||||
context.cb_history_messages = []
|
||||
|
||||
|
||||
@then("coverage boost the result should be an empty list")
|
||||
def step_cb_empty_list(context: Context) -> None:
|
||||
assert context.cb_lc_messages == [], (
|
||||
f"Expected empty list, got {context.cb_lc_messages}"
|
||||
)
|
||||
|
||||
|
||||
# ====================================================================
|
||||
# LangChainSessionCaller.invoke() tool_results
|
||||
# ====================================================================
|
||||
|
||||
|
||||
@given("coverage boost a LangChainSessionCaller with a stub LLM and empty history")
|
||||
def step_cb_caller_with_stub(context: Context) -> None:
|
||||
stub_llm = MagicMock()
|
||||
stub_llm.invoke.return_value = MagicMock(
|
||||
content="response text",
|
||||
tool_calls=[],
|
||||
response_metadata={},
|
||||
usage_metadata={},
|
||||
)
|
||||
context.cb_caller = LangChainSessionCaller(llm=stub_llm, history=[])
|
||||
context.cb_stub_llm = stub_llm
|
||||
|
||||
|
||||
@when(
|
||||
"coverage boost invoke is called with tool_results containing one success and one failure"
|
||||
)
|
||||
def step_cb_invoke_tool_results(context: Context) -> None:
|
||||
context.cb_caller.invoke(prompt="test prompt", tool_schemas=[])
|
||||
context.cb_tool_result_response = context.cb_caller.invoke(
|
||||
prompt="test prompt",
|
||||
tool_schemas=[],
|
||||
tool_results=[
|
||||
{
|
||||
"success": True,
|
||||
"output": {"result": "ok"},
|
||||
"call_id": "c1",
|
||||
"tool_name": "tool1",
|
||||
},
|
||||
{
|
||||
"success": False,
|
||||
"error": "failed",
|
||||
"call_id": "c2",
|
||||
"tool_name": "tool2",
|
||||
},
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
@then("coverage boost the accumulated messages should include tool result messages")
|
||||
def step_cb_accumulated_has_tool_results(context: Context) -> None:
|
||||
from langchain_core.messages.tool import ToolMessage
|
||||
|
||||
tool_msgs = [
|
||||
m for m in context.cb_caller._accumulated if isinstance(m, ToolMessage)
|
||||
]
|
||||
assert len(tool_msgs) >= 2, f"Expected >= 2 ToolMessages, got {len(tool_msgs)}"
|
||||
|
||||
|
||||
# ====================================================================
|
||||
# LangChainSessionCaller.invoke() with tool_calls in response
|
||||
# ====================================================================
|
||||
|
||||
|
||||
@given(
|
||||
"coverage boost a LangChainSessionCaller with a stub LLM that returns tool calls"
|
||||
)
|
||||
def step_cb_caller_tool_calls(context: Context) -> None:
|
||||
stub_llm = MagicMock()
|
||||
resp = MagicMock(
|
||||
content="response with tool calls",
|
||||
tool_calls=[
|
||||
{"name": "test_tool", "args": {"arg1": "val1"}, "id": "call_123"},
|
||||
],
|
||||
response_metadata={},
|
||||
usage_metadata={},
|
||||
)
|
||||
stub_llm.invoke.return_value = resp
|
||||
context.cb_caller = LangChainSessionCaller(llm=stub_llm, history=[])
|
||||
|
||||
|
||||
@when("coverage boost invoke is called for the first time")
|
||||
def step_cb_invoke_first_time(context: Context) -> None:
|
||||
context.cb_first_invoke_result = context.cb_caller.invoke(
|
||||
prompt="test prompt", tool_schemas=[]
|
||||
)
|
||||
|
||||
|
||||
@then("coverage boost the LLMResponse should contain the extracted tool calls")
|
||||
def step_cb_llm_response_has_tool_calls(context: Context) -> None:
|
||||
assert isinstance(context.cb_first_invoke_result, LLMResponse)
|
||||
assert len(context.cb_first_invoke_result.tool_calls) >= 1, (
|
||||
f"Expected >= 1 tool calls, got {len(context.cb_first_invoke_result.tool_calls)}"
|
||||
)
|
||||
tc = context.cb_first_invoke_result.tool_calls[0]
|
||||
assert tc.name == "test_tool"
|
||||
assert tc.arguments == {"arg1": "val1"}
|
||||
|
||||
|
||||
# ====================================================================
|
||||
# _build_lc_messages_from_history (SessionWorkflow method)
|
||||
# ====================================================================
|
||||
|
||||
|
||||
@given("coverage boost a SessionWorkflow with a stub service and no registry")
|
||||
def step_cb_workflow_with_stub(context: Context) -> None:
|
||||
svc = MagicMock()
|
||||
svc.get.return_value = MagicMock(actor_name="openai/gpt-4")
|
||||
svc.get_messages.return_value = []
|
||||
svc.append_message.return_value = MagicMock(spec=SessionMessage)
|
||||
svc.update_token_usage.return_value = None
|
||||
context.cb_workflow = SessionWorkflow(session_service=svc, provider_registry=None)
|
||||
|
||||
|
||||
@given("coverage boost session history without a system message")
|
||||
def step_cb_history_no_system(context: Context) -> None:
|
||||
context.cb_no_system_history = [
|
||||
SessionMessage(
|
||||
message_id="01KJ1N8YDN05N29P5RZV4SPDW0",
|
||||
role=MessageRole.USER,
|
||||
content="Hello",
|
||||
sequence=1,
|
||||
timestamp="2026-01-01T00:00:00Z",
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
@when("coverage boost _build_lc_messages_from_history is called with a prompt")
|
||||
def step_cb_call_build_lc(context: Context) -> None:
|
||||
context.cb_built_messages = context.cb_workflow._build_lc_messages_from_history(
|
||||
context.cb_no_system_history, "test prompt"
|
||||
)
|
||||
|
||||
|
||||
@then(
|
||||
"coverage boost the first message should be a SystemMessage with the session system prompt"
|
||||
)
|
||||
def step_cb_first_is_system(context: Context) -> None:
|
||||
from langchain_core.messages import SystemMessage
|
||||
|
||||
assert len(context.cb_built_messages) > 0, "Expected at least one message"
|
||||
first = context.cb_built_messages[0]
|
||||
assert isinstance(first, SystemMessage), (
|
||||
f"Expected SystemMessage, got {type(first)}"
|
||||
)
|
||||
assert "CleverAgents orchestrator" in first.content, (
|
||||
f"Expected system prompt content, got {first.content}"
|
||||
)
|
||||
|
||||
|
||||
# ====================================================================
|
||||
# _MinimalStubLLM
|
||||
# ====================================================================
|
||||
|
||||
|
||||
@given("coverage boost a _MinimalStubLLM instance")
|
||||
def step_cb_minimal_stub(context: Context) -> None:
|
||||
context.cb_stub = _MinimalStubLLM()
|
||||
|
||||
|
||||
@when("coverage boost invoke on the stub is called")
|
||||
def step_cb_stub_invoke(context: Context) -> None:
|
||||
context.cb_stub_response = context.cb_stub.invoke([])
|
||||
|
||||
|
||||
@then('coverage boost the stub response content should be "(no LLM configured)"')
|
||||
def step_cb_stub_content(context: Context) -> None:
|
||||
assert context.cb_stub_response.content == "(no LLM configured)"
|
||||
|
||||
|
||||
@then("coverage boost the stub response should have empty tool_calls")
|
||||
def step_cb_stub_empty_tool_calls(context: Context) -> None:
|
||||
assert context.cb_stub_response.tool_calls == []
|
||||
|
||||
|
||||
@when("coverage boost stream on the stub is called")
|
||||
def step_cb_stub_stream(context: Context) -> None:
|
||||
context.cb_stub_stream_chunks = list(context.cb_stub.stream([]))
|
||||
|
||||
|
||||
@then('coverage boost it should yield a chunk with content "(no LLM configured)"')
|
||||
def step_cb_stub_chunk_content(context: Context) -> None:
|
||||
assert len(context.cb_stub_stream_chunks) >= 1, "Expected at least one chunk"
|
||||
assert context.cb_stub_stream_chunks[0].content == "(no LLM configured)"
|
||||
@@ -15,18 +15,21 @@ from unittest.mock import MagicMock, patch
|
||||
from behave import given, then, when
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from cleveragents.application.services.session_workflow import TellResult
|
||||
from cleveragents.cli.commands.session import app as session_app
|
||||
from cleveragents.domain.models.core.session import SessionService
|
||||
|
||||
runner = CliRunner()
|
||||
|
||||
_ULID = "01KJ1N8YDN05N29P5RZV4SPDVZ"
|
||||
_STUB_RESPONSE = "Acknowledged: Hello world"
|
||||
|
||||
|
||||
def _mock_service() -> MagicMock:
|
||||
"""Create a MagicMock that passes isinstance checks for SessionService."""
|
||||
svc = MagicMock(spec=SessionService)
|
||||
svc.append_message.return_value = None
|
||||
svc.get_messages.return_value = []
|
||||
return svc
|
||||
|
||||
|
||||
@@ -37,6 +40,33 @@ def _patch_service(context: object, svc: MagicMock) -> None:
|
||||
context._cleanup_handlers.append(patcher.stop) # type: ignore[attr-defined]
|
||||
|
||||
|
||||
def _mock_workflow(response: str = _STUB_RESPONSE) -> MagicMock:
|
||||
"""Create a mock SessionWorkflow with canned tell and tell_stream returns."""
|
||||
wf = MagicMock()
|
||||
wf.tell.return_value = TellResult(
|
||||
session_id=_ULID,
|
||||
user_message="",
|
||||
assistant_message=response,
|
||||
input_tokens=5,
|
||||
output_tokens=5,
|
||||
cost=0.0,
|
||||
duration_ms=0.0,
|
||||
tool_calls_count=0,
|
||||
)
|
||||
wf.tell_stream.return_value = iter([response])
|
||||
return wf
|
||||
|
||||
|
||||
def _patch_workflow(context: object, wf: MagicMock) -> None:
|
||||
"""Patch _build_session_workflow and register cleanup."""
|
||||
patcher = patch(
|
||||
"cleveragents.cli.commands.session._build_session_workflow",
|
||||
return_value=wf,
|
||||
)
|
||||
patcher.start()
|
||||
context._cleanup_handlers.append(patcher.stop) # type: ignore[attr-defined]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Background
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -45,8 +75,11 @@ def _patch_service(context: object, svc: MagicMock) -> None:
|
||||
@given("a session tell stream redaction mock service")
|
||||
def step_setup_mock_service(context: object) -> None:
|
||||
svc = _mock_service()
|
||||
wf = _mock_workflow()
|
||||
_patch_service(context, svc)
|
||||
_patch_workflow(context, wf)
|
||||
context.mock_svc = svc # type: ignore[attr-defined]
|
||||
context.mock_wf = wf # type: ignore[attr-defined]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -117,10 +150,15 @@ def step_stdout_write_not_called(context: object) -> None:
|
||||
return original_write(s)
|
||||
|
||||
svc = _mock_service()
|
||||
wf = _mock_workflow()
|
||||
sensitive_prompt = "my-api-key=sk-secret1234567890abcdef"
|
||||
|
||||
with (
|
||||
mock_patch("cleveragents.cli.commands.session._service", svc),
|
||||
mock_patch(
|
||||
"cleveragents.cli.commands.session._build_session_workflow",
|
||||
return_value=wf,
|
||||
),
|
||||
mock_patch("sys.stdout.write", side_effect=tracking_write),
|
||||
):
|
||||
runner.invoke(
|
||||
@@ -159,8 +197,8 @@ def step_output_via_console(context: object) -> None:
|
||||
that it was NOT split into individual characters in the captured output.
|
||||
"""
|
||||
result = context.result # type: ignore[attr-defined]
|
||||
# The assistant content for "Hello world" is "Acknowledged: Hello world"
|
||||
expected = "Acknowledged: Hello world"
|
||||
# After real LLM wiring, the stub workflow returns _STUB_RESPONSE.
|
||||
expected = _STUB_RESPONSE
|
||||
assert expected in result.output, (
|
||||
f"Expected '{expected}' in streaming output.\nGot:\n{result.output}"
|
||||
)
|
||||
|
||||
@@ -47,6 +47,7 @@ dependencies = [
|
||||
"tomlkit>=0.13.0", # TOML writing with comment preservation for config CLI
|
||||
"tenacity>=8.2.0", # Retry framework for service layer resilience
|
||||
"aiohttp>=3.13.4", # CVE-2026-34515 mitigation: open redirect vulnerability
|
||||
"pyyaml>=6.0.3", # Security: address known YAML parsing vulnerabilities
|
||||
"a2a-sdk>=0.3.0,<1.0.0", # A2A Python SDK — required transport for local (stdio) and server (HTTP) modes (ADR-047); pinned <1.0.0 (removed legacy A2AClient)
|
||||
]
|
||||
|
||||
|
||||
@@ -97,7 +97,7 @@ def list_operations() -> None:
|
||||
facade = A2aLocalFacade()
|
||||
ops = facade.list_operations()
|
||||
expected = {"session.create", "plan.create", "plan.execute", "context.get"}
|
||||
if expected.issubset(set(ops)) and len(ops) == 42:
|
||||
if expected.issubset(set(ops)) and len(ops) == 44:
|
||||
print("a2a-list-operations-ok")
|
||||
else:
|
||||
print(f"FAIL: ops={ops}", file=sys.stderr)
|
||||
|
||||
@@ -10,7 +10,7 @@ import sys
|
||||
import tempfile
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
# Ensure local source tree is importable
|
||||
_SRC = str(Path(__file__).resolve().parents[1] / "src")
|
||||
@@ -20,6 +20,7 @@ if _SRC not in sys.path:
|
||||
from typer.testing import CliRunner # noqa: E402
|
||||
from ulid import ULID # noqa: E402
|
||||
|
||||
from cleveragents.application.services.session_workflow import TellResult # noqa: E402
|
||||
from cleveragents.cli.commands import session as session_mod # noqa: E402
|
||||
from cleveragents.cli.commands.session import app as session_app # noqa: E402
|
||||
from cleveragents.domain.models.core.session import ( # noqa: E402
|
||||
@@ -284,13 +285,27 @@ def import_rich_panels() -> None:
|
||||
def tell_message() -> None:
|
||||
sid = str(ULID())
|
||||
svc = _setup_service()
|
||||
svc.get.return_value = _mock_session(session_id=sid)
|
||||
svc.append_message.side_effect = [
|
||||
_mock_message(MessageRole.USER, "Hello", 0),
|
||||
_mock_message(MessageRole.ASSISTANT, "Acknowledged: Hello", 1),
|
||||
]
|
||||
# Session must have actor_name so tell can proceed without --actor flag.
|
||||
svc.get.return_value = _mock_session(session_id=sid, actor_name="openai/gpt-4")
|
||||
svc.get_messages.return_value = []
|
||||
# Build a mock workflow that returns a canned TellResult.
|
||||
mock_wf = MagicMock()
|
||||
mock_wf.tell.return_value = TellResult(
|
||||
session_id=sid,
|
||||
user_message="Hello",
|
||||
assistant_message="Acknowledged: Hello",
|
||||
input_tokens=5,
|
||||
output_tokens=5,
|
||||
cost=0.0,
|
||||
duration_ms=0.0,
|
||||
tool_calls_count=0,
|
||||
)
|
||||
try:
|
||||
result = runner.invoke(session_app, ["tell", "--session", sid, "Hello"])
|
||||
with patch(
|
||||
"cleveragents.cli.commands.session._build_session_workflow",
|
||||
return_value=mock_wf,
|
||||
):
|
||||
result = runner.invoke(session_app, ["tell", "--session", sid, "Hello"])
|
||||
assert result.exit_code == 0, f"exit={result.exit_code}: {result.output}"
|
||||
assert "Acknowledged" in result.output
|
||||
print("session-cli-tell-message-ok")
|
||||
|
||||
@@ -0,0 +1,213 @@
|
||||
"""Helper script for session_tell_llm.robot integration tests.
|
||||
|
||||
Each subcommand is a self-contained check that prints a sentinel on success.
|
||||
Tests verify SessionWorkflow.tell() / tell_stream() behaviour with a stub LLM.
|
||||
|
||||
No real API keys or network access required — all LLM calls are intercepted
|
||||
by a monkey-patched ``_resolve_llm`` that returns a minimal stub.
|
||||
"""
|
||||
|
||||
# ruff: noqa: E402, I001
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
# Ensure local source tree is importable
|
||||
_SRC = str(Path(__file__).resolve().parents[1] / "src")
|
||||
if _SRC not in sys.path:
|
||||
sys.path.insert(0, _SRC)
|
||||
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from cleveragents.application.services.session_service import (
|
||||
PersistentSessionService,
|
||||
)
|
||||
from cleveragents.application.services.session_workflow import (
|
||||
SessionWorkflow,
|
||||
)
|
||||
from cleveragents.domain.models.core.session import (
|
||||
MessageRole,
|
||||
SessionActorNotConfiguredError,
|
||||
)
|
||||
from cleveragents.infrastructure.database.models import Base
|
||||
from cleveragents.infrastructure.database.repositories import (
|
||||
SessionMessageRepository,
|
||||
SessionRepository,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Stub LLM — returns deterministic responses without LLM API calls
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_STUB_TEXT = "This is a real response from the stub LLM actor."
|
||||
|
||||
|
||||
class _StubLLM:
|
||||
"""Minimal LLM stub that behaves like a LangChain chat model."""
|
||||
|
||||
def invoke(self, messages: Any, **kwargs: Any) -> Any:
|
||||
class _Resp:
|
||||
content = _STUB_TEXT
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.tool_calls: list[Any] = []
|
||||
self.response_metadata: dict[str, Any] = {
|
||||
"usage": {"input_tokens": 10, "output_tokens": 20}
|
||||
}
|
||||
|
||||
return _Resp()
|
||||
|
||||
def stream(self, messages: Any, **kwargs: Any) -> Iterator[Any]:
|
||||
for word in _STUB_TEXT.split():
|
||||
|
||||
class _Chunk:
|
||||
def __init__(self, t: str) -> None:
|
||||
self.content = t + " "
|
||||
|
||||
yield _Chunk(word)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixture helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _make_db_session_factory() -> Any:
|
||||
"""Create an in-memory SQLite engine and return a SQLAlchemy session factory."""
|
||||
engine = create_engine("sqlite:///:memory:", echo=False)
|
||||
Base.metadata.create_all(engine)
|
||||
return sessionmaker(bind=engine, expire_on_commit=False)
|
||||
|
||||
|
||||
def _make_workflow(
|
||||
actor_name: str | None = "openai/gpt-4",
|
||||
llm_factory: Any = None,
|
||||
) -> tuple[SessionWorkflow, Any]:
|
||||
"""Build a real SessionWorkflow backed by an in-memory SQLite database."""
|
||||
db_factory = _make_db_session_factory()
|
||||
db_session = db_factory()
|
||||
|
||||
def get_db() -> Any:
|
||||
return db_session
|
||||
|
||||
session_repo = SessionRepository(get_db)
|
||||
message_repo = SessionMessageRepository(get_db)
|
||||
svc = PersistentSessionService(session_repo=session_repo, message_repo=message_repo)
|
||||
|
||||
# Create a real session in the DB
|
||||
session = svc.create(actor_name=actor_name)
|
||||
|
||||
factory = llm_factory if llm_factory is not None else lambda _actor: _StubLLM()
|
||||
wf = SessionWorkflow(session_service=svc, llm_factory=factory)
|
||||
# No monkey-patching needed — llm_factory provides clean DI (addresses M8)
|
||||
|
||||
return wf, session
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Subcommands
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def tell_persists() -> None:
|
||||
"""Verify tell() returns real LLM response and persists messages."""
|
||||
wf, session = _make_workflow(actor_name="openai/gpt-4")
|
||||
result = wf.tell(
|
||||
session_id=session.session_id,
|
||||
prompt="What can you help me with?",
|
||||
)
|
||||
assert result.assistant_message == _STUB_TEXT, (
|
||||
f"Expected stub response, got: {result.assistant_message!r}"
|
||||
)
|
||||
assert result.input_tokens > 0, "input_tokens should be > 0"
|
||||
assert result.output_tokens > 0, "output_tokens should be > 0"
|
||||
# Verify messages were persisted via the service
|
||||
messages = wf._session_service.get_messages(session.session_id)
|
||||
roles = [m.role for m in messages]
|
||||
assert MessageRole.USER in roles, "User message not persisted"
|
||||
assert MessageRole.ASSISTANT in roles, "Assistant message not persisted"
|
||||
print("session-tell-persists-ok")
|
||||
|
||||
|
||||
def tell_no_actor() -> None:
|
||||
"""Verify tell() raises SessionActorNotConfiguredError when no actor set."""
|
||||
wf, session = _make_workflow(actor_name=None)
|
||||
try:
|
||||
wf.tell(session_id=session.session_id, prompt="Hello")
|
||||
print(
|
||||
"FAIL: expected SessionActorNotConfiguredError not raised",
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(1)
|
||||
except SessionActorNotConfiguredError as exc:
|
||||
assert "actor" in str(exc).lower(), (
|
||||
f"Error message should mention 'actor', got: {exc}"
|
||||
)
|
||||
print("session-tell-no-actor-ok")
|
||||
|
||||
|
||||
def tell_stream() -> None:
|
||||
"""Verify tell_stream() yields tokens from the stub LLM."""
|
||||
wf, session = _make_workflow(actor_name="openai/gpt-4")
|
||||
tokens: list[str] = []
|
||||
for token in wf.tell_stream(
|
||||
session_id=session.session_id,
|
||||
prompt="Hello",
|
||||
):
|
||||
tokens.append(token)
|
||||
assert len(tokens) > 0, "Expected tokens to be yielded, got none"
|
||||
full_response = "".join(tokens).strip()
|
||||
assert len(full_response) > 0, f"Expected non-empty stream, got: {full_response!r}"
|
||||
print("session-tell-stream-ok")
|
||||
|
||||
|
||||
def tell_actor_override() -> None:
|
||||
"""Verify tell() uses actor_override instead of session actor."""
|
||||
captured_actor: list[str] = []
|
||||
|
||||
def _tracking_resolve(actor_name: str) -> Any:
|
||||
captured_actor.append(actor_name)
|
||||
return _StubLLM()
|
||||
|
||||
wf, session = _make_workflow(
|
||||
actor_name="anthropic/claude-3-haiku",
|
||||
llm_factory=_tracking_resolve,
|
||||
)
|
||||
|
||||
result = wf.tell(
|
||||
session_id=session.session_id,
|
||||
prompt="Hello",
|
||||
actor_override="openai/gpt-4",
|
||||
)
|
||||
assert captured_actor, "Actor resolver was never called"
|
||||
assert captured_actor[0] == "openai/gpt-4", (
|
||||
f"Expected 'openai/gpt-4' actor, got {captured_actor[0]!r}"
|
||||
)
|
||||
assert result.assistant_message is not None
|
||||
print("session-tell-actor-override-ok")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Entry point
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
COMMANDS = {
|
||||
"tell-persists": tell_persists,
|
||||
"tell-no-actor": tell_no_actor,
|
||||
"tell-stream": tell_stream,
|
||||
"tell-actor-override": tell_actor_override,
|
||||
}
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) < 2 or sys.argv[1] not in COMMANDS:
|
||||
print(
|
||||
f"Usage: {sys.argv[0]} <command>\nCommands: {', '.join(COMMANDS)}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(1)
|
||||
COMMANDS[sys.argv[1]]()
|
||||
@@ -0,0 +1,56 @@
|
||||
*** Settings ***
|
||||
Documentation Integration tests for session tell real LLM actor invocation.
|
||||
...
|
||||
... Verifies that ``agents session tell`` calls ``SessionWorkflow.tell()``
|
||||
... (not the stub), persists the response, and handles the no-actor
|
||||
... error case. Uses a stub LLM actor so no real API keys are
|
||||
... required.
|
||||
Resource ${CURDIR}/common.resource
|
||||
Suite Setup Setup Test Environment With Database Isolation
|
||||
Suite Teardown Cleanup Test Environment
|
||||
|
||||
*** Variables ***
|
||||
${HELPER} ${CURDIR}/helper_session_tell_llm.py
|
||||
|
||||
*** Test Cases ***
|
||||
Session Tell Workflow Returns Real Response
|
||||
[Documentation] Verify SessionWorkflow.tell() returns a real (stub) response
|
||||
... and persists user + assistant messages.
|
||||
[Tags] session_tell_llm tdd_issue tdd_issue_5784
|
||||
${result}= Run Process ${PYTHON} ${HELPER} tell-persists
|
||||
... cwd=${WORKSPACE}
|
||||
Log ${result.stdout}
|
||||
Log ${result.stderr}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} session-tell-persists-ok
|
||||
|
||||
Session Tell No Actor Raises Clear Error
|
||||
[Documentation] Verify SessionWorkflow.tell() raises SessionActorNotConfiguredError
|
||||
... when no actor is configured and no --actor override is supplied.
|
||||
[Tags] session_tell_llm tdd_issue tdd_issue_5784
|
||||
${result}= Run Process ${PYTHON} ${HELPER} tell-no-actor
|
||||
... cwd=${WORKSPACE}
|
||||
Log ${result.stdout}
|
||||
Log ${result.stderr}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} session-tell-no-actor-ok
|
||||
|
||||
Session Tell Stream Yields Tokens
|
||||
[Documentation] Verify SessionWorkflow.tell_stream() yields tokens from the LLM.
|
||||
[Tags] session_tell_llm tdd_issue tdd_issue_5784
|
||||
${result}= Run Process ${PYTHON} ${HELPER} tell-stream
|
||||
... cwd=${WORKSPACE}
|
||||
Log ${result.stdout}
|
||||
Log ${result.stderr}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} session-tell-stream-ok
|
||||
|
||||
Session Tell Actor Override Works
|
||||
[Documentation] Verify that the actor_override parameter overrides the session actor.
|
||||
[Tags] session_tell_llm tdd_issue tdd_issue_5784
|
||||
${result}= Run Process ${PYTHON} ${HELPER} tell-actor-override
|
||||
... cwd=${WORKSPACE}
|
||||
Log ${result.stdout}
|
||||
Log ${result.stderr}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} session-tell-actor-override-ok
|
||||
+115
-18
@@ -21,7 +21,7 @@ response so the facade never crashes due to missing wiring.
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from typing import TYPE_CHECKING, Any
|
||||
from typing import Any, cast
|
||||
|
||||
import structlog
|
||||
from ulid import ULID
|
||||
@@ -30,22 +30,27 @@ from cleveragents.a2a.errors import (
|
||||
A2aOperationNotFoundError,
|
||||
map_domain_error,
|
||||
)
|
||||
from cleveragents.a2a.events import A2aEventQueue
|
||||
from cleveragents.a2a.models import (
|
||||
A2aErrorDetail,
|
||||
A2aRequest,
|
||||
A2aResponse,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from cleveragents.a2a.events import A2aEventQueue
|
||||
from cleveragents.application.services.plan_lifecycle_service import (
|
||||
PlanLifecycleService,
|
||||
)
|
||||
from cleveragents.application.services.resource_registry_service import (
|
||||
ResourceRegistryService,
|
||||
)
|
||||
from cleveragents.domain.models.core.session import SessionService
|
||||
from cleveragents.tool.registry import ToolRegistry
|
||||
from cleveragents.application.services.plan_lifecycle_service import (
|
||||
PlanLifecycleService,
|
||||
)
|
||||
from cleveragents.application.services.resource_registry_service import (
|
||||
ResourceRegistryService,
|
||||
)
|
||||
from cleveragents.application.services.session_workflow import SessionWorkflow
|
||||
from cleveragents.core.exceptions import DatabaseError
|
||||
from cleveragents.domain.models.core.session import (
|
||||
SessionActorNotConfiguredError,
|
||||
SessionNotFoundError,
|
||||
SessionService,
|
||||
)
|
||||
from cleveragents.providers.registry import ProviderRegistry
|
||||
from cleveragents.tool.registry import ToolRegistry
|
||||
|
||||
logger: structlog.stdlib.BoundLogger = structlog.get_logger(__name__)
|
||||
|
||||
@@ -53,6 +58,12 @@ logger: structlog.stdlib.BoundLogger = structlog.get_logger(__name__)
|
||||
# Supported operations
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Standard A2A protocol operations (message/send and message/stream).
|
||||
_STANDARD_A2A_OPERATIONS: list[str] = [
|
||||
"message/send",
|
||||
"message/stream",
|
||||
]
|
||||
|
||||
# Spec-aligned _cleveragents/ extension method names (ADR-047).
|
||||
_EXTENSION_OPERATIONS: list[str] = [
|
||||
# Plan lifecycle
|
||||
@@ -109,7 +120,9 @@ _LEGACY_OPERATIONS: list[str] = [
|
||||
"event.subscribe",
|
||||
]
|
||||
|
||||
_SUPPORTED_OPERATIONS: list[str] = _EXTENSION_OPERATIONS + _LEGACY_OPERATIONS
|
||||
_SUPPORTED_OPERATIONS: list[str] = (
|
||||
_STANDARD_A2A_OPERATIONS + _EXTENSION_OPERATIONS + _LEGACY_OPERATIONS
|
||||
)
|
||||
|
||||
|
||||
class A2aLocalFacade:
|
||||
@@ -134,25 +147,33 @@ class A2aLocalFacade:
|
||||
|
||||
@property
|
||||
def _session_service(self) -> SessionService | None:
|
||||
return self._services.get("session_service") # type: ignore[return-value]
|
||||
return cast(SessionService | None, self._services.get("session_service"))
|
||||
|
||||
@property
|
||||
def _provider_registry(self) -> ProviderRegistry | None:
|
||||
return cast(ProviderRegistry | None, self._services.get("provider_registry"))
|
||||
|
||||
@property
|
||||
def _session_workflow(self) -> SessionWorkflow | None:
|
||||
return cast(SessionWorkflow | None, self._services.get("session_workflow"))
|
||||
|
||||
@property
|
||||
def _plan_lifecycle_service(self) -> PlanLifecycleService | None:
|
||||
svc: object | None = self._services.get("plan_lifecycle_service")
|
||||
return svc # type: ignore[return-value]
|
||||
return cast(PlanLifecycleService | None, svc)
|
||||
|
||||
@property
|
||||
def _tool_registry(self) -> ToolRegistry | None:
|
||||
return self._services.get("tool_registry") # type: ignore[return-value]
|
||||
return cast(ToolRegistry | None, self._services.get("tool_registry"))
|
||||
|
||||
@property
|
||||
def _resource_registry_service(self) -> ResourceRegistryService | None:
|
||||
svc: object | None = self._services.get("resource_registry_service")
|
||||
return svc # type: ignore[return-value]
|
||||
return cast(ResourceRegistryService | None, svc)
|
||||
|
||||
@property
|
||||
def _event_queue(self) -> A2aEventQueue | None:
|
||||
return self._services.get("event_queue") # type: ignore[return-value]
|
||||
return cast(A2aEventQueue | None, self._services.get("event_queue"))
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Public API
|
||||
@@ -164,6 +185,10 @@ class A2aLocalFacade:
|
||||
Returns an :class:`A2aResponse` with ``result`` set on success or
|
||||
``error`` set when the operation fails. Domain exceptions
|
||||
are mapped to A2A error codes via :func:`map_domain_error`.
|
||||
|
||||
Handlers may re-raise domain exceptions (e.g.
|
||||
``SessionNotFoundError``) to signal that the caller should receive
|
||||
the original exception rather than an A2A error envelope.
|
||||
"""
|
||||
if not isinstance(request, A2aRequest):
|
||||
raise TypeError("request must be an A2aRequest instance")
|
||||
@@ -184,6 +209,11 @@ class A2aLocalFacade:
|
||||
)
|
||||
except A2aOperationNotFoundError:
|
||||
raise
|
||||
except (SessionNotFoundError, SessionActorNotConfiguredError, DatabaseError):
|
||||
# Let domain exceptions propagate — they are re-raised by
|
||||
# handlers that want the caller (e.g. CLI) to receive the
|
||||
# original exception type for proper error handling.
|
||||
raise
|
||||
except Exception as exc:
|
||||
elapsed = (time.monotonic() - start) * 1000.0
|
||||
code, message = map_domain_error(exc)
|
||||
@@ -242,6 +272,9 @@ class A2aLocalFacade:
|
||||
"""
|
||||
if self._handler_map is None:
|
||||
self._handler_map = {
|
||||
# --- Standard A2A operations ---
|
||||
"message/send": self._handle_message_send,
|
||||
"message/stream": self._handle_message_stream,
|
||||
# --- _cleveragents/ extension methods (spec-aligned) ---
|
||||
# Plan lifecycle
|
||||
"_cleveragents/plan/use": self._handle_plan_create,
|
||||
@@ -336,6 +369,70 @@ class A2aLocalFacade:
|
||||
self._cleanup_session_devcontainers(session_id)
|
||||
return {"status": "closed"}
|
||||
|
||||
def _get_or_build_session_workflow(self) -> SessionWorkflow:
|
||||
"""Return the registered ``SessionWorkflow`` or build a fresh one."""
|
||||
workflow = self._session_workflow
|
||||
if workflow is not None:
|
||||
return workflow
|
||||
|
||||
# Lazy-build using registered services.
|
||||
svc = self._session_service
|
||||
if svc is None:
|
||||
raise RuntimeError("Session service not available — create a session first")
|
||||
registry = self._provider_registry
|
||||
return SessionWorkflow(
|
||||
session_service=svc,
|
||||
provider_registry=registry,
|
||||
)
|
||||
|
||||
def _handle_message_send(self, params: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Handle A2A ``message/send`` — invoke orchestrator actor (non-streaming).
|
||||
|
||||
Required params:
|
||||
``session_id``: Target session ULID.
|
||||
``message``: User message text.
|
||||
|
||||
Optional params:
|
||||
``actor``: Actor override (``namespace/name``).
|
||||
"""
|
||||
session_id = params.get("session_id", "")
|
||||
message = params.get("message", "")
|
||||
actor_override: str | None = params.get("actor") or None
|
||||
|
||||
if not session_id:
|
||||
raise ValueError("session_id is required")
|
||||
if not message:
|
||||
raise ValueError("message is required")
|
||||
|
||||
workflow = self._get_or_build_session_workflow()
|
||||
result = workflow.tell(
|
||||
session_id=session_id,
|
||||
prompt=message,
|
||||
actor_override=actor_override,
|
||||
)
|
||||
|
||||
return {
|
||||
"session_id": result.session_id,
|
||||
"assistant_message": result.assistant_message,
|
||||
"usage": result.usage_dict(),
|
||||
}
|
||||
|
||||
def _handle_message_stream(self, params: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Handle A2A ``message/stream`` — invoke orchestrator actor (streaming).
|
||||
|
||||
In local-mode A2A, streaming is not natively supported via the
|
||||
synchronous :meth:`dispatch` response model. This handler falls
|
||||
back to non-streaming (``message/send``) and returns the complete
|
||||
response with a ``"streamed": false`` flag so callers can detect
|
||||
the degradation (addresses M4).
|
||||
|
||||
Callers that require real token-by-token streaming should use
|
||||
:meth:`~SessionWorkflow.tell_stream` directly.
|
||||
"""
|
||||
result = self._handle_message_send(params)
|
||||
result["streamed"] = False
|
||||
return result
|
||||
|
||||
def _cleanup_session_devcontainers(self, session_id: str) -> None:
|
||||
"""Best-effort stop of devcontainers associated with a session.
|
||||
|
||||
|
||||
@@ -0,0 +1,307 @@
|
||||
"""Session caller — LangChain-based LLM invocation for session conversations.
|
||||
|
||||
Implements the :class:`LangChainSessionCaller` that wraps a LangChain chat
|
||||
model with full session history context, plus token usage extraction and
|
||||
cost estimation helpers. Used by :class:`SessionWorkflow` in
|
||||
:mod:`cleveragents.application.services.session_workflow`.
|
||||
|
||||
Forgejo: #5784
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterator
|
||||
from typing import Any
|
||||
|
||||
from langchain_core.messages import AIMessage, HumanMessage, SystemMessage
|
||||
from langchain_core.messages.tool import ToolMessage
|
||||
|
||||
from cleveragents.domain.models.core.session import MessageRole
|
||||
from cleveragents.tool.actor_runtime import LLMResponse, LLMToolCall
|
||||
|
||||
# Token cost estimation constants (USD per 1K tokens).
|
||||
# These are rough defaults; real cost comes from the LLM response metadata.
|
||||
_COST_PER_1K_INPUT = 0.001
|
||||
_COST_PER_1K_OUTPUT = 0.002
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Token extraction and cost estimation helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def extract_content(response: Any) -> str:
|
||||
"""Extract plain text content from a LangChain response object."""
|
||||
raw_content = getattr(response, "content", None)
|
||||
if raw_content is None:
|
||||
raw_content = getattr(response, "text", None)
|
||||
if raw_content is None:
|
||||
raw_content = str(response)
|
||||
if isinstance(raw_content, list):
|
||||
parts: list[str] = []
|
||||
for chunk in raw_content:
|
||||
if isinstance(chunk, dict):
|
||||
parts.append(str(chunk.get("text") or chunk.get("content", "")))
|
||||
else:
|
||||
parts.append(str(chunk))
|
||||
return " ".join(parts)
|
||||
return str(raw_content)
|
||||
|
||||
|
||||
def extract_token_usage(response: Any) -> tuple[int, int]:
|
||||
"""Extract (input_tokens, output_tokens) from a LangChain response.
|
||||
|
||||
Attempts ``response_metadata`` and ``usage_metadata`` attributes.
|
||||
Returns (0, 0) when no usage information is present.
|
||||
"""
|
||||
# LangChain ≥ 0.2: response_metadata may carry provider-specific usage
|
||||
meta = getattr(response, "response_metadata", None) or {}
|
||||
usage = meta.get("usage") or meta.get("token_usage") or {}
|
||||
if usage:
|
||||
try:
|
||||
inp = int(usage.get("input_tokens") or usage.get("prompt_tokens") or 0)
|
||||
out = int(usage.get("output_tokens") or usage.get("completion_tokens") or 0)
|
||||
return inp, out
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
|
||||
# LangChain ≥ 0.3: usage_metadata attribute
|
||||
usage_meta = getattr(response, "usage_metadata", None) or {}
|
||||
if usage_meta:
|
||||
try:
|
||||
inp = int(usage_meta.get("input_tokens") or 0)
|
||||
out = int(usage_meta.get("output_tokens") or 0)
|
||||
return inp, out
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
|
||||
return 0, 0
|
||||
|
||||
|
||||
def estimate_cost(input_tokens: int, output_tokens: int) -> float:
|
||||
"""Estimate LLM cost in USD from token counts."""
|
||||
raw = input_tokens * _COST_PER_1K_INPUT + output_tokens * _COST_PER_1K_OUTPUT
|
||||
return raw / 1000.0
|
||||
|
||||
|
||||
def history_to_langchain_messages(
|
||||
messages: list[Any],
|
||||
) -> list[Any]:
|
||||
"""Convert session message history to LangChain message objects.
|
||||
|
||||
Maps:
|
||||
- ``MessageRole.SYSTEM`` → ``SystemMessage``
|
||||
- ``MessageRole.USER`` → ``HumanMessage``
|
||||
- ``MessageRole.ASSISTANT`` → ``AIMessage``
|
||||
- ``MessageRole.TOOL`` → ``ToolMessage``
|
||||
|
||||
Returns an ordered list of LangChain message objects, empty if
|
||||
no history is available.
|
||||
"""
|
||||
lc_messages: list[Any] = []
|
||||
for msg in messages:
|
||||
role = msg.role
|
||||
content = msg.content
|
||||
if role == MessageRole.SYSTEM:
|
||||
lc_messages.append(SystemMessage(content=content))
|
||||
elif role == MessageRole.USER:
|
||||
lc_messages.append(HumanMessage(content=content))
|
||||
elif role == MessageRole.ASSISTANT:
|
||||
lc_messages.append(AIMessage(content=content))
|
||||
elif role == MessageRole.TOOL:
|
||||
tool_id = msg.tool_call_id or "unknown"
|
||||
lc_messages.append(ToolMessage(content=content, tool_call_id=tool_id))
|
||||
else:
|
||||
# Unknown role — default to human message to preserve context
|
||||
lc_messages.append(HumanMessage(content=content))
|
||||
return lc_messages
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# LangChainSessionCaller
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class LangChainSessionCaller:
|
||||
"""LLMCaller implementation for session conversations.
|
||||
|
||||
Wraps a LangChain chat model with full session history context.
|
||||
Implements the :class:`~cleveragents.tool.actor_runtime.LLMCaller`
|
||||
protocol so it can be used with :class:`ToolCallingRuntime`.
|
||||
|
||||
The caller is stateful within a single ``tell()`` invocation: it
|
||||
accumulates the LangChain message thread across tool-call loop
|
||||
iterations. Create a fresh instance for each ``tell()`` call.
|
||||
|
||||
Args:
|
||||
llm: A LangChain ``BaseLanguageModel`` (chat model).
|
||||
history: Pre-converted list of LangChain message objects
|
||||
representing the session history *before* the current
|
||||
user message.
|
||||
session_system_prompt: Optional system prompt prepended to
|
||||
the message list when no system message exists in history.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
llm: Any,
|
||||
history: list[Any],
|
||||
session_system_prompt: str | None = (
|
||||
"You are the CleverAgents orchestrator. You interpret the user's "
|
||||
"natural-language requests and help them manage their software "
|
||||
"development workflows. Be concise, accurate, and helpful."
|
||||
),
|
||||
) -> None:
|
||||
self._llm = llm
|
||||
self._last_response: Any = None
|
||||
# Build initial accumulated messages: optional system + history
|
||||
self._accumulated: list[Any] = []
|
||||
|
||||
# Prepend system prompt only when history doesn't already have one
|
||||
has_system = any(isinstance(m, SystemMessage) for m in history)
|
||||
if session_system_prompt and not has_system:
|
||||
self._accumulated.append(SystemMessage(content=session_system_prompt))
|
||||
self._accumulated.extend(history)
|
||||
self._first_call = True
|
||||
# Accumulated real token usage across all invoke() calls (M1).
|
||||
self._total_input_tokens = 0
|
||||
self._total_output_tokens = 0
|
||||
|
||||
def invoke(
|
||||
self,
|
||||
prompt: str,
|
||||
tool_schemas: list[dict[str, Any]],
|
||||
tool_results: list[dict[str, Any]] | None = None,
|
||||
actor_config: dict[str, Any] | None = None,
|
||||
) -> Any:
|
||||
"""Invoke the LLM with the accumulated conversation context.
|
||||
|
||||
On the first call (``tool_results is None``):
|
||||
Appends the user message and invokes the LLM.
|
||||
|
||||
On subsequent calls (``tool_results`` present):
|
||||
Appends the previous AI response (if available) and tool
|
||||
results, then invokes the LLM again.
|
||||
|
||||
Args:
|
||||
prompt: The original user prompt (used only on the first call).
|
||||
tool_schemas: Provider tool schemas (unused — no tool binding).
|
||||
tool_results: Tool execution results from the previous iteration.
|
||||
actor_config: Actor configuration dict (currently unused).
|
||||
|
||||
Returns:
|
||||
:class:`~cleveragents.tool.actor_runtime.LLMResponse` with the
|
||||
LLM's text response.
|
||||
"""
|
||||
if self._first_call:
|
||||
# First iteration: append the user's prompt and invoke
|
||||
self._accumulated.append(HumanMessage(content=prompt))
|
||||
self._first_call = False
|
||||
elif tool_results is not None:
|
||||
# Subsequent iteration: add previous AI response + tool results
|
||||
if self._last_response is not None:
|
||||
self._accumulated.append(self._last_response)
|
||||
for tr in tool_results:
|
||||
tool_result_content = (
|
||||
str(tr.get("output", {}))
|
||||
if tr.get("success")
|
||||
else tr.get("error", "error")
|
||||
)
|
||||
self._accumulated.append(
|
||||
ToolMessage(
|
||||
content=tool_result_content,
|
||||
tool_call_id=(
|
||||
tr.get("call_id") or tr.get("tool_name", "unknown")
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
# Invoke the LLM (no tool binding — session tell is conversational)
|
||||
response = self._llm.invoke(self._accumulated)
|
||||
self._last_response = response
|
||||
|
||||
# Extract real token usage from the LLM response metadata (M1).
|
||||
inp, out = extract_token_usage(response)
|
||||
self._total_input_tokens += inp
|
||||
self._total_output_tokens += out
|
||||
|
||||
content = extract_content(response)
|
||||
|
||||
# Extract any tool calls from the response
|
||||
tool_calls: list[LLMToolCall] = []
|
||||
raw_tool_calls = getattr(response, "tool_calls", None) or []
|
||||
for tc in raw_tool_calls:
|
||||
if isinstance(tc, dict):
|
||||
name = tc.get("name") or tc.get("id", "")
|
||||
args = tc.get("args") or tc.get("arguments") or {}
|
||||
call_id = tc.get("id", "")
|
||||
if name:
|
||||
tool_calls.append(
|
||||
LLMToolCall(name=name, arguments=args, call_id=call_id)
|
||||
)
|
||||
|
||||
return LLMResponse(content=content, tool_calls=tool_calls)
|
||||
|
||||
def get_token_usage(self) -> tuple[int, int]:
|
||||
"""Return accumulated ``(input_tokens, output_tokens)`` across all calls.
|
||||
|
||||
Tokens are extracted from real LLM response metadata via
|
||||
:func:`extract_token_usage`. Returns ``(0, 0)`` when the LLM
|
||||
provides no usage metadata (stub / minimal mode).
|
||||
"""
|
||||
return self._total_input_tokens, self._total_output_tokens
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Minimal stub LLM response helpers (used by SessionWorkflow._make_stub_llm)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def estimate_tokens(prompt: str, response: str) -> tuple[int, int]:
|
||||
"""Estimate token counts using a simple word-based heuristic.
|
||||
|
||||
Returns ``(input_tokens, output_tokens)`` based on the
|
||||
conventional approximation of ~4 characters per token.
|
||||
"""
|
||||
input_tokens = max(1, len(prompt) // 4)
|
||||
output_tokens = max(1, len(response) // 4)
|
||||
return input_tokens, output_tokens
|
||||
|
||||
|
||||
class _StubResp:
|
||||
"""Minimal LLM response object for stub mode."""
|
||||
|
||||
content = "(no LLM configured)"
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.tool_calls: list[Any] = []
|
||||
self.response_metadata: dict[str, Any] = {}
|
||||
|
||||
|
||||
class _StubChunk:
|
||||
"""Minimal streaming chunk for stub mode."""
|
||||
|
||||
content = "(no LLM configured)"
|
||||
|
||||
|
||||
class _MinimalStubLLM:
|
||||
"""Minimal LLM stub for environments without langchain_community."""
|
||||
|
||||
def invoke(self, messages: Any, **kwargs: Any) -> _StubResp: # pragma: no cover
|
||||
"""Return a stub response."""
|
||||
return _StubResp()
|
||||
|
||||
def stream(
|
||||
self, messages: Any, **kwargs: Any
|
||||
) -> Iterator[_StubChunk]: # pragma: no cover
|
||||
"""Return a single stub chunk."""
|
||||
yield _StubChunk()
|
||||
|
||||
|
||||
__all__ = [
|
||||
"LangChainSessionCaller",
|
||||
"estimate_cost",
|
||||
"extract_content",
|
||||
"extract_token_usage",
|
||||
"history_to_langchain_messages",
|
||||
]
|
||||
@@ -328,6 +328,23 @@ class PersistentSessionService(SessionService):
|
||||
except (KeyError, ValueError, TypeError) as exc:
|
||||
raise SessionImportError(f"Invalid import data: {exc}") from exc
|
||||
|
||||
def get_messages(self, session_id: str) -> list[SessionMessage]:
|
||||
"""Retrieve all messages for a session in order.
|
||||
|
||||
Args:
|
||||
session_id: The ULID of the session.
|
||||
|
||||
Returns:
|
||||
Ordered list of SessionMessage objects (by sequence number).
|
||||
|
||||
Raises:
|
||||
SessionNotFoundError: If no session with the given ID exists.
|
||||
"""
|
||||
session = self._session_repo.get_by_id(session_id)
|
||||
if session is None:
|
||||
raise SessionNotFoundError(f"Session '{session_id}' not found")
|
||||
return self._message_repo.get_for_session(session_id)
|
||||
|
||||
def update_token_usage(
|
||||
self,
|
||||
session_id: str,
|
||||
|
||||
@@ -0,0 +1,467 @@
|
||||
"""Session workflow — orchestrator LLM invocation for ``agents session tell``.
|
||||
|
||||
Replaces the M3 stub with real LLM actor invocation. The user prompt is
|
||||
sent to the session's bound orchestrator actor via the A2A ``message/send`` /
|
||||
``message/stream`` operations routed through :class:`A2aLocalFacade`.
|
||||
|
||||
Call chain in local mode::
|
||||
|
||||
tell CLI ──▸ A2aLocalFacade.dispatch("message/send")
|
||||
──▸ SessionWorkflow.tell()
|
||||
──▸ LangChainSessionCaller.invoke()
|
||||
──▸ LangChain chat model (real LLM)
|
||||
|
||||
Forgejo: #5784
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from collections.abc import Callable, Iterator
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import structlog
|
||||
from langchain_core.messages import HumanMessage, SystemMessage
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
try:
|
||||
from langchain_community.llms import FakeListLLM
|
||||
except ImportError: # pragma: no cover — langchain_community optional
|
||||
FakeListLLM = None
|
||||
|
||||
from cleveragents.application.services.prompt_sanitizer import PromptSanitizer
|
||||
from cleveragents.application.services.session_caller import (
|
||||
LangChainSessionCaller,
|
||||
_MinimalStubLLM,
|
||||
estimate_cost,
|
||||
estimate_tokens,
|
||||
extract_content,
|
||||
history_to_langchain_messages,
|
||||
)
|
||||
from cleveragents.domain.models.core.session import (
|
||||
MessageRole,
|
||||
SessionActorNotConfiguredError,
|
||||
SessionMessage,
|
||||
)
|
||||
from cleveragents.tool.actor_runtime import (
|
||||
LLMCaller,
|
||||
ToolCallingRuntime,
|
||||
ToolCallRunResult,
|
||||
)
|
||||
from cleveragents.tool.registry import ToolRegistry
|
||||
from cleveragents.tool.runner import ToolRunner
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from cleveragents.domain.models.core.session import SessionService
|
||||
from cleveragents.providers.registry import ProviderRegistry
|
||||
|
||||
logger = structlog.get_logger(__name__)
|
||||
_sanitizer = PromptSanitizer()
|
||||
|
||||
# System prompt for the orchestrator actor in session context.
|
||||
_SESSION_SYSTEM_PROMPT = (
|
||||
"You are the CleverAgents orchestrator. You interpret the user's "
|
||||
"natural-language requests and help them manage their software "
|
||||
"development workflows. Be concise, accurate, and helpful."
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TellResult
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TellResult(BaseModel):
|
||||
"""Result of a :meth:`SessionWorkflow.tell` invocation.
|
||||
|
||||
Attributes:
|
||||
session_id: Session ULID.
|
||||
user_message: Original user prompt.
|
||||
assistant_message: Orchestrator response text.
|
||||
input_tokens: Input token count.
|
||||
output_tokens: Output token count.
|
||||
cost: Estimated cost in USD.
|
||||
duration_ms: Wall-clock LLM invocation duration (ms).
|
||||
tool_calls_count: Number of tool calls made.
|
||||
"""
|
||||
|
||||
session_id: str = Field(..., description="ULID of the session")
|
||||
user_message: str = Field(..., description="Original user prompt")
|
||||
assistant_message: str = Field(..., description="Orchestrator response text")
|
||||
input_tokens: int = Field(0, ge=0, description="Input token count")
|
||||
output_tokens: int = Field(0, ge=0, description="Output token count")
|
||||
cost: float = Field(0.0, ge=0.0, description="Estimated cost in USD")
|
||||
duration_ms: float = Field(0.0, ge=0.0, description="LLM invocation duration (ms)")
|
||||
tool_calls_count: int = Field(0, ge=0, description="Number of tool calls made")
|
||||
|
||||
model_config = ConfigDict(
|
||||
str_strip_whitespace=True,
|
||||
validate_assignment=True,
|
||||
)
|
||||
|
||||
def usage_dict(self) -> dict[str, Any]:
|
||||
"""Return a JSON-serializable usage summary dict."""
|
||||
return {
|
||||
"input_tokens": self.input_tokens,
|
||||
"output_tokens": self.output_tokens,
|
||||
"cost_usd": round(self.cost, 6),
|
||||
"duration_ms": round(self.duration_ms, 1),
|
||||
"tool_calls": self.tool_calls_count,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SessionWorkflow
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class SessionWorkflow:
|
||||
"""Orchestrates ``agents session tell`` — real LLM actor invocation.
|
||||
|
||||
Resolves the session's actor (or ``--actor`` override), invokes the LLM
|
||||
via :class:`ToolCallingRuntime`, persists the response, and records
|
||||
token usage via :class:`SessionService`.
|
||||
|
||||
Args:
|
||||
session_service: Session persistence service.
|
||||
provider_registry: Optional provider registry for LLM resolution.
|
||||
When ``None``, falls back to a stub response.
|
||||
tool_registry: Optional ``ToolRegistry`` (defaults to empty).
|
||||
llm_factory: Optional ``(actor_name: str) -> Any`` test injection point.
|
||||
Tests inject stub LLMs through this factory.
|
||||
max_iterations: Max tool-call loop iterations (default 25).
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
session_service: SessionService,
|
||||
provider_registry: ProviderRegistry | None = None,
|
||||
tool_registry: ToolRegistry | None = None,
|
||||
llm_factory: Callable[[str], Any] | None = None,
|
||||
max_iterations: int = 25,
|
||||
) -> None:
|
||||
self._session_service = session_service
|
||||
self._provider_registry = provider_registry
|
||||
self._tool_registry = tool_registry or ToolRegistry()
|
||||
self._llm_factory = llm_factory
|
||||
self._max_iterations = max_iterations
|
||||
self._logger = logger.bind(component="session_workflow")
|
||||
# Populated by tell_stream() so callers can read real usage metrics
|
||||
# after the stream is exhausted (addresses M3, M4).
|
||||
self.last_stream_usage: dict[str, Any] = {}
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Public API
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def tell(
|
||||
self,
|
||||
session_id: str,
|
||||
prompt: str,
|
||||
actor_override: str | None = None,
|
||||
) -> TellResult:
|
||||
"""Send a user prompt to the session's orchestrator actor (non-streaming).
|
||||
|
||||
Appends the user message, resolves the actor, invokes the LLM,
|
||||
persists the assistant response, and records token usage.
|
||||
|
||||
Raises:
|
||||
SessionNotFoundError: If the session does not exist.
|
||||
SessionActorNotConfiguredError: If no actor is configured.
|
||||
ValueError: If prompt is empty.
|
||||
"""
|
||||
if not prompt or not prompt.strip():
|
||||
raise ValueError("prompt must not be empty")
|
||||
|
||||
session = self._session_service.get(session_id)
|
||||
actor_name = actor_override or session.actor_name
|
||||
if not actor_name:
|
||||
raise SessionActorNotConfiguredError(
|
||||
"Session has no actor configured. "
|
||||
"Create a session with --actor <actor> or pass --actor."
|
||||
)
|
||||
|
||||
# Load history before appending the user message so the slice in
|
||||
# _invoke_actor reads a stable snapshot (addresses m11).
|
||||
# TODO(#5784-M9): If _invoke_actor() raises below, the session retains
|
||||
# an orphaned user message with no paired assistant response. Implement
|
||||
# compensation logic that removes the orphaned message on failure, or
|
||||
# persist the user message transactionally with the assistant response.
|
||||
# Sanitize the prompt before passing to the LLM (addresses M7).
|
||||
sanitized_prompt = _sanitizer.sanitize_user_input(prompt).sanitized
|
||||
|
||||
self._session_service.append_message(
|
||||
session_id=session_id,
|
||||
role=MessageRole.USER,
|
||||
content=prompt,
|
||||
)
|
||||
|
||||
start = time.monotonic()
|
||||
run_result, real_input_tokens, real_output_tokens = self._invoke_actor(
|
||||
session_id=session_id,
|
||||
prompt=sanitized_prompt,
|
||||
actor_name=actor_name,
|
||||
)
|
||||
duration_ms = (time.monotonic() - start) * 1000.0
|
||||
|
||||
assistant_content = run_result.content or ""
|
||||
|
||||
# Persist assistant response
|
||||
self._session_service.append_message(
|
||||
session_id=session_id,
|
||||
role=MessageRole.ASSISTANT,
|
||||
content=assistant_content if assistant_content else "(no response)",
|
||||
)
|
||||
|
||||
# Use real token counts from LLM response metadata when available,
|
||||
# falling back to the heuristic estimate otherwise (M1).
|
||||
if real_input_tokens > 0 or real_output_tokens > 0:
|
||||
input_tokens = real_input_tokens
|
||||
output_tokens = real_output_tokens
|
||||
else:
|
||||
input_tokens, output_tokens = estimate_tokens(
|
||||
sanitized_prompt, assistant_content
|
||||
)
|
||||
cost = estimate_cost(input_tokens, output_tokens)
|
||||
self._session_service.update_token_usage(
|
||||
session_id=session_id,
|
||||
input_tokens=input_tokens,
|
||||
output_tokens=output_tokens,
|
||||
cost=cost,
|
||||
)
|
||||
|
||||
self._logger.info(
|
||||
"session.tell.complete",
|
||||
session_id=session_id,
|
||||
actor=actor_name,
|
||||
input_tokens=input_tokens,
|
||||
output_tokens=output_tokens,
|
||||
tool_calls=len(run_result.tool_call_history),
|
||||
duration_ms=round(duration_ms, 1),
|
||||
)
|
||||
|
||||
return TellResult(
|
||||
session_id=session_id,
|
||||
user_message=prompt,
|
||||
assistant_message=assistant_content,
|
||||
input_tokens=input_tokens,
|
||||
output_tokens=output_tokens,
|
||||
cost=cost,
|
||||
duration_ms=round(duration_ms, 1),
|
||||
tool_calls_count=len(run_result.tool_call_history),
|
||||
)
|
||||
|
||||
def tell_stream(
|
||||
self,
|
||||
session_id: str,
|
||||
prompt: str,
|
||||
actor_override: str | None = None,
|
||||
) -> Iterator[str]:
|
||||
"""Stream the assistant's response token-by-token.
|
||||
|
||||
Loads session history before appending the user message (addresses C1).
|
||||
After the stream is exhausted, the full response is persisted, token
|
||||
usage is recorded, and ``last_stream_usage`` is populated (M3, M4).
|
||||
|
||||
Yields:
|
||||
Successive text tokens from the LLM.
|
||||
|
||||
Raises:
|
||||
SessionNotFoundError: If the session does not exist.
|
||||
SessionActorNotConfiguredError: If no actor is configured.
|
||||
ValueError: If prompt is empty.
|
||||
"""
|
||||
if not prompt or not prompt.strip():
|
||||
raise ValueError("prompt must not be empty")
|
||||
|
||||
# Reset stale usage data before any operation that could raise (M4).
|
||||
self.last_stream_usage = {}
|
||||
|
||||
session = self._session_service.get(session_id)
|
||||
actor_name = actor_override or session.actor_name
|
||||
if not actor_name:
|
||||
raise SessionActorNotConfiguredError(
|
||||
"Session has no actor configured. "
|
||||
"Create a session with --actor <actor> or pass --actor."
|
||||
)
|
||||
|
||||
# Load history BEFORE appending the user message (addresses C1).
|
||||
# Build the LC message list from existing history + the new prompt
|
||||
# so the LLM does not receive a duplicate HumanMessage.
|
||||
existing_history = self._session_service.get_messages(session_id)
|
||||
|
||||
# Sanitize the prompt before passing to the LLM (addresses M7).
|
||||
sanitized_prompt = _sanitizer.sanitize_user_input(prompt).sanitized
|
||||
lc_messages = self._build_lc_messages_from_history(
|
||||
existing_history, sanitized_prompt
|
||||
)
|
||||
|
||||
# Append user message to the session (persisted separately from
|
||||
# the LC message list used for the LLM call).
|
||||
self._session_service.append_message(
|
||||
session_id=session_id,
|
||||
role=MessageRole.USER,
|
||||
content=prompt,
|
||||
)
|
||||
|
||||
llm = self._resolve_llm(actor_name)
|
||||
|
||||
start = time.monotonic()
|
||||
collected_chunks: list[str] = []
|
||||
|
||||
# try/finally guarantees persistence even when the consumer
|
||||
# breaks early. The stream is explicitly closed in the finally
|
||||
# block to release resources (M8).
|
||||
stream = llm.stream(lc_messages)
|
||||
try:
|
||||
for chunk in stream:
|
||||
token = extract_content(chunk)
|
||||
if token:
|
||||
collected_chunks.append(token)
|
||||
yield token
|
||||
finally:
|
||||
if hasattr(stream, "close"):
|
||||
stream.close()
|
||||
duration_ms = (time.monotonic() - start) * 1000.0
|
||||
full_content = "".join(collected_chunks)
|
||||
|
||||
# Persist and record usage
|
||||
self._session_service.append_message(
|
||||
session_id=session_id,
|
||||
role=MessageRole.ASSISTANT,
|
||||
content=full_content if full_content else "(no response)",
|
||||
)
|
||||
input_tokens, output_tokens = estimate_tokens(
|
||||
sanitized_prompt, full_content
|
||||
)
|
||||
cost = estimate_cost(input_tokens, output_tokens)
|
||||
self._session_service.update_token_usage(
|
||||
session_id=session_id,
|
||||
input_tokens=input_tokens,
|
||||
output_tokens=output_tokens,
|
||||
cost=cost,
|
||||
)
|
||||
|
||||
# Populate last_stream_usage so CLI can read real metrics (M3, M4)
|
||||
self.last_stream_usage = {
|
||||
"input_tokens": input_tokens,
|
||||
"output_tokens": output_tokens,
|
||||
"cost_usd": cost,
|
||||
"duration_ms": round(duration_ms, 1),
|
||||
"tool_calls": 0,
|
||||
}
|
||||
|
||||
self._logger.info(
|
||||
"session.tell_stream.complete",
|
||||
session_id=session_id,
|
||||
actor=actor_name,
|
||||
duration_ms=round(duration_ms, 1),
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Private helpers
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _resolve_llm(self, actor_name: str) -> Any:
|
||||
"""Create a LangChain LLM for the given actor name.
|
||||
|
||||
Delegates to ``llm_factory`` (test injection), ``ProviderRegistry``,
|
||||
or falls back to a stub when neither is available.
|
||||
"""
|
||||
# Test injection point — skips provider registry when llm_factory is set.
|
||||
# Must be checked before _provider_registry fallback so tests can inject
|
||||
# stubs even when the registry is None (the default).
|
||||
if self._llm_factory is not None:
|
||||
result = self._llm_factory(actor_name)
|
||||
if result is not None:
|
||||
return result
|
||||
# Factory returned None; fall through to provider registry
|
||||
|
||||
if self._provider_registry is None:
|
||||
return self._make_stub_llm()
|
||||
|
||||
from cleveragents.application.services.strategy_resolution import (
|
||||
_parse_actor_name,
|
||||
)
|
||||
|
||||
provider_type, model_id = _parse_actor_name(actor_name)
|
||||
return self._provider_registry.create_llm(
|
||||
provider_type=provider_type,
|
||||
model_id=model_id,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _make_stub_llm() -> Any:
|
||||
"""Return a deterministic stub LLM for environments without API keys."""
|
||||
if FakeListLLM is not None:
|
||||
return FakeListLLM(responses=["(no LLM configured)"] * 100)
|
||||
return _MinimalStubLLM()
|
||||
|
||||
def _build_lc_messages_from_history(
|
||||
self,
|
||||
history_messages: list[SessionMessage],
|
||||
prompt: str,
|
||||
) -> list[Any]:
|
||||
"""Build LangChain messages from pre-loaded history + new prompt.
|
||||
|
||||
Used by ``tell_stream()`` to avoid the duplicate-user-message bug (C1).
|
||||
The caller controls when history is sampled.
|
||||
"""
|
||||
lc_history = history_to_langchain_messages(history_messages)
|
||||
|
||||
messages: list[Any] = []
|
||||
if not any(isinstance(m, SystemMessage) for m in lc_history):
|
||||
messages.append(SystemMessage(content=_SESSION_SYSTEM_PROMPT))
|
||||
messages.extend(lc_history)
|
||||
messages.append(HumanMessage(content=prompt))
|
||||
return messages
|
||||
|
||||
def _invoke_actor(
|
||||
self,
|
||||
session_id: str,
|
||||
prompt: str,
|
||||
actor_name: str,
|
||||
) -> tuple[ToolCallRunResult, int, int]:
|
||||
"""Run the tool-calling loop against the resolved LLM.
|
||||
|
||||
Returns ``(run_result, input_tokens, output_tokens)`` with real token
|
||||
counts from LLM response metadata. Callers fall back to
|
||||
``estimate_tokens()`` when both counts are zero.
|
||||
"""
|
||||
llm = self._resolve_llm(actor_name)
|
||||
|
||||
# Load history — only SessionNotFoundError is handled;
|
||||
# infrastructure errors propagate (addresses M5).
|
||||
# TODO(#5784-m7): Session message history (assistant/tool roles) is fed to
|
||||
# the LLM without sanitization. Apply PromptSanitizer to historical
|
||||
# messages to prevent prompt injection from prior assistant responses.
|
||||
history_msgs = self._session_service.get_messages(session_id)
|
||||
|
||||
# Exclude the user message we just appended (last message)
|
||||
# so it is not duplicated when LangChainSessionCaller appends it
|
||||
lc_history = history_to_langchain_messages(history_msgs[:-1])
|
||||
|
||||
caller: LLMCaller = LangChainSessionCaller(
|
||||
llm=llm,
|
||||
history=lc_history,
|
||||
)
|
||||
|
||||
runner = ToolRunner(registry=self._tool_registry)
|
||||
runtime = ToolCallingRuntime(
|
||||
registry=self._tool_registry,
|
||||
runner=runner,
|
||||
llm_caller=caller,
|
||||
max_iterations=self._max_iterations,
|
||||
)
|
||||
|
||||
run_result = runtime.run_tool_loop(prompt=prompt)
|
||||
# Read real token usage accumulated by LangChainSessionCaller (M1).
|
||||
real_input, real_output = caller.get_token_usage()
|
||||
return run_result, real_input, real_output
|
||||
|
||||
|
||||
__all__ = [
|
||||
"LangChainSessionCaller",
|
||||
"SessionWorkflow",
|
||||
"TellResult",
|
||||
]
|
||||
@@ -16,7 +16,7 @@ from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import sys
|
||||
import re
|
||||
import threading
|
||||
from collections import OrderedDict
|
||||
from pathlib import Path
|
||||
@@ -28,17 +28,20 @@ from rich.panel import Panel
|
||||
from rich.table import Table
|
||||
|
||||
from cleveragents.a2a.models import A2aRequest
|
||||
from cleveragents.application.services.session_workflow import SessionWorkflow
|
||||
from cleveragents.cli.formatting import OutputFormat, format_output
|
||||
from cleveragents.core.exceptions import DatabaseError
|
||||
from cleveragents.domain.models.core.session import (
|
||||
MessageRole,
|
||||
Session,
|
||||
SessionActorNotConfiguredError,
|
||||
SessionExportError,
|
||||
SessionImportError,
|
||||
SessionMessage,
|
||||
SessionNotFoundError,
|
||||
SessionService,
|
||||
)
|
||||
from cleveragents.providers.registry import ProviderRegistry
|
||||
|
||||
# Create sub-app for session commands
|
||||
app = typer.Typer(help="Manage interactive sessions.")
|
||||
@@ -58,7 +61,7 @@ _MCP_LOGGER_NAME = "cleveragents.mcp"
|
||||
_mcp_logger_lock = threading.Lock()
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Module-level service accessor (patchable in tests)
|
||||
# Module-level service and workflow accessors (patchable in tests)
|
||||
# ---------------------------------------------------------------------------
|
||||
_service: SessionService | None = None
|
||||
|
||||
@@ -87,11 +90,43 @@ def _reset_session_service() -> None:
|
||||
_service = None
|
||||
|
||||
|
||||
def _build_session_workflow() -> SessionWorkflow:
|
||||
"""Build a :class:`SessionWorkflow` wired to the CLI session service.
|
||||
|
||||
Returns a fully configured workflow using the same ``SessionService``
|
||||
that the CLI commands use. Tests can replace this function or its
|
||||
dependencies by patching ``_service`` (for the service layer) and
|
||||
``_get_provider_registry`` (for the LLM layer).
|
||||
"""
|
||||
service = _get_session_service()
|
||||
provider_registry = _get_provider_registry()
|
||||
return SessionWorkflow(
|
||||
session_service=service,
|
||||
provider_registry=provider_registry,
|
||||
)
|
||||
|
||||
|
||||
def _get_provider_registry() -> ProviderRegistry | None:
|
||||
"""Attempt to load the provider registry; return ``None`` on failure.
|
||||
|
||||
Isolated in its own function so tests can patch it independently
|
||||
to avoid network / credential errors in CI.
|
||||
"""
|
||||
try:
|
||||
from cleveragents.providers.registry import get_provider_registry
|
||||
|
||||
return get_provider_registry()
|
||||
except (ImportError, RuntimeError, OSError) as exc:
|
||||
_log.warning("Provider registry unavailable: %s", exc)
|
||||
return None
|
||||
|
||||
|
||||
def _facade_dispatch(operation: str, params: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Route an operation through the A2A local facade.
|
||||
|
||||
Returns the response data dict on success. Raises on error
|
||||
so that CLI commands can render the appropriate error message.
|
||||
Returns the response data dict on success. Domain errors are mapped
|
||||
back to their original exception types so CLI error handlers work
|
||||
correctly. Unknown / infrastructure errors raise ``RuntimeError``.
|
||||
|
||||
stdout/stderr are redirected during facade construction to prevent
|
||||
structlog or Rich output from polluting CLI output captured by
|
||||
@@ -107,9 +142,25 @@ def _facade_dispatch(operation: str, params: dict[str, Any]) -> dict[str, Any]:
|
||||
from cleveragents.a2a.cli_bootstrap import get_facade
|
||||
|
||||
facade = get_facade()
|
||||
# Wire the CLI's session service and workflow into the facade
|
||||
# so that message/send and message/stream handlers use the same
|
||||
# SessionService and SessionWorkflow as the rest of the CLI.
|
||||
# Tests that patch _get_session_service() and
|
||||
# _build_session_workflow() therefore also control the facade.
|
||||
with contextlib.suppress(Exception):
|
||||
facade.register_service("session_service", _get_session_service())
|
||||
with contextlib.suppress(Exception):
|
||||
facade.register_service("session_workflow", _build_session_workflow())
|
||||
request = A2aRequest(method=operation, params=params)
|
||||
response = facade.dispatch(request)
|
||||
if response.error is not None:
|
||||
# Map A2A error codes back to domain exceptions so CLI error
|
||||
# handlers work correctly when routing through the facade.
|
||||
# Domain exceptions that handlers explicitly re-raise
|
||||
# (SessionNotFoundError, SessionActorNotConfiguredError,
|
||||
# DatabaseError) propagate directly through the facade
|
||||
# dispatch, so they never reach this point. Any error here
|
||||
# is an unknown/infrastructure failure.
|
||||
raise RuntimeError(response.error.message)
|
||||
return dict(response.result or {})
|
||||
|
||||
@@ -855,8 +906,9 @@ def tell(
|
||||
) -> None:
|
||||
"""Send a message to a session.
|
||||
|
||||
Appends a user message and generates an assistant response. For M3, the
|
||||
actor execution is stubbed — the assistant echoes an acknowledgement.
|
||||
Appends a user message and invokes the session's bound orchestrator actor
|
||||
(or ``--actor`` override) via the A2A ``message/send`` operation.
|
||||
The actor's real response is persisted and token usage is tracked.
|
||||
|
||||
Examples:
|
||||
agents session tell --session 01HXYZ... "Hello, world"
|
||||
@@ -865,49 +917,16 @@ def tell(
|
||||
agents session tell --session 01HXYZ... --format json "Hello"
|
||||
"""
|
||||
try:
|
||||
service = _get_session_service()
|
||||
|
||||
# Append user message
|
||||
service.append_message(
|
||||
_do_session_tell(
|
||||
session_id=session_id,
|
||||
role=MessageRole.USER,
|
||||
content=prompt,
|
||||
prompt=prompt,
|
||||
actor_override=actor,
|
||||
stream=stream,
|
||||
fmt=fmt,
|
||||
)
|
||||
|
||||
# Stub actor execution: generate simple assistant response
|
||||
assistant_content = (
|
||||
f"Acknowledged: {prompt[:100]}"
|
||||
if not actor
|
||||
else f"[{actor}] Acknowledged: {prompt[:100]}"
|
||||
)
|
||||
service.append_message(
|
||||
session_id=session_id,
|
||||
role=MessageRole.ASSISTANT,
|
||||
content=assistant_content,
|
||||
)
|
||||
|
||||
data = {
|
||||
"session_id": session_id,
|
||||
"user_message": prompt,
|
||||
"assistant_message": assistant_content,
|
||||
}
|
||||
|
||||
if fmt not in (OutputFormat.RICH.value, OutputFormat.COLOR.value):
|
||||
typer.echo(format_output(data, fmt))
|
||||
return
|
||||
|
||||
if stream:
|
||||
# Simulate streaming by printing character by character
|
||||
for char in assistant_content:
|
||||
sys.stdout.write(char)
|
||||
sys.stdout.flush()
|
||||
sys.stdout.write("\n")
|
||||
else:
|
||||
from rich.markup import escape
|
||||
|
||||
console.print(f"[dim]user:[/dim] {escape(prompt)}")
|
||||
console.print(f"[cyan]assistant:[/cyan] {escape(assistant_content)}")
|
||||
|
||||
except SessionActorNotConfiguredError as exc:
|
||||
console.print(f"[red]Error:[/red] {exc}")
|
||||
raise typer.Exit(1) from exc
|
||||
except SessionNotFoundError as exc:
|
||||
console.print(f"[red]Session not found:[/red] {session_id}")
|
||||
raise typer.Exit(1) from exc
|
||||
@@ -918,3 +937,95 @@ def tell(
|
||||
"Hint: run 'agents init' to initialise the database."
|
||||
)
|
||||
raise typer.Exit(1) from exc
|
||||
except Exception as exc:
|
||||
if not isinstance(exc, typer.Exit):
|
||||
_log.exception("session tell failed with unexpected error")
|
||||
console.print(
|
||||
"[red]Error:[/red] An unexpected error occurred. "
|
||||
"Check the logs for details."
|
||||
)
|
||||
raise typer.Exit(1) from exc
|
||||
raise
|
||||
|
||||
|
||||
def _do_session_tell(
|
||||
session_id: str,
|
||||
prompt: str,
|
||||
actor_override: str | None,
|
||||
stream: bool,
|
||||
fmt: OutputFormat,
|
||||
) -> None:
|
||||
"""Core implementation of ``session tell``, extracted for testability.
|
||||
|
||||
Both streaming and non-streaming paths route through
|
||||
:class:`~cleveragents.a2a.facade.A2aLocalFacade` per the spec's
|
||||
A2A protocol mapping (message/send, message/stream; addresses C3, C5).
|
||||
"""
|
||||
# Validate --actor format before passing to the workflow (m8).
|
||||
if actor_override is not None and not re.match(
|
||||
r"^[a-z0-9][a-z0-9_-]*/[a-z0-9][a-z0-9_-]*$",
|
||||
actor_override,
|
||||
):
|
||||
console.print(
|
||||
f"[red]Error:[/red] Invalid actor name {actor_override!r}. "
|
||||
"Expected format: 'namespace/name' "
|
||||
"(e.g. 'openai/gpt-4')."
|
||||
)
|
||||
raise typer.Exit(1)
|
||||
|
||||
# Route through A2aLocalFacade per the spec's A2A protocol mapping.
|
||||
# message/send → SessionWorkflow.tell() (non-streaming)
|
||||
# message/stream → falls back to non-streaming with streamed=false
|
||||
# (true SSE-based streaming is deferred; acceptable per the spec).
|
||||
operation = "message/stream" if stream else "message/send"
|
||||
result_dict = _facade_dispatch(
|
||||
operation,
|
||||
{
|
||||
"session_id": session_id,
|
||||
"message": prompt,
|
||||
"actor": actor_override,
|
||||
},
|
||||
)
|
||||
|
||||
assistant_content = result_dict.get("assistant_message", "")
|
||||
usage = result_dict.get("usage", {})
|
||||
|
||||
if fmt not in (OutputFormat.RICH.value, OutputFormat.COLOR.value):
|
||||
data: dict[str, object] = {
|
||||
"session_id": session_id,
|
||||
"user_message": prompt,
|
||||
"assistant_message": assistant_content,
|
||||
"usage": usage,
|
||||
}
|
||||
typer.echo(format_output(data, fmt))
|
||||
return
|
||||
|
||||
from rich.markup import escape
|
||||
|
||||
console.print(f"[dim]user:[/dim] {escape(prompt)}")
|
||||
console.print(f"[cyan]assistant:[/cyan] {escape(assistant_content)}")
|
||||
_print_usage_panel(
|
||||
input_tokens=int(usage.get("input_tokens", 0)),
|
||||
output_tokens=int(usage.get("output_tokens", 0)),
|
||||
cost=float(usage.get("cost_usd", 0.0)),
|
||||
duration_ms=float(usage.get("duration_ms", 0.0)),
|
||||
tool_calls=int(usage.get("tool_calls", 0)),
|
||||
)
|
||||
|
||||
|
||||
def _print_usage_panel(
|
||||
input_tokens: int,
|
||||
output_tokens: int,
|
||||
cost: float,
|
||||
duration_ms: float,
|
||||
tool_calls: int,
|
||||
) -> None:
|
||||
"""Render a Rich Usage panel summarising token and cost metrics."""
|
||||
lines = [
|
||||
f"[bold]Input tokens:[/bold] {input_tokens}",
|
||||
f"[bold]Output tokens:[/bold] {output_tokens}",
|
||||
f"[bold]Est. cost:[/bold] ${cost:.6f}",
|
||||
f"[bold]Duration:[/bold] {duration_ms / 1000:.1f}s",
|
||||
f"[bold]Tool calls:[/bold] {tool_calls}",
|
||||
]
|
||||
console.print(Panel("\n".join(lines), title="Usage", expand=False))
|
||||
|
||||
@@ -280,6 +280,7 @@ from cleveragents.domain.models.core.session import (
|
||||
LinkedPlan,
|
||||
MessageRole,
|
||||
Session,
|
||||
SessionActorNotConfiguredError,
|
||||
SessionExportError,
|
||||
SessionImportError,
|
||||
SessionMessage,
|
||||
@@ -506,6 +507,7 @@ __all__ = [
|
||||
"ServiceRetryPolicy",
|
||||
"ServiceRetryPolicyRegistry",
|
||||
"Session",
|
||||
"SessionActorNotConfiguredError",
|
||||
"SessionCostBudget",
|
||||
"SessionExportError",
|
||||
"SessionImportError",
|
||||
|
||||
@@ -595,6 +595,14 @@ class SessionImportError(SessionServiceError):
|
||||
"""Raised when session import fails (schema mismatch, corrupt data)."""
|
||||
|
||||
|
||||
class SessionActorNotConfiguredError(SessionServiceError):
|
||||
"""Raised when session tell is invoked with no actor configured.
|
||||
|
||||
Occurs when neither the session has a bound actor nor an ``--actor``
|
||||
override was supplied to the ``agents session tell`` command.
|
||||
"""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Service Contract (Abstract)
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -716,6 +724,20 @@ class SessionService(ABC):
|
||||
SessionImportError: If import fails (bad schema, corrupt data).
|
||||
"""
|
||||
|
||||
@abstractmethod
|
||||
def get_messages(self, session_id: str) -> list[SessionMessage]:
|
||||
"""Retrieve all messages for a session in order.
|
||||
|
||||
Args:
|
||||
session_id: The ULID of the session.
|
||||
|
||||
Returns:
|
||||
Ordered list of SessionMessage objects (by sequence number).
|
||||
|
||||
Raises:
|
||||
SessionNotFoundError: If no session with the given ID exists.
|
||||
"""
|
||||
|
||||
@abstractmethod
|
||||
def update_token_usage(
|
||||
self,
|
||||
|
||||
@@ -140,6 +140,7 @@ if TYPE_CHECKING:
|
||||
CorrectionAttemptRecord,
|
||||
)
|
||||
from cleveragents.domain.models.core.decision import Decision
|
||||
from cleveragents.domain.models.core.session import SessionMessage
|
||||
|
||||
from cleveragents.domain.models.core.correction import (
|
||||
CORRECTION_ATTEMPT_TERMINAL_STATES,
|
||||
@@ -4374,7 +4375,7 @@ class SessionMessageRepository:
|
||||
session_id: str,
|
||||
limit: int | None = None,
|
||||
offset: int = 0,
|
||||
) -> list[Any]:
|
||||
) -> list[SessionMessage]:
|
||||
"""Retrieve messages for a session with optional pagination.
|
||||
|
||||
Args:
|
||||
|
||||
@@ -77,6 +77,12 @@ class LLMToolCall(BaseModel):
|
||||
arguments: dict[str, Any] = Field(
|
||||
default_factory=dict, description="Arguments for the tool"
|
||||
)
|
||||
call_id: str = Field(
|
||||
"",
|
||||
min_length=0,
|
||||
max_length=256,
|
||||
description="Provider-assigned unique call ID for this tool call",
|
||||
)
|
||||
|
||||
model_config = ConfigDict(
|
||||
str_strip_whitespace=True,
|
||||
@@ -401,6 +407,7 @@ class ToolCallingRuntime:
|
||||
# Build result for LLM feedback
|
||||
result_dict: dict[str, Any] = {
|
||||
"tool_name": tool_call.name,
|
||||
"call_id": tool_call.call_id,
|
||||
"success": success,
|
||||
"output": output,
|
||||
}
|
||||
|
||||
@@ -446,6 +446,7 @@ dependencies = [
|
||||
{ name = "pydantic" },
|
||||
{ name = "pydantic-settings" },
|
||||
{ name = "python-ulid" },
|
||||
{ name = "pyyaml" },
|
||||
{ name = "restrictedpython" },
|
||||
{ name = "rx" },
|
||||
{ name = "structlog" },
|
||||
@@ -528,6 +529,7 @@ requires-dist = [
|
||||
{ name = "pytest", marker = "extra == 'dev'", specifier = ">=8.0.0" },
|
||||
{ name = "pytest-asyncio", marker = "extra == 'dev'", specifier = ">=0.23.0" },
|
||||
{ name = "pytest-cov", marker = "extra == 'dev'", specifier = ">=4.1.0" },
|
||||
{ name = "pyyaml", specifier = ">=6.0.3" },
|
||||
{ name = "python-ulid", specifier = ">=2.7.0" },
|
||||
{ name = "radon", marker = "extra == 'dev'", specifier = ">=6.0.1" },
|
||||
{ name = "restrictedpython", specifier = ">=7.0" },
|
||||
|
||||
Reference in New Issue
Block a user