87a7ce35d7
CI / benchmark-regression (push) Has been skipped
CI / helm (push) Successful in 38s
CI / lint (push) Successful in 1m14s
CI / build (push) Successful in 1m10s
CI / push-validation (push) Successful in 49s
CI / quality (push) Successful in 1m35s
CI / typecheck (push) Successful in 1m39s
CI / security (push) Successful in 1m45s
CI / integration_tests (push) Successful in 3m44s
CI / e2e_tests (push) Successful in 4m29s
CI / unit_tests (push) Successful in 5m10s
CI / docker (push) Successful in 1m55s
CI / coverage (push) Successful in 11m8s
CI / status-check (push) Successful in 3s
CI / benchmark-publish (push) Successful in 1h20m21s
CI / benchmark-publish (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Failing after 1m35s
CI / docker (pull_request) Successful in 1m26s
CI / unit_tests (pull_request) Successful in 6m40s
CI / push-validation (pull_request) Successful in 1m24s
CI / quality (pull_request) Successful in 4m4s
CI / integration_tests (pull_request) Successful in 5m44s
CI / e2e_tests (pull_request) Failing after 6m5s
CI / helm (pull_request) Successful in 1m10s
CI / build (pull_request) Successful in 2m39s
CI / lint (pull_request) Successful in 2m58s
CI / typecheck (pull_request) Successful in 3m56s
CI / security (pull_request) Successful in 3m56s
CI / coverage (pull_request) Successful in 10m52s
CI / status-check (pull_request) Failing after 3s
Replace the M3 stub in `agents session tell` with real orchestrator actor
invocation via `SessionWorkflow.tell()`. The stub echoed a canned
`Acknowledged: ...` response without calling any LLM or actor; this
commit wires up the full pipeline:
Architecture:
- New `SessionWorkflow` (Application layer) orchestrates LLM invocation
for session tell. Accepts a `ProviderRegistry` and `ToolRegistry`;
falls back to `FakeListLLM` when no provider is configured.
- `LangChainSessionCaller` implements the `LLMCaller` protocol, building
history-aware LangChain message lists from the session conversation and
invoking the LLM via `ToolCallingRuntime.run_tool_loop()`.
- `TellResult` (Pydantic BaseModel) carries the assistant response plus
token usage (input_tokens, output_tokens, cost_usd, duration_ms).
Domain / service layer:
- `SessionActorNotConfiguredError` added to the session domain model;
raised when tell is invoked with no actor on the session and no
`--actor` override. Clear message; CLI exits with code 1.
- `SessionService.get_messages()` abstract method added (+ implementation
in `PersistentSessionService`) to load ordered message history.
A2A facade:
- `A2aLocalFacade` gains `message/send` and `message/stream` standard
A2A operation handlers that route to `SessionWorkflow.tell()`.
Total supported operations count: 42 → 44.
CLI:
- `session tell` command delegates to `_build_session_workflow()`
(patchable factory) instead of directly calling `SessionService`.
- Non-streaming: `SessionWorkflow.tell()` returns `TellResult`; output
includes a Usage panel (Rich/Plain) or a `usage` object (JSON/YAML).
- Streaming: `SessionWorkflow.tell_stream()` yields tokens; CLI prints
them via `console.print` (not raw `sys.stdout.write`).
- Token usage recorded via `SessionService.update_token_usage()` in
both paths.
Tests:
- New Behave feature `session_tell_llm.feature` (4 scenarios): real LLM
response persisted; streaming yields tokens; no-actor exits code 1;
`--actor` override resolves correctly.
- New Robot suite `session_tell_llm.robot` (4 tests): end-to-end with
stub LLM injected via monkey-patching `_resolve_llm`.
- Updated all existing `session tell` test steps to patch
`_build_session_workflow` returning a mock `TellResult` so tests
remain isolated from the LLM layer.
- Updated operation-count assertions (42 → 44) in
`a2a_cli_facade_integration`, `consolidated_misc`,
`m6_autonomy_acceptance` feature files and steps.
Cycle 7 (Review ID 8088) fixes:
- Blocker 4: Split `session_workflow.py` (was 801 lines) into two files:
`session_workflow.py` (467 lines) and `session_caller.py` (318 lines).
Extracted: `LangChainSessionCaller`, `extract_content`,
`extract_token_usage`, `estimate_cost`, `estimate_tokens`,
`history_to_langchain_messages`, and stub classes.
- Blocker 5: Route CLI streaming path through
`_facade_dispatch("message/stream", ...)` instead of directly
calling `workflow.tell_stream()`. The facade's
`_handle_message_stream` falls back to non-streaming with
`streamed: false` (acceptable for this milestone per the spec).
- Updated streaming test assertion to validate full response presence
(no longer checks token-by-token word positions, since the facade
fallback returns a complete message).
ISSUES CLOSED: #5784
405 lines
13 KiB
Python
405 lines
13 KiB
Python
"""Helper script for session_cli.robot smoke tests.
|
|
|
|
Each subcommand is a self-contained check that prints a sentinel on success.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
import sys
|
|
import tempfile
|
|
from datetime import datetime
|
|
from pathlib import Path
|
|
from unittest.mock import MagicMock, patch
|
|
|
|
# Ensure local source tree is importable
|
|
_SRC = str(Path(__file__).resolve().parents[1] / "src")
|
|
if _SRC not in sys.path:
|
|
sys.path.insert(0, _SRC)
|
|
|
|
from typer.testing import CliRunner # noqa: E402
|
|
from ulid import ULID # noqa: E402
|
|
|
|
from cleveragents.application.services.session_workflow import TellResult # noqa: E402
|
|
from cleveragents.cli.commands import session as session_mod # noqa: E402
|
|
from cleveragents.cli.commands.session import app as session_app # noqa: E402
|
|
from cleveragents.domain.models.core.session import ( # noqa: E402
|
|
MessageRole,
|
|
Session,
|
|
SessionMessage,
|
|
SessionNotFoundError,
|
|
SessionTokenUsage,
|
|
)
|
|
|
|
runner = CliRunner()
|
|
|
|
|
|
def _mock_session(
|
|
session_id: str | None = None,
|
|
actor_name: str | None = None,
|
|
messages: list[SessionMessage] | None = None,
|
|
) -> Session:
|
|
"""Create a test Session instance."""
|
|
return Session(
|
|
session_id=session_id or str(ULID()),
|
|
actor_name=actor_name,
|
|
namespace="local",
|
|
messages=messages or [],
|
|
token_usage=SessionTokenUsage(
|
|
input_tokens=100,
|
|
output_tokens=50,
|
|
estimated_cost=0.005,
|
|
),
|
|
created_at=datetime.now(),
|
|
updated_at=datetime.now(),
|
|
)
|
|
|
|
|
|
def _mock_message(
|
|
role: MessageRole = MessageRole.USER,
|
|
content: str = "Hello",
|
|
sequence: int = 0,
|
|
) -> SessionMessage:
|
|
return SessionMessage(
|
|
message_id=str(ULID()),
|
|
role=role,
|
|
content=content,
|
|
sequence=sequence,
|
|
timestamp=datetime.now(),
|
|
)
|
|
|
|
|
|
def _setup_service() -> MagicMock:
|
|
"""Create and install a mock service."""
|
|
svc = MagicMock()
|
|
session_mod._service = svc
|
|
return svc
|
|
|
|
|
|
def _teardown() -> None:
|
|
session_mod._service = None
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Subcommands
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def create_default() -> None:
|
|
svc = _setup_service()
|
|
svc.create.return_value = _mock_session()
|
|
try:
|
|
result = runner.invoke(session_app, ["create"])
|
|
assert result.exit_code == 0, f"exit={result.exit_code}: {result.output}"
|
|
print("session-cli-create-default-ok")
|
|
finally:
|
|
_teardown()
|
|
|
|
|
|
def create_actor() -> None:
|
|
svc = _setup_service()
|
|
svc.create.return_value = _mock_session(actor_name="openai/gpt-4")
|
|
try:
|
|
result = runner.invoke(session_app, ["create", "--actor", "openai/gpt-4"])
|
|
assert result.exit_code == 0, f"exit={result.exit_code}: {result.output}"
|
|
assert "openai/gpt-4" in result.output
|
|
print("session-cli-create-actor-ok")
|
|
finally:
|
|
_teardown()
|
|
|
|
|
|
def list_empty() -> None:
|
|
svc = _setup_service()
|
|
svc.list.return_value = []
|
|
try:
|
|
result = runner.invoke(session_app, ["list"])
|
|
assert result.exit_code == 0
|
|
assert "No sessions found" in result.output
|
|
print("session-cli-list-empty-ok")
|
|
finally:
|
|
_teardown()
|
|
|
|
|
|
def list_populated() -> None:
|
|
svc = _setup_service()
|
|
svc.list.return_value = [
|
|
_mock_session(actor_name="openai/gpt-4"),
|
|
_mock_session(),
|
|
]
|
|
try:
|
|
result = runner.invoke(session_app, ["list"])
|
|
assert result.exit_code == 0
|
|
assert "Sessions" in result.output
|
|
print("session-cli-list-populated-ok")
|
|
finally:
|
|
_teardown()
|
|
|
|
|
|
def show_valid() -> None:
|
|
sid = str(ULID())
|
|
svc = _setup_service()
|
|
session = _mock_session(
|
|
session_id=sid,
|
|
messages=[
|
|
_mock_message(MessageRole.USER, "Hello", 0),
|
|
_mock_message(MessageRole.ASSISTANT, "Hi there", 1),
|
|
],
|
|
)
|
|
svc.get.return_value = session
|
|
try:
|
|
result = runner.invoke(session_app, ["show", sid])
|
|
assert result.exit_code == 0, f"exit={result.exit_code}: {result.output}"
|
|
assert "Session Summary" in result.output
|
|
print("session-cli-show-valid-ok")
|
|
finally:
|
|
_teardown()
|
|
|
|
|
|
def show_not_found() -> None:
|
|
svc = _setup_service()
|
|
svc.get.side_effect = SessionNotFoundError("not found")
|
|
try:
|
|
result = runner.invoke(session_app, ["show", "INVALID"])
|
|
assert result.exit_code != 0
|
|
assert "Session not found" in result.output
|
|
print("session-cli-show-not-found-ok")
|
|
finally:
|
|
_teardown()
|
|
|
|
|
|
def delete_yes() -> None:
|
|
sid = str(ULID())
|
|
svc = _setup_service()
|
|
svc.get.return_value = _mock_session(session_id=sid)
|
|
svc.delete.return_value = None
|
|
try:
|
|
result = runner.invoke(session_app, ["delete", sid, "--yes"])
|
|
assert result.exit_code == 0, f"exit={result.exit_code}: {result.output}"
|
|
assert "deleted" in result.output
|
|
print("session-cli-delete-yes-ok")
|
|
finally:
|
|
_teardown()
|
|
|
|
|
|
def export_import_roundtrip() -> None:
|
|
"""Test export to file then import back, verifying Rich panels are rendered."""
|
|
sid = str(ULID())
|
|
svc = _setup_service()
|
|
|
|
session = _mock_session(session_id=sid, actor_name="openai/gpt-4")
|
|
svc.get.return_value = session
|
|
export_data = session.as_export_dict()
|
|
svc.export_session.return_value = export_data
|
|
|
|
fd, path = tempfile.mkstemp(suffix=".json")
|
|
os.close(fd)
|
|
os.unlink(path)
|
|
|
|
try:
|
|
# Export — verify Rich panels are rendered
|
|
result = runner.invoke(session_app, ["export", sid, "--output", path])
|
|
assert result.exit_code == 0, f"export exit={result.exit_code}: {result.output}"
|
|
assert os.path.exists(path), "Export file was not created"
|
|
assert "Session Export" in result.output, (
|
|
f"Missing 'Session Export' panel in output:\n{result.output}"
|
|
)
|
|
assert "Contents" in result.output, (
|
|
f"Missing 'Contents' panel in output:\n{result.output}"
|
|
)
|
|
assert "Integrity" in result.output, (
|
|
f"Missing 'Integrity' panel in output:\n{result.output}"
|
|
)
|
|
assert "Export completed" in result.output, (
|
|
f"Missing 'Export completed' in output:\n{result.output}"
|
|
)
|
|
|
|
# Set up import mock
|
|
imported = _mock_session(actor_name="openai/gpt-4")
|
|
svc.import_session.return_value = imported
|
|
|
|
# Import
|
|
result = runner.invoke(session_app, ["import", "--input", path])
|
|
assert result.exit_code == 0, f"import exit={result.exit_code}: {result.output}"
|
|
assert "Session Import" in result.output
|
|
assert "Validation" in result.output
|
|
assert "Merge" in result.output
|
|
assert "Import completed" in result.output
|
|
|
|
print("session-cli-export-import-roundtrip-ok")
|
|
finally:
|
|
if os.path.exists(path):
|
|
os.unlink(path)
|
|
_teardown()
|
|
|
|
|
|
def import_rich_panels() -> None:
|
|
"""Test that session import renders Session Import, Validation, and Merge panels."""
|
|
import json as _json
|
|
|
|
svc = _setup_service()
|
|
|
|
session = _mock_session(actor_name="openai/gpt-4")
|
|
imported = _mock_session(actor_name="openai/gpt-4")
|
|
svc.import_session.return_value = imported
|
|
|
|
fd, path = tempfile.mkstemp(suffix=".json")
|
|
with os.fdopen(fd, "w") as fh:
|
|
_json.dump(session.as_export_dict(), fh, default=str)
|
|
|
|
try:
|
|
result = runner.invoke(session_app, ["import", "--input", path])
|
|
assert result.exit_code == 0, f"exit={result.exit_code}: {result.output}"
|
|
assert "Session Import" in result.output, (
|
|
f"Missing 'Session Import' panel: {result.output}"
|
|
)
|
|
assert "Validation" in result.output, (
|
|
f"Missing 'Validation' panel: {result.output}"
|
|
)
|
|
assert "Merge" in result.output, f"Missing 'Merge' panel: {result.output}"
|
|
assert "Import completed" in result.output, (
|
|
f"Missing 'Import completed': {result.output}"
|
|
)
|
|
assert "Input:" in result.output, f"Missing 'Input:' field: {result.output}"
|
|
assert "Session ID:" in result.output, (
|
|
f"Missing 'Session ID:' field: {result.output}"
|
|
)
|
|
assert "Checksum:" in result.output, (
|
|
f"Missing 'Checksum:' field: {result.output}"
|
|
)
|
|
assert "Actor Ref:" in result.output, (
|
|
f"Missing 'Actor Ref:' field: {result.output}"
|
|
)
|
|
assert "Existing:" in result.output, (
|
|
f"Missing 'Existing:' field: {result.output}"
|
|
)
|
|
assert "Strategy:" in result.output, (
|
|
f"Missing 'Strategy:' field: {result.output}"
|
|
)
|
|
print("session-cli-import-rich-panels-ok")
|
|
finally:
|
|
if os.path.exists(path):
|
|
os.unlink(path)
|
|
_teardown()
|
|
|
|
|
|
def tell_message() -> None:
|
|
sid = str(ULID())
|
|
svc = _setup_service()
|
|
# Session must have actor_name so tell can proceed without --actor flag.
|
|
svc.get.return_value = _mock_session(session_id=sid, actor_name="openai/gpt-4")
|
|
svc.get_messages.return_value = []
|
|
# Build a mock workflow that returns a canned TellResult.
|
|
mock_wf = MagicMock()
|
|
mock_wf.tell.return_value = TellResult(
|
|
session_id=sid,
|
|
user_message="Hello",
|
|
assistant_message="Acknowledged: Hello",
|
|
input_tokens=5,
|
|
output_tokens=5,
|
|
cost=0.0,
|
|
duration_ms=0.0,
|
|
tool_calls_count=0,
|
|
)
|
|
try:
|
|
with patch(
|
|
"cleveragents.cli.commands.session._build_session_workflow",
|
|
return_value=mock_wf,
|
|
):
|
|
result = runner.invoke(session_app, ["tell", "--session", sid, "Hello"])
|
|
assert result.exit_code == 0, f"exit={result.exit_code}: {result.output}"
|
|
assert "Acknowledged" in result.output
|
|
print("session-cli-tell-message-ok")
|
|
finally:
|
|
_teardown()
|
|
|
|
|
|
def export_rich_panels() -> None:
|
|
"""Test that export renders all three spec-required Rich panels."""
|
|
sid = str(ULID())
|
|
svc = _setup_service()
|
|
|
|
session = _mock_session(session_id=sid, actor_name="openai/gpt-4")
|
|
svc.get.return_value = session
|
|
export_data = session.as_export_dict()
|
|
svc.export_session.return_value = export_data
|
|
|
|
fd, path = tempfile.mkstemp(suffix=".json")
|
|
os.close(fd)
|
|
os.unlink(path)
|
|
|
|
try:
|
|
result = runner.invoke(session_app, ["export", sid, "--output", path])
|
|
assert result.exit_code == 0, f"exit={result.exit_code}: {result.output}"
|
|
assert "Session Export" in result.output, (
|
|
f"Missing 'Session Export' panel:\n{result.output}"
|
|
)
|
|
assert "Contents" in result.output, (
|
|
f"Missing 'Contents' panel:\n{result.output}"
|
|
)
|
|
assert "Integrity" in result.output, (
|
|
f"Missing 'Integrity' panel:\n{result.output}"
|
|
)
|
|
assert "Export completed" in result.output, (
|
|
f"Missing 'Export completed':\n{result.output}"
|
|
)
|
|
assert sid in result.output, f"Session ID missing from output:\n{result.output}"
|
|
print("session-cli-export-rich-panels-ok")
|
|
finally:
|
|
if os.path.exists(path):
|
|
os.unlink(path)
|
|
_teardown()
|
|
|
|
|
|
def export_stdout_rich_panels() -> None:
|
|
"""Test that stdout export also renders Rich panels."""
|
|
sid = str(ULID())
|
|
svc = _setup_service()
|
|
|
|
session = _mock_session(session_id=sid, actor_name="openai/gpt-4")
|
|
svc.get.return_value = session
|
|
export_data = session.as_export_dict()
|
|
svc.export_session.return_value = export_data
|
|
|
|
try:
|
|
result = runner.invoke(session_app, ["export", sid])
|
|
assert result.exit_code == 0, f"exit={result.exit_code}: {result.output}"
|
|
assert "Session Export" in result.output, (
|
|
f"Missing 'Session Export' panel:\n{result.output}"
|
|
)
|
|
assert "(stdout)" in result.output, (
|
|
f"Missing '(stdout)' indicator:\n{result.output}"
|
|
)
|
|
assert "Export completed" in result.output, (
|
|
f"Missing 'Export completed':\n{result.output}"
|
|
)
|
|
print("session-cli-export-stdout-rich-panels-ok")
|
|
finally:
|
|
_teardown()
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Dispatcher
|
|
# ---------------------------------------------------------------------------
|
|
|
|
_COMMANDS: dict[str, object] = {
|
|
"create-default": create_default,
|
|
"create-actor": create_actor,
|
|
"list-empty": list_empty,
|
|
"list-populated": list_populated,
|
|
"show-valid": show_valid,
|
|
"show-not-found": show_not_found,
|
|
"delete-yes": delete_yes,
|
|
"export-import-roundtrip": export_import_roundtrip,
|
|
"import-rich-panels": import_rich_panels,
|
|
"export-rich-panels": export_rich_panels,
|
|
"export-stdout-rich-panels": export_stdout_rich_panels,
|
|
"tell-message": tell_message,
|
|
}
|
|
|
|
if __name__ == "__main__":
|
|
if len(sys.argv) < 2 or sys.argv[1] not in _COMMANDS:
|
|
print(f"Usage: {sys.argv[0]} <{'|'.join(_COMMANDS)}>", file=sys.stderr)
|
|
sys.exit(1)
|
|
cmd = _COMMANDS[sys.argv[1]]
|
|
cmd() # type: ignore[operator]
|