Robot Framework integration test suite for Specification Workflow Example 5:
Database Schema Migration with Safety Nets. Exercises the review automation
profile with custom resource types, custom skills with database tools,
phased child plan execution, checkpointing, and rollback.
- 7 test cases covering: custom resource type registration, review profile
behavior, custom skill with 3 database tools, action creation with
review profile, checkpoint/rollback via managers, 5-phase sequential
SubplanService.spawn with fail-fast, and plan lifecycle through
strategize-to-execute
- Uses mocked LLM providers (CLEVERAGENTS_TESTING_USE_MOCK_AI=true)
- All Run Process calls have timeout=60s on_timeout=kill
- Fix environment-dependent test failures: mock shutil.disk_usage in
disk space and diagnostics --check Behave steps so tests pass
regardless of CI runner disk space; Robot diagnostics test tolerates
disk-space errors
ISSUES CLOSED: #769
The format_output() function returned a string that callers passed to
Rich console.print(), which wraps long lines at terminal width. This
injected literal newline characters into JSON string values (e.g. in
definition_of_done fields), producing invalid JSON that downstream
parsers could not decode (JSONDecodeError: Invalid control character).
For machine-readable formats (json, yaml, plain), format_output() now
writes the rendered output directly to sys.stdout and returns an empty
string. This preserves the exact serialization from json.dumps/
yaml.dump without Rich text processing artifacts.
Refs: #746
Add LLMTrace Pydantic v2 domain model with all required fields (trace_id,
plan_id, decision_id, actor, provider, model, prompt_tokens, completion_tokens,
cost_usd, latency_ms, tool_calls, context_hash, streaming, retry_count, error).
Define 14 OperationalMetricKey values (PLAN_DURATION_MS, PLAN_TOTAL_COST_USD,
PLAN_DECISION_COUNT, ACTOR_INVOCATION_COUNT, ACTOR_LATENCY_MS,
TOOL_INVOCATION_COUNT, TOOL_ERROR_RATE, CONTEXT_BUILD_TIME_MS,
CONTEXT_TOKEN_COUNT, LLM_CALL_COUNT, LLM_TOTAL_TOKENS, LLM_TOTAL_COST_USD,
LLM_AVG_LATENCY_MS, SUBPLAN_COUNT) with MetricEntry model and MetricCollector.
Add llm_traces database table (LLMTraceModel) with LLMTraceRepository for
persistence. TraceService provides recording, querying, metric computation,
plan lifecycle hooks, and optional LangSmith forwarding when
LANGCHAIN_TRACING_V2=true.
Wired into DI container as trace_service. Includes 28 Behave BDD scenarios,
6 Robot Framework smoke tests, 3 ASV benchmark suites, and reference
documentation.
Fix pre-existing cli_core server_mode test flake by mocking
resolve_server_mode in test steps to avoid stale config file interference.
Closes#500
Add structured data builders and Rich renderers for version, info, and
diagnostics commands. Support --format rich/plain/json/yaml output parity
and --check flag for diagnostics (exits non-zero on errors).
Includes Behave scenarios, Robot smoke tests, ASV benchmarks, and updates
existing tests to match the new Rich panel output format.
hamza.khyari
Luis Mendes
freemo
khyari hamza