- Add `cost` field to `_plan_spec_dict` in plan.py so `plan status --format
json` includes cost metadata in output (was previously omitted)
- Fix `_make_plan_with_cost` helper: replace broken
`Plan.namespaced_name.__class__(...)` (returns FieldInfo, not NamespacedName)
with `NamespacedName.parse("local/test-plan")`; add `NamespacedName` import
- Remove `<plan_id>` / `<session_id>` angle-bracket placeholders from feature
file When steps: these are not Scenario Outline templates so they were
passed literally to `_validate_plan_ulid()` which rejected them; steps now
use `context.plan_id` / `context.session_id` set by the Given steps
- Rename `I run plan status for the plan` step to
`I run plan status for the plan with cost reporting` to avoid AmbiguousStep
collision with `plan_cli_spec_alignment_steps.py`
- Fix session `estimated_cost` assertions: the value is nested under
`token_usage` (not top-level) and is a formatted string `"$0.0080"`,
not a float
- Apply ruff format to step file
ISSUES CLOSED: #5250
Rewrite cost_reporting_cli_steps.py to properly test plan status and
session show cost reporting in CLI output.
- Add @when step implementations that execute CLI via CliRunner
- Add proper @given fixtures that create Plan and Session domain objects
with CostMetadata/SessionTokenUsage data
- Replace setattr-based assertions with direct context.result assertions
- Import json module and add _unwrap_envelope helper for CLI spec envelopes
- Follow existing test patterns from session_cli_steps.py
Closes#10616
---
Automated by CleverAgents Bot
Supervisor: PR Fix | Agent: task-implementor
Added BDD feature file and step implementations for cost reporting in CLI commands.
- Plan status now includes cost metadata in JSON output
- Session show includes estimated cost in JSON output
- Both commands display cost information in rich output format
ISSUES CLOSED: #5250
All 9 test cases previously only called `project create` + `project show`
despite documentation claiming to test correction, plan execution, and
subplan spawning workflows. This commit fixes that.
Changes per reviewer blockers:
Blocker 1 (hollow stubs): Each test now invokes the workflow its name
and documentation describe:
- test_correction_workflow.robot: calls `plan correct --mode revert/append`
(with --dry-run for mode tests; actual correction for state transition test)
- test_project_plan_workflow.robot: Plan Execution Workflow now calls
`plan use` + `plan execute` + `plan status`; Project Creation Workflow
uses resource-based project creation and verifies list output
- test_subplan_workflow.robot: Subplan Spawning and Merge Result Validation
inspect `plan tree` for spec-required child_plans/decision_ids fields;
Three-Way Merge Workflow exercises `plan apply`
Blocker 2 (temp dir leaks): All tests now use `Create Temp Git Repo`
(creates inside SUITE_HOME, cleaned by E2E Suite Teardown) instead of
`Create Temp Directory` (tempfile.mkdtemp outside SUITE_HOME, leaked).
The unused `Set Up E2E Project Test` local keyword (which created an
orphaned suite-variable temp dir) is removed from all three files.
Each file gains a proper suite setup keyword that registers the
local/code-review action needed by plan lifecycle tests, following
the pattern established in m6_acceptance.robot.
Address PR #10614 review findings:
- Added [Timeout] 30 minutes to all 9 test cases to prevent CI hangs
- Removed duplicate Create Temp Directory keyword from each file (moved to common_e2e.resource)
- Added Set Up E2E Project Test keyword for consistent temp dir creation via TEST NAME variable
- Added msg= parameters to Should Not Be Empty assertions for better debugging
Signed-off-by: HAL9000 <hal9000@noreply.git.cleverthis.com>
- Add test_project_plan_workflow.robot for project creation and plan execution workflows
- Add test_correction_workflow.robot for revert and append mode correction workflows
- Add test_subplan_workflow.robot for subplan spawning and three-way merge workflows
- All tests use Robot Framework with real CLI execution (no mocking)
- Tests validate spec-required output formats
- Tests skip gracefully if LLM API keys are not configured
Closes#5259
Click consumes the bare '--' end-of-options marker before tokens reach
ctx.args, so the `if not key:` defensive branch in `attach` cannot be
exercised from CLI invocation. Mark it `# pragma: no cover` so
diff-coverage stops flagging lines 323/324/326 on this PR.
Verified with a direct CliRunner invocation: passing '--' (alone or
followed by other tokens) never produces a token whose `[2:]` is empty
— Click strips it.
Closes#3684
Restore the two behaviours that master had but were lost in this branch:
1. Convert hyphens to underscores in named-option keys before forwarding
to the service layer (--coverage-threshold → coverage_threshold). This
matches Typer/Click's universal convention and the documented
attach_validation(args=...) contract, where keys must be Python-
identifier-style strings.
2. Detect a missing value when the next token starts with -- (e.g.
"--threshold --strict true" no longer silently sets threshold to the
literal string "--strict"; it errors with a clear "Missing value for
option" message).
Update the Behave step definitions and the Robot helper to assert the
underscore-normalised key (coverage_threshold), matching the restored
behaviour. The CLI-level option name (--coverage-threshold) is unchanged
in the feature file and Robot test — only the dict key the service
receives is normalised.
The new step file defined a Then step that collided with the pre-existing
definition in validation_attach_type_guard_steps.py:220, causing behave to
abort step-module loading with AmbiguousStep across every worker and
failing the unit_tests gate with 32 errored features.
Remove the duplicate from the new file and bridge context.last_result =
context.named_opts_result after each invoke so the pre-existing,
context.last_result-based step covers the new feature file's scenarios.
Closes#3684
Refactors the 'agents validation attach' command to accept extra validation
arguments as named CLI options using '--key value' format (e.g.
'--coverage-threshold 90'), as required by the specification.
Changes:
- validation.py: Replace positional 'key=value' Argument with Typer context
settings (allow_extra_args=True, ignore_unknown_options=True) to capture
'--key value' named options from ctx.args. Strips '--' prefix and maps
tokens to {key: value} dict entries. Rejects bare tokens (not '--key value')
with a clear error message.
- features/tdd_validation_attach_named_options.feature: New TDD Behave
scenarios covering single/multiple named options, no-args case, and
rejection of old positional key=value format.
- features/steps/tdd_validation_attach_named_options_steps.py: Step
definitions for the new feature.
- robot/validation_attach_named_options.robot: Robot Framework integration
tests verifying spec-compliant '--key value' option format.
- robot/helper_validation_attach_named_options.py: Helper script for the
Robot Framework tests.
- features/steps/tdd_cli_incomplete_subcommand_registration_steps.py: Fix
pre-existing CliRunner(mix_stderr=False) incompatibility with current Typer.
- features/steps/tool_runtime_steps.py: Fix pre-existing AmbiguousStep error
by converting conflicting parse-based step definitions to regex matchers.
- features/consolidated_tool.feature: Update step text to match renamed step.
Closes#3684
Replace four time.sleep(0.01) timestamp guards in
memory_service_coverage_steps.py with a deterministic
_wait_for_clock_advance() helper that busy-waits on the monotonic
clock until the UTC clock actually advances past the recorded
before timestamp, bounded by a 2-second deadline.
Also introduce memory_service_clock_wait.feature and its step
definitions to verify the helper raises AssertionError on deadline
exceeded and returns normally once the clock advances.
Closes#9963
Reformat the lint session's ruff check call in noxfile.py to a multi-line
form so ruff format --check passes (the single-line form exceeded 88 chars).
Add CHANGELOG.md entry under [Unreleased] ### Changed for issue #10848.
ISSUES CLOSED: #10848
- features/steps/tdd_plan_generation_validate_steps.py: drop duplicate
@given/@then registrations (live in plan_generation_langgraph_coverage_
steps.py; redefinition caused AmbiguousStep errors crashing all 8
behave-parallel workers); have the @when step feed the docstring as the
FakeListLLM response so each scenario tests _validate()'s parsing of a
specific PASS/FAIL signal rather than the input code.
- features/tdd_plan_generation_validate_logic.feature: add the required
@tdd_issue tag alongside @tdd_issue_10746 (enforced by
features/environment.py); tighten scenario 4 wording to remove the
reference to the obsolete length guard.
- robot/plan_generation_graph.robot: drop assertion for handle_retry node
(retry is a conditional edge, not a fifth node) and update node-count
check from 5 to 4; update Should Retry test to assert _should_retry does
NOT mutate state (read-only contract for LangGraph conditional edges).
- src/cleveragents/agents/graphs/plan_generation.py:
* _validate: persist retry_count increment in return dict (FAIL path and
exception path) so LangGraph propagates it through the state graph.
* _should_retry: remove state mutation (conditional-edge functions are
read-only in LangGraph; mutations were silently dropped, causing
retry_count to remain 0 forever and the graph to loop infinitely).
Adjust comparison to retry_count <= max_retries because _validate has
already incremented before _should_retry runs.
* __init__: add max_context_files parameter (default 5, validated > 0)
and wire it into _format_context_summary in place of the hardcoded 5,
implementing the configurable-limits contract tested by
features/agent_configurable_limits.feature.
ISSUES CLOSED: #10746
- Added Given step handler 'I have a langgraph PlanGenerationGraph instance'
to create a PlanGenerationGraph with FakeListLLM in the test setup.
- Added Then step handler 'the langgraph validation status should be "{status}"'
to assert PASS/FAIL results from _validate().
- Fixed TDD tag format: replaced '@tdd_issue @tdd_issue_10746' with '@tdd_issue_10746'
per project convention (single tdd tag, not two).
These changes resolve all three review blocking issues for PR #10867:
1. Missing Given step handler causing test execution failures.
2. Missing Then step handler causing Behave StepDefinitionNotFoundError.
3. TDD tag format violation preventing CI from properly tagging tests.
The core code fix (removing length-based bypass in _validate) was already
correctly implemented and does not need changes.
ISSUES CLOSED: #10867
The `agents actor run` and `actor_run` Typer commands wrapped their main
try/except around `_resolve_config_files` with a bare `except
click.exceptions.Exit: raise` clause to let resolver-raised exits
propagate cleanly. Modern Typer re-exports `typer.Exit` from its vendored
click (`typer._click.exceptions.Exit`), which is NOT a subclass of
`click.exceptions.Exit`. So `_resolve_config_files`'s
`raise typer.Exit(code=2)` for unknown-actor / no-config-data / bad-blob
cases fell through to the generic `except Exception` clause, which
re-raised as `typer.Exit(code=3)` and replaced the targeted stderr
("not found in registry") with a generic "Unexpected error" message.
Behave scenarios at `actor_run_signature.feature:38` and `:56` and three
Robot integration scenarios verified the original exit-code/message
contract and failed. Catch `(typer.Exit, click.exceptions.Exit)` instead.
The five errored scenarios in `actor_run_signature_resolve_steps.py` and
`actor_run_signature_security_steps.py` had the same root cause from the
test side: `except (SystemExit, click.exceptions.Exit)` could not catch
the raised `typer.Exit`. Widen the tuple to include `typer.Exit`.
memory_service.py's `SQLChatMessageHistory(connection_string=...)` call
broke against langchain-community 0.4.2: the keyword was renamed
`connection` (which now accepts a URL string OR an Engine). Update the
call site and refresh the local `typings/langchain_community/...`
pyright stub to match the upstream signature so typecheck stays clean.
ISSUES CLOSED: #10921
- Remove misplaced pytest test files: tests/unit/a2a_test_http_transport.py,
tests/unit/__init__.py, and features/steps/test_a2a_http_transport_pytest.py.
Project layout uses Behave in features/ exclusively per CONTRIBUTING.md.
- Resolve AmbiguousStep crash in features/steps/a2a_facade_steps.py by
deduplicating step_transport_connect / step_transport_disconnect /
"the transport should not be connected" definitions left over from the
pre-implementation stub.
- Remove all `# type: ignore[arg-type]` comments (zero-tolerance policy).
- Fix ruff lint failures in src/cleveragents/a2a/transport.py: drop unused
imports (Any, map_domain_error, BaseHandler, OpenerDirector), wrap long
log lines (E501), and switch ssl.VerifyMode literal 0 to CERT_NONE for
pyright compliance.
- Update Robot helpers (robot/helper_a2a_facade.py,
robot/helper_m6_autonomy_acceptance.py) and the m6 / consolidated Behave
scenarios to verify the new server-mode lifecycle (connect succeeds with
valid URL, send-before-connect raises RuntimeError, invalid scheme raises
ValueError) instead of the obsolete "stub raises A2aNotAvailableError"
contract.
- Broaden the "I try to connect via the transport to ..." regex so the
invalid-URL scenario outline matches the empty-string / quoted / None
example cells; alias "I disconnect the transport" with @then so it is
reachable from `And` after a `Then` keyword.
Replace the A2aHttpTransport stub with a working HTTP(S) transport that:
- Connects to remote A2A servers via HTTPS (with configurable TLS verification)
- Sends JSON-RPC 2.0 requests over HTTP POST
- Parses JSON-RPC responses into A2aResponse objects
- Handles HTTP errors (4xx/5xx) with structured error mapping
- Supports JWT Bearer token authentication
- Validates connection state before send operations
Added comprehensive test coverage:
- Behave BDD scenarios for validation and lifecycle testing
- Pytest unit tests with mocked HTTP responses covering success,
errors, network failures, auth tokens, and roundtrip serialization
Three artifacts of the bad merge resolution on this branch are now corrected:
- [tool.coverage.report] section restored. The Robot integration test
``Coverage Threshold :: Noxfile Contains Coverage Threshold Constant``
asserts pyproject.toml contains ``fail_under = 96.5`` as the single
source for the coverage floor. The section was lost during merge-conflict
resolution; ``noxfile._read_coverage_fail_under`` was falling back to its
hard-coded default and the robot test was failing as a result.
- ``fastapi>=0.115.0`` (was 0.100.0). Master pins 0.115.0; the older floor
on this branch came in with the auto-scratch fix and is now bumped to
match.
- Duplicate ``langchain-anthropic>=0.2.0`` entry removed (line 40). Master
declares it once; the duplicate is a stray from the same bad merge.
Refs: #9055
Three step patterns in pyyaml_security_steps.py clashed with existing
step files, causing all Behave features to error at load time:
- "I call load_yaml_text with YAML text" clashed with actor_config_coverage_boost_steps.py:103
- "the load_yaml_text result should have key" clashed with actor_config_coverage_boost_steps.py:90
- "a ValueError should be raised" clashed with lsp_registry_steps.py:475
Rename all three to unique patterns and update pyyaml_security.feature
to match. Also fix typings/behave/runner.pyi ruff format (.pyi convention:
single blank line before class, no blank lines between stub methods) and
add missing fastapi>=0.100.0 to pyproject.toml (asgi_app.py imports
fastapi but it was absent from declared dependencies, causing typecheck
and integration test failures).
Refs: #9055
Remove ``# type: ignore[import-untyped]`` comments from features/steps/pyyaml_security_steps.py, replacing them with proper .pyi stubs for behave.runner.Context in typings/behave/runner.pyi.
Refs: #9055
Scenario 2 in features/pyyaml_security.feature referenced three step
definitions that were not implemented in the step file, causing
StepDefinitionNotFoundError and CI unit_tests failure:
- When I call load_yaml_text with YAML text "..."
- Then the load_yaml_text result should have key "..." equal to "..."
Added both missing step definitions with proper type annotations.
Also fixed dead code (except (ValueError, Exception) -> except Exception,
ruff B014) and toned down the alarmist assertion message per reviewer
feedback.
Added explicit pyyaml>=6.0.3 constraint to pyproject.toml to address
CVE-2017-18342 and related advisories. A codebase-wide audit confirmed
all YAML loading uses yaml.safe_load() exclusively via
cleveragents.actor.yaml_loader. Added BDD regression scenarios in
features/pyyaml_security.feature to verify the version constraint and
safe-load enforcement are maintained. Updated CHANGELOG.md with a
security entry.
ISSUES CLOSED: #9055
Address CI lint + unit_tests failures on the namespace-lock test suite:
- Step file: drop unused imports (yaml, ExitStack, ValidationError) and
unused noqa directives flagged by ruff F401/RUF100; rewrite the
collected-errors list to use iterable unpacking (RUF005); register the
"I create an ActorLoader with initial actors from multiple namespaces"
step under both @given and @when so scenario 3 is no longer reported as
undefined; remove the dead _error_collector inner function the reviewer
flagged.
- Step regex fix: the concurrent-modifications step pattern ended with
'' (two single quotes) instead of `` (two backticks), so it never
matched the feature file's `(triggering ``discover()``)` literal.
- Test fixtures: add the required `description` field to _make_actor_yaml
in both the BDD step file and tests/actor/test_loader_list_actors_thread_safety.py
so ActorConfigSchema validation passes (previously every scenario
errored at discover() with "description: Field required").
- Concurrent worker names: collapse three-slash actor names like
"conc/ns{i % 2}/concurrent_{i}" to the single-slash form
"conc/concurrent_{i}" required by the schema's namespaced-name rule.
- Test file: drop unused `yaml` import and three unused F841 assignments
in _list_worker; apply ruff format.
- Feature file: switch the @issue_8660 TDD tag to @tdd_issue_8588 to
match the CONTRIBUTING.md tag convention for the bug issue this PR
closes.
ISSUES CLOSED: #8588
HAL9000
HAL9000
controller-ci-rerun
cleveragents-auto
freemo
CleverThis