e4c01492d5
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 15s
CI / lint (pull_request) Successful in 3m18s
CI / quality (pull_request) Successful in 3m47s
CI / typecheck (pull_request) Successful in 3m52s
CI / security (pull_request) Successful in 4m1s
CI / integration_tests (pull_request) Successful in 6m53s
CI / unit_tests (pull_request) Successful in 7m9s
CI / docker (pull_request) Successful in 1m9s
CI / e2e_tests (pull_request) Successful in 11m5s
CI / coverage (pull_request) Successful in 11m15s
CI / status-check (pull_request) Successful in 1s
CI / lint (push) Successful in 3m16s
CI / build (push) Successful in 31s
CI / typecheck (push) Successful in 4m0s
CI / security (push) Successful in 4m1s
CI / benchmark-regression (push) Has been skipped
CI / quality (push) Successful in 4m8s
CI / integration_tests (push) Successful in 6m10s
CI / unit_tests (push) Successful in 7m35s
CI / docker (push) Successful in 1m9s
CI / e2e_tests (push) Successful in 9m58s
CI / coverage (push) Successful in 11m7s
CI / status-check (push) Successful in 1s
CI / benchmark-publish (push) Failing after 17m22s
CI / benchmark-regression (pull_request) Successful in 56m7s
Aligned the `agents actor run` command signature with the specification by introducing positional NAME and PROMPT arguments. The --config/-c option is preserved as an optional fallback for direct YAML invocation. When NAME is provided without --config, the actor is resolved from the Actor Registry. Updated both actor_run.py and actor.py run commands. Added backward compatibility: if --config is provided, it takes precedence over name-based resolution. Review fixes applied (code review round 1): - P1-1: Narrowed bare `except Exception` to `except NotFoundError` in _resolve_config_files to avoid masking infrastructure errors. - P1-2: Moved _resolve_config_files call inside the try block in run() so container/registry init errors get user-friendly messages. Added `except click.exceptions.Exit: raise` to let typer.Exit propagate through the broadened try scope. - P1-3: Added atexit.register cleanup for temp files created by _resolve_config_files (resource leak fix). - P1-4: Added CHANGELOG.md entry for the breaking CLI change. - P2-1: Extracted duplicated _resolve_config_files to shared module `_resolve_actor.py`; both actor.py and actor_run.py now import it. - P2-2: Added guard for actors with no configuration data (config_blob=None) to produce a clear error instead of invalid YAML. - P2-3/P2-4: Added 5 BDD scenarios exercising the real resolve_config_files function (registry path, yaml_text path, config_blob fallback, no-config-data error, not-found error). - P2-5: Added @coverage tags to all new BDD scenarios. - P3-1: Added timeout=120s and on_timeout=kill to Robot tests. - Fixed rxpy_route_validation.robot tests that used the removed --prompt/-p option (replaced with positional NAME + PROMPT args). Review fixes applied (code review round 2): - P2-1: Aligned actor_run.py exception handler from `CleverAgentsException` to `CleverAgentsError`, matching actor.py so infrastructure errors from registry resolution get user-friendly messages instead of falling through to the generic handler. - P2-2: Changed `yaml.dump` to `yaml.safe_dump` in _resolve_actor.py for fail-fast behavior on unexpected types, consistent with the codebase's dominant pattern. - P3-6: Replaced defensive `getattr(actor, ...)` calls with direct Pydantic model attribute access (`actor.yaml_text`, `actor.config_blob`) for type-checker coverage. - P3-1: Switched BDD temp file cleanup from post-assertion `unlink()` to `context.add_cleanup()` for leak-proof teardown. - P2-3/P3-2/P3-3/P3-4: Added 3 BDD edge-case scenarios (empty config_blob dict, infrastructure error propagation, empty string name) and 1 Robot test case (actor_app registry resolution). Review fixes applied (code review round 3): - P1-1: Migrated 48 remaining `-p` invocations across 9 Robot test files to the new positional `NAME PROMPT` pattern (context_delete_all_yes, load_context_test, scientific_paper_e2e_test, routing_prefix_stripping, scientific_paper_basic, scientific_paper_writer_test, context_management_test, initial_next_command_test, system_prompt_template_rendering). - P2-1: Documented `--config/-c` as a spec deviation in `_resolve_actor.py` module docstring (spec lines 4562-4566 define `actor run` with no --config option; issue #901 AC accepts keeping it as optional). - P2-2: Corrected `--config` help text from "fallback" to "overrides registry-based name resolution" — the option takes precedence, not the other way around. - P3-1: Added `.strip()` to `yaml_text` emptiness check in `_resolve_actor.py` to handle whitespace-only values that would otherwise bypass the `config_blob` fallback. - P3-2: Added `from None` to the no-configuration-data `typer.Exit(code=2)` for consistency with the not-found path. - P3-3: Added BDD scenario testing `--config` precedence for `actor_run_app` (was only tested for `actor_app`). - P3-4: Strengthened config_blob BDD scenario to verify generated YAML is parseable via `yaml.safe_load` round-trip. - P3-7: Replaced hardcoded `/tmp/dummy.yaml` with `tempfile.gettempdir()` for portability. - P3-8: Moved 5 inline imports to module level per CONTRIBUTING.md §1292-1294 (3x `import click`, 1x InfrastructureError in steps; 1x `import typer` in robot helper). - P3-9: Added `encoding="utf-8"` to `_write_yaml` in Robot helper for consistency with production code. Review fixes applied (code review round 4): - P2-1: Wrapped `yaml.safe_dump` in `_resolve_actor.py` with `try/except yaml.YAMLError` so non-serialisable config_blob values produce a user-friendly error message instead of a raw traceback. - P2-2: Moved remaining inline `import yaml` to module level in `actor_run_signature_steps.py` per CONTRIBUTING.md §1292-1294. - P2-3: Replaced 3 bare `assert` statements in Robot helper `helper_actor_run_signature.py` with diagnostic `if/print/sys.exit` pattern matching the rest of the file, improving failure diagnostics. Review fixes applied (code review round 5): - P3-4: Replaced per-call `atexit.register(lambda)` in `_resolve_actor.py` with a module-level `_temp_files` set and a single `atexit` handler (`_cleanup_temp_files`) to prevent unbounded handler accumulation in same-process usage (e.g. test suites running multiple CliRunner invocations). - P3-1/P3-2/P3-3: Added 4 BDD scenarios: whitespace-only `yaml_text` fallback to config_blob, config-precedence registry-not-consulted assertion for both `actor_app` and `actor_run_app`, multiple `--config` files with positional NAME. - P4-1: Strengthened error-path BDD assertions to verify error message content (not-found, no-config-data, serialisation-error) alongside exit codes via captured stderr. - P4-2: Added Robot test case for `actor_app` unknown-name error path and corresponding helper function. ISSUES CLOSED: #901
144 lines
4.2 KiB
Python
144 lines
4.2 KiB
Python
"""Helper script for skill_actor_run.robot smoke tests.
|
|
|
|
Tests the ``--skill`` flag on the ``actor-run run`` CLI command.
|
|
Each subcommand is a self-contained check that prints a sentinel on success.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
import sys
|
|
import tempfile
|
|
from pathlib import Path
|
|
from unittest.mock import MagicMock, patch
|
|
|
|
# Ensure local source tree is importable
|
|
_SRC = str(Path(__file__).resolve().parents[1] / "src")
|
|
if _SRC not in sys.path:
|
|
sys.path.insert(0, _SRC)
|
|
|
|
from typer.testing import CliRunner # noqa: E402
|
|
|
|
from cleveragents.cli.commands.actor_run import app as actor_run_app # noqa: E402
|
|
|
|
runner = CliRunner()
|
|
|
|
_SIMPLE_YAML = """\
|
|
name: smoke-actor
|
|
type: custom
|
|
tools:
|
|
- operation: identity
|
|
"""
|
|
|
|
|
|
def _write_yaml(content: str) -> str:
|
|
fd, path = tempfile.mkstemp(suffix=".yaml")
|
|
with os.fdopen(fd, "w") as fh:
|
|
fh.write(content)
|
|
return path
|
|
|
|
|
|
def unknown_skill_exits_with_error() -> None:
|
|
"""Verify that ``--skill`` with an unknown skill name exits with code 2."""
|
|
path = _write_yaml(_SIMPLE_YAML)
|
|
try:
|
|
result = runner.invoke(
|
|
actor_run_app,
|
|
[
|
|
"--config",
|
|
path,
|
|
"test-actor",
|
|
"hello",
|
|
"--skill",
|
|
"local/nonexistent-skill",
|
|
],
|
|
)
|
|
if result.exit_code == 2 and "not found in registry" in result.output:
|
|
print("skill-actor-run-unknown-skill-ok")
|
|
else:
|
|
print(
|
|
f"FAIL: expected exit code 2 with 'not found in registry', "
|
|
f"got code={result.exit_code}",
|
|
file=sys.stderr,
|
|
)
|
|
print(result.output, file=sys.stderr)
|
|
if result.exception:
|
|
import traceback
|
|
|
|
traceback.print_exception(
|
|
type(result.exception),
|
|
result.exception,
|
|
result.exception.__traceback__,
|
|
file=sys.stderr,
|
|
)
|
|
sys.exit(1)
|
|
finally:
|
|
Path(path).unlink(missing_ok=True)
|
|
|
|
|
|
def skill_flag_accepted() -> None:
|
|
"""Verify that ``--skill`` with a valid skill name is accepted by the CLI."""
|
|
from cleveragents.application.services.skill_service import SkillService
|
|
from cleveragents.domain.models.core.skill import Skill
|
|
|
|
# Build a real in-memory SkillService with a registered skill
|
|
svc = SkillService()
|
|
skill = Skill(
|
|
name="local/smoke-skill",
|
|
description="Smoke test skill",
|
|
tool_refs=["builtin/read_file"],
|
|
)
|
|
svc._skills["local/smoke-skill"] = skill
|
|
|
|
mock_container = MagicMock()
|
|
mock_container.skill_service.return_value = svc
|
|
|
|
path = _write_yaml(_SIMPLE_YAML)
|
|
try:
|
|
with patch(
|
|
"cleveragents.reactive.application.get_container",
|
|
return_value=mock_container,
|
|
):
|
|
result = runner.invoke(
|
|
actor_run_app,
|
|
[
|
|
"--config",
|
|
path,
|
|
"test-actor",
|
|
"hello",
|
|
"--skill",
|
|
"local/smoke-skill",
|
|
],
|
|
)
|
|
# The command may fail for other reasons (no real LLM, etc.) but
|
|
# the important thing is it did NOT fail with "not found in registry"
|
|
# and the skill was accepted.
|
|
if "not found in registry" in (result.output or ""):
|
|
print("FAIL: skill was rejected as not found", file=sys.stderr)
|
|
print(result.output, file=sys.stderr)
|
|
sys.exit(1)
|
|
print("skill-actor-run-flag-accepted-ok")
|
|
finally:
|
|
Path(path).unlink(missing_ok=True)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Main dispatcher
|
|
# ---------------------------------------------------------------------------
|
|
|
|
_COMMANDS = {
|
|
"unknown-skill": unknown_skill_exits_with_error,
|
|
"skill-flag-accepted": skill_flag_accepted,
|
|
}
|
|
|
|
|
|
def main() -> None:
|
|
if len(sys.argv) < 2 or sys.argv[1] not in _COMMANDS:
|
|
print(f"Usage: {sys.argv[0]} <{'|'.join(_COMMANDS)}>", file=sys.stderr)
|
|
sys.exit(2)
|
|
_COMMANDS[sys.argv[1]]()
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|