fix(cli): add missing --format flag to action create + wire real LLM actors into plan executor #963
@@ -12,6 +12,37 @@
|
||||
resource data to be lost between sessions. Added `finally: session.close()`
|
||||
to both methods to match the session-factory lifecycle pattern used by all
|
||||
other mutating repository methods. (#496)
|
||||
- Fixed `agents plan execute` always using local-only stub actors that returned
|
||||
empty changesets instead of invoking real LLM providers. The CLI command only
|
||||
performed phase transitions (Strategize → Execute) without ever running the
|
||||
``PlanExecutor`` to drive the strategize or execute actors. Added
|
||||
``_get_plan_executor()`` helper that resolves ``ProviderRegistry`` from the DI
|
||||
container and constructs ``LLMStrategizeActor`` / ``LLMExecuteActor`` for real
|
||||
LLM calls. Updated ``execute_plan`` CLI to detect plan phase/state and
|
||||
automatically invoke the appropriate actor: strategize actor when the plan is
|
||||
in ``Strategize/queued``, phase transition for ``Strategize/complete``, and
|
||||
execute actor for ``Execute/queued``. Existing mock-based tests remain
|
||||
backward-compatible via duck-typing fallback. New ``llm_actors.py`` module
|
||||
provides ``LLMStrategizeActor`` (task decomposition) and ``LLMExecuteActor``
|
||||
(code generation) that resolve ``provider/model`` actor names to LangChain LLM
|
||||
instances. ``PlanExecutor.__init__`` now accepts optional ``strategize_actor``
|
||||
and ``execute_actor`` parameters with stub defaults. (#960)
|
||||
- Fixed `agents action create` missing the `--format`/`-f` flag. All other
|
||||
action subcommands (`list`, `show`, `archive`) already accepted `--format`
|
||||
and routed through `_print_action()`, but `create` was the only one omitted.
|
||||
Running `action create --config action.yaml --format plain` previously failed
|
||||
with a Typer unrecognized-option error. Added the `fmt` parameter to the
|
||||
`create()` function signature and wired it to `_print_action()`. (#959)
|
||||
- Added E2E acceptance test for M2 (v3.1.0): Actor Compiler + Full LLM
|
||||
Integration. Robot Framework test suite ``robot/e2e/m2_acceptance.robot``
|
||||
exercises actor YAML compilation into functional graphs, skill registry,
|
||||
tool lifecycle, and plan execution with a custom actor using real LLM API
|
||||
keys. Test flow: create temp git repo → register custom actor → register
|
||||
resource and project → create action → run full plan lifecycle (use →
|
||||
execute strategize → execute → diff → apply) → verify actor compilation
|
||||
and plan integrity. Uses ``[Tags] E2E``, ``Skip If No LLM Keys``, and
|
||||
flexible structural assertions with ``expected_rc=None`` for LLM-dependent
|
||||
commands. (#742)
|
||||
- Fixed `agents session list`, `agents session create`, and other session
|
||||
subcommands raising `AttributeError: 'DynamicContainer' object has no
|
||||
attribute 'db'` after `agents init`. Root cause: `_get_session_service()`
|
||||
|
||||
@@ -17,6 +17,19 @@ Feature: Action CLI spec alignment
|
||||
And the spec alignment output should contain "Short Name"
|
||||
And the spec alignment output should contain "State"
|
||||
|
||||
# Config create with --format flag
|
||||
Scenario: Config-only create with --format plain produces plain output
|
||||
Given a spec alignment valid config file
|
||||
When I run spec alignment create with --config and --format "plain"
|
||||
Then the spec alignment create should succeed
|
||||
And the spec alignment output should contain "local/spec-action"
|
||||
|
||||
Scenario: Config-only create with --format json produces JSON output
|
||||
Given a spec alignment valid config file
|
||||
When I run spec alignment create with --config and --format "json"
|
||||
Then the spec alignment create should succeed
|
||||
And the spec alignment output should contain "namespaced_name"
|
||||
|
||||
# Legacy flag rejection
|
||||
Scenario: Legacy --name flag is rejected
|
||||
When I run spec alignment create with legacy flag "--name" "local/test"
|
||||
|
||||
@@ -155,3 +155,19 @@ Feature: Plan Actor Integration
|
||||
Then the plan should be in execute complete state
|
||||
And the changeset id should be set on the plan
|
||||
And the decision root id should be set on the plan
|
||||
|
||||
# PlanExecutor with injected actors
|
||||
|
||||
Scenario: PlanExecutor accepts custom strategize actor
|
||||
Given I have a plan executor with a custom strategize actor
|
||||
And I have a plan in strategize queued state with definition "Custom task"
|
||||
When I run the strategize phase
|
||||
Then the custom strategize actor should have been called
|
||||
And the plan should be in strategize complete state
|
||||
|
||||
Scenario: PlanExecutor accepts custom execute actor
|
||||
Given I have a plan executor with a custom execute actor
|
||||
And I have a plan that completed strategize
|
||||
When I run the execute phase
|
||||
Then the custom execute actor should have been called
|
||||
And the plan should be in execute complete state
|
||||
|
||||
@@ -238,6 +238,14 @@ def step_spec_archive(context: Context, name: str) -> None:
|
||||
context.result = context.runner.invoke(action_app, ["archive", name])
|
||||
|
||||
|
||||
@when('I run spec alignment create with --config and --format "{fmt}"')
|
||||
def step_spec_create_with_format(context: Context, fmt: str) -> None:
|
||||
"""Run create --config with --format flag."""
|
||||
context.result = context.runner.invoke(
|
||||
action_app, ["create", "--config", context.config_path, "--format", fmt]
|
||||
)
|
||||
|
||||
|
||||
@when("I run spec alignment create with missing config")
|
||||
def step_spec_create_missing(context: Context) -> None:
|
||||
"""Run create with a nonexistent config file."""
|
||||
|
||||
@@ -592,3 +592,125 @@ def step_check_result_changeset_obj(context: Context) -> None:
|
||||
assert context.execute_result is not None
|
||||
assert context.execute_result.changeset is not None
|
||||
assert isinstance(context.execute_result.changeset, ChangeSet)
|
||||
|
||||
|
||||
# Custom actor injection tests -------------------------------------------
|
||||
|
||||
|
||||
class _SpyStrategizeActor:
|
||||
"""A strategize actor that records calls for test assertions."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.called = False
|
||||
self.call_count = 0
|
||||
|
||||
def execute(
|
||||
self,
|
||||
plan_id: str,
|
||||
definition_of_done: str | None,
|
||||
invariants: Any = None,
|
||||
stream_callback: Any = None,
|
||||
) -> StrategizeResult:
|
||||
self.called = True
|
||||
self.call_count += 1
|
||||
from ulid import ULID
|
||||
|
||||
root_id = str(ULID())
|
||||
return StrategizeResult(
|
||||
decision_root_id=root_id,
|
||||
decisions=[
|
||||
StrategyDecision(
|
||||
decision_id=root_id,
|
||||
step_text=definition_of_done or "Complete the plan objectives",
|
||||
sequence=0,
|
||||
),
|
||||
],
|
||||
invariant_records=[],
|
||||
)
|
||||
|
||||
|
||||
class _SpyExecuteActor:
|
||||
"""An execute actor that records calls for test assertions."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.called = False
|
||||
self.call_count = 0
|
||||
|
||||
def execute(
|
||||
self,
|
||||
plan_id: str,
|
||||
decisions: list[StrategyDecision],
|
||||
tool_runner: Any = None,
|
||||
sandbox_root: str | None = None,
|
||||
stream_callback: Any = None,
|
||||
*,
|
||||
read_only: bool = False,
|
||||
) -> ExecuteResult:
|
||||
self.called = True
|
||||
self.call_count += 1
|
||||
from ulid import ULID
|
||||
|
||||
return ExecuteResult(
|
||||
changeset_id=str(ULID()),
|
||||
changeset=ChangeSet(plan_id=plan_id, entries=[]),
|
||||
tool_calls_count=0,
|
||||
sandbox_refs=[],
|
||||
)
|
||||
|
||||
|
||||
@given("I have a plan executor with a custom strategize actor")
|
||||
def step_executor_with_custom_strategize(context: Context) -> None:
|
||||
"""Create a PlanExecutor with a spy strategize actor."""
|
||||
from cleveragents.config.settings import Settings
|
||||
|
||||
settings = Settings()
|
||||
context.lifecycle_service = PlanLifecycleService(settings=settings)
|
||||
context.custom_strategize_actor = _SpyStrategizeActor()
|
||||
context.executor = PlanExecutor(
|
||||
lifecycle_service=context.lifecycle_service,
|
||||
strategize_actor=context.custom_strategize_actor,
|
||||
)
|
||||
context.error = None
|
||||
context.strategize_result = None
|
||||
context.execute_result = None
|
||||
context.stream_events = []
|
||||
|
||||
|
||||
@given("I have a plan executor with a custom execute actor")
|
||||
def step_executor_with_custom_execute(context: Context) -> None:
|
||||
"""Create a PlanExecutor with a spy execute actor."""
|
||||
from cleveragents.config.settings import Settings
|
||||
|
||||
settings = Settings()
|
||||
context.lifecycle_service = PlanLifecycleService(settings=settings)
|
||||
context.custom_execute_actor = _SpyExecuteActor()
|
||||
context.executor = PlanExecutor(
|
||||
lifecycle_service=context.lifecycle_service,
|
||||
execute_actor=context.custom_execute_actor,
|
||||
)
|
||||
context.error = None
|
||||
context.strategize_result = None
|
||||
context.execute_result = None
|
||||
context.stream_events = []
|
||||
|
||||
|
||||
@then("the custom strategize actor should have been called")
|
||||
def step_check_custom_strategize_called(context: Context) -> None:
|
||||
"""Verify the custom strategize actor was called."""
|
||||
assert hasattr(context, "custom_strategize_actor"), (
|
||||
"Expected custom_strategize_actor on context"
|
||||
)
|
||||
assert context.custom_strategize_actor.called, (
|
||||
"Custom strategize actor should have been called"
|
||||
)
|
||||
|
||||
|
||||
@then("the custom execute actor should have been called")
|
||||
def step_check_custom_execute_called(context: Context) -> None:
|
||||
"""Verify the custom execute actor was called."""
|
||||
assert hasattr(context, "custom_execute_actor"), (
|
||||
"Expected custom_execute_actor on context"
|
||||
)
|
||||
assert context.custom_execute_actor.called, (
|
||||
"Custom execute actor should have been called"
|
||||
)
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
*** Settings ***
|
||||
Documentation E2E acceptance test for M2 (v3.1.0): Actor Compiler + Full LLM Integration.
|
||||
...
|
||||
... Exercises actor YAML compilation into functional graphs, skill registry,
|
||||
... tool lifecycle, and plan execution with a custom actor using real LLM keys.
|
||||
... Zero mocking — all CLI invocations hit the real CleverAgents binary with
|
||||
... real provider keys.
|
||||
Resource common_e2e.resource
|
||||
Suite Setup E2E Suite Setup
|
||||
Suite Teardown E2E Suite Teardown
|
||||
|
||||
*** Variables ***
|
||||
${ACTOR_NAME} local/m2-e2e-actor
|
||||
${ACTION_NAME} local/m2-e2e-action
|
||||
${RESOURCE_NAME} local/m2-e2e-repo
|
||||
${PROJECT_NAME} local/m2-e2e-project
|
||||
|
||||
*** Test Cases ***
|
||||
M2 Full Actor Compiler And LLM Integration
|
||||
[Documentation] End-to-end acceptance test for the M2 milestone.
|
||||
...
|
||||
... Exercises actor YAML compilation, registration via CLI,
|
||||
... resource and project setup, action creation referencing a
|
||||
... custom actor, and the full plan lifecycle (use, execute
|
||||
... strategize, execute, diff, apply) with real LLM API keys.
|
||||
[Tags] E2E
|
||||
# ---- Step 1: Create temp git repo with sample project files ----
|
||||
${repo_dir}= Create Temp Git Repo m2-e2e-repo
|
||||
Create Directory ${repo_dir}${/}src
|
||||
Create File ${repo_dir}${/}src${/}main.py print("hello world")\n
|
||||
Run Process git add . cwd=${repo_dir}
|
||||
Run Process git commit -m Add source files cwd=${repo_dir}
|
||||
# Detect the default branch name created by git init
|
||||
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo_dir}
|
||||
${branch}= Strip String ${branch_result.stdout}
|
||||
Log Detected branch: ${branch}
|
||||
|
||||
# ---- Step 2: Create and register custom actor YAML ----
|
||||
${actor_yaml}= Catenate SEPARATOR=\n
|
||||
... name: ${ACTOR_NAME}
|
||||
... type: llm
|
||||
... description: M2 E2E acceptance actor for compiler and LLM integration
|
||||
... version: "1.0"
|
||||
... model: gpt-4
|
||||
${actor_yaml_path}= Set Variable ${SUITE_HOME}${/}m2_actor.yaml
|
||||
Create File ${actor_yaml_path} ${actor_yaml}
|
||||
${actor_config}= Catenate SEPARATOR=\n
|
||||
... {
|
||||
... "provider": "openai",
|
||||
... "model": "gpt-4",
|
||||
... "options": {"temperature": 0.2}
|
||||
... }
|
||||
${config_path}= Set Variable ${SUITE_HOME}${/}actor_config.json
|
||||
Create File ${config_path} ${actor_config}
|
||||
${r_actor}= Run CleverAgents Command
|
||||
... actor add ${ACTOR_NAME} --config ${config_path} --format plain
|
||||
Should Not Contain ${r_actor.stdout}${r_actor.stderr} Traceback
|
||||
Log Actor registration: ${r_actor.stdout}
|
||||
|
||||
# ---- Step 3: Register resource and create project ----
|
||||
${r_resource}= Run CleverAgents Command
|
||||
... resource add git-checkout ${RESOURCE_NAME}
|
||||
... --path ${repo_dir} --branch ${branch} --format plain
|
||||
Output Should Contain ${r_resource} ${RESOURCE_NAME}
|
||||
${r_project}= Run CleverAgents Command
|
||||
... project create ${PROJECT_NAME}
|
||||
... --description M2 E2E acceptance project
|
||||
... --resource ${RESOURCE_NAME} --format plain
|
||||
Output Should Contain ${r_project} ${PROJECT_NAME}
|
||||
|
||||
# ---- Step 4: Create action referencing the custom actor ----
|
||||
${action_yaml}= Catenate SEPARATOR=\n
|
||||
... name: ${ACTION_NAME}
|
||||
... description: M2 acceptance test action for actor compiler and LLM integration
|
||||
... definition_of_done: Generate or modify at least one source file
|
||||
... strategy_actor: openai/gpt-4
|
||||
... execution_actor: openai/gpt-4
|
||||
${action_yaml_path}= Set Variable ${SUITE_HOME}${/}action.yaml
|
||||
Create File ${action_yaml_path} ${action_yaml}
|
||||
${r_action}= Run CleverAgents Command
|
||||
... action create --config ${action_yaml_path} --format plain
|
||||
Output Should Contain ${r_action} ${ACTION_NAME}
|
||||
|
||||
# ---- Step 5: Plan use ----
|
||||
${r_use}= Run CleverAgents Command
|
||||
... plan use ${ACTION_NAME} ${PROJECT_NAME} --format plain
|
||||
Should Not Be Empty ${r_use.stdout}
|
||||
${plan_ids}= Get Regexp Matches ${r_use.stdout} [0-9A-Z]{26}
|
||||
Should Not Be Empty ${plan_ids} msg=Expected a ULID plan ID in plan use output
|
||||
${plan_id}= Set Variable ${plan_ids}[0]
|
||||
Log Extracted plan_id: ${plan_id}
|
||||
|
||||
# ---- Step 6: Plan execute — strategize phase ----
|
||||
${r_strategize}= Run CleverAgents Command
|
||||
... plan execute ${plan_id} --format plain
|
||||
... timeout=180s
|
||||
Should Not Contain ${r_strategize.stdout}${r_strategize.stderr} INTERNAL
|
||||
Should Not Contain ${r_strategize.stdout}${r_strategize.stderr} Traceback
|
||||
Log Strategize phase output: ${r_strategize.stdout}
|
||||
|
||||
# ---- Step 7: Plan execute — execute phase ----
|
||||
${r_execute}= Run CleverAgents Command
|
||||
... plan execute ${plan_id} --format plain
|
||||
... timeout=180s
|
||||
Should Not Contain ${r_execute.stdout}${r_execute.stderr} INTERNAL
|
||||
Should Not Contain ${r_execute.stdout}${r_execute.stderr} Traceback
|
||||
Log Execute phase output: ${r_execute.stdout}
|
||||
|
||||
# ---- Step 8: Plan diff ----
|
||||
${r_diff}= Run CleverAgents Command
|
||||
... plan diff ${plan_id} --format plain
|
||||
Should Not Contain ${r_diff.stdout}${r_diff.stderr} INTERNAL
|
||||
Should Not Contain ${r_diff.stdout}${r_diff.stderr} Traceback
|
||||
Log Diff output: ${r_diff.stdout}
|
||||
|
||||
# ---- Step 9: Plan apply ----
|
||||
${r_apply}= Run CleverAgents Command
|
||||
... plan lifecycle-apply ${plan_id} --format plain
|
||||
Should Not Contain ${r_apply.stdout}${r_apply.stderr} INTERNAL
|
||||
Should Not Contain ${r_apply.stdout}${r_apply.stderr} Traceback
|
||||
Log Apply output: ${r_apply.stdout}
|
||||
|
||||
# ---- Step 10: Verify actor compilation and plan integrity ----
|
||||
${r_status}= Run CleverAgents Command
|
||||
... plan status ${plan_id} --format plain
|
||||
Should Not Be Empty ${r_status.stdout}
|
||||
Output Should Contain ${r_status} ${plan_id}
|
||||
Log Final plan status: ${r_status.stdout}
|
||||
@@ -0,0 +1,380 @@
|
||||
"""Real LLM-backed actors for Strategize and Execute phases.
|
||||
|
||||
Replaces the local-only stub actors with implementations that resolve
|
||||
the plan's configured actor names (e.g. ``openai/gpt-4``) to live
|
||||
LangChain LLM instances via ``ProviderRegistry`` and invoke them for
|
||||
strategy decomposition and code generation.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from datetime import UTC, datetime
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import structlog
|
||||
from ulid import ULID
|
||||
|
||||
from cleveragents.application.services.plan_executor import (
|
||||
ExecuteResult,
|
||||
StrategizeResult,
|
||||
StrategyDecision,
|
||||
StreamCallback,
|
||||
)
|
||||
from cleveragents.core.exceptions import ValidationError
|
||||
from cleveragents.domain.models.core.plan import PlanInvariant
|
||||
from cleveragents.tool.builtins.changeset import ChangeSet, ChangeSetEntry
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from cleveragents.providers.registry import ProviderRegistry
|
||||
|
||||
logger = structlog.get_logger(__name__)
|
||||
|
||||
# Maximum characters of LLM response to log for debugging
|
||||
_LOG_RESPONSE_CHARS = 500
|
||||
|
||||
|
||||
def _parse_actor_name(actor_name: str) -> tuple[str, str]:
|
||||
"""Split ``provider/model`` into *(provider_type, model_id)*.
|
||||
|
||||
Examples:
|
||||
``openai/gpt-4`` → ``("openai", "gpt-4")``
|
||||
``anthropic/claude-3`` → ``("anthropic", "claude-3")``
|
||||
``gpt-4`` → ``("openai", "gpt-4")`` (default provider)
|
||||
"""
|
||||
if not actor_name:
|
||||
return ("openai", "gpt-4")
|
||||
parts = actor_name.split("/", 1)
|
||||
if len(parts) == 2:
|
||||
return (parts[0], parts[1])
|
||||
return ("openai", actor_name)
|
||||
|
||||
|
||||
class LLMStrategizeActor:
|
||||
"""Strategize actor that uses a real LLM to decompose tasks.
|
||||
|
||||
Resolves the plan's ``strategy_actor`` name to a live LLM via
|
||||
``ProviderRegistry.create_llm()`` and asks it to break the
|
||||
``definition_of_done`` into discrete implementation steps, returning
|
||||
them as a list of ``StrategyDecision`` objects.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
provider_registry: ProviderRegistry,
|
||||
lifecycle_service: Any,
|
||||
) -> None:
|
||||
if provider_registry is None:
|
||||
raise ValidationError("provider_registry must not be None")
|
||||
if lifecycle_service is None:
|
||||
raise ValidationError("lifecycle_service must not be None")
|
||||
self._registry = provider_registry
|
||||
self._lifecycle = lifecycle_service
|
||||
self._logger = logger.bind(actor="llm_strategize")
|
||||
|
||||
def execute(
|
||||
self,
|
||||
plan_id: str,
|
||||
definition_of_done: str | None,
|
||||
invariants: list[PlanInvariant] | None = None,
|
||||
stream_callback: StreamCallback | None = None,
|
||||
) -> StrategizeResult:
|
||||
"""Invoke the LLM to decompose *definition_of_done* into steps."""
|
||||
if not plan_id:
|
||||
raise ValidationError("plan_id must not be empty")
|
||||
|
||||
if stream_callback is not None:
|
||||
stream_callback(
|
||||
"strategize_started", {"plan_id": plan_id, "phase": "strategize"}
|
||||
)
|
||||
|
||||
# Resolve actor name from plan → action
|
||||
plan = self._lifecycle.get_plan(plan_id)
|
||||
action = self._lifecycle.get_action(plan.action_name)
|
||||
actor_name = action.strategy_actor or "openai/gpt-4"
|
||||
provider_type, model_id = _parse_actor_name(actor_name)
|
||||
|
||||
self._logger.info(
|
||||
"Resolving LLM for strategize",
|
||||
plan_id=plan_id,
|
||||
actor_name=actor_name,
|
||||
provider=provider_type,
|
||||
model=model_id,
|
||||
)
|
||||
|
||||
llm = self._registry.create_llm(provider_type=provider_type, model_id=model_id)
|
||||
|
||||
dod = definition_of_done or "Complete the plan objectives"
|
||||
prompt = (
|
||||
"You are an expert software architect. Analyze the following task "
|
||||
"and break it into concrete, sequential implementation steps.\n\n"
|
||||
f"Task:\n{dod}\n\n"
|
||||
"Return ONLY a numbered list (1., 2., …) of steps. "
|
||||
"Each step must be a single, actionable change. "
|
||||
"Do not include commentary before or after the list."
|
||||
)
|
||||
|
||||
from langchain_core.messages import HumanMessage
|
||||
|
||||
response = llm.invoke([HumanMessage(content=prompt)])
|
||||
content = response.content if hasattr(response, "content") else str(response)
|
||||
|
||||
self._logger.debug(
|
||||
"LLM strategize response",
|
||||
plan_id=plan_id,
|
||||
response_preview=content[:_LOG_RESPONSE_CHARS],
|
||||
)
|
||||
|
||||
decisions = self._parse_decisions(content)
|
||||
|
||||
root_id = str(ULID())
|
||||
result_decisions: list[StrategyDecision] = []
|
||||
for idx, step_text in enumerate(decisions):
|
||||
result_decisions.append(
|
||||
StrategyDecision(
|
||||
decision_id=str(ULID()) if idx > 0 else root_id,
|
||||
step_text=step_text,
|
||||
sequence=idx,
|
||||
parent_id=root_id if idx > 0 else None,
|
||||
)
|
||||
)
|
||||
|
||||
invariant_records: list[dict[str, Any]] = []
|
||||
for inv in invariants or []:
|
||||
invariant_records.append(
|
||||
{
|
||||
"text": inv.text,
|
||||
"source": inv.source.value,
|
||||
"enforced": True,
|
||||
"enforcement_note": "llm: accepted without reconciliation",
|
||||
}
|
||||
)
|
||||
|
||||
if stream_callback is not None:
|
||||
stream_callback(
|
||||
"strategize_decisions",
|
||||
{
|
||||
"plan_id": plan_id,
|
||||
"decision_count": len(result_decisions),
|
||||
"root_id": root_id,
|
||||
},
|
||||
)
|
||||
|
||||
result = StrategizeResult(
|
||||
decision_root_id=root_id,
|
||||
decisions=result_decisions,
|
||||
invariant_records=invariant_records,
|
||||
)
|
||||
|
||||
if stream_callback is not None:
|
||||
stream_callback(
|
||||
"strategize_complete",
|
||||
{"plan_id": plan_id, "decision_count": len(result_decisions)},
|
||||
)
|
||||
|
||||
self._logger.info(
|
||||
"LLM strategize completed",
|
||||
plan_id=plan_id,
|
||||
decision_count=len(result_decisions),
|
||||
)
|
||||
return result
|
||||
|
||||
@staticmethod
|
||||
def _parse_decisions(llm_output: str) -> list[str]:
|
||||
"""Parse numbered steps from LLM response text."""
|
||||
lines = llm_output.strip().splitlines()
|
||||
steps: list[str] = []
|
||||
for line in lines:
|
||||
cleaned = line.strip()
|
||||
if not cleaned:
|
||||
continue
|
||||
# Strip numbered prefix (e.g. "1.", "2)", "1 -")
|
||||
match = re.match(r"^\d+[\.\)\-\:]\s*", cleaned)
|
||||
if match:
|
||||
cleaned = cleaned[match.end() :].strip()
|
||||
# Strip bullet prefixes
|
||||
for prefix in ("-", "*", "•"):
|
||||
if cleaned.startswith(prefix):
|
||||
cleaned = cleaned[len(prefix) :].strip()
|
||||
break
|
||||
if cleaned:
|
||||
steps.append(cleaned)
|
||||
return steps if steps else ["Complete the plan objectives"]
|
||||
|
||||
|
||||
class LLMExecuteActor:
|
||||
"""Execute actor that uses a real LLM to generate code changes.
|
||||
|
||||
Resolves the plan's ``execution_actor`` name to a live LLM via
|
||||
``ProviderRegistry.create_llm()`` and asks it to produce file
|
||||
changes for the strategy decisions, returning a ``ChangeSet``.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
provider_registry: ProviderRegistry,
|
||||
lifecycle_service: Any,
|
||||
) -> None:
|
||||
if provider_registry is None:
|
||||
raise ValidationError("provider_registry must not be None")
|
||||
if lifecycle_service is None:
|
||||
raise ValidationError("lifecycle_service must not be None")
|
||||
self._registry = provider_registry
|
||||
self._lifecycle = lifecycle_service
|
||||
self._logger = logger.bind(actor="llm_execute")
|
||||
|
||||
def execute(
|
||||
self,
|
||||
plan_id: str,
|
||||
decisions: list[StrategyDecision],
|
||||
tool_runner: Any | None = None,
|
||||
sandbox_root: str | None = None,
|
||||
stream_callback: StreamCallback | None = None,
|
||||
*,
|
||||
read_only: bool = False,
|
||||
) -> ExecuteResult:
|
||||
"""Invoke the LLM to generate file changes for the given decisions."""
|
||||
if not plan_id:
|
||||
raise ValidationError("plan_id must not be empty")
|
||||
|
||||
if stream_callback is not None:
|
||||
stream_callback("execute_started", {"plan_id": plan_id, "phase": "execute"})
|
||||
|
||||
# Resolve actor name from plan → action
|
||||
plan = self._lifecycle.get_plan(plan_id)
|
||||
action = self._lifecycle.get_action(plan.action_name)
|
||||
actor_name = action.execution_actor or "openai/gpt-4"
|
||||
provider_type, model_id = _parse_actor_name(actor_name)
|
||||
|
||||
self._logger.info(
|
||||
"Resolving LLM for execute",
|
||||
plan_id=plan_id,
|
||||
actor_name=actor_name,
|
||||
provider=provider_type,
|
||||
model=model_id,
|
||||
)
|
||||
|
||||
llm = self._registry.create_llm(provider_type=provider_type, model_id=model_id)
|
||||
|
||||
# Build a prompt summarising the decisions
|
||||
steps_text = "\n".join(f"{d.sequence + 1}. {d.step_text}" for d in decisions)
|
||||
prompt = (
|
||||
"You are an expert software engineer. "
|
||||
"Implement the following steps by producing file changes.\n\n"
|
||||
f"Steps:\n{steps_text}\n\n"
|
||||
"For each file you create or modify, output a block:\n"
|
||||
"FILE: <path>\n```\n<full file content>\n```\n\n"
|
||||
"Only output file blocks. Do not add commentary."
|
||||
)
|
||||
|
||||
from langchain_core.messages import HumanMessage
|
||||
|
||||
response = llm.invoke([HumanMessage(content=prompt)])
|
||||
content = response.content if hasattr(response, "content") else str(response)
|
||||
|
||||
self._logger.debug(
|
||||
"LLM execute response",
|
||||
plan_id=plan_id,
|
||||
response_preview=content[:_LOG_RESPONSE_CHARS],
|
||||
)
|
||||
|
||||
# Stream per-decision progress
|
||||
for decision in decisions:
|
||||
if stream_callback is not None:
|
||||
stream_callback(
|
||||
"execute_step",
|
||||
{
|
||||
"plan_id": plan_id,
|
||||
"decision_id": decision.decision_id,
|
||||
"step": decision.step_text,
|
||||
"sequence": decision.sequence,
|
||||
},
|
||||
)
|
||||
|
||||
# Parse LLM output into changeset entries
|
||||
entries = self._parse_file_blocks(content, plan_id)
|
||||
changeset_id = str(ULID())
|
||||
changeset = ChangeSet(plan_id=plan_id, entries=entries)
|
||||
|
||||
# Write generated files to sandbox when available
|
||||
if sandbox_root is not None and not read_only:
|
||||
self._write_to_sandbox(entries, sandbox_root, content)
|
||||
|
||||
sandbox_refs: list[str] = []
|
||||
if sandbox_root is not None:
|
||||
sandbox_refs.append(sandbox_root)
|
||||
|
||||
if stream_callback is not None:
|
||||
stream_callback(
|
||||
"execute_complete",
|
||||
{
|
||||
"plan_id": plan_id,
|
||||
"changeset_id": changeset_id,
|
||||
"tool_calls_count": len(entries),
|
||||
},
|
||||
)
|
||||
|
||||
self._logger.info(
|
||||
"LLM execute completed",
|
||||
plan_id=plan_id,
|
||||
changeset_id=changeset_id,
|
||||
entry_count=len(entries),
|
||||
)
|
||||
|
||||
return ExecuteResult(
|
||||
changeset_id=changeset_id,
|
||||
changeset=changeset,
|
||||
tool_calls_count=len(entries),
|
||||
sandbox_refs=sandbox_refs,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _parse_file_blocks(llm_output: str, plan_id: str) -> list[ChangeSetEntry]:
|
||||
"""Extract ``FILE: <path>`` + fenced code blocks from LLM output."""
|
||||
entries: list[ChangeSetEntry] = []
|
||||
# Pattern: FILE: <path> followed by a fenced code block
|
||||
pattern = re.compile(
|
||||
r"FILE:\s*(.+?)\s*\n```[^\n]*\n(.*?)```",
|
||||
re.DOTALL,
|
||||
)
|
||||
for match in pattern.finditer(llm_output):
|
||||
path = match.group(1).strip()
|
||||
entries.append(
|
||||
ChangeSetEntry(
|
||||
operation="create",
|
||||
path=path,
|
||||
resource_id=plan_id,
|
||||
tool_name="llm_execute",
|
||||
timestamp=datetime.now(tz=UTC),
|
||||
metadata={"source": "llm", "plan_id": plan_id},
|
||||
)
|
||||
)
|
||||
return entries
|
||||
|
||||
@staticmethod
|
||||
def _write_to_sandbox(
|
||||
entries: list[ChangeSetEntry],
|
||||
sandbox_root: str,
|
||||
llm_output: str,
|
||||
) -> None:
|
||||
"""Write generated file contents to the sandbox directory."""
|
||||
import os
|
||||
|
||||
pattern = re.compile(
|
||||
r"FILE:\s*(.+?)\s*\n```[^\n]*\n(.*?)```",
|
||||
re.DOTALL,
|
||||
)
|
||||
for match in pattern.finditer(llm_output):
|
||||
path = match.group(1).strip()
|
||||
content = match.group(2)
|
||||
full_path = os.path.join(sandbox_root, path)
|
||||
os.makedirs(os.path.dirname(full_path), exist_ok=True)
|
||||
try:
|
||||
with open(full_path, "w") as fh:
|
||||
fh.write(content)
|
||||
except OSError:
|
||||
logger.warning(
|
||||
"Failed to write generated file to sandbox",
|
||||
path=full_path,
|
||||
exc_info=True,
|
||||
)
|
||||
@@ -272,6 +272,8 @@ class PlanExecutor:
|
||||
checkpoint_manager: CheckpointManager | None = None,
|
||||
guardrail_service: AutonomyGuardrailService | None = None,
|
||||
metrics_emitter: MetricsEmitter | None = None,
|
||||
strategize_actor: Any | None = None,
|
||||
execute_actor: Any | None = None,
|
||||
) -> None:
|
||||
"""Initialize the plan executor.
|
||||
|
||||
@@ -289,6 +291,12 @@ class PlanExecutor:
|
||||
enforcing step limits, budgets, and wall-clock time.
|
||||
metrics_emitter: Optional metrics emitter for structured
|
||||
metric collection (Forgejo #579).
|
||||
strategize_actor: Optional custom strategize actor. When
|
||||
``None``, the default ``StrategizeStubActor`` is used.
|
||||
Pass an ``LLMStrategizeActor`` for real LLM execution.
|
||||
execute_actor: Optional custom execute actor. When ``None``,
|
||||
the default ``ExecuteStubActor`` is used. Pass an
|
||||
``LLMExecuteActor`` for real LLM execution.
|
||||
"""
|
||||
if lifecycle_service is None:
|
||||
raise ValidationError("lifecycle_service must not be None")
|
||||
@@ -300,8 +308,8 @@ class PlanExecutor:
|
||||
self._checkpoint_manager = checkpoint_manager
|
||||
self._guardrail_service = guardrail_service
|
||||
self._metrics_emitter = metrics_emitter
|
||||
self._strategize_actor = StrategizeStubActor()
|
||||
self._execute_actor = ExecuteStubActor()
|
||||
self._strategize_actor = strategize_actor or StrategizeStubActor()
|
||||
self._execute_actor = execute_actor or ExecuteStubActor()
|
||||
self._logger = logger.bind(service="plan_executor")
|
||||
|
||||
def _try_emit_metric(
|
||||
|
||||
@@ -213,6 +213,7 @@ def create(
|
||||
exists=False,
|
||||
),
|
||||
],
|
||||
fmt: Annotated[str, typer.Option("--format", "-f", help=_FORMAT_HELP)] = "rich",
|
||||
) -> None:
|
||||
"""Create a new action from a YAML configuration file.
|
||||
|
||||
@@ -222,6 +223,7 @@ def create(
|
||||
|
||||
Examples:
|
||||
agents action create --config ./actions/code-coverage.yaml
|
||||
agents action create --config ./actions/code-coverage.yaml --format json
|
||||
"""
|
||||
try:
|
||||
# Load and validate config via ActionConfigSchema
|
||||
@@ -247,7 +249,7 @@ def create(
|
||||
tags=action.tags,
|
||||
)
|
||||
|
||||
_print_action(action, title="Action Created")
|
||||
_print_action(action, title="Action Created", fmt=fmt)
|
||||
|
||||
except FileNotFoundError as e:
|
||||
console.print(f"[red]Config file error:[/red] {e}")
|
||||
|
||||
@@ -1201,15 +1201,39 @@ def _get_lifecycle_service():
|
||||
return container.plan_lifecycle_service()
|
||||
|
||||
|
||||
def _get_plan_executor():
|
||||
"""Get a PlanExecutor wired with the container's lifecycle service.
|
||||
def _get_plan_executor() -> Any:
|
||||
"""Build a ``PlanExecutor`` wired with real LLM actors.
|
||||
|
||||
Returns a PlanExecutor that uses stub actors for M1 phase processing.
|
||||
Resolves the ``ProviderRegistry`` and ``PlanLifecycleService`` from
|
||||
the DI container and constructs ``LLMStrategizeActor`` /
|
||||
``LLMExecuteActor`` so that ``plan execute`` invocations drive real
|
||||
LLM calls instead of the local-only stub actors.
|
||||
"""
|
||||
from cleveragents.application.container import get_container
|
||||
from cleveragents.application.services.llm_actors import (
|
||||
LLMExecuteActor,
|
||||
LLMStrategizeActor,
|
||||
)
|
||||
from cleveragents.application.services.plan_executor import PlanExecutor
|
||||
|
||||
lifecycle = _get_lifecycle_service()
|
||||
return PlanExecutor(lifecycle_service=lifecycle)
|
||||
container = get_container()
|
||||
registry = container.provider_registry()
|
||||
lifecycle_service = _get_lifecycle_service()
|
||||
|
||||
strategize_actor = LLMStrategizeActor(
|
||||
provider_registry=registry,
|
||||
lifecycle_service=lifecycle_service,
|
||||
)
|
||||
execute_actor = LLMExecuteActor(
|
||||
provider_registry=registry,
|
||||
lifecycle_service=lifecycle_service,
|
||||
)
|
||||
|
||||
return PlanExecutor(
|
||||
lifecycle_service=lifecycle_service,
|
||||
strategize_actor=strategize_actor,
|
||||
execute_actor=execute_actor,
|
||||
)
|
||||
|
||||
|
||||
def _print_lifecycle_plan(plan: Any, title: str = "Plan") -> None:
|
||||
|
||||
Reference in New Issue
Block a user