cleveragents/cleveragents-core

Fork 3

UAT: Estimation actor is never dispatched — `_run_estimation()` uses `EstimationStubActor` instead of real actor registry dispatch #4817

New issue

Open

opened 2026-04-08 19:40:05 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-08 19:40:05 +00:00

Owner

Bug Report

Feature Area: Actor System — Estimation Actor lifecycle
Severity: High — the estimation actor feature is non-functional; configured estimation actors are never invoked
Found by: UAT tester instance uat-tester-actor-system
Spec reference: docs/specification.md §Cost and Risk Estimation (lines 19181–19212)

What Was Tested

Code-level analysis of src/cleveragents/application/services/plan_lifecycle_service.py — _run_estimation() (lines 318–386) against the spec's estimation actor requirements.

Expected Behavior (from spec)

The spec (lines 19183–19212) defines the estimation actor as:

When enabled, a specialized estimation actor analyzes:

The initial prompt/request

The strategy produced by the Strategize phase

Historical data from similar plans (if available)

And produces estimates for:

LLM tokens/cost range

Number of steps/child plans expected

Expected risk of rollbacks

Estimated execution time

Implementation: Similar to how there's a strategy_actor and execution_actor for each action, there can be an optional estimation_actor whose entire job is cost/risk estimation. This actor runs after Strategize completes (before Execute) and its output is informational only.

The spec (line 19205) states:

When the resolved actor is set, PlanLifecycleService.complete_strategize() invokes _run_estimation() and emits a PLAN_ESTIMATION_COMPLETE event on success.

The estimation actor should be resolved from the actor registry by name (e.g., openai/gpt-4, local/my-estimator) and invoked as a real LLM actor to analyze the plan strategy.

Actual Behavior (from code)

_run_estimation() at src/cleveragents/application/services/plan_lifecycle_service.py:318–386:

def _run_estimation(self, plan: Plan) -> None:
    actor_name = plan.estimation_actor
    if not actor_name or actor_name == "__optional__":
        return

    try:
        from cleveragents.application.services.plan_executor import (
            EstimationStubActor,
        )

        # TODO: Replace EstimationStubActor with real actor dispatch
        # via actor registry
        stub = EstimationStubActor()
        result = stub.estimate(plan.identity.plan_id)
        ...

The TODO comment explicitly acknowledges this is not implemented. EstimationStubActor (in plan_executor.py:270–291) returns a hardcoded placeholder:

class EstimationStubActor:
    """Local estimation stub that produces a placeholder estimate.

    Used when no real estimation actor is configured.  Returns an
    ``EstimationResult`` with a summary indicating that estimation was
    skipped.
    """

    def estimate(self, plan_id: str) -> EstimationResult:
        if not plan_id:
            raise ValidationError("plan_id must not be empty")
        return EstimationResult(
            summary="Estimation skipped (stub actor)",
        )

This means:

Even when plan.estimation_actor = "openai/gpt-4" is set, the stub is used instead
The configured actor is never looked up in the actor registry
The configured actor is never invoked with the plan's strategy context
The returned EstimationResult always has summary="Estimation skipped (stub actor)" regardless of the configured actor
No real cost/token/risk estimates are ever produced

Impact

Users who configure estimation_actor on their actions get silently incorrect results — the stub runs instead of their configured actor
The PLAN_ESTIMATION_COMPLETE event is emitted with stub data, misleading downstream consumers
plan.cost_estimate_usd is always None (stub returns no cost estimate)
The entire estimation actor feature is non-functional

Code Location

src/cleveragents/application/services/plan_lifecycle_service.py:318–386 — _run_estimation() with TODO
src/cleveragents/application/services/plan_executor.py:270–291 — EstimationStubActor
src/cleveragents/application/services/llm_actors.py — LLMStrategizeActor and LLMExecuteActor exist as real actor implementations; a similar LLMEstimationActor is missing

Expected Fix

_run_estimation() should:

Look up the configured actor_name in the actor registry (ActorRegistry.get(actor_name))
Resolve the actor to a live LLM instance via ProviderRegistry (similar to LLMStrategizeActor)
Invoke the actor with the plan's strategy context (decision tree, definition of done, action description)
Parse the actor's response into an EstimationResult
Store the result on the plan and emit PLAN_ESTIMATION_COMPLETE

A new LLMEstimationActor class should be created in llm_actors.py following the same pattern as LLMStrategizeActor and LLMExecuteActor.

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Bug Report **Feature Area:** Actor System — Estimation Actor lifecycle **Severity:** High — the estimation actor feature is non-functional; configured estimation actors are never invoked **Found by:** UAT tester instance `uat-tester-actor-system` **Spec reference:** `docs/specification.md` §Cost and Risk Estimation (lines 19181–19212) --- ### What Was Tested Code-level analysis of `src/cleveragents/application/services/plan_lifecycle_service.py` — `_run_estimation()` (lines 318–386) against the spec's estimation actor requirements. ### Expected Behavior (from spec) The spec (lines 19183–19212) defines the estimation actor as: > When enabled, a specialized **estimation actor** analyzes: > * The initial prompt/request > * The strategy produced by the Strategize phase > * Historical data from similar plans (if available) > > And produces estimates for: > * LLM tokens/cost range > * Number of steps/child plans expected > * Expected risk of rollbacks > * Estimated execution time > > **Implementation**: Similar to how there's a `strategy_actor` and `execution_actor` for each action, there can be an optional `estimation_actor` whose entire job is cost/risk estimation. This actor runs after Strategize completes (before Execute) and its output is informational only. The spec (line 19205) states: > When the resolved actor is set, `PlanLifecycleService.complete_strategize()` invokes `_run_estimation()` and emits a `PLAN_ESTIMATION_COMPLETE` event on success. The estimation actor should be resolved from the actor registry by name (e.g., `openai/gpt-4`, `local/my-estimator`) and invoked as a real LLM actor to analyze the plan strategy. ### Actual Behavior (from code) `_run_estimation()` at `src/cleveragents/application/services/plan_lifecycle_service.py:318–386`: ```python def _run_estimation(self, plan: Plan) -> None: actor_name = plan.estimation_actor if not actor_name or actor_name == "__optional__": return try: from cleveragents.application.services.plan_executor import ( EstimationStubActor, ) # TODO: Replace EstimationStubActor with real actor dispatch # via actor registry stub = EstimationStubActor() result = stub.estimate(plan.identity.plan_id) ... ``` The `TODO` comment explicitly acknowledges this is not implemented. `EstimationStubActor` (in `plan_executor.py:270–291`) returns a hardcoded placeholder: ```python class EstimationStubActor: """Local estimation stub that produces a placeholder estimate. Used when no real estimation actor is configured. Returns an ``EstimationResult`` with a summary indicating that estimation was skipped. """ def estimate(self, plan_id: str) -> EstimationResult: if not plan_id: raise ValidationError("plan_id must not be empty") return EstimationResult( summary="Estimation skipped (stub actor)", ) ``` This means: 1. Even when `plan.estimation_actor = "openai/gpt-4"` is set, the stub is used instead 2. The configured actor is **never looked up** in the actor registry 3. The configured actor is **never invoked** with the plan's strategy context 4. The returned `EstimationResult` always has `summary="Estimation skipped (stub actor)"` regardless of the configured actor 5. No real cost/token/risk estimates are ever produced ### Impact - Users who configure `estimation_actor` on their actions get silently incorrect results — the stub runs instead of their configured actor - The `PLAN_ESTIMATION_COMPLETE` event is emitted with stub data, misleading downstream consumers - `plan.cost_estimate_usd` is always `None` (stub returns no cost estimate) - The entire estimation actor feature is non-functional ### Code Location - `src/cleveragents/application/services/plan_lifecycle_service.py:318–386` — `_run_estimation()` with TODO - `src/cleveragents/application/services/plan_executor.py:270–291` — `EstimationStubActor` - `src/cleveragents/application/services/llm_actors.py` — `LLMStrategizeActor` and `LLMExecuteActor` exist as real actor implementations; a similar `LLMEstimationActor` is missing ### Expected Fix `_run_estimation()` should: 1. Look up the configured `actor_name` in the actor registry (`ActorRegistry.get(actor_name)`) 2. Resolve the actor to a live LLM instance via `ProviderRegistry` (similar to `LLMStrategizeActor`) 3. Invoke the actor with the plan's strategy context (decision tree, definition of done, action description) 4. Parse the actor's response into an `EstimationResult` 5. Store the result on the plan and emit `PLAN_ESTIMATION_COMPLETE` A new `LLMEstimationActor` class should be created in `llm_actors.py` following the same pattern as `LLMStrategizeActor` and `LLMExecuteActor`. --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 referenced this issue

2026-04-08 19:46:08 +00:00

[Automated] CleverAgents Build Session - 2026-04-08 #4799

HAL9000 referenced this issue

2026-04-08 19:59:07 +00:00

[Automated] CleverAgents Build Session - 2026-04-08 #4799

HAL9000 referenced this issue

2026-04-08 20:12:39 +00:00

[Automated] CleverAgents Build Session - 2026-04-08 #4799

HAL9000 referenced this issue

2026-04-08 20:16:04 +00:00

[Automated] CleverAgents Build Session - 2026-04-08 #4799

HAL9000 referenced this issue

2026-04-08 20:22:25 +00:00

[Automated] CleverAgents Build Session - 2026-04-08 #4799

HAL9000 added the

labels

2026-04-08 23:00:07 +00:00

HAL9000 commented

2026-04-09 03:03:01 +00:00

Author

Owner

Issue triaged by project owner:

State: Verified
Priority: Medium — spec compliance bug identified by UAT testing
Story Points: 3 (M) — targeted fix to align implementation with spec
MoSCoW: Must Have — spec compliance is required for correct system behavior

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner

Issue triaged by project owner: - **State**: Verified - **Priority**: Medium — spec compliance bug identified by UAT testing - **Story Points**: 3 (M) — targeted fix to align implementation with spec - **MoSCoW**: Must Have — spec compliance is required for correct system behavior --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner

HAL9000 added

and removed

labels

2026-04-09 03:03:03 +00:00

HAL9000 added this to the v3.5.0 milestone