UAT: `response_format` field in actor config not wired to LLM provider calls in `LLMStrategizeActor` and `LLMExecuteActor` #3824

New issue

Open

opened 2026-04-06 06:47:23 +00:00 by freemo · 0 comments

freemo commented

2026-04-06 06:47:23 +00:00

Owner

Background and Context

The ActorConfigSchema defines a response_format field (src/cleveragents/actor/schema.py line 720) that is validated and parsed from actor YAML configuration but is never passed to the LLM when invoking it. Both LLMStrategizeActor.execute() and LLMExecuteActor.execute() in src/cleveragents/application/services/llm_actors.py contain explicit TODO comments acknowledging this gap.

The specification (line 20424) defines config.response_format as: "JSON schema for structured output from the LLM. When set, the model is constrained to produce output matching this schema." The spec also states (line 31665): "Constrains model output to match the schema." This is particularly critical for estimation actors which must produce structured JSON output.

What Was Tested

The response_format field in ActorConfigSchema and its propagation to LLM provider calls in LLMStrategizeActor and LLMExecuteActor.

Expected Behavior (from spec)

When an actor YAML defines response_format: {type: "json_schema", ...}, the LLM invocation in both LLMStrategizeActor.execute() and LLMExecuteActor.execute() must pass this schema to the provider call so that the model is constrained to produce output matching the schema.

Actual Behavior

The response_format field is parsed and validated but silently ignored during execution. Both actor execute methods contain explicit TODO comments:

# TODO(#650): Wire actor-configured response_format into provider calls
# when structured-output enforcement is implemented in runtime execution.
response = llm.invoke([HumanMessage(content=prompt)])

The response_format from the actor's YAML configuration is never retrieved from the action/actor config and never passed to llm.invoke(). This means:

Estimation actors that define response_format for structured JSON output will receive unstructured text responses
Any actor relying on response_format for output schema enforcement will silently receive non-conforming output
The spec requirement for structured output enforcement is not implemented

Code Locations

src/cleveragents/application/services/llm_actors.py lines 149–150 — LLMStrategizeActor.execute() TODO
src/cleveragents/application/services/llm_actors.py lines 351–352 — LLMExecuteActor.execute() TODO
src/cleveragents/actor/schema.py lines 720–818 — response_format field definition and validation
src/cleveragents/actor/role_validation.py lines 59–79 — role validation checks for response_format but it is never wired

Steps to Reproduce

Create an actor YAML with response_format: {type: "json_schema", ...}
Run a plan that uses this actor for strategize or execute
Observe that the LLM response is unstructured text, not constrained by the schema
The response_format field is parsed and validated but silently ignored during execution

Impact

This affects all actors that define response_format for structured output, particularly estimation actors. The spec explicitly requires this feature for the estimation actor role. Without it, estimation actors cannot reliably produce structured JSON output for cost/risk/duration estimation.

Backlog note: This issue was discovered during autonomous operation
on milestone v3.6.0. It does not block milestone completion and has been
placed in the backlog for human review and future milestone assignment.

Metadata

Branch: fix/response-format-wiring-llm-actors
Commit Message: fix(llm-actors): wire actor config response_format into LLM provider invocations
Milestone: Backlog
Parent Epic: #392

Subtasks

Retrieve response_format from actor config in LLMStrategizeActor.execute() (llm_actors.py line ~149)
Retrieve response_format from actor config in LLMExecuteActor.execute() (llm_actors.py line ~351)
Pass response_format to llm.invoke() (or equivalent structured-output binding) in both actors
Verify that when response_format is None or unset, behaviour is unchanged (no regression)
Remove the TODO comments once wired
Tests (Behave): Add BDD scenarios covering response_format propagation for both strategize and execute actors
Tests (Behave): Add scenario verifying unstructured fallback when response_format is absent
Tests (Robot): Add integration test confirming structured output is enforced end-to-end for an estimation actor
Verify coverage >= 97% via nox -s coverage_report
Run nox (all default sessions), fix any errors

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
response_format from actor YAML config is correctly retrieved and passed to the LLM provider in both LLMStrategizeActor and LLMExecuteActor.
Actors with response_format defined produce structured output conforming to the schema.
Actors without response_format continue to work as before (no regression).
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
All nox stages pass
Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-uat-tester

## Background and Context The `ActorConfigSchema` defines a `response_format` field (`src/cleveragents/actor/schema.py` line 720) that is validated and parsed from actor YAML configuration but is **never passed to the LLM** when invoking it. Both `LLMStrategizeActor.execute()` and `LLMExecuteActor.execute()` in `src/cleveragents/application/services/llm_actors.py` contain explicit TODO comments acknowledging this gap. The specification (line 20424) defines `config.response_format` as: *"JSON schema for structured output from the LLM. When set, the model is constrained to produce output matching this schema."* The spec also states (line 31665): *"Constrains model output to match the schema."* This is particularly critical for estimation actors which must produce structured JSON output. ## What Was Tested The `response_format` field in `ActorConfigSchema` and its propagation to LLM provider calls in `LLMStrategizeActor` and `LLMExecuteActor`. ## Expected Behavior (from spec) When an actor YAML defines `response_format: {type: "json_schema", ...}`, the LLM invocation in both `LLMStrategizeActor.execute()` and `LLMExecuteActor.execute()` must pass this schema to the provider call so that the model is constrained to produce output matching the schema. ## Actual Behavior The `response_format` field is parsed and validated but silently ignored during execution. Both actor execute methods contain explicit TODO comments: ```python # TODO(#650): Wire actor-configured response_format into provider calls # when structured-output enforcement is implemented in runtime execution. response = llm.invoke([HumanMessage(content=prompt)]) ``` The `response_format` from the actor's YAML configuration is never retrieved from the action/actor config and never passed to `llm.invoke()`. This means: 1. Estimation actors that define `response_format` for structured JSON output will receive unstructured text responses 2. Any actor relying on `response_format` for output schema enforcement will silently receive non-conforming output 3. The spec requirement for structured output enforcement is not implemented ## Code Locations - `src/cleveragents/application/services/llm_actors.py` lines 149–150 — `LLMStrategizeActor.execute()` TODO - `src/cleveragents/application/services/llm_actors.py` lines 351–352 — `LLMExecuteActor.execute()` TODO - `src/cleveragents/actor/schema.py` lines 720–818 — `response_format` field definition and validation - `src/cleveragents/actor/role_validation.py` lines 59–79 — role validation checks for `response_format` but it is never wired ## Steps to Reproduce 1. Create an actor YAML with `response_format: {type: "json_schema", ...}` 2. Run a plan that uses this actor for strategize or execute 3. Observe that the LLM response is unstructured text, not constrained by the schema 4. The `response_format` field is parsed and validated but silently ignored during execution ## Impact This affects all actors that define `response_format` for structured output, particularly estimation actors. The spec explicitly requires this feature for the estimation actor role. Without it, estimation actors cannot reliably produce structured JSON output for cost/risk/duration estimation. --- > **Backlog note:** This issue was discovered during autonomous operation > on milestone v3.6.0. It does not block milestone completion and has been > placed in the backlog for human review and future milestone assignment. --- ## Metadata - **Branch**: `fix/response-format-wiring-llm-actors` - **Commit Message**: `fix(llm-actors): wire actor config response_format into LLM provider invocations` - **Milestone**: Backlog - **Parent Epic**: #392 ## Subtasks - [ ] Retrieve `response_format` from actor config in `LLMStrategizeActor.execute()` (`llm_actors.py` line ~149) - [ ] Retrieve `response_format` from actor config in `LLMExecuteActor.execute()` (`llm_actors.py` line ~351) - [ ] Pass `response_format` to `llm.invoke()` (or equivalent structured-output binding) in both actors - [ ] Verify that when `response_format` is `None` or unset, behaviour is unchanged (no regression) - [ ] Remove the TODO comments once wired - [ ] Tests (Behave): Add BDD scenarios covering `response_format` propagation for both strategize and execute actors - [ ] Tests (Behave): Add scenario verifying unstructured fallback when `response_format` is absent - [ ] Tests (Robot): Add integration test confirming structured output is enforced end-to-end for an estimation actor - [ ] Verify coverage >= 97% via `nox -s coverage_report` - [ ] Run `nox` (all default sessions), fix any errors ## Definition of Done This issue is complete when: - [ ] All subtasks above are completed and checked off. - [ ] `response_format` from actor YAML config is correctly retrieved and passed to the LLM provider in both `LLMStrategizeActor` and `LLMExecuteActor`. - [ ] Actors with `response_format` defined produce structured output conforming to the schema. - [ ] Actors without `response_format` continue to work as before (no regression). - [ ] A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - [ ] The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - [ ] The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - [ ] All nox stages pass - [ ] Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-uat-tester

freemo added a new dependency

2026-04-06 06:47:43 +00:00

#392 Epic: Actor YAML & Compiler

freemo added the

Points

label

2026-04-06 06:50:02 +00:00

freemo referenced this issue

2026-04-06 06:55:25 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775

freemo referenced this issue

2026-04-06 07:04:26 +00:00