UAT: `role_validation.py` only validates `ESTIMATION` role — `STRATEGY`, `EXECUTION`, `INVARIANT_RECONCILIATION`, and `REVIEW` roles have no validation #3723

New issue

Open

opened 2026-04-05 22:18:34 +00:00 by freemo · 0 comments

freemo commented

2026-04-05 22:18:34 +00:00

Owner

Metadata

Branch: fix/actor-role-validation-all-roles
Commit Message: fix(actor): implement role-aware validation for all specialized actor roles in role_validation.py
Milestone: None (Backlog)
Parent Epic: #230

Background and Context

Per docs/specification.md §Role Validation:

"The system defines several specialized actor roles that are critical for the plan lifecycle. The ActorService must validate that any actor assigned to these roles meets the required interface."

Strategy Actor: Must be capable of generating a decision tree from a plan's definition.

Execution Actor: Must be able to execute tool calls and manage sandbox operations.

Estimation Actor: Must provide cost and duration estimates for a given plan.

Invariant Reconciliation Actor: Must be able to reconcile and enforce invariants.

Review Actor: Must be able to review and approve plan steps.

The RoleHint enum in src/cleveragents/actor/role_validation.py correctly defines all five roles:

class RoleHint(StrEnum):
    STRATEGY = "strategy"
    EXECUTION = "execution"
    ESTIMATION = "estimation"
    INVARIANT_RECONCILIATION = "invariant_reconciliation"
    REVIEW = "review"

However, the actor_role_warnings() function only validates the ESTIMATION role. All other roles return an empty warnings list immediately:

def actor_role_warnings(config: object) -> list[str]:
    ...
    if role_hint != RoleHint.ESTIMATION:
        return []  # BUG: all non-ESTIMATION roles bypass validation entirely
    ...

Current Behavior

actor_role_warnings() in src/cleveragents/actor/role_validation.py:

RoleHint.ESTIMATION: Validates that response_format is defined and context_view is "strategist".
RoleHint.STRATEGY: No validation — returns [] immediately.
RoleHint.EXECUTION: No validation — returns [] immediately.
RoleHint.INVARIANT_RECONCILIATION: No validation — returns [] immediately.
RoleHint.REVIEW: No validation — returns [] immediately.

Expected Behavior

Per the specification, each specialized role must have appropriate validation:

STRATEGY: Should validate that the actor is capable of generating a decision tree (e.g., has context_view set to "strategist", has a system_prompt defined, and is of type LLM or GRAPH).
EXECUTION: Should validate that the actor can execute tool calls (e.g., has tools defined or is of type TOOL or GRAPH, and has context_view set to "executor").
ESTIMATION: Already validated — requires response_format and context_view == "strategist".
INVARIANT_RECONCILIATION: Should validate that the actor can reconcile invariants (e.g., has response_format for structured output and appropriate context_view).
REVIEW: Should validate that the actor can review and approve plan steps (e.g., has context_view set to "reviewer").

Impact

Actors assigned to STRATEGY, EXECUTION, INVARIANT_RECONCILIATION, or REVIEW roles receive no compatibility warnings, even if they are misconfigured for those roles.
The plan lifecycle can silently use incompatible actors for critical roles, leading to runtime failures or incorrect behavior.
The spec's role validation contract is only partially implemented.

Code Location

src/cleveragents/actor/role_validation.py — actor_role_warnings() function (approximately lines 40–65)

Steps to Reproduce

Create an actor YAML with role_hint: strategy but no system_prompt and context_view: executor.
Call actor_role_warnings(config).
Expected: Warnings about missing system_prompt and incorrect context_view for a strategy actor.
Actual: Empty list returned — no warnings.

Subtasks

Write TDD failing Behave scenarios for each role (@tdd_expected_fail) asserting that actor_role_warnings() returns appropriate warnings for misconfigured STRATEGY, EXECUTION, INVARIANT_RECONCILIATION, and REVIEW actors
Implement validation logic in actor_role_warnings() for RoleHint.STRATEGY:
- Warn if context_view is not "strategist"
- Warn if system_prompt is not defined (for LLM/GRAPH types)
Implement validation logic for RoleHint.EXECUTION:
- Warn if context_view is not "executor"
- Warn if no tools are defined and actor type is LLM (execution actors need tools)
Implement validation logic for RoleHint.INVARIANT_RECONCILIATION:
- Warn if response_format is not defined (structured output required for reconciliation)
- Warn if context_view is not "strategist" or "full"
Implement validation logic for RoleHint.REVIEW:
- Warn if context_view is not "reviewer"
Remove @tdd_expected_fail tags once fixes are in place
Verify nox -e typecheck passes
Verify nox -e unit_tests passes with coverage >= 97%
Run nox (all default sessions), fix any errors

Definition of Done

actor_role_warnings() returns appropriate warnings for misconfigured actors of all five role types
RoleHint.STRATEGY, RoleHint.EXECUTION, RoleHint.INVARIANT_RECONCILIATION, and RoleHint.REVIEW all have validation logic
Behave BDD scenarios cover all five role types
All nox stages pass
Coverage >= 97%

Backlog note: This issue was discovered during autonomous UAT testing of the Actor System feature area. It does not block milestone completion and has been placed in the backlog for human review and future milestone assignment.

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-new-issue-creator

## Metadata - **Branch**: `fix/actor-role-validation-all-roles` - **Commit Message**: `fix(actor): implement role-aware validation for all specialized actor roles in role_validation.py` - **Milestone**: None (Backlog) - **Parent Epic**: #230 ## Background and Context Per `docs/specification.md` §Role Validation: > "The system defines several specialized actor roles that are critical for the plan lifecycle. The `ActorService` must validate that any actor assigned to these roles meets the required interface." > > - **Strategy Actor**: Must be capable of generating a decision tree from a plan's definition. > - **Execution Actor**: Must be able to execute tool calls and manage sandbox operations. > - **Estimation Actor**: Must provide cost and duration estimates for a given plan. > - **Invariant Reconciliation Actor**: Must be able to reconcile and enforce invariants. > - **Review Actor**: Must be able to review and approve plan steps. The `RoleHint` enum in `src/cleveragents/actor/role_validation.py` correctly defines all five roles: ```python class RoleHint(StrEnum): STRATEGY = "strategy" EXECUTION = "execution" ESTIMATION = "estimation" INVARIANT_RECONCILIATION = "invariant_reconciliation" REVIEW = "review" ``` However, the `actor_role_warnings()` function only validates the `ESTIMATION` role. All other roles return an empty warnings list immediately: ```python def actor_role_warnings(config: object) -> list[str]: ... if role_hint != RoleHint.ESTIMATION: return [] # BUG: all non-ESTIMATION roles bypass validation entirely ... ``` ## Current Behavior `actor_role_warnings()` in `src/cleveragents/actor/role_validation.py`: - `RoleHint.ESTIMATION`: Validates that `response_format` is defined and `context_view` is `"strategist"`. - `RoleHint.STRATEGY`: **No validation** — returns `[]` immediately. - `RoleHint.EXECUTION`: **No validation** — returns `[]` immediately. - `RoleHint.INVARIANT_RECONCILIATION`: **No validation** — returns `[]` immediately. - `RoleHint.REVIEW`: **No validation** — returns `[]` immediately. ## Expected Behavior Per the specification, each specialized role must have appropriate validation: - **`STRATEGY`**: Should validate that the actor is capable of generating a decision tree (e.g., has `context_view` set to `"strategist"`, has a `system_prompt` defined, and is of type `LLM` or `GRAPH`). - **`EXECUTION`**: Should validate that the actor can execute tool calls (e.g., has tools defined or is of type `TOOL` or `GRAPH`, and has `context_view` set to `"executor"`). - **`ESTIMATION`**: Already validated — requires `response_format` and `context_view == "strategist"`. - **`INVARIANT_RECONCILIATION`**: Should validate that the actor can reconcile invariants (e.g., has `response_format` for structured output and appropriate `context_view`). - **`REVIEW`**: Should validate that the actor can review and approve plan steps (e.g., has `context_view` set to `"reviewer"`). ## Impact - Actors assigned to `STRATEGY`, `EXECUTION`, `INVARIANT_RECONCILIATION`, or `REVIEW` roles receive no compatibility warnings, even if they are misconfigured for those roles. - The plan lifecycle can silently use incompatible actors for critical roles, leading to runtime failures or incorrect behavior. - The spec's role validation contract is only partially implemented. ## Code Location `src/cleveragents/actor/role_validation.py` — `actor_role_warnings()` function (approximately lines 40–65) ## Steps to Reproduce 1. Create an actor YAML with `role_hint: strategy` but no `system_prompt` and `context_view: executor`. 2. Call `actor_role_warnings(config)`. 3. **Expected**: Warnings about missing `system_prompt` and incorrect `context_view` for a strategy actor. 4. **Actual**: Empty list returned — no warnings. ## Subtasks - [ ] Write TDD failing Behave scenarios for each role (`@tdd_expected_fail`) asserting that `actor_role_warnings()` returns appropriate warnings for misconfigured `STRATEGY`, `EXECUTION`, `INVARIANT_RECONCILIATION`, and `REVIEW` actors - [ ] Implement validation logic in `actor_role_warnings()` for `RoleHint.STRATEGY`: - Warn if `context_view` is not `"strategist"` - Warn if `system_prompt` is not defined (for LLM/GRAPH types) - [ ] Implement validation logic for `RoleHint.EXECUTION`: - Warn if `context_view` is not `"executor"` - Warn if no tools are defined and actor type is `LLM` (execution actors need tools) - [ ] Implement validation logic for `RoleHint.INVARIANT_RECONCILIATION`: - Warn if `response_format` is not defined (structured output required for reconciliation) - Warn if `context_view` is not `"strategist"` or `"full"` - [ ] Implement validation logic for `RoleHint.REVIEW`: - Warn if `context_view` is not `"reviewer"` - [ ] Remove `@tdd_expected_fail` tags once fixes are in place - [ ] Verify `nox -e typecheck` passes - [ ] Verify `nox -e unit_tests` passes with coverage >= 97% - [ ] Run `nox` (all default sessions), fix any errors ## Definition of Done - [ ] `actor_role_warnings()` returns appropriate warnings for misconfigured actors of all five role types - [ ] `RoleHint.STRATEGY`, `RoleHint.EXECUTION`, `RoleHint.INVARIANT_RECONCILIATION`, and `RoleHint.REVIEW` all have validation logic - [ ] Behave BDD scenarios cover all five role types - [ ] All nox stages pass - [ ] Coverage >= 97% > **Backlog note:** This issue was discovered during autonomous UAT testing of the Actor System feature area. It does not block milestone completion and has been placed in the backlog for human review and future milestone assignment. --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-new-issue-creator

freemo added a new dependency

2026-04-05 22:18:54 +00:00

#230 Stage A1: Plan Data Model

freemo referenced this issue

2026-04-05 22:20:24 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 (Resumed) #3654

freemo referenced this issue

2026-04-05 22:20:38 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 (Resumed) #3654

freemo referenced this issue

2026-04-05 22:22:14 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 (Resumed) #3654