TDD: First-run setup does not auto-generate thinking-effort presets from actor schema #10405

New issue

Open

opened 2026-04-18 09:33:03 +00:00 by HAL9000 · 0 comments

HAL9000 commented

2026-04-18 09:33:03 +00:00

Owner

Metadata

Commit message: test(tui): add failing Behave scenario for thinking-effort preset generation on first run
Branch name: bugfix/m8-first-run-thinking-effort-presets
Tracks bug: #10394

Background and Context

This is the TDD counterpart to bug issue #10394. A failing Behave scenario is required to prove that create_default_persona_for_actor() does not generate thinking-effort presets on first run (the bug), before the fix is implemented.

The scenario is tagged @tdd_expected_fail so it is expected to fail until the fix in #10394 is merged.

Failing Behave Test

The following Behave feature file and step definitions should be added to prove the bug exists:

Feature file: `features/tui/first_run_thinking_effort_presets.feature`

@tdd_issue @tdd_issue_10394 @tdd_expected_fail
Feature: First-run setup auto-generates thinking-effort presets from actor schema

  Background:
    Given a clean persona registry
    And an actor "test_actor" with thinking-effort parameters "low", "medium", "high"

  @tdd_issue @tdd_issue_10394 @tdd_expected_fail
  Scenario: create_default_persona_for_actor generates thinking-effort presets
    When I call create_default_persona_for_actor with actor "test_actor"
    Then the returned persona should have an argument preset named "low"
    And the returned persona should have an argument preset named "medium"
    And the returned persona should have an argument preset named "high"
    And the "low" preset should contain thinking-effort overrides
    And the "medium" preset should contain thinking-effort overrides
    And the "high" preset should contain thinking-effort overrides

  @tdd_issue @tdd_issue_10394 @tdd_expected_fail
  Scenario: Persona saved to registry includes thinking-effort presets
    When I call create_default_persona_for_actor with actor "test_actor"
    Then the persona saved in the registry should have an argument preset named "low"
    And the persona saved in the registry should have an argument preset named "medium"
    And the persona saved in the registry should have an argument preset named "high"

  @tdd_issue @tdd_issue_10394 @tdd_expected_fail
  Scenario: Actor with no thinking-effort parameters falls back to default preset only
    Given an actor "minimal_actor" with no thinking-effort parameters
    When I call create_default_persona_for_actor with actor "minimal_actor"
    Then the returned persona should have only the mandatory "default" preset

Step definitions: `features/steps/tui/first_run_thinking_effort_presets_steps.py`

"""
Step definitions for TDD scenario: first-run thinking-effort preset generation.

These steps are expected to FAIL until bug #10394 is fixed.
Tagged: @tdd_issue @tdd_issue_10394 @tdd_expected_fail
"""
from unittest.mock import MagicMock, patch

from behave import given, then, when


@given('a clean persona registry')
def step_clean_persona_registry(context):
    context.registry = MagicMock()
    context.registry.save = MagicMock()
    context.registry.set_last_persona = MagicMock()
    context.persona = None


@given('an actor "{actor_name}" with thinking-effort parameters {params}')
def step_actor_with_thinking_effort_params(context, actor_name, params):
    # Parse quoted param names from the step text
    import re
    param_names = re.findall(r'"([^"]+)"', params)
    context.actor_name = actor_name
    context.actor_thinking_effort_params = param_names
    # Mock the actor schema to expose thinking-effort parameters
    context.actor_schema_mock = {
        "thinking_effort": {
            "type": "string",
            "enum": param_names,
        }
    }


@given('an actor "{actor_name}" with no thinking-effort parameters')
def step_actor_without_thinking_effort_params(context, actor_name):
    context.actor_name = actor_name
    context.actor_thinking_effort_params = []
    context.actor_schema_mock = {}


@when('I call create_default_persona_for_actor with actor "{actor_name}"')
def step_call_create_default_persona_for_actor(context, actor_name):
    from cleveragents.tui.first_run import create_default_persona_for_actor

    with patch(
        "cleveragents.tui.first_run.get_actor_schema",
        return_value=context.actor_schema_mock,
    ):
        context.persona = create_default_persona_for_actor(
            registry=context.registry,
            actor=actor_name,
        )


@then('the returned persona should have an argument preset named "{preset_name}"')
def step_persona_has_preset(context, preset_name):
    preset_names = [p.name for p in context.persona.argument_presets]
    assert preset_name in preset_names, (
        f"Expected preset '{preset_name}' in persona argument_presets, "
        f"but got: {preset_names}"
    )


@then('the "{preset_name}" preset should contain thinking-effort overrides')
def step_preset_has_thinking_effort_overrides(context, preset_name):
    preset = next(
        (p for p in context.persona.argument_presets if p.name == preset_name),
        None,
    )
    assert preset is not None, f"Preset '{preset_name}' not found"
    assert preset.overrides, (
        f"Preset '{preset_name}' has no overrides (expected thinking-effort overrides)"
    )
    assert "thinking_effort" in preset.overrides, (
        f"Preset '{preset_name}' overrides do not contain 'thinking_effort': "
        f"{preset.overrides}"
    )


@then('the persona saved in the registry should have an argument preset named "{preset_name}"')
def step_saved_persona_has_preset(context, preset_name):
    # Verify the persona passed to registry.save() has the preset
    assert context.registry.save.called, "registry.save() was never called"
    saved_persona = context.registry.save.call_args[0][0]
    preset_names = [p.name for p in saved_persona.argument_presets]
    assert preset_name in preset_names, (
        f"Expected preset '{preset_name}' in saved persona argument_presets, "
        f"but got: {preset_names}"
    )


@then('the returned persona should have only the mandatory "default" preset')
def step_persona_has_only_default_preset(context):
    preset_names = [p.name for p in context.persona.argument_presets]
    assert preset_names == ["default"], (
        f"Expected only ['default'] preset for actor with no thinking-effort params, "
        f"but got: {preset_names}"
    )

Why These Tests Will Fail (Proving the Bug)

The current create_default_persona_for_actor() implementation:

Does not call any get_actor_schema() function
Does not generate low, medium, or high presets
Creates a Persona with only the mandatory default preset

Therefore, the @tdd_expected_fail scenarios above will fail with:

AssertionError: Expected preset 'low' in persona argument_presets, but got: ['default']

This failure proves the bug described in #10394.

Acceptance Criteria

Feature file added at features/tui/first_run_thinking_effort_presets.feature
Step definitions added at features/steps/tui/first_run_thinking_effort_presets_steps.py
All scenarios tagged with @tdd_issue, @tdd_issue_10394, and @tdd_expected_fail
Running behave --tags=@tdd_issue_10394 confirms the scenarios FAIL (proving the bug)
After bug #10394 is fixed, these scenarios should PASS (remove @tdd_expected_fail tag)

Subtasks

Add feature file features/tui/first_run_thinking_effort_presets.feature
Add step definitions features/steps/tui/first_run_thinking_effort_presets_steps.py
Run behave --tags=@tdd_issue_10394 and confirm all scenarios fail as expected
Commit and push to branch bugfix/m8-first-run-thinking-effort-presets

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
The failing Behave scenarios are committed and pushed to the branch bugfix/m8-first-run-thinking-effort-presets.
Running behave --tags=@tdd_issue_10394 confirms the scenarios fail (proving the bug in #10394).
Once bug #10394 is fixed and merged, these scenarios should pass — at that point, remove the @tdd_expected_fail tag and close this issue.

Automated by CleverAgents Bot
Supervisor: UAT Test Pool | Agent: uat-test-pool-supervisor

## Metadata - **Commit message:** `test(tui): add failing Behave scenario for thinking-effort preset generation on first run` - **Branch name:** `bugfix/m8-first-run-thinking-effort-presets` - **Tracks bug:** #10394 ## Background and Context This is the TDD counterpart to bug issue #10394. A failing Behave scenario is required to prove that `create_default_persona_for_actor()` does **not** generate thinking-effort presets on first run (the bug), before the fix is implemented. The scenario is tagged `@tdd_expected_fail` so it is expected to fail until the fix in #10394 is merged. ## Failing Behave Test The following Behave feature file and step definitions should be added to prove the bug exists: ### Feature file: `features/tui/first_run_thinking_effort_presets.feature` ```gherkin @tdd_issue @tdd_issue_10394 @tdd_expected_fail Feature: First-run setup auto-generates thinking-effort presets from actor schema Background: Given a clean persona registry And an actor "test_actor" with thinking-effort parameters "low", "medium", "high" @tdd_issue @tdd_issue_10394 @tdd_expected_fail Scenario: create_default_persona_for_actor generates thinking-effort presets When I call create_default_persona_for_actor with actor "test_actor" Then the returned persona should have an argument preset named "low" And the returned persona should have an argument preset named "medium" And the returned persona should have an argument preset named "high" And the "low" preset should contain thinking-effort overrides And the "medium" preset should contain thinking-effort overrides And the "high" preset should contain thinking-effort overrides @tdd_issue @tdd_issue_10394 @tdd_expected_fail Scenario: Persona saved to registry includes thinking-effort presets When I call create_default_persona_for_actor with actor "test_actor" Then the persona saved in the registry should have an argument preset named "low" And the persona saved in the registry should have an argument preset named "medium" And the persona saved in the registry should have an argument preset named "high" @tdd_issue @tdd_issue_10394 @tdd_expected_fail Scenario: Actor with no thinking-effort parameters falls back to default preset only Given an actor "minimal_actor" with no thinking-effort parameters When I call create_default_persona_for_actor with actor "minimal_actor" Then the returned persona should have only the mandatory "default" preset ``` ### Step definitions: `features/steps/tui/first_run_thinking_effort_presets_steps.py` ```python """ Step definitions for TDD scenario: first-run thinking-effort preset generation. These steps are expected to FAIL until bug #10394 is fixed. Tagged: @tdd_issue @tdd_issue_10394 @tdd_expected_fail """ from unittest.mock import MagicMock, patch from behave import given, then, when @given('a clean persona registry') def step_clean_persona_registry(context): context.registry = MagicMock() context.registry.save = MagicMock() context.registry.set_last_persona = MagicMock() context.persona = None @given('an actor "{actor_name}" with thinking-effort parameters {params}') def step_actor_with_thinking_effort_params(context, actor_name, params): # Parse quoted param names from the step text import re param_names = re.findall(r'"([^"]+)"', params) context.actor_name = actor_name context.actor_thinking_effort_params = param_names # Mock the actor schema to expose thinking-effort parameters context.actor_schema_mock = { "thinking_effort": { "type": "string", "enum": param_names, } } @given('an actor "{actor_name}" with no thinking-effort parameters') def step_actor_without_thinking_effort_params(context, actor_name): context.actor_name = actor_name context.actor_thinking_effort_params = [] context.actor_schema_mock = {} @when('I call create_default_persona_for_actor with actor "{actor_name}"') def step_call_create_default_persona_for_actor(context, actor_name): from cleveragents.tui.first_run import create_default_persona_for_actor with patch( "cleveragents.tui.first_run.get_actor_schema", return_value=context.actor_schema_mock, ): context.persona = create_default_persona_for_actor( registry=context.registry, actor=actor_name, ) @then('the returned persona should have an argument preset named "{preset_name}"') def step_persona_has_preset(context, preset_name): preset_names = [p.name for p in context.persona.argument_presets] assert preset_name in preset_names, ( f"Expected preset '{preset_name}' in persona argument_presets, " f"but got: {preset_names}" ) @then('the "{preset_name}" preset should contain thinking-effort overrides') def step_preset_has_thinking_effort_overrides(context, preset_name): preset = next( (p for p in context.persona.argument_presets if p.name == preset_name), None, ) assert preset is not None, f"Preset '{preset_name}' not found" assert preset.overrides, ( f"Preset '{preset_name}' has no overrides (expected thinking-effort overrides)" ) assert "thinking_effort" in preset.overrides, ( f"Preset '{preset_name}' overrides do not contain 'thinking_effort': " f"{preset.overrides}" ) @then('the persona saved in the registry should have an argument preset named "{preset_name}"') def step_saved_persona_has_preset(context, preset_name): # Verify the persona passed to registry.save() has the preset assert context.registry.save.called, "registry.save() was never called" saved_persona = context.registry.save.call_args[0][0] preset_names = [p.name for p in saved_persona.argument_presets] assert preset_name in preset_names, ( f"Expected preset '{preset_name}' in saved persona argument_presets, " f"but got: {preset_names}" ) @then('the returned persona should have only the mandatory "default" preset') def step_persona_has_only_default_preset(context): preset_names = [p.name for p in context.persona.argument_presets] assert preset_names == ["default"], ( f"Expected only ['default'] preset for actor with no thinking-effort params, " f"but got: {preset_names}" ) ``` ## Why These Tests Will Fail (Proving the Bug) The current `create_default_persona_for_actor()` implementation: 1. Does **not** call any `get_actor_schema()` function 2. Does **not** generate `low`, `medium`, or `high` presets 3. Creates a `Persona` with only the mandatory `default` preset Therefore, the `@tdd_expected_fail` scenarios above will fail with: ``` AssertionError: Expected preset 'low' in persona argument_presets, but got: ['default'] ``` This failure proves the bug described in #10394. ## Acceptance Criteria - [ ] Feature file added at `features/tui/first_run_thinking_effort_presets.feature` - [ ] Step definitions added at `features/steps/tui/first_run_thinking_effort_presets_steps.py` - [ ] All scenarios tagged with `@tdd_issue`, `@tdd_issue_10394`, and `@tdd_expected_fail` - [ ] Running `behave --tags=@tdd_issue_10394` confirms the scenarios FAIL (proving the bug) - [ ] After bug #10394 is fixed, these scenarios should PASS (remove `@tdd_expected_fail` tag) ## Subtasks - [ ] Add feature file `features/tui/first_run_thinking_effort_presets.feature` - [ ] Add step definitions `features/steps/tui/first_run_thinking_effort_presets_steps.py` - [ ] Run `behave --tags=@tdd_issue_10394` and confirm all scenarios fail as expected - [ ] Commit and push to branch `bugfix/m8-first-run-thinking-effort-presets` ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - The failing Behave scenarios are committed and pushed to the branch `bugfix/m8-first-run-thinking-effort-presets`. - Running `behave --tags=@tdd_issue_10394` confirms the scenarios fail (proving the bug in #10394). - Once bug #10394 is fixed and merged, these scenarios should pass — at that point, remove the `@tdd_expected_fail` tag and close this issue. --- **Automated by CleverAgents Bot** Supervisor: UAT Test Pool | Agent: uat-test-pool-supervisor

HAL9000 added the

labels

2026-04-18 09:34:28 +00:00

HAL9000 referenced this issue