cleveragents/cleveragents-core

Fork 3

TDD: Write failing test for #1076 — use_action() does not propagate automation_profile #1098

New issue

Closed

opened 2026-03-22 16:30:10 +00:00 by freemo · 2 comments

freemo commented

2026-03-22 16:30:10 +00:00

Owner

Metadata

Commit Message: test: add TDD bug-capture test for #1076 — use_action automation_profile propagation
Branch: tdd/m4-use-action-automation-profile

Background and Context

This is the TDD counterpart to bug #1076. Per the project's Test-Driven Development workflow for bugs (see CONTRIBUTING.md > Bug Fix Workflow), the first step in fixing any bug is to write a test that captures the buggy behavior. The test is tagged with @tdd_bug, @tdd_bug_1076, and @tdd_expected_fail so that it passes CI while the bug is still unfixed. Once the fix is implemented in #1076, the @tdd_expected_fail tag will be removed and the test will run normally.

See #1076 for full bug details.

Expected Behavior

A new test exists that:

Captures the exact failure described in #1076.
Is tagged with @tdd_bug, @tdd_bug_1076, and @tdd_expected_fail.
Passes CI via the expected-failure mechanism (the underlying assertion fails, confirming the bug exists, but the tag inversion causes the test to pass).
Would fail CI if the bug were fixed without removing the @tdd_expected_fail tag.

Acceptance Criteria

A test is written that captures the bug behavior described in #1076.
The test is tagged with @tdd_bug, @tdd_bug_1076, and @tdd_expected_fail.
The @tdd_expected_fail tag causes the test to pass CI (the underlying assertion fails as expected, proving the bug exists).
The test is specific enough that it will pass normally (without the tag) only when the bug is genuinely fixed.
Tag validation rules pass: @tdd_bug_1076 has corresponding @tdd_bug, and @tdd_expected_fail has both.
A pull request is opened from the branch to master, CI passes, and the PR is merged through the normal merge process.

Definition of Done

This issue is complete when:

All subtasks below are completed and checked off.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the test and what bug behavior it captures.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, CI passes, and the PR is merged before this issue is marked done.

Subtasks

Code: Analyze bug #1076 to identify the exact failure condition, including the inputs, state, and code path that trigger the bug.
Code: Determine the appropriate test type (Behave unit test, Robot integration test, or both) and file location for the reproducing test.
Tests (Behave): Write a Behave scenario in features/ that captures the bug. Tag the scenario with @tdd_bug, @tdd_bug_1076, and @tdd_expected_fail. The scenario must exercise the specific code path that triggers the bug and assert the correct expected behavior (which currently fails due to the bug). Name the scenario descriptively to indicate it is a bug regression test.
Tests (Robot): If the bug involves integration-level behavior, add a Robot test in robot/ with equivalent tags. If purely unit-level, mark N/A with justification.
Docs: Add a comment in the test file explaining this test captures bug #1076 and uses @tdd_expected_fail until the fix is merged.
Quality: Verify CI passes with the tagged test. Confirm the underlying assertion fails for the correct reason.
Quality: Verify tag validation rules pass.
Quality: Verify coverage >=97% via nox -s coverage_report. If coverage is <97% then review the current unit test coverage report at build/coverage.xml and use it to write new Behave based unit tests to improve code coverage.
Quality: Run nox (all default sessions), fix any errors if needed ensuring nox passes across entire code base.

## Metadata - **Commit Message**: `test: add TDD bug-capture test for #1076 — use_action automation_profile propagation` - **Branch**: `tdd/m4-use-action-automation-profile` ## Background and Context This is the TDD counterpart to bug #1076. Per the project's Test-Driven Development workflow for bugs (see `CONTRIBUTING.md` > Bug Fix Workflow), the first step in fixing any bug is to write a test that captures the buggy behavior. The test is tagged with `@tdd_bug`, `@tdd_bug_1076`, and `@tdd_expected_fail` so that it passes CI while the bug is still unfixed. Once the fix is implemented in #1076, the `@tdd_expected_fail` tag will be removed and the test will run normally. See #1076 for full bug details. ## Expected Behavior A new test exists that: 1. Captures the exact failure described in #1076. 2. Is tagged with `@tdd_bug`, `@tdd_bug_1076`, and `@tdd_expected_fail`. 3. Passes CI via the expected-failure mechanism (the underlying assertion fails, confirming the bug exists, but the tag inversion causes the test to pass). 4. Would fail CI if the bug were fixed without removing the `@tdd_expected_fail` tag. ## Acceptance Criteria - [x] A test is written that captures the bug behavior described in #1076. - [x] The test is tagged with `@tdd_bug`, `@tdd_bug_1076`, and `@tdd_expected_fail`. - [x] The `@tdd_expected_fail` tag causes the test to pass CI (the underlying assertion fails as expected, proving the bug exists). - [x] The test is specific enough that it will pass normally (without the tag) only when the bug is genuinely fixed. - [x] Tag validation rules pass: `@tdd_bug_1076` has corresponding `@tdd_bug`, and `@tdd_expected_fail` has both. - [ ] A pull request is opened from the branch to `master`, CI passes, and the PR is merged through the normal merge process. ## Definition of Done This issue is complete when: - All subtasks below are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the test and what bug behavior it captures. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, CI passes, and the PR is **merged** before this issue is marked done. ## Subtasks - [x] Code: Analyze bug #1076 to identify the exact failure condition, including the inputs, state, and code path that trigger the bug. - [x] Code: Determine the appropriate test type (Behave unit test, Robot integration test, or both) and file location for the reproducing test. - [x] Tests (Behave): Write a Behave scenario in `features/` that captures the bug. Tag the scenario with `@tdd_bug`, `@tdd_bug_1076`, and `@tdd_expected_fail`. The scenario must exercise the specific code path that triggers the bug and assert the correct expected behavior (which currently fails due to the bug). Name the scenario descriptively to indicate it is a bug regression test. - [x] Tests (Robot): If the bug involves integration-level behavior, add a Robot test in `robot/` with equivalent tags. If purely unit-level, mark N/A with justification. - [x] Docs: Add a comment in the test file explaining this test captures bug #1076 and uses `@tdd_expected_fail` until the fix is merged. - [x] Quality: Verify CI passes with the tagged test. Confirm the underlying assertion fails for the correct reason. - [x] Quality: Verify tag validation rules pass. - [x] Quality: Verify coverage >=97% via `nox -s coverage_report`. If coverage is <97% then review the current unit test coverage report at `build/coverage.xml` and use it to write new Behave based unit tests to improve code coverage. - [x] Quality: Run `nox` (all default sessions), fix any errors if needed ensuring nox passes across **entire** code base.

freemo added this to the v3.3.0 milestone

2026-03-22 16:30:10 +00:00

freemo added the

labels

2026-03-22 16:30:10 +00:00

brent.edwards was assigned by freemo

2026-03-22 16:30:10 +00:00

freemo added a new dependency

2026-03-22 16:32:53 +00:00

#1076 bug(plan): use_action() does not resolve or propagate automation_profile to created Plan

freemo referenced this issue

2026-03-22 16:32:54 +00:00

bug(plan): use_action() does not resolve or propagate automation_profile to created Plan #1076

freemo referenced this issue

2026-03-22 16:35:31 +00:00

fix(plan): use_action() does not propagate automation_profile from Action to Plan #1060

freemo referenced this issue

2026-03-22 16:41:41 +00:00

bug(plan): use_action() does not resolve or propagate automation_profile to created Plan #1076

freemo added the

State

Verified

label

2026-03-22 16:54:44 +00:00

brent.edwards added

and removed

labels

2026-03-22 20:43:36 +00:00

brent.edwards commented

2026-03-22 21:39:15 +00:00

Member

Implementation Notes — Subtask Analysis and Test Design

Bug Analysis (Subtask 1)

Analyzed bug #1076. The exact failure condition is:

Code path: PlanLifecycleService.use_action() in cleveragents.application.services.plan_lifecycle_service (commit master, around the Plan() constructor call within use_action).
Root cause: The Plan() constructor call in use_action() does not include automation_profile in its keyword arguments. The field is simply omitted.
Inputs that trigger the bug: Any call to use_action() — regardless of whether the Action has automation_profile set, the bug is always triggered because the field is never resolved or passed.
Expected behavior per spec: The Plan's automation_profile should be resolved at plan use time using the four-level precedence chain: plan > action > project > global (spec lines 18919, 18967). The default is "supervised".
Actual behavior: plan.automation_profile is always None.

Test Type Decision (Subtask 2)

This is a unit-level bug in PlanLifecycleService.use_action() — a pure service method. The test is written as a Behave scenario in features/tdd_use_action_automation_profile.feature with step definitions in features/steps/tdd_use_action_automation_profile_steps.py. Robot integration tests are N/A for this bug because:

The bug is entirely within the service layer (no CLI, no I/O, no external dependencies).
Existing CLI-level tests in plan_lifecycle_commands_coverage.feature and plan_cli_spec_alignment.feature already cover the --automation-profile CLI flag routing — the bug is that the service ignores the resolved profile when constructing the Plan.

Test Scenarios (Subtask 3)

Two scenarios capture the bug:

"Plan inherits automation_profile from action when action has a profile set" — Creates an Action with automation_profile="full-auto", uses it via use_action(), and asserts the Plan's automation_profile is not None, has profile_name="full-auto", and provenance="action".
"Plan gets global default automation_profile when action has no profile" — Creates an Action without automation_profile, uses it via use_action(), and asserts the Plan's automation_profile is not None, has profile_name="supervised" (the global default), and provenance="global".

Both scenarios fail at the assertion level (confirming the bug), but pass CI because the @tdd_expected_fail tag inverts the result.

Test Verification

Ran nox -s unit_tests -- features/tdd_use_action_automation_profile.feature — both scenarios show ASSERT FAILED messages proving the bug, and both pass CI via the inversion mechanism. Feature result: 1 feature passed, 0 failed, 0 skipped; 2 scenarios passed, 0 failed, 0 skipped; 12 steps passed, 0 failed, 0 skipped.

## Implementation Notes — Subtask Analysis and Test Design ### Bug Analysis (Subtask 1) Analyzed bug #1076. The exact failure condition is: - **Code path:** `PlanLifecycleService.use_action()` in `cleveragents.application.services.plan_lifecycle_service` (commit `master`, around the `Plan()` constructor call within `use_action`). - **Root cause:** The `Plan()` constructor call in `use_action()` does not include `automation_profile` in its keyword arguments. The field is simply omitted. - **Inputs that trigger the bug:** Any call to `use_action()` — regardless of whether the Action has `automation_profile` set, the bug is always triggered because the field is never resolved or passed. - **Expected behavior per spec:** The Plan's `automation_profile` should be resolved at `plan use` time using the four-level precedence chain: plan > action > project > global (spec lines 18919, 18967). The default is `"supervised"`. - **Actual behavior:** `plan.automation_profile` is always `None`. ### Test Type Decision (Subtask 2) This is a **unit-level** bug in `PlanLifecycleService.use_action()` — a pure service method. The test is written as a **Behave scenario** in `features/tdd_use_action_automation_profile.feature` with step definitions in `features/steps/tdd_use_action_automation_profile_steps.py`. Robot integration tests are **N/A** for this bug because: 1. The bug is entirely within the service layer (no CLI, no I/O, no external dependencies). 2. Existing CLI-level tests in `plan_lifecycle_commands_coverage.feature` and `plan_cli_spec_alignment.feature` already cover the `--automation-profile` CLI flag routing — the bug is that the service ignores the resolved profile when constructing the Plan. ### Test Scenarios (Subtask 3) Two scenarios capture the bug: 1. **"Plan inherits automation_profile from action when action has a profile set"** — Creates an Action with `automation_profile="full-auto"`, uses it via `use_action()`, and asserts the Plan's `automation_profile` is not `None`, has `profile_name="full-auto"`, and `provenance="action"`. 2. **"Plan gets global default automation_profile when action has no profile"** — Creates an Action without `automation_profile`, uses it via `use_action()`, and asserts the Plan's `automation_profile` is not `None`, has `profile_name="supervised"` (the global default), and `provenance="global"`. Both scenarios fail at the assertion level (confirming the bug), but pass CI because the `@tdd_expected_fail` tag inverts the result. ### Test Verification Ran `nox -s unit_tests -- features/tdd_use_action_automation_profile.feature` — both scenarios show `ASSERT FAILED` messages proving the bug, and both pass CI via the inversion mechanism. Feature result: `1 feature passed, 0 failed, 0 skipped; 2 scenarios passed, 0 failed, 0 skipped; 12 steps passed, 0 failed, 0 skipped`.

brent.edwards commented

2026-03-23 01:14:30 +00:00

Member

Quality Gate Results

All required quality gates pass:

Gate	Result	Notes
`nox -s lint`	✅ PASS	All checks passed (ruff lint + format)
`nox -s typecheck`	✅ PASS	0 errors, 1 pre-existing warning
`nox -s unit_tests`	✅ PASS	462 features, 12232 scenarios passed. 2 pre-existing flaky failures in `garbage_collection.feature` (pass when run individually — parallel timing issue, not related to this change)
`nox -s integration_tests`	✅ PASS	1676 tests, 1676 passed, 0 failed
`nox -s e2e_tests`	✅ PASS	37 tests, 37 passed, 0 failed
`nox -s coverage_report`	✅ PASS	98% coverage (threshold: 97%)

TDD Tag Validation

Feature tagged @tdd_expected_fail @tdd_bug @tdd_bug_1076 — all three tags present
@tdd_bug_1076 has corresponding @tdd_bug ✅
@tdd_expected_fail has both @tdd_bug and @tdd_bug_1076 ✅
Both scenarios show ASSERT FAILED at the assertion level, confirming the bug exists ✅
Both scenarios pass CI via the inversion mechanism ✅

Robot (N/A)

Robot integration tests not needed for this bug — it's entirely within the service layer (PlanLifecycleService.use_action()) with no CLI, I/O, or external dependencies involved. Existing CLI-level tests in plan_lifecycle_commands_coverage.feature and plan_cli_spec_alignment.feature already cover the --automation-profile CLI flag routing.

## Quality Gate Results All required quality gates pass: | Gate | Result | Notes | |------|--------|-------| | `nox -s lint` | ✅ PASS | All checks passed (ruff lint + format) | | `nox -s typecheck` | ✅ PASS | 0 errors, 1 pre-existing warning | | `nox -s unit_tests` | ✅ PASS | 462 features, 12232 scenarios passed. 2 pre-existing flaky failures in `garbage_collection.feature` (pass when run individually — parallel timing issue, not related to this change) | | `nox -s integration_tests` | ✅ PASS | 1676 tests, 1676 passed, 0 failed | | `nox -s e2e_tests` | ✅ PASS | 37 tests, 37 passed, 0 failed | | `nox -s coverage_report` | ✅ PASS | 98% coverage (threshold: 97%) | ### TDD Tag Validation - Feature tagged `@tdd_expected_fail @tdd_bug @tdd_bug_1076` — all three tags present - `@tdd_bug_1076` has corresponding `@tdd_bug` ✅ - `@tdd_expected_fail` has both `@tdd_bug` and `@tdd_bug_1076` ✅ - Both scenarios show `ASSERT FAILED` at the assertion level, confirming the bug exists ✅ - Both scenarios pass CI via the inversion mechanism ✅ ### Robot (N/A) Robot integration tests not needed for this bug — it's entirely within the service layer (`PlanLifecycleService.use_action()`) with no CLI, I/O, or external dependencies involved. Existing CLI-level tests in `plan_lifecycle_commands_coverage.feature` and `plan_cli_spec_alignment.feature` already cover the `--automation-profile` CLI flag routing.

brent.edwards referenced this issue from a commit

2026-03-23 01:14:55 +00:00

test: add TDD bug-capture test for #1076 — use_action automation_profile propagation

~~brent.edwards referenced this issue 2026-03-23 01:15:10 +00:00~~

test: add TDD bug-capture test for #1076 — use_action automation_profile propagation #1116

brent.edwards added a new dependency

2026-03-23 01:16:01 +00:00

#1116 test: add TDD bug-capture test for #1076 — use_action automation_profile propagation

brent.edwards added

State

In Review

and removed