feat(autonomy): automation profile resolution precedence correct (plan > action > global) #854

Closed
opened 2026-03-13 22:00:38 +00:00 by freemo · 9 comments
Owner

Metadata

  • Commit Message: feat(autonomy): automation profile resolution precedence correct
  • Branch: feature/m6-automation-profiles

Background

M6 (v3.5.0) acceptance criterion: automation profile resolution must follow the correct four-level precedence: plan-level > action-level > project-level > global default. Per the testing spec (docs/development/testing.md), there are 8 built-in profiles, custom profile support, YAML loading, and the four-level resolution precedence chain.

Per the E2E suite (m6_e2e_verification.robot line 103), the test verifies SubplanConfig accepts all execution modes (SEQUENTIAL, PARALLEL, DEPENDENCY_ORDERED) and all merge strategies, with bounds for max_parallel and timeout configuration.

Expected Behavior

  1. Automation profiles resolve in correct precedence: plan > action > project > global
  2. 8 built-in profiles are available and correctly configured
  3. Custom profiles can be loaded from YAML
  4. Profile resolution is transparent — logs show which level was resolved
  5. SubplanConfig correctly reflects the resolved profile settings

Acceptance Criteria

  • Four-level resolution precedence works correctly
  • 8 built-in profiles available and functional
  • Custom YAML profile loading works
  • Profile resolution is logged for debugging
  • Robot E2E test Subplan Config Supports All Execution Modes passes
  • Unit tests cover: each precedence level, override behavior, custom profiles, edge cases

Supporting Information

  • E2E test: robot/m6_e2e_verification.robot line 103
  • Testing spec: docs/development/testing.md — Automation Profiles section
  • Related: guard enforcement (sibling issue)

Subtasks

  • Implement four-level automation profile resolution
  • Configure 8 built-in profiles
  • Implement YAML custom profile loading
  • Add profile resolution logging
  • Tests (Behave): Add scenarios for profile resolution
  • Tests (Robot): Verify E2E acceptance test passes
  • Verify coverage >=97% via nox -s coverage_report
  • Run nox (all default sessions), fix any errors

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
## Metadata - **Commit Message**: `feat(autonomy): automation profile resolution precedence correct` - **Branch**: `feature/m6-automation-profiles` ## Background M6 (v3.5.0) acceptance criterion: automation profile resolution must follow the correct four-level precedence: plan-level > action-level > project-level > global default. Per the testing spec (`docs/development/testing.md`), there are 8 built-in profiles, custom profile support, YAML loading, and the four-level resolution precedence chain. Per the E2E suite (`m6_e2e_verification.robot` line 103), the test verifies SubplanConfig accepts all execution modes (SEQUENTIAL, PARALLEL, DEPENDENCY_ORDERED) and all merge strategies, with bounds for max_parallel and timeout configuration. ## Expected Behavior 1. Automation profiles resolve in correct precedence: plan > action > project > global 2. 8 built-in profiles are available and correctly configured 3. Custom profiles can be loaded from YAML 4. Profile resolution is transparent — logs show which level was resolved 5. SubplanConfig correctly reflects the resolved profile settings ## Acceptance Criteria - [x] Four-level resolution precedence works correctly - [x] 8 built-in profiles available and functional - [x] Custom YAML profile loading works - [x] Profile resolution is logged for debugging - [x] Robot E2E test `Subplan Config Supports All Execution Modes` passes - [x] Unit tests cover: each precedence level, override behavior, custom profiles, edge cases ## Supporting Information - E2E test: `robot/m6_e2e_verification.robot` line 103 - Testing spec: `docs/development/testing.md` — Automation Profiles section - Related: guard enforcement (sibling issue) ## Subtasks - [x] Implement four-level automation profile resolution - [x] Configure 8 built-in profiles - [x] Implement YAML custom profile loading - [x] Add profile resolution logging - [x] Tests (Behave): Add scenarios for profile resolution - [x] Tests (Robot): Verify E2E acceptance test passes - [x] Verify coverage >=97% via `nox -s coverage_report` - [x] Run `nox` (all default sessions), fix any errors ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done.
freemo added this to the v3.5.0 milestone 2026-03-13 22:00:59 +00:00
Member

Phase 1 update:

  • Verified issue is open and now assigned to brent.edwards.
  • Verified milestone v3.5.0 and transitioned state label to State/In Progress.
  • Created isolated workspace at /tmp/cleveragents-854 and checked out branch feature/m6-automation-profiles.
  • Reviewed docs/specification.md, CONTRIBUTING.md, and docs/timeline.md.

Initial implementation plan:

  1. Implement automation profile resolution at plan creation in cleveragents.application.services.plan_lifecycle_service.PlanLifecycleService.use_action with precedence plan > action > project > global.
  2. Reuse cleveragents.application.services.automation_profile_service.AutomationProfileService for built-in/custom profile lookup and resolution behavior, and persist provenance via AutomationProfileRef.
  3. Add structured logging at profile-resolution time for transparency (resolved profile + provenance + inputs).
  4. Convert existing TDD coverage around profile propagation into active regression coverage and add/adjust Behave scenarios for precedence levels and override behavior.
  5. Run full quality gates and report results before PR creation.
Phase 1 update: - Verified issue is open and now assigned to `brent.edwards`. - Verified milestone `v3.5.0` and transitioned state label to `State/In Progress`. - Created isolated workspace at `/tmp/cleveragents-854` and checked out branch `feature/m6-automation-profiles`. - Reviewed `docs/specification.md`, `CONTRIBUTING.md`, and `docs/timeline.md`. Initial implementation plan: 1. Implement automation profile resolution at plan creation in `cleveragents.application.services.plan_lifecycle_service.PlanLifecycleService.use_action` with precedence `plan > action > project > global`. 2. Reuse `cleveragents.application.services.automation_profile_service.AutomationProfileService` for built-in/custom profile lookup and resolution behavior, and persist provenance via `AutomationProfileRef`. 3. Add structured logging at profile-resolution time for transparency (resolved profile + provenance + inputs). 4. Convert existing TDD coverage around profile propagation into active regression coverage and add/adjust Behave scenarios for precedence levels and override behavior. 5. Run full quality gates and report results before PR creation.
Member

Implementation journal update:

Completed code path changes:

  • Updated cleveragents.application.services.plan_lifecycle_service.PlanLifecycleService.use_action to resolve and persist Plan.automation_profile at plan creation.
  • Added internal resolver PlanLifecycleService._resolve_plan_profile_ref implementing precedence plan > action > project > global and storing provenance in AutomationProfileRef.
  • Added structured resolution logging in PlanLifecycleService including resolved profile, provenance, and contributing sources.
  • Updated cleveragents.cli.commands.plan.use_action to pass --automation-profile directly into service-level plan creation so plan-level precedence is resolved at creation time.

TDD/Regression test updates:

  • Converted features/tdd_use_action_automation_profile.feature from expected-fail capture into active regression coverage for this issue.
  • Updated features/steps/tdd_use_action_automation_profile_steps.py to use an isolated ConfigService instance wired into PlanLifecycleService for deterministic project/global profile resolution checks.

Validation run:

  • nox -e unit_tests -- features/tdd_use_action_automation_profile.feature (3 scenarios passed).

Next step: run full nox quality gates and finalize PR metadata/state transitions.

Implementation journal update: Completed code path changes: - Updated `cleveragents.application.services.plan_lifecycle_service.PlanLifecycleService.use_action` to resolve and persist `Plan.automation_profile` at plan creation. - Added internal resolver `PlanLifecycleService._resolve_plan_profile_ref` implementing precedence `plan > action > project > global` and storing provenance in `AutomationProfileRef`. - Added structured resolution logging in `PlanLifecycleService` including resolved profile, provenance, and contributing sources. - Updated `cleveragents.cli.commands.plan.use_action` to pass `--automation-profile` directly into service-level plan creation so plan-level precedence is resolved at creation time. TDD/Regression test updates: - Converted `features/tdd_use_action_automation_profile.feature` from expected-fail capture into active regression coverage for this issue. - Updated `features/steps/tdd_use_action_automation_profile_steps.py` to use an isolated `ConfigService` instance wired into `PlanLifecycleService` for deterministic project/global profile resolution checks. Validation run: - `nox -e unit_tests -- features/tdd_use_action_automation_profile.feature` ✅ (3 scenarios passed). Next step: run full nox quality gates and finalize PR metadata/state transitions.
Member

Implementation update for #854 (automation profile precedence)

Design and implementation notes

  • Implemented precedence resolution in src/cleveragents/application/services/plan_lifecycle_service.py at service level (PlanLifecycleService.use_action + PlanLifecycleService._resolve_plan_profile_ref) so plan creation now resolves in the required order: plan override > action profile > project config > global config.
  • Injected ConfigService into PlanLifecycleService to resolve core.automation-profile at ConfigLevel.PROJECT and then ConfigLevel.GLOBAL.
  • Added provenance-aware AutomationProfileRef creation so resolved source is represented (PLAN, ACTION, PROJECT, GLOBAL).
  • Added structured debug logging in PlanLifecycleService._resolve_plan_profile_ref to make profile resolution transparent during debugging and audits.
  • Updated CLI wiring in src/cleveragents/cli/commands/plan.py so plan use --automation-profile passes the override into PlanLifecycleService.use_action directly; retained explicit plan-level override assignment post-create for CLI output compatibility in mocked CLI paths.

TDD / test updates

  • Activated the issue regression feature by converting features/tdd_use_action_automation_profile.feature from expected-fail tagging to active issue tagging.
  • Updated features/steps/tdd_use_action_automation_profile_steps.py to use an isolated ConfigService fixture and inject it into PlanLifecycleService for deterministic project/global precedence tests.

Validation results

  • PASS: nox -e unit_tests -- features/tdd_use_action_automation_profile.feature
  • PASS: nox -e unit_tests -- features/consolidated_automation_profile.feature (includes built-ins, precedence, and YAML custom profile coverage)
  • PASS: nox -e unit_tests -- features/automation_profile_cli.feature
  • PASS: nox -e integration_tests -- --test "Subplan Config Supports All Execution Modes" robot/m6_e2e_verification.robot
  • PASS: nox -e lint
  • PASS: nox -e typecheck
  • PASS: nox -e coverage_report (summary met threshold: 97%)

Current blocker on final subtask

  • Full default nox run and full-suite unit/integration/e2e still show broad baseline failures/timeouts unrelated to the #854 delta in this branch snapshot. I have left the final checklist item (Run nox (all default sessions), fix any errors) unchecked pending direction on whether to absorb baseline stabilization into this issue or split as separate stabilization work.
Implementation update for #854 (automation profile precedence) Design and implementation notes - Implemented precedence resolution in `src/cleveragents/application/services/plan_lifecycle_service.py` at service level (`PlanLifecycleService.use_action` + `PlanLifecycleService._resolve_plan_profile_ref`) so plan creation now resolves in the required order: plan override > action profile > project config > global config. - Injected `ConfigService` into `PlanLifecycleService` to resolve `core.automation-profile` at `ConfigLevel.PROJECT` and then `ConfigLevel.GLOBAL`. - Added provenance-aware `AutomationProfileRef` creation so resolved source is represented (`PLAN`, `ACTION`, `PROJECT`, `GLOBAL`). - Added structured debug logging in `PlanLifecycleService._resolve_plan_profile_ref` to make profile resolution transparent during debugging and audits. - Updated CLI wiring in `src/cleveragents/cli/commands/plan.py` so `plan use --automation-profile` passes the override into `PlanLifecycleService.use_action` directly; retained explicit plan-level override assignment post-create for CLI output compatibility in mocked CLI paths. TDD / test updates - Activated the issue regression feature by converting `features/tdd_use_action_automation_profile.feature` from expected-fail tagging to active issue tagging. - Updated `features/steps/tdd_use_action_automation_profile_steps.py` to use an isolated `ConfigService` fixture and inject it into `PlanLifecycleService` for deterministic project/global precedence tests. Validation results - PASS: `nox -e unit_tests -- features/tdd_use_action_automation_profile.feature` - PASS: `nox -e unit_tests -- features/consolidated_automation_profile.feature` (includes built-ins, precedence, and YAML custom profile coverage) - PASS: `nox -e unit_tests -- features/automation_profile_cli.feature` - PASS: `nox -e integration_tests -- --test "Subplan Config Supports All Execution Modes" robot/m6_e2e_verification.robot` - PASS: `nox -e lint` - PASS: `nox -e typecheck` - PASS: `nox -e coverage_report` (summary met threshold: 97%) Current blocker on final subtask - Full default `nox` run and full-suite unit/integration/e2e still show broad baseline failures/timeouts unrelated to the #854 delta in this branch snapshot. I have left the final checklist item (`Run nox (all default sessions), fix any errors`) unchecked pending direction on whether to absorb baseline stabilization into this issue or split as separate stabilization work.
Member

Submission update for #854

  • Branch: feature/m6-automation-profiles
  • Commit: ec0a2f4c
  • PR: #1196
  • Issue state transitioned to State/In Review
  • PR metadata set: milestone v3.5.0, label Type/Feature, and dependency link to issue #854

I will monitor PR checks and respond with fixes if any required check fails.

Submission update for #854 - Branch: `feature/m6-automation-profiles` - Commit: `ec0a2f4c` - PR: https://git.cleverthis.com/cleveragents/cleveragents-core/pulls/1196 - Issue state transitioned to `State/In Review` - PR metadata set: milestone `v3.5.0`, label `Type/Feature`, and dependency link to issue `#854` I will monitor PR checks and respond with fixes if any required check fails.
Member

CI monitoring note:

  • PR #1196 checks have been registered by the forge as 14 contexts.
  • Combined commit status currently reports failure, but individual contexts are still returning state: null from the status API payload, so a concrete failing stage could not yet be extracted via API.
  • I will continue from this point by reconciling failing checks once concrete stage conclusions are available.
CI monitoring note: - PR #1196 checks have been registered by the forge as 14 contexts. - Combined commit status currently reports `failure`, but individual contexts are still returning `state: null` from the status API payload, so a concrete failing stage could not yet be extracted via API. - I will continue from this point by reconciling failing checks once concrete stage conclusions are available.
Member

Implementation update:

  • Stabilized automation-profile-related lifecycle tests in src/cleveragents/application/services/plan_lifecycle_service.py and Behave steps (features/steps/plan_lifecycle_service_steps.py) so phase/state transitions are idempotent under auto-progress (e.g., CI profile) and still respect expected state invariants.
  • Fixed a race/flakiness path in Robot helper robot/helper_tdd_plan_explain_plan_id.py by pre-initializing per-suite SQLite DB from build/.template-migrated.db before DI/container initialization. This avoids intermittent Alembic migration races observed under high parallel pabot load.
  • Increased process timeouts in long-running Robot suites/helpers (robot/m3_e2e_verification.robot, robot/tdd_*, robot/helper_container_resolve_crash.py) to absorb CI variability and reduce false negatives.
  • Updated Robot resource DAG setup (robot/resource_dag.robot) to reuse a shared SQLAlchemy session in inline Python snippets, preventing race-like behavior from per-call session factory semantics in these tests.
  • Adjusted WF07 lifecycle helper (robot/helper_wf07_cicd.py) to account for valid auto-progression endpoints (apply/queued and apply/applied) and guard transition calls by current phase/state.

Quality gates executed and passing on branch feature/m6-automation-profiles:

  • nox -e integration_tests (1819/1819 passed)
  • Full nox default suite
    • lint
    • format
    • typecheck
    • security_scan
    • dead_code
    • unit_tests
    • integration_tests
    • docs
    • build
    • benchmark
    • coverage_report
  • Coverage summary: 97% ((summary) 77527 total, 1962 missing).

This resolves the remaining issue subtask to run full nox and fix any errors.

Implementation update: - Stabilized automation-profile-related lifecycle tests in `src/cleveragents/application/services/plan_lifecycle_service.py` and Behave steps (`features/steps/plan_lifecycle_service_steps.py`) so phase/state transitions are idempotent under auto-progress (e.g., CI profile) and still respect expected state invariants. - Fixed a race/flakiness path in Robot helper `robot/helper_tdd_plan_explain_plan_id.py` by pre-initializing per-suite SQLite DB from `build/.template-migrated.db` before DI/container initialization. This avoids intermittent Alembic migration races observed under high parallel `pabot` load. - Increased process timeouts in long-running Robot suites/helpers (`robot/m3_e2e_verification.robot`, `robot/tdd_*`, `robot/helper_container_resolve_crash.py`) to absorb CI variability and reduce false negatives. - Updated Robot resource DAG setup (`robot/resource_dag.robot`) to reuse a shared SQLAlchemy session in inline Python snippets, preventing race-like behavior from per-call session factory semantics in these tests. - Adjusted WF07 lifecycle helper (`robot/helper_wf07_cicd.py`) to account for valid auto-progression endpoints (`apply/queued` and `apply/applied`) and guard transition calls by current phase/state. Quality gates executed and passing on branch `feature/m6-automation-profiles`: - `nox -e integration_tests` ✅ (1819/1819 passed) - Full `nox` default suite ✅ - lint - format - typecheck - security_scan - dead_code - unit_tests - integration_tests - docs - build - benchmark - coverage_report - Coverage summary: **97%** (`(summary) 77527 total, 1962 missing`). This resolves the remaining issue subtask to run full nox and fix any errors.
Member

Pushed follow-up commit 859b74a5 to feature/m6-automation-profiles with integration stability fixes and full nox verification. PR #1196 is updated and CI checks are now running for the new head SHA.

Pushed follow-up commit `859b74a5` to `feature/m6-automation-profiles` with integration stability fixes and full nox verification. PR #1196 is updated and CI checks are now running for the new head SHA.
Member

Final status: PR #1196 (head 859b74a5) now has all required checks green, including benchmark-regression. PR description has been updated to reflect full nox and CI success.

Final status: PR #1196 (head `859b74a5`) now has all required checks green, including benchmark-regression. PR description has been updated to reflect full nox and CI success.
Member

Implementation Notes — PR Fix Round (Review #2963)

Review Comment 1: Bare except Exception: in _resolve_plan_profile_ref (BLOCKING)

What was changed: Narrowed the exception handler in PlanLifecycleService._resolve_plan_profile_ref from bare except Exception: to except (KeyError, ValueError, OSError) as exc:. Added warning-level structured logging via self._logger.warning() that records the error message, error type, and the settings fallback value.

Rationale: Per CONTRIBUTING.md §Exception Propagation, exceptions should not be silently suppressed. The ConfigService.resolve() method can raise:

  • KeyError — missing dict keys during config resolution
  • ValueError — unregistered config key or TOML parse errors (including tomllib.TOMLDecodeError which is a ValueError subclass)
  • OSError — file I/O issues (includes FileNotFoundError, PermissionError)

Any other exception type (e.g., AttributeError, ImportError) would indicate a programming error and should propagate rather than being silently caught.

Key code location: cleveragents.application.services.plan_lifecycle_service.PlanLifecycleService._resolve_plan_profile_ref — the except clause near the end of the config service resolution block.

Review Comment 2: Tag inconsistency in feature file (BLOCKING)

What was changed: Updated features/tdd_use_action_automation_profile.feature:

  • Tag changed from @tdd_issue @tdd_issue_854 to @tdd_bug @tdd_bug_1076
  • Title changed from "TDD Issue #1076" to "TDD Bug #1076"

Rationale: The TDD test was originally written for bug #1076 (use_action() doesn't propagate automation_profile). The reviewer confirmed @tdd_bug is the correct convention (not @tdd_issue). The @tdd_expected_fail tag was correctly removed since the bug is now fixed.

Additional: Test coverage for exception handler and plan-level override

Added two new BDD scenarios to cover the new code paths:

  1. "Plan falls back to settings default when config service raises an error" — exercises the narrowed exception handler by injecting a _BrokenConfigService that always raises ValueError on resolve(). Verifies fallback to "manual" profile.
  2. "Plan-level profile overrides action-level profile" — exercises the plan-level precedence path (lines 939-941), verifying that automation_profile="ci" overrides the action's "full-auto" profile.

These scenarios restored coverage to >= 97% after the rebase onto latest master.

Rebase

Branch was rebased onto origin/master (532ea100) to incorporate recent commits. One merge conflict in robot/resource_dag.robot (variable naming for shared session factory) was resolved by keeping master's convention (shared_session/lambda: shared_session).

Commit: d35280ff

## Implementation Notes — PR Fix Round (Review #2963) ### Review Comment 1: Bare `except Exception:` in `_resolve_plan_profile_ref` (BLOCKING) **What was changed:** Narrowed the exception handler in `PlanLifecycleService._resolve_plan_profile_ref` from bare `except Exception:` to `except (KeyError, ValueError, OSError) as exc:`. Added warning-level structured logging via `self._logger.warning()` that records the error message, error type, and the settings fallback value. **Rationale:** Per CONTRIBUTING.md §Exception Propagation, exceptions should not be silently suppressed. The `ConfigService.resolve()` method can raise: - `KeyError` — missing dict keys during config resolution - `ValueError` — unregistered config key or TOML parse errors (including `tomllib.TOMLDecodeError` which is a `ValueError` subclass) - `OSError` — file I/O issues (includes `FileNotFoundError`, `PermissionError`) Any other exception type (e.g., `AttributeError`, `ImportError`) would indicate a programming error and should propagate rather than being silently caught. **Key code location:** `cleveragents.application.services.plan_lifecycle_service.PlanLifecycleService._resolve_plan_profile_ref` — the `except` clause near the end of the config service resolution block. ### Review Comment 2: Tag inconsistency in feature file (BLOCKING) **What was changed:** Updated `features/tdd_use_action_automation_profile.feature`: - Tag changed from `@tdd_issue @tdd_issue_854` to `@tdd_bug @tdd_bug_1076` - Title changed from "TDD Issue #1076" to "TDD Bug #1076" **Rationale:** The TDD test was originally written for bug #1076 (use_action() doesn't propagate automation_profile). The reviewer confirmed `@tdd_bug` is the correct convention (not `@tdd_issue`). The `@tdd_expected_fail` tag was correctly removed since the bug is now fixed. ### Additional: Test coverage for exception handler and plan-level override Added two new BDD scenarios to cover the new code paths: 1. **"Plan falls back to settings default when config service raises an error"** — exercises the narrowed exception handler by injecting a `_BrokenConfigService` that always raises `ValueError` on `resolve()`. Verifies fallback to "manual" profile. 2. **"Plan-level profile overrides action-level profile"** — exercises the plan-level precedence path (lines 939-941), verifying that `automation_profile="ci"` overrides the action's "full-auto" profile. These scenarios restored coverage to >= 97% after the rebase onto latest master. ### Rebase Branch was rebased onto `origin/master` (532ea100) to incorporate recent commits. One merge conflict in `robot/resource_dag.robot` (variable naming for shared session factory) was resolved by keeping master's convention (`shared_session`/`lambda: shared_session`). Commit: `d35280ff`
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cleveragents/cleveragents-core#854
No description provided.