cleveragents/cleveragents-core

Fork 3

test(cli): cover action and plan extensions #326

New issue

Closed

opened 2026-02-22 23:41:19 +00:00 by freemo · 4 comments

freemo commented

2026-02-22 23:41:19 +00:00

Owner

Tests (Behave): Add scenarios for automation_profile resolution, invariant ordering, and actor override errors.
Tests (Behave): Add output snapshot assertions for extended fields in JSON/YAML/table formats.
Tests (Robot): Add Robot test that validates action show includes optional actors/invariants when set.
Update docs/development/testing.md with CLI extension fixtures.
Tests (ASV): Add benchmarks/cli_extension_tests_bench.py for extended scenario runtime baseline.
Verify coverage >=97% via nox -s coverage_report.
Run nox (all default sessions, including benchmark), fix any errors.

- [x] Tests (Behave): Add scenarios for automation_profile resolution, invariant ordering, and actor override errors. - [x] Tests (Behave): Add output snapshot assertions for extended fields in JSON/YAML/table formats. - [x] Tests (Robot): Add Robot test that validates `action show` includes optional actors/invariants when set. - [x] Update `docs/development/testing.md` with CLI extension fixtures. - [x] Tests (ASV): Add `benchmarks/cli_extension_tests_bench.py` for extended scenario runtime baseline. - [ ] Verify coverage >=97% via `nox -s coverage_report`. - [ ] Run `nox` (all default sessions, including benchmark), fix any errors.

freemo added this to the (deleted) milestone

2026-02-22 23:41:19 +00:00

freemo added the

labels

2026-02-22 23:41:19 +00:00

brent.edwards was assigned by freemo

2026-02-22 23:41:19 +00:00

freemo added a new dependency

2026-02-22 23:54:25 +00:00

#346 Epic: Foundation Phases 0-1

freemo removed a dependency

2026-02-22 23:56:03 +00:00

#346 Epic: Foundation Phases 0-1

freemo added a new dependency

2026-02-22 23:56:03 +00:00

#364 Epic: Additional CLI Commands & UX

freemo modified the milestone from (deleted) to v3.1.0

2026-02-23 00:07:06 +00:00

freemo commented

2026-02-23 18:32:53 +00:00

Author

Owner

Expected completion updated (Day 15 rebaseline): Day 35 / 2026-03-15 (previously Day 26 / 2026-03-06)

**Expected completion updated (Day 15 rebaseline):** Day 35 / 2026-03-15 (previously Day 26 / 2026-03-06)

freemo added the due date 2026-02-20

2026-02-23 18:41:52 +00:00

freemo referenced this issue

2026-02-24 03:42:31 +00:00

Epic: Additional CLI Commands & UX #364

brent.edwards added

and removed

labels

2026-02-24 19:04:25 +00:00

brent.edwards referenced this issue from a commit

2026-02-24 20:11:48 +00:00

test(cli): cover action and plan extensions

brent.edwards commented

2026-02-24 20:12:13 +00:00

Member

Implementation Summary

Branch: feature/m4-cli-extension-tests

Files Created/Modified

File	Action	Description
`features/cli_extensions.feature`	Modified	Added 27 new scenarios: profile resolution (9), invariant ordering (2), actor error cases (7), output snapshots (9)
`features/steps/cli_extensions_steps.py`	Modified	Added step definitions for ordering verification, JSON/YAML validation, error output assertions
`robot/cli_extensions.robot`	Modified	Added 4 test cases: action show extended, action show JSON, invariant ordering, profile resolution
`robot/helper_cli_extensions.py`	Modified	Added helper functions for action show, invariant ordering, and profile resolution tests
`benchmarks/cli_extension_tests_bench.py`	Created	New ASV benchmark with 4 suites: ProfileResolutionSuite, InvariantOrderingSuite, ActorValidationErrorSuite, OutputFormatRenderingSuite
`benchmarks/cli_extensions_bench.py`	Modified	Fixed pyright type errors (added missing required parameters to mock constructors)
`docs/development/testing.md`	Modified	Added CLI Extension Test Fixtures section documenting fixture patterns, test catalog, and run commands

Test Coverage

Behave: 57 total scenarios in cli_extensions.feature (27 new)
Robot: 9 total test cases in cli_extensions.robot (4 new)
ASV: 12 new benchmarks across 4 suites

Quality Gates Verified

nox -s lint — passed
nox -s typecheck — passed (0 errors)
nox -s format -- --check — passed (744 files formatted)
nox -s unit_tests -- features/cli_extensions.feature — 57 scenarios passed
nox -s integration_tests -- --suite cli_extensions — 9 tests passed

Key Design Decisions

Extended existing cli_extensions_steps.py rather than creating a new step file, per naming convention
Added json import to step file for output snapshot validation
Used validate_namespaced_actor regex pattern edge cases for comprehensive error testing
Removed empty-string test cases (empty strings are falsy in Python, so if automation_profile: skips them — this is correct behavior, not an error case)

## Implementation Summary Branch: `feature/m4-cli-extension-tests` ### Files Created/Modified | File | Action | Description | |------|--------|-------------| | `features/cli_extensions.feature` | Modified | Added 27 new scenarios: profile resolution (9), invariant ordering (2), actor error cases (7), output snapshots (9) | | `features/steps/cli_extensions_steps.py` | Modified | Added step definitions for ordering verification, JSON/YAML validation, error output assertions | | `robot/cli_extensions.robot` | Modified | Added 4 test cases: action show extended, action show JSON, invariant ordering, profile resolution | | `robot/helper_cli_extensions.py` | Modified | Added helper functions for action show, invariant ordering, and profile resolution tests | | `benchmarks/cli_extension_tests_bench.py` | Created | New ASV benchmark with 4 suites: ProfileResolutionSuite, InvariantOrderingSuite, ActorValidationErrorSuite, OutputFormatRenderingSuite | | `benchmarks/cli_extensions_bench.py` | Modified | Fixed pyright type errors (added missing required parameters to mock constructors) | | `docs/development/testing.md` | Modified | Added CLI Extension Test Fixtures section documenting fixture patterns, test catalog, and run commands | ### Test Coverage - **Behave**: 57 total scenarios in `cli_extensions.feature` (27 new) - **Robot**: 9 total test cases in `cli_extensions.robot` (4 new) - **ASV**: 12 new benchmarks across 4 suites ### Quality Gates Verified - `nox -s lint` — passed - `nox -s typecheck` — passed (0 errors) - `nox -s format -- --check` — passed (744 files formatted) - `nox -s unit_tests -- features/cli_extensions.feature` — 57 scenarios passed - `nox -s integration_tests -- --suite cli_extensions` — 9 tests passed ### Key Design Decisions 1. Extended existing `cli_extensions_steps.py` rather than creating a new step file, per naming convention 2. Added `json` import to step file for output snapshot validation 3. Used `validate_namespaced_actor` regex pattern edge cases for comprehensive error testing 4. Removed empty-string test cases (empty strings are falsy in Python, so `if automation_profile:` skips them — this is correct behavior, not an error case)

brent.edwards referenced this issue from a pull request that will close it,

2026-02-24 20:12:39 +00:00

test(cli): cover action and plan extensions #420

brent.edwards added

and removed

labels

2026-02-24 20:12:45 +00:00

freemo commented

2026-02-24 21:54:44 +00:00

Author

Owner

Parent Epic: #364 (Actor System & Tool Integration)

brent.edwards referenced this issue from a commit

2026-02-25 02:55:52 +00:00

chore(pr): address code review feedback for PR #420

brent.edwards commented

2026-02-25 02:56:11 +00:00

Member

Code Review Feedback Addressed

Pushed commit f3ddb48 to feature/m4-cli-extension-tests addressing all four items from Jeff's review on PR #420:

Required Changes

CHANGELOG.md — Added entry under ## Unreleased describing the CLI extension test suite addition.
CONTRIBUTORS.md — Added Brent E. Edwards <brent.edwards@cleverthis.com> to contributors list and details section.

Advisory Changes

benchmarks/cli_extension_tests_bench.py — Replaced broad except Exception: pass with except ValidationError: pass in ActorValidationErrorSuite.time_invalid_actor_rejection. The function validate_namespaced_actor raises cleveragents.core.exceptions.ValidationError, so catching the specific type is both correct and aligned with CONTRIBUTING.md error handling guidelines.
robot/helper_cli_extensions.py — Moved import json from inside action_show_json() function body to module-level imports at top of file, per CONTRIBUTING.md import guidelines.

Verification

nox -e lint — passed
nox -e typecheck — 0 errors, 0 warnings
nox -e format -- --check — 747 files formatted
nox -e unit_tests — 279 features, 5864 scenarios, 25457 steps passed, 0 failures
nox -e integration_tests — all suites passed
nox -e coverage_report — 97.2% (threshold: 97%)

## Code Review Feedback Addressed Pushed commit `f3ddb48` to `feature/m4-cli-extension-tests` addressing all four items from Jeff's review on PR #420: ### Required Changes 1. **CHANGELOG.md** — Added entry under `## Unreleased` describing the CLI extension test suite addition. 2. **CONTRIBUTORS.md** — Added `Brent E. Edwards <brent.edwards@cleverthis.com>` to contributors list and details section. ### Advisory Changes 3. **benchmarks/cli_extension_tests_bench.py** — Replaced broad `except Exception: pass` with `except ValidationError: pass` in `ActorValidationErrorSuite.time_invalid_actor_rejection`. The function `validate_namespaced_actor` raises `cleveragents.core.exceptions.ValidationError`, so catching the specific type is both correct and aligned with CONTRIBUTING.md error handling guidelines. 4. **robot/helper_cli_extensions.py** — Moved `import json` from inside `action_show_json()` function body to module-level imports at top of file, per CONTRIBUTING.md import guidelines. ### Verification - `nox -e lint` — passed - `nox -e typecheck` — 0 errors, 0 warnings - `nox -e format -- --check` — 747 files formatted - `nox -e unit_tests` — 279 features, 5864 scenarios, 25457 steps passed, 0 failures - `nox -e integration_tests` — all suites passed - `nox -e coverage_report` — 97.2% (threshold: 97%)

brent.edwards referenced this issue

2026-02-25 07:43:37 +00:00

test: consolidated Brent QA batch — issues #156, #169, #326, #402, #403 #431

brent.edwards closed this issue

2026-02-25 20:46:19 +00:00

brent.edwards referenced this issue from a commit

2026-02-25 20:46:20 +00:00

Merge pull request 'test: consolidated Brent QA batch — issues #156, #169, #326, #402, #403' (#431) from develop-brent-4 into master

brent.edwards added

State

Completed

and removed