hurui200320
2005b8ef82
CI / push-validation (push) Successful in 18s
CI / build (push) Successful in 19s
CI / helm (push) Successful in 24s
CI / lint (push) Successful in 29s
CI / security (push) Successful in 1m11s
CI / e2e_tests (push) Successful in 2m56s
CI / quality (push) Successful in 3m40s
CI / typecheck (push) Successful in 3m59s
CI / integration_tests (push) Successful in 4m3s
CI / unit_tests (push) Successful in 4m55s
CI / docker (push) Successful in 10s
CI / coverage (push) Successful in 10m44s
CI / status-check (push) Successful in 1s
CI / benchmark-regression (push) Has been skipped
CI / benchmark-publish (push) Successful in 1h13m28s
## Summary Replaces all 234 bare `@skip` occurrences across 82 Behave feature files with the correct TDD issue-capture tagging system described in CONTRIBUTING.md § Bug Fix Workflow. Previously, the noxfile ran Behave with `--tags=not @skip`, silently excluding all `@skip`-tagged scenarios from every CI run. These tests never ran, never inverted results via the `@tdd_expected_fail` mechanism, and never contributed to coverage — defeating the purpose of TDD issue-capture testing. Every `@skip` occurrence had a commented-out hint line immediately above it showing the intended proper tags (e.g., `# @tdd_issue @tdd_issue_4272 @tdd_expected_fail @skip`), confirming they were all intended for conversion. ## Changes ### Mechanical conversion (234 replacements across 82 files) - Extracted the proper TDD tags from the comment hint above each `@skip` line, removed `@skip` from the tag set, and replaced the `@skip` line with those tags. - Removed the now-redundant comment hint lines alongside each replacement. ### Bug-fixed scenarios — `@tdd_expected_fail` removed (84 scenarios) - After conversion, ran `nox -s unit_tests` to identify which newly-enabled `@tdd_expected_fail` scenarios now **pass** (their referenced bugs have already been fixed). Removed `@tdd_expected_fail` from those 84 scenarios and their corresponding feature-level tags, leaving only the permanent `@tdd_issue @tdd_issue_<N>` regression-guard tags. - Affected features include: `tdd_tool_runner_env_precedence`, `tdd_automation_profile_session_leak`, `tls_certificate_check`, `project_create_persist`, `resource_type_bootstrap_*`, and 18 others. ### Noxfile cleanup - Removed all four `--tags=not @skip` arguments from `noxfile.py` (unit_tests and coverage sessions). With zero `@skip` tags remaining in the codebase, this filter was dead code and its presence would mislead future maintainers into thinking `@skip` is still a supported escape mechanism. ### Regression guard files - Split the regression guards into two focused files: - `tdd_regression_guards_exec_env.feature` for bug #4281 (exec-env precedence) - `tdd_regression_guards_session_list.feature` for bug #4271 (session list summary) - Each file carries only its own `@tdd_issue` tags at the feature level, avoiding cross-contamination via Behave tag inheritance. The `Background` step (`session-list-summary mock`) only appears in the session-list file where it is actually needed. ### Duplicate tag cleanup - Removed duplicate `@tdd_issue @tdd_issue_4287` tag lines in `tdd_skill_add_regression.feature` (lines 20 and 29). ### Inline comment for retained `@tdd_expected_fail` - Added inline comment in `ci_workflow_validation.feature:134` explaining why this specific #4227 scenario retains `@tdd_expected_fail` despite #4227 being closed (CI YAML does not encode threshold as a machine-readable value). ### Known edge cases — `@tdd_expected_fail` retained (closed issues, fix on master, scenarios still fail) The following issues are **closed** and their fixes **are on master**, but the specific test assertions still fail because the fixes address other aspects of the bugs. The `@tdd_expected_fail` tags are functionally correct and must remain until the specific scenario assertions pass: - `tdd_exec_env_resolution_precedence.feature` — bug #1080 (closed 2026-03-31). The precedence-level-2-vs-4 scenario still fails. - `session_list_summary_dedup.feature` — bug #3046 (closed 2026-04-05). The dedup-consistency scenarios still fail. - `actor_add_update_enforcement.feature` — bug #2609 (closed 2026-04-05). The enforcement scenarios still fail. - `ci_workflow_validation.feature:134` — #4227 (closed 2026-04-08). The CI YAML threshold assertion still fails. ## Verification - `grep -r "@skip" features/ --include="*.feature"` → **zero results** ✓ - `grep -n "tags=not @skip" noxfile.py` → **zero results** ✓ - `nox -s unit_tests` → **629 features passed, 0 failed** ✓ (up from ~545 before this PR) - CI all green (coverage ≥ 97%) ✓ - Integration tests (Robot Framework) do not use `@skip` — confirmed no action needed ✓ - E2E tests (Robot Framework) do not use `@skip` — confirmed no action needed ✓ - `CHANGELOG.md` updated with entry for this change ✓ - `CONTRIBUTORS.md` — Rui Hu already listed ✓ ## Issues Addressed Closes #7025 Co-authored-by: CleverThis <hal9000@cleverthis.com> Reviewed-on: #7221 Reviewed-by: HAL 9000 <HAL9000@cleverthis.com> Reviewed-by: HAL9001 <hal9001@cleverthis.com> Co-authored-by: Rui Hu <rui.hu@cleverthis.com> Co-committed-by: Rui Hu <rui.hu@cleverthis.com>
367 lines
18 KiB
Gherkin
367 lines
18 KiB
Gherkin
Feature: CLI extensions for plan and action commands
|
|
As a developer using the CleverAgents CLI
|
|
I want automation profile, invariant, and actor override flags
|
|
So that I can customize plan creation and view extended details
|
|
|
|
Background:
|
|
Given a cli extensions test runner
|
|
And a cli extensions mocked lifecycle service
|
|
|
|
# --- plan use: automation profile ---
|
|
|
|
Scenario: Plan use with --automation-profile persists to plan metadata
|
|
Given a cli extensions action exists
|
|
When I run plan use with automation profile flag "trusted"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan should have automation profile "trusted"
|
|
|
|
Scenario: Plan use with invalid automation profile is rejected
|
|
Given a cli extensions action exists
|
|
When I run plan use with automation profile flag "nonexistent-profile"
|
|
Then the cli extensions plan use should fail
|
|
|
|
# --- plan use: invariant flags ---
|
|
|
|
Scenario: Plan use with --invariant flags persists to plan metadata
|
|
Given a cli extensions action exists
|
|
When I run plan use with invariant flags "No warnings" and "Keep compat"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions service received invariants "No warnings" and "Keep compat"
|
|
|
|
Scenario: Plan use with single invariant flag
|
|
Given a cli extensions action exists
|
|
When I run plan use with single invariant "All tests must pass"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions service received single invariant "All tests must pass"
|
|
|
|
# --- plan use: actor override validation (valid namespaced) ---
|
|
|
|
Scenario: Plan use with valid namespaced strategy-actor
|
|
Given a cli extensions action exists
|
|
When I run plan use with strategy actor flag "openai/gpt-4"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan has strategy actor "openai/gpt-4"
|
|
|
|
Scenario: Plan use with valid namespaced execution-actor
|
|
Given a cli extensions action exists
|
|
When I run plan use with execution actor flag "anthropic/claude-3"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan has execution actor "anthropic/claude-3"
|
|
|
|
Scenario: Plan use with valid namespaced estimation-actor
|
|
Given a cli extensions action exists
|
|
When I run plan use with estimation actor flag "openai/gpt-4"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan has estimation actor "openai/gpt-4"
|
|
|
|
Scenario: Plan use with valid namespaced invariant-actor
|
|
Given a cli extensions action exists
|
|
When I run plan use with invariant actor flag "openai/gpt-4"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan has invariant actor "openai/gpt-4"
|
|
|
|
# --- plan use: actor override validation (invalid format) ---
|
|
|
|
Scenario: Plan use rejects invalid strategy-actor format
|
|
Given a cli extensions action exists
|
|
When I run plan use with strategy actor flag "bad-format"
|
|
Then the cli extensions plan use should fail with actor validation error
|
|
|
|
Scenario: Plan use rejects invalid execution-actor format
|
|
Given a cli extensions action exists
|
|
When I run plan use with execution actor flag "no_slash"
|
|
Then the cli extensions plan use should fail with actor validation error
|
|
|
|
Scenario: Plan use rejects invalid estimation-actor format
|
|
Given a cli extensions action exists
|
|
When I run plan use with estimation actor flag "UPPERCASE/gpt"
|
|
Then the cli extensions plan use should fail with actor validation error
|
|
|
|
Scenario: Plan use rejects invalid invariant-actor format
|
|
Given a cli extensions action exists
|
|
When I run plan use with invariant actor flag "spaces are bad"
|
|
Then the cli extensions plan use should fail with actor validation error
|
|
|
|
# --- plan status: showing automation_profile ---
|
|
|
|
Scenario: Plan status shows automation profile in table
|
|
Given cli extensions plans exist with automation profile "trusted"
|
|
When I run cli extensions plan status
|
|
Then the cli extensions status output should contain "trusted"
|
|
|
|
Scenario: Plan status shows automation profile in JSON
|
|
Given cli extensions plans exist with automation profile "manual"
|
|
When I run cli extensions plan status with format "json"
|
|
Then the cli extensions status json output should contain automation profile "manual"
|
|
|
|
# --- plan list: showing invariants when present ---
|
|
|
|
Scenario: Plan list shows invariant count in table
|
|
Given cli extensions plans exist with invariants
|
|
When I run cli extensions plan list
|
|
Then the cli extensions list output should contain invariant count
|
|
|
|
Scenario: Plan list shows invariants in JSON
|
|
Given cli extensions plans exist with invariants
|
|
When I run cli extensions plan list with format "json"
|
|
Then the cli extensions list json output should contain invariants
|
|
|
|
# --- action show: optional actors ---
|
|
|
|
Scenario: Action show displays estimation actor when present
|
|
Given a cli extensions action with estimation actor "openai/gpt-4"
|
|
When I run cli extensions action show
|
|
Then the cli extensions action output should contain "Estimation Actor"
|
|
And the cli extensions action output should contain "openai/gpt-4"
|
|
|
|
Scenario: Action show displays invariant actor when present
|
|
Given a cli extensions action with invariant actor "anthropic/claude-3"
|
|
When I run cli extensions action show
|
|
Then the cli extensions action output should contain "Invariant Actor"
|
|
And the cli extensions action output should contain "anthropic/claude-3"
|
|
|
|
Scenario: Action show displays invariants when present
|
|
Given a cli extensions action with invariants
|
|
When I run cli extensions action show
|
|
Then the cli extensions action output should contain "Invariants"
|
|
|
|
Scenario: Action show displays inputs schema when present
|
|
Given a cli extensions action with inputs schema
|
|
When I run cli extensions action show
|
|
Then the cli extensions action output should contain "Inputs Schema"
|
|
|
|
Scenario: Action show in JSON includes optional actors
|
|
Given a cli extensions action with estimation actor "openai/gpt-4"
|
|
When I run cli extensions action show with format "json"
|
|
Then the cli extensions action json should contain "estimation_actor"
|
|
|
|
Scenario: Action show in JSON includes inputs schema
|
|
Given a cli extensions action with inputs schema
|
|
When I run cli extensions action show with format "json"
|
|
Then the cli extensions action json should contain "inputs_schema"
|
|
|
|
# --- plan use: automation_profile + invariants combined ---
|
|
|
|
Scenario: Plan use with automation profile and invariants together
|
|
Given a cli extensions action exists
|
|
When I run plan use with profile "trusted" and invariant "No regressions"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan should have automation profile "trusted"
|
|
And the cli extensions service received single invariant "No regressions"
|
|
|
|
# --- plan status: single plan view with invariants ---
|
|
|
|
Scenario: Plan status single plan view renders invariants in rich panel
|
|
Given cli extensions single plan exists with invariants
|
|
When I run cli extensions plan status for single plan
|
|
Then the cli extensions status output should contain "Invariants"
|
|
And the cli extensions status output should contain "No warnings"
|
|
And the cli extensions status output should contain "plan"
|
|
|
|
Scenario: Plan status single plan view renders automation profile in rich panel
|
|
Given cli extensions single plan exists with profile "trusted"
|
|
When I run cli extensions plan status for single plan
|
|
Then the cli extensions status output should contain "Automation Profile"
|
|
And the cli extensions status output should contain "trusted"
|
|
|
|
# --- action show: automation profile display ---
|
|
|
|
Scenario: Action show displays automation profile when present
|
|
Given a cli extensions action with automation profile "trusted"
|
|
When I run cli extensions action show
|
|
Then the cli extensions action output should contain "Automation Profile"
|
|
And the cli extensions action output should contain "trusted"
|
|
|
|
Scenario: Action show in JSON without invariant actor omits field
|
|
Given a cli extensions action without invariant actor
|
|
When I run cli extensions action show with format "json"
|
|
Then the cli extensions action json should not contain "invariant_actor"
|
|
|
|
# --- validate_namespaced_actor function ---
|
|
|
|
Scenario: validate_namespaced_actor accepts valid names
|
|
Then validate_namespaced_actor accepts "openai/gpt-4"
|
|
And validate_namespaced_actor accepts "anthropic/claude-3"
|
|
And validate_namespaced_actor accepts "local/my-actor"
|
|
|
|
Scenario: validate_namespaced_actor rejects invalid names
|
|
Then validate_namespaced_actor rejects "no-slash"
|
|
And validate_namespaced_actor rejects "UPPER/case"
|
|
And validate_namespaced_actor rejects "/leading-slash"
|
|
And validate_namespaced_actor rejects "trail/ing/"
|
|
|
|
# ==========================================================================
|
|
# DEEP COVERAGE: Automation profile resolution logic (#326)
|
|
# ==========================================================================
|
|
|
|
Scenario: Plan use resolves valid builtin profile "manual"
|
|
Given a cli extensions action exists
|
|
When I run plan use with automation profile flag "manual"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan should have automation profile "manual"
|
|
|
|
Scenario: Plan use resolves valid builtin profile "review"
|
|
Given a cli extensions action exists
|
|
When I run plan use with automation profile flag "review"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan should have automation profile "review"
|
|
|
|
Scenario: Plan use resolves valid builtin profile "supervised"
|
|
Given a cli extensions action exists
|
|
When I run plan use with automation profile flag "supervised"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan should have automation profile "supervised"
|
|
|
|
Scenario: Plan use resolves valid builtin profile "cautious"
|
|
Given a cli extensions action exists
|
|
When I run plan use with automation profile flag "cautious"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan should have automation profile "cautious"
|
|
|
|
Scenario: Plan use resolves valid builtin profile "auto"
|
|
Given a cli extensions action exists
|
|
When I run plan use with automation profile flag "auto"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan should have automation profile "auto"
|
|
|
|
Scenario: Plan use rejects profile name with special characters
|
|
Given a cli extensions action exists
|
|
When I run plan use with automation profile flag "tr@sted!"
|
|
Then the cli extensions plan use should fail
|
|
|
|
Scenario: Plan use rejects profile name with spaces
|
|
Given a cli extensions action exists
|
|
When I run plan use with automation profile flag "my profile"
|
|
Then the cli extensions plan use should fail
|
|
|
|
Scenario: Plan use profile resolution provides available profiles on error
|
|
Given a cli extensions action exists
|
|
When I run plan use with automation profile flag "nonexistent"
|
|
Then the cli extensions plan use should fail
|
|
And the cli extensions error output mentions available profiles
|
|
|
|
# ==========================================================================
|
|
# DEEP COVERAGE: Invariant ordering preservation (#326)
|
|
# ==========================================================================
|
|
|
|
Scenario: Plan use with three invariants preserves insertion order
|
|
Given a cli extensions action exists
|
|
When I run plan use with three invariants "Alpha first" and "Beta second" and "Gamma third"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions service received invariants in order "Alpha first" then "Beta second" then "Gamma third"
|
|
|
|
Scenario: Plan use with five invariants preserves insertion order
|
|
Given a cli extensions action exists
|
|
When I run plan use with five invariants "A" and "B" and "C" and "D" and "E"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions service received five invariants in exact order "A" "B" "C" "D" "E"
|
|
|
|
# ==========================================================================
|
|
# DEEP COVERAGE: Actor override error cases (#326)
|
|
# ==========================================================================
|
|
|
|
Scenario: Plan use rejects actor with double slashes
|
|
Given a cli extensions action exists
|
|
When I run plan use with strategy actor flag "open//ai"
|
|
Then the cli extensions plan use should fail with actor validation error
|
|
|
|
Scenario: Plan use rejects actor with special characters in namespace
|
|
Given a cli extensions action exists
|
|
When I run plan use with strategy actor flag "open@ai/gpt-4"
|
|
Then the cli extensions plan use should fail with actor validation error
|
|
|
|
Scenario: Plan use rejects actor with numeric-start namespace
|
|
Given a cli extensions action exists
|
|
When I run plan use with strategy actor flag "123/model"
|
|
Then the cli extensions plan use should fail with actor validation error
|
|
|
|
Scenario: Plan use rejects actor with only a slash
|
|
Given a cli extensions action exists
|
|
When I run plan use with strategy actor flag "/"
|
|
Then the cli extensions plan use should fail with actor validation error
|
|
|
|
Scenario: Plan use rejects actor with whitespace
|
|
Given a cli extensions action exists
|
|
When I run plan use with execution actor flag "open ai/gpt"
|
|
Then the cli extensions plan use should fail with actor validation error
|
|
|
|
Scenario: Plan use rejects actor with missing name after slash
|
|
Given a cli extensions action exists
|
|
When I run plan use with estimation actor flag "openai/"
|
|
Then the cli extensions plan use should fail with actor validation error
|
|
|
|
Scenario: validate_namespaced_actor rejects double slash
|
|
Then validate_namespaced_actor rejects "ns//name"
|
|
|
|
Scenario: validate_namespaced_actor rejects special characters
|
|
Then validate_namespaced_actor rejects "ns@special/name"
|
|
And validate_namespaced_actor rejects "ns/name#bad"
|
|
|
|
Scenario: validate_namespaced_actor rejects numeric-leading namespace
|
|
Then validate_namespaced_actor rejects "1abc/model"
|
|
|
|
# ==========================================================================
|
|
# DEEP COVERAGE: Output snapshot assertions JSON/YAML/table (#326)
|
|
# ==========================================================================
|
|
|
|
@tdd_issue @tdd_issue_4360 @tdd_expected_fail
|
|
Scenario: Plan status in JSON format contains required keys
|
|
Given cli extensions plans exist with automation profile "trusted"
|
|
When I run cli extensions plan status with format "json"
|
|
Then the cli extensions output is valid JSON
|
|
And the cli extensions json output contains key "automation_profile"
|
|
And the cli extensions json output contains key "namespaced_name"
|
|
And the cli extensions json output contains key "phase"
|
|
|
|
Scenario: Plan status in YAML format contains required fields
|
|
Given cli extensions plans exist with automation profile "trusted"
|
|
When I run cli extensions plan status with format "yaml"
|
|
Then the cli extensions output contains yaml field "automation_profile"
|
|
And the cli extensions output contains yaml field "namespaced_name"
|
|
|
|
@tdd_issue @tdd_issue_4361 @tdd_expected_fail
|
|
Scenario: Action show in JSON format contains all required keys
|
|
Given a cli extensions action with estimation actor "openai/gpt-4"
|
|
When I run cli extensions action show with format "json"
|
|
Then the cli extensions output is valid JSON
|
|
And the cli extensions json output contains key "estimation_actor"
|
|
And the cli extensions json output contains key "strategy_actor"
|
|
And the cli extensions json output contains key "execution_actor"
|
|
And the cli extensions json output contains key "description"
|
|
|
|
Scenario: Action show in YAML format contains required fields
|
|
Given a cli extensions action with estimation actor "openai/gpt-4"
|
|
When I run cli extensions action show with format "yaml"
|
|
Then the cli extensions output contains yaml field "estimation_actor"
|
|
And the cli extensions output contains yaml field "strategy_actor"
|
|
|
|
@tdd_issue @tdd_issue_4362 @tdd_expected_fail
|
|
Scenario: Action show in JSON includes invariants list when present
|
|
Given a cli extensions action with invariants
|
|
When I run cli extensions action show with format "json"
|
|
Then the cli extensions output is valid JSON
|
|
And the cli extensions json output contains key "invariants"
|
|
And the cli extensions json invariants list has length 2
|
|
|
|
Scenario: Plan list in YAML shows invariant data
|
|
Given cli extensions plans exist with invariants
|
|
When I run cli extensions plan list with format "yaml"
|
|
Then the cli extensions output contains yaml field "invariants"
|
|
|
|
Scenario: Plan list in JSON preserves invariant text
|
|
Given cli extensions plans exist with invariants
|
|
When I run cli extensions plan list with format "json"
|
|
Then the cli extensions output is valid JSON
|
|
And the cli extensions json output string contains "No warnings"
|
|
And the cli extensions json output string contains "Keep compat"
|
|
|
|
Scenario: Action show in table format contains actor information
|
|
Given a cli extensions action with estimation actor "openai/gpt-4"
|
|
When I run cli extensions action show with format "table"
|
|
Then the cli extensions action output should contain "openai/gpt-4"
|
|
|
|
Scenario: Plan status in table format contains profile name
|
|
Given cli extensions plans exist with automation profile "cautious"
|
|
When I run cli extensions plan status
|
|
Then the cli extensions status output should contain "cautious"
|