2005b8ef82
CI / push-validation (push) Successful in 18s
CI / build (push) Successful in 19s
CI / helm (push) Successful in 24s
CI / lint (push) Successful in 29s
CI / security (push) Successful in 1m11s
CI / e2e_tests (push) Successful in 2m56s
CI / quality (push) Successful in 3m40s
CI / typecheck (push) Successful in 3m59s
CI / integration_tests (push) Successful in 4m3s
CI / unit_tests (push) Successful in 4m55s
CI / docker (push) Successful in 10s
CI / coverage (push) Successful in 10m44s
CI / status-check (push) Successful in 1s
CI / benchmark-regression (push) Has been skipped
CI / benchmark-publish (push) Successful in 1h13m28s
## Summary Replaces all 234 bare `@skip` occurrences across 82 Behave feature files with the correct TDD issue-capture tagging system described in CONTRIBUTING.md § Bug Fix Workflow. Previously, the noxfile ran Behave with `--tags=not @skip`, silently excluding all `@skip`-tagged scenarios from every CI run. These tests never ran, never inverted results via the `@tdd_expected_fail` mechanism, and never contributed to coverage — defeating the purpose of TDD issue-capture testing. Every `@skip` occurrence had a commented-out hint line immediately above it showing the intended proper tags (e.g., `# @tdd_issue @tdd_issue_4272 @tdd_expected_fail @skip`), confirming they were all intended for conversion. ## Changes ### Mechanical conversion (234 replacements across 82 files) - Extracted the proper TDD tags from the comment hint above each `@skip` line, removed `@skip` from the tag set, and replaced the `@skip` line with those tags. - Removed the now-redundant comment hint lines alongside each replacement. ### Bug-fixed scenarios — `@tdd_expected_fail` removed (84 scenarios) - After conversion, ran `nox -s unit_tests` to identify which newly-enabled `@tdd_expected_fail` scenarios now **pass** (their referenced bugs have already been fixed). Removed `@tdd_expected_fail` from those 84 scenarios and their corresponding feature-level tags, leaving only the permanent `@tdd_issue @tdd_issue_<N>` regression-guard tags. - Affected features include: `tdd_tool_runner_env_precedence`, `tdd_automation_profile_session_leak`, `tls_certificate_check`, `project_create_persist`, `resource_type_bootstrap_*`, and 18 others. ### Noxfile cleanup - Removed all four `--tags=not @skip` arguments from `noxfile.py` (unit_tests and coverage sessions). With zero `@skip` tags remaining in the codebase, this filter was dead code and its presence would mislead future maintainers into thinking `@skip` is still a supported escape mechanism. ### Regression guard files - Split the regression guards into two focused files: - `tdd_regression_guards_exec_env.feature` for bug #4281 (exec-env precedence) - `tdd_regression_guards_session_list.feature` for bug #4271 (session list summary) - Each file carries only its own `@tdd_issue` tags at the feature level, avoiding cross-contamination via Behave tag inheritance. The `Background` step (`session-list-summary mock`) only appears in the session-list file where it is actually needed. ### Duplicate tag cleanup - Removed duplicate `@tdd_issue @tdd_issue_4287` tag lines in `tdd_skill_add_regression.feature` (lines 20 and 29). ### Inline comment for retained `@tdd_expected_fail` - Added inline comment in `ci_workflow_validation.feature:134` explaining why this specific #4227 scenario retains `@tdd_expected_fail` despite #4227 being closed (CI YAML does not encode threshold as a machine-readable value). ### Known edge cases — `@tdd_expected_fail` retained (closed issues, fix on master, scenarios still fail) The following issues are **closed** and their fixes **are on master**, but the specific test assertions still fail because the fixes address other aspects of the bugs. The `@tdd_expected_fail` tags are functionally correct and must remain until the specific scenario assertions pass: - `tdd_exec_env_resolution_precedence.feature` — bug #1080 (closed 2026-03-31). The precedence-level-2-vs-4 scenario still fails. - `session_list_summary_dedup.feature` — bug #3046 (closed 2026-04-05). The dedup-consistency scenarios still fail. - `actor_add_update_enforcement.feature` — bug #2609 (closed 2026-04-05). The enforcement scenarios still fail. - `ci_workflow_validation.feature:134` — #4227 (closed 2026-04-08). The CI YAML threshold assertion still fails. ## Verification - `grep -r "@skip" features/ --include="*.feature"` → **zero results** ✓ - `grep -n "tags=not @skip" noxfile.py` → **zero results** ✓ - `nox -s unit_tests` → **629 features passed, 0 failed** ✓ (up from ~545 before this PR) - CI all green (coverage ≥ 97%) ✓ - Integration tests (Robot Framework) do not use `@skip` — confirmed no action needed ✓ - E2E tests (Robot Framework) do not use `@skip` — confirmed no action needed ✓ - `CHANGELOG.md` updated with entry for this change ✓ - `CONTRIBUTORS.md` — Rui Hu already listed ✓ ## Issues Addressed Closes #7025 Co-authored-by: CleverThis <hal9000@cleverthis.com> Reviewed-on: #7221 Reviewed-by: HAL 9000 <HAL9000@cleverthis.com> Reviewed-by: HAL9001 <hal9001@cleverthis.com> Co-authored-by: Rui Hu <rui.hu@cleverthis.com> Co-committed-by: Rui Hu <rui.hu@cleverthis.com>
289 lines
15 KiB
Gherkin
289 lines
15 KiB
Gherkin
Feature: Config CLI safety-net coverage
|
|
As a developer
|
|
I want thorough safety-net tests for every function in config.py
|
|
So that 100% line and branch coverage is maintained as the code evolves
|
|
|
|
# =====================================================================
|
|
# _normalize_key (L91-96)
|
|
# =====================================================================
|
|
|
|
Scenario: safety-net _normalize_key returns registry key as-is
|
|
When the safety-net normalizer processes key "core.log.level"
|
|
Then the safety-net normalized result should be "core.log.level"
|
|
|
|
Scenario: safety-net _normalize_key converts underscores to dots for registry match
|
|
When the safety-net normalizer processes key "plan_concurrency"
|
|
Then the safety-net normalized result should be "plan.concurrency"
|
|
|
|
Scenario: safety-net _normalize_key strips whitespace
|
|
When the safety-net normalizer processes key " core.log.level "
|
|
Then the safety-net normalized result should be "core.log.level"
|
|
|
|
Scenario: safety-net _normalize_key returns unrecognized key as-is
|
|
When the safety-net normalizer processes key "unknown_random_key"
|
|
Then the safety-net normalized result should be "unknown_random_key"
|
|
|
|
# =====================================================================
|
|
# _is_secret_key (L116-118)
|
|
# =====================================================================
|
|
|
|
Scenario: safety-net _is_secret_key detects api_key pattern
|
|
When the safety-net secret checker inspects key "openai_api_key"
|
|
Then the safety-net secret check result should be true
|
|
|
|
Scenario: safety-net _is_secret_key detects token pattern
|
|
When the safety-net secret checker inspects key "auth_token"
|
|
Then the safety-net secret check result should be true
|
|
|
|
Scenario: safety-net _is_secret_key detects password pattern
|
|
When the safety-net secret checker inspects key "db_password"
|
|
Then the safety-net secret check result should be true
|
|
|
|
Scenario: safety-net _is_secret_key detects secret pattern
|
|
When the safety-net secret checker inspects key "client_secret"
|
|
Then the safety-net secret check result should be true
|
|
|
|
Scenario: safety-net _is_secret_key rejects non-secret key
|
|
When the safety-net secret checker inspects key "log_level"
|
|
Then the safety-net secret check result should be false
|
|
|
|
Scenario: safety-net _is_secret_key is case insensitive
|
|
When the safety-net secret checker inspects key "API_KEY"
|
|
Then the safety-net secret check result should be true
|
|
|
|
# =====================================================================
|
|
# _mask_value (L121-123)
|
|
# =====================================================================
|
|
|
|
Scenario: safety-net _mask_value always returns four asterisks
|
|
When the safety-net masker masks value "super-secret-123"
|
|
Then the safety-net masked output should be "****"
|
|
|
|
Scenario: safety-net _mask_value masks empty string
|
|
When the safety-net masker masks an empty string value
|
|
Then the safety-net masked output should be "****"
|
|
|
|
# =====================================================================
|
|
# _env_var_for_key (L158-160)
|
|
# =====================================================================
|
|
|
|
Scenario: safety-net _env_var_for_key builds correct env var name
|
|
When the safety-net env var builder processes key "log_level"
|
|
Then the safety-net env var name should be "CLEVERAGENTS_LOG_LEVEL"
|
|
|
|
Scenario: safety-net _env_var_for_key uppercases the key
|
|
When the safety-net env var builder processes key "server_port"
|
|
Then the safety-net env var name should be "CLEVERAGENTS_SERVER_PORT"
|
|
|
|
# =====================================================================
|
|
# _read_config_file (L126-131)
|
|
# =====================================================================
|
|
|
|
Scenario: safety-net _read_config_file returns empty dict when file absent
|
|
Given a safety-net isolated temp config directory
|
|
When the safety-net reader reads the config file
|
|
Then the safety-net read result should be an empty dict
|
|
|
|
Scenario: safety-net _read_config_file reads existing toml data
|
|
Given a safety-net isolated temp config directory
|
|
And a safety-net toml config file containing key "debug_enabled" with value "true"
|
|
When the safety-net reader reads the config file
|
|
Then the safety-net read result should contain key "debug_enabled"
|
|
|
|
# =====================================================================
|
|
# _write_config_file (L134-155) - create new file
|
|
# =====================================================================
|
|
|
|
Scenario: safety-net _write_config_file creates file when absent
|
|
Given a safety-net isolated temp config directory
|
|
When the safety-net writer writes key "log_level" with value "INFO"
|
|
Then the safety-net config file should exist and contain key "log_level"
|
|
|
|
# =====================================================================
|
|
# _settings_fields (L62-70)
|
|
# =====================================================================
|
|
|
|
Scenario: safety-net _settings_fields returns populated dictionary
|
|
When the safety-net fields loader retrieves all settings fields
|
|
Then the safety-net fields result should be a non-empty dict
|
|
|
|
# =====================================================================
|
|
# _validate_key (L99-113) - unknown key path
|
|
# =====================================================================
|
|
|
|
Scenario: safety-net _validate_key raises on unknown key with helpful message
|
|
When the safety-net validator checks unknown key "zzz_nonexistent_xxy"
|
|
Then the safety-net validator should raise BadParameter with "Unknown configuration key"
|
|
|
|
Scenario: safety-net _validate_key accepts valid key and returns normalized form
|
|
When the safety-net validator checks valid key "core.log.level"
|
|
Then the safety-net validator should return "core.log.level"
|
|
|
|
# =====================================================================
|
|
# _resolve_source (L163-174) - default path
|
|
# =====================================================================
|
|
|
|
Scenario: safety-net _resolve_source returns default when no env or file
|
|
Given a safety-net isolated temp config directory
|
|
When the safety-net source resolver checks key "core.log.level"
|
|
Then the safety-net resolved source should be "default"
|
|
|
|
Scenario: safety-net _resolve_source returns env_var when env var is set
|
|
Given a safety-net isolated temp config directory
|
|
And the safety-net env var "CLEVERAGENTS_LOG_LEVEL" is set to "WARNING"
|
|
When the safety-net source resolver checks key "core.log.level"
|
|
Then the safety-net resolved source should be "env_var"
|
|
|
|
# =====================================================================
|
|
# _resolution_chain (L177-209)
|
|
# =====================================================================
|
|
|
|
Scenario: safety-net _resolution_chain returns six-entry list
|
|
Given a safety-net isolated temp config directory
|
|
When the safety-net chain builder builds chain for key "core.log.level"
|
|
Then the safety-net chain should have exactly 6 entries
|
|
And the safety-net chain sources should be "cli_flag, env_var, local, project, global, default"
|
|
|
|
# =====================================================================
|
|
# config_set - type coercion paths (L241-248)
|
|
# =====================================================================
|
|
|
|
Scenario: safety-net config set coerces boolean true
|
|
Given a safety-net isolated temp config directory
|
|
When the safety-net CLI sets key "index.auto-reindex" to value "true" with format "json"
|
|
Then the safety-net set output should be valid JSON
|
|
And the safety-net set JSON field "value" should be boolean true
|
|
|
|
Scenario: safety-net config set coerces boolean false
|
|
Given a safety-net isolated temp config directory
|
|
When the safety-net CLI sets key "index.auto-reindex" to value "false" with format "json"
|
|
Then the safety-net set output should be valid JSON
|
|
And the safety-net set JSON field "value" should be boolean false
|
|
|
|
Scenario: safety-net config set coerces integer value
|
|
Given a safety-net isolated temp config directory
|
|
When the safety-net CLI sets key "plan.concurrency" to value "8080" with format "json"
|
|
Then the safety-net set output should be valid JSON
|
|
And the safety-net set JSON field "value" should be integer 8080
|
|
|
|
Scenario: safety-net config set coerces float value
|
|
Given a safety-net isolated temp config directory
|
|
When the safety-net CLI sets key "plan.budget.warn-threshold" to value "3.14" with format "json"
|
|
Then the safety-net set output should be valid JSON
|
|
And the safety-net set JSON field "value" should be float 3.14
|
|
|
|
Scenario: safety-net config set keeps string when not numeric or bool
|
|
Given a safety-net isolated temp config directory
|
|
When the safety-net CLI sets key "core.log.level" to value "DEBUG" with format "json"
|
|
Then the safety-net set output should be valid JSON
|
|
And the safety-net set JSON field "value" should be string "DEBUG"
|
|
|
|
Scenario: safety-net config set captures previous value field
|
|
Given a safety-net isolated temp config directory
|
|
When the safety-net CLI sets key "core.log.level" to value "DEBUG" with format "json"
|
|
Then the safety-net set output should be valid JSON
|
|
And the safety-net set JSON should contain a "previous_value" field
|
|
|
|
Scenario: safety-net config set rich format shows panel output
|
|
Given a safety-net isolated temp config directory
|
|
When the safety-net CLI sets key "core.log.level" to value "DEBUG" with format "rich"
|
|
Then the safety-net set rich output should contain "Configuration Updated"
|
|
And the safety-net set rich output should contain "core.log.level"
|
|
|
|
# =====================================================================
|
|
# config_get - rich format (L319-335)
|
|
# =====================================================================
|
|
|
|
Scenario: safety-net config get rich format displays panel and chain
|
|
Given a safety-net isolated temp config directory
|
|
When the safety-net CLI gets key "core.log.level" with format "rich" and verbose
|
|
Then the safety-net get rich output should contain "Configuration Value"
|
|
And the safety-net get rich output should contain "Resolution chain"
|
|
|
|
@tdd_issue @tdd_issue_4238 @tdd_expected_fail
|
|
Scenario: safety-net config get yaml format produces valid YAML
|
|
Given a safety-net isolated temp config directory
|
|
When the safety-net CLI gets key "core.log.level" with format "yaml"
|
|
Then the safety-net get output should be valid YAML
|
|
And the safety-net get YAML should contain key "key"
|
|
|
|
Scenario: safety-net config get json format includes type field
|
|
Given a safety-net isolated temp config directory
|
|
When the safety-net CLI gets key "core.log.level" with format "json" and verbose
|
|
Then the safety-net get output should be valid JSON with type field
|
|
|
|
# =====================================================================
|
|
# config_list - YAML and plain formats (L426-431)
|
|
# =====================================================================
|
|
|
|
@tdd_issue @tdd_issue_4238 @tdd_expected_fail
|
|
Scenario: safety-net config list yaml format produces valid YAML
|
|
Given a safety-net isolated temp config directory
|
|
When the safety-net CLI lists config with format "yaml"
|
|
Then the safety-net list output should be valid YAML list
|
|
|
|
Scenario: safety-net config list plain format produces text output
|
|
Given a safety-net isolated temp config directory
|
|
When the safety-net CLI lists config with format "plain"
|
|
Then the safety-net list plain output should contain key-value lines
|
|
|
|
Scenario: safety-net config list table format produces table output
|
|
Given a safety-net isolated temp config directory
|
|
When the safety-net CLI lists config with format "table"
|
|
Then the safety-net list table output should contain column headers
|
|
|
|
# =====================================================================
|
|
# config_list - combined key + value filters (L391-399)
|
|
# =====================================================================
|
|
|
|
Scenario: safety-net config list with both key and value filter
|
|
Given a safety-net isolated temp config directory
|
|
When the safety-net CLI lists config with key pattern "log" and value filter "INFO"
|
|
Then the safety-net combined filter result should succeed
|
|
|
|
# =====================================================================
|
|
# config_list - Path serialisation in non-rich (L428-429)
|
|
# =====================================================================
|
|
|
|
Scenario: safety-net config list json format serialises Path values to strings
|
|
Given a safety-net isolated temp config directory
|
|
And safety-net settings fields include a Path-typed value
|
|
When the safety-net CLI lists config with format "json"
|
|
Then the safety-net list JSON output should not contain "PosixPath" or "WindowsPath"
|
|
|
|
# =====================================================================
|
|
# config_list - secret masking in list (L408-411)
|
|
# =====================================================================
|
|
|
|
Scenario: safety-net config list masks secret values by default
|
|
Given a safety-net isolated temp config directory
|
|
And the safety-net env var "AZURE_OPENAI_API_KEY" is set to "test-secret-value"
|
|
When the safety-net CLI lists all config in json format
|
|
Then the safety-net list JSON should contain masked "****" for key "provider.azure.api-key"
|
|
|
|
Scenario: safety-net config list reveals secrets with show-secrets flag
|
|
Given a safety-net isolated temp config directory
|
|
And the safety-net env var "AZURE_OPENAI_API_KEY" is set to "test-secret-value"
|
|
When the safety-net CLI lists all config in json format with show-secrets
|
|
Then the safety-net list JSON should contain value "test-secret-value" for key "provider.azure.api-key"
|
|
|
|
# =====================================================================
|
|
# config_list - modified flag (L405-407)
|
|
# =====================================================================
|
|
|
|
Scenario: safety-net config list marks modified values
|
|
Given a safety-net isolated temp config directory
|
|
And safety-net settings fields have a value different from default
|
|
When the safety-net CLI lists all config in json format
|
|
Then the safety-net list JSON should include a modified flag set to true
|
|
|
|
# =====================================================================
|
|
# config_get - resolution chain active marker (L333)
|
|
# =====================================================================
|
|
|
|
Scenario: safety-net config get rich format marks active source in chain
|
|
Given a safety-net isolated temp config directory
|
|
And a safety-net patched console for capturing rich output
|
|
When the safety-net CLI gets key "core.log.level" with format "rich" and verbose
|
|
Then the safety-net captured console output should contain "active"
|