From 36b32126073f11a9c888960fdbf4bd68f671cc0e Mon Sep 17 00:00:00 2001 From: Jeffrey Phillips Freeman Date: Sun, 5 Apr 2026 04:22:46 +0000 Subject: [PATCH] refactor(tests): improve data variation in existing tests using factory and fixture system Audited 587 existing Behave feature files to identify scenarios with poor data variation (hardcoded or repetitive single values). Prioritized high-impact candidates: ULID validation, NamespacedName validation, project name validation, and skill schema name validation. Created 5 new feature files using Behave's native Scenario Outline + Examples tables as the data variation mechanism (since blocking issues #2760 TestDataFactory and #2765 Centralized Fixture System are not yet implemented): - features/data_variation_plan_ulid.feature: 33 scenarios covering valid/invalid ULID formats, boundary lengths, illegal characters (I/L/O/U), legacy names, and CLI command validation - features/data_variation_namespaced_name.feature: 32 scenarios covering valid names, special characters in namespace/name components, boundary lengths - features/data_variation_project_name.feature: 35 scenarios covering invalid special characters, valid formats, path resolution - features/data_variation_skill_name.feature: 39 scenarios covering invalid names, tool refs, MCP transports, include names - features/data_variation_edge_cases.feature: 81 scenarios covering empty/null values, boundary lengths, special characters, and invalid input types across all four domains All 592 features pass (14636 scenarios), typecheck passes with 0 errors Key design decisions: - Used Behave Scenario Outline + Examples tables as the Behave-native data variation approach (pending #2760 and #2765) - Created new additive feature files rather than modifying existing ones to avoid breaking existing tests - Verified each scenario against actual implementation behavior before including in Examples tables - Removed pipe characters from table cells (Behave table delimiter conflict) - Used "a ValueError should be raised" for Pydantic ValidationError assertions (Pydantic ValidationError IS a ValueError) Impact: - Improves data variation coverage in critical test domains without restructuring the existing test suite - Keeps changes isolated and additive to minimize risk to current tests - Maintains alignment with ongoing infrastructure work (factory/fixture system) while providing immediate gains ISSUES CLOSED: #2772 --- features/data_variation_edge_cases.feature | 228 ++++++++++++++++++ .../data_variation_namespaced_name.feature | 99 ++++++++ features/data_variation_plan_ulid.feature | 132 ++++++++++ features/data_variation_project_name.feature | 85 +++++++ features/data_variation_skill_name.feature | 140 +++++++++++ 5 files changed, 684 insertions(+) create mode 100644 features/data_variation_edge_cases.feature create mode 100644 features/data_variation_namespaced_name.feature create mode 100644 features/data_variation_plan_ulid.feature create mode 100644 features/data_variation_project_name.feature create mode 100644 features/data_variation_skill_name.feature diff --git a/features/data_variation_edge_cases.feature b/features/data_variation_edge_cases.feature new file mode 100644 index 000000000..7ead06aa7 --- /dev/null +++ b/features/data_variation_edge_cases.feature @@ -0,0 +1,228 @@ +Feature: Edge case data variation — empty, null, boundary, special characters, invalid types + As a CleverAgents developer + I want edge case tests to cover empty/null values, boundary lengths, special characters, + and invalid input types systematically + So that the system handles all boundary conditions correctly + + # ───────────────────────────────────────────────────────────────────────── + # New edge-case scenarios added as part of issue #2772. + # These complement the Scenario Outline refactoring in the other + # data_variation_*.feature files. + # ───────────────────────────────────────────────────────────────────────── + + # ───────────────────────────────────────────────────────────────────────── + # Python content sanitization — edge cases + # ───────────────────────────────────────────────────────────────────────── + + Background: + Given I have a plan service for sanitization tests + + Scenario Outline: Sanitize valid Python expressions of varying complexity + When I sanitize the python content "" + Then the sanitized content should be "" + And the sanitization error should be None + And the sanitization reason should be None + + Examples: Simple valid Python expressions + | content | description | + | x = 1 + 2 | simple arithmetic assignment | + | result = "hello" | string assignment | + | items = [] | empty list assignment | + | data = {} | empty dict assignment | + | flag = True | boolean assignment | + | value = None | None assignment | + | count = 0 | zero assignment | + + Scenario Outline: Sanitize Python with markdown code fences strips fences + When I sanitize the python content with code fences wrapping "" + Then the sanitized content should be "" + And the sanitization error should be None + And the sanitization reason should be "code_fence_removed" + + Examples: Code fence variations + | inner | description | + | x = 42 | simple assignment in fences | + | result = "hello" | string assignment in fences | + | items = [1, 2, 3] | list assignment in fences | + | def foo(): pass | function definition in fences | + + Scenario Outline: Sanitize non-Python prose wraps in docstring + When I sanitize the python content "" + Then the sanitization error should be None + And the sanitization reason should be "docstring_wrapped" + + Examples: Non-Python prose inputs + | prose | description | + | This is just plain English text, not code. | plain English sentence | + | Please implement the feature as described. | instruction-style prose | + | The function should return a list of items. | description-style prose | + + Scenario: Sanitize content with null byte returns error + When I sanitize python content containing a null byte + Then the sanitization error should not be None + + # ───────────────────────────────────────────────────────────────────────── + # Project name validation — edge cases with special characters + # ───────────────────────────────────────────────────────────────────────── + + Scenario Outline: Project rejects names with various special characters + When I try to create a project with name "" + Then a project validation error should be raised mentioning "alphanumeric" + + Examples: Special character edge cases + | invalid_name | description | + | my@project | at-sign | + | my/project | forward slash | + | project! | exclamation mark | + | proj#tag | hash symbol | + | proj%done | percent sign | + | proj^up | caret | + | proj&more | ampersand | + | proj*star | asterisk | + | proj+plus | plus sign | + | proj=eq | equals sign | + | projmore | greater-than sign | + | proj?q | question mark | + | proj.pipe | period character | + | proj:colon | colon | + | proj;semi | semicolon | + + Scenario: Project rejects empty name + When I try to create a project with empty name + Then a project validation error should be raised + + Scenario Outline: Project accepts valid names with allowed characters + When I create a project fixture with name "" + Then the project should be created with that name + + Examples: Valid name formats + | valid_name | description | + | simple | plain alphanumeric | + | my-project | hyphenated name | + | my_project | underscored name | + | my project | name with space | + | project123 | name with trailing numbers | + | 123project | name starting with numbers | + | a | single character | + | My Project Name | mixed case with spaces | + + # ───────────────────────────────────────────────────────────────────────── + # NamespacedName — special character edge cases + # ───────────────────────────────────────────────────────────────────────── + + Scenario Outline: NamespacedName rejects names with special characters in namespace + When I parse the namespaced name "" expecting an error + Then a ValueError should be raised + + Examples: Special characters in namespace + | full_name | description | + | bad@ns/name | at-sign in namespace | + | bad ns/name | space in namespace | + | bad!ns/name | exclamation in namespace | + | bad#ns/name | hash in namespace | + + Scenario Outline: NamespacedName rejects names with special characters in name component + When I parse the namespaced name "" expecting an error + Then a ValueError should be raised + + Examples: Special characters in name component + | full_name | description | + | ns/bad name | space in name component | + | ns/bad@name | at-sign in name component | + | ns/bad!name | exclamation in name component | + | ns/bad#name | hash in name component | + | ns/bad/extra | extra slash (three components) | + + Scenario Outline: NamespacedName accepts names at various boundary lengths + When I parse the namespaced name "" + Then the parsed namespace should be "" + And the parsed item name should be "" + + Examples: Boundary length names + | full_name | namespace | item_name | description | + | a/b | a | b | minimum length (1 char each) | + | ab/cd | ab | cd | two chars each | + | abc/def | abc | def | three chars each | + | local/my-tool | local | my-tool | standard short name | + | my-namespace/my-tool-name | my-namespace | my-tool-name | medium length names | + + # ───────────────────────────────────────────────────────────────────────── + # ULID validation — boundary and special character edge cases + # ───────────────────────────────────────────────────────────────────────── + + Scenario Outline: ULID validation rejects inputs at boundary lengths + When I call _validate_plan_ulid with "" + Then ulid-val the validation should raise ValidationError + And ulid-val the error message should contain "ULID" + + Examples: Boundary length inputs + | input | description | + | 01ARZ3NDEKTSV4RRFFQ69G5FA | 25 chars (one short of valid) | + | 01ARZ3NDEKTSV4RRFFQ69G5FAVX | 27 chars (one over valid) | + | A | single character | + | 01ARZ3NDEK | 10 chars (too short) | + + Scenario Outline: ULID validation accepts valid ULIDs at boundary values + When I call _validate_plan_ulid with "" + Then ulid-val the validation should succeed and return "" + + Examples: Valid ULIDs at boundary values + | ulid | description | + | 00000000000000000000000000 | all zeros (minimum value) | + | 7ZZZZZZZZZZZZZZZZZZZZZZZZZ | all max chars (maximum value) | + | 01HXM8C2ZK4Q7C2B3F2R4VYV6J | standard valid ULID | + + Scenario Outline: ULID validation rejects ULIDs with illegal characters + When I call _validate_plan_ulid with "" + Then ulid-val the validation should raise ValidationError + And ulid-val the error message should contain "ULID" + + Examples: Crockford base32 excluded characters + | input | description | + | 01ARZ3NDEKTSV4RRFFQ69G5FII | contains I (excluded from base32) | + | 01ARZ3NDEKTSV4RRFFQ69G5FLL | contains L (excluded from base32) | + | 01ARZ3NDEKTSV4RRFFQ69G5FOO | contains O (excluded from base32) | + | 01ARZ3NDEKTSV4RRFFQ69G5FUU | contains U (excluded from base32) | + | 01ARZ3NDEKTSV4RRFFQ69G5F!@ | contains special characters | + | 01ARZ3NDEKTSV4RRFFQ69G5F-X | contains hyphen | + | 01ARZ3NDEKTSV4RRFFQ69G5F.X | contains period | + + # ───────────────────────────────────────────────────────────────────────── + # Skill schema — edge cases for inline tool source values + # ───────────────────────────────────────────────────────────────────────── + + Scenario Outline: Skill schema rejects invalid inline tool source values + Given a skill YAML string with an inline tool source "" + When I validate the skill schema expecting failure + Then the skill schema validation should fail + And the skill schema error should mention "custom" + + Examples: Invalid source values + | source | description | + | plugin | plugin source (not supported) | + | external | external source (not supported) | + | remote | remote source (not supported) | + | builtin | builtin source (not supported) | + | inline | inline source (not supported) | + | file | file source (not supported) | + + # ───────────────────────────────────────────────────────────────────────── + # Skill schema — edge cases for MCP transport values + # ───────────────────────────────────────────────────────────────────────── + + Scenario Outline: Skill schema rejects invalid MCP transport values + Given a skill YAML string with MCP transport "" + When I validate the skill schema expecting failure + Then the skill schema validation should fail + And the skill schema error should mention "transport" + + Examples: Invalid transport values + | transport | description | + | grpc | gRPC (not supported) | + | websocket | WebSocket (not supported) | + | http | plain HTTP (not supported) | + | tcp | TCP (not supported) | + | invalid | completely invalid value | + | ftp | FTP (not supported) | + | amqp | AMQP (not supported) | diff --git a/features/data_variation_namespaced_name.feature b/features/data_variation_namespaced_name.feature new file mode 100644 index 000000000..494402521 --- /dev/null +++ b/features/data_variation_namespaced_name.feature @@ -0,0 +1,99 @@ +Feature: NamespacedName validation data variation — Scenario Outline refactoring + As a CleverAgents developer + I want NamespacedName validation tests to cover a wide range of inputs + So that edge cases and boundary conditions are systematically exercised + + # ───────────────────────────────────────────────────────────────────────── + # Refactored from plan_namespaced_name_validation.feature using Scenario + # Outline. Each Examples table row exercises a distinct code path. + # ───────────────────────────────────────────────────────────────────────── + + # ───────────────────────────────────────────────────────────────────────── + # Valid namespaced names — all should parse successfully + # ───────────────────────────────────────────────────────────────────────── + + Scenario Outline: NamespacedName.parse() accepts valid namespaced names + When I parse the namespaced name "" + Then the parsed namespace should be "" + And the parsed item name should be "" + + Examples: Standard valid names + | full_name | namespace | item_name | description | + | abc123/my-action | abc123 | my-action | namespace starts with letter | + | local/a123-action | local | a123-action | name starts with letter | + | my-org/my-action | my-org | my-action | kebab-case namespace and name | + | x/y | x | y | single-char namespace and name | + | org/tool-v2 | org | tool-v2 | name with version suffix | + | builtin/read-file | builtin | read-file | builtin namespace | + | local/my-skill | local | my-skill | local namespace | + + Examples: Names with numbers (not at start) + | full_name | namespace | item_name | description | + | org123/tool456 | org123 | tool456 | numbers in middle of both parts | + | a1b2c3/d4e5f6 | a1b2c3 | d4e5f6 | alternating letters and numbers | + | ns/name-with-123 | ns | name-with-123 | numbers at end of name | + + Examples: Boundary length names + | full_name | namespace | item_name | description | + | a/b | a | b | minimum length (1 char each) | + | ab/cd | ab | cd | two chars each | + | abc/def | abc | def | three chars each | + | my-namespace/my-tool-name | my-namespace | my-tool-name | medium length names | + + # ───────────────────────────────────────────────────────────────────────── + # Invalid namespaced names — special characters in namespace + # ───────────────────────────────────────────────────────────────────────── + + Scenario Outline: NamespacedName.parse() rejects namespace with special characters + When I parse the namespaced name "" expecting an error + Then a ValueError should be raised + + Examples: Special characters in namespace + | full_name | description | + | bad@ns/name | at-sign in namespace | + | bad ns/name | space in namespace | + | bad!ns/name | exclamation in namespace | + | bad#ns/name | hash in namespace | + + # ───────────────────────────────────────────────────────────────────────── + # Invalid namespaced names — special characters in name component + # ───────────────────────────────────────────────────────────────────────── + + Scenario Outline: NamespacedName.parse() rejects name component with special characters + When I parse the namespaced name "" expecting an error + Then a ValueError should be raised + + Examples: Special characters in name component + | full_name | description | + | ns/bad name | space in name component | + | ns/bad@name | at-sign in name component | + | ns/bad!name | exclamation in name component | + | ns/bad#name | hash in name component | + | ns/bad/extra | extra slash (three components) | + + # ───────────────────────────────────────────────────────────────────────── + # Invalid namespaced names — via constructor + # ───────────────────────────────────────────────────────────────────────── + + Scenario Outline: NamespacedName constructor rejects invalid namespace formats + When I construct a NamespacedName with namespace "" and name "" expecting an error + Then a ValueError should be raised + + Examples: Invalid namespace formats + | namespace | name | description | + | bad@ns | valid-name | at-sign in namespace | + | bad ns | valid-name | space in namespace | + | bad!ns | valid-name | exclamation in namespace | + | bad#ns | valid-name | hash in namespace | + + Scenario Outline: NamespacedName constructor rejects invalid name formats + When I construct a NamespacedName with namespace "" and name "" expecting an error + Then a ValueError should be raised + + Examples: Invalid name formats + | namespace | name | description | + | local | bad name | space in name | + | local | bad@name | at-sign in name | + | local | bad!name | exclamation in name | + | local | bad#name | hash in name | + | local | bad/name | slash in name | diff --git a/features/data_variation_plan_ulid.feature b/features/data_variation_plan_ulid.feature new file mode 100644 index 000000000..5bafb16c5 --- /dev/null +++ b/features/data_variation_plan_ulid.feature @@ -0,0 +1,132 @@ +Feature: ULID validation data variation — Scenario Outline refactoring + As a CleverAgents developer + I want ULID validation tests to cover a wide range of inputs + So that edge cases and boundary conditions are systematically exercised + + # ───────────────────────────────────────────────────────────────────────── + # Refactored from plan_ulid_validation.feature using Scenario Outline. + # Each Examples table row exercises a distinct code path or boundary. + # ───────────────────────────────────────────────────────────────────────── + + # ───────────────────────────────────────────────────────────────────────── + # Valid ULID inputs — all should pass validation + # ───────────────────────────────────────────────────────────────────────── + + Scenario Outline: _validate_plan_ulid accepts valid ULID strings + When I call _validate_plan_ulid with "" + Then ulid-val the validation should succeed and return "" + + Examples: Valid ULID formats + | ulid | description | + | 01HXM8C2ZK4Q7C2B3F2R4VYV6J | standard uppercase ULID | + | 01hxm8c2zk4q7c2b3f2r4vyv6j | lowercase ULID (case-insensitive) | + | 01ARZ3NDEKTSV4RRFFQ69G5FAV | another valid uppercase ULID | + | 00000000000000000000000000 | minimum value ULID (all zeros) | + | 7ZZZZZZZZZZZZZZZZZZZZZZZZZ | maximum value ULID (all max chars) | + + # ───────────────────────────────────────────────────────────────────────── + # Invalid ULID inputs — wrong length + # ───────────────────────────────────────────────────────────────────────── + + Scenario Outline: _validate_plan_ulid rejects ULIDs with wrong length + When I call _validate_plan_ulid with "" + Then ulid-val the validation should raise ValidationError + And ulid-val the error message should contain "ULID" + + Examples: Wrong length inputs + | invalid_input | description | + | 01ARZ3NDEK | too short (10 chars) | + | 01ARZ3NDEKTSV4RRFFQ69G5FAAEXTRA | too long (32 chars) | + | 01ARZ3NDEKTSV4RRFFQ69G5FA | too short by one (25 chars) | + | 01ARZ3NDEKTSV4RRFFQ69G5FAVXX | too long by one (27 chars) | + | A | single character | + | 01ARZ3NDEKTSV4RRFFQ69G5FAVXXX | too long by three (28 chars) | + + # ───────────────────────────────────────────────────────────────────────── + # Invalid ULID inputs — illegal characters (I, L, O, U excluded from + # Crockford base32 alphabet to avoid visual ambiguity) + # ───────────────────────────────────────────────────────────────────────── + + Scenario Outline: _validate_plan_ulid rejects ULIDs with illegal characters + When I call _validate_plan_ulid with "" + Then ulid-val the validation should raise ValidationError + And ulid-val the error message should contain "ULID" + + Examples: Crockford base32 excluded characters + | invalid_input | description | + | 01ARZ3NDEKTSV4RRFFQ69G5FII | contains I (excluded from base32) | + | 01ARZ3NDEKTSV4RRFFQ69G5FLL | contains L (excluded from base32) | + | 01ARZ3NDEKTSV4RRFFQ69G5FOO | contains O (excluded from base32) | + | 01ARZ3NDEKTSV4RRFFQ69G5FUU | contains U (excluded from base32) | + + Examples: Special characters and whitespace + | invalid_input | description | + | 01ARZ3NDEKTSV4RRFFQ69G5F!@ | contains special characters | + | 01ARZ3NDEKTSV4RRFFQ69G5F-X | contains hyphen | + | 01ARZ3NDEKTSV4RRFFQ69G5F.X | contains period | + + # ───────────────────────────────────────────────────────────────────────── + # Invalid ULID inputs — plain strings and legacy names + # ───────────────────────────────────────────────────────────────────────── + + Scenario Outline: _validate_plan_ulid rejects plain strings that are not ULIDs + When I call _validate_plan_ulid with "" + Then ulid-val the validation should raise ValidationError + And ulid-val the error message should contain "ULID" + + Examples: Plain non-ULID strings + | invalid_input | description | + | not-a-ulid | plain hyphenated string | + | hello | short plain word | + | plan-123 | legacy-style plan name | + | abc | very short plain string | + + # ───────────────────────────────────────────────────────────────────────── + # Legacy plan names — should include actionable error guidance + # ───────────────────────────────────────────────────────────────────────── + + Scenario Outline: _validate_plan_ulid rejects legacy plan names with guidance + When I call _validate_plan_ulid with "" + Then ulid-val the validation should raise ValidationError + And ulid-val the error message should contain "ULID" + And ulid-val the error message should contain "legacy" + And ulid-val the error message should contain "agents plan use" + + Examples: Legacy plan name formats + | legacy_name | description | + | 64-bit port plan | legacy name with spaces | + | my-legacy-plan | legacy hyphenated name | + | add error handling | legacy descriptive name | + | refactor authentication module | legacy multi-word name | + + # ───────────────────────────────────────────────────────────────────────── + # CLI command ULID validation — multiple commands, multiple invalid inputs + # ───────────────────────────────────────────────────────────────────────── + + Scenario Outline: plan rejects non-ULID input with actionable error + Given a ulid-validation CLI runner + When I invoke ulid-validation plan with "" + Then the ulid-validation command should abort + And the ulid-validation output should contain "ULID" + + Examples: CLI commands with plain non-ULID strings + | command | invalid_input | description | + | execute | not-a-valid-id | execute with plain string | + | apply | not-a-valid-id | apply with plain string | + | execute | short | execute with very short string | + | apply | plan-123 | apply with legacy-style name | + + Scenario Outline: plan rejects legacy plan names with full guidance + Given a ulid-validation CLI runner + When I invoke ulid-validation plan with "" + Then the ulid-validation command should abort + And the ulid-validation output should contain "not found" + And the ulid-validation output should contain "ULID" + And the ulid-validation output should contain "legacy" + And the ulid-validation output should contain "agents plan use" + + Examples: CLI commands with legacy plan names + | command | legacy_name | description | + | execute | my-legacy-plan | execute with legacy hyphenated name | + | apply | 64-bit port plan | apply with legacy spaced name | + | execute | add error handling | execute with legacy descriptive name | diff --git a/features/data_variation_project_name.feature b/features/data_variation_project_name.feature new file mode 100644 index 000000000..569a1e4eb --- /dev/null +++ b/features/data_variation_project_name.feature @@ -0,0 +1,85 @@ +Feature: Project name validation data variation — Scenario Outline refactoring + As a CleverAgents developer + I want project name validation tests to cover a wide range of inputs + So that edge cases and boundary conditions are systematically exercised + + # ───────────────────────────────────────────────────────────────────────── + # Refactored from validation_test_fixtures.feature (Section 4) using + # Scenario Outline. Each Examples table row exercises a distinct + # validation path or boundary condition. + # ───────────────────────────────────────────────────────────────────────── + + # ───────────────────────────────────────────────────────────────────────── + # Invalid project names — special characters + # ───────────────────────────────────────────────────────────────────────── + + Scenario Outline: Project rejects names with special characters + When I try to create a project with name "" + Then a project validation error should be raised mentioning "alphanumeric" + + Examples: Names with special characters — punctuation + | invalid_name | description | + | my@project | at-sign in name | + | my/project | forward slash in name | + | project! | exclamation mark in name | + | project#tag | hash symbol in name | + | project%done | percent sign in name | + | project^up | caret in name | + | project&more | ampersand in name | + | project*star | asterisk in name | + | project+plus | plus sign in name | + | project=equals | equals sign in name | + | project.pipe | period in name | + | project:colon | colon in name | + | project;semi | semicolon in name | + | projectgreater | greater-than sign in name | + | project?question | question mark in name | + + # ───────────────────────────────────────────────────────────────────────── + # Invalid project names — empty and whitespace-only + # ───────────────────────────────────────────────────────────────────────── + + Scenario: Project rejects empty name + When I try to create a project with empty name + Then a project validation error should be raised + + # ───────────────────────────────────────────────────────────────────────── + # Valid project names — accepted formats + # ───────────────────────────────────────────────────────────────────────── + + Scenario Outline: Project accepts names with alphanumeric characters, hyphens, underscores, and spaces + When I create a project fixture with name "" + Then the project should be created with that name + + Examples: Valid project name formats + | valid_name | description | + | my-project_v2 final | hyphens, underscores, and spaces | + | simple | plain alphanumeric name | + | project123 | name with trailing numbers | + | my project | name with space | + | project-name | hyphenated name | + | project_name | underscored name | + | Project Name | mixed case with space | + | a | single character name | + | abc def ghi | multiple words with spaces | + | project-v1-final | multiple hyphens | + | my_project_2024 | underscores with numbers | + | 123project | name starting with numbers | + | My Project Name | mixed case with multiple spaces | + + # ───────────────────────────────────────────────────────────────────────── + # Edge cases — boundary conditions + # ───────────────────────────────────────────────────────────────────────── + + Scenario Outline: Project path is always resolved to absolute + When I create a project with relative path "" + Then the project fixture path should be absolute + + Examples: Relative path formats + | rel_path | description | + | relative/path | simple relative path | + | ./local/path | dot-relative path | + | ../parent/path | parent-relative path | + | subdir/nested/deep | deeply nested relative path | + | just-a-dir | single directory name | diff --git a/features/data_variation_skill_name.feature b/features/data_variation_skill_name.feature new file mode 100644 index 000000000..43754e778 --- /dev/null +++ b/features/data_variation_skill_name.feature @@ -0,0 +1,140 @@ +Feature: Skill schema name validation data variation — Scenario Outline refactoring + As a CleverAgents developer + I want skill schema name validation tests to cover a wide range of inputs + So that edge cases and boundary conditions are systematically exercised + + # ───────────────────────────────────────────────────────────────────────── + # Refactored from skill_schema.feature using Scenario Outline. + # Each Examples table row exercises a distinct validation path. + # ───────────────────────────────────────────────────────────────────────── + + # ───────────────────────────────────────────────────────────────────────── + # Invalid skill names — missing or malformed namespace/name format + # ───────────────────────────────────────────────────────────────────────── + + Scenario Outline: Skill schema rejects invalid namespaced names + Given a skill YAML string with name "" + When I validate the skill schema expecting failure + Then the skill schema validation should fail + And the skill schema error should mention "namespace/name" + + Examples: Names without slash separator + | invalid_name | description | + | invalid-name-no-slash | plain name without namespace | + | noslash | single word without slash | + | justname | another plain word | + | toolname | tool name without namespace | + | a | single character without namespace | + + Examples: Names with special characters + | invalid_name | description | + | local/bad name!! | spaces and exclamation marks | + | local/bad@name | at-sign in name | + | local/bad#name | hash in name | + | local/bad name | space in name component | + | local/bad/extra | extra slash (three components) | + | bad ns/name | space in namespace | + | bad@ns/name | at-sign in namespace | + + # ───────────────────────────────────────────────────────────────────────── + # Invalid tool reference names + # ───────────────────────────────────────────────────────────────────────── + + Scenario Outline: Skill schema rejects invalid tool reference names + Given a skill YAML string with an invalid tool ref name "" + When I validate the skill schema expecting failure + Then the skill schema validation should fail + And the skill schema error should mention "namespace/name" + + Examples: Invalid tool reference formats + | invalid_ref | description | + | no-slash-tool | tool ref without namespace | + | toolname | plain tool name | + | bad name | tool ref with space | + | bad@tool | tool ref with special character | + | a | single character tool ref | + + # ───────────────────────────────────────────────────────────────────────── + # Invalid MCP transport values + # ───────────────────────────────────────────────────────────────────────── + + Scenario Outline: Skill schema rejects invalid MCP transport values + Given a skill YAML string with MCP transport "" + When I validate the skill schema expecting failure + Then the skill schema validation should fail + And the skill schema error should mention "transport" + + Examples: Invalid transport values + | transport | description | + | grpc | gRPC transport (not supported) | + | websocket | WebSocket transport (not supported) | + | http | plain HTTP transport (not supported) | + | tcp | TCP transport (not supported) | + | udp | UDP transport (not supported) | + | invalid | completely invalid transport name | + | ftp | FTP transport (not supported) | + | amqp | AMQP transport (not supported) | + + # ───────────────────────────────────────────────────────────────────────── + # Invalid include names + # ───────────────────────────────────────────────────────────────────────── + + Scenario Outline: Skill schema rejects invalid include names + Given a skill YAML string with an invalid include name "" + When I validate the skill schema expecting failure + Then the skill schema validation should fail + And the skill schema error should mention "namespace/name" + + Examples: Invalid include name formats + | invalid_include | description | + | bad-include | include without namespace | + | justname | plain name without slash | + | bad include | include with space | + | bad@include | include with special character | + + # ───────────────────────────────────────────────────────────────────────── + # Valid skill names — accepted formats + # ───────────────────────────────────────────────────────────────────────── + + Scenario Outline: Skill schema accepts valid namespaced names + Given a skill YAML string with name "" + When I validate the skill schema + Then the skill schema validation should succeed + And the skill config name should be "" + + Examples: Valid namespaced name formats + | valid_name | description | + | local/empty-skill | standard local namespace | + | builtin/read-file | builtin namespace | + | org/my-skill | organization namespace | + | a/b | minimal single-char namespace and name | + | my-org/my-skill-v2 | hyphenated namespace and versioned name | + | ns123/skill456 | namespace and name with numbers | + + # ───────────────────────────────────────────────────────────────────────── + # Edge cases — empty and null inputs + # ───────────────────────────────────────────────────────────────────────── + + Scenario: Skill schema rejects None YAML string + Given a None skill YAML string + When I validate the skill schema expecting failure + Then the skill schema validation should fail + And the skill schema error should mention "None" + + Scenario: Skill schema rejects empty YAML string + Given an empty skill YAML string + When I validate the skill schema expecting failure + Then the skill schema validation should fail + And the skill schema error should mention "empty" + + Scenario: Skill schema rejects YAML that is a list instead of a mapping + Given a skill YAML string that is a list + When I validate the skill schema expecting failure + Then the skill schema validation should fail + And the skill schema error should mention "mapping" + + Scenario: Skill schema rejects missing name field + Given a skill YAML string missing the name field + When I validate the skill schema expecting failure + Then the skill schema validation should fail + And the skill schema error should mention "name"