cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity

refactor(tests): improve data variation in existing tests using factory and fixture system
CI / typecheck (pull_request) Successful in 1m2s
Details
CI / security (pull_request) Successful in 51s
Details
CI / quality (pull_request) Successful in 46s
Details
CI / lint (pull_request) Successful in 3m21s
Details
CI / build (pull_request) Successful in 19s
Details
CI / helm (pull_request) Successful in 23s
Details
CI / unit_tests (pull_request) Successful in 7m6s
Details
CI / e2e_tests (pull_request) Successful in 18m1s
Details
CI / integration_tests (pull_request) Successful in 22m52s
Details
CI / docker (pull_request) Successful in 21s
Details
CI / coverage (pull_request) Successful in 10m58s
Details
CI / status-check (pull_request) Successful in 1s
Details
CI / benchmark-publish (pull_request) Has been skipped
Details
CI / benchmark-regression (pull_request) Successful in 57m25s
Details

Browse Source 
Audited 587 existing Behave feature files to identify scenarios with poor data variation (hardcoded or repetitive single values). Prioritized high-impact candidates: ULID validation, NamespacedName validation, project name validation, and skill schema name validation. Created 5 new feature files using Behave's native Scenario Outline + Examples tables as the data variation mechanism (since blocking issues #2760 TestDataFactory and #2765 Centralized Fixture System are not yet implemented):
- features/data_variation_plan_ulid.feature: 33 scenarios covering valid/invalid ULID formats, boundary lengths, illegal characters (I/L/O/U), legacy names, and CLI command validation
- features/data_variation_namespaced_name.feature: 32 scenarios covering valid names, special characters in namespace/name components, boundary lengths
- features/data_variation_project_name.feature: 35 scenarios covering invalid special characters, valid formats, path resolution
- features/data_variation_skill_name.feature: 39 scenarios covering invalid names, tool refs, MCP transports, include names
- features/data_variation_edge_cases.feature: 81 scenarios covering empty/null values, boundary lengths, special characters, and invalid input types across all four domains

All 592 features pass (14636 scenarios), typecheck passes with 0 errors

Key design decisions:
- Used Behave Scenario Outline + Examples tables as the Behave-native data variation approach (pending #2760 and #2765)
- Created new additive feature files rather than modifying existing ones to avoid breaking existing tests
- Verified each scenario against actual implementation behavior before including in Examples tables
- Removed pipe characters from table cells (Behave table delimiter conflict)
- Used "a ValueError should be raised" for Pydantic ValidationError assertions (Pydantic ValidationError IS a ValueError)

Impact:
- Improves data variation coverage in critical test domains without restructuring the existing test suite
- Keeps changes isolated and additive to minimize risk to current tests
- Maintains alignment with ongoing infrastructure work (factory/fixture system) while providing immediate gains

ISSUES CLOSED: #2772
This commit is contained in:
Jeffrey Phillips Freeman
2026-04-05 04:22:46 +00:00
parent bbff42ac9a
commit 36b3212607
5 changed files with 684 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
features/data_variation_edge_cases.feature
+228
View File
@@ -0,0 +1,228 @@
Feature: Edge case data variation — empty, null, boundary, special characters, invalid types
As a CleverAgents developer
I want edge case tests to cover empty/null values, boundary lengths, special characters,
and invalid input types systematically
So that the system handles all boundary conditions correctly
# ─────────────────────────────────────────────────────────────────────────
# New edge-case scenarios added as part of issue #2772.
# These complement the Scenario Outline refactoring in the other
# data_variation_*.feature files.
# ─────────────────────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────
# Python content sanitization — edge cases
# ─────────────────────────────────────────────────────────────────────────
Background:
Given I have a plan service for sanitization tests
Scenario Outline: Sanitize valid Python expressions of varying complexity
When I sanitize the python content "<content>"
Then the sanitized content should be "<content>"
And the sanitization error should be None
And the sanitization reason should be None
Examples: Simple valid Python expressions
| content | description |
| x = 1 + 2 | simple arithmetic assignment |
| result = "hello" | string assignment |
| items = [] | empty list assignment |
| data = {} | empty dict assignment |
| flag = True | boolean assignment |
| value = None | None assignment |
| count = 0 | zero assignment |
Scenario Outline: Sanitize Python with markdown code fences strips fences
When I sanitize the python content with code fences wrapping "<inner>"
Then the sanitized content should be "<inner>"
And the sanitization error should be None
And the sanitization reason should be "code_fence_removed"
Examples: Code fence variations
| inner | description |
| x = 42 | simple assignment in fences |
| result = "hello" | string assignment in fences |
| items = [1, 2, 3] | list assignment in fences |
| def foo(): pass | function definition in fences |
Scenario Outline: Sanitize non-Python prose wraps in docstring
When I sanitize the python content "<prose>"
Then the sanitization error should be None
And the sanitization reason should be "docstring_wrapped"
Examples: Non-Python prose inputs
| prose | description |
| This is just plain English text, not code. | plain English sentence |
| Please implement the feature as described. | instruction-style prose |
| The function should return a list of items. | description-style prose |
Scenario: Sanitize content with null byte returns error
When I sanitize python content containing a null byte
Then the sanitization error should not be None
# ─────────────────────────────────────────────────────────────────────────
# Project name validation — edge cases with special characters
# ─────────────────────────────────────────────────────────────────────────
Scenario Outline: Project rejects names with various special characters
When I try to create a project with name "<invalid_name>"
Then a project validation error should be raised mentioning "alphanumeric"
Examples: Special character edge cases
| invalid_name | description |
| my@project | at-sign |
| my/project | forward slash |
| project! | exclamation mark |
| proj#tag | hash symbol |
| proj%done | percent sign |
| proj^up | caret |
| proj&more | ampersand |
| proj*star | asterisk |
| proj+plus | plus sign |
| proj=eq | equals sign |
| proj<less | less-than sign |
| proj>more | greater-than sign |
| proj?q | question mark |
| proj.pipe | period character |
| proj:colon | colon |
| proj;semi | semicolon |
Scenario: Project rejects empty name
When I try to create a project with empty name
Then a project validation error should be raised
Scenario Outline: Project accepts valid names with allowed characters
When I create a project fixture with name "<valid_name>"
Then the project should be created with that name
Examples: Valid name formats
| valid_name | description |
| simple | plain alphanumeric |
| my-project | hyphenated name |
| my_project | underscored name |
| my project | name with space |
| project123 | name with trailing numbers |
| 123project | name starting with numbers |
| a | single character |
| My Project Name | mixed case with spaces |
# ─────────────────────────────────────────────────────────────────────────
# NamespacedName — special character edge cases
# ─────────────────────────────────────────────────────────────────────────
Scenario Outline: NamespacedName rejects names with special characters in namespace
When I parse the namespaced name "<full_name>" expecting an error
Then a ValueError should be raised
Examples: Special characters in namespace
| full_name | description |
| bad@ns/name | at-sign in namespace |
| bad ns/name | space in namespace |
| bad!ns/name | exclamation in namespace |
| bad#ns/name | hash in namespace |
Scenario Outline: NamespacedName rejects names with special characters in name component
When I parse the namespaced name "<full_name>" expecting an error
Then a ValueError should be raised
Examples: Special characters in name component
| full_name | description |
| ns/bad name | space in name component |
| ns/bad@name | at-sign in name component |
| ns/bad!name | exclamation in name component |
| ns/bad#name | hash in name component |
| ns/bad/extra | extra slash (three components) |
Scenario Outline: NamespacedName accepts names at various boundary lengths
When I parse the namespaced name "<full_name>"
Then the parsed namespace should be "<namespace>"
And the parsed item name should be "<item_name>"
Examples: Boundary length names
| full_name | namespace | item_name | description |
| a/b | a | b | minimum length (1 char each) |
| ab/cd | ab | cd | two chars each |
| abc/def | abc | def | three chars each |
| local/my-tool | local | my-tool | standard short name |
| my-namespace/my-tool-name | my-namespace | my-tool-name | medium length names |
# ─────────────────────────────────────────────────────────────────────────
# ULID validation — boundary and special character edge cases
# ─────────────────────────────────────────────────────────────────────────
Scenario Outline: ULID validation rejects inputs at boundary lengths
When I call _validate_plan_ulid with "<input>"
Then ulid-val the validation should raise ValidationError
And ulid-val the error message should contain "ULID"
Examples: Boundary length inputs
| input | description |
| 01ARZ3NDEKTSV4RRFFQ69G5FA | 25 chars (one short of valid) |
| 01ARZ3NDEKTSV4RRFFQ69G5FAVX | 27 chars (one over valid) |
| A | single character |
| 01ARZ3NDEK | 10 chars (too short) |
Scenario Outline: ULID validation accepts valid ULIDs at boundary values
When I call _validate_plan_ulid with "<ulid>"
Then ulid-val the validation should succeed and return "<ulid>"
Examples: Valid ULIDs at boundary values
| ulid | description |
| 00000000000000000000000000 | all zeros (minimum value) |
| 7ZZZZZZZZZZZZZZZZZZZZZZZZZ | all max chars (maximum value) |
| 01HXM8C2ZK4Q7C2B3F2R4VYV6J | standard valid ULID |
Scenario Outline: ULID validation rejects ULIDs with illegal characters
When I call _validate_plan_ulid with "<input>"
Then ulid-val the validation should raise ValidationError
And ulid-val the error message should contain "ULID"
Examples: Crockford base32 excluded characters
| input | description |
| 01ARZ3NDEKTSV4RRFFQ69G5FII | contains I (excluded from base32) |
| 01ARZ3NDEKTSV4RRFFQ69G5FLL | contains L (excluded from base32) |
| 01ARZ3NDEKTSV4RRFFQ69G5FOO | contains O (excluded from base32) |
| 01ARZ3NDEKTSV4RRFFQ69G5FUU | contains U (excluded from base32) |
| 01ARZ3NDEKTSV4RRFFQ69G5F!@ | contains special characters |
| 01ARZ3NDEKTSV4RRFFQ69G5F-X | contains hyphen |
| 01ARZ3NDEKTSV4RRFFQ69G5F.X | contains period |
# ─────────────────────────────────────────────────────────────────────────
# Skill schema — edge cases for inline tool source values
# ─────────────────────────────────────────────────────────────────────────
Scenario Outline: Skill schema rejects invalid inline tool source values
Given a skill YAML string with an inline tool source "<source>"
When I validate the skill schema expecting failure
Then the skill schema validation should fail
And the skill schema error should mention "custom"
Examples: Invalid source values
| source | description |
| plugin | plugin source (not supported) |
| external | external source (not supported) |
| remote | remote source (not supported) |
| builtin | builtin source (not supported) |
| inline | inline source (not supported) |
| file | file source (not supported) |
# ─────────────────────────────────────────────────────────────────────────
# Skill schema — edge cases for MCP transport values
# ─────────────────────────────────────────────────────────────────────────
Scenario Outline: Skill schema rejects invalid MCP transport values
Given a skill YAML string with MCP transport "<transport>"
When I validate the skill schema expecting failure
Then the skill schema validation should fail
And the skill schema error should mention "transport"
Examples: Invalid transport values
| transport | description |
| grpc | gRPC (not supported) |
| websocket | WebSocket (not supported) |
| http | plain HTTP (not supported) |
| tcp | TCP (not supported) |
| invalid | completely invalid value |
| ftp | FTP (not supported) |
| amqp | AMQP (not supported) |
features/data_variation_namespaced_name.feature
+99
View File
@@ -0,0 +1,99 @@
Feature: NamespacedName validation data variation — Scenario Outline refactoring
As a CleverAgents developer
I want NamespacedName validation tests to cover a wide range of inputs
So that edge cases and boundary conditions are systematically exercised
# ─────────────────────────────────────────────────────────────────────────
# Refactored from plan_namespaced_name_validation.feature using Scenario
# Outline. Each Examples table row exercises a distinct code path.
# ─────────────────────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────
# Valid namespaced names — all should parse successfully
# ─────────────────────────────────────────────────────────────────────────
Scenario Outline: NamespacedName.parse() accepts valid namespaced names
When I parse the namespaced name "<full_name>"
Then the parsed namespace should be "<namespace>"
And the parsed item name should be "<item_name>"
Examples: Standard valid names
| full_name | namespace | item_name | description |
| abc123/my-action | abc123 | my-action | namespace starts with letter |
| local/a123-action | local | a123-action | name starts with letter |
| my-org/my-action | my-org | my-action | kebab-case namespace and name |
| x/y | x | y | single-char namespace and name |
| org/tool-v2 | org | tool-v2 | name with version suffix |
| builtin/read-file | builtin | read-file | builtin namespace |
| local/my-skill | local | my-skill | local namespace |
Examples: Names with numbers (not at start)
| full_name | namespace | item_name | description |
| org123/tool456 | org123 | tool456 | numbers in middle of both parts |
| a1b2c3/d4e5f6 | a1b2c3 | d4e5f6 | alternating letters and numbers |
| ns/name-with-123 | ns | name-with-123 | numbers at end of name |
Examples: Boundary length names
| full_name | namespace | item_name | description |
| a/b | a | b | minimum length (1 char each) |
| ab/cd | ab | cd | two chars each |
| abc/def | abc | def | three chars each |
| my-namespace/my-tool-name | my-namespace | my-tool-name | medium length names |
# ─────────────────────────────────────────────────────────────────────────
# Invalid namespaced names — special characters in namespace
# ─────────────────────────────────────────────────────────────────────────
Scenario Outline: NamespacedName.parse() rejects namespace with special characters
When I parse the namespaced name "<full_name>" expecting an error
Then a ValueError should be raised
Examples: Special characters in namespace
| full_name | description |
| bad@ns/name | at-sign in namespace |
| bad ns/name | space in namespace |
| bad!ns/name | exclamation in namespace |
| bad#ns/name | hash in namespace |
# ─────────────────────────────────────────────────────────────────────────
# Invalid namespaced names — special characters in name component
# ─────────────────────────────────────────────────────────────────────────
Scenario Outline: NamespacedName.parse() rejects name component with special characters
When I parse the namespaced name "<full_name>" expecting an error
Then a ValueError should be raised
Examples: Special characters in name component
| full_name | description |
| ns/bad name | space in name component |
| ns/bad@name | at-sign in name component |
| ns/bad!name | exclamation in name component |
| ns/bad#name | hash in name component |
| ns/bad/extra | extra slash (three components) |
# ─────────────────────────────────────────────────────────────────────────
# Invalid namespaced names — via constructor
# ─────────────────────────────────────────────────────────────────────────
Scenario Outline: NamespacedName constructor rejects invalid namespace formats
When I construct a NamespacedName with namespace "<namespace>" and name "<name>" expecting an error
Then a ValueError should be raised
Examples: Invalid namespace formats
| namespace | name | description |
| bad@ns | valid-name | at-sign in namespace |
| bad ns | valid-name | space in namespace |
| bad!ns | valid-name | exclamation in namespace |
| bad#ns | valid-name | hash in namespace |
Scenario Outline: NamespacedName constructor rejects invalid name formats
When I construct a NamespacedName with namespace "<namespace>" and name "<name>" expecting an error
Then a ValueError should be raised
Examples: Invalid name formats
| namespace | name | description |
| local | bad name | space in name |
| local | bad@name | at-sign in name |
| local | bad!name | exclamation in name |
| local | bad#name | hash in name |
| local | bad/name | slash in name |
features/data_variation_plan_ulid.feature
+132
View File
@@ -0,0 +1,132 @@
Feature: ULID validation data variation — Scenario Outline refactoring
As a CleverAgents developer
I want ULID validation tests to cover a wide range of inputs
So that edge cases and boundary conditions are systematically exercised
# ─────────────────────────────────────────────────────────────────────────
# Refactored from plan_ulid_validation.feature using Scenario Outline.
# Each Examples table row exercises a distinct code path or boundary.
# ─────────────────────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────
# Valid ULID inputs — all should pass validation
# ─────────────────────────────────────────────────────────────────────────
Scenario Outline: _validate_plan_ulid accepts valid ULID strings
When I call _validate_plan_ulid with "<ulid>"
Then ulid-val the validation should succeed and return "<ulid>"
Examples: Valid ULID formats
| ulid | description |
| 01HXM8C2ZK4Q7C2B3F2R4VYV6J | standard uppercase ULID |
| 01hxm8c2zk4q7c2b3f2r4vyv6j | lowercase ULID (case-insensitive) |
| 01ARZ3NDEKTSV4RRFFQ69G5FAV | another valid uppercase ULID |
| 00000000000000000000000000 | minimum value ULID (all zeros) |
| 7ZZZZZZZZZZZZZZZZZZZZZZZZZ | maximum value ULID (all max chars) |
# ─────────────────────────────────────────────────────────────────────────
# Invalid ULID inputs — wrong length
# ─────────────────────────────────────────────────────────────────────────
Scenario Outline: _validate_plan_ulid rejects ULIDs with wrong length
When I call _validate_plan_ulid with "<invalid_input>"
Then ulid-val the validation should raise ValidationError
And ulid-val the error message should contain "ULID"
Examples: Wrong length inputs
| invalid_input | description |
| 01ARZ3NDEK | too short (10 chars) |
| 01ARZ3NDEKTSV4RRFFQ69G5FAAEXTRA | too long (32 chars) |
| 01ARZ3NDEKTSV4RRFFQ69G5FA | too short by one (25 chars) |
| 01ARZ3NDEKTSV4RRFFQ69G5FAVXX | too long by one (27 chars) |
| A | single character |
| 01ARZ3NDEKTSV4RRFFQ69G5FAVXXX | too long by three (28 chars) |
# ─────────────────────────────────────────────────────────────────────────
# Invalid ULID inputs — illegal characters (I, L, O, U excluded from
# Crockford base32 alphabet to avoid visual ambiguity)
# ─────────────────────────────────────────────────────────────────────────
Scenario Outline: _validate_plan_ulid rejects ULIDs with illegal characters
When I call _validate_plan_ulid with "<invalid_input>"
Then ulid-val the validation should raise ValidationError
And ulid-val the error message should contain "ULID"
Examples: Crockford base32 excluded characters
| invalid_input | description |
| 01ARZ3NDEKTSV4RRFFQ69G5FII | contains I (excluded from base32) |
| 01ARZ3NDEKTSV4RRFFQ69G5FLL | contains L (excluded from base32) |
| 01ARZ3NDEKTSV4RRFFQ69G5FOO | contains O (excluded from base32) |
| 01ARZ3NDEKTSV4RRFFQ69G5FUU | contains U (excluded from base32) |
Examples: Special characters and whitespace
| invalid_input | description |
| 01ARZ3NDEKTSV4RRFFQ69G5F!@ | contains special characters |
| 01ARZ3NDEKTSV4RRFFQ69G5F-X | contains hyphen |
| 01ARZ3NDEKTSV4RRFFQ69G5F.X | contains period |
# ─────────────────────────────────────────────────────────────────────────
# Invalid ULID inputs — plain strings and legacy names
# ─────────────────────────────────────────────────────────────────────────
Scenario Outline: _validate_plan_ulid rejects plain strings that are not ULIDs
When I call _validate_plan_ulid with "<invalid_input>"
Then ulid-val the validation should raise ValidationError
And ulid-val the error message should contain "ULID"
Examples: Plain non-ULID strings
| invalid_input | description |
| not-a-ulid | plain hyphenated string |
| hello | short plain word |
| plan-123 | legacy-style plan name |
| abc | very short plain string |
# ─────────────────────────────────────────────────────────────────────────
# Legacy plan names — should include actionable error guidance
# ─────────────────────────────────────────────────────────────────────────
Scenario Outline: _validate_plan_ulid rejects legacy plan names with guidance
When I call _validate_plan_ulid with "<legacy_name>"
Then ulid-val the validation should raise ValidationError
And ulid-val the error message should contain "ULID"
And ulid-val the error message should contain "legacy"
And ulid-val the error message should contain "agents plan use"
Examples: Legacy plan name formats
| legacy_name | description |
| 64-bit port plan | legacy name with spaces |
| my-legacy-plan | legacy hyphenated name |
| add error handling | legacy descriptive name |
| refactor authentication module | legacy multi-word name |
# ─────────────────────────────────────────────────────────────────────────
# CLI command ULID validation — multiple commands, multiple invalid inputs
# ─────────────────────────────────────────────────────────────────────────
Scenario Outline: plan <command> rejects non-ULID input with actionable error
Given a ulid-validation CLI runner
When I invoke ulid-validation plan <command> with "<invalid_input>"
Then the ulid-validation command should abort
And the ulid-validation output should contain "ULID"
Examples: CLI commands with plain non-ULID strings
| command | invalid_input | description |
| execute | not-a-valid-id | execute with plain string |
| apply | not-a-valid-id | apply with plain string |
| execute | short | execute with very short string |
| apply | plan-123 | apply with legacy-style name |
Scenario Outline: plan <command> rejects legacy plan names with full guidance
Given a ulid-validation CLI runner
When I invoke ulid-validation plan <command> with "<legacy_name>"
Then the ulid-validation command should abort
And the ulid-validation output should contain "not found"
And the ulid-validation output should contain "ULID"
And the ulid-validation output should contain "legacy"
And the ulid-validation output should contain "agents plan use"
Examples: CLI commands with legacy plan names
| command | legacy_name | description |
| execute | my-legacy-plan | execute with legacy hyphenated name |
| apply | 64-bit port plan | apply with legacy spaced name |
| execute | add error handling | execute with legacy descriptive name |
features/data_variation_project_name.feature
+85
View File
@@ -0,0 +1,85 @@
Feature: Project name validation data variation — Scenario Outline refactoring
As a CleverAgents developer
I want project name validation tests to cover a wide range of inputs
So that edge cases and boundary conditions are systematically exercised
# ─────────────────────────────────────────────────────────────────────────
# Refactored from validation_test_fixtures.feature (Section 4) using
# Scenario Outline. Each Examples table row exercises a distinct
# validation path or boundary condition.
# ─────────────────────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────
# Invalid project names — special characters
# ─────────────────────────────────────────────────────────────────────────
Scenario Outline: Project rejects names with special characters
When I try to create a project with name "<invalid_name>"
Then a project validation error should be raised mentioning "alphanumeric"
Examples: Names with special characters — punctuation
| invalid_name | description |
| my@project | at-sign in name |
| my/project | forward slash in name |
| project! | exclamation mark in name |
| project#tag | hash symbol in name |
| project%done | percent sign in name |
| project^up | caret in name |
| project&more | ampersand in name |
| project*star | asterisk in name |
| project+plus | plus sign in name |
| project=equals | equals sign in name |
| project.pipe | period in name |
| project:colon | colon in name |
| project;semi | semicolon in name |
| project<less | less-than sign in name |
| project>greater | greater-than sign in name |
| project?question | question mark in name |
# ─────────────────────────────────────────────────────────────────────────
# Invalid project names — empty and whitespace-only
# ─────────────────────────────────────────────────────────────────────────
Scenario: Project rejects empty name
When I try to create a project with empty name
Then a project validation error should be raised
# ─────────────────────────────────────────────────────────────────────────
# Valid project names — accepted formats
# ─────────────────────────────────────────────────────────────────────────
Scenario Outline: Project accepts names with alphanumeric characters, hyphens, underscores, and spaces
When I create a project fixture with name "<valid_name>"
Then the project should be created with that name
Examples: Valid project name formats
| valid_name | description |
| my-project_v2 final | hyphens, underscores, and spaces |
| simple | plain alphanumeric name |
| project123 | name with trailing numbers |
| my project | name with space |
| project-name | hyphenated name |
| project_name | underscored name |
| Project Name | mixed case with space |
| a | single character name |
| abc def ghi | multiple words with spaces |
| project-v1-final | multiple hyphens |
| my_project_2024 | underscores with numbers |
| 123project | name starting with numbers |
| My Project Name | mixed case with multiple spaces |
# ─────────────────────────────────────────────────────────────────────────
# Edge cases — boundary conditions
# ─────────────────────────────────────────────────────────────────────────
Scenario Outline: Project path is always resolved to absolute
When I create a project with relative path "<rel_path>"
Then the project fixture path should be absolute
Examples: Relative path formats
| rel_path | description |
| relative/path | simple relative path |
| ./local/path | dot-relative path |
| ../parent/path | parent-relative path |
| subdir/nested/deep | deeply nested relative path |
| just-a-dir | single directory name |
features/data_variation_skill_name.feature
+140
View File
@@ -0,0 +1,140 @@
Feature: Skill schema name validation data variation — Scenario Outline refactoring
As a CleverAgents developer
I want skill schema name validation tests to cover a wide range of inputs
So that edge cases and boundary conditions are systematically exercised
# ─────────────────────────────────────────────────────────────────────────
# Refactored from skill_schema.feature using Scenario Outline.
# Each Examples table row exercises a distinct validation path.
# ─────────────────────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────
# Invalid skill names — missing or malformed namespace/name format
# ─────────────────────────────────────────────────────────────────────────
Scenario Outline: Skill schema rejects invalid namespaced names
Given a skill YAML string with name "<invalid_name>"
When I validate the skill schema expecting failure
Then the skill schema validation should fail
And the skill schema error should mention "namespace/name"
Examples: Names without slash separator
| invalid_name | description |
| invalid-name-no-slash | plain name without namespace |
| noslash | single word without slash |
| justname | another plain word |
| toolname | tool name without namespace |
| a | single character without namespace |
Examples: Names with special characters
| invalid_name | description |
| local/bad name!! | spaces and exclamation marks |
| local/bad@name | at-sign in name |
| local/bad#name | hash in name |
| local/bad name | space in name component |
| local/bad/extra | extra slash (three components) |
| bad ns/name | space in namespace |
| bad@ns/name | at-sign in namespace |
# ─────────────────────────────────────────────────────────────────────────
# Invalid tool reference names
# ─────────────────────────────────────────────────────────────────────────
Scenario Outline: Skill schema rejects invalid tool reference names
Given a skill YAML string with an invalid tool ref name "<invalid_ref>"
When I validate the skill schema expecting failure
Then the skill schema validation should fail
And the skill schema error should mention "namespace/name"
Examples: Invalid tool reference formats
| invalid_ref | description |
| no-slash-tool | tool ref without namespace |
| toolname | plain tool name |
| bad name | tool ref with space |
| bad@tool | tool ref with special character |
| a | single character tool ref |
# ─────────────────────────────────────────────────────────────────────────
# Invalid MCP transport values
# ─────────────────────────────────────────────────────────────────────────
Scenario Outline: Skill schema rejects invalid MCP transport values
Given a skill YAML string with MCP transport "<transport>"
When I validate the skill schema expecting failure
Then the skill schema validation should fail
And the skill schema error should mention "transport"
Examples: Invalid transport values
| transport | description |
| grpc | gRPC transport (not supported) |
| websocket | WebSocket transport (not supported) |
| http | plain HTTP transport (not supported) |
| tcp | TCP transport (not supported) |
| udp | UDP transport (not supported) |
| invalid | completely invalid transport name |
| ftp | FTP transport (not supported) |
| amqp | AMQP transport (not supported) |
# ─────────────────────────────────────────────────────────────────────────
# Invalid include names
# ─────────────────────────────────────────────────────────────────────────
Scenario Outline: Skill schema rejects invalid include names
Given a skill YAML string with an invalid include name "<invalid_include>"
When I validate the skill schema expecting failure
Then the skill schema validation should fail
And the skill schema error should mention "namespace/name"
Examples: Invalid include name formats
| invalid_include | description |
| bad-include | include without namespace |
| justname | plain name without slash |
| bad include | include with space |
| bad@include | include with special character |
# ─────────────────────────────────────────────────────────────────────────
# Valid skill names — accepted formats
# ─────────────────────────────────────────────────────────────────────────
Scenario Outline: Skill schema accepts valid namespaced names
Given a skill YAML string with name "<valid_name>"
When I validate the skill schema
Then the skill schema validation should succeed
And the skill config name should be "<valid_name>"
Examples: Valid namespaced name formats
| valid_name | description |
| local/empty-skill | standard local namespace |
| builtin/read-file | builtin namespace |
| org/my-skill | organization namespace |
| a/b | minimal single-char namespace and name |
| my-org/my-skill-v2 | hyphenated namespace and versioned name |
| ns123/skill456 | namespace and name with numbers |
# ─────────────────────────────────────────────────────────────────────────
# Edge cases — empty and null inputs
# ─────────────────────────────────────────────────────────────────────────
Scenario: Skill schema rejects None YAML string
Given a None skill YAML string
When I validate the skill schema expecting failure
Then the skill schema validation should fail
And the skill schema error should mention "None"
Scenario: Skill schema rejects empty YAML string
Given an empty skill YAML string
When I validate the skill schema expecting failure
Then the skill schema validation should fail
And the skill schema error should mention "empty"
Scenario: Skill schema rejects YAML that is a list instead of a mapping
Given a skill YAML string that is a list
When I validate the skill schema expecting failure
Then the skill schema validation should fail
And the skill schema error should mention "mapping"
Scenario: Skill schema rejects missing name field
Given a skill YAML string missing the name field
When I validate the skill schema expecting failure
Then the skill schema validation should fail
And the skill schema error should mention "name"