freemo
7b0533b34d
CI / lint (pull_request) Successful in 20s
CI / typecheck (pull_request) Successful in 56s
CI / security (pull_request) Successful in 1m0s
CI / quality (pull_request) Successful in 41s
CI / build (pull_request) Successful in 30s
CI / helm (pull_request) Successful in 23s
CI / unit_tests (pull_request) Successful in 7m15s
CI / e2e_tests (pull_request) Successful in 16m28s
CI / integration_tests (pull_request) Successful in 22m39s
CI / docker (pull_request) Successful in 1m25s
CI / coverage (pull_request) Successful in 11m4s
CI / status-check (pull_request) Successful in 1s
CI / benchmark-publish (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Successful in 57m3s
Enforce lowercase-only namespace/name pattern in both SkillConfigSchema and ActionConfigSchema by updating NAMESPACED_NAME_RE from the permissive [a-zA-Z0-9] pattern to the strict [a-z0-9] pattern. This fixes a UAT-identified bug where uppercase letters were incorrectly accepted in the namespace/name fields of skill and action configurations, violating the spec's naming convention (^[a-z0-9_-]+/[a-z0-9_-]+$). Changes: - src/cleveragents/skills/schema.py: Update NAMESPACED_NAME_RE to reject uppercase letters in namespace and name parts - src/cleveragents/action/schema.py: Apply the same lowercase-only fix - features/skill_schema.feature: Add BDD scenarios for uppercase namespace, uppercase name part, fully uppercase name, uppercase tool ref, and uppercase include name rejection - features/consolidated_action.feature: Add BDD scenarios for uppercase namespace, uppercase name part, and fully uppercase name rejection ISSUES CLOSED: #3029
297 lines
14 KiB
Gherkin
297 lines
14 KiB
Gherkin
Feature: Skill YAML schema validation
|
|
As a CleverAgents user
|
|
I want to define skills via YAML configuration files
|
|
So that I can create validated, reusable tool collections
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Valid schema scenarios
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Load a minimal valid skill YAML (name only)
|
|
Given a skill YAML string with only the name field
|
|
When I validate the skill schema
|
|
Then the skill schema validation should succeed
|
|
And the skill config name should be "local/empty-skill"
|
|
|
|
Scenario: Load a skill YAML with tool references
|
|
Given a skill YAML string with tool references
|
|
When I validate the skill schema
|
|
Then the skill schema validation should succeed
|
|
And the skill config name should be "local/file-reader"
|
|
And the skill config should have 3 tools
|
|
|
|
Scenario: Load a skill YAML with all sections populated
|
|
Given a skill YAML string with all sections populated
|
|
When I validate the skill schema
|
|
Then the skill schema validation should succeed
|
|
And the skill config name should be "local/full-skill"
|
|
And the skill config description should be "A fully populated skill"
|
|
And the skill config should have 2 tools
|
|
And the skill config should have 1 inline tools
|
|
And the skill config should have 1 includes
|
|
And the skill config should have 1 mcp servers
|
|
And the skill config should have 1 agent skill folders
|
|
|
|
Scenario: Load the single-tool example YAML
|
|
Given the skill YAML file "examples/skills/single-tool.yaml"
|
|
When I validate the skill schema from file
|
|
Then the skill schema validation should succeed
|
|
|
|
Scenario: Load the composed example YAML
|
|
Given the skill YAML file "examples/skills/composed.yaml"
|
|
When I validate the skill schema from file
|
|
Then the skill schema validation should succeed
|
|
|
|
Scenario: Load the inline-tool example YAML
|
|
Given the skill YAML file "examples/skills/inline-tool.yaml"
|
|
When I validate the skill schema from file
|
|
Then the skill schema validation should succeed
|
|
|
|
Scenario: Load the validation-only example YAML
|
|
Given the skill YAML file "examples/skills/validation-only.yaml"
|
|
When I validate the skill schema from file
|
|
Then the skill schema validation should succeed
|
|
|
|
Scenario: Load the mcp-backed example YAML
|
|
Given the skill YAML file "examples/skills/mcp-backed.yaml"
|
|
When I validate the skill schema from file
|
|
Then the skill schema validation should succeed
|
|
|
|
Scenario: Optional fields have correct defaults when omitted
|
|
Given a skill YAML string with only the name field
|
|
When I validate the skill schema
|
|
Then the skill schema validation should succeed
|
|
And the skill config tools list should be empty
|
|
And the skill config inline_tools list should be empty
|
|
And the skill config includes list should be empty
|
|
And the skill config mcp_servers list should be empty
|
|
And the skill config agent_skill_folders list should be empty
|
|
|
|
Scenario: Tool reference with override fields
|
|
Given a skill YAML string with tool overrides
|
|
When I validate the skill schema
|
|
Then the skill schema validation should succeed
|
|
And skill tool 0 name should be "builtin/shell_execute"
|
|
And skill tool 0 description should be "Execute commands in sandbox"
|
|
And skill tool 0 writes should be true
|
|
And skill tool 0 checkpointable should be false
|
|
|
|
Scenario: Inline tool with all fields
|
|
Given a skill YAML string with a full inline tool
|
|
When I validate the skill schema
|
|
Then the skill schema validation should succeed
|
|
And skill inline tool 0 name should be "run_migrations"
|
|
And skill inline tool 0 source should be "custom"
|
|
And skill inline tool 0 writes should be true
|
|
And skill inline tool 0 checkpointable should be true
|
|
|
|
Scenario: MCP server with stdio transport
|
|
Given a skill YAML string with a stdio MCP server
|
|
When I validate the skill schema
|
|
Then the skill schema validation should succeed
|
|
And skill mcp server 0 name should be "github"
|
|
And skill mcp server 0 transport should be "stdio"
|
|
And skill mcp server 0 command should be "npx"
|
|
|
|
Scenario: MCP server with SSE transport
|
|
Given a skill YAML string with an SSE MCP server
|
|
When I validate the skill schema
|
|
Then the skill schema validation should succeed
|
|
And skill mcp server 0 transport should be "sse"
|
|
And skill mcp server 0 url should be "https://mcp.example.com/sse"
|
|
|
|
Scenario: MCP server with tool filter
|
|
Given a skill YAML string with MCP tool filter
|
|
When I validate the skill schema
|
|
Then the skill schema validation should succeed
|
|
And skill mcp server 0 tool filter include should have 2 items
|
|
And skill mcp server 0 tool filter exclude should have 1 items
|
|
|
|
Scenario: Agent skill folder entries
|
|
Given a skill YAML string with agent skill folders
|
|
When I validate the skill schema
|
|
Then the skill schema validation should succeed
|
|
And the skill config should have 2 agent skill folders
|
|
And skill agent folder 0 path should be "./skills/deploy"
|
|
And skill agent folder 1 name should be "code-review"
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Invalid schema scenarios
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Missing name field produces a clear error
|
|
Given a skill YAML string missing the name field
|
|
When I validate the skill schema expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "name"
|
|
|
|
Scenario: Invalid namespaced name without slash
|
|
Given a skill YAML string with name "invalid-name-no-slash"
|
|
When I validate the skill schema expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "namespace/name"
|
|
|
|
Scenario: Invalid namespaced name with special characters
|
|
Given a skill YAML string with name "local/bad name!!"
|
|
When I validate the skill schema expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "namespace/name"
|
|
|
|
Scenario: Uppercase namespace is rejected by SkillConfigSchema
|
|
Given a skill YAML string with name "MyOrg/my-skill"
|
|
When I validate the skill schema expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "namespace/name"
|
|
|
|
Scenario: Uppercase name part is rejected by SkillConfigSchema
|
|
Given a skill YAML string with name "myorg/MySkill"
|
|
When I validate the skill schema expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "namespace/name"
|
|
|
|
Scenario: Fully uppercase namespaced name is rejected by SkillConfigSchema
|
|
Given a skill YAML string with name "MyOrg/MySkill"
|
|
When I validate the skill schema expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "namespace/name"
|
|
|
|
Scenario: Uppercase tool reference name is rejected by SkillToolRefSchema
|
|
Given a skill YAML string with an invalid tool ref name "MyOrg/MyTool"
|
|
When I validate the skill schema expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "namespace/name"
|
|
|
|
Scenario: Uppercase include name is rejected by SkillIncludeSchema
|
|
Given a skill YAML string with an invalid include name "MyOrg/MySkill"
|
|
When I validate the skill schema expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "namespace/name"
|
|
|
|
Scenario: Invalid tool reference name
|
|
Given a skill YAML string with an invalid tool ref name "no-slash-tool"
|
|
When I validate the skill schema expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "namespace/name"
|
|
|
|
Scenario: Inline tool missing source field
|
|
Given a skill YAML string with an inline tool missing source
|
|
When I validate the skill schema expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "source"
|
|
|
|
Scenario: Inline tool missing code field
|
|
Given a skill YAML string with an inline tool missing code
|
|
When I validate the skill schema expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "code"
|
|
|
|
Scenario: Inline tool with invalid source value
|
|
Given a skill YAML string with an inline tool source "plugin"
|
|
When I validate the skill schema expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "custom"
|
|
|
|
Scenario: Invalid include name
|
|
Given a skill YAML string with an invalid include name "bad-include"
|
|
When I validate the skill schema expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "namespace/name"
|
|
|
|
Scenario: Invalid MCP transport value
|
|
Given a skill YAML string with MCP transport "grpc"
|
|
When I validate the skill schema expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "transport"
|
|
|
|
Scenario: MCP server missing transport field
|
|
Given a skill YAML string with MCP server missing transport
|
|
When I validate the skill schema expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "transport"
|
|
|
|
Scenario: Agent skill folder missing path
|
|
Given a skill YAML string with agent folder missing path
|
|
When I validate the skill schema expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "path"
|
|
|
|
Scenario: Extra unknown field is rejected
|
|
Given a skill YAML string with an unknown top-level field
|
|
When I validate the skill schema expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "extra"
|
|
|
|
Scenario: None YAML string raises ValueError
|
|
Given a None skill YAML string
|
|
When I validate the skill schema expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "None"
|
|
|
|
Scenario: Empty YAML string raises ValueError
|
|
Given an empty skill YAML string
|
|
When I validate the skill schema expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "empty"
|
|
|
|
Scenario: YAML that is a list instead of a mapping
|
|
Given a skill YAML string that is a list
|
|
When I validate the skill schema expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "mapping"
|
|
|
|
Scenario: None file path raises ValueError
|
|
Given a None skill YAML file path
|
|
When I validate the skill schema from file expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "None"
|
|
|
|
Scenario: Non-existent file raises FileNotFoundError
|
|
Given the skill YAML file "examples/skills/nonexistent.yaml"
|
|
When I validate the skill schema from file expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "not found"
|
|
|
|
Scenario: Directory path raises ValueError
|
|
Given the skill YAML directory path "examples/skills"
|
|
When I validate the skill schema from file expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "not a file"
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Key normalization scenarios
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: camelCase keys are normalized to snake_case
|
|
Given a skill YAML string with camelCase keys
|
|
When I validate the skill schema
|
|
Then the skill schema validation should succeed
|
|
And the skill config should have 1 inline tools
|
|
And the skill config should have 1 mcp servers
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Environment variable interpolation
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Environment variable interpolation in YAML values
|
|
Given the environment variable "TEST_MCP_CMD" is set to "npx"
|
|
And a skill YAML string using env var "${TEST_MCP_CMD}" for mcp command
|
|
When I validate the skill schema
|
|
Then the skill schema validation should succeed
|
|
And skill mcp server 0 command should be "npx"
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Round-trip serialization
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Schema model serializes to dict correctly
|
|
Given a skill YAML string with all sections populated
|
|
When I validate the skill schema
|
|
Then the skill schema validation should succeed
|
|
And the skill config model_dump should contain key "name"
|
|
And the skill config model_dump should contain key "tools"
|
|
And the skill config model_dump should contain key "inline_tools"
|
|
And the skill config model_dump should contain key "includes"
|
|
And the skill config model_dump should contain key "mcp_servers"
|
|
And the skill config model_dump should contain key "agent_skill_folders"
|
|
|