Files
cleveragents-core/features/agent_skills_loader.feature
aditya cb82fc51df feat(skill): add agent skills loader
Implemented AgentSkillSpec loader that parses SKILL.md frontmatter and
progressive disclosure sections (discover/activate/deactivate) into
structured SkillStep objects with stable 1-based ordering.

Mapped Agent Skills to AgentSkillToolDescriptor with namespaced naming
(namespace/short_name), source="agent_skill", read-only defaults, and
AgentSkillResourceSlot bindings for scripts/, references/, and assets/
directories. All resource slots are unconditionally read_only.

Added explicit validation for missing frontmatter fields (name,
description) and invalid namespace format with actionable error messages.

Added docs/reference/agent_skills.md covering folder layout, SKILL.md
parsing rules, progressive disclosure model, and tool mapping.

Added Behave scenarios covering valid/invalid SKILL.md parsing,
namespaced naming, step ordering, missing frontmatter errors, progressive
disclosure lifecycle, tool mapping, and resource binding slots.

Added Robot Framework integration tests using the deploy-to-staging
example skill folder (robot/agent_skills_loader.robot).

Added ASV benchmarks for parsing throughput, folder load, progressive
disclosure lifecycle, and resource listing
(benchmarks/agent_skills_loader_bench.py).

ISSUES CLOSED: #160
2026-02-26 09:10:35 +00:00

696 lines
24 KiB
Gherkin

@phase2 @skills @agent_skills @agent_skills_loader
Feature: Agent Skills Loader
As a CleverAgents developer
I want to load Agent Skills Standard SKILL.md folders into structured tool definitions
So that actors can discover and invoke instruction-driven multi-step workflows
# ---------------------------------------------------------------------------
# AgentSkillSpec SKILL.md Parsing
# ---------------------------------------------------------------------------
@agent_skill_parse
Scenario: Parse a minimal SKILL.md with only required frontmatter fields
Given a SKILL.md file with frontmatter:
"""
---
name: local/deploy-staging
description: Deploy the current branch to the staging environment.
---
Follow these steps to deploy.
"""
When I parse the SKILL.md file
Then the parsed skill name should be "local/deploy-staging"
And the parsed skill description should be "Deploy the current branch to the staging environment."
And the parsed skill body should contain "Follow these steps to deploy."
@agent_skill_parse
Scenario: Parse a SKILL.md with all optional frontmatter fields
Given a SKILL.md file with frontmatter:
"""
---
name: local/code-review
description: Run a thorough code review checklist.
version: 1.2.3
compatibility:
min_agent_version: "3.0.0"
metadata:
author: aditya
tags: [review, quality]
allowed-tools:
- local/read-file
- local/search-files
---
## Code Review Steps
1. Check style
2. Verify tests
"""
When I parse the SKILL.md file
Then the parsed skill name should be "local/code-review"
And the parsed skill version should be "1.2.3"
And the parsed skill allowed tools should include "local/read-file"
And the parsed skill allowed tools should include "local/search-files"
And the parsed skill metadata should have key "author" with value "aditya"
@agent_skill_parse
Scenario: Parse SKILL.md body text is preserved exactly
Given a SKILL.md file with frontmatter:
"""
---
name: local/test-skill
description: Test skill for body parsing.
---
# Multi-line Instructions
Step 1: Do something important.
Step 2: Do another thing.
```python
code_example()
```
"""
When I parse the SKILL.md file
Then the parsed skill body should contain "Step 1: Do something important."
And the parsed skill body should contain "Step 2: Do another thing."
And the parsed skill body should contain "code_example()"
@agent_skill_parse @error_handling
Scenario: SKILL.md missing required name field raises error
Given a SKILL.md file with frontmatter:
"""
---
description: A skill without a name.
---
Some body text.
"""
When I parse the SKILL.md file expecting an error
Then the agent skill parse error should mention "name"
@agent_skill_parse @error_handling
Scenario: SKILL.md missing required description field raises error
Given a SKILL.md file with frontmatter:
"""
---
name: local/no-desc
---
Some body text.
"""
When I parse the SKILL.md file expecting an error
Then the agent skill parse error should mention "description"
@agent_skill_parse @error_handling
Scenario: SKILL.md with invalid namespaced name raises error
Given a SKILL.md file with frontmatter:
"""
---
name: invalid-name-no-slash
description: A skill with a bad name.
---
Body text.
"""
When I parse the SKILL.md file expecting an error
Then the agent skill parse error should mention "namespace"
@agent_skill_parse @error_handling
Scenario: File without YAML frontmatter delimiter raises error
Given a SKILL.md file with frontmatter:
"""
Just plain text without frontmatter.
No YAML delimiter here.
"""
When I parse the SKILL.md file expecting an error
Then the agent skill parse error should mention "frontmatter"
@agent_skill_parse @error_handling
Scenario: Empty SKILL.md file raises error
Given an empty SKILL.md file
When I parse the SKILL.md file expecting an error
Then the agent skill parse error should mention "frontmatter"
# ---------------------------------------------------------------------------
# AgentSkillFolder — Folder Discovery
# ---------------------------------------------------------------------------
@agent_skill_folder
Scenario: Discover a valid Agent Skills folder with SKILL.md
Given an agent skills folder with a valid SKILL.md:
"""
---
name: local/folder-skill
description: A skill loaded from a folder.
---
Instructions here.
"""
When I load the agent skills folder
Then the loaded agent skill name should be "local/folder-skill"
And the loaded agent skill description should be "A skill loaded from a folder."
@agent_skill_folder
Scenario: Agent Skills folder discovers scripts/ subfolder
Given an agent skills folder with a valid SKILL.md:
"""
---
name: local/scripted-skill
description: A skill with bundled scripts.
---
Use the scripts provided.
"""
And the folder contains a script file "deploy.py" with content "print('deploying')"
When I load the agent skills folder
Then the loaded agent skill script paths should include "deploy.py"
@agent_skill_folder
Scenario: Agent Skills folder discovers references/ subfolder
Given an agent skills folder with a valid SKILL.md:
"""
---
name: local/ref-skill
description: A skill with reference documents.
---
See references/ for more.
"""
And the folder contains a reference file "guide.md" with content "# Guide"
When I load the agent skills folder
Then the loaded agent skill reference paths should include "guide.md"
@agent_skill_folder
Scenario: Agent Skills folder discovers assets/ subfolder
Given an agent skills folder with a valid SKILL.md:
"""
---
name: local/asset-skill
description: A skill with assets.
---
See assets/ for resources.
"""
And the folder contains an asset file "template.json" with content "{}"
When I load the agent skills folder
Then the loaded agent skill asset paths should include "template.json"
@agent_skill_folder
Scenario: Agent Skills folder with no optional subfolders loads successfully
Given an agent skills folder with a valid SKILL.md:
"""
---
name: local/bare-skill
description: A minimal folder skill with no subfolders.
---
Minimal instructions.
"""
When I load the agent skills folder
Then the loaded agent skill script paths should be empty
And the loaded agent skill reference paths should be empty
And the loaded agent skill asset paths should be empty
@agent_skill_folder @error_handling
Scenario: Loading a folder without SKILL.md raises error
Given an agent skills folder without a SKILL.md file
When I load the agent skills folder expecting an error
Then the agent skill load error should mention "SKILL.md"
@agent_skill_folder @error_handling
Scenario: Loading a non-existent folder raises error
Given a non-existent agent skills folder path
When I load the agent skills folder expecting an error
Then the agent skill load error should mention "not found"
# ---------------------------------------------------------------------------
# Tool Mapping — AgentSkillSpec → ToolDescriptor
# ---------------------------------------------------------------------------
@agent_skill_tool_mapping
Scenario: Agent skill maps to a tool with source agent_skill
Given an agent skills folder with a valid SKILL.md:
"""
---
name: local/mapped-skill
description: Maps to a tool descriptor.
---
Do the task.
"""
When I load the agent skills folder
And I convert the loaded agent skill to a tool descriptor
Then the tool descriptor source should be "agent_skill"
And the tool descriptor name should be "local/mapped-skill"
@agent_skill_tool_mapping
Scenario: Agent skill tool descriptor is read-only by default
Given an agent skills folder with a valid SKILL.md:
"""
---
name: local/readonly-skill
description: Should default to read-only.
---
Read things.
"""
When I load the agent skills folder
And I convert the loaded agent skill to a tool descriptor
Then the tool descriptor should have read_only set to True
And the tool descriptor should have writes set to False
@agent_skill_tool_mapping
Scenario: Agent skill tool descriptor includes allowed-tools in metadata
Given an agent skills folder with a valid SKILL.md:
"""
---
name: local/allowed-tools-skill
description: Skill with allowed tools declared.
allowed-tools:
- local/read-file
- local/git-status
---
Use allowed tools only.
"""
When I load the agent skills folder
And I convert the loaded agent skill to a tool descriptor
Then the tool descriptor allowed tools should include "local/read-file"
And the tool descriptor allowed tools should include "local/git-status"
@agent_skill_tool_mapping
Scenario: Agent skill tool descriptor agent_skill_path points to folder
Given an agent skills folder with a valid SKILL.md:
"""
---
name: local/path-skill
description: Path should be captured in descriptor.
---
Instructions.
"""
When I load the agent skills folder
And I convert the loaded agent skill to a tool descriptor
Then the tool descriptor agent_skill_path should be set
# ---------------------------------------------------------------------------
# Progressive Disclosure — Three-Tier Model
# ---------------------------------------------------------------------------
@progressive_disclosure
Scenario: Discovery tier loads only metadata (name + description)
Given an agent skills folder with a valid SKILL.md:
"""
---
name: local/disclosure-skill
description: Progressive disclosure test skill.
---
Full instructions that should only load on activate.
Step 1: do this.
Step 2: do that.
"""
When I call discover on the agent skill
Then the discover result should have name "local/disclosure-skill"
And the discover result should have description "Progressive disclosure test skill."
And the discover result body should be empty
@progressive_disclosure
Scenario: Activation tier loads the full SKILL.md body
Given an agent skills folder with a valid SKILL.md:
"""
---
name: local/activate-skill
description: Activation loads the body.
---
Step 1: do this.
Step 2: do that.
"""
When I call discover on the agent skill
And I call activate on the agent skill
Then the activate result body should contain "Step 1: do this."
And the activate result body should contain "Step 2: do that."
@progressive_disclosure
Scenario: Resource tier lists available scripts on demand
Given an agent skills folder with a valid SKILL.md:
"""
---
name: local/resource-skill
description: Resource tier test skill.
---
Use the scripts.
"""
And the folder contains a script file "helper.py" with content "# helper"
When I call discover on the agent skill
And I list available resources
Then the resources list should include "helper.py"
@progressive_disclosure
Scenario: Deactivation clears the loaded body
Given an agent skills folder with a valid SKILL.md:
"""
---
name: local/deactivate-skill
description: Deactivation clears body.
---
Step 1: do this.
"""
When I call discover on the agent skill
And I call activate on the agent skill
And I call deactivate on the agent skill
Then the agent skill body should be cleared
# ---------------------------------------------------------------------------
# Namespaced Naming
# ---------------------------------------------------------------------------
@agent_skill_naming
Scenario: Skill with local namespace is accepted
Given a SKILL.md file with frontmatter:
"""
---
name: local/my-skill
description: A valid local skill.
---
Body.
"""
When I parse the SKILL.md file
Then the parsed skill name should be "local/my-skill"
@agent_skill_naming
Scenario: Skill with custom namespace is accepted
Given a SKILL.md file with frontmatter:
"""
---
name: devops/deploy-pipeline
description: A skill in the devops namespace.
---
Body.
"""
When I parse the SKILL.md file
Then the parsed skill name should be "devops/deploy-pipeline"
@agent_skill_naming
Scenario: Skill name with underscores and hyphens is accepted
Given a SKILL.md file with frontmatter:
"""
---
name: local/my_complex-skill_v2
description: Valid name with mixed characters.
---
Body.
"""
When I parse the SKILL.md file
Then the parsed skill name should be "local/my_complex-skill_v2"
@agent_skill_naming @error_handling
Scenario: Skill name with special characters is rejected
Given a SKILL.md file with frontmatter:
"""
---
name: local/bad name!
description: Invalid name.
---
Body.
"""
When I parse the SKILL.md file expecting an error
Then the agent skill parse error should mention "namespace"
# ---------------------------------------------------------------------------
# Stable Step Ordering
# ---------------------------------------------------------------------------
@agent_skill_ordering
Scenario: Steps in SKILL.md body are preserved in stable order
Given a SKILL.md file with frontmatter:
"""
---
name: local/ordered-skill
description: Ordered steps skill.
---
Step A: First action.
Step B: Second action.
Step C: Third action.
"""
When I parse the SKILL.md file
Then the parsed skill body steps should appear in order: "Step A", "Step B", "Step C"
# ---------------------------------------------------------------------------
# Folder path normalization and safety
# ---------------------------------------------------------------------------
@agent_skill_safety
Scenario: Script paths are normalized to relative paths within the folder
Given an agent skills folder with a valid SKILL.md:
"""
---
name: local/safe-skill
description: Path normalization test.
---
Use scripts.
"""
And the folder contains a script file "run.py" with content "# run"
When I load the agent skills folder
Then all script paths should be relative to the skill folder
# ---------------------------------------------------------------------------
# Edge case coverage — branch and statement completeness
# ---------------------------------------------------------------------------
@agent_skill_parse @error_handling @coverage
Scenario: AgentSkillSpec.from_file on a non-existent path raises FileNotFoundError
Given a non-existent SKILL.md path
When I parse the non-existent SKILL.md file expecting an error
Then the agent skill parse error should mention "SKILL.md"
@agent_skill_parse @error_handling @coverage
Scenario: SKILL.md with unclosed frontmatter delimiter raises error
Given a SKILL.md file with frontmatter:
"""
---
name: local/unclosed-skill
description: This frontmatter is never closed.
"""
When I parse the SKILL.md file expecting an error
Then the agent skill parse error should mention "frontmatter"
@agent_skill_parse @error_handling @coverage
Scenario: SKILL.md with empty frontmatter block raises error
Given a SKILL.md with an empty frontmatter block
When I parse the SKILL.md file expecting an error
Then the agent skill parse error should mention "frontmatter"
@agent_skill_parse @error_handling @coverage
Scenario: SKILL.md with malformed YAML in frontmatter raises error
Given a SKILL.md with malformed YAML frontmatter
When I parse the SKILL.md file expecting an error
Then the agent skill parse error should mention "YAML"
@agent_skill_parse @error_handling @coverage
Scenario: SKILL.md with non-mapping YAML frontmatter raises error
Given a SKILL.md with a YAML list as frontmatter instead of a mapping
When I parse the SKILL.md file expecting an error
Then the agent skill parse error should mention "mapping"
@agent_skill_folder @coverage
Scenario: AgentSkillLoader folder property returns resolved path
Given an agent skills folder with a valid SKILL.md:
"""
---
name: local/prop-skill
description: Property access test skill.
---
Body text.
"""
When I load the agent skills folder
Then the loaded agent skill folder property should be set
@agent_skill_folder @error_handling @coverage
Scenario: Loading a path that is a file not a directory raises error
Given an agent skills folder path that is a file
When I load the agent skills folder expecting an error
Then the agent skill load error should mention "not a directory"
@agent_skill_parse @coverage
Scenario: SKILL.md with non-dict compatibility is handled gracefully
Given a SKILL.md file with frontmatter:
"""
---
name: local/bad-compat
description: Skill with non-dict compatibility field.
compatibility: "just a string"
---
Body text.
"""
When I parse the SKILL.md file
Then the parsed skill name should be "local/bad-compat"
@agent_skill_parse @coverage
Scenario: SKILL.md with non-dict metadata is handled gracefully
Given a SKILL.md file with frontmatter:
"""
---
name: local/bad-meta
description: Skill with non-dict metadata field.
metadata: "just a string"
---
Body text.
"""
When I parse the SKILL.md file
Then the parsed skill name should be "local/bad-meta"
@agent_skill_folder @coverage
Scenario: Sub-directories inside scripts/ are skipped during discovery
Given an agent skills folder with a valid SKILL.md:
"""
---
name: local/nested-skill
description: Skill with nested directories in scripts/.
---
Use scripts.
"""
And the folder contains a script file "run.py" with content "# run"
And the scripts folder contains a nested sub-directory
When I load the agent skills folder
Then the loaded agent skill script paths should include "run.py"
And the loaded agent skill script paths should not include any directories
@agent_skill_loader @error_handling @coverage
Scenario: AgentSkillLoader constructor rejects None spec
When I construct an AgentSkillLoader with None spec expecting an error
Then the agent skill load error should mention "spec"
@agent_skill_loader @error_handling @coverage
Scenario: AgentSkillLoader constructor rejects None folder
When I construct an AgentSkillLoader with None folder expecting an error
Then the agent skill load error should mention "folder"
@agent_skill_folder @error_handling @coverage
Scenario: AgentSkillLoader.from_folder rejects None path
When I call from_folder with None expecting an error
Then the agent skill load error should mention "folder"
# ---------------------------------------------------------------------------
# Structured Steps — frontmatter steps list → SkillStep objects
# ---------------------------------------------------------------------------
@agent_skill_steps
Scenario: SKILL.md with steps list in frontmatter parses into SkillStep objects
Given a SKILL.md file with frontmatter:
"""
---
name: local/step-skill
description: Skill with structured steps.
steps:
- Run tests to verify everything passes
- Push branch to remote repository
- Deploy to staging environment
---
Detailed instructions go here.
"""
When I parse the SKILL.md file
Then the parsed skill steps should have 3 entries
And the parsed skill step 1 content should be "Run tests to verify everything passes"
And the parsed skill step 2 content should be "Push branch to remote repository"
And the parsed skill step 3 content should be "Deploy to staging environment"
@agent_skill_steps
Scenario: Structured steps maintain stable 1-based index ordering
Given a SKILL.md file with frontmatter:
"""
---
name: local/ordered-steps-skill
description: Ordered structured steps.
steps:
- Alpha step
- Beta step
- Gamma step
---
Body text.
"""
When I parse the SKILL.md file
Then the parsed skill step 1 index should be 1
And the parsed skill step 2 index should be 2
And the parsed skill step 3 index should be 3
@agent_skill_steps
Scenario: SKILL.md without steps field has empty steps list
Given a SKILL.md file with frontmatter:
"""
---
name: local/no-steps-skill
description: Skill without steps field.
---
Body text.
"""
When I parse the SKILL.md file
Then the parsed skill steps should have 0 entries
@agent_skill_steps @error_handling
Scenario: SKILL.md with steps as a non-list raises actionable error
Given a SKILL.md file with frontmatter:
"""
---
name: local/bad-steps-skill
description: Skill with invalid steps field type.
steps: "just a string instead of a list"
---
Body text.
"""
When I parse the SKILL.md file expecting an error
Then the agent skill parse error should mention "steps"
@agent_skill_steps @error_handling
Scenario: SKILL.md with an empty step entry raises actionable error
Given a SKILL.md file with frontmatter:
"""
---
name: local/empty-step-skill
description: Skill with an empty step in the list.
steps:
- First valid step
-
---
Body text.
"""
When I parse the SKILL.md file expecting an error
Then the agent skill parse error should mention "steps"
# ---------------------------------------------------------------------------
# Resource Binding Slots — read-only slots for discovered directories
# ---------------------------------------------------------------------------
@agent_skill_resource_slots
Scenario: Tool descriptor includes resource binding slots for discovered directories
Given an agent skills folder with a valid SKILL.md:
"""
---
name: local/slotted-skill
description: Skill with resource directories.
---
Use scripts and references.
"""
And the folder contains a script file "run.py" with content "# run"
And the folder contains a reference file "guide.md" with content "# Guide"
When I load the agent skills folder
And I convert the loaded agent skill to a tool descriptor
Then the tool descriptor should have 2 resource slot(s)
And the tool descriptor resource slot "scripts" should have access "read_only"
And the tool descriptor resource slot "references" should have access "read_only"
@agent_skill_resource_slots
Scenario: Tool descriptor with no resource directories has empty resource slots
Given an agent skills folder with a valid SKILL.md:
"""
---
name: local/no-slots-skill
description: Skill with no resource directories.
---
Instructions.
"""
When I load the agent skills folder
And I convert the loaded agent skill to a tool descriptor
Then the tool descriptor should have 0 resource slot(s)
And the loaded skill resource_slots property should be empty
@agent_skill_resource_slots
Scenario: Discover descriptor includes resource binding slots
Given an agent skills folder with a valid SKILL.md:
"""
---
name: local/discover-slots-skill
description: Skill exposing slots at discover time.
---
Instructions.
"""
And the folder contains an asset file "config.json" with content "{}"
When I call discover on the agent skill
Then the discover result should have 1 resource slot(s)
And the discover result resource slot "assets" should have access "read_only"