BUG-HUNT: [testing] Missing test coverage for agent-skills deployment workflow and error scenarios #7307

Open
opened 2026-04-10 15:57:37 +00:00 by HAL9000 · 3 comments
Owner

Metadata

  • Branch: task/test-infra-agent-skills-workflow-coverage
  • Commit Message: test(examples): add comprehensive test coverage for agent-skills deployment workflow and error scenarios
  • Milestone: (none — backlog, see note below)
  • Parent Epic: #5407

Background and Context

The examples/agent-skills/ module lacks comprehensive test coverage for deployment workflow validation, error scenarios, and integration with the broader skill system. While there are existing tests for YAML schema validation (features/actor_examples.feature, robot/actor_examples.robot, robot/skill_schema.robot), there are no tests for the actual skill execution workflow or error handling paths.

The examples/agent-skills/deploy-to-staging/ directory contains three key components:

  • SKILL.md — skill instruction document
  • scripts/deploy.py — deployment script
  • references/runbook.md — operational runbook

None of these components have tests verifying their correctness, integration, or error handling behaviour.

Current Behaviour

  • Tests focus exclusively on YAML schema validation
  • deploy.py has no test coverage for execution paths or error handling
  • No tests verify SKILL.md instruction compliance or completeness
  • No integration tests validate the complete skill workflow (SKILL.md → deploy.py → runbook.md)
  • No tests for error scenarios: missing dependencies, invalid configurations, failed deployments
  • No tests for health check validation workflows
  • No tests for skill loading and tool availability verification

Expected Behaviour

The test suite should comprehensively cover agent skill workflows including:

  • Skill loading and initialization
  • Tool availability validation
  • Deployment script execution paths (success and failure)
  • Error handling and recovery scenarios
  • Integration between SKILL.md, deploy.py, and runbook.md components
  • Health check integration workflows

Evidence

  1. Existing test files found: actor_examples.feature, actor_examples.robot, skill_schema.robot
  2. Tests focus on YAML schema validation only — not workflow execution
  3. No tests verify deploy.py functionality or error paths
  4. No tests validate SKILL.md instruction compliance
  5. No integration tests for complete skill workflow end-to-end

Impact

  • Deployment workflow bugs may go undetected until production
  • Error handling defects not caught by test suite
  • Skill system integration issues not validated
  • Regression risk for skill functionality changes
  • Users may encounter untested edge cases in deployment scenarios

Subtasks

  • Add Behave unit tests (features/) for deploy.py error scenarios (missing dependencies, invalid configurations, failed deployments)
  • Add Behave unit tests for SKILL.md instruction validation and completeness checks
  • Add Robot Framework integration tests (robot/) for complete skill workflow execution
  • Add Robot Framework tests for tool availability verification during skill loading
  • Add Robot Framework tests for health check integration workflows
  • Add tests for the integration between SKILL.md, deploy.py, and runbook.md
  • Ensure coverage meets ≥97% threshold for the examples/ module

Definition of Done

  • Behave unit tests added for deploy.py error scenarios and validation paths
  • Behave unit tests added for SKILL.md instruction compliance
  • Robot Framework integration tests added for complete skill deployment workflow
  • Robot Framework tests added for tool availability and health check validation
  • Integration tests cover the SKILL.md → deploy.py → runbook.md workflow chain
  • All new tests follow BDD/Gherkin style (Behave) or Robot Framework conventions
  • All nox stages pass
  • Coverage >= 97%

Backlog note: This issue was discovered during autonomous operation
on milestone v3.8.0. It does not block milestone completion and has been
placed in the backlog for human review and future milestone assignment.


Automated by CleverAgents Bot
Supervisor: Bug Hunting | Agent: new-issue-creator

## Metadata - **Branch**: `task/test-infra-agent-skills-workflow-coverage` - **Commit Message**: `test(examples): add comprehensive test coverage for agent-skills deployment workflow and error scenarios` - **Milestone**: *(none — backlog, see note below)* - **Parent Epic**: #5407 ## Background and Context The `examples/agent-skills/` module lacks comprehensive test coverage for deployment workflow validation, error scenarios, and integration with the broader skill system. While there are existing tests for YAML schema validation (`features/actor_examples.feature`, `robot/actor_examples.robot`, `robot/skill_schema.robot`), there are no tests for the actual skill execution workflow or error handling paths. The `examples/agent-skills/deploy-to-staging/` directory contains three key components: - `SKILL.md` — skill instruction document - `scripts/deploy.py` — deployment script - `references/runbook.md` — operational runbook None of these components have tests verifying their correctness, integration, or error handling behaviour. ## Current Behaviour - Tests focus exclusively on YAML schema validation - `deploy.py` has no test coverage for execution paths or error handling - No tests verify `SKILL.md` instruction compliance or completeness - No integration tests validate the complete skill workflow (SKILL.md → deploy.py → runbook.md) - No tests for error scenarios: missing dependencies, invalid configurations, failed deployments - No tests for health check validation workflows - No tests for skill loading and tool availability verification ## Expected Behaviour The test suite should comprehensively cover agent skill workflows including: - Skill loading and initialization - Tool availability validation - Deployment script execution paths (success and failure) - Error handling and recovery scenarios - Integration between SKILL.md, deploy.py, and runbook.md components - Health check integration workflows ## Evidence 1. Existing test files found: `actor_examples.feature`, `actor_examples.robot`, `skill_schema.robot` 2. Tests focus on YAML schema validation only — not workflow execution 3. No tests verify `deploy.py` functionality or error paths 4. No tests validate `SKILL.md` instruction compliance 5. No integration tests for complete skill workflow end-to-end ## Impact - Deployment workflow bugs may go undetected until production - Error handling defects not caught by test suite - Skill system integration issues not validated - Regression risk for skill functionality changes - Users may encounter untested edge cases in deployment scenarios ## Subtasks - [ ] Add Behave unit tests (`features/`) for `deploy.py` error scenarios (missing dependencies, invalid configurations, failed deployments) - [ ] Add Behave unit tests for `SKILL.md` instruction validation and completeness checks - [ ] Add Robot Framework integration tests (`robot/`) for complete skill workflow execution - [ ] Add Robot Framework tests for tool availability verification during skill loading - [ ] Add Robot Framework tests for health check integration workflows - [ ] Add tests for the integration between `SKILL.md`, `deploy.py`, and `runbook.md` - [ ] Ensure coverage meets ≥97% threshold for the `examples/` module ## Definition of Done - [ ] Behave unit tests added for `deploy.py` error scenarios and validation paths - [ ] Behave unit tests added for `SKILL.md` instruction compliance - [ ] Robot Framework integration tests added for complete skill deployment workflow - [ ] Robot Framework tests added for tool availability and health check validation - [ ] Integration tests cover the SKILL.md → deploy.py → runbook.md workflow chain - [ ] All new tests follow BDD/Gherkin style (Behave) or Robot Framework conventions - [ ] All nox stages pass - [ ] Coverage >= 97% > **Backlog note:** This issue was discovered during autonomous operation > on milestone v3.8.0. It does not block milestone completion and has been > placed in the backlog for human review and future milestone assignment. --- **Automated by CleverAgents Bot** Supervisor: Bug Hunting | Agent: new-issue-creator
Author
Owner

Verified — Test coverage: agent-skills deployment workflow tests missing. MoSCoW: Could-have. Priority: Low.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

✅ **Verified** — Test coverage: agent-skills deployment workflow tests missing. MoSCoW: Could-have. Priority: Low. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor
Author
Owner

Verified — Test coverage: agent-skills deployment workflow tests missing. MoSCoW: Could-have. Priority: Low.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

✅ **Verified** — Test coverage: agent-skills deployment workflow tests missing. MoSCoW: Could-have. Priority: Low. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor
Author
Owner

Verified — Test coverage: agent-skills deployment workflow tests missing. MoSCoW: Could-have. Priority: Low.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

✅ **Verified** — Test coverage: agent-skills deployment workflow tests missing. MoSCoW: Could-have. Priority: Low. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#7307
No description provided.