TEST-INFRA: [missing-test-levels] Add missing Robot and ASV tests for the cli module #2460

Open
opened 2026-04-03 18:29:28 +00:00 by freemo · 1 comment
Owner

Metadata

  • Branch: feature/cli-robot-asv-tests
  • Commit Message: test(cli): add Robot integration tests and ASV performance benchmarks for cli module
  • Milestone: v3.8.0
  • Parent Epic: #1678

Description

The cleveragents.cli module is missing Robot integration tests and ASV performance benchmarks. The CONTRIBUTING.md file states that all source modules should have all three levels of tests: Behave (unit), Robot (integration), and ASV (performance benchmarks).

Currently, the cli module only has Behave unit tests. This gap means:

  • Integration-level regressions in the CLI (e.g., argument parsing, command dispatch, output formatting) are not caught by the Robot test suite.
  • Performance regressions in critical CLI commands are not tracked over time by ASV.

Proposed Solution

  1. Create Robot integration tests for the cli module in the robot/ directory. These tests should cover the main functionalities of the CLI, such as running commands, handling arguments, and producing the correct output. Mocking is strictly prohibited in Robot tests — real dependencies must be used.
  2. Create ASV performance benchmarks for the cli module in the benchmarks/ directory. These benchmarks should measure the performance of critical CLI commands and track their performance over time.

Subtasks

  • Audit the cleveragents.cli module to identify all public commands and entry points requiring Robot coverage
  • Create robot/cli/ directory and write Robot Framework .robot test files covering CLI command execution, argument handling, and output validation
  • Create benchmarks/cli/ directory and write ASV benchmark classes for critical CLI commands
  • Run nox -e integration_tests locally to verify all new Robot tests pass with real dependencies
  • Run nox -e benchmarks (or equivalent ASV session) to verify benchmarks execute without error
  • Run nox -e coverage_report to confirm overall coverage remains >= 97%
  • Update any relevant documentation or test inventory files

Definition of Done

  • Robot integration tests for the cli module are created in robot/cli/ and pass via nox -e integration_tests
  • ASV performance benchmarks for the cli module are created in benchmarks/cli/ and execute successfully
  • No mocking is used in Robot tests — all tests use real dependencies
  • All nox stages pass (nox -e lint, nox -e typecheck, nox -e unit_tests, nox -e integration_tests, nox -e coverage_report)
  • Coverage >= 97%
  • PR is submitted referencing this issue and linked as a dependency

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `feature/cli-robot-asv-tests` - **Commit Message**: `test(cli): add Robot integration tests and ASV performance benchmarks for cli module` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Description The `cleveragents.cli` module is missing Robot integration tests and ASV performance benchmarks. The `CONTRIBUTING.md` file states that all source modules should have all three levels of tests: Behave (unit), Robot (integration), and ASV (performance benchmarks). Currently, the `cli` module only has Behave unit tests. This gap means: - Integration-level regressions in the CLI (e.g., argument parsing, command dispatch, output formatting) are not caught by the Robot test suite. - Performance regressions in critical CLI commands are not tracked over time by ASV. ## Proposed Solution 1. Create Robot integration tests for the `cli` module in the `robot/` directory. These tests should cover the main functionalities of the CLI, such as running commands, handling arguments, and producing the correct output. Mocking is strictly prohibited in Robot tests — real dependencies must be used. 2. Create ASV performance benchmarks for the `cli` module in the `benchmarks/` directory. These benchmarks should measure the performance of critical CLI commands and track their performance over time. ## Subtasks - [ ] Audit the `cleveragents.cli` module to identify all public commands and entry points requiring Robot coverage - [ ] Create `robot/cli/` directory and write Robot Framework `.robot` test files covering CLI command execution, argument handling, and output validation - [ ] Create `benchmarks/cli/` directory and write ASV benchmark classes for critical CLI commands - [ ] Run `nox -e integration_tests` locally to verify all new Robot tests pass with real dependencies - [ ] Run `nox -e benchmarks` (or equivalent ASV session) to verify benchmarks execute without error - [ ] Run `nox -e coverage_report` to confirm overall coverage remains >= 97% - [ ] Update any relevant documentation or test inventory files ## Definition of Done - [ ] Robot integration tests for the `cli` module are created in `robot/cli/` and pass via `nox -e integration_tests` - [ ] ASV performance benchmarks for the `cli` module are created in `benchmarks/cli/` and execute successfully - [ ] No mocking is used in Robot tests — all tests use real dependencies - [ ] All nox stages pass (`nox -e lint`, `nox -e typecheck`, `nox -e unit_tests`, `nox -e integration_tests`, `nox -e coverage_report`) - [ ] Coverage >= 97% - [ ] PR is submitted referencing this issue and linked as a dependency --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator
freemo added this to the v3.8.0 milestone 2026-04-03 18:29:33 +00:00
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • MoSCoW: Should Have — Important spec requirement or quality improvement. Should be included in the milestone if possible.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **MoSCoW**: Should Have — Important spec requirement or quality improvement. Should be included in the milestone if possible. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
freemo removed this from the v3.8.0 milestone 2026-04-06 20:59:04 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#2460
No description provided.