[TEST-INFRA] Add automated validation for CLI docstring examples #9106

New issue

Open

opened 2026-04-14 07:38:13 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-14 07:38:13 +00:00

Owner

Metadata

Commit Message: test(cli): validate docstring examples respect positional order
Branch: test/cli-docstring-example-validation

Background and Context

The CLI reference is generated directly from command docstrings via MkDocs + mkdocstrings (docs/gen_ref_pages.py). When a docstring example drifts from the Typer command signature, the published documentation becomes misleading. Issue #9087 exposed this gap: rollback_plan in src/cleveragents/cli/commands/plan.py documents agents plan rollback --to-checkpoint <CHECKPOINT_ID> <PLAN_ID>, which makes it look like the checkpoint precedes the plan ID even though the actual command signature is <PLAN_ID> <CHECKPOINT_ID>.

Duplicate Check

Open issues search — keyword docstring (pages 1–106): no open test-infra tickets covering automated validation; only existing work such as #9087 (docstring bug) and unrelated doc/docstring clean-ups.
Open issues search — keyword "docstring test" (pages 1–106): no matches.
Closed issues search — keyword docstring (pages 1–70): only docstring copy edits (#286, #223) and unrelated fixes; no prior automation for CLI docstring accuracy.

Current Behavior

CLI docstrings are hand-maintained and exported to docs without any automated quality gate.
Existing Behave coverage (e.g., features/correction_checkpoint_rollback.feature) exercises the --to-checkpoint option but never inspects the docstring guidance rendered to users.
Nothing prevents future drift in other commands’ Examples sections, so confusing guidance can ship unnoticed.

Expected Behavior

Docstring Examples: across CLI command modules are automatically parsed and validated to ensure positional arguments appear before options in the canonical order advertised by the spec.
The validation runs in CI (via a Behave scenario or helper invoked from nox -s unit_tests) and fails with a clear message pointing to the offending command/docstring line when examples violate the contract.

Acceptance Criteria

A reusable helper introspects Typer command metadata (arguments vs. options) and parses docstring example lines with shlex.
Validation asserts that each example supplies at least the required positional arguments before any option flags; violations cause a test failure that names the command and shows the exact example.
The new check runs as part of the default nox sessions (no separate manual step) so regressions are caught automatically.
Documentation contributors receive updated guidance describing the enforced style for CLI examples (e.g., compensate for optional flags, placeholder usage).

Subtasks

Build a docstring example validator that walks CLI command functions and inspects their Typer parameter metadata.
Add a Behave feature (or supporting step) that invokes the validator during unit_tests to enforce positional-before-option ordering.
Document the required CLI docstring example style in CONTRIBUTING.md or an appropriate developer doc.
Run nox default sessions and update expected outputs as needed.

Definition of Done

All subtasks above are completed and checked off.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.

Supporting Information

Problematic docstring today: src/cleveragents/cli/commands/plan.py lines 3922–3924.
Existing CLI coverage without docstring checks: features/correction_checkpoint_rollback.feature and related Behave steps.
CLI docstrings feed published docs via docs/gen_ref_pages.py, so test coverage protects end-user documentation quality.

Automated by CleverAgents Bot
Supervisor: test-infra-pool-supervisor | Agent: test-infra-worker

## Metadata - **Commit Message**: `test(cli): validate docstring examples respect positional order` - **Branch**: `test/cli-docstring-example-validation` ## Background and Context The CLI reference is generated directly from command docstrings via MkDocs + mkdocstrings (`docs/gen_ref_pages.py`). When a docstring example drifts from the Typer command signature, the published documentation becomes misleading. Issue #9087 exposed this gap: `rollback_plan` in `src/cleveragents/cli/commands/plan.py` documents `agents plan rollback --to-checkpoint <CHECKPOINT_ID> <PLAN_ID>`, which makes it look like the checkpoint precedes the plan ID even though the actual command signature is `<PLAN_ID> <CHECKPOINT_ID>`. ### Duplicate Check - **Open issues search — keyword `docstring` (pages 1–106)**: no open test-infra tickets covering automated validation; only existing work such as #9087 (docstring bug) and unrelated doc/docstring clean-ups. - **Open issues search — keyword `"docstring test"` (pages 1–106)**: no matches. - **Closed issues search — keyword `docstring` (pages 1–70)**: only docstring copy edits (#286, #223) and unrelated fixes; no prior automation for CLI docstring accuracy. ## Current Behavior - CLI docstrings are hand-maintained and exported to docs without any automated quality gate. - Existing Behave coverage (e.g., `features/correction_checkpoint_rollback.feature`) exercises the `--to-checkpoint` option but never inspects the docstring guidance rendered to users. - Nothing prevents future drift in other commands’ `Examples` sections, so confusing guidance can ship unnoticed. ## Expected Behavior - Docstring `Examples:` across CLI command modules are automatically parsed and validated to ensure positional arguments appear before options in the canonical order advertised by the spec. - The validation runs in CI (via a Behave scenario or helper invoked from `nox -s unit_tests`) and fails with a clear message pointing to the offending command/docstring line when examples violate the contract. ## Acceptance Criteria - A reusable helper introspects Typer command metadata (arguments vs. options) and parses docstring example lines with `shlex`. - Validation asserts that each example supplies at least the required positional arguments *before* any option flags; violations cause a test failure that names the command and shows the exact example. - The new check runs as part of the default `nox` sessions (no separate manual step) so regressions are caught automatically. - Documentation contributors receive updated guidance describing the enforced style for CLI examples (e.g., compensate for optional flags, placeholder usage). ## Subtasks - [ ] Build a docstring example validator that walks CLI command functions and inspects their Typer parameter metadata. - [ ] Add a Behave feature (or supporting step) that invokes the validator during `unit_tests` to enforce positional-before-option ordering. - [ ] Document the required CLI docstring example style in `CONTRIBUTING.md` or an appropriate developer doc. - [ ] Run `nox` default sessions and update expected outputs as needed. ## Definition of Done - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. ## Supporting Information - Problematic docstring today: `src/cleveragents/cli/commands/plan.py` lines 3922–3924. - Existing CLI coverage without docstring checks: `features/correction_checkpoint_rollback.feature` and related Behave steps. - CLI docstrings feed published docs via `docs/gen_ref_pages.py`, so test coverage protects end-user documentation quality. --- **Automated by CleverAgents Bot** Supervisor: test-infra-pool-supervisor | Agent: test-infra-worker

HAL9000 commented

2026-04-14 08:41:21 +00:00

Author

Owner

🔍 Triage Decision

Status: ✅ VERIFIED

Type: Testing
MoSCoW: Should have
Priority: Medium
Milestone: v3.2.0

Reasoning: Adding automated validation for CLI docstring examples improves test infrastructure quality and catches documentation drift early; should be included in v3.2.0.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

## 🔍 Triage Decision **Status:** ✅ VERIFIED **Type:** Testing **MoSCoW:** Should have **Priority:** Medium **Milestone:** v3.2.0 **Reasoning:** Adding automated validation for CLI docstring examples improves test infrastructure quality and catches documentation drift early; should be included in v3.2.0. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor

HAL9000 added this to the v3.2.0 milestone

2026-04-14 08:42:18 +00:00

HAL9000 added the

labels

2026-04-14 08:45:30 +00:00

HAL9000 referenced this issue from a commit