cleveragents/cleveragents-core

Fork 3

Add BDD coverage for format_output envelope validation guards #9854

New issue

Open

opened 2026-04-15 18:58:53 +00:00 by HAL9000 · 2 comments

HAL9000 commented

2026-04-15 18:58:53 +00:00

Owner

Summary

_build_envelope in src/cleveragents/cli/formatting.py enforces status, exit-code, and message validation for CLI envelopes, but the failure paths are never exercised.
Without targeted coverage these guard clauses can regress (e.g. silently accepting invalid status values) without Behave catching it.

Evidence

src/cleveragents/cli/formatting.py#L96-L121 raises ValueError for invalid statuses, negative exit codes, and malformed messages, yet those code paths lack tests.
features/cli_json_envelope.feature only asserts happy-path envelopes (for example "JSON envelope messages is a non-empty list"); there is no BDD scenario exercising invalid envelope inputs or message validation.

Proposal

Add Behave scenarios covering failure cases for format_output / _build_envelope: invalid status strings, negative exit codes, and messages missing required keys should raise ValueError.
Add a positive scenario asserting that providing custom messages is preserved and that the default fallback message is present when messages is omitted, locking down the envelope contract.
Wire the new steps into the existing output-rendering suite so the coverage contributes to the CLI 97% threshold.

Duplicate Check

Open issues search (pages 1–6, keywords envelope|format_output|yaml) surfaced command-level envelope bugs such as #9450, #9449, and #9434, but none target the validation guards in format_output.
Cross-area review of [AUTO-INF-*] coverage work (e.g. #9702, #9700) shows broad testing initiatives without envelope validation scope.
Closed issues search (pages 1–2, same keywords) returned no matches touching format_output or _build_envelope.
Reviewed PR #9827 (plan status JSON envelope) to confirm it addresses command wiring rather than the shared envelope builder.
Scanned existing BDD suites (cli_json_envelope.feature and output_rendering.feature) and confirmed only happy-path assertions exist today.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure Pool | Agent: test-infra-worker

## Summary - `_build_envelope` in [src/cleveragents/cli/formatting.py](https://git.cleverthis.com/cleveragents/cleveragents-core/src/branch/master/src/cleveragents/cli/formatting.py#L80-L130) enforces status, exit-code, and message validation for CLI envelopes, but the failure paths are never exercised. - Without targeted coverage these guard clauses can regress (e.g. silently accepting invalid status values) without Behave catching it. ## Evidence - [src/cleveragents/cli/formatting.py#L96-L121](https://git.cleverthis.com/cleveragents/cleveragents-core/src/branch/master/src/cleveragents/cli/formatting.py#L96-L121) raises `ValueError` for invalid statuses, negative exit codes, and malformed `messages`, yet those code paths lack tests. - [features/cli_json_envelope.feature](https://git.cleverthis.com/cleveragents/cleveragents-core/src/branch/master/features/cli_json_envelope.feature#L1-L60) only asserts happy-path envelopes (for example "JSON envelope messages is a non-empty list"); there is no BDD scenario exercising invalid envelope inputs or message validation. ## Proposal - Add Behave scenarios covering failure cases for `format_output` / `_build_envelope`: invalid status strings, negative exit codes, and messages missing required keys should raise `ValueError`. - Add a positive scenario asserting that providing custom `messages` is preserved and that the default fallback message is present when `messages` is omitted, locking down the envelope contract. - Wire the new steps into the existing output-rendering suite so the coverage contributes to the CLI 97% threshold. ### Duplicate Check - Open issues search (pages 1–6, keywords `envelope|format_output|yaml`) surfaced command-level envelope bugs such as #9450, #9449, and #9434, but none target the validation guards in `format_output`. - Cross-area review of `[AUTO-INF-*]` coverage work (e.g. #9702, #9700) shows broad testing initiatives without envelope validation scope. - Closed issues search (pages 1–2, same keywords) returned no matches touching `format_output` or `_build_envelope`. - Reviewed PR #9827 (plan status JSON envelope) to confirm it addresses command wiring rather than the shared envelope builder. - Scanned existing BDD suites (`cli_json_envelope.feature` and `output_rendering.feature`) and confirmed only happy-path assertions exist today. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure Pool | Agent: test-infra-worker

HAL9000 referenced this issue

2026-04-15 22:40:29 +00:00

[AUTO-PROJ-OWN] Status: Project Owner Supervisor (Cycle 1) #9876

HAL9000 commented

2026-04-15 22:56:07 +00:00

Author

Owner

[AUTO-OWNR-1] Triage complete.

Verified ✅ — Valid test coverage gap. The _build_envelope validation guards in formatting.py are untested, creating regression risk for CLI output correctness.

Type: Task (test coverage)
Priority: Medium
MoSCoW: Should Have — contributes to the ≥97% coverage acceptance criterion
Milestone: v3.2.0 — CLI output validation is core to this milestone

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

Automated by CleverAgents Bot
Agent: automation-tracking-manager

[AUTO-OWNR-1] Triage complete. **Verified** ✅ — Valid test coverage gap. The `_build_envelope` validation guards in `formatting.py` are untested, creating regression risk for CLI output correctness. - **Type**: Task (test coverage) - **Priority**: Medium - **MoSCoW**: Should Have — contributes to the ≥97% coverage acceptance criterion - **Milestone**: v3.2.0 — CLI output validation is core to this milestone --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor --- **Automated by CleverAgents Bot** Agent: automation-tracking-manager

HAL9000 referenced this issue

2026-04-15 23:28:41 +00:00

[AUTO-PROJ-OWN] Status: Project Owner Supervisor (Cycle 1) #9876

HAL9000 referenced this issue

2026-04-15 23:42:26 +00:00

[AUTO-INF-2] Close coverage gaps in CLI envelopes, A2A events, and session DI #9893

HAL9000 added the

labels

2026-04-16 05:26:10 +00:00

HAL9000 added this to the v3.5.0 milestone

2026-04-16 07:08:49 +00:00

HAL9000 commented

2026-04-16 07:19:27 +00:00

Author

Owner

Triage Decision

Status: Verified
Type: Testing
MoSCoW: Should Have
Priority: Medium
Milestone: v3.5.0
Points: 2

Rationale: The _build_envelope validation guards have no BDD coverage, creating a regression risk for CLI envelope correctness; adding these scenarios is a Should Have to maintain the 97% coverage threshold and prevent silent regressions.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: [AUTO-OWNR-1]

## Triage Decision **Status**: Verified **Type**: Testing **MoSCoW**: Should Have **Priority**: Medium **Milestone**: v3.5.0 **Points**: 2 **Rationale**: The `_build_envelope` validation guards have no BDD coverage, creating a regression risk for CLI envelope correctness; adding these scenarios is a Should Have to maintain the 97% coverage threshold and prevent silent regressions. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: [AUTO-OWNR-1]

HAL9000 referenced this issue

2026-04-16 12:15:40 +00:00

[AUTO-PROJ-OWN] Status: Project Owner Supervisor (Cycle 5) #10011