[AUTO-INF-6] Test Data Quality: Restore validation edge fixtures to Behave coverage #9129

New issue

Open

opened 2026-04-14 08:21:57 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-14 08:21:57 +00:00

Owner

"## Summary\n- docs/development/testing.md still documents a Behave suite named features/validation_edge_cases.feature with 25 scenarios, implying the validation fixtures should be exercised by Behave.\n- The features/ directory in master contains no validation_edge_cases.feature (nor a companion step module), so the fixtures under features/fixtures/validation/ are not covered by Behave anymore.\n- The only remaining coverage is robot/validation_edge.robot, which shells out to robot/helper_validation_edge.py; those helpers only perform structural assertions on the JSON fixture files (string/field presence) and never invoke the real SemanticValidationRule implementations.\n\n## Evidence\n- Documentation reference: docs/development/testing.md ("### Behave Suite: features/validation_edge_cases.feature") still lists the missing Behave suite and its supposed scenarios.\n- Feature directory listing via GET /api/v1/repos/cleveragents/cleveragents-core/contents/features?ref=master shows no validation_edge_cases.feature file.\n- robot/validation_edge.robot + robot/helper_validation_edge.py only assert that fixture files load and contain expected strings; they never run the validation rules (SemanticValidationRule.check).\n\n## Impact\n- Validation edge-case fixture data (malformed tool outputs, missing resources, timeout simulations, etc.) is no longer exercised against the validation engine in Behave, so regressions in SemanticValidationRule implementations can ship undetected.\n- Documentation promises Behave coverage that does not exist, creating confusion for contributors auditing validation coverage.\n- The JSON fixtures risk bit-rot because only superficial structural checks run in CI.\n\n## Proposal\n1. Re-introduce a Behave feature (e.g. features/validation_edge_cases.feature) with step definitions that iterate the fixtures via features/fixtures/validation/suite_helpers.py and run each through the mapped SemanticValidationRule (run_fixture / verify_fixture_result).\n2. Extend suite_helpers.ALL_FIXTURE_FILENAMES (or add a companion list) so the new scenarios cover every JSON file in features/fixtures/validation/, not only the legacy ones.\n3. Keep the Robot structural smoke tests for quick sanity checks, but ensure they delegate to the same helper or at minimum rename their documentation to "fixture lint" to avoid implying semantic coverage.\n4. Update docs/development/testing.md to reflect the restored Behave suite (or to accurately describe the Robot-only coverage if Behave is intentionally omitted).\n\n### Duplicate Check\n- Open issues search (q=validation edge, state=open): no tickets covering the missing Behave suite; reviewed #9048 (broader test-data realism work) and confirmed it does not address validation fixture coverage.\n- Cross-area search (label AUTO-INF-6): no existing AUTO-INF-6 items related to validation fixtures.\n- Closed issues search (q=validation edge, state=closed): nearest matches (e.g. #8316) target other validation tasks; none restore Behave coverage for the validation edge fixtures.\n\n---\nAutomated by CleverAgents Bot\nSupervisor: Test Infrastructure Pool | Agent: test-infra-pool-supervisor\n"

"## Summary\n- `docs/development/testing.md` still documents a Behave suite named `features/validation_edge_cases.feature` with 25 scenarios, implying the validation fixtures should be exercised by Behave.\n- The `features/` directory in `master` contains no `validation_edge_cases.feature` (nor a companion step module), so the fixtures under `features/fixtures/validation/` are not covered by Behave anymore.\n- The only remaining coverage is `robot/validation_edge.robot`, which shells out to `robot/helper_validation_edge.py`; those helpers only perform structural assertions on the JSON fixture files (string/field presence) and never invoke the real `SemanticValidationRule` implementations.\n\n## Evidence\n- Documentation reference: `docs/development/testing.md` (\"### Behave Suite: `features/validation_edge_cases.feature`\") still lists the missing Behave suite and its supposed scenarios.\n- Feature directory listing via `GET /api/v1/repos/cleveragents/cleveragents-core/contents/features?ref=master` shows no `validation_edge_cases.feature` file.\n- `robot/validation_edge.robot` + `robot/helper_validation_edge.py` only assert that fixture files load and contain expected strings; they never run the validation rules (`SemanticValidationRule.check`).\n\n## Impact\n- Validation edge-case fixture data (malformed tool outputs, missing resources, timeout simulations, etc.) is no longer exercised against the validation engine in Behave, so regressions in `SemanticValidationRule` implementations can ship undetected.\n- Documentation promises Behave coverage that does not exist, creating confusion for contributors auditing validation coverage.\n- The JSON fixtures risk bit-rot because only superficial structural checks run in CI.\n\n## Proposal\n1. Re-introduce a Behave feature (e.g. `features/validation_edge_cases.feature`) with step definitions that iterate the fixtures via `features/fixtures/validation/suite_helpers.py` and run each through the mapped `SemanticValidationRule` (`run_fixture` / `verify_fixture_result`).\n2. Extend `suite_helpers.ALL_FIXTURE_FILENAMES` (or add a companion list) so the new scenarios cover every JSON file in `features/fixtures/validation/`, not only the legacy ones.\n3. Keep the Robot structural smoke tests for quick sanity checks, but ensure they delegate to the same helper or at minimum rename their documentation to \"fixture lint\" to avoid implying semantic coverage.\n4. Update `docs/development/testing.md` to reflect the restored Behave suite (or to accurately describe the Robot-only coverage if Behave is intentionally omitted).\n\n### Duplicate Check\n- Open issues search (`q=validation edge`, `state=open`): no tickets covering the missing Behave suite; reviewed #9048 (broader test-data realism work) and confirmed it does not address validation fixture coverage.\n- Cross-area search (label `AUTO-INF-6`): no existing AUTO-INF-6 items related to validation fixtures.\n- Closed issues search (`q=validation edge`, `state=closed`): nearest matches (e.g. #8316) target other validation tasks; none restore Behave coverage for the validation edge fixtures.\n\n---\n**Automated by CleverAgents Bot**\nSupervisor: Test Infrastructure Pool | Agent: test-infra-pool-supervisor\n"

HAL9000 added this to the v3.2.0 milestone

2026-04-14 08:23:06 +00:00

HAL9000 added the

Rows
Columns