[AUTO-INF-11] Improve Test Data Quality #8332

New issue

Closed

opened 2026-04-13 09:02:56 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-13 09:02:56 +00:00

Owner

Summary

Robot and Behave fixtures rely on static credentials and timestamps, which makes the suites brittle and unrealistic when production policies shift.
Many fixtures use placeholder values or cover only the happy path, so regression detection for edge cases is weak.
Robot helper scripts execute tiny inline assertions with hardcoded data, providing little coverage of error handling or negative flows.

Findings

Static authentication fixture with narrow coverage – robot/wf02_test_generation_common.py (lines 123-138, 292-312) writes a temporary tests/conftest.py whose auth_credentials fixture always returns ("admin", "secret") and generates tests that accept only that pair. Any policy change (role-based credentials, password rotation, multi-factor) breaks WF02 scenarios and there is no coverage of invalid inputs.
Hardcoded timestamps and identifiers across Behave steps – features/steps/plan_cli_coverage_steps.py (lines 69-98) and features/steps/session_service_coverage_steps.py (lines 235-253) embed 2025-2026 datetimes, ULIDs, and session IDs directly inside helper factories. When output formats or timezone handling change, dozens of steps must be updated by hand. Similar patterns appear in legacy_migrator_steps.py.
Placeholder fixture content that diverges from production – features/fixtures/validation/api_surface_changes.json relies on names like old_handler_xyz and flags alias gaps as "expected". These inputs do not mimic real import graphs or dependency metadata, so semantic validation regressions can slip through unnoticed.
Robot helper scripts limited to happy paths – Files such as robot/scripts/test_mapper_configured.py and robot/scripts/test_missing_tool.py assert against minimal dictionaries (e.g., { "dest": "/path" }) and never exercise malformed inputs, missing keys, or type mismatches. Error-handling code paths remain untested.

Recommendations

Introduce reusable data builders or factory helpers (possibly backed by factory_boy-style patterns) so credentials, plans, sessions, and resource records can vary without editing every step file.
Adopt faker or hypothesis strategies for IDs, timestamps, and paths, paired with freezegun/timezone helpers so assertions focus on relative behavior instead of specific calendar dates.
Expand fixtures to include negative and edge-case variants (invalid credentials, multiple plan states, import alias cases that should be caught) and document expected fields alongside the data files.
Promote Robot helper scripts into parametrized pytest suites that share fixtures with Behave where possible, providing clearer failure output and coverage of both success and failure paths.

Duplicate Check

Searched open issues for [AUTO-INF-11] and "test data quality" — no matches found.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure Pool | Agent: test-infra-worker

## Summary - Robot and Behave fixtures rely on static credentials and timestamps, which makes the suites brittle and unrealistic when production policies shift. - Many fixtures use placeholder values or cover only the happy path, so regression detection for edge cases is weak. - Robot helper scripts execute tiny inline assertions with hardcoded data, providing little coverage of error handling or negative flows. ## Findings 1. **Static authentication fixture with narrow coverage** – `robot/wf02_test_generation_common.py` (lines 123-138, 292-312) writes a temporary `tests/conftest.py` whose `auth_credentials` fixture always returns `("admin", "secret")` and generates tests that accept only that pair. Any policy change (role-based credentials, password rotation, multi-factor) breaks WF02 scenarios and there is no coverage of invalid inputs. 2. **Hardcoded timestamps and identifiers across Behave steps** – `features/steps/plan_cli_coverage_steps.py` (lines 69-98) and `features/steps/session_service_coverage_steps.py` (lines 235-253) embed 2025-2026 datetimes, ULIDs, and session IDs directly inside helper factories. When output formats or timezone handling change, dozens of steps must be updated by hand. Similar patterns appear in `legacy_migrator_steps.py`. 3. **Placeholder fixture content that diverges from production** – `features/fixtures/validation/api_surface_changes.json` relies on names like `old_handler_xyz` and flags alias gaps as "expected". These inputs do not mimic real import graphs or dependency metadata, so semantic validation regressions can slip through unnoticed. 4. **Robot helper scripts limited to happy paths** – Files such as `robot/scripts/test_mapper_configured.py` and `robot/scripts/test_missing_tool.py` assert against minimal dictionaries (e.g., `{ "dest": "/path" }`) and never exercise malformed inputs, missing keys, or type mismatches. Error-handling code paths remain untested. ## Recommendations - Introduce reusable data builders or factory helpers (possibly backed by `factory_boy`-style patterns) so credentials, plans, sessions, and resource records can vary without editing every step file. - Adopt `faker` or hypothesis strategies for IDs, timestamps, and paths, paired with `freezegun`/timezone helpers so assertions focus on relative behavior instead of specific calendar dates. - Expand fixtures to include negative and edge-case variants (invalid credentials, multiple plan states, import alias cases that should be caught) and document expected fields alongside the data files. - Promote Robot helper scripts into parametrized pytest suites that share fixtures with Behave where possible, providing clearer failure output and coverage of both success and failure paths. ### Duplicate Check - Searched open issues for `[AUTO-INF-11]` and "test data quality" — no matches found. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure Pool | Agent: test-infra-worker

Rows
Columns