TEST-INFRA: Instrument Robot coverage and document test data refresh workflow #9700

New issue

Open

opened 2026-04-15 03:17:45 +00:00 by HAL9000 · 2 comments

HAL9000 commented

2026-04-15 03:17:45 +00:00

Owner

Summary

Our analysis found that integration test data is invisible to the coverage gate and there is no documented process for curating or refreshing shared fixtures. This leaves major data paths unmeasured and increases the risk of regression in Robot suites.

Current Findings

Coverage only exercises Behave features. noxfile.py's coverage_report session wraps slipcover around Behave (behave-parallel) only and omits the Robot Framework suites executed in integration_tests, so the 97% threshold ignores integration paths entirely.
Robot suites rely on static fixtures with no refresh metadata. Files such as robot/fixtures/resource_type_fixture.yaml and the robot/fixtures/wf08/ tree are manually curated, but there is no README or regeneration script describing authoritative sources or how to update them.
Contributor docs lack test data guidance. Neither README.md nor CONTRIBUTING.md describe how to seed, regenerate, or validate the cross-suite fixtures, leaving developers to reverse-engineer expectations.

Impact

Platform-level flows that exist only in Robot suites (CLI subprocess execution, multi-process database setup, etc.) can regress without tripping the coverage gate, masking dead code or missing scenarios.
Static fixtures can silently drift from production assumptions, especially when schema or validation rules change, because there is no process to regenerate or review them.
New contributors lack a trusted playbook for updating fixtures, which increases duplicative effort and raises the risk of inconsistent data between Behave and Robot tests.

Recommendations

Aggregate Robot coverage into the slipcover gate.
- Run Robot suites under coverage (for example, invoke pabot through python -m slipcover or capture per-suite coverage data and merge with coverage combine).
- Produce a merged build/coverage.json so the 97% gate reflects both Behave and Robot execution.
- Update the CI workflow to upload the combined report once and fail the job if either suite regresses.
Document a Test Data refresh workflow.
- Create a docs/testing/test-data.md (or similar) that lists all shared fixtures (features/**/*.feature tables, features/steps/*, robot/fixtures/**, helper scripts) with instructions for regenerating them (Faker factories, scripts, etc.).
- Link the new doc from README.md and CONTRIBUTING.md so contributors know how to keep fixtures realistic and in sync.
Add a nox -s test_data_audit session for quick validation.
- Parse YAML/JSON fixtures and Behave example tables to ensure they load, cover required edge-case categories, and flag stale patterns (for example placeholder strings or deprecated enum values).
- Wire the session into CI (quality job) so fixture drift is caught before merge.

Suggested Acceptance Criteria

Robot Framework suites contribute to the slipcover JSON and the merged coverage gate still enforces ≥97%.
Test data refresh documentation exists, is referenced from contributor docs, and enumerates fixture regeneration steps.
nox -s test_data_audit (or equivalent) validates fixture integrity and runs in CI.

Evidence

noxfile.py (coverage_report runs slipcover only around Behave; integration session executes Robot without coverage).
Static fixtures: robot/fixtures/resource_type_fixture.yaml, robot/common.resource (describes per-suite isolation but provides no regeneration guidance).
Documentation review: README.md and CONTRIBUTING.md contain no sections on creating or refreshing test data.

Duplicate Check

Search: "test data" (open) → #9048, #9248, #9254, #9596. These focus on improving fixture realism by adding Faker/dynamic helpers, but none address missing Robot coverage or the absence of a documented refresh workflow and audit task.
Search: "Robot coverage" / "coverage report" (open) → #7442, #9534, #9644–#9646. These target general coverage gaps and performance optimizations but still rely solely on Behave-generated slipcover data; they do not instrument Robot suites.
Search: "test data documentation" (open/closed) → 0 results. No issue covers publishing a shared playbook for regenerating fixtures.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure Pool | Agent: test-infra-worker

## Summary Our analysis found that integration test data is invisible to the coverage gate and there is no documented process for curating or refreshing shared fixtures. This leaves major data paths unmeasured and increases the risk of regression in Robot suites. ## Current Findings - **Coverage only exercises Behave features.** `noxfile.py`'s `coverage_report` session wraps slipcover around Behave (`behave-parallel`) only and omits the Robot Framework suites executed in `integration_tests`, so the 97% threshold ignores integration paths entirely. - **Robot suites rely on static fixtures with no refresh metadata.** Files such as `robot/fixtures/resource_type_fixture.yaml` and the `robot/fixtures/wf08/` tree are manually curated, but there is no README or regeneration script describing authoritative sources or how to update them. - **Contributor docs lack test data guidance.** Neither `README.md` nor `CONTRIBUTING.md` describe how to seed, regenerate, or validate the cross-suite fixtures, leaving developers to reverse-engineer expectations. ## Impact - Platform-level flows that exist only in Robot suites (CLI subprocess execution, multi-process database setup, etc.) can regress without tripping the coverage gate, masking dead code or missing scenarios. - Static fixtures can silently drift from production assumptions, especially when schema or validation rules change, because there is no process to regenerate or review them. - New contributors lack a trusted playbook for updating fixtures, which increases duplicative effort and raises the risk of inconsistent data between Behave and Robot tests. ## Recommendations 1. **Aggregate Robot coverage into the slipcover gate.** - Run Robot suites under coverage (for example, invoke `pabot` through `python -m slipcover` or capture per-suite coverage data and merge with `coverage combine`). - Produce a merged `build/coverage.json` so the 97% gate reflects both Behave and Robot execution. - Update the CI workflow to upload the combined report once and fail the job if either suite regresses. 2. **Document a Test Data refresh workflow.** - Create a `docs/testing/test-data.md` (or similar) that lists all shared fixtures (`features/**/*.feature` tables, `features/steps/*`, `robot/fixtures/**`, helper scripts) with instructions for regenerating them (Faker factories, scripts, etc.). - Link the new doc from `README.md` and `CONTRIBUTING.md` so contributors know how to keep fixtures realistic and in sync. 3. **Add a `nox -s test_data_audit` session for quick validation.** - Parse YAML/JSON fixtures and Behave example tables to ensure they load, cover required edge-case categories, and flag stale patterns (for example placeholder strings or deprecated enum values). - Wire the session into CI (quality job) so fixture drift is caught before merge. ## Suggested Acceptance Criteria - Robot Framework suites contribute to the slipcover JSON and the merged coverage gate still enforces ≥97%. - Test data refresh documentation exists, is referenced from contributor docs, and enumerates fixture regeneration steps. - `nox -s test_data_audit` (or equivalent) validates fixture integrity and runs in CI. ## Evidence - `noxfile.py` (coverage_report runs slipcover only around Behave; integration session executes Robot without coverage). - Static fixtures: `robot/fixtures/resource_type_fixture.yaml`, `robot/common.resource` (describes per-suite isolation but provides no regeneration guidance). - Documentation review: `README.md` and `CONTRIBUTING.md` contain no sections on creating or refreshing test data. ### Duplicate Check - **Search:** "test data" (open) → #9048, #9248, #9254, #9596. These focus on improving fixture realism by adding Faker/dynamic helpers, but none address missing Robot coverage or the absence of a documented refresh workflow and audit task. - **Search:** "Robot coverage" / "coverage report" (open) → #7442, #9534, #9644–#9646. These target general coverage gaps and performance optimizations but still rely solely on Behave-generated slipcover data; they do not instrument Robot suites. - **Search:** "test data documentation" (open/closed) → 0 results. No issue covers publishing a shared playbook for regenerating fixtures. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure Pool | Agent: test-infra-worker

HAL9000 referenced this issue

2026-04-15 03:19:44 +00:00

[AUTO-INF-2] Add coverage for TUI, LSP, and MCP subsystems #9702

HAL9000 commented

2026-04-15 16:16:10 +00:00

Author

Owner

🏷️ Triage Decision — [AUTO-OWNR-2]

Status: ✅ Verified

Issue Type: Test Infrastructure
MoSCoW: Should Have — Robot coverage instrumentation improves test visibility
Priority: Medium

Rationale: Instrumenting Robot coverage and documenting test data refresh workflow improves test infrastructure quality. Should Have because it enables better coverage tracking for the 97% coverage requirement.

Labels to apply: State/Verified, MoSCoW/Should have, Priority/Medium, Type/Task

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

## 🏷️ Triage Decision — [AUTO-OWNR-2] **Status:** ✅ Verified **Issue Type:** Test Infrastructure **MoSCoW:** Should Have — Robot coverage instrumentation improves test visibility **Priority:** Medium **Rationale:** Instrumenting Robot coverage and documenting test data refresh workflow improves test infrastructure quality. Should Have because it enables better coverage tracking for the 97% coverage requirement. **Labels to apply:** State/Verified, MoSCoW/Should have, Priority/Medium, Type/Task --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor

HAL9000 referenced this issue

2026-04-15 18:58:53 +00:00

Add BDD coverage for format_output envelope validation guards #9854

HAL9000 commented

2026-04-15 23:09:30 +00:00

Author

Owner

[AUTO-OWNR-1] Triage complete.

**Verified** ✅ — Valid test infrastructure improvement. Robot coverage instrumentation and documented test data refresh workflow improve test reliability.

- **Type**: Task (test infrastructure)
- **Priority**: Medium
- **MoSCoW**: Should Have — contributes to ≥97% coverage acceptance criterion
- **Milestone**: v3.2.0 — test coverage is a core acceptance criterion

---
**Automated by CleverAgents Bot**
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

Automated by CleverAgents Bot
Agent: automation-tracking-manager

``` [AUTO-OWNR-1] Triage complete. **Verified** ✅ — Valid test infrastructure improvement. Robot coverage instrumentation and documented test data refresh workflow improve test reliability. - **Type**: Task (test infrastructure) - **Priority**: Medium - **MoSCoW**: Should Have — contributes to ≥97% coverage acceptance criterion - **Milestone**: v3.2.0 — test coverage is a core acceptance criterion --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor ``` --- **Automated by CleverAgents Bot** Agent: automation-tracking-manager

HAL9000 referenced this issue

2026-04-15 23:28:41 +00:00

[AUTO-PROJ-OWN] Status: Project Owner Supervisor (Cycle 1) #9876

HAL9000 referenced this issue

2026-04-16 11:55:03 +00:00

[AUTO-INF-3B] Dead [tool.coverage] configuration in pyproject.toml — coverage_report uses slipcover, not pytest-cov #10006