TEST-INFRA: [test-architecture] Remove tests focused on coverage, not behavior #1837

New issue

Open

opened 2026-04-02 23:56:43 +00:00 by freemo · 1 comment

freemo commented

2026-04-02 23:56:43 +00:00

Owner

Metadata

Branch: refactor/test-architecture-remove-coverage-boost-tests
Commit Message: refactor(test-infra): remove coverage-only feature files and replace with behavior-focused BDD scenarios
Milestone: v3.8.0
Parent Epic: #1678

Background and Context

The test suite contains a number of feature files with names like coverage_boost.feature. These files contain scenarios that are not focused on testing specific behaviors, but rather on exercising uncovered lines of code to increase coverage metrics. This is a violation of the BDD philosophy, which emphasizes testing behavior over simply increasing coverage numbers.

This is a significant issue with the test architecture. It indicates that the team is more focused on meeting a coverage target than on writing meaningful tests that document and verify the behavior of the system. This can lead to a false sense of security, as high coverage numbers do not necessarily mean that the application is well-tested.

Expected Behavior

All feature files in the test suite should describe meaningful, user-facing behaviors. No feature file should exist solely for the purpose of increasing coverage metrics. Coverage should be a by-product of thorough behavioral testing, not a goal in itself.

It is recommended that the team:

Stop writing tests solely for the purpose of increasing coverage.
Review the existing coverage_boost feature files and either remove them or rewrite them to focus on specific, meaningful behaviors.
Focus on writing BDD scenarios that describe how users interact with the system and what the expected outcomes are.

Acceptance Criteria

No feature file named coverage_boost.feature (or similar coverage-only names) exists in the test suite.
All remaining feature files describe specific, meaningful behaviors of the system.
Any previously coverage-only scenarios that happen to test real behavior have been rewritten as proper BDD scenarios with clear Given/When/Then structure describing user-observable outcomes.
Coverage remains at or above the project threshold (≥97%) after the removal/rewrite, demonstrating that genuine behavioral tests provide adequate coverage.

Subtasks

Audit the test suite and identify all feature files that exist solely to boost coverage (e.g., coverage_boost.feature, coverage_*.feature, or similar)
For each identified file, determine whether any scenarios describe real behavior worth preserving
Remove feature files that contain no meaningful behavioral scenarios
Rewrite any salvageable scenarios as proper BDD scenarios focused on user-observable behavior, placing them in the appropriate feature file
Verify that coverage remains ≥97% after changes via nox -s coverage_report
Run nox (all default sessions) and fix any errors
Update any documentation or contributing guidelines that may have encouraged coverage-boosting tests

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
No feature files exist in the test suite whose sole purpose is to increase coverage metrics.
All feature files describe specific, meaningful, user-observable behaviors.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
All nox stages pass.
Coverage >= 97%.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `refactor/test-architecture-remove-coverage-boost-tests` - **Commit Message**: `refactor(test-infra): remove coverage-only feature files and replace with behavior-focused BDD scenarios` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context The test suite contains a number of feature files with names like `coverage_boost.feature`. These files contain scenarios that are not focused on testing specific behaviors, but rather on exercising uncovered lines of code to increase coverage metrics. This is a violation of the BDD philosophy, which emphasizes testing behavior over simply increasing coverage numbers. This is a significant issue with the test architecture. It indicates that the team is more focused on meeting a coverage target than on writing meaningful tests that document and verify the behavior of the system. This can lead to a false sense of security, as high coverage numbers do not necessarily mean that the application is well-tested. ## Expected Behavior All feature files in the test suite should describe meaningful, user-facing behaviors. No feature file should exist solely for the purpose of increasing coverage metrics. Coverage should be a by-product of thorough behavioral testing, not a goal in itself. It is recommended that the team: 1. Stop writing tests solely for the purpose of increasing coverage. 2. Review the existing `coverage_boost` feature files and either remove them or rewrite them to focus on specific, meaningful behaviors. 3. Focus on writing BDD scenarios that describe how users interact with the system and what the expected outcomes are. ## Acceptance Criteria - No feature file named `coverage_boost.feature` (or similar coverage-only names) exists in the test suite. - All remaining feature files describe specific, meaningful behaviors of the system. - Any previously coverage-only scenarios that happen to test real behavior have been rewritten as proper BDD scenarios with clear Given/When/Then structure describing user-observable outcomes. - Coverage remains at or above the project threshold (≥97%) after the removal/rewrite, demonstrating that genuine behavioral tests provide adequate coverage. ## Subtasks - [ ] Audit the test suite and identify all feature files that exist solely to boost coverage (e.g., `coverage_boost.feature`, `coverage_*.feature`, or similar) - [ ] For each identified file, determine whether any scenarios describe real behavior worth preserving - [ ] Remove feature files that contain no meaningful behavioral scenarios - [ ] Rewrite any salvageable scenarios as proper BDD scenarios focused on user-observable behavior, placing them in the appropriate feature file - [ ] Verify that coverage remains ≥97% after changes via `nox -s coverage_report` - [ ] Run `nox` (all default sessions) and fix any errors - [ ] Update any documentation or contributing guidelines that may have encouraged coverage-boosting tests ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - No feature files exist in the test suite whose sole purpose is to increase coverage metrics. - All feature files describe specific, meaningful, user-observable behaviors. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage >= 97%. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

freemo added this to the v3.8.0 milestone

2026-04-02 23:58:20 +00:00

freemo added a new dependency

2026-04-03 00:01:38 +00:00

#1678 Epic: CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing

freemo commented

2026-04-03 00:04:40 +00:00

Author

Owner

Issue triaged by project owner:

State: Verified
MoSCoW: MoSCoW/Could Have — CI/test infrastructure improvement.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **MoSCoW**: MoSCoW/Could Have — CI/test infrastructure improvement. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner

freemo added the

MoSCoW

Could have

label

2026-04-03 00:04:40 +00:00