TEST-INFRA: [test-data-quality] Use Scenario Outlines for better test data parameterization #1784

Open
opened 2026-04-02 23:49:23 +00:00 by freemo · 0 comments
Owner

Metadata

  • Branch: task/v3.8.0-test-data-quality-scenario-outlines
  • Commit Message: test(infra): refactor hardcoded feature file scenarios to use Scenario Outlines for parameterization
  • Milestone: v3.8.0
  • Parent Epic: #1634

Background and Context

Area: test-data-quality

Some scenarios in the feature files use hardcoded values that would be better represented as examples in a Scenario Outline. For instance, in a2a_sse_streaming.feature, the event type and plan ID are hardcoded. Using a Scenario Outline would make it easier to add more test cases with different data, improving test coverage and maintainability.

Hardcoded values in scenarios create several problems as the test suite grows:

  • Reduced Coverage: It is impractical to manually duplicate scenarios for every data permutation, leading to gaps in test coverage.
  • Poor Maintainability: When data models change, every hardcoded scenario must be updated individually, increasing the risk of inconsistency.
  • Reduced Readability: Scenario Outlines with an Examples table make the intent of parameterized tests immediately clear to readers, serving as living documentation.
  • Missed Edge Cases: Parameterized outlines make it trivial to add new data rows (e.g., boundary values, invalid inputs) without duplicating scenario logic.

Expected Behavior

Feature files that currently use hardcoded values for repeated or parameterizable scenarios are refactored to use Gherkin Scenario Outline with Examples tables. This allows multiple data combinations to be tested with a single scenario definition, improving both coverage and maintainability.

Acceptance Criteria

  • All identified scenarios with hardcoded, parameterizable values (e.g., event types, plan IDs in a2a_sse_streaming.feature) are converted to Scenario Outline with Examples tables.
  • Each Examples table includes at minimum the original hardcoded value plus at least one additional representative value.
  • All step definitions are updated as needed to support the parameterized placeholders.
  • No existing test coverage is lost; all previously passing scenarios continue to pass.
  • The refactored feature files remain readable and serve as clear living documentation.

Subtasks

  • Audit all feature files to identify scenarios with hardcoded values suitable for Scenario Outline parameterization.
  • Refactor identified scenarios in a2a_sse_streaming.feature (event type, plan ID) to use Scenario Outline + Examples.
  • Refactor any other identified feature files with similar hardcoded patterns.
  • Update step definitions to handle parameterized placeholders introduced by the outlines.
  • Add additional Examples rows to improve coverage (e.g., boundary values, alternative valid inputs).
  • Tests (Behave): Verify all refactored scenarios pass with the full Examples table.
  • Verify coverage >= 97% via nox -s coverage_report.
  • Run nox (all default sessions), fix any errors.

Definition of Done

  • All identified hardcoded scenarios are converted to Scenario Outline with Examples tables.
  • Each Examples table contains at least two data rows (original value + at least one additional).
  • All step definitions support the parameterized placeholders without regression.
  • All subtasks above are completed and checked off.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
  • All nox stages pass.
  • Coverage >= 97%.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `task/v3.8.0-test-data-quality-scenario-outlines` - **Commit Message**: `test(infra): refactor hardcoded feature file scenarios to use Scenario Outlines for parameterization` - **Milestone**: v3.8.0 - **Parent Epic**: #1634 ## Background and Context **Area**: `test-data-quality` Some scenarios in the feature files use hardcoded values that would be better represented as examples in a Scenario Outline. For instance, in `a2a_sse_streaming.feature`, the event type and plan ID are hardcoded. Using a Scenario Outline would make it easier to add more test cases with different data, improving test coverage and maintainability. Hardcoded values in scenarios create several problems as the test suite grows: - **Reduced Coverage**: It is impractical to manually duplicate scenarios for every data permutation, leading to gaps in test coverage. - **Poor Maintainability**: When data models change, every hardcoded scenario must be updated individually, increasing the risk of inconsistency. - **Reduced Readability**: Scenario Outlines with an `Examples` table make the intent of parameterized tests immediately clear to readers, serving as living documentation. - **Missed Edge Cases**: Parameterized outlines make it trivial to add new data rows (e.g., boundary values, invalid inputs) without duplicating scenario logic. ## Expected Behavior Feature files that currently use hardcoded values for repeated or parameterizable scenarios are refactored to use Gherkin `Scenario Outline` with `Examples` tables. This allows multiple data combinations to be tested with a single scenario definition, improving both coverage and maintainability. ## Acceptance Criteria - All identified scenarios with hardcoded, parameterizable values (e.g., event types, plan IDs in `a2a_sse_streaming.feature`) are converted to `Scenario Outline` with `Examples` tables. - Each `Examples` table includes at minimum the original hardcoded value plus at least one additional representative value. - All step definitions are updated as needed to support the parameterized placeholders. - No existing test coverage is lost; all previously passing scenarios continue to pass. - The refactored feature files remain readable and serve as clear living documentation. ## Subtasks - [ ] Audit all feature files to identify scenarios with hardcoded values suitable for `Scenario Outline` parameterization. - [ ] Refactor identified scenarios in `a2a_sse_streaming.feature` (event type, plan ID) to use `Scenario Outline` + `Examples`. - [ ] Refactor any other identified feature files with similar hardcoded patterns. - [ ] Update step definitions to handle parameterized placeholders introduced by the outlines. - [ ] Add additional `Examples` rows to improve coverage (e.g., boundary values, alternative valid inputs). - [ ] Tests (Behave): Verify all refactored scenarios pass with the full `Examples` table. - [ ] Verify coverage >= 97% via `nox -s coverage_report`. - [ ] Run `nox` (all default sessions), fix any errors. ## Definition of Done - [ ] All identified hardcoded scenarios are converted to `Scenario Outline` with `Examples` tables. - [ ] Each `Examples` table contains at least two data rows (original value + at least one additional). - [ ] All step definitions support the parameterized placeholders without regression. - [ ] All subtasks above are completed and checked off. - [ ] A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - [ ] The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - [ ] The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - [ ] All nox stages pass. - [ ] Coverage >= 97%. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator
freemo added this to the v3.8.0 milestone 2026-04-02 23:49:44 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#1784
No description provided.