TEST-INFRA: [test-data-quality] Introduce a Test Data Factory #2760

Open
opened 2026-04-04 19:15:15 +00:00 by freemo · 1 comment
Owner

Background and Context

Our current test suite relies heavily on hardcoded values (e.g., plan IDs, owner IDs, project names, tool names) embedded directly within Behave feature files and step definitions. This makes tests brittle and difficult to maintain: any change to the shape of these identifiers requires hunting down and updating every hardcoded occurrence. It also limits the variety of data exercised by our tests, reducing their effectiveness at surfacing edge-case bugs.

Introducing a dedicated test data factory — backed by a library such as faker — will allow step definitions to request realistic, varied data on demand, decoupling test logic from specific literal values.

Expected Behaviour

A TestDataFactory module (located in features/mocks/ per the project's mocking convention) provides factory methods for all common domain data types. Step definitions import and call these factories instead of embedding literal values. The factory is statically typed, fully documented, and covered by Behave scenarios.

Acceptance Criteria

  • A TestDataFactory class (or equivalent module) exists under features/mocks/.
  • Factory methods cover at minimum: plan IDs, owner IDs, project names, and tool names.
  • All existing hardcoded literal values in feature files are replaced with factory calls where practical.
  • The factory module is statically typed and passes nox -e typecheck.
  • Behave scenarios cover the factory's public API.
  • Coverage remains ≥ 97% (nox -e coverage_report).
  • The factory is documented (docstrings + usage notes in features/mocks/README.md or equivalent).

Supporting Information

  • Project mocking convention: all mocks, fakes, and stubs must live in features/mocks/ (CONTRIBUTING.md).
  • Candidate library: faker — widely used, actively maintained, MIT-licensed.
  • Integration tests (Robot Framework, robot/) must use real services; the factory is for unit-test (Behave) use only.

Metadata

  • Branch: task/test-data-quality-introduce-factory
  • Commit Message: feat(tests): introduce TestDataFactory in features/mocks for generated, varied test data
  • Milestone: v3.8.0
  • Parent Epic: #1678

Subtasks

  • Research and select a suitable test data generation library (evaluate faker; confirm licence and maintenance status).
  • Add the chosen library to the project's test dependencies via Hatch / pyproject.toml.
  • Create features/mocks/test_data_factory.py with a statically-typed TestDataFactory class.
  • Implement factory methods: plan_id(), owner_id(), project_name(), tool_name(), and any other commonly hardcoded types.
  • Write Behave scenarios (features/test_data_factory.feature) covering the factory's public API.
  • Refactor existing feature files to replace hardcoded literal values with factory calls.
  • Add documentation (docstrings on all public methods; usage notes in features/mocks/README.md or inline).
  • Run nox -e typecheck — fix any type errors.
  • Run nox -e unit_tests — all scenarios pass.
  • Run nox -e coverage_report — coverage ≥ 97%.
  • Run nox (all default sessions) — fix any remaining errors.

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • TestDataFactory is implemented under features/mocks/, is statically typed, and is documented.
  • Hardcoded literal values in existing Behave feature files are replaced with factory calls where practical.
  • Behave scenarios covering the factory's public API are present and passing.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
  • All nox stages pass.
  • Coverage ≥ 97%.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Background and Context Our current test suite relies heavily on hardcoded values (e.g., plan IDs, owner IDs, project names, tool names) embedded directly within Behave feature files and step definitions. This makes tests brittle and difficult to maintain: any change to the shape of these identifiers requires hunting down and updating every hardcoded occurrence. It also limits the variety of data exercised by our tests, reducing their effectiveness at surfacing edge-case bugs. Introducing a dedicated test data factory — backed by a library such as `faker` — will allow step definitions to request realistic, varied data on demand, decoupling test logic from specific literal values. ## Expected Behaviour A `TestDataFactory` module (located in `features/mocks/` per the project's mocking convention) provides factory methods for all common domain data types. Step definitions import and call these factories instead of embedding literal values. The factory is statically typed, fully documented, and covered by Behave scenarios. ## Acceptance Criteria - A `TestDataFactory` class (or equivalent module) exists under `features/mocks/`. - Factory methods cover at minimum: plan IDs, owner IDs, project names, and tool names. - All existing hardcoded literal values in feature files are replaced with factory calls where practical. - The factory module is statically typed and passes `nox -e typecheck`. - Behave scenarios cover the factory's public API. - Coverage remains ≥ 97% (`nox -e coverage_report`). - The factory is documented (docstrings + usage notes in `features/mocks/README.md` or equivalent). ## Supporting Information - Project mocking convention: all mocks, fakes, and stubs must live in `features/mocks/` (CONTRIBUTING.md). - Candidate library: [`faker`](https://faker.readthedocs.io/) — widely used, actively maintained, MIT-licensed. - Integration tests (Robot Framework, `robot/`) must use real services; the factory is for unit-test (Behave) use only. ## Metadata - **Branch**: `task/test-data-quality-introduce-factory` - **Commit Message**: `feat(tests): introduce TestDataFactory in features/mocks for generated, varied test data` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Subtasks - [ ] Research and select a suitable test data generation library (evaluate `faker`; confirm licence and maintenance status). - [ ] Add the chosen library to the project's test dependencies via Hatch / `pyproject.toml`. - [ ] Create `features/mocks/test_data_factory.py` with a statically-typed `TestDataFactory` class. - [ ] Implement factory methods: `plan_id()`, `owner_id()`, `project_name()`, `tool_name()`, and any other commonly hardcoded types. - [ ] Write Behave scenarios (`features/test_data_factory.feature`) covering the factory's public API. - [ ] Refactor existing feature files to replace hardcoded literal values with factory calls. - [ ] Add documentation (docstrings on all public methods; usage notes in `features/mocks/README.md` or inline). - [ ] Run `nox -e typecheck` — fix any type errors. - [ ] Run `nox -e unit_tests` — all scenarios pass. - [ ] Run `nox -e coverage_report` — coverage ≥ 97%. - [ ] Run `nox` (all default sessions) — fix any remaining errors. ## Definition of Done This issue is complete when: - [ ] All subtasks above are completed and checked off. - [ ] `TestDataFactory` is implemented under `features/mocks/`, is statically typed, and is documented. - [ ] Hardcoded literal values in existing Behave feature files are replaced with factory calls where practical. - [ ] Behave scenarios covering the factory's public API are present and passing. - [ ] A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - [ ] The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - [ ] The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - [ ] All nox stages pass. - [ ] Coverage ≥ 97%. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator
freemo added this to the v3.8.0 milestone 2026-04-04 19:15:21 +00:00
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • Priority: Medium
  • MoSCoW: Could Have — Test data factory is a test infrastructure improvement. Improves test maintainability but is not blocking feature work.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **Priority**: Medium - **MoSCoW**: Could Have — Test data factory is a test infrastructure improvement. Improves test maintainability but is not blocking feature work. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#2760
No description provided.