TEST-INFRA: [test-data-quality] Replace static decision tree fixtures with dynamic data generation #3388

Open
opened 2026-04-05 16:21:45 +00:00 by freemo · 0 comments
Owner

Metadata

  • Branch: test/backlog/decision-tree-dynamic-fixtures
  • Commit Message: test(decision-tree): replace static fixtures with dynamic data generation
  • Milestone: Backlog
  • Parent Epic: #394

Background and Context

The test fixtures for decision trees, located in features/fixtures/m3/decision_tree_outputs.json, are currently implemented as a static JSON file. This approach has several drawbacks that impact the quality and maintainability of our tests:

  • Limited Test Coverage: The tests always run against the same, hardcoded data. This means we are not testing a wide variety of scenarios, and we may be missing bugs that only appear with different data combinations or edge cases.
  • Maintenance Overhead: Any change to the decision tree data model requires manual updates to this large JSON file, which is error-prone and time-consuming.
  • Poor Edge Case Handling: The current fixtures only cover a few "happy path" scenarios. There are no tests for edge cases like empty decision trees, trees with many branches, or trees with invalid data.

Proposed Solution

Replace the static JSON fixtures with a dynamic data generation strategy using:

  • Test Data Factories: A factory library (like factory_boy) could be used to generate decision tree objects on the fly, with the ability to customize the data for each test case.
  • Property-Based Testing: A library like hypothesis could be used to generate a wide range of valid and invalid decision tree structures, allowing us to test for a much broader set of inputs.

Subtasks

  • Audit existing usage of features/fixtures/m3/decision_tree_outputs.json across all test suites
  • Design and implement a DecisionTreeFactory (e.g., using factory_boy) for dynamic fixture generation
  • Evaluate and optionally integrate hypothesis for property-based testing of decision tree structures
  • Migrate all existing tests that reference the static JSON fixture to use the new factory/generation strategy
  • Add new test scenarios covering edge cases: empty trees, deeply nested branches, invalid data, maximum branching factor
  • Remove features/fixtures/m3/decision_tree_outputs.json once all references are migrated
  • Run nox (all default sessions), fix any errors
  • Verify coverage >= 97% via nox -s coverage_report

Definition of Done

  • The static JSON file features/fixtures/m3/decision_tree_outputs.json is removed
  • A test data generation strategy (e.g., using factory_boy factories) is implemented for decision trees
  • The existing tests are updated to use the new data generation strategy
  • New tests are added to cover edge cases and a wider variety of scenarios
  • All subtasks above are completed and checked off
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done
  • All nox stages pass
  • Coverage >= 97%

Backlog note: This issue was discovered during autonomous operation
on milestone v3.2.0. It does not block milestone completion and has been
placed in the backlog for human review and future milestone assignment.


Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator
Acting on behalf of: Test Infrastructure

## Metadata - **Branch**: `test/backlog/decision-tree-dynamic-fixtures` - **Commit Message**: `test(decision-tree): replace static fixtures with dynamic data generation` - **Milestone**: Backlog - **Parent Epic**: #394 ## Background and Context The test fixtures for decision trees, located in `features/fixtures/m3/decision_tree_outputs.json`, are currently implemented as a static JSON file. This approach has several drawbacks that impact the quality and maintainability of our tests: - **Limited Test Coverage**: The tests always run against the same, hardcoded data. This means we are not testing a wide variety of scenarios, and we may be missing bugs that only appear with different data combinations or edge cases. - **Maintenance Overhead**: Any change to the decision tree data model requires manual updates to this large JSON file, which is error-prone and time-consuming. - **Poor Edge Case Handling**: The current fixtures only cover a few "happy path" scenarios. There are no tests for edge cases like empty decision trees, trees with many branches, or trees with invalid data. ## Proposed Solution Replace the static JSON fixtures with a dynamic data generation strategy using: - **Test Data Factories**: A factory library (like `factory_boy`) could be used to generate decision tree objects on the fly, with the ability to customize the data for each test case. - **Property-Based Testing**: A library like `hypothesis` could be used to generate a wide range of valid and invalid decision tree structures, allowing us to test for a much broader set of inputs. ## Subtasks - [ ] Audit existing usage of `features/fixtures/m3/decision_tree_outputs.json` across all test suites - [ ] Design and implement a `DecisionTreeFactory` (e.g., using `factory_boy`) for dynamic fixture generation - [ ] Evaluate and optionally integrate `hypothesis` for property-based testing of decision tree structures - [ ] Migrate all existing tests that reference the static JSON fixture to use the new factory/generation strategy - [ ] Add new test scenarios covering edge cases: empty trees, deeply nested branches, invalid data, maximum branching factor - [ ] Remove `features/fixtures/m3/decision_tree_outputs.json` once all references are migrated - [ ] Run `nox` (all default sessions), fix any errors - [ ] Verify coverage >= 97% via `nox -s coverage_report` ## Definition of Done - [ ] The static JSON file `features/fixtures/m3/decision_tree_outputs.json` is removed - [ ] A test data generation strategy (e.g., using `factory_boy` factories) is implemented for decision trees - [ ] The existing tests are updated to use the new data generation strategy - [ ] New tests are added to cover edge cases and a wider variety of scenarios - [ ] All subtasks above are completed and checked off - [ ] A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation - [ ] The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly - [ ] The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done - [ ] All nox stages pass - [ ] Coverage >= 97% > **Backlog note:** This issue was discovered during autonomous operation > on milestone v3.2.0. It does not block milestone completion and has been > placed in the backlog for human review and future milestone assignment. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator Acting on behalf of: Test Infrastructure
freemo added this to the v3.7.0 milestone 2026-04-05 17:19:35 +00:00
freemo removed this from the v3.7.0 milestone 2026-04-06 23:52:25 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Blocks
#394 Epic: Decision Framework
cleveragents/cleveragents-core
Reference
cleveragents/cleveragents-core#3388
No description provided.