test: add TDD bug-capture test for #1039 — missing validation YAML (#1132)
CI / build (push) Successful in 25s
CI / helm (push) Successful in 33s
CI / lint (push) Successful in 3m19s
CI / typecheck (push) Successful in 4m14s
CI / benchmark-regression (push) Has been skipped
CI / security (push) Successful in 4m29s
CI / quality (push) Successful in 4m12s
CI / integration_tests (push) Successful in 7m29s
CI / unit_tests (push) Successful in 7m51s
CI / docker (push) Successful in 1m30s
CI / e2e_tests (push) Successful in 12m50s
CI / coverage (push) Successful in 12m19s
CI / status-check (push) Successful in 1s
CI / benchmark-publish (push) Has been cancelled
CI / build (push) Successful in 25s
CI / helm (push) Successful in 33s
CI / lint (push) Successful in 3m19s
CI / typecheck (push) Successful in 4m14s
CI / benchmark-regression (push) Has been skipped
CI / security (push) Successful in 4m29s
CI / quality (push) Successful in 4m12s
CI / integration_tests (push) Successful in 7m29s
CI / unit_tests (push) Successful in 7m51s
CI / docker (push) Successful in 1m30s
CI / e2e_tests (push) Successful in 12m50s
CI / coverage (push) Successful in 12m19s
CI / status-check (push) Successful in 1s
CI / benchmark-publish (push) Has been cancelled
## Summary This PR adds a Behave TDD bug-capture test that proves bug #1039 exists: the specification references `validations/unit-tests.yaml` as an example validation configuration file (Workflow Examples, Example 1, Step 1) but the file does not exist in the project's `examples/validations/` directory. ## Changes - **New feature file:** `features/tdd_missing_validation_unit_tests_yaml.feature` — four scenarios tagged `@tdd_expected_fail @tdd_bug @tdd_bug_1039` that verify: 1. The file `examples/validations/unit-tests.yaml` exists 2. The YAML config contains `name: local/unit-tests` 3. The validation mode is `required` 4. The description field is non-empty - **New step file:** `features/steps/tdd_missing_validation_unit_tests_yaml_steps.py` — step implementations that locate the project root and check file existence/content All four scenarios currently fail with `AssertionError` (the file doesn't exist, proving the bug exists). The `@tdd_expected_fail` tag inverts the results so the test suite passes CI. ## Robot Integration Test N/A — this bug is purely about a missing example file referenced by the specification. There is no integration-level behavior to test. ## Quality Gate Results | Gate | Result | |------|--------| | `nox -s lint` | passed | | `nox -s typecheck` | passed (0 errors) | | `nox -s unit_tests` | 462 features, 12234 scenarios, 0 failures | | `nox -s integration_tests` | 1672 tests, 0 failures | | `nox -s e2e_tests` | 37 tests, 0 failures | | `nox -s coverage_report` | 98% (>= 97% threshold) | Closes #1103 Reviewed-on: #1132 Reviewed-by: Jeffrey Phillips Freeman <jeffrey.freeman@cleverthis.com> Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com> Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
This commit was merged in pull request #1132.
This commit is contained in:
@@ -233,6 +233,7 @@ jobs:
|
||||
|
||||
e2e_tests:
|
||||
runs-on: docker
|
||||
timeout-minutes: 45
|
||||
container:
|
||||
image: python:3.13-slim
|
||||
steps:
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
"""Step definitions for TDD Bug #1039 — Missing validations/unit-tests.yaml.
|
||||
|
||||
This module captures the bug described in issue #1039: the specification
|
||||
references ``validations/unit-tests.yaml`` as a validation configuration file
|
||||
but the file does not exist in the project. These steps verify the file's
|
||||
existence and expected content.
|
||||
|
||||
The scenarios in ``tdd_missing_validation_unit_tests_yaml.feature`` are tagged
|
||||
with ``@tdd_expected_fail`` until the fix is merged. Once the file is
|
||||
created, the tag will be removed and the test will run normally as a
|
||||
regression guard.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
from behave import given, then, when
|
||||
from behave.runner import Context
|
||||
|
||||
|
||||
@given("the project root directory is known for validation yaml test")
|
||||
def step_project_root_known(context: Context) -> None:
|
||||
"""Locate the project root directory for validation YAML tests."""
|
||||
# Walk up from the features directory to find the project root
|
||||
features_dir = Path(__file__).resolve().parent.parent
|
||||
project_root = features_dir.parent
|
||||
context.tdd_1039_project_root = project_root
|
||||
|
||||
|
||||
@when('I check for the validation config file "{relative_path}"')
|
||||
def step_check_validation_config_file(context: Context, relative_path: str) -> None:
|
||||
"""Check whether a validation config file exists at the given path."""
|
||||
project_root: Path = context.tdd_1039_project_root
|
||||
config_path = project_root / relative_path
|
||||
context.tdd_1039_config_path = config_path
|
||||
context.tdd_1039_config_exists = config_path.is_file()
|
||||
|
||||
|
||||
@then("the validation config file should exist")
|
||||
def step_validation_config_should_exist(context: Context) -> None:
|
||||
"""Assert that the validation config file exists."""
|
||||
assert context.tdd_1039_config_exists, (
|
||||
f"Expected validation config file to exist at "
|
||||
f"{context.tdd_1039_config_path} but it was not found. "
|
||||
f"Bug #1039: the specification references this file but it is missing."
|
||||
)
|
||||
|
||||
|
||||
@when('I load the validation config file "{relative_path}"')
|
||||
def step_load_validation_config_file(context: Context, relative_path: str) -> None:
|
||||
"""Attempt to load and parse a validation config YAML file."""
|
||||
project_root: Path = context.tdd_1039_project_root
|
||||
config_path = project_root / relative_path
|
||||
|
||||
assert config_path.is_file(), (
|
||||
f"Cannot load validation config: file does not exist at "
|
||||
f"{config_path}. Bug #1039: this file is referenced by the "
|
||||
f"specification but is missing from the project."
|
||||
)
|
||||
|
||||
raw = config_path.read_text(encoding="utf-8")
|
||||
config = yaml.safe_load(raw)
|
||||
|
||||
assert isinstance(config, dict), (
|
||||
f"Validation config at {config_path} is not a YAML mapping."
|
||||
)
|
||||
|
||||
context.tdd_1039_config = config
|
||||
|
||||
|
||||
@then('the validation config should have name "{expected_name}"')
|
||||
def step_validation_config_name(context: Context, expected_name: str) -> None:
|
||||
"""Assert the validation config has the expected name."""
|
||||
config: dict[str, object] = context.tdd_1039_config
|
||||
actual_name = config.get("name")
|
||||
assert actual_name == expected_name, (
|
||||
f"Expected validation config name to be '{expected_name}' "
|
||||
f"but got '{actual_name}'."
|
||||
)
|
||||
|
||||
|
||||
@then('the validation config should have mode "{expected_mode}"')
|
||||
def step_validation_config_mode(context: Context, expected_mode: str) -> None:
|
||||
"""Assert the validation config has the expected mode."""
|
||||
config: dict[str, object] = context.tdd_1039_config
|
||||
actual_mode = config.get("mode")
|
||||
assert actual_mode == expected_mode, (
|
||||
f"Expected validation config mode to be '{expected_mode}' "
|
||||
f"but got '{actual_mode}'."
|
||||
)
|
||||
|
||||
|
||||
@then("the validation config should have a non-empty description")
|
||||
def step_validation_config_has_description(context: Context) -> None:
|
||||
"""Assert the validation config has a non-empty description."""
|
||||
config: dict[str, object] = context.tdd_1039_config
|
||||
description = config.get("description")
|
||||
assert description and isinstance(description, str) and description.strip(), (
|
||||
"Expected validation config to have a non-empty 'description' field."
|
||||
)
|
||||
@@ -0,0 +1,42 @@
|
||||
@tdd_expected_fail @tdd_issue @tdd_issue_1039
|
||||
Feature: TDD Bug #1039 — Missing validations/unit-tests.yaml configuration
|
||||
As a developer
|
||||
I want to verify that the validation configuration file referenced by the
|
||||
specification exists and conforms to the expected schema
|
||||
So that the bug is captured and will be caught by a regression test
|
||||
|
||||
The specification (Workflow Examples, Example 1, Step 1) references a
|
||||
validation configuration file ``validations/unit-tests.yaml`` that is
|
||||
used with the ``agents validation add`` command to register a required
|
||||
validation named ``local/unit-tests``. According to the specification
|
||||
output, this validation should run ``pytest tests/``, have description
|
||||
"Unit tests", mode "required", and timeout 300s (default).
|
||||
|
||||
However, the file ``validations/unit-tests.yaml`` does not exist anywhere
|
||||
in the project. The ``examples/validations/`` directory contains only
|
||||
``required-validation.yaml`` and ``wrapped-validation.yaml``, but not
|
||||
the ``unit-tests.yaml`` referenced by the specification.
|
||||
|
||||
This test captures bug #1039 and uses ``@tdd_expected_fail`` until the
|
||||
fix is merged. Once the file is created, the tag will be removed and
|
||||
the test will run normally as a regression guard.
|
||||
|
||||
Scenario: The unit-tests.yaml validation config file exists in the project
|
||||
Given the project root directory is known for validation yaml test
|
||||
When I check for the validation config file "examples/validations/unit-tests.yaml"
|
||||
Then the validation config file should exist
|
||||
|
||||
Scenario: The unit-tests.yaml config loads as valid YAML with expected name
|
||||
Given the project root directory is known for validation yaml test
|
||||
When I load the validation config file "examples/validations/unit-tests.yaml"
|
||||
Then the validation config should have name "local/unit-tests"
|
||||
|
||||
Scenario: The unit-tests.yaml config defines a required validation mode
|
||||
Given the project root directory is known for validation yaml test
|
||||
When I load the validation config file "examples/validations/unit-tests.yaml"
|
||||
Then the validation config should have mode "required"
|
||||
|
||||
Scenario: The unit-tests.yaml config includes a description
|
||||
Given the project root directory is known for validation yaml test
|
||||
When I load the validation config file "examples/validations/unit-tests.yaml"
|
||||
Then the validation config should have a non-empty description
|
||||
Reference in New Issue
Block a user