forked from cleveragents/cleveragents-core
Luis Mendes
c8cd7eab82
Implemented the three-tag TDD bug-capture system in Behave environment hooks: - Added tag validation in before_scenario: @tdd_bug_<N> requires @tdd_bug, @tdd_expected_fail requires both @tdd_bug and @tdd_bug_<N> - Added result inversion via Scenario.run() wrapper installed in before_all: scenarios tagged @tdd_expected_fail that fail are reported as passed (expected failure), and scenarios that unexpectedly pass are reported as failed with guidance to remove the tag - Added helper functions validate_tdd_tags() and should_invert_result() - Added inline documentation referencing CONTRIBUTING.md > TDD Bug Test Tags - Added Behave test scenarios for tag validation and inversion behavior - Extract apply_tdd_inversion() as a public, testable function that encapsulates all inversion logic with proper guards - Refactored handle_tdd_expected_fail() to delegate to apply_tdd_inversion() after tag validation, eliminating ~55 lines of duplicated inversion logic - Tag validation errors in handle_tdd_expected_fail() are now logged at WARNING level instead of being silently swallowed - Add hook_failed guard: never invert infrastructure/hook errors - Add was_dry_run guard: skip inversion when no test actually ran - Add non-AssertionError guard: warn and skip inversion for exceptions that likely indicate infrastructure problems, not the captured bug - Log exception details at DEBUG level before clearing (previously discarded silently) - Attach synthetic AssertionError to last step on unexpected pass so the failure reason appears in standard Behave formatter output ISSUES CLOSED: #627
166 lines
9.2 KiB
Gherkin
166 lines
9.2 KiB
Gherkin
@infrastructure
|
|
Feature: TDD expected-fail handler infrastructure
|
|
Verify that ``apply_tdd_inversion`` (the production code path called by the
|
|
``Scenario.run()`` monkey-patch) and ``handle_tdd_expected_fail`` (the
|
|
standalone entry point) correctly invert scenario AND step-level status for
|
|
TDD bug-capture tests.
|
|
|
|
# --- apply_tdd_inversion (production path) ---
|
|
|
|
Scenario: apply_tdd_inversion inverts a failed scenario with failed and skipped steps to passed
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_bug @tdd_bug_999" with status "failed"
|
|
And the mock scenario has a step "broken step" with status "failed"
|
|
And the mock scenario has a step "skipped step" with status "skipped"
|
|
When apply_tdd_inversion processes the scenario with failed True
|
|
Then the scenario status should be "passed"
|
|
And the step "broken step" should have status "passed"
|
|
And the step "skipped step" should have status "passed"
|
|
And the step "broken step" should have error_message cleared
|
|
And the step "skipped step" should have error_message cleared
|
|
|
|
Scenario: apply_tdd_inversion fails a passed scenario that still carries @tdd_expected_fail
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_bug @tdd_bug_999" with status "passed"
|
|
And the mock scenario has a step "last step" with status "passed"
|
|
When apply_tdd_inversion processes the scenario with failed False
|
|
Then the scenario status should be "failed"
|
|
And the step "last step" should have a synthetic error_message
|
|
|
|
Scenario: apply_tdd_inversion does not invert when hook_failed is set
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_bug @tdd_bug_999" with status "failed" and hook_failed
|
|
And the mock scenario has a step "broken step" with status "failed"
|
|
When apply_tdd_inversion processes the scenario with failed True
|
|
Then the scenario status should be "failed"
|
|
And the step "broken step" should have status "failed"
|
|
|
|
Scenario: apply_tdd_inversion does not invert during dry-run mode
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_bug @tdd_bug_999" with status "passed" and was_dry_run
|
|
When apply_tdd_inversion processes the scenario with failed False
|
|
Then the apply_tdd_inversion result should be False
|
|
|
|
Scenario: apply_tdd_inversion does not invert non-AssertionError exceptions
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_bug @tdd_bug_999" with status "failed"
|
|
And the mock scenario has an infrastructure-error step "infra step" with exception "RuntimeError"
|
|
When apply_tdd_inversion processes the scenario with failed True
|
|
Then the scenario status should be "failed"
|
|
And the step "infra step" should have status "failed"
|
|
|
|
Scenario: apply_tdd_inversion ignores scenarios without @tdd_expected_fail
|
|
Given a mock scenario tagged "@tdd_bug @tdd_bug_999" with status "failed"
|
|
And the mock scenario has a step "broken step" with status "failed"
|
|
When apply_tdd_inversion processes the scenario with failed True
|
|
Then the scenario status should be "failed"
|
|
And the step "broken step" should have status "failed"
|
|
|
|
# --- handle_tdd_expected_fail (standalone entry point) ---
|
|
|
|
Scenario: Handler inverts a failed scenario with failed and skipped steps to passed
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_bug @tdd_bug_999" with status "failed"
|
|
And the mock scenario has a step "broken step" with status "failed"
|
|
And the mock scenario has a step "skipped step" with status "skipped"
|
|
When the TDD expected-fail handler processes the scenario
|
|
Then the scenario status should be "passed"
|
|
And the step "broken step" should have status "passed"
|
|
And the step "skipped step" should have status "passed"
|
|
And the step "broken step" should have error_message cleared
|
|
And the step "skipped step" should have error_message cleared
|
|
|
|
Scenario: Handler fails a passed scenario that still carries @tdd_expected_fail
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_bug @tdd_bug_999" with status "passed"
|
|
And the mock scenario has a step "last step" with status "passed"
|
|
When the TDD expected-fail handler processes the scenario
|
|
Then the scenario status should be "failed"
|
|
And the step "last step" should have a synthetic error_message
|
|
|
|
Scenario: Handler rejects @tdd_expected_fail without @tdd_bug
|
|
Given a mock scenario tagged "@tdd_expected_fail" with status "failed"
|
|
When the TDD expected-fail handler processes the scenario
|
|
Then the scenario status should be "failed"
|
|
|
|
Scenario: Handler rejects @tdd_expected_fail without @tdd_bug_N
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_bug" with status "failed"
|
|
When the TDD expected-fail handler processes the scenario
|
|
Then the scenario status should be "failed"
|
|
|
|
Scenario: Handler rejects @tdd_bug_N without @tdd_bug unconditionally
|
|
Given a mock scenario tagged "@tdd_bug_999" with status "passed"
|
|
When the TDD expected-fail handler processes the scenario
|
|
Then the scenario status should be "failed"
|
|
|
|
Scenario: Handler ignores scenarios without @tdd_expected_fail
|
|
Given a mock scenario tagged "@tdd_bug @tdd_bug_999" with status "failed"
|
|
When the TDD expected-fail handler processes the scenario
|
|
Then the scenario status should be "failed"
|
|
|
|
Scenario: Handler does not invert non-AssertionError exceptions
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_bug @tdd_bug_999" with status "failed"
|
|
And the mock scenario has an infrastructure-error step "infra step" with exception "RuntimeError"
|
|
When the TDD expected-fail handler processes the scenario
|
|
Then the scenario status should be "failed"
|
|
And the step "infra step" should have status "failed"
|
|
|
|
# --- apply_tdd_inversion: mixed exception types (S3) ---
|
|
|
|
Scenario: apply_tdd_inversion does not invert when mixed exceptions present
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_bug @tdd_bug_999" with status "failed"
|
|
And the mock scenario has a step "bug step" with status "failed"
|
|
And the mock scenario has an infrastructure-error step "infra step" with exception "RuntimeError"
|
|
When apply_tdd_inversion processes the scenario with failed True
|
|
Then the scenario status should be "failed"
|
|
And the step "bug step" should have status "failed"
|
|
And the step "infra step" should have status "failed"
|
|
|
|
Scenario: Handler does not invert when mixed exceptions present
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_bug @tdd_bug_999" with status "failed"
|
|
And the mock scenario has a step "bug step" with status "failed"
|
|
And the mock scenario has an infrastructure-error step "infra step" with exception "RuntimeError"
|
|
When the TDD expected-fail handler processes the scenario
|
|
Then the scenario status should be "failed"
|
|
And the step "bug step" should have status "failed"
|
|
And the step "infra step" should have status "failed"
|
|
|
|
# --- selective step reset preserves already-passed steps (TC-3) ---
|
|
|
|
Scenario: apply_tdd_inversion preserves already-passed steps during inversion
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_bug @tdd_bug_999" with status "failed"
|
|
And the mock scenario has a step "ok step" with status "passed"
|
|
And the mock scenario has a step "broken step" with status "failed"
|
|
And the mock scenario has a step "skipped step" with status "skipped"
|
|
When apply_tdd_inversion processes the scenario with failed True
|
|
Then the scenario status should be "passed"
|
|
And the step "ok step" should have status "passed"
|
|
And the step "broken step" should have status "passed"
|
|
And the step "skipped step" should have status "passed"
|
|
And the step "broken step" should have error_message cleared
|
|
And the step "skipped step" should have error_message cleared
|
|
|
|
Scenario: Handler preserves already-passed steps during inversion
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_bug @tdd_bug_999" with status "failed"
|
|
And the mock scenario has a step "ok step" with status "passed"
|
|
And the mock scenario has a step "broken step" with status "failed"
|
|
And the mock scenario has a step "skipped step" with status "skipped"
|
|
When the TDD expected-fail handler processes the scenario
|
|
Then the scenario status should be "passed"
|
|
And the step "ok step" should have status "passed"
|
|
And the step "broken step" should have status "passed"
|
|
And the step "skipped step" should have status "passed"
|
|
And the step "broken step" should have error_message cleared
|
|
And the step "skipped step" should have error_message cleared
|
|
|
|
# --- before_scenario hook_failed regression test (TC-2) ---
|
|
|
|
Scenario: before_scenario sets hook_failed on invalid TDD tags
|
|
Given a mock scenario tagged "@tdd_expected_fail" with status "untested"
|
|
When before_scenario is called with the mock scenario
|
|
Then the scenario hook_failed flag should be True
|
|
And the scenario status should be "failed"
|
|
|
|
# --- _install_tdd_expected_fail_patch (S2) ---
|
|
|
|
Scenario: _install_tdd_expected_fail_patch sets the patched flag
|
|
When the TDD expected-fail patch installation status is checked
|
|
Then the Scenario class should have _tdd_run_patched set to True
|
|
|
|
Scenario: _install_tdd_expected_fail_patch is idempotent
|
|
When _install_tdd_expected_fail_patch is called twice
|
|
Then the Scenario.run method should not be double-wrapped
|