hurui200320
f67e8a2e07
CI / lint (pull_request) Successful in 18s
CI / build (pull_request) Successful in 17s
CI / helm (pull_request) Successful in 18s
CI / quality (pull_request) Successful in 53s
CI / typecheck (pull_request) Successful in 56s
CI / security (pull_request) Successful in 57s
CI / push-validation (pull_request) Successful in 40s
CI / unit_tests (pull_request) Successful in 3m13s
CI / integration_tests (pull_request) Successful in 4m23s
CI / e2e_tests (pull_request) Successful in 4m37s
CI / docker (pull_request) Successful in 1m34s
CI / coverage (pull_request) Successful in 10m46s
CI / status-check (pull_request) Successful in 1s
CI / lint (push) Successful in 17s
CI / quality (push) Successful in 17s
CI / build (push) Successful in 24s
CI / helm (push) Successful in 24s
CI / push-validation (push) Successful in 37s
CI / typecheck (push) Successful in 52s
CI / security (push) Successful in 52s
CI / e2e_tests (push) Successful in 3m13s
CI / unit_tests (push) Successful in 6m37s
CI / integration_tests (push) Successful in 6m39s
CI / docker (push) Successful in 1m35s
CI / coverage (push) Successful in 11m13s
CI / status-check (push) Successful in 1s
Surface the non-AssertionError guard warning in standard Behave output by emitting to stderr in addition to the structured logger, and add infrastructure coverage that asserts this guard path is visible during test runs. Document the @tdd_expected_fail expectation that bug-signaling failures must use AssertionError so infrastructure exceptions are not accidentally treated as expected bug failures. ISSUES CLOSED: #8294
184 lines
10 KiB
Gherkin
184 lines
10 KiB
Gherkin
@infrastructure
|
|
Feature: TDD expected-fail handler infrastructure
|
|
Verify that ``apply_tdd_inversion`` (the production code path called by the
|
|
``Scenario.run()`` monkey-patch) and ``handle_tdd_expected_fail`` (the
|
|
standalone entry point) correctly invert scenario AND step-level status for
|
|
TDD issue-capture tests.
|
|
|
|
# --- apply_tdd_inversion (production path) ---
|
|
|
|
Scenario: apply_tdd_inversion inverts a failed scenario with failed and skipped steps to passed
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_issue @tdd_issue_999" with status "failed"
|
|
And the mock scenario has a step "broken step" with status "failed"
|
|
And the mock scenario has a step "skipped step" with status "skipped"
|
|
When apply_tdd_inversion processes the scenario with failed True
|
|
Then the scenario status should be "passed"
|
|
And the step "broken step" should have status "passed"
|
|
And the step "skipped step" should have status "passed"
|
|
And the step "broken step" should have error_message cleared
|
|
And the step "skipped step" should have error_message cleared
|
|
|
|
Scenario: apply_tdd_inversion fails a passed scenario that still carries @tdd_expected_fail
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_issue @tdd_issue_999" with status "passed"
|
|
And the mock scenario has a step "last step" with status "passed"
|
|
When apply_tdd_inversion processes the scenario with failed False
|
|
Then the scenario status should be "failed"
|
|
And the step "last step" should have a synthetic error_message
|
|
|
|
Scenario: apply_tdd_inversion does not invert when hook_failed is set
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_issue @tdd_issue_999" with status "failed" and hook_failed
|
|
And the mock scenario has a step "broken step" with status "failed"
|
|
When apply_tdd_inversion processes the scenario with failed True
|
|
Then the scenario status should be "failed"
|
|
And the step "broken step" should have status "failed"
|
|
|
|
Scenario: apply_tdd_inversion does not invert during dry-run mode
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_issue @tdd_issue_999" with status "passed" and was_dry_run
|
|
When apply_tdd_inversion processes the scenario with failed False
|
|
Then the apply_tdd_inversion result should be False
|
|
|
|
Scenario: apply_tdd_inversion does not invert non-AssertionError exceptions
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_issue @tdd_issue_999" with status "failed"
|
|
And the mock scenario has an infrastructure-error step "infra step" with exception "RuntimeError"
|
|
When apply_tdd_inversion processes the scenario with failed True
|
|
Then the scenario status should be "failed"
|
|
And the step "infra step" should have status "failed"
|
|
|
|
Scenario: apply_tdd_inversion surfaces non-AssertionError guard warning to stderr
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_issue @tdd_issue_999" with status "failed"
|
|
And the mock scenario has an infrastructure-error step "infra step" with exception "RuntimeError"
|
|
When apply_tdd_inversion processes the scenario with failed True while capturing stderr
|
|
Then the scenario status should be "failed"
|
|
And the step "infra step" should have status "failed"
|
|
And the apply_tdd_inversion result should be True
|
|
And stderr output should contain "Non-assertion exception in expected-fail scenario"
|
|
|
|
Scenario: apply_tdd_inversion does not emit guard warning for AssertionError exceptions
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_issue @tdd_issue_999" with status "failed"
|
|
And the mock scenario has a step "bug step" with status "failed"
|
|
When apply_tdd_inversion processes the scenario with failed True while capturing stderr
|
|
Then the scenario status should be "passed"
|
|
And the step "bug step" should have status "passed"
|
|
And the apply_tdd_inversion result should be False
|
|
And stderr output should not contain "Non-assertion exception"
|
|
|
|
Scenario: apply_tdd_inversion ignores scenarios without @tdd_expected_fail
|
|
Given a mock scenario tagged "@tdd_issue @tdd_issue_999" with status "failed"
|
|
And the mock scenario has a step "broken step" with status "failed"
|
|
When apply_tdd_inversion processes the scenario with failed True
|
|
Then the scenario status should be "failed"
|
|
And the step "broken step" should have status "failed"
|
|
|
|
# --- handle_tdd_expected_fail (standalone entry point) ---
|
|
|
|
Scenario: Handler inverts a failed scenario with failed and skipped steps to passed
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_issue @tdd_issue_999" with status "failed"
|
|
And the mock scenario has a step "broken step" with status "failed"
|
|
And the mock scenario has a step "skipped step" with status "skipped"
|
|
When the TDD expected-fail handler processes the scenario
|
|
Then the scenario status should be "passed"
|
|
And the step "broken step" should have status "passed"
|
|
And the step "skipped step" should have status "passed"
|
|
And the step "broken step" should have error_message cleared
|
|
And the step "skipped step" should have error_message cleared
|
|
|
|
Scenario: Handler fails a passed scenario that still carries @tdd_expected_fail
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_issue @tdd_issue_999" with status "passed"
|
|
And the mock scenario has a step "last step" with status "passed"
|
|
When the TDD expected-fail handler processes the scenario
|
|
Then the scenario status should be "failed"
|
|
And the step "last step" should have a synthetic error_message
|
|
|
|
Scenario: Handler rejects @tdd_expected_fail without @tdd_issue
|
|
Given a mock scenario tagged "@tdd_expected_fail" with status "failed"
|
|
When the TDD expected-fail handler processes the scenario
|
|
Then the scenario status should be "failed"
|
|
|
|
Scenario: Handler rejects @tdd_expected_fail without @tdd_issue_N
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_issue" with status "failed"
|
|
When the TDD expected-fail handler processes the scenario
|
|
Then the scenario status should be "failed"
|
|
|
|
Scenario: Handler rejects @tdd_issue_N without @tdd_issue unconditionally
|
|
Given a mock scenario tagged "@tdd_issue_999" with status "passed"
|
|
When the TDD expected-fail handler processes the scenario
|
|
Then the scenario status should be "failed"
|
|
|
|
Scenario: Handler ignores scenarios without @tdd_expected_fail
|
|
Given a mock scenario tagged "@tdd_issue @tdd_issue_999" with status "failed"
|
|
When the TDD expected-fail handler processes the scenario
|
|
Then the scenario status should be "failed"
|
|
|
|
Scenario: Handler does not invert non-AssertionError exceptions
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_issue @tdd_issue_999" with status "failed"
|
|
And the mock scenario has an infrastructure-error step "infra step" with exception "RuntimeError"
|
|
When the TDD expected-fail handler processes the scenario
|
|
Then the scenario status should be "failed"
|
|
And the step "infra step" should have status "failed"
|
|
|
|
# --- apply_tdd_inversion: mixed exception types (S3) ---
|
|
|
|
Scenario: apply_tdd_inversion does not invert when mixed exceptions present
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_issue @tdd_issue_999" with status "failed"
|
|
And the mock scenario has a step "bug step" with status "failed"
|
|
And the mock scenario has an infrastructure-error step "infra step" with exception "RuntimeError"
|
|
When apply_tdd_inversion processes the scenario with failed True
|
|
Then the scenario status should be "failed"
|
|
And the step "bug step" should have status "failed"
|
|
And the step "infra step" should have status "failed"
|
|
|
|
Scenario: Handler does not invert when mixed exceptions present
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_issue @tdd_issue_999" with status "failed"
|
|
And the mock scenario has a step "bug step" with status "failed"
|
|
And the mock scenario has an infrastructure-error step "infra step" with exception "RuntimeError"
|
|
When the TDD expected-fail handler processes the scenario
|
|
Then the scenario status should be "failed"
|
|
And the step "bug step" should have status "failed"
|
|
And the step "infra step" should have status "failed"
|
|
|
|
# --- selective step reset preserves already-passed steps (TC-3) ---
|
|
|
|
Scenario: apply_tdd_inversion preserves already-passed steps during inversion
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_issue @tdd_issue_999" with status "failed"
|
|
And the mock scenario has a step "ok step" with status "passed"
|
|
And the mock scenario has a step "broken step" with status "failed"
|
|
And the mock scenario has a step "skipped step" with status "skipped"
|
|
When apply_tdd_inversion processes the scenario with failed True
|
|
Then the scenario status should be "passed"
|
|
And the step "ok step" should have status "passed"
|
|
And the step "broken step" should have status "passed"
|
|
And the step "skipped step" should have status "passed"
|
|
And the step "broken step" should have error_message cleared
|
|
And the step "skipped step" should have error_message cleared
|
|
|
|
Scenario: Handler preserves already-passed steps during inversion
|
|
Given a mock scenario tagged "@tdd_expected_fail @tdd_issue @tdd_issue_999" with status "failed"
|
|
And the mock scenario has a step "ok step" with status "passed"
|
|
And the mock scenario has a step "broken step" with status "failed"
|
|
And the mock scenario has a step "skipped step" with status "skipped"
|
|
When the TDD expected-fail handler processes the scenario
|
|
Then the scenario status should be "passed"
|
|
And the step "ok step" should have status "passed"
|
|
And the step "broken step" should have status "passed"
|
|
And the step "skipped step" should have status "passed"
|
|
And the step "broken step" should have error_message cleared
|
|
And the step "skipped step" should have error_message cleared
|
|
|
|
# --- before_scenario hook_failed regression test (TC-2) ---
|
|
|
|
Scenario: before_scenario sets hook_failed on invalid TDD tags
|
|
Given a mock scenario tagged "@tdd_expected_fail" with status "untested"
|
|
When before_scenario is called with the mock scenario
|
|
Then the scenario hook_failed flag should be True
|
|
And the scenario status should be "failed"
|
|
|
|
# --- _install_tdd_expected_fail_patch (S2) ---
|
|
|
|
Scenario: _install_tdd_expected_fail_patch sets the patched flag
|
|
When the TDD expected-fail patch installation status is checked
|
|
Then the Scenario class should have _tdd_run_patched set to True
|
|
|
|
Scenario: _install_tdd_expected_fail_patch is idempotent
|
|
When _install_tdd_expected_fail_patch is called twice
|
|
Then the Scenario.run method should not be double-wrapped
|