51aab18411
CI / benchmark-publish (pull_request) Has been skipped
CI / helm (pull_request) Successful in 26s
CI / lint (pull_request) Successful in 29s
CI / push-validation (pull_request) Successful in 28s
CI / quality (pull_request) Successful in 31s
CI / build (pull_request) Successful in 32s
CI / typecheck (pull_request) Successful in 59s
CI / security (pull_request) Successful in 59s
CI / e2e_tests (pull_request) Successful in 3m0s
CI / integration_tests (pull_request) Successful in 4m3s
CI / unit_tests (pull_request) Successful in 5m21s
CI / docker (pull_request) Successful in 1m19s
CI / coverage (pull_request) Successful in 10m25s
CI / status-check (pull_request) Successful in 1s
CI / push-validation (push) Successful in 17s
CI / build (push) Successful in 27s
CI / helm (push) Successful in 29s
CI / quality (push) Successful in 32s
CI / lint (push) Successful in 35s
CI / security (push) Successful in 51s
CI / typecheck (push) Successful in 51s
CI / benchmark-regression (push) Has been skipped
CI / e2e_tests (push) Successful in 3m7s
CI / integration_tests (push) Successful in 4m0s
CI / unit_tests (push) Successful in 4m54s
CI / docker (push) Successful in 21s
CI / coverage (push) Successful in 10m24s
CI / status-check (push) Successful in 1s
CI / benchmark-regression (pull_request) Successful in 59m2s
CI / benchmark-publish (push) Successful in 1h13m43s
Add three guard conditions to the Robot Framework tdd_expected_fail_listener end_test() function, paralleling the Behave apply_tdd_inversion() guards in features/environment.py. These guards prevent the listener from blindly inverting ALL test failures to passes, which was masking infrastructure errors and causing flaky CI behavior. Guards added: 1. Setup/teardown error guard (_has_setup_teardown_failure): checks result.setup/teardown status and message prefix to detect infrastructure failures where the test body never executed. 2. Non-assertion failure guard (_is_infrastructure_error): checks result.message against known infrastructure error patterns (keyword resolution errors, Python exception types, network errors) to avoid inverting failures that are not the captured bug. 3. Dry-run guard: detects Robot Framework dry-run mode by checking if all body keywords have status NOT RUN, preventing meaningless inversions when no test actually executed. Also fixes collateral issues exposed by the guards: - Fixed Variable Should Exist syntax error in e2e test files (single space was being parsed as part of keyword name instead of separator) - Removed tdd_expected_fail from 4 context assembly e2e tests where the bugs appear to be fixed (previously masked by syntax error + blind inversion) ISSUES CLOSED: #5436
80 lines
2.3 KiB
Python
80 lines
2.3 KiB
Python
"""Guard command helpers for Robot TDD listener validation fixtures."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import sys
|
|
from collections.abc import Callable
|
|
|
|
from _tdd_fixture_runner import run_fixture, run_fixture_dryrun
|
|
|
|
__all__ = [
|
|
"GUARD_COMMANDS",
|
|
"cmd_dry_run_guard",
|
|
"cmd_infra_error_guard",
|
|
"cmd_setup_error_guard",
|
|
"cmd_teardown_error_guard",
|
|
]
|
|
|
|
|
|
def cmd_setup_error_guard() -> int:
|
|
"""Verify setup error guard prevents inversion."""
|
|
status, message = run_fixture("tdd_expected_fail_setup_error")
|
|
if status == "FAIL" and "Setup failed" in message:
|
|
print("tdd-setup-error-guard-ok")
|
|
return 0
|
|
print(
|
|
f"FAIL: Expected FAIL with 'Setup failed' message but got "
|
|
f"{status}. Message: {message}",
|
|
file=sys.stderr,
|
|
)
|
|
return 1
|
|
|
|
|
|
def cmd_infra_error_guard() -> int:
|
|
"""Verify infrastructure error guard prevents inversion."""
|
|
status, message = run_fixture("tdd_expected_fail_infra_error")
|
|
if status == "FAIL" and "No keyword with name" in message:
|
|
print("tdd-infra-error-guard-ok")
|
|
return 0
|
|
print(
|
|
f"FAIL: Expected FAIL with 'No keyword with name' message but "
|
|
f"got {status}. Message: {message}",
|
|
file=sys.stderr,
|
|
)
|
|
return 1
|
|
|
|
|
|
def cmd_teardown_error_guard() -> int:
|
|
"""Verify teardown error guard prevents inversion."""
|
|
status, message = run_fixture("tdd_expected_fail_teardown_error")
|
|
if status == "FAIL" and "Teardown failed" in message:
|
|
print("tdd-teardown-error-guard-ok")
|
|
return 0
|
|
print(
|
|
f"FAIL: Expected FAIL with 'Teardown failed' message but got "
|
|
f"{status}. Message: {message}",
|
|
file=sys.stderr,
|
|
)
|
|
return 1
|
|
|
|
|
|
def cmd_dry_run_guard() -> int:
|
|
"""Verify dry-run guard prevents inversion."""
|
|
status, message = run_fixture_dryrun("tdd_expected_fail_fails")
|
|
if status == "PASS":
|
|
print("tdd-dry-run-guard-ok")
|
|
return 0
|
|
print(
|
|
f"FAIL: Expected PASS in dry-run mode but got {status}. Message: {message}",
|
|
file=sys.stderr,
|
|
)
|
|
return 1
|
|
|
|
|
|
GUARD_COMMANDS: dict[str, Callable[[], int]] = {
|
|
"setup-error-guard": cmd_setup_error_guard,
|
|
"infra-error-guard": cmd_infra_error_guard,
|
|
"teardown-error-guard": cmd_teardown_error_guard,
|
|
"dry-run-guard": cmd_dry_run_guard,
|
|
}
|