Files
cleveragents-core/robot/helper_tdd_guard_commands.py
hurui200320 51aab18411
CI / benchmark-publish (pull_request) Has been skipped
CI / helm (pull_request) Successful in 26s
CI / lint (pull_request) Successful in 29s
CI / push-validation (pull_request) Successful in 28s
CI / quality (pull_request) Successful in 31s
CI / build (pull_request) Successful in 32s
CI / typecheck (pull_request) Successful in 59s
CI / security (pull_request) Successful in 59s
CI / e2e_tests (pull_request) Successful in 3m0s
CI / integration_tests (pull_request) Successful in 4m3s
CI / unit_tests (pull_request) Successful in 5m21s
CI / docker (pull_request) Successful in 1m19s
CI / coverage (pull_request) Successful in 10m25s
CI / status-check (pull_request) Successful in 1s
CI / push-validation (push) Successful in 17s
CI / build (push) Successful in 27s
CI / helm (push) Successful in 29s
CI / quality (push) Successful in 32s
CI / lint (push) Successful in 35s
CI / security (push) Successful in 51s
CI / typecheck (push) Successful in 51s
CI / benchmark-regression (push) Has been skipped
CI / e2e_tests (push) Successful in 3m7s
CI / integration_tests (push) Successful in 4m0s
CI / unit_tests (push) Successful in 4m54s
CI / docker (push) Successful in 21s
CI / coverage (push) Successful in 10m24s
CI / status-check (push) Successful in 1s
CI / benchmark-regression (pull_request) Successful in 59m2s
CI / benchmark-publish (push) Successful in 1h13m43s
fix(testing): add guard logic to Robot tdd_expected_fail listener to prevent flaky CI
Add three guard conditions to the Robot Framework tdd_expected_fail_listener
end_test() function, paralleling the Behave apply_tdd_inversion() guards in
features/environment.py. These guards prevent the listener from blindly
inverting ALL test failures to passes, which was masking infrastructure errors
and causing flaky CI behavior.

Guards added:
1. Setup/teardown error guard (_has_setup_teardown_failure): checks
   result.setup/teardown status and message prefix to detect infrastructure
   failures where the test body never executed.
2. Non-assertion failure guard (_is_infrastructure_error): checks
   result.message against known infrastructure error patterns (keyword
   resolution errors, Python exception types, network errors) to avoid
   inverting failures that are not the captured bug.
3. Dry-run guard: detects Robot Framework dry-run mode by checking if all
   body keywords have status NOT RUN, preventing meaningless inversions
   when no test actually executed.

Also fixes collateral issues exposed by the guards:
- Fixed Variable Should Exist syntax error in e2e test files (single space
  was being parsed as part of keyword name instead of separator)
- Removed tdd_expected_fail from 4 context assembly e2e tests where the
  bugs appear to be fixed (previously masked by syntax error + blind
  inversion)

ISSUES CLOSED: #5436
2026-04-10 05:21:24 +00:00

80 lines
2.3 KiB
Python

"""Guard command helpers for Robot TDD listener validation fixtures."""
from __future__ import annotations
import sys
from collections.abc import Callable
from _tdd_fixture_runner import run_fixture, run_fixture_dryrun
__all__ = [
"GUARD_COMMANDS",
"cmd_dry_run_guard",
"cmd_infra_error_guard",
"cmd_setup_error_guard",
"cmd_teardown_error_guard",
]
def cmd_setup_error_guard() -> int:
"""Verify setup error guard prevents inversion."""
status, message = run_fixture("tdd_expected_fail_setup_error")
if status == "FAIL" and "Setup failed" in message:
print("tdd-setup-error-guard-ok")
return 0
print(
f"FAIL: Expected FAIL with 'Setup failed' message but got "
f"{status}. Message: {message}",
file=sys.stderr,
)
return 1
def cmd_infra_error_guard() -> int:
"""Verify infrastructure error guard prevents inversion."""
status, message = run_fixture("tdd_expected_fail_infra_error")
if status == "FAIL" and "No keyword with name" in message:
print("tdd-infra-error-guard-ok")
return 0
print(
f"FAIL: Expected FAIL with 'No keyword with name' message but "
f"got {status}. Message: {message}",
file=sys.stderr,
)
return 1
def cmd_teardown_error_guard() -> int:
"""Verify teardown error guard prevents inversion."""
status, message = run_fixture("tdd_expected_fail_teardown_error")
if status == "FAIL" and "Teardown failed" in message:
print("tdd-teardown-error-guard-ok")
return 0
print(
f"FAIL: Expected FAIL with 'Teardown failed' message but got "
f"{status}. Message: {message}",
file=sys.stderr,
)
return 1
def cmd_dry_run_guard() -> int:
"""Verify dry-run guard prevents inversion."""
status, message = run_fixture_dryrun("tdd_expected_fail_fails")
if status == "PASS":
print("tdd-dry-run-guard-ok")
return 0
print(
f"FAIL: Expected PASS in dry-run mode but got {status}. Message: {message}",
file=sys.stderr,
)
return 1
GUARD_COMMANDS: dict[str, Callable[[], int]] = {
"setup-error-guard": cmd_setup_error_guard,
"infra-error-guard": cmd_infra_error_guard,
"teardown-error-guard": cmd_teardown_error_guard,
"dry-run-guard": cmd_dry_run_guard,
}