Commit Graph

1 Commits

Author SHA1 Message Date
hurui200320 51aab18411 fix(testing): add guard logic to Robot tdd_expected_fail listener to prevent flaky CI
CI / benchmark-publish (pull_request) Has been skipped
CI / helm (pull_request) Successful in 26s
CI / lint (pull_request) Successful in 29s
CI / push-validation (pull_request) Successful in 28s
CI / quality (pull_request) Successful in 31s
CI / build (pull_request) Successful in 32s
CI / typecheck (pull_request) Successful in 59s
CI / security (pull_request) Successful in 59s
CI / e2e_tests (pull_request) Successful in 3m0s
CI / integration_tests (pull_request) Successful in 4m3s
CI / unit_tests (pull_request) Successful in 5m21s
CI / docker (pull_request) Successful in 1m19s
CI / coverage (pull_request) Successful in 10m25s
CI / status-check (pull_request) Successful in 1s
CI / push-validation (push) Successful in 17s
CI / build (push) Successful in 27s
CI / helm (push) Successful in 29s
CI / quality (push) Successful in 32s
CI / lint (push) Successful in 35s
CI / security (push) Successful in 51s
CI / typecheck (push) Successful in 51s
CI / benchmark-regression (push) Has been skipped
CI / e2e_tests (push) Successful in 3m7s
CI / integration_tests (push) Successful in 4m0s
CI / unit_tests (push) Successful in 4m54s
CI / docker (push) Successful in 21s
CI / coverage (push) Successful in 10m24s
CI / status-check (push) Successful in 1s
CI / benchmark-regression (pull_request) Successful in 59m2s
CI / benchmark-publish (push) Successful in 1h13m43s
Add three guard conditions to the Robot Framework tdd_expected_fail_listener
end_test() function, paralleling the Behave apply_tdd_inversion() guards in
features/environment.py. These guards prevent the listener from blindly
inverting ALL test failures to passes, which was masking infrastructure errors
and causing flaky CI behavior.

Guards added:
1. Setup/teardown error guard (_has_setup_teardown_failure): checks
   result.setup/teardown status and message prefix to detect infrastructure
   failures where the test body never executed.
2. Non-assertion failure guard (_is_infrastructure_error): checks
   result.message against known infrastructure error patterns (keyword
   resolution errors, Python exception types, network errors) to avoid
   inverting failures that are not the captured bug.
3. Dry-run guard: detects Robot Framework dry-run mode by checking if all
   body keywords have status NOT RUN, preventing meaningless inversions
   when no test actually executed.

Also fixes collateral issues exposed by the guards:
- Fixed Variable Should Exist syntax error in e2e test files (single space
  was being parsed as part of keyword name instead of separator)
- Removed tdd_expected_fail from 4 context assembly e2e tests where the
  bugs appear to be fixed (previously masked by syntax error + blind
  inversion)

ISSUES CLOSED: #5436
2026-04-10 05:21:24 +00:00