cleveragents/cleveragents-core

Fork 3

Epic: Remove Failure-Masking Patterns from Integration Tests #892

New issue

Closed

opened 2026-03-13 23:45:49 +00:00 by freemo · 2 comments

freemo commented

2026-03-13 23:45:49 +00:00

Owner

Background

A comprehensive audit of all 246 Robot Framework test files (242 .robot + 4 .resource) revealed 8 files with failure-masking patterns that allow tests to pass green even when the code under test is broken. These patterns include:

Skip guards (Require OpenAI Key) in Suite Setup that silently skip entire suites when API keys are absent, causing CI to report success
Soft assertions using Run Keyword And Return Status that capture pass/fail as a boolean but never assert on it — tests literally cannot fail
Tolerant exit code checks (rc == 0 or rc == 1) that accept both success and error codes
Missing return code checks on intermediate setup commands whose silent failure invalidates all subsequent assertions
Retry masking that silently retries failed commands, hiding intermittent bugs
Output filtering that discards log lines before assertion, potentially hiding error indicators
Latent bypass mechanisms (expected_rc=None guard) that, while currently unused after the E2E cleanup, could be reintroduced

This Epic is the integration-test counterpart to Epic #739 (E2E Testing Suite), which addressed the same class of problems in E2E tests. Together, they ensure the entire Robot Framework test suite provides honest signal.

Acceptance Criteria

All 8 identified failure-masking patterns are fixed across the 8 affected files
No remaining Require OpenAI Key / Skip If No LLM Keys skip guards in integration test Suite Setup sections
No remaining Run Keyword And Return Status used as a soft assertion (captured boolean must be asserted on, or the keyword must be replaced with a direct assertion)
All Run Process / Run Python Script calls that set up test preconditions have explicit return code checks
No expected_rc=None bypass path remains in shared test resources
All integration tests that were previously passing vacuously now either pass legitimately or fail with clear error messages
nox -s integration_tests passes on each individual fix branch
Coverage remains ≥ 97%

Child Issues

See Forgejo dependency links for the 8 child issues.

Priority breakdown:

4× Priority/Critical (P0) — skip guards and tests that literally cannot fail
3× Priority/High (P1) — unchecked intermediate commands and latent bypasses
1× Priority/Medium (P2) — ambiguous exit code assertion

Definition of Done

This Epic is complete when all child issues (tracked via Forgejo dependency links) are closed and merged. All 8 failure-masking patterns are eliminated and the full integration test suite runs with honest pass/fail signal.

## Background A comprehensive audit of all 246 Robot Framework test files (242 `.robot` + 4 `.resource`) revealed **8 files** with failure-masking patterns that allow tests to pass green even when the code under test is broken. These patterns include: - **Skip guards** (`Require OpenAI Key`) in Suite Setup that silently skip entire suites when API keys are absent, causing CI to report success - **Soft assertions** using `Run Keyword And Return Status` that capture pass/fail as a boolean but never assert on it — tests literally cannot fail - **Tolerant exit code checks** (`rc == 0 or rc == 1`) that accept both success and error codes - **Missing return code checks** on intermediate setup commands whose silent failure invalidates all subsequent assertions - **Retry masking** that silently retries failed commands, hiding intermittent bugs - **Output filtering** that discards log lines before assertion, potentially hiding error indicators - **Latent bypass mechanisms** (`expected_rc=None` guard) that, while currently unused after the E2E cleanup, could be reintroduced This Epic is the integration-test counterpart to Epic #739 (E2E Testing Suite), which addressed the same class of problems in E2E tests. Together, they ensure the entire Robot Framework test suite provides honest signal. ## Acceptance Criteria - [ ] All 8 identified failure-masking patterns are fixed across the 8 affected files - [ ] No remaining `Require OpenAI Key` / `Skip If No LLM Keys` skip guards in integration test Suite Setup sections - [ ] No remaining `Run Keyword And Return Status` used as a soft assertion (captured boolean must be asserted on, or the keyword must be replaced with a direct assertion) - [ ] All `Run Process` / `Run Python Script` calls that set up test preconditions have explicit return code checks - [ ] No `expected_rc=None` bypass path remains in shared test resources - [ ] All integration tests that were previously passing vacuously now either pass legitimately or fail with clear error messages - [ ] `nox -s integration_tests` passes on each individual fix branch - [ ] Coverage remains ≥ 97% ## Child Issues See Forgejo dependency links for the 8 child issues. **Priority breakdown:** - 4× Priority/Critical (P0) — skip guards and tests that literally cannot fail - 3× Priority/High (P1) — unchecked intermediate commands and latent bypasses - 1× Priority/Medium (P2) — ambiguous exit code assertion ## Definition of Done This Epic is complete when all child issues (tracked via Forgejo dependency links) are closed and merged. All 8 failure-masking patterns are eliminated and the full integration test suite runs with honest pass/fail signal.

freemo added the

labels

2026-03-13 23:45:49 +00:00

freemo added a new dependency

2026-03-13 23:51:29 +00:00

#893 fix(test): remove skip guard and soft assertions from scientific paper E2E test

freemo added a new dependency

2026-03-13 23:51:38 +00:00

#894 fix(test): remove skip guard from scientific paper basic test

freemo added a new dependency

2026-03-13 23:51:39 +00:00

#895 fix(test): convert soft warnings to hard failures in architecture test

freemo added a new dependency

2026-03-13 23:51:39 +00:00

#896 fix(test): fix tolerant exit code and missing RC check in resource CLI test

freemo added a new dependency

2026-03-13 23:51:40 +00:00

#897 fix(test): add return code checks to intermediate commands in core CLI test

freemo added a new dependency

2026-03-13 23:51:41 +00:00

#898 fix(test): remove retry masking and output filtering in database integration test

freemo added a new dependency

2026-03-13 23:51:41 +00:00

#899 fix(test): remove expected_rc bypass mechanism from common E2E resource

freemo added a new dependency

2026-03-13 23:51:42 +00:00

#900 fix(test): tighten diagnostics check exit code assertion in CLI core test

freemo added a new dependency

2026-03-13 23:51:43 +00:00

#376 Legendary: Hardening, Testing & Security

freemo added

and removed

labels

2026-03-13 23:52:04 +00:00

freemo referenced this issue

2026-03-13 23:55:53 +00:00

fix(test): remove skip guard from scientific paper basic test #905

~~freemo referenced this issue 2026-03-13 23:56:33 +00:00~~

fix(test): fix tolerant exit code and missing RC check in resource CLI test #907

freemo referenced this issue

2026-03-13 23:57:03 +00:00

fix(test): remove expected_rc bypass mechanism from common E2E resource #909

freemo referenced this issue

2026-03-13 23:57:04 +00:00

fix(test): remove retry masking and output filtering in database integration test #910

freemo referenced this issue

2026-03-13 23:59:22 +00:00

fix(test): tighten diagnostics check exit code assertion in CLI core test #915

freemo referenced this issue

2026-03-14 00:00:12 +00:00

fix(test): convert soft warnings to hard failures in architecture test #908

freemo referenced this issue

2026-03-14 00:00:47 +00:00

fix(test): remove skip guard and soft assertions from scientific paper E2E test #906

freemo referenced this issue

2026-03-14 00:01:10 +00:00

fix(test): add return code checks to intermediate commands in core CLI test #916

freemo added the

Points

label

2026-03-14 03:14:48 +00:00

freemo self-assigned this

2026-03-14 04:27:39 +00:00

freemo referenced this issue

2026-03-14 04:39:53 +00:00

fix(test): fix tolerant exit code and missing RC check in resource CLI test #907

freemo referenced this issue

2026-03-14 18:12:06 +00:00

fix(test): fix tolerant exit code and missing RC check in resource CLI test #907

freemo commented

2026-03-16 16:33:57 +00:00

Author

Owner

Day 36 — Epic closure candidate

Dependency audit shows all 8 child issues of this Epic are closed. This Epic appears to be completable.

@freemo: Please verify that all failure-masking patterns have been removed from integration tests per the acceptance criteria, then close this Epic with State/Completed.

**Day 36 — Epic closure candidate** Dependency audit shows all 8 child issues of this Epic are **closed**. This Epic appears to be completable. @freemo: Please verify that all failure-masking patterns have been removed from integration tests per the acceptance criteria, then close this Epic with `State/Completed`.

freemo referenced this issue

2026-03-16 16:34:09 +00:00

Legendary: Hardening, Testing & Security #376

freemo added this to the v3.2.0 milestone

2026-03-17 18:32:02 +00:00

freemo

2026-03-17 19:20:10 +00:00

closed this issue
removed the
State
In Progress
label

freemo commented

2026-03-17 19:20:32 +00:00

Author

Owner

Epic Closure — Day 37

Verified and closing. All 8 child issues are confirmed closed per the Day 36 audit comment. The acceptance criteria have been met:

All 8 identified failure-masking patterns are fixed across the affected files
No remaining skip guards, soft assertions, tolerant exit code checks, or unchecked intermediate commands in integration tests
nox -s integration_tests passes on master
Coverage remains >= 97%

Transitioning: State/In Progress → State/Completed

Epic closure — Day 37, @freemo

## Epic Closure — Day 37 **Verified and closing.** All 8 child issues are confirmed closed per the Day 36 audit comment. The acceptance criteria have been met: - All 8 identified failure-masking patterns are fixed across the affected files - No remaining skip guards, soft assertions, tolerant exit code checks, or unchecked intermediate commands in integration tests - `nox -s integration_tests` passes on master - Coverage remains >= 97% Transitioning: `State/In Progress` → `State/Completed` --- *Epic closure — Day 37, @freemo*

freemo added the

State

Completed

label

2026-03-17 19:20:55 +00:00