fix(test): remove skip guard and soft assertions from scientific paper E2E test #893

Closed
opened 2026-03-13 23:46:23 +00:00 by freemo · 0 comments
Owner

Metadata

  • Commit Message: fix(test): remove skip guard and soft assertions from scientific paper E2E test
  • Branch: fix/integration-paper-e2e-skip-guard

Background

The scientific paper E2E test (robot/scientific_paper_e2e_test.robot, 283 lines) contains multiple failure-masking patterns that allow tests to pass green even when the underlying functionality is broken.

Problems Identified

  1. Skip guard in Suite Setup (lines 280–282): Require OpenAI Key silently skips the entire suite when OPENAI_API_KEY is absent. Robot Framework treats skipped tests as non-failures (exit code 0), so CI reports green.

  2. Unchecked return code (line 73): ${rc_ok}= Run Keyword And Return Status Should Be Equal As Integers ${result.rc} 0 captures the RC check result as a boolean but never asserts on it. The test cannot fail on a non-zero exit code.

  3. Soft LaTeX presence check (lines 137–145): LaTeX file existence is checked via Run Keyword And Return Status, and if absent, the test merely logs a warning and continues — all subsequent LaTeX-dependent assertions are silently skipped.

  4. Soft latex_source context field check (lines 158–163): Same pattern — Run Keyword And Return Status captures whether the context field exists, then skips assertions if it doesn't.

  5. Dead stderr check (line 130): Should Not Contain ${result.stderr} Error is meaningless because stderr=STDOUT was used on line 127, making ${result.stderr} always empty.

Acceptance Criteria

  • Require OpenAI Key skip guard is removed from Suite Setup
  • Line 73: Run Keyword And Return Status is replaced with a direct Should Be Equal As Integers ${result.rc} 0 assertion
  • Lines 137–145: LaTeX file existence uses File Should Exist directly (hard failure)
  • Lines 158–163: latex_source context field check uses a direct assertion
  • Line 130: Dead stderr check is either removed or stderr=STDOUT is removed so stderr is actually captured
  • nox -s integration_tests passes
  • No new Run Keyword And Return Status used as soft assertion remains

Subtasks

  • Remove Require OpenAI Key from Suite Setup and the keyword definition if it exists locally
  • Replace Run Keyword And Return Status RC check with direct assertion on line 73
  • Convert LaTeX file soft-check to hard File Should Exist assertion
  • Convert latex_source context field soft-check to hard assertion
  • Fix or remove dead stderr check on line 130
  • Run nox -s integration_tests and verify the test suite passes
  • Verify coverage >= 97% via nox -s coverage_report

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
## Metadata - **Commit Message**: `fix(test): remove skip guard and soft assertions from scientific paper E2E test` - **Branch**: `fix/integration-paper-e2e-skip-guard` ## Background The scientific paper E2E test (`robot/scientific_paper_e2e_test.robot`, 283 lines) contains multiple failure-masking patterns that allow tests to pass green even when the underlying functionality is broken. ### Problems Identified 1. **Skip guard in Suite Setup** (lines 280–282): `Require OpenAI Key` silently skips the entire suite when `OPENAI_API_KEY` is absent. Robot Framework treats skipped tests as non-failures (exit code 0), so CI reports green. 2. **Unchecked return code** (line 73): `${rc_ok}= Run Keyword And Return Status Should Be Equal As Integers ${result.rc} 0` captures the RC check result as a boolean but **never asserts on it**. The test cannot fail on a non-zero exit code. 3. **Soft LaTeX presence check** (lines 137–145): LaTeX file existence is checked via `Run Keyword And Return Status`, and if absent, the test merely logs a warning and continues — all subsequent LaTeX-dependent assertions are silently skipped. 4. **Soft `latex_source` context field check** (lines 158–163): Same pattern — `Run Keyword And Return Status` captures whether the context field exists, then skips assertions if it doesn't. 5. **Dead stderr check** (line 130): `Should Not Contain ${result.stderr} Error` is meaningless because `stderr=STDOUT` was used on line 127, making `${result.stderr}` always empty. ## Acceptance Criteria - [ ] `Require OpenAI Key` skip guard is removed from Suite Setup - [ ] Line 73: `Run Keyword And Return Status` is replaced with a direct `Should Be Equal As Integers ${result.rc} 0` assertion - [ ] Lines 137–145: LaTeX file existence uses `File Should Exist` directly (hard failure) - [ ] Lines 158–163: `latex_source` context field check uses a direct assertion - [ ] Line 130: Dead `stderr` check is either removed or `stderr=STDOUT` is removed so stderr is actually captured - [ ] `nox -s integration_tests` passes - [ ] No new `Run Keyword And Return Status` used as soft assertion remains ## Subtasks - [ ] Remove `Require OpenAI Key` from Suite Setup and the keyword definition if it exists locally - [ ] Replace `Run Keyword And Return Status` RC check with direct assertion on line 73 - [ ] Convert LaTeX file soft-check to hard `File Should Exist` assertion - [ ] Convert `latex_source` context field soft-check to hard assertion - [ ] Fix or remove dead stderr check on line 130 - [ ] Run `nox -s integration_tests` and verify the test suite passes - [ ] Verify coverage >= 97% via `nox -s coverage_report` ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done.
freemo added this to the v3.2.0 milestone 2026-03-13 23:46:23 +00:00
freemo 2026-03-14 00:56:37 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#893
No description provided.