fix(test): convert soft warnings to hard failures in architecture test #895

Closed
opened 2026-03-13 23:46:42 +00:00 by freemo · 0 comments
Owner

Metadata

  • Commit Message: fix(test): convert soft warnings to hard failures in architecture test
  • Branch: fix/integration-architecture-warn-to-fail

Background

The architecture test (robot/architecture.robot, 109 lines) contains multiple failure-masking patterns where checks are captured as booleans but never cause test failure.

Problems Identified

  1. Docstring soft-check (lines 35–36): Run Keyword And Return Status Should Contain ${content} """ captures the docstring check as a boolean (${has_docstring}), then Run Keyword If not ${has_docstring} Log Missing docstring WARN merely logs a warning. The test can never fail — it always passes green regardless of whether docstrings are present.

  2. File existence soft-checks (lines 70, 82): File Should Exist is wrapped in Run Keyword And Return Status, capturing the result in ${exists}. If the file doesn't exist, all assertions inside the IF ${exists} block are silently skipped. The test passes vacuously.

Acceptance Criteria

  • Lines 35–36: Docstring check uses a hard Should Contain assertion that fails the test if docstrings are missing
  • Lines 70, 82: File existence checks use direct File Should Exist assertions (hard failure)
  • All assertions that were previously inside conditional IF ${exists} blocks now execute unconditionally
  • No remaining Run Keyword And Return Status used as a soft assertion in this file
  • nox -s integration_tests passes

Subtasks

  • Replace Run Keyword And Return Status + Log WARN pattern with direct Should Contain for docstring check
  • Replace Run Keyword And Return Status file existence checks with direct File Should Exist
  • Remove conditional IF ${exists} blocks — make assertions unconditional
  • Run nox -s integration_tests and verify the test suite passes
  • Verify coverage >= 97% via nox -s coverage_report

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
## Metadata - **Commit Message**: `fix(test): convert soft warnings to hard failures in architecture test` - **Branch**: `fix/integration-architecture-warn-to-fail` ## Background The architecture test (`robot/architecture.robot`, 109 lines) contains multiple failure-masking patterns where checks are captured as booleans but never cause test failure. ### Problems Identified 1. **Docstring soft-check** (lines 35–36): `Run Keyword And Return Status Should Contain ${content} """` captures the docstring check as a boolean (`${has_docstring}`), then `Run Keyword If not ${has_docstring} Log Missing docstring WARN` merely logs a warning. The test **can never fail** — it always passes green regardless of whether docstrings are present. 2. **File existence soft-checks** (lines 70, 82): `File Should Exist` is wrapped in `Run Keyword And Return Status`, capturing the result in `${exists}`. If the file doesn't exist, all assertions inside the `IF ${exists}` block are silently skipped. The test passes vacuously. ## Acceptance Criteria - [ ] Lines 35–36: Docstring check uses a hard `Should Contain` assertion that fails the test if docstrings are missing - [ ] Lines 70, 82: File existence checks use direct `File Should Exist` assertions (hard failure) - [ ] All assertions that were previously inside conditional `IF ${exists}` blocks now execute unconditionally - [ ] No remaining `Run Keyword And Return Status` used as a soft assertion in this file - [ ] `nox -s integration_tests` passes ## Subtasks - [ ] Replace `Run Keyword And Return Status` + `Log WARN` pattern with direct `Should Contain` for docstring check - [ ] Replace `Run Keyword And Return Status` file existence checks with direct `File Should Exist` - [ ] Remove conditional `IF ${exists}` blocks — make assertions unconditional - [ ] Run `nox -s integration_tests` and verify the test suite passes - [ ] Verify coverage >= 97% via `nox -s coverage_report` ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done.
freemo added this to the v3.2.0 milestone 2026-03-13 23:46:42 +00:00
freemo 2026-03-14 03:31:43 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#895
No description provided.