feat(tests): replace all @skip tags with proper @tdd_expected_fail tags or remove them across the entire codebase #7025

Closed
opened 2026-04-10 07:05:41 +00:00 by hurui200320 · 1 comment
Member

Metadata

  • Commit Message: feat(tests): replace all @skip tags with proper @tdd_expected_fail tags or remove them across the entire codebase
  • Branch: feat/skip-tag-to-tdd-expected-fail

Background and Context

A codebase-wide audit identified 234 occurrences of @skip tags across 82 Behave feature files. These were introduced as a temporary workaround before the @tdd_expected_fail infrastructure (documented in CONTRIBUTING.md § Bug Fix Workflow) was fully implemented.

The project's noxfile.py runs Behave with --tags=not @skip, meaning all @skip-tagged scenarios are excluded from CI entirely — they never run, never invert results, and never contribute to coverage. This defeats the purpose of TDD issue-capture testing.

Every single @skip occurrence has a commented-out line immediately above it showing the intended proper tags (e.g., # @tdd_issue @tdd_issue_4272 @tdd_expected_fail @skip). This confirms all 234 cases are candidates for conversion.

Existing issues (#5692, #5661, #5650, #5648, #5590, #5583, #5530, #6115, #4443) each address individual files in isolation. This ticket covers the comprehensive codebase-wide sweep.

Current Behavior

  • 82 feature files contain @skip tags (234 total occurrences)
  • noxfile.py passes --tags=not @skip to all Behave runs, so these tests never execute
  • 14 features have @skip at the feature level (entire feature excluded)
  • 220 individual scenarios have @skip at the scenario level
  • All @skip occurrences have preceding comments showing the intended @tdd_expected_fail @tdd_issue @tdd_issue_<N> tags

Expected Behavior

Each @skip occurrence is handled based on whether its referenced bug issue is open or closed:

Category A — Bug still OPEN: Replace @skip with @tdd_expected_fail @tdd_issue @tdd_issue_<N> (as shown in the comment above). The test runs via CI with result inversion (fail → pass while bug exists; pass → fail if bug is fixed without removing tag).

Category B — Bug already CLOSED/FIXED: Remove @skip entirely (and remove @tdd_expected_fail if present), keeping @tdd_issue @tdd_issue_<N> as permanent regression guard tags. The test runs normally and must pass.

In both categories:

  • The commented-out hint line above each @skip is removed (it becomes redundant after replacement)
  • The @skip tag itself is removed

Acceptance Criteria

  • All 234 @skip occurrences across 82 feature files are resolved (either replaced or removed)
  • For Category A (bug open): scenario is tagged @tdd_expected_fail @tdd_issue @tdd_issue_<N> and passes CI via result inversion
  • For Category B (bug closed): scenario runs as a normal regression guard and passes
  • No @skip tags remain in any .feature file (confirmed by grep -r "@skip" features/ --include="*.feature" returning empty)
  • The commented-out hint lines (# @tdd_issue ... @skip) are removed alongside the @skip replacement
  • nox -s unit_tests passes with all converted tests included
  • nox -s coverage_report meets or exceeds 97% threshold

Supporting Information

Scope breakdown by issue number (Category A — open bugs, need @tdd_expected_fail):

Feature-level @skip files (entire feature excluded):

  • tdd_tool_runner_env_precedence.feature (#2592/#4290)
  • tdd_skill_add_regression.feature (#980/#4287)
  • tdd_server_connect_atomic_writes.feature (#993)
  • tdd_missing_validation_unit_tests_yaml.feature (#1039)
  • tdd_mcp_error_content_key.feature (#2158/#4284)
  • tdd_json_decode_crash_persistence.feature (#989)
  • tdd_indentation_library_try_except.feature (#2845/#4282)
  • tdd_exec_env_resolution_precedence.feature (#1080/#4281)
  • tdd_correction_checkpoint_wiring.feature (#986/#4279)
  • tdd_context_tier_runtime.feature (#821/#4278)
  • tdd_checkpoint_real_rollback.feature (#822/#4276)
  • tdd_automation_profile_session_leak.feature (#987/#4274)
  • tdd_a2a_sdk_dependency.feature (#2922/#4273)
  • session_list_summary_dedup.feature (#3046/#4271)

Related issues (narrowly scoped, this ticket supersedes them): #5692, #5661, #5650, #5648, #5590, #5583, #5530, #6115, #4443

Subtasks

  • Audit all 234 @skip occurrences: check referenced issue numbers and determine if each is open (Category A) or closed (Category B)
  • Category B cleanup: for each closed-issue @skip, remove @skip + @tdd_expected_fail comment/tags, keep @tdd_issue @tdd_issue_<N>, verify scenario passes normally
  • Category A conversion: for each open-issue @skip, replace with @tdd_expected_fail @tdd_issue @tdd_issue_<N> from the commented hint, remove the comment line
  • Handle the 3 special cases where only @tdd_expected_fail is needed (feature already carries @tdd_issue @tdd_issue_<N> at feature level): tdd_cli_incomplete_subcommand_registration.feature:33, tdd_event_bus_exception_swallow.feature:30, tdd_exec_env_resolution_precedence.feature:35
  • Handle the 14 feature-level @skip tags (replace or remove at feature level, then handle each scenario individually)
  • Remove all commented hint lines (e.g., # @tdd_issue @tdd_issue_4272 @tdd_expected_fail @skip) after replacing the @skip tag below them
  • Run grep -r "@skip" features/ --include="*.feature" and confirm empty output
  • Run nox -s unit_tests — verify all converted tests pass CI
  • Run nox -s coverage_report — verify coverage stays ≥ 97%

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
## Metadata - **Commit Message**: `feat(tests): replace all @skip tags with proper @tdd_expected_fail tags or remove them across the entire codebase` - **Branch**: `feat/skip-tag-to-tdd-expected-fail` ## Background and Context A codebase-wide audit identified **234 occurrences** of `@skip` tags across **82 Behave feature files**. These were introduced as a temporary workaround before the `@tdd_expected_fail` infrastructure (documented in CONTRIBUTING.md § Bug Fix Workflow) was fully implemented. The project's `noxfile.py` runs Behave with `--tags=not @skip`, meaning all `@skip`-tagged scenarios are **excluded from CI entirely** — they never run, never invert results, and never contribute to coverage. This defeats the purpose of TDD issue-capture testing. Every single `@skip` occurrence has a commented-out line immediately above it showing the intended proper tags (e.g., `# @tdd_issue @tdd_issue_4272 @tdd_expected_fail @skip`). This confirms all 234 cases are candidates for conversion. Existing issues (#5692, #5661, #5650, #5648, #5590, #5583, #5530, #6115, #4443) each address individual files in isolation. This ticket covers the **comprehensive codebase-wide sweep**. ## Current Behavior - 82 feature files contain `@skip` tags (234 total occurrences) - `noxfile.py` passes `--tags=not @skip` to all Behave runs, so these tests never execute - 14 features have `@skip` at the **feature level** (entire feature excluded) - 220 individual scenarios have `@skip` at the **scenario level** - All `@skip` occurrences have preceding comments showing the intended `@tdd_expected_fail @tdd_issue @tdd_issue_<N>` tags ## Expected Behavior Each `@skip` occurrence is handled based on whether its referenced bug issue is open or closed: **Category A — Bug still OPEN:** Replace `@skip` with `@tdd_expected_fail @tdd_issue @tdd_issue_<N>` (as shown in the comment above). The test runs via CI with result inversion (fail → pass while bug exists; pass → fail if bug is fixed without removing tag). **Category B — Bug already CLOSED/FIXED:** Remove `@skip` entirely (and remove `@tdd_expected_fail` if present), keeping `@tdd_issue @tdd_issue_<N>` as permanent regression guard tags. The test runs normally and must pass. In both categories: - The commented-out hint line above each `@skip` is removed (it becomes redundant after replacement) - The `@skip` tag itself is removed ## Acceptance Criteria - [ ] All 234 `@skip` occurrences across 82 feature files are resolved (either replaced or removed) - [ ] For Category A (bug open): scenario is tagged `@tdd_expected_fail @tdd_issue @tdd_issue_<N>` and passes CI via result inversion - [ ] For Category B (bug closed): scenario runs as a normal regression guard and passes - [ ] No `@skip` tags remain in any `.feature` file (confirmed by `grep -r "@skip" features/ --include="*.feature"` returning empty) - [ ] The commented-out hint lines (`# @tdd_issue ... @skip`) are removed alongside the `@skip` replacement - [ ] `nox -s unit_tests` passes with all converted tests included - [ ] `nox -s coverage_report` meets or exceeds 97% threshold ## Supporting Information **Scope breakdown by issue number (Category A — open bugs, need @tdd_expected_fail):** - Issues #4229–#4295: 50+ scenarios across 30+ files (batch of related skip-to-tdd issues) - Issues #523, #524, #589, #590, #1543, #2145, #2147, #2609, #2911: older open bugs **Feature-level @skip files (entire feature excluded):** - `tdd_tool_runner_env_precedence.feature` (#2592/#4290) - `tdd_skill_add_regression.feature` (#980/#4287) - `tdd_server_connect_atomic_writes.feature` (#993) - `tdd_missing_validation_unit_tests_yaml.feature` (#1039) - `tdd_mcp_error_content_key.feature` (#2158/#4284) - `tdd_json_decode_crash_persistence.feature` (#989) - `tdd_indentation_library_try_except.feature` (#2845/#4282) - `tdd_exec_env_resolution_precedence.feature` (#1080/#4281) - `tdd_correction_checkpoint_wiring.feature` (#986/#4279) - `tdd_context_tier_runtime.feature` (#821/#4278) - `tdd_checkpoint_real_rollback.feature` (#822/#4276) - `tdd_automation_profile_session_leak.feature` (#987/#4274) - `tdd_a2a_sdk_dependency.feature` (#2922/#4273) - `session_list_summary_dedup.feature` (#3046/#4271) **Related issues (narrowly scoped, this ticket supersedes them):** #5692, #5661, #5650, #5648, #5590, #5583, #5530, #6115, #4443 ## Subtasks - [ ] Audit all 234 `@skip` occurrences: check referenced issue numbers and determine if each is open (Category A) or closed (Category B) - [ ] Category B cleanup: for each closed-issue `@skip`, remove `@skip` + `@tdd_expected_fail` comment/tags, keep `@tdd_issue @tdd_issue_<N>`, verify scenario passes normally - [ ] Category A conversion: for each open-issue `@skip`, replace with `@tdd_expected_fail @tdd_issue @tdd_issue_<N>` from the commented hint, remove the comment line - [ ] Handle the 3 special cases where only `@tdd_expected_fail` is needed (feature already carries `@tdd_issue @tdd_issue_<N>` at feature level): `tdd_cli_incomplete_subcommand_registration.feature:33`, `tdd_event_bus_exception_swallow.feature:30`, `tdd_exec_env_resolution_precedence.feature:35` - [ ] Handle the 14 feature-level `@skip` tags (replace or remove at feature level, then handle each scenario individually) - [ ] Remove all commented hint lines (e.g., `# @tdd_issue @tdd_issue_4272 @tdd_expected_fail @skip`) after replacing the `@skip` tag below them - [ ] Run `grep -r "@skip" features/ --include="*.feature"` and confirm empty output - [ ] Run `nox -s unit_tests` — verify all converted tests pass CI - [ ] Run `nox -s coverage_report` — verify coverage stays ≥ 97% ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done.
hurui200320 added this to the v3.5.0 milestone 2026-04-10 09:19:55 +00:00
Author
Member

Implementation complete. Pull request submitted: #7221

PR #7221 blocks this issue (this issue depends on PR #7221). Note: the Forgejo dependency API returned an error when setting the explicit UI dependency link for a PR-to-issue cross-link; the Closes #7025 keyword in the PR description handles the functional dependency.

Implementation complete. Pull request submitted: #7221 PR #7221 **blocks** this issue (this issue depends on PR #7221). Note: the Forgejo dependency API returned an error when setting the explicit UI dependency link for a PR-to-issue cross-link; the `Closes #7025` keyword in the PR description handles the functional dependency.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#7025
No description provided.