TEST-INFRA: [flaky-tests] Improve isolation of wf14_server_mode.robot E2E tests #6272

Open
opened 2026-04-09 19:45:32 +00:00 by HAL9000 · 0 comments
Owner

Metadata

  • Branch: feat/test-infra/flaky-tests-wf14-server-mode-isolation
  • Commit Message: test(infra): improve isolation of wf14_server_mode.robot E2E tests
  • Milestone: Backlog (no milestone — see backlog note below)
  • Parent Epic: #5407

Description

The wf14_server_mode.robot E2E test suite is a candidate for flakiness due to its stateful nature and lack of complete isolation between test cases. The suite relies on a single Suite Setup and Suite Teardown to manage the test environment, which can lead to state leaking between test cases and causing intermittent failures.

Analysis

The wf14_server_mode.robot test suite performs the following stateful operations:

  • Modifies global configuration settings (server.url, server.token, core.namespace).
  • Creates actions and actors with names that could potentially collide in parallel runs.
  • Relies on the filesystem to create temporary YAML files.

While the suite uses a RUN_SUFFIX to mitigate name collisions, a more robust solution is needed to ensure complete test isolation.

Proposed Solution

To improve the reliability of the wf14_server_mode.robot test suite, the following changes are proposed:

  1. Isolate Test Cases: Replace the Suite Setup and Suite Teardown with Test Setup and Test Teardown. This will ensure that each test case runs in a clean, isolated environment.
  2. Use Temporary Namespaces: Instead of a hardcoded namespace (myteam), generate a unique namespace for each test case. This will prevent any potential for resource conflicts between parallel test runs.
  3. Implement Health Checks: Before each test run, perform a health check to verify that the environment is clean. This check should ensure that no leftover configurations, actions, or actors from previous runs exist.

Duplicate Check

  • Search query: flaky — Results: 0
  • Search query: e2e — Results: many, but none related to this specific issue
  • Search query: flaky e2e — Results: 0
  • Search query: stateful — Results: 0
  • Search query: test isolation — Results: 0

None of the existing issues cover the specific problem of test isolation in the wf14_server_mode.robot E2E test suite.

Subtasks

  • Audit wf14_server_mode.robot for all stateful operations and shared state
  • Replace Suite Setup / Suite Teardown with Test Setup / Test Teardown for each test case
  • Implement unique namespace generation per test case (replacing hardcoded myteam)
  • Implement pre-test health check keyword to verify clean environment before each test
  • Implement post-test teardown to clean up all resources created during the test
  • Verify refactored tests pass consistently in CI pipeline (no flakiness over multiple runs)
  • Run nox -s e2e_tests to confirm all E2E tests pass
  • Run nox (all default sessions) and fix any errors

Definition of Done

  • The wf14_server_mode.robot test suite is refactored to use Test Setup and Test Teardown for each test case
  • Each test case in the suite uses a unique, temporary namespace
  • A pre-test health check is implemented to ensure a clean test environment
  • The refactored tests pass consistently in the CI pipeline
  • All nox stages pass
  • Coverage >= 97%

Backlog note: This issue was discovered during autonomous operation
on milestone v3.8.0. It does not block milestone completion and has been
placed in the backlog for human review and future milestone assignment.


Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: new-issue-creator

## Metadata - **Branch**: `feat/test-infra/flaky-tests-wf14-server-mode-isolation` - **Commit Message**: `test(infra): improve isolation of wf14_server_mode.robot E2E tests` - **Milestone**: Backlog (no milestone — see backlog note below) - **Parent Epic**: #5407 ## Description The `wf14_server_mode.robot` E2E test suite is a candidate for flakiness due to its stateful nature and lack of complete isolation between test cases. The suite relies on a single `Suite Setup` and `Suite Teardown` to manage the test environment, which can lead to state leaking between test cases and causing intermittent failures. ### Analysis The `wf14_server_mode.robot` test suite performs the following stateful operations: * Modifies global configuration settings (`server.url`, `server.token`, `core.namespace`). * Creates actions and actors with names that could potentially collide in parallel runs. * Relies on the filesystem to create temporary YAML files. While the suite uses a `RUN_SUFFIX` to mitigate name collisions, a more robust solution is needed to ensure complete test isolation. ### Proposed Solution To improve the reliability of the `wf14_server_mode.robot` test suite, the following changes are proposed: 1. **Isolate Test Cases**: Replace the `Suite Setup` and `Suite Teardown` with `Test Setup` and `Test Teardown`. This will ensure that each test case runs in a clean, isolated environment. 2. **Use Temporary Namespaces**: Instead of a hardcoded namespace (`myteam`), generate a unique namespace for each test case. This will prevent any potential for resource conflicts between parallel test runs. 3. **Implement Health Checks**: Before each test run, perform a health check to verify that the environment is clean. This check should ensure that no leftover configurations, actions, or actors from previous runs exist. ### Duplicate Check * Search query: `flaky` — Results: 0 * Search query: `e2e` — Results: many, but none related to this specific issue * Search query: `flaky e2e` — Results: 0 * Search query: `stateful` — Results: 0 * Search query: `test isolation` — Results: 0 None of the existing issues cover the specific problem of test isolation in the `wf14_server_mode.robot` E2E test suite. ## Subtasks - [ ] Audit `wf14_server_mode.robot` for all stateful operations and shared state - [ ] Replace `Suite Setup` / `Suite Teardown` with `Test Setup` / `Test Teardown` for each test case - [ ] Implement unique namespace generation per test case (replacing hardcoded `myteam`) - [ ] Implement pre-test health check keyword to verify clean environment before each test - [ ] Implement post-test teardown to clean up all resources created during the test - [ ] Verify refactored tests pass consistently in CI pipeline (no flakiness over multiple runs) - [ ] Run `nox -s e2e_tests` to confirm all E2E tests pass - [ ] Run `nox` (all default sessions) and fix any errors ## Definition of Done - [ ] The `wf14_server_mode.robot` test suite is refactored to use `Test Setup` and `Test Teardown` for each test case - [ ] Each test case in the suite uses a unique, temporary namespace - [ ] A pre-test health check is implemented to ensure a clean test environment - [ ] The refactored tests pass consistently in the CI pipeline - [ ] All nox stages pass - [ ] Coverage >= 97% > **Backlog note:** This issue was discovered during autonomous operation > on milestone v3.8.0. It does not block milestone completion and has been > placed in the backlog for human review and future milestone assignment. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: new-issue-creator
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#6272
No description provided.