cleveragents/cleveragents-core

Fork 3

BUG-HUNT: [resource] Memory exhaustion risk from unbounded StringIO buffers in parallel test runner worker #7311

New issue

Open

opened 2026-04-10 16:09:08 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-10 16:09:08 +00:00

Owner

Metadata

Branch: bugfix/parallel-runner-bounded-output-buffers
Commit Message: fix(tests): implement bounded output buffers in parallel runner worker to prevent memory exhaustion
Milestone: (backlog — see note below)
Parent Epic: #1678

Background and Context

The parallel Behave test runner (scripts/run_behave_parallel.py) captures all stdout and stderr from worker processes in unbounded io.StringIO buffers. Tests that produce large amounts of output — verbose logging, data dumps, long error traces — can cause memory exhaustion in worker processes, potentially crashing the test runner or destabilising the host system.

Current Behavior

In _worker_run_features() (lines 212–236), two io.StringIO buffers are allocated with no size limit:

def _worker_run_features(
    payload: tuple[list[str], list[str]],
) -> tuple[bool, str, str, Summary]:
    feature_paths, behave_args = payload

    stdout_buf = io.StringIO()  # No size limit
    stderr_buf = io.StringIO()  # No size limit

    runner = _make_runner(feature_paths, behave_args)

    with redirect_stdout(stdout_buf), redirect_stderr(stderr_buf):
        failed = runner.run()

    summary = _extract_summary(runner)
    return failed, stdout_buf.getvalue(), stderr_buf.getvalue(), summary

The full buffer contents are then returned to the parent process via IPC, potentially doubling memory usage. There is no cap, no overflow handling, and no warning when output grows large.

Expected Behavior

Worker processes should implement bounded output capture to prevent memory exhaustion:

Bounded StringIO buffers with configurable size limits and overflow handling.
Fallback to temporary files for large outputs instead of in-memory buffers.
Output size monitoring with warnings when thresholds are approached.
Truncation of extremely large outputs with a summary message indicating truncation occurred.

Acceptance Criteria

Worker output buffers are bounded (configurable limit, with a sensible default).
When a buffer exceeds its limit, output is either truncated with a clear message or spilled to a temporary file.
The parent process is never handed a string larger than the configured limit per worker.
A warning is emitted (to the parent's stderr) when truncation or spill occurs.
All existing parallel runner tests continue to pass.
No regression in test runner behaviour for normal-sized outputs.

Supporting Information

File: scripts/run_behave_parallel.py
Function: _worker_run_features()
Lines: 212–236
Related issue: #7292 (platform compatibility failure in the same function)
Severity: Medium likelihood — occurs with verbose tests or tests that produce large output; High impact when triggered (OOM crash).

Subtasks

Implement a BoundedStringIO wrapper (or equivalent) with a configurable byte/character cap
Replace bare io.StringIO() allocations in _worker_run_features() with the bounded variant
Add overflow handling: truncate with a sentinel message, or spill to tempfile.SpooledTemporaryFile
Emit a warning to the parent process when truncation/spill occurs
Tests (Behave): Add scenario for worker producing output exceeding the buffer limit
Tests (Behave): Verify truncation message is present in returned output
Tests (Behave): Verify no OOM condition when a worker emits very large output
Verify coverage ≥97% via nox -s coverage_report
Run nox (all default sessions), fix any errors

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
All nox stages pass.
Coverage ≥ 97%.

Backlog note: This issue was discovered during autonomous operation
on milestone v3.8.0. It does not block milestone completion and has been
placed in the backlog for human review and future milestone assignment.

TDD Note

After this bug issue is verified, a corresponding Type/Testing issue will be created for TDD. The test will use tags: @tdd_issue, @tdd_issue_<this-issue-number>, and @tdd_expected_fail to prove the bug exists before fixing it.

Automated by CleverAgents Bot
Supervisor: Bug Hunter | Agent: new-issue-creator

## Metadata - **Branch**: `bugfix/parallel-runner-bounded-output-buffers` - **Commit Message**: `fix(tests): implement bounded output buffers in parallel runner worker to prevent memory exhaustion` - **Milestone**: *(backlog — see note below)* - **Parent Epic**: #1678 ## Background and Context The parallel Behave test runner (`scripts/run_behave_parallel.py`) captures all stdout and stderr from worker processes in unbounded `io.StringIO` buffers. Tests that produce large amounts of output — verbose logging, data dumps, long error traces — can cause memory exhaustion in worker processes, potentially crashing the test runner or destabilising the host system. ## Current Behavior In `_worker_run_features()` (lines 212–236), two `io.StringIO` buffers are allocated with no size limit: ```python def _worker_run_features( payload: tuple[list[str], list[str]], ) -> tuple[bool, str, str, Summary]: feature_paths, behave_args = payload stdout_buf = io.StringIO() # No size limit stderr_buf = io.StringIO() # No size limit runner = _make_runner(feature_paths, behave_args) with redirect_stdout(stdout_buf), redirect_stderr(stderr_buf): failed = runner.run() summary = _extract_summary(runner) return failed, stdout_buf.getvalue(), stderr_buf.getvalue(), summary ``` The full buffer contents are then returned to the parent process via IPC, potentially doubling memory usage. There is no cap, no overflow handling, and no warning when output grows large. ## Expected Behavior Worker processes should implement bounded output capture to prevent memory exhaustion: 1. Bounded `StringIO` buffers with configurable size limits and overflow handling. 2. Fallback to temporary files for large outputs instead of in-memory buffers. 3. Output size monitoring with warnings when thresholds are approached. 4. Truncation of extremely large outputs with a summary message indicating truncation occurred. ## Acceptance Criteria - [ ] Worker output buffers are bounded (configurable limit, with a sensible default). - [ ] When a buffer exceeds its limit, output is either truncated with a clear message or spilled to a temporary file. - [ ] The parent process is never handed a string larger than the configured limit per worker. - [ ] A warning is emitted (to the parent's stderr) when truncation or spill occurs. - [ ] All existing parallel runner tests continue to pass. - [ ] No regression in test runner behaviour for normal-sized outputs. ## Supporting Information - **File**: `scripts/run_behave_parallel.py` - **Function**: `_worker_run_features()` - **Lines**: 212–236 - **Related issue**: #7292 (platform compatibility failure in the same function) - **Severity**: Medium likelihood — occurs with verbose tests or tests that produce large output; High impact when triggered (OOM crash). ## Subtasks - [ ] Implement a `BoundedStringIO` wrapper (or equivalent) with a configurable byte/character cap - [ ] Replace bare `io.StringIO()` allocations in `_worker_run_features()` with the bounded variant - [ ] Add overflow handling: truncate with a sentinel message, or spill to `tempfile.SpooledTemporaryFile` - [ ] Emit a warning to the parent process when truncation/spill occurs - [ ] Tests (Behave): Add scenario for worker producing output exceeding the buffer limit - [ ] Tests (Behave): Verify truncation message is present in returned output - [ ] Tests (Behave): Verify no OOM condition when a worker emits very large output - [ ] Verify coverage ≥97% via `nox -s coverage_report` - [ ] Run `nox` (all default sessions), fix any errors ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage ≥ 97%. > **Backlog note:** This issue was discovered during autonomous operation > on milestone v3.8.0. It does not block milestone completion and has been > placed in the backlog for human review and future milestone assignment. ### TDD Note After this bug issue is verified, a corresponding `Type/Testing` issue will be created for TDD. The test will use tags: `@tdd_issue`, `@tdd_issue_<this-issue-number>`, and `@tdd_expected_fail` to prove the bug exists before fixing it. --- **Automated by CleverAgents Bot** Supervisor: Bug Hunter | Agent: new-issue-creator

HAL9000 added a new dependency

2026-04-10 16:09:19 +00:00

#1678 Epic: CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing

HAL9000 commented

2026-04-10 16:15:56 +00:00

Author

Owner

⚠️ Label Application Required

This issue was created by an automated agent. The label API endpoints are restricted in this environment. Please apply the following labels manually:

State/Unverified (ID: 846)
Type/Bug (ID: 849)
Priority/Backlog (ID: 862)

The parent Epic dependency link to #1678 has been set (this issue blocks #1678).

Automated by CleverAgents Bot
Supervisor: Bug Hunter | Agent: new-issue-creator

⚠️ **Label Application Required** This issue was created by an automated agent. The label API endpoints are restricted in this environment. Please apply the following labels manually: - `State/Unverified` (ID: 846) - `Type/Bug` (ID: 849) - `Priority/Backlog` (ID: 862) The parent Epic dependency link to #1678 has been set (this issue blocks #1678). --- **Automated by CleverAgents Bot** Supervisor: Bug Hunter | Agent: new-issue-creator

HAL9000 added the

labels

2026-04-10 18:44:22 +00:00

HAL9000 added this to the v3.5.0 milestone

2026-04-10 19:06:07 +00:00

HAL9000 added