cleveragents/cleveragents-core

Fork 3

BUG-HUNT: [concurrency] Platform compatibility failure in parallel test runner — hardcoded fork() start method crashes on Windows and restricted Unix environments #7292

New issue

Open

opened 2026-04-10 15:20:57 +00:00 by HAL9000 · 3 comments

HAL9000 commented

2026-04-10 15:20:57 +00:00

Owner

Metadata

Branch: bugfix/m8-parallel-runner-platform-compat
Commit Message: fix(tests): detect platform capabilities and fall back to compatible multiprocessing start method in parallel runner
Milestone: v3.8.0
Parent Epic: #1678

Background

The parallel Behave test runner (scripts/run_behave_parallel.py) hardcodes the "fork" multiprocessing start method in its main() function without any platform detection or compatibility validation. The fork start method is only available on Unix-like systems and is not supported on Windows, which uses spawn by default. Additionally, some Unix environments (e.g., macOS since Python 3.8+, restricted container environments, or systems with POSIX_SPAWN restrictions) may not support fork reliably.

This causes the entire test suite to crash immediately on any non-Unix platform or restricted environment, with unclear error messages that do not indicate the root cause.

Current Behavior

# scripts/run_behave_parallel.py, lines 294-297
ctx = multiprocessing.get_context("fork")
with ctx.Pool(processes=min(processes, len(chunks))) as pool:
    results = pool.map(
        _worker_run_features,
        [(chunk, other_args) for chunk in chunks],
    )

The script unconditionally calls multiprocessing.get_context("fork"). On Windows or restricted environments, this raises a ValueError: cannot find context for fork or similar, crashing the entire test suite with an unhelpful traceback.

Expected Behavior

The script should:

Detect the current platform's multiprocessing capabilities at runtime.
Use "fork" when available (Linux/Unix with fork support).
Fall back gracefully to "spawn" or "forkserver" on platforms that don't support "fork".
As a last resort, fall back to sequential execution mode with a clear warning message explaining the platform limitation.
Provide an explicit, actionable error message if parallel execution is unavailable.

Evidence

# Problematic code — no platform guard:
ctx = multiprocessing.get_context("fork")  # Crashes on Windows, some macOS configs

Python's multiprocessing module documents that "fork" is only available on Unix. The correct cross-platform approach is to use multiprocessing.get_start_method() to detect the default, or check sys.platform before selecting the context.

Suggested Fix

import sys

def _get_multiprocessing_context() -> multiprocessing.context.BaseContext:
    """Return the best available multiprocessing context for this platform."""
    if sys.platform == "win32":
        return multiprocessing.get_context("spawn")
    try:
        return multiprocessing.get_context("fork")
    except ValueError:
        # Fall back to spawn if fork is unavailable
        return multiprocessing.get_context("spawn")

Additionally, the caller should catch OSError / ValueError from Pool creation and fall back to sequential mode with a clear warning.

Subtasks

Add platform detection helper _get_multiprocessing_context() in scripts/run_behave_parallel.py
Replace hardcoded multiprocessing.get_context("fork") call with the platform-aware helper
Add graceful fallback to sequential mode when no parallel context is available
Emit a clear, actionable warning message when falling back to sequential mode
Add type annotations to the new helper function
Verify fix does not regress existing Unix/Linux parallel execution behaviour

Definition of Done

multiprocessing.get_context("fork") is no longer called unconditionally
Platform detection logic selects the appropriate start method at runtime
Sequential fallback is triggered (with warning) when no parallel context is available
All nox stages pass
Coverage >= 97%
No # type: ignore suppressions introduced
Pyright type checking passes on the modified file

TDD Note: After this bug issue is verified, a corresponding Type/Testing issue will be created for TDD. The test will use tags: @tdd_issue, @tdd_issue_<this-issue-number>, and @tdd_expected_fail to prove the bug exists before fixing it.

Automated by CleverAgents Bot
Supervisor: Bug Hunter | Agent: new-issue-creator

## Metadata - **Branch**: `bugfix/m8-parallel-runner-platform-compat` - **Commit Message**: `fix(tests): detect platform capabilities and fall back to compatible multiprocessing start method in parallel runner` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background The parallel Behave test runner (`scripts/run_behave_parallel.py`) hardcodes the `"fork"` multiprocessing start method in its `main()` function without any platform detection or compatibility validation. The `fork` start method is only available on Unix-like systems and is not supported on Windows, which uses `spawn` by default. Additionally, some Unix environments (e.g., macOS since Python 3.8+, restricted container environments, or systems with `POSIX_SPAWN` restrictions) may not support `fork` reliably. This causes the entire test suite to crash immediately on any non-Unix platform or restricted environment, with unclear error messages that do not indicate the root cause. ## Current Behavior ```python # scripts/run_behave_parallel.py, lines 294-297 ctx = multiprocessing.get_context("fork") with ctx.Pool(processes=min(processes, len(chunks))) as pool: results = pool.map( _worker_run_features, [(chunk, other_args) for chunk in chunks], ) ``` The script unconditionally calls `multiprocessing.get_context("fork")`. On Windows or restricted environments, this raises a `ValueError: cannot find context for fork` or similar, crashing the entire test suite with an unhelpful traceback. ## Expected Behavior The script should: 1. Detect the current platform's multiprocessing capabilities at runtime. 2. Use `"fork"` when available (Linux/Unix with fork support). 3. Fall back gracefully to `"spawn"` or `"forkserver"` on platforms that don't support `"fork"`. 4. As a last resort, fall back to sequential execution mode with a clear warning message explaining the platform limitation. 5. Provide an explicit, actionable error message if parallel execution is unavailable. ## Evidence ```python # Problematic code — no platform guard: ctx = multiprocessing.get_context("fork") # Crashes on Windows, some macOS configs ``` Python's `multiprocessing` module documents that `"fork"` is only available on Unix. The correct cross-platform approach is to use `multiprocessing.get_start_method()` to detect the default, or check `sys.platform` before selecting the context. ## Suggested Fix ```python import sys def _get_multiprocessing_context() -> multiprocessing.context.BaseContext: """Return the best available multiprocessing context for this platform.""" if sys.platform == "win32": return multiprocessing.get_context("spawn") try: return multiprocessing.get_context("fork") except ValueError: # Fall back to spawn if fork is unavailable return multiprocessing.get_context("spawn") ``` Additionally, the caller should catch `OSError` / `ValueError` from `Pool` creation and fall back to sequential mode with a clear warning. ## Subtasks - [ ] Add platform detection helper `_get_multiprocessing_context()` in `scripts/run_behave_parallel.py` - [ ] Replace hardcoded `multiprocessing.get_context("fork")` call with the platform-aware helper - [ ] Add graceful fallback to sequential mode when no parallel context is available - [ ] Emit a clear, actionable warning message when falling back to sequential mode - [ ] Add type annotations to the new helper function - [ ] Verify fix does not regress existing Unix/Linux parallel execution behaviour ## Definition of Done - [ ] `multiprocessing.get_context("fork")` is no longer called unconditionally - [ ] Platform detection logic selects the appropriate start method at runtime - [ ] Sequential fallback is triggered (with warning) when no parallel context is available - [ ] All nox stages pass - [ ] Coverage >= 97% - [ ] No `# type: ignore` suppressions introduced - [ ] Pyright type checking passes on the modified file > **TDD Note:** After this bug issue is verified, a corresponding `Type/Testing` issue will be created for TDD. The test will use tags: `@tdd_issue`, `@tdd_issue_<this-issue-number>`, and `@tdd_expected_fail` to prove the bug exists before fixing it. --- **Automated by CleverAgents Bot** Supervisor: Bug Hunter | Agent: new-issue-creator

HAL9000 added this to the v3.8.0 milestone

2026-04-10 15:21:04 +00:00

HAL9000 added the

labels

2026-04-10 15:25:27 +00:00

HAL9000 added a new dependency

2026-04-10 15:25:33 +00:00

#1678 Epic: CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing

HAL9000 referenced this issue

2026-04-10 16:09:08 +00:00

BUG-HUNT: [resource] Memory exhaustion risk from unbounded StringIO buffers in parallel test runner worker #7311

HAL9000 added

and removed

labels

2026-04-14 06:23:54 +00:00

HAL9000 commented

2026-04-14 06:23:54 +00:00

Author

Owner

✅ Verified — Critical compatibility bug: hardcoded fork() crashes on Windows/restricted Unix. MoSCoW: Must-have. Priority: Critical.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

✅ **Verified** — Critical compatibility bug: hardcoded fork() crashes on Windows/restricted Unix. MoSCoW: Must-have. Priority: Critical. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor

HAL9000 commented

2026-04-14 06:29:32 +00:00

Author

Owner

✅ Verified — Critical compatibility bug: hardcoded fork() crashes on Windows/restricted Unix. MoSCoW: Must-have. Priority: Critical.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor