Fix flaky database_integration robot suite on CI #387

New issue

Closed

opened 2026-02-23 16:58:12 +00:00 by freemo · 1 comment

freemo commented

2026-02-23 16:58:12 +00:00

Owner

Parent

Epic: #362 (Security & Safety Hardening)

Problem

The Database Integration robot suite intermittently fails on the CI server with an empty error message:

Database Can Be Initialized :: Verify database initialization works   | FAIL |
Python script execution failed:

The root cause is that under heavy parallel CI load (N pabot workers each spawning Python subprocesses via Run Python Script), a subprocess can be killed by OOM or CPU starvation before its output buffer is flushed. This results in a non-zero exit code with empty stdout — making the failure impossible to diagnose.

The test passes on re-run without code changes, confirming the non-deterministic nature.

Root Cause

The Run Python Script keyword in robot/database_integration.robot had several issues:

NamedTemporaryFile fd leak — file object not explicitly closed before Create File writes to it
30s timeout too tight — insufficient headroom under parallel CI resource contention
No retry mechanism — transient signal-killed processes cause immediate test failure
Poor error reporting — return code not included in failure message

Fix

Applied in PR #386:

Replace NamedTemporaryFile with mkstemp + explicit os.close(fd)
Increase subprocess timeout from 30s to 60s
Retry once when rc != 0 and stdout is empty (characteristic of OOM/SIGKILL)
Include return code in failure message for future diagnosis

Verification

nox -e lint — PASS
nox -e typecheck — PASS
nox -e integration_tests — Database Integration suite passes (both instances, 218-223s)

## Parent Epic: #362 (Security & Safety Hardening) ## Problem The `Database Integration` robot suite intermittently fails on the CI server with an empty error message: ``` Database Can Be Initialized :: Verify database initialization works | FAIL | Python script execution failed: ``` The root cause is that under heavy parallel CI load (N pabot workers each spawning Python subprocesses via `Run Python Script`), a subprocess can be killed by OOM or CPU starvation before its output buffer is flushed. This results in a non-zero exit code with empty stdout — making the failure impossible to diagnose. The test passes on re-run without code changes, confirming the non-deterministic nature. ## Root Cause The `Run Python Script` keyword in `robot/database_integration.robot` had several issues: 1. **`NamedTemporaryFile` fd leak** — file object not explicitly closed before `Create File` writes to it 2. **30s timeout too tight** — insufficient headroom under parallel CI resource contention 3. **No retry mechanism** — transient signal-killed processes cause immediate test failure 4. **Poor error reporting** — return code not included in failure message ## Fix Applied in PR #386: - Replace `NamedTemporaryFile` with `mkstemp` + explicit `os.close(fd)` - Increase subprocess timeout from 30s to 60s - Retry once when `rc != 0` and stdout is empty (characteristic of OOM/SIGKILL) - Include return code in failure message for future diagnosis ## Verification - `nox -e lint` — PASS - `nox -e typecheck` — PASS - `nox -e integration_tests` — Database Integration suite passes (both instances, 218-223s)

freemo added the

labels

2026-02-23 16:59:34 +00:00

freemo added this to the v3.3.0 milestone

2026-02-23 16:59:35 +00:00

freemo commented

2026-02-23 16:59:42 +00:00

Author

Owner

Fixed by PR #386 (ci-int-fix branch).

Changes in robot/database_integration.robot:

mkstemp instead of NamedTemporaryFile (fd leak fix)
Timeout 30s → 60s
Retry once on empty-output failure (signal-killed process)
Include rc in error messages

Fixed by PR #386 (`ci-int-fix` branch). Changes in `robot/database_integration.robot`: - `mkstemp` instead of `NamedTemporaryFile` (fd leak fix) - Timeout 30s → 60s - Retry once on empty-output failure (signal-killed process) - Include rc in error messages

freemo closed this issue

2026-02-23 16:59:47 +00:00

freemo added a new dependency

2026-02-23 17:03:50 +00:00

#376 Legendary: Hardening, Testing & Security

freemo added a new dependency

2026-02-23 17:03:55 +00:00

#386 fix(test): harden database_integration robot suite against CI flakiness

freemo removed a dependency

2026-02-23 17:12:58 +00:00

#376 Legendary: Hardening, Testing & Security

freemo added a new dependency