Files
cleveragents-core/tools/controller/worker/session_sidecar.py
T
drew d71046b9a0 fix(controller): batch D — PID-reuse, AWAITING_CI escape, flake bound, scheduler skip
Four safety items from the consolidated adversarial-review punch list.

ITEM 8 — PID-reuse hazard in janitor:
The janitor SIGKILL'd whatever process happened to live at the
sidecar's recorded subprocess_pid. Between sidecar write and janitor
sweep, the OS can reuse the PID for an unrelated process; the janitor
was killing innocents under fork-heavy workloads.

Fix:
- ``session_sidecar.py``: added ``subprocess_starttime`` field
  (Optional[int]) + ``read_proc_starttime(pid)`` helper that reads
  ``/proc/{pid}/stat`` field 22 (clock ticks since boot — monotonic
  for a (boot, pid) pair).
- ``WorkerSession.from_dict`` filters unknown keys so forward + back
  compat with sidecars from earlier/later versions is preserved.
- ``janitor._pid_alive`` and ``_kill_with_grace`` accept
  ``expected_starttime``; on mismatch they short-circuit and DON'T
  signal the impostor.
- ``_kill_with_grace`` return semantics tightened: True iff a signal
  was actually delivered (False for "PID gone" / "PID reused"). The
  ``sessions_killed`` counter now reflects real kills.

ITEM 9 — AWAITING_CI escape from infinite poll:
Previously AWAITING_CI could only exit via ``ci_green`` /
``ci_red_*`` / ``ci_flake_retry`` — if CI hangs forever (runner
outage, broken integration, etc.) the workflow had no controller-
driven STUCK path; only operator_unstick could rescue it.

Fix: new ``ci_polling_exhausted`` event → STUCK. The master's
AWAITING_CI poll handler is the natural place to emit it once a
threshold passes (deferred to a follow-up — Phase 1k+ ships the
event in the table; the timer fires it).

ITEM 10 — ci_flake_retry was unbounded:
The ``ci_flake_retry`` self-loop on AWAITING_CI had no encoded
ceiling. Pathological flaky CI could loop forever (the docstring
said "retry once per gate" but nothing enforced it).

Fix:
- New ``workflows.ci_flake_retries_remaining`` column (server_default
  '1', default 1 — operators tune via ``CONTROLLER_CI_FLAKE_RETRIES``
  at startup or via direct UPDATE).
- New ``ci_flake_retries_exhausted`` event → ESCALATING. Master
  decrements the column on each ci_flake_retry; at 0 the next CI
  failure routes through ci_red_* (regular path) or this new
  event (escalates if the operator wants a hard ceiling).

ITEM 11 — scheduler now skips PAUSED workflows:
Without this, the scheduler could enqueue a fresh attempt for a
PAUSED workflow between two reconciliation ticks (race: label
removed at T+0, reconciliation runs at T+300, scheduler ticks at
T+30 with stale DB state). The window is at most one attempt of
worker work.

Fix: ``schedule_next_attempts`` SQL now lists only
{ANALYZING, IMPLEMENTING, REVIEWING, CONFLICT_RESOLVING, ESCALATING}
explicitly; PAUSED is excluded by absence. Reconciliation owns the
PAUSED → resume transition; scheduler doesn't touch it.

Schema additions:
- ``workflows.ci_flake_retries_remaining`` (INTEGER NOT NULL DEFAULT 1)
- ``workflows.awaiting_ci_started_at`` (TIMESTAMP NULL) — for the
  poll-exhaustion timer (timer impl deferred; column is staged).

Tests:
- TestReadProcStarttime — 3 tests (Linux skip-guard) for the
  /proc/pid/stat parser (self-pid > 0, missing pid is None,
  invalid pid is None).
- TestJanitor::test_pid_reuse_defended_via_starttime — pins the
  contract end-to-end (real subprocess + fabricated wrong starttime
  → janitor doesn't signal).
- TestPhase1kPlusTransitions — 5 tests pinning the new events +
  proving the load-bearing invariants still pass.
- test_scheduler_skips_paused_workflows — pins item 11.

Total: 603 controller tests pass (+10 net), 0 regressions.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 15:30:45 -04:00

160 lines
5.4 KiB
Python

"""Worker session sidecar: per-workspace file tracking the live
OpenCode session + subprocess for orphan recovery.
Per plan v6: at attempt spawn, the worker writes
``{workspace}/worker.session`` BEFORE issuing the OpenCode prompt
that begins agent work. The sidecar records:
- ``opencode_server_url``: which OpenCode the worker is talking to
(different worker machines may use different OpenCode instances).
- ``session_id``: the OpenCode session id (needed for the
cancel-via-API path on orphan cleanup).
- ``subprocess_pid``: the local OS pid that owns the session (fallback
for SIGKILL when the API cancel fails).
- ``spawned_by_controller_pid``: this worker process's PID. Lets the
janitor identify "sessions spawned by a worker that is no longer
running on this machine."
- ``instance_id``: the worker instance the lock is held by. Janitor
cross-checks against DB.
- ``spawned_at``: ISO timestamp.
Janitor (janitor.py) reads every sidecar on this machine at worker
startup; sessions whose owning instance is no longer alive get
cancelled (OpenCode API first, SIGKILL fallback) and the sidecar +
workspace are deleted.
"""
from __future__ import annotations
import json
import os
from dataclasses import asdict, dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
@dataclass(frozen=True)
class WorkerSession:
"""The sidecar's shape. Frozen dataclass = JSON serialisable +
immutable after write.
``subprocess_starttime`` (Phase 1k+ refinement): the Linux process
start-time read from ``/proc/{pid}/stat`` field 22 at sidecar
write. Used by the janitor to defend against PID reuse — between
sidecar write and janitor sweep, the OS may have reused the PID
for an unrelated process; without starttime confirmation the
janitor would SIGTERM/SIGKILL the wrong process. Optional (None)
for cross-platform compatibility + backward compat with sidecars
written before this column existed.
"""
opencode_server_url: str
session_id: str
subprocess_pid: int
spawned_by_controller_pid: int
instance_id: str
spawned_at: str # ISO 8601 UTC
subprocess_starttime: int | None = None # /proc/pid/stat field 22
def to_dict(self) -> dict[str, Any]:
return asdict(self)
@classmethod
def from_dict(cls, data: dict[str, Any]) -> "WorkerSession":
# Forward-compat: tolerate sidecars missing newer fields (e.g.
# subprocess_starttime added in Phase 1k+) AND sidecars carrying
# extra fields a future version might add.
known = {k: v for k, v in data.items() if k in cls.__dataclass_fields__}
return cls(**known)
def now_iso() -> str:
return datetime.now(timezone.utc).isoformat()
def read_proc_starttime(pid: int) -> int | None:
"""Read a Linux process's start time from ``/proc/{pid}/stat`` field
22 (in clock ticks since system boot).
Returns None for non-Linux hosts, missing PID, permission denied,
or any parse error. Callers should treat None as "starttime
confirmation unavailable" — fall back to PID-alone semantics for
that process.
The starttime is monotonic for a given (boot, pid): two processes
with the same PID on the same boot have the same starttime IFF
they are the same process. PID reuse across reboots is handled by
a reboot also resetting the starttime baseline.
"""
if pid <= 0:
return None
try:
with open(f"/proc/{pid}/stat", "rb") as f:
raw = f.read()
except (OSError, PermissionError):
return None
# /proc/[pid]/stat field 2 is the (parenthesized) comm; a comm that
# contains spaces or close-parens breaks naive split(). Take the
# text after the LAST ')' as the start of fields 3+.
end_paren = raw.rfind(b")")
if end_paren < 0:
return None
fields = raw[end_paren + 1:].split()
# Field indexing on the split portion: field 3 is at index 0
# (state), field 22 (starttime) is at index 19.
if len(fields) < 20:
return None
try:
return int(fields[19])
except ValueError:
return None
def write_sidecar(path: Path, session: WorkerSession) -> None:
"""Write the sidecar JSON atomically (write to ``.tmp`` + rename).
Atomic so a partial write isn't read by the janitor. fsync before
rename ensures durability across power loss.
"""
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_suffix(path.suffix + ".tmp")
payload = json.dumps(session.to_dict(), sort_keys=True, indent=2)
with open(tmp, "w", encoding="utf-8") as f:
f.write(payload)
f.flush()
os.fsync(f.fileno())
os.replace(tmp, path)
def read_sidecar(path: Path) -> WorkerSession | None:
"""Read + parse the sidecar JSON. Returns ``None`` for missing,
empty, or malformed sidecars (janitor treats those as
fully-orphaned + cleans up)."""
if not path.is_file():
return None
try:
text = path.read_text(encoding="utf-8")
except OSError:
return None
if not text.strip():
return None
try:
data = json.loads(text)
except json.JSONDecodeError:
return None
if not isinstance(data, dict):
return None
try:
return WorkerSession.from_dict(data)
except (TypeError, KeyError):
return None
__all__ = [
"WorkerSession",
"now_iso",
"read_proc_starttime",
"read_sidecar",
"write_sidecar",
]