d71046b9a0
Four safety items from the consolidated adversarial-review punch list.
ITEM 8 — PID-reuse hazard in janitor:
The janitor SIGKILL'd whatever process happened to live at the
sidecar's recorded subprocess_pid. Between sidecar write and janitor
sweep, the OS can reuse the PID for an unrelated process; the janitor
was killing innocents under fork-heavy workloads.
Fix:
- ``session_sidecar.py``: added ``subprocess_starttime`` field
(Optional[int]) + ``read_proc_starttime(pid)`` helper that reads
``/proc/{pid}/stat`` field 22 (clock ticks since boot — monotonic
for a (boot, pid) pair).
- ``WorkerSession.from_dict`` filters unknown keys so forward + back
compat with sidecars from earlier/later versions is preserved.
- ``janitor._pid_alive`` and ``_kill_with_grace`` accept
``expected_starttime``; on mismatch they short-circuit and DON'T
signal the impostor.
- ``_kill_with_grace`` return semantics tightened: True iff a signal
was actually delivered (False for "PID gone" / "PID reused"). The
``sessions_killed`` counter now reflects real kills.
ITEM 9 — AWAITING_CI escape from infinite poll:
Previously AWAITING_CI could only exit via ``ci_green`` /
``ci_red_*`` / ``ci_flake_retry`` — if CI hangs forever (runner
outage, broken integration, etc.) the workflow had no controller-
driven STUCK path; only operator_unstick could rescue it.
Fix: new ``ci_polling_exhausted`` event → STUCK. The master's
AWAITING_CI poll handler is the natural place to emit it once a
threshold passes (deferred to a follow-up — Phase 1k+ ships the
event in the table; the timer fires it).
ITEM 10 — ci_flake_retry was unbounded:
The ``ci_flake_retry`` self-loop on AWAITING_CI had no encoded
ceiling. Pathological flaky CI could loop forever (the docstring
said "retry once per gate" but nothing enforced it).
Fix:
- New ``workflows.ci_flake_retries_remaining`` column (server_default
'1', default 1 — operators tune via ``CONTROLLER_CI_FLAKE_RETRIES``
at startup or via direct UPDATE).
- New ``ci_flake_retries_exhausted`` event → ESCALATING. Master
decrements the column on each ci_flake_retry; at 0 the next CI
failure routes through ci_red_* (regular path) or this new
event (escalates if the operator wants a hard ceiling).
ITEM 11 — scheduler now skips PAUSED workflows:
Without this, the scheduler could enqueue a fresh attempt for a
PAUSED workflow between two reconciliation ticks (race: label
removed at T+0, reconciliation runs at T+300, scheduler ticks at
T+30 with stale DB state). The window is at most one attempt of
worker work.
Fix: ``schedule_next_attempts`` SQL now lists only
{ANALYZING, IMPLEMENTING, REVIEWING, CONFLICT_RESOLVING, ESCALATING}
explicitly; PAUSED is excluded by absence. Reconciliation owns the
PAUSED → resume transition; scheduler doesn't touch it.
Schema additions:
- ``workflows.ci_flake_retries_remaining`` (INTEGER NOT NULL DEFAULT 1)
- ``workflows.awaiting_ci_started_at`` (TIMESTAMP NULL) — for the
poll-exhaustion timer (timer impl deferred; column is staged).
Tests:
- TestReadProcStarttime — 3 tests (Linux skip-guard) for the
/proc/pid/stat parser (self-pid > 0, missing pid is None,
invalid pid is None).
- TestJanitor::test_pid_reuse_defended_via_starttime — pins the
contract end-to-end (real subprocess + fabricated wrong starttime
→ janitor doesn't signal).
- TestPhase1kPlusTransitions — 5 tests pinning the new events +
proving the load-bearing invariants still pass.
- test_scheduler_skips_paused_workflows — pins item 11.
Total: 603 controller tests pass (+10 net), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
160 lines
5.4 KiB
Python
160 lines
5.4 KiB
Python
"""Worker session sidecar: per-workspace file tracking the live
|
|
OpenCode session + subprocess for orphan recovery.
|
|
|
|
Per plan v6: at attempt spawn, the worker writes
|
|
``{workspace}/worker.session`` BEFORE issuing the OpenCode prompt
|
|
that begins agent work. The sidecar records:
|
|
|
|
- ``opencode_server_url``: which OpenCode the worker is talking to
|
|
(different worker machines may use different OpenCode instances).
|
|
- ``session_id``: the OpenCode session id (needed for the
|
|
cancel-via-API path on orphan cleanup).
|
|
- ``subprocess_pid``: the local OS pid that owns the session (fallback
|
|
for SIGKILL when the API cancel fails).
|
|
- ``spawned_by_controller_pid``: this worker process's PID. Lets the
|
|
janitor identify "sessions spawned by a worker that is no longer
|
|
running on this machine."
|
|
- ``instance_id``: the worker instance the lock is held by. Janitor
|
|
cross-checks against DB.
|
|
- ``spawned_at``: ISO timestamp.
|
|
|
|
Janitor (janitor.py) reads every sidecar on this machine at worker
|
|
startup; sessions whose owning instance is no longer alive get
|
|
cancelled (OpenCode API first, SIGKILL fallback) and the sidecar +
|
|
workspace are deleted.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import os
|
|
from dataclasses import asdict, dataclass
|
|
from datetime import datetime, timezone
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class WorkerSession:
|
|
"""The sidecar's shape. Frozen dataclass = JSON serialisable +
|
|
immutable after write.
|
|
|
|
``subprocess_starttime`` (Phase 1k+ refinement): the Linux process
|
|
start-time read from ``/proc/{pid}/stat`` field 22 at sidecar
|
|
write. Used by the janitor to defend against PID reuse — between
|
|
sidecar write and janitor sweep, the OS may have reused the PID
|
|
for an unrelated process; without starttime confirmation the
|
|
janitor would SIGTERM/SIGKILL the wrong process. Optional (None)
|
|
for cross-platform compatibility + backward compat with sidecars
|
|
written before this column existed.
|
|
"""
|
|
|
|
opencode_server_url: str
|
|
session_id: str
|
|
subprocess_pid: int
|
|
spawned_by_controller_pid: int
|
|
instance_id: str
|
|
spawned_at: str # ISO 8601 UTC
|
|
subprocess_starttime: int | None = None # /proc/pid/stat field 22
|
|
|
|
def to_dict(self) -> dict[str, Any]:
|
|
return asdict(self)
|
|
|
|
@classmethod
|
|
def from_dict(cls, data: dict[str, Any]) -> "WorkerSession":
|
|
# Forward-compat: tolerate sidecars missing newer fields (e.g.
|
|
# subprocess_starttime added in Phase 1k+) AND sidecars carrying
|
|
# extra fields a future version might add.
|
|
known = {k: v for k, v in data.items() if k in cls.__dataclass_fields__}
|
|
return cls(**known)
|
|
|
|
|
|
def now_iso() -> str:
|
|
return datetime.now(timezone.utc).isoformat()
|
|
|
|
|
|
def read_proc_starttime(pid: int) -> int | None:
|
|
"""Read a Linux process's start time from ``/proc/{pid}/stat`` field
|
|
22 (in clock ticks since system boot).
|
|
|
|
Returns None for non-Linux hosts, missing PID, permission denied,
|
|
or any parse error. Callers should treat None as "starttime
|
|
confirmation unavailable" — fall back to PID-alone semantics for
|
|
that process.
|
|
|
|
The starttime is monotonic for a given (boot, pid): two processes
|
|
with the same PID on the same boot have the same starttime IFF
|
|
they are the same process. PID reuse across reboots is handled by
|
|
a reboot also resetting the starttime baseline.
|
|
"""
|
|
if pid <= 0:
|
|
return None
|
|
try:
|
|
with open(f"/proc/{pid}/stat", "rb") as f:
|
|
raw = f.read()
|
|
except (OSError, PermissionError):
|
|
return None
|
|
# /proc/[pid]/stat field 2 is the (parenthesized) comm; a comm that
|
|
# contains spaces or close-parens breaks naive split(). Take the
|
|
# text after the LAST ')' as the start of fields 3+.
|
|
end_paren = raw.rfind(b")")
|
|
if end_paren < 0:
|
|
return None
|
|
fields = raw[end_paren + 1:].split()
|
|
# Field indexing on the split portion: field 3 is at index 0
|
|
# (state), field 22 (starttime) is at index 19.
|
|
if len(fields) < 20:
|
|
return None
|
|
try:
|
|
return int(fields[19])
|
|
except ValueError:
|
|
return None
|
|
|
|
|
|
def write_sidecar(path: Path, session: WorkerSession) -> None:
|
|
"""Write the sidecar JSON atomically (write to ``.tmp`` + rename).
|
|
|
|
Atomic so a partial write isn't read by the janitor. fsync before
|
|
rename ensures durability across power loss.
|
|
"""
|
|
path.parent.mkdir(parents=True, exist_ok=True)
|
|
tmp = path.with_suffix(path.suffix + ".tmp")
|
|
payload = json.dumps(session.to_dict(), sort_keys=True, indent=2)
|
|
with open(tmp, "w", encoding="utf-8") as f:
|
|
f.write(payload)
|
|
f.flush()
|
|
os.fsync(f.fileno())
|
|
os.replace(tmp, path)
|
|
|
|
|
|
def read_sidecar(path: Path) -> WorkerSession | None:
|
|
"""Read + parse the sidecar JSON. Returns ``None`` for missing,
|
|
empty, or malformed sidecars (janitor treats those as
|
|
fully-orphaned + cleans up)."""
|
|
if not path.is_file():
|
|
return None
|
|
try:
|
|
text = path.read_text(encoding="utf-8")
|
|
except OSError:
|
|
return None
|
|
if not text.strip():
|
|
return None
|
|
try:
|
|
data = json.loads(text)
|
|
except json.JSONDecodeError:
|
|
return None
|
|
if not isinstance(data, dict):
|
|
return None
|
|
try:
|
|
return WorkerSession.from_dict(data)
|
|
except (TypeError, KeyError):
|
|
return None
|
|
|
|
|
|
__all__ = [
|
|
"WorkerSession",
|
|
"now_iso",
|
|
"read_proc_starttime",
|
|
"read_sidecar",
|
|
"write_sidecar",
|
|
]
|