drew
30dfd92021
Per-PR workspace umbrella manager (per plan v6) + the worker.session
sidecar that tracks live OpenCode session metadata for orphan cleanup
+ the startup janitor that sweeps orphaned workspaces from previous
worker crashes.
tools/controller/worker/:
- workspace.py: PerPRWorkspace + WorkspaceIdentity (frozen dataclass
yielding the canonical pr-{owner}-{repo}-{N} dir name). ensure_present
is idempotent. clone_if_absent runs `git clone --no-single-branch`
if .git/ missing; idempotent on re-call. fetch_and_validate runs
`git fetch origin --prune`, compares origin/<ref> to expected
head_sha, raises StaleInputError on mismatch (v6 stale-input fix),
then `git reset --hard <expected_sha>` + `git clean -fdx` to wipe
worktree residue from prior attempts. remove() rm -rf's the
workspace.
- session_sidecar.py: WorkerSession frozen dataclass +
write_sidecar (atomic via tmp+rename+fsync) + read_sidecar (tolerates
missing/empty/malformed/wrong-shape gracefully). Sidecar captures
opencode_server_url / session_id / subprocess_pid /
spawned_by_controller_pid / instance_id / spawned_at.
- janitor.py: sweep_orphans pre-queries the DB once for the set of
live instance_ids (workflow_attempts.status='in_progress'), then
scans workspace_root for pr-* dirs. For each:
- no sidecar → just delete (crashed pre-spawn)
- sidecar's instance_id is in live set → preserve (active worker)
- else → orphan. Try cancel_callback (production wires to OpenCode
cancel API); fall back to SIGTERM/grace/SIGKILL on the
subprocess_pid. Then delete sidecar + workspace. JanitorReport
summarises each sweep for structured logging.
Key v6 design points implemented:
- worker.session sidecar atomicity → orphan detection is robust
against partial writes (e.g., worker crashed mid-spawn).
- Cancel-callback-then-SIGKILL fallback → production prefers the
graceful OpenCode-side cancel; tests inject a fake.
- _kill_with_grace returns True after SIGKILL delivery; zombie
reaping is the parent's responsibility, not the janitor's.
27 new tests:
- WorkspaceIdentity (format + frozen)
- PerPRWorkspace paths + ensure_present idempotency
- clone_if_absent (creates worktree, idempotent, raises without URL)
- fetch_and_validate (matching sha resets clean; mismatched raises
StaleInputError; unknown ref raises)
- Sidecar I/O round-trip + atomicity + missing/empty/malformed handling
- Janitor: empty root / no-sidecar / active-lock / orphan-with-dead-pid
/ orphan-with-live-pid / cancel-callback (3 sub-paths: ok / fails /
raises) / mixed-workspaces / non-pr-skip
Total: 184 controller tests; full auto_agents suite 2546 pass.
106 lines
3.2 KiB
Python
106 lines
3.2 KiB
Python
"""Worker session sidecar: per-workspace file tracking the live
|
|
OpenCode session + subprocess for orphan recovery.
|
|
|
|
Per plan v6: at attempt spawn, the worker writes
|
|
``{workspace}/worker.session`` BEFORE issuing the OpenCode prompt
|
|
that begins agent work. The sidecar records:
|
|
|
|
- ``opencode_server_url``: which OpenCode the worker is talking to
|
|
(different worker machines may use different OpenCode instances).
|
|
- ``session_id``: the OpenCode session id (needed for the
|
|
cancel-via-API path on orphan cleanup).
|
|
- ``subprocess_pid``: the local OS pid that owns the session (fallback
|
|
for SIGKILL when the API cancel fails).
|
|
- ``spawned_by_controller_pid``: this worker process's PID. Lets the
|
|
janitor identify "sessions spawned by a worker that is no longer
|
|
running on this machine."
|
|
- ``instance_id``: the worker instance the lock is held by. Janitor
|
|
cross-checks against DB.
|
|
- ``spawned_at``: ISO timestamp.
|
|
|
|
Janitor (janitor.py) reads every sidecar on this machine at worker
|
|
startup; sessions whose owning instance is no longer alive get
|
|
cancelled (OpenCode API first, SIGKILL fallback) and the sidecar +
|
|
workspace are deleted.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import os
|
|
from dataclasses import asdict, dataclass
|
|
from datetime import datetime, timezone
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class WorkerSession:
|
|
"""The sidecar's shape. Frozen dataclass = JSON serialisable +
|
|
immutable after write."""
|
|
|
|
opencode_server_url: str
|
|
session_id: str
|
|
subprocess_pid: int
|
|
spawned_by_controller_pid: int
|
|
instance_id: str
|
|
spawned_at: str # ISO 8601 UTC
|
|
|
|
def to_dict(self) -> dict[str, Any]:
|
|
return asdict(self)
|
|
|
|
@classmethod
|
|
def from_dict(cls, data: dict[str, Any]) -> "WorkerSession":
|
|
return cls(**data)
|
|
|
|
|
|
def now_iso() -> str:
|
|
return datetime.now(timezone.utc).isoformat()
|
|
|
|
|
|
def write_sidecar(path: Path, session: WorkerSession) -> None:
|
|
"""Write the sidecar JSON atomically (write to ``.tmp`` + rename).
|
|
|
|
Atomic so a partial write isn't read by the janitor. fsync before
|
|
rename ensures durability across power loss.
|
|
"""
|
|
path.parent.mkdir(parents=True, exist_ok=True)
|
|
tmp = path.with_suffix(path.suffix + ".tmp")
|
|
payload = json.dumps(session.to_dict(), sort_keys=True, indent=2)
|
|
with open(tmp, "w", encoding="utf-8") as f:
|
|
f.write(payload)
|
|
f.flush()
|
|
os.fsync(f.fileno())
|
|
os.replace(tmp, path)
|
|
|
|
|
|
def read_sidecar(path: Path) -> WorkerSession | None:
|
|
"""Read + parse the sidecar JSON. Returns ``None`` for missing,
|
|
empty, or malformed sidecars (janitor treats those as
|
|
fully-orphaned + cleans up)."""
|
|
if not path.is_file():
|
|
return None
|
|
try:
|
|
text = path.read_text(encoding="utf-8")
|
|
except OSError:
|
|
return None
|
|
if not text.strip():
|
|
return None
|
|
try:
|
|
data = json.loads(text)
|
|
except json.JSONDecodeError:
|
|
return None
|
|
if not isinstance(data, dict):
|
|
return None
|
|
try:
|
|
return WorkerSession.from_dict(data)
|
|
except (TypeError, KeyError):
|
|
return None
|
|
|
|
|
|
__all__ = [
|
|
"WorkerSession",
|
|
"now_iso",
|
|
"read_sidecar",
|
|
"write_sidecar",
|
|
]
|