drew
eab476e48e
The dequeue+lock+heartbeat+runner+loop machinery. Production
OpenCode + MCP invocation slots in via the agent_runner callable
(Phase 1c-2). This commit is the structural foundation:
tools/controller/worker/:
- identity.py: build_instance_id() → "{hostname}/{pid}/{worker_uuid}"
per plan v9 (slash delimiter; IPv6-safe; uuid4 prefix for
per-instance uniqueness).
- heartbeat.py: Heartbeat thread that updates lock_heartbeat_at
every interval (default 30s). v9 simplified: TTL-only (no activity
tracking). UPDATE … WHERE locked_by_instance=us; rowcount=0 →
lost_lock_event.set() and thread exits, letting reaper handle it.
- runner.py: run_one_attempt() drives one attempt end-to-end.
Starts heartbeat → invokes agent_runner → on success writes
status='complete' + output_payload; on WorkerError writes
status='failed' with outcome label; on WorkerLostLock or detected
stolen-lock-at-write returns aborted (no DB write — reaper has
already re-pended). Defense-in-depth: even if agent returns
successfully, lost_lock_event.is_set() check skips the write.
- loop.py: worker_main_loop() polls the DB for pending attempts up
to MAX_CONCURRENT_WORKERS_PER_MACHINE, submits each to a
ThreadPoolExecutor. Honors stop_event for graceful shutdown
(drains in-flight before exit).
tools/controller/db/session.py: StaticPool for in-memory SQLite so
the heartbeat thread + runner write + dequeue all see the same DB
(without this, ":memory:" gives each connection an independent DB).
16 new tests in test_worker.py: instance ID format/uniqueness;
heartbeat tick (hold + steal); runner happy path; 5 error paths
(worker error / unexpected exception / WorkerLostLock raised /
stolen lock at write / lost_lock_event set defense-in-depth); 4
loop scenarios (single attempt, role filter skip, empty queue
exit-on-stop, explicit instance_id).
Total: 157 controller tests; full auto_agents suite 2519 pass.
34 lines
1.1 KiB
Python
34 lines
1.1 KiB
Python
"""Worker instance identity.
|
|
|
|
Per plan v9 ``instance_id`` format: ``{hostname}/{pid}/{worker_uuid}``.
|
|
|
|
- ``hostname`` — which machine. Lets operator ``controller-cli`` /
|
|
log queries say "kill all workers on host X."
|
|
- ``pid`` — which process. Lets operator inspect logs / strace.
|
|
- ``worker_uuid`` — unique per worker instance even if pid recycles
|
|
(Linux pid wrap-around or worker restart with same pid).
|
|
|
|
Slash delimiter (not ``:``) so IPv6 hostnames don't trip parsing.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
import socket
|
|
import uuid
|
|
|
|
|
|
def build_instance_id(*, hostname: str | None = None) -> str:
|
|
"""Generate a fresh worker instance id.
|
|
|
|
Call once at worker startup and keep the result for the worker's
|
|
lifetime. Each attempt holds the same instance_id; the heartbeat
|
|
UPDATEs use it as the WHERE clause's authoritative owner check.
|
|
"""
|
|
host = hostname or socket.gethostname() or "unknown-host"
|
|
pid = os.getpid()
|
|
worker_uuid = uuid.uuid4().hex[:12]
|
|
return f"{host}/{pid}/{worker_uuid}"
|
|
|
|
|
|
__all__ = ["build_instance_id"]
|