The dequeue+lock+heartbeat+runner+loop machinery. Production
OpenCode + MCP invocation slots in via the agent_runner callable
(Phase 1c-2). This commit is the structural foundation:
tools/controller/worker/:
- identity.py: build_instance_id() → "{hostname}/{pid}/{worker_uuid}"
per plan v9 (slash delimiter; IPv6-safe; uuid4 prefix for
per-instance uniqueness).
- heartbeat.py: Heartbeat thread that updates lock_heartbeat_at
every interval (default 30s). v9 simplified: TTL-only (no activity
tracking). UPDATE … WHERE locked_by_instance=us; rowcount=0 →
lost_lock_event.set() and thread exits, letting reaper handle it.
- runner.py: run_one_attempt() drives one attempt end-to-end.
Starts heartbeat → invokes agent_runner → on success writes
status='complete' + output_payload; on WorkerError writes
status='failed' with outcome label; on WorkerLostLock or detected
stolen-lock-at-write returns aborted (no DB write — reaper has
already re-pended). Defense-in-depth: even if agent returns
successfully, lost_lock_event.is_set() check skips the write.
- loop.py: worker_main_loop() polls the DB for pending attempts up
to MAX_CONCURRENT_WORKERS_PER_MACHINE, submits each to a
ThreadPoolExecutor. Honors stop_event for graceful shutdown
(drains in-flight before exit).
tools/controller/db/session.py: StaticPool for in-memory SQLite so
the heartbeat thread + runner write + dequeue all see the same DB
(without this, ":memory:" gives each connection an independent DB).
16 new tests in test_worker.py: instance ID format/uniqueness;
heartbeat tick (hold + steal); runner happy path; 5 error paths
(worker error / unexpected exception / WorkerLostLock raised /
stolen lock at write / lost_lock_event set defense-in-depth); 4
loop scenarios (single attempt, role filter skip, empty queue
exit-on-stop, explicit instance_id).
Total: 157 controller tests; full auto_agents suite 2519 pass.
