The dequeue+lock+heartbeat+runner+loop machinery. Production
OpenCode + MCP invocation slots in via the agent_runner callable
(Phase 1c-2). This commit is the structural foundation:
tools/controller/worker/:
- identity.py: build_instance_id() → "{hostname}/{pid}/{worker_uuid}"
per plan v9 (slash delimiter; IPv6-safe; uuid4 prefix for
per-instance uniqueness).
- heartbeat.py: Heartbeat thread that updates lock_heartbeat_at
every interval (default 30s). v9 simplified: TTL-only (no activity
tracking). UPDATE … WHERE locked_by_instance=us; rowcount=0 →
lost_lock_event.set() and thread exits, letting reaper handle it.
- runner.py: run_one_attempt() drives one attempt end-to-end.
Starts heartbeat → invokes agent_runner → on success writes
status='complete' + output_payload; on WorkerError writes
status='failed' with outcome label; on WorkerLostLock or detected
stolen-lock-at-write returns aborted (no DB write — reaper has
already re-pended). Defense-in-depth: even if agent returns
successfully, lost_lock_event.is_set() check skips the write.
- loop.py: worker_main_loop() polls the DB for pending attempts up
to MAX_CONCURRENT_WORKERS_PER_MACHINE, submits each to a
ThreadPoolExecutor. Honors stop_event for graceful shutdown
(drains in-flight before exit).
tools/controller/db/session.py: StaticPool for in-memory SQLite so
the heartbeat thread + runner write + dequeue all see the same DB
(without this, ":memory:" gives each connection an independent DB).
16 new tests in test_worker.py: instance ID format/uniqueness;
heartbeat tick (hold + steal); runner happy path; 5 error paths
(worker error / unexpected exception / WorkerLostLock raised /
stolen lock at write / lost_lock_event set defense-in-depth); 4
loop scenarios (single attempt, role filter skip, empty queue
exit-on-stop, explicit instance_id).
Total: 157 controller tests; full auto_agents suite 2519 pass.
Five SQLAlchemy 2.0 declarative models implementing plan v6/v9's
unified workflow schema. Cross-dialect (SQLite for tests + local dev,
Postgres for multi-machine production). Lock columns on
workflow_attempts implement the multi-machine-safe dequeue protocol
(plan v5).
Modules:
- tools/controller/db/models.py:
- Workflow (kind discriminator pr/issue, unique on owner+repo+kind+
entity_number, parent_workflow_id FK for issue→PR linkage)
- WorkflowAttempt (status/locked_by_instance/locked_at/
lock_heartbeat_at/lock_ttl_seconds/pickup_count + CHECK
constraints on status enum and pickup_count≥0; partial indexes
on the pending/in_progress/complete hot paths)
- ControllerEvent (Forgejo-write replay support kept in-schema even
though v9 simplified to Forgejo-first protocol; allows v3-style
upgrade later without migration)
- FlakeHistory (composite PK; supports the v6 flake-learning
heuristic)
- CIObservation (raw CI state history; 90-day retention to be
enforced by a sweep task)
- AutoincrementPk variant (Integer on SQLite where it autoincrements
via rowid; BigInteger on Postgres for BIGSERIAL); JsonColumn
variant (JSON on SQLite, JSONB on Postgres)
- tools/controller/db/session.py: build_engine (per-dialect tuning —
SQLite WAL + foreign_keys + busy_timeout; Postgres pool_pre_ping);
create_all (idempotent); session_scope (transactional context).
- tools/controller/db/dequeue.py: dequeue_one (one row atomically;
Postgres path uses SELECT FOR UPDATE SKIP LOCKED, SQLite path uses
UPDATE-WHERE-id-IN-SELECT-LIMIT-1 with RETURNING). Bumps pickup_count
on dequeue; respects max_pickups guard (default 3 per v6 blocker fix).
Returns DequeueResult dataclass with role/tier/pickup_count and
reason on miss.
- tools/controller/db/payload_guard.py: enforce_input_payload_size
with 4MB cap and 5-step truncation priority (older_summary → oldest
verbatim → comments → full_diff → CI failure excerpts). Raises
PayloadTooLargeError after all steps exhausted; master maps to
workflow STUCK with reason='input-too-large'.
- pyproject.toml: new optional extras `controller-db` pinning
sqlalchemy + psycopg2-binary (latter installed only for prod
multi-machine deploy; tests use stdlib sqlite3 via SQLAlchemy's
SQLite dialect which is already pulled in transitively via alembic).
37 new tests across test_db_schema/dequeue/payload_guard; 141
controller tests total; full auto_agents suite 2503 pass (no
regressions).
drew