drew
36b133ec5e
Five SQLAlchemy 2.0 declarative models implementing plan v6/v9's
unified workflow schema. Cross-dialect (SQLite for tests + local dev,
Postgres for multi-machine production). Lock columns on
workflow_attempts implement the multi-machine-safe dequeue protocol
(plan v5).
Modules:
- tools/controller/db/models.py:
- Workflow (kind discriminator pr/issue, unique on owner+repo+kind+
entity_number, parent_workflow_id FK for issue→PR linkage)
- WorkflowAttempt (status/locked_by_instance/locked_at/
lock_heartbeat_at/lock_ttl_seconds/pickup_count + CHECK
constraints on status enum and pickup_count≥0; partial indexes
on the pending/in_progress/complete hot paths)
- ControllerEvent (Forgejo-write replay support kept in-schema even
though v9 simplified to Forgejo-first protocol; allows v3-style
upgrade later without migration)
- FlakeHistory (composite PK; supports the v6 flake-learning
heuristic)
- CIObservation (raw CI state history; 90-day retention to be
enforced by a sweep task)
- AutoincrementPk variant (Integer on SQLite where it autoincrements
via rowid; BigInteger on Postgres for BIGSERIAL); JsonColumn
variant (JSON on SQLite, JSONB on Postgres)
- tools/controller/db/session.py: build_engine (per-dialect tuning —
SQLite WAL + foreign_keys + busy_timeout; Postgres pool_pre_ping);
create_all (idempotent); session_scope (transactional context).
- tools/controller/db/dequeue.py: dequeue_one (one row atomically;
Postgres path uses SELECT FOR UPDATE SKIP LOCKED, SQLite path uses
UPDATE-WHERE-id-IN-SELECT-LIMIT-1 with RETURNING). Bumps pickup_count
on dequeue; respects max_pickups guard (default 3 per v6 blocker fix).
Returns DequeueResult dataclass with role/tier/pickup_count and
reason on miss.
- tools/controller/db/payload_guard.py: enforce_input_payload_size
with 4MB cap and 5-step truncation priority (older_summary → oldest
verbatim → comments → full_diff → CI failure excerpts). Raises
PayloadTooLargeError after all steps exhausted; master maps to
workflow STUCK with reason='input-too-large'.
- pyproject.toml: new optional extras `controller-db` pinning
sqlalchemy + psycopg2-binary (latter installed only for prod
multi-machine deploy; tests use stdlib sqlite3 via SQLAlchemy's
SQLite dialect which is already pulled in transitively via alembic).
37 new tests across test_db_schema/dequeue/payload_guard; 141
controller tests total; full auto_agents suite 2503 pass (no
regressions).
59 lines
1.3 KiB
Python
59 lines
1.3 KiB
Python
"""Controller DB layer.
|
|
|
|
SQLAlchemy 2.0 declarative schema for the workflows, attempts,
|
|
events, flake-history, and ci-observations tables. The same schema
|
|
serves SQLite (tests + local dev) and Postgres (production; multi-
|
|
machine workers per plan v5).
|
|
|
|
The DB connection is configured by ``CLEVERAGENTS_DB_URL``:
|
|
postgresql+psycopg2://user@host:5432/cleveragents
|
|
sqlite:///./controller.db
|
|
sqlite:///:memory: (tests)
|
|
|
|
The dequeue helper (``dequeue.py``) abstracts over the two dialects'
|
|
different concurrency primitives:
|
|
- Postgres: ``SELECT … FOR UPDATE SKIP LOCKED``
|
|
- SQLite: ``BEGIN IMMEDIATE`` + serialized UPDATE
|
|
"""
|
|
from .models import (
|
|
Base,
|
|
CIObservation,
|
|
ControllerEvent,
|
|
FlakeHistory,
|
|
Workflow,
|
|
WorkflowAttempt,
|
|
)
|
|
from .session import (
|
|
build_engine,
|
|
create_all,
|
|
session_scope,
|
|
)
|
|
from .dequeue import (
|
|
DequeueResult,
|
|
dequeue_one,
|
|
)
|
|
from .payload_guard import (
|
|
PayloadTooLargeError,
|
|
enforce_input_payload_size,
|
|
)
|
|
|
|
__all__ = [
|
|
# models
|
|
"Base",
|
|
"CIObservation",
|
|
"ControllerEvent",
|
|
"FlakeHistory",
|
|
"Workflow",
|
|
"WorkflowAttempt",
|
|
# session
|
|
"build_engine",
|
|
"create_all",
|
|
"session_scope",
|
|
# dequeue
|
|
"DequeueResult",
|
|
"dequeue_one",
|
|
# payload guard
|
|
"PayloadTooLargeError",
|
|
"enforce_input_payload_size",
|
|
]
|