cleveragents-core/docs/adr/ADR-019-storage-and-persistence.md

adr_number, title, status_history, tier, authors, superseded_by, related_adrs, acceptance

adr_number	title	status_history	tier	authors	superseded_by	related_adrs	acceptance
19	Storage and Persistence	2026-02-16 Proposed Jeffrey Phillips Freeman 2026-02-16 Accepted Jeffrey Phillips Freeman	3	Jeffrey Phillips Freeman	null	number title relationship 1 Layered Architecture Persistence adapters reside in the Infrastructure Layer behind domain ports number title relationship 3 Dependency Injection Repository implementations are injected via the DI container number title relationship 4 Data Validation Pydantic models define the schemas that map to SQLAlchemy ORM models number title relationship 5 Technical Stack SQLAlchemy, SQLite, and PostgreSQL are the chosen persistence technologies	votes_for votes_against abstentions voter comment Jeffrey Phillips Freeman <Jeffrey.Freeman@CleverThis.com> Repository pattern with Unit of Work and dual-database support covers both local and server deployment cleanly

Context

CleverAgents must persist a rich data model — plans with hierarchical decision trees, resources with DAG relationships, actors, tools, skills, sessions, invariants, automation profiles, checkpoints, correction attempts, and more. The persistence layer must support local single-user mode (zero-configuration) and server multi-user mode (concurrent access). It must integrate with the layered architecture, supporting repository pattern interfaces in the domain layer and concrete implementations in the infrastructure layer.

Decision Drivers

• Local single-user mode requires zero-configuration storage with no external database server
• Server multi-user mode requires concurrent access and connection pooling (PostgreSQL)
• Domain layer code must remain decoupled from persistence mechanics via repository interfaces
• A rich data model (plans, decision trees, resources, actors, sessions, checkpoints) needs version-controlled schema migrations
• Entity IDs must be globally unique and lexicographically sortable by creation time without coordination

Decision

CleverAgents uses SQLite as the primary database for local mode and SQLAlchemy as the ORM for database abstraction across all modes. The infrastructure layer implements the repository pattern with Unit of Work for transactional consistency. Alembic manages version-controlled schema migrations. IDs use ULIDs for lexicographically sortable, time-ordered identification.

Design

SQLite for Local Mode

SQLite provides zero-configuration, file-based, ACID-compliant storage. WAL (Write-Ahead Logging) mode is enabled for concurrent reads during plan execution (important when multiple plans or background indexing are running). The database file is stored at <core.data-dir>/cleveragents.db by default.

SQLAlchemy ORM

SQLAlchemy provides:

• Declarative model mapping: Domain entities map to database tables via SQLAlchemy declarative models in the infrastructure layer.
• Session management: SQLAlchemy sessions manage the lifecycle of database transactions.
• Unit of Work pattern: Changes are accumulated in a session and committed atomically, ensuring transactional consistency.
• Dialect abstraction: The same repository implementations work with SQLite (local mode) and PostgreSQL (server mode) by switching the SQLAlchemy dialect. No query changes needed.

Repository Pattern

Domain Layer defines repository interfaces as Protocol classes:

class PlanRepository(Protocol):
    def get(self, plan_id: ULID) -> Plan: ...
    def save(self, plan: Plan) -> None: ...
    def find_by_project(self, project_name: str) -> list[Plan]: ...

Infrastructure Layer provides concrete implementations:

class SQLAlchemyPlanRepository:
    def __init__(self, session_factory: SessionFactory): ...
    def get(self, plan_id: ULID) -> Plan: ...
    def save(self, plan: Plan) -> None: ...
    def find_by_project(self, project_name: str) -> list[Plan]: ...

The DI container (ADR-003) maps protocol to implementation.

Alembic Migrations

Schema migrations are version-controlled with Alembic:

• Migrations support both upgrade and downgrade operations.
• Migrations are auto-applied on first run and during agents init.
• Migration files live alongside the infrastructure code.

ULID Identifiers

All entity IDs (plans, decisions, resources, correction attempts, etc.) use ULIDs generated by python-ulid (>= 2.7.0). ULIDs are:

• Lexicographically sortable by creation time: Efficient time-range queries without separate timestamp indexes.
• Globally unique: No coordination needed for distributed ID generation.
• 128-bit: Stored as 26-character CROCKFORD Base32 strings.

Key Database Tables

The schema includes tables for: plans, decisions, decision_dependencies, correction_attempts, actions, resources, resource_types, resource_links (project-resource mapping), projects, actors, tools, skills, sessions, invariants, automation_profiles, checkpoints, validation_results, and configuration/metadata tables.

Server Mode Database

For server mode, the infrastructure layer swaps SQLite for PostgreSQL via the same SQLAlchemy interface. The repository implementations remain unchanged — only the connection dialect and URL change. PostgreSQL provides concurrent multi-user access, connection pooling, and horizontal scaling capabilities.

Backup and Retention

• Backup snapshots stored at <core.data-dir>/backups, managed by core.backup.retention-days (default: 7).
• Backups created during project deletion, agents init resets, and correction history cleanup.
• Automatic purge of backups older than the retention period.

Constraints

• Domain Layer code must never import SQLAlchemy or reference database tables. All persistence goes through repository protocol interfaces.
• Repository implementations must use the Unit of Work pattern — no auto-commit, no implicit transactions.
• All entity IDs must be ULIDs. Sequential integer IDs are not permitted for domain entities.
• Schema changes must go through Alembic migrations. Direct schema modifications are prohibited.
• The database must support concurrent read access via WAL mode (SQLite) or native concurrency (PostgreSQL).

Consequences

Positive

• SQLite provides zero-configuration local mode — no database server to install or manage.
• SQLAlchemy's dialect abstraction enables seamless switching between SQLite and PostgreSQL for server mode.
• The repository pattern ensures domain logic is completely decoupled from persistence mechanics.
• ULIDs provide time-ordered IDs without requiring database sequences or timestamp indexes.
• Alembic migrations provide reversible, version-controlled schema evolution.

Negative

• SQLite has limited concurrent write support (single-writer), which could bottleneck parallel plan execution.
• SQLAlchemy ORM overhead may impact performance for high-frequency operations (e.g., decision recording during active execution).
• The repository pattern adds boilerplate for each entity type (protocol definition + implementation).

Risks

• SQLite WAL mode has platform-specific behavior and limitations (e.g., network filesystem incompatibility).
• Alembic migrations that fail partway through can leave the database in an inconsistent state.
• ULID string storage is less space-efficient than integer or binary ID types.

Alternatives Considered

Raw SQL without ORM — More performant for simple queries but loses the dialect abstraction, session management, and declarative model mapping that SQLAlchemy provides. The portability between SQLite and PostgreSQL would require maintaining two sets of SQL queries.

DuckDB or other embedded databases — Optimized for analytical workloads rather than the transactional OLTP patterns (insert, update, lookup by ID) that dominate CleverAgents usage.

Compliance

• Repository isolation tests: Tests verify that domain code never imports from sqlalchemy or alembic.
• Migration tests: Tests verify that all Alembic migrations can be applied (upgrade) and reversed (downgrade) cleanly against a fresh database.
• Dialect portability tests: Key repository operations are tested against both SQLite and PostgreSQL to verify dialect abstraction.
• ULID ordering tests: Tests verify that ULID-based queries return results in creation-time order.
• Concurrency tests: Tests verify that concurrent read operations work correctly under WAL mode.
• Backup/restore tests: Tests verify that backup creation and restoration produce consistent database state.