feat(server): implement PostgreSQL storage backend for server mode
Add PostgreSQL support as the server-mode storage backend alongside existing SQLite for local mode. Verify all ORM models are dialect- agnostic, configure connection pooling for multi-user access, add Docker Compose for local PG development, and wire database URL selection based on deployment mode. Changes: - Add psycopg2-binary dependency to pyproject.toml - Add server_mode, db_pool_size, db_max_overflow, db_pool_recycle settings to Settings with environment variable support - Add resolve_database_url() and is_postgresql() to Settings for mode-aware database URL resolution - Configure UnitOfWork engine creation with pool_size, max_overflow, pool_recycle, and pool_pre_ping for PostgreSQL connections - Update MigrationRunner to handle both SQLite and PostgreSQL backends - Add compare_type=True to Alembic env.py for dialect-aware migrations - Add docker-compose.yml with PostgreSQL 16-alpine for local development - Add Behave BDD feature (14 scenarios) covering settings, pool config, engine creation, ORM dialect compatibility, and migration runner - Add Robot Framework integration tests (12 test cases) for the abstraction layer with requires_postgresql tag for live PG tests ISSUES CLOSED: #878
This commit is contained in:
@@ -0,0 +1,165 @@
|
||||
*** Settings ***
|
||||
Documentation Integration tests for PostgreSQL storage backend abstraction layer.
|
||||
... Tests the dialect-agnostic ORM models, connection configuration,
|
||||
... and database URL selection logic using SQLite as the test backend.
|
||||
... Tests tagged @requires_postgresql are excluded from default CI runs.
|
||||
Library OperatingSystem
|
||||
Library String
|
||||
Library Collections
|
||||
Library Process
|
||||
Library indentation_library.py
|
||||
Resource ${CURDIR}/common.resource
|
||||
|
||||
*** Variables ***
|
||||
${PYTHON} python
|
||||
|
||||
*** Test Cases ***
|
||||
|
||||
Settings Resolve Database URL Local Mode
|
||||
[Documentation] Verify resolve_database_url returns SQLite in local mode
|
||||
${script}= Catenate SEPARATOR=\n
|
||||
... import os
|
||||
... os.environ.pop('CLEVERAGENTS_SERVER_MODE', None)
|
||||
... os.environ.pop('CLEVERAGENTS_DATABASE_URL', None)
|
||||
... from cleveragents.config.settings import Settings
|
||||
... s = Settings()
|
||||
... print(s.resolve_database_url())
|
||||
${result}= Run Process ${PYTHON} -c ${script}
|
||||
... cwd=${CURDIR}/..
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Start With ${result.stdout.strip()} sqlite
|
||||
|
||||
Settings Resolve Database URL Server Mode
|
||||
[Documentation] Verify resolve_database_url returns PostgreSQL when server_mode is on
|
||||
${script}= Catenate SEPARATOR=\n
|
||||
... import os
|
||||
... os.environ['CLEVERAGENTS_SERVER_MODE'] = 'true'
|
||||
... os.environ.pop('CLEVERAGENTS_DATABASE_URL', None)
|
||||
... from cleveragents.config.settings import Settings
|
||||
... s = Settings()
|
||||
... print(s.resolve_database_url())
|
||||
${result}= Run Process ${PYTHON} -c ${script}
|
||||
... cwd=${CURDIR}/..
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Start With ${result.stdout.strip()} postgresql
|
||||
|
||||
Settings Is PostgreSQL Returns True For PG URL
|
||||
[Documentation] Verify is_postgresql detects PostgreSQL URLs
|
||||
${script}= Catenate SEPARATOR=\n
|
||||
... import os
|
||||
... os.environ['CLEVERAGENTS_SERVER_MODE'] = 'true'
|
||||
... os.environ['CLEVERAGENTS_DATABASE_URL'] = 'postgresql://localhost/db'
|
||||
... from cleveragents.config.settings import Settings
|
||||
... s = Settings()
|
||||
... print(s.is_postgresql())
|
||||
${result}= Run Process ${PYTHON} -c ${script}
|
||||
... cwd=${CURDIR}/..
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Be Equal As Strings ${result.stdout.strip()} True
|
||||
|
||||
Settings Is PostgreSQL Returns False For SQLite URL
|
||||
[Documentation] Verify is_postgresql returns False for SQLite URLs
|
||||
${script}= Catenate SEPARATOR=\n
|
||||
... import os
|
||||
... os.environ.pop('CLEVERAGENTS_SERVER_MODE', None)
|
||||
... os.environ.pop('CLEVERAGENTS_DATABASE_URL', None)
|
||||
... from cleveragents.config.settings import Settings
|
||||
... s = Settings()
|
||||
... print(s.is_postgresql())
|
||||
${result}= Run Process ${PYTHON} -c ${script}
|
||||
... cwd=${CURDIR}/..
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Be Equal As Strings ${result.stdout.strip()} False
|
||||
|
||||
UnitOfWork Creates SQLite Engine With Pool Parameters
|
||||
[Documentation] UnitOfWork stores pool parameters and creates an SQLite engine
|
||||
${result}= Run Process ${PYTHON} -c
|
||||
... from cleveragents.infrastructure.database.unit_of_work import UnitOfWork; u \= UnitOfWork('sqlite:///:memory:', prompt_for_migration\=lambda m: True, pool_size\=8, max_overflow\=15, pool_recycle\=900); print(u._pool_size, u._max_overflow, u._pool_recycle, u.engine.dialect.name)
|
||||
... cwd=${CURDIR}/..
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout.strip()} 8 15 900 sqlite
|
||||
|
||||
ORM Models Use Dialect-Agnostic Types
|
||||
[Documentation] All ORM column types are portable across SQLite and PostgreSQL
|
||||
${script}= Catenate SEPARATOR=\n
|
||||
... from cleveragents.infrastructure.database.models import Base
|
||||
... PORTABLE = {'VARCHAR','STRING','TEXT','INTEGER','BIGINTEGER','SMALLINTEGER',
|
||||
... 'FLOAT','NUMERIC','BOOLEAN','DATE','DATETIME','TIME','TIMESTAMP','JSON',
|
||||
... 'BLOB','LARGEBINARY','ENUM'}
|
||||
... for t in Base.metadata.tables.values():
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}for c in t.columns:
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}tn = type(c.type).__name__.upper()
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}if tn not in PORTABLE and not any(b in tn for b in PORTABLE):
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}raise AssertionError(f'{t.name}.{c.name} has non-portable type {tn}')
|
||||
... print('ALL_PORTABLE')
|
||||
${result}= Run Process ${PYTHON} -c ${script}
|
||||
... cwd=${CURDIR}/..
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout.strip()} ALL_PORTABLE
|
||||
|
||||
ORM Models Source Contains No SQLite Specific Code
|
||||
[Documentation] models.py should not import sqlite3 or use PRAGMAs
|
||||
${script}= Catenate SEPARATOR=\n
|
||||
... import inspect
|
||||
... from cleveragents.infrastructure.database import models
|
||||
... src = inspect.getsource(models)
|
||||
... for forbidden in ['sqlite3', 'PRAGMA']:
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}assert forbidden not in src, f'Found {forbidden}'
|
||||
... print('CLEAN')
|
||||
${result}= Run Process ${PYTHON} -c ${script}
|
||||
... cwd=${CURDIR}/..
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout.strip()} CLEAN
|
||||
|
||||
Table Names Are Valid PostgreSQL Identifiers
|
||||
[Documentation] All table names are lowercase SQL identifiers
|
||||
${script}= Catenate SEPARATOR=\n
|
||||
... import re
|
||||
... from cleveragents.infrastructure.database.models import Base
|
||||
... pat = re.compile(r'^[a-z_][a-z0-9_]*$')
|
||||
... for t in Base.metadata.tables.values():
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}assert pat.match(t.name), f'Invalid table: {t.name}'
|
||||
... print('VALID')
|
||||
${result}= Run Process ${PYTHON} -c ${script}
|
||||
... cwd=${CURDIR}/..
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout.strip()} VALID
|
||||
|
||||
Docker Compose File Exists For PostgreSQL Development
|
||||
[Documentation] docker-compose.yml exists at project root
|
||||
File Should Exist ${CURDIR}/../docker-compose.yml
|
||||
|
||||
Pool Defaults Are Multi-User Appropriate
|
||||
[Documentation] Default pool settings: pool_size=5, max_overflow=10, pool_recycle=1800
|
||||
${script}= Catenate SEPARATOR=\n
|
||||
... import os
|
||||
... os.environ['CLEVERAGENTS_SERVER_MODE'] = 'true'
|
||||
... os.environ.pop('CLEVERAGENTS_DATABASE_URL', None)
|
||||
... from cleveragents.config.settings import Settings
|
||||
... s = Settings()
|
||||
... print(s.db_pool_size, s.db_max_overflow, s.db_pool_recycle)
|
||||
${result}= Run Process ${PYTHON} -c ${script}
|
||||
... cwd=${CURDIR}/..
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Be Equal As Strings ${result.stdout.strip()} 5 10 1800
|
||||
|
||||
Migration Runner Accepts PostgreSQL URL
|
||||
[Documentation] MigrationRunner can be constructed with a PostgreSQL URL
|
||||
${result}= Run Process ${PYTHON} -c
|
||||
... from cleveragents.infrastructure.database.migration_runner import MigrationRunner; r \= MigrationRunner('postgresql://localhost/test'); print(r.database_url)
|
||||
... cwd=${CURDIR}/..
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Start With ${result.stdout.strip()} postgresql
|
||||
|
||||
PostgreSQL Live Connection Test
|
||||
[Documentation] Connect to a real PostgreSQL instance and run migrations.
|
||||
... Requires a running PostgreSQL server and CLEVERAGENTS_DATABASE_URL
|
||||
... pointing to it. Excluded from default CI runs.
|
||||
[Tags] requires_postgresql
|
||||
${pg_url}= Get Environment Variable CLEVERAGENTS_DATABASE_URL default=SKIP
|
||||
Skip If '${pg_url}' == 'SKIP' PostgreSQL not available
|
||||
${result}= Run Process ${PYTHON} -c
|
||||
... from cleveragents.infrastructure.database.migration_runner import MigrationRunner; r \= MigrationRunner('${pg_url}'); r.init_or_upgrade(); print('OK')
|
||||
... cwd=${CURDIR}/..
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout.strip()} OK
|
||||
Reference in New Issue
Block a user