Fix alembic.ini and alembic/ not included in wheel distribution #4180

Closed
opened 2026-04-06 18:14:59 +00:00 by CoreRasurae · 3 comments
Member

Metadata

  • Commit Message: fix(database): include alembic files in package distribution
  • Branch: fix/alembic-distribution

Background and Context

When running agents init inside a Docker container built from the current Dockerfile, the application fails with:

FileNotFoundError: Alembic configuration file (alembic.ini) not found. 
Searched upward from: /usr/local/lib/python3.13/site-packages/cleveragents

The root cause is that alembic files (alembic.ini and the alembic/ directory) are located at the repository root but are not included in the Python wheel distribution. When the wheel is built and installed, Python packaging only includes files from the declared packages directory (src/cleveragents) and explicitly listed include files in pyproject.toml. Since the alembic files don't meet either criterion, they are silently omitted.

Affected code:

  • MigrationRunner._find_alembic_ini() in src/cleveragents/infrastructure/database/migration_runner.py:35-89 — searches for alembic.ini by walking upward from the package root, assuming it's in a predictable location relative to the installed package
  • Dockerfile (lines 16-17) — only copies src/ directory, not the repository-root alembic/ directory and alembic.ini
  • pyproject.toml (lines 109-111) — wheel build configuration only includes src/cleveragents package

This is a critical blocker for containerized deployments and any installation method that relies on the wheel distribution.

Current Behavior

  1. Docker image is built with python -m build --wheel
  2. Wheel is installed with pip install /tmp/*.whl
  3. When agents init is executed, it calls MigrationRunner.init_or_upgrade()
  4. init_or_upgrade() triggers alembic_cfg property, which calls _find_alembic_ini()
  5. _find_alembic_ini() searches for alembic.ini by walking up 10 levels from src/cleveragents or infrastructure/database
  6. The file is not found because it's not part of the installed wheel
  7. FileNotFoundError is raised and wrapped as INTERNAL [500] error

Expected Behavior

When running agents init in any deployment mode (Docker, local install, editable install), the application should:

  1. Successfully locate alembic.ini and the migration scripts
  2. Apply any pending database migrations without errors
  3. Initialize the database schema correctly

Acceptance Criteria

  1. Alembic files are properly packagedalembic.ini and the alembic/ directory (with all migration scripts) are included in the built wheel
  2. MigrationRunner works in all deployment contextsagents init succeeds in:
    • Docker containers (wheel-based install)
    • Local pip install (wheel or editable)
    • Development environments (editable install with pip install -e .)
  3. No modification to alembic search logic — the current search strategy in _find_alembic_ini() remains unchanged; we're only ensuring files are present
  4. Migration tests pass — existing migration-related tests in the test suite continue to pass with the new structure

Supporting Information

  • Configuration issue: pyproject.toml lines 109-111 define the wheel build configuration but do not include alembic files
  • Docker issue: Dockerfile lines 16-17 only copy source files, not repository-root assets
  • Test the fix: Run docker build -t cleveragents . && docker run cleveragents agents init test-project — this should succeed without errors

Solution Options

The project owner and maintainers must choose between Option 1 (Recommended) and Option 2 below. Both solve the problem; they represent different philosophical approaches to package organization.

Approach: Reorganize alembic files into the standard Python package structure.

Changes:

  1. Move alembic/ directory from repository root to src/cleveragents/infrastructure/database/migrations/
  2. Move alembic.ini to src/cleveragents/infrastructure/database/migrations/alembic.ini
  3. Update alembic.ini — change script_location from %(here)s/alembic to %(here)s/versions (the migrations directory)
  4. Update MigrationRunner._find_alembic_ini() to search from src/cleveragents/infrastructure/database/migrations/ as the primary anchor point
  5. No changes required to pyproject.toml (files in src/cleveragents/ are automatically included)
  6. Update Dockerfile — no changes needed; alembic files are now part of the package

Advantages:

  • Standard Python packaging practice — alembic files are part of the package code
  • Works in all deployment modes without special configuration (wheel, editable, local install)
  • No changes to pyproject.toml required
  • No Docker special cases required
  • Portable and maintainable long-term
  • If the package is published to PyPI, alembic files are automatically included

Disadvantages:

  • ⚠️ Reorganizes directory structure (may require refactoring of any other code that references alembic paths)
  • ⚠️ Alembic files are now "buried" in src/cleveragents/infrastructure/database/migrations/ rather than at repo root (but this is standard practice)

Subtasks

  • Implement chosen solution:
    • Move alembic/ directory to src/cleveragents/infrastructure/database/migrations/
    • Move alembic.ini to src/cleveragents/infrastructure/database/migrations/alembic.ini
    • Update alembic.ini script_location setting to correct relative path
    • Update MigrationRunner._find_alembic_ini() to search from new location
    • Update any documentation that references alembic file locations
  • Tests (Behave): Add scenarios for database initialization in containerized/packaged contexts
  • Tests (Robot): Add integration test for agents init in a fresh environment
  • Manual testing: Build Docker image and verify agents init <project> succeeds
  • Verify coverage ≥97% via nox -s coverage_report
  • Run full test suite: nox (all default sessions), fix any errors
  • Update documentation: Update any relevant docs that mention alembic file organization or installation

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines describing the implementation
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done
  • agents init successfully runs in a Docker container built from the updated Dockerfile without FileNotFoundError
  • All tests pass, including migration-specific tests
  • Test coverage remains ≥97%
## Metadata - **Commit Message**: `fix(database): include alembic files in package distribution` - **Branch**: `fix/alembic-distribution` ## Background and Context When running `agents init` inside a Docker container built from the current Dockerfile, the application fails with: ``` FileNotFoundError: Alembic configuration file (alembic.ini) not found. Searched upward from: /usr/local/lib/python3.13/site-packages/cleveragents ``` The root cause is that alembic files (`alembic.ini` and the `alembic/` directory) are located at the repository root but are **not included** in the Python wheel distribution. When the wheel is built and installed, Python packaging only includes files from the declared `packages` directory (`src/cleveragents`) and explicitly listed `include` files in `pyproject.toml`. Since the alembic files don't meet either criterion, they are silently omitted. **Affected code:** - `MigrationRunner._find_alembic_ini()` in `src/cleveragents/infrastructure/database/migration_runner.py:35-89` — searches for `alembic.ini` by walking upward from the package root, assuming it's in a predictable location relative to the installed package - Dockerfile (lines 16-17) — only copies `src/` directory, not the repository-root `alembic/` directory and `alembic.ini` - `pyproject.toml` (lines 109-111) — wheel build configuration only includes `src/cleveragents` package This is a critical blocker for containerized deployments and any installation method that relies on the wheel distribution. ## Current Behavior 1. Docker image is built with `python -m build --wheel` 2. Wheel is installed with `pip install /tmp/*.whl` 3. When `agents init` is executed, it calls `MigrationRunner.init_or_upgrade()` 4. `init_or_upgrade()` triggers `alembic_cfg` property, which calls `_find_alembic_ini()` 5. `_find_alembic_ini()` searches for `alembic.ini` by walking up 10 levels from `src/cleveragents` or `infrastructure/database` 6. The file is not found because it's not part of the installed wheel 7. `FileNotFoundError` is raised and wrapped as `INTERNAL [500]` error ## Expected Behavior When running `agents init` in any deployment mode (Docker, local install, editable install), the application should: 1. Successfully locate `alembic.ini` and the migration scripts 2. Apply any pending database migrations without errors 3. Initialize the database schema correctly ## Acceptance Criteria 1. **Alembic files are properly packaged** — `alembic.ini` and the `alembic/` directory (with all migration scripts) are included in the built wheel 2. **MigrationRunner works in all deployment contexts** — `agents init` succeeds in: - Docker containers (wheel-based install) - Local pip install (wheel or editable) - Development environments (editable install with `pip install -e .`) 3. **No modification to alembic search logic** — the current search strategy in `_find_alembic_ini()` remains unchanged; we're only ensuring files are present 4. **Migration tests pass** — existing migration-related tests in the test suite continue to pass with the new structure ## Supporting Information - **Configuration issue**: `pyproject.toml` lines 109-111 define the wheel build configuration but do not include alembic files - **Docker issue**: `Dockerfile` lines 16-17 only copy source files, not repository-root assets - **Test the fix**: Run `docker build -t cleveragents . && docker run cleveragents agents init test-project` — this should succeed without errors --- ## Solution Options The project owner and maintainers must choose between **Option 1** (Recommended) and **Option 2** below. Both solve the problem; they represent different philosophical approaches to package organization. ### Option 1: Move alembic files into the package ⭐ **RECOMMENDED** **Approach:** Reorganize alembic files into the standard Python package structure. **Changes:** 1. Move `alembic/` directory from repository root to `src/cleveragents/infrastructure/database/migrations/` 2. Move `alembic.ini` to `src/cleveragents/infrastructure/database/migrations/alembic.ini` 3. Update `alembic.ini` — change `script_location` from `%(here)s/alembic` to `%(here)s/versions` (the migrations directory) 4. Update `MigrationRunner._find_alembic_ini()` to search from `src/cleveragents/infrastructure/database/migrations/` as the primary anchor point 5. No changes required to `pyproject.toml` (files in `src/cleveragents/` are automatically included) 6. Update `Dockerfile` — no changes needed; alembic files are now part of the package **Advantages:** - ✅ Standard Python packaging practice — alembic files are part of the package code - ✅ Works in all deployment modes without special configuration (wheel, editable, local install) - ✅ No changes to `pyproject.toml` required - ✅ No Docker special cases required - ✅ Portable and maintainable long-term - ✅ If the package is published to PyPI, alembic files are automatically included **Disadvantages:** - ⚠️ Reorganizes directory structure (may require refactoring of any other code that references alembic paths) - ⚠️ Alembic files are now "buried" in `src/cleveragents/infrastructure/database/migrations/` rather than at repo root (but this is standard practice) --- ## Subtasks - [ ] **Implement chosen solution**: - [ ] Move `alembic/` directory to `src/cleveragents/infrastructure/database/migrations/` - [ ] Move `alembic.ini` to `src/cleveragents/infrastructure/database/migrations/alembic.ini` - [ ] Update `alembic.ini` `script_location` setting to correct relative path - [ ] Update `MigrationRunner._find_alembic_ini()` to search from new location - [ ] Update any documentation that references alembic file locations - [ ] **Tests (Behave)**: Add scenarios for database initialization in containerized/packaged contexts - [ ] **Tests (Robot)**: Add integration test for `agents init` in a fresh environment - [ ] **Manual testing**: Build Docker image and verify `agents init <project>` succeeds - [ ] **Verify coverage** ≥97% via `nox -s coverage_report` - [ ] **Run full test suite**: `nox` (all default sessions), fix any errors - [ ] **Update documentation**: Update any relevant docs that mention alembic file organization or installation ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines describing the implementation - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done - `agents init` successfully runs in a Docker container built from the updated Dockerfile without `FileNotFoundError` - All tests pass, including migration-specific tests - Test coverage remains ≥97%
Author
Member

I believe the best solution is to for Option 1 since:
Option 1 does NOT violate either CONTRIBUTING.md nor docs/specification.md — in fact, it aligns better with CONTRIBUTING.md's philosophy because:

  1. It moves a non-standard file from the root into an appropriate subdirectory
  2. It co-locates database migrations with other database code (infrastructure/database)
  3. It follows Python packaging best practices (migrations are part of the package)

Please advise.

I believe the best solution is to for Option 1 since: Option 1 does NOT violate either CONTRIBUTING.md nor docs/specification.md — in fact, it aligns better with CONTRIBUTING.md's philosophy because: 1. It moves a non-standard file from the root into an appropriate subdirectory 2. It co-locates database migrations with other database code (infrastructure/database) 3. It follows Python packaging best practices (migrations are part of the package) Please advise.
HAL9000 self-assigned this 2026-04-08 18:49:12 +00:00
HAL9000 added this to the v3.2.0 milestone 2026-04-08 18:49:12 +00:00
Owner

Issue assigned to @HAL9000 and milestone set to v3.2.0.

Milestone Rationale: This is a fundamental build/distribution issue — alembic files are missing from the wheel, which causes agents init to fail in Docker containers. This blocks basic deployment and is core infrastructure that belongs in the earliest milestone.

Assignment Rationale: Default assignment to HAL9000 to maintain velocity. This is a build configuration fix.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner

Issue assigned to @HAL9000 and milestone set to **v3.2.0**. **Milestone Rationale**: This is a fundamental build/distribution issue — alembic files are missing from the wheel, which causes `agents init` to fail in Docker containers. This blocks basic deployment and is core infrastructure that belongs in the earliest milestone. **Assignment Rationale**: Default assignment to HAL9000 to maintain velocity. This is a build configuration fix. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner
Author
Member

Implementation In Progress

This issue is being addressed by PR #9408: fix(database): include alembic files in package distribution

PR Blocks This Issue - Once merged, this PR will resolve issue #4180 by:

  • Moving alembic configuration and migration files into the Python package structure
  • Ensuring alembic files are included in wheel distributions
  • Allowing agents init to work correctly in Docker containers and other wheel-based installations

Please review the PR for implementation details.

## Implementation In Progress This issue is being addressed by **PR #9408**: [fix(database): include alembic files in package distribution](https://git.cleverthis.com/cleveragents/cleveragents-core/pulls/9408) **PR Blocks This Issue** - Once merged, this PR will resolve issue #4180 by: - Moving alembic configuration and migration files into the Python package structure - Ensuring alembic files are included in wheel distributions - Allowing `agents init` to work correctly in Docker containers and other wheel-based installations Please review the PR for implementation details.
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#4180
No description provided.