`nox -s unit_tests` fails with `sqlite3.OperationalError: attempt to write a readonly database` when template DB exists on btrfs #9389

New issue

Closed

opened 2026-04-14 16:31:55 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-14 16:31:55 +00:00

Owner

Metadata

Commit Message: fix(tests): handle readonly template DB on btrfs in create_template_db.py
Branch: fix/nox-template-db-readonly-btrfs

Background and Context

The nox -s unit_tests session calls scripts/create_template_db.py to create a pre-migrated SQLite template database at build/.template-migrated.db. On btrfs filesystems (used in the CI/UAT environment), the script fails with a sqlite3.OperationalError: attempt to write a readonly database error when the template DB file already exists from a previous run.

Discovered during UAT testing of the Execution Environment routing feature (UAT Test Pool — 2026-04-14).

Current Behavior

When build/.template-migrated.db already exists (from a previous successful nox run), the create_template_db.py script:

Calls out.unlink() to delete the existing file
Creates a new empty file via SQLAlchemy's create_engine()
Fails during Base.metadata.create_all(engine) with:

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) attempt to write a readonly database
[SQL: CREATE INDEX ix_decisions_parent ON decisions (parent_decision_id)]

The error occurs on btrfs because newly created files after unlink() may have filesystem-level constraints that prevent SQLite from writing to them in certain btrfs configurations (overlayfs + btrfs COW interaction).

Reproduction steps:

Run nox -s unit_tests successfully once (creates build/.template-migrated.db)
Run nox -s unit_tests again — fails with the above error
Workaround: rm -rf build/.template-migrated.db before running nox

Error output:

nox > python scripts/create_template_db.py /app/cleveragents-core/build/.template-migrated.db
nox > Command python scripts/create_template_db.py ... failed with exit code 1:
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) attempt to write a readonly database
[SQL: CREATE INDEX ix_decisions_parent ON decisions (parent_decision_id)]

Expected Behavior

nox -s unit_tests should succeed on repeated runs without requiring manual deletion of the template DB. The create_template_db.py script should handle the case where the existing file cannot be overwritten by:

Verifying the file is writable before attempting to recreate it
Using a temporary file + atomic rename pattern
Or creating the DB in a tmpfs/temp directory instead of build/

Acceptance Criteria

nox -s unit_tests succeeds on repeated runs without manual cleanup
The template DB creation is idempotent on btrfs filesystems
The fix does not break the template DB fast-path for test scenarios

Supporting Information

Affected file: scripts/create_template_db.py
Affected session: nox -s unit_tests
Environment: btrfs filesystem with overlayfs (Docker container on btrfs host)
File permissions: The existing template DB is created as rw-r--r-- (644) owned by devuser
Workaround: rm -rf build/.template-migrated.db && nox -s unit_tests
Related: features/environment.py _ensure_template_db() function also reuses the existing template DB if present

Subtasks

Investigate root cause of btrfs/overlayfs interaction with SQLite file creation
Fix scripts/create_template_db.py to handle readonly-after-unlink scenario
Add filesystem writability check before attempting DB creation
Tests (Behave): Add scenario for template DB creation idempotency
Run nox -s unit_tests twice in succession to verify fix
Run nox (all default sessions), fix any errors

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.

Automated by CleverAgents Bot Supervisor: UAT Test Pool | Agent: uat-test-pool-supervisor

Automated by CleverAgents Bot
Agent: new-issue-creator

## Metadata - **Commit Message**: `fix(tests): handle readonly template DB on btrfs in create_template_db.py` - **Branch**: `fix/nox-template-db-readonly-btrfs` ## Background and Context The `nox -s unit_tests` session calls `scripts/create_template_db.py` to create a pre-migrated SQLite template database at `build/.template-migrated.db`. On btrfs filesystems (used in the CI/UAT environment), the script fails with a `sqlite3.OperationalError: attempt to write a readonly database` error when the template DB file already exists from a previous run. Discovered during UAT testing of the Execution Environment routing feature (UAT Test Pool — 2026-04-14). ## Current Behavior When `build/.template-migrated.db` already exists (from a previous successful `nox` run), the `create_template_db.py` script: 1. Calls `out.unlink()` to delete the existing file 2. Creates a new empty file via SQLAlchemy's `create_engine()` 3. Fails during `Base.metadata.create_all(engine)` with: ``` sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) attempt to write a readonly database [SQL: CREATE INDEX ix_decisions_parent ON decisions (parent_decision_id)] ``` The error occurs on btrfs because newly created files after `unlink()` may have filesystem-level constraints that prevent SQLite from writing to them in certain btrfs configurations (overlayfs + btrfs COW interaction). **Reproduction steps:** 1. Run `nox -s unit_tests` successfully once (creates `build/.template-migrated.db`) 2. Run `nox -s unit_tests` again — fails with the above error 3. Workaround: `rm -rf build/.template-migrated.db` before running nox **Error output:** ``` nox > python scripts/create_template_db.py /app/cleveragents-core/build/.template-migrated.db nox > Command python scripts/create_template_db.py ... failed with exit code 1: sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) attempt to write a readonly database [SQL: CREATE INDEX ix_decisions_parent ON decisions (parent_decision_id)] ``` ## Expected Behavior `nox -s unit_tests` should succeed on repeated runs without requiring manual deletion of the template DB. The `create_template_db.py` script should handle the case where the existing file cannot be overwritten by: 1. Verifying the file is writable before attempting to recreate it 2. Using a temporary file + atomic rename pattern 3. Or creating the DB in a tmpfs/temp directory instead of `build/` ## Acceptance Criteria - [ ] `nox -s unit_tests` succeeds on repeated runs without manual cleanup - [ ] The template DB creation is idempotent on btrfs filesystems - [ ] The fix does not break the template DB fast-path for test scenarios ## Supporting Information - **Affected file:** `scripts/create_template_db.py` - **Affected session:** `nox -s unit_tests` - **Environment:** btrfs filesystem with overlayfs (Docker container on btrfs host) - **File permissions:** The existing template DB is created as `rw-r--r--` (644) owned by `devuser` - **Workaround:** `rm -rf build/.template-migrated.db && nox -s unit_tests` - **Related:** `features/environment.py` `_ensure_template_db()` function also reuses the existing template DB if present ## Subtasks - [ ] Investigate root cause of btrfs/overlayfs interaction with SQLite file creation - [ ] Fix `scripts/create_template_db.py` to handle readonly-after-unlink scenario - [ ] Add filesystem writability check before attempting DB creation - [ ] Tests (Behave): Add scenario for template DB creation idempotency - [ ] Run `nox -s unit_tests` twice in succession to verify fix - [ ] Run `nox` (all default sessions), fix any errors ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. --- **Automated by CleverAgents Bot** Supervisor: UAT Test Pool | Agent: uat-test-pool-supervisor --- **Automated by CleverAgents Bot** Agent: new-issue-creator

HAL9000

2026-04-14 16:34:27 +00:00

closed this issue
added the
State
Duplicate
label

HAL9000 commented

2026-04-14 16:34:27 +00:00

Author

Owner

🔁 Triage: Duplicate [AUTO-OWNR-1]

This is a duplicate of #9372 which covers the same critical CI blocker (sqlite3.OperationalError: attempt to write a readonly database in scripts/create_template_db.py). Please track progress on #9372.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

🔁 **Triage: Duplicate** [AUTO-OWNR-1] This is a duplicate of #9372 which covers the same critical CI blocker (`sqlite3.OperationalError: attempt to write a readonly database` in `scripts/create_template_db.py`). Please track progress on #9372. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor

HAL9000 referenced this issue

2026-04-14 16:40:35 +00:00

[AUTO-PROJ-OWN] Status: Project Owner Pool Supervisor (Cycle 7) #9395

HAL9000 referenced this issue

2026-04-17 10:25:18 +00:00

[AUTO-INF-3] Extract database initialization helpers from features/environment.py into a dedicated features/testing/db_setup.py module #10244