freemo
7a298ede6e
Implemented plan-level and project-level locking with configurable timeouts. Added locks table via Alembic migration storing owner_id, resource_type, resource_id, acquired_at, and expires_at. Locks enforced in PlanLifecycleService transitions. Support for re-entrant acquisition, lock renewal, graceful shutdown release, and startup cleanup of expired locks. Added diagnostics check for stale lock reporting. ISSUES CLOSED: #327
3.9 KiB
Concurrency Locks
CleverAgents uses advisory locking to prevent concurrent modifications to plans and projects. The lock mechanism is backed by a SQLite table and enforced at the application layer before lifecycle transitions.
Overview
The LockService provides mutual-exclusion primitives for two resource
types: plan and project. Each resource can have at most one
active lock at any time. Locks are identified by the tuple
(resource_type, resource_id) and scoped to an owner (typically
the session or process identity).
Lock Lifecycle
acquire --> active --> release / expire
|
+--> renew (extend TTL)
- Acquire: Creates a new lock row or re-acquires for the same owner.
- Renew: Extends the TTL of an active lock.
- Release: Explicitly removes the lock.
- Expire: The lock becomes stale after
expires_atpasses.
Re-entrant Acquisition
If the same owner calls acquire() on a resource it already holds,
the TTL is silently extended. No error is raised.
Conflict Handling
If a different owner attempts to acquire a lock that is still active
(not expired), a LockConflictError is raised with details about the
current holder. Expired locks from other owners are transparently
replaced.
TTL Defaults and Renewal Strategy
| Constant | Default | Description |
|---|---|---|
DEFAULT_LOCK_TTL_SECS |
300 | Default lock lifetime (5 min) |
MAX_LOCK_TTL_SECS |
3600 | Maximum allowed TTL (1 hr) |
MIN_LOCK_TTL_SECS |
5 | Minimum allowed TTL (5 sec) |
For long-running phases (e.g. Execute), the caller should periodically
call renew() with the desired TTL before the current lock expires.
A recommended renewal interval is ttl / 2 (e.g. every 150 seconds
for the default 300-second TTL).
Database Schema
The locks table (migration m4_001_concurrency_locks):
| Column | Type | Description |
|---|---|---|
id |
INTEGER PK | Auto-increment primary key |
owner_id |
VARCHAR(255) | Lock owner identity |
resource_type |
VARCHAR(50) | plan or project |
resource_id |
VARCHAR(255) | Resource identifier |
acquired_at |
VARCHAR(30) | ISO-8601 acquisition time |
expires_at |
VARCHAR(30) | ISO-8601 expiry time |
A unique constraint on (resource_type, resource_id) enforces at most
one lock per resource.
Startup Cleanup
On application startup, cleanup_expired() is called to purge any
locks whose expires_at has passed. This handles the case where a
previous process terminated without releasing its locks.
Graceful Shutdown
During graceful shutdown, release_all_for_owner(owner_id) removes all
locks held by the current process so that resources become immediately
available to other instances.
Diagnostics
The agents diagnostics command includes a Stale locks check that
reports the number of expired-but-not-yet-cleaned locks. A non-zero
count indicates that the cleanup routine has not run recently and may
warrant investigation.
Integration Points
- PlanLifecycleService: Transitions acquire a plan-level lock before mutating phase/state and release it after persistence.
- Startup: The application container calls
cleanup_expired()during initialisation.
Error Types
| Exception | Condition |
|---|---|
LockConflictError |
Another owner holds an active lock |
LockExpiredError |
Attempted to renew an already-expired lock |
ValidationError |
Invalid parameters (empty strings, bad TTL) |