freemo
7a298ede6e
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 16s
CI / quality (pull_request) Successful in 30s
CI / build (pull_request) Successful in 24s
CI / security (pull_request) Successful in 40s
CI / typecheck (pull_request) Successful in 1m0s
CI / integration_tests (pull_request) Successful in 4m14s
CI / unit_tests (pull_request) Successful in 16m25s
CI / docker (pull_request) Successful in 55s
CI / benchmark-regression (pull_request) Successful in 22m55s
CI / coverage (pull_request) Successful in 38m4s
CI / lint (push) Successful in 13s
CI / build (push) Successful in 15s
CI / quality (push) Successful in 28s
CI / typecheck (push) Successful in 31s
CI / benchmark-regression (push) Has been skipped
CI / security (push) Successful in 31s
CI / integration_tests (push) Successful in 3m19s
CI / benchmark-publish (push) Successful in 14m23s
CI / unit_tests (push) Successful in 14m44s
CI / docker (push) Successful in 38s
CI / coverage (push) Successful in 32m22s
Implemented plan-level and project-level locking with configurable timeouts. Added locks table via Alembic migration storing owner_id, resource_type, resource_id, acquired_at, and expires_at. Locks enforced in PlanLifecycleService transitions. Support for re-entrant acquisition, lock renewal, graceful shutdown release, and startup cleanup of expired locks. Added diagnostics check for stale lock reporting. ISSUES CLOSED: #327
102 lines
3.9 KiB
Markdown
102 lines
3.9 KiB
Markdown
# Concurrency Locks
|
|
|
|
CleverAgents uses advisory locking to prevent concurrent modifications
|
|
to plans and projects. The lock mechanism is backed by a SQLite table
|
|
and enforced at the application layer before lifecycle transitions.
|
|
|
|
## Overview
|
|
|
|
The `LockService` provides mutual-exclusion primitives for two resource
|
|
types: **plan** and **project**. Each resource can have at most one
|
|
active lock at any time. Locks are identified by the tuple
|
|
`(resource_type, resource_id)` and scoped to an **owner** (typically
|
|
the session or process identity).
|
|
|
|
## Lock Lifecycle
|
|
|
|
```text
|
|
acquire --> active --> release / expire
|
|
|
|
|
+--> renew (extend TTL)
|
|
```
|
|
|
|
1. **Acquire**: Creates a new lock row or re-acquires for the same owner.
|
|
2. **Renew**: Extends the TTL of an active lock.
|
|
3. **Release**: Explicitly removes the lock.
|
|
4. **Expire**: The lock becomes stale after `expires_at` passes.
|
|
|
|
## Re-entrant Acquisition
|
|
|
|
If the same owner calls `acquire()` on a resource it already holds,
|
|
the TTL is silently extended. No error is raised.
|
|
|
|
## Conflict Handling
|
|
|
|
If a **different** owner attempts to acquire a lock that is still active
|
|
(not expired), a `LockConflictError` is raised with details about the
|
|
current holder. Expired locks from other owners are transparently
|
|
replaced.
|
|
|
|
## TTL Defaults and Renewal Strategy
|
|
|
|
| Constant | Default | Description |
|
|
|---------------------------|---------|--------------------------------|
|
|
| `DEFAULT_LOCK_TTL_SECS` | 300 | Default lock lifetime (5 min) |
|
|
| `MAX_LOCK_TTL_SECS` | 3600 | Maximum allowed TTL (1 hr) |
|
|
| `MIN_LOCK_TTL_SECS` | 5 | Minimum allowed TTL (5 sec) |
|
|
|
|
For long-running phases (e.g. Execute), the caller should periodically
|
|
call `renew()` with the desired TTL before the current lock expires.
|
|
A recommended renewal interval is `ttl / 2` (e.g. every 150 seconds
|
|
for the default 300-second TTL).
|
|
|
|
## Database Schema
|
|
|
|
The `locks` table (migration `m4_001_concurrency_locks`):
|
|
|
|
| Column | Type | Description |
|
|
|-----------------|-------------|-------------------------------|
|
|
| `id` | INTEGER PK | Auto-increment primary key |
|
|
| `owner_id` | VARCHAR(255) | Lock owner identity |
|
|
| `resource_type` | VARCHAR(50) | `plan` or `project` |
|
|
| `resource_id` | VARCHAR(255) | Resource identifier |
|
|
| `acquired_at` | VARCHAR(30) | ISO-8601 acquisition time |
|
|
| `expires_at` | VARCHAR(30) | ISO-8601 expiry time |
|
|
|
|
A unique constraint on `(resource_type, resource_id)` enforces at most
|
|
one lock per resource.
|
|
|
|
## Startup Cleanup
|
|
|
|
On application startup, `cleanup_expired()` is called to purge any
|
|
locks whose `expires_at` has passed. This handles the case where a
|
|
previous process terminated without releasing its locks.
|
|
|
|
## Graceful Shutdown
|
|
|
|
During graceful shutdown, `release_all_for_owner(owner_id)` removes all
|
|
locks held by the current process so that resources become immediately
|
|
available to other instances.
|
|
|
|
## Diagnostics
|
|
|
|
The `agents diagnostics` command includes a **Stale locks** check that
|
|
reports the number of expired-but-not-yet-cleaned locks. A non-zero
|
|
count indicates that the cleanup routine has not run recently and may
|
|
warrant investigation.
|
|
|
|
## Integration Points
|
|
|
|
- **PlanLifecycleService**: Transitions acquire a plan-level lock before
|
|
mutating phase/state and release it after persistence.
|
|
- **Startup**: The application container calls `cleanup_expired()` during
|
|
initialisation.
|
|
|
|
## Error Types
|
|
|
|
| Exception | Condition |
|
|
|---------------------|------------------------------------------------|
|
|
| `LockConflictError` | Another owner holds an active lock |
|
|
| `LockExpiredError` | Attempted to renew an already-expired lock |
|
|
| `ValidationError` | Invalid parameters (empty strings, bad TTL) |
|