cleveragents/cleveragents-core

Fork 3

BUG-HUNT: [resource] RepoIndexingService._resource_locks grows unboundedly — per-resource locks never evicted except on remove_index #7445

New issue

Open

opened 2026-04-10 19:37:58 +00:00 by HAL9000 · 3 comments

HAL9000 commented

2026-04-10 19:37:58 +00:00

Owner

Bug Report: Resource Management — RepoIndexingService Lock Dictionary Memory Leak

Severity Assessment

Impact: The _resource_locks dictionary in RepoIndexingService grows one entry per unique resource_id ever indexed, and is only cleaned up when remove_index() is explicitly called. In environments with many resources (e.g., many projects/repositories indexed over the service's lifetime), this accumulates threading.RLock objects in memory indefinitely. Each RLock is a lightweight OS primitive, but the dictionary itself can become very large.
Likelihood: Medium — affects long-running server processes with many resources
Priority: Medium

Location

File: src/cleveragents/application/services/repo_indexing_service.py
Function: RepoIndexingService._resource_lock()
Lines: 73–76

Description

def _resource_lock(self, resource_id: str) -> threading.RLock:
    """Return a per-*resource_id* reentrant lock."""
    with self._locks_guard:
        return self._resource_locks.setdefault(resource_id, threading.RLock())

Each time a new resource_id is encountered, a new threading.RLock() is added to _resource_locks. This dictionary is only cleaned up in one place:

def remove_index(self, resource_id: str) -> bool:
    # ...
    with self._locks_guard:
        self._resource_locks.pop(resource_id, None)  # ← only cleaned here

If resources are indexed but never explicitly removed (the common case — resources persist and are just re-indexed over time), the lock dictionary grows indefinitely. For a server running for years indexing thousands of repositories, this becomes a significant memory accumulation of threading.RLock objects.

Additionally, the cleanup_stale_indexing() called in __init__ does NOT clean up corresponding lock entries for stale resources, creating orphaned lock entries.

Evidence

# src/cleveragents/application/services/repo_indexing_service.py, lines 73-76
def _resource_lock(self, resource_id: str) -> threading.RLock:
    with self._locks_guard:
        return self._resource_locks.setdefault(resource_id, threading.RLock())
        # ↑ Never removed unless remove_index() is called

Expected Behavior

The lock dictionary should be bounded in size. Locks for resources that haven't been accessed recently should be eligible for eviction.

Actual Behavior

_resource_locks grows indefinitely, one entry per unique resource_id ever indexed.

Suggested Fix

Use a weakref dictionary or LRU cache for the locks:

import weakref
self._resource_locks: weakref.WeakValueDictionary[str, threading.RLock] = weakref.WeakValueDictionary()

This way, locks that are no longer held by any caller are automatically garbage collected. However, this requires ensuring callers hold a reference to the lock while using it (which the with self._resource_lock(resource_id): pattern doesn't do directly).

A safer alternative: use a bounded LRU cache for locks:

from functools import lru_cache
# Or implement a simple fixed-size LRU lock cache

Or at minimum, add a periodic cleanup that removes locks for resources not accessed in the last N hours.

TDD Note

After this bug issue is verified, a corresponding Type/Testing issue will be created for TDD with tags: @tdd_issue, @tdd_issue_, @tdd_expected_fail.

Automated by CleverAgents Bot
Supervisor: Bug Detection Pool | Agent: bug-hunt-pool-supervisor

## Bug Report: Resource Management — RepoIndexingService Lock Dictionary Memory Leak ### Severity Assessment - **Impact**: The `_resource_locks` dictionary in `RepoIndexingService` grows one entry per unique `resource_id` ever indexed, and is only cleaned up when `remove_index()` is explicitly called. In environments with many resources (e.g., many projects/repositories indexed over the service's lifetime), this accumulates `threading.RLock` objects in memory indefinitely. Each `RLock` is a lightweight OS primitive, but the dictionary itself can become very large. - **Likelihood**: Medium — affects long-running server processes with many resources - **Priority**: Medium ### Location - **File**: `src/cleveragents/application/services/repo_indexing_service.py` - **Function**: `RepoIndexingService._resource_lock()` - **Lines**: 73–76 ### Description ```python def _resource_lock(self, resource_id: str) -> threading.RLock: """Return a per-*resource_id* reentrant lock.""" with self._locks_guard: return self._resource_locks.setdefault(resource_id, threading.RLock()) ``` Each time a new `resource_id` is encountered, a new `threading.RLock()` is added to `_resource_locks`. This dictionary is only cleaned up in one place: ```python def remove_index(self, resource_id: str) -> bool: # ... with self._locks_guard: self._resource_locks.pop(resource_id, None) # ← only cleaned here ``` If resources are indexed but never explicitly removed (the common case — resources persist and are just re-indexed over time), the lock dictionary grows indefinitely. For a server running for years indexing thousands of repositories, this becomes a significant memory accumulation of `threading.RLock` objects. Additionally, the `cleanup_stale_indexing()` called in `__init__` does NOT clean up corresponding lock entries for stale resources, creating orphaned lock entries. ### Evidence ```python # src/cleveragents/application/services/repo_indexing_service.py, lines 73-76 def _resource_lock(self, resource_id: str) -> threading.RLock: with self._locks_guard: return self._resource_locks.setdefault(resource_id, threading.RLock()) # ↑ Never removed unless remove_index() is called ``` ### Expected Behavior The lock dictionary should be bounded in size. Locks for resources that haven't been accessed recently should be eligible for eviction. ### Actual Behavior `_resource_locks` grows indefinitely, one entry per unique resource_id ever indexed. ### Suggested Fix Use a `weakref` dictionary or LRU cache for the locks: ```python import weakref self._resource_locks: weakref.WeakValueDictionary[str, threading.RLock] = weakref.WeakValueDictionary() ``` This way, locks that are no longer held by any caller are automatically garbage collected. However, this requires ensuring callers hold a reference to the lock while using it (which the `with self._resource_lock(resource_id):` pattern doesn't do directly). A safer alternative: use a bounded LRU cache for locks: ```python from functools import lru_cache # Or implement a simple fixed-size LRU lock cache ``` Or at minimum, add a periodic cleanup that removes locks for resources not accessed in the last N hours. ### Category resource ### TDD Note After this bug issue is verified, a corresponding Type/Testing issue will be created for TDD with tags: @tdd_issue, @tdd_issue_<this-issue-number>, @tdd_expected_fail. --- **Automated by CleverAgents Bot** Supervisor: Bug Detection Pool | Agent: bug-hunt-pool-supervisor

HAL9000 referenced this issue

2026-04-10 19:38:52 +00:00

[AUTO-BUG-SUP] Bug Hunt Status (Cycle 3) #7449

HAL9000 added the

labels

2026-04-10 20:33:15 +00:00

HAL9000 referenced this issue

2026-04-10 20:39:58 +00:00

[AUTO-BUG-SUP] Bug Hunt Status (Cycle 4) #7470

HAL9000 referenced this issue

2026-04-12 03:16:59 +00:00

[AUTO-BUG-POOL] Bug Detection Report (Cycle 1) #7709

freemo added the

Type

Bug

label

2026-04-12 18:34:09 +00:00

HAL9000 added

and removed

labels

2026-04-14 06:24:03 +00:00

HAL9000 commented

2026-04-14 06:24:03 +00:00

Author

Owner

✅ Verified — Resource leak: RepoIndexingService._resource_locks grows unboundedly. MoSCoW: Should-have. Priority: Medium.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

✅ **Verified** — Resource leak: RepoIndexingService._resource_locks grows unboundedly. MoSCoW: Should-have. Priority: Medium. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor

HAL9000 commented

2026-04-14 06:29:40 +00:00

Author

Owner

✅ Verified — Resource leak: RepoIndexingService._resource_locks grows unboundedly. MoSCoW: Should-have. Priority: Medium.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor