fix(actor): move namespace filter inside lock in ActorLoader.list_actors
CI / helm (pull_request) Successful in 53s
CI / push-validation (pull_request) Successful in 53s
CI / build (pull_request) Successful in 1m30s
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 1m48s
CI / quality (pull_request) Successful in 2m3s
CI / typecheck (pull_request) Successful in 2m17s
CI / security (pull_request) Successful in 2m25s
CI / benchmark-regression (pull_request) Failing after 1m6s
CI / unit_tests (pull_request) Failing after 6m5s
CI / docker (pull_request) Has been skipped
CI / coverage (pull_request) Has been skipped
CI / e2e_tests (pull_request) Successful in 6m9s
CI / integration_tests (pull_request) Failing after 6m13s
CI / status-check (pull_request) Failing after 4s
CI / helm (pull_request) Successful in 53s
CI / push-validation (pull_request) Successful in 53s
CI / build (pull_request) Successful in 1m30s
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 1m48s
CI / quality (pull_request) Successful in 2m3s
CI / typecheck (pull_request) Successful in 2m17s
CI / security (pull_request) Successful in 2m25s
CI / benchmark-regression (pull_request) Failing after 1m6s
CI / unit_tests (pull_request) Failing after 6m5s
CI / docker (pull_request) Has been skipped
CI / coverage (pull_request) Has been skipped
CI / e2e_tests (pull_request) Successful in 6m9s
CI / integration_tests (pull_request) Failing after 6m13s
CI / status-check (pull_request) Failing after 4s
Fixes a race condition (TOCTOU vulnerability) in ActorLoader.list_actors() where the namespace filter was applied outside the self._lock block. Another thread could modify the actors dictionary between the filtering operation and the return, causing inconsistent results or RuntimeError: dictionary changed size during iteration. The conditional block that filters actors by namespace is now executed inside the lock, ensuring atomic access to the actors dictionary. A concurrency test using threading.Barrier verifies thread safety under concurrent access patterns. Closes #8588
This commit is contained in:
@@ -18,6 +18,15 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
(`features/actor_subgraph_cycle_detection.feature`) and a Robot Framework
|
||||
integration test (`robot/actor_compiler.robot`) to prevent regressions.
|
||||
|
||||
- **ActorLoader.list_actors TOCTOU race condition** (#8588): Moved the namespace
|
||||
filter inside the ``with self._lock:`` block so that the actors dictionary is read
|
||||
and filtered atomically. Previously an actor ``.clear()`` could run between the
|
||||
dictionary read and the filter step, causing the loader to return stale results or
|
||||
raise ``RuntimeError: dictionary changed size during iteration``. Added a
|
||||
``@unit @actor @concurrency`` Behave scenario in ``features/actor_loading.feature``
|
||||
that exercises ``list_actors(namespace=...)`` and ``clear()`` on multiple threads
|
||||
via ``threading.Barrier``.
|
||||
|
||||
### Changed
|
||||
|
||||
- **`agents session list` now displays full 26-character session ULIDs** (#10970): The Rich table
|
||||
|
||||
@@ -253,3 +253,19 @@ Feature: Actor registry and loader
|
||||
And I run discovery
|
||||
Then the loader should contain actor "assistants/simple"
|
||||
And the loader should have 0 warnings
|
||||
|
||||
# ────────────────────────────────────────────────────────
|
||||
# Concurrency and race conditions
|
||||
# ────────────────────────────────────────────────────────
|
||||
|
||||
@unit @actor @concurrency
|
||||
Scenario: list_actors with namespace filter is thread-safe under concurrent clear/discover
|
||||
Given a temporary actors directory with files
|
||||
| filename | name | type | model |
|
||||
| rev.yaml | assistants/reviewer | llm | gpt-4 |
|
||||
| fmt.yaml | utilities/formatter | llm | gpt-4 |
|
||||
| ana.yaml | assistants/analyzer | llm | gpt-4 |
|
||||
When I create an actor loader with that directory as search root
|
||||
And I run discovery
|
||||
And 5 threads concurrently call list_actors with namespace "assistants" while another thread calls clear
|
||||
Then all concurrent list_actors calls should return consistent results
|
||||
|
||||
@@ -3,6 +3,8 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
from behave import given, then, when
|
||||
@@ -386,3 +388,82 @@ def step_given_list_yaml(context: Context, filename: str) -> None:
|
||||
d = _fresh_dir(context, "listyaml")
|
||||
context._actor_dir = d
|
||||
(d / filename).write_text("- item_one\n- item_two\n")
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────
|
||||
# Concurrency test steps
|
||||
# ──────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@when(
|
||||
"{n:d} threads concurrently call list_actors with namespace "
|
||||
'"{namespace}" while another thread calls clear'
|
||||
)
|
||||
def step_when_concurrent_list_actors_with_clear(
|
||||
context: Context, n: int, namespace: str
|
||||
) -> None:
|
||||
"""Test race condition: list_actors with namespace filter under concurrent clear."""
|
||||
|
||||
loader: ActorLoader = context._loader
|
||||
results: list[list[ActorConfigSchema]] = []
|
||||
errors: list[Exception] = []
|
||||
barrier = threading.Barrier(n + 1) # n list_actors threads + 1 clear thread
|
||||
|
||||
def list_actors_worker() -> None:
|
||||
try:
|
||||
barrier.wait() # All threads start simultaneously
|
||||
result = loader.list_actors(namespace=namespace)
|
||||
results.append(result)
|
||||
except Exception as exc:
|
||||
errors.append(exc)
|
||||
|
||||
def clear_worker() -> None:
|
||||
try:
|
||||
barrier.wait() # All threads start simultaneously
|
||||
# Small delay to let list_actors threads acquire the lock first
|
||||
|
||||
time.sleep(0.001)
|
||||
loader.clear()
|
||||
except Exception as exc:
|
||||
errors.append(exc)
|
||||
|
||||
# Create n threads for list_actors calls
|
||||
list_threads = [threading.Thread(target=list_actors_worker) for _ in range(n)]
|
||||
# Create 1 thread for clear
|
||||
clear_thread = threading.Thread(target=clear_worker)
|
||||
|
||||
# Start all threads
|
||||
for t in list_threads:
|
||||
t.start()
|
||||
clear_thread.start()
|
||||
|
||||
# Wait for all threads to complete
|
||||
for t in list_threads:
|
||||
t.join()
|
||||
clear_thread.join()
|
||||
|
||||
assert not errors, f"Threads raised exceptions: {errors}"
|
||||
|
||||
# Store results and namespace for assertion
|
||||
context._concurrent_list_results = results
|
||||
context._concurrent_namespace = namespace
|
||||
|
||||
|
||||
@then("all concurrent list_actors calls should return consistent results")
|
||||
def step_then_concurrent_results_consistent(context: Context) -> None:
|
||||
"""Verify that all concurrent list_actors calls returned consistent results."""
|
||||
results: list[list[ActorConfigSchema]] = context._concurrent_list_results
|
||||
namespace: str = context._concurrent_namespace
|
||||
|
||||
# Each result should only contain actors from the requested namespace.
|
||||
# When clear() runs concurrently, some threads may see the full list while
|
||||
# others see an empty list, but no result should have stale/partial data.
|
||||
if not results:
|
||||
return
|
||||
|
||||
# Check that each result only contains actors from the requested namespace
|
||||
for i, result in enumerate(results):
|
||||
for actor_config in result:
|
||||
assert actor_config.name.startswith(f"{namespace}/"), (
|
||||
f"Result {i} contains actor from wrong namespace: {actor_config.name}"
|
||||
)
|
||||
|
||||
@@ -241,10 +241,9 @@ class ActorLoader:
|
||||
"""
|
||||
with self._lock:
|
||||
configs = [e.config for e in self._actors.values()]
|
||||
|
||||
if namespace is not None:
|
||||
prefix = f"{namespace}/"
|
||||
configs = [c for c in configs if c.name.startswith(prefix)]
|
||||
if namespace is not None:
|
||||
prefix = f"{namespace}/"
|
||||
configs = [c for c in configs if c.name.startswith(prefix)]
|
||||
|
||||
return configs
|
||||
|
||||
|
||||
Reference in New Issue
Block a user