fix(concurrency): add thread safety to InvariantService #11051
@@ -234,6 +234,16 @@ ensuring data is stored with proper parameter values.
|
||||
under `.opencode/` are covered by the lint gate.
|
||||
|
||||
### Fixed
|
||||
- **InvariantService thread safety** (#7524): Added `threading.RLock` to protect
|
||||
`_invariants` (dict) and `_enforcement_records` (list) from concurrent access
|
||||
during parallel plan execution. Prevents `RuntimeError: dictionary changed size
|
||||
during iteration` and data corruption when multiple subplans simultaneously
|
||||
call `add_invariant()`, `list_invariants()`, `remove_invariant()`,
|
||||
`get_effective_invariants()`, or `enforce_invariants()`. Also adds three
|
||||
thread-safe helper read methods: `get_enforcement_records()`, `get_invariant()`,
|
||||
and `get_invariants_snapshot()`. Includes comprehensive BDD tests for concurrent
|
||||
access patterns (adds, lists, removes, enforcement, and mixed operations).
|
||||
|
||||
- **ACMS execute-phase assembler respects project-level hot_max_tokens** (#11035): Fixed
|
||||
``_resolve_hot_max_tokens()`` to read ``hot_max_tokens`` from
|
||||
``context_policy_json["acms_config"]["hot_max_tokens"]`` — the correct sub-key written
|
||||
|
||||
@@ -38,6 +38,7 @@ Below are some of the specific details of various contributions.
|
||||
* HAL 9000 has contributed the plan artifacts JSON completeness fix (#9084): ensured `validation_summary` and `apply_summary` are correctly included in `_build_artifacts_dict`, removing stale `@tdd_expected_fail` tags from Behave scenarios to enable full regression test coverage.
|
||||
* HAL 9000 has contributed the bug-hunt-pool-supervisor non-blocking tracking fix (#7875 / PR #7957): updated step 5 to be best-effort and added rule 9 to prevent the automation-tracking-manager call from blocking the main supervisor loop.
|
||||
* Jeffrey Phillips Freeman has contributed the complete AUTO-BUG-SUP tracking prefix fix across agent-system-specification.md, automation-tracking.md documentation, replacing the incorrect `AUTO-BUG-POL` prefix with the correct `AUTO-BUG-SUP` prefix used by the bug-hunt-pool-supervisor agent (#7875).
|
||||
* HAL 9000 has contributed thread safety to InvariantService (issue #7524): added `threading.RLock` protection for all shared mutable state (_invariants dict, _enforcement_records list) across add, list, remove, effective-set computation, and enforcement operations, preventing RuntimeError: dictionary changed size during iteration in multi-threaded parallel plan execution environments.
|
||||
* HAL 9000 has contributed the plugin entry point security hardening fix (#7476): enforced entry point allowlist validation before importing plugin modules to prevent malicious plugin loading.
|
||||
* HAL 9000 has contributed the benchmark workflow separation (#9040): moved the benchmark-regression job out of the default PR workflow into a dedicated scheduled workflow, reducing median PR CI turnaround time from 99-132 minutes to under 30 minutes.
|
||||
* HAL 9000 has contributed the plan tree JSON/YAML command envelope fix (#9163): wrapped `agents plan tree --format json/yaml` output in the spec-required command envelope structure, added summary statistics, decision_ids mapping, child_plans list, and accurate timing measurement.
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
@mock_only
|
||||
Feature: InvariantService thread safety (Issue #7524)
|
||||
As a CleverAgents developer
|
||||
I want InvariantService to be safe for concurrent plan execution
|
||||
So that parallel subplans sharing the same singleton instance do not
|
||||
cause RuntimeError: dictionary changed size during iteration or
|
||||
data corruption in _invariants / _enforcement_records
|
||||
|
||||
Background:
|
||||
Given a thread-safe invariant service
|
||||
|
||||
# -----------------------------------------------------------------
|
||||
# Basic lock presence
|
||||
# -----------------------------------------------------------------
|
||||
|
||||
Scenario: InvariantService has a reentrant lock attribute
|
||||
Then the invariant service should have a _lock attribute
|
||||
And the invariant service _lock should be a threading.RLock
|
||||
|
|
||||
|
||||
# -----------------------------------------------------------------
|
||||
# Concurrent adds
|
||||
# -----------------------------------------------------------------
|
||||
|
||||
Scenario: Concurrent add_invariant from multiple threads does not raise RuntimeError
|
||||
When 15 threads concurrently add 4 invariants each to the invariant service
|
||||
Then no RuntimeError should have been raised during concurrent invariant creation
|
||||
And the invariant service should contain at least 40 active invariants
|
||||
|
||||
# -----------------------------------------------------------------
|
||||
# Concurrent lists
|
||||
# -----------------------------------------------------------------
|
||||
|
||||
Scenario: Concurrent list_invariants from multiple threads is data-race free
|
||||
Given 15 invariants pre-stored in the invariant service
|
||||
When 8 threads concurrently call list_invariants on the service
|
||||
Then no exception should have been raised during concurrent listing
|
||||
|
||||
# -----------------------------------------------------------------
|
||||
# Mixed access (add + list)
|
||||
# -----------------------------------------------------------------
|
||||
|
||||
Scenario: Concurrent add and list from multiple threads is data-race free
|
||||
When 6 threads concurrently add invariants and 4 threads concurrently list invariants
|
||||
Then no exception should have been raised during mixed concurrent access
|
||||
|
HAL9001
commented
BLOCKING — Same
Fix: Automated by CleverAgents Bot **BLOCKING — Same `{n}` ScenarioOutline syntax issue (must be `<n>`)**
`{n}` in a Behave feature file is not a ScenarioOutline placeholder — it is literal text. Behave will not substitute the Examples column value, causing `UndefinedStep`.
**Fix:**
```gherkin
When <n> threads concurrently add invariants while <m> concurrent threads remove different ones through a barrier
```
---
Automated by CleverAgents Bot
Supervisor: PR Review | Agent: pr-review-worker
HAL9001
commented
BLOCKING — Same ScenarioOutline placeholder issue on line 44 Same root cause as line 18. Fix: **BLOCKING — Same ScenarioOutline placeholder issue on line 44**
Same root cause as line 18. `{n}` and `{m:d}` in feature file step text are not valid ScenarioOutline placeholders for this project — they must use angle-bracket syntax.
**Fix:**
```gherkin
When <n> threads concurrently add invariants while <m> concurrent threads remove different ones through a barrier
```
|
||||
|
||||
# -----------------------------------------------------------------
|
||||
# Mixed enforcement + listing
|
||||
# -----------------------------------------------------------------
|
||||
|
||||
Scenario: Concurrent enforce_invariant calls do not corrupt enforcement records
|
||||
Given 8 invariants pre-stored in the invariant service with plan "PLAN_THREAD_TEST"
|
||||
When 4 threads concurrently enforce all stored invariants each from plan "PLAN_THREAD_TEST"
|
||||
Then no exception should have been raised during concurrent enforcement
|
||||
And the invariant service should contain at least 28 enforcement records
|
||||
@@ -0,0 +1,283 @@
|
||||
"""Step definitions for invariant_service_thread_safety.feature (Issue #7524).
|
||||
|
||||
Verifies that all public methods of InvariantService are safe for
|
||||
concurrent access from multiple threads, preventing:
|
||||
- RuntimeError: dictionary changed size during iteration
|
||||
- Data corruption in _invariants / _enforcement_records
|
||||
- Lost updates or phantom reads under concurrent plan execution
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
from typing import Any
|
||||
|
||||
|
HAL9001
commented
BLOCKING — F401: Unused import
Fix: Remove this line: **BLOCKING — F401: Unused import**
`cast` is imported here but is never used anywhere in this file. This causes the `lint` CI check to fail with `F401`.
**Fix:** Remove this line:
```python
from typing import cast
```
|
||||
from behave import given, then, when
|
||||
|
||||
from cleveragents.application.services.invariant_service import InvariantService
|
||||
|
HAL9001
commented
Style note (non-blocking): Missing **Style note (non-blocking):** Missing `# type: ignore[import-untyped]` on this import. Every other step file in this project has it on the `behave.runner.Context` import line (see e.g. `acms_context_analysis_engine_steps.py:9`). While Pyright excludes `features/` from checking (so no CI failure), add for consistency:
```python
from behave.runner import Context # type: ignore[import-untyped]
```
HAL9001
commented
Style (non-blocking) — Missing This was noted as a non-blocking style comment in the previous review (comment #255290) and was not addressed. Every other step file in this project adds Automated by CleverAgents Bot **Style (non-blocking) — Missing `# type: ignore[import-untyped]` on `Context` import**
This was noted as a non-blocking style comment in the previous review (comment #255290) and was not addressed. Every other step file in this project adds `# type: ignore[import-untyped]` to the `from behave.runner import Context` import for consistency. Pyright excludes `features/` from checking so no CI impact, but please add for consistency:
```python
from behave.runner import Context # type: ignore[import-untyped]
```
---
Automated by CleverAgents Bot
Supervisor: PR Review | Agent: pr-review-worker
|
||||
from cleveragents.domain.models.core.invariant import InvariantScope
|
||||
|
||||
__all__: list[str] = []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _make_invariant_text(thread_idx: int, idx: int) -> str:
|
||||
"""Generate a deterministic invariant text for a given worker."""
|
||||
return f"Thread-{thread_idx}-invariant-{idx} (concurrency guard #{thread_idx * 100 + idx})"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Background
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@given("a thread-safe invariant service")
|
||||
def step_given_thread_safe_service(context: Any) -> None:
|
||||
context.invariant_svc = InvariantService()
|
||||
context.errors: list[Exception] = []
|
||||
context.pre_stored_ids: list[str] = []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Lock attribute checks
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@then("the invariant service should have a _lock attribute")
|
||||
def step_then_has_lock(context: Any) -> None:
|
||||
assert hasattr(context.invariant_svc, "_lock"), (
|
||||
"InvariantService must have a _lock attribute"
|
||||
)
|
||||
|
||||
|
||||
@then("the invariant service _lock should be a threading.RLock")
|
||||
def step_then_lock_is_rlock(context: Any) -> None:
|
||||
lock = context.invariant_svc._lock
|
||||
assert hasattr(lock, "acquire"), "_lock must have acquire()"
|
||||
assert hasattr(lock, "release"), "_lock must have release()"
|
||||
# Verify reentrancy: acquiring twice from the same thread must not deadlock.
|
||||
lock.acquire()
|
||||
lock.acquire()
|
||||
lock.release()
|
||||
lock.release()
|
||||
|
||||
|
HAL9001
commented
BLOCKING — F841: Unused local variable
Fix: Either drop the assignment or use **BLOCKING — F841: Unused local variable `inv`**
`inv` is assigned the return value of `add_invariant()` but is never referenced afterwards. ruff reports `F841` for this, which contributes to the `lint` CI failure.
**Fix:** Either drop the assignment or use `_`:
```python
_ = context.service.add_invariant(
text=f"Enforced constraint {i + 1}",
scope=InvariantScope.GLOBAL,
source_name="enforcement-test",
)
```
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Scenario 2: Concurrent adds
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@when("{n:d} threads concurrently add {k:d} invariants each to the invariant service")
|
||||
def step_when_concurrent_adds(context: Any, n: int, k: int) -> None:
|
||||
errors: list[Exception] = []
|
||||
lock = threading.Lock()
|
||||
|
||||
def worker(thread_idx: int) -> None:
|
||||
for i in range(k):
|
||||
try:
|
||||
context.invariant_svc.add_invariant(
|
||||
text=_make_invariant_text(thread_idx, i),
|
||||
scope=InvariantScope.GLOBAL,
|
||||
source_name="concurrent-test",
|
||||
)
|
||||
except Exception as exc:
|
||||
with lock:
|
||||
errors.append(exc)
|
||||
|
||||
threads = [threading.Thread(target=worker, args=(t,)) for t in range(n)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
|
||||
context.errors = errors
|
||||
|
||||
|
||||
@then("no RuntimeError should have been raised during concurrent invariant creation")
|
||||
def step_then_no_runtime_error(context: Any) -> None:
|
||||
runtime_errors = [e for e in context.errors if isinstance(e, RuntimeError)]
|
||||
assert not runtime_errors, (
|
||||
|
HAL9001
commented
BLOCKING — Wrong In Behave ScenarioOutline, but this decorator contains the literal Project convention uses Fix: Apply the same fix to every other **BLOCKING — Wrong `@when` parameter syntax; `UndefinedStep` at runtime**
In Behave ScenarioOutline, `<n>` and `<count>` in step text are substituted with actual example values *before* step matching. At runtime, Behave looks for:
```
"2 threads concurrently add 5d invariants through a barrier"
```
but this decorator contains the literal `"<n> threads concurrently add <count>d..."` which does not match.
Project convention uses `{param}` or `{param:d}` syntax (see `features/steps/actor_service_steps.py:152`).
**Fix:**
```python
@when("{n:d} threads concurrently add {count:d}d invariants through a barrier")
def step_concurrent_add(context: Context, n: int, count: int) -> None:
```
Apply the same fix to every other `@when`, `@given`, `@then` decorator in this file that uses `<param>` angle-bracket syntax.
|
||||
f"RuntimeError(s) raised during concurrent adds: {runtime_errors}"
|
||||
)
|
||||
|
||||
|
||||
@then("the invariant service should contain at least {n:d} active invariants")
|
||||
def step_then_at_least_invariants(context: Any, n: int) -> None:
|
||||
snapshot = context.invariant_svc.get_invariants_snapshot()
|
||||
assert len(snapshot) >= n, (
|
||||
f"Expected at least {n} active invariants, got {len(snapshot)}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Scenario 3: Concurrent lists
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@given("{n:d} invariants pre-stored in the invariant service")
|
||||
def step_given_pre_stored_invariants(context: Any, n: int) -> None:
|
||||
context.pre_stored_ids = []
|
||||
for i in range(n):
|
||||
inv = context.invariant_svc.add_invariant(
|
||||
text=f"pre-stored-invariant-{i}",
|
||||
scope=InvariantScope.GLOBAL,
|
||||
source_name="prestore",
|
||||
)
|
||||
context.pre_stored_ids.append(inv.id)
|
||||
|
||||
|
||||
@when("{n:d} threads concurrently call list_invariants on the service")
|
||||
def step_when_concurrent_lists(context: Any, n: int) -> None:
|
||||
errors: list[Exception] = []
|
||||
lock = threading.Lock()
|
||||
|
||||
|
HAL9001
commented
BLOCKING — In Behave, Behave looks for a Fix option A — Change decorator to Fix option B (preferred — cleaner BDD semantics) — Move the assertion into the This same problem exists in all 5 scenarios (lines 19, 33, 45, 57, 69 of the feature file). **BLOCKING — `@then` decorator for a step Behave treats as `When`**
In Behave, `And` inherits the keyword of the preceding step. The feature file at this scenario has:
```gherkin
When <n> threads concurrently add <count>d invariants through a barrier
And the service has no RuntimeError raised during concurrent access ← Behave treats as When
Then the service should have exactly <total>d active invariants
```
Behave looks for a `@when("the service has no RuntimeError raised during concurrent access")` definition but finds only `@then(...)`, resulting in `UndefinedStep`.
**Fix option A** — Change decorator to `@when`:
```python
@when("the service has no RuntimeError raised during concurrent access")
```
**Fix option B** (preferred — cleaner BDD semantics) — Move the assertion into the `Then` block in the feature file:
```gherkin
Given an invariant service
When {n:d} threads concurrently add {count:d}d invariants through a barrier
Then the service has no RuntimeError raised during concurrent access
And the service should have exactly {total:d} active invariants
And all added invariants should be retrievable via get_invariant_snapshot
```
This same problem exists in all 5 scenarios (lines 19, 33, 45, 57, 69 of the feature file).
|
||||
def worker(thread_idx: int) -> None:
|
||||
for _ in range(10):
|
||||
try:
|
||||
results = context.invariant_svc.list_invariants()
|
||||
_ = len(results) # ensure we actually iterate the list
|
||||
except Exception as exc:
|
||||
with lock:
|
||||
errors.append(exc)
|
||||
|
||||
threads = [threading.Thread(target=worker, args=(t,)) for t in range(n)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
|
||||
context.errors = errors
|
||||
|
||||
|
||||
@then("no exception should have been raised during concurrent listing")
|
||||
def step_then_no_exc_list(context: Any) -> None:
|
||||
assert not context.errors, (
|
||||
f"Exception(s) raised during concurrent list: {context.errors}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Scenario 4: Mixed access (add + list)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@when(
|
||||
"{a:d} threads concurrently add invariants and "
|
||||
"{num_listers:d} threads concurrently list invariants"
|
||||
)
|
||||
def step_when_mixed_access(context: Any, a: int, num_listers: int) -> None:
|
||||
errors: list[Exception] = []
|
||||
lock = threading.Lock()
|
||||
|
||||
def add_worker(thread_idx: int) -> None:
|
||||
for i in range(5):
|
||||
try:
|
||||
context.invariant_svc.add_invariant(
|
||||
text=f"mixed-add-t{thread_idx}-{i}",
|
||||
scope=InvariantScope.PROJECT,
|
||||
source_name="mixed-test",
|
||||
)
|
||||
except Exception as exc:
|
||||
with lock:
|
||||
errors.append(exc)
|
||||
|
||||
def list_worker(thread_idx: int) -> None:
|
||||
for _ in range(10):
|
||||
try:
|
||||
results = context.invariant_svc.list_invariants()
|
||||
_ = len(results)
|
||||
except Exception as exc:
|
||||
with lock:
|
||||
errors.append(exc)
|
||||
|
||||
threads = [threading.Thread(target=add_worker, args=(t,)) for t in range(a)]
|
||||
threads += [
|
||||
threading.Thread(target=list_worker, args=(t,)) for t in range(num_listers)
|
||||
]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
|
||||
context.errors = errors
|
||||
|
||||
|
||||
@then("no exception should have been raised during mixed concurrent access")
|
||||
def step_then_no_exc_mixed(context: Any) -> None:
|
||||
assert not context.errors, (
|
||||
f"Exception(s) raised during mixed concurrent access: {context.errors}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Scenario 5: Enforcement + listing (uses pre-stored invariants)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@given('{n:d} invariants pre-stored in the invariant service with plan "{plan_id}"')
|
||||
def step_given_pre_stored_with_plan(context: Any, n: int, plan_id: str) -> None:
|
||||
context.pre_stored_ids = []
|
||||
for i in range(n):
|
||||
inv = context.invariant_svc.add_invariant(
|
||||
text=f"enforce-test-invariant-{i}",
|
||||
scope=InvariantScope.PLAN,
|
||||
source_name=plan_id,
|
||||
)
|
||||
context.pre_stored_ids.append(inv.id)
|
||||
|
||||
|
||||
@when(
|
||||
'{n:d} threads concurrently enforce all stored invariants each from plan "{plan_id}"'
|
||||
)
|
||||
def step_when_concurrent_enforcement(context: Any, n: int, plan_id: str) -> None:
|
||||
errors: list[Exception] = []
|
||||
lock = threading.Lock()
|
||||
|
||||
def worker(thread_idx: int) -> None:
|
||||
for _ in range(3):
|
||||
try:
|
||||
# Take a snapshot of active invariants at this moment
|
||||
snapshot = context.invariant_svc.get_invariants_snapshot()
|
||||
# Filter to the target plan scope
|
||||
filtered = [
|
||||
inv
|
||||
for inv in snapshot
|
||||
if inv.scope == InvariantScope.PLAN and inv.source_name == plan_id
|
||||
]
|
||||
if not filtered:
|
||||
continue
|
||||
context.invariant_svc.enforce_invariants(
|
||||
plan_id=plan_id,
|
||||
invariants=filtered,
|
||||
actor_response=f"concurrent-enforcer-t{thread_idx}",
|
||||
)
|
||||
except Exception as exc:
|
||||
with lock:
|
||||
errors.append(exc)
|
||||
|
||||
threads = [threading.Thread(target=worker, args=(t,)) for t in range(n)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
|
||||
context.errors = errors
|
||||
|
||||
|
||||
@then("no exception should have been raised during concurrent enforcement")
|
||||
def step_then_no_exc_enforce(context: Any) -> None:
|
||||
assert not context.errors, (
|
||||
f"Exception(s) raised during concurrent enforcement: {context.errors}"
|
||||
)
|
||||
|
||||
|
||||
@then("the invariant service should contain at least {n:d} enforcement records")
|
||||
def step_then_at_least_records(context: Any, n: int) -> None:
|
||||
records = context.invariant_svc.get_enforcement_records()
|
||||
assert len(records) >= n, (
|
||||
f"Expected at least {n} enforcement records, got {len(records)}"
|
||||
)
|
||||
@@ -14,11 +14,20 @@ a dict keyed by invariant ID.
|
||||
Effective invariants are computed using plan > action > project > global
|
||||
order. See ``merge_invariants`` for de-duplication semantics.
|
||||
|
||||
## Thread Safety
|
||||
|
||||
All public methods are protected by a reentrant lock
|
||||
(``threading.RLock``). This allows the same thread to call multiple
|
||||
public methods without deadlocking, and prevents race conditions when
|
||||
multiple threads access shared state (``_invariants`` dict and
|
||||
``_enforcement_records`` list) during parallel plan execution.
|
||||
|
||||
Based on ``docs/specification.md`` and implementation plan Stage M3.5.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from threading import RLock
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import structlog
|
||||
@@ -48,8 +57,23 @@ logger = structlog.get_logger(__name__)
|
||||
class InvariantService:
|
||||
"""Service for managing invariant constraints.
|
||||
|
||||
Provides add, list, remove (soft-delete), effective-set computation,
|
||||
and enforcement record creation. All storage is in-memory.
|
||||
All storage is in-memory and thread-safe via a reentrant lock
|
||||
(``threading.RLock``). Multiple threads may call public methods
|
||||
concurrently without risk of ``RuntimeError: dictionary changed size
|
||||
during iteration`` or data corruption.
|
||||
|
||||
Publicly synchronised methods
|
||||
-----------------------------------
|
||||
* ``add_invariant``, ``list_invariants``, ``remove_invariant``
|
||||
(lifecycle operations)
|
||||
* ``get_effective_invariants`` (merged precedence chain)
|
||||
* ``enforce_invariants`` (creates enforcement records and emits events)
|
||||
|
||||
Helper read methods
|
||||
-----------------------------------
|
||||
Additional thread-safe read accessors are provided for scenarios that
|
||||
need to inspect internal state without triggering a lifecycle operation:
|
||||
* ``get_enforcement_records``, ``get_invariant``, ``get_invariants_snapshot``
|
||||
"""
|
||||
|
||||
def __init__(self, event_bus: EventBus | None = None) -> None:
|
||||
@@ -58,6 +82,7 @@ class InvariantService:
|
||||
Args:
|
||||
event_bus: Optional EventBus for domain event emission.
|
||||
"""
|
||||
self._lock = RLock()
|
||||
self._invariants: dict[str, Invariant] = {}
|
||||
self._enforcement_records: list[InvariantEnforcementRecord] = []
|
||||
self._logger = logger.bind(service="invariant")
|
||||
@@ -72,6 +97,9 @@ class InvariantService:
|
||||
) -> Invariant:
|
||||
"""Add a new invariant with validation.
|
||||
|
||||
Thread-safe: acquires the internal reentrant lock before accessing
|
||||
shared state.
|
||||
|
||||
Args:
|
||||
text: The natural-language constraint text.
|
||||
scope: The scope at which this invariant applies.
|
||||
@@ -99,13 +127,14 @@ class InvariantService:
|
||||
source_name=source_name.strip(),
|
||||
)
|
||||
|
||||
self._invariants[invariant.id] = invariant
|
||||
self._logger.info(
|
||||
"Invariant added",
|
||||
invariant_id=invariant.id,
|
||||
scope=scope.value,
|
||||
source_name=source_name,
|
||||
)
|
||||
with self._lock:
|
||||
self._invariants[invariant.id] = invariant
|
||||
self._logger.info(
|
||||
"Invariant added",
|
||||
invariant_id=invariant.id,
|
||||
scope=scope.value,
|
||||
source_name=source_name,
|
||||
)
|
||||
return invariant
|
||||
|
||||
def list_invariants(
|
||||
@@ -116,6 +145,11 @@ class InvariantService:
|
||||
) -> list[Invariant]:
|
||||
"""Filter and list invariants.
|
||||
|
||||
Thread-safe: acquires the internal reentrant lock before accessing
|
||||
shared state. When ``effective=True`` calls are delegated through
|
||||
``get_effective_invariants()`` which also locks -- RLock makes this
|
||||
path deadlock-free (re-entrant acquisition).
|
||||
|
||||
Args:
|
||||
scope: Filter by scope (None = all scopes).
|
||||
source_name: Filter by source name (None = all sources).
|
||||
@@ -132,19 +166,23 @@ class InvariantService:
|
||||
project_name=source_name if scope == InvariantScope.PROJECT else None,
|
||||
)
|
||||
|
||||
result = [inv for inv in self._invariants.values() if inv.active]
|
||||
with self._lock:
|
||||
result = [inv for inv in self._invariants.values() if inv.active]
|
||||
|
||||
if scope is not None:
|
||||
result = [inv for inv in result if inv.scope == scope]
|
||||
if scope is not None:
|
||||
result = [inv for inv in result if inv.scope == scope]
|
||||
|
||||
if source_name is not None:
|
||||
result = [inv for inv in result if inv.source_name == source_name]
|
||||
if source_name is not None:
|
||||
result = [inv for inv in result if inv.source_name == source_name]
|
||||
|
||||
return result
|
||||
|
||||
def remove_invariant(self, invariant_id: str) -> Invariant:
|
||||
"""Soft-delete an invariant by setting active=False.
|
||||
|
||||
Thread-safe: acquires the internal reentrant lock before accessing
|
||||
shared state.
|
||||
|
||||
Args:
|
||||
invariant_id: The ULID of the invariant to remove.
|
||||
|
||||
@@ -157,16 +195,19 @@ class InvariantService:
|
||||
if not invariant_id or not invariant_id.strip():
|
||||
raise ValidationError("Invariant ID must not be empty")
|
||||
|
||||
inv = self._invariants.get(invariant_id)
|
||||
if inv is None:
|
||||
raise NotFoundError(
|
||||
resource_type="invariant",
|
||||
resource_id=invariant_id,
|
||||
with self._lock:
|
||||
inv = self._invariants.get(invariant_id)
|
||||
if inv is None:
|
||||
raise NotFoundError(
|
||||
resource_type="invariant",
|
||||
resource_id=invariant_id,
|
||||
)
|
||||
# Invariant is frozen (immutable); create a new instance with active=False
|
||||
deactivated = inv.model_copy(update={"active": False})
|
||||
self._invariants[invariant_id] = deactivated
|
||||
self._logger.info(
|
||||
"Invariant removed (soft-delete)", invariant_id=invariant_id
|
||||
)
|
||||
# Invariant is frozen (immutable); create a new instance with active=False
|
||||
deactivated = inv.model_copy(update={"active": False})
|
||||
self._invariants[invariant_id] = deactivated
|
||||
self._logger.info("Invariant removed (soft-delete)", invariant_id=invariant_id)
|
||||
return deactivated
|
||||
|
||||
def get_effective_invariants(
|
||||
@@ -180,6 +221,11 @@ class InvariantService:
|
||||
Collects active invariants from each scope tier and merges them
|
||||
using plan > action > project > global precedence.
|
||||
|
||||
Thread-safe: acquires the internal reentrant lock before accessing
|
||||
shared state. If called from another locked public method (e.g.
|
||||
``list_invariants`` with ``effective=True``), RLock handles the
|
||||
re-entrant acquisition so no deadlock occurs.
|
||||
|
||||
Args:
|
||||
plan_id: Optional plan identifier to collect plan-scoped
|
||||
invariants.
|
||||
@@ -194,28 +240,28 @@ class InvariantService:
|
||||
Returns:
|
||||
Merged, de-duplicated list of effective invariants.
|
||||
"""
|
||||
active = [inv for inv in self._invariants.values() if inv.active]
|
||||
|
||||
plan_invs = [
|
||||
inv
|
||||
for inv in active
|
||||
if inv.scope == InvariantScope.PLAN
|
||||
and (plan_id is None or inv.source_name == plan_id)
|
||||
]
|
||||
action_invs = [
|
||||
inv
|
||||
for inv in active
|
||||
if inv.scope == InvariantScope.ACTION
|
||||
and action_name is not None # Only include when explicitly requested
|
||||
and (inv.source_name == action_name or action_name == "*")
|
||||
]
|
||||
project_invs = [
|
||||
inv
|
||||
for inv in active
|
||||
if inv.scope == InvariantScope.PROJECT
|
||||
and (project_name is None or inv.source_name == project_name)
|
||||
]
|
||||
global_invs = [inv for inv in active if inv.scope == InvariantScope.GLOBAL]
|
||||
with self._lock:
|
||||
active = [inv for inv in self._invariants.values() if inv.active]
|
||||
plan_invs = [
|
||||
inv
|
||||
for inv in active
|
||||
if inv.scope == InvariantScope.PLAN
|
||||
and (plan_id is None or inv.source_name == plan_id)
|
||||
]
|
||||
action_invs = [
|
||||
inv
|
||||
for inv in active
|
||||
if inv.scope == InvariantScope.ACTION
|
||||
and action_name is not None # Only include when explicitly requested
|
||||
and (inv.source_name == action_name or action_name == "*")
|
||||
]
|
||||
project_invs = [
|
||||
inv
|
||||
for inv in active
|
||||
if inv.scope == InvariantScope.PROJECT
|
||||
and (project_name is None or inv.source_name == project_name)
|
||||
]
|
||||
global_invs = [inv for inv in active if inv.scope == InvariantScope.GLOBAL]
|
||||
|
||||
return merge_invariants(plan_invs, action_invs, project_invs, global_invs)
|
||||
|
||||
@@ -416,6 +462,10 @@ class InvariantService:
|
||||
Called by the Invariant Reconciliation Actor at the start of
|
||||
Strategize to record ``invariant_enforced`` decisions.
|
||||
|
||||
Thread-safe: acquires the internal reentrant lock before accessing
|
||||
shared state (both ``_enforcement_records`` list and ``_invariants``
|
||||
dict for event emission lookups).
|
||||
|
||||
Args:
|
||||
plan_id: The plan being checked.
|
||||
invariants: The invariants to enforce.
|
||||
@@ -435,44 +485,27 @@ class InvariantService:
|
||||
raise ValidationError("Plan ID must not be empty")
|
||||
|
||||
violated_ids = set(violated_invariant_ids or [])
|
||||
records: list[InvariantEnforcementRecord] = []
|
||||
response = actor_response or ""
|
||||
for inv in invariants:
|
||||
enforced = inv.id not in violated_ids
|
||||
record = InvariantEnforcementRecord(
|
||||
invariant_id=inv.id,
|
||||
enforced=enforced,
|
||||
actor_response=response,
|
||||
decision_id=str(ULID()),
|
||||
)
|
||||
records.append(record)
|
||||
if not enforced and self._event_bus is not None:
|
||||
try:
|
||||
self._event_bus.emit(
|
||||
DomainEvent(
|
||||
event_type=EventType.INVARIANT_VIOLATED,
|
||||
plan_id=plan_id,
|
||||
details={
|
||||
"invariant_id": inv.id,
|
||||
"invariant_text": inv.text,
|
||||
"scope": inv.scope.value,
|
||||
},
|
||||
)
|
||||
)
|
||||
except Exception:
|
||||
self._logger.warning(
|
||||
"event_bus_emit_failed",
|
||||
event_type="INVARIANT_VIOLATED",
|
||||
plan_id=plan_id,
|
||||
exc_info=True,
|
||||
)
|
||||
|
||||
self._enforcement_records.extend(records)
|
||||
self._logger.info(
|
||||
"Invariants enforced",
|
||||
plan_id=plan_id,
|
||||
count=len(records),
|
||||
)
|
||||
with self._lock:
|
||||
records: list[InvariantEnforcementRecord] = []
|
||||
for inv in invariants:
|
||||
enforced = inv.id not in violated_ids
|
||||
record = InvariantEnforcementRecord(
|
||||
invariant_id=inv.id,
|
||||
enforced=enforced,
|
||||
actor_response=response,
|
||||
decision_id=str(ULID()),
|
||||
)
|
||||
records.append(record)
|
||||
|
||||
self._enforcement_records.extend(records)
|
||||
self._logger.info(
|
||||
"Invariants enforced",
|
||||
plan_id=plan_id,
|
||||
count=len(records),
|
||||
)
|
||||
|
||||
if self._event_bus is not None:
|
||||
for record in records:
|
||||
try:
|
||||
@@ -494,7 +527,33 @@ class InvariantService:
|
||||
plan_id=plan_id,
|
||||
exc_info=True,
|
||||
)
|
||||
# Emit a single INVARIANT_RECONCILED for the batch
|
||||
|
||||
# Emit INVARIANT_VIOLATED events outside the lock to reduce
|
||||
# hold time on _lock (event bus I/O should not block
|
||||
# concurrent invariants access).
|
||||
for inv in invariants:
|
||||
if inv.id in violated_ids:
|
||||
try:
|
||||
self._event_bus.emit(
|
||||
DomainEvent(
|
||||
event_type=EventType.INVARIANT_VIOLATED,
|
||||
plan_id=plan_id,
|
||||
details={
|
||||
"invariant_id": inv.id,
|
||||
"invariant_text": inv.text,
|
||||
"scope": inv.scope.value,
|
||||
},
|
||||
)
|
||||
)
|
||||
except Exception:
|
||||
self._logger.warning(
|
||||
"event_bus_emit_failed",
|
||||
event_type="INVARIANT_VIOLATED",
|
||||
plan_id=plan_id,
|
||||
exc_info=True,
|
||||
)
|
||||
|
||||
# Emit a single INVARIANT_RECONCILED for the batch outside lock
|
||||
try:
|
||||
self._event_bus.emit(
|
||||
DomainEvent(
|
||||
@@ -512,4 +571,52 @@ class InvariantService:
|
||||
plan_id=plan_id,
|
||||
exc_info=True,
|
||||
)
|
||||
|
||||
return records
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
# Thread-safe helper read accessors #
|
||||
# ------------------------------------------------------------------ #
|
||||
|
||||
def get_enforcement_records(self) -> list[InvariantEnforcementRecord]:
|
||||
"""Return a snapshot of all enforcement records.
|
||||
|
||||
Thread-safe: acquires the internal reentrant lock to build copies of
|
||||
each record so callers receive a consistent, independent snapshot.
|
||||
|
||||
Returns:
|
||||
A list of ``InvariantEnforcementRecord`` objects copied from
|
||||
the internal registry.
|
||||
"""
|
||||
with self._lock:
|
||||
return [record.model_copy() for record in self._enforcement_records]
|
||||
|
||||
def get_invariant(self, invariant_id: str) -> Invariant | None:
|
||||
"""Return a single invariant by ID, or ``None`` if absent.
|
||||
|
||||
Thread-safe: acquires the internal reentrant lock before lookup.
|
||||
|
||||
Args:
|
||||
invariant_id: The ULID of the invariant to retrieve.
|
||||
|
||||
Returns:
|
||||
The ``Invariant`` if found, otherwise ``None``.
|
||||
"""
|
||||
with self._lock:
|
||||
inv = self._invariants.get(invariant_id)
|
||||
if inv is not None:
|
||||
return inv.model_copy()
|
||||
return None
|
||||
|
||||
def get_invariants_snapshot(self) -> list[Invariant]:
|
||||
"""Return a snapshot of all active invariants currently registered.
|
||||
|
||||
Thread-safe: acquires the internal reentrant lock to build a
|
||||
consistent copy of the active set at the moment of invocation.
|
||||
|
||||
Returns:
|
||||
A list of ``Invariant`` objects (copies) that are currently
|
||||
active (``active=True``).
|
||||
"""
|
||||
with self._lock:
|
||||
return [inv.model_copy() for inv in self._invariants.values() if inv.active]
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
"""Thread-safety regressions for InvariantService."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
from collections.abc import Callable
|
||||
|
||||
from cleveragents.application.services.invariant_service import InvariantService
|
||||
from cleveragents.domain.models.core.invariant import InvariantScope
|
||||
|
||||
|
||||
def _run_threads(*workers: Callable[[], None]) -> list[BaseException]:
|
||||
errors: list[BaseException] = []
|
||||
errors_lock = threading.Lock()
|
||||
start = threading.Barrier(len(workers))
|
||||
|
||||
def guarded(worker: Callable[[], None]) -> None:
|
||||
try:
|
||||
start.wait(timeout=5)
|
||||
worker()
|
||||
except BaseException as exc:
|
||||
with errors_lock:
|
||||
errors.append(exc)
|
||||
|
||||
threads = [threading.Thread(target=guarded, args=(worker,)) for worker in workers]
|
||||
for thread in threads:
|
||||
thread.start()
|
||||
for thread in threads:
|
||||
thread.join(timeout=10)
|
||||
if thread.is_alive():
|
||||
errors.append(TimeoutError(f"{thread.name} did not finish"))
|
||||
|
||||
return errors
|
||||
|
||||
|
||||
def test_add_list_and_remove_are_safe_under_concurrent_access() -> None:
|
||||
service = InvariantService()
|
||||
initial = [
|
||||
service.add_invariant(
|
||||
text=f"initial invariant {index}",
|
||||
scope=InvariantScope.GLOBAL,
|
||||
source_name="thread-test",
|
||||
)
|
||||
for index in range(20)
|
||||
]
|
||||
|
||||
def add_worker(worker_index: int) -> Callable[[], None]:
|
||||
def run() -> None:
|
||||
for index in range(40):
|
||||
service.add_invariant(
|
||||
text=f"worker {worker_index} invariant {index}",
|
||||
scope=InvariantScope.PROJECT,
|
||||
source_name="thread-test",
|
||||
)
|
||||
|
||||
return run
|
||||
|
||||
def list_worker() -> None:
|
||||
for _ in range(200):
|
||||
assert all(inv.active for inv in service.list_invariants())
|
||||
assert len(service.get_invariants_snapshot()) >= 1
|
||||
|
||||
def remove_worker() -> None:
|
||||
for inv in initial:
|
||||
service.remove_invariant(inv.id)
|
||||
|
||||
errors = _run_threads(
|
||||
add_worker(1),
|
||||
add_worker(2),
|
||||
add_worker(3),
|
||||
list_worker,
|
||||
list_worker,
|
||||
remove_worker,
|
||||
)
|
||||
|
||||
assert errors == []
|
||||
assert len(service.list_invariants(scope=InvariantScope.PROJECT)) == 120
|
||||
assert len(service.list_invariants(scope=InvariantScope.GLOBAL)) == 0
|
||||
|
||||
|
||||
def test_enforce_invariants_is_safe_under_concurrent_access() -> None:
|
||||
service = InvariantService()
|
||||
plan_id = "PLAN_THREAD_TEST"
|
||||
invariants = [
|
||||
service.add_invariant(
|
||||
text=f"plan invariant {index}",
|
||||
scope=InvariantScope.PLAN,
|
||||
source_name=plan_id,
|
||||
)
|
||||
for index in range(12)
|
||||
]
|
||||
|
||||
def enforce_worker(worker_index: int) -> Callable[[], None]:
|
||||
def run() -> None:
|
||||
for _ in range(25):
|
||||
service.enforce_invariants(
|
||||
plan_id=plan_id,
|
||||
invariants=invariants,
|
||||
actor_response=f"worker-{worker_index}",
|
||||
)
|
||||
|
||||
return run
|
||||
|
||||
def records_reader() -> None:
|
||||
for _ in range(200):
|
||||
records = service.get_enforcement_records()
|
||||
assert len(records) % len(invariants) == 0
|
||||
|
||||
errors = _run_threads(
|
||||
enforce_worker(1),
|
||||
enforce_worker(2),
|
||||
enforce_worker(3),
|
||||
enforce_worker(4),
|
||||
records_reader,
|
||||
records_reader,
|
||||
)
|
||||
|
||||
assert errors == []
|
||||
assert len(service.get_enforcement_records()) == 4 * 25 * len(invariants)
|
||||
BLOCKING — Wrong ScenarioOutline placeholder syntax:
{n}must be<n>In a Behave feature file,
Scenario Outlinecolumn values are substituted using angle-bracket syntax<param>. Curly-brace syntax ({n:d}) is reserved exclusively for step definition decorators in Python files.With
{n}here, Behave will NOT substitute the Examples table value. It tries to match the literal text{n} threads concurrently add {count:d} invariants through a barrieragainst the step definition pattern{n:d} threads concurrently add {count:d} invariants through a barrier, which will produceUndefinedStepat runtime.Fix:
Apply the same fix to all ScenarioOutline step lines in this file that use
{param}or{param:d}syntax — change every one to<param>(no type specifier in the feature file; the type specifier belongs only in the step decorator).See
features/acms_pipeline.featurefor the project-standard<param>ScenarioOutline pattern.Automated by CleverAgents Bot
Supervisor: PR Review | Agent: pr-review-worker
BLOCKING — ScenarioOutline placeholder syntax:
{n}should be<n>This line uses
{n}as a ScenarioOutline placeholder, but this project uses angle-bracket<param>syntax for ScenarioOutline substitution (standard Gherkin convention, confirmed byfeatures/cli.feature,features/actor_service_coverage.feature, etc.).Curly-brace
{param}is the step-capture syntax in step definitions only. Since{n}is not a valid outline placeholder, no substitution occurs and Behave passes the literal text"{n} threads concurrently add {count:d} invariants through a barrier"to the step matcher — which does not match the step definition"{n:d} threads concurrently...". This causesUndefinedStepand is the primary cause of theunit_testsCI failure.Fix:
Apply the same fix to every outline placeholder throughout this feature file: replace all
{param}and{param:d}in step text with<param>(no type specifiers in feature file text).