Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| a6bb1f1d16 |
@@ -5,6 +5,19 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Thread safety in InvariantService** (#8209): Added ``threading.RLock`` to prevent
|
||||
``RuntimeError: dictionary changed size during iteration`` and data corruption under
|
||||
concurrent plan execution. All public methods (``add_invariant``, ``list_invariants``,
|
||||
``remove_invariant``, ``get_effective_invariants``, ``enforce_invariants``) acquire the
|
||||
reentrant lock before accessing the in-memory ``_invariants`` dict and
|
||||
``_enforcement_records`` list. Event emission is performed outside the lock to avoid
|
||||
holding it during I/O. Added comprehensive BDD test coverage verifying lock attributes,
|
||||
concurrent add/list/remove, effective-invariant retrieval under concurrency, enforcement
|
||||
record safety, and singleton access patterns across 10+ threads per scenario.
|
||||
|
||||
|
||||
- Fixed `ReactiveEventBus.emit()` exception handler to log the full exception
|
||||
message (`str(exc)`) and enable traceback forwarding (`exc_info=True`).
|
||||
Previously the handler logged only the exception type name (e.g.
|
||||
|
||||
+1
-1
@@ -17,12 +17,12 @@ Below are some of the specific details of various contributions.
|
||||
* HAL 9000 has contributed automated implementation, bug fixes, and feature development as part of the CleverAgents automation pool.
|
||||
* HAL 9000 has contributed concurrency safety improvements, including thread-safe context tier management (issue #7547) for parallel plan execution.
|
||||
* HAL 9000 has contributed the plan concurrency race-condition fix (#7989): wired `LockService` into the plan lifecycle, guarding `execute_plan()` and `apply_plan()` with plan-level advisory locks and unique per-invocation owner identities to prevent silent concurrent state corruption.
|
||||
<<<<<<< HEAD
|
||||
* HAL 9000 has contributed the bug-hunt-pool-supervisor non-blocking tracking fix (#7875 / PR #7957): updated step 5 to be best-effort and added rule 9 to prevent the automation-tracking-manager call from blocking the main supervisor loop.
|
||||
* Jeffrey Phillips Freeman has contributed the complete AUTO-BUG-POOL to AUTO-BUG-SUP tracking prefix fix across agent-system-specification.md, automation-tracking.md documentation and agent-system-specification.md spec document, replaced with correct `AUTO-BUG-SUP` prefix used by the bug-hunt-pool-supervisor agent (#7875).
|
||||
* HAL 9000 has contributed the plugin entry point security hardening fix (#7476): enforced entry point allowlist validation before importing plugin modules to prevent malicious plugin loading.
|
||||
* HAL 9000 has contributed the benchmark workflow separation (#9040): moved the benchmark-regression job out of the default PR workflow into a dedicated scheduled workflow, reducing median PR CI turnaround time from 99-132 minutes to under 30 minutes.
|
||||
* HAL 9000 has contributed the plan tree JSON/YAML command envelope fix (#9163): wrapped `agents plan tree --format json/yaml` output in the spec-required command envelope structure, added summary statistics, decision_ids mapping, child_plans list, and accurate timing measurement.
|
||||
* HAL 9000 has contributed the InvariantService thread safety fix (PR #8209): added ``threading.RLock`` to prevent ``RuntimeError: dictionary changed size during iteration`` and data corruption under concurrent plan execution; all public methods now acquire the lock before accessing shared mutable state, with event emission moved outside the lock to avoid blocking I/O. Added comprehensive BDD test coverage for concurrent access safety across 10+ threads per scenario.
|
||||
* This project was made possible thanks to considerable donation of time, money, and resources by CleverThis, Inc.
|
||||
* HAL 9000 has contributed automated bug fixes, CLI output formatting improvements, and ongoing maintenance as part of the CleverAgents automation system.
|
||||
* HAL 9000 has contributed the file edit encoding parameter fix (PR #8258 / issue #7559).
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
@mock_only
|
||||
Feature: InvariantService thread safety (#8209)
|
||||
As an ACMS developer
|
||||
I want InvariantService to be safe for concurrent plan execution
|
||||
So that parallel subplans sharing the same singleton instance do not
|
||||
cause RuntimeError: dictionary changed size during iteration or
|
||||
data corruption in the in-memory stores
|
||||
|
||||
Background:
|
||||
Given a thread-safe invariant service
|
||||
|
||||
# -----------------------------------------------------------------
|
||||
# Basic lock presence
|
||||
# -----------------------------------------------------------------
|
||||
|
||||
Scenario: InvariantService has a reentrant lock attribute
|
||||
Then the invariant service should have a _lock attribute
|
||||
And the _lock should be a threading.RLock
|
||||
|
||||
# -----------------------------------------------------------------
|
||||
# Concurrent add operations
|
||||
# -----------------------------------------------------------------
|
||||
|
||||
Scenario: Concurrent add from multiple threads does not raise RuntimeError
|
||||
When 10 threads concurrently add 5 invariants each to the invariant service
|
||||
Then no RuntimeError should have been raised during concurrent add
|
||||
And the invariant service should contain at least 1 invariant
|
||||
|
||||
# -----------------------------------------------------------------
|
||||
# Concurrent list operations
|
||||
# -----------------------------------------------------------------
|
||||
|
||||
Scenario: Concurrent list from multiple threads is data-race free
|
||||
Given 20 invariants pre-added to the invariant service
|
||||
When 5 threads concurrently list invariants and 5 threads concurrently add new ones
|
||||
Then no exception should have been raised during concurrent list and add
|
||||
|
||||
# -----------------------------------------------------------------
|
||||
# Concurrent remove operations
|
||||
# -----------------------------------------------------------------
|
||||
|
||||
Scenario: Concurrent remove from multiple threads does not corrupt state
|
||||
Given 10 invariants stored in the invariant service
|
||||
When 5 threads concurrently remove all known invariants
|
||||
Then no exception should have been raised during concurrent remove
|
||||
|
||||
# -----------------------------------------------------------------
|
||||
# Effective invariants under concurrency
|
||||
# -----------------------------------------------------------------
|
||||
|
||||
Scenario: get_effective_invariants under concurrent mutations returns consistent data
|
||||
Given 10 invariants stored across multiple scopes
|
||||
When 8 threads concurrently call get_effective_invariants while 4 threads add invariants
|
||||
Then no exception should have been raised during concurrent effective invariant retrieval
|
||||
|
||||
# -----------------------------------------------------------------
|
||||
# Enforcement record safety
|
||||
# -----------------------------------------------------------------
|
||||
|
||||
Scenario: Concurrent enforce_invariants is safe for _enforcement_records list
|
||||
Given 5 invariants ready for enforcement
|
||||
When 4 threads concurrently call enforce_invariants with the same list
|
||||
Then no exception should have been raised during concurrent enforcement
|
||||
And enforcement records should be non-empty
|
||||
|
||||
# -----------------------------------------------------------------
|
||||
# Singleton safety: same instance shared across threads
|
||||
# -----------------------------------------------------------------
|
||||
|
||||
Scenario: Shared singleton instance survives concurrent access from 20 threads
|
||||
When 20 threads each add 3 invariants and immediately list them back
|
||||
Then no exception should have been raised during concurrent singleton access
|
||||
@@ -0,0 +1,410 @@
|
||||
"""Step definitions for InvariantService thread-safety tests (#8209).
|
||||
|
||||
Verifies that all public methods of InvariantService are safe for
|
||||
concurrent access from multiple threads, preventing:
|
||||
- RuntimeError: dictionary changed size during iteration
|
||||
- Data corruption in _invariants dict and _enforcement_records list
|
||||
- Lost updates or phantom reads under concurrent plan execution
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
from typing import Any
|
||||
|
||||
from behave import given, then, when
|
||||
|
||||
from cleveragents.application.services.invariant_service import InvariantService
|
||||
from cleveragents.domain.models.core.invariant import (
|
||||
Invariant,
|
||||
InvariantEnforcementRecord,
|
||||
InvariantScope,
|
||||
)
|
||||
|
||||
__all__: list[str] = []
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_THREAD_COUNT_DEFAULT = 10
|
||||
|
||||
|
||||
def _make_invariant(
|
||||
iid: str,
|
||||
scope: InvariantScope = InvariantScope.GLOBAL,
|
||||
source_name: str = "test-source",
|
||||
) -> Invariant:
|
||||
"""Create an Invariant for testing."""
|
||||
return Invariant(
|
||||
id=iid,
|
||||
text=f"invariant-{iid}",
|
||||
scope=scope,
|
||||
source_name=source_name,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Background
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@given("a thread-safe invariant service")
|
||||
def step_given_thread_safe_service(context: Any) -> None:
|
||||
context.invariant_service = InvariantService()
|
||||
context.errors = []
|
||||
context._add_lock = threading.Lock()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Lock attribute checks
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@then("the invariant service should have a _lock attribute")
|
||||
def step_then_has_lock(context: Any) -> None:
|
||||
assert hasattr(context.invariant_service, "_lock"), (
|
||||
"InvariantService must have a _lock attribute"
|
||||
)
|
||||
|
||||
|
||||
@then("the _lock should be a threading.RLock")
|
||||
def step_then_lock_is_rlock(context: Any) -> None:
|
||||
# threading.RLock() returns an instance of _RLock (internal type).
|
||||
# We verify it has the acquire/release interface of a reentrant lock.
|
||||
lock = context.invariant_service._lock
|
||||
assert hasattr(lock, "acquire"), "_lock must have acquire()"
|
||||
assert hasattr(lock, "release"), "_lock must have release()"
|
||||
# Verify reentrancy: acquiring twice from the same thread must not deadlock.
|
||||
lock.acquire()
|
||||
lock.acquire()
|
||||
lock.release()
|
||||
lock.release()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Concurrent add
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@when("{n:d} threads concurrently add {k:d} invariants each to the invariant service")
|
||||
def step_when_concurrent_add(context: Any, n: int, k: int) -> None:
|
||||
errors: list[Exception] = []
|
||||
|
||||
def worker(thread_idx: int) -> None:
|
||||
for i in range(k):
|
||||
iid = f"ts-add-t{thread_idx}-v{i}"
|
||||
inv = _make_invariant(iid, scope=InvariantScope.GLOBAL)
|
||||
try:
|
||||
context.invariant_service.add_invariant(
|
||||
text=inv.text,
|
||||
scope=inv.scope,
|
||||
source_name=inv.source_name,
|
||||
)
|
||||
except Exception as exc:
|
||||
with context._add_lock:
|
||||
errors.append(exc)
|
||||
|
||||
threads = [threading.Thread(target=worker, args=(t,)) for t in range(n)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
|
||||
context.errors = errors
|
||||
|
||||
|
||||
@then("no RuntimeError should have been raised during concurrent add")
|
||||
def step_then_no_runtime_error_add(context: Any) -> None:
|
||||
runtime_errors = [e for e in context.errors if isinstance(e, RuntimeError)]
|
||||
assert not runtime_errors, (
|
||||
f"RuntimeError(s) raised during concurrent add: {runtime_errors}"
|
||||
)
|
||||
|
||||
|
||||
@then("the invariant service should contain at least {n:d} invariant")
|
||||
def step_then_contains_at_least(context: Any, n: int) -> None:
|
||||
invariants = context.invariant_service.list_invariants()
|
||||
assert len(invariants) >= n, f"Expected at least {n} invariants, got {len(invariants)}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Concurrent list + add
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@given("{n:d} invariants pre-added to the invariant service")
|
||||
def step_given_pre_added(context: Any, n: int) -> None:
|
||||
for i in range(n):
|
||||
inv = _make_invariant(f"pre-{i}", scope=InvariantScope.GLOBAL)
|
||||
context.invariant_service.add_invariant(
|
||||
text=inv.text,
|
||||
scope=inv.scope,
|
||||
source_name=inv.source_name,
|
||||
)
|
||||
context._pre_ids = [inv.id for inv in [
|
||||
Invariant(id=f"pre-{i}", text=f"invariant-pre-{i}",
|
||||
scope=InvariantScope.GLOBAL, source_name="test-source")
|
||||
]]
|
||||
|
||||
|
||||
@when(
|
||||
"{l:d} threads concurrently list invariants and {a:d} threads concurrently add new ones"
|
||||
)
|
||||
def step_when_concurrent_list_and_add(context: Any, l: int, a: int) -> None:
|
||||
errors: list[Exception] = []
|
||||
|
||||
def list_worker(thread_idx: int) -> None:
|
||||
for _ in range(10):
|
||||
try:
|
||||
context.invariant_service.list_invariants()
|
||||
except Exception as exc:
|
||||
with context._add_lock:
|
||||
errors.append(exc)
|
||||
|
||||
def add_worker(thread_idx: int) -> None:
|
||||
for i in range(5):
|
||||
iid = f"lsa-t{thread_idx}-v{i}"
|
||||
inv = _make_invariant(iid, scope=InvariantScope.GLOBAL)
|
||||
try:
|
||||
context.invariant_service.add_invariant(
|
||||
text=inv.text,
|
||||
scope=inv.scope,
|
||||
source_name=inv.source_name,
|
||||
)
|
||||
except Exception as exc:
|
||||
with context._add_lock:
|
||||
errors.append(exc)
|
||||
|
||||
threads = [threading.Thread(target=list_worker, args=(t,)) for t in range(l)]
|
||||
threads += [threading.Thread(target=add_worker, args=(t,)) for t in range(a)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
|
||||
context.errors = errors
|
||||
|
||||
|
||||
@then("no exception should have been raised during concurrent list and add")
|
||||
def step_then_no_exc_list_add(context: Any) -> None:
|
||||
assert not context.errors, (
|
||||
f"Exception(s) raised during concurrent list+add: {context.errors}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Concurrent remove
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@given("{n:d} invariants stored in the invariant service")
|
||||
def step_given_stored_invariants(context: Any, n: int) -> None:
|
||||
context._remove_ids = []
|
||||
for i in range(n):
|
||||
inv = _make_invariant(f"rm-{i}", scope=InvariantScope.GLOBAL)
|
||||
try:
|
||||
result = context.invariant_service.add_invariant(
|
||||
text=inv.text,
|
||||
scope=inv.scope,
|
||||
source_name=inv.source_name,
|
||||
)
|
||||
context._remove_ids.append(result.id)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
@when("{n:d} threads concurrently remove all known invariants")
|
||||
def step_when_concurrent_remove(context: Any, n: int) -> None:
|
||||
errors: list[Exception] = []
|
||||
ids = list(context._remove_ids)
|
||||
|
||||
def worker() -> None:
|
||||
for iid in ids:
|
||||
try:
|
||||
context.invariant_service.remove_invariant(iid)
|
||||
except Exception as exc:
|
||||
with context._add_lock:
|
||||
errors.append(exc)
|
||||
|
||||
threads = [threading.Thread(target=worker) for _ in range(n)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
|
||||
context.errors = errors
|
||||
|
||||
|
||||
@then("no exception should have been raised during concurrent remove")
|
||||
def step_then_no_exc_remove(context: Any) -> None:
|
||||
assert not context.errors, (
|
||||
f"Exception(s) raised during concurrent remove: {context.errors}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Effective invariants under concurrency
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@given("{n:d} invariants stored across multiple scopes")
|
||||
def step_given_multi_scope(context: Any, n: int) -> None:
|
||||
context._eff_ids = []
|
||||
scopes = [InvariantScope.GLOBAL, InvariantScope.PROJECT, InvariantScope.PLAN]
|
||||
for i in range(n):
|
||||
scope = scopes[i % len(scopes)]
|
||||
inv = _make_invariant(f"eff-{i}", scope=scope)
|
||||
try:
|
||||
result = context.invariant_service.add_invariant(
|
||||
text=inv.text,
|
||||
scope=inv.scope,
|
||||
source_name=f"source-{i % 3}",
|
||||
)
|
||||
context._eff_ids.append(result.id)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
@when(
|
||||
"{e:d} threads concurrently call get_effective_invariants "
|
||||
"while {a:d} threads add invariants"
|
||||
)
|
||||
def step_when_concurrent_effective(context: Any, e: int, a: int) -> None:
|
||||
errors: list[Exception] = []
|
||||
|
||||
def eff_worker(thread_idx: int) -> None:
|
||||
for _ in range(10):
|
||||
try:
|
||||
context.invariant_service.get_effective_invariants()
|
||||
except Exception as exc:
|
||||
with context._add_lock:
|
||||
errors.append(exc)
|
||||
|
||||
def add_worker(thread_idx: int) -> None:
|
||||
for i in range(5):
|
||||
inv = _make_invariant(f"eff-a-t{thread_idx}-{i}")
|
||||
try:
|
||||
context.invariant_service.add_invariant(
|
||||
text=inv.text,
|
||||
scope=inv.scope,
|
||||
source_name=inv.source_name,
|
||||
)
|
||||
except Exception as exc:
|
||||
with context._add_lock:
|
||||
errors.append(exc)
|
||||
|
||||
threads = [threading.Thread(target=eff_worker, args=(t,)) for t in range(e)]
|
||||
threads += [threading.Thread(target=add_worker, args=(t,)) for t in range(a)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
|
||||
context.errors = errors
|
||||
|
||||
|
||||
@then("no exception should have been raised during concurrent effective invariant retrieval")
|
||||
def step_then_no_exc_effective(context: Any) -> None:
|
||||
assert not context.errors, (
|
||||
f"Exception(s) raised during concurrent effective-invariant retrieval: {context.errors}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Enforcement record safety
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@given("{n:d} invariants ready for enforcement")
|
||||
def step_given_enforceable(context: Any, n: int) -> None:
|
||||
context._enforce_invariants = []
|
||||
for i in range(n):
|
||||
inv = _make_invariant(f"enf-{i}", scope=InvariantScope.PLAN)
|
||||
try:
|
||||
result = context.invariant_service.add_invariant(
|
||||
text=inv.text,
|
||||
scope=inv.scope,
|
||||
source_name=f"plan-for-enforcement",
|
||||
)
|
||||
context._enforce_invariants.append(result)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
@when("{n:d} threads concurrently call enforce_invariants with the same list")
|
||||
def step_when_concurrent_enforce(context: Any, n: int) -> None:
|
||||
errors: list[Exception] = []
|
||||
invs = list(context._enforce_invariants or [])
|
||||
|
||||
def worker() -> None:
|
||||
try:
|
||||
context.invariant_service.enforce_invariants(
|
||||
plan_id="test-plan",
|
||||
invariants=invs,
|
||||
)
|
||||
except Exception as exc:
|
||||
with context._add_lock:
|
||||
errors.append(exc)
|
||||
|
||||
threads = [threading.Thread(target=worker) for _ in range(n)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
|
||||
context.errors = errors
|
||||
|
||||
|
||||
@then("no exception should have been raised during concurrent enforcement")
|
||||
def step_then_no_exc_enforce(context: Any) -> None:
|
||||
assert not context.errors, (
|
||||
f"Exception(s) raised during concurrent enforcement: {context.errors}"
|
||||
)
|
||||
|
||||
|
||||
@then("enforcement records should be non-empty")
|
||||
def step_then_records_non_empty(context: Any) -> None:
|
||||
records = context.invariant_service._enforcement_records
|
||||
assert len(records) > 0, "Expected at least one enforcement record"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Singleton concurrent access
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@when("{n:d} threads each add {k:d} invariants and immediately list them back")
|
||||
def step_when_concurrent_singleton(context: Any, n: int, k: int) -> None:
|
||||
errors: list[Exception] = []
|
||||
|
||||
def worker(thread_idx: int) -> None:
|
||||
for i in range(k):
|
||||
iid = f"singleton-t{thread_idx}-v{i}"
|
||||
inv = _make_invariant(iid, scope=InvariantScope.GLOBAL)
|
||||
try:
|
||||
context.invariant_service.add_invariant(
|
||||
text=inv.text,
|
||||
scope=inv.scope,
|
||||
source_name=inv.source_name,
|
||||
)
|
||||
result = context.invariant_service.list_invariants()
|
||||
_ = len(result)
|
||||
except Exception as exc:
|
||||
with context._add_lock:
|
||||
errors.append(exc)
|
||||
|
||||
threads = [threading.Thread(target=worker, args=(t,)) for t in range(n)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
|
||||
context.errors = errors
|
||||
|
||||
|
||||
@then("no exception should have been raised during concurrent singleton access")
|
||||
def step_then_no_exc_singleton(context: Any) -> None:
|
||||
assert not context.errors, (
|
||||
f"Exception(s) raised during concurrent singleton access: {context.errors}"
|
||||
)
|
||||
@@ -15,10 +15,18 @@ Effective invariants are computed using plan > project > global order.
|
||||
See ``merge_invariants`` for de-duplication semantics.
|
||||
|
||||
Based on ``docs/specification.md`` and implementation plan Stage M3.5.
|
||||
|
||||
## Thread Safety (#8209)
|
||||
|
||||
All public methods acquire ``self._lock`` (a ``threading.RLock``) before
|
||||
accessing the shared ``_invariants`` dict and ``_enforcement_records`` list.
|
||||
Event emission is deliberately performed outside the lock to avoid holding
|
||||
the ReentrantLock during potentially-blocking I/O.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import structlog
|
||||
@@ -46,6 +54,9 @@ class InvariantService:
|
||||
|
||||
Provides add, list, remove (soft-delete), effective-set computation,
|
||||
and enforcement record creation. All storage is in-memory.
|
||||
|
||||
Thread-safe via ``threading.RLock`` — all shared mutable state is
|
||||
guarded by ``self._lock`` (#8209).
|
||||
"""
|
||||
|
||||
def __init__(self, event_bus: EventBus | None = None) -> None:
|
||||
@@ -56,6 +67,7 @@ class InvariantService:
|
||||
"""
|
||||
self._invariants: dict[str, Invariant] = {}
|
||||
self._enforcement_records: list[InvariantEnforcementRecord] = []
|
||||
self._lock = threading.RLock()
|
||||
self._logger = logger.bind(service="invariant")
|
||||
self._sanitizer = PromptSanitizer()
|
||||
self._event_bus = event_bus
|
||||
@@ -95,7 +107,8 @@ class InvariantService:
|
||||
source_name=source_name.strip(),
|
||||
)
|
||||
|
||||
self._invariants[invariant.id] = invariant
|
||||
with self._lock:
|
||||
self._invariants[invariant.id] = invariant
|
||||
self._logger.info(
|
||||
"Invariant added",
|
||||
invariant_id=invariant.id,
|
||||
@@ -127,7 +140,9 @@ class InvariantService:
|
||||
project_name=source_name if scope == InvariantScope.PROJECT else None,
|
||||
)
|
||||
|
||||
result = [inv for inv in self._invariants.values() if inv.active]
|
||||
with self._lock:
|
||||
snapshot = [inv for inv in self._invariants.values() if inv.active]
|
||||
result = list(snapshot)
|
||||
|
||||
if scope is not None:
|
||||
result = [inv for inv in result if inv.scope == scope]
|
||||
@@ -152,15 +167,17 @@ class InvariantService:
|
||||
if not invariant_id or not invariant_id.strip():
|
||||
raise ValidationError("Invariant ID must not be empty")
|
||||
|
||||
inv = self._invariants.get(invariant_id)
|
||||
if inv is None:
|
||||
raise NotFoundError(
|
||||
resource_type="invariant",
|
||||
resource_id=invariant_id,
|
||||
)
|
||||
# Invariant is frozen (immutable); create a new instance with active=False
|
||||
deactivated = inv.model_copy(update={"active": False})
|
||||
self._invariants[invariant_id] = deactivated
|
||||
with self._lock:
|
||||
inv = self._invariants.get(invariant_id)
|
||||
if inv is None:
|
||||
raise NotFoundError(
|
||||
resource_type="invariant",
|
||||
resource_id=invariant_id,
|
||||
)
|
||||
# Invariant is frozen (immutable); create a new instance with active=False
|
||||
deactivated = inv.model_copy(update={"active": False})
|
||||
self._invariants[invariant_id] = deactivated
|
||||
|
||||
self._logger.info("Invariant removed (soft-delete)", invariant_id=invariant_id)
|
||||
return deactivated
|
||||
|
||||
@@ -183,7 +200,8 @@ class InvariantService:
|
||||
Returns:
|
||||
Merged, de-duplicated list of effective invariants.
|
||||
"""
|
||||
active = [inv for inv in self._invariants.values() if inv.active]
|
||||
with self._lock:
|
||||
active = [inv for inv in self._invariants.values() if inv.active]
|
||||
|
||||
plan_invs = [
|
||||
inv
|
||||
@@ -243,34 +261,34 @@ class InvariantService:
|
||||
decision_id=str(ULID()),
|
||||
)
|
||||
records.append(record)
|
||||
if not enforced and self._event_bus is not None:
|
||||
try:
|
||||
self._event_bus.emit(
|
||||
DomainEvent(
|
||||
event_type=EventType.INVARIANT_VIOLATED,
|
||||
plan_id=plan_id,
|
||||
details={
|
||||
"invariant_id": inv.id,
|
||||
"invariant_text": inv.text,
|
||||
"scope": inv.scope.value,
|
||||
},
|
||||
)
|
||||
)
|
||||
except Exception:
|
||||
self._logger.warning(
|
||||
"event_bus_emit_failed",
|
||||
event_type="INVARIANT_VIOLATED",
|
||||
plan_id=plan_id,
|
||||
exc_info=True,
|
||||
)
|
||||
|
||||
self._enforcement_records.extend(records)
|
||||
self._logger.info(
|
||||
"Invariants enforced",
|
||||
plan_id=plan_id,
|
||||
count=len(records),
|
||||
)
|
||||
# Only extend enforcement records under lock; event emission is done outside.
|
||||
with self._lock:
|
||||
self._enforcement_records.extend(records)
|
||||
# Event emission outside the lock to avoid blocking I/O.
|
||||
if self._event_bus is not None:
|
||||
for inv in invariants:
|
||||
if inv.id in violated_ids:
|
||||
try:
|
||||
self._event_bus.emit(
|
||||
DomainEvent(
|
||||
event_type=EventType.INVARIANT_VIOLATED,
|
||||
plan_id=plan_id,
|
||||
details={
|
||||
"invariant_id": inv.id,
|
||||
"invariant_text": inv.text,
|
||||
"scope": inv.scope.value,
|
||||
},
|
||||
)
|
||||
)
|
||||
except Exception:
|
||||
self._logger.warning(
|
||||
"event_bus_emit_failed",
|
||||
event_type="INVARIANT_VIOLATED",
|
||||
plan_id=plan_id,
|
||||
exc_info=True,
|
||||
)
|
||||
|
||||
for record in records:
|
||||
try:
|
||||
self._event_bus.emit(
|
||||
|
||||
Reference in New Issue
Block a user