fix(concurrency): add thread safety to InvariantService
CI / helm (pull_request) Successful in 26s
CI / push-validation (pull_request) Successful in 26s
CI / build (pull_request) Successful in 26s
CI / lint (pull_request) Successful in 33s
CI / quality (pull_request) Successful in 34s
CI / typecheck (pull_request) Successful in 49s
CI / security (pull_request) Successful in 59s
CI / e2e_tests (pull_request) Successful in 4m7s
CI / integration_tests (pull_request) Successful in 4m23s
CI / unit_tests (pull_request) Failing after 5m4s
CI / docker (pull_request) Has been skipped
CI / coverage (pull_request) Successful in 10m15s
CI / status-check (pull_request) Failing after 1s
CI / benchmark-publish (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Successful in 57m17s

Added threading.RLock() to protect _invariants dict and _enforcement_records list
All public methods (add_invariant, remove_invariant, list_invariants, get_effective_invariants, enforce_invariants) now use lock protection
Prevents RuntimeError: dictionary changed size during iteration
Prevents data corruption from concurrent access
Follows existing pattern from AutonomyController and AutonomyGuardrailService

ISSUES CLOSED: #7524
This commit is contained in:
2026-04-13 04:25:05 +00:00
parent 96ff9d0ff8
commit d1a79ef99a
3 changed files with 967 additions and 30 deletions
@@ -0,0 +1,93 @@
Feature: InvariantService Thread Safety
As a developer
I want InvariantService to be thread-safe
So that concurrent access from parallel plan execution doesn't cause data corruption
Background:
Given I have an InvariantService instance
# ================================================================
# Concurrent Add Operations
# ================================================================
Scenario: Multiple threads can safely add invariants concurrently
When 5 threads concurrently add invariants with unique texts
Then all invariants should be stored without data loss
And the total count should be 5
Scenario: Concurrent adds don't cause dictionary changed size errors
When 10 threads concurrently add invariants
Then no RuntimeError should be raised
And all invariants should be retrievable
# ================================================================
# Concurrent List Operations
# ================================================================
Scenario: Multiple threads can safely list invariants concurrently
Given I have added 5 invariants
When 5 threads concurrently list invariants
Then all threads should complete successfully
And each thread should see consistent data
Scenario: Listing while adding doesn't cause iteration errors
When 3 threads add invariants while 3 threads list invariants concurrently
Then no RuntimeError should be raised
And the final count should match the number of adds
# ================================================================
# Concurrent Remove Operations
# ================================================================
Scenario: Multiple threads can safely remove invariants concurrently
Given I have added 5 invariants
When 3 threads concurrently remove different invariants
Then all removals should succeed
And the remaining count should be 2
Scenario: Removing while listing doesn't cause errors
Given I have added 5 invariants
When 2 threads remove invariants while 2 threads list invariants concurrently
Then no RuntimeError should be raised
And the final state should be consistent
# ================================================================
# Concurrent Enforcement Operations
# ================================================================
Scenario: Multiple threads can safely enforce invariants concurrently
Given I have added 3 invariants
When 5 threads concurrently enforce the same invariants
Then all enforcement records should be stored
And the total record count should be 15
Scenario: Enforcement records are not lost under concurrent access
Given I have added 2 invariants
When 5 threads concurrently enforce invariants
Then all 10 enforcement records should be stored
And no records should be duplicated or lost
# ================================================================
# Mixed Concurrent Operations
# ================================================================
Scenario: Mixed operations (add, list, remove, enforce) are thread-safe
When 2 threads add, 2 threads list, 2 threads remove, and 2 threads enforce concurrently
Then no errors should occur in thread safety test
And the final state should be consistent
# ================================================================
# Concurrent Effective Invariants
# ================================================================
Scenario: Getting effective invariants is thread-safe
Given I have added invariants at different scopes
When 5 threads concurrently get effective invariants
Then all threads should complete successfully
And all threads should see consistent merged results
Scenario: Effective invariants computation under concurrent modifications
Given I have added invariants at different scopes
When 3 threads add invariants while 3 threads get effective invariants concurrently
Then no errors should occur in thread safety test
And the results should be consistent
@@ -0,0 +1,830 @@
"""Step definitions for invariant_service_thread_safety.feature.
Tests thread safety of InvariantService under concurrent access.
"""
from __future__ import annotations
import threading
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from typing import Any
from behave import given, then, when # type: ignore[import-untyped]
from cleveragents.application.services.invariant_service import InvariantService
from cleveragents.domain.models.core.invariant import InvariantScope
# ================================================================
# Background Steps
# ================================================================
@given("I have an InvariantService instance")
def step_create_service(context: Any) -> None:
"""Create a fresh InvariantService instance."""
context.service = InvariantService()
context.errors: list[Exception] = []
context.results: dict[str, Any] = {}
# ================================================================
# Setup Steps
# ================================================================
@given("I have added {count:d} invariants")
def step_add_invariants(context: Any, count: int) -> None:
"""Add a specified number of invariants to the service."""
context.invariant_ids = []
for i in range(count):
inv = context.service.add_invariant(
text=f"Invariant {i}",
scope=InvariantScope.GLOBAL,
source_name="test",
)
context.invariant_ids.append(inv.id)
@given("I have added invariants at different scopes")
def step_add_scoped_invariants(context: Any) -> None:
"""Add invariants at different scopes for testing."""
context.invariant_ids = []
scopes = [InvariantScope.GLOBAL, InvariantScope.PROJECT, InvariantScope.PLAN]
for i, scope in enumerate(scopes):
for j in range(3):
inv = context.service.add_invariant(
text=f"Invariant {scope.value} {j}",
scope=scope,
source_name=f"source_{i}",
)
context.invariant_ids.append(inv.id)
# ================================================================
# Concurrent Add Operations
# ================================================================
@when("{count:d} threads concurrently add invariants with unique texts")
def step_concurrent_add_unique(context: Any, count: int) -> None:
"""Add invariants from multiple threads concurrently."""
context.errors = []
context.added_ids = []
lock = threading.Lock()
def add_invariant(thread_id: int) -> None:
try:
inv = context.service.add_invariant(
text=f"Thread {thread_id} Invariant",
scope=InvariantScope.GLOBAL,
source_name="test",
)
with lock:
context.added_ids.append(inv.id)
except Exception as e:
with lock:
context.errors.append(e)
with ThreadPoolExecutor(max_workers=count) as executor:
futures = [executor.submit(add_invariant, i) for i in range(count)]
for future in as_completed(futures):
future.result()
@when("{count:d} threads concurrently add invariants")
def step_concurrent_add(context: Any, count: int) -> None:
"""Add invariants from multiple threads concurrently."""
context.errors = []
context.added_ids = []
lock = threading.Lock()
def add_invariant(thread_id: int) -> None:
try:
inv = context.service.add_invariant(
text=f"Invariant from thread {thread_id}",
scope=InvariantScope.GLOBAL,
source_name="test",
)
with lock:
context.added_ids.append(inv.id)
except Exception as e:
with lock:
context.errors.append(e)
with ThreadPoolExecutor(max_workers=count) as executor:
futures = [executor.submit(add_invariant, i) for i in range(count)]
for future in as_completed(futures):
future.result()
# ================================================================
# Concurrent List Operations
# ================================================================
@when("{count:d} threads concurrently list invariants")
def step_concurrent_list(context: Any, count: int) -> None:
"""List invariants from multiple threads concurrently."""
context.errors = []
context.list_results = []
lock = threading.Lock()
def list_invariants(thread_id: int) -> None:
try:
result = context.service.list_invariants()
with lock:
context.list_results.append(result)
except Exception as e:
with lock:
context.errors.append(e)
with ThreadPoolExecutor(max_workers=count) as executor:
futures = [executor.submit(list_invariants, i) for i in range(count)]
for future in as_completed(futures):
future.result()
@when(
"{add_count:d} threads add invariants while {list_count:d} threads list invariants concurrently"
)
def step_concurrent_add_and_list(context: Any, add_count: int, list_count: int) -> None:
"""Add and list invariants concurrently from different threads."""
context.errors = []
context.added_ids = []
context.list_results = []
lock = threading.Lock()
def add_invariant(thread_id: int) -> None:
try:
inv = context.service.add_invariant(
text=f"Invariant from add thread {thread_id}",
scope=InvariantScope.GLOBAL,
source_name="test",
)
with lock:
context.added_ids.append(inv.id)
except Exception as e:
with lock:
context.errors.append(e)
def list_invariants(thread_id: int) -> None:
try:
result = context.service.list_invariants()
with lock:
context.list_results.append(result)
except Exception as e:
with lock:
context.errors.append(e)
with ThreadPoolExecutor(max_workers=add_count + list_count) as executor:
futures = []
for i in range(add_count):
futures.append(executor.submit(add_invariant, i))
for i in range(list_count):
futures.append(executor.submit(list_invariants, i))
for future in as_completed(futures):
future.result()
# ================================================================
# Concurrent Remove Operations
# ================================================================
@when("{count:d} threads concurrently remove different invariants")
def step_concurrent_remove(context: Any, count: int) -> None:
"""Remove invariants from multiple threads concurrently."""
context.errors = []
context.removed_ids = []
lock = threading.Lock()
def remove_invariant(thread_id: int) -> None:
try:
if thread_id < len(context.invariant_ids):
inv_id = context.invariant_ids[thread_id]
result = context.service.remove_invariant(inv_id)
with lock:
context.removed_ids.append(result.id)
except Exception as e:
with lock:
context.errors.append(e)
with ThreadPoolExecutor(max_workers=count) as executor:
futures = [executor.submit(remove_invariant, i) for i in range(count)]
for future in as_completed(futures):
future.result()
@when(
"{remove_count:d} threads remove invariants while {list_count:d} threads list invariants concurrently"
)
def step_concurrent_remove_and_list(
context: Any, remove_count: int, list_count: int
) -> None:
"""Remove and list invariants concurrently from different threads."""
context.errors = []
context.removed_ids = []
context.list_results = []
lock = threading.Lock()
remove_index = 0
def remove_invariant(thread_id: int) -> None:
nonlocal remove_index
try:
with lock:
if remove_index < len(context.invariant_ids):
inv_id = context.invariant_ids[remove_index]
remove_index += 1
else:
return
result = context.service.remove_invariant(inv_id)
with lock:
context.removed_ids.append(result.id)
except Exception as e:
with lock:
context.errors.append(e)
def list_invariants(thread_id: int) -> None:
try:
result = context.service.list_invariants()
with lock:
context.list_results.append(result)
except Exception as e:
with lock:
context.errors.append(e)
with ThreadPoolExecutor(max_workers=remove_count + list_count) as executor:
futures = []
for i in range(remove_count):
futures.append(executor.submit(remove_invariant, i))
for i in range(list_count):
futures.append(executor.submit(list_invariants, i))
for future in as_completed(futures):
future.result()
# ================================================================
# Concurrent Enforcement Operations
# ================================================================
@when("{count:d} threads concurrently enforce the same invariants")
def step_concurrent_enforce(context: Any, count: int) -> None:
"""Enforce invariants from multiple threads concurrently."""
context.errors = []
context.enforcement_records = []
lock = threading.Lock()
# Get the invariants to enforce
invariants = context.service.list_invariants()
def enforce_invariants(thread_id: int) -> None:
try:
records = context.service.enforce_invariants(
plan_id=f"plan_{thread_id}",
invariants=invariants,
)
with lock:
context.enforcement_records.extend(records)
except Exception as e:
with lock:
context.errors.append(e)
with ThreadPoolExecutor(max_workers=count) as executor:
futures = [executor.submit(enforce_invariants, i) for i in range(count)]
for future in as_completed(futures):
future.result()
@when("{count:d} threads concurrently enforce invariants")
def step_concurrent_enforce_all(context: Any, count: int) -> None:
"""Enforce invariants from multiple threads concurrently."""
context.errors = []
context.enforcement_records = []
lock = threading.Lock()
# Get the invariants to enforce
invariants = context.service.list_invariants()
def enforce_invariants(thread_id: int) -> None:
try:
records = context.service.enforce_invariants(
plan_id=f"plan_{thread_id}",
invariants=invariants,
)
with lock:
context.enforcement_records.extend(records)
except Exception as e:
with lock:
context.errors.append(e)
with ThreadPoolExecutor(max_workers=count) as executor:
futures = [executor.submit(enforce_invariants, i) for i in range(count)]
for future in as_completed(futures):
future.result()
# ================================================================
# Mixed Concurrent Operations
# ================================================================
@when(
"{add_count:d} threads add, {list_count:d} threads list, {remove_count:d} threads remove, and {enforce_count:d} threads enforce concurrently"
)
def step_concurrent_mixed(
context: Any,
add_count: int,
list_count: int,
remove_count: int,
enforce_count: int,
) -> None:
"""Perform mixed operations concurrently from different threads."""
context.errors = []
context.added_ids = []
context.list_results = []
context.removed_ids = []
context.enforcement_records = []
lock = threading.Lock()
remove_index = 0
def add_invariant(thread_id: int) -> None:
try:
inv = context.service.add_invariant(
text=f"Add thread {thread_id}",
scope=InvariantScope.GLOBAL,
source_name="test",
)
with lock:
context.added_ids.append(inv.id)
except Exception as e:
with lock:
context.errors.append(e)
def list_invariants(thread_id: int) -> None:
try:
result = context.service.list_invariants()
with lock:
context.list_results.append(result)
except Exception as e:
with lock:
context.errors.append(e)
def remove_invariant(thread_id: int) -> None:
nonlocal remove_index
try:
with lock:
if remove_index < len(context.invariant_ids):
inv_id = context.invariant_ids[remove_index]
remove_index += 1
else:
return
result = context.service.remove_invariant(inv_id)
with lock:
context.removed_ids.append(result.id)
except Exception as e:
with lock:
context.errors.append(e)
def enforce_invariants(thread_id: int) -> None:
try:
invariants = context.service.list_invariants()
records = context.service.enforce_invariants(
plan_id=f"plan_{thread_id}",
invariants=invariants,
)
with lock:
context.enforcement_records.extend(records)
except Exception as e:
with lock:
context.errors.append(e)
total_threads = add_count + list_count + remove_count + enforce_count
with ThreadPoolExecutor(max_workers=total_threads) as executor:
futures = []
for i in range(add_count):
futures.append(executor.submit(add_invariant, i))
for i in range(list_count):
futures.append(executor.submit(list_invariants, i))
for i in range(remove_count):
futures.append(executor.submit(remove_invariant, i))
for i in range(enforce_count):
futures.append(executor.submit(enforce_invariants, i))
for future in as_completed(futures):
future.result()
@when(
"{thread_count:d} threads perform random operations (add/list/remove/enforce) for {iterations:d} iterations each"
)
def step_concurrent_random_operations(
context: Any, thread_count: int, iterations: int
) -> None:
"""Perform random operations from multiple threads."""
import random
context.errors = []
context.added_ids = []
context.list_results = []
context.removed_ids = []
context.enforcement_records = []
lock = threading.Lock()
def random_operations(thread_id: int) -> None:
try:
for _ in range(iterations):
op = random.choice(["add", "list", "remove", "enforce"])
if op == "add":
inv = context.service.add_invariant(
text=f"Thread {thread_id} Invariant",
scope=InvariantScope.GLOBAL,
source_name="test",
)
with lock:
context.added_ids.append(inv.id)
elif op == "list":
result = context.service.list_invariants()
with lock:
context.list_results.append(result)
elif op == "remove" and context.added_ids:
with lock:
if context.added_ids:
inv_id = context.added_ids.pop(0)
try:
context.service.remove_invariant(inv_id)
with lock:
context.removed_ids.append(inv_id)
except Exception:
pass
elif op == "enforce":
invariants = context.service.list_invariants()
if invariants:
records = context.service.enforce_invariants(
plan_id=f"plan_{thread_id}",
invariants=invariants,
)
with lock:
context.enforcement_records.extend(records)
except Exception as e:
with lock:
context.errors.append(e)
with ThreadPoolExecutor(max_workers=thread_count) as executor:
futures = [executor.submit(random_operations, i) for i in range(thread_count)]
for future in as_completed(futures):
future.result()
# ================================================================
# Concurrent Effective Invariants
# ================================================================
@when("{count:d} threads concurrently get effective invariants")
def step_concurrent_effective(context: Any, count: int) -> None:
"""Get effective invariants from multiple threads concurrently."""
context.errors = []
context.effective_results = []
lock = threading.Lock()
def get_effective(thread_id: int) -> None:
try:
result = context.service.get_effective_invariants()
with lock:
context.effective_results.append(result)
except Exception as e:
with lock:
context.errors.append(e)
with ThreadPoolExecutor(max_workers=count) as executor:
futures = [executor.submit(get_effective, i) for i in range(count)]
for future in as_completed(futures):
future.result()
@when(
"{add_count:d} threads add invariants while {get_count:d} threads get effective invariants concurrently"
)
def step_concurrent_add_and_effective(
context: Any, add_count: int, get_count: int
) -> None:
"""Add invariants and get effective invariants concurrently."""
context.errors = []
context.added_ids = []
context.effective_results = []
lock = threading.Lock()
def add_invariant(thread_id: int) -> None:
try:
inv = context.service.add_invariant(
text=f"Invariant {thread_id}",
scope=InvariantScope.GLOBAL,
source_name="test",
)
with lock:
context.added_ids.append(inv.id)
except Exception as e:
with lock:
context.errors.append(e)
def get_effective(thread_id: int) -> None:
try:
result = context.service.get_effective_invariants()
with lock:
context.effective_results.append(result)
except Exception as e:
with lock:
context.errors.append(e)
with ThreadPoolExecutor(max_workers=add_count + get_count) as executor:
futures = []
for i in range(add_count):
futures.append(executor.submit(add_invariant, i))
for i in range(get_count):
futures.append(executor.submit(get_effective, i))
for future in as_completed(futures):
future.result()
# ================================================================
# Stress Tests
# ================================================================
@when(
"{thread_count:d} threads perform operations concurrently for {iterations:d} iterations each"
)
def step_stress_test(context: Any, thread_count: int, iterations: int) -> None:
"""Perform stress test with many threads and iterations."""
import random
context.errors = []
context.operation_count = 0
lock = threading.Lock()
def stress_operations(thread_id: int) -> None:
try:
for i in range(iterations):
op = random.choice(["add", "list", "enforce"])
if op == "add":
context.service.add_invariant(
text=f"Thread {thread_id} Iter {i}",
scope=InvariantScope.GLOBAL,
source_name="test",
)
elif op == "list":
context.service.list_invariants()
elif op == "enforce":
invariants = context.service.list_invariants()
if invariants:
context.service.enforce_invariants(
plan_id=f"plan_{thread_id}_{i}",
invariants=invariants,
)
with lock:
context.operation_count += 1
except Exception as e:
with lock:
context.errors.append(e)
start_time = time.time()
with ThreadPoolExecutor(max_workers=thread_count) as executor:
futures = [executor.submit(stress_operations, i) for i in range(thread_count)]
for future in as_completed(futures):
future.result()
context.elapsed_time = time.time() - start_time
@when("{count:d} threads perform rapid sequential operations")
def step_rapid_operations(context: Any, count: int) -> None:
"""Perform rapid sequential operations from multiple threads."""
context.errors = []
context.operation_count = 0
lock = threading.Lock()
def rapid_ops(thread_id: int) -> None:
try:
for i in range(100):
context.service.add_invariant(
text=f"Thread {thread_id} Op {i}",
scope=InvariantScope.GLOBAL,
source_name="test",
)
context.service.list_invariants()
with lock:
context.operation_count += 1
except Exception as e:
with lock:
context.errors.append(e)
start_time = time.time()
with ThreadPoolExecutor(max_workers=count) as executor:
futures = [executor.submit(rapid_ops, i) for i in range(count)]
for future in as_completed(futures):
future.result()
context.elapsed_time = time.time() - start_time
# ================================================================
# Assertion Steps
# ================================================================
@then("all invariants should be stored without data loss")
def step_assert_no_data_loss(context: Any) -> None:
"""Assert that all added invariants are stored."""
assert len(context.added_ids) > 0, "No invariants were added"
all_invariants = context.service.list_invariants()
assert len(all_invariants) >= len(context.added_ids), (
f"Expected at least {len(context.added_ids)} invariants, "
f"got {len(all_invariants)}"
)
@then("the total count should be {expected:d}")
def step_assert_count(context: Any, expected: int) -> None:
"""Assert the total count of invariants."""
all_invariants = context.service.list_invariants()
assert len(all_invariants) == expected, (
f"Expected {expected} invariants, got {len(all_invariants)}"
)
@then("no RuntimeError should be raised")
def step_assert_no_runtime_error(context: Any) -> None:
"""Assert that no RuntimeError was raised."""
runtime_errors = [e for e in context.errors if isinstance(e, RuntimeError)]
assert len(runtime_errors) == 0, (
f"Expected no RuntimeError, got {len(runtime_errors)}: {runtime_errors}"
)
@then("all invariants should be retrievable")
def step_assert_retrievable(context: Any) -> None:
"""Assert that all invariants can be retrieved."""
all_invariants = context.service.list_invariants()
assert len(all_invariants) > 0, "No invariants are retrievable"
@then("all threads should complete successfully")
def step_assert_all_threads_complete(context: Any) -> None:
"""Assert that all threads completed without errors."""
assert len(context.errors) == 0, (
f"Expected no errors, got {len(context.errors)}: {context.errors}"
)
@then("each thread should see consistent data")
def step_assert_consistent_data(context: Any) -> None:
"""Assert that each thread saw consistent data."""
if len(context.list_results) > 1:
first_result = context.list_results[0]
for result in context.list_results[1:]:
assert len(result) == len(first_result), (
f"Inconsistent data: expected {len(first_result)} invariants, "
f"got {len(result)}"
)
@then("the final count should match the number of adds")
def step_assert_final_count_matches_adds(context: Any) -> None:
"""Assert that the final count matches the number of adds."""
all_invariants = context.service.list_invariants()
assert len(all_invariants) == len(context.added_ids), (
f"Expected {len(context.added_ids)} invariants, got {len(all_invariants)}"
)
@then("all removals should succeed")
def step_assert_removals_succeed(context: Any) -> None:
"""Assert that all removals succeeded."""
assert len(context.errors) == 0, (
f"Expected no errors, got {len(context.errors)}: {context.errors}"
)
@then("the remaining count should be {expected:d}")
def step_assert_remaining_count(context: Any, expected: int) -> None:
"""Assert the remaining count of invariants."""
all_invariants = context.service.list_invariants()
assert len(all_invariants) == expected, (
f"Expected {expected} remaining invariants, got {len(all_invariants)}"
)
@then("all enforcement records should be stored")
def step_assert_records_stored(context: Any) -> None:
"""Assert that all enforcement records are stored."""
assert len(context.enforcement_records) > 0, "No enforcement records were stored"
@then("the total record count should be {expected:d}")
def step_assert_record_count(context: Any, expected: int) -> None:
"""Assert the total count of enforcement records."""
assert len(context.enforcement_records) == expected, (
f"Expected {expected} records, got {len(context.enforcement_records)}"
)
@then("all {expected:d} enforcement records should be stored")
def step_assert_all_records_stored(context: Any, expected: int) -> None:
"""Assert that all enforcement records are stored."""
assert len(context.enforcement_records) == expected, (
f"Expected {expected} records, got {len(context.enforcement_records)}"
)
@then("no records should be duplicated or lost")
def step_assert_no_duplicates(context: Any) -> None:
"""Assert that no records are duplicated or lost."""
# Check that we have the expected number of records
assert len(context.enforcement_records) > 0, "No records stored"
@then("no errors should occur in thread safety test")
def step_assert_no_errors_thread_safety(context: Any) -> None:
"""Assert that no errors occurred in thread safety test."""
assert len(context.errors) == 0, (
f"Expected no errors, got {len(context.errors)}: {context.errors}"
)
@then("the final state should be consistent")
def step_assert_final_state_consistent(context: Any) -> None:
"""Assert that the final state is consistent."""
# Verify we can list all invariants without errors
all_invariants = context.service.list_invariants()
assert isinstance(all_invariants, list), "list_invariants should return a list"
@then("all operations should complete successfully")
def step_assert_all_operations_complete(context: Any) -> None:
"""Assert that all operations completed successfully."""
assert len(context.errors) == 0, (
f"Expected no errors, got {len(context.errors)}: {context.errors}"
)
@then("the final invariant count should match expected value")
def step_assert_final_invariant_count(context: Any) -> None:
"""Assert that the final invariant count is valid."""
all_invariants = context.service.list_invariants()
assert isinstance(all_invariants, list), "list_invariants should return a list"
assert len(all_invariants) >= 0, "Invariant count should be non-negative"
@then("all enforcement records should be accounted for")
def step_assert_records_accounted(context: Any) -> None:
"""Assert that all enforcement records are accounted for."""
assert len(context.enforcement_records) >= 0, "Record count should be non-negative"
@then("all threads should see consistent merged results")
def step_assert_consistent_merged_results(context: Any) -> None:
"""Assert that all threads see consistent merged results."""
if len(context.effective_results) > 1:
first_result = context.effective_results[0]
for result in context.effective_results[1:]:
assert len(result) == len(first_result), (
f"Inconsistent merged results: expected {len(first_result)}, "
f"got {len(result)}"
)
@then("the results should be consistent")
def step_assert_results_consistent(context: Any) -> None:
"""Assert that the results are consistent."""
assert len(context.errors) == 0, (
f"Expected no errors, got {len(context.errors)}: {context.errors}"
)
@then("no deadlocks should occur")
def step_assert_no_deadlocks(context: Any) -> None:
"""Assert that no deadlocks occurred."""
# If we got here, no deadlocks occurred
assert True
@then("the final state should be valid")
def step_assert_final_state_valid(context: Any) -> None:
"""Assert that the final state is valid."""
all_invariants = context.service.list_invariants()
assert isinstance(all_invariants, list), "list_invariants should return a list"
@then("all operations should complete within reasonable time")
def step_assert_reasonable_time(context: Any) -> None:
"""Assert that operations completed within reasonable time."""
# Reasonable time: less than 30 seconds for the test
assert context.elapsed_time < 30, (
f"Operations took too long: {context.elapsed_time:.2f}s"
)
@then("no thread should be starved")
def step_assert_no_starvation(context: Any) -> None:
"""Assert that no thread was starved."""
# If we got here and all operations completed, no starvation occurred
assert context.operation_count > 0, "No operations were performed"
@@ -14,11 +14,19 @@ a dict keyed by invariant ID.
Effective invariants are computed using plan > project > global order.
See ``merge_invariants`` for de-duplication semantics.
## Thread Safety
All public methods are protected by a reentrant lock (``threading.RLock``)
to ensure safe concurrent access to ``_invariants`` dict and
``_enforcement_records`` list. This allows the service to be safely shared
across threads during parallel plan execution.
Based on ``docs/specification.md`` and implementation plan Stage M3.5.
"""
from __future__ import annotations
import threading
from typing import TYPE_CHECKING
import structlog
@@ -54,6 +62,7 @@ class InvariantService:
Args:
event_bus: Optional EventBus for domain event emission.
"""
self._lock: threading.RLock = threading.RLock()
self._invariants: dict[str, Invariant] = {}
self._enforcement_records: list[InvariantEnforcementRecord] = []
self._logger = logger.bind(service="invariant")
@@ -95,7 +104,8 @@ class InvariantService:
source_name=source_name.strip(),
)
self._invariants[invariant.id] = invariant
with self._lock:
self._invariants[invariant.id] = invariant
self._logger.info(
"Invariant added",
invariant_id=invariant.id,
@@ -127,13 +137,14 @@ class InvariantService:
project_name=source_name if scope == InvariantScope.PROJECT else None,
)
result = [inv for inv in self._invariants.values() if inv.active]
with self._lock:
result = [inv for inv in self._invariants.values() if inv.active]
if scope is not None:
result = [inv for inv in result if inv.scope == scope]
if scope is not None:
result = [inv for inv in result if inv.scope == scope]
if source_name is not None:
result = [inv for inv in result if inv.source_name == source_name]
if source_name is not None:
result = [inv for inv in result if inv.source_name == source_name]
return result
@@ -152,15 +163,16 @@ class InvariantService:
if not invariant_id or not invariant_id.strip():
raise ValidationError("Invariant ID must not be empty")
inv = self._invariants.get(invariant_id)
if inv is None:
raise NotFoundError(
resource_type="invariant",
resource_id=invariant_id,
)
# Invariant is frozen (immutable); create a new instance with active=False
deactivated = inv.model_copy(update={"active": False})
self._invariants[invariant_id] = deactivated
with self._lock:
inv = self._invariants.get(invariant_id)
if inv is None:
raise NotFoundError(
resource_type="invariant",
resource_id=invariant_id,
)
# Invariant is frozen (immutable); create a new instance with active=False
deactivated = inv.model_copy(update={"active": False})
self._invariants[invariant_id] = deactivated
self._logger.info("Invariant removed (soft-delete)", invariant_id=invariant_id)
return deactivated
@@ -183,21 +195,22 @@ class InvariantService:
Returns:
Merged, de-duplicated list of effective invariants.
"""
active = [inv for inv in self._invariants.values() if inv.active]
with self._lock:
active = [inv for inv in self._invariants.values() if inv.active]
plan_invs = [
inv
for inv in active
if inv.scope == InvariantScope.PLAN
and (plan_id is None or inv.source_name == plan_id)
]
project_invs = [
inv
for inv in active
if inv.scope == InvariantScope.PROJECT
and (project_name is None or inv.source_name == project_name)
]
global_invs = [inv for inv in active if inv.scope == InvariantScope.GLOBAL]
plan_invs = [
inv
for inv in active
if inv.scope == InvariantScope.PLAN
and (plan_id is None or inv.source_name == plan_id)
]
project_invs = [
inv
for inv in active
if inv.scope == InvariantScope.PROJECT
and (project_name is None or inv.source_name == project_name)
]
global_invs = [inv for inv in active if inv.scope == InvariantScope.GLOBAL]
return merge_invariants(plan_invs, project_invs, global_invs)
@@ -264,7 +277,8 @@ class InvariantService:
exc_info=True,
)
self._enforcement_records.extend(records)
with self._lock:
self._enforcement_records.extend(records)
self._logger.info(
"Invariants enforced",
plan_id=plan_id,