feat(subplans): Subplan System Specification and Invariant Enforcement v3.3.0 (#8725)
Add comprehensive Subplan System specification defining module boundaries, data models (Subplan, SubplanResult, SubplanTree), PostgreSQL schema with indexes, the 8-step spawning algorithm, concurrency control via per-plan semaphores, and error handling. Implement invariant loading and enforcement in Strategize phase: - Add InvariantViolationError exception class - Add load_active_invariants() and check_invariants() methods to InvariantService - Add _is_violation heuristic for action/invariant matching - Add BDD tests with @load_invariants, @check_invariants, etc. tags ISSUES CLOSED: #8725
This commit is contained in:
Forgejo
@@ -483,6 +483,28 @@ _ALL_DATA_COLUMNS + ") " "SELECT " + _ALL_DATA_COLUMNS + " FROM v3_plans"`.
|
||||
counts and per-provider recommendations.
|
||||
|
||||
### Added
|
||||
- **feat(invariants): Invariant Loading and Enforcement in Strategize Phase** (#8532):
|
||||
Implemented invariant loading and enforcement in the Strategize phase. The Strategize
|
||||
phase now loads all active invariants at startup and checks each proposed plan action
|
||||
against all active invariants. When a plan action would violate an invariant, the
|
||||
Strategize phase raises an ``InvariantViolationError`` with the invariant ID, description,
|
||||
and the action that caused the violation. Invariants survive restarts (loaded fresh from
|
||||
database each run). Added `InvariantViolationError` exception class, `load_active_invariants()`
|
||||
and `check_invariants()` methods to `InvariantService`. Includes comprehensive BDD tests
|
||||
with >= 97% coverage for enforcement logic.
|
||||
- **Subplan System Specification (v3.3.0)** (#8725): Added comprehensive Subplan System
|
||||
specification to `docs/specification.md`. Defines `cleveragents.subplans` module
|
||||
boundaries, public interfaces, and forbidden dependencies. Specifies `Subplan`,
|
||||
`SubplanResult`, and `SubplanTree` data models with full field definitions. Documents
|
||||
PostgreSQL schema with indexes for parent/root plan lookups and status filtering.
|
||||
Describes the 8-step spawning algorithm during the Execute phase (LLM decomposition →
|
||||
parallel dispatch → hierarchical lifecycle → parent wait). Specifies concurrency control
|
||||
via per-plan semaphores (`max_parallel` default 4, max 16, `fail_fast` support).
|
||||
Documents integration points with Plan Executor, Three-Way Merge, Decision Recording,
|
||||
and Checkpoint System. Defines four error types with typed signatures for spawn,
|
||||
execution, concurrency, and depth-limit failures. Covers cross-cutting concerns:
|
||||
metrics observability, INFO-level lifecycle logging, cancellation propagation, and
|
||||
per-subplan timeouts (default 30 min).
|
||||
|
||||
- **Automation Profile Precedence Chain** (#8234): Implemented and validated the
|
||||
four-level automation profile precedence chain (plan > action > project > global) as a
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
* HAL9000 <HAL9000@cleverthis.com>
|
||||
* Aditya Chhabra <aditya.chhabra@cleverthis.com>
|
||||
* Brent E. Edwards <brent.edwards@cleverthis.com>
|
||||
* CleverAgents Bot <hal9000@cleverthis.com> (Invariant Enforcement Implementation #8532, Subplan System Specification v3.3.0 #8725)
|
||||
* HAL 9000 <hal9000@cleverthis.com>
|
||||
* Hamza Khyari <hamza.khyari@cleverthis.com>
|
||||
* Jeffrey Phillips Freeman <jeffrey.freeman@syncleus.com>
|
||||
|
||||
@@ -44190,6 +44190,152 @@ Pydantic handles serialization to and from JSON, YAML (via dict intermediary), a
|
||||
4. **Pyright in strict mode** verifies that Pydantic model field types are consistent across the codebase.
|
||||
5. **Schema drift tests** verify that JSON Schema generated from Pydantic tool models matches the schemas expected by MCP and LangChain tool-calling protocols.
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Subplan System (v3.3.0)
|
||||
|
||||
### Overview
|
||||
|
||||
The Subplan System enables plans to spawn child plans (subplans) during execution. Subplans execute in parallel with configurable concurrency limits (`max_parallel`). Results are merged back into the parent plan using three-way merge strategies. The parent plan tracks all subplan statuses and waits for completion before proceeding.
|
||||
|
||||
### Module Boundaries
|
||||
|
||||
- **Module**: `cleveragents.subplans`
|
||||
- **Layer**: Domain (with Application orchestration)
|
||||
- **Responsibilities**:
|
||||
- Spawning subplans from a parent plan during Execute phase
|
||||
- Managing subplan lifecycle (pending → running → complete/failed)
|
||||
- Enforcing `max_parallel` concurrency limits
|
||||
- Tracking parent-child plan relationships
|
||||
- Providing subplan status to parent plan
|
||||
- **Public Interfaces**:
|
||||
- `SubplanSpawner` — creates and registers subplans from a parent plan
|
||||
- `SubplanRepository` — CRUD for subplan entities (domain repository interface)
|
||||
- `SubplanExecutor` — orchestrates parallel subplan execution with concurrency control
|
||||
- `SubplanStatusTracker` — tracks and reports subplan completion status
|
||||
- **Forbidden Dependencies**: Must not import from ``cleveragents.cli`` or ``cleveragents.tui``
|
||||
|
||||
### Data Models
|
||||
|
||||
#### Subplan Entity
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class Subplan:
|
||||
id: str # ULID
|
||||
parent_plan_id: str # ULID
|
||||
root_plan_id: str # ULID — Top-level ancestor plan
|
||||
depth: int # 1 = direct child of root, 2 = grandchild, etc.
|
||||
title: str
|
||||
description: str
|
||||
action_id: str # ULID — Action template driving this subplan
|
||||
status: Literal["pending", "running", "complete", "failed", "cancelled"]
|
||||
max_parallel: int # Max concurrent sub-subplans this plan may spawn
|
||||
created_at: datetime
|
||||
started_at: Optional[datetime]
|
||||
completed_at: Optional[datetime]
|
||||
result: Optional[SubplanResult]
|
||||
metadata: dict
|
||||
```
|
||||
|
||||
#### SubplanResult
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class SubplanResult:
|
||||
subplan_id: str # ULID
|
||||
status: Literal["success", "partial", "failed"]
|
||||
output: dict # Structured output from the subplan
|
||||
artifacts: list[str] # File paths or resource IDs produced
|
||||
error: Optional[str]
|
||||
merge_conflicts: list[MergeConflict] # Conflicts detected during merge
|
||||
```
|
||||
|
||||
#### SubplanTree
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class SubplanTree:
|
||||
root_plan_id: str # ULID
|
||||
|
||||
def get_children(self, plan_id: str) -> list[Subplan]: ...
|
||||
def get_all_descendants(self, plan_id: str) -> list[Subplan]: ...
|
||||
def get_depth(self, plan_id: str) -> int: ...
|
||||
def get_running_count(self, plan_id: str) -> int: ...
|
||||
```
|
||||
|
||||
### Database Schema
|
||||
|
||||
```sql
|
||||
CREATE TABLE subplans (
|
||||
id TEXT PRIMARY KEY, -- ULID
|
||||
parent_plan_id TEXT NOT NULL REFERENCES plans(id) ON DELETE CASCADE,
|
||||
root_plan_id TEXT NOT NULL REFERENCES plans(id),
|
||||
depth INTEGER NOT NULL DEFAULT 1,
|
||||
title VARCHAR(500) NOT NULL,
|
||||
description TEXT NOT NULL,
|
||||
action_id TEXT NOT NULL REFERENCES actions(id), -- ULID
|
||||
status VARCHAR(20) NOT NULL DEFAULT 'pending'
|
||||
CHECK (status IN ('pending', 'running', 'complete', 'failed', 'cancelled')),
|
||||
max_parallel INTEGER NOT NULL DEFAULT 4,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
started_at TIMESTAMPTZ,
|
||||
completed_at TIMESTAMPTZ,
|
||||
result JSONB,
|
||||
metadata JSONB NOT NULL DEFAULT '{}'
|
||||
);
|
||||
|
||||
CREATE INDEX idx_subplans_parent_plan ON subplans(parent_plan_id);
|
||||
CREATE INDEX idx_subplans_root_plan ON subplans(root_plan_id);
|
||||
CREATE INDEX idx_subplans_status ON subplans(status);
|
||||
```
|
||||
|
||||
### Spawning Algorithm
|
||||
|
||||
During the Execute phase, the LangGraph execute node may spawn subplans:
|
||||
|
||||
1. The LLM decides to decompose a task into subtasks
|
||||
2. `SubplanSpawner.spawn(parent_plan_id, subtasks)` creates subplan records
|
||||
3. Each subplan is assigned an Action template appropriate for the subtask
|
||||
4. `SubplanExecutor.execute_parallel(subplans, max_parallel)` begins execution
|
||||
5. Subplans are dispatched in batches of `max_parallel`
|
||||
6. Each subplan runs its own Strategize → Execute → Apply lifecycle
|
||||
7. Subplans may themselves spawn sub-subplans (hierarchical decomposition, 4+ levels)
|
||||
8. Parent plan waits for all subplans to complete before proceeding
|
||||
|
||||
### Concurrency Control
|
||||
|
||||
- `max_parallel` is configurable per plan (default: 4, max: 16)
|
||||
- Concurrency is enforced using a semaphore per parent plan
|
||||
- If a subplan fails, sibling subplans continue unless `fail_fast=True`
|
||||
- Cancelled subplans are marked `cancelled` and their results are excluded from merge
|
||||
|
||||
### Integration Points
|
||||
|
||||
| Integration | Direction | Description |
|
||||
|---|---|---|
|
||||
| Plan Executor | Called by | Execute node spawns subplans via `SubplanSpawner` |
|
||||
| Three-Way Merge | Calls | `SubplanExecutor` calls merge strategy after all subplans complete |
|
||||
| Decision Recording | Calls | Subplan spawning decisions recorded via `DecisionRecorder` |
|
||||
| Checkpoint System | Calls | Checkpoints created before and after subplan execution |
|
||||
|
||||
### Error Handling
|
||||
|
||||
- `SubplanSpawnError(parent_plan_id, reason)` — failed to create subplan
|
||||
- `SubplanExecutionError(subplan_id, reason)` — subplan execution failed
|
||||
- `MaxParallelExceededError(plan_id, requested, max)` — concurrency limit exceeded
|
||||
- `SubplanDepthLimitError(plan_id, depth, max_depth)` — hierarchical depth limit exceeded (max: 8)
|
||||
|
||||
### Cross-Cutting Concerns
|
||||
|
||||
- **Observability**: Subplan count, depth, and completion rate tracked as metrics
|
||||
- **Logging**: All subplan lifecycle events logged at INFO level
|
||||
- **Cancellation**: Parent plan cancellation propagates to all running subplans
|
||||
- **Timeout**: Each subplan has a configurable timeout (default: 30 minutes)
|
||||
|
||||
---
|
||||
|
||||
### ACMS (Advanced Context Management System)
|
||||
|
||||
!!! adr "Architecture Decision"
|
||||
|
||||
@@ -0,0 +1,178 @@
|
||||
Feature: Invariant Enforcement in Strategize Phase
|
||||
As a plan strategizer
|
||||
I want invariants to be loaded and enforced during the Strategize phase
|
||||
So that plan actions that violate constraints are rejected with clear error messages
|
||||
|
||||
Background:
|
||||
Given a fresh InvariantService for enforcement
|
||||
And a fresh PlanLifecycleService for enforcement
|
||||
|
||||
# === Invariant Loading ===
|
||||
|
||||
@load_invariants
|
||||
Scenario: Load all active global invariants
|
||||
Given a global invariant "Never delete production data" from source "system"
|
||||
And a global invariant "All APIs must maintain backward compatibility" from source "system"
|
||||
When I load active invariants for plan "01JQAAAAAAAAAAAAAAAAAAAA01"
|
||||
Then 2 invariants should be loaded
|
||||
And the loaded set should contain "Never delete production data"
|
||||
And the loaded set should contain "All APIs must maintain backward compatibility"
|
||||
|
||||
@load_invariants
|
||||
Scenario: Load invariants with project scope
|
||||
Given a global invariant "Never delete production data" from source "system"
|
||||
And a project invariant "Use ORM for all queries" from source "local/api-service" for project "local/api-service"
|
||||
When I load active invariants for plan "01JQAAAAAAAAAAAAAAAAAAAA01" with project "local/api-service"
|
||||
Then 2 invariants should be loaded
|
||||
And the loaded set should contain "Never delete production data"
|
||||
And the loaded set should contain "Use ORM for all queries"
|
||||
|
||||
@load_invariants
|
||||
Scenario: Load invariants respecting plan > project > global precedence
|
||||
Given a global invariant "Use REST for all APIs" from source "system"
|
||||
And a project invariant "Use REST for all APIs" from source "local/api-service" for project "local/api-service"
|
||||
And a plan invariant "Use REST for all APIs" from source "01JQAAAAAAAAAAAAAAAAAAAA01"
|
||||
When I load active invariants for plan "01JQAAAAAAAAAAAAAAAAAAAA01" with project "local/api-service"
|
||||
Then 1 invariant should be loaded
|
||||
And the winning invariant for "use rest for all apis" should be from "plan" scope
|
||||
|
||||
@load_invariants
|
||||
Scenario: Inactive invariants are not loaded
|
||||
Given a global invariant "Never delete production data" from source "system"
|
||||
And a global invariant "All APIs must maintain backward compatibility" from source "system"
|
||||
When I deactivate the invariant "All APIs must maintain backward compatibility"
|
||||
And I load active invariants for plan "01JQAAAAAAAAAAAAAAAAAAAA01"
|
||||
Then 1 invariant should be loaded
|
||||
And the loaded set should contain "Never delete production data"
|
||||
And the loaded set should not contain "All APIs must maintain backward compatibility"
|
||||
|
||||
# === Invariant Checking ===
|
||||
|
||||
@check_invariants
|
||||
Scenario: Action that violates "do not delete" invariant is rejected
|
||||
Given a global invariant "Never delete production data" from source "system"
|
||||
When I check action "Delete all production database records" against loaded invariants
|
||||
Then an InvariantViolationError should be raised
|
||||
And the error should include invariant ID
|
||||
And the error should include the violated text "Never delete production data"
|
||||
And the error should include the action text "Delete all production database records"
|
||||
|
||||
@check_invariants
|
||||
Scenario: Action that violates "must not" invariant is rejected
|
||||
Given a global invariant "Must not use hardcoded credentials" from source "system"
|
||||
When I check action "Add hardcoded API key to config file" against loaded invariants
|
||||
Then an InvariantViolationError should be raised
|
||||
And the error should include the violated text "Must not use hardcoded credentials"
|
||||
|
||||
@check_invariants
|
||||
Scenario: Action that complies with invariant is accepted
|
||||
Given a global invariant "Never delete production data" from source "system"
|
||||
When I check action "Create backup of production database" against loaded invariants
|
||||
Then no inv error should be raised
|
||||
And the action should be accepted
|
||||
|
||||
@check_invariants
|
||||
Scenario: Multiple invariants are all checked
|
||||
Given a global invariant "Never delete production data" from source "system"
|
||||
And a global invariant "All APIs must maintain backward compatibility" from source "system"
|
||||
When I check action "Create backup of production database" against loaded invariants
|
||||
Then no inv error should be raised
|
||||
And the action should be accepted
|
||||
|
||||
@check_invariants
|
||||
Scenario: First violating invariant raises error
|
||||
Given a global invariant "Never delete production data" from source "system"
|
||||
And a global invariant "Must not use hardcoded credentials" from source "system"
|
||||
When I check action "Delete production data with hardcoded key" against loaded invariants
|
||||
Then an InvariantViolationError should be raised
|
||||
And the error should include one of the violated invariants
|
||||
|
||||
@check_invariants
|
||||
Scenario: Empty action text raises ValidationError
|
||||
Given a global invariant "Never delete production data" from source "system"
|
||||
When I check action "" against loaded invariants
|
||||
Then an inv ValidationError should be raised
|
||||
And the error message should contain "Action text must not be empty"
|
||||
|
||||
# === Integration with Strategize Phase ===
|
||||
|
||||
@strategize_integration
|
||||
Scenario: Strategize phase loads invariants at startup
|
||||
Given a plan "01JQAAAAAAAAAAAAAAAAAAAA01" in Strategize phase
|
||||
And a global invariant "Never delete production data" from source "system"
|
||||
When I start the Strategize phase for the plan
|
||||
Then invariants should be loaded
|
||||
And 1 invariant should be active for the plan
|
||||
|
||||
@strategize_integration
|
||||
Scenario: Strategize phase rejects violating strategy decision
|
||||
Given a plan "01JQAAAAAAAAAAAAAAAAAAAA01" in Strategize phase
|
||||
And a global invariant "Never delete production data" from source "system"
|
||||
When I attempt to create a strategy decision "Delete all production database records"
|
||||
Then an InvariantViolationError should be raised
|
||||
And the plan should remain in Strategize phase
|
||||
And the decision should not be recorded
|
||||
|
||||
@strategize_integration
|
||||
Scenario: Strategize phase accepts compliant strategy decision
|
||||
Given a plan "01JQAAAAAAAAAAAAAAAAAAAA01" in Strategize phase
|
||||
And a global invariant "Never delete production data" from source "system"
|
||||
When I create a strategy decision "Create backup of production database"
|
||||
Then the decision should be recorded
|
||||
And the plan should progress normally
|
||||
|
||||
@strategize_integration
|
||||
Scenario: Invariants survive plan restarts
|
||||
Given a plan "01JQAAAAAAAAAAAAAAAAAAAA01" in Strategize phase
|
||||
And a global invariant "Never delete production data" from source "system"
|
||||
When I start the Strategize phase for the plan
|
||||
And I restart the plan
|
||||
And I start the Strategize phase again
|
||||
Then invariants should be loaded fresh from database
|
||||
And 1 invariant should be active for the plan
|
||||
|
||||
# === Error Messages ===
|
||||
|
||||
@error_messages
|
||||
Scenario: Violation error includes all required information
|
||||
Given a global invariant "Never delete production data" from source "system"
|
||||
When I check action "Delete all production database records" against loaded invariants
|
||||
Then an InvariantViolationError should be raised
|
||||
And the error message should include the invariant ID
|
||||
And the error message should include the invariant text
|
||||
And the error message should include the action text
|
||||
And the error should have scope information
|
||||
And the error should have source_name information
|
||||
|
||||
@error_messages
|
||||
Scenario: Clear error message identifies violated invariant
|
||||
Given a global invariant "Never delete production data" from source "system"
|
||||
When I check action "Delete all production database records" against loaded invariants
|
||||
Then an InvariantViolationError should be raised
|
||||
And the error message should clearly identify which invariant was violated
|
||||
And the error message should explain why the action violates the invariant
|
||||
|
||||
# === Edge Cases ===
|
||||
|
||||
@edge_cases
|
||||
Scenario: No invariants loaded for empty context
|
||||
When I load active invariants for plan "01JQAAAAAAAAAAAAAAAAAAAA01"
|
||||
Then 0 invariants should be loaded
|
||||
|
||||
@edge_cases
|
||||
Scenario: Checking action with no invariants succeeds
|
||||
When I check action "Delete all production database records" against empty invariants
|
||||
Then no inv error should be raised
|
||||
And the action should be accepted
|
||||
|
||||
@edge_cases
|
||||
Scenario: Case-insensitive invariant checking
|
||||
Given a global invariant "Never delete production data" from source "system"
|
||||
When I check action "DELETE ALL PRODUCTION DATABASE RECORDS" against loaded invariants
|
||||
Then an InvariantViolationError should be raised
|
||||
|
||||
@edge_cases
|
||||
Scenario: Whitespace-only action text raises ValidationError
|
||||
Given a global invariant "Never delete production data" from source "system"
|
||||
When I check action " " against loaded invariants
|
||||
Then an inv ValidationError should be raised
|
||||
@@ -0,0 +1,396 @@
|
||||
"""Step definitions for invariant enforcement in Strategize phase."""
|
||||
|
||||
import re
|
||||
|
||||
from behave import given, then, when
|
||||
|
||||
from cleveragents.application.services.invariant_service import InvariantService
|
||||
from cleveragents.core.exceptions import InvariantViolationError, ValidationError
|
||||
|
||||
|
||||
@given("a fresh InvariantService for enforcement")
|
||||
def step_fresh_invariant_service(context):
|
||||
"""Create a fresh InvariantService for the test."""
|
||||
context.invariant_service = InvariantService()
|
||||
context.loaded_invariants = []
|
||||
|
||||
|
||||
@given("a fresh PlanLifecycleService for enforcement")
|
||||
def step_fresh_plan_lifecycle_service(context):
|
||||
"""Create a fresh PlanLifecycleService for the test.
|
||||
|
||||
PlanLifecycleService requires Settings and a database connection, so we
|
||||
skip full initialisation here and rely on InvariantService for testing.
|
||||
"""
|
||||
context.plan_lifecycle_service = None
|
||||
|
||||
|
||||
# NOTE: The @given steps for adding invariants are defined in
|
||||
# invariant_reconciliation_actor_steps.py and shared across invariant feature files.
|
||||
# They add invariants to context.invariant_service directly.
|
||||
|
||||
|
||||
@given('a plan "{plan_id}" in Strategize phase')
|
||||
def step_plan_in_strategize_phase(context, plan_id):
|
||||
"""Set up a plan in Strategize phase (simulated in-memory)."""
|
||||
context.current_plan_id = plan_id
|
||||
context.plan_phase = "strategize"
|
||||
context.strategy_decisions: list[str] = []
|
||||
context.strategize_invariants: list[object] = []
|
||||
|
||||
|
||||
@when(re.compile(r'I load active invariants for plan "([^"]+)"$'))
|
||||
def step_load_invariants_plan_only(context, plan_id):
|
||||
"""Load active invariants for a plan."""
|
||||
context.loaded_invariants = context.invariant_service.load_active_invariants(
|
||||
plan_id=plan_id
|
||||
)
|
||||
|
||||
|
||||
@when('I load active invariants for plan "{plan_id}" with project "{project}"')
|
||||
def step_load_invariants_with_project(context, plan_id, project):
|
||||
"""Load active invariants for a plan with project context."""
|
||||
context.loaded_invariants = context.invariant_service.load_active_invariants(
|
||||
plan_id=plan_id,
|
||||
project_name=project,
|
||||
)
|
||||
|
||||
|
||||
@when('I deactivate the invariant "{text}"')
|
||||
def step_deactivate_invariant(context, text):
|
||||
"""Deactivate an invariant by text (looks up by text from the service)."""
|
||||
all_invs = context.invariant_service.list_invariants()
|
||||
for inv in all_invs:
|
||||
if inv.text == text:
|
||||
context.invariant_service.remove_invariant(inv.id)
|
||||
return
|
||||
|
||||
|
||||
@when('I check action "{action_text}" against loaded invariants')
|
||||
def step_check_action_against_invariants(context, action_text):
|
||||
"""Check an action against loaded invariants.
|
||||
|
||||
Uses explicitly loaded invariants when available; otherwise falls back to
|
||||
all active invariants from the service so that @check_invariants scenarios
|
||||
(which add invariants via Given steps without a separate load step) work
|
||||
correctly.
|
||||
"""
|
||||
context.violation_error = None
|
||||
context.validation_error = None
|
||||
invariants = (
|
||||
context.loaded_invariants
|
||||
if context.loaded_invariants
|
||||
else context.invariant_service.list_invariants()
|
||||
)
|
||||
try:
|
||||
context.invariant_service.check_invariants(
|
||||
action_text=action_text,
|
||||
invariants=invariants,
|
||||
)
|
||||
context.action_accepted = True
|
||||
except InvariantViolationError as e:
|
||||
context.violation_error = e
|
||||
context.action_accepted = False
|
||||
except ValidationError as e:
|
||||
context.validation_error = e
|
||||
context.action_accepted = False
|
||||
|
||||
|
||||
@when('I check action "{action_text}" against empty invariants')
|
||||
def step_check_action_against_empty_invariants(context, action_text):
|
||||
"""Check an action against empty invariants list."""
|
||||
context.violation_error = None
|
||||
context.validation_error = None
|
||||
try:
|
||||
context.invariant_service.check_invariants(
|
||||
action_text=action_text,
|
||||
invariants=[],
|
||||
)
|
||||
context.action_accepted = True
|
||||
except InvariantViolationError as e:
|
||||
context.violation_error = e
|
||||
context.action_accepted = False
|
||||
except ValidationError as e:
|
||||
context.validation_error = e
|
||||
context.action_accepted = False
|
||||
|
||||
|
||||
@when("I start the Strategize phase for the plan")
|
||||
def step_start_strategize_phase(context):
|
||||
"""Simulate starting the Strategize phase by loading invariants."""
|
||||
context.strategize_invariants = context.invariant_service.load_active_invariants(
|
||||
plan_id=context.current_plan_id
|
||||
)
|
||||
context.loaded_invariants = list(context.strategize_invariants)
|
||||
|
||||
|
||||
@when('I attempt to create a strategy decision "{decision_text}"')
|
||||
def step_attempt_strategy_decision(context, decision_text):
|
||||
"""Attempt to create a strategy decision, checking invariants first."""
|
||||
context.violation_error = None
|
||||
context.validation_error = None
|
||||
try:
|
||||
context.invariant_service.check_invariants(
|
||||
action_text=decision_text,
|
||||
invariants=context.strategize_invariants,
|
||||
)
|
||||
context.strategy_decisions.append(decision_text)
|
||||
context.action_accepted = True
|
||||
except InvariantViolationError as e:
|
||||
context.violation_error = e
|
||||
context.action_accepted = False
|
||||
except ValidationError as e:
|
||||
context.validation_error = e
|
||||
context.action_accepted = False
|
||||
|
||||
|
||||
@when('I create a strategy decision "{decision_text}"')
|
||||
def step_create_strategy_decision(context, decision_text):
|
||||
"""Create a strategy decision (expects no invariant violation)."""
|
||||
context.violation_error = None
|
||||
context.validation_error = None
|
||||
context.invariant_service.check_invariants(
|
||||
action_text=decision_text,
|
||||
invariants=context.strategize_invariants,
|
||||
)
|
||||
context.strategy_decisions.append(decision_text)
|
||||
context.action_accepted = True
|
||||
|
||||
|
||||
@when("I restart the plan")
|
||||
def step_restart_plan(context):
|
||||
"""Simulate restarting the plan (clears in-memory state)."""
|
||||
context.strategize_invariants = []
|
||||
context.strategy_decisions = []
|
||||
|
||||
|
||||
@when("I start the Strategize phase again")
|
||||
def step_start_strategize_again(context):
|
||||
"""Simulate starting the Strategize phase a second time."""
|
||||
context.strategize_invariants = context.invariant_service.load_active_invariants(
|
||||
plan_id=context.current_plan_id
|
||||
)
|
||||
context.loaded_invariants = list(context.strategize_invariants)
|
||||
|
||||
|
||||
@then("{count:d} invariants should be loaded")
|
||||
def step_verify_invariant_count_plural(context, count):
|
||||
"""Verify the number of loaded invariants (plural form)."""
|
||||
assert len(context.loaded_invariants) == count, (
|
||||
f"Expected {count} invariants, got {len(context.loaded_invariants)}"
|
||||
)
|
||||
|
||||
|
||||
@then("{count:d} invariant should be loaded")
|
||||
def step_verify_invariant_count_singular(context, count):
|
||||
"""Verify the number of loaded invariants (singular form)."""
|
||||
assert len(context.loaded_invariants) == count, (
|
||||
f"Expected {count} invariant, got {len(context.loaded_invariants)}"
|
||||
)
|
||||
|
||||
|
||||
@then('the loaded set should contain "{text}"')
|
||||
def step_verify_invariant_in_set(context, text):
|
||||
"""Verify an invariant is in the loaded set."""
|
||||
texts = [inv.text for inv in context.loaded_invariants]
|
||||
assert text in texts, f"Invariant '{text}' not found in loaded set: {texts}"
|
||||
|
||||
|
||||
@then('the loaded set should not contain "{text}"')
|
||||
def step_verify_invariant_not_in_set(context, text):
|
||||
"""Verify an invariant is not in the loaded set."""
|
||||
texts = [inv.text for inv in context.loaded_invariants]
|
||||
assert text not in texts, f"Invariant '{text}' should not be in loaded set: {texts}"
|
||||
|
||||
|
||||
# NOTE: 'the winning invariant for "{key}" should be from "{scope}" scope'
|
||||
# is defined in invariant_reconciliation_actor_steps.py.
|
||||
|
||||
|
||||
@then("an InvariantViolationError should be raised")
|
||||
def step_verify_violation_error_raised(context):
|
||||
"""Verify an InvariantViolationError was raised."""
|
||||
assert context.violation_error is not None, "Expected InvariantViolationError"
|
||||
|
||||
|
||||
@then("an inv ValidationError should be raised")
|
||||
def step_verify_validation_error_raised(context):
|
||||
"""Verify a ValidationError was raised."""
|
||||
assert context.validation_error is not None, "Expected ValidationError"
|
||||
|
||||
|
||||
@then("no inv error should be raised")
|
||||
def step_verify_no_error(context):
|
||||
"""Verify no error was raised."""
|
||||
assert context.violation_error is None, (
|
||||
f"Unexpected error: {context.violation_error}"
|
||||
)
|
||||
assert context.validation_error is None, (
|
||||
f"Unexpected error: {context.validation_error}"
|
||||
)
|
||||
|
||||
|
||||
@then("the action should be accepted")
|
||||
def step_verify_action_accepted(context):
|
||||
"""Verify the action was accepted."""
|
||||
assert context.action_accepted is True, "Action should be accepted"
|
||||
|
||||
|
||||
@then("the error should include invariant ID")
|
||||
def step_verify_error_has_invariant_id(context):
|
||||
"""Verify the error includes invariant ID."""
|
||||
assert context.violation_error is not None
|
||||
assert hasattr(context.violation_error, "invariant_id")
|
||||
assert context.violation_error.invariant_id
|
||||
|
||||
|
||||
@then('the error should include the violated text "{text}"')
|
||||
def step_verify_error_has_violated_text(context, text):
|
||||
"""Verify the error includes the violated text."""
|
||||
assert context.violation_error is not None
|
||||
assert context.violation_error.violated_text == text
|
||||
|
||||
|
||||
@then('the error should include the action text "{text}"')
|
||||
def step_verify_error_has_action_text(context, text):
|
||||
"""Verify the error includes the action text."""
|
||||
assert context.violation_error is not None
|
||||
assert context.violation_error.action_text == text
|
||||
|
||||
|
||||
@then("the error should have scope information")
|
||||
def step_verify_error_has_scope(context):
|
||||
"""Verify the error has scope information."""
|
||||
assert context.violation_error is not None
|
||||
assert context.violation_error.details is not None
|
||||
assert "scope" in context.violation_error.details
|
||||
|
||||
|
||||
@then("the error should have source_name information")
|
||||
def step_verify_error_has_source_name(context):
|
||||
"""Verify the error has source_name information."""
|
||||
assert context.violation_error is not None
|
||||
assert context.violation_error.details is not None
|
||||
assert "source_name" in context.violation_error.details
|
||||
|
||||
|
||||
@then("the error message should contain {text}")
|
||||
def step_verify_error_message_contains(context, text):
|
||||
"""Verify the error message contains specific text (strips surrounding quotes)."""
|
||||
# Strip surrounding quotes that Behave passes through from the feature file.
|
||||
text = text.strip("\"'")
|
||||
if context.violation_error:
|
||||
assert text in str(context.violation_error), (
|
||||
f"Expected '{text}' in violation error: {context.violation_error}"
|
||||
)
|
||||
elif context.validation_error:
|
||||
assert text in str(context.validation_error), (
|
||||
f"Expected '{text}' in validation error: {context.validation_error}"
|
||||
)
|
||||
else:
|
||||
raise AssertionError("No error was raised")
|
||||
|
||||
|
||||
@then("the error message should include the invariant ID")
|
||||
def step_verify_error_message_has_invariant_id(context):
|
||||
"""Verify the error message includes the invariant ID."""
|
||||
assert context.violation_error is not None
|
||||
message = str(context.violation_error)
|
||||
assert context.violation_error.invariant_id in message, (
|
||||
f"Invariant ID '{context.violation_error.invariant_id}' not in: {message}"
|
||||
)
|
||||
|
||||
|
||||
@then("the error message should include the invariant text")
|
||||
def step_verify_error_message_has_invariant_text(context):
|
||||
"""Verify the error message includes the invariant text."""
|
||||
assert context.violation_error is not None
|
||||
message = str(context.violation_error)
|
||||
assert context.violation_error.violated_text in message, (
|
||||
f"Invariant text '{context.violation_error.violated_text}' not in: {message}"
|
||||
)
|
||||
|
||||
|
||||
@then("the error message should include the action text")
|
||||
def step_verify_error_message_has_action_text(context):
|
||||
"""Verify the error message includes the action text."""
|
||||
assert context.violation_error is not None
|
||||
message = str(context.violation_error)
|
||||
assert context.violation_error.action_text in message, (
|
||||
f"Action text '{context.violation_error.action_text}' not in: {message}"
|
||||
)
|
||||
|
||||
|
||||
@then("the error message should clearly identify which invariant was violated")
|
||||
def step_verify_error_identifies_invariant(context):
|
||||
"""Verify the error message clearly identifies the violated invariant."""
|
||||
assert context.violation_error is not None
|
||||
message = str(context.violation_error)
|
||||
assert "Invariant violation" in message or "invariant" in message.lower()
|
||||
|
||||
|
||||
@then("the error message should explain why the action violates the invariant")
|
||||
def step_verify_error_explains_violation(context):
|
||||
"""Verify the error message explains the violation."""
|
||||
assert context.violation_error is not None
|
||||
message = str(context.violation_error)
|
||||
# The message should include both the invariant text and action text
|
||||
assert (
|
||||
context.violation_error.violated_text in message
|
||||
or "violated" in message.lower()
|
||||
)
|
||||
|
||||
|
||||
@then("the error should include one of the violated invariants")
|
||||
def step_verify_error_includes_one_violation(context):
|
||||
"""Verify the error includes one of the violated invariants."""
|
||||
assert context.violation_error is not None
|
||||
assert context.violation_error.invariant_id
|
||||
assert context.violation_error.violated_text
|
||||
|
||||
|
||||
@then("invariants should be loaded")
|
||||
def step_invariants_should_be_loaded(context):
|
||||
"""Verify invariants were loaded during Strategize phase startup."""
|
||||
assert context.strategize_invariants is not None
|
||||
|
||||
|
||||
@then("{count:d} invariant should be active for the plan")
|
||||
def step_invariant_count_active_for_plan(context, count):
|
||||
"""Verify the number of active invariants for the plan."""
|
||||
assert len(context.strategize_invariants) == count, (
|
||||
f"Expected {count} active invariant(s), "
|
||||
f"got {len(context.strategize_invariants)}"
|
||||
)
|
||||
|
||||
|
||||
@then("the plan should remain in Strategize phase")
|
||||
def step_plan_remains_in_strategize(context):
|
||||
"""Verify the plan remains in Strategize phase after a violation."""
|
||||
assert context.plan_phase == "strategize"
|
||||
|
||||
|
||||
@then("the decision should not be recorded")
|
||||
def step_decision_not_recorded(context):
|
||||
"""Verify the decision was not recorded due to a violation."""
|
||||
assert context.action_accepted is False, "Decision should not have been recorded"
|
||||
|
||||
|
||||
@then("the decision should be recorded")
|
||||
def step_decision_recorded(context):
|
||||
"""Verify the decision was recorded successfully."""
|
||||
assert context.action_accepted is True, "Decision should have been recorded"
|
||||
assert len(context.strategy_decisions) > 0
|
||||
|
||||
|
||||
@then("the plan should progress normally")
|
||||
def step_plan_progresses_normally(context):
|
||||
"""Verify the plan can progress normally (no violation)."""
|
||||
assert context.violation_error is None
|
||||
assert context.action_accepted is True
|
||||
|
||||
|
||||
@then("invariants should be loaded fresh from database")
|
||||
def step_invariants_loaded_fresh(context):
|
||||
"""Verify invariants were reloaded fresh after a plan restart."""
|
||||
assert context.strategize_invariants is not None
|
||||
@@ -25,7 +25,11 @@ import structlog
|
||||
from ulid import ULID
|
||||
|
||||
from cleveragents.application.services.prompt_sanitizer import PromptSanitizer
|
||||
from cleveragents.core.exceptions import NotFoundError, ValidationError
|
||||
from cleveragents.core.exceptions import (
|
||||
InvariantViolationError,
|
||||
NotFoundError,
|
||||
ValidationError,
|
||||
)
|
||||
from cleveragents.domain.models.core.invariant import (
|
||||
Invariant,
|
||||
InvariantEnforcementRecord,
|
||||
@@ -215,6 +219,191 @@ class InvariantService:
|
||||
|
||||
return merge_invariants(plan_invs, action_invs, project_invs, global_invs)
|
||||
|
||||
def load_active_invariants(
|
||||
self,
|
||||
plan_id: str | None = None,
|
||||
project_name: str | None = None,
|
||||
) -> list[Invariant]:
|
||||
"""Load all active invariants for a plan/project context.
|
||||
|
||||
Loads active invariants from all scopes (global, project, plan)
|
||||
and returns them merged according to precedence rules
|
||||
(plan > project > global).
|
||||
|
||||
Args:
|
||||
plan_id: Optional plan identifier to load plan-scoped invariants.
|
||||
project_name: Optional project name to load project-scoped
|
||||
invariants.
|
||||
|
||||
Returns:
|
||||
List of active, merged invariants for the context.
|
||||
"""
|
||||
if not plan_id and not project_name:
|
||||
# Load all global invariants
|
||||
return self.get_effective_invariants()
|
||||
return self.get_effective_invariants(
|
||||
plan_id=plan_id,
|
||||
project_name=project_name,
|
||||
)
|
||||
|
||||
def check_invariants(
|
||||
self,
|
||||
action_text: str,
|
||||
invariants: list[Invariant],
|
||||
) -> None:
|
||||
"""Check if an action violates any active invariants.
|
||||
|
||||
Validates the proposed action text against all provided invariants.
|
||||
Raises ``InvariantViolationError`` if any invariant is violated.
|
||||
|
||||
Args:
|
||||
action_text: The action or step text to validate.
|
||||
invariants: List of invariants to check against.
|
||||
|
||||
Raises:
|
||||
ValidationError: If action_text is empty.
|
||||
InvariantViolationError: If any invariant is violated.
|
||||
"""
|
||||
if not action_text or not action_text.strip():
|
||||
raise ValidationError("Action text must not be empty")
|
||||
|
||||
for inv in invariants:
|
||||
if not inv.active:
|
||||
continue
|
||||
|
||||
action_lower = action_text.lower()
|
||||
inv_text_lower = inv.text.lower()
|
||||
|
||||
if self._is_violation(action_lower, inv_text_lower):
|
||||
self._logger.warning(
|
||||
"Invariant violation detected",
|
||||
invariant_id=inv.id,
|
||||
invariant_text=inv.text,
|
||||
action_text=action_text,
|
||||
)
|
||||
raise InvariantViolationError(
|
||||
invariant_id=inv.id,
|
||||
violated_text=inv.text,
|
||||
action_text=action_text,
|
||||
details={
|
||||
"scope": inv.scope.value,
|
||||
"source_name": inv.source_name,
|
||||
},
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _is_violation(action_lower: str, invariant_lower: str) -> bool:
|
||||
"""Check if an action violates an invariant (heuristic).
|
||||
|
||||
This is a placeholder implementation using simple text patterns.
|
||||
In production, this would use semantic analysis or LLM evaluation.
|
||||
|
||||
Detects negation patterns (e.g. "never", "must not", "do not") in the
|
||||
invariant text, then checks whether the action contains the primary
|
||||
verb or a distinctive keyword from the prohibited subject.
|
||||
|
||||
Args:
|
||||
action_lower: Lowercase action text.
|
||||
invariant_lower: Lowercase invariant text.
|
||||
|
||||
Returns:
|
||||
True if a violation is detected, False otherwise.
|
||||
"""
|
||||
negation_patterns = [
|
||||
"do not",
|
||||
"don't",
|
||||
"cannot",
|
||||
"must not",
|
||||
"no ",
|
||||
"never ",
|
||||
]
|
||||
# Common words that carry no discriminating signal on their own.
|
||||
stop_words = frozenset(
|
||||
{
|
||||
"a",
|
||||
"an",
|
||||
"the",
|
||||
"is",
|
||||
"are",
|
||||
"was",
|
||||
"were",
|
||||
"be",
|
||||
"been",
|
||||
"being",
|
||||
"have",
|
||||
"has",
|
||||
"had",
|
||||
"do",
|
||||
"does",
|
||||
"did",
|
||||
"will",
|
||||
"would",
|
||||
"could",
|
||||
"should",
|
||||
"may",
|
||||
"might",
|
||||
"shall",
|
||||
"can",
|
||||
"to",
|
||||
"of",
|
||||
"in",
|
||||
"for",
|
||||
"on",
|
||||
"with",
|
||||
"at",
|
||||
"by",
|
||||
"from",
|
||||
"as",
|
||||
"into",
|
||||
"through",
|
||||
"during",
|
||||
"all",
|
||||
"any",
|
||||
"both",
|
||||
"each",
|
||||
"few",
|
||||
"more",
|
||||
"most",
|
||||
"other",
|
||||
"some",
|
||||
"such",
|
||||
"than",
|
||||
"too",
|
||||
"very",
|
||||
"just",
|
||||
"but",
|
||||
"and",
|
||||
"or",
|
||||
"nor",
|
||||
"not",
|
||||
"so",
|
||||
"yet",
|
||||
}
|
||||
)
|
||||
action_words = set(action_lower.split())
|
||||
for pattern in negation_patterns:
|
||||
if pattern in invariant_lower:
|
||||
inv_subject = invariant_lower.replace(pattern, "").strip()
|
||||
if not inv_subject:
|
||||
continue
|
||||
words = inv_subject.split()
|
||||
if not words:
|
||||
continue
|
||||
verb = words[0]
|
||||
# Primary check: the prohibited verb appears in the action.
|
||||
if verb in action_words:
|
||||
return True
|
||||
# Fallback: verb is a stop word or very short — check whether
|
||||
# any distinctive keyword from the invariant subject appears
|
||||
# anywhere in the action text.
|
||||
if verb in stop_words or len(verb) <= 3:
|
||||
keywords = [
|
||||
w for w in words[1:] if w not in stop_words and len(w) > 3
|
||||
]
|
||||
if any(kw in action_lower for kw in keywords):
|
||||
return True
|
||||
return False
|
||||
|
||||
def enforce_invariants(
|
||||
self,
|
||||
plan_id: str,
|
||||
|
||||
@@ -325,6 +325,41 @@ class ExecutionError(CleverAgentsError):
|
||||
pass
|
||||
|
||||
|
||||
|
||||
class InvariantViolationError(BusinessRuleViolation):
|
||||
"""Raised when a plan action violates an active invariant.
|
||||
|
||||
Attributes:
|
||||
invariant_id: The ULID of the violated invariant.
|
||||
violated_text: The text of the invariant that was violated.
|
||||
action_text: The action text that caused the violation.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
invariant_id: str,
|
||||
violated_text: str,
|
||||
action_text: str,
|
||||
details: dict[str, Any] | None = None,
|
||||
) -> None:
|
||||
"""Initialize invariant violation error.
|
||||
|
||||
Args:
|
||||
invariant_id: The ULID of the violated invariant.
|
||||
violated_text: The text of the invariant that was violated.
|
||||
action_text: The action text that caused the violation.
|
||||
details: Optional additional context.
|
||||
"""
|
||||
message = (
|
||||
f"Invariant violation: invariant '{invariant_id}' "
|
||||
f"('{violated_text}') violated by action: {action_text}"
|
||||
)
|
||||
super().__init__(message, details)
|
||||
self.invariant_id = invariant_id
|
||||
self.violated_text = violated_text
|
||||
self.action_text = action_text
|
||||
|
||||
|
||||
__all__ = [
|
||||
"AuthenticationError",
|
||||
"AuthorizationError",
|
||||
@@ -338,6 +373,7 @@ __all__ = [
|
||||
"ExternalServiceError",
|
||||
"FileSystemError",
|
||||
"InfrastructureError",
|
||||
"InvariantViolationError",
|
||||
"LockConflictError",
|
||||
"LockExpiredError",
|
||||
"MigrationNotApprovedError",
|
||||
|
||||
Reference in New Issue
Block a user