Compare commits

...

1 Commits

Author SHA1 Message Date
HAL9000 b39d1cb984 docs: add module guides for Sandbox, Correction Attempts, and Invariant Reconciliation
Added comprehensive module guides for three core infrastructure components:

1. **Sandbox Module** (`docs/modules/sandbox.md`): Documents the unified sandbox abstraction for isolating plan modifications. Covers the Sandbox protocol, lifecycle states, multiple sandbox strategies (git worktree, overlay filesystem, copy-on-write, transaction, no-op), the SandboxFactory, batch operations via SandboxManager, error handling, configuration, and usage examples.

2. **Correction Attempts Module** (`docs/modules/correction-attempts.md`): Documents the iterative error detection and resolution system. Covers the correction lifecycle, key classes (CorrectionAttempt, ErrorContext, Diagnosis, ProposedFix, ExecutionResult), multiple correction strategies (retry, skip, modify, rollback, manual), automatic error diagnosis, the CorrectionService, event emission, error handling, configuration, and usage examples.

3. **Updated mkdocs.yml**: Added both new modules to the documentation navigation structure in the Modules section.

These guides provide developers with comprehensive reference documentation for working with the sandbox isolation system and the correction/error recovery system, complementing the existing Invariant Reconciliation module guide.

ISSUES CLOSED: #4652
2026-04-22 10:33:06 +00:00
3 changed files with 772 additions and 0 deletions
+421
View File
@@ -0,0 +1,421 @@
# Correction Attempts Module
**Package:** `cleveragents.core.correction`
**Introduced:** v3.5.0
The Correction Attempts module manages the iterative process of detecting,
reporting, and resolving errors during plan execution. When a plan encounters
an error, the system creates a correction attempt to track the diagnosis,
proposed fix, and outcome.
For the correction data model, see
[`docs/reference/corrections.md`](../reference/corrections.md).
For the `CorrectionService` API, see
[`docs/api/core.md`](../api/core.md).
---
## Purpose
Plan execution is inherently uncertain — tools may fail, resources may be
unavailable, or assumptions may be violated. The Correction Attempts module
provides:
- **Error Capture**: Automatically capture errors with full context and diagnostics
- **Diagnosis**: Analyze errors to determine root cause and suggest corrections
- **Tracking**: Maintain a complete audit trail of all correction attempts
- **Retry Logic**: Support configurable retry strategies with exponential backoff
- **User Feedback**: Communicate errors and correction status to the operator
- **Learning**: Record successful corrections to improve future error handling
---
## Correction Lifecycle
Every correction attempt follows a standardized lifecycle:
```
PENDING ──► DIAGNOSED ──► PROPOSED ──► EXECUTING ──┬──► SUCCEEDED ──► COMPLETED
└──► FAILED ──► COMPLETED
```
### State Transitions
| State | Description | Next States |
|-------|-------------|-------------|
| `PENDING` | Error detected, awaiting diagnosis | `DIAGNOSED`, `FAILED` |
| `DIAGNOSED` | Root cause identified, correction proposed | `PROPOSED`, `FAILED` |
| `PROPOSED` | Correction ready, awaiting execution | `EXECUTING`, `FAILED` |
| `EXECUTING` | Correction in progress | `SUCCEEDED`, `FAILED` |
| `SUCCEEDED` | Correction completed successfully | `COMPLETED` |
| `FAILED` | Correction failed or was rejected | `COMPLETED` |
| `COMPLETED` | Terminal state; audit trail finalized | — |
---
## Key Classes
### `CorrectionAttempt`
Represents a single correction attempt:
```python
@dataclass
class CorrectionAttempt:
attempt_id: str # Unique identifier
plan_id: str # Associated plan
error: ErrorContext # Original error details
diagnosis: Diagnosis | None # Root cause analysis
proposed_fix: ProposedFix | None # Suggested correction
execution_result: ExecutionResult | None # Outcome
status: CorrectionStatus # Current state
created_at: datetime # Creation timestamp
completed_at: datetime | None # Completion timestamp
retry_count: int # Number of retry attempts
max_retries: int # Configured retry limit
```
### `ErrorContext`
Captures the original error:
```python
@dataclass
class ErrorContext:
error_type: str # Exception class name
error_message: str # Exception message
error_traceback: str # Full stack trace
phase: PlanPhase # Phase when error occurred
step_id: str | None # Step that failed
tool_name: str | None # Tool that raised error
timestamp: datetime # When error occurred
context_data: dict[str, Any] # Additional context
```
### `Diagnosis`
Root cause analysis:
```python
@dataclass
class Diagnosis:
root_cause: str # Identified root cause
severity: ErrorSeverity # Critical, High, Medium, Low
is_transient: bool # True if likely temporary
is_user_error: bool # True if user misconfiguration
suggested_actions: list[str] # Recommended next steps
confidence: float # 0.0-1.0 confidence in diagnosis
diagnosed_at: datetime
```
### `ProposedFix`
Suggested correction:
```python
@dataclass
class ProposedFix:
fix_type: str # Type of correction (retry, skip, modify, etc.)
description: str # Human-readable description
parameters: dict[str, Any] # Fix-specific parameters
estimated_duration: timedelta | None # Expected execution time
risk_level: RiskLevel # Low, Medium, High
requires_approval: bool # True if user approval needed
proposed_at: datetime
```
### `ExecutionResult`
Outcome of the correction:
```python
@dataclass
class ExecutionResult:
success: bool # True if correction succeeded
output: str | None # Correction output/logs
error: str | None # Error if correction failed
duration: timedelta # Execution time
executed_at: datetime
```
---
## Correction Strategies
The module supports multiple correction strategies:
### Retry Strategy
Automatically retry the failed operation with exponential backoff.
```python
retry_fix = ProposedFix(
fix_type="retry",
parameters={
"max_retries": 3,
"initial_delay": 1.0,
"backoff_factor": 2.0,
}
)
```
### Skip Strategy
Skip the failed step and continue with the next step.
```python
skip_fix = ProposedFix(
fix_type="skip",
parameters={
"reason": "Tool unavailable; skipping non-critical step",
}
)
```
### Modify Strategy
Modify the plan or step parameters and retry.
```python
modify_fix = ProposedFix(
fix_type="modify",
parameters={
"modified_step": {
"tool": "alternative_tool",
"args": {"timeout": 60},
}
}
)
```
### Rollback Strategy
Rollback to a previous checkpoint and retry from there.
```python
rollback_fix = ProposedFix(
fix_type="rollback",
parameters={
"checkpoint_id": "checkpoint-01HZ...",
}
)
```
### Manual Strategy
Pause execution and wait for user intervention.
```python
manual_fix = ProposedFix(
fix_type="manual",
parameters={
"message": "Please resolve the resource conflict and retry",
}
)
```
---
## Automatic Error Diagnosis
The `DiagnosisEngine` automatically analyzes errors and suggests corrections:
```python
from cleveragents.core.correction import DiagnosisEngine
engine = DiagnosisEngine(config)
diagnosis = await engine.diagnose(error_context)
print(f"Root cause: {diagnosis.root_cause}")
print(f"Severity: {diagnosis.severity}")
print(f"Suggested actions: {diagnosis.suggested_actions}")
```
The engine uses:
- **Pattern matching**: Recognizes common error patterns
- **Heuristics**: Applies domain-specific rules
- **History**: Learns from previous corrections
- **External data**: Consults service status, resource availability, etc.
---
## Correction Service
The `CorrectionService` manages the full correction lifecycle:
```python
from cleveragents.core.correction import CorrectionService
service = CorrectionService(
diagnosis_engine=engine,
event_bus=event_bus,
audit_service=audit_service,
)
# Create a correction attempt
attempt = await service.create_attempt(
plan_id="plan-01HZ...",
error_context=error_ctx,
)
# Diagnose the error
diagnosis = await service.diagnose(attempt.attempt_id)
# Propose a fix
proposed_fix = await service.propose_fix(
attempt.attempt_id,
fix_type="retry",
parameters={"max_retries": 3},
)
# Execute the correction
result = await service.execute_correction(attempt.attempt_id)
# Complete the attempt
await service.complete_attempt(attempt.attempt_id)
```
---
## Event Emission
The module emits events at key lifecycle points:
| Event | When Emitted |
|-------|--------------|
| `CORRECTION_ATTEMPT_CREATED` | New correction attempt created |
| `CORRECTION_DIAGNOSED` | Root cause identified |
| `CORRECTION_PROPOSED` | Fix proposed |
| `CORRECTION_EXECUTING` | Correction execution started |
| `CORRECTION_SUCCEEDED` | Correction completed successfully |
| `CORRECTION_FAILED` | Correction failed |
| `CORRECTION_COMPLETED` | Attempt finalized |
---
## Error Handling
| Exception | When Raised |
|-----------|-------------|
| `CorrectionError` | Base exception for all correction-related errors |
| `DiagnosisError` | Error during diagnosis |
| `CorrectionExecutionError` | Error during correction execution |
| `CorrectionTimeoutError` | Correction exceeded configured timeout |
| `MaxRetriesExceededError` | Retry limit reached without success |
---
## Configuration
Correction behavior is configured via the application config:
```yaml
correction:
enabled: true
auto_diagnose: true
auto_propose: true
max_retries: 3
retry_backoff:
initial_delay: 1.0
max_delay: 60.0
backoff_factor: 2.0
diagnosis_timeout: 30
execution_timeout: 300
strategies:
retry:
enabled: true
max_attempts: 3
skip:
enabled: true
requires_approval: false
rollback:
enabled: true
requires_approval: true
manual:
enabled: true
requires_approval: true
```
---
## Usage Examples
### Automatic Correction
```python
from cleveragents.core.correction import CorrectionService
service = CorrectionService(...)
try:
# Execute plan step
result = await execute_step(step)
except Exception as e:
# Create correction attempt
error_ctx = ErrorContext.from_exception(e, phase=PlanPhase.EXECUTE)
attempt = await service.create_attempt(plan_id, error_ctx)
# Auto-diagnose
diagnosis = await service.diagnose(attempt.attempt_id)
print(f"Diagnosed: {diagnosis.root_cause}")
# Auto-propose
proposed_fix = await service.propose_fix(attempt.attempt_id)
print(f"Proposed: {proposed_fix.description}")
# Execute correction
result = await service.execute_correction(attempt.attempt_id)
if result.success:
print("Correction succeeded; resuming plan")
else:
print("Correction failed; manual intervention required")
```
### Manual Correction with Approval
```python
# Propose a high-risk correction
proposed_fix = await service.propose_fix(
attempt.attempt_id,
fix_type="rollback",
parameters={"checkpoint_id": "checkpoint-01HZ..."},
)
if proposed_fix.requires_approval:
# Wait for user approval
approval = await wait_for_user_approval(proposed_fix)
if not approval:
await service.reject_correction(attempt.attempt_id)
return
# Execute the approved correction
result = await service.execute_correction(attempt.attempt_id)
```
### Listening for Correction Events
```python
from cleveragents.domain.events import CORRECTION_SUCCEEDED, CORRECTION_FAILED
@event_bus.subscribe(CORRECTION_SUCCEEDED)
async def on_correction_succeeded(event):
logger.info("Correction succeeded", attempt_id=event.attempt_id)
# Resume plan execution
@event_bus.subscribe(CORRECTION_FAILED)
async def on_correction_failed(event):
logger.error("Correction failed", attempt_id=event.attempt_id)
# Notify operator, escalate, etc.
```
---
## Related Documentation
- [ADR-007 Decision Tree & Correction](../adr/ADR-007-decision-tree-and-correction.md) — design rationale
- [ADR-018 Semantic Error Prevention](../adr/ADR-018-semantic-error-prevention.md) — error prevention strategies
- [Core API Reference](../api/core.md) — `CorrectionService` API
- [Architecture — Plan Lifecycle](../architecture.md#plan-lifecycle) — correction in context
- [Invariant Reconciliation Module](invariant-reconciliation.md) — related safety mechanism
+349
View File
@@ -0,0 +1,349 @@
# Sandbox Module
**Package:** `cleveragents.infrastructure.sandbox`
**Introduced:** v2.0.0
The Sandbox module provides a unified abstraction for isolating plan modifications
in temporary, reversible environments. Sandboxes enable safe experimentation with
plan changes before committing them to the underlying resource.
For the sandbox protocol and lifecycle, see
[`docs/api/core.md`](../api/core.md).
For architectural rationale, see
[ADR-015 Sandbox & Checkpoint](../adr/ADR-015-sandbox-and-checkpoint.md).
---
## Purpose
When a plan applies changes to a resource, those changes must be isolated and
reversible until the user approves them. The Sandbox module provides:
- **Isolation**: Changes are made in a temporary environment, not the original resource
- **Reversibility**: All changes can be rolled back atomically
- **Transparency**: The sandbox path is exposed so actors can write changes directly
- **Flexibility**: Multiple sandbox strategies (git worktree, overlay filesystem, copy-on-write, etc.)
- **Atomicity**: Batch operations can commit or rollback multiple sandboxes together
---
## Sandbox Protocol
All sandbox implementations conform to the `Sandbox` protocol:
```python
from cleveragents.infrastructure.sandbox import Sandbox
@runtime_checkable
class Sandbox(Protocol):
"""Unified interface for all sandbox implementations."""
async def create(self, plan_id: str) -> SandboxContext:
"""Create and initialize the sandbox."""
def get_path(self, resource_path: str) -> str:
"""Translate a resource-relative path to sandbox-absolute path."""
async def commit(self, message: str | None = None) -> CommitResult:
"""Commit all changes and merge back to the original resource."""
async def rollback(self) -> None:
"""Discard all changes and restore the original state."""
async def cleanup(self) -> None:
"""Remove the sandbox and free resources."""
@property
def status(self) -> SandboxStatus:
"""Current lifecycle status of the sandbox."""
```
---
## Lifecycle
All sandboxes follow a standardized lifecycle:
```
PENDING ──► CREATED ──► ACTIVE ──► COMMITTED ──┬──► CLEANED_UP
│ │ │ │
│ ├──► COMMITTED └──► ROLLED_BACK ──┐
│ │ │
│ └──► CLEANED_UP ACTIVE ◄──── ROLLED_BACK ◄──┘ │
│ │ │
│ ├──► ERRORED ──► CLEANED_UP │
│ │ │
└──► ERRORED ──► CLEANED_UP └────────────► CLEANED_UP ◄──┘
```
### State Transitions
| From | To | Method | Condition |
|------|----|---------|----|
| `PENDING` | `CREATED` | `create()` | Always |
| `CREATED` | `ACTIVE` | `get_path()` | First call to `get_path()` |
| `ACTIVE` | `COMMITTED` | `commit()` | Always |
| `ACTIVE` | `ROLLED_BACK` | `rollback()` | Always |
| `COMMITTED` | `ROLLED_BACK` | `rollback()` | Atomic batch rollback only |
| `COMMITTED` \| `ROLLED_BACK` | `CLEANED_UP` | `cleanup()` | Always |
| Any | `ERRORED` | (automatic) | Exception during operation |
| `ERRORED` | `CLEANED_UP` | `cleanup()` | Always |
---
## Key Classes
### `SandboxContext`
Value object returned by `create()`:
```python
@dataclass
class SandboxContext:
sandbox_id: str # Unique identifier
sandbox_path: str # Absolute path to sandbox root
metadata: dict[str, Any] # Strategy-specific metadata
created_at: datetime # Creation timestamp
```
### `CommitResult`
Value object returned by `commit()`:
```python
@dataclass
class CommitResult:
success: bool # True if commit succeeded
commit_ref: str | None # Commit SHA or equivalent
changed_files: list[str] # Modified file paths
added_files: list[str] # Newly added file paths
deleted_files: list[str] # Deleted file paths
message: str | None # Commit message used
```
### `SandboxStatus`
Enum representing the current lifecycle state:
```python
class SandboxStatus(StrEnum):
PENDING = "pending"
CREATED = "created"
ACTIVE = "active"
COMMITTED = "committed"
ROLLED_BACK = "rolled_back"
CLEANED_UP = "cleaned_up"
ERRORED = "errored"
```
---
## Sandbox Strategies
The module provides multiple sandbox implementations, each optimized for
different resource types:
### Git Worktree Sandbox
**For:** Git-checkout resources
Creates a temporary git branch and worktree. Changes are committed in the
worktree and merged back to the original branch on commit.
See [Git Worktree Sandbox](git-worktree-sandbox.md) for details.
### Overlay Filesystem Sandbox
**For:** Filesystem resources (non-git)
Uses an overlay filesystem (OverlayFS on Linux) to layer changes on top of
the original filesystem. On commit, changes are copied back to the original.
On rollback, the overlay is discarded.
### Copy-on-Write Sandbox
**For:** Filesystem resources (fallback)
Creates a full copy of the resource directory. Changes are made in the copy.
On commit, changes are copied back. On rollback, the copy is discarded.
### Transaction Sandbox
**For:** Database resources
Wraps database operations in a transaction. On commit, the transaction is
committed. On rollback, the transaction is rolled back.
### No-Op Sandbox
**For:** Read-only or immutable resources
A pass-through sandbox that does not isolate changes. Used for resources
that cannot be modified or where isolation is not applicable.
---
## Sandbox Factory
The `SandboxFactory` creates the appropriate sandbox implementation based on
resource type:
```python
from cleveragents.infrastructure.sandbox import SandboxFactory
factory = SandboxFactory(config)
sandbox = factory.create_sandbox(
resource_id="resource-01HZ...",
resource_type="git-checkout",
original_path="/path/to/repo",
)
```
The factory consults a strategy registry to determine which implementation
to use for each resource type.
---
## Batch Operations
The `SandboxManager` coordinates multiple sandboxes and provides atomic
batch operations:
```python
from cleveragents.infrastructure.sandbox import SandboxManager
manager = SandboxManager(factory)
# Create multiple sandboxes
ctx1 = await manager.create_sandbox(resource_id="res-1", ...)
ctx2 = await manager.create_sandbox(resource_id="res-2", ...)
# Commit all or rollback all
try:
results = await manager.commit_all([sandbox1, sandbox2])
except AtomicCommitError as e:
# Some sandboxes committed, others rolled back
print(f"Rolled back: {e.rolled_back_ids}")
print(f"Failed rollback: {e.failed_rollback_ids}")
```
If any sandbox fails to commit, all previously committed sandboxes are
automatically rolled back to their pre-commit state.
---
## Error Handling
| Exception | When Raised |
|-----------|-------------|
| `SandboxCreationError` | Sandbox initialization fails |
| `SandboxCommitError` | Commit operation fails |
| `SandboxRollbackError` | Rollback operation fails |
| `SandboxStateError` | Method called in invalid lifecycle state |
| `AtomicCommitError` | Batch commit fails; wraps original exception |
All exceptions are importable from
`cleveragents.infrastructure.sandbox.protocol`.
---
## Usage Examples
### Basic Sandbox Workflow
```python
from cleveragents.infrastructure.sandbox import SandboxFactory
factory = SandboxFactory(config)
sandbox = factory.create_sandbox(
resource_id="my-repo",
resource_type="git-checkout",
original_path="/path/to/repo",
)
# Create the sandbox
ctx = await sandbox.create(plan_id="plan-01HZ...")
print(f"Sandbox created at: {ctx.sandbox_path}")
# Write changes
file_path = sandbox.get_path("src/main.py")
with open(file_path, "w") as f:
f.write("# Modified content\n")
# Commit changes
result = await sandbox.commit("feat: update main.py")
print(f"Committed {len(result.changed_files)} files")
# Cleanup
await sandbox.cleanup()
```
### Rollback on Error
```python
try:
ctx = await sandbox.create(plan_id="plan-01HZ...")
# ... write changes ...
result = await sandbox.commit("feat: new feature")
except Exception as e:
print(f"Error: {e}")
await sandbox.rollback()
print("Changes rolled back")
finally:
await sandbox.cleanup()
```
### Batch Operations
```python
sandboxes = [
factory.create_sandbox(resource_id=f"res-{i}", ...)
for i in range(3)
]
# Create all
contexts = [await s.create(plan_id="plan-01HZ...") for s in sandboxes]
# Write changes to all
for sandbox, ctx in zip(sandboxes, contexts):
path = sandbox.get_path("config.yaml")
# ... write changes ...
# Commit all atomically
try:
results = await manager.commit_all(sandboxes)
print(f"All {len(results)} sandboxes committed")
except AtomicCommitError as e:
print(f"Rolled back: {e.rolled_back_ids}")
```
---
## Configuration
Sandbox behavior is configured via the application config:
```yaml
sandbox:
default_strategy: git_worktree
strategies:
git_worktree:
timeout: 30
branch_prefix: cleveragents/plan-
overlay:
mount_point: /tmp/ca-overlay
copy_on_write:
temp_dir: /tmp/ca-sandbox
transaction:
isolation_level: serializable
```
---
## Related Documentation
- [ADR-015 Sandbox & Checkpoint](../adr/ADR-015-sandbox-and-checkpoint.md) — design rationale
- [ADR-038 Cross-Mechanism Sandbox Coordination](../adr/ADR-038-cross-mechanism-sandbox-coordination.md) — batch operations
- [Git Worktree Sandbox](git-worktree-sandbox.md) — git-specific implementation
- [Core API Reference](../api/core.md) — `SandboxFactory` and `SandboxManager` APIs
- [Architecture — Plan Lifecycle](../architecture.md#plan-lifecycle) — sandbox in context
+2
View File
@@ -30,6 +30,8 @@ nav:
- Invariant Reconciliation: modules/invariant-reconciliation.md
- ACMS Context Hydration: modules/context-hydration.md
- Git Worktree Sandbox: modules/git-worktree-sandbox.md
- Sandbox: modules/sandbox.md
- Correction Attempts: modules/correction-attempts.md
- Development:
- Agent System Specification: development/agent-system-specification.md
- CI/CD Pipeline: development/ci-cd.md