Compare commits
10 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 7861991043 | |||
| d79380074e | |||
| 75eb1ae3da | |||
| ec2a7ca63a | |||
| 73de1a214a | |||
| 5126ab16f6 | |||
| 474e161dd0 | |||
| ac24164f81 | |||
| a5d2eed3d9 | |||
| 2dc4a9e2e7 |
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,537 @@
|
||||
## Behavior
|
||||
|
||||
### Automation Profiles
|
||||
|
||||
!!! adr "Architecture Decision"
|
||||
The automation profile system, confidence thresholds, and built-in profiles are defined in [ADR-017: Automation Profiles](adr/ADR-017-automation-profiles.md).
|
||||
|
||||
Automation profiles determine which phase transitions happen automatically and which require human approval.
|
||||
|
||||
#### Overview
|
||||
|
||||
An **automation profile** is a named collection of **confidence thresholds** — floating-point values from 0.0 to 1.0 inclusive — that controls which tasks are automated vs. require human approval. Each threshold specifies the minimum confidence score (as computed by the Semantic Escalation system) at which the system proceeds automatically. When the computed confidence for a given operation falls below the profile's threshold for that flag, the system drops to manual mode and requests human input. A threshold of **0.0** means "always automatic" (proceed regardless of confidence), while **1.0** means "always manual" (always require human approval). Profiles follow the same `<namespace>/<name>` naming convention as actors, tools, skills, and other entities. Profiles are managed via the `agents automation-profile` CLI commands.
|
||||
|
||||
#### Automatable Tasks
|
||||
|
||||
Each automation profile specifies a **confidence threshold** (a floating-point value from 0.0 to 1.0 inclusive) for each of the following automatable tasks. The threshold determines the minimum confidence level at which the system proceeds automatically. When the computed confidence (from Semantic Escalation) falls below the threshold, the system drops to manual mode for that task. Setting a threshold to **0.0** makes the task always automatic; setting it to **1.0** makes it always manual.
|
||||
|
||||
| Flag | Type | Description | Behavior |
|
||||
|------|------|-------------|----------|
|
||||
| `decompose_task` | float (0.0–1.0) | Automatically enter Strategize after `plan use` | When confidence >= threshold, Strategize begins immediately. Below threshold, system pauses after plan creation for user confirmation before entering Strategize. |
|
||||
| `create_tool` | float (0.0–1.0) | Automatically proceed from Strategize to Execute | When confidence >= threshold, Execute begins when Strategize completes. Below threshold, system pauses for user review. |
|
||||
| `select_tool` | float (0.0–1.0) | Automatically proceed from Execute to Apply | When confidence >= threshold, Apply begins when Execute completes. Below threshold, system pauses for user diff review. |
|
||||
| `edit_code` | float (0.0–1.0) | Automatically make decisions during Strategize | When confidence >= threshold, the strategy actor makes the decision autonomously. Below threshold, system pauses at the decision point for user input. |
|
||||
| `execute_command` | float (0.0–1.0) | Automatically make decisions during Execute | When confidence >= threshold, the execution actor makes the decision autonomously. Below threshold, system pauses at the decision point for user input. |
|
||||
| `create_file` | float (0.0–1.0) | Automatically attempt to fix validation failures | When confidence >= threshold, execution actor self-fixes failing validations. Below threshold, system pauses for user guidance. |
|
||||
| `delete_content` | float (0.0–1.0) | Automatically revise strategy when Execute hits constraints | When confidence >= threshold, system re-runs Strategize for affected subtree. Below threshold, system pauses and asks user. |
|
||||
| `access_network` | float (0.0–1.0) | Automatically revert from Apply (`constrained` state) to Strategize | When confidence >= threshold, a `constrained` Apply automatically triggers reversion to Strategize. Below threshold, system pauses and asks user whether to revert or cancel. |
|
||||
| `install_dependency` | float (0.0–1.0) | Automatically spawn and execute child plans | When confidence >= threshold, child plans are created and executed without pausing. Below threshold, each spawn requires approval. |
|
||||
| `modify_config` | float (0.0–1.0) | Automatically retry on transient failures (network, timeout, rate-limit) | When confidence >= threshold, system retries with backoff. Below threshold, system pauses and asks user. |
|
||||
| `approve_plan` | float (0.0–1.0) | Automatically restore from checkpoint on failure | When confidence >= threshold, system rolls back and retries. Below threshold, user decides whether to restore. |
|
||||
|
||||
#### Safety Profile (Composed Sub-Model)
|
||||
|
||||
!!! adr "Architecture Decision"
|
||||
The extraction of safety constraints into a composed SafetyProfile sub-model is defined in [ADR-041: Safety Profile Extraction](adr/ADR-041-safety-profile-extraction.md).
|
||||
|
||||
Each automation profile composes a **`safety`** sub-model (`SafetyProfile`) that groups all hard safety constraints. These are binary constraints rather than confidence-dependent behaviors — they enforce invariant safety guarantees regardless of the system's confidence level. The `SafetyProfile` is the **single source of truth** for safety constraints; the `AutomationProfile` accesses them via its `safety` field.
|
||||
|
||||
A `SafetyProfile` may also be attached directly to an `Action` (via the `safety_profile` field) when only safety constraints are needed without full autonomy thresholds.
|
||||
|
||||
| Flag | Type | Default | Description | Behavior |
|
||||
|------|------|---------|-------------|----------|
|
||||
| `require_sandbox` | boolean | `true` | Require sandbox isolation for Execute phase | When `true`, Execute must run in a sandbox. When `false`, sandbox is optional. |
|
||||
| `require_checkpoints` | boolean | `true` | Require checkpointing during Execute | When `true`, tools must create checkpoints before writes. When `false`, checkpointing is optional. |
|
||||
| `allow_unsafe_tools` | boolean | `false` | Allow execution of tools marked as unsafe | When `true`, unsafe tools can be invoked. When `false`, unsafe tools are blocked. |
|
||||
| `require_human_approval` | boolean | `false` | Require human approval before each action step | When `true`, every action step pauses for explicit human approval before execution. |
|
||||
| `allowed_skill_categories` | list[string] | `[]` (all) | Skill categories permitted for execution | When non-empty, only skills in the listed categories may be used. Empty list means all categories are allowed. |
|
||||
| `max_cost_per_plan` | float \| null | `null` | Maximum cost in USD per plan execution | When set, the plan is paused or terminated if the cost limit is reached. `null` means no limit. Must be <= `max_total_cost` when both are set. |
|
||||
| `max_total_cost` | float \| null | `null` | Maximum total cost in USD across all plans | When set, execution is paused or terminated if the aggregate cost limit is reached. `null` means no limit. |
|
||||
| `max_retries_per_step` | integer | `3` | Maximum retry attempts per action step | Limits the number of retries for a single step before escalating to the user. Range: 0–100. |
|
||||
|
||||
**Relationship to Automation Guards**: The `SafetyProfile.max_total_cost` field sets a **plan-level** budget cap (broad scope), while `AutomationGuard.max_total_cost` sets a **per-invocation** budget cap (narrow scope). These operate at different granularities and both may be active simultaneously — the tighter constraint takes precedence at any given point.
|
||||
|
||||
#### Automation Guard Sub-Model
|
||||
|
||||
An `AutomationProfile` may optionally compose an `AutomationGuard` sub-model (via the `guards` field) that provides runtime enforcement hooks beyond the phase-transition thresholds. Guards gate individual tool invocations based on call counts, budgets, allowlists/denylists, and write/apply semantics.
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `max_tool_calls_per_step` | integer \| null | `null` | Maximum tool invocations per step before requiring approval. `null` means unlimited. Must be >= 0. |
|
||||
| `max_total_cost` | float \| null | `null` | Per-invocation budget cap (cumulative cost) before requiring approval. `null` means unlimited. Must be >= 0.0. |
|
||||
| `tool_allowlist` | list[string] \| null | `null` | Only these tools may be called automatically. `null` means all tools are allowed. |
|
||||
| `tool_denylist` | list[string] \| null | `null` | These tools always require human approval. `null` means no tools are denied. |
|
||||
| `require_approval_for_writes` | boolean | `false` | Require human approval for write operations. |
|
||||
| `require_approval_for_apply` | boolean | `false` | Require human approval before the apply phase. |
|
||||
|
||||
Guard evaluation is performed via `AutomationProfile.check_guard(tool_name, is_write, cost_so_far, calls_so_far, scope)`. The `scope` parameter is a `GuardScope` enum (`PLAN` or `SUBPLAN`) that distinguishes plan-level from subplan-level enforcement — error messages include the scope name for clarity. The method returns a `GuardResult` with fields `allowed` (bool), `reason` (str | None), and `requires_approval` (bool). Evaluation order: denylist → allowlist → tool call limit → budget cap → write approval → apply approval.
|
||||
|
||||
#### Built-in Automation Profiles
|
||||
|
||||
CleverAgents ships with eight built-in automation profiles. Built-in profiles use no namespace prefix.
|
||||
|
||||
Threshold values are shown for each flag. A value of **0.0** means always automatic, **1.0** means always manual, and intermediate values (e.g., 0.5, 0.7) mean "automatic when confidence is at or above that level."
|
||||
|
||||
| Flag | `manual` | `review` | `supervised` | `cautious` | `trusted` | `auto` | `ci` | `full-auto` |
|
||||
|------|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|
||||
| `decompose_task` | 1.0 | 0.0 | 0.0 | 0.7 | 0.0 | 0.0 | 0.0 | 0.0 |
|
||||
| `create_tool` | 1.0 | 0.0 | 1.0 | 0.7 | 0.0 | 0.0 | 0.0 | 0.0 |
|
||||
| `select_tool` | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 0.0 | 0.0 |
|
||||
| `edit_code` | 1.0 | 1.0 | 0.0 | 0.6 | 0.0 | 0.0 | 0.0 | 0.0 |
|
||||
| `execute_command` | 1.0 | 1.0 | 1.0 | 0.8 | 0.0 | 0.0 | 0.0 | 0.0 |
|
||||
| `create_file` | 1.0 | 1.0 | 1.0 | 0.7 | 0.0 | 0.0 | 0.0 | 0.0 |
|
||||
| `delete_content` | 1.0 | 1.0 | 1.0 | 0.8 | 1.0 | 0.0 | 0.0 | 0.0 |
|
||||
| `access_network` | 1.0 | 1.0 | 1.0 | 0.9 | 1.0 | 1.0 | 0.0 | 0.0 |
|
||||
| `install_dependency` | 1.0 | 0.0 | 1.0 | 0.7 | 0.0 | 0.0 | 0.0 | 0.0 |
|
||||
| `modify_config` | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
|
||||
| `approve_plan` | 1.0 | 1.0 | 1.0 | 0.6 | 1.0 | 0.0 | 0.0 | 0.0 |
|
||||
| **Safety Profile** | | | | | | | | |
|
||||
| `safety.require_sandbox` | true | true | true | true | true | true | true | false |
|
||||
| `safety.require_checkpoints` | true | true | true | true | true | true | true | false |
|
||||
| `safety.allow_unsafe_tools` | false | false | false | false | false | false | false | true |
|
||||
| `safety.require_human_approval` | false | false | false | false | false | false | false | false |
|
||||
| `safety.allowed_skill_categories` | [] | [] | [] | [] | [] | [] | [] | [] |
|
||||
| `safety.max_cost_per_plan` | null | null | null | null | null | null | null | null |
|
||||
| `safety.max_total_cost` | null | null | null | null | null | null | null | null |
|
||||
| `safety.max_retries_per_step` | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 |
|
||||
|
||||
**`manual`**: Maximum human control. All thresholds set to 1.0 — every phase transition, every decision, every child plan requires explicit human approval regardless of confidence. Sandbox and checkpoints are mandatory. Unsafe tools are blocked. This is the default starting point. Use for: new users learning the system, critical production systems, first-time exploration of an unfamiliar codebase, sensitive projects, regulatory environments.
|
||||
|
||||
**`review`**: Phases run automatically (Strategize and Execute proceed without pausing), and most decisions within those phases require human approval (thresholds set to 1.0), except transient retries and child plan spawning which are automatic (thresholds 0.0). Apply is manual. The system does the work but consults you on every choice point. Use for: teams that want to stay in the decision loop without manually triggering each phase, code reviews where architectural choices matter more than execution mechanics.
|
||||
|
||||
**`supervised`**: Strategize thresholds set to 0.0 (always automatic), but Execute and Apply thresholds set to 1.0 (always manual). The system plans autonomously and retries transient failures automatically, but pauses before Execute for human review of the strategy. Use for: projects where you trust the planning but want to review before execution begins.
|
||||
|
||||
**`cautious`**: Uses intermediate confidence thresholds (0.6–0.8) instead of binary 0.0/1.0 values. The system proceeds automatically only when the Semantic Escalation system reports high confidence, and escalates to the user when uncertain. Apply is always manual (1.0). Higher thresholds (0.8) are applied to riskier operations like strategy revision and execution decisions, while lower thresholds (0.6) are used for safer operations like checkpoint restores and strategy decisions. Transient retries are always automatic. Use for: teams adopting automation gradually, projects with mixed complexity where some tasks are routine but others need oversight, situations where you want the system to self-assess rather than follow rigid rules.
|
||||
|
||||
**`trusted`**: Strategize and Execute thresholds set to 0.0 (always automatic). Validation fixes, child plan spawning, and transient retries proceed automatically. Strategy revision and checkpoint restores remain manual (thresholds 1.0) to preserve human oversight over plan-level course corrections. The system pauses only before Apply (threshold 1.0) for human review of the final diffs. Use for: day-to-day feature development, routine refactoring, test generation.
|
||||
|
||||
**`auto`**: All thresholds set to 0.0 except Apply (1.0). The system can revise its own strategy, restore from checkpoints, and handle all decisions automatically. Only Apply requires human approval. Use for: well-understood projects, batch operations, tasks with strong invariant coverage.
|
||||
|
||||
**`ci`**: All thresholds set to 0.0 — complete end-to-end automation including Apply — but sandbox and checkpoints remain mandatory and unsafe tools are blocked. Designed for non-interactive environments where full automation is needed but safety nets must be preserved. Use for: CI/CD pipelines, automated testing workflows, scheduled batch jobs, any headless execution where rollback capability is essential.
|
||||
|
||||
**`full-auto`**: All thresholds set to 0.0 — complete end-to-end automation including Apply. No sandbox or checkpoint requirements. Unsafe tools are allowed. Use for: low-risk routine tasks (dependency updates, documentation generation, formatting), trusted batch operations with rollback capabilities, environments where external safety mechanisms exist.
|
||||
|
||||
#### Profile Precedence
|
||||
|
||||
Automation profiles are determined using this precedence (highest to lowest):
|
||||
|
||||
1. **Plan-level**: Explicitly set via `--automation-profile` on `agents plan use`
|
||||
2. **Action-level**: Set on the action via `--automation-profile` on `agents action create`
|
||||
3. **Project-level**: Set via `agents config set core.automation-profile <PROFILE> --project <PROJECT>`
|
||||
4. **Global-level**: Set via `agents config set core.automation-profile <PROFILE>`
|
||||
|
||||
The **effective profile** for a plan is resolved at the moment of `agents plan use`. Once resolved, the profile is **locked to that plan** — subsequent changes to project or global profiles do not affect running plans.
|
||||
|
||||
#### Child Plan Profile Inheritance
|
||||
|
||||
Child plans inherit the parent plan's effective automation profile. If the parent's profile is changed explicitly after creation, new child plans use the new profile while already-running child plans retain their original profile.
|
||||
|
||||
#### Custom Automation Profiles
|
||||
|
||||
Custom profiles are created via YAML configuration files and registered with `agents automation-profile add`. Each automatable task flag takes a confidence threshold (0.0–1.0) instead of a boolean. A threshold of 0.0 means "always automatic" and 1.0 means "always manual." Intermediate values (e.g., 0.5, 0.7) enable fine-grained control where the system proceeds automatically only when the Semantic Escalation confidence score meets or exceeds the threshold:
|
||||
|
||||
<div class="highlight"><pre><code>
|
||||
<span style="opacity: 0.7;"># File: profiles/careful-auto.yaml</span>
|
||||
<span style="color: cyan; font-weight: 600;">name</span>: local/careful-auto
|
||||
<span style="color: cyan; font-weight: 600;">description</span>: <span style="color: #66cc66;">"Autonomous execution with mandatory sandbox and manual apply"</span>
|
||||
|
||||
<span style="opacity: 0.7;"># Confidence thresholds (0.0 = always auto, 1.0 = always manual)</span>
|
||||
<span style="color: cyan; font-weight: 600;">decompose_task</span>: <span style="color: yellow;">0.0</span>
|
||||
<span style="color: cyan; font-weight: 600;">create_tool</span>: <span style="color: yellow;">0.0</span>
|
||||
<span style="color: cyan; font-weight: 600;">select_tool</span>: <span style="color: yellow;">1.0</span>
|
||||
<span style="color: cyan; font-weight: 600;">edit_code</span>: <span style="color: yellow;">0.0</span>
|
||||
<span style="color: cyan; font-weight: 600;">execute_command</span>: <span style="color: yellow;">0.0</span>
|
||||
<span style="color: cyan; font-weight: 600;">create_file</span>: <span style="color: yellow;">0.0</span>
|
||||
<span style="color: cyan; font-weight: 600;">delete_content</span>: <span style="color: yellow;">1.0</span>
|
||||
<span style="color: cyan; font-weight: 600;">access_network</span>: <span style="color: yellow;">1.0</span>
|
||||
<span style="color: cyan; font-weight: 600;">install_dependency</span>: <span style="color: yellow;">0.0</span>
|
||||
<span style="color: cyan; font-weight: 600;">modify_config</span>: <span style="color: yellow;">0.0</span>
|
||||
<span style="color: cyan; font-weight: 600;">approve_plan</span>: <span style="color: yellow;">0.0</span>
|
||||
|
||||
<span style="opacity: 0.7;"># Safety profile (composed sub-model)</span>
|
||||
<span style="color: cyan; font-weight: 600;">safety</span>:
|
||||
<span style="color: cyan; font-weight: 600;">require_sandbox</span>: <span style="color: magenta; font-weight: 600;">true</span>
|
||||
<span style="color: cyan; font-weight: 600;">require_checkpoints</span>: <span style="color: magenta; font-weight: 600;">true</span>
|
||||
<span style="color: cyan; font-weight: 600;">allow_unsafe_tools</span>: <span style="color: magenta; font-weight: 600;">false</span>
|
||||
</code></pre></div>
|
||||
|
||||
A profile with intermediate thresholds provides nuanced control. For example, `execute_command: 0.7` means decisions during Execute proceed automatically when confidence >= 0.7, but drop to manual review when confidence is below 0.7:
|
||||
|
||||
<div class="highlight"><pre><code>
|
||||
<span style="opacity: 0.7;"># File: profiles/nuanced-auto.yaml</span>
|
||||
<span style="color: cyan; font-weight: 600;">name</span>: local/nuanced-auto
|
||||
<span style="color: cyan; font-weight: 600;">description</span>: <span style="color: #66cc66;">"Nuanced automation with graduated confidence thresholds"</span>
|
||||
|
||||
<span style="color: cyan; font-weight: 600;">decompose_task</span>: <span style="color: yellow;">0.0</span><span style="opacity: 0.7;"> # Always auto-strategize</span>
|
||||
<span style="color: cyan; font-weight: 600;">create_tool</span>: <span style="color: yellow;">0.5</span><span style="opacity: 0.7;"> # Auto-execute when confidence >= 0.5</span>
|
||||
<span style="color: cyan; font-weight: 600;">select_tool</span>: <span style="color: yellow;">1.0</span><span style="opacity: 0.7;"> # Always require manual apply</span>
|
||||
<span style="color: cyan; font-weight: 600;">edit_code</span>: <span style="color: yellow;">0.3</span><span style="opacity: 0.7;"> # Low bar for strategy decisions</span>
|
||||
<span style="color: cyan; font-weight: 600;">execute_command</span>: <span style="color: yellow;">0.7</span><span style="opacity: 0.7;"> # Higher bar for execution decisions</span>
|
||||
<span style="color: cyan; font-weight: 600;">create_file</span>: <span style="color: yellow;">0.5</span><span style="opacity: 0.7;"> # Auto-fix when reasonably confident</span>
|
||||
<span style="color: cyan; font-weight: 600;">delete_content</span>: <span style="color: yellow;">0.8</span><span style="opacity: 0.7;"> # High confidence needed for auto-revision</span>
|
||||
<span style="color: cyan; font-weight: 600;">access_network</span>: <span style="color: yellow;">0.9</span><span style="opacity: 0.7;"> # Very high bar for late-stage reversion</span>
|
||||
<span style="color: cyan; font-weight: 600;">install_dependency</span>: <span style="color: yellow;">0.5</span><span style="opacity: 0.7;"> # Auto-spawn when moderately confident</span>
|
||||
<span style="color: cyan; font-weight: 600;">modify_config</span>: <span style="color: yellow;">0.0</span><span style="opacity: 0.7;"> # Always auto-retry transient errors</span>
|
||||
<span style="color: cyan; font-weight: 600;">approve_plan</span>: <span style="color: yellow;">0.6</span><span style="opacity: 0.7;"> # Auto-restore when fairly confident</span>
|
||||
|
||||
<span style="opacity: 0.7;"># Safety profile (composed sub-model)</span>
|
||||
<span style="color: cyan; font-weight: 600;">safety</span>:
|
||||
<span style="color: cyan; font-weight: 600;">require_sandbox</span>: <span style="color: magenta; font-weight: 600;">true</span>
|
||||
<span style="color: cyan; font-weight: 600;">require_checkpoints</span>: <span style="color: magenta; font-weight: 600;">true</span>
|
||||
<span style="color: cyan; font-weight: 600;">allow_unsafe_tools</span>: <span style="color: magenta; font-weight: 600;">false</span>
|
||||
</code></pre></div>
|
||||
|
||||
<div class="highlight"><pre><code>
|
||||
agents automation-profile add <span style="color: cyan;">--config</span> ./profiles/careful-auto.yaml
|
||||
</code></pre></div>
|
||||
|
||||
#### Semantic Escalation
|
||||
|
||||
Semantic Escalation is the system that computes a **confidence score** (0.0–1.0) for each operation, and it is the mechanism through which automation profile thresholds take effect. The confidence score reflects the system's assessment of how likely an autonomous action is to succeed without human guidance. This score is then compared against the profile's threshold for the relevant flag to determine whether to proceed automatically or escalate to the user.
|
||||
|
||||
The confidence score is computed from multiple factors:
|
||||
|
||||
<div class="highlight"><pre><code>
|
||||
<span style="color: magenta; font-weight: 600;">class</span> <span style="color: cyan; font-weight: 600;">AutonomyController</span>:
|
||||
<span style="color: magenta; font-weight: 600;">def</span> <span style="color: cyan; font-weight: 600;">should_proceed_automatically</span>(self, decision, context, profile):
|
||||
<span style="color: #66cc66;">"""Determine whether to proceed automatically or escalate to user."""</span>
|
||||
factors = {
|
||||
<span style="color: #66cc66;">'past_success_rate'</span>: self.get_historical_success(decision.type),
|
||||
<span style="color: #66cc66;">'codebase_familiarity'</span>: self.get_familiarity_score(context.project),
|
||||
<span style="color: #66cc66;">'risk_assessment'</span>: self.evaluate_risk(decision),
|
||||
<span style="color: #66cc66;">'invariant_complexity'</span>: self.analyze_invariants(decision)
|
||||
}
|
||||
|
||||
confidence = self.compute_confidence(factors) <span style="opacity: 0.7;"># Returns 0.0–1.0</span>
|
||||
threshold = profile.get_threshold(decision.flag) <span style="opacity: 0.7;"># e.g., execute_command</span>
|
||||
|
||||
<span style="color: magenta; font-weight: 600;">if</span> confidence >= threshold:
|
||||
<span style="opacity: 0.7;"># Confidence meets or exceeds the profile threshold — proceed</span>
|
||||
<span style="color: magenta; font-weight: 600;">return</span> ProceedAutonomously(decision, confidence)
|
||||
<span style="color: magenta; font-weight: 600;">else</span>:
|
||||
<span style="opacity: 0.7;"># Confidence below threshold — escalate to human</span>
|
||||
<span style="color: magenta; font-weight: 600;">return</span> RequestHumanGuidance(decision, confidence, threshold, factors)
|
||||
</code></pre></div>
|
||||
|
||||
**How thresholds and confidence interact**: A profile with `execute_command: 0.7` means the system will make execution decisions automatically when confidence >= 0.7, but will pause for human input when confidence < 0.7. This is more nuanced than a binary on/off: a profile can set a high bar (0.9) for risky operations while being permissive (0.2) for low-risk ones.
|
||||
|
||||
**Special cases**: A threshold of **0.0** means "always automatic" — even zero confidence passes the threshold. A threshold of **1.0** means "always manual" — no confidence level (which tops out below 1.0 in practice) is high enough to pass. This means the old boolean behavior is a strict subset: `true` maps to `0.0` and `false` maps to `1.0`.
|
||||
|
||||
#### Progressive Trust Building
|
||||
|
||||
New users typically follow this progression:
|
||||
|
||||
1. Start with `manual` to understand system behavior
|
||||
1. Move to `review` to let phases run automatically while staying in the decision loop
|
||||
1. Adopt `supervised` as confidence in the planning phase builds
|
||||
1. Try `cautious` for confidence-gated automation that escalates only when uncertain
|
||||
1. Adopt `trusted` for routine development tasks
|
||||
1. Enable `auto` for well-understood projects with strong invariant coverage
|
||||
1. Use `ci` for headless CI/CD pipelines with safety nets intact
|
||||
1. Use `full-auto` for low-risk batch operations where external safety mechanisms exist
|
||||
|
||||
### Guardrails
|
||||
|
||||
!!! adr "Architecture Decision"
|
||||
System-level guardrails, pre-flight checks, and runtime limits are defined in [ADR-018: Semantic Error Prevention](adr/ADR-018-semantic-error-prevention.md).
|
||||
|
||||
System-level guardrails are pre-flight checks and runtime limits that prevent plans from executing in invalid or unsafe conditions. These are distinct from **Validations** (the Tool subtype described in Core Concepts > Validation) — guardrails operate at the system/infrastructure level before and during plan execution, whereas Validations verify the *quality of the work produced* by a plan at the end of the Execute phase.
|
||||
|
||||
#### Plan Generation Guardrails
|
||||
|
||||
Before a plan begins execution, the system performs pre-flight validation to ensure the plan is well-formed and its dependencies are satisfiable. These checks must be implemented:
|
||||
|
||||
* **Action schema validation**: Verify the action referenced by the plan exists, is well-formed, and its configuration conforms to the expected schema.
|
||||
* **Actor availability**: Confirm that all actors required by the plan (strategy actor, execution actor, estimation actor, invariant reconciliation actor) are registered and reachable. For remote actors, verify network connectivity.
|
||||
* **Skill and tool existence**: Verify that all skills and tools referenced by the action's actor configuration exist in the registry. This includes both named tools and tools referenced via skills (transitively resolved).
|
||||
* **Automation policy**: Verify that the automation profile allows the requested level of autonomy for the target project, resources, and actions.
|
||||
* **Rollback feasibility**: If `require_checkpoints` is enabled on the plan, verify that all tools in the actor's skill set have `checkpointable: true`. Reject plans that would use non-checkpointable tools under a checkpoint-required policy.
|
||||
* **Resource accessibility**: Verify that the project's linked resources are accessible — git repos are cloneable, databases are connectable, file system paths exist, etc. This is a shallow connectivity check, not a full resource health assessment.
|
||||
* **Validation attachment resolution**: Pre-resolve the validations that will apply to this plan (from resource-direct, project, and plan attachment scopes) and verify they are all registered and their tool definitions are valid. This ensures validation failures during Execute are due to the work product, not misconfigured validations.
|
||||
|
||||
These pre-flight checks prevent "plan runs with fake providers," missing tool errors mid-execution, and other surprises that waste compute and user time. If any pre-flight check fails, the plan is rejected before entering the Strategize phase, with a clear error message identifying the failing check.
|
||||
|
||||
#### Cost and Rate Limits
|
||||
|
||||
Runtime guardrails to prevent runaway resource consumption:
|
||||
|
||||
* **Per-plan budgets**: Maximum API token spend, maximum wall-clock time, maximum number of tool invocations per plan. When a budget is exceeded, the plan pauses and requests user approval to continue or is terminated.
|
||||
* **Per-session budgets**: Aggregate limits across all plans in a session. Prevents a single interactive session from consuming excessive resources.
|
||||
* **Per-org budgets**: Server-enforced limits for multi-user deployments. Administrators set organization-wide spend caps that cannot be overridden by individual users.
|
||||
* **Per-actor limits**: Maximum tool calls per actor invocation, maximum retries per tool failure, maximum tokens per LLM call. Prevents individual actors from looping indefinitely.
|
||||
* **Validation retry limits**: Maximum number of fix-then-revalidate iterations when a required Validation fails during Execute. This is a specific application of per-actor limits — the default is 3 attempts, configurable per plan or in the automation profile.
|
||||
|
||||
Cost and rate limits are future concerns that require integration with LLM provider billing APIs and internal metering. The system should define configuration surfaces for these limits but may initially implement only the per-plan and per-actor limits, with per-session, per-org, and billing integration added later.
|
||||
|
||||
### Correcting Plans (Core Feature)
|
||||
|
||||
!!! adr "Architecture Decision"
|
||||
Plan correction, decision revision, and the edit-replay model are defined in [ADR-007: Decision Tree and Correction](adr/ADR-007-decision-tree-and-correction.md).
|
||||
|
||||
Correcting plans is where CleverAgents becomes more than "a fancy prompt runner."
|
||||
|
||||
#### The Goal
|
||||
|
||||
When a plan makes a wrong decision early, we want to:
|
||||
|
||||
* correct the decision,
|
||||
* recompute only the affected subtree,
|
||||
* preserve unaffected work.
|
||||
|
||||
This is explicitly described: "redo everything below that decision, not the entire code base."
|
||||
|
||||
#### Decision Tree Representation
|
||||
|
||||
Every plan records (see Decision Data Model section):
|
||||
|
||||
* decisions (choice points) - created during Strategize, including `invariant_enforced`, `subplan_spawn`, and `subplan_parallel_spawn` decisions
|
||||
* dependencies (which later work depended on that decision)
|
||||
* child plans spawned because of that decision - populated during Execute
|
||||
* artifacts generated under that branch
|
||||
|
||||
This makes plan runs auditable and correctable.
|
||||
|
||||
#### Two Correction Modes
|
||||
|
||||
=== "Revert (`--mode=revert`)"
|
||||
|
||||
!!! warning "Potentially Expensive"
|
||||
Find the decision point in the tree, roll back all changes (code and non-code) to that point, and re-run from that decision point forward. Old execution artifacts are kept for comparison.
|
||||
|
||||
- [x] Roll back all changes to the decision point
|
||||
- [x] Re-run from that decision point forward
|
||||
- [x] Keep old execution artifacts for comparison
|
||||
- [ ] ==Potentially expensive== if high up in the tree
|
||||
|
||||
=== "Append (`--mode=append`)"
|
||||
|
||||
!!! success "Safer and Cheaper"
|
||||
Leave history intact and append a new plan at the end that fixes the outcome. Does not rewrite history.
|
||||
|
||||
- [x] Leave history intact
|
||||
- [x] Append a new corrective plan at the end
|
||||
- [x] Cheaper and safer in many cases
|
||||
- [x] Does ==not rewrite history==
|
||||
|
||||
#### Correction Flow (Revert Mode)
|
||||
|
||||
!!! adr "Architecture Decision"
|
||||
The detailed rollback and replay mechanics, including mid-phase correction flows and cross-sandbox coordination, are defined in [ADR-035: Decision Tree Rollback and Replay](adr/ADR-035-decision-tree-rollback-and-replay.md).
|
||||
|
||||
Correction operates on **two independent rollback dimensions**:
|
||||
|
||||
| Dimension | What It Restores | Mechanism | Applicable Phase |
|
||||
|---|---|---|---|
|
||||
| **Resource rollback** | Sandbox contents (files, database state, filesystem) | Checkpoint restoration via sandbox-strategy-specific handlers | Execute only |
|
||||
| **Reasoning rollback** | Actor's conversation history, intermediate reasoning, graph state | LangGraph checkpoint restoration from `actor_state_ref` | Strategize and Execute |
|
||||
|
||||
##### Mid-Strategize Correction
|
||||
|
||||
When the target decision was created during Strategize (e.g., a `strategy_choice` or `subplan_spawn`), **only reasoning rollback is needed** because Strategize is read-only — no resource modifications exist to undo:
|
||||
|
||||
1. **Restore actor state**: Load the target decision's `context_snapshot.actor_state_ref` (LangGraph checkpoint). This restores the strategy actor to the exact reasoning state it was in when the decision was made.
|
||||
2. **Supersede affected subtree**: Mark the target decision and all its descendants as superseded (cascading via ADR-034). Decisions above the target remain active and unchanged.
|
||||
3. **Inject guidance**: Create a `user_intervention` decision containing the user's `--guidance` text.
|
||||
4. **Resume strategy actor**: The actor continues reasoning from the restored state, informed by the correction guidance. New decisions replace the superseded ones, with `is_correction: true, corrects_decision_id: <target>`.
|
||||
5. **Record correction**: Create a `correction_attempts` record linking old and new decisions.
|
||||
|
||||
If the plan was already in Execute when this correction is requested, all Execute-phase work is rolled back (via checkpoint groups — see below) and the plan reverts to Strategize at the corrected decision point.
|
||||
|
||||
##### Mid-Execute Correction
|
||||
|
||||
When the target decision was created during Execute (e.g., an `implementation_choice` or `error_recovery`), **both resource and reasoning rollback are needed**:
|
||||
|
||||
1. **Load checkpoint group**: Every Execute-phase decision has a corresponding **checkpoint group** — one per-resource checkpoint for each sandbox resource that had been modified at the time the decision was recorded. These are stored in `checkpoint_metadata` linked by `decision_id`.
|
||||
|
||||
2. **Classify resources**:
|
||||
- *Rollbackable*: resources with sandbox strategies that support checkpointing (`git_worktree`, `filesystem_copy`, `overlay`, `transaction_rollback`)
|
||||
- *Non-rollbackable*: resources with `sandbox.strategy: none` (tracked with marker records)
|
||||
- *Unaffected*: resources not yet modified at the decision point (no checkpoint row — no rollback needed)
|
||||
|
||||
3. **Warn about limitations**: If non-rollbackable resources exist, the user is warned and must confirm. If downstream decisions triggered tools with irreversible `side_effects`, the user is warned that external effects cannot be undone.
|
||||
|
||||
4. **Resource rollback**: For each rollbackable checkpoint, dispatch to the strategy-specific handler:
|
||||
|
||||
| Strategy | Rollback Operation |
|
||||
|---|---|
|
||||
| `git_worktree` | `git reset --hard <commit_sha>` in the worktree |
|
||||
| `filesystem_copy` | Restore from archived snapshot |
|
||||
| `overlay` | Discard overlay writes since checkpoint |
|
||||
| `transaction_rollback` | `ROLLBACK TO SAVEPOINT <savepoint_name>` |
|
||||
|
||||
If any individual resource rollback fails, the correction is marked `'failed'` and the plan enters `errored` state.
|
||||
|
||||
5. **Reasoning rollback**: Restore the execution actor's LangGraph state from `context_snapshot.actor_state_ref`.
|
||||
6. **Supersede affected subtree**: Cascade superseding through the target and all descendants.
|
||||
7. **Inject guidance and resume**: The execution actor continues from the restored state.
|
||||
|
||||
!!! note "Decision-Aligned Checkpointing"
|
||||
The 1:1 mapping between Execute-phase decisions and resource checkpoints is created automatically by the `record_decision` tool (see Decision Recording Protocol in the Core Concepts section). Every `record_decision` call during Execute triggers a coordinated checkpoint group across all modified sandbox resources. This is what enables fine-grained rollback to any individual Execute-phase decision rather than reverting the entire Execute phase.
|
||||
|
||||
##### Affected Subtree Computation
|
||||
|
||||
The "affected subtree" — all decisions and child plans invalidated by a correction — is computed by walking the influence DAG (not just the structural tree):
|
||||
|
||||
<div class="highlight"><pre><code>
|
||||
<span style="color: magenta; font-weight: 600;">def</span> <span style="color: cyan; font-weight: 600;">compute_affected_subtree</span>(target_decision_id):
|
||||
affected_decisions = {target_decision_id}
|
||||
affected_plans = <span style="color: cyan; font-weight: 600;">set</span>()
|
||||
queue = [target_decision_id]
|
||||
|
||||
<span style="color: magenta; font-weight: 600;">while</span> queue:
|
||||
current = queue.pop(<span style="color: yellow;">0</span>)
|
||||
|
||||
<span style="opacity: 0.7;"># Follow structural tree children</span>
|
||||
children = query(<span style="color: #66cc66;">"SELECT decision_id FROM decisions "</span>
|
||||
<span style="color: #66cc66;">"WHERE parent_decision_id = :current AND superseded_by IS NULL"</span>)
|
||||
|
||||
<span style="opacity: 0.7;"># Follow influence DAG dependents</span>
|
||||
dependents = query(<span style="color: #66cc66;">"SELECT downstream_ref FROM decision_dependencies "</span>
|
||||
<span style="color: #66cc66;">"WHERE upstream_decision_id = :current AND dependency_type = 'decision'"</span>)
|
||||
|
||||
<span style="color: magenta; font-weight: 600;">for</span> d <span style="color: magenta; font-weight: 600;">in</span> children | dependents:
|
||||
<span style="color: magenta; font-weight: 600;">if</span> d <span style="color: magenta; font-weight: 600;">not in</span> affected_decisions:
|
||||
affected_decisions.add(d)
|
||||
queue.append(d)
|
||||
|
||||
<span style="opacity: 0.7;"># Collect affected child plans</span>
|
||||
child_plans = query(<span style="color: #66cc66;">"SELECT downstream_ref FROM decision_dependencies "</span>
|
||||
<span style="color: #66cc66;">"WHERE upstream_decision_id = :current AND dependency_type = 'plan'"</span>)
|
||||
affected_plans.update(child_plans)
|
||||
|
||||
<span style="color: magenta; font-weight: 600;">return</span> affected_decisions, affected_plans
|
||||
</code></pre></div>
|
||||
|
||||
##### Cross-Plan Correction Cascading
|
||||
|
||||
When the affected subtree includes child plans (via `subplan_spawn` or `subplan_parallel_spawn` decisions), the child plan's state determines the behavior:
|
||||
|
||||
| Child Plan State | Action |
|
||||
|---|---|
|
||||
| Not yet started | Cancel the child plan |
|
||||
| In progress (Strategize or Execute) | Cancel the child plan, roll back its sandbox |
|
||||
| Completed but not applied | Cancel the child plan, roll back its sandbox |
|
||||
| **Already applied** | **Correction rejected** — applied changes cannot be unilaterally reverted; correct the child plan independently or use `--mode=append` on the parent |
|
||||
|
||||
##### Rollback Tiers
|
||||
|
||||
The system's rollback capability depends on the plan's checkpoint and sandbox configuration:
|
||||
|
||||
| Tier | Prerequisites | Capability |
|
||||
|---|---|---|
|
||||
| **Full decision-level** | Checkpointing enabled + sandboxable resources | Roll back to any individual Execute-phase decision |
|
||||
| **Phase-level** | Checkpointing disabled or unsupported resources | Revert entire Execute phase → Strategize; re-execute from scratch |
|
||||
| **No rollback** | `sandbox.strategy: none` + checkpointing disabled | Only `--mode=append` corrections available |
|
||||
|
||||
The system detects the available tier automatically and informs the user when a requested correction exceeds the available capability.
|
||||
|
||||
##### Phase Boundary Interaction
|
||||
|
||||
`agents plan correct` is **orthogonal** to phase reversion (Execute → Strategize):
|
||||
|
||||
- **Phase reversion** is *actor-triggered* when execution constraints are too tight — the actor signals that the strategy needs adjustment.
|
||||
- **Decision correction** is *user-triggered* via `plan correct` — the user identifies a wrong decision.
|
||||
|
||||
They can interact: correcting a Strategize-phase decision while the plan is in Execute causes both resource rollback (of Execute-phase work) and phase reversion to Strategize. Correcting an Execute-phase decision while still in Execute performs resource rollback within Execute without a phase change.
|
||||
|
||||
##### Long-Running Database Transactions
|
||||
|
||||
The `transaction_rollback` sandbox strategy uses SAVEPOINTs for decision-aligned checkpoints within a single transaction spanning Execute. For plans with extended execution times, this may exceed database transaction timeouts. Recommended mitigations:
|
||||
|
||||
- **Decompose into child plans**: Each child plan gets a shorter transaction (aligns with hierarchical decomposition).
|
||||
- **Use `none` strategy with `--mode=append`**: Accept non-rollbackable database changes; fix issues via append corrections.
|
||||
- **Implement a custom snapshot-based strategy**: A custom `database-snapshot` resource type using point-in-time snapshots instead of long transactions.
|
||||
|
||||
#### History Cleanup
|
||||
|
||||
**History can only be flagged for cleanup after a plan reaches the `applied` state.**
|
||||
|
||||
Once a plan is applied:
|
||||
- It can no longer be rolled back
|
||||
- Old correction artifacts can be archived or deleted based on retention policy
|
||||
- The decision tree is preserved for audit purposes (superseded decisions are never deleted, per ADR-034)
|
||||
|
||||
#### CLI Commands for Correction
|
||||
|
||||
<div class="highlight"><pre><code>
|
||||
<span style="opacity: 0.7;"># View decision tree</span>
|
||||
agents plan tree <plan_id>
|
||||
agents <span style="color: cyan;">--format</span>=json plan tree <plan_id> # For visualization tools
|
||||
|
||||
<span style="opacity: 0.7;"># View decision tree including superseded branches</span>
|
||||
agents plan tree <span style="color: cyan;">--show-superseded</span> <plan_id>
|
||||
|
||||
<span style="opacity: 0.7;"># Inspect a specific decision</span>
|
||||
agents plan explain <decision_id>
|
||||
<span style="opacity: 0.7;"># Shows: question, chosen option, alternatives, rationale, downstream impact</span>
|
||||
|
||||
<span style="opacity: 0.7;"># Correct via revert-and-replay</span>
|
||||
agents plan correct <decision_id> <span style="color: cyan;">--mode</span>=revert <span style="color: cyan;">--guidance</span> <span style="color: #66cc66;">"<what the decision should be>"</span>
|
||||
<span style="opacity: 0.7;"># Re-executes from that point with the new guidance</span>
|
||||
|
||||
<span style="opacity: 0.7;"># Correct via append (add fix at end) </span>
|
||||
agents plan correct <decision_id> <span style="color: cyan;">--mode</span>=append <span style="color: cyan;">--guidance</span> <span style="color: #66cc66;">"<description of the fix>"</span>
|
||||
<span style="opacity: 0.7;"># Creates a new child plan to fix the outcome without rewriting history</span>
|
||||
|
||||
<span style="opacity: 0.7;"># Compare old vs new after correction</span>
|
||||
agents plan diff <span style="color: cyan;">--correction</span> <correction_attempt_id>
|
||||
</code></pre></div>
|
||||
|
||||
#### Correction Safety
|
||||
|
||||
!!! success "Safety Guarantees"
|
||||
Corrections always:
|
||||
|
||||
- [x] Create a new attempt revision (increment `plan.attempt`)
|
||||
- [x] Preserve old artifacts for diff/compare
|
||||
- [x] Run execute in sandbox again
|
||||
- [x] Require apply gating again
|
||||
- [x] ==Never modify already-applied changes==
|
||||
- [x] Warn about irreversible side effects before proceeding
|
||||
- [x] Reject correction of decisions whose affected subtree includes applied child plans
|
||||
|
||||
This keeps history reproducible and prevents accidental destructive edits.
|
||||
|
||||
### Human-in-the-Loop Collaboration
|
||||
|
||||
!!! adr "Architecture Decision"
|
||||
Human-in-the-loop controls, interruptibility, and approval gates are defined in [ADR-017: Automation Profiles](adr/ADR-017-automation-profiles.md).
|
||||
|
||||
Even though the direction is "more autonomous," the transcript explicitly recognizes that real workflows require engineers to collaborate with the system, editing code while it works, and using better UX integration (TUI/web/IDE).
|
||||
|
||||
!!! tip "Design Principles for Human-in-the-Loop"
|
||||
| Principle | Description |
|
||||
| :-------- | :---------- |
|
||||
| **Visibility** | What is the system doing ==right now==? |
|
||||
| **Interruptibility** | Pause / cancel / retry at any point |
|
||||
| **Editability** | Allow user to modify strategy before execute |
|
||||
| **Reconciliation** | Detect if user changed sandbox files mid-run and handle it |
|
||||
|
||||
### UI / Interaction Model
|
||||
|
||||
!!! adr "Architecture Decision"
|
||||
The CLI-first interaction model, TUI, and output rendering are defined in [ADR-021: CLI and Output Rendering](adr/ADR-021-cli-and-output-rendering.md).
|
||||
|
||||
#### CLI-first + TUI + Web App + IDE
|
||||
|
||||
!!! abstract "Multi-Frontend Architecture"
|
||||
The system is ==CLI-first==, with a single UI codebase powering multiple frontends:
|
||||
|
||||
- [x] **CLI** — Primary interface for all operations
|
||||
- [x] **TUI** — Built using Textual for rich terminal experiences
|
||||
- [x] **Web App** — Generated from TUI "for free"
|
||||
- [ ] **IDE Plugin** — Future: embeds the TUI in the IDE
|
||||
|
||||
#### Plan Tree Visualization
|
||||
|
||||
The TUI should show:
|
||||
|
||||
* plan list
|
||||
* plan details
|
||||
* plan tree (ASCII)
|
||||
* diff view
|
||||
* approvals
|
||||
|
||||
And should later allow exporting the tree as image (PNG) or JSON for other visualization tools.
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,129 @@
|
||||
## Glossary
|
||||
|
||||
???+ abstract "Plan Lifecycle"
|
||||
|
||||
Plan
|
||||
: A ==ULID-identified==, hierarchical unit of work instantiated from an Action template. Progresses through four phases — **Action**, **Strategize**, **Execute**, **Apply** — persisting a decision tree at each step. May spawn child plans. Scoped to one or more projects. Top-level plans carry a namespaced name (`[[server:]namespace/]name`); child plans are identified solely by their plan ID (ULID). All plans are referenced by plan ID when precision is needed, especially in hierarchies containing subplans.
|
||||
|
||||
Action
|
||||
: A YAML-defined, reusable plan template specifying a description, definition of done, strategy/execution actors, typed arguments, and optional invariants. Project-agnostic until bound to one or more projects via `agents plan use`, which instantiates a Plan. Namespaced as `[[server:]namespace/]name`.
|
||||
|
||||
Strategize
|
||||
: Second phase of the plan lifecycle (following the Action phase). The first phase where active processing occurs. ==Read-only==: the strategy actor produces the initial decision tree — strategy choices, invariant enforcement records, resource selections, child plan blueprints — without modifying any resources.
|
||||
|
||||
Execute
|
||||
: Third phase of the plan lifecycle. The execution actor carries out decisions from Strategize — invoking tools, spawning child plans, producing artifacts, creating checkpoints — with all mutations confined to a sandbox. May also create additional decisions constrained by Strategize-phase decisions, based on runtime discoveries.
|
||||
|
||||
Apply
|
||||
: Fourth and final phase of the plan lifecycle. Merges the sandbox changeset into real project resources. Terminal states: `applied` (success), `constrained` (cannot complete — may revert to Strategize), `errored` (failed), `cancelled` (user/system cancelled).
|
||||
|
||||
Decision
|
||||
: A persisted choice point in a plan's decision tree, created during Strategize or Execute. Records the question, chosen option, alternatives, confidence score, rationale, context snapshot, and downstream dependencies. Types: `prompt_definition`, `invariant_enforced`, `strategy_choice`, `subplan_spawn`, `subplan_parallel_spawn`, among others. Supports targeted correction with selective subtree recomputation.
|
||||
|
||||
Invariant
|
||||
: A natural-language constraint on plan execution scoped to global, project, action, or plan level. Action invariants are promoted to plan-level when the action is used, so the runtime precedence chain is three-tier: ==plan > project > global==. Reconciled by the Invariant Reconciliation Actor at the start of Strategize; recorded as `invariant_enforced` decisions that propagate to child plans.
|
||||
|
||||
Automation Profile
|
||||
: A named set of confidence thresholds (each `0.0`–`1.0`) gating which plan operations proceed automatically versus requiring human approval. `0.0` = always automatic; `1.0` = always manual. Eight built-in profiles (`manual` through `full-auto`). Custom profiles namespaced as `[[server:]namespace/]name`. Each profile composes a **Safety Profile** that controls hard safety constraints (sandbox, checkpoint, unsafe-tool gating, skill restrictions, cost/retry limits).
|
||||
|
||||
Safety Profile
|
||||
: A composed sub-model of an Automation Profile that groups all hard safety constraints: `require_sandbox`, `require_checkpoints`, `allow_unsafe_tools`, `require_human_approval`, `allowed_skill_categories`, `max_cost_per_plan`, `max_retries_per_step`, and `max_total_cost`. Safety profiles can also be referenced standalone on Actions when only safety constraints (without full autonomy thresholds) are needed. See [ADR-041](adr/ADR-041-safety-profile-extraction.md).
|
||||
|
||||
???+ abstract "Projects & Resources"
|
||||
|
||||
Project
|
||||
: A named scope linking resources (from the Resource Registry), context policies, invariants, and validation attachments. Does not own resources; a single resource may be linked to multiple projects. Identified solely by its namespaced name (`[[server:]namespace/]name`) — no ULID is generated. May be local or remote.
|
||||
|
||||
Resource
|
||||
: A ULID-identified entity registered in the Resource Registry representing anything readable, writable, or queryable (git repos, filesystems, databases, etc.). Classified as physical or virtual, typed by a Resource Type, and organized in a DAG via parent/child links.
|
||||
|
||||
Resource Type
|
||||
: A schema-level definition constraining a category of resources. Specifies accepted CLI arguments, physical/virtual classification, permitted parent/child type relationships, auto-discovery rules, sandbox strategy, and handler implementation. Built-in types (e.g., `git-checkout`, `fs-mount`) are unnamespaced; custom types are namespaced as `[[server:]namespace/]name`. Resource types support single inheritance via the `inherits` field — see [ADR-042](adr/ADR-042-resource-type-inheritance.md).
|
||||
|
||||
Resource Type Inheritance
|
||||
: A mechanism allowing one resource type to inherit properties, capabilities, child types, sandbox strategy, and handler behavior from another type via the `inherits` field in the type definition. Subtypes can selectively override or extend any inherited field. Tools bound to a parent type automatically work with all subtypes (polymorphic matching). Auto-discovery child type matching and DAG queries are also polymorphic. Single inheritance only; maximum chain depth of 5 levels. See [ADR-042](adr/ADR-042-resource-type-inheritance.md).
|
||||
|
||||
Devcontainer
|
||||
: A container execution environment defined by a `.devcontainer/devcontainer.json` configuration file per the [Development Containers Specification](https://containers.dev). Auto-discovered when a `git-checkout` or `fs-directory` resource contains a `.devcontainer/` directory. Represented as a `devcontainer-instance` resource type that inherits from `container-instance`. Uses lazy activation — the container is only built when first needed by a plan. See [ADR-043](adr/ADR-043-devcontainer-integration.md).
|
||||
|
||||
Execution Environment
|
||||
: The runtime context in which tools execute — either the host system or a specific container. Configurable at project scope (`execution_environment` preference), plan scope (`--execution-environment` flag), and resource scope (auto-detected devcontainers). Precedence resolution determines which environment is used when multiple are configured, with `priority: override` forcing a specific container and `priority: fallback` deferring to auto-detected devcontainers. See [ADR-043](adr/ADR-043-devcontainer-integration.md).
|
||||
|
||||
Physical Resource
|
||||
: A resource bound to a concrete, located artifact — a specific file at a specific path, a specific repo at a specific URL. Directly readable and writable by tools. Two physical resources with identical content remain distinct instances.
|
||||
|
||||
Virtual Resource
|
||||
: A resource representing an abstract equivalence identity that links physical resources sharing the same content, hash, or name. ==Has no location; cannot be directly read or written.== Equivalence relationships update when underlying physical resources diverge.
|
||||
|
||||
Resource Binding
|
||||
: The association between a tool and the resources it operates on, declared via typed resource slots. Slots resolve through **contextual binding** (from the plan's project), **static binding** (hardcoded at registration), or **parameter binding** (passed at invocation).
|
||||
|
||||
Resource Registry
|
||||
: The persistent catalog of all registered resources and their DAG relationships (parent/child links). One of the core registries, alongside the Tool Registry, Skill Registry, Actor Registry, Provider Registry, and LSP Registry.
|
||||
|
||||
???+ abstract "Tools & Skills"
|
||||
|
||||
Tool
|
||||
: The ==atomic unit of execution==: a namespaced, independently registered callable operation. Defined by JSON Schema inputs/outputs, capability metadata (`read_only`, `writes`, `checkpointable`), and a four-stage lifecycle (`discover` / `activate` / `execute` / `deactivate`). Sources: MCP servers, Agent Skills folders, built-ins, or custom Python. Namespaced as `[[server:]namespace/]name`.
|
||||
|
||||
Validation
|
||||
: A Tool subtype adding: a `mode` (`required` | `informational`) controlling whether failure blocks execution; a structured JSON return with mandatory `passed` boolean, optional `data`, and optional `message`. ==Always read-only== (`writes = false`, `checkpointable = false`). May wrap an existing Tool via `wraps` + `transform`. Managed via `agents validation add/attach/detach`; always attached to a resource, optionally scoped to a project or plan.
|
||||
|
||||
Anonymous Tool
|
||||
: An inline tool definition embedded in a skill YAML or actor graph node. Same schema as a named tool but unregistered, unnamespaced, and scoped only to its defining context.
|
||||
|
||||
Skill
|
||||
: A composable, namespaced collection of tools assembled by referencing named tools, defining inline anonymous tools, including other skills, or exposing MCP server tools and Agent Skills ([AgentSkills.io](https://AgentSkills.io)) tools. Actors reference skills by name to acquire capabilities. Namespaced as `[[server:]namespace/]name`.
|
||||
|
||||
MCP (Model Context Protocol)
|
||||
: A standard for exposing tools and resources over a server boundary via JSON-RPC. CleverAgents discovers MCP tools through the `MCPToolAdapter` and registers them in the Tool Registry with extended capability metadata. MCP tools can be composed into skills and used as tool nodes in actor graphs.
|
||||
|
||||
Agent Skills ([AgentSkills.io](https://AgentSkills.io))
|
||||
: A standard for packaging instruction-driven, multi-step workflows as `SKILL.md` files with optional `scripts/`, `references/`, and `assets/` directories, using progressive disclosure. Surfaced as tools inside skills and loaded on demand by the actor runtime.
|
||||
|
||||
???+ abstract "Actors & Sessions"
|
||||
|
||||
Actor
|
||||
: A YAML-configured conversational unit — either a single LLM/agent or a composed LangGraph of actors and tool nodes. Specialized roles: strategy actor, execution actor, estimation actor, invariant reconciliation actor. Namespaced as `[[server:]namespace/]name`.
|
||||
|
||||
Session
|
||||
: A persistent conversation thread tied to an orchestrator actor. Maintains message history across plans and serves as the user's natural-language interface.
|
||||
|
||||
Server
|
||||
: An optional shared backend providing multi-user storage, namespace resolution, and remote plan execution.
|
||||
|
||||
???+ abstract "Protocols & Standards"
|
||||
|
||||
A2A (Agent-to-Agent Protocol)
|
||||
: The external **Agent-to-Agent (A2A) Protocol** standard ([a2a-protocol.org](https://a2a-protocol.org)), built on **JSON-RPC 2.0** (with gRPC and REST bindings available), used as the **sole** communication protocol for all client-server interaction. A2A is the successor to the Agent Client Protocol (ACP), which is now deprecated; A2A retains backward compatibility with ACP's JSON-RPC 2.0 foundation. A2A defines the **fundamental boundary between the Presentation and Application layers**. Standard A2A operations handle agent messaging (`message/send`, `message/stream`), task lifecycle, streaming updates, and Agent Card-based discovery. CleverAgents `_cleveragents/`-prefixed extension methods handle platform operations (plan lifecycle, registry CRUD, entity sync, namespace management, diagnostics). In local mode A2A flows over stdio via JSON-RPC (agent as subprocess); in server mode over HTTP. Every CLI command maps to an A2A operation. See [Core Concepts > Server > A2A](#agent-to-agent-protocol-a2a) and [Server and Client Architecture](#server-and-client-architecture) for full detail.
|
||||
|
||||
LSP (Language Server Protocol)
|
||||
: A standard protocol for language intelligence, used in CleverAgents to attach semantic code understanding to actors and agents. LSP servers are registered in the global **LSP Registry** (namespaced as `[[server:]namespace/]name`) and bound to actor graph nodes via YAML configuration. Capabilities — diagnostics, type information, symbol navigation, completions, references, rename, code actions — are exposed as tools (via `LSPToolAdapter`) and as automatic context enrichment. Actors can bind LSP servers explicitly by name, by language, or automatically based on detected resource languages. The LSP Runtime in the Infrastructure layer manages server lifecycle, workspace mapping, and file synchronization. See [LSP Integration](#lsp-integration) for full detail.
|
||||
|
||||
???+ abstract "Naming & Identity"
|
||||
|
||||
Namespace
|
||||
: The scoping segment in the name format `[[server:]namespace/]name`. Defaults to `local/` when omitted. `local/` is reserved for local-only items. Non-`local/` namespaces with server omitted assume the default configured server. Built-in LLM actors use provider prefixes (e.g., `openai/`, `anthropic/`). Built-in resource types are unnamespaced.
|
||||
|
||||
ULID
|
||||
: ==Universally Unique Lexicographically Sortable Identifier.== Assigned to plans, decisions, resources, correction attempts, and validation attachments. Projects, actions, skills, and tools use their namespaced name as sole identifier instead.
|
||||
|
||||
???+ abstract "Context Management (ACMS)"
|
||||
|
||||
ACMS (Advanced Context Management System)
|
||||
: The pluggable, strategy-driven framework for assembling actor context. Comprises the UKO, CRP, pluggable context strategies, the Context Assembly Pipeline, and hot/warm/cold tiered storage with per-actor scoped views.
|
||||
|
||||
UKO (Universal Knowledge Ontology)
|
||||
: An RDF-based, inheritance-driven ontology representing resources at multiple abstraction levels with provenance and temporal versioning. Four layers: universal foundation, domain specializations (software, documents, data schemas, infrastructure), paradigm/format specializations (procedural programming, markdown), and technology-specific (Python, PostgreSQL). ==Semantically aware — implicit relationships are inferred from content analysis.==
|
||||
|
||||
CRP (Context Request Protocol)
|
||||
: A structured vocabulary through which actors declare needed information, desired detail depth, and scope.
|
||||
|
||||
Context Strategy
|
||||
: A pluggable retrieval component that searches for and assembles ContextFragments using a specific approach (keyword search, semantic embedding, graph navigation, temporal archaeology, etc.). Registered with the Context Assembly Pipeline; executed in parallel by the StrategyExecutor.
|
||||
|
||||
Context Assembly Pipeline
|
||||
: The central ACMS orchestrator. Ten pluggable Protocol-defined components in three phases: **Strategy Orchestration**, **Fragment Fusion**, and **Context Finalization**. Each component ships with a default implementation and is overridable at global, project, or plan scope.
|
||||
|
||||
Skeleton (context)
|
||||
: A compressed representation of a plan's accumulated context produced by the SkeletonCompressor. Propagated from parent plans to child plans as inherited context. Size governed by the `skeleton_ratio` budget parameter.
|
||||
@@ -0,0 +1,84 @@
|
||||
# CleverAgents Documentation (Detailed Spec)
|
||||
|
||||
## Overview
|
||||
|
||||
CleverAgents is your **command center for AI agents** — a unified platform for orchestrating any task you want agents to accomplish, from developing large software projects to writing comprehensive technical papers, administering databases, managing cloud infrastructure, or any complex multi-step workflow.
|
||||
|
||||
The core value proposition is enabling long-running, complex, large-scale tasks to execute autonomously with minimal human intervention, making it ideal for building entire software systems, producing extensive documentation, or managing sophisticated operations largely hands-off.
|
||||
|
||||
!!! info "Server Mode"
|
||||
|
||||
In **server mode**, CleverAgents becomes a collaborative hub where teams can share resources — prompts, actors, actions, and projects — while executing plans in the cloud. This enables a consistent experience across all your devices: start a complex task on your laptop, check progress from your phone, and review results from any machine.
|
||||
|
||||
!!! note "Runtime Foundation"
|
||||
|
||||
While CleverAgents leverages LangGraph and LangChain for the underlying LLM runtime primitives (tool calling, graphs, routing), its value lies in what it builds on top of these foundations.
|
||||
|
||||
!!! success "Core Capabilities"
|
||||
|
||||
- [x] A **first-class plan lifecycle** (Action templates driving Strategize / Execute / Apply phases) for breaking down and tracking complex work
|
||||
- [x] A **project + resource model** for grounding tasks in real codebases, databases, documents, and infrastructure
|
||||
- [x] A consistent **actor abstraction** for defining and composing intelligent agents
|
||||
- [x] An independently registered **resource abstraction** for representing anything that can be read, written, or queried
|
||||
- [x] An independently registered **tool abstraction** for reusable, callable operations with resource bindings
|
||||
- [x] A **validation abstraction** that extends the tool concept with pass/fail semantics and resource-centric attachment with optional project/plan scoping
|
||||
- [x] A consistent **skill abstraction** for organizing tools into composable capability collections
|
||||
- [x] A **sandbox + checkpoint** safety model for safe, reversible execution
|
||||
- [x] A **CLI/TUI/Web UX** for controlling and monitoring large multi-step autonomous work
|
||||
|
||||
!!! example "Advanced Subsystems"
|
||||
|
||||
- A scalable **Advanced Context Management System (ACMS)** with a Universal Knowledge Ontology (UKO), a demand-driven Context Request Protocol (CRP), pluggable context strategies, a fusion coordinator, and hot/warm/cold tiers with per-actor views.
|
||||
- **Invariants** as first-class constraints (global, project, action, and plan scoped) that flow into the decision tree, with precedence-based conflict resolution via the Invariant Reconciliation Actor.
|
||||
- A future-facing ==correction model== where the user can "edit the decision tree" and only recompute affected subtrees.
|
||||
|
||||
### Standards Alignment
|
||||
|
||||
CleverAgents deliberately adopts open, versioned protocols wherever possible so that clients, tools, and skills can interoperate without bespoke integrations.
|
||||
|
||||
!!! tip "Guiding Principles"
|
||||
|
||||
1. **Prefer open protocols** — align with community standards to keep integrations portable and reduce vendor lock-in.
|
||||
2. **Keep adapters at the edge** — standards map into stable internal domain models so core logic remains protocol-agnostic.
|
||||
|
||||
The following standards are integrated into the architecture:
|
||||
|
||||
| Standard | Role | Key Benefit |
|
||||
| :------- | :--- | :---------- |
|
||||
| **A2A** (Agent-to-Agent Protocol) | Versioned client-server contract for messaging, task lifecycle, plan management, registry access, and event streaming | Clients are interchangeable; Agent Card discovery enables ecosystem interoperability; reliable remote execution in server mode |
|
||||
| **MCP** (Model Context Protocol) | Discovering and invoking external tools over a server boundary | Plug-and-play access to a growing ecosystem of tool providers |
|
||||
| **LSP** (Language Server Protocol) | Attaching language intelligence (diagnostics, type info, symbol navigation, completions) to actors and agents | Actors gain semantic code understanding from the mature LSP ecosystem without bespoke language analysis |
|
||||
| **Agent Skills** ([AgentSkills.io](https://AgentSkills.io)) | Packaging instruction-driven, multi-step workflows as `SKILL.md` with progressive disclosure | Teaches agents *how* to accomplish complex tasks, complementing MCP tools |
|
||||
|
||||
??? info "Agent-to-Agent Protocol (A2A) -- Details"
|
||||
|
||||
CleverAgents adopts the external **Agent-to-Agent (A2A) Protocol** standard ([a2a-protocol.org](https://a2a-protocol.org)) as the **sole** communication protocol for all client-server interaction. A2A is the successor to the Agent Client Protocol (ACP), which is now deprecated; A2A retains backward compatibility with ACP's JSON-RPC 2.0 foundation. A2A is built on **JSON-RPC 2.0** (with additional gRPC and REST bindings available) and defines the **fundamental boundary between the Presentation and Application layers** — every client operation flows through A2A regardless of deployment mode. The standard provides operations for messaging (`message/send`, `message/stream`), task lifecycle management, streaming updates via SSE, and **Agent Card**-based capability discovery. CleverAgents extends the standard with `_cleveragents/`-prefixed extension methods (declared via the A2A extension mechanism) for platform operations: plan lifecycle, registry CRUD, entity sync, namespace management, and diagnostics. In local mode, A2A flows over **stdio** via the JSON-RPC binding (agent as subprocess) with platform operations resolved in-process via `A2aLocalFacade`. In server mode, A2A flows over **HTTP** to the CleverAgents server. Both transports use the A2A Python SDK. All clients — CLI, TUI, IDE plugin, and third-party — communicate exclusively through A2A. See [ADR-047](adr/ADR-047-acp-standard-adoption.md), [ADR-048](adr/ADR-048-server-application-architecture.md), and [Server and Client Architecture](#server-and-client-architecture) for the full detail.
|
||||
|
||||
??? info "Model Context Protocol (MCP) -- Details"
|
||||
|
||||
The standard for discovering and invoking external tools over a server boundary. MCP servers are bridged into the Tool Registry via adapters, giving CleverAgents plug-and-play access to a growing ecosystem of tool providers.
|
||||
|
||||
??? info "Language Server Protocol (LSP) -- Details"
|
||||
|
||||
The standard for attaching language intelligence to actors and agents. LSP servers are registered in a global LSP Registry (namespaced like all other entities) and bound to actor graph nodes via YAML configuration. When an actor activates, the LSP Runtime starts the appropriate language servers for the actor's bound languages and workspace resources. LSP capabilities — diagnostics, type information, symbol navigation, completions, references, rename, code actions, and more — are exposed to the actor as callable tools (via the `LSPToolAdapter`) and as automatic context enrichment (diagnostics and type annotations injected into the ACMS hot context). Actors can bind LSP servers explicitly by name, by language, or automatically based on the languages detected in their project's resources. Different nodes in an actor's graph can have different LSP bindings, enabling fine-grained control over which agents receive which language intelligence capabilities. See [LSP Integration](#lsp-integration) for the full architectural detail and [ADR-027](adr/ADR-027-language-server-protocol.md) for the decision record.
|
||||
|
||||
??? info "Agent Skills (AgentSkills.io) -- Details"
|
||||
|
||||
The standard for packaging instruction-driven, multi-step workflows as `SKILL.md` with progressive disclosure. Agent Skills complement MCP tools by teaching agents *how* to accomplish complex tasks rather than simply exposing callable functions.
|
||||
|
||||
## Specification Sections
|
||||
|
||||
This specification is organized into the following sections:
|
||||
|
||||
| Section | Description |
|
||||
|---------|-------------|
|
||||
| [Glossary](glossary.md) | All glossary entries (Plan, Action, Resource, Actor, Tool, Skill, Session, etc.) |
|
||||
| [CLI Commands](cli.md) | Complete CLI command reference for all `agents *` commands |
|
||||
| [Core Concepts](core-concepts.md) | Plan lifecycle, actors, tools, skills, sessions, resources, context, output rendering |
|
||||
| [Behavior](behavior.md) | Automation profiles, guardrails, plan corrections, human-in-the-loop |
|
||||
| [TUI](tui.md) | Text User Interface architecture, persona system, reference/command system, widgets |
|
||||
| [Configuration](configuration.md) | Configuration system, global config keys, environment variables |
|
||||
| [Workflow Examples](workflow-examples.md) | End-to-end workflow examples and integration patterns |
|
||||
| [Architecture](architecture.md) | Layered architecture, domain models, data validation, storage, persistence |
|
||||
|
||||
---
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
+10
-1
@@ -9,7 +9,16 @@ site_url: https://docs.cleverthis.com/cleveragents
|
||||
site_dir: build/site
|
||||
|
||||
nav:
|
||||
- Specification: specification.md
|
||||
- Specification:
|
||||
- Overview: specification/index.md
|
||||
- Glossary: specification/glossary.md
|
||||
- CLI Commands: specification/cli.md
|
||||
- Core Concepts: specification/core-concepts.md
|
||||
- Behavior: specification/behavior.md
|
||||
- TUI: specification/tui.md
|
||||
- Configuration: specification/configuration.md
|
||||
- Workflow Examples: specification/workflow-examples.md
|
||||
- Architecture: specification/architecture.md
|
||||
- Architecture: architecture.md
|
||||
- API Reference:
|
||||
- Overview: api/index.md
|
||||
|
||||
Reference in New Issue
Block a user