docs(spec): architecture cycle 25 — model_tier, autonomous shell blocking, in-actor compaction, uncertainty band escalation
CI / benchmark-publish (pull_request) Has been skipped
CI / push-validation (pull_request) Successful in 27s
CI / helm (pull_request) Successful in 29s
CI / build (pull_request) Successful in 37s
CI / quality (pull_request) Successful in 40s
CI / lint (pull_request) Successful in 45s
CI / typecheck (pull_request) Successful in 51s
CI / security (pull_request) Successful in 1m0s
CI / e2e_tests (pull_request) Successful in 3m16s
CI / integration_tests (pull_request) Successful in 4m4s
CI / unit_tests (pull_request) Successful in 5m9s
CI / docker (pull_request) Successful in 1m19s
CI / coverage (pull_request) Successful in 11m17s
CI / status-check (pull_request) Successful in 1s
CI / benchmark-regression (pull_request) Successful in 58m40s
CI / benchmark-publish (pull_request) Has been skipped
CI / push-validation (pull_request) Successful in 27s
CI / helm (pull_request) Successful in 29s
CI / build (pull_request) Successful in 37s
CI / quality (pull_request) Successful in 40s
CI / lint (pull_request) Successful in 45s
CI / typecheck (pull_request) Successful in 51s
CI / security (pull_request) Successful in 1m0s
CI / e2e_tests (pull_request) Successful in 3m16s
CI / integration_tests (pull_request) Successful in 4m4s
CI / unit_tests (pull_request) Successful in 5m9s
CI / docker (pull_request) Successful in 1m19s
CI / coverage (pull_request) Successful in 11m17s
CI / status-check (pull_request) Successful in 1s
CI / benchmark-regression (pull_request) Successful in 58m40s
Add spec coverage for four new architectural features: 1. Actor node model_tier field: Optional per-node model tier override (cheap/default/frontier) in graph route nodes. Tier-to-model mapping via new model.tiers.* config keys. Enables cost-optimised actor graphs without hardcoding model names. 2. Autonomous shell blocking: Safety Profile integration with ShellSafetyService. When allow_unsafe_tools=false and profile is headless (ci/full-auto), CRITICAL+HIGH shell patterns are hard-blocked rather than advisory. TUI remains advisory-only. 3. In-actor conversation history compaction: LangGraph actor runner hook that monitors accumulated message history and summarises old turns when threshold exceeded. Distinct from ACMS (which handles retrieval); this handles within-session accumulation. New actor.compaction.* config keys. 4. Uncertainty band LLM escalation: Optional two-stage AutonomyController augmentation. Stage 1 heuristic runs always (zero LLM cost). Stage 2 cheap LLM evaluator activates only when score falls within uncertainty band around threshold. Result caching and circuit breaker included. New escalation.classifier.* config keys. Closes: #6765, #6763, #6761, #6760 ISSUES CLOSED: #6765, #6763, #6761, #6760
This commit is contained in:
@@ -28522,6 +28522,22 @@ A `SafetyProfile` may also be attached directly to an `Action` (via the `safety_
|
||||
|
||||
**Relationship to Automation Guards**: The `SafetyProfile.max_total_cost` field sets a **plan-level** budget cap (broad scope), while `AutomationGuard.max_total_cost` sets a **per-invocation** budget cap (narrow scope). These operate at different granularities and both may be active simultaneously — the tighter constraint takes precedence at any given point.
|
||||
|
||||
**Shell command blocking in autonomous mode**: When `allow_unsafe_tools` is `false` AND the active automation profile is headless (`ci` or `full-auto`), the `ShellSafetyService` is configured in **blocking mode** for `CRITICAL` and `HIGH` danger-level patterns. This is distinct from the TUI's advisory-only shell danger detection (§Shell Danger Detection), which never blocks. The distinction is:
|
||||
|
||||
- **TUI (interactive)**: Shell danger detection is advisory only — warnings are shown but execution always proceeds. The human is present and can make informed decisions.
|
||||
- **Autonomous execution (`ci`/`full-auto` with `allow_unsafe_tools: false`)**: `CRITICAL` and `HIGH` patterns are hard-blocked. The sandbox handles most reversible operations, but certain patterns (recursive deletion of sandbox root, disk formatting, fork bombs) can destroy the sandbox itself before any checkpoint can help. Blocked commands produce a structured denial that is fed back into the actor's context, allowing the actor to reason about alternatives.
|
||||
|
||||
The `ShellSafetyService` `block_level` is determined at plan execution start based on the resolved Safety Profile:
|
||||
|
||||
```python
|
||||
if safety_profile.allow_unsafe_tools is False and profile.is_headless:
|
||||
block_level = ShellDangerLevel.HIGH # CRITICAL + HIGH patterns hard-block
|
||||
else:
|
||||
block_level = ShellDangerLevel.CRITICAL # Advisory-only (TUI behaviour, nothing actually blocks)
|
||||
```
|
||||
|
||||
`profile.is_headless` is `True` for `ci` and `full-auto` built-in profiles, and for any custom profile where all thresholds are `0.0` (fully automatic). This field is computed at profile resolution time and stored on the resolved profile object.
|
||||
|
||||
#### Automation Guard Sub-Model
|
||||
|
||||
An `AutomationProfile` may optionally compose an `AutomationGuard` sub-model (via the `guards` field) that provides runtime enforcement hooks beyond the phase-transition thresholds. Guards gate individual tool invocations based on call counts, budgets, allowlists/denylists, and write/apply semantics.
|
||||
@@ -28688,6 +28704,41 @@ The confidence score is computed from multiple factors:
|
||||
|
||||
**Special cases**: A threshold of **0.0** means "always automatic" — even zero confidence passes the threshold. A threshold of **1.0** means "always manual" — no confidence level (which tops out below 1.0 in practice) is high enough to pass. This means the old boolean behavior is a strict subset: `true` maps to `0.0` and `false` maps to `1.0`.
|
||||
|
||||
#### Uncertainty Band LLM Escalation (Optional Two-Stage Classifier)
|
||||
|
||||
The heuristic `AutonomyController` computes confidence from population-level statistics (past success rate, codebase familiarity, risk assessment, invariant complexity). These factors are blind to the specific content of the tool call being evaluated — they cannot distinguish `git status` from `git reset --hard HEAD~10` for the `execute_command` flag.
|
||||
|
||||
An optional **two-stage classifier** augments the heuristic when the confidence score falls within an **uncertainty band** around the active threshold. The LLM is only invoked for genuinely ambiguous decisions; clear-cut cases (well above or well below the threshold) proceed without any LLM call.
|
||||
|
||||
```
|
||||
Stage 1 (always runs): Heuristic AutonomyController — zero LLM cost
|
||||
├── score >= threshold + band → proceed automatically (no LLM call)
|
||||
├── score <= threshold - band → escalate to user (no LLM call)
|
||||
└── score within uncertainty band → Stage 2
|
||||
|
||||
Stage 2 (optional, only for ambiguous cases): Cheap LLM evaluator
|
||||
reads actual tool call name + args + context snapshot
|
||||
├── safe → proceed automatically
|
||||
└── unsafe → escalate or request confirmation
|
||||
```
|
||||
|
||||
**Configuration keys** (under the `escalation` namespace):
|
||||
|
||||
| Key | Type | Default | Description |
|
||||
|-----|------|---------|-------------|
|
||||
| `escalation.classifier.enabled` | boolean | `false` | Enable the two-stage LLM classifier. When `false`, the heuristic-only path is used. |
|
||||
| `escalation.classifier.model` | string | *(uses `context.summarize.model`)* | Model for Stage 2 evaluation. Should be a fast, cheap model (e.g., a small variant). |
|
||||
| `escalation.classifier.uncertainty-band` | float | `0.10` | Half-width of the uncertainty band around the threshold. A value of `0.10` means Stage 2 activates when `|score - threshold| <= 0.10`. Range: 0.0–0.5. |
|
||||
| `escalation.classifier.cache-ttl-seconds` | integer | `3600` | TTL for caching Stage 2 results by `(tool_name, argument_fingerprint, operation_type)`. Set to `0` to disable caching. |
|
||||
|
||||
**Integration point**: The `AutonomyController` accepts an optional `llm_classifier` dependency. When absent, behaviour is identical to the existing heuristic-only path. When present, the classifier is invoked only for decisions in the uncertainty band.
|
||||
|
||||
**Result caching**: Stage 2 results are cached by a hash of `(tool_name, argument_fingerprint, operation_type)` with the configured TTL. This avoids redundant LLM calls for identical repeated operations (e.g., the same `pytest` invocation called many times in a test loop).
|
||||
|
||||
**Circuit breaker**: After 3 consecutive Stage 2 failures (LLM error, timeout, or malformed response), the classifier is disabled for the remainder of the plan execution and the heuristic-only path is used. The circuit-breaker state is logged and surfaced in `agents plan guard`.
|
||||
|
||||
**Scope**: The two-stage classifier is a plan-execution concern, not a TUI concern. It operates during autonomous plan execution (Execute phase) and is irrelevant in interactive TUI sessions where a human is present.
|
||||
|
||||
#### Progressive Trust Building
|
||||
|
||||
New users typically follow this progression:
|
||||
@@ -30816,6 +30867,41 @@ Provider credential keys follow the pattern `provider.<name>.<field>`. Each prov
|
||||
| `provider.huggingface.token` | string | *(not set)* | `HF_TOKEN` | Access token for Hugging Face Inference API. Required when using Hugging Face-hosted models. The standard `HF_TOKEN` environment variable is checked for compatibility with Hugging Face tooling. |
|
||||
| `provider.openrouter.api-key` | string | *(not set)* | `OPENROUTER_API_KEY` | API key for OpenRouter. Required when using OpenRouter as a provider. |
|
||||
|
||||
##### `model.*` — Model Tier Mapping
|
||||
|
||||
The `model.tiers` table maps tier names to concrete model identifiers. These tiers are referenced by the optional `model_tier` field on graph nodes in actor YAML configuration, enabling per-node model selection without hardcoding model names in actor definitions.
|
||||
|
||||
| Key | Type | Default | Env Variable | Description |
|
||||
|-----|------|---------|-------------|-------------|
|
||||
| `model.tiers.cheap` | string | *(not set)* | `CLEVERAGENTS_MODEL_TIER_CHEAP` | Model identifier for the `cheap` tier. Intended for lightweight nodes (binary checks, simple lookups, type annotation verification). When unset, nodes with `model_tier: cheap` fall back to the actor's default model. |
|
||||
| `model.tiers.default` | string | *(not set)* | `CLEVERAGENTS_MODEL_TIER_DEFAULT` | Model identifier for the `default` tier. When set, this overrides the actor's configured model for nodes with `model_tier: default`. When unset, nodes with `model_tier: default` use the actor's configured model unchanged. |
|
||||
| `model.tiers.frontier` | string | *(not set)* | `CLEVERAGENTS_MODEL_TIER_FRONTIER` | Model identifier for the `frontier` tier. Intended for the most capable model available, used for complex reasoning nodes (architectural decisions, invariant conflict resolution, error recovery). When unset, nodes with `model_tier: frontier` fall back to the actor's default model. |
|
||||
|
||||
**Tier resolution algorithm** for a graph node with `model_tier: <tier>`:
|
||||
1. Look up `model.tiers.<tier>` in the resolved configuration.
|
||||
2. If set, use that model identifier for this node's LLM invocation.
|
||||
3. If not set, use the actor's configured `model` field (no change from current behaviour).
|
||||
|
||||
This is **configuration-driven, not automatic** — the actor author declares the tier in the YAML; the tier-to-model mapping is a deployment concern set in `config.toml` or environment variables.
|
||||
|
||||
##### `escalation.*` — Uncertainty Band LLM Escalation
|
||||
|
||||
| Key | Type | Default | Env Variable | Description |
|
||||
|-----|------|---------|-------------|-------------|
|
||||
| `escalation.classifier.enabled` | boolean | `false` | `CLEVERAGENTS_ESCALATION_CLASSIFIER` | Enable the optional two-stage LLM classifier for the `AutonomyController`. When `false`, the heuristic-only path is used for all decisions. |
|
||||
| `escalation.classifier.model` | string | *(uses `context.summarize.model`)* | `CLEVERAGENTS_ESCALATION_MODEL` | Model for Stage 2 LLM evaluation. Should be a fast, cheap model. When unset, falls back to `context.summarize.model`, then to the active actor's model. |
|
||||
| `escalation.classifier.uncertainty-band` | float | `0.10` | `CLEVERAGENTS_ESCALATION_BAND` | Half-width of the uncertainty band around the automation threshold. Stage 2 activates when `|score - threshold| <= band`. Range: 0.0–0.5. |
|
||||
| `escalation.classifier.cache-ttl-seconds` | integer | `3600` | `CLEVERAGENTS_ESCALATION_CACHE_TTL` | TTL in seconds for caching Stage 2 results by `(tool_name, argument_fingerprint, operation_type)`. Set to `0` to disable caching. |
|
||||
|
||||
##### `actor.compaction.*` — In-Actor Conversation History Compaction
|
||||
|
||||
| Key | Type | Default | Env Variable | Description |
|
||||
|-----|------|---------|-------------|-------------|
|
||||
| `actor.compaction.enabled` | boolean | `true` | `CLEVERAGENTS_ACTOR_COMPACTION` | Enable within-session message history compaction. When `true`, the compaction hook monitors accumulated message history and summarises old turns when the threshold is exceeded. |
|
||||
| `actor.compaction.threshold` | float | `0.75` | `CLEVERAGENTS_ACTOR_COMPACTION_THRESHOLD` | Fraction of the model's context window (minus ACMS hot budget) at which compaction triggers. Range: 0.1–0.95. |
|
||||
| `actor.compaction.keep-recent-turns` | integer | `8` | `CLEVERAGENTS_ACTOR_COMPACTION_KEEP` | Number of most-recent tool call turns to preserve intact during compaction. These turns are not summarised. |
|
||||
| `actor.compaction.model` | string | *(uses `context.summarize.model`)* | `CLEVERAGENTS_ACTOR_COMPACTION_MODEL` | Model to use for compaction summarisation. When unset, falls back to `context.summarize.model`, then to the active actor's model. |
|
||||
|
||||
#### Configuration Scoping
|
||||
|
||||
Several keys support **project-scoped** values in addition to the global default. When a project-scoped value is set, it applies only to plans targeting that project. Keys marked **Project-scopable** in the reference above support this feature.
|
||||
@@ -31465,6 +31551,11 @@ The following is the formal [JSON Schema](https://json-schema.org/) definition f
|
||||
<span style="color: cyan; font-weight: 600;">"retry_policy"</span>: { <span style="color: cyan; font-weight: 600;">"type"</span>: <span style="color: #66cc66;">"object"</span>, <span style="color: cyan; font-weight: 600;">"additionalProperties"</span>: <span style="color: magenta; font-weight: 600;">true</span>, <span style="color: cyan; font-weight: 600;">"description"</span>: <span style="color: #66cc66;">"Retry configuration."</span> },
|
||||
<span style="color: cyan; font-weight: 600;">"timeout"</span>: { <span style="color: cyan; font-weight: 600;">"type"</span>: <span style="color: #66cc66;">"integer"</span>, <span style="color: cyan; font-weight: 600;">"minimum"</span>: <span style="color: yellow;">1</span>, <span style="color: cyan; font-weight: 600;">"description"</span>: <span style="color: #66cc66;">"Timeout in seconds."</span> },
|
||||
<span style="color: cyan; font-weight: 600;">"parallel"</span>: { <span style="color: cyan; font-weight: 600;">"type"</span>: <span style="color: #66cc66;">"boolean"</span>, <span style="color: cyan; font-weight: 600;">"default"</span>: <span style="color: magenta; font-weight: 600;">false</span>, <span style="color: cyan; font-weight: 600;">"description"</span>: <span style="color: #66cc66;">"Allow parallel execution."</span> },
|
||||
<span style="color: cyan; font-weight: 600;">"model_tier"</span>: {
|
||||
<span style="color: cyan; font-weight: 600;">"type"</span>: <span style="color: #66cc66;">"string"</span>,
|
||||
<span style="color: cyan; font-weight: 600;">"enum"</span>: [<span style="color: #66cc66;">"cheap"</span>, <span style="color: #66cc66;">"default"</span>, <span style="color: #66cc66;">"frontier"</span>],
|
||||
<span style="color: cyan; font-weight: 600;">"description"</span>: <span style="color: #66cc66;">"Optional model tier override for agent-type nodes. When set, the node uses the model mapped to this tier in the global model.tiers config instead of the actor's default model. Only applies to nodes of type 'agent'. Absent means use the actor's default model."</span>
|
||||
},
|
||||
<span style="color: cyan; font-weight: 600;">"metadata"</span>: { <span style="color: cyan; font-weight: 600;">"type"</span>: <span style="color: #66cc66;">"object"</span>, <span style="color: cyan; font-weight: 600;">"additionalProperties"</span>: <span style="color: magenta; font-weight: 600;">true</span> }
|
||||
},
|
||||
<span style="color: cyan; font-weight: 600;">"required"</span>: [<span style="color: #66cc66;">"type"</span>],
|
||||
@@ -31611,6 +31702,9 @@ The following annotated YAML provides an easier-to-read overview of the same sch
|
||||
<span style="color: cyan; font-weight: 600;">retry_policy</span>: {}<span style="opacity: 0.7;"> # Retry configuration (optional)</span>
|
||||
<span style="color: cyan; font-weight: 600;">timeout</span>: <span style="color: yellow;">30</span><span style="opacity: 0.7;"> # Timeout in seconds (optional)</span>
|
||||
<span style="color: cyan; font-weight: 600;">parallel</span>: <span style="color: magenta; font-weight: 600;">false</span><span style="opacity: 0.7;"> # Allow parallel execution (optional)</span>
|
||||
<span style="color: cyan; font-weight: 600;">model_tier</span>: <span style="color: magenta;">cheap | default | frontier</span><span style="opacity: 0.7;"> # Model tier override for agent nodes (optional)</span>
|
||||
<span style="opacity: 0.7;"> # When set, uses the model mapped to this tier in model.tiers config</span>
|
||||
<span style="opacity: 0.7;"> # instead of the actor's default model. Absent = use actor default.</span>
|
||||
<span style="color: cyan; font-weight: 600;">metadata</span>: {}<span style="opacity: 0.7;"> # Arbitrary metadata (optional)</span>
|
||||
<span style="color: cyan; font-weight: 600;">edges</span>: # Graph edges (required for graph routes)
|
||||
- <span style="color: cyan; font-weight: 600;">source</span>: <span style="color: #66cc66;"><node_name></span><span style="opacity: 0.7;"> # Source node (required)</span>
|
||||
@@ -46737,6 +46831,59 @@ The following table shows which Protocol each pipeline slot implements and what
|
||||
| **Pipeline: PreambleGenerator** | `config.toml` `context.pipeline.preamble-generator` or project/plan YAML | TOML/YAML + Python module | Configuration-driven (scope chain) |
|
||||
| **Pipeline: SkeletonCompressor** | `config.toml` `context.pipeline.skeleton-compressor` or project/plan YAML | TOML/YAML + Python module | Configuration-driven (scope chain) |
|
||||
|
||||
### In-Actor Conversation History Compaction
|
||||
|
||||
The ACMS assembles context at the **start** of each actor invocation from the indexed knowledge graph. It does not manage what happens to the LangGraph message history **during** a long Execute actor session involving many tool call turns. If an Execute actor makes 40–60 tool calls, the accumulated LangGraph message history (tool calls + results + model responses) grows continuously and will eventually hit the model's context window limit.
|
||||
|
||||
This is a distinct concern from the ACMS: the ACMS solves retrieval; in-actor compaction solves accumulation.
|
||||
|
||||
#### Compaction Hook
|
||||
|
||||
A compaction hook is added to the LangGraph actor runner that:
|
||||
|
||||
1. **Monitors** accumulated message history token count after each tool-call turn
|
||||
2. **Triggers** when the count exceeds a configurable threshold (default: 75% of the model's context window minus the ACMS hot context budget)
|
||||
3. **Summarises** the tool call + result history using `context.summarize.model` (the existing cheap model config), replacing the oldest N turns with a compact summary
|
||||
4. **Preserves** the most recent K turns intact (configurable via `actor.compaction.keep-recent-turns`, default: 8) so the actor retains full fidelity for its current work
|
||||
5. **Circuit-breaker**: Disable compaction for the rest of the session after 3 consecutive failures (same pattern as ACMS `ParallelStrategyExecutor`)
|
||||
|
||||
#### Compaction Configuration Keys
|
||||
|
||||
| Key | Type | Default | Description |
|
||||
|-----|------|---------|-------------|
|
||||
| `actor.compaction.enabled` | boolean | `true` | Enable within-session message history compaction. |
|
||||
| `actor.compaction.threshold` | float | `0.75` | Fraction of the model's context window (minus ACMS hot budget) at which compaction triggers. Range: 0.1–0.95. |
|
||||
| `actor.compaction.keep-recent-turns` | integer | `8` | Number of most-recent tool call turns to preserve intact. These are not summarised. |
|
||||
| `actor.compaction.model` | string | *(uses `context.summarize.model`)* | Model to use for compaction summarisation. Defaults to the context summarisation model. |
|
||||
|
||||
#### Summary Structure
|
||||
|
||||
The compaction summarisation prompt produces a structured summary with these sections:
|
||||
|
||||
1. **Primary task context** — what the actor was working on
|
||||
2. **Key files and code sections touched** — file paths and relevant code areas
|
||||
3. **Decisions made and why** — significant choices and their rationale
|
||||
4. **Errors encountered and how they were resolved** — problems and their solutions
|
||||
5. **Tool calls made** — compact list with outcomes (success/failure, key results)
|
||||
6. **Pending items** — what still needs to be done to complete the current task
|
||||
|
||||
This structure preserves what an actor needs to resume effectively without the full message history.
|
||||
|
||||
#### Compaction vs. ACMS
|
||||
|
||||
| Concern | Solved by |
|
||||
|---------|-----------|
|
||||
| Context assembly at invocation start | ACMS (external knowledge retrieval) |
|
||||
| Message history overflow during session | In-actor compaction (internal accumulation) |
|
||||
|
||||
These are complementary, not competing. Both may be active simultaneously. The compaction hook operates on the LangGraph `messages` state key; the ACMS operates on the `context` state key.
|
||||
|
||||
#### Implementation Location
|
||||
|
||||
The compaction hook is implemented in `src/cleveragents/langgraph/` (either in `nodes.py` or a new `compaction.py` module). It is invoked as a post-tool-call hook in the actor execution loop, before the next LLM turn begins.
|
||||
|
||||
---
|
||||
|
||||
## Milestone Plan
|
||||
|
||||
This section defines the ordered milestone plan for CleverAgents v3.x, mapping architectural features to verifiable deliverables. Each milestone builds on the previous and is independently testable. Milestones v3.0.0 and v3.1.0 are **complete**. This plan covers v3.2.0 through v3.7.0 — the production-ready target.
|
||||
|
||||
Reference in New Issue
Block a user