Compare commits

...

1 Commits

Author SHA1 Message Date
HAL9000 a47b372ba5 docs(spec): update specification — validation gate blocks on empty validation summary
CI / benchmark-publish (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Failing after 19s
CI / push-validation (pull_request) Successful in 50s
CI / helm (pull_request) Successful in 52s
CI / lint (pull_request) Successful in 1m54s
CI / build (pull_request) Successful in 1m50s
CI / quality (pull_request) Successful in 2m15s
CI / typecheck (pull_request) Successful in 2m21s
CI / security (pull_request) Successful in 2m37s
CI / integration_tests (pull_request) Successful in 6m2s
CI / e2e_tests (pull_request) Failing after 6m27s
CI / unit_tests (pull_request) Successful in 9m33s
CI / docker (pull_request) Successful in 1m42s
CI / coverage (pull_request) Successful in 12m50s
CI / status-check (pull_request) Failing after 4s
Updated docs/specification.md to document the security invariant
introduced in PR #7786 (fixing issue #7508). The spec now explicitly
states that ApplyValidationSummary.all_required_passed returns False
when no validations have been run (empty summary), blocking apply.

Also updated CHANGELOG.md with a Changed entry under [Unreleased] and
added contributors entry for HAL 9000 spec maintenance.

ISSUES CLOSED: #8146
2026-05-07 14:11:16 +00:00
3 changed files with 25 additions and 4 deletions
+7
View File
@@ -529,6 +529,13 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
delays from 15s to 2s, and reduced idle sleep from 60s to 10s for dramatically
faster throughput.
- **Specification — Validation Gate Empty-Run Guard** (#8146): Updated `docs/specification.md`
to document the security invariant introduced in PR #7786 (fixing issue #7508). The spec now
explicitly states that `ApplyValidationSummary.all_required_passed` returns `False` when no
validations have been run (empty summary), blocking apply. Added a prominent danger admonition
block, updated the validation process results section, the `final_validation_results` data
model description, and two milestone acceptance criteria to reflect the corrected blocking
behavior for empty validation summaries and no-attachment runs.
### Fixed
+1
View File
@@ -21,6 +21,7 @@ Below are some of the specific details of various contributions.
* HAL 9000 has contributed the plugin entry point security hardening fix (#7476): enforced entry point allowlist validation before importing plugin modules to prevent malicious plugin loading.
* HAL 9000 has contributed the benchmark workflow separation (#9040): moved the benchmark-regression job out of the default PR workflow into a dedicated scheduled workflow, reducing median PR CI turnaround time from 99-132 minutes to under 30 minutes.
* HAL 9000 has contributed the agent-evolution-pool-supervisor PR metadata assignment (#7888): the supervisor now automatically looks up the Type/Automation label and earliest open milestone before dispatching improvement PR creation workers, ensuring all generated improvement PRs have correct Type labels and milestone assignments.
* HAL 9000 has contributed automated specification maintenance, documentation updates, and bot-driven PR authorship.
* This project was made possible thanks to considerable donation of time, money, and resources by CleverThis, Inc.
* HAL 9000 has contributed automated bug fixes, CLI output formatting improvements, and ongoing maintenance as part of the CleverAgents automation system.
* HAL 9000 has contributed the file edit encoding parameter fix (PR #8258 / issue #7559).
+17 -4
View File
@@ -20019,6 +20019,18 @@ Apply should perform (configurable) validations before committing:
Validation runs during **Execute**, not during Apply. By the time a plan reaches the Apply phase, all required validations have already passed. Apply commits the sandbox changes to the real resources and does not re-run validations.
!!! danger "Security Invariant — Empty Validation Summary Blocks Apply"
Apply **requires** that at least one required validation was executed during the Execute phase. If no validations were run — because no validations are attached to the plan's resources, or because the validation collection step produced an empty set — the apply gate **blocks** and the plan cannot proceed to Apply.
This is an intentional security invariant (introduced in PR #7786, fixing issue #7508): apply cannot proceed without at least one validation having run. A plan with no validation attachments is treated the same as a plan with failing validations — it cannot be applied.
**Blocking conditions for the apply gate**:
- Any required validation returned `passed: false` (validation failure)
- No required validations were executed at all (empty validation summary)
- A plan has no validation attachments (no validations configured)
The `ApplyValidationSummary.all_required_passed` property returns `True` **only when** at least one required validation was executed AND none of the required validations failed. An empty summary always yields `False`.
For full details on validation — including the Validation type system, modes, attachment scoping, failure handling, fix-then-revalidate loops, and data model — see the **Validation** section under Core Concepts.
#### Apply Data Model
@@ -22757,7 +22769,8 @@ Validation is the **final step** of the Execute phase. The execution actor's wor
2. **Collect applicable validations**: The system resolves all validations from the resource-direct, project, and plan scopes (as described in Attachment Resolution above).
3. **Execute validations**: Each applicable validation is invoked as a standard tool call. The validation reads the sandbox state and returns its structured result. Validations may be run in parallel since they are read-only and cannot interfere with each other.
4. **Process results**:
- **All required validations pass**: Execution is complete. The plan transitions to the review/Apply phase.
- **All required validations pass (and at least one ran)**: Execution is complete. The plan transitions to the review/Apply phase.
- **No required validations were executed (empty summary)**: The apply gate is blocked. This is treated equivalently to a required validation failure — the plan cannot proceed to Apply. This condition arises when no validations are attached to the plan's resources. See the security invariant note in [Validation in Apply](#validation-in-apply).
- **Any required validation fails**: The execution actor enters the fix-then-revalidate loop (see Validation Failure Handling below).
- **Informational validations fail**: Results are recorded in the plan's validation summary. No fix attempts are made.
@@ -23076,7 +23089,7 @@ The validation-related fields stored in the plan data model:
| Field | Location | Description |
|-------|----------|-------------|
| `validation_summary` | `plan.execution` | Array of validation results collected during Execute. Each entry includes the validation name, mode, `passed`, `message`, `data`, execution duration, and attempt number. |
| `final_validation_results` | `plan.apply` | Snapshot of the final validation state at the time execution completed (all required validations passing). Identical to the last state of `validation_summary` but stored separately for quick reference. |
| `final_validation_results` | `plan.apply` | Snapshot of the final validation state at the time execution completed (all required validations passing, and at least one required validation was executed). Identical to the last state of `validation_summary` but stored separately for quick reference. A plan can only reach the Apply phase if this snapshot is non-empty and all required entries have `passed: true`. |
| `validation_attempts` | `plan.execution` | Total number of validation invocations across all fix-then-revalidate loops. Useful for understanding how much effort was spent on validation remediation. |
| `validation_fix_history` | `plan.execution` | Per-validation log of fix attempts. Each entry records: the validation that failed, the fix the actor applied, and whether the subsequent re-validation passed. Enables auditing of the fix-then-revalidate process. |
@@ -47121,7 +47134,7 @@ This section defines the ordered milestone plan for CleverAgents v3.x, mapping a
| 6 | Hierarchical decomposition creates 4+ levels of subplans | §Subplan Architecture — Hierarchy | `plan tree` shows 4+ nesting levels for complex tasks |
| 7 | Parallel execution scales to 10+ concurrent subplans | §Subplan Execution — Parallel | 10 subplans execute concurrently without deadlock or resource starvation |
| 8 | Decision correction recomputes only affected subtree | §Correction Model — Selective Recomputation | Unaffected decisions preserved; only downstream decisions recomputed |
| 9 | Validation-gated apply: required validations must pass before apply | §Validation Abstraction — Apply Gating | Apply blocked when required validation fails; informational validation does not block |
| 9 | Validation-gated apply: required validations must pass before apply | §Validation Abstraction — Apply Gating | Apply blocked when required validation fails; apply also blocked when no validations were run (empty summary); informational validation does not block |
| 10 | A realistic porting task completes autonomously | §Autonomy Acceptance | End-to-end task (e.g., port a Python module) completes without human intervention |
| 11 | `agents automation-profile add/list/show` functional | §CLI Commands — automation-profile | Round-trip: add profile, list shows it, show displays details |
| 12 | `agents plan guard` shows active guards for a plan | §CLI Commands — plan guard | Command shows denylist, budget caps, tool limits |
@@ -47133,7 +47146,7 @@ This section defines the ordered milestone plan for CleverAgents v3.x, mapping a
- **Guard evaluation order**: Denylist checked first (fast reject), then budget caps, then tool call limits.
- **Autonomy threshold**: `0.0` = always automatic; `1.0` = always manual; eight built-in profiles from `manual` to `full-auto`.
- **Subtree recomputation**: Only decisions with `depends_on` edges to the corrected decision are recomputed; others preserved.
- **Validation gating**: `required` validations block apply; `informational` validations log but do not block.
- **Validation gating**: `required` validations block apply; an empty validation summary (no validations ran) also blocks apply; `informational` validations log but do not block.
#### Definition of Done