From 57881a075b6fc904aa39c182233cb6104690a9de Mon Sep 17 00:00:00 2001 From: HAL9000 Date: Fri, 8 May 2026 10:21:23 +0000 Subject: [PATCH] =?UTF-8?q?docs(spec):=20update=20specification=20?= =?UTF-8?q?=E2=80=94=20validation=20gate=20blocks=20on=20empty=20validatio?= =?UTF-8?q?n=20summary?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CHANGELOG.md | 7 +++++++ CONTRIBUTORS.md | 1 + docs/specification.md | 21 +++++++++++++++++---- 3 files changed, 25 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d0a222732..2c851beaa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -653,6 +653,13 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). delays from 15s to 2s, and reduced idle sleep from 60s to 10s for dramatically faster throughput. +- **Specification — Validation Gate Empty-Run Guard** (#8146): Updated `docs/specification.md` + to document the security invariant introduced in PR #7786 (fixing issue #7508). The spec now + explicitly states that `ApplyValidationSummary.all_required_passed` returns `False` when no + validations have been run (empty summary), blocking apply. Added a prominent danger admonition + block, updated the validation process results section, the `final_validation_results` data + model description, and two milestone acceptance criteria to reflect the corrected blocking + behavior for empty validation summaries and no-attachment runs. ### Fixed diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index a0aef2f4b..be00cf151 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -22,6 +22,7 @@ Below are some of the specific details of various contributions. * HAL 9000 has contributed the benchmark workflow separation (#9040): moved the benchmark-regression job out of the default PR workflow into a dedicated scheduled workflow, reducing median PR CI turnaround time from 99-132 minutes to under 30 minutes. * HAL 9000 has contributed the agent-evolution-pool-supervisor PR metadata assignment (#7888): the supervisor now automatically looks up the Type/Automation label and earliest open milestone before dispatching improvement PR creation workers, ensuring all generated improvement PRs have correct Type labels and milestone assignments. * HAL 9000 has contributed the decision recording hook for the Strategize phase (issue #8522): captures every decision point with question, chosen option, alternatives, confidence, rationale, and full context snapshot for replay and correction. +* HAL 9000 has contributed automated specification maintenance, documentation updates, and bot-driven PR authorship. * This project was made possible thanks to considerable donation of time, money, and resources by CleverThis, Inc. * HAL 9000 has contributed automated bug fixes, CLI output formatting improvements, and ongoing maintenance as part of the CleverAgents automation system. * HAL 9000 has contributed the file edit encoding parameter fix (PR #8258 / issue #7559). diff --git a/docs/specification.md b/docs/specification.md index 469f33f8d..ee41c7f68 100644 --- a/docs/specification.md +++ b/docs/specification.md @@ -20019,6 +20019,18 @@ Apply should perform (configurable) validations before committing: Validation runs during **Execute**, not during Apply. By the time a plan reaches the Apply phase, all required validations have already passed. Apply commits the sandbox changes to the real resources and does not re-run validations. +!!! danger "Security Invariant — Empty Validation Summary Blocks Apply" + Apply **requires** that at least one required validation was executed during the Execute phase. If no validations were run — because no validations are attached to the plan's resources, or because the validation collection step produced an empty set — the apply gate **blocks** and the plan cannot proceed to Apply. + + This is an intentional security invariant (introduced in PR #7786, fixing issue #7508): apply cannot proceed without at least one validation having run. A plan with no validation attachments is treated the same as a plan with failing validations — it cannot be applied. + + **Blocking conditions for the apply gate**: + - Any required validation returned `passed: false` (validation failure) + - No required validations were executed at all (empty validation summary) + - A plan has no validation attachments (no validations configured) + + The `ApplyValidationSummary.all_required_passed` property returns `True` **only when** at least one required validation was executed AND none of the required validations failed. An empty summary always yields `False`. + For full details on validation — including the Validation type system, modes, attachment scoping, failure handling, fix-then-revalidate loops, and data model — see the **Validation** section under Core Concepts. #### Apply Data Model @@ -22757,7 +22769,8 @@ Validation is the **final step** of the Execute phase. The execution actor's wor 2. **Collect applicable validations**: The system resolves all validations from the resource-direct, project, and plan scopes (as described in Attachment Resolution above). 3. **Execute validations**: Each applicable validation is invoked as a standard tool call. The validation reads the sandbox state and returns its structured result. Validations may be run in parallel since they are read-only and cannot interfere with each other. 4. **Process results**: - - **All required validations pass**: Execution is complete. The plan transitions to the review/Apply phase. + - **All required validations pass (and at least one ran)**: Execution is complete. The plan transitions to the review/Apply phase. + - **No required validations were executed (empty summary)**: The apply gate is blocked. This is treated equivalently to a required validation failure — the plan cannot proceed to Apply. This condition arises when no validations are attached to the plan's resources. See the security invariant note in [Validation in Apply](#validation-in-apply). - **Any required validation fails**: The execution actor enters the fix-then-revalidate loop (see Validation Failure Handling below). - **Informational validations fail**: Results are recorded in the plan's validation summary. No fix attempts are made. @@ -23076,7 +23089,7 @@ The validation-related fields stored in the plan data model: | Field | Location | Description | |-------|----------|-------------| | `validation_summary` | `plan.execution` | Array of validation results collected during Execute. Each entry includes the validation name, mode, `passed`, `message`, `data`, execution duration, and attempt number. | -| `final_validation_results` | `plan.apply` | Snapshot of the final validation state at the time execution completed (all required validations passing). Identical to the last state of `validation_summary` but stored separately for quick reference. | +| `final_validation_results` | `plan.apply` | Snapshot of the final validation state at the time execution completed (all required validations passing, and at least one required validation was executed). Identical to the last state of `validation_summary` but stored separately for quick reference. A plan can only reach the Apply phase if this snapshot is non-empty and all required entries have `passed: true`. | | `validation_attempts` | `plan.execution` | Total number of validation invocations across all fix-then-revalidate loops. Useful for understanding how much effort was spent on validation remediation. | | `validation_fix_history` | `plan.execution` | Per-validation log of fix attempts. Each entry records: the validation that failed, the fix the actor applied, and whether the subsequent re-validation passed. Enables auditing of the fix-then-revalidate process. | @@ -47123,7 +47136,7 @@ This section defines the ordered milestone plan for CleverAgents v3.x, mapping a | 6 | Hierarchical decomposition creates 4+ levels of subplans | §Subplan Architecture — Hierarchy | `plan tree` shows 4+ nesting levels for complex tasks | | 7 | Parallel execution scales to 10+ concurrent subplans | §Subplan Execution — Parallel | 10 subplans execute concurrently without deadlock or resource starvation | | 8 | Decision correction recomputes only affected subtree | §Correction Model — Selective Recomputation | Unaffected decisions preserved; only downstream decisions recomputed | -| 9 | Validation-gated apply: required validations must pass before apply | §Validation Abstraction — Apply Gating | Apply blocked when required validation fails; informational validation does not block | +| 9 | Validation-gated apply: required validations must pass before apply | §Validation Abstraction — Apply Gating | Apply blocked when required validation fails; apply also blocked when no validations were run (empty summary); informational validation does not block | | 10 | A realistic porting task completes autonomously | §Autonomy Acceptance | End-to-end task (e.g., port a Python module) completes without human intervention | | 11 | `agents automation-profile add/list/show` functional | §CLI Commands — automation-profile | Round-trip: add profile, list shows it, show displays details | | 12 | `agents plan guard` shows active guards for a plan | §CLI Commands — plan guard | Command shows denylist, budget caps, tool limits | @@ -47135,7 +47148,7 @@ This section defines the ordered milestone plan for CleverAgents v3.x, mapping a - **Guard evaluation order**: Denylist checked first (fast reject), then budget caps, then tool call limits. - **Autonomy threshold**: `0.0` = always automatic; `1.0` = always manual; eight built-in profiles from `manual` to `full-auto`. - **Subtree recomputation**: Only decisions with `depends_on` edges to the corrected decision are recomputed; others preserved. -- **Validation gating**: `required` validations block apply; `informational` validations log but do not block. +- **Validation gating**: `required` validations block apply; an empty validation summary (no validations ran) also blocks apply; `informational` validations log but do not block. #### Definition of Done -- 2.52.0