feat(autonomy): guard enforcement works (denylist, budget caps, tool call limits) #853

Closed
opened 2026-03-13 22:00:29 +00:00 by freemo · 2 comments
Owner

Metadata

  • Commit Message: feat(autonomy): guard enforcement works (denylist, budget caps, tool call limits)
  • Branch: feature/m6-guard-enforcement

Background

M6 (v3.5.0) acceptance criterion: autonomy guardrails must enforce safety constraints during autonomous execution. Per the testing spec (docs/development/testing.md), guards include: denylist/allowlist enforcement, cost budget caps, tool call limits, and write/apply approval gates.

Per the E2E suite (m6_e2e_verification.robot line 92), the test verifies SubplanFailureHandler behavior including fail_fast, sequential stop, retriable/non-retriable error handling, and max retry limits.

Expected Behavior

  1. Denylist enforcement prevents execution of denied operations
  2. Cost budget caps halt execution when budget is exceeded
  3. Tool call limits prevent runaway tool usage
  4. Write/apply approval gates require explicit approval before destructive operations
  5. Guard violations produce clear error messages with remediation guidance

Acceptance Criteria

  • Denylist enforcement blocks denied operations
  • Allowlist enforcement restricts to permitted operations only
  • Cost budget caps halt execution at threshold
  • Tool call limits enforced per-plan and per-subplan
  • Write/apply approval gates functional
  • Robot E2E test Subplan Failure Handler Retry And Stop Logic passes
  • Unit tests cover: each guard type individually, combined guards, edge cases

Supporting Information

  • E2E test: robot/m6_e2e_verification.robot line 92
  • Testing spec: docs/development/testing.md — Autonomy Guardrails section
  • Related: automation profiles (sibling issue)

Subtasks

  • Implement denylist/allowlist guard enforcement
  • Implement cost budget cap enforcement
  • Implement tool call limit enforcement
  • Implement write/apply approval gates
  • Add guard violation error messages
  • Tests (Behave): Add scenarios for each guard type
  • Tests (Robot): Verify E2E acceptance test passes
  • Verify coverage >=97% via nox -s coverage_report
  • Run nox (all default sessions), fix any errors

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
## Metadata - **Commit Message**: `feat(autonomy): guard enforcement works (denylist, budget caps, tool call limits)` - **Branch**: `feature/m6-guard-enforcement` ## Background M6 (v3.5.0) acceptance criterion: autonomy guardrails must enforce safety constraints during autonomous execution. Per the testing spec (`docs/development/testing.md`), guards include: denylist/allowlist enforcement, cost budget caps, tool call limits, and write/apply approval gates. Per the E2E suite (`m6_e2e_verification.robot` line 92), the test verifies SubplanFailureHandler behavior including fail_fast, sequential stop, retriable/non-retriable error handling, and max retry limits. ## Expected Behavior 1. Denylist enforcement prevents execution of denied operations 2. Cost budget caps halt execution when budget is exceeded 3. Tool call limits prevent runaway tool usage 4. Write/apply approval gates require explicit approval before destructive operations 5. Guard violations produce clear error messages with remediation guidance ## Acceptance Criteria - [x] Denylist enforcement blocks denied operations - [x] Allowlist enforcement restricts to permitted operations only - [x] Cost budget caps halt execution at threshold - [x] Tool call limits enforced per-plan and per-subplan - [x] Write/apply approval gates functional - [x] Robot E2E test `Subplan Failure Handler Retry And Stop Logic` passes - [x] Unit tests cover: each guard type individually, combined guards, edge cases ## Supporting Information - E2E test: `robot/m6_e2e_verification.robot` line 92 - Testing spec: `docs/development/testing.md` — Autonomy Guardrails section - Related: automation profiles (sibling issue) ## Subtasks - [x] Implement denylist/allowlist guard enforcement - [x] Implement cost budget cap enforcement - [x] Implement tool call limit enforcement - [x] Implement write/apply approval gates - [x] Add guard violation error messages - [x] Tests (Behave): Add scenarios for each guard type - [x] Tests (Robot): Verify E2E acceptance test passes - [x] Verify coverage >=97% via `nox -s coverage_report` - [x] Run `nox` (all default sessions), fix any errors ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done.
freemo added this to the v3.5.0 milestone 2026-03-13 22:00:58 +00:00
Member

Implementation journal update for #853

Completed scope:

  • Implemented guard enforcement behavior and messaging for denylist/allowlist, budget caps, tool-call limits, and write/apply approval gates in AutomationProfile.check_guard.
  • Added plan/subplan scope support for guard checks and propagated scope through AutomationProfileService.check_guard context handling.
  • Added Behave coverage for remediation messaging and subplan-specific tool-call scope behavior in:
    • features/consolidated_automation_profile.feature
    • features/steps/automation_profiles_guards_steps.py

Key design decisions:

  • check_guard(..., scope=...) now validates scope (plan or subplan) and includes scope in denial reasons to make operator diagnostics explicit.
  • Guard denial reasons include direct remediation guidance (e.g., adjust profile limits/allowlist/denylist or request approval), matching M6 acceptance language.
  • Tool-call cap message now reports observed count vs configured limit and clarifies whether cap was at plan or subplan scope.

Additional stabilization work completed during full-suite validation:

  • Addressed failing integration case in robot/resource_dag.robot (cycle-detection path) by using stable in-memory SQLite engine configuration (StaticPool + shared in-memory connection semantics) and explicit two-type A<->B cycle setup for deterministic cycle validation.
  • This was required to keep full nox green and was completed in the same branch per project quality rules.

Quality gate evidence:

  • Full nox run completed successfully after changes.
  • Session results: lint, format, typecheck, security_scan, dead_code, unit_tests, integration_tests, docs, build, benchmark, coverage_report = SUCCESS.
  • Coverage summary reported 97% (meets >=97 requirement).

Environment note:

  • Verified and ran nox with PYTHONPATH unset (PYTHONPATH=) to avoid importing /app/src and ensure tests execute against /tmp/cleveragents-853/src branch code.
Implementation journal update for #853 Completed scope: - Implemented guard enforcement behavior and messaging for denylist/allowlist, budget caps, tool-call limits, and write/apply approval gates in `AutomationProfile.check_guard`. - Added plan/subplan scope support for guard checks and propagated scope through `AutomationProfileService.check_guard` context handling. - Added Behave coverage for remediation messaging and subplan-specific tool-call scope behavior in: - `features/consolidated_automation_profile.feature` - `features/steps/automation_profiles_guards_steps.py` Key design decisions: - `check_guard(..., scope=...)` now validates scope (`plan` or `subplan`) and includes scope in denial reasons to make operator diagnostics explicit. - Guard denial reasons include direct remediation guidance (e.g., adjust profile limits/allowlist/denylist or request approval), matching M6 acceptance language. - Tool-call cap message now reports observed count vs configured limit and clarifies whether cap was at plan or subplan scope. Additional stabilization work completed during full-suite validation: - Addressed failing integration case in `robot/resource_dag.robot` (cycle-detection path) by using stable in-memory SQLite engine configuration (`StaticPool` + shared in-memory connection semantics) and explicit two-type A<->B cycle setup for deterministic cycle validation. - This was required to keep full nox green and was completed in the same branch per project quality rules. Quality gate evidence: - Full `nox` run completed successfully after changes. - Session results: lint, format, typecheck, security_scan, dead_code, unit_tests, integration_tests, docs, build, benchmark, coverage_report = SUCCESS. - Coverage summary reported 97% (meets >=97 requirement). Environment note: - Verified and ran nox with `PYTHONPATH` unset (`PYTHONPATH=`) to avoid importing `/app/src` and ensure tests execute against `/tmp/cleveragents-853/src` branch code.
Member

Implementation Notes — Review Fix Round (PR #1204, review #2910)

Addressed all four review comments from @freemo. Summary of changes:

1. Reverted resource_dag.robot (BLOCKING)

Restored robot/resource_dag.robot to its master state. The StaticPool migration and cycle-detection type refactoring were unrelated to guard enforcement and violated the Atomic Commits policy. The Robot fix should be submitted as a separate PR/issue.

2. Created GuardScope(StrEnum) (BLOCKING)

  • Added GuardScope enum to cleveragents.domain.models.core.automation_guard with members PLAN = "plan" and SUBPLAN = "subplan".
  • Updated AutomationProfile.check_guard() signature: scope parameter now typed as GuardScope with default GuardScope.PLAN.
  • Updated AutomationProfileService.evaluate_guard() to pass GuardScope.PLAN as default.
  • Updated Behave step step_check_guard_calls_with_scope in features/steps/automation_profiles_guards_steps.py to construct GuardScope(scope) from the string parameter.
  • Added GuardScope to __init__.py re-exports and __all__.
  • Used StrEnum (not str, Enum) to comply with ruff UP042 lint rule.

3. Removed redundant scope_label (MINOR)

Removed the intermediate scope_label variable. Now uses scope.value directly in all remediation messages, which is always either "plan" or "subplan".

4. Extracted remediation constants (MINOR)

Created six module-level constants in automation_guard.py:

  • REMEDIATION_DENYLIST, REMEDIATION_ALLOWLIST, REMEDIATION_TOOL_CALL_LIMIT
  • REMEDIATION_BUDGET, REMEDIATION_WRITE_APPROVAL, REMEDIATION_APPLY_APPROVAL

Each guard denial message now composes a dynamic prefix with the constant suffix, improving consistency and testability.

Quality Gates

  • nox -s lint passed
  • nox -s typecheck 0 errors, 0 warnings
  • nox -s unit_tests 508 features, 12989 scenarios, 0 failures
  • nox -s coverage_report 97% (79992 stmts, 2436 missed)
  • Rebased onto latest master (532ea100)
## Implementation Notes — Review Fix Round (PR #1204, review #2910) Addressed all four review comments from @freemo. Summary of changes: ### 1. Reverted `resource_dag.robot` (BLOCKING) Restored `robot/resource_dag.robot` to its master state. The StaticPool migration and cycle-detection type refactoring were unrelated to guard enforcement and violated the Atomic Commits policy. The Robot fix should be submitted as a separate PR/issue. ### 2. Created `GuardScope(StrEnum)` (BLOCKING) - Added `GuardScope` enum to `cleveragents.domain.models.core.automation_guard` with members `PLAN = "plan"` and `SUBPLAN = "subplan"`. - Updated `AutomationProfile.check_guard()` signature: `scope` parameter now typed as `GuardScope` with default `GuardScope.PLAN`. - Updated `AutomationProfileService.evaluate_guard()` to pass `GuardScope.PLAN` as default. - Updated Behave step `step_check_guard_calls_with_scope` in `features/steps/automation_profiles_guards_steps.py` to construct `GuardScope(scope)` from the string parameter. - Added `GuardScope` to `__init__.py` re-exports and `__all__`. - Used `StrEnum` (not `str, Enum`) to comply with ruff UP042 lint rule. ### 3. Removed redundant `scope_label` (MINOR) Removed the intermediate `scope_label` variable. Now uses `scope.value` directly in all remediation messages, which is always either `"plan"` or `"subplan"`. ### 4. Extracted remediation constants (MINOR) Created six module-level constants in `automation_guard.py`: - `REMEDIATION_DENYLIST`, `REMEDIATION_ALLOWLIST`, `REMEDIATION_TOOL_CALL_LIMIT` - `REMEDIATION_BUDGET`, `REMEDIATION_WRITE_APPROVAL`, `REMEDIATION_APPLY_APPROVAL` Each guard denial message now composes a dynamic prefix with the constant suffix, improving consistency and testability. ### Quality Gates - `nox -s lint` — ✅ passed - `nox -s typecheck` — ✅ 0 errors, 0 warnings - `nox -s unit_tests` — ✅ 508 features, 12989 scenarios, 0 failures - `nox -s coverage_report` — ✅ 97% (79992 stmts, 2436 missed) - Rebased onto latest `master` (532ea100)
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cleveragents/cleveragents-core#853
No description provided.