cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity
Files
c9e20c6b82ba3d19f4e81088b2528fa3b4cbaa8b
cleveragents-core/features/consolidated_automation_profile.feature
T
brent.edwards a8d91b94a7
CI / build (push) Successful in 16s
Details
CI / helm (push) Successful in 22s
Details
CI / lint (push) Successful in 39s
Details
CI / quality (push) Successful in 3m43s
Details
CI / typecheck (push) Successful in 3m57s
Details
CI / benchmark-regression (push) Has been skipped
Details
CI / security (push) Successful in 4m8s
Details
CI / unit_tests (push) Successful in 9m13s
Details
CI / docker (push) Successful in 1m34s
Details
CI / e2e_tests (push) Successful in 15m20s
Details
CI / coverage (push) Successful in 12m28s
Details
CI / integration_tests (push) Successful in 21m12s
Details
CI / status-check (push) Successful in 1s
Details
CI / benchmark-publish (push) Successful in 28m30s
Details
feat(autonomy): guard enforcement works (denylist, budget caps, tool call limits) (#1204) 
## Summary
- Enforced autonomy guard behavior for denylist/allowlist checks, budget caps, tool-call limits, and write/apply approval gates.
- Added scope-aware guard evaluation (plan vs subplan) using `GuardScope(StrEnum)` for type-safe scope handling.
- Extracted remediation guidance strings as module-level constants (`REMEDIATION_DENYLIST`, `REMEDIATION_ALLOWLIST`, etc.) for consistency and testability.
- Added Behave coverage to validate guard remediation messaging and subplan-scoped tool-call limit behavior.

## Review Fix Round (v2)
Addressed review #2910 by @freemo:
1. **Reverted `resource_dag.robot`** — Unrelated Robot SQLite/cycle-detection changes removed from this commit; will be submitted as a separate PR.
2. **`scope` parameter → `GuardScope` enum** — Created `GuardScope(StrEnum)` in `automation_guard.py` with `PLAN` and `SUBPLAN` members. Updated `check_guard()` signature and all call sites.
3. **Removed redundant `scope_label`** — Now uses `scope.value` directly.
4. **Extracted remediation constants** — Guidance strings moved to module-level constants in `automation_guard.py`.

## Validation
- `nox -s lint` — passed
- `nox -s typecheck` — passed (0 errors, 0 warnings)
- `nox -s unit_tests` — passed (508 features, 12989 scenarios, 0 failures)
- `nox -s coverage_report` — passed (97% coverage)
- Rebased onto latest `master` (532ea100)

Closes #853

Reviewed-on: #1204
Co-authored-by: Brent Edwards <brent.edwards@cleverthis.com>
Co-committed-by: Brent Edwards <brent.edwards@cleverthis.com>
2026-04-01 01:46:37 +00:00

828 lines
30 KiB
Gherkin
Raw Blame History

Feature: Consolidated Automation Profile
Combined scenarios from: automation_level_removed, automation_profile, automation_profile_crud_coverage, automation_profile_service, automation_profiles_guards
# ============================================================
# Originally from: automation_level_removed.feature
# Feature: automation_level legacy removal
# ============================================================
Scenario: Plan creation uses automation_profile only
Given I have a plan lifecycle service with automation level support
Given I have an available action for automation tests
When I use the action without explicit automation level on project "proj-001"
Then the automated plan phase should be "strategize"
Scenario: Plan display shows automation_profile not automation_level
Given I have a plan lifecycle service with automation level support
Given I have an available action for automation tests
When I use the action without explicit automation level on project "proj-001"
Then the automated plan phase should be "strategize"
Scenario: Pause sets manual automation profile
Given I have a plan lifecycle service with automation level support
Given I have an available action for automation tests
When I use the action without explicit automation level on project "proj-002"
And I pause the plan
Then the plan automation level should be "manual"
Scenario: Resume sets auto automation profile
Given I have a plan lifecycle service with automation level support
Given I have a paused plan that was previously "manual" in strategize phase
When I resume the plan without explicit level
Then the automated plan phase should be "strategize"
# ============================================================
# Originally from: automation_profile.feature
# Feature: Automation Profile Domain Model
# ============================================================
Scenario: Profile accepts threshold of 0.0
When I create a profile with decompose_task 0.0
Then the profile model should be created
And the profile decompose_task should be 0.0
Scenario: Profile accepts threshold of 0.5
When I create a profile with decompose_task 0.5
Then the profile model should be created
And the profile decompose_task should be 0.5
Scenario: Profile accepts threshold of 1.0
When I create a profile with decompose_task 1.0
Then the profile model should be created
And the profile decompose_task should be 1.0
# ---- Profile validation: invalid thresholds ----
Scenario: Profile rejects threshold below 0.0
When I try to create a profile with decompose_task -0.1
Then a profile validation error should be raised
And the profile error should mention "greater than or equal"
Scenario: Profile rejects threshold above 1.0
When I try to create a profile with decompose_task 1.1
Then a profile validation error should be raised
And the profile error should mention "less than or equal"
Scenario: Profile rejects create_tool below 0.0
When I try to create a profile with create_tool -0.5
Then a profile validation error should be raised
Scenario: Profile rejects create_tool above 1.0
When I try to create a profile with create_tool 2.0
Then a profile validation error should be raised
Scenario: Profile rejects select_tool below 0.0
When I try to create a profile with select_tool -0.01
Then a profile validation error should be raised
Scenario: Profile rejects edit_code above 1.0
When I try to create a profile with edit_code 1.5
Then a profile validation error should be raised
Scenario: Profile rejects execute_command below 0.0
When I try to create a profile with execute_command -1.0
Then a profile validation error should be raised
Scenario: Profile rejects create_file above 1.0
When I try to create a profile with create_file 9.9
Then a profile validation error should be raised
Scenario: Profile rejects delete_content below 0.0
When I try to create a profile with delete_content -0.001
Then a profile validation error should be raised
Scenario: Profile rejects access_network above 1.0
When I try to create a profile with access_network 1.01
Then a profile validation error should be raised
Scenario: Profile rejects install_dependency below 0.0
When I try to create a profile with install_dependency -0.5
Then a profile validation error should be raised
Scenario: Profile rejects modify_config above 1.0
When I try to create a profile with modify_config 100.0
Then a profile validation error should be raised
Scenario: Profile rejects approve_plan below 0.0
When I try to create a profile with approve_plan -0.1
Then a profile validation error should be raised
# ---- Built-in profiles load correctly ----
Scenario: Built-in manual profile loads with expected values
When I load the built-in profile "manual"
Then the profile model should be created
And the profile decompose_task should be 1.0
And the profile create_tool should be 1.0
And the profile select_tool should be 1.0
And the profile safety require_sandbox should be true
And the profile safety require_checkpoints should be true
And the profile safety allow_unsafe_tools should be false
Scenario: Built-in review profile loads with expected values
When I load the built-in profile "review"
Then the profile model should be created
And the profile decompose_task should be 0.0
And the profile create_tool should be 0.0
And the profile select_tool should be 1.0
And the profile install_dependency should be 0.0
Scenario: Built-in supervised profile loads with expected values
When I load the built-in profile "supervised"
Then the profile model should be created
And the profile decompose_task should be 0.0
And the profile create_tool should be 1.0
And the profile edit_code should be 0.0
And the profile install_dependency should be 1.0
Scenario: Built-in cautious profile loads with expected values
When I load the built-in profile "cautious"
Then the profile model should be created
And the profile decompose_task should be 0.7
And the profile create_tool should be 0.7
And the profile edit_code should be 0.6
And the profile execute_command should be 0.8
And the profile create_file should be 0.7
And the profile delete_content should be 0.8
And the profile access_network should be 0.9
And the profile install_dependency should be 0.7
And the profile approve_plan should be 0.6
Scenario: Built-in trusted profile loads with expected values
When I load the built-in profile "trusted"
Then the profile model should be created
And the profile decompose_task should be 0.0
And the profile create_tool should be 0.0
And the profile select_tool should be 1.0
And the profile delete_content should be 1.0
And the profile access_network should be 1.0
Scenario: Built-in auto profile loads with expected values
When I load the built-in profile "auto"
Then the profile model should be created
And the profile decompose_task should be 0.0
And the profile select_tool should be 1.0
And the profile access_network should be 1.0
And the profile delete_content should be 0.0
Scenario: Built-in ci profile loads with expected values
When I load the built-in profile "ci"
Then the profile model should be created
And the profile decompose_task should be 0.0
And the profile select_tool should be 0.0
And the profile safety require_sandbox should be true
And the profile safety allow_unsafe_tools should be false
Scenario: Built-in full-auto profile loads with expected values
When I load the built-in profile "full-auto"
Then the profile model should be created
And the profile decompose_task should be 0.0
And the profile select_tool should be 0.0
And the profile safety require_sandbox should be false
And the profile safety require_checkpoints should be false
And the profile safety allow_unsafe_tools should be true
# ---- All 8 built-in profiles exist ----
Scenario: All 8 built-in profiles are registered
Then there should be 8 built-in profiles
And built-in profile "manual" should exist
And built-in profile "review" should exist
And built-in profile "supervised" should exist
And built-in profile "cautious" should exist
And built-in profile "trusted" should exist
And built-in profile "auto" should exist
And built-in profile "ci" should exist
And built-in profile "full-auto" should exist
# ---- Custom profile from YAML dict loads correctly ----
Scenario: Custom profile from YAML config loads correctly
When I load a profile from config with name "acme/strict" and select_tool 0.8
Then the profile model should be created
And the profile name should be "acme/strict"
And the profile select_tool should be 0.8
Scenario: Profile from config missing name raises error
When I try to load a profile from config missing name
Then a profile config error should be raised with "name"
# ---- Name format validation ----
Scenario: Profile name accepts bare name
When I create a profile with name "manual"
Then the profile model should be created
And the profile name should be "manual"
Scenario: Profile name accepts namespaced name
When I create a profile with name "acme/strict"
Then the profile model should be created
And the profile name should be "acme/strict"
Scenario: Profile name rejects spaces
When I try to create a profile with invalid name "bad name"
Then a profile validation error should be raised
And the profile error should mention "Profile name"
Scenario: Profile name rejects empty string
When I try to create a profile with empty name
Then a profile validation error should be raised
Scenario: Profile name rejects double slash
When I try to create a profile with invalid name "bad//name"
Then a profile validation error should be raised
Scenario: Profile name rejects trailing slash
When I try to create a profile with invalid name "bad/"
Then a profile validation error should be raised
# ---- get_builtin_profile helper ----
Scenario: get_builtin_profile returns correct profile
When I call get_builtin_profile with "ci"
Then the profile model should be created
And the profile name should be "ci"
Scenario: get_builtin_profile raises for unknown name
When I try to call get_builtin_profile with "nonexistent"
Then a profile key error should be raised
# ---- Schema version ----
Scenario: Profile has default schema version
When I create a profile with name "test-profile"
Then the profile schema_version should be "1.0"
Scenario: Profile accepts custom schema version
When I create a profile with schema_version "2.0"
Then the profile schema_version should be "2.0"
# ---- Safety field defaults ----
Scenario: Profile safety fields have correct defaults
When I create a profile with name "test-defaults"
Then the profile safety require_sandbox should be true
And the profile safety require_checkpoints should be true
And the profile safety allow_unsafe_tools should be false
# ---- Description field ----
Scenario: Profile description defaults to empty string
When I create a profile with name "no-desc"
Then the profile description should be empty
Scenario: Profile description accepts custom value
When I create a profile with description "My custom profile"
Then the profile description should be "My custom profile"
# ---- validate_assignment enforcement ----
Scenario: Assigning invalid threshold raises error
When I create a profile with name "assign-test"
And I try to assign decompose_task 1.5 on the profile
Then a profile validation error should be raised
# ---- Built-in profile retry and checkpoint values ----
Scenario: Manual profile has modify_config 1.0
When I load the built-in profile "manual"
Then the profile modify_config should be 1.0
And the profile approve_plan should be 1.0
Scenario: Review profile has modify_config 0.0
When I load the built-in profile "review"
Then the profile modify_config should be 0.0
And the profile approve_plan should be 1.0
Scenario: Cautious profile has modify_config 0.0
When I load the built-in profile "cautious"
Then the profile modify_config should be 0.0
# ---- Model dump and round-trip ----
Scenario: Profile model_dump produces valid dict
When I create a profile and dump it with select_tool 0.5
Then the profile model dump should have key "name"
And the profile model dump should have key "select_tool"
And the profile model dump select_tool should be 0.5
# ============================================================
# Originally from: automation_profile_crud_coverage.feature
# Feature: Automation Profile CRUD operations with repository
# ============================================================
Scenario: Get custom profile from repository
Given a profile service with a mock repository
And the mock repository contains a profile named "acme/custom"
When I fetch profile "acme/custom"
Then the fetched profile name should be "acme/custom"
# ---- list_profiles with repository (lines 221-222) ----
Scenario: List profiles includes custom profiles from repository
Given a profile service with a mock repository
And the mock repository contains a profile named "acme/custom"
When I list all available profiles
Then the available profiles should include "manual"
And the available profiles should include "full-auto"
And the available profiles should include "acme/custom"
# ---- create_profile valid (lines 239-248) ----
Scenario: Create a custom profile with valid config
Given a profile service with a mock repository
When I create a profile with name "team/deploy" and description "Deploy profile"
Then the created profile name should be "team/deploy"
And the created profile description should be "Deploy profile"
And the mock repository should contain "team/deploy"
# ---- create_profile builtin name raises ValidationError (lines 240-241) ----
Scenario: Create profile with builtin name raises ValidationError
Given a profile service with a mock repository
When I try to create a profile with builtin name "manual"
Then a crud validation error should be raised
And the crud validation error message should mention "manual"
# ---- update_profile valid (lines 269-282) ----
Scenario: Update a custom profile with valid config
Given a profile service with a mock repository
And the mock repository contains a profile named "acme/custom"
When I update profile "acme/custom" with description "Updated description"
Then the updated profile name should be "acme/custom"
And the updated profile description should be "Updated description"
And the mock repository should contain "acme/custom"
# ---- update_profile builtin name raises ValidationError (line 269-270) ----
Scenario: Update builtin profile raises ValidationError
Given a profile service with a mock repository
When I try to update builtin profile "auto"
Then a crud validation error should be raised
And the crud validation error message should mention "auto"
# ---- update_profile without repo raises NotFoundError (lines 275-276) ----
Scenario: Update profile without repository raises NotFoundError
Given a profile service without a repository
When I try to update profile "acme/custom" without repo
Then a crud not found error should be raised
And the crud not found error message should mention "acme/custom"
# ---- delete_profile valid (lines 294-304) ----
Scenario: Delete a custom profile with valid name
Given a profile service with a mock repository
And the mock repository contains a profile named "acme/custom"
When I delete profile "acme/custom"
Then the mock repository should not contain "acme/custom"
# ---- delete_profile builtin name raises ValidationError (lines 294-295) ----
Scenario: Delete builtin profile raises ValidationError
Given a profile service with a mock repository
When I try to delete builtin profile "supervised"
Then a crud validation error should be raised
And the crud validation error message should mention "supervised"
# ---- delete_profile without repo raises NotFoundError (lines 298-299) ----
Scenario: Delete profile without repository raises NotFoundError
Given a profile service without a repository
When I try to delete profile "acme/custom" without repo
Then a crud not found error should be raised
And the crud not found error message should mention "acme/custom"
# ============================================================
# Originally from: automation_profile_service.feature
# Feature: Automation Profile Service
# ============================================================
Scenario: Plan-level profile takes highest precedence
Given an automation profile service with global default "manual"
When I resolve profile with plan "full-auto" action "auto" project "supervised"
Then the resolved profile name should be "full-auto"
Scenario: Action-level profile used when plan is not set
Given an automation profile service with global default "manual"
When I resolve profile with plan None action "auto" project "supervised"
Then the resolved profile name should be "auto"
Scenario: Project-level profile used when plan and action are not set
Given an automation profile service with global default "manual"
When I resolve profile with plan None action None project "supervised"
Then the resolved profile name should be "supervised"
Scenario: Global default used when no level is set
Given an automation profile service with global default "cautious"
When I resolve profile with plan None action None project None
Then the resolved profile name should be "cautious"
Scenario: Default global is manual when nothing configured
Given an automation profile service with no configuration
When I resolve profile with plan None action None project None
Then the resolved profile name should be "manual"
# ---- Missing profile errors ----
Scenario: Missing profile raises NotFoundError
Given an automation profile service with global default "manual"
When I try to get profile "nonexistent-profile"
Then a profile not found error should be raised
And the profile service error should mention "nonexistent-profile"
Scenario: Missing profile in resolve raises NotFoundError
Given an automation profile service with global default "does-not-exist"
When I try to resolve profile with all None
Then a profile not found error should be raised
# ---- Environment variable override ----
Scenario: Env var overrides empty global default
Given an automation profile service with no configuration
And the env var CLEVERAGENTS_AUTOMATION_PROFILE is set to "ci"
When I resolve profile with plan None action None project None
Then the resolved profile name should be "ci"
Scenario: Explicit global default takes precedence over env var
Given an automation profile service with global default "review"
And the env var CLEVERAGENTS_AUTOMATION_PROFILE is set to "ci"
When I resolve profile with plan None action None project None
Then the resolved profile name should be "review"
# ---- CRUD via repository ----
Scenario: List profiles includes built-ins
Given an automation profile service with global default "manual"
When I list all profiles
Then the profile list should include "manual"
And the profile list should include "full-auto"
And the profile list should have at least 8 entries
# ---- Get built-in profile ----
Scenario: Get built-in profile by name
Given an automation profile service with global default "manual"
When I get profile "auto"
Then the retrieved profile name should be "auto"
And the retrieved profile select_tool should be 1.0
# ---- Profile threshold checks ----
Scenario: Manual profile has all thresholds at 1.0
Given an automation profile service with global default "manual"
When I get profile "manual"
Then the retrieved profile decompose_task should be 1.0
And the retrieved profile create_tool should be 1.0
And the retrieved profile select_tool should be 1.0
Scenario: Full-auto profile has all thresholds at 0.0
Given an automation profile service with global default "manual"
When I get profile "full-auto"
Then the retrieved profile decompose_task should be 0.0
And the retrieved profile create_tool should be 0.0
And the retrieved profile select_tool should be 0.0
# ============================================================
# Originally from: automation_profiles_guards.feature
# Feature: Automation Profile Guards
# ============================================================
Scenario: Guard allows tool within call limits
Given a profile with max_tool_calls_per_step 5
When I check guard for tool "read_file" with 3 calls so far
Then the guard result should be allowed
Scenario: Guard allows tool with no guards configured
Given a profile with no guards
When I check guard for tool "any_tool"
Then the guard result should be allowed
# ---- Guard blocks tool exceeding max_tool_calls ----
Scenario: Guard blocks tool exceeding max_tool_calls
Given a profile with max_tool_calls_per_step 3
When I check guard for tool "read_file" with 3 calls so far
Then the guard result should not be allowed
And the guard reason should mention "limit"
And the guard should require approval
Scenario: Guard blocks tool at exact max_tool_calls boundary
Given a profile with max_tool_calls_per_step 0
When I check guard for tool "read_file" with 0 calls so far
Then the guard result should not be allowed
# ---- Guard blocks tool on denylist ----
Scenario: Guard blocks tool on denylist
Given a profile with denylist "shell_exec,file_delete"
When I check guard for tool "shell_exec"
Then the guard result should not be allowed
And the guard reason should mention "denylist"
And the guard should require approval
Scenario: Guard allows tool not on denylist
Given a profile with denylist "shell_exec,file_delete"
When I check guard for tool "read_file"
Then the guard result should be allowed
# ---- Guard allows tool on allowlist ----
Scenario: Guard allows tool on allowlist
Given a profile with allowlist "read_file,list_dir"
When I check guard for tool "read_file"
Then the guard result should be allowed
Scenario: Guard blocks tool not on allowlist
Given a profile with allowlist "read_file,list_dir"
When I check guard for tool "write_file"
Then the guard result should not be allowed
And the guard reason should mention "allowlist"
# ---- Guard requires approval for writes ----
Scenario: Guard requires approval for write operations
Given a profile with require_approval_for_writes true
When I check guard for tool "write_file" with is_write true
Then the guard result should not be allowed
And the guard reason should mention "Write"
And the guard should require approval
Scenario: Guard allows non-write operations when writes require approval
Given a profile with require_approval_for_writes true
When I check guard for tool "read_file" with is_write false
Then the guard result should be allowed
# ---- Guard requires approval for apply ----
Scenario: Guard requires approval for apply phase
Given a profile with require_approval_for_apply true
When I check guard for tool "__apply__"
Then the guard result should not be allowed
And the guard reason should mention "Apply"
And the guard should require approval
Scenario: Guard allows non-apply tools when apply requires approval
Given a profile with require_approval_for_apply true
When I check guard for tool "read_file"
Then the guard result should be allowed
# ---- Cost budget enforcement ----
Scenario: Guard blocks tool when cost budget exceeded
Given a profile with max_total_cost 10.0
When I check guard for tool "expensive_tool" with cost 10.0
Then the guard result should not be allowed
And the guard reason should mention "Cost"
And the guard should require approval
Scenario: Guard allows tool within cost budget
Given a profile with max_total_cost 10.0
When I check guard for tool "cheap_tool" with cost 5.0
Then the guard result should be allowed
# ---- Effective profile resolution ----
Scenario: Effective profile resolves plan-level first
Given a profile service with profiles "cautious" and "manual"
When I resolve effective profile with plan "cautious"
Then the resolved guard profile name should be "cautious"
Scenario: Effective profile falls back to action level
Given a profile service with profiles "cautious" and "manual"
When I resolve effective profile with action "cautious"
Then the resolved guard profile name should be "cautious"
Scenario: Effective profile falls back to default
Given a profile service with profiles "cautious" and "manual"
When I resolve effective profile with default "cautious"
Then the resolved guard profile name should be "cautious"
Scenario: Effective profile uses global when no overrides
Given a profile service with global default "manual"
When I resolve effective profile with no overrides
Then the resolved guard profile name should be "manual"
# ---- Custom profile YAML loading ----
Scenario: Load custom profile from YAML file
Given a YAML profile file with guards
When I load the profile from YAML
Then the loaded profile should have guards
And the loaded profile guards max_tool_calls_per_step should be 10
Scenario: Load custom profile from YAML via service
Given a YAML profile file with guards
When I load the profile via service from_yaml
Then the loaded profile should have guards
# ---- Built-in profile immutability ----
Scenario: Built-in profiles cannot be updated
Given a profile service
When I try to update built-in profile "manual"
Then a profile service validation error should be raised
Scenario: Built-in profiles cannot be deleted
Given a profile service
When I try to delete built-in profile "manual"
Then a profile service validation error should be raised
# ---- Guard model validation ----
Scenario: AutomationGuard rejects negative max_tool_calls
When I try to create a guard with max_tool_calls -1
Then a guard model validation error should be raised
Scenario: AutomationGuard rejects negative max_total_cost
When I try to create a guard with max_total_cost -1.0
Then a guard model validation error should be raised
Scenario: AutomationGuard accepts zero max_tool_calls
When I create a guard with max_tool_calls 0
Then the guard should be created
Scenario: AutomationGuard accepts None values
When I create a guard with defaults
Then the guard max_tool_calls_per_step should be None
And the guard max_total_cost should be None
# ---- GuardResult model ----
Scenario: GuardResult allowed with no reason
When I create a guard result allowed true
Then the guard result allowed should be true
And the guard result reason should be None
Scenario: GuardResult denied with reason
When I create a guard result denied with reason "blocked"
Then the guard result allowed should be false
And the guard result reason should be "blocked"
And the guard result requires_approval should be true
# ---- Service evaluate_guard ----
Scenario: Service evaluate_guard delegates to profile
Given a profile service
When I evaluate guard for profile "full-auto" tool "any_tool"
Then the guard result should be allowed
Scenario: Service evaluate_guard rejects empty profile name
Given a profile service
When I try to evaluate guard with empty profile name
Then a profile service validation error should be raised
Scenario: Service evaluate_guard rejects empty tool name
Given a profile service
When I try to evaluate guard with empty tool name
Then a profile service validation error should be raised
# ---- Profile with guards from config dict ----
Scenario: Profile from config dict with guards
When I create a profile from config with guards
Then the profile should have guards configured
And the profile guards should have denylist
# ---- check_guard argument validation ----
Scenario: check_guard rejects negative cost_so_far
Given a profile with max_total_cost 10.0
When I try to check guard with negative cost
Then a guard argument error should be raised
Scenario: check_guard rejects negative calls_so_far
Given a profile with max_tool_calls_per_step 5
When I try to check guard with negative calls
Then a guard argument error should be raised
Scenario: Guard denylist message includes remediation guidance
Given a profile with denylist "shell_exec"
When I check guard for tool "shell_exec"
Then the guard result should not be allowed
And the guard reason should mention "Remediation"
Scenario: Guard budget message includes remediation guidance
Given a profile with max_total_cost 10.0
When I check guard for tool "expensive_tool" with cost 10.0
Then the guard result should not be allowed
And the guard reason should mention "Remediation"
Scenario: Guard tool-call limit can be evaluated per subplan scope
Given a profile with max_tool_calls_per_step 2
When I check guard for tool "read_file" with 2 calls so far in "subplan" scope
Then the guard result should not be allowed
And the guard reason should mention "Subplan"
Scenario: Guard write approval message includes remediation guidance
Given a profile with require_approval_for_writes true
When I check guard for tool "write_file" with is_write true
Then the guard result should not be allowed
And the guard reason should mention "Remediation"
Reference in New Issue View Git Blame Copy Permalink