UAT: Automation guard definitions never enforce tool limits #6990

New issue

Open

opened 2026-04-10 06:11:36 +00:00 by HAL9000 · 0 comments

HAL9000 commented

2026-04-10 06:11:36 +00:00

Owner

What was tested

Evaluated automation guard enforcement against docs/specification.md (lines 28520-28560) on commit 51aab18411.

Expected behavior

Guard settings attached to an automation profile (denylist, allowlist, max_tool_calls_per_step, max_total_cost, write/apply approval flags) should run automatically before every tool invocation.
Runtime should follow the documented order (denylist → allowlist → tool call limit → budget cap → write approval → apply approval) and block or require approval when a guard trips.

Actual behavior

Guard evaluation is never wired into plan or tool execution. AutomationProfile.check_guard(...) is only referenced by the helper service; nothing calls AutomationProfileService.evaluate_guard(...).
ToolRuntime.execute and PlanExecutor execute tools without invoking the guard APIs, so denylist, budget, and call-limit values are ignored.

Steps to reproduce

From the repo root, run:

python3 - <<'PY'
import pathlib
root = pathlib.Path('src/cleveragents')
uses = [str(p.relative_to(root)) for p in root.rglob('*.py')
        if p.name != 'automation_profile.py' and 'check_guard(' in p.read_text(encoding='utf-8')]
print('check_guard references:', uses)
PY

Output shows check_guard only appears in application/services/automation_profile_service.py. A similar search for evaluate_guard( returns an empty list.
Inspect src/cleveragents/tool/lifecycle.py and src/cleveragents/application/services/plan_executor.py; neither calls the guard helper, so guard configuration never executes.

Code location

src/cleveragents/domain/models/core/automation_profile.py
src/cleveragents/application/services/automation_profile_service.py
src/cleveragents/tool/lifecycle.py
src/cleveragents/application/services/plan_executor.py

## What was tested - Evaluated automation guard enforcement against docs/specification.md (lines 28520-28560) on commit 51aab184112728471a44d5a91c334663cf8cd016. ## Expected behavior - Guard settings attached to an automation profile (denylist, allowlist, max_tool_calls_per_step, max_total_cost, write/apply approval flags) should run automatically before every tool invocation. - Runtime should follow the documented order (denylist → allowlist → tool call limit → budget cap → write approval → apply approval) and block or require approval when a guard trips. ## Actual behavior - Guard evaluation is never wired into plan or tool execution. `AutomationProfile.check_guard(...)` is only referenced by the helper service; nothing calls `AutomationProfileService.evaluate_guard(...)`. - `ToolRuntime.execute` and `PlanExecutor` execute tools without invoking the guard APIs, so denylist, budget, and call-limit values are ignored. ## Steps to reproduce 1. From the repo root, run: ```bash python3 - <<'PY' import pathlib root = pathlib.Path('src/cleveragents') uses = [str(p.relative_to(root)) for p in root.rglob('*.py') if p.name != 'automation_profile.py' and 'check_guard(' in p.read_text(encoding='utf-8')] print('check_guard references:', uses) PY ``` 2. Output shows `check_guard` only appears in `application/services/automation_profile_service.py`. A similar search for `evaluate_guard(` returns an empty list. 3. Inspect `src/cleveragents/tool/lifecycle.py` and `src/cleveragents/application/services/plan_executor.py`; neither calls the guard helper, so guard configuration never executes. ## Code location - `src/cleveragents/domain/models/core/automation_profile.py` - `src/cleveragents/application/services/automation_profile_service.py` - `src/cleveragents/tool/lifecycle.py` - `src/cleveragents/application/services/plan_executor.py`

Rows
Columns