cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity

test(security): cover safety profile enforcement #516

Closed
brent.edwards wants to merge 2 commits from feature/post-safety-profile-tests into master
pull from: feature/post-safety-profile-tests
merge into: cleveragents:master
Conversation 4 Commits 2 Files Changed 9
+731

2 Commits

Author SHA1 Message Date
brent.edwards 927c1f0e4b Merge master into feature/post-safety-profile-tests
CI / lint (pull_request) Successful in 24s
Details
CI / typecheck (pull_request) Successful in 45s
Details
CI / quality (pull_request) Successful in 43s
Details
CI / security (pull_request) Successful in 49s
Details
CI / benchmark-publish (pull_request) Has been skipped
Details
CI / build (pull_request) Successful in 24s
Details
CI / unit_tests (pull_request) Failing after 3m35s
Details
CI / docker (pull_request) Has been skipped
Details
CI / integration_tests (pull_request) Successful in 4m3s
Details
CI / benchmark-regression (pull_request) Failing after 33s
Details
CI / coverage (pull_request) Failing after 5m22s
Details
2026-03-04 00:06:17 +00:00
brent.edwards cb28ecf63d test(security): cover safety profile enforcement
CI / benchmark-publish (pull_request) Has been skipped
Details
CI / lint (pull_request) Successful in 16s
Details
CI / build (pull_request) Successful in 16s
Details
CI / quality (pull_request) Successful in 23s
Details
CI / typecheck (pull_request) Successful in 33s
Details
CI / security (pull_request) Successful in 35s
Details
CI / unit_tests (pull_request) Successful in 2m27s
Details
CI / docker (pull_request) Successful in 41s
Details
CI / integration_tests (pull_request) Successful in 3m10s
Details
CI / coverage (pull_request) Successful in 4m1s
Details
CI / benchmark-regression (pull_request) Successful in 25m55s
Details 
Add 20 new Behave BDD scenarios across two feature files to validate
SafetyProfile model constraints and Action-level safety integration:

safety_profile.feature (12 scenarios):
  - Boolean flag toggles (sandbox, checkpoints, unsafe_tools, human_approval)
  - Empty/deny-none skill categories semantics
  - Cost-without-total and total-without-cost partial bounds
  - Type validation (string rejected for numeric field)
  - Negative cost rejection via profile (-5.0)
  - Upper-bound retries acceptance (max_retries_per_step=100)
  - Restrictive full-constraint profile with content assertions

safety_profile_cost_retry.feature (8 scenarios):
  - Valid cost bounds with max_total_cost assertion
  - Cost-per-plan exceeds total rejection
  - Valid/invalid retry counts (0, 50, -1, 101)
  - Zero cost-per-plan acceptance
  - Equal cost-per-plan and total boundary acceptance
  - Missing profile defaults to no constraints

Additional changes:
  - Robot Framework: 2 smoke tests (validation rules, action attachment)
  - ASV benchmarks: safety_profile_tests_bench.py with setup()-based imports
  - Updated docs/development/testing.md with fixture documentation
  - Standardised exception handling to catch only ValidationError (not
    TypeError/ValueError) in step definitions
  - Added isinstance(ValidationError) assertions for error type checking
  - Fixed pre-existing server_mode test failures in cli_core.feature,
    cli_core.robot, server_stubs.robot, and helper_server_stubs.py by
    mocking resolve_server_mode in Behave steps and relaxing Robot
    assertions to accept both 'disabled' and 'stubbed'

ISSUES CLOSED: #333
 2026-03-03 20:13:33 +00:00