CoreRasurae
53455275ba
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 17s
CI / quality (pull_request) Successful in 18s
CI / lint (pull_request) Successful in 23s
CI / security (pull_request) Successful in 31s
CI / typecheck (pull_request) Successful in 35s
CI / integration_tests (pull_request) Successful in 3m46s
CI / unit_tests (pull_request) Successful in 11m44s
CI / docker (pull_request) Successful in 38s
CI / benchmark-regression (pull_request) Successful in 27m39s
CI / coverage (pull_request) Successful in 55m41s
CI / build (push) Successful in 15s
CI / quality (push) Successful in 16s
CI / lint (push) Successful in 19s
CI / typecheck (push) Successful in 32s
CI / benchmark-regression (push) Has been skipped
CI / security (push) Successful in 41s
CI / integration_tests (push) Successful in 2m55s
CI / unit_tests (push) Successful in 11m43s
CI / docker (push) Successful in 40s
CI / benchmark-publish (push) Successful in 16m11s
CI / coverage (push) Successful in 44m49s
Add runtime autonomy constraints (max steps, tool budget, required confirmations) and a structured audit trail for plan execution. New domain models: - AutonomyGuardrails: enforces step limits, tool budgets, and confirmation gates with validators and check methods - GuardrailAuditEntry: records each enforcement event with timestamp, event type, guard name, result, reason, and context - GuardrailAuditTrail: ordered collection of audit entries persisted to plan metadata New service: - AutonomyGuardrailService: high-level service for configuring guardrails per plan, checking constraints, recording audit entries, and serializing/restoring state via plan metadata Tests: - Behave: 69 scenarios covering model validation, step/budget/ confirmation checks, audit trail recording, and service operations - Robot: 8 test cases for autonomy guardrail CLI flag smoke testing - ASV: 6 benchmark suites measuring enforcement overhead Documentation: - Updated docs/reference/automation_profiles.md with guardrail fields, enforcement behavior, audit trail schema, and event type reference ISSUES CLOSED: #204
88 lines
4.2 KiB
Plaintext
88 lines
4.2 KiB
Plaintext
*** Settings ***
|
|
Documentation Smoke tests for autonomy guardrails and audit trail
|
|
Resource ${CURDIR}/common.resource
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
|
|
*** Variables ***
|
|
${HELPER_SCRIPT} robot/helper_autonomy_guardrails.py
|
|
|
|
*** Test Cases ***
|
|
Guardrail Step Limit Allows Within Bounds
|
|
[Documentation] Step limit allows when under limit
|
|
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} step-allow cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} step-allow-ok
|
|
|
|
Guardrail Step Limit Blocks At Max
|
|
[Documentation] Step limit blocks when at max steps
|
|
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} step-block cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} step-block-ok
|
|
|
|
Guardrail Budget Allows Within Limit
|
|
[Documentation] Budget allows when within limit
|
|
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} budget-allow cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} budget-allow-ok
|
|
|
|
Guardrail Budget Blocks Exceeded
|
|
[Documentation] Budget blocks when exceeded
|
|
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} budget-block cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} budget-block-ok
|
|
|
|
Guardrail Confirmation Required
|
|
[Documentation] Confirmation required for listed operation
|
|
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} confirm-required cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} confirm-required-ok
|
|
|
|
Guardrail Audit Trail Records Events
|
|
[Documentation] Audit trail records enforcement events
|
|
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} audit-trail cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} audit-trail-ok
|
|
|
|
Guardrail Service Metadata Persistence
|
|
[Documentation] Service can serialize and restore from metadata
|
|
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} metadata cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} metadata-ok
|
|
|
|
Guardrail Wall Clock Allows Within Limit
|
|
[Documentation] Wall-clock allows when within time limit
|
|
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} wall-clock-allow cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} wall-clock-allow-ok
|
|
|
|
Guardrail Wall Clock Blocks When Expired
|
|
[Documentation] Wall-clock blocks when time limit exceeded
|
|
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} wall-clock-block cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} wall-clock-block-ok
|
|
|
|
Guardrail Actor Tool Call Limit
|
|
[Documentation] Actor tool-call limit enforcement
|
|
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} actor-limit cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-limit-ok
|
|
|
|
Guardrail Audit Trail Eviction
|
|
[Documentation] Audit trail evicts oldest entries when full
|
|
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} eviction cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} eviction-ok
|
|
|
|
Guardrail Zero Budget Edge Case
|
|
[Documentation] Zero budget blocks any positive cost
|
|
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} zero-budget cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} zero-budget-ok
|
|
|
|
Guardrail CLI Flags Summary
|
|
[Documentation] Summary of autonomy guardrail features
|
|
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} summary cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} summary-ok
|