Files
cleveragents-core/robot/security_eval.robot
T
2026-02-21 16:32:03 +00:00

102 lines
4.9 KiB
Plaintext

*** Settings ***
Documentation Integration smoke tests for SEC1 config security scanner.
... Validates that the security_scanner module correctly
... detects dangerous patterns in config files and reports
... file + line numbers in structured violations.
Resource ${CURDIR}/common.resource
Suite Setup Setup Test Environment
Suite Teardown Cleanup Test Environment
*** Variables ***
${HELPER_SCRIPT} robot/helper_security_eval_scanner.py
*** Test Cases ***
Scanner Detects Eval In Config Content
[Documentation] Verify scanner flags eval() in YAML-like content
[Tags] security scanner
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} detect-eval cwd=${WORKSPACE}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} STATUS=ok
Should Contain ${result.stdout} TOKEN=eval(
Should Contain ${result.stdout} SEVERITY=CRITICAL
Scanner Detects Exec In Config Content
[Documentation] Verify scanner flags exec() in config content
[Tags] security scanner
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} detect-exec cwd=${WORKSPACE}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} STATUS=ok
Should Contain ${result.stdout} TOKEN=exec(
Scanner Detects Import Injection
[Documentation] Verify scanner flags __import__ in config content
[Tags] security scanner
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} detect-import cwd=${WORKSPACE}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} STATUS=ok
Should Contain ${result.stdout} TOKEN=__import__
Scanner Detects Os System Call
[Documentation] Verify scanner flags os.system in config content
[Tags] security scanner
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} detect-os-system cwd=${WORKSPACE}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} STATUS=ok
Should Contain ${result.stdout} TOKEN=os.system
Scanner Detects Subprocess Usage
[Documentation] Verify scanner flags subprocess calls in config content
[Tags] security scanner
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} detect-subprocess cwd=${WORKSPACE}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} STATUS=ok
Should Contain ${result.stdout} TOKEN=subprocess.call
Scanner Passes Clean Config
[Documentation] Verify scanner returns zero violations for safe content
[Tags] security scanner
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} clean-config cwd=${WORKSPACE}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} STATUS=ok
Should Contain ${result.stdout} VIOLATIONS=0
Scanner Reports Correct Line Numbers
[Documentation] Verify violations include accurate line numbers
[Tags] security scanner
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} line-numbers cwd=${WORKSPACE}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} STATUS=ok
Should Contain ${result.stdout} LINE=2
Validate Config Safety Raises On Violation
[Documentation] Verify validate_config_safety raises ConfigurationError
[Tags] security scanner validation
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} validate-raises cwd=${WORKSPACE}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} STATUS=ok
Should Not Contain ${result.stdout} STATUS=fail
Scanner Skips Comment Lines
[Documentation] Verify scanner ignores patterns inside comments
[Tags] security scanner
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} skip-comments cwd=${WORKSPACE}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} STATUS=ok
Should Contain ${result.stdout} VIOLATIONS=0
Scanner Detects Template Directives
[Documentation] Verify scanner flags Jinja2-style template directives
[Tags] security scanner
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} detect-template cwd=${WORKSPACE}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} STATUS=ok
Should Contain ${result.stdout} TOKEN={{
Scanner Skips Inline Comments
[Documentation] Verify scanner ignores patterns in inline comments
[Tags] security scanner
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} inline-comment cwd=${WORKSPACE}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} STATUS=ok
Should Contain ${result.stdout} VIOLATIONS=0