CoreRasurae
253 lines
11 KiB
Gherkin
253 lines
11 KiB
Gherkin
Feature: Explicit exception handling and error classification
|
|
As a security-conscious developer
|
|
I want structured error types with redaction and classification
|
|
So that errors are safe for users and diagnostics are preserved
|
|
|
|
Background:
|
|
Given a security exceptions test environment
|
|
|
|
# -- Error code mapping ---------------------------------------------------
|
|
|
|
Scenario: Classify a ValidationError as 422
|
|
Given a ValidationError with message "field is invalid"
|
|
When I classify the security exception
|
|
Then the error code should be 422
|
|
And the error code name should be "VALIDATION_FAILED"
|
|
And the error handling category should be "CLIENT"
|
|
|
|
Scenario: Classify a ResourceNotFoundError as 404
|
|
Given a ResourceNotFoundError for resource "plan" with id "01PLAN001"
|
|
When I classify the security exception
|
|
Then the error code should be 404
|
|
And the error code name should be "NOT_FOUND"
|
|
|
|
Scenario: Classify an AuthenticationError as 401
|
|
Given an AuthenticationError with message "invalid token"
|
|
When I classify the security exception
|
|
Then the error code should be 401
|
|
And the error code name should be "UNAUTHORIZED"
|
|
|
|
Scenario: Classify an AuthorizationError as 403
|
|
Given an AuthorizationError with message "access denied"
|
|
When I classify the security exception
|
|
Then the error code should be 403
|
|
And the error code name should be "FORBIDDEN"
|
|
|
|
Scenario: Classify a RateLimitError as 429
|
|
Given a RateLimitError with message "slow down"
|
|
When I classify the security exception
|
|
Then the error code should be 429
|
|
And the error code name should be "RATE_LIMITED"
|
|
|
|
Scenario: Classify a DatabaseError as 522
|
|
Given a DatabaseError with message "connection lost"
|
|
When I classify the security exception
|
|
Then the error code should be 522
|
|
And the error handling category should be "SERVER"
|
|
|
|
Scenario: Classify a ConfigurationError as 525
|
|
Given a ConfigurationError with message "missing key"
|
|
When I classify the security exception
|
|
Then the error code should be 525
|
|
|
|
Scenario: Classify a NetworkError as 524
|
|
Given a NetworkError with message "timeout"
|
|
When I classify the security exception
|
|
Then the error code should be 524
|
|
|
|
Scenario: Classify a FileSystemError as 523
|
|
Given a FileSystemError with message "permission denied"
|
|
When I classify the security exception
|
|
Then the error code should be 523
|
|
|
|
Scenario: Classify a ProviderError as 520
|
|
Given a ProviderError with message "model failed"
|
|
When I classify the security exception
|
|
Then the error code should be 520
|
|
|
|
Scenario: Classify a TokenLimitExceededError as 526
|
|
Given a TokenLimitExceededError with message "context too long"
|
|
When I classify the security exception
|
|
Then the error code should be 526
|
|
|
|
Scenario: Classify a BusinessRuleViolation as 400
|
|
Given a BusinessRuleViolation with message "invalid transition"
|
|
When I classify the security exception
|
|
Then the error code should be 400
|
|
|
|
Scenario: Classify a PlanError as 400
|
|
Given a PlanError with message "plan stuck"
|
|
When I classify the security exception
|
|
Then the error code should be 400
|
|
|
|
Scenario: Classify an unknown Exception as 500
|
|
Given a bare Exception with message "something broke"
|
|
When I classify the security exception
|
|
Then the error code should be 500
|
|
And the error handling category should be "SERVER"
|
|
|
|
Scenario: Classify an ExecutionError as 521
|
|
Given an ExecutionError with message "agent failed"
|
|
When I classify the security exception
|
|
Then the error code should be 521
|
|
|
|
Scenario: Classify a StreamRoutingError as 528
|
|
Given a StreamRoutingError with message "routing failed"
|
|
When I classify the security exception
|
|
Then the error code should be 528
|
|
|
|
Scenario: Classify a ModelNotAvailableError as 527
|
|
Given a ModelNotAvailableError with message "deprecated"
|
|
When I classify the security exception
|
|
Then the error code should be 527
|
|
|
|
Scenario: Classify a ResourceConflictError as 409
|
|
Given a ResourceConflictError with message "already exists"
|
|
When I classify the security exception
|
|
Then the error code should be 409
|
|
|
|
Scenario: Classify a MissingConfigurationError as 525
|
|
Given a MissingConfigurationError with message "key not set"
|
|
When I classify the security exception
|
|
Then the error code should be 525
|
|
|
|
Scenario: Classify an ExternalServiceError as 503
|
|
Given an ExternalServiceError with message "service down"
|
|
When I classify the security exception
|
|
Then the error code should be 503
|
|
|
|
# -- Secret redaction -----------------------------------------------------
|
|
|
|
Scenario: Redact API key from error details
|
|
Given redaction test details with key "api_key" and value "sk-abc123secret"
|
|
When I redact the error details
|
|
Then the redacted detail "api_key" should be "***REDACTED***"
|
|
|
|
Scenario: Redact password from error details
|
|
Given redaction test details with key "password" and value "my-secret-pass"
|
|
When I redact the error details
|
|
Then the redacted detail "password" should be "***REDACTED***"
|
|
|
|
Scenario: Redact OpenAI key pattern from string value
|
|
Given redaction test details with key "message" and value "Failed with key sk-abcdefghijklmnopqrstuvwxyz"
|
|
When I redact the error details
|
|
Then the redacted detail "message" should contain "***REDACTED***"
|
|
And the redacted detail "message" should not contain "sk-abcdefghijklmnopqrstuvwxyz"
|
|
|
|
Scenario: Redact JWT token pattern from string value
|
|
Given redaction test details with key "log" and value "auth eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9 failed"
|
|
When I redact the error details
|
|
Then the redacted detail "log" should contain "***REDACTED***"
|
|
|
|
Scenario: Redact nested dict values
|
|
Given redaction test details with nested key "config" containing secret_key "abc123"
|
|
When I redact the error details
|
|
Then the redacted nested detail "config" key "secret_key" should be "***REDACTED***"
|
|
|
|
Scenario: Non-sensitive values pass through unchanged
|
|
Given redaction test details with key "plan_id" and value "01PLAN001"
|
|
When I redact the error details
|
|
Then the redacted detail "plan_id" should be "01PLAN001"
|
|
|
|
Scenario: Redact GitHub PAT pattern from string value
|
|
Given redaction test details with key "msg" and value "token ghp_abcdefghijklmnopqrstuvwxyz0123456 expired"
|
|
When I redact the error details
|
|
Then the redacted detail "msg" should contain "***REDACTED***"
|
|
|
|
Scenario: Redact value with redact_value function
|
|
Given a redaction test string value "key=sk-abcdefghijklmnopqrstuvwxyz"
|
|
When I redact the string value
|
|
Then the redacted string should contain "***REDACTED***"
|
|
|
|
# -- Wrapping unexpected exceptions ----------------------------------------
|
|
|
|
Scenario: Wrap a bare Exception into CleverAgentsError
|
|
Given a bare Exception with message "internal crash"
|
|
When I wrap the unexpected exception
|
|
Then the wrapped error should be a CleverAgentsError
|
|
And the wrapped error message should be "An unexpected error occurred"
|
|
And the wrapped error details should have key "exception_type"
|
|
|
|
Scenario: Wrap preserves CleverAgentsError as-is
|
|
Given a CleverAgentsError with message "known issue"
|
|
When I wrap the unexpected exception
|
|
Then the wrapped error message should be "known issue"
|
|
|
|
Scenario: Wrap merges context into CleverAgentsError details
|
|
Given a CleverAgentsError with message "known issue"
|
|
And a wrapping context with key "plan_id" and value "01PLAN001"
|
|
When I wrap the unexpected exception
|
|
Then the wrapped error details should have key "plan_id"
|
|
|
|
Scenario: Wrap with custom safe message
|
|
Given a bare Exception with message "segfault"
|
|
When I wrap the unexpected exception with message "Operation failed"
|
|
Then the wrapped error message should be "Operation failed"
|
|
|
|
Scenario: Wrap redacts secrets in context
|
|
Given a bare Exception with message "oops"
|
|
And a wrapping context with key "api_key" and value "sk-secret123456789012345"
|
|
When I wrap the unexpected exception
|
|
Then the wrapped error details key "api_key" should be "***REDACTED***"
|
|
|
|
# -- CLI formatting -------------------------------------------------------
|
|
|
|
Scenario: Format error for CLI output
|
|
Given a ValidationError with message "name is required"
|
|
When I classify the security exception
|
|
And I format the error for CLI
|
|
Then the error CLI output should contain "422"
|
|
And the error CLI output should contain "VALIDATION_FAILED"
|
|
And the error CLI output should contain "name is required"
|
|
|
|
Scenario: Format error with details for CLI output
|
|
Given a detailed CleverAgentsError "db fail" with detail "table" as "plans"
|
|
When I classify the security exception
|
|
And I format the error for CLI
|
|
Then the error CLI output should contain "table: plans"
|
|
|
|
# -- Secret in message (S1) ------------------------------------------------
|
|
|
|
Scenario: classify_error redacts secrets embedded in exception message
|
|
Given a CleverAgentsError with message "key=sk-abcdefghijklmnopqrstuvwxyz leaked"
|
|
When I classify the security exception
|
|
Then the error info message should contain "***REDACTED***"
|
|
And the error info message should not contain "sk-abcdefghijklmnopqrstuvwxyz"
|
|
|
|
Scenario: Format CLI output redacts secrets from message
|
|
Given a CleverAgentsError with message "tok sk-abcdefghijklmnopqrstuvwxyz end"
|
|
When I classify the security exception
|
|
And I format the error for CLI
|
|
Then the error CLI output should contain "***REDACTED***"
|
|
And the error CLI output should not contain "sk-abcdefghijklmnopqrstuvwxyz"
|
|
|
|
# -- List value redaction (B2/T1) ------------------------------------------
|
|
|
|
Scenario: Redact secrets inside list values
|
|
Given redaction test details with key "keys" and list values "sk-abcdefghijklmnopqrstuvwxyz" and "normal"
|
|
When I redact the error details
|
|
Then the redacted detail "keys" should be a list
|
|
And the redacted list "keys" should contain "***REDACTED***"
|
|
And the redacted list "keys" should not contain "sk-abcdefghijklmnopqrstuvwxyz"
|
|
|
|
# -- Non-string non-dict passthrough (T4) ----------------------------------
|
|
|
|
Scenario: Integer values pass through redaction unchanged
|
|
Given redaction test details with key "count" and integer value 42
|
|
When I redact the error details
|
|
Then the redacted detail "count" should be integer 42
|
|
|
|
Scenario: Boolean values pass through redaction unchanged
|
|
Given redaction test details with key "enabled" and boolean value true
|
|
When I redact the error details
|
|
Then the redacted detail "enabled" should be boolean true
|
|
|
|
# -- Non-mutation of original details (B1) ---------------------------------
|
|
|
|
Scenario: Wrap does not mutate the original exception details dict
|
|
Given a tracked CleverAgentsError "original" with details key "a" value "1"
|
|
And a wrapping context with key "b" and value "2"
|
|
When I wrap the unexpected exception
|
|
Then the original exception details reference should not have key "b"
|
|
And the wrapped error details should have key "b"
|