freemo
36b3212607
CI / typecheck (pull_request) Successful in 1m2s
CI / security (pull_request) Successful in 51s
CI / quality (pull_request) Successful in 46s
CI / lint (pull_request) Successful in 3m21s
CI / build (pull_request) Successful in 19s
CI / helm (pull_request) Successful in 23s
CI / unit_tests (pull_request) Successful in 7m6s
CI / e2e_tests (pull_request) Successful in 18m1s
CI / integration_tests (pull_request) Successful in 22m52s
CI / docker (pull_request) Successful in 21s
CI / coverage (pull_request) Successful in 10m58s
CI / status-check (pull_request) Successful in 1s
CI / benchmark-publish (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Successful in 57m25s
Audited 587 existing Behave feature files to identify scenarios with poor data variation (hardcoded or repetitive single values). Prioritized high-impact candidates: ULID validation, NamespacedName validation, project name validation, and skill schema name validation. Created 5 new feature files using Behave's native Scenario Outline + Examples tables as the data variation mechanism (since blocking issues #2760 TestDataFactory and #2765 Centralized Fixture System are not yet implemented): - features/data_variation_plan_ulid.feature: 33 scenarios covering valid/invalid ULID formats, boundary lengths, illegal characters (I/L/O/U), legacy names, and CLI command validation - features/data_variation_namespaced_name.feature: 32 scenarios covering valid names, special characters in namespace/name components, boundary lengths - features/data_variation_project_name.feature: 35 scenarios covering invalid special characters, valid formats, path resolution - features/data_variation_skill_name.feature: 39 scenarios covering invalid names, tool refs, MCP transports, include names - features/data_variation_edge_cases.feature: 81 scenarios covering empty/null values, boundary lengths, special characters, and invalid input types across all four domains All 592 features pass (14636 scenarios), typecheck passes with 0 errors Key design decisions: - Used Behave Scenario Outline + Examples tables as the Behave-native data variation approach (pending #2760 and #2765) - Created new additive feature files rather than modifying existing ones to avoid breaking existing tests - Verified each scenario against actual implementation behavior before including in Examples tables - Removed pipe characters from table cells (Behave table delimiter conflict) - Used "a ValueError should be raised" for Pydantic ValidationError assertions (Pydantic ValidationError IS a ValueError) Impact: - Improves data variation coverage in critical test domains without restructuring the existing test suite - Keeps changes isolated and additive to minimize risk to current tests - Maintains alignment with ongoing infrastructure work (factory/fixture system) while providing immediate gains ISSUES CLOSED: #2772
229 lines
15 KiB
Gherkin
229 lines
15 KiB
Gherkin
Feature: Edge case data variation — empty, null, boundary, special characters, invalid types
|
|
As a CleverAgents developer
|
|
I want edge case tests to cover empty/null values, boundary lengths, special characters,
|
|
and invalid input types systematically
|
|
So that the system handles all boundary conditions correctly
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
# New edge-case scenarios added as part of issue #2772.
|
|
# These complement the Scenario Outline refactoring in the other
|
|
# data_variation_*.feature files.
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
# Python content sanitization — edge cases
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
|
|
Background:
|
|
Given I have a plan service for sanitization tests
|
|
|
|
Scenario Outline: Sanitize valid Python expressions of varying complexity
|
|
When I sanitize the python content "<content>"
|
|
Then the sanitized content should be "<content>"
|
|
And the sanitization error should be None
|
|
And the sanitization reason should be None
|
|
|
|
Examples: Simple valid Python expressions
|
|
| content | description |
|
|
| x = 1 + 2 | simple arithmetic assignment |
|
|
| result = "hello" | string assignment |
|
|
| items = [] | empty list assignment |
|
|
| data = {} | empty dict assignment |
|
|
| flag = True | boolean assignment |
|
|
| value = None | None assignment |
|
|
| count = 0 | zero assignment |
|
|
|
|
Scenario Outline: Sanitize Python with markdown code fences strips fences
|
|
When I sanitize the python content with code fences wrapping "<inner>"
|
|
Then the sanitized content should be "<inner>"
|
|
And the sanitization error should be None
|
|
And the sanitization reason should be "code_fence_removed"
|
|
|
|
Examples: Code fence variations
|
|
| inner | description |
|
|
| x = 42 | simple assignment in fences |
|
|
| result = "hello" | string assignment in fences |
|
|
| items = [1, 2, 3] | list assignment in fences |
|
|
| def foo(): pass | function definition in fences |
|
|
|
|
Scenario Outline: Sanitize non-Python prose wraps in docstring
|
|
When I sanitize the python content "<prose>"
|
|
Then the sanitization error should be None
|
|
And the sanitization reason should be "docstring_wrapped"
|
|
|
|
Examples: Non-Python prose inputs
|
|
| prose | description |
|
|
| This is just plain English text, not code. | plain English sentence |
|
|
| Please implement the feature as described. | instruction-style prose |
|
|
| The function should return a list of items. | description-style prose |
|
|
|
|
Scenario: Sanitize content with null byte returns error
|
|
When I sanitize python content containing a null byte
|
|
Then the sanitization error should not be None
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
# Project name validation — edge cases with special characters
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
|
|
Scenario Outline: Project rejects names with various special characters
|
|
When I try to create a project with name "<invalid_name>"
|
|
Then a project validation error should be raised mentioning "alphanumeric"
|
|
|
|
Examples: Special character edge cases
|
|
| invalid_name | description |
|
|
| my@project | at-sign |
|
|
| my/project | forward slash |
|
|
| project! | exclamation mark |
|
|
| proj#tag | hash symbol |
|
|
| proj%done | percent sign |
|
|
| proj^up | caret |
|
|
| proj&more | ampersand |
|
|
| proj*star | asterisk |
|
|
| proj+plus | plus sign |
|
|
| proj=eq | equals sign |
|
|
| proj<less | less-than sign |
|
|
| proj>more | greater-than sign |
|
|
| proj?q | question mark |
|
|
| proj.pipe | period character |
|
|
| proj:colon | colon |
|
|
| proj;semi | semicolon |
|
|
|
|
Scenario: Project rejects empty name
|
|
When I try to create a project with empty name
|
|
Then a project validation error should be raised
|
|
|
|
Scenario Outline: Project accepts valid names with allowed characters
|
|
When I create a project fixture with name "<valid_name>"
|
|
Then the project should be created with that name
|
|
|
|
Examples: Valid name formats
|
|
| valid_name | description |
|
|
| simple | plain alphanumeric |
|
|
| my-project | hyphenated name |
|
|
| my_project | underscored name |
|
|
| my project | name with space |
|
|
| project123 | name with trailing numbers |
|
|
| 123project | name starting with numbers |
|
|
| a | single character |
|
|
| My Project Name | mixed case with spaces |
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
# NamespacedName — special character edge cases
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
|
|
Scenario Outline: NamespacedName rejects names with special characters in namespace
|
|
When I parse the namespaced name "<full_name>" expecting an error
|
|
Then a ValueError should be raised
|
|
|
|
Examples: Special characters in namespace
|
|
| full_name | description |
|
|
| bad@ns/name | at-sign in namespace |
|
|
| bad ns/name | space in namespace |
|
|
| bad!ns/name | exclamation in namespace |
|
|
| bad#ns/name | hash in namespace |
|
|
|
|
Scenario Outline: NamespacedName rejects names with special characters in name component
|
|
When I parse the namespaced name "<full_name>" expecting an error
|
|
Then a ValueError should be raised
|
|
|
|
Examples: Special characters in name component
|
|
| full_name | description |
|
|
| ns/bad name | space in name component |
|
|
| ns/bad@name | at-sign in name component |
|
|
| ns/bad!name | exclamation in name component |
|
|
| ns/bad#name | hash in name component |
|
|
| ns/bad/extra | extra slash (three components) |
|
|
|
|
Scenario Outline: NamespacedName accepts names at various boundary lengths
|
|
When I parse the namespaced name "<full_name>"
|
|
Then the parsed namespace should be "<namespace>"
|
|
And the parsed item name should be "<item_name>"
|
|
|
|
Examples: Boundary length names
|
|
| full_name | namespace | item_name | description |
|
|
| a/b | a | b | minimum length (1 char each) |
|
|
| ab/cd | ab | cd | two chars each |
|
|
| abc/def | abc | def | three chars each |
|
|
| local/my-tool | local | my-tool | standard short name |
|
|
| my-namespace/my-tool-name | my-namespace | my-tool-name | medium length names |
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
# ULID validation — boundary and special character edge cases
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
|
|
Scenario Outline: ULID validation rejects inputs at boundary lengths
|
|
When I call _validate_plan_ulid with "<input>"
|
|
Then ulid-val the validation should raise ValidationError
|
|
And ulid-val the error message should contain "ULID"
|
|
|
|
Examples: Boundary length inputs
|
|
| input | description |
|
|
| 01ARZ3NDEKTSV4RRFFQ69G5FA | 25 chars (one short of valid) |
|
|
| 01ARZ3NDEKTSV4RRFFQ69G5FAVX | 27 chars (one over valid) |
|
|
| A | single character |
|
|
| 01ARZ3NDEK | 10 chars (too short) |
|
|
|
|
Scenario Outline: ULID validation accepts valid ULIDs at boundary values
|
|
When I call _validate_plan_ulid with "<ulid>"
|
|
Then ulid-val the validation should succeed and return "<ulid>"
|
|
|
|
Examples: Valid ULIDs at boundary values
|
|
| ulid | description |
|
|
| 00000000000000000000000000 | all zeros (minimum value) |
|
|
| 7ZZZZZZZZZZZZZZZZZZZZZZZZZ | all max chars (maximum value) |
|
|
| 01HXM8C2ZK4Q7C2B3F2R4VYV6J | standard valid ULID |
|
|
|
|
Scenario Outline: ULID validation rejects ULIDs with illegal characters
|
|
When I call _validate_plan_ulid with "<input>"
|
|
Then ulid-val the validation should raise ValidationError
|
|
And ulid-val the error message should contain "ULID"
|
|
|
|
Examples: Crockford base32 excluded characters
|
|
| input | description |
|
|
| 01ARZ3NDEKTSV4RRFFQ69G5FII | contains I (excluded from base32) |
|
|
| 01ARZ3NDEKTSV4RRFFQ69G5FLL | contains L (excluded from base32) |
|
|
| 01ARZ3NDEKTSV4RRFFQ69G5FOO | contains O (excluded from base32) |
|
|
| 01ARZ3NDEKTSV4RRFFQ69G5FUU | contains U (excluded from base32) |
|
|
| 01ARZ3NDEKTSV4RRFFQ69G5F!@ | contains special characters |
|
|
| 01ARZ3NDEKTSV4RRFFQ69G5F-X | contains hyphen |
|
|
| 01ARZ3NDEKTSV4RRFFQ69G5F.X | contains period |
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
# Skill schema — edge cases for inline tool source values
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
|
|
Scenario Outline: Skill schema rejects invalid inline tool source values
|
|
Given a skill YAML string with an inline tool source "<source>"
|
|
When I validate the skill schema expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "custom"
|
|
|
|
Examples: Invalid source values
|
|
| source | description |
|
|
| plugin | plugin source (not supported) |
|
|
| external | external source (not supported) |
|
|
| remote | remote source (not supported) |
|
|
| builtin | builtin source (not supported) |
|
|
| inline | inline source (not supported) |
|
|
| file | file source (not supported) |
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
# Skill schema — edge cases for MCP transport values
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
|
|
Scenario Outline: Skill schema rejects invalid MCP transport values
|
|
Given a skill YAML string with MCP transport "<transport>"
|
|
When I validate the skill schema expecting failure
|
|
Then the skill schema validation should fail
|
|
And the skill schema error should mention "transport"
|
|
|
|
Examples: Invalid transport values
|
|
| transport | description |
|
|
| grpc | gRPC (not supported) |
|
|
| websocket | WebSocket (not supported) |
|
|
| http | plain HTTP (not supported) |
|
|
| tcp | TCP (not supported) |
|
|
| invalid | completely invalid value |
|
|
| ftp | FTP (not supported) |
|
|
| amqp | AMQP (not supported) |
|