freemo
36b3212607
CI / typecheck (pull_request) Successful in 1m2s
CI / security (pull_request) Successful in 51s
CI / quality (pull_request) Successful in 46s
CI / lint (pull_request) Successful in 3m21s
CI / build (pull_request) Successful in 19s
CI / helm (pull_request) Successful in 23s
CI / unit_tests (pull_request) Successful in 7m6s
CI / e2e_tests (pull_request) Successful in 18m1s
CI / integration_tests (pull_request) Successful in 22m52s
CI / docker (pull_request) Successful in 21s
CI / coverage (pull_request) Successful in 10m58s
CI / status-check (pull_request) Successful in 1s
CI / benchmark-publish (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Successful in 57m25s
Audited 587 existing Behave feature files to identify scenarios with poor data variation (hardcoded or repetitive single values). Prioritized high-impact candidates: ULID validation, NamespacedName validation, project name validation, and skill schema name validation. Created 5 new feature files using Behave's native Scenario Outline + Examples tables as the data variation mechanism (since blocking issues #2760 TestDataFactory and #2765 Centralized Fixture System are not yet implemented): - features/data_variation_plan_ulid.feature: 33 scenarios covering valid/invalid ULID formats, boundary lengths, illegal characters (I/L/O/U), legacy names, and CLI command validation - features/data_variation_namespaced_name.feature: 32 scenarios covering valid names, special characters in namespace/name components, boundary lengths - features/data_variation_project_name.feature: 35 scenarios covering invalid special characters, valid formats, path resolution - features/data_variation_skill_name.feature: 39 scenarios covering invalid names, tool refs, MCP transports, include names - features/data_variation_edge_cases.feature: 81 scenarios covering empty/null values, boundary lengths, special characters, and invalid input types across all four domains All 592 features pass (14636 scenarios), typecheck passes with 0 errors Key design decisions: - Used Behave Scenario Outline + Examples tables as the Behave-native data variation approach (pending #2760 and #2765) - Created new additive feature files rather than modifying existing ones to avoid breaking existing tests - Verified each scenario against actual implementation behavior before including in Examples tables - Removed pipe characters from table cells (Behave table delimiter conflict) - Used "a ValueError should be raised" for Pydantic ValidationError assertions (Pydantic ValidationError IS a ValueError) Impact: - Improves data variation coverage in critical test domains without restructuring the existing test suite - Keeps changes isolated and additive to minimize risk to current tests - Maintains alignment with ongoing infrastructure work (factory/fixture system) while providing immediate gains ISSUES CLOSED: #2772
133 lines
9.3 KiB
Gherkin
133 lines
9.3 KiB
Gherkin
Feature: ULID validation data variation — Scenario Outline refactoring
|
|
As a CleverAgents developer
|
|
I want ULID validation tests to cover a wide range of inputs
|
|
So that edge cases and boundary conditions are systematically exercised
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
# Refactored from plan_ulid_validation.feature using Scenario Outline.
|
|
# Each Examples table row exercises a distinct code path or boundary.
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
# Valid ULID inputs — all should pass validation
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
|
|
Scenario Outline: _validate_plan_ulid accepts valid ULID strings
|
|
When I call _validate_plan_ulid with "<ulid>"
|
|
Then ulid-val the validation should succeed and return "<ulid>"
|
|
|
|
Examples: Valid ULID formats
|
|
| ulid | description |
|
|
| 01HXM8C2ZK4Q7C2B3F2R4VYV6J | standard uppercase ULID |
|
|
| 01hxm8c2zk4q7c2b3f2r4vyv6j | lowercase ULID (case-insensitive) |
|
|
| 01ARZ3NDEKTSV4RRFFQ69G5FAV | another valid uppercase ULID |
|
|
| 00000000000000000000000000 | minimum value ULID (all zeros) |
|
|
| 7ZZZZZZZZZZZZZZZZZZZZZZZZZ | maximum value ULID (all max chars) |
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
# Invalid ULID inputs — wrong length
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
|
|
Scenario Outline: _validate_plan_ulid rejects ULIDs with wrong length
|
|
When I call _validate_plan_ulid with "<invalid_input>"
|
|
Then ulid-val the validation should raise ValidationError
|
|
And ulid-val the error message should contain "ULID"
|
|
|
|
Examples: Wrong length inputs
|
|
| invalid_input | description |
|
|
| 01ARZ3NDEK | too short (10 chars) |
|
|
| 01ARZ3NDEKTSV4RRFFQ69G5FAAEXTRA | too long (32 chars) |
|
|
| 01ARZ3NDEKTSV4RRFFQ69G5FA | too short by one (25 chars) |
|
|
| 01ARZ3NDEKTSV4RRFFQ69G5FAVXX | too long by one (27 chars) |
|
|
| A | single character |
|
|
| 01ARZ3NDEKTSV4RRFFQ69G5FAVXXX | too long by three (28 chars) |
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
# Invalid ULID inputs — illegal characters (I, L, O, U excluded from
|
|
# Crockford base32 alphabet to avoid visual ambiguity)
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
|
|
Scenario Outline: _validate_plan_ulid rejects ULIDs with illegal characters
|
|
When I call _validate_plan_ulid with "<invalid_input>"
|
|
Then ulid-val the validation should raise ValidationError
|
|
And ulid-val the error message should contain "ULID"
|
|
|
|
Examples: Crockford base32 excluded characters
|
|
| invalid_input | description |
|
|
| 01ARZ3NDEKTSV4RRFFQ69G5FII | contains I (excluded from base32) |
|
|
| 01ARZ3NDEKTSV4RRFFQ69G5FLL | contains L (excluded from base32) |
|
|
| 01ARZ3NDEKTSV4RRFFQ69G5FOO | contains O (excluded from base32) |
|
|
| 01ARZ3NDEKTSV4RRFFQ69G5FUU | contains U (excluded from base32) |
|
|
|
|
Examples: Special characters and whitespace
|
|
| invalid_input | description |
|
|
| 01ARZ3NDEKTSV4RRFFQ69G5F!@ | contains special characters |
|
|
| 01ARZ3NDEKTSV4RRFFQ69G5F-X | contains hyphen |
|
|
| 01ARZ3NDEKTSV4RRFFQ69G5F.X | contains period |
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
# Invalid ULID inputs — plain strings and legacy names
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
|
|
Scenario Outline: _validate_plan_ulid rejects plain strings that are not ULIDs
|
|
When I call _validate_plan_ulid with "<invalid_input>"
|
|
Then ulid-val the validation should raise ValidationError
|
|
And ulid-val the error message should contain "ULID"
|
|
|
|
Examples: Plain non-ULID strings
|
|
| invalid_input | description |
|
|
| not-a-ulid | plain hyphenated string |
|
|
| hello | short plain word |
|
|
| plan-123 | legacy-style plan name |
|
|
| abc | very short plain string |
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
# Legacy plan names — should include actionable error guidance
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
|
|
Scenario Outline: _validate_plan_ulid rejects legacy plan names with guidance
|
|
When I call _validate_plan_ulid with "<legacy_name>"
|
|
Then ulid-val the validation should raise ValidationError
|
|
And ulid-val the error message should contain "ULID"
|
|
And ulid-val the error message should contain "legacy"
|
|
And ulid-val the error message should contain "agents plan use"
|
|
|
|
Examples: Legacy plan name formats
|
|
| legacy_name | description |
|
|
| 64-bit port plan | legacy name with spaces |
|
|
| my-legacy-plan | legacy hyphenated name |
|
|
| add error handling | legacy descriptive name |
|
|
| refactor authentication module | legacy multi-word name |
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
# CLI command ULID validation — multiple commands, multiple invalid inputs
|
|
# ─────────────────────────────────────────────────────────────────────────
|
|
|
|
Scenario Outline: plan <command> rejects non-ULID input with actionable error
|
|
Given a ulid-validation CLI runner
|
|
When I invoke ulid-validation plan <command> with "<invalid_input>"
|
|
Then the ulid-validation command should abort
|
|
And the ulid-validation output should contain "ULID"
|
|
|
|
Examples: CLI commands with plain non-ULID strings
|
|
| command | invalid_input | description |
|
|
| execute | not-a-valid-id | execute with plain string |
|
|
| apply | not-a-valid-id | apply with plain string |
|
|
| execute | short | execute with very short string |
|
|
| apply | plan-123 | apply with legacy-style name |
|
|
|
|
Scenario Outline: plan <command> rejects legacy plan names with full guidance
|
|
Given a ulid-validation CLI runner
|
|
When I invoke ulid-validation plan <command> with "<legacy_name>"
|
|
Then the ulid-validation command should abort
|
|
And the ulid-validation output should contain "not found"
|
|
And the ulid-validation output should contain "ULID"
|
|
And the ulid-validation output should contain "legacy"
|
|
And the ulid-validation output should contain "agents plan use"
|
|
|
|
Examples: CLI commands with legacy plan names
|
|
| command | legacy_name | description |
|
|
| execute | my-legacy-plan | execute with legacy hyphenated name |
|
|
| apply | 64-bit port plan | apply with legacy spaced name |
|
|
| execute | add error handling | execute with legacy descriptive name |
|