cleveragents-core/features/semantic_validation.feature

Feature: Semantic validation service
  As a plan executor
  I want semantic validation checks for Python source code
  So that syntax errors, missing imports, and broken references are caught early

  Background:
    Given a semantic validation test environment

  # ── Severity enum ──────────────────────────────────────────────

  Scenario: SemanticValidationSeverity has three levels
    Then the semantic severity enum has values "info", "warn", "error"

  # ── SemanticCheckResult model ──────────────────────────────────

  Scenario: SemanticCheckResult normalised output has passed message data
    Given a semantic check result with passed true and message "All good"
    Then the semantic check result passed is true
    And the semantic check result message is "All good"
    And the semantic check result data is none

  Scenario: SemanticCheckResult with failure and data
    Given a semantic check result with passed false and message "Error found" and data
    Then the semantic check result passed is false
    And the semantic check result has data key "detail"

  # ── SyntaxCheckRule ────────────────────────────────────────────

  Scenario: SyntaxCheckRule passes for valid Python
    Given valid Python source code
    When the syntax check rule runs
    Then the semantic check result passed is true

  Scenario: SyntaxCheckRule fails for invalid Python
    Given Python source code with a syntax error
    When the syntax check rule runs
    Then the semantic check result passed is false
    And the semantic check result message contains "Syntax error"
    And the semantic check result severity is "error"

  # ── MissingImportRule ──────────────────────────────────────────

  Scenario: MissingImportRule passes for valid imports
    Given Python source with standard imports
    When the missing import rule runs
    Then the semantic check result passed is true

  Scenario: MissingImportRule flags suspicious private imports
    Given Python source with suspicious private imports
    When the missing import rule runs
    Then the semantic check result passed is false
    And the semantic check result message contains "suspicious import"

  Scenario: MissingImportRule skips on syntax error
    Given Python source code with a syntax error
    When the missing import rule runs
    Then the semantic check result passed is true
    And the semantic check result message contains "Skipped"

  # ── BrokenReferenceRule ────────────────────────────────────────

  Scenario: BrokenReferenceRule passes for well-defined code
    Given Python source with all names defined
    When the broken reference rule runs
    Then the semantic check result passed is true

  Scenario: BrokenReferenceRule flags undefined names
    Given Python source with undefined references
    When the broken reference rule runs
    Then the semantic check result passed is false
    And the semantic check result message contains "broken reference"

  # ── DependencyCycleRule ────────────────────────────────────────

  Scenario: DependencyCycleRule passes for no duplicate relative imports
    Given Python source with unique relative imports
    When the dependency cycle rule runs
    Then the semantic check result passed is true

  Scenario: DependencyCycleRule flags duplicate relative imports
    Given Python source with duplicate relative imports
    When the dependency cycle rule runs
    Then the semantic check result passed is false
    And the semantic check result message contains "Duplicate relative imports"

  # ── APIMisuseRule ──────────────────────────────────────────────

  Scenario: APIMisuseRule passes for clean code
    Given Python source without API misuse
    When the API misuse rule runs
    Then the semantic check result passed is true

  Scenario: APIMisuseRule flags eval usage
    Given Python source using eval
    When the API misuse rule runs
    Then the semantic check result passed is false
    And the semantic check result message contains "API misuse"

  # ── MissingSymbolRule ──────────────────────────────────────────

  Scenario: MissingSymbolRule passes for well-scoped functions
    Given Python source with well-scoped functions
    When the missing symbol rule runs
    Then the semantic check result passed is true

  Scenario: MissingSymbolRule flags undefined symbols in functions
    Given Python source with undefined symbols in functions
    When the missing symbol rule runs
    Then the semantic check result passed is false
    And the semantic check result message contains "missing symbol"

  # ── SemanticRuleRegistry ───────────────────────────────────────

  Scenario: Default registry has all six built-in rules
    Given a default semantic rule registry
    Then the registry has 6 rules
    And the registry contains rule "syntax_error"
    And the registry contains rule "missing_import"
    And the registry contains rule "broken_reference"
    And the registry contains rule "duplicate_import"
    And the registry contains rule "api_misuse"
    And the registry contains rule "missing_symbol"

  Scenario: Registry lookup returns None for unknown rule
    Given a default semantic rule registry
    Then the registry returns None for rule "nonexistent"

  Scenario: Registry remove returns true for existing rule
    Given a default semantic rule registry
    When the rule "syntax_error" is removed from the registry
    Then the registry has 5 rules
    And the registry returns None for rule "syntax_error"

  Scenario: Registry remove returns false for unknown rule
    Given a default semantic rule registry
    Then removing rule "nonexistent" returns false

  # ── SemanticValidationCache ────────────────────────────────────

  Scenario: Cache returns None for uncached entries
    Given a semantic validation cache
    Then the cache returns None for rule "check" and hash "abc123"

  Scenario: Cache stores and retrieves results by rule and hash
    Given a semantic validation cache
    And a cached semantic result for rule "check" and hash "abc123"
    Then the cache returns the result for rule "check" and hash "abc123"

  Scenario: Cache invalidation removes entries
    Given a semantic validation cache
    And a cached semantic result for rule "check" and hash "abc123"
    When the cache entry for rule "check" and hash "abc123" is invalidated
    Then the cache returns None for rule "check" and hash "abc123"

  Scenario: Cache clear removes all entries
    Given a semantic validation cache
    And a cached semantic result for rule "a" and hash "h1"
    And a cached semantic result for rule "b" and hash "h2"
    When the semantic cache is cleared
    Then the semantic cache has 0 entries

  Scenario: Cache compute_hash returns consistent SHA-256
    Then the semantic cache hash for "hello" is consistent

  # ── Severity mapping ───────────────────────────────────────────

  Scenario: ERROR severity maps to REQUIRED mode
    Then semantic severity "error" maps to validation mode "required"

  Scenario: WARN severity maps to INFORMATIONAL mode
    Then semantic severity "warn" maps to validation mode "informational"

  Scenario: INFO severity maps to INFORMATIONAL mode
    Then semantic severity "info" maps to validation mode "informational"

  Scenario: resolve_severity uses default mapping
    Then resolve_severity for "syntax_error" returns "error"
    And resolve_severity for "missing_import" returns "warn"
    And resolve_severity for "duplicate_import" returns "error"
    And resolve_severity for "missing_symbol" returns "info"

  Scenario: resolve_severity uses custom mapping
    Given a custom severity mapping with "syntax_error" as "warn"
    Then resolve_severity with custom mapping for "syntax_error" returns "warn"

  Scenario: resolve_severity falls back to info for unknown rule
    Then resolve_severity for "unknown_rule" returns "info"

  # ── SemanticValidationService ──────────────────────────────────

  Scenario: Service check_file runs all rules on valid Python
    Given a semantic validation service
    And valid Python source code
    When the service checks the file
    Then all semantic check results passed

  Scenario: Service check_file detects syntax error
    Given a semantic validation service
    And Python source code with a syntax error
    When the service checks the file
    Then at least one semantic check result failed
    And a failing result has message containing "Syntax error"

  Scenario: Service check_file with specific rules
    Given a semantic validation service
    And valid Python source code
    When the service checks the file with rules "syntax_error"
    Then only 1 semantic check result is returned

  Scenario: Service returns empty when disabled
    Given a semantic validation service with semantic disabled
    And valid Python source code
    When the service checks the file
    Then 0 semantic check results are returned

  Scenario: Service returns empty for Python when python disabled
    Given a semantic validation service with python disabled
    And valid Python source code
    When the service checks the python file
    Then 0 semantic check results are returned

  Scenario: Service uses cache for repeated checks
    Given a semantic validation service
    And valid Python source code
    When the service checks the file twice
    Then the cache has entries

  # ── Pipeline integration ───────────────────────────────────────

  Scenario: as_pipeline_results returns normalised dicts
    Given a semantic validation service
    And valid Python source code
    When the service returns pipeline results
    Then each pipeline result has keys "passed" and "message" and "data"
    And each pipeline result has key "severity"
    And each pipeline result has key "mode"

  Scenario: normalise_output returns standard schema
    Given a semantic check result with passed true and message "OK"
    When the result is normalised
    Then the normalised output has keys "passed" and "message" and "data"

  # ── Config keys ────────────────────────────────────────────────

  Scenario: Config key constants are defined
    Then config key "validation.semantic.enabled" exists
    And config key "validation.semantic.python.enabled" exists
    And config key "validation.semantic.severity_mapping" exists

  # ── Required vs informational attachment modes ─────────────────

  Scenario: Error severity findings integrate as required in pipeline
    Given a semantic validation service
    And Python source code with a syntax error
    When the service returns pipeline results
    Then a pipeline result with severity "error" has mode "required"

  Scenario: Warn severity findings integrate as informational in pipeline
    Given a semantic validation service
    And Python source using eval
    When the service returns pipeline results
    Then a pipeline result with severity "warn" has mode "informational"

  # ── BrokenReferenceRule scope-awareness ─────────────────────────

  Scenario: BrokenReferenceRule passes for function-local variables
    Given Python source with function-local variables
    When the broken reference rule runs
    Then the semantic check result passed is true

  Scenario: BrokenReferenceRule passes for annotated assignments
    Given Python source with annotated assignment
    When the broken reference rule runs
    Then the semantic check result passed is true

  Scenario: BrokenReferenceRule passes for with-statement variables
    Given Python source with with-statement variable
    When the broken reference rule runs
    Then the semantic check result passed is true

  # ── MissingSymbolRule class/nested function support ────────────

  Scenario: MissingSymbolRule detects undefined symbols in class methods
    Given Python source with a class method using undefined symbol
    When the missing symbol rule runs
    Then the semantic check result passed is false
    And the semantic check result message contains "missing symbol"

  Scenario: MissingSymbolRule passes for nested function calls
    Given Python source with nested functions
    When the missing symbol rule runs
    Then the semantic check result passed is true

  # ── MissingSymbolRule comprehension scope ───────────────────────

  Scenario: MissingSymbolRule does not flag comprehension variables
    Given Python source with comprehension variables
    When the missing symbol rule runs
    Then the semantic check result passed is true

  # ── DuplicateImportRule alias ──────────────────────────────────

  Scenario: DuplicateImportRule passes for no duplicate relative imports
    Given Python source with unique relative imports
    When the duplicate import rule runs
    Then the semantic check result passed is true

  Scenario: DuplicateImportRule flags duplicate relative imports
    Given Python source with duplicate relative imports
    When the duplicate import rule runs
    Then the semantic check result passed is false
    And the semantic check result message contains "Duplicate relative imports"

  # ── APIMisuseRule AST-based detection ──────────────────────────

  Scenario: APIMisuseRule flags exec usage
    Given Python source using exec
    When the API misuse rule runs
    Then the semantic check result passed is false
    And the semantic check result message contains "API misuse"

  Scenario: APIMisuseRule flags os.popen usage
    Given Python source using os.popen
    When the API misuse rule runs
    Then the semantic check result passed is false
    And the semantic check result message contains "API misuse"

  Scenario: APIMisuseRule flags subprocess.run usage
    Given Python source using subprocess.run
    When the API misuse rule runs
    Then the semantic check result passed is false
    And the semantic check result message contains "API misuse"

  Scenario: APIMisuseRule ignores eval in string literals
    Given Python source with eval in a string literal
    When the API misuse rule runs
    Then the semantic check result passed is true

  # ── Non-Python file handling ───────────────────────────────────

  Scenario: Service returns empty for non-Python files
    Given a semantic validation service
    And valid Python source code
    And a non-Python filename
    When the service checks the non-python file
    Then 0 semantic check results are returned

  # ── Empty source ───────────────────────────────────────────────

  Scenario: Service handles empty source code
    Given a semantic validation service
    And empty Python source code
    When the service checks the file
    Then all semantic check results passed

  # ── Cache with max size ────────────────────────────────────────

  Scenario: Cache respects max size with LRU eviction
    Given a semantic validation cache with max size 2
    And a cached semantic result for rule "a" and hash "h1"
    And a cached semantic result for rule "b" and hash "h2"
    And a cached semantic result for rule "c" and hash "h3"
    Then the semantic cache has 2 entries

  # ── Safe initialisation and cleanup ────────────────────────────

  Scenario: Cache rejects invalid max size on construction
    Then creating a semantic validation cache with max size 0 raises ValueError

  Scenario: Cache put updates existing entry without growing
    Given a semantic validation cache
    And a cached semantic result for rule "r1" and hash "h1"
    When the cache entry for rule "r1" and hash "h1" is overwritten with message "updated"
    Then the cache returns the updated result for rule "r1" and hash "h1" with message "updated"
    And the semantic cache has 1 entries

  Scenario: Registry list_rules returns sorted names
    Given a default semantic rule registry
    Then the registry list_rules returns sorted rule names

  Scenario: Service exposes registry property for inspection
    Given a semantic validation service
    Then the service registry property returns a registry with 6 rules

  Scenario: resolve_severity falls back to info for invalid severity value
    Given a custom severity mapping with "syntax_error" as "bogus_value"
    Then resolve_severity with custom mapping for "syntax_error" returns "info"

  Scenario: Service functions with non-dict severity mapping config
    Given a semantic validation service with non-dict severity mapping
    And valid Python source code
    When the service checks the file
    Then all semantic check results passed

  # ── MissingImportRule from-import form ─────────────────────────

  Scenario: MissingImportRule flags suspicious from-import with private module
    Given Python source with from-import of private module
    When the missing import rule runs
    Then the semantic check result passed is false
    And the semantic check result message contains "suspicious import"

  # ── BrokenReferenceRule advanced scope patterns ────────────────

  Scenario: BrokenReferenceRule passes for code with varargs and kwargs
    Given Python source with varargs and kwargs function
    When the broken reference rule runs
    Then the semantic check result passed is true

  Scenario: BrokenReferenceRule passes for from-import bindings
    Given Python source with from-import bindings
    When the broken reference rule runs
    Then the semantic check result passed is true

  Scenario: BrokenReferenceRule passes for except handler variable
    Given Python source with except handler variable
    When the broken reference rule runs
    Then the semantic check result passed is true

  Scenario: BrokenReferenceRule passes for tuple and starred unpacking
    Given Python source with tuple and starred unpacking
    When the broken reference rule runs
    Then the semantic check result passed is true

  Scenario: BrokenReferenceRule passes for with-statement without as clause
    Given Python source with with-statement without as clause
    When the broken reference rule runs
    Then the semantic check result passed is true

  # ── APIMisuseRule safe attribute on flagged module ──────────────

  Scenario: APIMisuseRule passes for safe method call on flagged module
    Given Python source calling a safe method on a flagged module
    When the API misuse rule runs
    Then the semantic check result passed is true

  # ── MissingSymbolRule function-local binding patterns ──────────

  Scenario: MissingSymbolRule passes for function with all parameter kinds
    Given Python source with function using varargs kwonly and kwargs
    When the missing symbol rule runs
    Then the semantic check result passed is true

  Scenario: MissingSymbolRule passes for function with for-loop variable
    Given Python source with function containing for-loop variable
    When the missing symbol rule runs
    Then the semantic check result passed is true

  Scenario: MissingSymbolRule passes for function with with-statement variable
    Given Python source with function containing with-statement variable
    When the missing symbol rule runs
    Then the semantic check result passed is true

  Scenario: MissingSymbolRule passes for function with except handler variable
    Given Python source with function containing except handler variable
    When the missing symbol rule runs
    Then the semantic check result passed is true

  Scenario: MissingSymbolRule passes for function with local import
    Given Python source with function containing local import
    When the missing symbol rule runs
    Then the semantic check result passed is true

  Scenario: MissingSymbolRule passes for function with local from-import
    Given Python source with function containing local from-import
    When the missing symbol rule runs
    Then the semantic check result passed is true