HAL9000
f808abff86
Fix JSON syntax errors in .devcontainer/devcontainer.json (removed
invalid JS-style // comments) and .devcontainer/opencode.json (removed
90+ trailing commas). Apply auto-fixes for end-of-file and trailing
whitespace issues across 100+ files. Fix SIM105 ruff violations in
benchmarks/core_circuit_breaker_bench.py (use contextlib.suppress).
Note: The security fix from issue #7478 (validate_path startswith bypass)
was already delivered to master in commit e18ac5f2. This PR as currently
structured is non-atomic (35 commits across 10+ issues) and needs
significant restructure before merge. This commit only addresses the
CI/pre-commit failures.
ISSUES CLOSED: #7478
959 lines
42 KiB
Gherkin
959 lines
42 KiB
Gherkin
Feature: Consolidated Quality Review
|
|
Combined scenarios from: adr_compliance_checker_coverage, definition_of_done, diff_review, quality_gates_checker_coverage, review_playbook
|
|
|
|
# ============================================================
|
|
# Originally from: adr_compliance_checker_coverage.feature
|
|
# Feature: ADR compliance checker script coverage
|
|
# ============================================================
|
|
|
|
Scenario: ADR-002 checker returns empty list when application directory does not exist
|
|
Given a temporary source directory with no application subdirectory
|
|
When the ADR-002 async model checker runs against that source directory
|
|
Then the ADR-002 checker should return an empty violations list
|
|
|
|
|
|
Scenario: ADR-002 checker returns empty list when no threading imports found
|
|
Given a temporary source directory with a clean application module
|
|
When the ADR-002 async model checker runs against that source directory
|
|
Then the ADR-002 checker should return an empty violations list
|
|
|
|
|
|
Scenario: ADR-002 checker detects import threading violation
|
|
Given a temporary source directory with an application module importing threading directly
|
|
When the ADR-002 async model checker runs against that source directory
|
|
Then the ADR-002 checker should return a violation mentioning "imports threading directly"
|
|
|
|
|
|
Scenario: ADR-002 checker detects from threading import violation
|
|
Given a temporary source directory with an application module using from-threading-import
|
|
When the ADR-002 async model checker runs against that source directory
|
|
Then the ADR-002 checker should return a violation mentioning "imports threading directly"
|
|
|
|
# --- ADR-003: Dependency Injection Framework ---
|
|
|
|
|
|
Scenario: ADR-003 checker returns empty list when services directory does not exist
|
|
Given a temporary source directory with no services subdirectory
|
|
When the ADR-003 dependency injection checker runs against that source directory
|
|
Then the ADR-003 checker should return an empty violations list
|
|
|
|
|
|
Scenario: ADR-003 checker skips dunder init files
|
|
Given a temporary source directory with only an __init__.py in services
|
|
When the ADR-003 dependency injection checker runs against that source directory
|
|
Then the ADR-003 checker should return an empty violations list
|
|
|
|
|
|
Scenario: ADR-003 checker skips files with syntax errors
|
|
Given a temporary source directory with a syntactically invalid Python file in services
|
|
When the ADR-003 dependency injection checker runs against that source directory
|
|
Then the ADR-003 checker should return an empty violations list
|
|
|
|
|
|
Scenario: ADR-003 checker reports class with empty init having no injected dependencies
|
|
Given a temporary source directory with a service class whose init takes only self
|
|
When the ADR-003 dependency injection checker runs against that source directory
|
|
Then the ADR-003 checker should return a violation mentioning "no injected dependencies"
|
|
|
|
|
|
Scenario: ADR-003 checker passes class with proper dependency injection
|
|
Given a temporary source directory with a service class whose init takes self and a dependency
|
|
When the ADR-003 dependency injection checker runs against that source directory
|
|
Then the ADR-003 checker should return an empty violations list
|
|
|
|
|
|
Scenario: ADR-003 checker ignores classes without init method
|
|
Given a temporary source directory with a service class that has no init method
|
|
When the ADR-003 dependency injection checker runs against that source directory
|
|
Then the ADR-003 checker should return an empty violations list
|
|
|
|
# --- ADR-007: Repository Pattern for Persistence ---
|
|
|
|
|
|
Scenario: ADR-007 checker returns empty list when services directory does not exist
|
|
Given a temporary source directory with no services subdirectory for ADR-007
|
|
When the ADR-007 repository pattern checker runs against that source directory
|
|
Then the ADR-007 checker should return an empty violations list
|
|
|
|
|
|
Scenario: ADR-007 checker detects direct SQLAlchemy imports in services
|
|
Given a temporary source directory with a service file importing from sqlalchemy
|
|
When the ADR-007 repository pattern checker runs against that source directory
|
|
Then the ADR-007 checker should return a violation mentioning "SQLAlchemy directly"
|
|
|
|
|
|
Scenario: ADR-007 checker detects direct session query usage in services
|
|
Given a temporary source directory with a service file using session.query
|
|
When the ADR-007 repository pattern checker runs against that source directory
|
|
Then the ADR-007 checker should return a violation mentioning "session directly"
|
|
|
|
|
|
Scenario: ADR-007 checker detects direct session execute usage in services
|
|
Given a temporary source directory with a service file using session.execute
|
|
When the ADR-007 repository pattern checker runs against that source directory
|
|
Then the ADR-007 checker should return a violation mentioning "session directly"
|
|
|
|
|
|
Scenario: ADR-007 checker passes clean services with no SQLAlchemy usage
|
|
Given a temporary source directory with a clean service file using no SQLAlchemy
|
|
When the ADR-007 repository pattern checker runs against that source directory
|
|
Then the ADR-007 checker should return an empty violations list
|
|
|
|
# --- main() function ---
|
|
|
|
|
|
Scenario: ADR compliance main returns error when source directory does not exist
|
|
When the ADR compliance main function runs with a nonexistent source directory
|
|
Then the ADR compliance main should return exit code 1
|
|
And the ADR compliance main output should contain "not found"
|
|
|
|
|
|
Scenario: ADR compliance main returns success when all checks pass
|
|
Given a temporary project tree with clean source code for ADR compliance
|
|
When the ADR compliance main function runs against that project tree
|
|
Then the ADR compliance main should return exit code 0
|
|
And the ADR compliance main output should contain "All ADR compliance checks passed"
|
|
|
|
|
|
Scenario: ADR compliance main returns failure when violations are found
|
|
Given a temporary project tree with ADR violations in the source code
|
|
When the ADR compliance main function runs against that project tree
|
|
Then the ADR compliance main should return exit code 1
|
|
And the ADR compliance main output should contain "ADR compliance issue"
|
|
|
|
|
|
# ============================================================
|
|
# Originally from: definition_of_done.feature
|
|
# Feature: Definition-of-Done gating
|
|
# ============================================================
|
|
|
|
Scenario: Parse DoD text into criteria
|
|
Given a definition of done test environment
|
|
When I parse the DoD text "All tests pass\nNo lint errors\nCoverage above 90%"
|
|
Then there should be 3 criteria parsed
|
|
And criterion 0 text should be "All tests pass"
|
|
And criterion 1 text should be "No lint errors"
|
|
|
|
|
|
Scenario: Parse DoD with bullet points
|
|
Given a definition of done test environment
|
|
When I parse the DoD text "- Tests pass\n- No errors\n* Coverage ok"
|
|
Then there should be 3 criteria parsed
|
|
And criterion 0 text should be "Tests pass"
|
|
|
|
|
|
Scenario: Parse DoD with numbered items
|
|
Given a definition of done test environment
|
|
When I parse the DoD text "1. Tests pass\n2. No errors"
|
|
Then there should be 2 criteria parsed
|
|
And criterion 0 text should be "Tests pass"
|
|
And criterion 1 text should be "No errors"
|
|
|
|
|
|
Scenario: Parse empty DoD returns no criteria
|
|
Given a definition of done test environment
|
|
When I parse an empty DoD text
|
|
Then there should be 0 criteria parsed
|
|
|
|
# -- Template rendering --------------------------------------------------
|
|
|
|
|
|
Scenario: Render DoD template with plan arguments
|
|
Given a definition of done test environment
|
|
When I render the DoD template "Tests for {module} must pass" with argument "module" as "auth"
|
|
Then the rendered DoD should be "Tests for auth must pass"
|
|
|
|
|
|
Scenario: Render DoD template preserves unmatched placeholders
|
|
Given a definition of done test environment
|
|
When I render the DoD template "Deploy {service} to {env}" with argument "service" as "api"
|
|
Then the rendered DoD should contain "{env}"
|
|
|
|
# -- Text match evaluation -----------------------------------------------
|
|
|
|
|
|
Scenario: TextMatchEvaluator passes when context contains criterion text
|
|
Given a definition of done test environment
|
|
Given DoD criteria from "All tests pass"
|
|
And evaluation context with key "status" value "all tests pass"
|
|
When I evaluate the criteria with TextMatchEvaluator
|
|
Then all criteria should have passed
|
|
|
|
|
|
Scenario: TextMatchEvaluator fails when context lacks criterion text
|
|
Given a definition of done test environment
|
|
Given DoD criteria from "All tests pass"
|
|
And evaluation context with key "status" value "3 tests failed"
|
|
When I evaluate the criteria with TextMatchEvaluator
|
|
Then criterion 0 should have status "failed"
|
|
|
|
|
|
Scenario: TextMatchEvaluator with regex pattern passes on match
|
|
Given a definition of done test environment
|
|
Given a DoD criterion with text "Coverage check" and pattern "coverage.*9[0-9]"
|
|
And evaluation context with key "coverage" value "coverage=95%"
|
|
When I evaluate the criteria with TextMatchEvaluator
|
|
Then all criteria should have passed
|
|
|
|
|
|
Scenario: TextMatchEvaluator with regex pattern fails on no match
|
|
Given a definition of done test environment
|
|
Given a DoD criterion with text "Coverage check" and pattern "coverage.*9[0-9]"
|
|
And evaluation context with key "coverage" value "coverage=80%"
|
|
When I evaluate the criteria with TextMatchEvaluator
|
|
Then criterion 0 should have status "failed"
|
|
|
|
|
|
Scenario: TextMatchEvaluator skips when context is empty
|
|
Given a definition of done test environment
|
|
Given DoD criteria from "All tests pass"
|
|
And an empty evaluation context
|
|
When I evaluate the criteria with TextMatchEvaluator
|
|
Then criterion 0 should have status "skipped"
|
|
|
|
|
|
Scenario: TextMatchEvaluator handles invalid regex gracefully
|
|
Given a definition of done test environment
|
|
Given a DoD criterion with text "Bad regex" and pattern "[invalid"
|
|
And evaluation context with key "data" value "anything"
|
|
When I evaluate the criteria with TextMatchEvaluator
|
|
Then criterion 0 should have status "failed"
|
|
And criterion 0 reasoning should contain "Invalid regex"
|
|
|
|
# -- DoDSummary model ----------------------------------------------------
|
|
|
|
|
|
Scenario: DoDSummary reports all_passed when no failures
|
|
Given a definition of done test environment
|
|
Given a DoDSummary with 2 passed and 0 failed
|
|
Then the summary should report all passed
|
|
And the DoD summary total should be 2
|
|
|
|
|
|
Scenario: DoDSummary reports not all passed when failures exist
|
|
Given a definition of done test environment
|
|
Given a DoDSummary with 1 passed and 1 failed
|
|
Then the summary should not report all passed
|
|
And the summary failed count should be 1
|
|
|
|
|
|
Scenario: DoDSummary empty check
|
|
Given a definition of done test environment
|
|
Given an empty DoDSummary
|
|
Then the summary should be empty
|
|
And the summary should not report all passed
|
|
|
|
|
|
Scenario: DoDSummary to_cli_dict has expected fields
|
|
Given a definition of done test environment
|
|
Given a DoDSummary with 1 passed and 0 failed
|
|
Then the summary CLI dict should have key "total"
|
|
And the summary CLI dict should have key "all_passed"
|
|
And the summary CLI dict should have key "criteria"
|
|
|
|
|
|
Scenario: DoDSummary to_validation_summary is compatible
|
|
Given a definition of done test environment
|
|
Given a DoDSummary with 2 passed and 1 failed
|
|
Then the validation summary should have required_passed 2
|
|
And the validation summary should have required_failed 1
|
|
And the validation summary should have dod_evaluated true
|
|
|
|
# -- Evaluation context matching -----------------------------------------
|
|
|
|
|
|
Scenario: TextMatchEvaluator matches criterion text in context keys
|
|
Given a definition of done test environment
|
|
Given DoD criteria from "lint_errors"
|
|
And evaluation context with key "lint_errors" value "0"
|
|
When I evaluate the criteria with TextMatchEvaluator
|
|
Then all criteria should have passed
|
|
|
|
|
|
Scenario: Multiple criteria with mixed results
|
|
Given a definition of done test environment
|
|
Given DoD criteria from "tests pass\nlint errors"
|
|
And evaluation context with key "result" value "tests pass, no issues"
|
|
When I evaluate the criteria with TextMatchEvaluator
|
|
Then criterion 0 should have status "passed"
|
|
And criterion 1 should have status "failed"
|
|
|
|
# -- Coverage: edge cases ------------------------------------------------
|
|
|
|
|
|
Scenario: Parse DoD with blank lines between criteria
|
|
Given a definition of done test environment
|
|
When I parse the DoD text "Tests pass\n\nNo errors"
|
|
Then there should be 2 criteria parsed
|
|
|
|
|
|
Scenario: Parse DoD with whitespace only line
|
|
Given a definition of done test environment
|
|
When I parse the DoD text " \n- Tests pass"
|
|
Then there should be 1 criteria parsed
|
|
And criterion 0 text should be "Tests pass"
|
|
|
|
|
|
Scenario: TextMatchEvaluator matches via individual key lookup
|
|
Given a definition of done test environment
|
|
Given DoD criteria from "all_tests_passed"
|
|
And evaluation context with key "all_tests_passed" value "true"
|
|
When I evaluate the criteria with TextMatchEvaluator
|
|
Then all criteria should have passed
|
|
|
|
# -- T1: all-SKIPPED summary gate behavior --------------------------------
|
|
|
|
|
|
Scenario: DoDSummary all_passed is false when all criteria skipped
|
|
Given a definition of done test environment
|
|
Given a DoDSummary with all criteria skipped
|
|
Then the summary should not report all passed
|
|
And the summary skipped count should be 2
|
|
|
|
# -- T2: bullet prefix yielding empty text --------------------------------
|
|
|
|
|
|
Scenario: Parse DoD with bullet prefix but no text
|
|
Given a definition of done test environment
|
|
When I parse the DoD text "- \n- Tests pass"
|
|
Then there should be 1 criteria parsed
|
|
And criterion 0 text should be "Tests pass"
|
|
|
|
# -- T5: multi-argument template rendering --------------------------------
|
|
|
|
|
|
Scenario: Render DoD template with multiple arguments
|
|
Given a definition of done test environment
|
|
When I render the DoD template "{svc} in {env} OK" with arguments "svc"="api" and "env"="prod"
|
|
Then the rendered DoD should be "api in prod OK"
|
|
|
|
|
|
Scenario: Render DoD template prevents injection via values
|
|
Given a definition of done test environment
|
|
When I render the DoD template "Test {name}" with argument "name" as "{secret}"
|
|
Then the rendered DoD should be "Test {secret}"
|
|
|
|
# -- S3: length limit on template -----------------------------------------
|
|
|
|
|
|
Scenario: Render DoD rejects oversized template
|
|
Given a definition of done test environment
|
|
Given a DoD template that is 11000 characters long
|
|
When I render the oversized DoD template with argument "x" as "y"
|
|
Then the DoD render should fail with "exceeds maximum"
|
|
|
|
|
|
Scenario: Render DoD rejects oversized output
|
|
Given a definition of done test environment
|
|
Given a short DoD template with a huge argument value
|
|
When I render the DoD template that produces oversized output
|
|
Then the DoD render should fail with "exceeds maximum"
|
|
|
|
# -- S2: ReDoS protection -------------------------------------------------
|
|
|
|
|
|
Scenario: TextMatchEvaluator rejects oversized regex pattern
|
|
Given a definition of done test environment
|
|
Given a DoD criterion with text "long regex" and pattern that is 600 chars
|
|
And evaluation context with key "data" value "anything"
|
|
When I evaluate the criteria with TextMatchEvaluator
|
|
Then criterion 0 should have status "failed"
|
|
And criterion 0 reasoning should contain "exceeds maximum"
|
|
|
|
|
|
# ============================================================
|
|
# Originally from: diff_review.feature
|
|
# Feature: Diff review artifacts
|
|
# ============================================================
|
|
|
|
Scenario: Empty changeset produces empty review artifact
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
When I build a review artifact from the changeset
|
|
Then the review artifact should have 0 total changes
|
|
And the review artifact should have 0 groups
|
|
And the review artifact should not be truncated
|
|
|
|
|
|
Scenario: Single CREATE change produces a diff entry
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a change entry with operation "create" for path "src/main.py" in resource "RES001"
|
|
And after content for "src/main.py" is "print('hello')\n"
|
|
When I build a review artifact from the changeset
|
|
Then the review artifact should have 1 total changes
|
|
And the review artifact should have 1 groups
|
|
And group 0 should have 1 entries
|
|
And entry 0 in group 0 should have change type "create"
|
|
And entry 0 in group 0 should have path "src/main.py"
|
|
|
|
|
|
Scenario: MODIFY change generates unified diff
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a change entry with operation "modify" for path "src/app.py" in resource "RES001"
|
|
And before content for "src/app.py" is "x = 1\ny = 2\n"
|
|
And after content for "src/app.py" is "x = 1\ny = 3\n"
|
|
When I build a review artifact from the changeset
|
|
Then entry 0 in group 0 should have change type "modify"
|
|
And the diff text in entry 0 of group 0 should contain "---"
|
|
And the diff text in entry 0 of group 0 should contain "+++"
|
|
|
|
|
|
Scenario: DELETE change entry
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a change entry with operation "delete" for path "old.txt" in resource "RES001"
|
|
And before content for "old.txt" is "old content\n"
|
|
When I build a review artifact from the changeset
|
|
Then entry 0 in group 0 should have change type "delete"
|
|
And the diff text in entry 0 of group 0 should contain "---"
|
|
|
|
|
|
Scenario: Multiple changes across different resources
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a change entry with operation "create" for path "a.py" in resource "RES001"
|
|
And a change entry with operation "modify" for path "b.py" in resource "RES002"
|
|
And after content for "a.py" is "new\n"
|
|
And before content for "b.py" is "old\n"
|
|
And after content for "b.py" is "new\n"
|
|
When I build a review artifact from the changeset
|
|
Then the review artifact should have 2 total changes
|
|
And the review artifact should have 2 groups
|
|
|
|
|
|
Scenario: Changes grouped by resource
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a change entry with operation "create" for path "a.py" in resource "RES001"
|
|
And a change entry with operation "modify" for path "b.py" in resource "RES001"
|
|
And after content for "a.py" is "new\n"
|
|
And before content for "b.py" is "old\n"
|
|
And after content for "b.py" is "new\n"
|
|
When I build a review artifact from the changeset
|
|
Then the review artifact should have 1 groups
|
|
And group 0 should have 2 entries
|
|
|
|
|
|
Scenario: Binary file detection
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a change entry with operation "create" for path "image.bin" in resource "RES001"
|
|
And after content for "image.bin" is "binary\x00data"
|
|
When I build a review artifact from the changeset
|
|
Then entry 0 in group 0 should be binary
|
|
And the diff text in entry 0 of group 0 should contain "[binary file]"
|
|
|
|
|
|
Scenario: Rename detection with matching content hash
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a delete entry for path "old/file.py" in resource "RES001" with before_hash "abc123"
|
|
And a create entry for path "new/file.py" in resource "RES001" with after_hash "abc123"
|
|
When I build a review artifact from the changeset
|
|
Then the review artifact should have 1 groups
|
|
And group 0 should have 1 entries
|
|
And entry 0 in group 0 should be a rename
|
|
And entry 0 in group 0 rename should be from "old/file.py" to "new/file.py"
|
|
And group 0 should have 1 renamed
|
|
|
|
|
|
Scenario: Rename not detected when hashes differ
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a delete entry for path "old/file.py" in resource "RES001" with before_hash "abc123"
|
|
And a create entry for path "new/file.py" in resource "RES001" with after_hash "xyz789"
|
|
When I build a review artifact from the changeset
|
|
Then group 0 should have 2 entries
|
|
And entry 0 in group 0 should not be a rename
|
|
|
|
|
|
Scenario: Max diff size truncation
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a change entry with operation "create" for path "big.py" in resource "RES001"
|
|
And after content for "big.py" is a string of 100000 characters
|
|
When I build a review artifact with max diff size 100
|
|
Then the review artifact should be truncated
|
|
And entry 0 in group 0 should be truncated
|
|
|
|
|
|
Scenario: Stable ordering by resource then path
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a change entry with operation "create" for path "z.py" in resource "RES002"
|
|
And a change entry with operation "create" for path "a.py" in resource "RES001"
|
|
And after content for "z.py" is "z\n"
|
|
And after content for "a.py" is "a\n"
|
|
When I build a review artifact from the changeset
|
|
Then group 0 should have resource id "RES001"
|
|
And group 1 should have resource id "RES002"
|
|
|
|
|
|
Scenario: Before and after content hashes in metadata
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a hashed change entry "modify" path "f.py" resource "RES001" before_hash "hash1" after_hash "hash2"
|
|
And before content for "f.py" is "old\n"
|
|
And after content for "f.py" is "new\n"
|
|
When I build a review artifact from the changeset
|
|
Then entry 0 in group 0 should have before hash "hash1"
|
|
And entry 0 in group 0 should have after hash "hash2"
|
|
|
|
|
|
Scenario: File mode in metadata
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a moded change entry "modify" path "script.sh" resource "RES001" after_mode 0o755
|
|
And before content for "script.sh" is "#!/bin/sh\n"
|
|
And after content for "script.sh" is "#!/bin/bash\n"
|
|
When I build a review artifact from the changeset
|
|
Then entry 0 in group 0 should have file mode 493
|
|
|
|
|
|
Scenario: Serialize to plain text
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a change entry with operation "create" for path "new.py" in resource "RES001"
|
|
And after content for "new.py" is "hello\n"
|
|
And resource name "RES001" is "my-resource"
|
|
When I build a review artifact from the changeset
|
|
And I serialize the artifact to plain text
|
|
Then the plain text should contain "my-resource"
|
|
And the plain text should contain "new.py"
|
|
And the plain text should contain "Total changes: 1"
|
|
|
|
|
|
Scenario: Serialize to JSON
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a change entry with operation "create" for path "new.py" in resource "RES001"
|
|
And after content for "new.py" is "hello\n"
|
|
When I build a review artifact from the changeset
|
|
And I serialize the artifact to JSON
|
|
Then the JSON output should be valid JSON
|
|
And the JSON output should contain key "total_changes"
|
|
And the JSON output should contain key "groups"
|
|
|
|
|
|
Scenario: Serialize to rich format
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a change entry with operation "create" for path "new.py" in resource "RES001"
|
|
And after content for "new.py" is "hello\n"
|
|
When I build a review artifact from the changeset
|
|
And I serialize the artifact to rich format
|
|
Then the rich text should contain "[bold]"
|
|
And the rich text should contain "new.py"
|
|
|
|
|
|
Scenario: ResourceDiffGroup summary counts
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a change entry with operation "create" for path "a.py" in resource "RES001"
|
|
And a change entry with operation "modify" for path "b.py" in resource "RES001"
|
|
And a change entry with operation "delete" for path "c.py" in resource "RES001"
|
|
And after content for "a.py" is "new\n"
|
|
And before content for "b.py" is "old\n"
|
|
And after content for "b.py" is "new\n"
|
|
And before content for "c.py" is "old\n"
|
|
When I build a review artifact from the changeset
|
|
Then group 0 should have 1 added
|
|
And group 0 should have 1 modified
|
|
And group 0 should have 1 deleted
|
|
|
|
|
|
Scenario: ReviewArtifact has plan and changeset IDs
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
When I build a review artifact from the changeset
|
|
Then the review artifact plan ID should match the changeset plan ID
|
|
And the review artifact changeset ID should match the changeset ID
|
|
|
|
|
|
Scenario: Empty content produces empty diff
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a change entry with operation "modify" for path "empty.py" in resource "RES001"
|
|
When I build a review artifact from the changeset
|
|
Then the diff text in entry 0 of group 0 should be empty
|
|
|
|
|
|
Scenario: Multiple entries in same resource sorted by path
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a change entry with operation "create" for path "z.py" in resource "RES001"
|
|
And a change entry with operation "create" for path "a.py" in resource "RES001"
|
|
And after content for "z.py" is "z\n"
|
|
And after content for "a.py" is "a\n"
|
|
When I build a review artifact from the changeset
|
|
Then entry 0 in group 0 should have path "a.py"
|
|
And entry 1 in group 0 should have path "z.py"
|
|
|
|
|
|
Scenario: Second entry fully truncated when budget exhausted
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a change entry with operation "create" for path "a.py" in resource "RES001"
|
|
And a change entry with operation "create" for path "b.py" in resource "RES001"
|
|
And after content for "a.py" is a string of 1000 characters
|
|
And after content for "b.py" is a string of 1000 characters
|
|
When I build a review artifact with max diff size 50
|
|
Then the review artifact should be truncated
|
|
And entry 1 in group 0 should be truncated
|
|
And the diff text in entry 1 of group 0 should contain "[truncated]"
|
|
|
|
|
|
Scenario: JSON serialization includes hashes and file mode
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a hashed change entry "modify" path "f.py" resource "RES001" before_hash "aaa" after_hash "bbb"
|
|
And a moded change entry "modify" path "g.py" resource "RES001" after_mode 0o644
|
|
And before content for "f.py" is "old\n"
|
|
And after content for "f.py" is "new\n"
|
|
And before content for "g.py" is "a\n"
|
|
And after content for "g.py" is "b\n"
|
|
When I build a review artifact from the changeset
|
|
And I serialize the artifact to JSON
|
|
Then the JSON output should be valid JSON
|
|
And the JSON output should have entry with before hash "aaa"
|
|
And the JSON output should have entry with after hash "bbb"
|
|
And the JSON output should have entry with file mode
|
|
|
|
|
|
Scenario: JSON serialization includes rename fields
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a delete entry for path "old/r.py" in resource "RES001" with before_hash "h1"
|
|
And a create entry for path "new/r.py" in resource "RES001" with after_hash "h1"
|
|
When I build a review artifact from the changeset
|
|
And I serialize the artifact to JSON
|
|
Then the JSON output should be valid JSON
|
|
And the JSON output should have entry with rename from "old/r.py"
|
|
|
|
|
|
Scenario: Rich format shows truncation warning
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a change entry with operation "create" for path "big.py" in resource "RES001"
|
|
And after content for "big.py" is a string of 100000 characters
|
|
When I build a review artifact with max diff size 100
|
|
And I serialize the artifact to rich format
|
|
Then the rich text should contain "[yellow]"
|
|
And the rich text should contain "truncated"
|
|
|
|
|
|
Scenario: Rich format shows binary marker
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a change entry with operation "create" for path "bin.dat" in resource "RES001"
|
|
And after content for "bin.dat" is "binary\x00data"
|
|
When I build a review artifact from the changeset
|
|
And I serialize the artifact to rich format
|
|
Then the rich text should contain "[binary]"
|
|
|
|
|
|
Scenario: Rich format colorizes diff lines
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a change entry with operation "modify" for path "c.py" in resource "RES001"
|
|
And before content for "c.py" is "old\n"
|
|
And after content for "c.py" is "new\n"
|
|
When I build a review artifact from the changeset
|
|
And I serialize the artifact to rich format
|
|
Then the rich text should contain "[green]"
|
|
And the rich text should contain "[red]"
|
|
|
|
|
|
Scenario: Cross-resource delete and create are not renamed
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a delete entry for path "old/x.py" in resource "RES001" with before_hash "same"
|
|
And a create entry for path "new/x.py" in resource "RES002" with after_hash "same"
|
|
When I build a review artifact from the changeset
|
|
Then the review artifact should have 2 groups
|
|
And entry 0 in group 0 should not be a rename
|
|
And entry 0 in group 1 should not be a rename
|
|
|
|
|
|
Scenario: Multiple renames consume creates only once
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a delete entry for path "old/a.py" in resource "RES001" with before_hash "h1"
|
|
And a delete entry for path "old/b.py" in resource "RES001" with before_hash "h1"
|
|
And a create entry for path "new/a.py" in resource "RES001" with after_hash "h1"
|
|
When I build a review artifact from the changeset
|
|
Then group 0 should have 2 entries
|
|
|
|
|
|
Scenario: Rename entry truncated when budget exhausted
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a change entry with operation "create" for path "a.py" in resource "RES001"
|
|
And after content for "a.py" is a string of 1000 characters
|
|
And a delete entry for path "old/r.py" in resource "RES001" with before_hash "rh1"
|
|
And a create entry for path "new/r.py" in resource "RES001" with after_hash "rh1"
|
|
When I build a review artifact with max diff size 50
|
|
Then the review artifact should be truncated
|
|
And entry 1 in group 0 should be a rename
|
|
And entry 1 in group 0 should be truncated
|
|
And the diff text in entry 1 of group 0 should contain "[truncated]"
|
|
|
|
|
|
Scenario: Plain serialization shows binary marker
|
|
Given a plan ID "01PLAN000000000000000001"
|
|
And an empty changeset for the plan
|
|
Given a change entry with operation "create" for path "img.png" in resource "RES001"
|
|
And after content for "img.png" is "png\x00header"
|
|
When I build a review artifact from the changeset
|
|
And I serialize the artifact to plain text
|
|
Then the plain text should contain "[binary file]"
|
|
|
|
|
|
# ============================================================
|
|
# Originally from: quality_gates_checker_coverage.feature
|
|
# Feature: Quality gates checker script coverage
|
|
# ============================================================
|
|
|
|
Scenario: Coverage gate passes when coverage exceeds minimum threshold
|
|
Given the coverage tool reports a total of 92.5 percent
|
|
When the quality gates check_coverage function is called with minimum 85
|
|
Then the quality gates coverage result should be passed
|
|
And the quality gates coverage message should contain "92.5%"
|
|
|
|
|
|
Scenario: Coverage gate fails when coverage is below minimum threshold
|
|
Given the coverage tool reports a total of 70.0 percent
|
|
When the quality gates check_coverage function is called with minimum 85
|
|
Then the quality gates coverage result should be failed
|
|
And the quality gates coverage message should contain "70.0%"
|
|
|
|
|
|
Scenario: Coverage gate fails when coverage command returns non-zero exit code
|
|
Given the coverage tool returns a non-zero exit code with stderr "No data to report"
|
|
When the quality gates check_coverage function is called with minimum 85
|
|
Then the quality gates coverage result should be failed
|
|
And the quality gates coverage message should contain "Coverage command failed"
|
|
|
|
|
|
Scenario: Coverage gate fails when coverage output cannot be parsed as a number
|
|
Given the coverage tool returns unparseable output "not-a-number"
|
|
When the quality gates check_coverage function is called with minimum 85
|
|
Then the quality gates coverage result should be failed
|
|
And the quality gates coverage message should contain "Could not parse coverage output"
|
|
|
|
# --- check_typecheck ---
|
|
|
|
|
|
Scenario: Typecheck gate passes when pyright reports zero errors
|
|
Given pyright reports JSON output with zero errors
|
|
When the quality gates check_typecheck function is called
|
|
Then the quality gates typecheck result should be passed
|
|
And the quality gates typecheck message should contain "0 errors"
|
|
|
|
|
|
Scenario: Typecheck gate fails when pyright reports errors
|
|
Given pyright reports JSON output with 5 errors
|
|
When the quality gates check_typecheck function is called
|
|
Then the quality gates typecheck result should be failed
|
|
And the quality gates typecheck message should contain "5 errors"
|
|
|
|
|
|
Scenario: Typecheck gate falls back to exit code when JSON parsing fails and passes
|
|
Given pyright returns invalid JSON output with exit code 0
|
|
When the quality gates check_typecheck function is called
|
|
Then the quality gates typecheck result should be passed
|
|
And the quality gates typecheck message should contain "passed"
|
|
|
|
|
|
Scenario: Typecheck gate falls back to exit code when JSON parsing fails and fails
|
|
Given pyright returns invalid JSON output with exit code 1
|
|
When the quality gates check_typecheck function is called
|
|
Then the quality gates typecheck result should be failed
|
|
And the quality gates typecheck message should contain "failed (exit code 1)"
|
|
|
|
# --- check_security ---
|
|
|
|
|
|
Scenario: Security gate passes when bandit finds no high-severity issues
|
|
Given bandit reports JSON output with zero results
|
|
When the quality gates check_security function is called
|
|
Then the quality gates security result should be passed
|
|
And the quality gates security message should contain "0 high-severity issues"
|
|
|
|
|
|
Scenario: Security gate fails when bandit finds high-severity issues
|
|
Given bandit reports JSON output with 3 high-severity results
|
|
When the quality gates check_security function is called
|
|
Then the quality gates security result should be failed
|
|
And the quality gates security message should contain "3 high-severity issues"
|
|
|
|
|
|
Scenario: Security gate falls back to exit code on JSON parse error and passes
|
|
Given bandit returns invalid JSON output with exit code 0
|
|
When the quality gates check_security function is called
|
|
Then the quality gates security result should be passed
|
|
And the quality gates security message should contain "passed"
|
|
|
|
|
|
Scenario: Security gate falls back to exit code on JSON parse error and fails
|
|
Given bandit returns invalid JSON output with exit code 2
|
|
When the quality gates check_security function is called
|
|
Then the quality gates security result should be failed
|
|
And the quality gates security message should contain "failed (exit code 2)"
|
|
|
|
# --- check_dead_code ---
|
|
|
|
|
|
Scenario: Dead code gate passes when vulture finds no dead code
|
|
Given vulture returns exit code 0 with no output
|
|
When the quality gates check_dead_code function is called
|
|
Then the quality gates dead code result should be passed
|
|
And the quality gates dead code message should contain "none detected"
|
|
|
|
|
|
Scenario: Dead code gate fails when vulture detects dead code issues
|
|
Given vulture returns exit code 1 with 3 lines of dead code output
|
|
When the quality gates check_dead_code function is called
|
|
Then the quality gates dead code result should be failed
|
|
And the quality gates dead code message should contain "3 issue(s) found"
|
|
|
|
# --- check_complexity ---
|
|
|
|
|
|
Scenario: Complexity gate passes when no blocks exceed maximum grade
|
|
Given radon reports JSON output with zero complex blocks
|
|
When the quality gates check_complexity function is called with max grade "F"
|
|
Then the quality gates complexity result should be passed
|
|
And the quality gates complexity message should contain "no blocks with grade >= F"
|
|
|
|
|
|
Scenario: Complexity gate fails when blocks exceed maximum grade
|
|
Given radon reports JSON output with 2 complex blocks
|
|
When the quality gates check_complexity function is called with max grade "F"
|
|
Then the quality gates complexity result should be failed
|
|
And the quality gates complexity message should contain "2 block(s) with grade >= F"
|
|
|
|
|
|
Scenario: Complexity gate passes on JSON parse error as fallback
|
|
Given radon returns invalid JSON output
|
|
When the quality gates check_complexity function is called with max grade "F"
|
|
Then the quality gates complexity result should be passed
|
|
And the quality gates complexity message should contain "passed (no extreme complexity)"
|
|
|
|
# --- main function ---
|
|
|
|
|
|
Scenario: Quality gates main reports all gates passed when everything succeeds
|
|
Given all quality gate functions are mocked to return success
|
|
When the quality gates main function is invoked with default arguments
|
|
Then the quality gates main should return exit code 0
|
|
|
|
|
|
Scenario: Quality gates main reports failure when any gate fails
|
|
Given the quality gate coverage function is mocked to return failure
|
|
When the quality gates main function is invoked with default arguments
|
|
Then the quality gates main should return exit code 1
|
|
|
|
|
|
Scenario: Quality gates main creates build directory if it does not exist
|
|
Given a temporary working directory with no build subdirectory
|
|
When the quality gates main function is invoked from that temporary directory
|
|
Then a build directory should exist in the temporary working directory
|
|
|
|
|
|
# ============================================================
|
|
# Originally from: review_playbook.feature
|
|
# Feature: Review playbook documentation validation
|
|
# ============================================================
|
|
|
|
Scenario: Review playbook file exists
|
|
Given the review playbook file at "docs/development/review_playbook.md"
|
|
Then the review playbook file should exist
|
|
|
|
|
|
Scenario: Review playbook contains priority matrix
|
|
Given the review playbook file at "docs/development/review_playbook.md"
|
|
When I read the review playbook content
|
|
Then the playbook should contain the section "Priority Matrix"
|
|
|
|
|
|
Scenario: Review playbook contains routing table
|
|
Given the review playbook file at "docs/development/review_playbook.md"
|
|
When I read the review playbook content
|
|
Then the playbook should contain the section "PR Review Routing Table"
|
|
|
|
|
|
Scenario: Review playbook contains focus areas
|
|
Given the review playbook file at "docs/development/review_playbook.md"
|
|
When I read the review playbook content
|
|
Then the playbook should contain the section "Focus Areas and Skip Rules"
|
|
|
|
|
|
Scenario: Review playbook contains review SLA guidance
|
|
Given the review playbook file at "docs/development/review_playbook.md"
|
|
When I read the review playbook content
|
|
Then the playbook should contain the section "Review SLA Guidance"
|
|
|
|
|
|
Scenario: Review playbook contains architecture checklist
|
|
Given the review playbook file at "docs/development/review_playbook.md"
|
|
When I read the review playbook content
|
|
Then the playbook should contain the section "Architecture Review Checklist"
|
|
|
|
|
|
Scenario: Review playbook contains CLI checklist
|
|
Given the review playbook file at "docs/development/review_playbook.md"
|
|
When I read the review playbook content
|
|
Then the playbook should contain the section "CLI Review Checklist"
|
|
|
|
|
|
Scenario: Review playbook contains DB migration checklist
|
|
Given the review playbook file at "docs/development/review_playbook.md"
|
|
When I read the review playbook content
|
|
Then the playbook should contain the section "DB Migration Review Checklist"
|
|
|
|
|
|
Scenario: Review playbook contains security checklists
|
|
Given the review playbook file at "docs/development/review_playbook.md"
|
|
When I read the review playbook content
|
|
Then the playbook should contain the section "Security-Sensitive Change Checklists"
|
|
|
|
|
|
Scenario: Review playbook contains test matrix
|
|
Given the review playbook file at "docs/development/review_playbook.md"
|
|
When I read the review playbook content
|
|
Then the playbook should contain the section "Required Test Matrix for Reviewers"
|
|
|
|
|
|
Scenario: Review playbook contains acceptable vs blocking examples
|
|
Given the review playbook file at "docs/development/review_playbook.md"
|
|
When I read the review playbook content
|
|
Then the playbook should contain the section "Acceptable vs Blocking Findings"
|
|
|
|
|
|
Scenario: Review playbook references required nox sessions
|
|
Given the review playbook file at "docs/development/review_playbook.md"
|
|
When I read the review playbook content
|
|
Then the playbook should reference these nox sessions:
|
|
| session |
|
|
| lint |
|
|
| typecheck |
|
|
| unit_tests |
|
|
| integration_tests |
|
|
| coverage_report |
|
|
| security_scan |
|
|
| dead_code |
|
|
| complexity |
|
|
| benchmark |
|
|
|
|
|
|
Scenario: Review playbook defines all severity levels
|
|
Given the review playbook file at "docs/development/review_playbook.md"
|
|
When I read the review playbook content
|
|
Then the playbook should contain the text "P0"
|
|
And the playbook should contain the text "P1"
|
|
And the playbook should contain the text "P2"
|
|
And the playbook should contain the text "P3"
|