3e13411fcf
CI / lint (pull_request) Successful in 1m17s
CI / typecheck (pull_request) Successful in 2m2s
CI / security (pull_request) Successful in 2m5s
CI / quality (pull_request) Successful in 1m12s
CI / push-validation (pull_request) Successful in 47s
CI / helm (pull_request) Successful in 1m1s
CI / build (pull_request) Successful in 1m8s
CI / integration_tests (pull_request) Successful in 4m52s
CI / unit_tests (pull_request) Failing after 6m20s
CI / coverage (pull_request) Has been skipped
CI / docker (pull_request) Has been skipped
CI / status-check (pull_request) Failing after 3s
When action YAML references actors using namespace/name format (e.g. `strategy_actor: local/my-strategist`), `_parse_actor_name()` incorrectly treated the namespace prefix as a provider name, causing `ValueError: Unknown provider type: local`. Changes: - `_is_known_provider()`: New utility function (strategy_resolution.py) that checks whether a slash-separated first segment matches a known `ProviderType` value (openai, anthropic, etc.). Consolidated into a single implementation; llm_actors.py now imports from strategy_resolution.py. - `_parse_actor_name()`: When the first segment IS a known provider, preserves existing behaviour (provider/model). When it is NOT a known provider, treats the input as namespace/name and returns the full name as the model identifier with the default provider. Callers SHOULD pre-resolve namespace/name references via the actor registry before calling this function. Both implementations (strategy_resolution.py and llm_actors.py) updated; ProviderType import moved to top level. - `PlanLifecycleService.resolve_actor_provider_model()`: New method that resolves a namespaced actor name (e.g. local/my-strategist) to provider/model format by looking up the actor record and extracting its provider and model fields. - `LifecycleService` Protocol (strategy_resolution.py): Added `resolve_actor_provider_model` to the protocol, eliminating the need for `# type: ignore[union-attr]` at call sites. - `PlanLifecycleProtocol` (llm_actors.py): Added `resolve_actor_provider_model` to the protocol. - Caller pre-resolution: StrategyActor, LLMStrategizeActor, LLMExecuteActor, SessionWorkflow._resolve_llm(), and CLI session commands now pre-resolve actor names through the lifecycle service or via injected actor_resolver callables before calling `_parse_actor_name()`, ensuring namespace/name references are correctly mapped to their underlying LLM providers. - `_build_actor_resolver()` (cli/commands/session.py) and `_build_actor_resolver_for_session_workflow()` (a2a/facade.py): Moved `_is_known_provider` imports to top level; removed redundant `except (NotFoundError, Exception)`; added warning logging for outer exception handlers. - `make_mock_lifecycle()` (mock_strategy_llm.py) and `_make_mock_lifecycle()` (llm_actors_coverage_steps.py): Added `resolve_actor_provider_model` to mock lifecycle services for test compatibility. - Added `@tdd_issue @tdd_issue_11254` tags to all new BDD scenarios related to namespace/name disambiguation in llm_actors_coverage, strategy_actor_llm, and plan_lifecycle_service_coverage_boost_r4 feature files. - Updated docs/CHANGELOG.md with the fix entry. ISSUES CLOSED: #11254
779 lines
38 KiB
Gherkin
779 lines
38 KiB
Gherkin
@mock_only
|
|
Feature: LLM-powered Strategy Actor
|
|
Exercises the StrategyActor that uses an LLM to produce hierarchical
|
|
execution strategies with dependencies, resource requirements,
|
|
complexity estimates, and risk scores. Falls back to stub behaviour
|
|
when no LLM provider is configured.
|
|
|
|
Forgejo: #828
|
|
|
|
# ---------------------------------------------------------------
|
|
# StrategyActor initialization
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: StrategyActor initializes without provider (stub mode)
|
|
When I create a StrategyActor without a provider registry
|
|
Then the StrategyActor should be created successfully
|
|
And the StrategyActor should not have LLM capability
|
|
|
|
Scenario: StrategyActor initializes with provider (LLM mode)
|
|
Given a mock provider registry for strategy actor
|
|
When I create a StrategyActor with the provider registry
|
|
Then the StrategyActor should be created successfully
|
|
And the StrategyActor should have LLM capability
|
|
|
|
# ---------------------------------------------------------------
|
|
# StrategyActor.execute validation
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: StrategyActor rejects empty plan_id
|
|
Given a StrategyActor in stub mode
|
|
When I call strategy actor execute with empty plan_id
|
|
Then sa828 a ValidationError should be raised with message "plan_id must not be empty"
|
|
|
|
# ---------------------------------------------------------------
|
|
# Stub mode execution
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: StrategyActor stub mode parses bullet points
|
|
Given a StrategyActor in stub mode
|
|
When I execute strategy for plan "01HX00000000005T8B1NE00001" with definition "- Setup project\n- Implement feature\n- Write tests"
|
|
Then the strategy result should contain 3 decisions
|
|
And strategy decision 0 should contain "Setup project"
|
|
|
|
Scenario: StrategyActor stub mode uses default when no definition
|
|
Given a StrategyActor in stub mode
|
|
When I execute strategy for plan "01HX00000000005T8B1NE00002" with None definition
|
|
Then the strategy result should contain at least 1 decision
|
|
|
|
Scenario: StrategyActor stub mode with stream callback
|
|
Given a StrategyActor in stub mode
|
|
When I execute strategy for plan "01HX00000000005T8B1NE00003" with callback and definition "- Step one\n- Step two"
|
|
Then the strategy stream callback should have received "strategize_started"
|
|
And the strategy stream callback should have received "strategize_decisions"
|
|
And the strategy stream callback should have received "strategize_complete"
|
|
|
|
Scenario: StrategyActor stub mode with invariants
|
|
Given a StrategyActor in stub mode
|
|
When I execute strategy for plan "01HX00000000005T8B1NE00004" with invariants
|
|
Then the strategy result should contain invariant records
|
|
And the strategy invariant records should have enforcement notes
|
|
|
|
# ---------------------------------------------------------------
|
|
# LLM mode execution
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: StrategyActor LLM mode generates structured strategy from JSON
|
|
Given a StrategyActor with a mock LLM returning JSON strategy
|
|
When I execute strategy for plan "01HX0000000000QQMRNE000001" with definition "Build a REST API with authentication"
|
|
Then the strategy result should contain 5 decisions
|
|
And strategy decision 0 should contain "Set up project scaffolding"
|
|
|
|
Scenario: StrategyActor LLM mode falls back on numbered list response
|
|
Given a StrategyActor with a mock LLM returning numbered list
|
|
When I execute strategy for plan "01HX0000000000QQMRNE000002" with definition "Build a module"
|
|
Then the strategy result should contain 5 decisions
|
|
|
|
Scenario: StrategyActor LLM mode falls back on non-JSON text response
|
|
Given a StrategyActor with a mock LLM returning non-JSON text
|
|
When I execute strategy for plan "01HX0000000000QQMRNE000003" with definition "Build a feature"
|
|
Then the strategy result should contain 3 decisions
|
|
|
|
Scenario: StrategyActor LLM mode falls back to stub on LLM error
|
|
Given a StrategyActor with a failing mock LLM
|
|
When I execute strategy for plan "01HX0000000000QQMRNE000004" with definition "- Step A\n- Step B"
|
|
Then the strategy result should contain 2 decisions
|
|
And strategy decision 0 should contain "Step A"
|
|
|
|
Scenario: StrategyActor LLM mode with ACMS context
|
|
Given a StrategyActor with a mock LLM and ACMS pipeline
|
|
When I execute strategy for plan "01HX0000000000QQMRNE000005" with definition "Refactor the service layer"
|
|
Then the strategy result should contain 5 decisions
|
|
|
|
Scenario: StrategyActor LLM mode with failing ACMS pipeline
|
|
Given a StrategyActor with a mock LLM and failing ACMS pipeline
|
|
When I execute strategy for plan "01HX0000000000QQMRNE000006" with definition "Refactor the service layer"
|
|
Then the strategy result should contain 5 decisions
|
|
|
|
Scenario: StrategyActor LLM mode with resources and project context
|
|
Given a StrategyActor with a mock LLM returning JSON strategy
|
|
When I execute strategy for plan "01HX0000000000QQMRNE000007" with resources and context
|
|
Then the strategy result should contain 5 decisions
|
|
|
|
# ---------------------------------------------------------------
|
|
# Dependency graph validation
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: Validate acyclic dependency graph passes
|
|
When I validate a dependency graph with no cycles
|
|
Then the dependency validation should pass
|
|
|
|
Scenario: Validate empty dependency graph passes
|
|
When I validate an empty dependency graph
|
|
Then the dependency validation should pass
|
|
|
|
Scenario: Validate cyclic dependency graph raises PlanError
|
|
When I validate a dependency graph with a cycle
|
|
Then sa828 a PlanError should be raised with cycle message
|
|
|
|
# ---------------------------------------------------------------
|
|
# Strategy response parsing
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: Parse valid JSON strategy response
|
|
When I parse a JSON strategy response
|
|
Then I should get 5 strategy actions
|
|
And strategy action 0 should have description "Set up project scaffolding and configuration"
|
|
And strategy action 0 should have complexity "low"
|
|
And strategy action 0 should have risk score 0.1
|
|
And strategy action 1 should have resource requirements
|
|
|
|
Scenario: Parse numbered list strategy response
|
|
When I parse a numbered list strategy response
|
|
Then I should get 5 strategy actions
|
|
|
|
Scenario: Parse empty strategy response
|
|
When I parse an empty strategy response
|
|
Then I should get the default strategy action
|
|
|
|
Scenario: Parse JSON with partial fields
|
|
When I parse a JSON strategy response with partial fields
|
|
Then I should get 2 strategy actions
|
|
And strategy action 1 should have complexity "medium"
|
|
And strategy action 1 should have risk score 0.3
|
|
|
|
# ---------------------------------------------------------------
|
|
# Strategy tree and Decision conversion
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: Build Decision objects from strategy tree
|
|
Given a StrategyActor in stub mode
|
|
When I build decisions from strategy tree for plan "01HX0000000000DEC1NE000001"
|
|
Then I should get Decision objects of type strategy_choice
|
|
And the first Decision should be of type prompt_definition
|
|
And all decisions should have plan_id "01HX0000000000DEC1NE000001"
|
|
And decision sequence_numbers should be monotonically increasing from zero
|
|
|
|
# ---------------------------------------------------------------
|
|
# Prompt construction
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: Build strategy prompt with all context
|
|
When I build a strategy prompt with definition resources and context
|
|
Then the prompt should contain the definition of done
|
|
And the prompt should contain resource information
|
|
And the prompt should contain project context
|
|
And the prompt should contain ACMS context
|
|
|
|
Scenario: Build strategy prompt with minimal context
|
|
When I build a strategy prompt with only definition
|
|
Then the prompt should contain the definition of done
|
|
And the prompt should not contain resource information
|
|
|
|
# ---------------------------------------------------------------
|
|
# resolve_strategy_actor integration point
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: Resolve strategy actor with LLM config
|
|
When I resolve strategy actor with config "llm"
|
|
Then the resolved actor should be a StrategyActor
|
|
|
|
Scenario: Resolve strategy actor with stub config
|
|
When I resolve strategy actor with config "stub"
|
|
Then the resolved actor should be None
|
|
|
|
Scenario: Resolve strategy actor with provider registry
|
|
When I resolve strategy actor with a provider registry
|
|
Then the resolved actor should be a StrategyActor
|
|
|
|
Scenario: Resolve strategy actor with no config and no registry
|
|
When I resolve strategy actor with no config and no registry
|
|
Then the resolved actor should be None
|
|
|
|
# ---------------------------------------------------------------
|
|
# LLM mode with empty response
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: StrategyActor LLM mode handles empty LLM response
|
|
Given a StrategyActor with a mock LLM returning empty response
|
|
When I execute strategy for plan "01HX0000000000QQMRNE000008" with definition "Build a feature"
|
|
Then the strategy result should contain at least 1 decision
|
|
|
|
# ---------------------------------------------------------------
|
|
# _parse_actor_name helper
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: Parse strategy actor name with provider/model
|
|
When I parse strategy actor name "anthropic/claude-3"
|
|
Then the strategy parsed provider should be "anthropic"
|
|
And the strategy parsed model should be "claude-3"
|
|
|
|
Scenario: Parse strategy actor name with only model
|
|
When I parse strategy actor name "gpt-4"
|
|
Then the strategy parsed provider should be "openai"
|
|
And the strategy parsed model should be "gpt-4"
|
|
|
|
Scenario: Parse strategy actor name with empty string
|
|
When I parse strategy actor name ""
|
|
Then the strategy parsed provider should be "openai"
|
|
And the strategy parsed model should be "gpt-4"
|
|
|
|
@tdd_issue @tdd_issue_11254
|
|
Scenario: Parse strategy actor name with namespace/name format
|
|
When I parse strategy actor name "local/my-strategist"
|
|
Then the strategy parsed provider should be "openai"
|
|
And the strategy parsed model should be "local/my-strategist"
|
|
|
|
@tdd_issue @tdd_issue_11254
|
|
Scenario: Parse strategy actor name with org namespace/name format
|
|
When I parse strategy actor name "cleverthis/my-executor"
|
|
Then the strategy parsed provider should be "openai"
|
|
And the strategy parsed model should be "cleverthis/my-executor"
|
|
|
|
@tdd_issue @tdd_issue_11254
|
|
Scenario: Parse strategy actor name with unknown-provider-like segment
|
|
When I parse strategy actor name "unknown/gpt-4"
|
|
Then the strategy parsed provider should be "openai"
|
|
And the strategy parsed model should be "unknown/gpt-4"
|
|
|
|
@tdd_issue @tdd_issue_11254
|
|
Scenario: _is_known_provider recognises valid providers
|
|
When I check if "openai" is a known strategy provider
|
|
Then the strategy provider check should be true
|
|
|
|
@tdd_issue @tdd_issue_11254
|
|
Scenario: _is_known_provider rejects namespace prefix
|
|
When I check if "local" is a known strategy provider
|
|
Then the strategy provider check should be false
|
|
|
|
# ---------------------------------------------------------------
|
|
# Cyclic dependency detection through LLM execute path (H8)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: StrategyActor LLM mode raises PlanError on cyclic dependencies
|
|
Given a StrategyActor with a mock LLM returning cyclic dependencies
|
|
When I execute strategy for plan "01HX0000000000QQMRNE000009" expecting cycle error
|
|
Then sa828 a PlanError should be raised with cycle message
|
|
|
|
# ---------------------------------------------------------------
|
|
# Dependency edge resolution (M8)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: LLM JSON strategy resolves dependency edges correctly
|
|
Given a StrategyActor with a mock LLM returning JSON strategy
|
|
When I execute strategy for plan "01HX0000000000QQMRNE00000A" and inspect the tree
|
|
Then the strategy tree should have dependency edges
|
|
And strategy action 1 should depend on action 0
|
|
|
|
# ---------------------------------------------------------------
|
|
# LLM response without content attribute (M9)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: Parse strategy response from LLM without content attribute
|
|
When I parse a strategy response from an LLM without content attribute
|
|
Then the strategy result should contain decisions
|
|
|
|
# ---------------------------------------------------------------
|
|
# build_decisions with empty plan_id (M10)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: build_decisions rejects empty plan_id
|
|
Given a StrategyActor in stub mode
|
|
When I call build_decisions with empty plan_id
|
|
Then sa828 a ValidationError should be raised with message "plan_id must not be empty"
|
|
|
|
# ---------------------------------------------------------------
|
|
# Empty JSON array response (M11)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: Parse empty JSON array returns default action
|
|
When I parse an empty JSON array response
|
|
Then I should get the default strategy action
|
|
|
|
# ---------------------------------------------------------------
|
|
# LLM response with list content attribute (T1)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: StrategyActor handles LLM response where content is a list
|
|
When I parse a strategy response from an LLM with list content attribute
|
|
Then the strategy result should contain decisions
|
|
|
|
# ---------------------------------------------------------------
|
|
# build_decisions multi-level hierarchy (T2)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: build_decisions maps parent_id correctly across hierarchy levels
|
|
Given a StrategyActor in stub mode
|
|
When I build decisions from a multi-level strategy tree for plan "01HX0000000000DEC1NE000002"
|
|
Then the second-level decisions should reference the mid-level parent decision
|
|
|
|
# ---------------------------------------------------------------
|
|
# LLM prompt content verification (M6)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: LLM receives correct SystemMessage and HumanMessage prompt
|
|
Given a StrategyActor with a mock LLM returning JSON strategy
|
|
When I execute strategy for plan "01HX0000000000QQMRNE00000D" with resources and context
|
|
Then the mock LLM should have been called with a SystemMessage and a HumanMessage
|
|
And the HumanMessage should contain "Build REST API"
|
|
And the HumanMessage should contain "source-code"
|
|
And the HumanMessage should contain "FastAPI project with PostgreSQL backend"
|
|
|
|
# ---------------------------------------------------------------
|
|
# Prompt truncation (M7)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: build_strategy_prompt truncates oversized definition_of_done
|
|
When I build a strategy prompt with a definition exceeding the max length
|
|
Then the prompt length should be within the truncation limit
|
|
|
|
Scenario: build_strategy_prompt truncates oversized definition at word boundary
|
|
When I build a strategy prompt with a word-spaced definition exceeding the max length
|
|
Then the prompt length should be within the truncation limit
|
|
And the truncated definition should end with an ellipsis
|
|
|
|
# ---------------------------------------------------------------
|
|
# Risk score clamping (M9)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: Parse JSON with out-of-range risk score clamps to valid range
|
|
When I parse a JSON strategy response with risk score above 1.0
|
|
Then strategy action 0 should have risk score 1.0
|
|
|
|
# ---------------------------------------------------------------
|
|
# Self-dependency filtering (M10)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: Parse JSON with self-referencing dependency silently filters it
|
|
When I parse a JSON strategy response with self-dependency
|
|
Then the strategy tree should have no self-dependency edges
|
|
|
|
# ---------------------------------------------------------------
|
|
# Multi-slash actor name parsing (L2)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: Parse strategy actor name with multiple slashes
|
|
When I parse strategy actor name "provider/model/version"
|
|
Then the strategy parsed provider should be "openai"
|
|
And the strategy parsed model should be "provider/model/version"
|
|
|
|
# ---------------------------------------------------------------
|
|
# Actor name with empty segments (M2 fix)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: Parse strategy actor name with slash only defaults
|
|
When I parse strategy actor name "/"
|
|
Then the strategy parsed provider should be "openai"
|
|
And the strategy parsed model should be "gpt-4"
|
|
|
|
Scenario: Parse strategy actor name with empty provider uses default provider
|
|
When I parse strategy actor name "/model-x"
|
|
Then the strategy parsed provider should be "openai"
|
|
And the strategy parsed model should be "model-x"
|
|
|
|
Scenario: Parse strategy actor name with empty model uses default model
|
|
When I parse strategy actor name "provider/"
|
|
Then the strategy parsed provider should be "openai"
|
|
And the strategy parsed model should be "provider/"
|
|
|
|
# ---------------------------------------------------------------
|
|
# Whitespace-only definition_of_done (L5 fix)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: StrategyActor stub mode with whitespace-only definition
|
|
Given a StrategyActor in stub mode
|
|
When I execute strategy for plan "01HX00000000005T8B1NE00005" with definition " "
|
|
Then the strategy result should contain at least 1 decision
|
|
|
|
# ---------------------------------------------------------------
|
|
# build_decisions with empty actions list (L3 fix)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: build_decisions with empty actions returns empty list
|
|
Given a StrategyActor in stub mode
|
|
When I call build_decisions with empty actions for plan "01HX0000000000DEC1NE000003"
|
|
Then the decisions list should be empty
|
|
|
|
# ---------------------------------------------------------------
|
|
# Resource list truncation (L6 / M1 fix)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: build_strategy_prompt truncates oversized resource list
|
|
When I build a strategy prompt with a resource list exceeding the max count
|
|
Then the prompt should contain truncation indicator
|
|
|
|
# ---------------------------------------------------------------
|
|
# Lifecycle with strategy_actor=None (L4 fix)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: StrategyActor LLM mode with lifecycle returning null strategy_actor
|
|
Given a StrategyActor with a mock LLM and lifecycle with null strategy_actor
|
|
When I execute strategy for plan "01HX0000000000QQMRNE00000F" with definition "Build a feature"
|
|
Then the strategy result should contain decisions
|
|
|
|
# ---------------------------------------------------------------
|
|
# Duplicate step number collision (H1 review fix)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: LLM JSON with duplicate step numbers produces unique action IDs
|
|
When I parse a JSON strategy response with duplicate step numbers
|
|
Then all strategy tree actions should have unique action IDs
|
|
|
|
# ---------------------------------------------------------------
|
|
# _MAX_CONTEXT_CHARS truncation (M9 review fix)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: build_strategy_prompt truncates oversized project_context
|
|
When I build a strategy prompt with project_context exceeding the max length
|
|
Then the project_context in the prompt should be within the truncation limit
|
|
|
|
# ---------------------------------------------------------------
|
|
# ACMS context prompt inclusion (M10 review fix)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: LLM prompt includes ACMS context when pipeline is available
|
|
Given a StrategyActor with a mock LLM and ACMS pipeline
|
|
When I execute strategy for plan "01HX0000000000QQMRNE00000G" with resources and context
|
|
Then the mock LLM should have been called with a SystemMessage and a HumanMessage
|
|
And the HumanMessage should contain "Python 3.12, FastAPI, SQLAlchemy"
|
|
|
|
# ---------------------------------------------------------------
|
|
# LLM response str() fallback (L1 review fix)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: Parse strategy response from LLM with no content or text attribute
|
|
When I parse a strategy response from an LLM with no content or text attribute
|
|
Then the strategy result should contain decisions
|
|
|
|
# ---------------------------------------------------------------
|
|
# SystemMessage content verification (L2 review fix)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: LLM receives correct SystemMessage content
|
|
Given a StrategyActor with a mock LLM returning JSON strategy
|
|
When I execute strategy for plan "01HX0000000000QQMRNE00000H" with resources and context
|
|
Then the mock LLM should have been called with a SystemMessage and a HumanMessage
|
|
And the SystemMessage should contain "expert software architect"
|
|
|
|
# ---------------------------------------------------------------
|
|
# Resolve strategy actor with unknown config value (M5 cycle-2)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: Resolve strategy actor with unrecognised config value
|
|
When I resolve strategy actor with config "auto"
|
|
Then the resolved actor should be None
|
|
|
|
# ---------------------------------------------------------------
|
|
# Negative risk score clamping (M6 cycle-2)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: Parse JSON with negative risk score clamps to zero
|
|
When I parse a JSON strategy response with negative risk score
|
|
Then strategy action 0 should have risk score 0.0
|
|
|
|
# ---------------------------------------------------------------
|
|
# Non-sequential step numbers (L2 cycle-2)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: LLM JSON with non-sequential step numbers resolves correctly
|
|
When I parse a JSON strategy response with non-sequential step numbers
|
|
Then the strategy result should contain 3 decisions
|
|
And the strategy tree should have dependency edges
|
|
|
|
# ---------------------------------------------------------------
|
|
# Null JSON description filtering (H3 cycle-2)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: Parse JSON with null description drops the action
|
|
When I parse a JSON strategy response with null description
|
|
Then I should get 1 strategy actions
|
|
|
|
# ---------------------------------------------------------------
|
|
# Trailing bracketed commentary in LLM response (H4 cycle-2)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: Parse JSON with trailing bracketed commentary succeeds
|
|
When I parse a JSON strategy response with trailing brackets
|
|
Then I should get 2 strategy actions
|
|
And strategy action 0 should have description "Setup project"
|
|
|
|
# ---------------------------------------------------------------
|
|
# Review-fix scenarios (cycle-3)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: StrategyActor LLM mode emits stream callback events
|
|
Given a StrategyActor with a mock LLM returning JSON strategy
|
|
When I execute strategy for plan "01HX0000000000QQMRNE00000N" with callback and definition "Build a feature"
|
|
Then the strategy stream callback should have received "strategize_started"
|
|
And the strategy stream callback should have received "strategize_decisions"
|
|
And the strategy stream callback should have received "strategize_complete"
|
|
|
|
Scenario: build_strategy_prompt truncates oversized acms_context
|
|
When I build a strategy prompt with acms_context exceeding the max length
|
|
Then the acms_context in the prompt should be within the truncation limit
|
|
|
|
Scenario: LLM mode decision_root_id is a valid ULID
|
|
Given a StrategyActor with a mock LLM returning JSON strategy
|
|
When I execute strategy for plan "01HX0000000000QQMRNE00000P" with definition "Build a feature"
|
|
Then the strategy result decision_root_id should be a valid ULID
|
|
|
|
Scenario: build_decisions computes confidence_score from risk_score
|
|
Given a StrategyActor in stub mode
|
|
When I build decisions with known risk scores for plan "01HX0000000000DEC1NE000004"
|
|
Then each decision confidence_score should equal 1.0 minus risk_score
|
|
|
|
# ---------------------------------------------------------------
|
|
# Review-fix scenarios (cycle-4 — code review hardening)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: Parse JSON with NaN risk score defaults to 0.3
|
|
When I parse a JSON strategy response with NaN risk score
|
|
Then strategy action 0 should have risk score 0.3
|
|
|
|
Scenario: Parse JSON with Infinity risk score defaults to 0.3
|
|
When I parse a JSON strategy response with Inf risk score
|
|
Then strategy action 0 should have risk score 0.3
|
|
|
|
Scenario: Parse JSON with non-dict items filters them out
|
|
When I parse a JSON strategy response with non-dict items
|
|
Then I should get 1 strategy actions
|
|
And strategy action 0 should have description "Valid action among non-dict items"
|
|
|
|
Scenario: Parse numbered list with star and bullet markers
|
|
When I parse a strategy response with star and bullet markers
|
|
Then I should get 4 strategy actions
|
|
And strategy action 0 should have description "Set up project structure"
|
|
|
|
Scenario: LLM returning more than _MAX_ACTIONS truncates to cap
|
|
When I parse a JSON strategy response exceeding the action cap
|
|
Then the parsed action count should equal _MAX_ACTIONS
|
|
|
|
# ---------------------------------------------------------------
|
|
# Review-fix: hierarchical parent_id from dependency graph (B2)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: Strategy tree infers parent_id from first dependency
|
|
Given a StrategyActor with a mock LLM returning JSON strategy
|
|
When I execute strategy for plan "01HX0000000000QQMRNE0000B2" and inspect the tree
|
|
Then strategy action 1 parent_id should equal action 0 action_id
|
|
And strategy action 4 parent_id should equal action 2 action_id
|
|
|
|
Scenario: Strategy tree actions without dependencies parent to root
|
|
Given a StrategyActor with a mock LLM returning JSON strategy
|
|
When I execute strategy for plan "01HX0000000000QQMRNE0000B3" and inspect the tree
|
|
Then strategy action 0 parent_id should be None
|
|
|
|
# ---------------------------------------------------------------
|
|
# Review-fix: downstream_decision_ids from dependency edges (B3)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: build_decisions populates downstream_decision_ids from edges
|
|
Given a StrategyActor with a mock LLM returning JSON strategy
|
|
When I build decisions from LLM tree for plan "01HX0000000000QQMRNE0000B4"
|
|
Then root decision downstream_decision_ids should not be empty
|
|
And leaf decision downstream_decision_ids should be empty
|
|
|
|
Scenario: build_decisions with no edges leaves downstream empty
|
|
Given a StrategyActor in stub mode
|
|
When I build decisions from strategy tree for plan "01HX0000000000DEC1NE0000B5"
|
|
Then all decisions should have empty downstream_decision_ids
|
|
|
|
# ---------------------------------------------------------------
|
|
# Review-fix: _truncate_at_word edge cases (L3)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: _truncate_at_word returns short text unchanged
|
|
When I truncate "hello world" at 50 characters
|
|
Then the truncated text should be "hello world"
|
|
|
|
Scenario: _truncate_at_word cuts at word boundary
|
|
When I truncate "hello world foo bar" at 12 characters
|
|
Then the truncated text should be "hello..."
|
|
|
|
Scenario: _truncate_at_word handles empty string
|
|
When I truncate an empty string at 10 characters
|
|
Then the truncated text should be an empty string
|
|
|
|
# ---------------------------------------------------------------
|
|
# Review-fix: create_llm called with correct args (L4)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: LLM mode calls create_llm with resolved provider and model
|
|
Given a StrategyActor with a mock LLM returning JSON strategy
|
|
When I execute strategy for plan "01HX0000000000QQMRNEK4TEST" with definition "Build a feature"
|
|
Then create_llm should have been called with provider "openai" and model "gpt-4"
|
|
|
|
# ---------------------------------------------------------------
|
|
# Review-fix: non-numeric step fallback in _build_tree (L5)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: LLM JSON with non-numeric step field falls back to index
|
|
When I parse a JSON strategy response with non-numeric step field
|
|
Then the strategy result should contain decisions
|
|
|
|
# ---------------------------------------------------------------
|
|
# Review-fix: lifecycle exception fallback (R1)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: StrategyActor LLM mode falls back to default actor when lifecycle raises
|
|
Given a StrategyActor with a mock LLM and a failing lifecycle service
|
|
When I execute strategy for plan "01HX0000000000QQMRNER1TEST" with definition "Build a feature"
|
|
Then the strategy result should contain decisions
|
|
And create_llm should have been called with provider "openai" and model "gpt-4"
|
|
|
|
# ---------------------------------------------------------------
|
|
# Review-fix: PydanticValidationError re-raise (R2)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: StrategyActor LLM mode re-raises PydanticValidationError
|
|
Given a StrategyActor with a mock LLM returning Pydantic-invalid data
|
|
When I execute strategy for plan "01HX0000000000QQMRNER2TEST" expecting Pydantic error
|
|
Then a PydanticValidationError should have been raised
|
|
|
|
# ---------------------------------------------------------------
|
|
# Review-fix: self-loop cycle detection (R3)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: Validate self-loop dependency graph raises PlanError
|
|
When I validate a dependency graph with a self-loop
|
|
Then sa828 a PlanError should be raised with cycle message
|
|
|
|
# ---------------------------------------------------------------
|
|
# Review-fix: whitespace-only actor name (R4)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: Parse strategy actor name with whitespace-only defaults
|
|
When I parse strategy actor name " "
|
|
Then the strategy parsed provider should be "openai"
|
|
And the strategy parsed model should be "gpt-4"
|
|
|
|
# ---------------------------------------------------------------
|
|
# Post code-review hardening (PR #1175, review cycle 3)
|
|
# ---------------------------------------------------------------
|
|
|
|
# M1: XML tag injection sanitisation
|
|
Scenario: build_strategy_prompt escapes XML special characters in user content
|
|
When I build a strategy prompt with XML injection in definition_of_done
|
|
Then the prompt should not contain a raw closing XML tag
|
|
|
|
# M2: JSON parser retry on false-start preamble
|
|
Scenario: Parse JSON with preamble containing bracket fragment
|
|
When I parse a JSON strategy response with preamble bracket fragment
|
|
Then I should get 2 strategy actions
|
|
And strategy action 0 should have description "Real action one"
|
|
|
|
# L2: _truncate_at_word with max_chars below 3
|
|
Scenario: _truncate_at_word with max_chars below ellipsis length
|
|
When I truncate "hello world" at 2 characters
|
|
Then the truncated text should be "he"
|
|
|
|
# CR5-L1: _truncate_at_word with max_chars exactly at ellipsis length
|
|
Scenario: _truncate_at_word with max_chars exactly 3 returns only ellipsis
|
|
When I truncate "hello world" at 3 characters
|
|
Then the truncated text should be "..."
|
|
|
|
# L5: resolve_strategy_actor with both llm config and registry
|
|
Scenario: Resolve strategy actor with llm config and provider registry
|
|
When I resolve strategy actor with config "llm" and a provider registry
|
|
Then the resolved actor should be a StrategyActor
|
|
And the resolved actor should have LLM capability
|
|
|
|
# L7: build_decisions with unresolvable parent_id
|
|
Scenario: build_decisions falls back to root for unresolvable parent_id
|
|
Given a StrategyActor in stub mode
|
|
When I build decisions from a tree with unresolvable parent_id for plan "01HX0000000000DEC1NE000006"
|
|
Then the orphan decision parent should fall back to root decision
|
|
|
|
# ---------------------------------------------------------------
|
|
# Post code-review hardening (PR #1175, review cycle 4)
|
|
# ---------------------------------------------------------------
|
|
|
|
# CR4-S1a: XML injection sanitisation in resources field
|
|
Scenario: build_strategy_prompt escapes XML in resources field
|
|
When I build a strategy prompt with XML injection in resources
|
|
Then the prompt should not contain a raw closing available_resources tag
|
|
|
|
# CR4-S1b: XML injection sanitisation in project_context field
|
|
Scenario: build_strategy_prompt escapes XML in project_context field
|
|
When I build a strategy prompt with XML injection in project_context
|
|
Then the prompt should not contain a raw closing project_context tag
|
|
|
|
# CR4-S1c: XML injection sanitisation in acms_context field
|
|
Scenario: build_strategy_prompt escapes XML in acms_context field
|
|
When I build a strategy prompt with XML injection in acms_context
|
|
Then the prompt should not contain a raw closing code_analysis_context tag
|
|
|
|
# CR4-S1d: Ampersand sanitisation in user content
|
|
Scenario: build_strategy_prompt escapes ampersand in user content
|
|
When I build a strategy prompt with ampersand in definition_of_done
|
|
Then the prompt should contain the escaped ampersand entity
|
|
|
|
# CR4-T3: Retry budget exhaustion gracefully returns None
|
|
Scenario: Parse JSON with many false-start anchors degrades gracefully
|
|
When I parse a JSON strategy response with many false-start bracket anchors
|
|
Then the parsed actions should contain the real JSON action
|
|
|
|
# CR4-T4: Non-sequential step edge specificity
|
|
Scenario: LLM JSON with non-sequential steps resolves specific dependency edges
|
|
When I parse a JSON strategy response with non-sequential step numbers and inspect the tree
|
|
Then strategy tree action with step 20 should depend on action with step 10
|
|
And strategy tree action with step 30 should depend on action with step 20
|
|
|
|
# ---------------------------------------------------------------
|
|
# Post code-review hardening (PR #1175, review cycle 6)
|
|
# ---------------------------------------------------------------
|
|
|
|
# CR6-M2: Invariant constraints appear in LLM prompt
|
|
Scenario: LLM prompt includes constraints section when invariants are provided
|
|
When I build a strategy prompt with invariants
|
|
Then the prompt should contain a constraints section
|
|
And the prompt constraints should include invariant text "Must use Python 3.12+"
|
|
And the prompt constraints should include source label "plan"
|
|
|
|
# CR6-M3: XML sanitization of invariant text in prompt
|
|
Scenario: build_strategy_prompt escapes XML in invariant text
|
|
When I build a strategy prompt with XML injection in invariant text
|
|
Then the prompt should not contain a raw closing constraints tag
|
|
|
|
# CR6-M4: Invariant truncation cap
|
|
Scenario: build_strategy_prompt truncates oversized invariant list
|
|
When I build a strategy prompt with invariant list exceeding the max count
|
|
Then the prompt should contain invariant truncation indicator
|
|
|
|
# CR6-M5: _truncate_at_word with negative max_chars
|
|
Scenario: _truncate_at_word with negative max_chars returns empty string
|
|
When I truncate "hello world" at -5 characters
|
|
Then the truncated text should be an empty string
|
|
|
|
# CR6-L8: _truncate_at_word with text containing no spaces
|
|
Scenario: _truncate_at_word with no-space text truncates at hard limit
|
|
When I truncate "abcdefghijklmnopqrstuvwxyz" at 10 characters
|
|
Then the truncated text should be "abcdefg..."
|
|
|
|
# ---------------------------------------------------------------
|
|
# Post code-review hardening (PR #1175, review cycle 7)
|
|
# ---------------------------------------------------------------
|
|
|
|
# CR7-L3: Global JSON parse attempt cap terminates gracefully
|
|
Scenario: Parse JSON with pathological input exhausts global attempt cap
|
|
When I parse a JSON strategy response that exhausts the global attempt cap
|
|
Then the strategy result should contain at least 1 decision
|
|
|
|
# CR7-L4: build_decisions silently drops orphaned dependency edges
|
|
Scenario: build_decisions drops dependency edges referencing non-existent actions
|
|
Given a StrategyActor in stub mode
|
|
When I build decisions from a tree with orphaned dependency edges for plan "01HX0000000000DEC1NE000007"
|
|
Then the orphan edge should be silently dropped
|
|
|
|
# CR7-M2: execute rejects non-ULID plan_id
|
|
Scenario: StrategyActor rejects non-ULID plan_id format
|
|
Given a StrategyActor in stub mode
|
|
When I call strategy actor execute with non-ULID plan_id "not-a-valid-ulid"
|
|
Then sa828 a ValidationError should be raised with message "plan_id must be a valid ULID"
|
|
|
|
# CR7-M2b: build_decisions rejects non-ULID plan_id
|
|
Scenario: build_decisions rejects non-ULID plan_id format
|
|
Given a StrategyActor in stub mode
|
|
When I call build_decisions with non-ULID plan_id "not-a-valid-ulid"
|
|
Then sa828 a ValidationError should be raised with message "plan_id must be a valid ULID"
|