Files
cleveragents-core/robot/helper_strategy_actor.py
T
CoreRasurae d3cb534caf
CI / push-validation (pull_request) Successful in 17s
CI / helm (pull_request) Successful in 19s
CI / lint (pull_request) Successful in 27s
CI / security (pull_request) Successful in 32s
CI / quality (pull_request) Successful in 41s
CI / typecheck (pull_request) Successful in 48s
CI / build (pull_request) Successful in 3m18s
CI / e2e_tests (pull_request) Successful in 4m16s
CI / integration_tests (pull_request) Successful in 4m16s
CI / unit_tests (pull_request) Successful in 5m8s
CI / docker (pull_request) Successful in 1m32s
CI / coverage (pull_request) Successful in 10m53s
CI / status-check (pull_request) Successful in 1s
CI / push-validation (push) Successful in 18s
CI / helm (push) Successful in 23s
CI / build (push) Successful in 31s
CI / lint (push) Successful in 43s
CI / typecheck (push) Successful in 52s
CI / security (push) Successful in 52s
CI / e2e_tests (push) Successful in 3m37s
CI / quality (push) Successful in 3m44s
CI / integration_tests (push) Successful in 6m35s
CI / unit_tests (push) Failing after 7m36s
CI / docker (push) Has been skipped
CI / coverage (push) Successful in 13m39s
CI / status-check (push) Failing after 1s
feat(plan): implement LLM-powered strategy actor
Implement StrategyActor class for the plan strategize phase that uses an
LLM to produce hierarchical execution strategies with dependencies,
resource requirements, estimated complexity, and risk scores.

Key components:
- StrategyActor: Core actor with LLM prompt construction, response
  parsing (JSON and numbered-list fallback), and graceful degradation
  to StrategizeStubActor when no LLM provider is configured
- StrategyAction/StrategyTree: Pydantic models for the hierarchical
  action tree with dependency links
- validate_no_cycles(): Kahns algorithm (deque-based) for dependency
  graph cycle detection, raising PlanError on circular dependencies
- build_strategy_prompt(): Context-aware prompt construction using
  definition_of_done, resources, project context, and ACMS analysis
  with XML-delimited user content sections for prompt injection
  hardening
- parse_strategy_response(): Robust LLM output parsing with JSON
  extraction and numbered-list fallback
- resolve_strategy_actor(): Integration point for the existing
  actor.default.strategy config key (CLEVERAGENTS_DEFAULT_STRATEGY_ACTOR)
- Decision conversion producing strategy_choice Decision objects
- build_decisions() preserves tree hierarchy via parent_id mapping,
  populates downstream_decision_ids from dependency edges, and
  validates plan_id

Structural tree hierarchy (B2 review fix):
- _build_tree infers parent_id from the dependency graph: each
  actions first resolved dependency becomes its structural parent.
  Actions with no dependencies fall back to the root.  This produces
  hierarchical trees for agents plan tree rendering per spec
  Plan Decision Tree.

Downstream decision tracking (B3 review fix):
- build_decisions populates downstream_decision_ids from the strategy
  trees dependency edges using a pre-generated decision_id map so
  influence relationships between decisions are recorded per the spec
  Decision Record Structure.

Post code-review hardening (PR #1175):
- Broadened exception handling in execute() and ACMS retrieval to
  catch all LLM provider errors (openai, httpx, anthropic, etc.)
  with graceful fallback to stub mode (H1, H2)
- Added warning log for unresolvable dependency references so
  dropped edges are visible in structured logs (H3)
- Added XML-delimited user content sections and explicit data-only
  instructions in system prompt for prompt injection hardening (H4)
- Switched prompt truncation to word-boundary-safe _truncate_at_word()
  for all prompt input sections (M1)
- Fixed _parse_actor_name to preserve user-specified provider or
  model when only one segment is empty, instead of discarding both (M2)
- Annotated _build_invariant_records as placeholder pending the
  Invariant Reconciliation Actor implementation (M5)
- Documented resources/project_context params as future-wired
  through PlanExecutor.run_strategize() (M8)
- Added docstring noting supersession relationship with
  LLMStrategizeActor in llm_actors.py (M9)
- Added __all__ export definition (L2)
- Improved validate_no_cycles docstring edge direction semantics (L7)
- Cap JSON parse retry loop at _MAX_JSON_PARSE_RETRIES (10)

Post second code-review hardening (PR #1175, review cycle 2):
- Fixed _truncate_at_word docstring: documented max_chars >= 3
  precondition for the result-length guarantee (R-H1)
- Added warning log in build_decisions for unresolvable parent_id
  references, matching the existing _build_tree warning for
  unresolvable dependency references (R-H2)
- Fixed _parse_actor_name to handle whitespace-only input by adding
  actor_name.strip() check alongside the emptiness check (R-M1)
- Tightened ACMS scenario assertions from non-empty to expected
  count of 5 decisions (R-L3)
- Added timeout=60s on_timeout=kill to all Robot test cases for
  consistency with project patterns (R-M5)

Post third code-review hardening (PR #1175, review cycle 3):
- Added _sanitize_xml_content() to escape XML special characters
  (<, >, &) in user content before embedding into XML-delimited
  prompt sections, preventing prompt injection via forged closing
  tags (spec Prompt Injection Mitigation) (CR3-M1)
- Upgraded _try_parse_json() to multi-anchor retry: collects all
  [{ positions left-to-right and tries each as a candidate start,
  fixing false-start anchoring when LLM preamble contains [{
  fragments before the real JSON array (CR3-M2)
- Added _truncate_at_word() guard for max_chars < 3: returns a
  hard slice instead of word-boundary truncation when the ellipsis
  would exceed the limit (CR3-L2)
- Changed _build_tree collision fallback key from -(idx+1) to
  -(1_000_000+idx) to eliminate theoretical collision with
  LLM-produced negative step numbers (CR3-L3)
- Added forward-looking API docstring note to build_decisions()
  documenting that it is not yet wired into PlanExecutor and will
  be integrated once Decision persistence lands (CR3-M3)

Post fourth code-review hardening (PR #1175, review cycle 4):
- Fixed _try_parse_json per-anchor retry counter: reset retries=0
  at the start of each anchor iteration so false-start [{ anchors
  in LLM preamble text no longer exhaust the retry budget for the
  correct anchor (CR4-B1)
- Added known-limitations docstring to module header documenting
  missing decision types (resource_selection, subplan_spawn,
  invariant_enforced) as future work (CR4-D1)
- Rewrote XML injection assertion in test to use regex extraction
  instead of fragile chained .split() calls that could IndexError
  on structural changes (CR4-T5)

Post fifth code-review hardening (PR #1175, review cycle 5):
- Added warning log in build_decisions for empty-string parent_id
  (distinct from None) so the silent fallback to root is visible
  in structured logs for debuggability (CR5-B1)
- Added plan_id propagation assertion to build_decisions test
  scenarios verifying decision.plan_id matches the input (CR5-T1)
- Added sequence_number monotonicity assertion verifying decision
  sequence_numbers are zero-indexed and monotonically increasing
  (CR5-T2)
- Added _truncate_at_word boundary test for max_chars=3 (exactly
  ellipsis length) verifying correct "..." output (CR5-T3)
- Tightened false-start anchor test from permissive len>=1 to
  specific description match "Sole real action" (CR5-T4)
- Added word-boundary truncation test using space-separated input
  to exercise the rfind(" ") path under oversized DoD (CR5-T5)

Post sixth code-review hardening (PR #1175, review cycle 6):
- Added _MAX_INVARIANTS cap (100) for invariant list truncation in
  prompt to prevent token limit overflows, consistent with other prompt
  section caps (CR6-M4)
- Added negative max_chars guard in _truncate_at_word returning empty
  string instead of slicing from end (CR6-M5)
- Added global JSON parse attempt cap _MAX_GLOBAL_JSON_ATTEMPTS (50)
  across all anchors in _try_parse_json (CR6-L3)
- Moved re import to module level in strategy_parsing.py per
  CONTRIBUTING import guidelines (CR6-L4)
- Extracted _DEFAULT_DESCRIPTION constant to eliminate duplication
  between _default_action() and _build_tree() (CR6-L5)

Post seventh code-review hardening (PR #1175, review cycle 7):
- Decoupled _execute_stub from StrategizeStubActor._parse_steps
  private method by delegating to parse_strategy_response, removing
  cross-class private method dependency (CR7-M1)
- Added ULID format validation on plan_id in execute() and
  build_decisions() for spec-consistent argument validation per
  §Plan glossary and CONTRIBUTING §Argument Validation (CR7-M2)
- Constrained StrategyAction.estimated_complexity to
  Literal["low", "medium", "high"] at Pydantic model level per
  CONTRIBUTING §Type Safety (CR7-M5)
- Documented XML-tag prompt boundary deviation from spec
  [USER_CONTENT_START]/[USER_CONTENT_END] markers with rationale
  for the more structured approach (CR7-M6)
- Added _build_tree empty-input guard comment documenting orphaned
  root_id semantics (CR7-L1)
- Added _truncate_at_word > 0 intent comment explaining why
  position-0 space is intentionally excluded (CR7-L2)
- Added build_decisions context_snapshot future-work comment
  referencing spec §Decision Record Structure (CR7-L5)
- Used enumerate() in _build_tree first loop for idiomatic
  Python (CR7-L7)
- Fixed false-start anchor test (CR5-T4) broken by CR6-L3 global
  cap: reduced preamble fragments from 15 to 3 so total attempts
  stay within _MAX_GLOBAL_JSON_ATTEMPTS (CR7-T1)
- Fixed test plan_ids containing non-Crockford-Base32 characters
  (L→K) to pass ULID format validation (CR7-T2)

Tests:
- 105 Behave BDD scenarios in features/strategy_actor_llm.feature
  adding: global JSON attempt cap exhaustion (CR7-L3), orphaned
  dependency edge silent drop (CR7-L4), non-ULID plan_id rejection
  in execute() and build_decisions() (CR7-M2)
- 101 Behave BDD scenarios in features/strategy_actor_llm.feature
  including new scenarios for _truncate_at_word edge cases (L3),
  create_llm argument verification (L4), non-numeric step field
  fallback (L5), updated assertions for XML-delimited prompts
  and _parse_actor_name partial-segment preservation (M2),
  lifecycle exception fallback (R1), PydanticValidationError
  re-raise verification (R2), self-loop cycle detection (R3),
  whitespace-only actor name (R4), XML tag injection sanitisation
  (CR3-M1), preamble bracket fragment parsing (CR3-M2),
  _truncate_at_word sub-3 limit (CR3-L2), resolve_strategy_actor
  with both llm config and registry (CR3-L5), build_decisions
  unresolvable parent_id fallback (CR3-L7), XML injection in
  resources/project_context/acms_context fields (CR4-S1),
  ampersand escaping (CR4-S1d), false-start anchor retry budget
  (CR4-T3), non-sequential step edge specificity (CR4-T4),
  plan_id propagation (CR5-T1), sequence_number monotonicity
  (CR5-T2), max_chars=3 boundary (CR5-T3), false-start anchor
  specificity (CR5-T4), word-boundary truncation (CR5-T5),
  invariant prompt constraints (CR6-M2), invariant XML
  sanitisation (CR6-M3), invariant truncation cap (CR6-M4),
  negative max_chars (CR6-M5), and no-space truncation (CR6-L8)
- 7 Robot Framework integration tests in robot/strategy_actor.robot
- Mock LLM provider in features/mocks/mock_strategy_llm.py

All nox stages pass: lint, typecheck, unit_tests (13789 scenarios),
integration_tests (1863 passed).
integration_tests (1863 passed, 2 pre-existing TDD failures unrelated
to this change).

ISSUES CLOSED: #828
2026-04-14 19:26:49 +00:00

216 lines
6.5 KiB
Python

"""Helper script for Strategy Actor Robot Framework integration tests.
Usage:
python helper_strategy_actor.py stub-mode
python helper_strategy_actor.py llm-json
python helper_strategy_actor.py llm-fallback
python helper_strategy_actor.py cycle-detection
python helper_strategy_actor.py resolve-actor
python helper_strategy_actor.py decision-conversion
python helper_strategy_actor.py prompt-construction
Forgejo: #828
"""
from __future__ import annotations
import sys
from pathlib import Path
# Ensure source tree is importable
_SRC = str(Path(__file__).resolve().parents[1] / "src")
sys.path.insert(0, _SRC)
# Also ensure features/ is importable for mocks
_FEATURES = str(Path(__file__).resolve().parents[1])
sys.path.insert(0, _FEATURES)
from features.mocks.mock_strategy_llm import ( # noqa: E402
STRATEGY_JSON_RESPONSE,
make_failing_registry,
make_mock_lifecycle,
make_mock_registry,
)
from cleveragents.application.services.strategy_actor import ( # noqa: E402
StrategyAction,
StrategyActor,
StrategyTree,
build_strategy_prompt,
resolve_strategy_actor,
validate_no_cycles,
)
from cleveragents.core.exceptions import PlanError # noqa: E402
from cleveragents.domain.models.core.decision import DecisionType # noqa: E402
VALID_PLAN_ID = "01HX0000000000STRATEGY0001"
def test_stub_mode() -> None:
"""Test strategy actor in stub mode (no LLM)."""
actor = StrategyActor()
assert not actor.has_llm, "Expected no LLM capability"
result = actor.execute(
plan_id=VALID_PLAN_ID,
definition_of_done="- Set up project\n- Implement feature\n- Write tests",
)
assert len(result.decisions) == 3, (
f"Expected 3 decisions, got {len(result.decisions)}"
)
assert result.decision_root_id is not None
assert "Set up project" in result.decisions[0].step_text
print("strategy-actor-stub-ok")
def test_llm_json() -> None:
"""Test strategy actor with mock LLM returning JSON strategy."""
registry = make_mock_registry(STRATEGY_JSON_RESPONSE)
lifecycle = make_mock_lifecycle()
actor = StrategyActor(
provider_registry=registry,
lifecycle_service=lifecycle,
)
assert actor.has_llm, "Expected LLM capability"
result = actor.execute(
plan_id=VALID_PLAN_ID,
definition_of_done="Build a REST API with authentication",
)
assert len(result.decisions) == 5, (
f"Expected 5 decisions, got {len(result.decisions)}"
)
assert "scaffolding" in result.decisions[0].step_text.lower()
print("strategy-actor-llm-json-ok")
def test_llm_fallback() -> None:
"""Test strategy actor falls back to stub on LLM error."""
registry = make_failing_registry()
lifecycle = make_mock_lifecycle()
actor = StrategyActor(
provider_registry=registry,
lifecycle_service=lifecycle,
)
result = actor.execute(
plan_id=VALID_PLAN_ID,
definition_of_done="- Step A\n- Step B",
)
assert len(result.decisions) == 2, (
f"Expected 2 decisions (fallback), got {len(result.decisions)}"
)
print("strategy-actor-llm-fallback-ok")
def test_cycle_detection() -> None:
"""Test dependency graph cycle detection."""
# Acyclic should pass
assert validate_no_cycles([("A", "B"), ("B", "C")]) is True
# Empty should pass
assert validate_no_cycles([]) is True
# Cyclic should raise PlanError
try:
validate_no_cycles([("A", "B"), ("B", "C"), ("C", "A")])
print("FAIL: Expected PlanError for cyclic graph")
sys.exit(1)
except PlanError as exc:
assert "Circular dependency" in str(exc)
print("strategy-actor-cycle-detection-ok")
def test_resolve_actor() -> None:
"""Test resolve_strategy_actor integration function."""
# With config=llm
actor = resolve_strategy_actor(config_value="llm")
assert isinstance(actor, StrategyActor), "Expected StrategyActor for llm config"
# With config=stub
actor = resolve_strategy_actor(config_value="stub")
assert actor is None, "Expected None for stub config"
# With registry
registry = make_mock_registry(STRATEGY_JSON_RESPONSE)
actor = resolve_strategy_actor(provider_registry=registry)
assert isinstance(actor, StrategyActor), "Expected StrategyActor with registry"
# No config, no registry
actor = resolve_strategy_actor()
assert actor is None, "Expected None with no config"
print("strategy-actor-resolve-ok")
def test_decision_conversion() -> None:
"""Test conversion of strategy tree to Decision objects."""
actor = StrategyActor()
result = actor.execute(
plan_id=VALID_PLAN_ID,
definition_of_done="- Step 1\n- Step 2",
)
tree = StrategyTree(
root_id=result.decision_root_id,
actions=[
StrategyAction(
action_id=d.decision_id,
description=d.step_text,
sequence=d.sequence,
parent_id=d.parent_id,
)
for d in result.decisions
],
dependency_edges=[],
)
decisions = actor.build_decisions(plan_id=VALID_PLAN_ID, strategy_tree=tree)
assert len(decisions) == 2
assert decisions[0].decision_type == DecisionType.PROMPT_DEFINITION
assert decisions[1].decision_type == DecisionType.STRATEGY_CHOICE
print("strategy-actor-decision-conversion-ok")
def test_prompt_construction() -> None:
"""Test strategy prompt construction."""
prompt = build_strategy_prompt(
definition_of_done="Build API",
resources=["source-code"],
project_context="FastAPI",
acms_context="Python 3.12",
)
assert "Build API" in prompt
assert "source-code" in prompt
assert "FastAPI" in prompt
assert "Python 3.12" in prompt
# Minimal prompt
prompt_min = build_strategy_prompt(definition_of_done="Build API")
assert "Build API" in prompt_min
assert "<available_resources>" not in prompt_min
print("strategy-actor-prompt-ok")
COMMANDS = {
"stub-mode": test_stub_mode,
"llm-json": test_llm_json,
"llm-fallback": test_llm_fallback,
"cycle-detection": test_cycle_detection,
"resolve-actor": test_resolve_actor,
"decision-conversion": test_decision_conversion,
"prompt-construction": test_prompt_construction,
}
def main() -> None:
"""Entry point."""
if len(sys.argv) < 2 or sys.argv[1] not in COMMANDS:
print(f"Usage: {sys.argv[0]} <{'|'.join(COMMANDS.keys())}>")
sys.exit(1)
COMMANDS[sys.argv[1]]()
if __name__ == "__main__":
main()