149feae15f
CI / quality (pull_request) Successful in 41s
CI / lint (pull_request) Successful in 46s
CI / integration_tests (pull_request) Successful in 1m0s
CI / build (pull_request) Successful in 1m2s
CI / typecheck (pull_request) Successful in 1m6s
CI / security (pull_request) Successful in 1m5s
CI / unit_tests (pull_request) Successful in 3m39s
CI / coverage (pull_request) Failing after 3m35s
CI / status-check (pull_request) Failing after 3s
CI / lint (push) Successful in 40s
CI / typecheck (push) Successful in 1m4s
CI / security (push) Successful in 1m4s
CI / quality (push) Successful in 32s
CI / integration_tests (push) Successful in 51s
CI / build (push) Successful in 45s
CI / unit_tests (push) Successful in 3m38s
CI / coverage (push) Failing after 3m34s
CI / status-check (push) Failing after 3s
Implement validate_dict() in cleveractors/validation.py and export it from
cleveractors/__init__.py as the first router-facing API (ADR-2024, Wave 1).
validate_dict(config_dict, platform_limits) -> dict[str, Any]:
- Validates that both 'agents' and 'routes' top-level keys are present.
- Validates each agent's type against the spec-defined vocabulary:
{llm, tool, composite, chain, template_instance}.
- Validates LLM agent 'provider' values against platform_limits['allowed_providers']
when present, or the default allowlist {openai, anthropic, google} when absent.
- Validates route/edge field names: 'from'/'to' are rejected in favour of
'source'/'target' as required by the spec-conformant format (ADR-2025).
- Validates graph node types against the spec vocabulary:
{start, end, agent, function, tool, conditional, subgraph, message_router}.
- Validates stream operator types against the spec vocabulary (all defined types).
- Enforces structural platform limits from platform_limits when present:
max_graph_depth (DFS longest-path check), max_subgraph_depth (recursive
subgraph-reference depth), max_total_nodes (total nodes across all graph routes).
- Returns config_dict unchanged when all checks pass (identity contract).
- Pure static validator: no file I/O, no env-var reads, no app construction.
Review fixes (Cycle 1):
- M3: Added isinstance(node_name, str) guard for graph node keys in
_validate_graph_route, with BDD scenario for non-string node key.
- m1: Added # pragma: no branch to _count_total_nodes defensive guard in
_limits.py (matching the identical guards in _subgraph_children).
- m2: Added # pragma: no branch to dead defensive continue branches in
_validate_graph_route edge validation loop.
- m3: Added dedicated BDD scenario and step for dangling source node
(src not in nodes_raw branch), plus renamed existing scenario from
"source node" to "target node" for accuracy.
- m4: Deleted 4 lines of commented-out InlineYAMLJinja setup code in
features/environment.py.
- n1: Added depth <= 0 ValueError guard in build_subgraph_chain helper.
- n2: Renamed scenario "Edge referencing a non-existent source node" to
"Edge referencing a non-existent target node" (the step tested a target).
- n3: Changed > to >= in both _MAX_DFS_STEPS and _MAX_SUBGRAPH_STEPS guards
to prevent off-by-one allowing one extra iteration beyond documented ceiling.
- n4: Added adjacency target deduplication in _compute_graph_depth to avoid
wasting DFS step budget on parallel edges.
- Spec review: M1 (entry_point optional) and M2 (start/end implicit injection)
were verified against the authoritative spec at docs/specification.md in the
cleveragents-webapp repo (develop branch). The spec does NOT support either
claim; entry_point is required and start/end must be explicitly declared.
No changes needed for M1/M2.
Review fixes (Cycle 2):
- M1: Deduplicated _subgraph_children yields with a seen set to prevent
duplicate subgraph references from consuming DFS steps redundantly and
causing false "too complex" errors.
- M2: Changed adjacency value type from list[str] to set[str] in
_compute_graph_depth for O(1) deduplication, reducing worst case
from O(N^3) to O(N^2) for dense graphs.
- m1: Added BDD scenario for LLM agent without provider field failing
when default "openai" is not in custom allowlist.
- m2: Added positive BDD scenarios for valid node types start, tool,
conditional, message_router, function.
- m3: Added positive BDD scenarios for additional stream operator types
filter, transform, reduce.
- m5: Added depth_cache memoization to _compute_subgraph_depth to avoid
recomputing depths across graph routes sharing subgraph trees.
- m6: Moved cleveractors.validation._limits import from module level into
after_scenario, guarded by hasattr.
- n1: Split validate_dict_routes_steps.py (was 473 lines) into
validate_dict_routes_steps.py (generic route steps) and
validate_dict_graph_route_steps.py (graph-specific steps).
- n2: Updated _MAX_DFS_STEPS module comment to accurately state O(N+E)
complexity and that dense graphs can exhaust the ceiling quickly.
BDD Behave scenarios cover all requirements.
Robot Framework integration tests (robot/validate_dict.robot) cover API usage.
ISSUES CLOSED: #10
878 lines
45 KiB
Gherkin
878 lines
45 KiB
Gherkin
Feature: validate_dict public API — spec-conformant Actor Configuration validation
|
|
As a router platform integrating cleveractors-core
|
|
I want to call validate_dict(config_dict, platform_limits) to validate actor YAML dicts
|
|
So that only spec-conformant configs are accepted before they are stored in the database
|
|
|
|
Background:
|
|
Given the validate_dict function is imported from cleveractors
|
|
|
|
# ── Top-level key presence ─────────────────────────────────────────────────
|
|
|
|
Scenario: Valid minimal config returns the dict unchanged
|
|
Given a minimal valid config dict with agents and routes
|
|
When I call validate_dict with empty platform_limits
|
|
Then the function returns the config dict unchanged
|
|
And no exception is raised
|
|
|
|
Scenario: Config missing "agents" key raises ConfigurationError
|
|
Given a config dict without an "agents" key
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions "agents"
|
|
|
|
Scenario: Config missing "routes" key raises ConfigurationError
|
|
Given a config dict without a "routes" key
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions "routes"
|
|
|
|
# ── Agent type validation ──────────────────────────────────────────────────
|
|
|
|
Scenario: Config with a valid llm agent type passes validation
|
|
Given a config dict with an agent of type "llm"
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
Scenario: Config with a valid tool agent type passes validation
|
|
Given a config dict with an agent of type "tool"
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
Scenario: Config with a valid composite agent type passes validation
|
|
Given a config dict with an agent of type "composite"
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
Scenario: Config with a valid chain agent type passes validation
|
|
Given a config dict with an agent of type "chain"
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
Scenario: Config with a valid template_instance agent type passes validation
|
|
Given a config dict with an agent of type "template_instance"
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
Scenario: Config with an unknown agent type raises ConfigurationError
|
|
Given a config dict with an agent of type "unknown_type"
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions agent type validation
|
|
|
|
# ── LLM provider validation ────────────────────────────────────────────────
|
|
|
|
Scenario: LLM agent with openai provider passes default allowlist validation
|
|
Given a config dict with an llm agent using provider "openai"
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
Scenario: LLM agent with anthropic provider passes default allowlist validation
|
|
Given a config dict with an llm agent using provider "anthropic"
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
Scenario: LLM agent with google provider passes default allowlist validation
|
|
Given a config dict with an llm agent using provider "google"
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
Scenario: LLM agent with unknown provider and no allowlist raises ConfigurationError
|
|
Given a config dict with an llm agent using provider "unknown_provider"
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions provider validation
|
|
|
|
Scenario: LLM agent provider matches custom allowed_providers list
|
|
Given a config dict with an llm agent using provider "groq"
|
|
When I call validate_dict with platform_limits containing allowed_providers "groq,openai"
|
|
Then no exception is raised
|
|
|
|
Scenario: LLM agent provider not in custom allowed_providers raises ConfigurationError
|
|
Given a config dict with an llm agent using provider "anthropic"
|
|
When I call validate_dict with platform_limits containing allowed_providers "openai"
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions provider validation
|
|
|
|
Scenario: LLM agent without explicit provider uses default openai and passes
|
|
Given a config dict with an llm agent without a provider field
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
# ── Route / edge / node / operator validation ──────────────────────────────
|
|
|
|
Scenario: Graph route with source/target edge fields passes validation
|
|
Given a config dict with a graph route using source/target edges
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
Scenario: Graph route edge using "from" field raises ConfigurationError
|
|
Given a config dict with a graph route edge using "from" instead of "source"
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions edge field names
|
|
|
|
Scenario: Graph route edge using "to" field raises ConfigurationError
|
|
Given a config dict with a graph route edge using "to" instead of "target"
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions edge field names
|
|
|
|
Scenario: Graph route with valid node type "agent" passes validation
|
|
Given a config dict with a graph route containing a node of type "agent"
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
Scenario: Graph route with invalid node type raises ConfigurationError
|
|
Given a config dict with a graph route containing a node of type "invalid_node"
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions node type validation
|
|
|
|
Scenario: Stream route with valid operator type "map" passes validation
|
|
Given a config dict with a stream route containing a "map" operator
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
Scenario: Stream route with invalid operator type raises ConfigurationError
|
|
Given a config dict with a stream route containing an invalid operator type
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions operator type validation
|
|
|
|
# ── Structural limit enforcement ───────────────────────────────────────────
|
|
|
|
Scenario: Graph within max_graph_depth limit passes validation
|
|
Given a config dict with a graph route of 4 edges
|
|
When I call validate_dict with platform_limits max_graph_depth of 5
|
|
Then no exception is raised
|
|
|
|
Scenario: Graph exceeding max_graph_depth raises ConfigurationError
|
|
Given a config dict with a graph route of 4 edges
|
|
When I call validate_dict with platform_limits max_graph_depth of 3
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions graph depth
|
|
|
|
Scenario: Actor within max_total_nodes limit passes validation
|
|
Given a config dict with a graph route containing 3 non-end nodes
|
|
When I call validate_dict with platform_limits max_total_nodes of 10
|
|
Then no exception is raised
|
|
|
|
Scenario: Actor exceeding max_total_nodes raises ConfigurationError
|
|
Given a config dict with a graph route containing 5 non-end nodes
|
|
When I call validate_dict with platform_limits max_total_nodes of 3
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions total nodes
|
|
|
|
Scenario: Nested subgraph within max_subgraph_depth limit passes validation
|
|
Given a config dict with nested subgraphs of depth 2
|
|
When I call validate_dict with platform_limits max_subgraph_depth of 3
|
|
Then no exception is raised
|
|
|
|
Scenario: Nested subgraph exceeding max_subgraph_depth raises ConfigurationError
|
|
Given a config dict with nested subgraphs of depth 4
|
|
When I call validate_dict with platform_limits max_subgraph_depth of 3
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions subgraph depth
|
|
|
|
# ── Return value contract ──────────────────────────────────────────────────
|
|
|
|
Scenario: Valid config is returned unchanged (identity contract)
|
|
Given a fully valid spec-conformant config dict
|
|
When I call validate_dict with empty platform_limits
|
|
Then the returned dict is the same object as the input dict
|
|
|
|
# ── Callable without file/env/app side effects ────────────────────────────
|
|
|
|
Scenario: Bridge route type passes validation
|
|
Given a config dict with a valid bridge route
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
# ── Case-insensitive validation ────────────────────────────────────────────
|
|
|
|
Scenario: LLM agent with provider "OpenAI" passes default allowlist
|
|
Given a config dict with an llm agent using provider "OpenAI" in mixed case
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
Scenario: Graph route with node type "Agent" passes validation
|
|
Given a config dict with a graph route containing a node of type "Agent" in mixed case
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
# ── Boundary-exact structural limit tests ──────────────────────────────────
|
|
|
|
Scenario: Graph at exact max_graph_depth limit passes validation
|
|
Given a config dict with a graph route of exactly 5 edges for boundary test
|
|
When I call validate_dict with platform_limits max_graph_depth of 5
|
|
Then no exception is raised
|
|
|
|
Scenario: Actor at exact max_total_nodes limit passes validation
|
|
Given a config dict with a graph route containing exactly 5 non-end nodes for boundary test
|
|
When I call validate_dict with platform_limits max_total_nodes of 6
|
|
Then no exception is raised
|
|
|
|
Scenario: Nested subgraph at exact max_subgraph_depth limit passes validation
|
|
Given a config dict with nested subgraphs of exactly 3 levels for boundary test
|
|
When I call validate_dict with platform_limits max_subgraph_depth of 3
|
|
Then no exception is raised
|
|
|
|
# ── Zero-value boundary tests ──────────────────────────────────────────────
|
|
|
|
Scenario: max_graph_depth=0 raises ConfigurationError when graph has depth 1
|
|
Given a config dict with a graph route of 1 edge
|
|
When I call validate_dict with platform_limits max_graph_depth of 0
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions graph depth
|
|
|
|
Scenario: max_total_nodes=0 raises ConfigurationError when config has any nodes
|
|
Given a config dict with a graph route containing 1 non-end node
|
|
When I call validate_dict with platform_limits max_total_nodes of 0
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions total nodes
|
|
|
|
Scenario: max_subgraph_depth=0 raises ConfigurationError when config has a subgraph
|
|
Given a config dict with nested subgraphs of depth 1
|
|
When I call validate_dict with platform_limits max_subgraph_depth of 0
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions subgraph depth
|
|
|
|
# ── Empty collection boundary tests ────────────────────────────────────────
|
|
|
|
Scenario: Empty agents dict with valid routes passes validation
|
|
Given a config dict with empty agents and valid routes
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
Scenario: Empty routes dict with valid agents passes validation
|
|
Given a config dict with valid agents and empty routes
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
Scenario: Empty allowed_providers list raises ConfigurationError for LLM agent
|
|
Given a config dict with an llm agent using provider "openai"
|
|
When I call validate_dict with platform_limits containing allowed_providers ""
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions provider validation
|
|
|
|
# ── Negative platform limits rejected ──────────────────────────────────────
|
|
|
|
Scenario: Negative max_graph_depth raises ConfigurationError
|
|
Given a config dict with a graph route of 1 edge
|
|
When I call validate_dict with platform_limits max_graph_depth of -1
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions platform_limits type validation
|
|
|
|
Scenario: Negative max_total_nodes raises ConfigurationError
|
|
Given a config dict with a graph route containing 1 non-end node
|
|
When I call validate_dict with platform_limits max_total_nodes of -1
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions platform_limits type validation
|
|
|
|
Scenario: Negative max_subgraph_depth raises ConfigurationError
|
|
Given a config dict with nested subgraphs of depth 1
|
|
When I call validate_dict with platform_limits max_subgraph_depth of -1
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions platform_limits type validation
|
|
|
|
# ── Crash-bug regression tests ─────────────────────────────────────────────
|
|
|
|
Scenario: Non-string LLM provider raises ConfigurationError instead of crashing
|
|
Given a config dict with an llm agent using a non-string provider value
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions non-string provider
|
|
|
|
Scenario: Non-string node type raises ConfigurationError instead of crashing
|
|
Given a config dict with a graph route containing a non-string node type
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions non-string node type
|
|
|
|
Scenario: Edge missing source and target fields raises ConfigurationError
|
|
Given a config dict with a graph route where an edge lacks source and target
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions source and target fields
|
|
|
|
Scenario: Graph route missing nodes key raises ConfigurationError
|
|
Given a config dict with a graph route missing the nodes key
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions "nodes"
|
|
|
|
Scenario: Graph route missing edges key raises ConfigurationError
|
|
Given a config dict with a graph route missing the edges key
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions "edges"
|
|
|
|
Scenario: Stream operator missing type field raises ConfigurationError
|
|
Given a config dict with a stream route containing an operator missing type
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions missing type
|
|
|
|
Scenario: Bare string as allowed_providers raises ConfigurationError
|
|
Given a config dict with platform_limits allowed_providers as a bare string "openai"
|
|
When I call validate_dict with platform_limits containing the bare string
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions allowed_providers type validation
|
|
|
|
Scenario: Non-int max_graph_depth raises ConfigurationError
|
|
Given a config dict with platform_limits containing a non-int "max_graph_depth" value
|
|
When I call validate_dict with platform_limits containing a non-int limit value
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions platform_limits type validation
|
|
|
|
Scenario: agents section that is not a dict raises ConfigurationError
|
|
Given a config dict where agents is a list instead of a mapping
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
|
|
Scenario: routes section that is not a dict raises ConfigurationError
|
|
Given a config dict where routes is a list instead of a mapping
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
|
|
Scenario: Agent definition that is not a dict raises ConfigurationError
|
|
Given a config dict where an agent definition is a string instead of a mapping
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
|
|
Scenario: Agent with agent_template key but no type skips type validation
|
|
Given a config dict with an agent that has an agent_template key instead of type
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
Scenario: Agent with no type and no template key raises ConfigurationError
|
|
Given a config dict with an agent that has no type and no template key
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
|
|
Scenario: Agent with template key but no type skips type validation
|
|
Given a config dict with an agent that has a template key instead of type
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
Scenario: LLM agent with non-dict config raises ConfigurationError
|
|
Given a config dict with an llm agent that has a non-dict config
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions config must be a mapping
|
|
|
|
Scenario: Route definition that is not a dict raises ConfigurationError
|
|
Given a config dict where a route definition is a string instead of a mapping
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
|
|
Scenario: Route with unknown type raises ConfigurationError
|
|
Given a config dict with a route of unknown type
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
|
|
Scenario: Graph route with non-list edges raises ConfigurationError
|
|
Given a config dict with a graph route where edges is not a list
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions "edges"
|
|
|
|
Scenario: Graph route with non-dict edge entry raises ConfigurationError
|
|
Given a config dict with a graph route where one edge is a string
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions edge must be a mapping
|
|
|
|
Scenario: Graph route with non-dict node entry raises ConfigurationError
|
|
Given a config dict with a graph route where one node is a string
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
|
|
Scenario: Graph route with non-dict nodes section raises ConfigurationError
|
|
Given a config dict with a graph route where nodes is not a dict
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions "nodes"
|
|
|
|
Scenario: Stream route with non-list operators raises ConfigurationError
|
|
Given a config dict with a stream route where operators is not a list
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
|
|
Scenario: Stream route with non-dict operator entry raises ConfigurationError
|
|
Given a config dict with a stream route where one operator is a string
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions operator must be a mapping
|
|
|
|
Scenario: Stream operator with empty type string raises ConfigurationError
|
|
Given a config dict with a stream route where one operator has an empty type
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions operator type validation
|
|
|
|
Scenario: Route with non-dict value raises ConfigurationError before structural limits
|
|
Given a config dict where a route value is not a dict for structural limits
|
|
When I call validate_dict with platform_limits max_graph_depth of 5
|
|
Then a ConfigurationError is raised
|
|
|
|
Scenario: Route with non-dict value raises ConfigurationError before total nodes count
|
|
Given a config dict with a non-dict route value for total nodes counting
|
|
When I call validate_dict with platform_limits max_total_nodes of 100
|
|
Then a ConfigurationError is raised
|
|
|
|
Scenario: Graph depth raises ConfigurationError when nodes field is not a dict
|
|
Given a config dict with a graph route that has non-dict nodes for depth calculation
|
|
When I call validate_dict with platform_limits max_graph_depth of 5
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions "nodes"
|
|
|
|
Scenario: Graph depth check rejects disconnected entry_point
|
|
Given a config dict with a graph route whose entry_point does not match any node
|
|
When I call validate_dict with platform_limits max_graph_depth of 5
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions entry_point not found
|
|
|
|
Scenario: Graph depth limit handles cycles gracefully
|
|
Given a config dict with a graph route containing a cycle
|
|
When I call validate_dict with platform_limits max_graph_depth of 5
|
|
Then no exception is raised
|
|
|
|
Scenario: Graph depth calculation raises ConfigurationError for non-dict edge entries
|
|
Given a config dict with a graph route where edges list contains a non-dict entry
|
|
When I call validate_dict with platform_limits max_graph_depth of 5
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions edge must be a mapping
|
|
|
|
Scenario: Subgraph depth cycle guard prevents infinite recursion
|
|
Given a config dict with two routes that reference each other as subgraphs
|
|
When I call validate_dict with platform_limits max_subgraph_depth of 10
|
|
Then no exception is raised
|
|
|
|
Scenario: Subgraph depth raises ConfigurationError for non-existent subgraph reference
|
|
Given a config dict with a subgraph node referencing a missing route
|
|
When I call validate_dict with platform_limits max_subgraph_depth of 5
|
|
Then a ConfigurationError is raised
|
|
|
|
Scenario: Graph route with non-dict node definitions raises ConfigurationError before subgraph depth check
|
|
Given a config dict with a graph route containing non-dict node values
|
|
When I call validate_dict with platform_limits max_subgraph_depth of 5
|
|
Then a ConfigurationError is raised
|
|
|
|
Scenario: Graph route with non-dict nodes field raises ConfigurationError before subgraph depth check
|
|
Given a config dict with a graph route that has a non-dict nodes field and subgraph limit
|
|
When I call validate_dict with platform_limits max_subgraph_depth of 5
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions "nodes"
|
|
|
|
# ── Strengthened error message assertions ──────────────────────────────────
|
|
|
|
Scenario: Non-string agent type raises ConfigurationError with specific message
|
|
Given a config dict with an agent that has a non-string type
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions non-string agent type
|
|
|
|
Scenario: Non-string route type raises ConfigurationError with specific message
|
|
Given a config dict with a route that has a non-string type
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions non-string route type
|
|
|
|
Scenario: Subgraph node with non-string reference raises ConfigurationError
|
|
Given a config dict with a subgraph node referencing a missing route via a list
|
|
When I call validate_dict with platform_limits max_subgraph_depth of 5
|
|
Then a ConfigurationError is raised
|
|
|
|
# ── Cycle 3 fixes: entry_point, edge type, subgraph validation ────────────
|
|
|
|
Scenario: Graph route missing entry_point raises ConfigurationError
|
|
Given a config dict with a graph route that has no entry_point field
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions "entry_point"
|
|
|
|
Scenario: Graph route with non-string entry_point raises ConfigurationError in route validation
|
|
Given a config dict with a graph route whose entry_point is not a string
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions "entry_point"
|
|
|
|
Scenario: Graph route edge with non-string source raises ConfigurationError
|
|
Given a config dict with a graph route where an edge source is a list
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions source and target fields
|
|
|
|
Scenario: Graph route node with empty string subgraph reference raises ConfigurationError
|
|
Given a config dict with a graph route containing a subgraph node with empty reference
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions subgraph reference
|
|
|
|
Scenario: Graph route node with non-string subgraph reference raises ConfigurationError
|
|
Given a config dict with a graph route containing a subgraph node with list reference
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions subgraph reference
|
|
|
|
Scenario: Route missing type field entirely raises ConfigurationError
|
|
Given a config dict with a route that has no type field
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions "type"
|
|
|
|
# ── Cycle 4 fixes: empty-string source/target, empty entry_point ──────────
|
|
|
|
Scenario: Graph route edge with empty string source raises ConfigurationError
|
|
Given a config dict with a graph route where an edge has an empty source
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions source and target fields
|
|
|
|
Scenario: Graph route edge with empty string target raises ConfigurationError
|
|
Given a config dict with a graph route where an edge has an empty target
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions source and target fields
|
|
|
|
Scenario: Graph route with empty entry_point string raises ConfigurationError
|
|
Given a config dict with a graph route that has an empty entry_point string
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions "entry_point"
|
|
|
|
# ── Cycle 4: subgraph pointing to non-graph route ─────────────────────────
|
|
|
|
Scenario: Subgraph node referencing a stream route raises ConfigurationError
|
|
Given a config dict with a subgraph node referencing a stream route
|
|
When I call validate_dict with platform_limits max_subgraph_depth of 5
|
|
Then a ConfigurationError is raised
|
|
|
|
Scenario: Subgraph node referencing a bridge route raises ConfigurationError
|
|
Given a config dict with a subgraph node referencing a bridge route
|
|
When I call validate_dict with platform_limits max_subgraph_depth of 5
|
|
Then a ConfigurationError is raised
|
|
|
|
# ── Coverage gap: non-string stream operator type ─────────────────────────
|
|
|
|
Scenario: Stream operator with non-string type raises ConfigurationError
|
|
Given a config dict with a stream route where one operator has a non-string type
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions non-string operator type
|
|
|
|
# ── Coverage gap: non-string element in allowed_providers ──────────────────
|
|
|
|
Scenario: Allowed_providers list with a non-string element raises ConfigurationError
|
|
Given a config dict with llm agent and allowed_providers containing a non-string element
|
|
When I call validate_dict with platform_limits containing the mixed-type list
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions allowed_providers type validation
|
|
|
|
# ── Coverage gap: invalid types on max_total_nodes / max_subgraph_depth ────
|
|
|
|
Scenario: Boolean True as max_total_nodes raises ConfigurationError
|
|
Given a config dict with a graph route containing 3 non-end nodes
|
|
When I call validate_dict with platform_limits containing bool max_total_nodes
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions platform_limits type validation
|
|
|
|
Scenario: Non-int string as max_total_nodes raises ConfigurationError
|
|
Given a config dict with a graph route containing 3 non-end nodes
|
|
When I call validate_dict with platform_limits containing string max_total_nodes
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions platform_limits type validation
|
|
|
|
Scenario: Boolean True as max_subgraph_depth raises ConfigurationError
|
|
Given a config dict with nested subgraphs of depth 2
|
|
When I call validate_dict with platform_limits containing bool max_subgraph_depth
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions platform_limits type validation
|
|
|
|
Scenario: Non-int string as max_subgraph_depth raises ConfigurationError
|
|
Given a config dict with nested subgraphs of depth 2
|
|
When I call validate_dict with platform_limits containing string max_subgraph_depth
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions platform_limits type validation
|
|
|
|
# ── Crash-bug regression: non-string entry_point ────────────────────────────
|
|
|
|
Scenario: Non-string entry_point raises ConfigurationError instead of crashing
|
|
Given a config dict with a graph route whose entry_point is not a string
|
|
When I call validate_dict with platform_limits max_graph_depth of 5
|
|
Then a ConfigurationError is raised
|
|
|
|
# ── Crash-bug regression: allowed_providers as dict ─────────────────────────
|
|
|
|
Scenario: Dict as allowed_providers raises ConfigurationError
|
|
Given a config dict with platform_limits containing a dict for allowed_providers
|
|
When I call validate_dict with platform_limits containing a dict as allowed_providers
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions allowed_providers type validation
|
|
|
|
# ── Crash-bug regression: boolean platform_limits ───────────────────────────
|
|
|
|
Scenario: Boolean True as max_graph_depth raises ConfigurationError
|
|
Given a config dict with platform_limits containing a bool for max_graph_depth
|
|
When I call validate_dict with platform_limits containing a bool as max_graph_depth
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions platform_limits type validation
|
|
|
|
# ── Cycle 5: None argument guards ─────────────────────────────────────────
|
|
|
|
Scenario: validate_dict with None config_dict raises ConfigurationError
|
|
When I call validate_dict with config_dict set to None and empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions config_dict and dict
|
|
|
|
Scenario: validate_dict with None platform_limits raises ConfigurationError
|
|
Given a minimal valid config dict with agents and routes
|
|
When I call validate_dict with platform_limits set to None
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions platform_limits and dict
|
|
|
|
# ── Cycle 5: missing node type key ────────────────────────────────────────
|
|
|
|
Scenario: Graph node missing type field produces clear error message
|
|
Given a config dict with a graph route where a node is missing the type key
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions missing type
|
|
|
|
# ── Cycle 5: ConfigurationError exported from top-level package ───────────
|
|
|
|
Scenario: ConfigurationError is importable from the top-level cleveractors package
|
|
When I import ConfigurationError from the cleveractors package
|
|
Then the import succeeds and ConfigurationError is a class
|
|
|
|
Scenario: ConfigurationError is listed in cleveractors.__all__
|
|
When I check the cleveractors package __all__
|
|
Then ConfigurationError is listed in the __all__
|
|
|
|
# ── Cycle 5: non-dict config_dict and platform_limits ──────────────────────
|
|
|
|
Scenario: validate_dict with list config_dict raises ConfigurationError
|
|
When I call validate_dict with config_dict set to an empty list and empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions config_dict and dict
|
|
|
|
Scenario: validate_dict with string platform_limits raises ConfigurationError
|
|
Given a minimal valid config dict with agents and routes
|
|
When I call validate_dict with platform_limits set to a string
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions platform_limits and dict
|
|
|
|
# ── Cycle 7: DFS step guard branches ──────────────────────────────────────
|
|
|
|
Scenario: DFS step guard in _compute_graph_depth raises ConfigurationError
|
|
Given I monkey-patch _MAX_DFS_STEPS to 3
|
|
And a config dict with a graph route of 10 edges
|
|
When I call validate_dict with platform_limits max_graph_depth of 100
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions too complex to validate depth
|
|
|
|
Scenario: Subgraph step guard in _compute_subgraph_depth raises ConfigurationError
|
|
Given I monkey-patch _MAX_SUBGRAPH_STEPS to 3
|
|
And a config dict with nested subgraphs of depth 4
|
|
When I call validate_dict with platform_limits max_subgraph_depth of 10
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions too complex to validate
|
|
|
|
# ── Cycle 7: Stronger platform_limits error assertions ────────────────────
|
|
|
|
Scenario: Negative limit error message mentions "must be >= 0"
|
|
Given a config dict with a graph route containing 1 non-end node
|
|
When I call validate_dict with platform_limits max_total_nodes of -1
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions "must be >= 0"
|
|
|
|
Scenario: Type error limit message mentions "must be an integer"
|
|
Given a config dict with a graph route containing 3 non-end nodes
|
|
When I call validate_dict with platform_limits containing string max_total_nodes
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions "must be an integer"
|
|
|
|
# ── Cycle 8: Subgraph reference validation without max_subgraph_depth ─────
|
|
|
|
Scenario: Subgraph referencing missing route raises error without max_subgraph_depth
|
|
Given a config dict with a subgraph node referencing a missing route without max_subgraph_depth
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
|
|
Scenario: Subgraph referencing stream route raises error without max_subgraph_depth
|
|
Given a config dict with a subgraph node referencing a stream route without max_subgraph_depth
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
|
|
Scenario: Subgraph referencing bridge route raises error without max_subgraph_depth
|
|
Given a config dict with a subgraph node referencing a bridge route without max_subgraph_depth
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
|
|
# ── Cycle 8: Edge endpoint validation against declared nodes ─────────────
|
|
|
|
Scenario: Edge referencing a non-existent target node raises ConfigurationError
|
|
Given a config dict with a graph route where an edge references a non-existent node
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions edge endpoint not declared
|
|
|
|
Scenario: Edge referencing a non-existent source node raises ConfigurationError
|
|
Given a config dict with a graph route where an edge source references a non-existent node
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions edge endpoint not declared
|
|
|
|
# ── Cycle 8: LLM provider whitespace stripping ───────────────────────────
|
|
|
|
Scenario: LLM agent provider with surrounding whitespace passes validation
|
|
Given a config dict with an llm agent using provider " openai " with whitespace
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
# ── Review fix: reserved agent names (§4.2) ─────────────────────────────
|
|
|
|
Scenario: Agent name beginning with double underscore raises ConfigurationError
|
|
Given a config dict with an agent named "__internal_agent"
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions reserved agent name
|
|
|
|
# ── Review fix: reserved route names (§5.1) ─────────────────────────────
|
|
|
|
Scenario: Route name beginning with double underscore raises ConfigurationError
|
|
Given a config dict with a route named "__input__"
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions reserved route name
|
|
|
|
# ── Review fix: route type explicitly set to None ──────────────────────
|
|
|
|
Scenario: Route with type set to None raises ConfigurationError
|
|
Given a config dict with a route whose type is null
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions "type"
|
|
|
|
# ── Review fix: allowed_providers as set and frozenset ──────────────────
|
|
|
|
Scenario: Allowed_providers as a Python set passes validation
|
|
Given a config dict with an llm agent using provider "anthropic"
|
|
When I call validate_dict with an allowed_providers set containing anthropic and openai
|
|
Then no exception is raised
|
|
|
|
Scenario: Allowed_providers as a Python frozenset passes validation
|
|
Given a config dict with an llm agent using provider "anthropic"
|
|
When I call validate_dict with an allowed_providers frozenset containing anthropic and openai
|
|
Then no exception is raised
|
|
|
|
# ── Review fix: max_total_nodes aggregation across multiple graph routes ──
|
|
|
|
Scenario: max_total_nodes aggregates across multiple graph routes
|
|
Given a config dict with two graph routes of 3 nodes each
|
|
When I call validate_dict with platform_limits max_total_nodes of 5
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions total nodes
|
|
|
|
Scenario: max_total_nodes at exact limit across multiple graph routes
|
|
Given a config dict with two graph routes of 3 nodes each
|
|
When I call validate_dict with platform_limits max_total_nodes of 6
|
|
Then no exception is raised
|
|
|
|
# ── Review fix: allowed_providers as a tuple ────────────────────────────
|
|
|
|
Scenario: Allowed_providers as a Python tuple passes validation
|
|
Given a config dict with an llm agent using provider "anthropic"
|
|
When I call validate_dict with an allowed_providers tuple containing anthropic and openai
|
|
Then no exception is raised
|
|
|
|
# ── Review fix: empty operators list on stream route ───────────────────
|
|
|
|
Scenario: Stream route with empty operators list passes validation
|
|
Given a config dict with a stream route containing no operators
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
# ── Review fix: non-string agent key guard ─────────────────────────────
|
|
|
|
Scenario: Non-string key in agents dict raises ConfigurationError
|
|
Given a config dict with a non-string agent key
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions agent key must be a string
|
|
|
|
# ── Review fix: non-string route key guard ─────────────────────────────
|
|
|
|
Scenario: Non-string key in routes dict raises ConfigurationError
|
|
Given a config dict with a non-string route key
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions route key must be a string
|
|
|
|
# ── Review fix: non-string graph node key guard ──────────────────────
|
|
|
|
Scenario: Non-string key in graph route nodes raises ConfigurationError
|
|
Given a config dict with a graph route that has a non-string node key
|
|
When I call validate_dict with empty platform_limits
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions node key must be a string
|
|
|
|
# ── Review Cycle 2: LLM default provider exclusion test ──────────────
|
|
|
|
Scenario: LLM agent without provider fails when default openai not in allowed_providers
|
|
Given a config dict with an llm agent without a provider field
|
|
When I call validate_dict with platform_limits containing allowed_providers "anthropic,google"
|
|
Then a ConfigurationError is raised
|
|
And the error message mentions provider validation
|
|
|
|
# ── Review Cycle 2: positive valid node type coverage ─────────────────
|
|
|
|
Scenario: Graph route with valid node type "start" passes validation
|
|
Given a config dict with a graph route containing a node of type "start"
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
Scenario: Graph route with valid node type "tool" passes validation
|
|
Given a config dict with a graph route containing a node of type "tool"
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
Scenario: Graph route with valid node type "conditional" passes validation
|
|
Given a config dict with a graph route containing a node of type "conditional"
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
Scenario: Graph route with valid node type "message_router" passes validation
|
|
Given a config dict with a graph route containing a node of type "message_router"
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
Scenario: Graph route with valid node type "function" passes validation
|
|
Given a config dict with a graph route containing a node of type "function"
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
# ── Review Cycle 2: positive valid operator type coverage ─────────────
|
|
|
|
Scenario: Stream route with valid operator type "filter" passes validation
|
|
Given a config dict with a stream route containing a "filter" operator
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
Scenario: Stream route with valid operator type "transform" passes validation
|
|
Given a config dict with a stream route containing a "transform" operator
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
Scenario: Stream route with valid operator type "reduce" passes validation
|
|
Given a config dict with a stream route containing a "reduce" operator
|
|
When I call validate_dict with empty platform_limits
|
|
Then no exception is raised
|
|
|
|
# ── Review Cycle 2: duplicate subgraph reference de-duplication ───────
|
|
|
|
Scenario: Route with duplicate subgraph references does not cause false too-complex error
|
|
Given a config dict with a graph route containing two subgraph nodes referencing the same route
|
|
When I call validate_dict with platform_limits max_subgraph_depth of 5
|
|
Then no exception is raised |