diff --git a/docs/adr/ADR-010-actor-and-agent-architecture.md b/docs/adr/ADR-010-actor-and-agent-architecture.md index 8b31f6f4a..cdfab0d74 100644 --- a/docs/adr/ADR-010-actor-and-agent-architecture.md +++ b/docs/adr/ADR-010-actor-and-agent-architecture.md @@ -125,6 +125,7 @@ Agent behavior is configurable without code changes: prompt templates, tool sets | [ADR-022](ADR-022-langchain-langgraph-integration.md) | LangChain/LangGraph Integration | Actors are implemented as LangGraph StateGraph instances | | [ADR-030](ADR-030-skill-abstraction-definition.md) | Skill Abstraction Definition | Skills are the unit of capability assignment to actors | | [ADR-031](ADR-031-actor-abstraction-definition.md) | Actor Abstraction Definition | Formalizes the canonical definition of what an actor is | +| [ADR-032](ADR-032-jinja2-yaml-template-preprocessing.md) | Jinja2 YAML Template Preprocessing | Defines the two-phase Jinja2 + env var preprocessing pipeline for actor YAML files | ## Acceptance diff --git a/docs/adr/ADR-031-actor-abstraction-definition.md b/docs/adr/ADR-031-actor-abstraction-definition.md index 664ee51b0..88a26db21 100644 --- a/docs/adr/ADR-031-actor-abstraction-definition.md +++ b/docs/adr/ADR-031-actor-abstraction-definition.md @@ -179,6 +179,7 @@ There is no mechanism for an actor to gain tool capabilities outside the skill s | [ADR-028](ADR-028-agent-skills-standard.md) | Agent Skills Standard (AgentSkills.io) | Agent Skills appear as tool nodes in actor graphs and extend actor knowledge | | [ADR-029](ADR-029-model-context-protocol.md) | Model Context Protocol (MCP) Adoption | MCP tools appear as tool nodes in actor graphs via skill composition | | [ADR-030](ADR-030-skill-abstraction-definition.md) | Skill Abstraction Definition | Skills are the unit of capability assignment to actors; the skill-actor binding model | +| [ADR-032](ADR-032-jinja2-yaml-template-preprocessing.md) | Jinja2 YAML Template Preprocessing | Defines how actor YAML files are preprocessed with Jinja2 templates and environment variable interpolation before parsing | ## Acceptance diff --git a/docs/adr/ADR-032-jinja2-yaml-template-preprocessing.md b/docs/adr/ADR-032-jinja2-yaml-template-preprocessing.md new file mode 100644 index 000000000..4eccd10a3 --- /dev/null +++ b/docs/adr/ADR-032-jinja2-yaml-template-preprocessing.md @@ -0,0 +1,210 @@ +# ADR-032: Jinja2 YAML Template Preprocessing + +**Status:** Accepted +**Date:** 2026-02-19 +**Supersedes:** None +**Author(s):** Jeffrey Phillips Freeman +**Approver(s):** Jeffrey Phillips Freeman + +## Context + +CleverAgents actor configuration files are written in YAML and define complex multi-actor workflows with system prompts, routing logic, and shared context. Many real-world configurations require dynamic content in system prompts — for example, a brainstorming actor needs to reference paper requirements collected during a prior discovery stage, or a vetting actor needs to conditionally change its behavior based on whether auto-finish mode is active. + +Standard YAML offers no native mechanism for dynamic string interpolation, conditional blocks, or iterative generation. Without a template preprocessor, authors would need to either: (a) hardcode all prompt content statically, severely limiting reusability; (b) write custom Python code for each dynamic configuration, undermining the declarative YAML-only philosophy; or (c) rely on runtime string replacement, which cannot express conditionals, loops, or filters. + +Furthermore, actor configurations frequently need environment variable substitution (e.g., API keys, directory paths) with default fallbacks and automatic type coercion, which YAML also lacks natively. + +The v2 system already used Jinja2 templating in its reactive configuration files. Preserving and formalizing this capability ensures backward compatibility and provides a proven, well-understood solution for dynamic YAML content. + +## Decision + +CleverAgents adopts a **two-phase YAML processing pipeline** for actor configuration files: + +1. **Phase 1 — Jinja2 Template Rendering**: Before YAML parsing, the raw file content is processed through a sandboxed Jinja2 template engine. This phase resolves `{{ variable }}` expressions, `{% if/for/block %}` control structures, and `{# comment #}` blocks into static YAML text. + +2. **Phase 2 — YAML Parsing and Environment Variable Interpolation**: The rendered YAML text is parsed by PyYAML's `safe_load`, then all string values matching `${VAR}` or `${VAR:default}` patterns are recursively replaced with environment variable values, with automatic type coercion for booleans, integers, and floats. + +This two-phase approach cleanly separates template-time logic (Jinja2, evaluated once at load time with a context dictionary) from runtime configuration (environment variables, evaluated at parse time from the OS environment). + +## Design + +### Phase 1: Jinja2 Template Engine (`YAMLTemplateEngine`) + +The `YAMLTemplateEngine` class provides the Jinja2 preprocessing layer: + +#### Sandboxed Execution + +All template rendering uses `jinja2.sandbox.SandboxedEnvironment`, which prevents templates from executing arbitrary Python code, accessing the filesystem, or performing other unsafe operations. The sandbox restricts attribute access and method calls to a safe subset. + +#### Template Delimiters + +Standard Jinja2 delimiters are used: + +| Delimiter | Purpose | Example | +|-----------|---------|---------| +| `{{ ... }}` | Variable expression | `{{ context.paper_details.topic }}` | +| `{% ... %}` | Block statement (if, for, block, etc.) | `{% if context.auto_finish_active %}` | +| `{# ... #}` | Comment (stripped from output) | `{# This is a template comment #}` | + +#### Custom Filters + +The engine registers four custom Jinja2 filters for YAML-specific use: + +| Filter | Purpose | Example | +|--------|---------|---------| +| `yaml` | Serializes a value to YAML string | `{{ my_dict \| yaml }}` | +| `indent` | Indents text by N spaces | `{{ content \| indent(4) }}` | +| `sum` | Sums a numeric sequence | `{{ values \| sum }}` | +| `selectattr` | Selects attribute values from a sequence | `{{ items \| selectattr('name') }}` | + +All standard Jinja2 built-in filters (`tojson`, `default`, `lower`, `upper`, `trim`, `join`, `replace`, `length`, `first`, `last`, `sort`, `unique`, `map`, `reject`, `select`, `batch`, `slice`, `int`, `float`, `string`, `list`, `dictsort`, `e`, `escape`, `safe`, `truncate`, `wordwrap`, `center`, `format`, `title`, `capitalize`, `striptags`, `urlencode`, `abs`, `round`, `pprint`, `groupby`, `max`, `min`, `random`, `filesizeformat`, `xmlattr`, `wordcount`, `reverse`) are also available. + +#### Exposed Built-in Functions + +Safe Python built-in functions are exposed in the template context: + +| Function | Purpose | +|----------|---------| +| `range()` | Generates integer sequences | +| `abs()` | Absolute value | +| `round()` | Rounds a number | +| `len()` | Length of a sequence | +| `min()` | Minimum of a sequence | +| `max()` | Maximum of a sequence | +| `sum()` | Sum of a sequence | + +#### Context Resolution + +Template variables are resolved from a context dictionary provided at render time. The context dictionary supports a nested `context` key convention: + +- `{{ context.paper_details.topic }}` resolves `context["paper_details"]["topic"]` +- `{{ context.brainstorming_summary }}` resolves `context["brainstorming_summary"]` + +The `global_context` top-level key in the actor configuration file populates this context dictionary at load time. At runtime, actor invocation context, plan context, and session context are merged into the template context following a defined precedence order. + +#### Template Detection + +The engine detects Jinja2 content by scanning for `{%` or `{{` markers in the raw YAML text. Files without these markers bypass Jinja2 processing entirely and are parsed as plain YAML, incurring no template overhead. + +#### Deferred Rendering + +When Jinja2 markers are detected but no context is available at load time (e.g., the context will only be available at runtime), the engine performs **deferred rendering**: it attempts to parse the YAML as-is (with template markers intact as literal strings). The templates are preserved in the parsed structure and rendered later when context becomes available. + +#### YAML Post-Processing + +After Jinja2 rendering, the engine applies post-processing to fix common issues introduced by template expansion: + +1. **Blank line removal**: Removes extraneous blank lines generated by `{% %}` block statements between YAML keys. +2. **Multi-colon line splitting**: Fixes lines where template expansion produces multiple YAML key-value pairs on a single line. +3. **Indentation correction**: Inserts indentation hints for `{% for %}` loops to ensure generated YAML maintains correct indentation. + +#### Template Protection for System Prompts + +A special protection mechanism preserves Jinja2 syntax in `system_prompt` fields. During initial loading, Jinja2 delimiters inside the YAML file are temporarily replaced with sentinel markers (`<<>>`, `<<>>`, `<<>>`, `<<>>`). After YAML parsing, these sentinels are restored to their original Jinja2 syntax exclusively within `system_prompt` fields. This allows system prompts to contain Jinja2 templates that are evaluated later at runtime (when the actor's context is available), rather than at file-load time. + +### Phase 2: Environment Variable Interpolation + +After YAML parsing, all string values in the configuration are recursively scanned for the pattern `${VAR_NAME}` or `${VAR_NAME:default_value}`: + +| Pattern | Behavior | +|---------|----------| +| `${VAR}` | Replaced with `os.environ["VAR"]`. Raises `ValueError` if not set. | +| `${VAR:default}` | Replaced with `os.environ.get("VAR", "default")`. | + +#### Automatic Type Coercion + +After substitution, the resulting string value is coerced to its natural type: + +| String Value | Coerced Type | Result | +|-------------|-------------|--------| +| `"true"` / `"false"` (case-insensitive) | `bool` | `True` / `False` | +| Digits only (with optional leading `-`) | `int` | e.g., `42`, `-7` | +| Digits with single `.` | `float` | e.g., `3.14`, `-0.5` | +| Anything else | `str` | Unchanged | + +This coercion applies to both environment variable values and their defaults, enabling configurations like `max_retries: ${MAX_RETRIES:3}` to produce an integer `3` rather than a string `"3"`. + +#### Recursion + +Environment variable interpolation is applied recursively to all nested dictionaries and lists in the parsed configuration, ensuring variables are resolved regardless of nesting depth. + +## Constraints + +- Template rendering must use `jinja2.sandbox.SandboxedEnvironment` exclusively. Unsandboxed Jinja2 environments are not permitted for actor configuration processing. +- Environment variable interpolation must not raise exceptions for undefined variables when a default value is provided via the `${VAR:default}` syntax. +- Template delimiters in `system_prompt` fields are preserved for runtime evaluation rather than consumed at load time. This is the only field that receives this special treatment. +- Files without Jinja2 markers (`{%` or `{{`) must bypass the template engine entirely with no behavioral difference from direct YAML parsing. +- The Jinja2 template phase must complete before environment variable interpolation begins. The two phases must not be interleaved. + +## Consequences + +### Positive +- Actor configurations can express dynamic, context-dependent system prompts without writing Python code, preserving the declarative YAML-only philosophy. +- Conditional blocks (`{% if %}`) enable a single configuration file to define behavior for multiple modes (e.g., auto-finish vs. interactive), reducing configuration file proliferation. +- Environment variable interpolation with defaults provides a clean mechanism for per-environment configuration (development vs. production) without modifying YAML files. +- Automatic type coercion eliminates the need for explicit casting in configuration consumers. +- Full backward compatibility with v2 configurations that already use Jinja2 templates. +- The sandboxed environment prevents template injection attacks in user-provided configuration files. + +### Negative +- Jinja2 template syntax errors are reported at YAML load time rather than at validation time, making debugging harder because the error context shows rendered (or partially rendered) output rather than the source template. +- The template protection mechanism (sentinel replacement) for `system_prompt` fields adds complexity and could interact poorly with unusual YAML structures. +- Authors must understand both YAML syntax and Jinja2 syntax, increasing the learning curve for configuration authoring. + +### Risks +- Complex template logic in YAML files may become difficult to maintain and debug, especially when templates generate YAML structure (keys and values) rather than just string content. +- The post-processing heuristics (blank line removal, colon splitting) may produce incorrect results for edge cases in generated YAML. +- Deferred rendering may silently produce incorrect configurations if the expected context variables are not available at runtime. + +## Alternatives Considered + +**No template engine (static YAML only)** — Would require separate YAML files for each configuration variant (e.g., auto-finish mode vs. interactive mode) and prevent dynamic prompt content. Rejected because it undermines the goal of single-file, self-contained actor definitions for complex workflows. + +**Mako or Cheetah templates** — Alternative Python template engines. Jinja2 is the most widely used, best documented, and provides a sandboxed execution mode specifically designed for untrusted templates. Jinja2 is also already a transitive dependency via LangChain. Rejected in favor of Jinja2. + +**Runtime-only string interpolation (no load-time templates)** — Would defer all template evaluation to runtime. This prevents generating YAML structure from templates (e.g., generating actor definitions with `{% for %}`), which is needed for configuration DRY patterns. The two-phase approach (load-time Jinja2 + runtime env vars) provides the correct separation. Rejected. + +**YAML anchors and aliases** — YAML's built-in reuse mechanism. Anchors support structural reuse but not conditionals, loops, or string interpolation. Useful as a complement to Jinja2 templates but not a replacement. Not rejected — can be used alongside Jinja2. + +## Compliance + +- **Sandbox enforcement tests**: Verify that the template engine uses `SandboxedEnvironment` and that attempts to access unsafe operations (file I/O, `os.system`, `eval`) are rejected. +- **Template detection tests**: Verify that files without `{%` or `{{` markers bypass the template engine entirely. +- **Environment variable interpolation tests**: Verify `${VAR}`, `${VAR:default}`, type coercion (bool, int, float), and error behavior for missing variables without defaults. +- **Template protection tests**: Verify that Jinja2 syntax in `system_prompt` fields survives the load-parse cycle and is available for runtime evaluation. +- **Post-processing tests**: Verify that rendered YAML with blank lines, multi-colon lines, and indentation issues is correctly fixed. +- **Filter and built-in tests**: Verify that all custom filters (`yaml`, `indent`, `sum`, `selectattr`) and exposed built-ins (`range`, `abs`, `round`, `len`, `min`, `max`, `sum`) work correctly in template expressions. +- **BDD feature coverage**: `actor_config_coverage.feature`, `actor_config_new_coverage.feature`, and `yaml_template_engine_coverage.feature` provide end-to-end coverage of the two-phase pipeline. + +## Related ADRs + +| ADR | Title | Relationship | +|-----|-------|-------------| +| [ADR-010](ADR-010-actor-and-agent-architecture.md) | Actor and Agent Architecture | Defines the actor configuration model that this ADR's template engine processes | +| [ADR-024](ADR-024-configuration-system.md) | Configuration System | Defines the overall configuration approach; this ADR extends it with Jinja2 preprocessing for actor YAML files | +| [ADR-031](ADR-031-actor-abstraction-definition.md) | Actor Abstraction Definition | Defines what an actor is; this ADR defines how actor YAML files are preprocessed before validation | +| [ADR-005](ADR-005-technical-stack.md) | Technical Stack | Jinja2 is added as a required dependency in the technical stack | + +## Acceptance + +### Votes For + +| Voter | Comment | +|-------|---------| +| Jeffrey Phillips Freeman | Two-phase Jinja2 + env var preprocessing provides the right balance of power and safety for declarative actor configurations | + +**Total: 1** + +### Votes Against + +| Voter | Comment | +|-------|---------| + +**Total: 0** + +### Abstentions + +| Voter | Comment | +|-------|---------| + +**Total: 0** diff --git a/docs/adr/index.md b/docs/adr/index.md index 7b05e24d7..e4ef27a79 100644 --- a/docs/adr/index.md +++ b/docs/adr/index.md @@ -118,10 +118,11 @@ These ADRs address external integrations, operational interfaces, and deployment | [ADR-029](ADR-029-model-context-protocol.md) | Model Context Protocol (MCP) Adoption | MCP tool discovery, registry integration, skill composition, and actor-graph usage. | | [ADR-030](ADR-030-skill-abstraction-definition.md) | Skill Abstraction Definition | Canonical definition of a skill as a composable collection of tools from four sources (MCP, Agent Skills, built-in, custom). | | [ADR-031](ADR-031-actor-abstraction-definition.md) | Actor Abstraction Definition | Canonical definition of an actor as anything conversational — single LLM, composed graph, or hierarchical orchestrator. | +| [ADR-032](ADR-032-jinja2-yaml-template-preprocessing.md) | Jinja2 YAML Template Preprocessing | Two-phase YAML processing: sandboxed Jinja2 template rendering followed by environment variable interpolation with type coercion. | ## Creating a New ADR -1. Assign the next sequential number (e.g., `ADR-032`). +1. Assign the next sequential number (e.g., `ADR-033`). 2. Create a file named `ADR-0NN-.md` in this directory. 3. Copy the template structure from any existing ADR. 4. Fill in the metadata table with `Status: Proposed`, the current date, author(s), and leave approver(s) blank. diff --git a/docs/gen_ref_pages.py b/docs/gen_ref_pages.py index 9e78e058e..7188f3c83 100644 --- a/docs/gen_ref_pages.py +++ b/docs/gen_ref_pages.py @@ -79,6 +79,17 @@ for path in sorted(SRC_DIR.rglob("*.py")): # Include manually-authored reference pages that live alongside the # auto-generated API docs. nav[("actors_schema",)] = "actors_schema.md" +nav[("automation_profile_service",)] = "automation_profile_service.md" +nav[("automation_profiles",)] = "automation_profiles.md" +nav[("changeset_model",)] = "changeset_model.md" +nav[("cli_system_commands",)] = "cli_system_commands.md" +nav[("database_schema",)] = "database_schema.md" +nav[("diagnostics_checks",)] = "diagnostics_checks.md" +nav[("project_context_cli",)] = "project_context_cli.md" +nav[("project_context_policy",)] = "project_context_policy.md" +nav[("resource_dag",)] = "resource_dag.md" +nav[("resource_types_builtin",)] = "resource_types_builtin.md" +nav[("tool_bindings",)] = "tool_bindings.md" # Write the literate-nav summary file for the Reference section with mkdocs_gen_files.open("reference/SUMMARY.md", "w") as nav_file: diff --git a/docs/reference/tool_bindings.md b/docs/reference/tool_bindings.md index d0068ba44..c8e8aaaa0 100644 --- a/docs/reference/tool_bindings.md +++ b/docs/reference/tool_bindings.md @@ -127,6 +127,6 @@ resource_slots: See: -- [`BindingResolutionService`](../../src/cleveragents/application/services/binding_resolution_service.py) -- [`BindingResult`](../../src/cleveragents/domain/models/core/resource_slot.py) -- [`ResourceSlot`](../../src/cleveragents/domain/models/core/tool.py) +- [`BindingResolutionService`](cleveragents/application/services/binding_resolution_service.md) +- [`BindingResult`](cleveragents/domain/models/core/resource_slot.md) +- [`ResourceSlot`](cleveragents/domain/models/core/tool.md) diff --git a/docs/specification.md b/docs/specification.md index b6838ef93..6b5b05313 100644 --- a/docs/specification.md +++ b/docs/specification.md @@ -19018,6 +19018,167 @@ Example actor configuration (see `examples/` directory for full examples): task_description: "Default task" +#### Jinja2 Template Preprocessing + +!!! adr "Architecture Decision" + The two-phase Jinja2 + environment variable preprocessing pipeline for actor YAML files is defined in [ADR-032: Jinja2 YAML Template Preprocessing](adr/ADR-032-jinja2-yaml-template-preprocessing.md). + +Actor configuration YAML files are processed through a **two-phase pipeline** before the resulting data structure is validated and loaded: + +1. **Phase 1 — Jinja2 Template Rendering**: The raw file content is run through a sandboxed Jinja2 template engine, resolving `{{ }}` expressions, `{% %}` control structures, and `{# #}` comments into static YAML text. +2. **Phase 2 — Environment Variable Interpolation**: After YAML parsing, all string values matching `${VAR}` or `${VAR:default}` are recursively replaced with OS environment variable values, with automatic type coercion. + +!!! warning "Execution Order" + Jinja2 templates are evaluated **before** YAML parsing. Environment variables are interpolated **after** YAML parsing. The two phases are never interleaved. + +##### Jinja2 Template Syntax + +The engine uses standard Jinja2 delimiters inside the YAML file: + +| Delimiter | Purpose | Example | +|-----------|---------|---------| +| `{{ ... }}` | Variable expression | `{{ context.paper_details.topic }}` | +| `{% ... %}` | Block statement (`if`, `for`, `block`, etc.) | `{% if context.auto_finish_active %}` | +| `{# ... #}` | Comment (stripped from output) | `{# This is ignored #}` | + +**Variable expressions** are used to inject dynamic values into prompts and configuration fields: + +```yaml +system_prompt: | + You are writing a paper on {{ context.paper_details.topic | tojson }}. + Target length: {{ context.paper_details.length | tojson }} words. + Audience: {{ context.paper_details.audience | tojson }}. +``` + +**Conditional blocks** enable a single configuration to define behavior for multiple modes: + +```yaml +system_prompt: | + You are a research assistant. + {% if context.auto_finish_active %} + Auto-finish mode is active. Do not ask questions. Proceed autonomously. + {% else %} + Engage in interactive conversation to refine the output. + {% endif %} +``` + +**For loops** generate repetitive YAML structure from data: + +```yaml +system_prompt: | + Available sections: + {% for section in context.section_paths %} + - {{ section }} + {% endfor %} +``` + +##### Sandboxed Execution + +All template rendering uses `jinja2.sandbox.SandboxedEnvironment`, which prevents templates from: + +- Executing arbitrary Python code +- Accessing the filesystem +- Calling `os.system`, `eval`, `exec`, or similar unsafe operations +- Accessing private attributes of objects + +##### Custom Jinja2 Filters + +The engine registers four custom filters in addition to all standard Jinja2 built-in filters: + +| Filter | Purpose | Example | +|--------|---------|---------| +| `yaml` | Serializes any value to a YAML-formatted string | `{{ my_dict \| yaml }}` | +| `indent` | Indents text by N spaces (default 2) | `{{ content \| indent(4) }}` | +| `sum` | Sums a numeric iterable | `{{ values \| sum }}` | +| `selectattr` | Selects a named attribute from each item in a sequence | `{{ items \| selectattr('name') }}` | + +All standard Jinja2 built-in filters are also available, including: `tojson`, `default`, `lower`, `upper`, `trim`, `join`, `replace`, `length`, `first`, `last`, `sort`, `unique`, `map`, `reject`, `select`, `batch`, `slice`, `int`, `float`, `string`, `list`, `dictsort`, `escape`, `safe`, `truncate`, `wordwrap`, `center`, `format`, `title`, `capitalize`, `striptags`, `urlencode`, `abs`, `round`, `pprint`, `groupby`, `max`, `min`, `random`, `filesizeformat`, `wordcount`, `reverse`. + +##### Exposed Built-in Functions + +The following safe Python built-in functions are exposed in the template context and can be called directly in expressions: + +| Function | Purpose | Example | +|----------|---------|---------| +| `range()` | Generates integer sequences | `{% for i in range(5) %}` | +| `abs()` | Absolute value | `{{ abs(score) }}` | +| `round()` | Rounds a number | `{{ round(value, 2) }}` | +| `len()` | Length of a collection | `{{ len(items) }}` | +| `min()` | Minimum value | `{{ min(scores) }}` | +| `max()` | Maximum value | `{{ max(scores) }}` | +| `sum()` | Sum of values | `{{ sum(counts) }}` | + +##### Template Context Resolution + +Template variables are resolved from a context dictionary. The context supports a nested `context` key convention: + +- `{{ context.paper_details.topic }}` → resolves `context["paper_details"]["topic"]` +- `{{ context.brainstorming_summary }}` → resolves `context["brainstorming_summary"]` + +The `global_context` top-level key in the actor configuration file populates this dictionary at load time. At runtime, the actor invocation context, plan context, and session context are merged with the following precedence: **runtime context > plan context > session context > global_context from YAML**. + +##### Template Detection and Bypass + +The engine detects Jinja2 content by scanning for `{%` or `{{` markers in the raw text. Files without these markers bypass Jinja2 processing entirely and are parsed as plain YAML with zero template overhead. + +##### Deferred Rendering + +When Jinja2 markers are present but no context is available at load time, the engine performs **deferred rendering**: it parses the YAML with template markers intact as literal strings. Templates are preserved in the parsed structure and rendered later when runtime context becomes available. This is the mechanism by which `system_prompt` fields retain their Jinja2 templates for runtime evaluation. + +##### Template Protection for `system_prompt` Fields + +A special protection mechanism ensures Jinja2 syntax inside `system_prompt` fields survives the YAML loading process: + +1. During loading, all Jinja2 delimiters in the raw file are temporarily replaced with sentinel markers: + - `{{` → `<<>>` + - `}}` → `<<>>` + - `{%` → `<<>>` + - `%}` → `<<>>` + +2. The protected text is parsed as YAML. + +3. After parsing, the sentinels are restored to Jinja2 syntax **only within `system_prompt` fields** (recursively through all nested dictionaries and lists). + +This allows system prompts to contain Jinja2 templates that are evaluated at **runtime** (when the actor's full context is available) rather than at file-load time. + +##### YAML Post-Processing + +After Jinja2 rendering, the engine applies automatic post-processing to fix common issues: + +1. **Blank line cleanup**: Removes extraneous blank lines generated by `{% %}` block tags between YAML keys. +2. **Multi-colon line splitting**: Detects and splits lines where template expansion produces multiple `key: value` pairs on a single line. +3. **Indentation correction**: Inserts indentation hints for `{% for %}` loops to ensure generated YAML maintains correct structure. + +##### Environment Variable Interpolation + +After YAML parsing, all string values are recursively scanned for environment variable references: + +| Pattern | Behavior | +|---------|----------| +| `${VAR}` | Replaced with `os.environ["VAR"]`. Raises `ValueError` if not set. | +| `${VAR:default}` | Replaced with `os.environ.get("VAR", "default")`. Uses default if not set. | + +**Automatic type coercion** is applied after substitution: + +| Substituted Value | Coerced Type | Example | +|------------------|-------------|---------| +| `"true"` / `"false"` (case-insensitive) | `bool` | `True` / `False` | +| Digits only (with optional leading `-`) | `int` | `42`, `-7` | +| Digits with single `.` | `float` | `3.14`, `-0.5` | +| Anything else | `str` | Unchanged | + +This enables configurations like: + +```yaml +env_vars: + WORK_DIR: ${HOME}/workspace # String: /home/user/workspace + LOG_LEVEL: ${LOG_LEVEL:info} # String: info (default) + MAX_RETRIES: ${MAX_RETRIES:3} # Integer: 3 (coerced from default) + DEBUG: ${DEBUG_MODE:false} # Boolean: False (coerced from default) +``` + +Environment variable interpolation is applied recursively to all nested dictionaries and lists, ensuring resolution at any depth. + #### Actor Arguments **All actors can receive arguments when invoked**, including built-in actors. Arguments are passed when: @@ -19036,20 +19197,22 @@ For built-in actors (like `openai/gpt-4`), common arguments include: Actors can reference other actors **by name**: -

-actors:
-  complex_workflow:
-    type: llm
-    config:
-      actor: local/base-analyzer # References another actor
-
+```yaml +actors: + complex_workflow: + type: llm + config: + actor: local/base-analyzer # References another registered actor +``` **Load order matters**: Referenced actors must be loaded/defined before actors that depend on them. This enables hierarchical composition where: * Actor A's graph can include nodes that call Actor B (by name) * Actor B itself is a graph that might call Actor C -* And so on... +* And so on (no depth limit) + +Circular actor references are **prohibited** and detected at registration time. #### Actor vs Agent (Relationship) @@ -19063,21 +19226,164 @@ This enables hierarchical composition where: * a multi-step graph, * a wrapper around a third-party system (as long as it's "text in → text out" conversationally). -#### Actor Definition Fields (From Notes + Extended) +#### Actor Definition Fields (Complete Reference) -A robust actor schema should include: +The complete set of fields available in an actor definition. For the formal JSON Schema and additional annotated examples, see [Actor Configuration Files](#actor-configuration-files) in the Configuration section. -* `name` (namespaced, **required** in the YAML config file — defines the actor's registered identity) -* `provider` (LLM provider or runtime target) -* `model` -* `system_prompt` (or prompt template) -* `skill_access_policy` (which skill categories/names are allowed or denied) -* `graph_descriptor` (for composite actors) -* `memory_policy` (per-plan/per-actor—see memory section) -* `context_view_policy` (what context this actor sees — includes ACMS settings: `preferred_strategies`, `default_breadth`, `default_depth`, `depth_gradient`, `skeleton_budget_ratio`, `temporal_scope`, `auto_refresh`, `refresh_threshold`) -* `limits` (token limits, tool call limits, retries) -* `cost_policy` (caps, budgets) -* `metadata` (use cases, version) +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `name` | string | Yes | Namespaced actor name in `/` format. Used as the registered identity. | +| `type` | string | Yes | Actor type: `llm` (language model), `tool` (tool collection), or `graph` (multi-node workflow). | +| `description` | string | Yes | Human-readable description of what the actor does. | +| `version` | string | No | Schema version (default: `"1.0"`). | +| `model` | string | Yes (LLM/GRAPH) | LLM model identifier (e.g., `gpt-4`, `claude-3.5-sonnet`). | +| `system_prompt` | string | No | System prompt text. Supports Jinja2 templates for dynamic content. | +| `tools` | list | Yes (TOOL) | List of tool references (strings) and/or inline tool definitions. | +| `context_view` | string | No | Role-based context filtering: `strategist`, `executor`, `reviewer`, or `full`. | +| `memory` | object | No | Conversation history settings (see Memory Configuration below). | +| `context` | object | No | File inclusion and context window settings (see Context Configuration below). | +| `route` | object | Yes (GRAPH) | Graph topology (see Route Configuration below). | +| `env_vars` | object | No | Environment variable key-value mappings. | +| `skills` | list[string] | No | List of skill names this actor can use. Each entry is a namespaced skill name (e.g., `local/file-ops`). Skills provide tool capabilities to the actor. See [Actor References to Skills and Tools](#actor-references-to-skills-and-tools). | + +**Additional fields available in the v2/runtime actor definition format** (within the `config:` block of actors defined inside the `actors:` top-level key): + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `config.provider` | string | Yes (LLM) | LLM provider identifier: `openai`, `anthropic`, `google`, `azure`, `openrouter`, etc. | +| `config.model` | string | Yes (LLM) | Model identifier within the provider. | +| `config.actor` | string | No | Combined `provider/model` format (alternative to separate `provider` + `model`). | +| `config.system_prompt` | string | No | System prompt text with Jinja2 template support. | +| `config.temperature` | float | No | Sampling temperature (0.0 to 2.0). Lower = more deterministic. | +| `config.max_tokens` | integer | No | Maximum tokens in the generated response. | +| `config.memory_enabled` | boolean | No | Enable conversation memory (default: `false`). | +| `config.max_history` | integer | No | Maximum conversation turns retained in memory (default: `50`). | +| `config.unsafe` | boolean | No | Allow this actor to perform unsafe operations (default: `false`). | +| `config.options` | object | No | Provider-specific options passed through to the underlying LLM API. | +| `config.tools` | list | Yes (tool) | List of inline tool definitions (each with `name` and `code`). | +| `config.response_format` | object | No | JSON schema for structured output from the LLM. | + +##### Memory Configuration + +| Field | Type | Default | Description | +|-------|------|---------|-------------| +| `enabled` | boolean | `true` | Whether to maintain conversation history. | +| `max_messages` | integer | `null` (unlimited) | Maximum number of messages to retain. | +| `max_tokens` | integer | `null` (unlimited) | Maximum tokens in retained history. | +| `summarize_old` | boolean | `false` | Whether to summarize old messages instead of discarding them. | + +##### Context Configuration + +| Field | Type | Default | Description | +|-------|------|---------|-------------| +| `include_files` | list[string] | `[]` | File paths to include in the actor's context. | +| `include_dirs` | list[string] | `[]` | Directory paths to include in the actor's context. | +| `exclude_patterns` | list[string] | `[]` | Glob patterns to exclude from context (e.g., `"**/__pycache__/**"`). | +| `max_context_tokens` | integer | `null` (model default) | Maximum size of the context window in tokens. | + +##### Context View + +The `context_view` field controls role-based context filtering: + +| Value | Purpose | Includes | +|-------|---------|----------| +| `strategist` | High-level planning view | Project structure, goals, constraints, architectural summaries | +| `executor` | Implementation view | Source code, file contents, specific task details | +| `reviewer` | Validation view | Changes, diffs, test results, review criteria | +| `full` | Complete view (use sparingly) | All available context from all categories | + +##### Type-Specific Requirements + +| Actor Type | Required Fields | Optional Fields | +|-----------|----------------|-----------------| +| `llm` | `model` | `system_prompt`, `tools`, `context_view`, `memory`, `context`, `env_vars` | +| `tool` | `tools` (at least one) | `context_view`, `env_vars` | +| `graph` | `model`, `route` | `system_prompt`, `tools`, `context_view`, `memory`, `context`, `env_vars` | + +##### Tool Definitions (Inline) + +Tools in an actor can be either **string references** to registered tools or **inline definitions**: + +```yaml +tools: + # String reference to a registered tool + - files/read_file + - files/list_directory + + # Inline tool definition + - name: utils/count_lines + description: Count the number of lines in a file + parameters: + - name: file_path + type: str + description: Path to the file + required: true + default: null + code: | + def count_lines(file_path: str) -> int: + with open(file_path, 'r', encoding='utf-8') as f: + return len(f.readlines()) +``` + +Each inline tool parameter supports: + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `name` | string | Yes | Parameter name (must be a valid Python identifier). | +| `type` | string | Yes | Python type annotation as string (e.g., `str`, `int`, `list[str]`). | +| `description` | string | Yes | Human-readable description. | +| `required` | boolean | No | Whether the parameter must be provided (default: `true`). | +| `default` | any | No | Default value if not provided (only for optional parameters). | + +Inline tool `name` must follow the `namespace/tool_name` format. The `code` field contains Python source code that defines a callable function. + +##### Route Configuration (Graph Topology) + +For `type: graph` actors, the `route` field defines the graph structure: + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `nodes` | list[NodeDefinition] | Yes | All nodes in the graph. | +| `edges` | list[EdgeDefinition] | Yes | All edges connecting nodes. | +| `entry_node` | string | Yes | ID of the starting node. | +| `exit_nodes` | list[string] | Yes | IDs of terminal nodes. | + +**Node Definition:** + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `id` | string | Yes | Unique node identifier (alphanumeric with underscores/hyphens). | +| `type` | string | Yes | Node type: `agent`, `tool`, `conditional`, or `subgraph`. | +| `name` | string | Yes | Human-readable node name. | +| `description` | string | Yes | Node purpose and behavior. | +| `config` | object | No | Type-specific configuration (see below). | + +**Node type-specific config:** + +| Node Type | Config Fields | Description | +|-----------|---------------|-------------| +| `agent` | `model`, `prompt`, `tools` | LLM agent with optional tools | +| `tool` | `tool_name`, `parameters` | Deterministic tool execution | +| `conditional` | `conditions[].check`, `conditions[].route_to` | Routes based on state conditions (Python expressions) | +| `subgraph` | `actor_path` | Embeds another actor as a nested workflow | + +**Edge Definition:** + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `from_node` | string | Yes | Source node ID. | +| `to_node` | string | Yes | Target node ID. | +| `condition` | string | No | Python expression for conditional routing. | +| `priority` | integer | No | Edge priority for multiple outgoing edges (higher = evaluated first, default: `0`). | + +**Graph Validation:** + +- All node IDs must be unique within the graph. +- The `entry_node` must reference an existing node ID. +- All `exit_nodes` must reference existing node IDs. +- All edge `from_node` and `to_node` must reference existing node IDs. +- The graph must be **acyclic** — cycles are detected via DFS and rejected at validation time. +- All nodes must be reachable from the entry node. #### Actor Composition and Graphs @@ -19146,6 +19452,339 @@ This is a powerful simplification: actors provide intelligence, tools provide ca return {"valid": True} +#### Multi-Actor Configuration File (Complete Structure) + +A single actor configuration YAML file can define an entire multi-actor system — multiple actors, graph routing, stream processing, context sharing, and message routing — all in one file. This is the format used for complex workflows like the scientific paper writer. + +##### Top-Level Keys + +| Key | Type | Required | Description | +|-----|------|----------|-------------| +| `name` | string | Yes | Namespaced actor name (`/`). | +| `cleveragents` | object | No | Metadata: version, logging, template engine, safety, default actor. | +| `actors` (or `agents`) | object | Yes | Map of actor names to definitions. Both key names are accepted. | +| `routes` | object | No | Map of route names to stream or graph topology definitions. | +| `merges` | list | No | Stream merge operations combining multiple sources into one target. | +| `splits` | list | No | Stream split operations dividing one source into multiple targets. | +| `publications` | list | No | Output stream names (e.g., `["__output__"]`). | +| `templates` | object | No | Reusable template definitions for Jinja2 inheritance. | +| `instances` | object | No | Instantiated templates with bound parameters. | +| `global_context` | object | No | Key-value pairs accessible to all actors via `{{ context.key }}`. | +| `context` | object | No | Alternative context block with `global:` sub-key. | +| `prompts` | object | No | Named prompt templates referenceable by actors. | +| `pipelines` | object | No | Hybrid pipeline definitions combining stream and graph stages. | + +##### `cleveragents` Metadata Block + +```yaml +cleveragents: + version: "3.0" # Schema version (default: "3.0") + logging: + level: "INFO" # DEBUG, INFO, WARNING, ERROR (default: INFO) + template_engine: "JINJA2" # JINJA2 or NONE (default: JINJA2) + unsafe: false # Allow unsafe operations (default: false) + default_actor: my_actor # Default actor when multiple defined +``` + +##### Route Definitions + +Routes connect actors via **stream** or **graph** topologies: + +**Stream Routes** — reactive processing pipelines: + +```yaml +routes: + chat_stream: + type: stream + stream_type: cold # cold (default), hot, or replay + operators: + - type: map # map or graph_execute + params: + agent: chat_agent # Actor name for map operators + publications: + - __output__ # Output stream name + subscriptions: + - __input__ # Input stream name + buffer_size: 10 # Stream buffer size (default: 10) + initial_value: null # Initial value (optional) +``` + +**Graph Routes** — LangGraph-based directed graph workflows: + +```yaml +routes: + main: + type: graph + entry_point: start # Entry node name (required) + nodes: + start: + type: START # Special start node + end: + type: END # Special end node + router: + type: message_router # Message-based routing node + rules: # Routing rules (see below) + - ... + my_actor_node: + type: agent # Actor-backed node + agent: my_actor # References actor by name + metadata: {} # Optional metadata + edges: + - source: start + target: router + - source: router + target: my_actor_node + condition: + context_value: next_node + equals: my_actor_node + - source: my_actor_node + target: end + checkpointing: false # Enable checkpointing (default: false) + checkpoint_dir: null # Checkpoint storage directory + enable_time_travel: false # Enable time travel debugging (default: false) + parallel_execution: false # Allow parallel node execution (default: false) + state_class: null # Custom state class name +``` + +##### Graph Node Types + +| Type | Purpose | Key Fields | +|------|---------|------------| +| `agent` | Node backed by an actor | `agent: ` | +| `tool` | Node invoking tools | `tools: [, ...]` | +| `function` | Node backed by a Python function | `function: ` | +| `conditional` | Branching node | `condition: { ... }` | +| `subgraph` | Delegates to another graph route | `subgraph: ` | +| `start` / `START` | Explicit start node | (none) | +| `end` / `END` | Terminal node | (none) | +| `message_router` | Content-based message routing | `rules: [...]` | + +##### Message Router Node + +The `message_router` node type routes messages to different actors based on message content. It uses a rules-based system: + +```yaml +router: + type: message_router + rules: + # Prefix-based routing + - type: prefix + match: "GOTO_BRAINSTORMING" + target: brainstorming + strip_match: true # Remove the prefix before forwarding + + # Contains-based routing + - type: contains + match: "SET_TOPIC:" + target: discovery + + # Suffix-based routing (catch-all) + - type: suffix + match: "" # Empty string matches everything + target: workflow_controller +``` + +Each rule specifies: + +| Field | Type | Description | +|-------|------|-------------| +| `type` | string | Match type: `prefix`, `contains`, or `suffix`. | +| `match` | string | Pattern to match against the message content. | +| `target` | string | Node name to route the message to. | +| `strip_match` | boolean | Whether to strip the matched pattern from the message (default: `false`). | + +Rules are evaluated in order; the first matching rule determines the target node. + +##### Routing Prefixes (Inter-Actor Communication) + +Actors communicate with each other and the routing system via **routing prefixes** — special string prefixes prepended to output text that the message router interprets: + +| Prefix Pattern | Purpose | Example | +|----------------|---------|---------| +| `GOTO_:` | Route to a specific node | `GOTO_BRAINSTORMING:Start the brainstorm` | +| `SET_:` | Set a context field value | `SET_TOPIC:Quantum computing` | +| `ROUTE_:` | Route to a sub-target | `ROUTE_ASK_TOPIC:What topic?` | +| `COMMAND_OUTPUT:` | Display output directly to user | `COMMAND_OUTPUT:Help text here` | +| `DISCOVERY_RESPONSE:` | Response from discovery stage | `DISCOVERY_RESPONSE:Topic set` | + +Tool actors return these prefixes as their `result` variable. The message router parses the prefix and routes accordingly. The message content after the colon is forwarded to the target node. + +##### Conditional Edges + +Graph edges can include conditions that control routing based on graph state: + +```yaml +edges: + # Unconditional edge + - source: start + target: router + + # Conditional edge — routes based on context value + - source: router + target: brainstorming + condition: + context_value: next_node + equals: brainstorming + + # Conditional edge — routes based on boolean flag + - source: passthrough + target: auto_driver + condition: + context_value: auto_finish_active + equals: true +``` + +##### Merges and Splits + +**Merges** combine multiple input streams into a single stream: + +```yaml +merges: + - sources: [__input__] # Special __input__ = user input + target: main # Route name to send merged input to +``` + +**Splits** divide a single stream into multiple output streams: + +```yaml +splits: + - source: main_output + targets: [log_stream, display_stream] +``` + +Special stream names: +- `__input__` — the user's input +- `__output__` — the final output displayed to the user + +##### Inline Tool Code Model + +Tool-type actors define their behavior entirely in inline Python code within the `code:` field: + +```yaml +my_tool_actor: + type: tool + config: + tools: + - name: my_tool + code: | + import sys + # Available variables: + # input_data — the input text/message passed to this tool + # context — shared mutable context dictionary + # result — set this variable to define the tool's output + + msg = input_data or '' + context['last_input'] = msg + + if msg.startswith('!help'): + result = "COMMAND_OUTPUT:Available commands: !help, !next" + else: + result = f"GOTO_PROCESSOR:{msg}" + + print(f"DEBUG: {result}", file=sys.stderr) +``` + +The inline code execution model provides three implicit variables: + +| Variable | Type | Description | +|----------|------|-------------| +| `input_data` | string | The input text/message passed to the tool. | +| `context` | dict | Shared mutable context dictionary. Changes persist across invocations. | +| `result` | string | **Set this variable** to define the tool's output. | + +The `context` dictionary is the primary mechanism for inter-actor state sharing. All actors in the same configuration share the same context, enabling data flow between stages. + +##### Context Sharing + +The `context` dictionary (accessible in tool code and Jinja2 templates) serves as shared state: + +```yaml +# Set via global_context in YAML: +global_context: + writing_stage: intro + paper_details: + topic: null + length: null + audience: null + +# Or via context.global: +context: + global: + conversation_mode: true + default_actor: openai/gpt-4 +``` + +At runtime, tool actors read and modify context freely: + +```python +# In tool code: +context['writing_stage'] = 'brainstorming' # Update stage +topic = context.get('paper_details', {}).get('topic') # Read nested value +context.setdefault('history', []).append(msg) # Append to list +``` + +In Jinja2 templates (system prompts): + +```yaml +system_prompt: | + Paper topic: {{ context.paper_details.topic | tojson }} + {% if context.auto_finish_active %} + Proceed autonomously. + {% endif %} +``` + +##### Stream-to-Graph Bridge + +Routes can include bridge configuration for dynamic topology changes: + +```yaml +routes: + adaptive: + type: stream + bridge: + upgrade_conditions: + message_count_threshold: 5 + downgrade_conditions: + idle_timeout: 30 + state_extractor: "extract_graph_state" + state_flattener: "flatten_to_stream" + preserve_subscriptions: true + preserve_checkpointing: true +``` + +##### Publications + +The `publications` key defines output streams at the route or top level: + +```yaml +# Route-level publications +routes: + chat_stream: + type: stream + publications: + - __output__ + +# Top-level publications +publications: + - __output__ +``` + +#### Actor Configuration File Loading + +Actor configuration files can be loaded from either JSON or YAML format: + +1. The loader first attempts JSON parsing (`json.loads`) +2. If JSON parsing fails, it falls back to the YAML pipeline (Jinja2 preprocessing + `yaml.safe_load` + environment variable interpolation) + +The resolution order for `provider` and `model` values when loading: + +1. CLI override (`--provider`, `--model`) +2. Top-level `provider` / `model` keys +3. Top-level `provider_type` / `model_id` aliases +4. v2-extracted values from `actors..config.provider` / `.model` + +For `unsafe` flag: the result is `true` if **any** of the following is `true`: the top-level `unsafe` key, the v2-extracted `unsafe` flag, or the CLI `--unsafe` flag. + ### Agent !!! adr "Architecture Decision" @@ -26504,7 +27143,7 @@ All CleverAgents configuration files share a consistent set of design principles !!! adr "Architecture Decision" The actor configuration schema and actor registration model are defined in [ADR-010: Actor and Agent Architecture](adr/ADR-010-actor-and-agent-architecture.md). -Actor configuration files define intelligent agents — anything conversational, from a single LLM to an entire graph of interconnected actors and tools. Actors are registered via `agents actor add --config `. +Actor configuration files define intelligent agents — anything conversational, from a single LLM to an entire graph of interconnected actors and tools. Actors are registered via `agents actor add --config `. For behavioral documentation including Jinja2 template preprocessing, actor composition, multi-actor workflows, and field-level descriptions, see the [Actor](#actor) section. For the Jinja2 preprocessing pipeline specifically, see [Jinja2 Template Preprocessing](#jinja2-template-preprocessing) and [ADR-032](adr/ADR-032-jinja2-yaml-template-preprocessing.md). #### JSON Schema @@ -26753,9 +27392,19 @@ The following is the formal [JSON Schema](https://json-schema.org/) definition f "required": ["name", "code"], "additionalProperties": false } + }, + "response_format": { + "type": "object", + "description": "JSON schema for structured output from the LLM. When set, the model is constrained to produce output matching this schema.", + "additionalProperties": true } }, "additionalProperties": false + }, + "skills": { + "type": "array", + "description": "List of skill names this actor can use. Each entry is a namespaced skill name (e.g., 'local/file-ops'). Skills provide tool capabilities to the actor.", + "items": { "type": "string", "pattern": "^[a-z0-9_-]+/[a-z0-9_-]+$" } } }, "required": ["type", "config"], @@ -26984,6 +27633,12 @@ The following annotated YAML provides an easier-to-read overview of the same sch def run(input_data): return {"result": input_data["query"]} + response_format: {} # JSON schema for structured LLM output (optional, LLM only) + + # ── Skills (both LLM and tool actors) ──────────────────────── + skills: # Skill references providing tool capabilities (optional) + - <namespace>/<skill-name> # e.g., local/file-ops, local/git-ops + # ─── Routes ───────────────────────────────────────────────────────── routes: <route_name>: @@ -27111,6 +27766,8 @@ The following annotated YAML provides an easier-to-read overview of the same sch | `config.unsafe` | boolean | `false` | Allow this specific actor to perform unsafe operations. | | `config.options` | object | `{}` | Provider-specific options passed through to the underlying LLM API. | | `config.tools` | list | Yes (tool) | List of inline tool definitions for tool-type actors. Each tool has `name` and `code`. | +| `config.response_format` | object | No | JSON schema for structured output from the LLM. Constrains model output to match the schema. | +| `skills` | list | No | List of namespaced skill names (e.g., `local/file-ops`) providing tool capabilities to the actor. | **Node Types (Graph Routes)** @@ -27570,6 +28227,423 @@ An actor that routes between different specialists based on input classification --- +**Example 6: Environment-Aware Actor with Jinja2 Conditionals (Simple Jinja2)** + +A single-actor configuration demonstrating the most common Jinja2 preprocessing features: `{{ variable }}` interpolation, `{% if %}` conditionals, the `| default` filter, and `${VAR:default}` environment variable substitution. See [ADR-032](adr/ADR-032-jinja2-yaml-template-preprocessing.md) for the full Jinja2 preprocessing specification. + +

+# support-bot.yaml
+# Register: agents actor add --config support-bot.yaml
+#
+# Jinja2 preprocessing (Phase 1) resolves {{ }} and {% %} at load time.
+# Phase 2: env var interpolation resolves ${VAR} after parsing.
+
+name: local/support-bot
+
+cleveragents:
+  version: "3.0"
+  template_engine: "JINJA2"
+
+actors:
+  support:
+    type: llm
+    config:
+      # Phase 2: environment variables with defaults and type coercion
+      provider: ${LLM_PROVIDER:anthropic}
+      model: ${LLM_MODEL:claude-3.5-sonnet}
+      temperature: 0.4
+      max_tokens: ${MAX_TOKENS:4096}       # coerced to int automatically
+      memory_enabled: ${ENABLE_MEMORY:true}  # coerced to bool automatically
+      system_prompt: |
+        You are a {{ context.role }} for {{ context.company }}.
+
+        {% if context.tier == "enterprise" %}
+        This is an enterprise customer. Provide priority support with
+        detailed technical explanations and offer to escalate issues
+        to the engineering team when needed.
+        {% else %}
+        Provide friendly, concise support. Direct complex issues to
+        the documentation at {{ context.docs_url }}.
+        {% endif %}
+
+        Always respond in {{ context.language | default("English") }}.
+
+routes:
+  main:
+    type: stream
+    operators:
+      - type: map
+        params:
+          agent: support
+    publications:
+      - output
+
+merges:
+  - sources: [output]
+    target: final
+
+# global_context populates the {{ context.* }} namespace at load time
+global_context:
+  company: "Acme Corp"
+  role: "technical support specialist"
+  tier: "enterprise"
+  docs_url: "https://docs.acme.example.com"
+
+ +This example shows the three most common Jinja2 preprocessing patterns: variable interpolation (`{{ context.company }}`), conditional blocks (`{% if context.tier == "enterprise" %}`), and the `| default` filter for safe fallbacks. Environment variables (`${LLM_PROVIDER:anthropic}`) are resolved in Phase 2 after YAML parsing, with automatic type coercion converting `"4096"` to `int` and `"true"` to `bool`. + +--- + +**Example 7: Multi-Actor Paper Writer with Advanced Jinja2 (Advanced Jinja2)** + +A comprehensive multi-actor configuration exercising the full Jinja2 preprocessing feature set. Crucially, this example places `{% for %}` and `{% if %}` directives **outside any YAML value** — directly at the structural level where YAML keys and list items would normally appear. The raw file is **not valid YAML** until after Jinja2 Phase 1 preprocessing renders it into static YAML text. Features demonstrated include: structural `{% for %}` loops that generate actor definitions, graph nodes, and graph edges; structural `{% if %}` conditionals that include or exclude entire actor blocks, route definitions, and merge sources; `{% if %}` nested inside `{% for %}` at the structural level; `loop.index`; Jinja2 type tests (`is mapping`); `.get()` with defaults; string slicing (`[:200]`); ternary expressions; arithmetic; `| tojson` / `| length` / `| upper` / `| join` filters; `{# comment #}` blocks; and deeply nested `global_context`. See [ADR-032](adr/ADR-032-jinja2-yaml-template-preprocessing.md) for full details. + +

+# paper-writer.yaml
+# Register: agents actor add --config paper-writer.yaml
+#
+# IMPORTANT: This file is NOT valid YAML as written. Jinja2 directives
+# ({% for %}, {% if %}, {% endif %}, {% endfor %}) appear at the structural
+# level — where YAML keys and list items would normally be — making the
+# raw file unparseable by any YAML parser. Phase 1 (Jinja2 preprocessing)
+# renders these directives into static YAML text BEFORE the YAML parser
+# ever sees the file. Jinja2 syntax inside system_prompt fields is
+# preserved for deferred runtime rendering.
+
+name: local/paper-writer
+
+cleveragents:
+  version: "3.0"
+  template_engine: "JINJA2"
+  logging:
+    level: "${LOG_LEVEL:INFO}"
+  unsafe: ${ALLOW_UNSAFE:false}       # Phase 2: coerced to bool
+  default_actor: orchestrator
+
+{# ─── Template comment: stripped from output, never reaches YAML parser ─── #}
+
+actors:
+  # ── Orchestrator: uses deferred Jinja2 in system_prompt ─────────────
+  orchestrator:
+    type: llm
+    config:
+      provider: ${LLM_PROVIDER:anthropic}
+      model: ${PRIMARY_MODEL:claude-3.5-sonnet}
+      temperature: 0.7
+      max_tokens: ${MAX_TOKENS:8192}
+      memory_enabled: true
+      max_history: ${MAX_HISTORY:100}
+      system_prompt: |
+        You are the lead orchestrator for a research paper.
+        Topic: {{ context.paper_details.topic | tojson }}
+        Audience: {{ context.paper_details.audience | tojson }}
+        Max length: {{ context.paper_details.length | tojson }} words
+
+        {# ── Vetted sources: for-loop with type test, .get(), slicing ── #}
+        {% if context.vetted_sources and context.vetted_sources|length > 0 %}
+        The following {{ context.vetted_sources|length }} vetted sources:
+        {% for source in context.vetted_sources %}
+        {{ loop.index }}.
+        {% if source is mapping %}
+          {{ source.get('citation', 'Untitled') }}
+          {% if source.get('summary') %}
+          — {{ source.get('summary')[:200] }}
+          {% if source.get('summary')|length > 200 %}
+          ...
+          {% endif %}
+          {% endif %}
+        {% else %}
+          {{ source }}
+        {% endif %}
+        {% endfor %}
+        {% else %}
+        No vetted sources are available yet. Begin with the discovery phase.
+        {% endif %}
+
+        {% if context.deadline %}
+        DEADLINE: {{ context.deadline }}. Prioritize accordingly.
+        {% endif %}
+
+        {# ── Section plan: ternary expression highlights current section ── #}
+        Section plan:
+        {% for section in context.sections %}
+        {% set m = ">>> " if section == context.current_section else "    " %}
+        {{ m }}{{ loop.index }}. {{ section }}
+        {% endfor %}
+
+        {# ── Arithmetic in expressions ── #}
+        Progress: section {{ context.current_section_index + 1 }}
+        of {{ context.sections|length }}.
+
+  # ── Writer: deferred templates with nested conditionals ─────────────
+  writer:
+    type: llm
+    config:
+      provider: ${LLM_PROVIDER:anthropic}
+      model: ${PRIMARY_MODEL:claude-3.5-sonnet}
+      temperature: 0.5
+      max_tokens: 16384
+      system_prompt: |
+        You are writing section "{{ context.current_section }}" of a paper
+        on {{ context.paper_details.topic }}.
+
+        {% if context.section_content %}
+        Previous draft:
+        {% set sec = context.current_section %}
+        {{ context.section_content.get(sec, 'No prior draft.') }}
+        {% endif %}
+
+        {# ── Nested: loop inside conditional ── #}
+        {% if context.review_feedback and context.review_feedback|length > 0 %}
+        Reviewer feedback to address:
+        {% for fb in context.review_feedback %}
+        [{{ fb.reviewer }}] ({{ fb.severity }}): {{ fb.comment }}
+        {% endfor %}
+        {% endif %}
+
+        Format: {{ context.paper_details.get('format', 'markdown') | upper }}
+
+  # ── STRUCTURAL {% if %}: the entire assembler actor definition — its YAML
+  # ── key and all nested content — is conditionally included. The {% if %}
+  # ── and {% endif %} lines occupy positions where YAML keys would be,
+  # ── making this raw text invalid YAML. After Phase 1 rendering, either
+  # ── the full assembler: block appears or nothing does.
+  {% if context.enable_assembly %}
+  assembler:
+    type: llm
+    config:
+      actor: anthropic/claude-3.5-sonnet
+      temperature: 0.3
+      max_tokens: 32768
+      system_prompt: |
+        Assemble the final paper from these completed sections:
+        {% for path in context.sections %}
+        --- {{ path }} ---
+        {{ context.section_content.get(path, '[MISSING]') }}
+        {% endfor %}
+
+        Total sections: {{ context.sections|length }}
+        Target length: {{ context.paper_details.length }} words
+
+        {% if context.latex_errors %}
+        Previous compilation errors (last 2000 chars):
+        {{ context.latex_errors[-2000:] }}
+        {% endif %}
+  {% endif %}
+
+  # ── STRUCTURAL {% for %}: GENERATE one reviewer actor per entry in
+  # ── context.reviewers. The {% for %} line sits where a YAML key would
+  # ── be — not inside any string value. A YAML parser would reject this.
+  # This {% for %} runs at Phase 1 and produces static YAML actor definitions.
+  # With 3 reviewers in global_context, the rendered YAML contains 3 actors:
+  # reviewer_methods, reviewer_domain, reviewer_style.
+  {% for reviewer in context.reviewers %}
+  reviewer_{{ reviewer.id }}:
+    type: llm
+    config:
+      provider: {{ reviewer.get('provider', 'openai') }}
+      model: {{ reviewer.get('model', 'gpt-4') }}
+      temperature: 0.2
+      max_tokens: 4096
+      system_prompt: |
+        You are {{ reviewer.name }}, an expert reviewer
+        specializing in {{ reviewer.specialty }}.
+
+        Evaluate the paper section for:
+        {# ── Nested loop: iterate criteria inside reviewer loop ── #}
+        {% for criterion in reviewer.criteria %}
+        - {{ criterion }}
+        {% endfor %}
+
+        Severity ratings: Critical, Major, Minor, Suggestion.
+
+        {% if context.review_mode == "strict" %}
+        Apply strict academic standards. Flag all unsupported claims.
+        {% else %}
+        Focus on substantive issues. Ignore minor style preferences.
+        {% endif %}
+  {% endfor %}
+
+# ── Routes: load-time loop generates graph nodes and edges ──────────
+routes:
+  writing_graph:
+    type: graph
+    nodes:
+      plan:
+        type: agent
+        agent: orchestrator
+      draft:
+        type: agent
+        agent: writer
+      # STRUCTURAL {% for %}: generates review_methods, review_domain,
+      # review_style as concrete YAML keys — invalid YAML until rendered
+      {% for reviewer in context.reviewers %}
+      review_{{ reviewer.id }}:
+        type: agent
+        agent: reviewer_{{ reviewer.id }}
+      {% endfor %}
+      # STRUCTURAL {% if %}: the assemble node only exists when assembly
+      # is enabled — matches the conditional assembler actor above
+      {% if context.enable_assembly %}
+      assemble:
+        type: agent
+        agent: assembler
+      {% endif %}
+    edges:
+      - source: plan
+        target: draft
+      # STRUCTURAL {% for %}: generates edge sets for each reviewer.
+      # Contains a NESTED STRUCTURAL {% if %} — the assemble edge only
+      # appears when enable_assembly is true. Both directives sit where
+      # YAML list items would be — completely invalid YAML until rendered.
+      {% for reviewer in context.reviewers %}
+      - source: draft
+        target: review_{{ reviewer.id }}
+      - source: review_{{ reviewer.id }}
+        target: draft
+        condition:
+          has_critical_feedback: true
+      # {% if %} NESTED inside {% for %}: each reviewer gets an edge
+      # to assemble only when the assembler exists
+      {% if context.enable_assembly %}
+      - source: review_{{ reviewer.id }}
+        target: assemble
+        condition:
+          review_passed: true
+      {% endif %}
+      {% endfor %}
+    entry_point: plan
+    checkpointing: true
+    checkpoint_dir: "${CHECKPOINT_DIR:/tmp/paper_checkpoints}"
+    parallel_execution: true
+
+  output_stream:
+    type: stream
+    operators:
+      - type: graph_execute
+        params:
+          graph: writing_graph
+    publications:
+      - paper_output
+
+  # ── STRUCTURAL {% if %}: this entire route definition — the YAML key
+  # ── "progress_stream:" and all its children — only exists in the
+  # ── rendered output when enable_monitoring is true. A YAML parser
+  # ── would choke on the bare {% if %} line sitting where it expects
+  # ── a mapping key.
+  {% if context.enable_monitoring %}
+  progress_stream:
+    type: stream
+    stream_type: hot
+    operators:
+      - type: map
+        params:
+          agent: orchestrator
+    subscriptions:
+      - writing_updates
+    publications:
+      - progress_output
+    buffer_size: 50
+  {% endif %}
+
+merges:
+  - sources:
+      - paper_output
+      # STRUCTURAL {% if %} inside a YAML list: this list item only
+      # appears in the rendered YAML when the condition is true.
+      # The raw file has a {% if %} line where a "- value" is expected.
+      {% if context.enable_monitoring %}
+      - progress_output
+      {% endif %}
+    target: final
+
+# ── global_context: deeply nested structures drive all template rendering ──
+global_context:
+  paper_details:
+    topic: "Alignment techniques in large language models"
+    audience: "ML researchers"
+    length: 8000
+    publication: "NeurIPS 2026"
+    format: "latex"
+  sections:
+    - "Abstract"
+    - "Introduction"
+    - "Related Work"
+    - "Methodology"
+    - "Experiments"
+    - "Results > Quantitative"
+    - "Results > Qualitative"
+    - "Discussion"
+    - "Conclusion"
+  current_section: "Introduction"
+  current_section_index: 1
+  deadline: "2026-06-01"
+  review_mode: "strict"
+  # These flags drive the structural {% if %} conditionals above.
+  # Set to false to exclude the assembler actor and monitoring route entirely.
+  enable_assembly: true
+  enable_monitoring: true
+  # The reviewers list drives the structural {% for %} loops that generate
+  # actor definitions, graph nodes, and graph edges at load time.
+  reviewers:
+    - id: methods
+      name: "Dr. Methods"
+      specialty: "research methodology"
+      provider: "openai"
+      model: "gpt-4"
+      criteria:
+        - "Statistical validity"
+        - "Reproducibility of experiments"
+        - "Clarity of methodology description"
+    - id: domain
+      name: "Dr. Domain"
+      specialty: "AI alignment"
+      provider: "anthropic"
+      model: "claude-3.5-sonnet"
+      criteria:
+        - "Technical accuracy"
+        - "Completeness of literature review"
+        - "Novelty of contributions"
+    - id: style
+      name: "Prof. Style"
+      specialty: "academic writing"
+      criteria:
+        - "Clarity and readability"
+        - "Logical flow between sections"
+        - "Proper citation format"
+
+ +This configuration demonstrates every major Jinja2 preprocessing feature: + +| Feature | Where Used | +|---------|-----------| +| **Structural `{% if %}`** wrapping entire YAML blocks | Assembler actor def, `assemble` graph node, `progress_stream` route, merge source list item | +| **Structural `{% for %}`** generating YAML keys/items | Reviewer actor defs, graph nodes, graph edges | +| **`{% if %}` nested inside `{% for %}`** at structural level | Reviewer→assemble edge (conditional per-reviewer) | +| `{{ context.X.Y }}` — nested variable access | Orchestrator, writer, assembler `system_prompt` | +| `{{ X \| tojson }}` — safe serialization filter | Orchestrator prompt: topic, audience, length | +| `{{ X \| length }}` — collection length | Source count, section count | +| `{{ X \| upper }}` — string transformation | Writer prompt: format output | +| `{{ X \| default("Y") }}` — fallback defaults | Reviewer `.get('provider', 'openai')` | +| `{% if X and X\|length > 0 %}` — compound conditions | Vetted sources, review feedback, deadline | +| `{% for X in Y %}` / `loop.index` — iteration | Sources list, sections list, reviewers list | +| `{# comment #}` — template comments | Stripped from rendered output | +| `source is mapping` — Jinja2 type test | Vetted sources loop | +| `.get('key', 'default')` — safe dict access | Section content, paper format | +| `[:200]` / `[-2000:]` — string/list slicing | Source summaries, LaTeX errors | +| `"X" if cond else "Y"` — ternary expressions | Section plan current-section marker | +| `{{ index + 1 }}` — arithmetic | Section progress counter | +| Nested `{% for %}` inside `{% for %}` | Reviewer criteria inside reviewer loop | +| `${VAR:default}` — env var with type coercion | Provider, model, tokens, unsafe flag | +| `global_context` with nested dicts/lists | `paper_details`, `sections`, `reviewers` | +| Deferred rendering in `system_prompt` | All `system_prompt` fields preserve `{{ }}` for runtime | + +**Why the raw file is not valid YAML**: The `{% for %}`, `{% endfor %}`, `{% if %}`, and `{% endif %}` directives in this file appear at positions where the YAML parser expects mapping keys or list items — for example, `{% if context.enable_assembly %}` sits at the same indentation level as `assembler:` under the `actors:` mapping, and `{% if context.enable_monitoring %}` sits where a route name key would be under `routes:`. A YAML parser would reject these lines as syntax errors. Jinja2 Phase 1 preprocessing resolves all directives into plain text *before* the YAML parser ever runs, producing a valid static YAML document. + +**Load-time vs. runtime rendering**: The structural `{% for %}` and `{% if %}` directives in the `actors`, `routes`, and `merges` sections run at **load time** (Phase 1) and produce static YAML — the three reviewer entries expand into three concrete actor definitions (`reviewer_methods`, `reviewer_domain`, `reviewer_style`), their corresponding graph nodes and edges, and the assembler and monitoring route conditionally appear or disappear. In contrast, Jinja2 syntax inside `system_prompt` fields is **preserved** through the load-parse cycle (via the template protection mechanism) and evaluated at **runtime** when the actor's execution context is available. + +--- + ### Skill Configuration Files !!! adr "Architecture Decision" diff --git a/mkdocs.yml b/mkdocs.yml index 7bddd90d3..54c587750 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -50,6 +50,7 @@ nav: - ADR-029 Model Context Protocol (MCP) Adoption: adr/ADR-029-model-context-protocol.md - ADR-030 Skill Abstraction Definition: adr/ADR-030-skill-abstraction-definition.md - ADR-031 Actor Abstraction Definition: adr/ADR-031-actor-abstraction-definition.md + - ADR-032 Jinja2 YAML Template Preprocessing: adr/ADR-032-jinja2-yaml-template-preprocessing.md theme: name: material