Docs: Added JINJA2 details to docs

docs/adr/ADR-010-actor-and-agent-architecture.md

@@ -125,6 +125,7 @@ Agent behavior is configurable without code changes: prompt templates, tool sets
 | [ADR-022](ADR-022-langchain-langgraph-integration.md) | LangChain/LangGraph Integration | Actors are implemented as LangGraph StateGraph instances |
 | [ADR-030](ADR-030-skill-abstraction-definition.md) | Skill Abstraction Definition | Skills are the unit of capability assignment to actors |
 | [ADR-031](ADR-031-actor-abstraction-definition.md) | Actor Abstraction Definition | Formalizes the canonical definition of what an actor is |
+| [ADR-032](ADR-032-jinja2-yaml-template-preprocessing.md) | Jinja2 YAML Template Preprocessing | Defines the two-phase Jinja2 + env var preprocessing pipeline for actor YAML files |
 
 ## Acceptance
 

docs/adr/ADR-031-actor-abstraction-definition.md

@@ -179,6 +179,7 @@ There is no mechanism for an actor to gain tool capabilities outside the skill s
 | [ADR-028](ADR-028-agent-skills-standard.md) | Agent Skills Standard (AgentSkills.io) | Agent Skills appear as tool nodes in actor graphs and extend actor knowledge |
 | [ADR-029](ADR-029-model-context-protocol.md) | Model Context Protocol (MCP) Adoption | MCP tools appear as tool nodes in actor graphs via skill composition |
 | [ADR-030](ADR-030-skill-abstraction-definition.md) | Skill Abstraction Definition | Skills are the unit of capability assignment to actors; the skill-actor binding model |
+| [ADR-032](ADR-032-jinja2-yaml-template-preprocessing.md) | Jinja2 YAML Template Preprocessing | Defines how actor YAML files are preprocessed with Jinja2 templates and environment variable interpolation before parsing |
 
 ## Acceptance
 

docs/adr/ADR-032-jinja2-yaml-template-preprocessing.md

@@ -0,0 +1,210 @@
+# ADR-032: Jinja2 YAML Template Preprocessing
+
+**Status:** Accepted  
+**Date:** 2026-02-19  
+**Supersedes:** None  
+**Author(s):** Jeffrey Phillips Freeman <Jeffrey.Freeman@CleverThis.com>  
+**Approver(s):** Jeffrey Phillips Freeman <Jeffrey.Freeman@CleverThis.com>
+
+## Context
+
+CleverAgents actor configuration files are written in YAML and define complex multi-actor workflows with system prompts, routing logic, and shared context. Many real-world configurations require dynamic content in system prompts — for example, a brainstorming actor needs to reference paper requirements collected during a prior discovery stage, or a vetting actor needs to conditionally change its behavior based on whether auto-finish mode is active.
+
+Standard YAML offers no native mechanism for dynamic string interpolation, conditional blocks, or iterative generation. Without a template preprocessor, authors would need to either: (a) hardcode all prompt content statically, severely limiting reusability; (b) write custom Python code for each dynamic configuration, undermining the declarative YAML-only philosophy; or (c) rely on runtime string replacement, which cannot express conditionals, loops, or filters.
+
+Furthermore, actor configurations frequently need environment variable substitution (e.g., API keys, directory paths) with default fallbacks and automatic type coercion, which YAML also lacks natively.
+
+The v2 system already used Jinja2 templating in its reactive configuration files. Preserving and formalizing this capability ensures backward compatibility and provides a proven, well-understood solution for dynamic YAML content.
+
+## Decision
+
+CleverAgents adopts a **two-phase YAML processing pipeline** for actor configuration files:
+
+1. **Phase 1 — Jinja2 Template Rendering**: Before YAML parsing, the raw file content is processed through a sandboxed Jinja2 template engine. This phase resolves `{{ variable }}` expressions, `{% if/for/block %}` control structures, and `{# comment #}` blocks into static YAML text.
+
+2. **Phase 2 — YAML Parsing and Environment Variable Interpolation**: The rendered YAML text is parsed by PyYAML's `safe_load`, then all string values matching `${VAR}` or `${VAR:default}` patterns are recursively replaced with environment variable values, with automatic type coercion for booleans, integers, and floats.
+
+This two-phase approach cleanly separates template-time logic (Jinja2, evaluated once at load time with a context dictionary) from runtime configuration (environment variables, evaluated at parse time from the OS environment).
+
+## Design
+
+### Phase 1: Jinja2 Template Engine (`YAMLTemplateEngine`)
+
+The `YAMLTemplateEngine` class provides the Jinja2 preprocessing layer:
+
+#### Sandboxed Execution
+
+All template rendering uses `jinja2.sandbox.SandboxedEnvironment`, which prevents templates from executing arbitrary Python code, accessing the filesystem, or performing other unsafe operations. The sandbox restricts attribute access and method calls to a safe subset.
+
+#### Template Delimiters
+
+Standard Jinja2 delimiters are used:
+
+| Delimiter | Purpose | Example |
+|-----------|---------|---------|
+| `{{ ... }}` | Variable expression | `{{ context.paper_details.topic }}` |
+| `{% ... %}` | Block statement (if, for, block, etc.) | `{% if context.auto_finish_active %}` |
+| `{# ... #}` | Comment (stripped from output) | `{# This is a template comment #}` |
+
+#### Custom Filters
+
+The engine registers four custom Jinja2 filters for YAML-specific use:
+
+| Filter | Purpose | Example |
+|--------|---------|---------|
+| `yaml` | Serializes a value to YAML string | `{{ my_dict \| yaml }}` |
+| `indent` | Indents text by N spaces | `{{ content \| indent(4) }}` |
+| `sum` | Sums a numeric sequence | `{{ values \| sum }}` |
+| `selectattr` | Selects attribute values from a sequence | `{{ items \| selectattr('name') }}` |
+
+All standard Jinja2 built-in filters (`tojson`, `default`, `lower`, `upper`, `trim`, `join`, `replace`, `length`, `first`, `last`, `sort`, `unique`, `map`, `reject`, `select`, `batch`, `slice`, `int`, `float`, `string`, `list`, `dictsort`, `e`, `escape`, `safe`, `truncate`, `wordwrap`, `center`, `format`, `title`, `capitalize`, `striptags`, `urlencode`, `abs`, `round`, `pprint`, `groupby`, `max`, `min`, `random`, `filesizeformat`, `xmlattr`, `wordcount`, `reverse`) are also available.
+
+#### Exposed Built-in Functions
+
+Safe Python built-in functions are exposed in the template context:
+
+| Function | Purpose |
+|----------|---------|
+| `range()` | Generates integer sequences |
+| `abs()` | Absolute value |
+| `round()` | Rounds a number |
+| `len()` | Length of a sequence |
+| `min()` | Minimum of a sequence |
+| `max()` | Maximum of a sequence |
+| `sum()` | Sum of a sequence |
+
+#### Context Resolution
+
+Template variables are resolved from a context dictionary provided at render time. The context dictionary supports a nested `context` key convention:
+
+- `{{ context.paper_details.topic }}` resolves `context["paper_details"]["topic"]`
+- `{{ context.brainstorming_summary }}` resolves `context["brainstorming_summary"]`
+
+The `global_context` top-level key in the actor configuration file populates this context dictionary at load time. At runtime, actor invocation context, plan context, and session context are merged into the template context following a defined precedence order.
+
+#### Template Detection
+
+The engine detects Jinja2 content by scanning for `{%` or `{{` markers in the raw YAML text. Files without these markers bypass Jinja2 processing entirely and are parsed as plain YAML, incurring no template overhead.
+
+#### Deferred Rendering
+
+When Jinja2 markers are detected but no context is available at load time (e.g., the context will only be available at runtime), the engine performs **deferred rendering**: it attempts to parse the YAML as-is (with template markers intact as literal strings). The templates are preserved in the parsed structure and rendered later when context becomes available.
+
+#### YAML Post-Processing
+
+After Jinja2 rendering, the engine applies post-processing to fix common issues introduced by template expansion:
+
+1. **Blank line removal**: Removes extraneous blank lines generated by `{% %}` block statements between YAML keys.
+2. **Multi-colon line splitting**: Fixes lines where template expansion produces multiple YAML key-value pairs on a single line.
+3. **Indentation correction**: Inserts indentation hints for `{% for %}` loops to ensure generated YAML maintains correct indentation.
+
+#### Template Protection for System Prompts
+
+A special protection mechanism preserves Jinja2 syntax in `system_prompt` fields. During initial loading, Jinja2 delimiters inside the YAML file are temporarily replaced with sentinel markers (`<<<TEMPLATE_START>>>`, `<<<TEMPLATE_END>>>`, `<<<BLOCK_START>>>`, `<<<BLOCK_END>>>`). After YAML parsing, these sentinels are restored to their original Jinja2 syntax exclusively within `system_prompt` fields. This allows system prompts to contain Jinja2 templates that are evaluated later at runtime (when the actor's context is available), rather than at file-load time.
+
+### Phase 2: Environment Variable Interpolation
+
+After YAML parsing, all string values in the configuration are recursively scanned for the pattern `${VAR_NAME}` or `${VAR_NAME:default_value}`:
+
+| Pattern | Behavior |
+|---------|----------|
+| `${VAR}` | Replaced with `os.environ["VAR"]`. Raises `ValueError` if not set. |
+| `${VAR:default}` | Replaced with `os.environ.get("VAR", "default")`. |
+
+#### Automatic Type Coercion
+
+After substitution, the resulting string value is coerced to its natural type:
+
+| String Value | Coerced Type | Result |
+|-------------|-------------|--------|
+| `"true"` / `"false"` (case-insensitive) | `bool` | `True` / `False` |
+| Digits only (with optional leading `-`) | `int` | e.g., `42`, `-7` |
+| Digits with single `.` | `float` | e.g., `3.14`, `-0.5` |
+| Anything else | `str` | Unchanged |
+
+This coercion applies to both environment variable values and their defaults, enabling configurations like `max_retries: ${MAX_RETRIES:3}` to produce an integer `3` rather than a string `"3"`.
+
+#### Recursion
+
+Environment variable interpolation is applied recursively to all nested dictionaries and lists in the parsed configuration, ensuring variables are resolved regardless of nesting depth.
+
+## Constraints
+
+- Template rendering must use `jinja2.sandbox.SandboxedEnvironment` exclusively. Unsandboxed Jinja2 environments are not permitted for actor configuration processing.
+- Environment variable interpolation must not raise exceptions for undefined variables when a default value is provided via the `${VAR:default}` syntax.
+- Template delimiters in `system_prompt` fields are preserved for runtime evaluation rather than consumed at load time. This is the only field that receives this special treatment.
+- Files without Jinja2 markers (`{%` or `{{`) must bypass the template engine entirely with no behavioral difference from direct YAML parsing.
+- The Jinja2 template phase must complete before environment variable interpolation begins. The two phases must not be interleaved.
+
+## Consequences
+
+### Positive
+- Actor configurations can express dynamic, context-dependent system prompts without writing Python code, preserving the declarative YAML-only philosophy.
+- Conditional blocks (`{% if %}`) enable a single configuration file to define behavior for multiple modes (e.g., auto-finish vs. interactive), reducing configuration file proliferation.
+- Environment variable interpolation with defaults provides a clean mechanism for per-environment configuration (development vs. production) without modifying YAML files.
+- Automatic type coercion eliminates the need for explicit casting in configuration consumers.
+- Full backward compatibility with v2 configurations that already use Jinja2 templates.
+- The sandboxed environment prevents template injection attacks in user-provided configuration files.
+
+### Negative
+- Jinja2 template syntax errors are reported at YAML load time rather than at validation time, making debugging harder because the error context shows rendered (or partially rendered) output rather than the source template.
+- The template protection mechanism (sentinel replacement) for `system_prompt` fields adds complexity and could interact poorly with unusual YAML structures.
+- Authors must understand both YAML syntax and Jinja2 syntax, increasing the learning curve for configuration authoring.
+
+### Risks
+- Complex template logic in YAML files may become difficult to maintain and debug, especially when templates generate YAML structure (keys and values) rather than just string content.
+- The post-processing heuristics (blank line removal, colon splitting) may produce incorrect results for edge cases in generated YAML.
+- Deferred rendering may silently produce incorrect configurations if the expected context variables are not available at runtime.
+
+## Alternatives Considered
+
+**No template engine (static YAML only)** — Would require separate YAML files for each configuration variant (e.g., auto-finish mode vs. interactive mode) and prevent dynamic prompt content. Rejected because it undermines the goal of single-file, self-contained actor definitions for complex workflows.
+
+**Mako or Cheetah templates** — Alternative Python template engines. Jinja2 is the most widely used, best documented, and provides a sandboxed execution mode specifically designed for untrusted templates. Jinja2 is also already a transitive dependency via LangChain. Rejected in favor of Jinja2.
+
+**Runtime-only string interpolation (no load-time templates)** — Would defer all template evaluation to runtime. This prevents generating YAML structure from templates (e.g., generating actor definitions with `{% for %}`), which is needed for configuration DRY patterns. The two-phase approach (load-time Jinja2 + runtime env vars) provides the correct separation. Rejected.
+
+**YAML anchors and aliases** — YAML's built-in reuse mechanism. Anchors support structural reuse but not conditionals, loops, or string interpolation. Useful as a complement to Jinja2 templates but not a replacement. Not rejected — can be used alongside Jinja2.
+
+## Compliance
+
+- **Sandbox enforcement tests**: Verify that the template engine uses `SandboxedEnvironment` and that attempts to access unsafe operations (file I/O, `os.system`, `eval`) are rejected.
+- **Template detection tests**: Verify that files without `{%` or `{{` markers bypass the template engine entirely.
+- **Environment variable interpolation tests**: Verify `${VAR}`, `${VAR:default}`, type coercion (bool, int, float), and error behavior for missing variables without defaults.
+- **Template protection tests**: Verify that Jinja2 syntax in `system_prompt` fields survives the load-parse cycle and is available for runtime evaluation.
+- **Post-processing tests**: Verify that rendered YAML with blank lines, multi-colon lines, and indentation issues is correctly fixed.
+- **Filter and built-in tests**: Verify that all custom filters (`yaml`, `indent`, `sum`, `selectattr`) and exposed built-ins (`range`, `abs`, `round`, `len`, `min`, `max`, `sum`) work correctly in template expressions.
+- **BDD feature coverage**: `actor_config_coverage.feature`, `actor_config_new_coverage.feature`, and `yaml_template_engine_coverage.feature` provide end-to-end coverage of the two-phase pipeline.
+
+## Related ADRs
+
+| ADR | Title | Relationship |
+|-----|-------|-------------|
+| [ADR-010](ADR-010-actor-and-agent-architecture.md) | Actor and Agent Architecture | Defines the actor configuration model that this ADR's template engine processes |
+| [ADR-024](ADR-024-configuration-system.md) | Configuration System | Defines the overall configuration approach; this ADR extends it with Jinja2 preprocessing for actor YAML files |
+| [ADR-031](ADR-031-actor-abstraction-definition.md) | Actor Abstraction Definition | Defines what an actor is; this ADR defines how actor YAML files are preprocessed before validation |
+| [ADR-005](ADR-005-technical-stack.md) | Technical Stack | Jinja2 is added as a required dependency in the technical stack |
+
+## Acceptance
+
+### Votes For
+
+| Voter | Comment |
+|-------|---------|
+| Jeffrey Phillips Freeman <Jeffrey.Freeman@CleverThis.com> | Two-phase Jinja2 + env var preprocessing provides the right balance of power and safety for declarative actor configurations |
+
+**Total: 1**
+
+### Votes Against
+
+| Voter | Comment |
+|-------|---------|
+
+**Total: 0**
+
+### Abstentions
+
+| Voter | Comment |
+|-------|---------|
+
+**Total: 0**

docs/adr/index.md

@@ -118,10 +118,11 @@ These ADRs address external integrations, operational interfaces, and deployment
 | [ADR-029](ADR-029-model-context-protocol.md) | Model Context Protocol (MCP) Adoption | MCP tool discovery, registry integration, skill composition, and actor-graph usage. |
 | [ADR-030](ADR-030-skill-abstraction-definition.md) | Skill Abstraction Definition | Canonical definition of a skill as a composable collection of tools from four sources (MCP, Agent Skills, built-in, custom). |
 | [ADR-031](ADR-031-actor-abstraction-definition.md) | Actor Abstraction Definition | Canonical definition of an actor as anything conversational — single LLM, composed graph, or hierarchical orchestrator. |
+| [ADR-032](ADR-032-jinja2-yaml-template-preprocessing.md) | Jinja2 YAML Template Preprocessing | Two-phase YAML processing: sandboxed Jinja2 template rendering followed by environment variable interpolation with type coercion. |
 
 ## Creating a New ADR
 
-1. Assign the next sequential number (e.g., `ADR-032`).
+1. Assign the next sequential number (e.g., `ADR-033`).
 2. Create a file named `ADR-0NN-<kebab-case-title>.md` in this directory.
 3. Copy the template structure from any existing ADR.
 4. Fill in the metadata table with `Status: Proposed`, the current date, author(s), and leave approver(s) blank.

docs/gen_ref_pages.py

@@ -79,6 +79,17 @@ for path in sorted(SRC_DIR.rglob("*.py")):
 # Include manually-authored reference pages that live alongside the
 # auto-generated API docs.
 nav[("actors_schema",)] = "actors_schema.md"
+nav[("automation_profile_service",)] = "automation_profile_service.md"
+nav[("automation_profiles",)] = "automation_profiles.md"
+nav[("changeset_model",)] = "changeset_model.md"
+nav[("cli_system_commands",)] = "cli_system_commands.md"
+nav[("database_schema",)] = "database_schema.md"
+nav[("diagnostics_checks",)] = "diagnostics_checks.md"
+nav[("project_context_cli",)] = "project_context_cli.md"
+nav[("project_context_policy",)] = "project_context_policy.md"
+nav[("resource_dag",)] = "resource_dag.md"
+nav[("resource_types_builtin",)] = "resource_types_builtin.md"
+nav[("tool_bindings",)] = "tool_bindings.md"
 
 # Write the literate-nav summary file for the Reference section
 with mkdocs_gen_files.open("reference/SUMMARY.md", "w") as nav_file:

docs/reference/tool_bindings.md

@@ -127,6 +127,6 @@ resource_slots:
 
 See:
 
-- [`BindingResolutionService`](../../src/cleveragents/application/services/binding_resolution_service.py)
-- [`BindingResult`](../../src/cleveragents/domain/models/core/resource_slot.py)
-- [`ResourceSlot`](../../src/cleveragents/domain/models/core/tool.py)
+- [`BindingResolutionService`](cleveragents/application/services/binding_resolution_service.md)
+- [`BindingResult`](cleveragents/domain/models/core/resource_slot.md)
+- [`ResourceSlot`](cleveragents/domain/models/core/tool.md)

mkdocs.yml

@@ -50,6 +50,7 @@ nav:
     - ADR-029 Model Context Protocol (MCP) Adoption: adr/ADR-029-model-context-protocol.md
     - ADR-030 Skill Abstraction Definition: adr/ADR-030-skill-abstraction-definition.md
     - ADR-031 Actor Abstraction Definition: adr/ADR-031-actor-abstraction-definition.md
+    - ADR-032 Jinja2 YAML Template Preprocessing: adr/ADR-032-jinja2-yaml-template-preprocessing.md
 
 theme:
   name: material