cleveragents-core/docs/adr/ADR-032-jinja2-yaml-template-preprocessing.md

adr_number, title, status_history, tier, authors, superseded_by, related_adrs, acceptance

adr_number	title	status_history	tier	authors	superseded_by	related_adrs	acceptance
32	Jinja2 YAML Template Preprocessing	2026-02-19 Proposed Jeffrey Phillips Freeman 2026-02-19 Accepted Jeffrey Phillips Freeman	4	Jeffrey Phillips Freeman	null	number title relationship 10 Actor and Agent Architecture Defines the actor configuration model that this ADR's template engine processes number title relationship 24 Configuration System Defines the overall configuration approach; this ADR extends it with Jinja2 preprocessing for actor YAML files number title relationship 31 Actor Abstraction Definition Defines what an actor is; this ADR defines how actor YAML files are preprocessed before validation number title relationship 5 Technical Stack Jinja2 is added as a required dependency in the technical stack	votes_for votes_against abstentions voter comment Jeffrey Phillips Freeman <Jeffrey.Freeman@CleverThis.com> Two-phase Jinja2 + env var preprocessing provides the right balance of power and safety for declarative actor configurations

Context

CleverAgents actor configuration files are written in YAML and define complex multi-actor workflows with system prompts, routing logic, and shared context. Many real-world configurations require dynamic content in system prompts — for example, a brainstorming actor needs to reference paper requirements collected during a prior discovery stage, or a vetting actor needs to conditionally change its behavior based on whether auto-finish mode is active.

Standard YAML offers no native mechanism for dynamic string interpolation, conditional blocks, or iterative generation. Without a template preprocessor, authors would need to either: (a) hardcode all prompt content statically, severely limiting reusability; (b) write custom Python code for each dynamic configuration, undermining the declarative YAML-only philosophy; or (c) rely on runtime string replacement, which cannot express conditionals, loops, or filters.

Furthermore, actor configurations frequently need environment variable substitution (e.g., API keys, directory paths) with default fallbacks and automatic type coercion, which YAML also lacks natively.

The v2 system already used Jinja2 templating in its reactive configuration files. Preserving and formalizing this capability ensures backward compatibility and provides a proven, well-understood solution for dynamic YAML content.

Decision Drivers

• Actor configurations require dynamic content in system prompts (e.g., context-dependent instructions, conditional behavior) that static YAML cannot express
• The declarative YAML-only philosophy must be preserved; authors should not need to write Python code for dynamic configurations
• Template-time logic (conditionals, loops, variable expansion) and runtime configuration (environment variables) must be cleanly separated into distinct phases
• Jinja2 templates in system_prompt fields must survive file loading and be preserved for runtime evaluation when actor context becomes available
• Template execution must be sandboxed to prevent arbitrary code execution or filesystem access from user-provided configuration files
• Backward compatibility with v2 configurations that already use Jinja2 templating must be maintained

Decision

CleverAgents adopts a two-phase YAML processing pipeline for actor configuration files:

1. Phase 1 — Jinja2 Template Rendering: Before YAML parsing, the raw file content is processed through a sandboxed Jinja2 template engine. This phase resolves {{ variable }} expressions, {% if/for/block %} control structures, and {# comment #} blocks into static YAML text.

2. Phase 2 — YAML Parsing and Environment Variable Interpolation: The rendered YAML text is parsed by PyYAML's safe_load, then all string values matching ${VAR} or ${VAR:default} patterns are recursively replaced with environment variable values, with automatic type coercion for booleans, integers, and floats.

This two-phase approach cleanly separates template-time logic (Jinja2, evaluated once at load time with a context dictionary) from runtime configuration (environment variables, evaluated at parse time from the OS environment).

Design

Phase 1: Jinja2 Template Engine (YAMLTemplateEngine)

The YAMLTemplateEngine class provides the Jinja2 preprocessing layer:

Sandboxed Execution

All template rendering uses jinja2.sandbox.SandboxedEnvironment, which prevents templates from executing arbitrary Python code, accessing the filesystem, or performing other unsafe operations. The sandbox restricts attribute access and method calls to a safe subset.

Template Delimiters

Standard Jinja2 delimiters are used:

Delimiter	Purpose	Example
{{ ... }}	Variable expression	{{ context.paper_details.topic }}
{% ... %}	Block statement (if, for, block, etc.)	{% if context.auto_finish_active %}
{# ... #}	Comment (stripped from output)	{# This is a template comment #}

Custom Filters

The engine registers four custom Jinja2 filters for YAML-specific use:

Filter	Purpose	Example
yaml	Serializes a value to YAML string	{{ my_dict | yaml }}
indent	Indents text by N spaces	{{ content | indent(4) }}
sum	Sums a numeric sequence	{{ values | sum }}
selectattr	Selects attribute values from a sequence	{{ items | selectattr('name') }}

All standard Jinja2 built-in filters (tojson, default, lower, upper, trim, join, replace, length, first, last, sort, unique, map, reject, select, batch, slice, int, float, string, list, dictsort, e, escape, safe, truncate, wordwrap, center, format, title, capitalize, striptags, urlencode, abs, round, pprint, groupby, max, min, random, filesizeformat, xmlattr, wordcount, reverse) are also available.

Exposed Built-in Functions

Safe Python built-in functions are exposed in the template context:

Function	Purpose
range()	Generates integer sequences
abs()	Absolute value
round()	Rounds a number
len()	Length of a sequence
min()	Minimum of a sequence
max()	Maximum of a sequence
sum()	Sum of a sequence

Context Resolution

Template variables are resolved from a context dictionary provided at render time. The context dictionary supports a nested context key convention:

• {{ context.paper_details.topic }} resolves context["paper_details"]["topic"]
• {{ context.brainstorming_summary }} resolves context["brainstorming_summary"]

The global_context top-level key in the actor configuration file populates this context dictionary at load time. At runtime, actor invocation context, plan context, and session context are merged into the template context following a defined precedence order.

Template Detection

The engine detects Jinja2 content by scanning for {% or {{ markers in the raw YAML text. Files without these markers bypass Jinja2 processing entirely and are parsed as plain YAML, incurring no template overhead.

Deferred Rendering

When Jinja2 markers are detected but no context is available at load time (e.g., the context will only be available at runtime), the engine performs deferred rendering: it attempts to parse the YAML as-is (with template markers intact as literal strings). The templates are preserved in the parsed structure and rendered later when context becomes available.

YAML Post-Processing

After Jinja2 rendering, the engine applies post-processing to fix common issues introduced by template expansion:

1. Blank line removal: Removes extraneous blank lines generated by {% %} block statements between YAML keys.
2. Multi-colon line splitting: Fixes lines where template expansion produces multiple YAML key-value pairs on a single line.
3. Indentation correction: Inserts indentation hints for {% for %} loops to ensure generated YAML maintains correct indentation.

Template Protection for System Prompts

A special protection mechanism preserves Jinja2 syntax in system_prompt fields. During initial loading, Jinja2 delimiters inside the YAML file are temporarily replaced with sentinel markers (<<<TEMPLATE_START>>>, <<<TEMPLATE_END>>>, <<<BLOCK_START>>>, <<<BLOCK_END>>>). After YAML parsing, these sentinels are restored to their original Jinja2 syntax exclusively within system_prompt fields. This allows system prompts to contain Jinja2 templates that are evaluated later at runtime (when the actor's context is available), rather than at file-load time.

Phase 2: Environment Variable Interpolation

After YAML parsing, all string values in the configuration are recursively scanned for the pattern ${VAR_NAME} or ${VAR_NAME:default_value}:

Pattern	Behavior
${VAR}	Replaced with os.environ["VAR"]. Raises ValueError if not set.
${VAR:default}	Replaced with os.environ.get("VAR", "default").

Automatic Type Coercion

After substitution, the resulting string value is coerced to its natural type:

String Value	Coerced Type	Result
"true" / "false" (case-insensitive)	bool	True / False
Digits only (with optional leading -)	int	e.g., 42, -7
Digits with single .	float	e.g., 3.14, -0.5
Anything else	str	Unchanged

This coercion applies to both environment variable values and their defaults, enabling configurations like max_retries: ${MAX_RETRIES:3} to produce an integer 3 rather than a string "3".

Recursion

Environment variable interpolation is applied recursively to all nested dictionaries and lists in the parsed configuration, ensuring variables are resolved regardless of nesting depth.

Constraints

• Template rendering must use jinja2.sandbox.SandboxedEnvironment exclusively. Unsandboxed Jinja2 environments are not permitted for actor configuration processing.
• Environment variable interpolation must not raise exceptions for undefined variables when a default value is provided via the ${VAR:default} syntax.
• Template delimiters in system_prompt fields are preserved for runtime evaluation rather than consumed at load time. This is the only field that receives this special treatment.
• Files without Jinja2 markers ({% or {{) must bypass the template engine entirely with no behavioral difference from direct YAML parsing.
• The Jinja2 template phase must complete before environment variable interpolation begins. The two phases must not be interleaved.

Consequences

Positive

• Actor configurations can express dynamic, context-dependent system prompts without writing Python code, preserving the declarative YAML-only philosophy.
• Conditional blocks ({% if %}) enable a single configuration file to define behavior for multiple modes (e.g., auto-finish vs. interactive), reducing configuration file proliferation.
• Environment variable interpolation with defaults provides a clean mechanism for per-environment configuration (development vs. production) without modifying YAML files.
• Automatic type coercion eliminates the need for explicit casting in configuration consumers.
• Full backward compatibility with v2 configurations that already use Jinja2 templates.
• The sandboxed environment prevents template injection attacks in user-provided configuration files.

Negative

• Jinja2 template syntax errors are reported at YAML load time rather than at validation time, making debugging harder because the error context shows rendered (or partially rendered) output rather than the source template.
• The template protection mechanism (sentinel replacement) for system_prompt fields adds complexity and could interact poorly with unusual YAML structures.
• Authors must understand both YAML syntax and Jinja2 syntax, increasing the learning curve for configuration authoring.

Risks

• Complex template logic in YAML files may become difficult to maintain and debug, especially when templates generate YAML structure (keys and values) rather than just string content.
• The post-processing heuristics (blank line removal, colon splitting) may produce incorrect results for edge cases in generated YAML.
• Deferred rendering may silently produce incorrect configurations if the expected context variables are not available at runtime.

Alternatives Considered

No template engine (static YAML only) — Would require separate YAML files for each configuration variant (e.g., auto-finish mode vs. interactive mode) and prevent dynamic prompt content. Rejected because it undermines the goal of single-file, self-contained actor definitions for complex workflows.

Mako or Cheetah templates — Alternative Python template engines. Jinja2 is the most widely used, best documented, and provides a sandboxed execution mode specifically designed for untrusted templates. Jinja2 is also already a transitive dependency via LangChain. Rejected in favor of Jinja2.

Runtime-only string interpolation (no load-time templates) — Would defer all template evaluation to runtime. This prevents generating YAML structure from templates (e.g., generating actor definitions with {% for %}), which is needed for configuration DRY patterns. The two-phase approach (load-time Jinja2 + runtime env vars) provides the correct separation. Rejected.

YAML anchors and aliases — YAML's built-in reuse mechanism. Anchors support structural reuse but not conditionals, loops, or string interpolation. Useful as a complement to Jinja2 templates but not a replacement. Not rejected — can be used alongside Jinja2.

Compliance

• Sandbox enforcement tests: Verify that the template engine uses SandboxedEnvironment and that attempts to access unsafe operations (file I/O, os.system, eval) are rejected.
• Template detection tests: Verify that files without {% or {{ markers bypass the template engine entirely.
• Environment variable interpolation tests: Verify ${VAR}, ${VAR:default}, type coercion (bool, int, float), and error behavior for missing variables without defaults.
• Template protection tests: Verify that Jinja2 syntax in system_prompt fields survives the load-parse cycle and is available for runtime evaluation.
• Post-processing tests: Verify that rendered YAML with blank lines, multi-colon lines, and indentation issues is correctly fixed.
• Filter and built-in tests: Verify that all custom filters (yaml, indent, sum, selectattr) and exposed built-ins (range, abs, round, len, min, max, sum) work correctly in template expressions.
• BDD feature coverage: actor_config_coverage.feature, actor_config_new_coverage.feature, and yaml_template_engine_coverage.feature provide end-to-end coverage of the two-phase pipeline.