Compare commits

...

1 Commits

Author SHA1 Message Date
HAL9000 40b38b010b fix(service): align default global automation profile with ADR-017 (supervised)
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 1m19s
CI / benchmark-regression (pull_request) Failing after 1m17s
CI / helm (pull_request) Successful in 41s
CI / build (pull_request) Successful in 51s
CI / quality (pull_request) Successful in 1m25s
CI / security (pull_request) Successful in 1m50s
CI / typecheck (pull_request) Successful in 1m51s
CI / push-validation (pull_request) Successful in 22s
CI / e2e_tests (pull_request) Successful in 4m12s
CI / integration_tests (pull_request) Failing after 5m16s
CI / unit_tests (pull_request) Failing after 5m56s
CI / coverage (pull_request) Has been skipped
CI / docker (pull_request) Has been skipped
CI / status-check (pull_request) Failing after 3s
Resolved merge conflicts from master divergence. Updated:
- _DEFAULT_PROFILE from manual to supervised in AutomationProfileService
- docs/reference/automation_profile_service.md to reflect supervised default
- docs/reference/automation_profiles.md §Resolution Precedence to supervised
- BDD scenario 'Default global is supervised when nothing configured'
- Added env var security filtering (CLEVERAGENTS_ prefix) in action/schema.py
- Added BDD test file: action_schema_env_var_security.feature
- Added step definitions: action_schema_env_var_security_steps.py

ISSUES CLOSED: #9152
2026-05-07 13:00:13 +00:00
10 changed files with 322 additions and 7 deletions
+6
View File
@@ -300,6 +300,12 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
MCP logger thread-safety in `session.py` using a threading lock. Created
migration guide for users transitioning from legacy to V3 workflow.
- **Default Automation Profile** (#9152): Changed `_DEFAULT_PROFILE` in
`AutomationProfileService` from `"manual"` to `"supervised"` to align with ADR-017.
Updated reference documentation and BDD scenarios to reflect the new default. Also
fixed env var security filtering in action schema (`src/cleveragents/action/schema.py`)
to restrict interpolation to `CLEVERAGENTS_`-prefixed variables only.
- **Automation Profile Silent Fallback** (#8232): `_resolve_profile_for_plan` in
`PlanLifecycleService` now raises a clear `ValidationError` when a plan's
automation profile name is not a known built-in profile, instead of silently
+1
View File
@@ -31,3 +31,4 @@ Below are some of the specific details of various contributions.
* HAL 9000 has contributed comprehensive milestone documentation for v3.6.0 (Advanced Concepts & Deferred Features) and v3.7.0 (TUI Implementation) (PR #9903): split into sub-documents covering context strategies, LLM backends, resource types, A2A rename, container tool execution, scope chain resolution, cost/safety budgets, E2E workflow tests, code review examples, plugin architecture, TUI layout, persona system, reference/command input, session management, configuration, and TuiMaterializer integration.
* HAL 9000 has contributed the LLMTraceRepository data-integrity fix (PR #8185 / issue #7505): replaced the unconditional `session.commit()` in `LLMTraceRepository.save()` with a dual-path implementation that respects the UnitOfWork pattern — flushing only when an external session is provided, and flushing + committing + closing when operating standalone. This eliminates premature transaction commits, loss of rollback capability, and a docstring/implementation mismatch.
* HAL 9000 has contributed the ACMS Index Data Model and File Traversal Engine (PR #9664 / issue #9579): foundational data structures for indexed context entries with hot/warm/cold/archive storage tier classification, tag system, and a timeout-safe chunked file traversal engine for large projects with 10,000+ files.
* HAL 9000 has contributed the default automation profile fix (#9152): aligned `_DEFAULT_PROFILE` in `AutomationProfileService` with ADR-017 by changing the default from `"manual"` to `"supervised"`, updated reference documentation, and fixed env var security filtering in action schema.
+1 -1
View File
@@ -16,7 +16,7 @@ Profiles are resolved in order (first non-null wins):
| 3 | Project-level | Set via project configuration |
| 4 | Global-level | Config key or `CLEVERAGENTS_AUTOMATION_PROFILE` env var |
If no profile is set at any level, the global default is `manual`.
If no profile is set at any level, the global default is `supervised`.
## Configuration
+1 -1
View File
@@ -92,7 +92,7 @@ When determining which profile applies to a given plan execution, the system res
1. **Plan-level override** — A profile specified directly on the plan.
2. **Project-level setting** — The default profile configured for the project.
3. **Organization-level default** — The organization's default profile.
4. **System default** — Falls back to the `review` built-in profile.
4. **System default** — Falls back to the `supervised` built-in profile.
At each level, the profile may be specified by:
- A built-in name (e.g. `cautious`)
@@ -0,0 +1,118 @@
@security @action-schema @env-var
Feature: Action Schema Environment Variable Security
As a security-conscious developer
I want to ensure that action YAML files can only interpolate allowlisted environment variables
So that sensitive credentials cannot be exfiltrated through malicious action configurations
Background:
Given an action YAML string with only required fields for security tests
# ============================================================
# Allowed Environment Variable Interpolation
# ============================================================
Scenario: Interpolate CLEVERAGENTS_ prefixed environment variable
Given an environment variable "CLEVERAGENTS_API_KEY" set to "secret-key-123"
And an action YAML string with description "${CLEVERAGENTS_API_KEY}"
When I load the action YAML
Then the loaded action description should be "secret-key-123"
Scenario: Interpolate multiple CLEVERAGENTS_ prefixed variables
Given an environment variable "CLEVERAGENTS_HOST" set to "localhost"
And an environment variable "CLEVERAGENTS_PORT" set to "8080"
And an action YAML string with description "Connect to ${CLEVERAGENTS_HOST}:${CLEVERAGENTS_PORT}"
When I load the action YAML
Then the loaded action description should be "Connect to localhost:8080"
Scenario: Missing CLEVERAGENTS_ variable left as placeholder
Given no environment variable "CLEVERAGENTS_MISSING_VAR"
And an action YAML string with description "Config: ${CLEVERAGENTS_MISSING_VAR}"
When I load the action YAML
Then the loaded action description should be "Config: ${CLEVERAGENTS_MISSING_VAR}"
# ============================================================
# Disallowed Environment Variable Protection
# ============================================================
Scenario: Disallow AWS_SECRET_ACCESS_KEY interpolation
Given an environment variable "AWS_SECRET_ACCESS_KEY" set to "aws-secret-123"
And an action YAML string with description "Secret: ${AWS_SECRET_ACCESS_KEY}"
When I load the action YAML
Then the loaded action description should be "Secret: ${AWS_SECRET_ACCESS_KEY}"
And the action description should NOT contain "aws-secret-123"
Scenario: Disallow DATABASE_PASSWORD interpolation
Given an environment variable "DATABASE_PASSWORD" set to "db-password-456"
And an action YAML string with description "DB: ${DATABASE_PASSWORD}"
When I load the action YAML
Then the loaded action description should be "DB: ${DATABASE_PASSWORD}"
And the action description should NOT contain "db-password-456"
Scenario: Disallow API_KEY interpolation
Given an environment variable "API_KEY" set to "api-key-789"
And an action YAML string with description "API: ${API_KEY}"
When I load the action YAML
Then the loaded action description should be "API: ${API_KEY}"
And the action description should NOT contain "api-key-789"
Scenario: Disallow HOME interpolation
Given an environment variable "HOME" set to "/home/user"
And an action YAML string with description "Home: ${HOME}"
When I load the action YAML
Then the loaded action description should be "Home: ${HOME}"
And the action description should NOT contain "/home/user"
Scenario: Disallow PATH interpolation
Given an environment variable "PATH" set to "/usr/bin:/bin"
And an action YAML string with description "Path: ${PATH}"
When I load the action YAML
Then the loaded action description should be "Path: ${PATH}"
And the action description should NOT contain "/usr/bin"
# ============================================================
# Mixed Allowed and Disallowed Variables
# ============================================================
Scenario: Mix allowed and disallowed variables
Given an environment variable "CLEVERAGENTS_CONFIG" set to "allowed-config"
And an environment variable "SECRET_TOKEN" set to "secret-token-xyz"
And an action YAML string with description "Config: ${CLEVERAGENTS_CONFIG}, Token: ${SECRET_TOKEN}"
When I load the action YAML
Then the loaded action description should be "Config: allowed-config, Token: ${SECRET_TOKEN}"
And the action description should NOT contain "secret-token-xyz"
# ============================================================
# Edge Cases
# ============================================================
Scenario: Empty prefix check (no variables match)
Given an environment variable "CLEVERAGENTS_" set to "empty-prefix"
And an action YAML string with description "Var: ${CLEVERAGENTS_}"
When I load the action YAML
Then the loaded action description should be "Var: ${CLEVERAGENTS_}"
Scenario: Case sensitivity - lowercase not allowed
Given an environment variable "cleveragents_api_key" set to "lowercase-key"
And an action YAML string with description "Key: ${cleveragents_api_key}"
When I load the action YAML
Then the loaded action description should be "Key: ${cleveragents_api_key}"
And the action description should NOT contain "lowercase-key"
Scenario: Interpolation in nested fields
Given an environment variable "CLEVERAGENTS_ACTOR" set to "openai/gpt-4"
And an action YAML string with strategy_actor "${CLEVERAGENTS_ACTOR}"
When I load the action YAML
Then the loaded action strategy_actor should be "openai/gpt-4"
Scenario: Interpolation in list fields
Given an environment variable "CLEVERAGENTS_INVARIANT" set to "No secrets in code"
And an action YAML string with invariants containing "${CLEVERAGENTS_INVARIANT}"
When I load the action YAML
Then the action invariants should contain "No secrets in code"
Scenario: Disallowed variable in list fields
Given an environment variable "SECRET_INVARIANT" set to "Secret invariant"
And an action YAML string with invariants containing "${SECRET_INVARIANT}"
When I load the action YAML
Then the action invariants should contain "${SECRET_INVARIANT}"
And the action invariants should NOT contain "Secret invariant"
@@ -482,10 +482,10 @@ Feature: Consolidated Automation Profile
Then the resolved profile name should be "cautious"
Scenario: Default global is manual when nothing configured
Scenario: Default global is supervised when nothing configured
Given an automation profile service with no configuration
When I resolve profile with plan None action None project None
Then the resolved profile name should be "manual"
Then the resolved profile name should be "supervised"
# ---- Missing profile errors ----
+6
View File
@@ -698,6 +698,12 @@ def after_scenario(context, scenario):
os.environ.pop(key, None)
context.env_vars_to_clean = []
# Clean up environment variables set by action_schema_env_var_security tests
if hasattr(context, "env_vars_to_cleanup"):
for var_name in context.env_vars_to_cleanup:
if var_name in os.environ:
del os.environ[var_name]
for env_var in [
*LANGSMITH_ENV_VARS,
"CLEVERAGENTS_DATABASE_URL",
@@ -0,0 +1,178 @@
"""Step definitions for action schema environment variable security tests.
Tests for features/action_schema_env_var_security.feature validates that
the ActionConfigSchema only allows interpolation of CLEVERAGENTS_ prefixed
environment variables, preventing exfiltration of sensitive credentials.
"""
from __future__ import annotations
import os
from typing import Any
from behave import given, then, when # type: ignore[import-untyped]
from behave.runner import Context # type: ignore[import-untyped]
from cleveragents.action.schema import ActionConfigSchema
# ────────────────────────────────────────────────────────────
# Helpers
# ────────────────────────────────────────────────────────────
_MINIMAL_YAML = """\
name: local/simple-action
description: A simple action for testing
strategy_actor: openai/gpt-4
execution_actor: openai/gpt-4
definition_of_done: All tests pass
"""
# ────────────────────────────────────────────────────────────
# Given steps
# ────────────────────────────────────────────────────────────
@given("an action YAML string with only required fields for security tests")
def step_given_minimal_yaml(context: Context) -> None:
"""Provide a minimal valid action YAML string."""
context.action_yaml_string = _MINIMAL_YAML
@given('an environment variable "{var_name}" set to "{var_value}"')
def step_given_env_var_set(context: Context, var_name: str, var_value: str) -> None:
"""Set an environment variable for testing."""
os.environ[var_name] = var_value
# Store for cleanup
if not hasattr(context, "env_vars_to_cleanup"):
context.env_vars_to_cleanup = []
context.env_vars_to_cleanup.append(var_name)
@given('an environment variable "{var_name}" set "{var_value}"')
def step_given_env_var_set_alt(context: Context, var_name: str, var_value: str) -> None:
"""Set an environment variable for testing (alternative syntax)."""
os.environ[var_name] = var_value
# Store for cleanup
if not hasattr(context, "env_vars_to_cleanup"):
context.env_vars_to_cleanup = []
context.env_vars_to_cleanup.append(var_name)
@given('no environment variable "{var_name}"')
def step_given_no_env_var(context: Context, var_name: str) -> None:
"""Ensure an environment variable is not set."""
if var_name in os.environ:
del os.environ[var_name]
@given('an action YAML string with description "{description}"')
def step_given_yaml_with_description(context: Context, description: str) -> None:
"""Provide YAML with a custom description value."""
# Use a YAML template with quoted description to handle special chars like ':'
yaml_template = _MINIMAL_YAML.replace(
"description: A simple action for testing", "description: PLACEHOLDER"
)
context.action_yaml_string = yaml_template.replace(
"description: PLACEHOLDER", f"description: '{description}'"
)
@given('an action YAML string with strategy_actor "{actor}"')
def step_given_yaml_with_strategy_actor(context: Context, actor: str) -> None:
"""Provide YAML with a custom strategy_actor value."""
context.action_yaml_string = _MINIMAL_YAML.replace("openai/gpt-4", actor, 1)
@given('an action YAML string with invariants containing "{invariant}"')
def step_given_yaml_with_invariants(context: Context, invariant: str) -> None:
"""Provide YAML with invariants containing a specific value."""
yaml_with_invariants = _MINIMAL_YAML + f"\ninvariants:\n - {invariant}\n"
context.action_yaml_string = yaml_with_invariants
# ────────────────────────────────────────────────────────────
# When steps
# ────────────────────────────────────────────────────────────
@when("I load the action YAML")
def step_when_load_action_yaml(context: Context) -> None:
"""Load and validate the action YAML."""
try:
context.action_config = ActionConfigSchema.from_yaml(context.action_yaml_string)
context.load_error = None
except Exception as e:
context.action_config = None
context.load_error = e
# ────────────────────────────────────────────────────────────
# Then steps
# ────────────────────────────────────────────────────────────
@then('the loaded action description should be "{expected_description}"')
def step_then_description_is(context: Context, expected_description: str) -> None:
"""Verify the action description matches the expected value."""
assert context.action_config is not None, "Action config should be loaded"
assert context.action_config.description == expected_description, (
f"Expected description '{expected_description}', got '{context.action_config.description}'"
)
@then('the action description should NOT contain "{text}"')
def step_then_description_not_contains(context: Context, text: str) -> None:
"""Verify the action description does not contain the specified text."""
assert context.action_config is not None, "Action config should be loaded"
assert text not in context.action_config.description, (
f"Description should not contain '{text}', but got '{context.action_config.description}'"
)
@then('the loaded action strategy_actor should be "{expected_actor}"')
def step_then_strategy_actor_is(context: Context, expected_actor: str) -> None:
"""Verify the action strategy_actor matches the expected value."""
assert context.action_config is not None, "Action config should be loaded"
assert context.action_config.strategy_actor == expected_actor, (
f"Expected strategy_actor '{expected_actor}', got '{context.action_config.strategy_actor}'"
)
@then('the action invariants should contain "{expected_invariant}"')
def step_then_invariants_contain(context: Context, expected_invariant: str) -> None:
"""Verify the action invariants contain the expected value."""
assert context.action_config is not None, "Action config should be loaded"
assert expected_invariant in context.action_config.invariants, (
f"Expected invariant '{expected_invariant}' not found in {context.action_config.invariants}"
)
@then('the action invariants should NOT contain "{text}"')
def step_then_invariants_not_contain(context: Context, text: str) -> None:
"""Verify the action invariants do not contain the specified text."""
assert context.action_config is not None, "Action config should be loaded"
for invariant in context.action_config.invariants:
assert text not in invariant, (
f"Invariant should not contain '{text}', but got '{invariant}'"
)
# ────────────────────────────────────────────────────────────
# Cleanup
# ────────────────────────────────────────────────────────────
def cleanup_env_vars(context: Context) -> None:
"""Clean up environment variables set during testing."""
if hasattr(context, "env_vars_to_cleanup"):
for var_name in context.env_vars_to_cleanup:
if var_name in os.environ:
del os.environ[var_name]
context.env_vars_to_cleanup = []
# Hook to cleanup after each scenario
def after_scenario(context: Context, scenario: Any) -> None: # type: ignore[no-untyped-def]
"""Clean up after each scenario."""
cleanup_env_vars(context)
+6
View File
@@ -477,4 +477,10 @@ def _interpolate_env_vars(data: dict[str, Any]) -> dict[str, Any]:
def _env_replacer(match: re.Match[str]) -> str:
"""Replace a single ``${VAR}`` match with its env value."""
var_name = match.group(1)
# Only allow CLEVERAGENTS_ prefixed variables with at least one char after prefix
# (bare "CLEVERAGENTS_" with nothing after it is not a valid variable name)
_prefix = "CLEVERAGENTS_"
if not var_name.startswith(_prefix) or len(var_name) <= len(_prefix):
return match.group(0)
# Return the env value if it exists, otherwise leave as-is
return os.environ.get(var_name, match.group(0))
@@ -38,7 +38,7 @@ if TYPE_CHECKING:
logger = logging.getLogger(__name__)
_ENV_VAR = "CLEVERAGENTS_AUTOMATION_PROFILE"
_DEFAULT_PROFILE = "manual"
_DEFAULT_PROFILE = "supervised"
class AutomationProfileService:
@@ -62,7 +62,7 @@ class AutomationProfileService:
When ``None``, only built-in profiles are available.
global_default: Global default profile name from config.
Falls back to ``CLEVERAGENTS_AUTOMATION_PROFILE``
env var, then ``"manual"``.
env var, then ``"supervised"``.
"""
self._repo = repo
self._global_default = global_default
@@ -71,7 +71,7 @@ class AutomationProfileService:
def effective_global_default(self) -> str:
"""Return the resolved global default profile name.
Precedence: explicit config > env var > ``"manual"``.
Precedence: explicit config > env var > ``"supervised"``.
"""
if self._global_default:
return self._global_default