cleveragents-core/docs/v3.6.0-features.md

# CleverAgents v3.6.0 Features and Enhancements

## Overview

CleverAgents v3.6.0 introduces significant improvements to the core automation framework, focusing on advanced context management, enhanced security profiles, and improved observability. This release builds upon the v3.5.0 foundation with critical enhancements for enterprise deployments.

## Table of Contents

1. [Advanced Context Management](#advanced-context-management)
2. [Enhanced Security Profiles](#enhanced-security-profiles)
3. [Improved Observability](#improved-observability)
4. [Performance Optimizations](#performance-optimizations)
5. [API Enhancements](#api-enhancements)
6. [Migration Guide](#migration-guide)

## Advanced Context Management

### Context Tier Hydration

v3.6.0 introduces a sophisticated context tier hydration system that enables dynamic context loading based on automation requirements.

#### Key Features

- **Lazy Loading**: Context tiers are loaded on-demand, reducing memory footprint
- **Hierarchical Context**: Support for multi-level context inheritance and composition
- **Context Caching**: Intelligent caching of frequently accessed context data
- **Tier Prioritization**: Configurable priority chains for context resolution

#### Example: Configuring Context Tiers

```yaml
# automation-profile.yaml
automation_profile:
  name: advanced-context-profile
  context_tiers:
    - tier: system
      priority: 1
      cache_ttl: 3600
      lazy_load: true
    - tier: project
      priority: 2
      cache_ttl: 1800
      lazy_load: true
    - tier: session
      priority: 3
      cache_ttl: 300
      lazy_load: false
```

#### Using Context Tiers in Code

```python
from cleveragents.context import ContextManager

# Initialize context manager with tier configuration
context_mgr = ContextManager(
    tiers=['system', 'project', 'session'],
    cache_enabled=True
)

# Load context for a specific tier
system_context = await context_mgr.load_tier('system')

# Resolve context with priority chain
resolved_context = await context_mgr.resolve_context(
    required_keys=['api_key', 'project_id'],
    tier_priority=['session', 'project', 'system']
)
```

### Dynamic Context Analysis

The new context analysis system provides real-time insights into context usage patterns and optimization opportunities.

#### Features

- **Usage Metrics**: Track context access patterns and frequency
- **Performance Analysis**: Identify bottlenecks in context resolution
- **Optimization Recommendations**: Automatic suggestions for tier reorganization
- **Context Validation**: Continuous validation of context integrity

#### Example: Analyzing Context Performance

```python
from cleveragents.context import ContextAnalyzer

analyzer = ContextAnalyzer()

# Analyze context usage over a time window
analysis = await analyzer.analyze_usage(
    time_window='1h',
    include_metrics=['access_count', 'resolution_time', 'cache_hit_rate']
)

print(f"Cache Hit Rate: {analysis.cache_hit_rate:.2%}")
print(f"Avg Resolution Time: {analysis.avg_resolution_time}ms")

# Get optimization recommendations
recommendations = await analyzer.get_recommendations()
for rec in recommendations:
    print(f"Recommendation: {rec.title} - Impact: {rec.estimated_impact}")
```

## Enhanced Security Profiles

### Layered Boundary Enforcement

v3.6.0 introduces a comprehensive layered security architecture with multiple enforcement boundaries.

#### Security Layers

1. **Input Validation Layer**: Validates all incoming data against defined schemas
2. **Execution Boundary Layer**: Enforces resource access restrictions
3. **Output Sanitization Layer**: Ensures sensitive data is redacted
4. **Audit Layer**: Comprehensive logging of all security-relevant events

#### Configuring Security Profiles

```yaml
# security-profile.yaml
security_profile:
  name: enterprise-hardened
  layers:
    input_validation:
      enabled: true
      strict_mode: true
      max_payload_size: 10485760  # 10MB
      allowed_content_types:
        - application/json
        - application/yaml
    
    execution_boundary:
      enabled: true
      resource_limits:
        cpu_percent: 80
        memory_mb: 2048
        timeout_seconds: 300
      allowed_operations:
        - read
        - write
        - execute
    
    output_sanitization:
      enabled: true
      redaction_patterns:
        - pattern: '\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'
          replacement: '[EMAIL_REDACTED]'
        - pattern: 'sk-[A-Za-z0-9]{48}'
          replacement: '[API_KEY_REDACTED]'
    
    audit:
      enabled: true
      log_level: INFO
      retention_days: 90
```

### Trusted Profile Composition

Create complex security profiles by composing trusted base profiles.

```python
from cleveragents.security import SecurityProfileBuilder

# Create a custom profile by composing trusted profiles
builder = SecurityProfileBuilder()

profile = (builder
    .inherit_from('enterprise-hardened')
    .add_resource_restriction('database', ['read', 'write'])
    .add_resource_restriction('filesystem', ['read'])
    .set_timeout(600)
    .enable_audit_logging()
    .build()
)

# Apply the profile to an automation
automation.apply_security_profile(profile)
```

## Improved Observability

### Comprehensive Metrics Collection

v3.6.0 provides detailed metrics for monitoring automation performance and health.

#### Available Metrics

- **Execution Metrics**: Duration, success rate, error rate
- **Resource Metrics**: CPU usage, memory consumption, I/O operations
- **Context Metrics**: Context resolution time, cache hit rates
- **Security Metrics**: Policy violations, audit events

#### Example: Collecting and Analyzing Metrics

```python
from cleveragents.observability import MetricsCollector

collector = MetricsCollector()

# Collect metrics during automation execution
with collector.track_execution('my-automation'):
    # Your automation code here
    await automation.execute()

# Retrieve metrics
metrics = collector.get_metrics('my-automation')

print(f"Execution Time: {metrics.duration_ms}ms")
print(f"Success Rate: {metrics.success_rate:.2%}")
print(f"Memory Peak: {metrics.memory_peak_mb}MB")
print(f"Cache Hit Rate: {metrics.cache_hit_rate:.2%}")
```

### Distributed Tracing

Full support for distributed tracing across multiple automation instances.

```python
from cleveragents.observability import TracingContext

# Initialize tracing context
tracing = TracingContext(
    service_name='cleveragents-automation',
    trace_sample_rate=0.1  # Sample 10% of traces
)

# Trace automation execution
with tracing.span('automation-execution') as span:
    span.set_attribute('automation.name', 'my-automation')
    span.set_attribute('automation.version', '1.0.0')
    
    result = await automation.execute()
    
    span.set_attribute('automation.status', result.status)
    span.set_attribute('automation.duration_ms', result.duration_ms)
```

### Enhanced Logging

Structured logging with contextual information and correlation IDs.

```python
from cleveragents.observability import StructuredLogger

logger = StructuredLogger('my-automation')

# Log with context
logger.info(
    'Automation started',
    extra={
        'automation_id': 'auto-123',
        'user_id': 'user-456',
        'project_id': 'proj-789',
        'correlation_id': 'corr-abc123'
    }
)

# Logs are automatically formatted with context
# Output: {"timestamp": "2024-01-15T10:30:45Z", "level": "INFO", "message": "Automation started", "automation_id": "auto-123", ...}
```

## Performance Optimizations

### Parallel Execution Improvements

Enhanced parallel execution with better resource management.

```python
from cleveragents.execution import ParallelExecutor

executor = ParallelExecutor(
    max_workers=8,
    queue_size=100,
    timeout_seconds=300
)

# Execute multiple automations in parallel
results = await executor.execute_batch([
    automation_1,
    automation_2,
    automation_3
])

# Monitor execution progress
for result in executor.stream_results():
    print(f"Completed: {result.automation_id} - Status: {result.status}")
```

### Caching Enhancements

Improved caching strategy with multiple cache backends.

```python
from cleveragents.caching import CacheManager, RedisBackend

# Configure multi-tier caching
cache_mgr = CacheManager(
    backends=[
        ('memory', {'max_size': 1000}),
        ('redis', {'host': 'localhost', 'port': 6379})
    ],
    ttl_seconds=3600
)

# Use cache with fallback
value = await cache_mgr.get_or_compute(
    key='expensive-computation',
    compute_fn=expensive_function,
    ttl_seconds=1800
)
```

## API Enhancements

### New REST Endpoints

v3.6.0 introduces new REST endpoints for better automation management.

#### Context Management Endpoints

```
GET    /api/v1/contexts              - List all contexts
POST   /api/v1/contexts              - Create new context
GET    /api/v1/contexts/{id}         - Get context details
PUT    /api/v1/contexts/{id}         - Update context
DELETE /api/v1/contexts/{id}         - Delete context
GET    /api/v1/contexts/{id}/tiers   - List context tiers
```

#### Security Profile Endpoints

```
GET    /api/v1/security-profiles              - List profiles
POST   /api/v1/security-profiles              - Create profile
GET    /api/v1/security-profiles/{id}         - Get profile
PUT    /api/v1/security-profiles/{id}         - Update profile
DELETE /api/v1/security-profiles/{id}         - Delete profile
POST   /api/v1/security-profiles/{id}/validate - Validate profile
```

#### Metrics Endpoints

```
GET    /api/v1/metrics                        - Get system metrics
GET    /api/v1/automations/{id}/metrics       - Get automation metrics
GET    /api/v1/metrics/export                 - Export metrics (Prometheus format)
```

## Migration Guide

### From v3.5.0 to v3.6.0

#### Breaking Changes

1. **Context API Changes**: The old `get_context()` method is deprecated. Use `ContextManager.resolve_context()` instead.
2. **Security Profile Format**: YAML format has changed. See examples above for new structure.

#### Migration Steps

1. **Update Context Usage**

```python
# Old (v3.5.0)
context = automation.get_context('api_key')

# New (v3.6.0)
context_mgr = ContextManager()
context = await context_mgr.resolve_context(
    required_keys=['api_key'],
    tier_priority=['session', 'project', 'system']
)
```

2. **Update Security Profiles**

```yaml
# Old format (v3.5.0)
security:
  strict_mode: true
  timeout: 300

# New format (v3.6.0)
security_profile:
  name: my-profile
  layers:
    input_validation:
      enabled: true
      strict_mode: true
    execution_boundary:
      enabled: true
      resource_limits:
        timeout_seconds: 300
```

3. **Update Metrics Collection**

```python
# Old (v3.5.0)
metrics = automation.get_metrics()

# New (v3.6.0)
collector = MetricsCollector()
with collector.track_execution('automation-id'):
    await automation.execute()
metrics = collector.get_metrics('automation-id')
```

### Deprecation Timeline

- **v3.6.0**: Old APIs available with deprecation warnings
- **v3.7.0**: Old APIs still available but warnings increased
- **v3.8.0**: Old APIs removed

## Best Practices

### Context Management

1. **Use Tier Prioritization**: Define clear tier priorities for context resolution
2. **Enable Caching**: Use context caching for frequently accessed data
3. **Monitor Context Performance**: Regularly analyze context usage patterns
4. **Validate Context**: Always validate context data before use

### Security

1. **Use Layered Profiles**: Compose security profiles from trusted base profiles
2. **Enable Audit Logging**: Always enable audit logging in production
3. **Regular Security Reviews**: Periodically review and update security profiles
4. **Principle of Least Privilege**: Grant only necessary permissions

### Observability

1. **Enable Distributed Tracing**: Use tracing for multi-service automations
2. **Collect Metrics**: Monitor key metrics for performance optimization
3. **Structured Logging**: Use structured logging for better analysis
4. **Set Up Alerts**: Configure alerts for critical metrics

## Troubleshooting

### Context Resolution Issues

**Problem**: Context resolution is slow
**Solution**: 
- Enable context caching
- Optimize tier prioritization
- Use lazy loading for large contexts

### Security Profile Validation Failures

**Problem**: Security profile validation fails
**Solution**:
- Check profile syntax against schema
- Validate resource restrictions
- Review audit logs for details

### Metrics Collection Overhead

**Problem**: Metrics collection is impacting performance
**Solution**:
- Reduce trace sample rate
- Use sampling for high-volume metrics
- Consider async metric collection

## Additional Resources

- [Context Hydration Module Guide](../modules/context-hydration.md)
- [Security Profiles Reference](../reference/security-profiles.md)
- [Observability Best Practices](../reference/observability.md)
- [API Reference](../api/v1.md)

## Support and Feedback

For issues, questions, or feedback regarding v3.6.0 features:

- **GitHub Issues**: https://git.cleverthis.com/cleveragents/cleveragents-core/issues
- **Documentation**: https://docs.cleverthis.com
- **Community Chat**: https://chat.cleverthis.com

---

**Last Updated**: April 2024
**Version**: 3.6.0