temp/docs/development/automation-tracking.md

Automation Tracking System

Overview

The CleverAgents automation tracking system provides structured, searchable tracking for all autonomous agent activities. This system replaced the previous approach of using a single shared session state issue with individual tracking issues for each agent.

Benefits

• Reduced Noise: Each agent has its own tracking issues instead of all posting to one shared issue
• Better Searchability: Use GitHub/Forgejo search to find specific agent activities
• Cleaner History: Individual issues provide focused tracking without cross-agent interference
• Easier Debugging: Agent-specific issues make it easier to trace problems
• Automatic Cleanup: Stale tracking issues are automatically cleaned up

Issue Format Standards

Tracking Issue Title Format

All automation tracking issues MUST use this standardized title format:

[AUTO-<AGENT_PREFIX>] <TYPE> (Cycle <N>)

Components:

• AUTO-<AGENT_PREFIX>: Standardized prefix for the agent type
• <TYPE>: Type of tracking (Status, Health, Report, etc.)
• Cycle <N>: Current cycle number for the agent

Announcement Issue Title Format

For emergency announcements or important messages:

[AUTO-<AGENT_PREFIX>] Announce: <MESSAGE_SUMMARY>

Agent Prefixes

Agent	Prefix	Example Title
session-persister	AUTO-SESSION	[AUTO-SESSION] Checkpoint (Cycle 15)
implementation-orchestrator	AUTO-IMP-POOL	[AUTO-IMP-POOL] Health Report (Cycle 42)
system-watchdog	AUTO-WATCHDOG	[AUTO-WATCHDOG] System Health (Cycle 8)
backlog-groomer	AUTO-GROOMER	[AUTO-GROOMER] Grooming Report (Cycle 23)
human-liaison	AUTO-LIAISON	[AUTO-LIAISON] Status Update (Cycle 67)

Required Labels

All automation tracking issues MUST include:

• Automation Tracking - Primary label for filtering
• Additional agent-specific labels as appropriate

Interval Reporting and Health Monitoring

Standardized Interval Declaration

All periodic tracking issues MUST include the expected reporting interval in their description using this standardized format:

**Reporting Interval**: <interval> (Next report expected: <timestamp>)

This enables automated health monitoring by the system-watchdog to detect stalled or crashed agents.

Agent Reporting Intervals

Agent	Issue Type	Interval	Example
implementation-orchestrator	Status Update	Every 5 cycles (~variable timing)	Every worker dispatch cycle
implementation-orchestrator	Health Report	Every 10 cycles (~variable timing)	Comprehensive pool health
backlog-groomer	Grooming Report	Every 5 minutes	Backlog maintenance cycle
backlog-groomer	Health Report	Every 50 minutes (10 cycles)	Grooming health status
human-liaison	Status Update	Every 20 minutes (10 cycles)	Human activity monitoring
system-watchdog	Health Report	Every 30 minutes (6 cycles)	System-wide health check
session-persister	Checkpoint	Event-driven (variable)	After significant state changes

Automated Health Monitoring

The system-watchdog automatically monitors all automation tracking issues:

1. Scans tracking issues with "Automation Tracking" label every 5 minutes
2. Calculates staleness by comparing current time to issue creation time
3. Detects stalled agents when tracking issue is >20% overdue from expected interval
4. Triggers automated recovery actions:
   • Kills stalled agent sessions via OpenCode Server API (port 4096)
   • Performs root cause analysis by examining session messages and agent definitions
   • Creates high-priority diagnostic issues with findings and remediation suggestions
   • Closes stale tracking issues with recovery notes
   • Posts alerts for manual intervention where needed

Automated Recovery Process

When a stalled agent is detected:

1. Session Termination: All matching agent sessions are gracefully terminated
2. Root Cause Analysis: Automated analysis examines:
   • Recent session messages and tool call history
   • Related Forgejo issues and PR status
   • Agent definition and configuration
   • System resource availability
3. Diagnostic Issue Creation: Creates issue with format [AUTO-RECOVERY] <Agent> Agent Failure Analysis
4. Tracking Issue Cleanup: Closes stale tracking issue with recovery notes
5. Human Notification: Diagnostic issue requires manual review and restart

Recovery Issue Format

# Agent Failure Analysis — <Agent Name>

**Agent Prefix**: <prefix>
**Detection Time**: <ISO timestamp>
**Stale Issue**: #<issue_number>  
**Time Overdue**: <minutes> minutes
**Staleness Ratio**: <ratio>x expected interval

## Root Cause Analysis
[Automated analysis of session messages, Forgejo issues, and agent definition]

## Recovery Actions Taken
1. ✅ Session Termination: Killed <count> stalled sessions
2. ✅ Tracking Cleanup: Closed stale tracking issue  
3. ✅ Root Cause Analysis: Completed automated failure analysis
4. 🔄 Manual Intervention: Required per recommendations

## Recommended Next Steps
[Specific recommendations based on failure analysis]

Health Check Algorithm

# Pseudocode for health monitoring
for issue in automation_tracking_issues:
    if issue.title.matches("[AUTO-*] * (Cycle *)"):
        expected_interval = get_expected_interval(issue.agent_prefix, issue.type)
        time_since_creation = now() - issue.created_at
        staleness_threshold = expected_interval * 1.2  # 20% tolerance
        
        if time_since_creation > staleness_threshold:
            mark_agent_as_stalled(issue.agent_prefix)
            trigger_recovery_actions(issue.agent_prefix, issue)

Standardized Tracking Issue Templates

Common Header Format

All tracking issues MUST start with this standardized header:

# <Agent Name> <Issue Type> — <Timestamp>

**Agent**: <agent-name>
**Cycle**: <cycle-number>
**Reporting Interval**: <interval> (Next report expected: <next-timestamp>)
**Status**: <active|warning|error>

## Summary

<Brief status summary>

## Details

<Detailed information specific to agent type>

## Health Indicators

<Health metrics and status indicators>

## Next Actions

<Planned actions for next cycle>

---
**Automated by CleverAgents Bot**  
Supervisor: <supervisor-name> | Agent: <agent-name>

Template Variables

• <agent-name>: The agent type (e.g., "implementation-orchestrator", "backlog-groomer")
• <cycle-number>: Current cycle number for this agent
• <interval>: Expected time between reports (e.g., "5 minutes", "10 cycles")
• <next-timestamp>: When the next report is expected (ISO 8601 format)
• <status>: Current agent health status
  • active: Normal operation
  • warning: Non-critical issues detected
  • error: Critical issues requiring attention

Implementation Requirements

For All Agents

Every agent that creates tracking issues must implement:

1. Cleanup Function: Delete previous cycle tracking issues before creating new ones
2. Standardized Titles: Use the exact format specified above
3. Proper Labels: Always include "Automation Tracking" label
4. Structured Content: Use consistent formatting for issue bodies

Standard Tracking Functions

Each agent should implement these functions:

## Automation Tracking Functions

### Delete Previous Tracking Issues
1. Search for issues with title pattern `[AUTO-<PREFIX>] <TYPE> (Cycle *)`
2. Filter for issues created by this agent
3. Close and delete previous cycle issues
4. Preserve announcement issues (different title pattern)

### Create New Tracking Issue
1. Generate title with current cycle number
2. Include "Automation Tracking" label
3. Add structured content with timestamp and status
4. Link to relevant repositories/PRs as needed

Agent-Specific Implementation Details

session-persister

• Cycle Frequency: Every checkpoint (variable timing)
• Issue Type: Checkpoint
• Content: Session state, active agents, configuration
• Cleanup: Deletes previous checkpoint issue

implementation-orchestrator

• Cycle Frequency: Health reports every 10 cycles, status every 5 cycles
• Issue Types: Health Report, Status Update
• Content: Worker status, queue health, completion metrics
• Cleanup: Deletes previous health/status issues separately

system-watchdog

• Cycle Frequency: Every monitoring cycle (variable timing)
• Issue Types: System Health, Alert
• Content: Quality gates, system violations, corrective actions
• Cleanup: Deletes previous system health issues

backlog-groomer

• Cycle Frequency: Every grooming cycle (~30 minutes)
• Issue Types: Grooming Report, Scope Alert
• Content: Issues processed, duplicates found, orphans discovered
• Cleanup: Deletes previous grooming reports, acts as backup cleanup agent

human-liaison

• Cycle Frequency: Status every 10 cycles
• Issue Types: Status Update, Human Activity Summary
• Content: Human interactions, triage decisions, outstanding items
• Cleanup: Deletes previous status updates

Searching and Filtering

Finding Tracking Issues

All automation tracking:

label:"Automation Tracking"

Specific agent tracking:

label:"Automation Tracking" [AUTO-SESSION] in:title
label:"Automation Tracking" [AUTO-IMP-POOL] in:title
label:"Automation Tracking" [AUTO-WATCHDOG] in:title
label:"Automation Tracking" [AUTO-GROOMER] in:title
label:"Automation Tracking" [AUTO-LIAISON] in:title

Recent tracking issues (last 24 hours):

label:"Automation Tracking" created:>2024-01-01T00:00:00Z

Announcement issues only:

label:"Automation Tracking" "Announce:" in:title

Cleanup Protocol

Primary Cleanup (Each Agent)

• Each agent deletes its own previous cycle tracking issues before creating new ones
• Preserves announcement issues which use different title format
• Runs cleanup before every new tracking issue creation

Secondary Cleanup (backlog-groomer)

The backlog-groomer acts as a backup cleanup agent:

• Scans for stale automation tracking issues older than 7 days
• Deletes abandoned tracking issues from agents that may have crashed
• Preserves announcement issues and recently created tracking issues
• Reports cleanup activity in its own tracking issues

Emergency Cleanup (Manual)

If tracking issues accumulate excessively:

1. Use GitHub/Forgejo bulk operations with label filter
2. Search: label:"Automation Tracking" created:<7-days-ago
3. Bulk close/delete stale tracking issues
4. Preserve announcement issues

Troubleshooting

Common Issues

"Automation Tracking" label missing:

• Ensure label exists in repository
• Check agent permission to add labels
• Verify label name spelling (case sensitive)

Tracking issues not being cleaned up:

• Check if agent cleanup function is running
• Verify backlog-groomer is active and functioning
• Check for agent crashes or hangs preventing cleanup

Too many tracking issues:

• May indicate cleanup functions not working
• Check agent logs for cleanup errors
• Run manual cleanup if necessary

Cannot find specific agent tracking:

• Verify agent prefix in search
• Check if agent is actually running and creating issues
• Confirm agent is using correct title format

Debugging Steps

1. Check agent status: Verify agent is running and healthy
2. Verify title format: Ensure exact compliance with format standards
3. Check labels: Confirm "Automation Tracking" label is applied
4. Review cleanup logs: Look for cleanup function errors
5. Manual search: Use repository issue search to verify tracking issues

Migration Notes

This system replaced the previous shared session state issue approach. Key differences:

• Before: All agents posted comments to single issue #[session-state-issue-number]
• After: Each agent creates individual tracking issues with standardized titles
• Benefit: Eliminates noise, improves searchability, enables targeted cleanup

The migration was completed by updating agent definitions in .opencode/agents/ directory to:

• Remove session state issue posting
• Add tracking issue creation functions
• Implement cleanup protocols
• Apply standardized formatting

Related Documentation

• System Watchdog - Specific documentation for watchdog agent
• Quality Automation - Quality gate automation details
• Ops Runbook - General operational procedures