cleveragents-core/docs/development/automation-tracking.md

Automation Tracking System

Overview

The CleverAgents automation tracking system provides structured, searchable tracking for all autonomous agent activities. This system replaced the previous approach of using a single shared session state issue with individual tracking issues for each agent.

Benefits

• Reduced Noise: Each agent has its own tracking issues instead of all posting to one shared issue
• Better Searchability: Use GitHub/Forgejo search to find specific agent activities
• Cleaner History: Individual issues provide focused tracking without cross-agent interference
• Easier Debugging: Agent-specific issues make it easier to trace problems
• Automatic Cleanup: Stale tracking issues are automatically cleaned up

Issue Format Standards

Tracking Issue Title Format

All automation tracking issues MUST use this standardized title format:

[AUTO-<AGENT_PREFIX>] <TYPE> (Cycle <N>)

Components:

• AUTO-<AGENT_PREFIX>: Standardized prefix for the agent type
• <TYPE>: Type of tracking (Status, Health, Report, etc.)
• Cycle <N>: Current cycle number for the agent

Announcement Issue Title Format

For emergency announcements or important messages:

[AUTO-<AGENT_PREFIX>] Announce: <MESSAGE_SUMMARY>

Agent Prefixes

Agent	Prefix	Example Title
session-persister	AUTO-SESSION	[AUTO-SESSION] Checkpoint (Cycle 15)
implementation-orchestrator	AUTO-IMP-POOL	[AUTO-IMP-POOL] Health Report (Cycle 42)
system-watchdog	AUTO-WATCHDOG	[AUTO-WATCHDOG] System Health (Cycle 8)
backlog-groomer	AUTO-GROOMER	[AUTO-GROOMER] Grooming Report (Cycle 23)
human-liaison	AUTO-LIAISON	[AUTO-LIAISON] Status Update (Cycle 67)
product-builder	AUTO-PROD-BLDR	[AUTO-PROD-BLDR] Build Session (Cycle 5)
architect	AUTO-ARCH	[AUTO-ARCH] Architecture Report (Cycle 3)
timeline-updater	AUTO-TIME	[AUTO-TIME] Timeline Update (Cycle 12)
docs-writer	AUTO-DOCS	[AUTO-DOCS] Documentation Report (Cycle 7)
architecture-guard	AUTO-GUARD	[AUTO-GUARD] Guard Report (Cycle 9)
continuous-pr-reviewer	AUTO-REV-POOL	[AUTO-REV-POOL] Review Status (Cycle 4)
uat-tester	AUTO-UAT-POOL	[AUTO-UAT-POOL] UAT Status (Cycle 6)
project-owner	AUTO-PROJ-OWN	[AUTO-PROJ-OWN] Project Status (Cycle 11)
agent-evolver	AUTO-EVLV	[AUTO-EVLV] Agent Evolution Report (Cycle 10)
bug-hunter	AUTO-BUG-POOL	[AUTO-BUG-POOL] Bug Detection Pool Status (Cycle 60)
spec-updater	AUTO-SPEC	[AUTO-SPEC] Specification Update Report (Cycle 1)
test-infra-improver	AUTO-INF-POOL	[AUTO-INF-POOL] Infrastructure Analysis Report (Cycle 2)

Required Labels

All automation tracking issues MUST include:

• Automation Tracking - Primary label for filtering
• Type/Automation - Type label for automation issues
• State/In Progress - State label while agent is active
• Priority/Medium - Default priority (adjust as needed)

Centralized Tracking Manager

The automation-tracking-manager subagent is the single source of truth for all automation tracking operations. It was introduced to prevent cycle reuse issues where agents would incorrectly comment on old tracking issues instead of creating new ones.

Why Centralized?

Without centralization, agents experienced:

• Cycle reuse: Agents posting future cycle reports as comments on old issues
• Duplicate issues: Multiple tracking issues created for the same cycle
• Missing cycles: Agents skipping cycle numbers due to stale issue references
• Inconsistent cleanup: Previous cycle issues not always deleted before creating new ones

How It Works

The automation-tracking-manager subagent:

1. Maintains a persistent cycle counter per agent prefix
2. Searches for and closes the previous cycle's tracking issue
3. Creates a new tracking issue with the correct sequential cycle number
4. Applies all required labels atomically
5. Returns the new issue number to the calling agent

Usage

Agents that use the centralized manager delegate all tracking operations to it:

# Example: implementation-orchestrator delegating to tracking manager
# (Inside agent definition — pseudocode)
TRACKING_RESULT=$(call_subagent automation-tracking-manager \
  --prefix "AUTO-IMP-POOL" \
  --type "Health Report" \
  --body "...")
NEW_ISSUE_NUMBER=$(echo "$TRACKING_RESULT" | jq -r '.issue_number')

Migrated Agents

The following agents have been migrated to use the centralized tracking manager:

• system-watchdog (most critical — was skipping cycles)
• implementation-orchestrator (was creating duplicate issues)
• timeline-updater
• project-owner
• product-builder
• backlog-groomer

Interval Reporting and Health Monitoring

Standardized Interval Declaration

All periodic tracking issues MUST include the expected reporting interval in their description using this standardized format:

**Reporting Interval**: <interval> (Next report expected: <timestamp>)

This enables automated health monitoring by the system-watchdog to detect stalled or crashed agents.

Agent Reporting Intervals

Agent	Issue Type	Interval	Example
implementation-orchestrator	Status Update	Every 5 cycles (~variable timing)	Every worker dispatch cycle
implementation-orchestrator	Health Report	Every 10 cycles (~variable timing)	Comprehensive pool health
backlog-groomer	Grooming Report	Every 5 minutes	Backlog maintenance cycle
backlog-groomer	Health Report	Every 50 minutes (10 cycles)	Grooming health status
human-liaison	Status Update	Every 20 minutes (10 cycles)	Human activity monitoring
system-watchdog	Health Report	Every 30 minutes (6 cycles)	System-wide health check
session-persister	Checkpoint	Event-driven (variable)	After significant state changes
product-builder	Build Session	Every 10 cycles (~variable timing)	Full build session status
architect	Architecture Report	Every 10 cycles (~variable timing)	Architecture supervision status
timeline-updater	Timeline Update	Every 10 cycles (~variable timing)	Timeline maintenance status
docs-writer	Documentation Report	Every 10 cycles (~3.3 hours)	Documentation update status
architecture-guard	Guard Report	Every 10 cycles (~variable timing)	Codebase coherence check
continuous-pr-reviewer	Review Status	Every 10 cycles (~variable timing)	PR review pool status
uat-tester	UAT Status	Every 10 cycles (~variable timing)	UAT testing progress
project-owner	Project Status	Every 10 cycles (~variable timing)	Project ownership decisions
agent-evolver	Evolution Report	Every 10 cycles (~variable timing)	Agent improvement proposals
bug-hunter	Bug Detection Status	Every 10 cycles (~variable timing)	Bug detection pool health
spec-updater	Specification Update	Every 10 cycles (~variable timing)	Spec evolution status
test-infra-improver	Infrastructure Report	Every 10 cycles (~variable timing)	Test infrastructure analysis

Automated Health Monitoring

The system-watchdog automatically monitors all automation tracking issues:

1. Scans tracking issues with "Automation Tracking" label every 5 minutes
2. Calculates staleness by comparing current time to issue creation time
3. Detects stalled agents when tracking issue is >20% overdue from expected interval
4. Triggers automated recovery actions:
   • Kills stalled agent sessions via OpenCode Server API (port 4096)
   • Performs root cause analysis by examining session messages and agent definitions
   • Creates high-priority diagnostic issues with findings and remediation suggestions
   • Closes stale tracking issues with recovery notes
   • Posts alerts for manual intervention where needed

Automated Recovery Process

When a stalled agent is detected:

1. Session Termination: All matching agent sessions are gracefully terminated
2. Root Cause Analysis: Automated analysis examines:
   • Recent session messages and tool call history
   • Related Forgejo issues and PR status
   • Agent definition and configuration
   • System resource availability
3. Diagnostic Issue Creation: Creates issue with format [AUTO-RECOVERY] <Agent> Agent Failure Analysis
4. Tracking Issue Cleanup: Closes stale tracking issue with recovery notes
5. Human Notification: Diagnostic issue requires manual review and restart

Recovery Issue Format

# Agent Failure Analysis — <Agent Name>

**Agent Prefix**: <prefix>
**Detection Time**: <ISO timestamp>
**Stale Issue**: #<issue_number>  
**Time Overdue**: <minutes> minutes
**Staleness Ratio**: <ratio>x expected interval

## Root Cause Analysis
[Automated analysis of session messages, Forgejo issues, and agent definition]

## Recovery Actions Taken
1. ✅ Session Termination: Killed <count> stalled sessions
2. ✅ Tracking Cleanup: Closed stale tracking issue  
3. ✅ Root Cause Analysis: Completed automated failure analysis
4. 🔄 Manual Intervention: Required per recommendations

## Recommended Next Steps
[Specific recommendations based on failure analysis]

Health Check Algorithm

# Pseudocode for health monitoring
for issue in automation_tracking_issues:
    if issue.title.matches("[AUTO-*] * (Cycle *)"):
        expected_interval = get_expected_interval(issue.agent_prefix, issue.type)
        time_since_creation = now() - issue.created_at
        staleness_threshold = expected_interval * 1.2  # 20% tolerance
        
        if time_since_creation > staleness_threshold:
            mark_agent_as_stalled(issue.agent_prefix)
            trigger_recovery_actions(issue.agent_prefix, issue)

Standardized Tracking Issue Templates

Common Header Format

All tracking issues MUST start with this standardized header:

# <Agent Name> <Issue Type> — <Timestamp>

**Agent**: <agent-name>
**Cycle**: <cycle-number>
**Reporting Interval**: <interval> (Next report expected: <next-timestamp>)
**Status**: <active|warning|error>

## Summary

<Brief status summary>

## Details

<Detailed information specific to agent type>

## Health Indicators

<Health metrics and status indicators>

## Next Actions

<Planned actions for next cycle>

---
**Automated by CleverAgents Bot**  
Supervisor: <supervisor-name> | Agent: <agent-name>

Template Variables

• <agent-name>: The agent type (e.g., "implementation-orchestrator", "backlog-groomer")
• <cycle-number>: Current cycle number for this agent
• <interval>: Expected time between reports (e.g., "5 minutes", "10 cycles")
• <next-timestamp>: When the next report is expected (ISO 8601 format)
• <status>: Current agent health status
  • active: Normal operation
  • warning: Non-critical issues detected
  • error: Critical issues requiring attention

Implementation Requirements

For All Agents

Every agent that creates tracking issues must implement:

1. Cleanup Function: Delete previous cycle tracking issues before creating new ones
2. Standardized Titles: Use the exact format specified above
3. Proper Labels: Always include "Automation Tracking", "Type/Automation", "State/In Progress", and "Priority/Medium" labels
4. Structured Content: Use consistent formatting for issue bodies

Standard Tracking Functions

Each agent should implement these functions:

## Automation Tracking Functions

### Delete Previous Tracking Issues
1. Search for issues with title pattern `[AUTO-<PREFIX>] <TYPE> (Cycle *)`
2. Filter for issues created by this agent
3. Close and delete previous cycle issues
4. Preserve announcement issues (different title pattern)

### Create New Tracking Issue
1. Generate title with current cycle number
2. Include "Automation Tracking", "Type/Automation", "State/In Progress", "Priority/Medium" labels
3. Add structured content with timestamp and status
4. Link to relevant repositories/PRs as needed

Agent-Specific Implementation Details

session-persister

• Cycle Frequency: Every checkpoint (variable timing)
• Issue Type: Checkpoint
• Content: Session state, active agents, configuration
• Cleanup: Deletes previous checkpoint issue

implementation-orchestrator

• Cycle Frequency: Health reports every 10 cycles, status every 5 cycles
• Issue Types: Health Report, Status Update
• Content: Worker status, queue health, completion metrics
• Cleanup: Deletes previous health/status issues separately

system-watchdog

• Cycle Frequency: Every monitoring cycle (variable timing)
• Issue Types: System Health, Alert
• Content: Quality gates, system violations, corrective actions
• Cleanup: Deletes previous system health issues

backlog-groomer

• Cycle Frequency: Every grooming cycle (~30 minutes)
• Issue Types: Grooming Report, Scope Alert
• Content: Issues processed, duplicates found, orphans discovered
• Cleanup: Deletes previous grooming reports, acts as backup cleanup agent

human-liaison

• Cycle Frequency: Status every 10 cycles
• Issue Types: Status Update, Human Activity Summary
• Content: Human interactions, triage decisions, outstanding items
• Cleanup: Deletes previous status updates

Searching and Filtering

Finding Tracking Issues

All automation tracking:

label:"Automation Tracking"

Specific agent tracking:

label:"Automation Tracking" [AUTO-SESSION] in:title
label:"Automation Tracking" [AUTO-IMP-POOL] in:title
label:"Automation Tracking" [AUTO-WATCHDOG] in:title
label:"Automation Tracking" [AUTO-GROOMER] in:title
label:"Automation Tracking" [AUTO-LIAISON] in:title
label:"Automation Tracking" [AUTO-PROD-BLDR] in:title
label:"Automation Tracking" [AUTO-ARCH] in:title
label:"Automation Tracking" [AUTO-TIME] in:title
label:"Automation Tracking" [AUTO-DOCS] in:title
label:"Automation Tracking" [AUTO-GUARD] in:title
label:"Automation Tracking" [AUTO-REV-POOL] in:title
label:"Automation Tracking" [AUTO-UAT-POOL] in:title
label:"Automation Tracking" [AUTO-PROJ-OWN] in:title
label:"Automation Tracking" [AUTO-EVLV] in:title
label:"Automation Tracking" [AUTO-BUG-POOL] in:title
label:"Automation Tracking" [AUTO-SPEC] in:title
label:"Automation Tracking" [AUTO-INF-POOL] in:title

Recent tracking issues (last 24 hours):

label:"Automation Tracking" created:>2024-01-01T00:00:00Z

Announcement issues only:

label:"Automation Tracking" "Announce:" in:title

Cleanup Protocol

Primary Cleanup (Each Agent)

• Each agent deletes its own previous cycle tracking issues before creating new ones
• Preserves announcement issues which use different title format
• Runs cleanup before every new tracking issue creation

Secondary Cleanup (backlog-groomer)

The backlog-groomer acts as a backup cleanup agent:

• Scans for stale automation tracking issues older than 7 days
• Deletes abandoned tracking issues from agents that may have crashed
• Preserves announcement issues and recently created tracking issues
• Reports cleanup activity in its own tracking issues

Emergency Cleanup (Manual)

If tracking issues accumulate excessively:

1. Use GitHub/Forgejo bulk operations with label filter
2. Search: label:"Automation Tracking" created:<7-days-ago
3. Bulk close/delete stale tracking issues
4. Preserve announcement issues

Troubleshooting

Common Issues

"Automation Tracking" label missing:

• Ensure label exists in repository
• Check agent permission to add labels
• Verify label name spelling (case sensitive)

Tracking issues not being cleaned up:

• Check if agent cleanup function is running
• Verify backlog-groomer is active and functioning
• Check for agent crashes or hangs preventing cleanup

Too many tracking issues:

• May indicate cleanup functions not working
• Check agent logs for cleanup errors
• Run manual cleanup if necessary

Cannot find specific agent tracking:

• Verify agent prefix in search
• Check if agent is actually running and creating issues
• Confirm agent is using correct title format

Agents posting to old tracking issues (cycle reuse):

• Agent is not using the centralized automation-tracking-manager
• Migrate agent to use the centralized manager subagent
• Check that previous cycle cleanup runs before new issue creation

Debugging Steps

1. Check agent status: Verify agent is running and healthy
2. Verify title format: Ensure exact compliance with format standards
3. Check labels: Confirm "Automation Tracking" label is applied
4. Review cleanup logs: Look for cleanup function errors
5. Manual search: Use repository issue search to verify tracking issues

Migration Notes

This system replaced the previous shared session state issue approach. Key differences:

• Before: All agents posted comments to single issue #[session-state-issue-number]
• After: Each agent creates individual tracking issues with standardized titles
• Benefit: Eliminates noise, improves searchability, enables targeted cleanup

The migration was completed by updating agent definitions in .opencode/agents/ directory to:

• Remove session state issue posting
• Add tracking issue creation functions
• Implement cleanup protocols
• Apply standardized formatting

A second migration introduced the centralized automation-tracking-manager subagent to prevent cycle reuse issues where agents were commenting on old tracking issues instead of creating new ones.

Related Documentation

• System Watchdog - Specific documentation for watchdog agent
• Quality Automation - Quality gate automation details
• Ops Runbook - General operational procedures