temp/docs/development/automation-tracking.md

# Automation Tracking System

## Overview

The CleverAgents automation tracking system provides structured, searchable tracking for all autonomous agent activities. This system replaced the previous approach of using a single shared session state issue with individual tracking issues for each agent.

## Benefits

- **Reduced Noise**: Each agent has its own tracking issues instead of all posting to one shared issue
- **Better Searchability**: Use GitHub/Forgejo search to find specific agent activities
- **Cleaner History**: Individual issues provide focused tracking without cross-agent interference
- **Easier Debugging**: Agent-specific issues make it easier to trace problems
- **Automatic Cleanup**: Stale tracking issues are automatically cleaned up

## Issue Format Standards

### Tracking Issue Title Format

All automation tracking issues MUST use this standardized title format:

```
[AUTO-<AGENT_PREFIX>] <TYPE> (Cycle <N>)
```

**Components:**
- `AUTO-<AGENT_PREFIX>`: Standardized prefix for the agent type
- `<TYPE>`: Type of tracking (Status, Health, Report, etc.)
- `Cycle <N>`: Current cycle number for the agent

### Announcement Issue Title Format

For emergency announcements or important messages:

```
[AUTO-<AGENT_PREFIX>] Announce: <MESSAGE_SUMMARY>
```

### Agent Prefixes

| Agent | Prefix | Example Title |
|-------|--------|---------------|
| session-persister | `AUTO-SESSION` | `[AUTO-SESSION] Checkpoint (Cycle 15)` |
| implementation-pool-supervisor | `AUTO-IMP-POOL` | `[AUTO-IMP-POOL] Health Report (Cycle 42)` |
| system-watchdog-pool-supervisor | `AUTO-WATCHDOG` | `[AUTO-WATCHDOG] System Health (Cycle 8)` |
| backlog-grooming-pool-supervisor | `AUTO-GROOMER` | `[AUTO-GROOMER] Grooming Report (Cycle 23)` |
| human-liaison-pool-supervisor | `AUTO-LIAISON` | `[AUTO-LIAISON] Status Update (Cycle 67)` |
| product-builder | `AUTO-PROD-BLDR` | `[AUTO-PROD-BLDR] Build Session (Cycle 5)` |
| architecture-pool-supervisor | `AUTO-ARCH` | `[AUTO-ARCH] Architecture Report (Cycle 3)` |
| timeline-update-pool-supervisor | `AUTO-TIME` | `[AUTO-TIME] Timeline Update (Cycle 12)` |
| documentation-pool-supervisor | `AUTO-DOCS` | `[AUTO-DOCS] Documentation Report (Cycle 7)` |
| architecture-guard-pool-supervisor | `AUTO-GUARD` | `[AUTO-GUARD] Guard Report (Cycle 9)` |
| pr-review-pool-supervisor | `AUTO-REV-POOL` | `[AUTO-REV-POOL] Review Status (Cycle 4)` |
| uat-test-pool-supervisor | `AUTO-UAT-POOL` | `[AUTO-UAT-POOL] UAT Status (Cycle 6)` |
| project-owner-pool-supervisor | `AUTO-PROJ-OWN` | `[AUTO-PROJ-OWN] Project Status (Cycle 11)` |
| agent-evolution-pool-supervisor | `AUTO-EVLV` | `[AUTO-EVLV] Agent Evolution Report (Cycle 10)` |
| bug-hunt-pool-supervisor | `AUTO-BUG-POOL` | `[AUTO-BUG-POOL] Bug Detection Pool Status (Cycle 60)` |
| spec-update-pool-supervisor | `AUTO-SPEC` | `[AUTO-SPEC] Specification Update Report (Cycle 1)` |
| test-infra-pool-supervisor | `AUTO-INF-POOL` | `[AUTO-INF-POOL] Infrastructure Analysis Report (Cycle 2)` |

## Required Labels

All automation tracking issues MUST include:
- **`Automation Tracking`** - Primary label for filtering
- **`Type/Automation`** - Type label for automation issues
- **`State/In Progress`** - State label while agent is active
- **`Priority/Medium`** - Default priority (adjust as needed)

## Centralized Tracking Manager

The **`automation-tracking-manager`** subagent is the single source of truth for all
automation tracking operations. It was introduced to prevent cycle reuse issues where
agents would incorrectly comment on old tracking issues instead of creating new ones.

### Why Centralized?

Without centralization, agents experienced:
- **Cycle reuse**: Agents posting future cycle reports as comments on old issues
- **Duplicate issues**: Multiple tracking issues created for the same cycle
- **Missing cycles**: Agents skipping cycle numbers due to stale issue references
- **Inconsistent cleanup**: Previous cycle issues not always deleted before creating new ones

### How It Works

The `automation-tracking-manager` subagent:
1. Maintains a persistent cycle counter per agent prefix
2. Searches for and closes the previous cycle's tracking issue
3. Creates a new tracking issue with the correct sequential cycle number
4. Applies all required labels atomically
5. Returns the new issue number to the calling agent

### Usage

Agents that use the centralized manager delegate all tracking operations to it:

```bash
# Example: implementation-pool-supervisor delegating to tracking manager
# (Inside agent definition — pseudocode)
TRACKING_RESULT=$(call_subagent automation-tracking-manager \
  --prefix "AUTO-IMP-POOL" \
  --type "Health Report" \
  --body "...")
NEW_ISSUE_NUMBER=$(echo "$TRACKING_RESULT" | jq -r '.issue_number')
```

### Migrated Agents

The following agents have been migrated to use the centralized tracking manager:
- `system-watchdog-pool-supervisor` (most critical — was skipping cycles)
- `implementation-pool-supervisor` (was creating duplicate issues)
- `timeline-update-pool-supervisor`
- `project-owner-pool-supervisor`
- `product-builder`
- `backlog-grooming-pool-supervisor`

## Interval Reporting and Health Monitoring

### Standardized Interval Declaration

All periodic tracking issues MUST include the expected reporting interval in their description using this standardized format:

```markdown
**Reporting Interval**: <interval> (Next report expected: <timestamp>)
```

This enables automated health monitoring by the system-watchdog to detect stalled or crashed agents.

### Agent Reporting Intervals

| Agent | Issue Type | Interval | Example |
|-------|------------|----------|---------|
| implementation-pool-supervisor | Status Update | Every 5 cycles (~variable timing) | Every worker dispatch cycle |
| implementation-pool-supervisor | Health Report | Every 10 cycles (~variable timing) | Comprehensive pool health |
| backlog-grooming-pool-supervisor | Grooming Report | Every 5 minutes | Backlog maintenance cycle |
| backlog-grooming-pool-supervisor | Health Report | Every 50 minutes (10 cycles) | Grooming health status |
| human-liaison-pool-supervisor | Status Update | Every 20 minutes (10 cycles) | Human activity monitoring |
| system-watchdog-pool-supervisor | Health Report | Every 30 minutes (6 cycles) | System-wide health check |
| session-persister | Checkpoint | Event-driven (variable) | After significant state changes |
| product-builder | Build Session | Every 10 cycles (~variable timing) | Full build session status |
| architecture-pool-supervisor | Architecture Report | Every 10 cycles (~variable timing) | Architecture supervision status |
| timeline-update-pool-supervisor | Timeline Update | Every 10 cycles (~variable timing) | Timeline maintenance status |
| documentation-pool-supervisor | Documentation Report | Every 10 cycles (~3.3 hours) | Documentation update status |
| architecture-guard-pool-supervisor | Guard Report | Every 10 cycles (~variable timing) | Codebase coherence check |
| pr-review-pool-supervisor | Review Status | Every 10 cycles (~variable timing) | PR review pool status |
| uat-test-pool-supervisor | UAT Status | Every 10 cycles (~variable timing) | UAT testing progress |
| project-owner-pool-supervisor | Project Status | Every 10 cycles (~variable timing) | Project ownership decisions |
| agent-evolution-pool-supervisor | Evolution Report | Every 10 cycles (~variable timing) | Agent improvement proposals |
| bug-hunt-pool-supervisor | Bug Detection Status | Every 10 cycles (~variable timing) | Bug detection pool health |
| spec-update-pool-supervisor | Specification Update | Every 10 cycles (~variable timing) | Spec evolution status |
| test-infra-pool-supervisor | Infrastructure Report | Every 10 cycles (~variable timing) | Test infrastructure analysis |

### Automated Health Monitoring

The **system-watchdog** automatically monitors all automation tracking issues:

1. **Scans tracking issues** with "Automation Tracking" label every 5 minutes
2. **Calculates staleness** by comparing current time to issue creation time
3. **Detects stalled agents** when tracking issue is >20% overdue from expected interval
4. **Triggers automated recovery actions**:
   - Kills stalled agent sessions via OpenCode Server API (port 4096)
   - Performs root cause analysis by examining session messages and agent definitions  
   - Creates high-priority diagnostic issues with findings and remediation suggestions
   - Closes stale tracking issues with recovery notes
   - Posts alerts for manual intervention where needed

### Automated Recovery Process

When a stalled agent is detected:

1. **Session Termination**: All matching agent sessions are gracefully terminated
2. **Root Cause Analysis**: Automated analysis examines:
   - Recent session messages and tool call history
   - Related Forgejo issues and PR status  
   - Agent definition and configuration
   - System resource availability
3. **Diagnostic Issue Creation**: Creates issue with format `[AUTO-RECOVERY] <Agent> Agent Failure Analysis`
4. **Tracking Issue Cleanup**: Closes stale tracking issue with recovery notes
5. **Human Notification**: Diagnostic issue requires manual review and restart

### Recovery Issue Format

```markdown
# Agent Failure Analysis — <Agent Name>

**Agent Prefix**: <prefix>
**Detection Time**: <ISO timestamp>
**Stale Issue**: #<issue_number>  
**Time Overdue**: <minutes> minutes
**Staleness Ratio**: <ratio>x expected interval

## Root Cause Analysis
[Automated analysis of session messages, Forgejo issues, and agent definition]

## Recovery Actions Taken
1. ✅ Session Termination: Killed <count> stalled sessions
2. ✅ Tracking Cleanup: Closed stale tracking issue  
3. ✅ Root Cause Analysis: Completed automated failure analysis
4. 🔄 Manual Intervention: Required per recommendations

## Recommended Next Steps
[Specific recommendations based on failure analysis]
```

### Health Check Algorithm

```python
# Pseudocode for health monitoring
for issue in automation_tracking_issues:
    if issue.title.matches("[AUTO-*] * (Cycle *)"):
        expected_interval = get_expected_interval(issue.agent_prefix, issue.type)
        time_since_creation = now() - issue.created_at
        staleness_threshold = expected_interval * 1.2  # 20% tolerance
        
        if time_since_creation > staleness_threshold:
            mark_agent_as_stalled(issue.agent_prefix)
            trigger_recovery_actions(issue.agent_prefix, issue)
```

## Standardized Tracking Issue Templates

### Common Header Format

All tracking issues MUST start with this standardized header:

```markdown
# <Agent Name> <Issue Type> — <Timestamp>

**Agent**: <agent-name>
**Cycle**: <cycle-number>
**Reporting Interval**: <interval> (Next report expected: <next-timestamp>)
**Status**: <active|warning|error>

## Summary

<Brief status summary>

## Details

<Detailed information specific to agent type>

## Health Indicators

<Health metrics and status indicators>

## Next Actions

<Planned actions for next cycle>

---
**Automated by CleverAgents Bot**  
Supervisor: <supervisor-name> | Agent: <agent-name>
```

### Template Variables

- **`<agent-name>`**: The agent type (e.g., "implementation-orchestrator", "backlog-groomer")  
- **`<cycle-number>`**: Current cycle number for this agent
- **`<interval>`**: Expected time between reports (e.g., "5 minutes", "10 cycles")
- **`<next-timestamp>`**: When the next report is expected (ISO 8601 format)
- **`<status>`**: Current agent health status
  - `active`: Normal operation
  - `warning`: Non-critical issues detected  
  - `error`: Critical issues requiring attention

## Implementation Requirements

### For All Agents

Every agent that creates tracking issues must implement:

1. **Cleanup Function**: Delete previous cycle tracking issues before creating new ones
2. **Standardized Titles**: Use the exact format specified above
3. **Proper Labels**: Always include "Automation Tracking", "Type/Automation", "State/In Progress", and "Priority/Medium" labels
4. **Structured Content**: Use consistent formatting for issue bodies

### Standard Tracking Functions

Each agent should implement these functions:

```markdown
## Automation Tracking Functions

### Delete Previous Tracking Issues
1. Search for issues with title pattern `[AUTO-<PREFIX>] <TYPE> (Cycle *)`
2. Filter for issues created by this agent
3. Close and delete previous cycle issues
4. Preserve announcement issues (different title pattern)

### Create New Tracking Issue
1. Generate title with current cycle number
2. Include "Automation Tracking", "Type/Automation", "State/In Progress", "Priority/Medium" labels
3. Add structured content with timestamp and status
4. Link to relevant repositories/PRs as needed
```

### Agent-Specific Implementation Details

#### session-persister
- **Cycle Frequency**: Every checkpoint (variable timing)
- **Issue Type**: Checkpoint
- **Content**: Session state, active agents, configuration
- **Cleanup**: Deletes previous checkpoint issue

#### implementation-pool-supervisor  
- **Cycle Frequency**: Health reports every 10 cycles, status every 5 cycles
- **Issue Types**: Health Report, Status Update
- **Content**: Worker status, queue health, completion metrics
- **Cleanup**: Deletes previous health/status issues separately

#### system-watchdog-pool-supervisor
- **Cycle Frequency**: Every monitoring cycle (variable timing)
- **Issue Types**: System Health, Alert
- **Content**: Quality gates, system violations, corrective actions
- **Cleanup**: Deletes previous system health issues

#### backlog-grooming-pool-supervisor
- **Cycle Frequency**: Every grooming cycle (~30 minutes)
- **Issue Types**: Grooming Report, Scope Alert
- **Content**: Issues processed, duplicates found, orphans discovered
- **Cleanup**: Deletes previous grooming reports, acts as backup cleanup agent

#### human-liaison-pool-supervisor
- **Cycle Frequency**: Status every 10 cycles
- **Issue Types**: Status Update, Human Activity Summary  
- **Content**: Human interactions, triage decisions, outstanding items
- **Cleanup**: Deletes previous status updates

## Searching and Filtering

### Finding Tracking Issues

**All automation tracking:**
```
label:"Automation Tracking"
```

**Specific agent tracking:**
```
label:"Automation Tracking" [AUTO-SESSION] in:title
label:"Automation Tracking" [AUTO-IMP-POOL] in:title
label:"Automation Tracking" [AUTO-WATCHDOG] in:title
label:"Automation Tracking" [AUTO-GROOMER] in:title
label:"Automation Tracking" [AUTO-LIAISON] in:title
label:"Automation Tracking" [AUTO-PROD-BLDR] in:title
label:"Automation Tracking" [AUTO-ARCH] in:title
label:"Automation Tracking" [AUTO-TIME] in:title
label:"Automation Tracking" [AUTO-DOCS] in:title
label:"Automation Tracking" [AUTO-GUARD] in:title
label:"Automation Tracking" [AUTO-REV-POOL] in:title
label:"Automation Tracking" [AUTO-UAT-POOL] in:title
label:"Automation Tracking" [AUTO-PROJ-OWN] in:title
label:"Automation Tracking" [AUTO-EVLV] in:title
label:"Automation Tracking" [AUTO-BUG-POOL] in:title
label:"Automation Tracking" [AUTO-SPEC] in:title
label:"Automation Tracking" [AUTO-INF-POOL] in:title
```

**Recent tracking issues (last 24 hours):**
```
label:"Automation Tracking" created:>2024-01-01T00:00:00Z
```

**Announcement issues only:**
```
label:"Automation Tracking" "Announce:" in:title
```

## Cleanup Protocol

### Primary Cleanup (Each Agent)
- Each agent deletes its own previous cycle tracking issues before creating new ones
- Preserves announcement issues which use different title format
- Runs cleanup before every new tracking issue creation

### Secondary Cleanup (backlog-groomer)
The backlog-groomer acts as a backup cleanup agent:
- Scans for stale automation tracking issues older than 7 days
- Deletes abandoned tracking issues from agents that may have crashed
- Preserves announcement issues and recently created tracking issues
- Reports cleanup activity in its own tracking issues

### Emergency Cleanup (Manual)
If tracking issues accumulate excessively:
1. Use GitHub/Forgejo bulk operations with label filter
2. Search: `label:"Automation Tracking" created:<7-days-ago`
3. Bulk close/delete stale tracking issues
4. Preserve announcement issues

## Troubleshooting

### Common Issues

**"Automation Tracking" label missing:**
- Ensure label exists in repository
- Check agent permission to add labels
- Verify label name spelling (case sensitive)

**Tracking issues not being cleaned up:**
- Check if agent cleanup function is running
- Verify backlog-groomer is active and functioning
- Check for agent crashes or hangs preventing cleanup

**Too many tracking issues:**
- May indicate cleanup functions not working
- Check agent logs for cleanup errors
- Run manual cleanup if necessary

**Cannot find specific agent tracking:**
- Verify agent prefix in search
- Check if agent is actually running and creating issues
- Confirm agent is using correct title format

**Agents posting to old tracking issues (cycle reuse):**
- Agent is not using the centralized `automation-tracking-manager`
- Migrate agent to use the centralized manager subagent
- Check that previous cycle cleanup runs before new issue creation

### Debugging Steps

1. **Check agent status**: Verify agent is running and healthy
2. **Verify title format**: Ensure exact compliance with format standards
3. **Check labels**: Confirm "Automation Tracking" label is applied
4. **Review cleanup logs**: Look for cleanup function errors
5. **Manual search**: Use repository issue search to verify tracking issues

## Migration Notes

This system replaced the previous shared session state issue approach. Key differences:

- **Before**: All agents posted comments to single issue #[session-state-issue-number]
- **After**: Each agent creates individual tracking issues with standardized titles
- **Benefit**: Eliminates noise, improves searchability, enables targeted cleanup

The migration was completed by updating agent definitions in `.opencode/agents/` directory to:
- Remove session state issue posting
- Add tracking issue creation functions
- Implement cleanup protocols
- Apply standardized formatting

A second migration introduced the centralized `automation-tracking-manager` subagent
to prevent cycle reuse issues where agents were commenting on old tracking issues
instead of creating new ones.

## Related Documentation

- [System Watchdog](system-watchdog.md) - Specific documentation for watchdog agent
- [Quality Automation](quality-automation.md) - Quality gate automation details
- [Ops Runbook](ops-runbook.md) - General operational procedures