cleveragents-core/docs/development/automation-tracking.md

# Automation Tracking System

## Overview

The CleverAgents automation tracking system provides structured, searchable tracking for all autonomous agent activities. This system replaced the previous approach of using a single shared session state issue with individual tracking issues for each agent.

All tracking operations are centralized through the **`automation-tracking-manager`** subagent — the single source of truth for cycle numbers, issue creation, cleanup, and state reads. Agents MUST NOT create or close tracking issues directly; they invoke the manager subagent instead.

## Benefits

- **Reduced Noise**: Each agent has its own tracking issues instead of all posting to one shared issue
- **Better Searchability**: Use GitHub/Forgejo search to find specific agent activities
- **Cleaner History**: Individual issues provide focused tracking without cross-agent interference
- **Easier Debugging**: Agent-specific issues make it easier to trace problems
- **Automatic Cleanup**: Stale tracking issues are automatically cleaned up
- **Cycle Continuity**: Sequential cycle numbers persist across agent restarts via the centralized manager
- **No Duplicate Issues**: The manager ensures exactly one tracking issue per cycle per agent

## Issue Format Standards

### Tracking Issue Title Format

All automation tracking issues MUST use this standardized title format:

```
[AUTO-<AGENT_PREFIX>] <TYPE> (Cycle <N>)
```

**Components:**
- `AUTO-<AGENT_PREFIX>`: Standardized prefix for the agent type
- `<TYPE>`: Type of tracking (Status, Health, Report, etc.)
- `Cycle <N>`: Current cycle number for the agent

### Announcement Issue Title Format

For emergency announcements or important messages:

```
[AUTO-<AGENT_PREFIX>] Announce: <MESSAGE_SUMMARY>
```

### Agent Prefixes

All prefixes are registered in the `automation-tracking-manager` subagent, which is the single source of truth for the prefix registry:

| Agent | Prefix | Example Title |
|-------|--------|---------------|
| session-persister | `AUTO-SESSION` | `[AUTO-SESSION] Checkpoint (Cycle 15)` |
| implementation-pool-supervisor | `AUTO-IMP-POOL` | `[AUTO-IMP-POOL] Health Report (Cycle 42)` |
| system-watchdog-pool-supervisor | `AUTO-WATCHDOG` | `[AUTO-WATCHDOG] System Health (Cycle 8)` |
| backlog-grooming-pool-supervisor | `AUTO-GROOMER` | `[AUTO-GROOMER] Grooming Report (Cycle 23)` |
| human-liaison-pool-supervisor | `AUTO-LIAISON` | `[AUTO-LIAISON] Status Update (Cycle 67)` |
| product-builder | `AUTO-PROD-BLDR` | `[AUTO-PROD-BLDR] Build Session (Cycle 5)` |
| architecture-pool-supervisor | `AUTO-ARCH` | `[AUTO-ARCH] Architecture Report (Cycle 3)` |
| timeline-update-pool-supervisor | `AUTO-TIME` | `[AUTO-TIME] Timeline Update (Cycle 12)` |
| documentation-pool-supervisor, docs-writer | `AUTO-DOCS` | `[AUTO-DOCS] Documentation Report (Cycle 7)` |
| architecture-guard-pool-supervisor | `AUTO-GUARD` | `[AUTO-GUARD] Guard Report (Cycle 9)` |
| pr-review-pool-supervisor | `AUTO-REV-POOL` | `[AUTO-REV-POOL] Review Status (Cycle 4)` |
| pr-fix-pool-supervisor | `AUTO-FIX-POOL` | `[AUTO-FIX-POOL] PR Fix Status (Cycle 5)` |
| pr-merge-pool-supervisor | `AUTO-MERGE` | `[AUTO-MERGE] PR Merge Status (Cycle 3)` |
| uat-test-pool-supervisor | `AUTO-UAT-POOL` | `[AUTO-UAT-POOL] UAT Status (Cycle 6)` |
| project-owner-pool-supervisor | `AUTO-PROJ-OWN` | `[AUTO-PROJ-OWN] Project Status (Cycle 11)` |
| agent-evolution-pool-supervisor | `AUTO-EVLV` | `[AUTO-EVLV] Agent Evolution Report (Cycle 10)` |
| bug-hunt-pool-supervisor | `AUTO-BUG-SUP` | `[AUTO-BUG-SUP] Bug Hunt Status (Cycle 25)` |
| spec-update-pool-supervisor | `AUTO-SPEC` | `[AUTO-SPEC] Specification Update Report (Cycle 1)` |
| test-infra-pool-supervisor | `AUTO-INF-POOL` | `[AUTO-INF-POOL] Infrastructure Analysis Report (Cycle 2)` |
| epic-planner | `AUTO-EPIC` | `[AUTO-EPIC] Epic Planning Update (Cycle 9)` |

## Default Labels

The `automation-tracking-manager` automatically applies the **`Automation Tracking`**
label to every tracking issue so that system-watchdog health checks and global searches
remain reliable. Teams may add additional labels such as `Type/Automation`,
`State/In Progress`, or `Priority/Medium` to reflect their workflow, but those labels are
optional and are not applied automatically today. Agents MUST NOT attempt to add labels
directly — the manager delegates to `forgejo-label-manager` and handles all label
application on their behalf.

## Centralized Tracking Manager

All tracking operations are handled by the **`automation-tracking-manager`** subagent
(`.opencode/agents/automation-tracking-manager.md`). The manager is the single source of
truth for cycle numbers, issue creation, cleanup, and state reads. It was introduced to
eliminate cycle reuse, duplicate issues, and inconsistent cleanup that occurred when
agents managed tracking issues individually.

### Why Centralization Matters

- **Cycle reuse prevention**: agents never post future reports to stale issues.
- **Duplicate avoidance**: exactly one tracking issue exists per cycle per prefix.
- **Sequential numbering**: cycle counters persist across crashes and restarts.
- **Consistent cleanup**: previous cycles close automatically before new ones open.

### Operations

| Operation | Description |
|-----------|-------------|
| `CREATE_TRACKING_ISSUE` | Closes the previous cycle, determines the next cycle number, and creates the new issue |
| `UPDATE_TRACKING_ISSUE` | Adds a comment to the current open tracking issue |
| `CLOSE_TRACKING_ISSUE` | Closes the current tracking issue with a closure comment |
| `READ_TRACKING_STATE` | Reads the latest tracking issue and extracts cycle metadata |
| `GET_NEXT_CYCLE_NUMBER` | Scans open and closed issues to determine the next sequential cycle number |

### Cycle Number Continuity

The manager inspects both open and closed tracking issues to determine the highest cycle
number for each prefix, then increments by one. This guarantees sequential numbering even
when agents restart or experience failures.

### Rules Enforced by the Manager

1. **One issue per cycle** — the previous cycle is closed before a new one opens.
2. **Sequential cycle numbers** — cycle counters persist across restarts.
3. **Label enforcement** — the `Automation Tracking` label is applied to every issue.
4. **Cleanup before creation** — stale tracking issues are never left behind.
5. **Comment validation** — updates only target the current, open tracking issue.

### Integration Requirements

Agents MUST NOT:

- Create tracking issues directly via the Forgejo API
- Manually manage or increment cycle numbers
- Close tracking issues without going through the manager
- Comment on tracking issues directly

Instead, every agent invokes the `automation-tracking-manager` subagent for tracking
operations. Example pseudocode:

```bash
TRACKING_RESULT=$(call_subagent automation-tracking-manager \
  --prefix "AUTO-IMP-POOL" \
  --type "Health Report" \
  --body "...")
NEW_ISSUE_NUMBER=$(echo "$TRACKING_RESULT" | jq -r '.issue_number')
```

The initial migration covered `system-watchdog-pool-supervisor`,
`implementation-pool-supervisor`, `timeline-update-pool-supervisor`,
`project-owner-pool-supervisor`, `product-builder`, and
`backlog-grooming-pool-supervisor`, with subsequent agents added as the registry expanded.

## Interval Reporting and Health Monitoring

### Standardized Interval Declaration

All periodic tracking issues MUST include the expected reporting interval in their description using this standardized format:

```markdown
**Reporting Interval**: <interval> (Next report expected: <timestamp>)
```

This enables automated health monitoring by the system-watchdog to detect stalled or crashed agents.

### Agent Reporting Intervals

| Agent | Issue Type | Interval | Example |
|-------|------------|----------|---------|
| implementation-pool-supervisor | Status Update | Every 5 cycles (~variable timing) | Every worker dispatch cycle |
| implementation-pool-supervisor | Health Report | Every 10 cycles (~variable timing) | Comprehensive pool health |
| backlog-grooming-pool-supervisor | Grooming Report | Every 5 minutes | Backlog maintenance cycle |
| backlog-grooming-pool-supervisor | Health Report | Every 50 minutes (10 cycles) | Grooming health status |
| human-liaison-pool-supervisor | Status Update | Every 20 minutes (10 cycles) | Human activity monitoring |
| system-watchdog-pool-supervisor | Health Report | Every 30 minutes (6 cycles) | System-wide health check |
| session-persister | Session State Tracking | Event-driven (variable) | After significant state changes |
| documentation-pool-supervisor / docs-writer | Documentation Report | Every 10 cycles (~3.3 hours) | Documentation monitoring cycle |
| product-builder | Build Session | Every 10 cycles (~variable timing) | Full build session status |
| architecture-pool-supervisor | Architecture Report | Every 10 cycles (~variable timing) | Architecture supervision status |
| architecture-guard-pool-supervisor | Guard Report | Every 10 cycles (~variable timing) | Codebase coherence check |
| timeline-update-pool-supervisor | Timeline Update | Per-commit (variable) | After timeline-worthy events |
| pr-review-pool-supervisor | Review Status | Every 10 cycles (~variable timing) | PR review pool status |
| pr-fix-pool-supervisor | PR Fix Status | Every 10 cycles (~variable timing) | PR CI fix pool status |
| pr-merge-pool-supervisor | PR Merge Status | Every 10 cycles (~variable timing) | Automated PR merge status |
| uat-test-pool-supervisor | UAT Status | Every 10 cycles (~variable timing) | UAT testing progress |
| project-owner-pool-supervisor | Project Status | Per-triage-cycle (variable) | After issue triage passes |
| agent-evolution-pool-supervisor | Evolution Report | Every 10 cycles (~variable timing) | Agent improvement proposals |
| epic-planner | Epic Planning Update | Every 10 cycles (~variable timing) | Workstream planning summary |
| bug-hunt-pool-supervisor | Bug Detection Status | Every 10 cycles (~variable timing) | Bug detection pool health |
| spec-update-pool-supervisor | Specification Update | Every 10 cycles (~variable timing) | Spec evolution status |
| test-infra-pool-supervisor | Infrastructure Report | Every 10 cycles (~variable timing) | Test infrastructure analysis |

### Automated Health Monitoring

The **system-watchdog** automatically monitors all automation tracking issues:

1. **Scans tracking issues** with "Automation Tracking" label every 5 minutes
2. **Calculates staleness** by comparing current time to issue creation time
3. **Detects stalled agents** when tracking issue is >20% overdue from expected interval
4. **Triggers automated recovery actions**:
   - Kills stalled agent sessions via OpenCode Server API (port 4096)
   - Performs root cause analysis by examining session messages and agent definitions  
   - Creates high-priority diagnostic issues with findings and remediation suggestions
   - Closes stale tracking issues with recovery notes
   - Posts alerts for manual intervention where needed

### Automated Recovery Process

When a stalled agent is detected:

1. **Session Termination**: All matching agent sessions are gracefully terminated
2. **Root Cause Analysis**: Automated analysis examines:
   - Recent session messages and tool call history
   - Related Forgejo issues and PR status  
   - Agent definition and configuration
   - System resource availability
3. **Diagnostic Issue Creation**: Creates issue with format `[AUTO-RECOVERY] <Agent> Agent Failure Analysis`
4. **Tracking Issue Cleanup**: Closes stale tracking issue with recovery notes
5. **Human Notification**: Diagnostic issue requires manual review and restart

### Recovery Issue Format

```markdown
# Agent Failure Analysis — <Agent Name>

**Agent Prefix**: <prefix>
**Detection Time**: <ISO timestamp>
**Stale Issue**: #<issue_number>  
**Time Overdue**: <minutes> minutes
**Staleness Ratio**: <ratio>x expected interval

## Root Cause Analysis
[Automated analysis of session messages, Forgejo issues, and agent definition]

## Recovery Actions Taken
1. ✅ Session Termination: Killed <count> stalled sessions
2. ✅ Tracking Cleanup: Closed stale tracking issue  
3. ✅ Root Cause Analysis: Completed automated failure analysis
4. 🔄 Manual Intervention: Required per recommendations

## Recommended Next Steps
[Specific recommendations based on failure analysis]
```

### Health Check Algorithm

```python
# Pseudocode for health monitoring
for issue in automation_tracking_issues:
    if issue.title.matches("[AUTO-*] * (Cycle *)"):
        expected_interval = get_expected_interval(issue.agent_prefix, issue.type)
        time_since_creation = now() - issue.created_at
        staleness_threshold = expected_interval * 1.2  # 20% tolerance
        
        if time_since_creation > staleness_threshold:
            mark_agent_as_stalled(issue.agent_prefix)
            trigger_recovery_actions(issue.agent_prefix, issue)
```

## Standardized Tracking Issue Templates

### Common Header Format

All tracking issues MUST start with this standardized header:

```markdown
# <Agent Name> <Issue Type> — <Timestamp>

**Agent**: <agent-name>
**Cycle**: <cycle-number>
**Reporting Interval**: <interval> (Next report expected: <next-timestamp>)
**Status**: <active|warning|error>

## Summary

<Brief status summary>

## Details

<Detailed information specific to agent type>

## Health Indicators

<Health metrics and status indicators>

## Next Actions

<Planned actions for next cycle>

---
**Automated by CleverAgents Bot**  
Supervisor: <supervisor-name> | Agent: <agent-name>
```

### Template Variables

- **`<agent-name>`**: The agent type (e.g., "implementation-orchestrator", "backlog-groomer")  
- **`<cycle-number>`**: Current cycle number for this agent
- **`<interval>`**: Expected time between reports (e.g., "5 minutes", "10 cycles")
- **`<next-timestamp>`**: When the next report is expected (ISO 8601 format)
- **`<status>`**: Current agent health status
  - `active`: Normal operation
  - `warning`: Non-critical issues detected  
  - `error`: Critical issues requiring attention

## Implementation Requirements

### For All Agents

Every agent that creates tracking issues must implement:

1. **Call the manager** for every create, update, or close operation — no direct Forgejo API usage.
2. **Provide standardized metadata** (prefix, issue type, reporting interval) when invoking the manager so titles remain consistent.
3. **Populate the body using the template**, including summary, details, health indicators, and next actions.
4. **Capture meaningful health signals** (queues, failure counts, outstanding actions) that supervisors can act on.
5. **Persist the returned issue number** for subsequent updates during the same cycle.

### Agent-Specific Implementation Details

#### session-persister
- **Cycle Frequency**: Every checkpoint (variable timing)
- **Issue Type**: Session State Tracking
- **Content**: Session state, active agents, configuration
- **Cleanup**: Handled by automation-tracking-manager

#### implementation-pool-supervisor  
- **Cycle Frequency**: Status updates every 5 cycles; comprehensive health reports every 10 cycles
- **Issue Types**: Status Update, Health Report
- **Content**: Worker status, queue health, completion metrics
- **Cleanup**: Handled by automation-tracking-manager

#### system-watchdog-pool-supervisor
- **Cycle Frequency**: Every monitoring cycle (variable timing)
- **Issue Type**: System Health Report
- **Content**: Quality gates, system violations, corrective actions
- **Cleanup**: Handled by automation-tracking-manager

#### backlog-grooming-pool-supervisor
- **Cycle Frequency**: Grooming report every 5 minutes; scope alerts every 50 minutes (10 cycles)
- **Issue Types**: Grooming Report, Scope Alert
- **Content**: Issues processed, duplicates found, orphans discovered
- **Cleanup**: Handled by automation-tracking-manager

#### human-liaison-pool-supervisor
- **Cycle Frequency**: Status every 10 cycles
- **Issue Type**: Human Activity Report
- **Content**: Human interactions, triage decisions, outstanding items
- **Cleanup**: Handled by automation-tracking-manager

#### docs-writer
- **Cycle Frequency**: Documentation reports every 10 cycles (~3.3 hours)
- **Issue Type**: Documentation Report
- **Content**: Docs created/updated/skipped, commit hash, milestone coverage
- **Cleanup**: Handled by automation-tracking-manager

#### timeline-updater
- **Cycle Frequency**: Per-commit (variable timing)
- **Issue Type**: Timeline Update
- **Content**: Timeline changes, schedule adherence entries
- **Cleanup**: Handled by automation-tracking-manager

#### project-owner
- **Cycle Frequency**: Per-triage-cycle (variable timing)
- **Issue Type**: Project Triage Report
- **Content**: Issues triaged, MoSCoW assignments, priority decisions
- **Cleanup**: Handled by automation-tracking-manager

#### product-builder
- **Cycle Frequency**: Per-build-cycle (variable timing)
- **Issue Type**: Product Builder Status
- **Content**: Build orchestration status, worker pool health
- **Cleanup**: Handled by automation-tracking-manager

#### docs-writer
- **Cycle Frequency**: Documentation reports every 10 cycles (~3.3 hours)
- **Issue Types**: Documentation Report
- **Content**: Docs created/updated/skipped, commit hash, milestone coverage
- **Cleanup**: Deletes previous documentation report issues

## Searching and Filtering

### Finding Tracking Issues

**All automation tracking:**
```
label:"Automation Tracking"
```

**Specific agent tracking:**
```
label:"Automation Tracking" [AUTO-SESSION] in:title
label:"Automation Tracking" [AUTO-IMP-POOL] in:title
label:"Automation Tracking" [AUTO-WATCHDOG] in:title
label:"Automation Tracking" [AUTO-GROOMER] in:title
label:"Automation Tracking" [AUTO-LIAISON] in:title
label:"Automation Tracking" [AUTO-PROD-BLDR] in:title
label:"Automation Tracking" [AUTO-ARCH] in:title
label:"Automation Tracking" [AUTO-TIME] in:title
label:"Automation Tracking" [AUTO-DOCS] in:title
label:"Automation Tracking" [AUTO-GUARD] in:title
label:"Automation Tracking" [AUTO-REV-POOL] in:title
label:"Automation Tracking" [AUTO-FIX-POOL] in:title
label:"Automation Tracking" [AUTO-MERGE] in:title
label:"Automation Tracking" [AUTO-UAT-POOL] in:title
label:"Automation Tracking" [AUTO-PROJ-OWN] in:title
label:"Automation Tracking" [AUTO-EVLV] in:title
label:"Automation Tracking" [AUTO-EPIC] in:title
label:"Automation Tracking" [AUTO-BUG-SUP] in:title
label:"Automation Tracking" [AUTO-SPEC] in:title
label:"Automation Tracking" [AUTO-INF-POOL] in:title
```

**Recent tracking issues (last 24 hours):**
```
label:"Automation Tracking" created:>2024-01-01T00:00:00Z
```

**Announcement issues only:**
```
label:"Automation Tracking" "Announce:" in:title
```

## Cross-Agent Announcements

The automation tracking system supports emergency announcements and important messages
that need to be visible to all agents.

### Announcement Operations

The `automation-tracking-manager` subagent supports the following announcement operations:

| Operation | Description |
|-----------|-------------|
| `CREATE_ANNOUNCEMENT_ISSUE` | Create a new announcement with priority support |
| `CLOSE_ANNOUNCEMENT_ISSUE` | Close an announcement when it is no longer relevant |
| `LIST_TRACKING_ISSUES` | List all current tracking and announcement issues |
| `READ_ANNOUNCEMENTS` | Read announcements from other agents (cross-agent awareness) |
| `REVIEW_OWN_ANNOUNCEMENTS` | Review and clean up own announcements |

### Announcement Lifecycle

1. **Supervisors** read critical announcements before each cycle
2. **Workers** read announcements from system agents and orchestrators
3. **Priority-based filtering** reduces noise (Critical/High/Medium/Low)
4. **Backlog-groomer** performs intelligent cleanup with age thresholds by priority:
   - Critical: 72 hours
   - High: 48 hours
   - Medium: 24 hours
   - Low: 12 hours

### Announcement Title Format

```
[AUTO-<AGENT_PREFIX>] Announce: <MESSAGE_SUMMARY>
```

Example: `[AUTO-WATCHDOG] Announce: CRITICAL: CI pipeline down — all merges blocked`

## Cleanup Protocol

### Primary Cleanup (automation-tracking-manager)
The `automation-tracking-manager` subagent handles all primary cleanup:
- Closes the previous cycle's tracking issue before creating a new one
- Preserves announcement issues (different title format — no `(Cycle N)` suffix)
- Runs cleanup atomically as part of every `CREATE_TRACKING_ISSUE` operation

### Secondary Cleanup (backlog-groomer)
The backlog-groomer acts as a backup cleanup agent:
- Scans for stale automation tracking issues older than 7 days
- Closes abandoned tracking issues from agents that may have crashed
- Preserves announcement issues and recently created tracking issues
- Reports cleanup activity in its own tracking issues

### Emergency Cleanup (Manual)
If tracking issues accumulate excessively:
1. Use GitHub/Forgejo bulk operations with label filter
2. Search: `label:"Automation Tracking" created:<7-days-ago`
3. Bulk close/delete stale tracking issues
4. Preserve announcement issues

## Troubleshooting

### Common Issues

**"Automation Tracking" label missing:**
- Ensure label exists in repository
- Check agent permission to add labels
- Verify label name spelling (case sensitive)

**Tracking issues not being cleaned up:**
- Check if agent cleanup function is running
- Verify backlog-groomer is active and functioning
- Check for agent crashes or hangs preventing cleanup

**Too many tracking issues:**
- May indicate cleanup functions not working
- Check agent logs for cleanup errors
- Run manual cleanup if necessary

**Cannot find specific agent tracking:**
- Verify agent prefix in search
- Check if agent is actually running and creating issues
- Confirm agent is using correct title format

**Agents posting to old tracking issues (cycle reuse):**
- Agent is not using the centralized `automation-tracking-manager`
- Migrate agent to use the centralized manager subagent
- Check that previous cycle cleanup runs before new issue creation

### Debugging Steps

1. **Check agent status**: Verify agent is running and healthy
2. **Verify title format**: Ensure exact compliance with format standards
3. **Check labels**: Confirm "Automation Tracking" label is applied
4. **Review cleanup logs**: Look for cleanup function errors
5. **Manual search**: Use repository issue search to verify tracking issues

## Migration Notes

### Phase 1 — Individual Tracking Issues (completed)

Replaced the previous shared session state issue approach:

- **Before**: All agents posted comments to single issue #[session-state-issue-number]
- **After**: Each agent creates individual tracking issues with standardized titles
- **Benefit**: Eliminates noise, improves searchability, enables targeted cleanup

### Phase 2 — Centralized Tracking Manager (completed)

Replaced direct Forgejo API calls in each agent with the `automation-tracking-manager` subagent:

- **Before**: Each agent implemented its own tracking issue creation, cleanup, and cycle management
- **After**: All agents delegate to `automation-tracking-manager` as the single source of truth
- **Benefit**: Eliminates duplicate issues, ensures cycle continuity across restarts, prevents agents from commenting on stale issues
- **Breaking change**: `shared/automation_tracking.md` was removed; agents now use the subagent pattern

The migration updated these agents: `system-watchdog`, `implementation-orchestrator`,
`timeline-updater`, `project-owner`, `product-builder`, `backlog-groomer`.

A second migration introduced the centralized `automation-tracking-manager` subagent
to prevent cycle reuse issues where agents were commenting on old tracking issues
instead of creating new ones.

## Related Documentation

- [System Watchdog](system-watchdog.md) - Specific documentation for watchdog agent
- [Documentation Writer](docs-writer.md) - Documentation writer agent reference
- [Quality Automation](quality-automation.md) - Quality gate automation details
- [Ops Runbook](ops-runbook.md) - General operational procedures