cleveragents-core/docs/development/automation-tracking.md

Automation Tracking System

Overview

The CleverAgents automation tracking system provides structured, searchable tracking for all autonomous agent activities. This system replaced the previous approach of using a single shared session state issue with individual tracking issues for each agent.

All tracking operations are centralized through the automation-tracking-manager subagent — the single source of truth for cycle numbers, issue creation, cleanup, and state reads. Agents MUST NOT create or close tracking issues directly; they invoke the manager subagent instead.

Benefits

• Reduced Noise: Each agent has its own tracking issues instead of all posting to one shared issue
• Better Searchability: Use GitHub/Forgejo search to find specific agent activities
• Cleaner History: Individual issues provide focused tracking without cross-agent interference
• Easier Debugging: Agent-specific issues make it easier to trace problems
• Automatic Cleanup: Stale tracking issues are automatically cleaned up
• Cycle Continuity: Sequential cycle numbers persist across agent restarts via the centralized manager
• No Duplicate Issues: The manager ensures exactly one tracking issue per cycle per agent

Issue Format Standards

Tracking Issue Title Format

All automation tracking issues MUST use this standardized title format:

[AUTO-<AGENT_PREFIX>] <TYPE> (Cycle <N>)

Components:

• AUTO-<AGENT_PREFIX>: Standardized prefix for the agent type
• <TYPE>: Type of tracking (Status, Health, Report, etc.)
• Cycle <N>: Current cycle number for the agent

Announcement Issue Title Format

For emergency announcements or important messages:

[AUTO-<AGENT_PREFIX>] Announce: <MESSAGE_SUMMARY>

Agent Prefixes

All prefixes are registered in the automation-tracking-manager subagent, which is the single source of truth for the prefix registry:

Agent	Prefix	Example Title
session-persister	AUTO-SESSION	[AUTO-SESSION] Checkpoint (Cycle 15)
implementation-pool-supervisor	AUTO-IMP-POOL	[AUTO-IMP-POOL] Health Report (Cycle 42)
system-watchdog-pool-supervisor	AUTO-WATCHDOG	[AUTO-WATCHDOG] System Health (Cycle 8)
backlog-grooming-pool-supervisor	AUTO-GROOMER	[AUTO-GROOMER] Grooming Report (Cycle 23)
human-liaison-pool-supervisor	AUTO-LIAISON	[AUTO-LIAISON] Status Update (Cycle 67)
product-builder	AUTO-PROD-BLDR	[AUTO-PROD-BLDR] Build Session (Cycle 5)
architecture-pool-supervisor	AUTO-ARCH	[AUTO-ARCH] Architecture Report (Cycle 3)
timeline-update-pool-supervisor	AUTO-TIME	[AUTO-TIME] Timeline Update (Cycle 12)
documentation-pool-supervisor, docs-writer	AUTO-DOCS	[AUTO-DOCS] Documentation Report (Cycle 7)
architecture-guard-pool-supervisor	AUTO-GUARD	[AUTO-GUARD] Guard Report (Cycle 9)
pr-review-pool-supervisor	AUTO-REV-POOL	[AUTO-REV-POOL] Review Status (Cycle 4)
pr-fix-pool-supervisor	AUTO-FIX-POOL	[AUTO-FIX-POOL] PR Fix Status (Cycle 5)
pr-merge-pool-supervisor	AUTO-MERGE	[AUTO-MERGE] PR Merge Status (Cycle 3)
uat-test-pool-supervisor	AUTO-UAT-POOL	[AUTO-UAT-POOL] UAT Status (Cycle 6)
project-owner-pool-supervisor	AUTO-PROJ-OWN	[AUTO-PROJ-OWN] Project Status (Cycle 11)
agent-evolution-pool-supervisor	AUTO-EVLV	[AUTO-EVLV] Agent Evolution Report (Cycle 10)
bug-hunt-pool-supervisor	AUTO-BUG-SUP	[AUTO-BUG-SUP] Bug Hunt Status (Cycle 25)
spec-update-pool-supervisor	AUTO-SPEC	[AUTO-SPEC] Specification Update Report (Cycle 1)
test-infra-pool-supervisor	AUTO-INF-POOL	[AUTO-INF-POOL] Infrastructure Analysis Report (Cycle 2)
epic-planner	AUTO-EPIC	[AUTO-EPIC] Epic Planning Update (Cycle 9)

Default Labels

The automation-tracking-manager automatically applies the Automation Tracking label to every tracking issue so that system-watchdog health checks and global searches remain reliable. Teams may add additional labels such as Type/Automation, State/In Progress, or Priority/Medium to reflect their workflow, but those labels are optional and are not applied automatically today. Agents MUST NOT attempt to add labels directly — the manager delegates to forgejo-label-manager and handles all label application on their behalf.

Centralized Tracking Manager

All tracking operations are handled by the automation-tracking-manager subagent (.opencode/agents/automation-tracking-manager.md). The manager is the single source of truth for cycle numbers, issue creation, cleanup, and state reads. It was introduced to eliminate cycle reuse, duplicate issues, and inconsistent cleanup that occurred when agents managed tracking issues individually.

Why Centralization Matters

• Cycle reuse prevention: agents never post future reports to stale issues.
• Duplicate avoidance: exactly one tracking issue exists per cycle per prefix.
• Sequential numbering: cycle counters persist across crashes and restarts.
• Consistent cleanup: previous cycles close automatically before new ones open.

Operations

Operation	Description
CREATE_TRACKING_ISSUE	Closes the previous cycle, determines the next cycle number, and creates the new issue
UPDATE_TRACKING_ISSUE	Adds a comment to the current open tracking issue
CLOSE_TRACKING_ISSUE	Closes the current tracking issue with a closure comment
READ_TRACKING_STATE	Reads the latest tracking issue and extracts cycle metadata
GET_NEXT_CYCLE_NUMBER	Scans open and closed issues to determine the next sequential cycle number

Cycle Number Continuity

The manager inspects both open and closed tracking issues to determine the highest cycle number for each prefix, then increments by one. This guarantees sequential numbering even when agents restart or experience failures.

Rules Enforced by the Manager

1. One issue per cycle — the previous cycle is closed before a new one opens.
2. Sequential cycle numbers — cycle counters persist across restarts.
3. Label enforcement — the Automation Tracking label is applied to every issue.
4. Cleanup before creation — stale tracking issues are never left behind.
5. Comment validation — updates only target the current, open tracking issue.

Integration Requirements

Agents MUST NOT:

• Create tracking issues directly via the Forgejo API
• Manually manage or increment cycle numbers
• Close tracking issues without going through the manager
• Comment on tracking issues directly

Instead, every agent invokes the automation-tracking-manager subagent for tracking operations. Example pseudocode:

TRACKING_RESULT=$(call_subagent automation-tracking-manager \
  --prefix "AUTO-IMP-POOL" \
  --type "Health Report" \
  --body "...")
NEW_ISSUE_NUMBER=$(echo "$TRACKING_RESULT" | jq -r '.issue_number')

The initial migration covered system-watchdog-pool-supervisor, implementation-pool-supervisor, timeline-update-pool-supervisor, project-owner-pool-supervisor, product-builder, and backlog-grooming-pool-supervisor, with subsequent agents added as the registry expanded.

Interval Reporting and Health Monitoring

Standardized Interval Declaration

All periodic tracking issues MUST include the expected reporting interval in their description using this standardized format:

**Reporting Interval**: <interval> (Next report expected: <timestamp>)

This enables automated health monitoring by the system-watchdog to detect stalled or crashed agents.

Agent Reporting Intervals

Agent	Issue Type	Interval	Example
implementation-pool-supervisor	Status Update	Every 5 cycles (~variable timing)	Every worker dispatch cycle
implementation-pool-supervisor	Health Report	Every 10 cycles (~variable timing)	Comprehensive pool health
backlog-grooming-pool-supervisor	Grooming Report	Every 5 minutes	Backlog maintenance cycle
backlog-grooming-pool-supervisor	Health Report	Every 50 minutes (10 cycles)	Grooming health status
human-liaison-pool-supervisor	Status Update	Every 20 minutes (10 cycles)	Human activity monitoring
system-watchdog-pool-supervisor	Health Report	Every 30 minutes (6 cycles)	System-wide health check
session-persister	Session State Tracking	Event-driven (variable)	After significant state changes
documentation-pool-supervisor / docs-writer	Documentation Report	Every 10 cycles (~3.3 hours)	Documentation monitoring cycle
product-builder	Build Session	Every 10 cycles (~variable timing)	Full build session status
architecture-pool-supervisor	Architecture Report	Every 10 cycles (~variable timing)	Architecture supervision status
architecture-guard-pool-supervisor	Guard Report	Every 10 cycles (~variable timing)	Codebase coherence check
timeline-update-pool-supervisor	Timeline Update	Per-commit (variable)	After timeline-worthy events
pr-review-pool-supervisor	Review Status	Every 10 cycles (~variable timing)	PR review pool status
pr-fix-pool-supervisor	PR Fix Status	Every 10 cycles (~variable timing)	PR CI fix pool status
pr-merge-pool-supervisor	PR Merge Status	Every 10 cycles (~variable timing)	Automated PR merge status
uat-test-pool-supervisor	UAT Status	Every 10 cycles (~variable timing)	UAT testing progress
project-owner-pool-supervisor	Project Status	Per-triage-cycle (variable)	After issue triage passes
agent-evolution-pool-supervisor	Evolution Report	Every 10 cycles (~variable timing)	Agent improvement proposals
epic-planner	Epic Planning Update	Every 10 cycles (~variable timing)	Workstream planning summary
bug-hunt-pool-supervisor	Bug Detection Status	Every 10 cycles (~variable timing)	Bug detection pool health
spec-update-pool-supervisor	Specification Update	Every 10 cycles (~variable timing)	Spec evolution status
test-infra-pool-supervisor	Infrastructure Report	Every 10 cycles (~variable timing)	Test infrastructure analysis

Automated Health Monitoring

The system-watchdog automatically monitors all automation tracking issues:

1. Scans tracking issues with "Automation Tracking" label every 5 minutes
2. Calculates staleness by comparing current time to issue creation time
3. Detects stalled agents when tracking issue is >20% overdue from expected interval
4. Triggers automated recovery actions:
   • Kills stalled agent sessions via OpenCode Server API (port 4096)
   • Performs root cause analysis by examining session messages and agent definitions
   • Creates high-priority diagnostic issues with findings and remediation suggestions
   • Closes stale tracking issues with recovery notes
   • Posts alerts for manual intervention where needed

Automated Recovery Process

When a stalled agent is detected:

1. Session Termination: All matching agent sessions are gracefully terminated
2. Root Cause Analysis: Automated analysis examines:
   • Recent session messages and tool call history
   • Related Forgejo issues and PR status
   • Agent definition and configuration
   • System resource availability
3. Diagnostic Issue Creation: Creates issue with format [AUTO-RECOVERY] <Agent> Agent Failure Analysis
4. Tracking Issue Cleanup: Closes stale tracking issue with recovery notes
5. Human Notification: Diagnostic issue requires manual review and restart

Recovery Issue Format

# Agent Failure Analysis — <Agent Name>

**Agent Prefix**: <prefix>
**Detection Time**: <ISO timestamp>
**Stale Issue**: #<issue_number>  
**Time Overdue**: <minutes> minutes
**Staleness Ratio**: <ratio>x expected interval

## Root Cause Analysis
[Automated analysis of session messages, Forgejo issues, and agent definition]

## Recovery Actions Taken
1. ✅ Session Termination: Killed <count> stalled sessions
2. ✅ Tracking Cleanup: Closed stale tracking issue  
3. ✅ Root Cause Analysis: Completed automated failure analysis
4. 🔄 Manual Intervention: Required per recommendations

## Recommended Next Steps
[Specific recommendations based on failure analysis]

Health Check Algorithm

# Pseudocode for health monitoring
for issue in automation_tracking_issues:
    if issue.title.matches("[AUTO-*] * (Cycle *)"):
        expected_interval = get_expected_interval(issue.agent_prefix, issue.type)
        time_since_creation = now() - issue.created_at
        staleness_threshold = expected_interval * 1.2  # 20% tolerance
        
        if time_since_creation > staleness_threshold:
            mark_agent_as_stalled(issue.agent_prefix)
            trigger_recovery_actions(issue.agent_prefix, issue)

Standardized Tracking Issue Templates

Common Header Format

All tracking issues MUST start with this standardized header:

# <Agent Name> <Issue Type> — <Timestamp>

**Agent**: <agent-name>
**Cycle**: <cycle-number>
**Reporting Interval**: <interval> (Next report expected: <next-timestamp>)
**Status**: <active|warning|error>

## Summary

<Brief status summary>

## Details

<Detailed information specific to agent type>

## Health Indicators

<Health metrics and status indicators>

## Next Actions

<Planned actions for next cycle>

---
**Automated by CleverAgents Bot**  
Supervisor: <supervisor-name> | Agent: <agent-name>

Template Variables

• <agent-name>: The agent type (e.g., "implementation-orchestrator", "backlog-groomer")
• <cycle-number>: Current cycle number for this agent
• <interval>: Expected time between reports (e.g., "5 minutes", "10 cycles")
• <next-timestamp>: When the next report is expected (ISO 8601 format)
• <status>: Current agent health status
  • active: Normal operation
  • warning: Non-critical issues detected
  • error: Critical issues requiring attention

Implementation Requirements

For All Agents

Every agent that creates tracking issues must implement:

1. Call the manager for every create, update, or close operation — no direct Forgejo API usage.
2. Provide standardized metadata (prefix, issue type, reporting interval) when invoking the manager so titles remain consistent.
3. Populate the body using the template, including summary, details, health indicators, and next actions.
4. Capture meaningful health signals (queues, failure counts, outstanding actions) that supervisors can act on.
5. Persist the returned issue number for subsequent updates during the same cycle.

Agent-Specific Implementation Details

session-persister

• Cycle Frequency: Every checkpoint (variable timing)
• Issue Type: Session State Tracking
• Content: Session state, active agents, configuration
• Cleanup: Handled by automation-tracking-manager

implementation-pool-supervisor

• Cycle Frequency: Status updates every 5 cycles; comprehensive health reports every 10 cycles
• Issue Types: Status Update, Health Report
• Content: Worker status, queue health, completion metrics
• Cleanup: Handled by automation-tracking-manager

system-watchdog-pool-supervisor

• Cycle Frequency: Every monitoring cycle (variable timing)
• Issue Type: System Health Report
• Content: Quality gates, system violations, corrective actions
• Cleanup: Handled by automation-tracking-manager

backlog-grooming-pool-supervisor

• Cycle Frequency: Grooming report every 5 minutes; scope alerts every 50 minutes (10 cycles)
• Issue Types: Grooming Report, Scope Alert
• Content: Issues processed, duplicates found, orphans discovered
• Cleanup: Handled by automation-tracking-manager

human-liaison-pool-supervisor

• Cycle Frequency: Status every 10 cycles
• Issue Type: Human Activity Report
• Content: Human interactions, triage decisions, outstanding items
• Cleanup: Handled by automation-tracking-manager

docs-writer

• Cycle Frequency: Documentation reports every 10 cycles (~3.3 hours)
• Issue Type: Documentation Report
• Content: Docs created/updated/skipped, commit hash, milestone coverage
• Cleanup: Handled by automation-tracking-manager

timeline-updater

• Cycle Frequency: Per-commit (variable timing)
• Issue Type: Timeline Update
• Content: Timeline changes, schedule adherence entries
• Cleanup: Handled by automation-tracking-manager

project-owner

• Cycle Frequency: Per-triage-cycle (variable timing)
• Issue Type: Project Triage Report
• Content: Issues triaged, MoSCoW assignments, priority decisions
• Cleanup: Handled by automation-tracking-manager

product-builder

• Cycle Frequency: Per-build-cycle (variable timing)
• Issue Type: Product Builder Status
• Content: Build orchestration status, worker pool health
• Cleanup: Handled by automation-tracking-manager

docs-writer

• Cycle Frequency: Documentation reports every 10 cycles (~3.3 hours)
• Issue Types: Documentation Report
• Content: Docs created/updated/skipped, commit hash, milestone coverage
• Cleanup: Deletes previous documentation report issues

Searching and Filtering

Finding Tracking Issues

All automation tracking:

label:"Automation Tracking"

Specific agent tracking:

label:"Automation Tracking" [AUTO-SESSION] in:title
label:"Automation Tracking" [AUTO-IMP-POOL] in:title
label:"Automation Tracking" [AUTO-WATCHDOG] in:title
label:"Automation Tracking" [AUTO-GROOMER] in:title
label:"Automation Tracking" [AUTO-LIAISON] in:title
label:"Automation Tracking" [AUTO-PROD-BLDR] in:title
label:"Automation Tracking" [AUTO-ARCH] in:title
label:"Automation Tracking" [AUTO-TIME] in:title
label:"Automation Tracking" [AUTO-DOCS] in:title
label:"Automation Tracking" [AUTO-GUARD] in:title
label:"Automation Tracking" [AUTO-REV-POOL] in:title
label:"Automation Tracking" [AUTO-FIX-POOL] in:title
label:"Automation Tracking" [AUTO-MERGE] in:title
label:"Automation Tracking" [AUTO-UAT-POOL] in:title
label:"Automation Tracking" [AUTO-PROJ-OWN] in:title
label:"Automation Tracking" [AUTO-EVLV] in:title
label:"Automation Tracking" [AUTO-EPIC] in:title
label:"Automation Tracking" [AUTO-BUG-SUP] in:title
label:"Automation Tracking" [AUTO-SPEC] in:title
label:"Automation Tracking" [AUTO-INF-POOL] in:title

Recent tracking issues (last 24 hours):

label:"Automation Tracking" created:>2024-01-01T00:00:00Z

Announcement issues only:

label:"Automation Tracking" "Announce:" in:title

Cross-Agent Announcements

The automation tracking system supports emergency announcements and important messages that need to be visible to all agents.

Announcement Operations

The automation-tracking-manager subagent supports the following announcement operations:

Operation	Description
CREATE_ANNOUNCEMENT_ISSUE	Create a new announcement with priority support
CLOSE_ANNOUNCEMENT_ISSUE	Close an announcement when it is no longer relevant
LIST_TRACKING_ISSUES	List all current tracking and announcement issues
READ_ANNOUNCEMENTS	Read announcements from other agents (cross-agent awareness)
REVIEW_OWN_ANNOUNCEMENTS	Review and clean up own announcements

Announcement Lifecycle

1. Supervisors read critical announcements before each cycle
2. Workers read announcements from system agents and orchestrators
3. Priority-based filtering reduces noise (Critical/High/Medium/Low)
4. Backlog-groomer performs intelligent cleanup with age thresholds by priority:
   • Critical: 72 hours
   • High: 48 hours
   • Medium: 24 hours
   • Low: 12 hours

Announcement Title Format

[AUTO-<AGENT_PREFIX>] Announce: <MESSAGE_SUMMARY>

Example: [AUTO-WATCHDOG] Announce: CRITICAL: CI pipeline down — all merges blocked

Cleanup Protocol

Primary Cleanup (automation-tracking-manager)

The automation-tracking-manager subagent handles all primary cleanup:

• Closes the previous cycle's tracking issue before creating a new one
• Preserves announcement issues (different title format — no (Cycle N) suffix)
• Runs cleanup atomically as part of every CREATE_TRACKING_ISSUE operation

Secondary Cleanup (backlog-groomer)

The backlog-groomer acts as a backup cleanup agent:

• Scans for stale automation tracking issues older than 7 days
• Closes abandoned tracking issues from agents that may have crashed
• Preserves announcement issues and recently created tracking issues
• Reports cleanup activity in its own tracking issues

Emergency Cleanup (Manual)

If tracking issues accumulate excessively:

1. Use GitHub/Forgejo bulk operations with label filter
2. Search: label:"Automation Tracking" created:<7-days-ago
3. Bulk close/delete stale tracking issues
4. Preserve announcement issues

Troubleshooting

Common Issues

"Automation Tracking" label missing:

• Ensure label exists in repository
• Check agent permission to add labels
• Verify label name spelling (case sensitive)

Tracking issues not being cleaned up:

• Check if agent cleanup function is running
• Verify backlog-groomer is active and functioning
• Check for agent crashes or hangs preventing cleanup

Too many tracking issues:

• May indicate cleanup functions not working
• Check agent logs for cleanup errors
• Run manual cleanup if necessary

Cannot find specific agent tracking:

• Verify agent prefix in search
• Check if agent is actually running and creating issues
• Confirm agent is using correct title format

Agents posting to old tracking issues (cycle reuse):

• Agent is not using the centralized automation-tracking-manager
• Migrate agent to use the centralized manager subagent
• Check that previous cycle cleanup runs before new issue creation

Debugging Steps

1. Check agent status: Verify agent is running and healthy
2. Verify title format: Ensure exact compliance with format standards
3. Check labels: Confirm "Automation Tracking" label is applied
4. Review cleanup logs: Look for cleanup function errors
5. Manual search: Use repository issue search to verify tracking issues

Migration Notes

Phase 1 — Individual Tracking Issues (completed)

Replaced the previous shared session state issue approach:

• Before: All agents posted comments to single issue #[session-state-issue-number]
• After: Each agent creates individual tracking issues with standardized titles
• Benefit: Eliminates noise, improves searchability, enables targeted cleanup

Phase 2 — Centralized Tracking Manager (completed)

Replaced direct Forgejo API calls in each agent with the automation-tracking-manager subagent:

• Before: Each agent implemented its own tracking issue creation, cleanup, and cycle management
• After: All agents delegate to automation-tracking-manager as the single source of truth
• Benefit: Eliminates duplicate issues, ensures cycle continuity across restarts, prevents agents from commenting on stale issues
• Breaking change: shared/automation_tracking.md was removed; agents now use the subagent pattern

The migration updated these agents: system-watchdog, implementation-orchestrator, timeline-updater, project-owner, product-builder, backlog-groomer.

A second migration introduced the centralized automation-tracking-manager subagent to prevent cycle reuse issues where agents were commenting on old tracking issues instead of creating new ones.

Related Documentation

• System Watchdog - Specific documentation for watchdog agent
• Documentation Writer - Documentation writer agent reference
• Quality Automation - Quality gate automation details
• Ops Runbook - General operational procedures