HAL9000/cleveragents-core
0
0
Fork 2
forked from cleveragents/cleveragents-core
Code Pull Requests 3 Activity

feat: complete comprehensive worker tracking system implementation

Browse Source 
- Enhanced product-builder with detailed session monitoring via OpenCode API
- Added comprehensive worker status reporting with session details, targets, and activity
- Enhanced implementation-orchestrator with detailed worker tracking and health monitoring
- Enhanced continuous-pr-reviewer with detailed worker status and progress reporting
- Enhanced uat-tester with detailed worker monitoring and testing progress
- All supervisors now provide detailed visibility into worker activities and health
- Added actual cycle time calculation using timestamps throughout all agents
- Implemented proper tracking issue lifecycle (delete previous, create new each cycle)
- Added stale worker detection and restart functionality across all pool supervisors
- Ensured redundant monitoring between product-builder and individual supervisors

This completes the comprehensive worker tracking system providing full visibility
into all 16 supervisors and their workers with detailed status reporting, automatic
restarts, and actual timing data.
This commit is contained in:
Jeffrey Phillips Freeman
2026-04-09 02:44:00 +00:00
parent 30b457b090
commit 50096391b5
4 changed files with 488 additions and 144 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.opencode/agents/continuous-pr-reviewer.md
+87 -22
View File
@@ -487,54 +487,116 @@ function calculate_review_priority(pr, reason):
return priority
function post_health_signal():
local next_health_time=$(date -d "+60 minutes" -Iseconds) # Every 10 cycles * 30s + margin
local tracking_body="# PR Review Pool Health Report — $(date +'%Y-%m-%d %H:%M:%S')
# Calculate actual cycle time
local current_timestamp=$(date +%s)
local cycle_time_display="60 minutes (estimated)"
if [[ -n "$LAST_TRACKING_TIMESTAMP" ]]; then
local elapsed_seconds=$((current_timestamp - LAST_TRACKING_TIMESTAMP))
local cycle_time_minutes=$((elapsed_seconds / 60))
cycle_time_display="${cycle_time_minutes} minutes"
fi
# Get detailed worker information from OpenCode API
local SERVER="http://localhost:4096"
local detailed_reviewers=""
# Query each active reviewer session for detailed status
for pr_num in "${!active_reviews[@]}"; do
local session_id="${active_reviews[$pr_num][session_id]}"
local focus_areas="${active_reviews[$pr_num][review_focus]}"
local dispatched_at="${active_reviews[$pr_num][dispatched_at]}"
if [[ -n "$session_id" ]]; then
# Get session status
local session_status=$(curl -s "${SERVER}/session/${session_id}" | jq -r '.status // "unknown"' 2>/dev/null)
# Get recent messages to understand current review progress
local recent_messages=$(curl -s "${SERVER}/session/${session_id}/messages?limit=3" | jq -r '.[-1].content // "No recent activity"' 2>/dev/null)
local last_activity=$(curl -s "${SERVER}/session/${session_id}/messages?limit=1" | jq -r '.[-1].timestamp // "unknown"' 2>/dev/null)
# Calculate time since last activity
local activity_display="unknown"
if [[ "$last_activity" != "unknown" ]]; then
local last_activity_timestamp=$(date -d "$last_activity" +%s 2>/dev/null || echo "0")
local current_time=$(date +%s)
local minutes_since_activity=$(( (current_time - last_activity_timestamp) / 60 ))
activity_display="${minutes_since_activity}m ago"
fi
# Calculate duration since assignment
local duration="unknown"
if [[ "$dispatched_at" != "unknown" ]]; then
local start_timestamp=$(date -d "$dispatched_at" +%s 2>/dev/null || echo "0")
local duration_minutes=$(( (current_time - start_timestamp) / 60 ))
if [[ $duration_minutes -lt 60 ]]; then
duration="${duration_minutes}m"
else
duration="$((duration_minutes/60))h $((duration_minutes%60))m"
fi
fi
# Extract work summary from recent message (first 100 chars)
local work_summary=$(echo "$recent_messages" | head -c 100 | tr '\n' ' ')
if [[ ${#work_summary} -eq 100 ]]; then
work_summary="${work_summary}..."
fi
detailed_reviewers+="| #$pr_num | $session_id | $session_status | $focus_areas | $duration | $activity_display | $work_summary |\n"
fi
done
if [[ -z "$detailed_reviewers" ]]; then
detailed_reviewers="| - | - | - | - | - | - | No active reviewers |\n"
fi
local tracking_body="# PR Review Pool Status — $(date +'%Y-%m-%d %H:%M:%S')
**Agent**: continuous-pr-reviewer
**Cycle**: $cycle
**Reporting Interval**: 60 minutes (Next report expected: $next_health_time)
**Cycle Time**: $cycle_time_display
**Reporting Interval**: Every 10 cycles (~60 minutes)
**Status**: active
## Summary
Review pool managing ${#active_reviews[@]} active reviewers with ${#prs_needing_review[@]} PRs in queue and ${idle_cycles} idle cycles.
## Details
## Detailed Reviewer Status
**Active Reviewers**: ${#active_reviews[@]}/$N
| PR | Session ID | Status | Focus Areas | Duration | Last Activity | Recent Thinking |
|----|------------|--------|-------------|----------|---------------|-----------------|
$detailed_reviewers
## Pool Health
**Pool Status**: Active - managing PR review workload
**Active Reviewers**: ${#active_reviews[@]} / $N
**PRs Needing Review**: ${#prs_needing_review[@]} PRs queued
**Recently Reviewed**: ${#recently_reviewed[@]} PRs tracked
**Reviewer Utilization**: ${#active_reviews[@]}/$N ($(( ${#active_reviews[@]} * 100 / N ))%)
### Active Reviews
### Queue Status
| PR | Session | Focus Areas | Duration |
|----|---------|-------------|----------|
$(for pr_num in "${!active_reviews[@]}"; do
local session="${active_reviews[$pr_num][session_id]:0:8}..."
local focus="${active_reviews[$pr_num][review_focus]}"
local duration="$(( ($(date +%s) - $(date -d "${active_reviews[$pr_num][dispatched_at]}" +%s)) / 60 ))min"
echo "| #$pr_num | $session | $focus | $duration |"
done)
### Recent Activity
- **Last Action**: $(date +'%Y-%m-%d %H:%M:%S')
- **Idle Cycles**: $idle_cycles
- **Review Throughput**: ${#recently_reviewed[@]} PRs reviewed total
**PRs Pending Review**: ${#prs_needing_review[@]}
$(for pr in "${prs_needing_review[@]}"; do
echo "- PR #$pr (priority: $(calculate_review_priority $pr))"
done | head -5)
## Health Indicators
- **Reviewer Utilization**: ${#active_reviews[@]}/$N ($(( ${#active_reviews[@]} * 100 / N ))%)
- **Queue Health**: ${#prs_needing_review[@]} PRs pending review
- **System Status**: Operational and responsive
- **Idle Cycles**: $idle_cycles
- **Stale Reviewers**: $(echo -e "$detailed_reviewers" | grep -c "unknown\|[3-9][0-9]m ago\|[0-9][0-9][0-9]m ago") (inactive >30min)
## Next Actions
- Continue monitoring PR queue for review opportunities
- Dispatch reviewers to ${#prs_needing_review[@]} pending PRs
- Maintain focus area rotation for comprehensive reviews
- Next health report in ~10 cycles (60 minutes)
- Check for stale reviewers and restart if needed
- Next status update in ~10 cycles
## Inter-Agent Coordination
@@ -549,6 +611,9 @@ Supervisor: PR Review Pool | Agent: continuous-pr-reviewer"
cleanup_previous_reviewer_tracking
create_reviewer_tracking_issue $cycle "$tracking_body"
# Store timestamp for next cycle time calculation
export LAST_TRACKING_TIMESTAMP="$current_timestamp"
}
```
.opencode/agents/implementation-orchestrator.md
+106 -11
View File
@@ -329,6 +329,98 @@ if pr_work_queue:
cycle_time_display = "5 minutes (estimated)"
self.last_tracking_timestamp = current_timestamp
# Import required modules for API calls
import json
from datetime import timezone
# Get detailed worker information from OpenCode API
SERVER = "http://localhost:4096"
detailed_workers = []
# Query each active worker session for detailed status
all_worker_sessions = {**active_pr_workers, **active_issue_workers}
for work_id, session_info in all_worker_sessions.items():
session_id = session_info.get('session_id')
work_type = session_info.get('work_type', 'unknown')
assigned_at = session_info.get('assigned_at', 'unknown')
if session_id:
try:
# Get session status
session_status_cmd = f"curl -s {SERVER}/session/{session_id}"
session_status_result = bash(session_status_cmd, timeout=10000)
session_status = "unknown"
if session_status_result:
session_data = json.loads(session_status_result)
session_status = session_data.get('status', 'unknown')
# Get recent messages to understand current work
messages_cmd = f"curl -s {SERVER}/session/{session_id}/messages?limit=3"
messages_result = bash(messages_cmd, timeout=10000)
recent_thinking = "No recent activity"
last_check_time = "unknown"
if messages_result:
messages = json.loads(messages_result)
if messages and len(messages) > 0:
last_message = messages[-1]
recent_thinking = last_message.get('content', '')[:150] + "..." if len(last_message.get('content', '')) > 150 else last_message.get('content', '')
last_check_time = last_message.get('timestamp', 'unknown')
# Calculate time since last activity
if last_check_time != 'unknown':
try:
last_time = datetime.fromisoformat(last_check_time.replace('Z', '+00:00'))
time_diff = datetime.now(timezone.utc) - last_time
minutes_ago = int(time_diff.total_seconds() / 60)
last_check_time = f"{minutes_ago}m ago"
except:
last_check_time = "unknown"
# Calculate duration since assignment
duration = "unknown"
if assigned_at != 'unknown':
try:
start_time = datetime.fromisoformat(assigned_at.replace('Z', '+00:00'))
duration_diff = datetime.now(timezone.utc) - start_time
duration_minutes = int(duration_diff.total_seconds() / 60)
if duration_minutes < 60:
duration = f"{duration_minutes}m"
else:
duration = f"{duration_minutes//60}h {duration_minutes%60}m"
except:
duration = "unknown"
detailed_workers.append({
'session_id': session_id,
'work_type': work_type,
'target': f"#{work_id}" if work_type in ['issue', 'pr'] else str(work_id),
'status': session_status,
'duration': duration,
'last_check': last_check_time,
'recent_thinking': recent_thinking
})
except Exception as e:
# Fallback for failed API calls
detailed_workers.append({
'session_id': session_id or 'unknown',
'work_type': work_type,
'target': f"#{work_id}",
'status': 'unknown',
'duration': 'unknown',
'last_check': 'unknown',
'recent_thinking': f'API error: {str(e)[:50]}'
})
# Build detailed worker table
worker_table_rows = ""
for worker in detailed_workers:
worker_table_rows += f"| {worker['session_id']} | {worker['work_type']} | {worker['target']} | {worker['status']} | {worker['duration']} | {worker['last_check']} | {worker['recent_thinking']} |\n"
if not detailed_workers:
worker_table_rows = "| - | - | - | - | - | - | No active workers |\n"
tracking_body = f"""# Implementation Pool Status — {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}
**Agent**: implementation-orchestrator
@@ -341,19 +433,21 @@ if pr_work_queue:
Pool managing {len(active_pr_workers)} PR fixes and {len(active_issue_workers)} issue implementations with {len(pr_work_queue) + len(issue_work_queue)} items queued.
## Details
## Detailed Worker Status
**Active Workers**: {len(detailed_workers)}/{os.getenv('CA_MAX_PARALLEL_WORKERS', 4)}
| Session ID | Type | Target | Status | Duration | Last Check | Recent Thinking |
|------------|------|--------|--------|----------|------------|-----------------|
{worker_table_rows}
## Pool Health
**Pool Status**: {pool_status}
**Active Workers**: {len(active_pr_workers)} PR fixes, {len(active_issue_workers)} issue implementations
**Queue Status**: {len(pr_work_queue)} PRs, {len(issue_work_queue)} issues pending
**Success Rate**: {success_count}/{total_workers} workers succeeded (last 10 workers)
**Max Workers**: {os.getenv('CA_MAX_PARALLEL_WORKERS', 4)}
### Worker Overview
| Worker | Type | Target | Status | Branch | Duration |
|--------|------|--------|--------|--------|----------|
{worker_table_rows}
**Worker Utilization**: {int(len(detailed_workers)/max(int(os.getenv('CA_MAX_PARALLEL_WORKERS', 4)),1)*100)}%
### Queue Status
@@ -367,14 +461,15 @@ Pool managing {len(active_pr_workers)} PR fixes and {len(active_issue_workers)}
- **Worker Success Rate**: {success_count}/{total_workers} ({int(success_count/max(total_workers,1)*100)}%)
- **Queue Health**: {len(pr_work_queue) + len(issue_work_queue)} items pending
- **Active Workers**: {len(active_pr_workers + active_issue_workers)}/{os.getenv('CA_MAX_PARALLEL_WORKERS', 4)}
- **Worker Utilization**: {int(len(active_pr_workers + active_issue_workers)/max(int(os.getenv('CA_MAX_PARALLEL_WORKERS', 4)),1)*100)}%
- **Active Workers**: {len(detailed_workers)}/{os.getenv('CA_MAX_PARALLEL_WORKERS', 4)}
- **Stale Workers**: {len([w for w in detailed_workers if 'unknown' in w['last_check'] or ('m ago' in w['last_check'] and int(w['last_check'].split('m')[0]) > 15)])} (inactive >15min)
## Next Actions
- Continue monitoring {len(active_pr_workers + active_issue_workers)} active workers
- Continue monitoring {len(detailed_workers)} active workers
- Process {len(pr_work_queue) + len(issue_work_queue)} queued items
- Maintain PR-first priority
- Check for stale workers and restart if needed
- Next status update in ~5 cycles
---
.opencode/agents/product-builder.md
+187 -91
View File
@@ -1177,114 +1177,204 @@ Supervisor: Product Builder | Agent: product-builder"
cycle_time_display="10 minutes (estimated)"
fi
# Count supervisors by checking sessions
supervisor_counts = {}
worker_counts = {}
# Get detailed session information
echo "[MONITOR] Gathering detailed session information..."
ALL_SESSIONS_DETAILED=$(bash("curl -s ${SERVER}/session", timeout=30000))
SESSION_STATUS_DATA=$(bash("curl -s ${SERVER}/session/status", timeout=30000))
# Parse ALL_SESSIONS to count supervisors and workers
# Pool supervisors
supervisor_counts["implementor-pool"] = $(echo "${ALL_SESSIONS}" | grep -c '\[AUTO-IMP-SUP\]' || echo 0)
supervisor_counts["reviewer-pool"] = $(echo "${ALL_SESSIONS}" | grep -c '\[AUTO-REV-SUP\]' || echo 0)
supervisor_counts["tester-pool"] = $(echo "${ALL_SESSIONS}" | grep -c '\[AUTO-UAT-SUP\]' || echo 0)
supervisor_counts["hunter-pool"] = $(echo "${ALL_SESSIONS}" | grep -c '\[AUTO-BUG-SUP\]' || echo 0)
supervisor_counts["test-infra-pool"] = $(echo "${ALL_SESSIONS}" | grep -c '\[AUTO-INF-SUP\]' || echo 0)
# Parse sessions and build detailed worker information
worker_details = {}
supervisor_details = {}
# Process each session to get detailed information
echo "$ALL_SESSIONS_DETAILED" | jq -c '.[]' | while read -r session; do
session_id=$(echo "$session" | jq -r '.id')
session_title=$(echo "$session" | jq -r '.title')
session_created=$(echo "$session" | jq -r '.created_at')
# Get session status
session_status=$(echo "$SESSION_STATUS_DATA" | jq -r --arg id "$session_id" '.[] | select(.id == $id) | .status // "unknown"')
# Get recent messages to understand what the session is doing
session_messages=$(bash("curl -s ${SERVER}/session/${session_id}/messages?limit=5", timeout=15000))
last_message=$(echo "$session_messages" | jq -r '.[-1].content // "No messages"' 2>/dev/null)
last_activity=$(echo "$session_messages" | jq -r '.[-1].timestamp // "unknown"' 2>/dev/null)
# Calculate time since last activity
if [[ "$last_activity" != "unknown" ]]; then
last_activity_timestamp=$(date -d "$last_activity" +%s 2>/dev/null || echo "0")
current_time=$(date +%s)
minutes_since_activity=$(( (current_time - last_activity_timestamp) / 60 ))
activity_display="${minutes_since_activity}m ago"
else
activity_display="unknown"
fi
# Extract work summary from last message (first 100 chars)
work_summary=$(echo "$last_message" | head -c 100 | tr '\n' ' ')
if [[ ${#work_summary} -eq 100 ]]; then
work_summary="${work_summary}..."
fi
# Categorize session by title prefix
if [[ "$session_title" =~ \[AUTO-IMP-SUP\] ]]; then
supervisor_details["implementor-pool"]="$session_id|$session_status|$activity_display|$work_summary"
elif [[ "$session_title" =~ \[AUTO-REV-SUP\] ]]; then
supervisor_details["reviewer-pool"]="$session_id|$session_status|$activity_display|$work_summary"
elif [[ "$session_title" =~ \[AUTO-UAT-SUP\] ]]; then
supervisor_details["tester-pool"]="$session_id|$session_status|$activity_display|$work_summary"
elif [[ "$session_title" =~ \[AUTO-BUG-SUP\] ]]; then
supervisor_details["hunter-pool"]="$session_id|$session_status|$activity_display|$work_summary"
elif [[ "$session_title" =~ \[AUTO-INF-SUP\] ]]; then
supervisor_details["test-infra-pool"]="$session_id|$session_status|$activity_display|$work_summary"
elif [[ "$session_title" =~ \[AUTO-ARCH\] ]]; then
supervisor_details["architect"]="$session_id|$session_status|$activity_display|$work_summary"
elif [[ "$session_title" =~ \[AUTO-EPIC\] ]]; then
supervisor_details["epic-planner"]="$session_id|$session_status|$activity_display|$work_summary"
elif [[ "$session_title" =~ \[AUTO-HUMAN\] ]]; then
supervisor_details["human-liaison"]="$session_id|$session_status|$activity_display|$work_summary"
elif [[ "$session_title" =~ \[AUTO-EVLV\] ]]; then
supervisor_details["agent-evolver"]="$session_id|$session_status|$activity_display|$work_summary"
elif [[ "$session_title" =~ \[AUTO-GUARD\] ]]; then
supervisor_details["arch-guard"]="$session_id|$session_status|$activity_display|$work_summary"
elif [[ "$session_title" =~ \[AUTO-SPEC\] ]]; then
supervisor_details["spec-updater"]="$session_id|$session_status|$activity_display|$work_summary"
elif [[ "$session_title" =~ \[AUTO-BLOG\] ]]; then
supervisor_details["backlog-groomer"]="$session_id|$session_status|$activity_display|$work_summary"
elif [[ "$session_title" =~ \[AUTO-DOCS\] ]]; then
supervisor_details["docs-writer"]="$session_id|$session_status|$activity_display|$work_summary"
elif [[ "$session_title" =~ \[AUTO-TIME\] ]]; then
supervisor_details["timeline-updater"]="$session_id|$session_status|$activity_display|$work_summary"
elif [[ "$session_title" =~ \[AUTO-OWNR\] ]]; then
supervisor_details["project-owner"]="$session_id|$session_status|$activity_display|$work_summary"
elif [[ "$session_title" =~ \[AUTO-WDOG\] ]]; then
supervisor_details["system-watchdog"]="$session_id|$session_status|$activity_display|$work_summary"
# Worker sessions
elif [[ "$session_title" =~ \[AUTO-IMP\] ]]; then
# Extract issue/PR number from title
work_target=$(echo "$session_title" | grep -o '#[0-9]\+' | head -1)
worker_details["implementor"]="${worker_details["implementor"]}|$session_id:$session_status:$work_target:$activity_display:$work_summary"
elif [[ "$session_title" =~ \[AUTO-REV\] ]]; then
work_target=$(echo "$session_title" | grep -o '#[0-9]\+' | head -1)
worker_details["reviewer"]="${worker_details["reviewer"]}|$session_id:$session_status:$work_target:$activity_display:$work_summary"
elif [[ "$session_title" =~ \[AUTO-UAT\] ]]; then
work_target=$(echo "$session_title" | grep -o 'Feature [^]]*' | head -1)
worker_details["tester"]="${worker_details["tester"]}|$session_id:$session_status:$work_target:$activity_display:$work_summary"
elif [[ "$session_title" =~ \[AUTO-BUG\] ]]; then
work_target=$(echo "$session_title" | grep -o 'Module [^]]*' | head -1)
worker_details["hunter"]="${worker_details["hunter"]}|$session_id:$session_status:$work_target:$activity_display:$work_summary"
elif [[ "$session_title" =~ \[AUTO-INF\] ]]; then
work_target=$(echo "$session_title" | grep -o 'Analysis [^]]*' | head -1)
worker_details["infra"]="${worker_details["infra"]}|$session_id:$session_status:$work_target:$activity_display:$work_summary"
fi
done
# Count active supervisors and workers
supervisor_count=0
total_workers=0
# Build detailed supervisor status table
supervisor_status="## Supervisor Status (16 Total)\n\n"
supervisor_status+="| Supervisor | Status | Session ID | Last Activity | Current Work |\n"
supervisor_status+="|------------|--------|------------|---------------|---------------|\n"
# Pool supervisors with worker details
for pool in "implementor-pool" "reviewer-pool" "tester-pool" "hunter-pool" "test-infra-pool"; do
if [[ -n "${supervisor_details[$pool]}" ]]; then
IFS='|' read -r sess_id status activity work <<< "${supervisor_details[$pool]}"
supervisor_status+="| $pool | ✓ $status | $sess_id | $activity | $work |\n"
supervisor_count=$((supervisor_count + 1))
else
supervisor_status+="| $pool | ❌ MISSING | - | - | Not running |\n"
fi
done
# Singleton supervisors
supervisor_counts["architect"] = $(echo "${ALL_SESSIONS}" | grep -c '\[AUTO-ARCH\]' || echo 0)
supervisor_counts["epic-planner"] = $(echo "${ALL_SESSIONS}" | grep -c '\[AUTO-EPIC\]' || echo 0)
supervisor_counts["human-liaison"] = $(echo "${ALL_SESSIONS}" | grep -c '\[AUTO-HUMAN\]' || echo 0)
supervisor_counts["agent-evolver"] = $(echo "${ALL_SESSIONS}" | grep -c '\[AUTO-EVLV\]' || echo 0)
supervisor_counts["arch-guard"] = $(echo "${ALL_SESSIONS}" | grep -c '\[AUTO-GUARD\]' || echo 0)
supervisor_counts["spec-updater"] = $(echo "${ALL_SESSIONS}" | grep -c '\[AUTO-SPEC\]' || echo 0)
supervisor_counts["backlog-groomer"] = $(echo "${ALL_SESSIONS}" | grep -c '\[AUTO-BLOG\]' || echo 0)
supervisor_counts["docs-writer"] = $(echo "${ALL_SESSIONS}" | grep -c '\[AUTO-DOCS\]' || echo 0)
supervisor_counts["timeline-updater"] = $(echo "${ALL_SESSIONS}" | grep -c '\[AUTO-TIME\]' || echo 0)
supervisor_counts["project-owner"] = $(echo "${ALL_SESSIONS}" | grep -c '\[AUTO-OWNR\]' || echo 0)
supervisor_counts["system-watchdog"] = $(echo "${ALL_SESSIONS}" | grep -c '\[AUTO-WDOG\]' || echo 0)
for singleton in "architect" "epic-planner" "human-liaison" "agent-evolver" "arch-guard" "spec-updater" "backlog-groomer" "docs-writer" "timeline-updater" "project-owner" "system-watchdog"; do
if [[ -n "${supervisor_details[$singleton]}" ]]; then
IFS='|' read -r sess_id status activity work <<< "${supervisor_details[$singleton]}"
supervisor_status+="| $singleton | ✓ $status | $sess_id | $activity | $work |\n"
supervisor_count=$((supervisor_count + 1))
else
supervisor_status+="| $singleton | ❌ MISSING | - | - | Not running |\n"
fi
done
# Count workers
worker_counts["implementor"] = $(echo "${ALL_SESSIONS}" | grep -c '\[AUTO-IMP\]' || echo 0)
worker_counts["reviewer"] = $(echo "${ALL_SESSIONS}" | grep -c '\[AUTO-REV\]' || echo 0)
worker_counts["tester"] = $(echo "${ALL_SESSIONS}" | grep -c '\[AUTO-UAT\]' || echo 0)
worker_counts["hunter"] = $(echo "${ALL_SESSIONS}" | grep -c '\[AUTO-BUG\]' || echo 0)
worker_counts["infra"] = $(echo "${ALL_SESSIONS}" | grep -c '\[AUTO-INF\]' || echo 0)
# Build detailed worker status tables
worker_status=""
# Build supervisor status table
supervisor_status = "## Supervisor Status (16 Total)\n\n"
supervisor_status += "| Supervisor | Expected | Actual | Status | Workers |\n"
supervisor_status += "|------------|----------|--------|--------|----------|\n"
# Pool supervisors
supervisor_status += "| Implementation Pool | 1 | ${supervisor_counts['implementor-pool']} | "
supervisor_status += "${supervisor_counts['implementor-pool'] == 1 ? '✓' : '❌ MISSING'} | "
supervisor_status += "${worker_counts['implementor']}/${N_FULL} |\n"
supervisor_status += "| PR Review Pool | 1 | ${supervisor_counts['reviewer-pool']} | "
supervisor_status += "${supervisor_counts['reviewer-pool'] == 1 ? '✓' : '❌ MISSING'} | "
supervisor_status += "${worker_counts['reviewer']}/${N_HALF} |\n"
supervisor_status += "| UAT Tester Pool | 1 | ${supervisor_counts['tester-pool']} | "
supervisor_status += "${supervisor_counts['tester-pool'] == 1 ? '✓' : '❌ MISSING'} | "
supervisor_status += "${worker_counts['tester']}/${N_QUARTER} |\n"
supervisor_status += "| Bug Hunter Pool | 1 | ${supervisor_counts['hunter-pool']} | "
supervisor_status += "${supervisor_counts['hunter-pool'] == 1 ? '✓' : '❌ MISSING'} | "
supervisor_status += "${worker_counts['hunter']}/${N_QUARTER} |\n"
supervisor_status += "| Test Infra Pool | 1 | ${supervisor_counts['test-infra-pool']} | "
supervisor_status += "${supervisor_counts['test-infra-pool'] == 1 ? '✓' : '❌ MISSING'} | "
supervisor_status += "${worker_counts['infra']}/${N_QUARTER} |\n"
# Singleton supervisors
for singleton in ["architect", "epic-planner", "human-liaison", "agent-evolver",
"arch-guard", "spec-updater", "backlog-groomer", "docs-writer",
"timeline-updater", "project-owner", "system-watchdog"]:
supervisor_status += "| ${singleton} | 1 | ${supervisor_counts[singleton]} | "
supervisor_status += "${supervisor_counts[singleton] == 1 ? '✓' : '❌ MISSING'} | N/A |\n"
# Count total active supervisors
total_active = sum(supervisor_counts.values())
for pool_type in "implementor" "reviewer" "tester" "hunter" "infra"; do
case $pool_type in
"implementor") max_workers=$N_FULL; pool_name="Implementation Pool" ;;
"reviewer") max_workers=$N_HALF; pool_name="PR Review Pool" ;;
"tester") max_workers=$N_QUARTER; pool_name="UAT Testing Pool" ;;
"hunter") max_workers=$N_QUARTER; pool_name="Bug Hunter Pool" ;;
"infra") max_workers=$N_QUARTER; pool_name="Test Infrastructure Pool" ;;
esac
# Count and parse workers for this pool
worker_list="${worker_details[$pool_type]}"
if [[ -n "$worker_list" ]]; then
# Count workers (each worker is separated by |, first char is |)
worker_count=$(echo "$worker_list" | tr '|' '\n' | grep -c ':')
total_workers=$((total_workers + worker_count))
else
worker_count=0
fi
worker_status+="\n### $pool_name Workers ($worker_count/$max_workers active)\n\n"
if [[ $worker_count -gt 0 ]]; then
worker_status+="| Session ID | Status | Working On | Last Activity | Recent Thinking |\n"
worker_status+="|------------|--------|------------|---------------|------------------|\n"
# Parse each worker
echo "$worker_list" | tr '|' '\n' | grep ':' | while IFS=':' read -r sess_id status target activity thinking; do
worker_status+="| $sess_id | $status | $target | $activity | $thinking |\n"
done
else
worker_status+="*No active workers*\n"
fi
done
# Clean up previous tracking issue and create new one
cleanup_previous_product_builder_tracking()
# Increment cycle counter
cycle += 1
cycle=$((cycle + 1))
# Build tracking issue body
tracking_body = "# Product Builder Status — $(date +'%Y-%m-%d %H:%M:%S')
# Build comprehensive tracking issue body
tracking_body="# Product Builder Status — $(date +'%Y-%m-%d %H:%M:%S')
**Agent**: product-builder
**Cycle**: ${cycle}
**Cycle Time**: ${cycle_time_display}
**Cycle**: $cycle
**Cycle Time**: $cycle_time_display
**Reporting Interval**: Every 10 heartbeats (~10 minutes)
**Status**: ${total_active == 16 ? 'Healthy - All supervisors active' : 'UNHEALTHY - Missing supervisors'}
**Status**: $([ $supervisor_count -eq 16 ] && echo "Healthy - All supervisors active" || echo "UNHEALTHY - Missing supervisors")
## Summary
Managing ${total_active}/16 supervisors with ${sum(worker_counts.values())} total workers across all pools.
Managing $supervisor_count/16 supervisors with $total_workers total workers across all pools.
${supervisor_status}
$supervisor_status
## Detailed Worker Status
$worker_status
## Health Indicators
- **Supervisors Active**: ${total_active}/16 ${total_active < 16 ? '⚠️ MISSING SUPERVISORS' : '✓'}
- **Supervisors Relaunched**: ${supervisors_relaunched} (since session start)
- **Heartbeat Count**: ${heartbeat_count}
- **Supervisors Active**: $supervisor_count/16 $([ $supervisor_count -lt 16 ] && echo "⚠️ MISSING SUPERVISORS" || echo "✓")
- **Total Workers**: $total_workers
- **Supervisors Relaunched**: $supervisors_relaunched (since session start)
- **Heartbeat Count**: $heartbeat_count
- **Session Duration**: $((heartbeat_count * 60 / 3600)) hours
## Worker Pool Summary
- **Implementation**: ${worker_counts['implementor']}/${N_FULL} workers
- **PR Review**: ${worker_counts['reviewer']}/${N_HALF} workers
- **UAT Testing**: ${worker_counts['tester']}/${N_QUARTER} workers
- **Bug Hunting**: ${worker_counts['hunter']}/${N_QUARTER} workers
- **Test Infrastructure**: ${worker_counts['infra']}/${N_QUARTER} workers
## Next Actions
- Continue monitoring all 16 supervisors
- Re-launch any missing supervisors immediately
- Monitor worker health and activity
- Check convergence status at next interval
- Next tracking issue in ~10 minutes
@@ -1292,18 +1382,24 @@ ${supervisor_status}
**Automated by CleverAgents Bot**
Supervisor: Product Builder | Agent: product-builder"
create_product_builder_tracking_issue ${cycle} "${tracking_body}"
create_product_builder_tracking_issue $cycle "$tracking_body"
# If any supervisors are missing, re-launch them immediately
if total_active < 16:
echo "[CRITICAL] Only ${total_active}/16 supervisors active - re-launching missing ones"
# Re-check which specific supervisors are missing and launch them
for supervisor_name, expected_count in supervisor_counts.items():
if expected_count == 0:
echo "[RELAUNCH] Missing supervisor: ${supervisor_name}"
metadata = SUPERVISOR_METADATA[supervisor_name]
launch_supervisor(metadata["agent"], supervisor_name, metadata["tag"], metadata["prompt"])
supervisors_relaunched += 1
if [[ $supervisor_count -lt 16 ]]; then
echo "[CRITICAL] Only $supervisor_count/16 supervisors active - re-launching missing ones"
# Check each supervisor type and re-launch if missing
for supervisor_type in "implementor-pool" "reviewer-pool" "tester-pool" "hunter-pool" "test-infra-pool" "architect" "epic-planner" "human-liaison" "agent-evolver" "arch-guard" "spec-updater" "backlog-groomer" "docs-writer" "timeline-updater" "project-owner" "system-watchdog"; do
if [[ -z "${supervisor_details[$supervisor_type]}" ]]; then
echo "[RELAUNCH] Missing supervisor: $supervisor_type"
metadata=${SUPERVISOR_METADATA[$supervisor_type]}
if [[ -n "$metadata" ]]; then
launch_supervisor "${metadata[agent]}" "$supervisor_type" "${metadata[tag]}" "${metadata[prompt]}"
supervisors_relaunched=$((supervisors_relaunched + 1))
fi
fi
done
fi
fi
for comment in recent_comments:
.opencode/agents/uat-tester.md
+108 -20
View File
@@ -337,39 +337,127 @@ LOOP:
# ── Step 4: Create individual tracking issue for progress ─────────
if cycle % 60 == 0: # Every ~10 minutes with 10-second monitoring
# Calculate actual cycle time
current_timestamp=$(date +%s)
if [[ -n "$LAST_TRACKING_TIMESTAMP" ]]; then
elapsed_seconds=$((current_timestamp - LAST_TRACKING_TIMESTAMP))
cycle_time_minutes=$((elapsed_seconds / 60))
cycle_time_display="${cycle_time_minutes} minutes"
else
cycle_time_display="10 minutes (estimated)"
fi
# Get detailed worker information from OpenCode API
SERVER="http://localhost:4096"
detailed_workers=""
# Query each active worker session for detailed status
for area in "${!active[@]}"; do
session_id="${active[$area]}"
if [[ -n "$session_id" ]]; then
# Get session status
session_status=$(curl -s "${SERVER}/session/${session_id}" | jq -r '.status // "unknown"' 2>/dev/null)
# Get recent messages to understand current testing progress
recent_messages=$(curl -s "${SERVER}/session/${session_id}/messages?limit=3" | jq -r '.[-1].content // "No recent activity"' 2>/dev/null)
last_activity=$(curl -s "${SERVER}/session/${session_id}/messages?limit=1" | jq -r '.[-1].timestamp // "unknown"' 2>/dev/null)
# Calculate time since last activity
activity_display="unknown"
if [[ "$last_activity" != "unknown" ]]; then
last_activity_timestamp=$(date -d "$last_activity" +%s 2>/dev/null || echo "0")
current_time=$(date +%s)
minutes_since_activity=$(( (current_time - last_activity_timestamp) / 60 ))
activity_display="${minutes_since_activity}m ago"
fi
# Calculate duration since assignment
duration="unknown"
if [[ -n "${worker_start_times[$area]}" ]]; then
start_timestamp="${worker_start_times[$area]}"
duration_minutes=$(( (current_time - start_timestamp) / 60 ))
if [[ $duration_minutes -lt 60 ]]; then
duration="${duration_minutes}m"
else
duration="$((duration_minutes/60))h $((duration_minutes%60))m"
fi
fi
# Extract work summary from recent message (first 100 chars)
work_summary=$(echo "$recent_messages" | head -c 100 | tr '\n' ' ')
if [[ ${#work_summary} -eq 100 ]]; then
work_summary="${work_summary}..."
fi
detailed_workers+="| $area | $session_id | $session_status | $duration | $activity_display | $work_summary |\n"
fi
done
if [[ -z "$detailed_workers" ]]; then
detailed_workers="| - | - | - | - | - | No active workers |\n"
fi
# Clean up previous tracking issue and create new one
cleanup_previous_uat_tracking
tracking_body="
[HEALTH] uat-tester | Iteration: $cycle | Status: active
tracking_body="# UAT Testing Pool Status — $(date +'%Y-%m-%d %H:%M:%S')
**Progress Summary:**
- Type: pool-supervisor
- Active workers: ${#active[@]} / $N
- Work completed: ${#tested_areas[@]}/${#feature_areas[@]} areas tested
- Coverage: $((${#tested_areas[@]} * 100 / ${#feature_areas[@]}))%
- Bugs filed: $bugs_found_total
- Documentation:
- Examples generated: $docs_generated_total
- Categories covered: ${example_categories_covered[@]}
- Last action: $brief_description
- Next check: in 10 minutes
**Agent**: uat-tester
**Cycle**: $cycle
**Cycle Time**: $cycle_time_display
**Reporting Interval**: Every 60 cycles (~10 minutes)
**Status**: active
**Active Testing Areas:**
$(for area in "${!active[@]}"; do echo "- $area (session: ${active[$area]})"; done)
## Summary
**Completed Areas:**
Pool managing ${#active[@]} active testers with ${#tested_areas[@]}/${#feature_areas[@]} areas tested ($(( ${#tested_areas[@]} * 100 / ${#feature_areas[@]} ))% coverage).
## Detailed Worker Status
**Active Workers**: ${#active[@]}/$N
| Feature Area | Session ID | Status | Duration | Last Activity | Recent Thinking |
|--------------|------------|--------|----------|---------------|-----------------|
$detailed_workers
## Testing Progress
**Coverage**: ${#tested_areas[@]}/${#feature_areas[@]} areas tested ($(( ${#tested_areas[@]} * 100 / ${#feature_areas[@]} ))%)
**Bugs Filed**: $bugs_found_total
**Documentation Generated**: $docs_generated_total examples
**Categories Covered**: ${example_categories_covered[@]}
### Completed Areas
$(for area in "${tested_areas[@]}"; do echo "- ✓ $area"; done)
**Remaining Areas:**
$(for area in "${untested[@]}"; do echo "- ○ $area"; done)
### Remaining Areas
$(for area in "${untested[@]}"; do echo "- ○ $area"; done | head -10)
$(if [[ ${#untested[@]} -gt 10 ]]; then echo "- ... and $((${#untested[@]} - 10)) more"; fi)
## Health Indicators
- **Worker Utilization**: ${#active[@]}/$N ($(( ${#active[@]} * 100 / N ))%)
- **Testing Coverage**: $(( ${#tested_areas[@]} * 100 / ${#feature_areas[@]} ))%
- **Bug Discovery Rate**: $bugs_found_total bugs found
- **Stale Workers**: $(echo -e "$detailed_workers" | grep -c "unknown\|[3-9][0-9]m ago\|[0-9][0-9][0-9]m ago") (inactive >30min)
## Next Actions
- Continue testing ${#untested[@]} remaining feature areas
- Monitor ${#active[@]} active workers for completion
- Generate documentation from successful test runs
- Check for stale workers and restart if needed
- Next status update in ~60 cycles
---
**Automated by CleverAgents Bot**
Supervisor: UAT Testing | Agent: uat-tester
"
Supervisor: UAT Testing | Agent: uat-tester"
create_uat_tracking_issue $cycle "$tracking_body"
# Store timestamp for next cycle time calculation
export LAST_TRACKING_TIMESTAMP="$current_timestamp"
# ── IMMEDIATELY loop back ────────────────────────────────────
```