cleveragents/cleveragents-core

Fork 3

Proposal: improve issue-implementor — add crash-safe health posting and graceful exit signaling #3483

New issue

Open

opened 2026-04-05 18:32:43 +00:00 by freemo · 1 comment

freemo commented

2026-04-05 18:32:43 +00:00

Owner

Agent Improvement Proposal

Pattern Detected

Type: workflow_fix
Affected Agent: issue-implementor (pool supervisor mode, defined in issue-implementor.md)
Evidence:

During Session 3 (issue #3377), the implementor pool supervisor crashed twice without posting any error or exit message:

First crash (~16:36Z): The product builder discovered that only 1 of 32 expected workers was running. Comment: "CRITICAL FAILURE: Implementation Pool Severely Under-Performing — Expected: 32 parallel implementation workers, Actual: 1 worker." The supervisor session had silently exited/crashed.
Recovery launched (~16:37Z): A new implementor supervisor was launched with emergency recovery mode. It scaled to 32 workers at full capacity.
Second crash (~17:39Z): The recovered supervisor also crashed silently. Comment: "[CRITICAL] IMPLEMENTOR SUPERVISOR CRASHED - RECOVERY IN PROGRESS — Original implementor-pool supervisor CRASHED/EXITED. Was processing 32 PRs at full capacity (100% utilization). Session no longer appears in OpenCode Server status list. Silent failure - no error message posted."

Root cause: The issue-implementor agent definition does not include:

A "last will" health comment mechanism (posting a final status before exit)
Frequent enough health heartbeats for the product builder to detect crashes quickly
Any crash-safe signaling that would survive an unexpected exit

Impact:

The implementation pipeline was running at 3% capacity (1/32 workers) for an unknown period before the product builder detected the problem
Two separate crashes went undetected until the product builder's monitoring cycle caught them
164 PRs accumulated in the backlog while the supervisor was down
The product builder had to implement emergency recovery procedures

Proposed Change

Modify the issue-implementor.md agent definition to add crash resilience:

Add frequent health heartbeats — The supervisor should post a brief health comment on the session state issue every 5 minutes (not just when dispatching workers). This allows the product builder to detect crashes within 5-10 minutes instead of potentially much longer.
Add a "starting cycle" marker — At the beginning of each supervision cycle, post a brief "alive" signal. If the supervisor crashes mid-cycle, the absence of the next "alive" signal is detectable.
Add worker count validation — After each dispatch cycle, verify that the number of active workers matches expectations. If workers are silently dying, the supervisor should detect and report this.
Add graceful exit signaling — Before any intentional exit (e.g., all work complete), post a clear "EXITING" message on the session state issue. This distinguishes intentional exits from crashes.

Expected Impact

Crashes detected within 5-10 minutes instead of potentially 30+ minutes
Clear distinction between intentional exits and crashes
Worker count drift detected and reported proactively
Reduced PR backlog accumulation during supervisor downtime

Risk Assessment

Low risk: The changes only add monitoring and signaling. No dispatch or worker management logic is modified.
Potential concern: More frequent health comments could add noise to the session state issue. However, 5-minute intervals are reasonable and much less noisy than the test-infra-improver's 17-second intervals that prompted proposal #3385.

This is a proposal from the agent evolver. A human must approve this issue before the change will be implemented. To approve: remove the needs feedback label, add State/Verified, or comment with approval.

Automated by CleverAgents Bot
Supervisor: Agent Evolver | Agent: ca-agent-evolver

## Agent Improvement Proposal ### Pattern Detected **Type**: workflow_fix **Affected Agent**: issue-implementor (pool supervisor mode, defined in `issue-implementor.md`) **Evidence**: During Session 3 (issue #3377), the implementor pool supervisor crashed **twice** without posting any error or exit message: 1. **First crash (~16:36Z)**: The product builder discovered that only 1 of 32 expected workers was running. Comment: "CRITICAL FAILURE: Implementation Pool Severely Under-Performing — Expected: 32 parallel implementation workers, Actual: 1 worker." The supervisor session had silently exited/crashed. 2. **Recovery launched (~16:37Z)**: A new implementor supervisor was launched with emergency recovery mode. It scaled to 32 workers at full capacity. 3. **Second crash (~17:39Z)**: The recovered supervisor also crashed silently. Comment: "[CRITICAL] IMPLEMENTOR SUPERVISOR CRASHED - RECOVERY IN PROGRESS — Original implementor-pool supervisor CRASHED/EXITED. Was processing 32 PRs at full capacity (100% utilization). Session no longer appears in OpenCode Server status list. Silent failure - no error message posted." **Root cause**: The `issue-implementor` agent definition does not include: - A "last will" health comment mechanism (posting a final status before exit) - Frequent enough health heartbeats for the product builder to detect crashes quickly - Any crash-safe signaling that would survive an unexpected exit **Impact**: - The implementation pipeline was running at 3% capacity (1/32 workers) for an unknown period before the product builder detected the problem - Two separate crashes went undetected until the product builder's monitoring cycle caught them - 164 PRs accumulated in the backlog while the supervisor was down - The product builder had to implement emergency recovery procedures ### Proposed Change Modify the `issue-implementor.md` agent definition to add crash resilience: 1. **Add frequent health heartbeats** — The supervisor should post a brief health comment on the session state issue every 5 minutes (not just when dispatching workers). This allows the product builder to detect crashes within 5-10 minutes instead of potentially much longer. 2. **Add a "starting cycle" marker** — At the beginning of each supervision cycle, post a brief "alive" signal. If the supervisor crashes mid-cycle, the absence of the next "alive" signal is detectable. 3. **Add worker count validation** — After each dispatch cycle, verify that the number of active workers matches expectations. If workers are silently dying, the supervisor should detect and report this. 4. **Add graceful exit signaling** — Before any intentional exit (e.g., all work complete), post a clear "EXITING" message on the session state issue. This distinguishes intentional exits from crashes. ### Expected Impact - Crashes detected within 5-10 minutes instead of potentially 30+ minutes - Clear distinction between intentional exits and crashes - Worker count drift detected and reported proactively - Reduced PR backlog accumulation during supervisor downtime ### Risk Assessment - **Low risk**: The changes only add monitoring and signaling. No dispatch or worker management logic is modified. - **Potential concern**: More frequent health comments could add noise to the session state issue. However, 5-minute intervals are reasonable and much less noisy than the test-infra-improver's 17-second intervals that prompted proposal #3385. --- *This is a proposal from the agent evolver. A human must approve this issue before the change will be implemented. To approve: remove the `needs feedback` label, add `State/Verified`, or comment with approval.* --- **Automated by CleverAgents Bot** Supervisor: Agent Evolver | Agent: ca-agent-evolver

freemo referenced this issue

2026-04-05 18:33:22 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 #3377

freemo referenced this issue

2026-04-05 19:04:08 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 #3377

freemo referenced this issue

2026-04-05 19:08:56 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 #3377

freemo referenced this issue

2026-04-05 19:36:07 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 #3377

HAL9000 referenced this issue

2026-04-08 19:33:22 +00:00

[Automated] CleverAgents Build Session - 2026-04-08 #4799

HAL9000 commented

2026-04-08 20:05:58 +00:00

Owner

Issue triaged by project owner:

State: Verified — Valid agent improvement proposal with strong evidence. The implementation pool supervisor crashed twice silently during Session 3, leaving the system at 3% capacity (1/32 workers) for an unknown period. 164 PRs accumulated in the backlog.
Priority: High — Silent supervisor crashes that go undetected for 30+ minutes are a critical reliability issue. The system cannot self-heal if it doesn't know it's broken.
Milestone: v3.5.0 — Autonomy Hardening milestone (crash resilience is core to this milestone)
Story Points: 3 (M) — Add health heartbeats (5-min interval), "starting cycle" markers, worker count validation, and graceful exit signaling to the issue-implementor agent definition
MoSCoW: Must Have — Crash detection within 5-10 minutes (vs 30+ minutes currently) is essential for production reliability. The system must be able to detect and recover from supervisor failures quickly.
Parent Epic: #397 (Epic: Server & Autonomy Infrastructure)

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner

Issue triaged by project owner: - **State**: Verified — Valid agent improvement proposal with strong evidence. The implementation pool supervisor crashed twice silently during Session 3, leaving the system at 3% capacity (1/32 workers) for an unknown period. 164 PRs accumulated in the backlog. - **Priority**: High — Silent supervisor crashes that go undetected for 30+ minutes are a critical reliability issue. The system cannot self-heal if it doesn't know it's broken. - **Milestone**: v3.5.0 — Autonomy Hardening milestone (crash resilience is core to this milestone) - **Story Points**: 3 (M) — Add health heartbeats (5-min interval), "starting cycle" markers, worker count validation, and graceful exit signaling to the issue-implementor agent definition - **MoSCoW**: Must Have — Crash detection within 5-10 minutes (vs 30+ minutes currently) is essential for production reliability. The system must be able to detect and recover from supervisor failures quickly. - **Parent Epic**: #397 (Epic: Server & Autonomy Infrastructure) --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner

HAL9000 added this to the v3.5.0 milestone

2026-04-08 20:06:28 +00:00

HAL9000 added a new dependency

2026-04-08 20:06:48 +00:00

#397 Epic: Server & Autonomy Infrastructure

HAL9000 referenced this issue

2026-04-08 20:07:14 +00:00

[Automated] CleverAgents Build Session - 2026-04-08 #4799