cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity

docs(skill): add full redundancy and self-healing documentation to cleveragents-system skill
CI / push-validation (push) Successful in 22s
Details
CI / build (push) Successful in 24s
Details
CI / helm (push) Successful in 32s
Details
CI / lint (push) Successful in 33s
Details
CI / typecheck (push) Successful in 48s
Details
CI / quality (push) Successful in 53s
Details
CI / security (push) Successful in 1m9s
Details
CI / integration_tests (push) Successful in 4m12s
Details
CI / e2e_tests (push) Successful in 6m43s
Details
CI / unit_tests (push) Successful in 7m1s
Details
CI / coverage (push) Successful in 6m59s
Details
CI / docker (push) Successful in 1m33s
Details
CI / status-check (push) Successful in 1s
Details
CI / benchmark-publish (push) Successful in 1h13m54s
Details
CI / benchmark-regression (push) Has been skipped
Details

Browse Source 
Add new references/redundancy/README.md (280 lines) covering:
- Three-layer redundancy architecture overview (product-builder / watchdog / supervisors)
  with the key insight that each layer uses a different observation mechanism to
  prevent blind spots between layers
- Layer 1 (product-builder): fast cycle (60s liveness), deep inspection (5-min message
  reading with anti-pattern catalogue: error loops, circular patterns, policy violations,
  context exhaustion), worker health check (pool count vs expected), hourly verification
- Layer 2 (system-watchdog): independent 5-min audit using Forgejo tracking issue
  STALENESS rather than OpenCode session status — catches frozen-but-alive sessions
  that appear healthy to product-builder; session introspection for anti-pattern
  detection; clear role separation (watchdog detects, product-builder restarts)
- Layer 3 (supervisor self-monitoring): per-cycle worker health checks, stuck
  detection (15-min threshold), completed vs crashed distinction, pool filling
- State persistence as the foundation of self-healing: everything externalized
  to Forgejo (tracking issues, attempt comments, claim protocol, announcements)
- Supervisor crash-recovery pattern: session crash → product-builder detects ≤60s →
  relaunch → READ_TRACKING_STATE first → light/moderate/full recovery based on
  offline duration → resume from recovered state
- Worker crash-recovery pattern: crash → supervisor detects in next cycle →
  Forgejo evidence check → re-dispatch at same or escalated tier
- Two independent health signals table: OpenCode (session presence/status, latency
  60s) vs Forgejo (tracking staleness, latency 2×interval) — what each catches
- Complete failure mode catalogue (13 failure types with: who detects it, how,
  recovery action, and whether recovery is automatic or requires human)
- async-agent-monitor health classifications: healthy/stuck/idle/finished/errored
  with threshold and configurable idle_threshold_minutes parameter
- Redundancy gaps and limitations: product-builder has no watcher; watchdog
  detects but cannot restart; worker downtime latency varies by supervisor sleep

Expand SKILL.md (539 → 775 lines, 10 → 13 decision trees):
- Significantly expand 'Is something wrong?' tree: now lists every failure
  type with which layer detects it, how detection works, and recovery action
  (supervisor missing, frozen, error loop, waiting for input, worker crashed,
  worker frozen, supervisor stopped dispatching, orphaned claim, CI violations,
  multiple supervisors down, product-builder crash)
- Add new 'How does the system self-heal?' tree: full three-layer redundancy
  decision tree with per-layer mechanics (fast/deep/hourly cadences), the
  Forgejo persistence foundation, complete supervisor crash-recovery pattern,
  complete worker crash-recovery pattern, and the single-point-of-failure note
- Update Key Numbers table: add worker health check and hourly cycle entries;
  clarify session health threshold is configurable; add watchdog staleness
  threshold (2×interval); add supervisor max downtime (≤60s); add worker
  re-dispatch latency (varies by sleep interval)
- Update frontmatter description to cover self-healing and redundancy
- Update reference index to describe the new redundancy reference file

ISSUES CLOSED: #0
This commit is contained in:
Jeffrey Phillips Freeman
2026-04-15 20:18:54 +00:00
parent e1e65c85ae
commit c9dc70004c
2 changed files with 443 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.opencode/skills/cleveragents-system/SKILL.md
+163 -27
View File
@@ -19,8 +19,15 @@ description: |
the announcement relevancy matrix determines which agents consume which
announcements at what priority threshold, how claim/heartbeat/release
coordination prevents duplicate work, how supervisors recover state from
previous tracking issues on startup, and any other operational question
about how the autonomous system runs.
previous tracking issues on startup, how the three-layer redundancy and
self-healing system works (product-builder watches supervisors every 60s,
system-watchdog audits every 5min via Forgejo staleness, each supervisor
monitors its own workers every cycle), how crashed supervisors are detected
and relaunched within 60 seconds, how crashed workers are detected and
re-dispatched, how the two independent health signals (OpenCode session API
and Forgejo tracking issue staleness) cross-check each other, what failure
modes exist and how each is detected and recovered, and any other operational
question about how the autonomous system runs.
Covers: agent hierarchy (product-builder → 17 pool supervisors → workers →
utility subagents), all 17 supervisor prefixes (AUTO-PROD-BLDR through
@@ -57,6 +64,7 @@ references:
- tier-system
- credential-flow
- coordination
- redundancy
---
# CleverAgents System — Self-Reflection Skill
@@ -542,34 +550,157 @@ Credential propagation in the system:
### "Is something wrong with the system?"
```
Diagnosing system health:
Diagnosing system health — which layer detects which failure:
│
├─ A supervisor session is missing or not in busy state?
│ ├─ product-builder detects this in its fast cycle (every 60s)
│ └─ product-builder restarts the supervisor immediately with fresh session
├─ Supervisor session MISSING or not in busy state
│ ├─ Who detects: product-builder (fast cycle, every 60s)
│ ├─ How: searches session list for supervisor tag; absent or non-busy = problem
│ └─ Recovery: product-builder stops old session (if present), relaunches within 60s
│
├─ A worker appears stuck (no progress for 15+ minutes)?
│ ├─ product-builder reads messages from long-running sessions every 5 cycles
│ ├─ If stuck: stop the session and relaunch it
│ └─ Supervisor should keep its pool filled
├─ Supervisor EXISTS but is FROZEN (busy but no Forgejo output)
│ ├─ Who detects: system-watchdog (5-min cycle) via Forgejo tracking issue staleness
│ │ → issue age > 2× estimated_cycle_interval = supervisor stopped reporting
│ ├─ Also: product-builder (deep inspection) reads session messages directly
│ └─ Recovery: product-builder detects via deep inspection and relaunches
│
├─ A supervisor's tracking issue is older than 2× its cycle interval?
│ ├─ system-watchdog detects this
│ └─ system-watchdog creates announcement and notifies product-builder
├─ Supervisor in an ERROR LOOP (same error repeating)
│ ├─ Who detects: system-watchdog reads session messages via async-agent-manager
│ │ → same error message 3+ times = error loop
│ ├─ Also: product-builder deep inspection every 5 cycles
│ └─ Recovery: system-watchdog creates announcement; product-builder relaunches
│
├─ A PR or issue has a work claim that expired without a release?
│ ├─ system-watchdog detects expired claims via coordination protocol parsing
│ └─ system-watchdog dispatches a worker to clean up and reopen the work
├─ Supervisor WAITING FOR INPUT (session idle, asking a question)
│ ├─ Who detects: product-builder fast cycle → session status = idle
│ ├─ Recovery: product-builder reads messages, understands question, provides answer
│ └─ Without this: supervisor waits indefinitely — a silent failure
│
├─ CI quality gate violations?
│ ├─ system-watchdog creates Priority/CI-Blocker announcement
│ └─ ALL agents consume CI-Blocker announcements (universal baseline)
├─ Worker CRASHED (session gone, no completion evidence on Forgejo)
│ ├─ Who detects: owning supervisor (every cycle) → session gone + no PR/comment created
│ └─ Recovery: supervisor re-dispatches a new worker at same (or escalated) tier
│
└─ Looking for active automation agents?
├─ Worker FROZEN (busy but no message activity for 15+ min)
│ ├─ Who detects: owning supervisor via async-agent-monitor (classifies as 'stuck')
│ └─ Recovery: supervisor stops session and re-dispatches
│
├─ Supervisor STOPPED DISPATCHING WORKERS (pool consistently undercount)
│ ├─ Who detects: product-builder worker health check (every 5 cycles)
│ │ → supervisor running > 5 min but active workers < expected
│ └─ Recovery: product-builder relaunches the supervisor
│
├─ Work claim EXPIRED without release (orphaned claim)
│ ├─ Who detects: system-watchdog parses claim comments on issues/PRs
│ └─ Recovery: system-watchdog dispatches fix worker to clean up
│
├─ CI quality gate violations
│ ├─ Who detects: system-watchdog → checks CI run results, branch protection
│ └─ Action: system-watchdog creates Priority/CI-Blocker announcement
│ → ALL agents consume CI-Blocker immediately (universal baseline)
│
├─ MULTIPLE supervisors down simultaneously
│ ├─ Who detects: product-builder (repeated relaunch failures); system-watchdog
│ └─ Action: Priority/Critical announcement; human intervention may be needed
│
├─ PRODUCT-BUILDER itself crashed
│ ├─ Who detects: NO ONE — product-builder has no external watcher
│ └─ Recovery: requires human to restart product-builder manually
│
└─ Finding active automation agents (discovery)
├─ Search Forgejo for open issues with label "Automation Tracking"
├─ Title format: "[AUTO-PREFIX] Status: Type (Cycle N)"
├─ The most recent open status issue for each prefix = that agent's current state
└─ Read its body and comments for detailed status
├─ Status issue titles: "[AUTO-PREFIX] Status: Type (Cycle N)"
├─ Most recent open status issue per prefix = that agent's current state
├─ If no status issue for a prefix → agent is down or has never run
└─ Read body and comments for detailed current state
```
### "How does the system self-heal?"
```
Three independent layers provide overlapping redundancy:
│
├─ LAYER 1: product-builder watches ALL supervisors
│ │
│ ├─ FAST CYCLE — every 60 seconds:
│ │ ├─ Verify each supervisor session exists AND is in 'busy' state
│ │ │ → Missing or errored → stop old session → relaunch immediately
│ │ │ → Session is 'idle' (waiting for input) → read messages → answer question
│ │ └─ This is the PRIMARY recovery mechanism — 60s maximum downtime per supervisor
│ │
│ ├─ DEEP INSPECTION — every 5 fast cycles (~5 minutes):
│ │ ├─ Read last several messages from each supervisor session
│ │ ├─ Signs that trigger stop-and-relaunch:
│ │ │ ├─ Same error repeated 3+ times (error loop)
│ │ │ ├─ No work activity for 15+ minutes (frozen)
│ │ │ ├─ Circular tool calls (logic bug)
│ │ │ └─ Agent doing work outside its responsibilities
│ │ └─ Worker health check: count active workers vs expected; if supervisor
│ │ stopped dispatching → stop it and relaunch
│ │
│ └─ HOURLY CYCLE — every ~60 fast cycles:
│ ├─ Run product-verifier for full completion assessment
│ ├─ Create comprehensive status tracking issue
│ └─ Detect convergence (all milestones done → report completion)
│
├─ LAYER 2: system-watchdog AUDITS ALL supervisors independently
│ │
│ ├─ Runs every 5 minutes — completely independent of product-builder
│ │
│ ├─ Uses a DIFFERENT health signal than product-builder:
│ │ ├─ product-builder → OpenCode session API (is the session present/busy?)
│ │ └─ system-watchdog → Forgejo tracking issue staleness
│ │ → if issue_age > 2 × cycle_interval → supervisor stopped reporting
│ │ → catches frozen-but-alive sessions that appear healthy to product-builder
│ │
│ ├─ Deep session introspection via async-agent-manager:
│ │ ├─ Error loops, circular patterns, policy violations, context exhaustion
│ │ └─ Creates Priority/Critical announcements for critical findings
│ │
│ └─ system-watchdog creates announcements; product-builder acts on them
│ (watchdog detects, product-builder restarts — clear separation of roles)
│
├─ LAYER 3: each supervisor monitors its own workers every cycle
│ │
│ ├─ Searches for worker sessions by tag pattern every cycle
│ ├─ Reads messages from long-running workers to verify progress
│ ├─ Classifies: healthy / stuck (>15min idle) / completed / errored
│ ├─ Restarts errored or stuck workers immediately
│ └─ Keeps pool filled: if workers < capacity AND work exists → dispatch more
│
├─ THE FORGEJO PERSISTENCE FOUNDATION (why self-healing works at all)
│ │
│ ├─ All state is externalized to Forgejo — nothing lives only in session memory
│ │ ├─ Supervisor state → tracking issues (updated every N cycles)
│ │ ├─ Worker results → issue/PR comments (posted before worker exits)
│ │ ├─ Escalation tier → attempt comments (each attempt records its tier)
│ │ └─ Work claims → CLAIM/HEARTBEAT/RELEASE comments on issues/PRs
│ │
│ ├─ SUPERVISOR CRASH RECOVERY PATTERN:
│ │ 1. Session crashes (context exhaustion, API error, OpenCode bug)
│ │ 2. product-builder detects in next fast cycle (≤60s)
│ │ 3. product-builder relaunches with fresh session
│ │ 4. New supervisor calls READ_TRACKING_STATE FIRST
│ │ 5. Reads: cycle number, offline duration, what was being worked on
│ │ 6. If offline < 30 min: light resume; 30-120 min: re-verify; >120 min: full re-scan
│ │ 7. Creates new tracking issue (closes old one)
│ │ 8. Re-dispatches workers for in-progress items
│ │ → No data loss: all work is in Forgejo; gap in tracking reveals downtime
│ │
│ └─ WORKER CRASH RECOVERY PATTERN:
│ 1. Worker crashes mid-task (session gone)
│ 2. Supervisor detects in next cycle (2–60 min depending on sleep)
│ 3. Checks Forgejo: was a PR created? was a completion comment posted?
│ → Yes: worker completed → mark done
│ → No: worker crashed → re-dispatch
│ 4. New worker reads all previous comments → determines last tier used
│ 5. Re-dispatches at same tier (pure crash) or next tier (known failure)
│
└─ THE SINGLE POINT OF FAILURE
└─ product-builder has no external watcher
→ if product-builder crashes: supervisors keep running until their sessions
naturally end or hit context limits, but will not be relaunched
→ system-watchdog will create announcements as supervisors go down one by one
→ human must restart product-builder
→ once restarted: product-builder re-reads all supervisor tracking states,
relaunches any that are down, and resumes normal monitoring
```
---
@@ -618,10 +749,14 @@ Example: `bash("sleep 120", timeout=180000)` — never let the timeout be less t
| Forgejo API timeout | 30 seconds | |
| Git operation timeout | 5 minutes | |
| Test execution timeout | 15 minutes | |
| Session health threshold | 15 minutes idle = stuck | Deep inspection cadence |
| Product-builder fast cycle | 60 seconds | |
| Product-builder deep inspection | Every 5 fast cycles (~5 min) | |
| Product-builder hourly cycle | Every ~60 fast cycles | |
| Session health threshold | 15 minutes idle = stuck | Configurable via idle_threshold_minutes |
| Product-builder fast cycle | 60 seconds | Liveness check + question-answering |
| Product-builder deep inspection | Every 5 fast cycles (~5 min) | Message reading, anti-pattern detection |
| Product-builder worker health check | Every 5 fast cycles (~5 min) | Pool count vs expected |
| Product-builder hourly cycle | Every ~60 fast cycles | Full verification + status ticket |
| Watchdog staleness threshold | 2 × estimated_cycle_interval | If tracking issue older → supervisor frozen |
| Supervisor max downtime | ≤60 seconds | product-builder fast cycle detects crash |
| Worker max downtime before re-dispatch | Supervisor's sleep interval | 2 min (IMP) to 60 min (TIME) |
| Max temp dir age | 24 hours | Cleaned automatically |
| Forgejo MCP page limit | 50 per page | Always paginate exhaustively |
@@ -637,3 +772,4 @@ Example: `bash("sleep 120", timeout=180000)` — never let the timeout be less t
| `references/tier-system/` | Model tiers, tier selectors, progressive escalation, model assignments |
| `references/credential-flow/` | Environment variables, how credentials pass through the hierarchy |
| `references/coordination/` | Claim/heartbeat/release protocol, work-item coordination, conflict resolution |
| `references/redundancy/` | Three-layer redundancy architecture, product-builder monitoring cycles, watchdog staleness detection, supervisor and worker crash-recovery patterns, failure mode catalogue, async-agent-monitor health classifications |
.opencode/skills/cleveragents-system/references/redundancy/README.md
+280
View File
@@ -0,0 +1,280 @@
# Redundancy and Self-Healing — How the System Stays Alive
## Overview
The system has no single point of failure for ordinary operation. Three independent layers watch for problems simultaneously, using two different observation mechanisms (OpenCode session API and Forgejo tracking issues) to cross-check each other.
```
Layer 1 — product-builder watches ALL supervisors (every 60 seconds)
Layer 2 — system-watchdog audits ALL supervisors (every 5 minutes)
Layer 3 — each supervisor monitors its own workers (every cycle)
```
No layer depends on another to detect problems — if product-builder is distracted or slow, system-watchdog independently detects frozen supervisors via Forgejo. If a supervisor's pool drains, product-builder notices the undercount before system-watchdog checks in.
---
## Layer 1: Product-Builder Monitoring
Product-builder (`AUTO-PROD-BLDR`) runs an infinite monitoring loop with three cadences.
### Fast Cycle — every 60 seconds
Performs two checks per cycle:
**1. Supervisor liveness check**:
- Search for each supervisor's session tag via `async-agent-manager`
- A session must exist AND be in `busy` state
- If the session is `completed`, `errored`, or **missing** → relaunch immediately
- Before relaunching: verify the old session is stopped (prevent duplicates)
- Relaunched supervisors recover their own state from Forgejo (no data briefing needed from product-builder)
**2. Stuck-supervisor detection (waiting for input)**:
- If a supervisor's session is `idle` (waiting for a question answer) → read its messages
- Understand what question it's asking, provide an answer
- Without this, a supervisor that hit a decision point would wait indefinitely
### Deep Inspection — every 5 fast cycles (~5 minutes)
For each supervisor, reads the last several messages from the session (including internal reasoning):
| Sign of problem | Action |
|-----------------|--------|
| Same error message repeated 3+ times | Stop and relaunch |
| No work activity for 15+ minutes | Stop and relaunch |
| Circular tool call pattern (A → B → A → B) | Stop and relaunch |
| Agent doing work outside its responsibilities | Stop and relaunch |
| Agent still `busy` but producing no Forgejo output | Stop and relaunch |
### Worker Health Check — every 5 fast cycles (~5 minutes)
For each supervisor (except PR Merge — see below):
1. Search for sessions matching the supervisor's worker tag pattern
2. Count active (`busy`) workers
3. Compare against expected count for this supervisor
4. If supervisor has run > 5 minutes but consistently fewer workers than expected: investigate
- Read supervisor messages to distinguish "no work available" (normal) from "stopped dispatching" (broken)
- If broken: stop supervisor and relaunch
**PR Merge Pool exception**: `AUTO-PRMRG-SUP` calls `pr-merge-worker` as a blocking Task subagent — there are no `[AUTO-PRMRG-N]` async sessions. Skip worker count check entirely. Health = supervisor session active + tracking issue fresh.
### Hourly Cycle — every ~60 fast cycles
1. Invoke `product-verifier` for full product completion assessment
2. Convergence check: are all milestones done, all issues closed, all PRs merged?
3. Create new status tracking issue with: supervisor states, worker counts vs expected, per-worker summaries, completion results, corrective actions taken since last report
### Product-Builder's Own Longevity
Product-builder manages its own context window explicitly:
- Keeps in memory: supervisor prefix registry, all supervisor session IDs, heartbeat counter, credentials, N tier values
- Discards periodically: old tool call outputs, past monitoring results
- Everything else is reconstructable from the OpenCode session API and Forgejo
**Critical limit**: product-builder itself has no external watcher. If it crashes or exhausts its context, a human must restart it. This is the only single point of failure in the system. All supervisor-level crashes self-heal; product-builder-level crashes require human intervention.
---
## Layer 2: System Watchdog
System-watchdog (`AUTO-WDOG`) runs independently every 5 minutes and uses a **completely different observation mechanism** from product-builder.
### Primary Signal: Forgejo Tracking Issue Staleness
While product-builder uses the OpenCode session API (is the session present? is it busy?), system-watchdog uses **Forgejo tracking issues**:
1. Read the status tracking issue for every supervisor (issues with `Automation Tracking` label and title starting with `[{prefix}] Status:`)
2. Note the issue's creation timestamp and the `Estimated Cycle Interval` field
3. If `issue_age > 2 × estimated_cycle_interval` → supervisor has stopped reporting → may be frozen
4. Cross-reference with session status via `async-agent-manager` to confirm
This cross-referencing catches a class of bugs that OpenCode alone cannot: a supervisor session might still be `busy` (session exists, appears active) but has stopped doing any real work — it's spinning in a tight loop or stuck processing a failed operation. The Forgejo tracking issue goes stale even though the session appears healthy.
### Session Introspection
Via `async-agent-manager`, system-watchdog can read supervisor session messages to detect:
| Pattern | Diagnosis | Action |
|---------|-----------|--------|
| Same error message repeated 3+ times | Error loop — supervisor can't proceed | Create announcement; dispatch fix worker |
| Tool call sequence A → B → A → B cycling | Circular pattern — logic bug | Create announcement; dispatch fix worker |
| Forbidden tool calls (e.g., direct curl to localhost:4096) | Policy violation | Create Priority/High announcement |
| Increasingly short responses, loss of coherence | Context exhaustion — session needs reset | Create announcement; product-builder will detect and relaunch |
| No new Forgejo activity despite busy session | Zombie session | Create Priority/Critical announcement |
### Watchdog Audit Categories (per 5-minute cycle)
1. **Supervisor health**: tracking issue staleness + session cross-reference
2. **Quality gate enforcement**: branch protection active, merged PRs had CI, no false merges
3. **Ticket state correctness**: State/Completed on closed issues, Blocked label on paused issues
4. **PR health**: no stale PRs, reviews within 48h, no unreviewed merges
5. **Dependency and label integrity**: correct dependency direction, required labels present, no duplicate tracking issues
### Watchdog's Limitations
System-watchdog monitors supervisors but does NOT restart them. It creates:
- Announcements for product-builder and other agents to act on
- `needs feedback` issues for problems requiring human or agent-definition changes
- Fix workers (via `async-agent-manager`) for correctable issues like label mismatches
Product-builder is responsible for actually relaunching failed supervisors.
---
## Layer 3: Supervisor Self-Monitoring of Workers
Each pool supervisor monitors its own workers every cycle as part of its main loop.
### Worker Health Check (per supervisor cycle)
```
Every cycle:
1. Search sessions by tag pattern (e.g., [AUTO-IMP-ISSUE-*] and [AUTO-IMP-PR-*])
2. For each worker session:
a. Get session status (busy / idle / gone)
b. If busy → check last message timestamp (via async-agent-manager)
c. If last message > 15 minutes ago → session is stuck
d. If stuck or errored → stop session and mark work item for re-dispatch
e. If gone → worker completed (normal); check if the work is done on Forgejo
3. Count active workers; if below pool capacity, dispatch new workers
```
### Worker Completion vs Worker Crash
Workers completing normally is expected. The supervisor distinguishes:
| Observation | Meaning | Action |
|-------------|---------|--------|
| Session gone, Forgejo shows PR created / issue closed | Worker completed successfully | No action needed |
| Session gone, no Forgejo evidence of completion | Worker crashed before finishing | Re-dispatch at same or next tier |
| Session `idle` (waiting for input) | Worker stuck on a question | Read messages; answer if possible; else restart |
| Session `busy` but no new Forgejo output for 15+ min | Worker frozen | Stop and re-dispatch |
### Errored Worker Re-dispatch
When re-dispatching after a crash:
- Same issue/PR, new worker session with same tag pattern
- New worker reads issue comments to understand what was previously attempted
- If the previous attempt left a clear failure comment → escalate tier
- If no clear failure comment (pure crash) → re-attempt at same tier
### Supervisor Pool Filling
Each pool supervisor maintains a sliding window of active workers:
- If active workers < pool capacity AND work is available → dispatch new workers
- The pool is continuously filled, not batch-dispatched
- A supervisor that stops filling its pool triggers product-builder's worker health check and gets relaunched
---
## State Persistence: The Foundation of Self-Healing
Self-healing only works because all state is externalized to Forgejo. No state lives only in a session's memory.
### What Gets Persisted and Where
| State | Persisted To | How |
|-------|-------------|-----|
| Supervisor work status | Forgejo tracking issues | `CREATE_TRACKING_ISSUE` every N cycles |
| Worker attempt results | Forgejo issue/PR comments | Worker posts attempt comment before exiting |
| Escalation tier | Forgejo issue/PR comments | Attempt comment records current tier |
| Work-item claims | Forgejo issue/PR comments | CLAIM/HEARTBEAT/RELEASE protocol |
| Announcements | Forgejo announcement issues | Persist until explicitly closed |
### The Supervisor Crash-Recovery Pattern
```
1. Supervisor runs → writes state to Forgejo tracking issues every ~N cycles
2. Supervisor session crashes (context exhaustion, API error, OpenCode bug, etc.)
3. product-builder detects: session gone or errored in next fast cycle (≤60s later)
4. product-builder stops old session (if still present), launches fresh session
5. Fresh supervisor starts and IMMEDIATELY calls READ_TRACKING_STATE
6. Reads: cycle number, offline duration in minutes, previous issue body + all comments
7. If offline < 30 min: light recovery — resume from last known state
If offline 30-120 min: re-verify key Forgejo state before resuming
If offline > 120 min: full re-scan as if fresh start, using hints from state
8. Calls CREATE_TRACKING_ISSUE (closes old status issue, opens new one)
9. Re-dispatches workers for items that were in progress
10. Normal operation resumes; gap in tracking issues reveals the downtime period
```
**No data is lost** because:
- All work results are in Forgejo (PRs, comments, issue state changes)
- The supervisor's state ticket describes what it was working on
- Worker attempt comments describe escalation history
- The supervisor simply re-reads Forgejo to reconstruct its state
### The Worker Crash-Recovery Pattern
```
1. Worker runs → posts attempt comment on issue/PR at start and end
2. Worker crashes mid-task
3. Session gone from OpenCode
4. Supervisor notices in next cycle (2-30 min depending on sleep interval)
5. Supervisor checks Forgejo: was there a completion comment? was a PR created?
- Yes: worker completed → mark work done
- No: worker crashed → re-dispatch
6. New worker reads all previous comments on the issue/PR
7. Determines: what tier was last used? what was last attempted? did it partially succeed?
8. New worker starts from the appropriate resumption point
```
---
## The Two Independent Health Signals
Understanding which signal catches which failure:
| Signal | Source | Latency | What It Catches |
|--------|--------|---------|-----------------|
| Session presence/status | OpenCode API (`GET /session/status`) | 60 seconds | Outright crashes, completed sessions, sessions waiting for input |
| Tracking issue staleness | Forgejo (`Automation Tracking` label) | 2× cycle interval | Frozen-but-alive sessions, infinite loops, context exhaustion |
Together they provide complete coverage. A session can appear `busy` in OpenCode while being completely stuck (no Forgejo output) — only the staleness check catches this. A session can stop posting to Forgejo because it crashed — only the OpenCode check catches this.
---
## Detecting Specific Failure Modes
| Failure | How Detected | Recovery |
|---------|-------------|---------|
| Supervisor session crashed | product-builder: session missing in fast cycle | product-builder relaunches within 60s |
| Supervisor context exhausted | product-builder: deep inspection sees incoherence; system-watchdog: tracking issue stale | product-builder stops+relaunches |
| Supervisor in error loop | system-watchdog: reads same error 3+ times in messages | system-watchdog announces; product-builder relaunches |
| Supervisor stopped dispatching workers | product-builder: worker count below expected; system-watchdog: stale tracking + no Forgejo activity | product-builder relaunches |
| Worker crashed mid-task | supervisor: session gone, no completion evidence on Forgejo | supervisor re-dispatches at same tier |
| Worker frozen (busy but no output) | supervisor: last message > 15 min old; async-agent-monitor classifies as `stuck` | supervisor stops and re-dispatches |
| Worker repeatedly failing same problem | supervisor: reads attempt comments, escalation tier logic | supervisor escalates tier; eventually human escalation |
| Session answering questions (paused for input) | product-builder: session `idle` in fast cycle; reads messages to find the question | product-builder provides an answer |
| Multiple supervisors down | product-builder: repeated relaunch failures; system-watchdog: multiple stale tracking issues | product-builder: Priority/Critical announcement; human intervention |
| product-builder itself crashed | No automatic detection | Human must restart |
---
## The async-agent-monitor Subagent
Used by supervisors and product-builder to perform health checks without directly calling localhost:4096.
Classifies sessions as:
| Classification | Condition | Threshold |
|---------------|-----------|-----------|
| `healthy` | `busy` + recent message activity | Last message < 15 min ago (configurable) |
| `stuck` | `busy` + no recent message activity | Last message ≥ 15 min ago |
| `idle` | OpenCode status = `idle` | Waiting for user input |
| `finished` | Session gone (deleted by runtime) | Session no longer in list |
| `errored` | Session in error state | Error part in last messages |
The 15-minute threshold is configurable by the caller via `idle_threshold_minutes`. For short-cycle supervisors (120s sleep) this may be set lower; for long-cycle supervisors (1800s sleep) the default may be appropriate.
---
## Redundancy Gaps and Limitations
| Limitation | Impact | Mitigation |
|------------|--------|-----------|
| product-builder has no external watcher | product-builder crash requires human restart | product-builder manages its own context carefully; system is designed to minimize context growth |
| Watchdog monitors correctness, not liveness | system-watchdog cannot restart supervisors | Creates announcements for product-builder to act on; product-builder does the actual restart |
| Worker crash detection has latency | Latency = supervisor's sleep interval (2–60 min) | Short-cycle supervisors (2 min) detect quickly; long-cycle supervisors tolerate more latency for their lower-urgency work |
| State recovery based on Forgejo comments | If Forgejo is unavailable, state cannot be read | Forgejo unavailability is detected immediately; product-builder creates Priority/Critical announcement |
| No hot-standby for supervisors | Only one instance of each supervisor type | product-builder relaunches within 60s; work items are reclaimed via claim protocol |