Add new references/redundancy/README.md (280 lines) covering:
- Three-layer redundancy architecture overview (product-builder / watchdog / supervisors)
with the key insight that each layer uses a different observation mechanism to
prevent blind spots between layers
- Layer 1 (product-builder): fast cycle (60s liveness), deep inspection (5-min message
reading with anti-pattern catalogue: error loops, circular patterns, policy violations,
context exhaustion), worker health check (pool count vs expected), hourly verification
- Layer 2 (system-watchdog): independent 5-min audit using Forgejo tracking issue
STALENESS rather than OpenCode session status — catches frozen-but-alive sessions
that appear healthy to product-builder; session introspection for anti-pattern
detection; clear role separation (watchdog detects, product-builder restarts)
- Layer 3 (supervisor self-monitoring): per-cycle worker health checks, stuck
detection (15-min threshold), completed vs crashed distinction, pool filling
- State persistence as the foundation of self-healing: everything externalized
to Forgejo (tracking issues, attempt comments, claim protocol, announcements)
- Supervisor crash-recovery pattern: session crash → product-builder detects ≤60s →
relaunch → READ_TRACKING_STATE first → light/moderate/full recovery based on
offline duration → resume from recovered state
- Worker crash-recovery pattern: crash → supervisor detects in next cycle →
Forgejo evidence check → re-dispatch at same or escalated tier
- Two independent health signals table: OpenCode (session presence/status, latency
60s) vs Forgejo (tracking staleness, latency 2×interval) — what each catches
- Complete failure mode catalogue (13 failure types with: who detects it, how,
recovery action, and whether recovery is automatic or requires human)
- async-agent-monitor health classifications: healthy/stuck/idle/finished/errored
with threshold and configurable idle_threshold_minutes parameter
- Redundancy gaps and limitations: product-builder has no watcher; watchdog
detects but cannot restart; worker downtime latency varies by supervisor sleep
Expand SKILL.md (539 → 775 lines, 10 → 13 decision trees):
- Significantly expand 'Is something wrong?' tree: now lists every failure
type with which layer detects it, how detection works, and recovery action
(supervisor missing, frozen, error loop, waiting for input, worker crashed,
worker frozen, supervisor stopped dispatching, orphaned claim, CI violations,
multiple supervisors down, product-builder crash)
- Add new 'How does the system self-heal?' tree: full three-layer redundancy
decision tree with per-layer mechanics (fast/deep/hourly cadences), the
Forgejo persistence foundation, complete supervisor crash-recovery pattern,
complete worker crash-recovery pattern, and the single-point-of-failure note
- Update Key Numbers table: add worker health check and hourly cycle entries;
clarify session health threshold is configurable; add watchdog staleness
threshold (2×interval); add supervisor max downtime (≤60s); add worker
re-dispatch latency (varies by sleep interval)
- Update frontmatter description to cover self-healing and redundancy
- Update reference index to describe the new redundancy reference file
ISSUES CLOSED: #0
freemo
0c527b9580