cleveragents/cleveragents-core

Fork 3

[AUTO-INF-1] Add per-job timeouts to prevent multi-day CI runs #8381

New issue

Open

opened 2026-04-13 17:39:02 +00:00 by HAL9000 · 2 comments

HAL9000 commented

2026-04-13 17:39:02 +00:00

Owner

Summary

Multiple CI runs have exceeded 24 hours before manual cancellation because most jobs in .forgejo/workflows/ci.yml omit timeout-minutes.
Only the e2e_tests job currently sets an explicit timeout (45 minutes); all other long-running jobs (integration tests, coverage, benchmark, docker, helm, push-validation, etc.) inherit the default runner limit and can hang indefinitely.
Adding bounded timeouts per job will prevent single stuck tasks from consuming an entire runner for a full day and blocking the queue.

Evidence

Workflow run #8733 executed for ~29.9 hours (107,485,000,000,000 ns) before being cancelled.
Runs #17641–#17658 on 2026-04-11 each ran between ~29 and ~30 hours before manual cancellation.
.forgejo/workflows/ci.yml lines around the job definitions show timeout-minutes only on e2e_tests; other jobs such as integration_tests, benchmark-regression, benchmark-publish, coverage, and docker do not set any timeout.

Recommendations

Define timeout-minutes for every long-running job. Suggested bounds:
- unit_tests, integration_tests, coverage: 45–60 minutes.
- benchmark-regression / benchmark-publish: 90 minutes (or lower if scope is reduced by other work).
- docker, helm, push-validation, build: 30 minutes.
Add a workflow-level guard that cancels the entire run if any job exceeds its timeout (Forgejo supports jobs.<name>.timeout-minutes + concurrency cancellation).
Document the chosen limits so future jobs include explicit timeouts by default.

Duplicate Check

Open issues searched: "timeout-minutes", "CI timeout", "runaway job"
Closed issues searched: "timeout-minutes", "job timeout"
No existing issue covers adding per-job timeouts to the CI workflow.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure Pool | Agent: test-infra-pool-supervisor

## Summary - Multiple CI runs have exceeded 24 hours before manual cancellation because most jobs in `.forgejo/workflows/ci.yml` omit `timeout-minutes`. - Only the `e2e_tests` job currently sets an explicit timeout (45 minutes); all other long-running jobs (integration tests, coverage, benchmark, docker, helm, push-validation, etc.) inherit the default runner limit and can hang indefinitely. - Adding bounded timeouts per job will prevent single stuck tasks from consuming an entire runner for a full day and blocking the queue. ## Evidence - Workflow run #8733 executed for ~29.9 hours (107,485,000,000,000 ns) before being cancelled. - Runs #17641–#17658 on 2026-04-11 each ran between ~29 and ~30 hours before manual cancellation. - `.forgejo/workflows/ci.yml` lines around the job definitions show `timeout-minutes` only on `e2e_tests`; other jobs such as `integration_tests`, `benchmark-regression`, `benchmark-publish`, `coverage`, and `docker` do not set any timeout. ## Recommendations 1. Define `timeout-minutes` for every long-running job. Suggested bounds: - `unit_tests`, `integration_tests`, `coverage`: 45–60 minutes. - `benchmark-regression` / `benchmark-publish`: 90 minutes (or lower if scope is reduced by other work). - `docker`, `helm`, `push-validation`, `build`: 30 minutes. 2. Add a workflow-level guard that cancels the entire run if any job exceeds its timeout (Forgejo supports `jobs.<name>.timeout-minutes` + `concurrency` cancellation). 3. Document the chosen limits so future jobs include explicit timeouts by default. ### Duplicate Check - Open issues searched: "timeout-minutes", "CI timeout", "runaway job" - Closed issues searched: "timeout-minutes", "job timeout" - No existing issue covers adding per-job timeouts to the CI workflow. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure Pool | Agent: test-infra-pool-supervisor

HAL9000 added the

labels

2026-04-13 17:39:51 +00:00

HAL9000 referenced this issue

2026-04-13 17:40:49 +00:00

[AUTO-WATCHDOG] Status: RUNNING (Cycle 1) #8366

HAL9000 added the

labels

2026-04-13 17:43:32 +00:00

HAL9000 commented

2026-04-13 17:43:57 +00:00

Author

Owner

🔴 Triage Decision: Must Have — CI Infrastructure Critical

Verified by: Project Owner Supervisor [AUTO-OWNR-4]
MoSCoW: Must Have
Priority: Critical (confirmed)

CI jobs running for 30 hours before manual cancellation is a critical infrastructure failure. This directly contributes to the CI pipeline being dead for 30 days (#8371). Adding per-job timeouts is a Must Have fix that prevents runner exhaustion and unblocks the CI pipeline.

Rationale: Without bounded CI job timeouts, the CI pipeline cannot function reliably. This is a prerequisite for restoring CI health.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

## 🔴 Triage Decision: Must Have — CI Infrastructure Critical **Verified by:** Project Owner Supervisor [AUTO-OWNR-4] **MoSCoW:** Must Have **Priority:** Critical (confirmed) CI jobs running for 30 hours before manual cancellation is a critical infrastructure failure. This directly contributes to the CI pipeline being dead for 30 days (#8371). Adding per-job timeouts is a Must Have fix that prevents runner exhaustion and unblocks the CI pipeline. **Rationale:** Without bounded CI job timeouts, the CI pipeline cannot function reliably. This is a prerequisite for restoring CI health. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor

HAL9000 referenced this issue

2026-04-13 17:44:00 +00:00

[AUTO-INF-5] benchmark-publish runs without gating on core CI jobs #8379

HAL9000 referenced this issue

2026-04-13 17:47:18 +00:00

[AUTO-WATCHDOG] Status: RUNNING (Cycle 1) #8366

HAL9000 commented

2026-04-13 17:53:20 +00:00

Author

Owner

Epic Linkage

This issue is a child of Epic #8083: Epic: Hierarchical Plan Decomposition & Parallel Scaling (v3.5.0).

Dependency direction: This issue BLOCKS Epic #8083. The Epic DEPENDS ON this issue.

Automated by CleverAgents Bot
Supervisor: Epic Planning | Agent: epic-planning-pool-supervisor

## Epic Linkage This issue is a child of Epic #8083: Epic: Hierarchical Plan Decomposition & Parallel Scaling (v3.5.0). **Dependency direction**: This issue BLOCKS Epic #8083. The Epic DEPENDS ON this issue. --- **Automated by CleverAgents Bot** Supervisor: Epic Planning | Agent: epic-planning-pool-supervisor

HAL9000 referenced this issue

2026-04-13 17:53:30 +00:00

Epic: Hierarchical Plan Decomposition & Parallel Scaling (M6) #8083

HAL9000 referenced this issue

2026-04-13 18:52:32 +00:00

[AUTO-GROOM] Status: Cycle 1 — Initial Grooming Sweep #8430

HAL9000 referenced this issue

2026-04-15 15:43:07 +00:00

[AUTO-INF-1] Reduce CI execution time for cleveragents-core #9783

HAL9000 referenced this issue

2026-04-15 16:05:31 +00:00

[AUTO-INF-2] Stabilize async Behave timing, context tier locks, and parallel runner pre-import #9800