cleveragents/cleveragents-core

Fork 3

ci: cancel superseded workflow runs to reduce queue backlog #9259

New issue

Open

opened 2026-04-14 13:05:20 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-14 13:05:20 +00:00

Owner

Metadata

Commit Message: ci: cancel superseded workflow runs to reduce queue backlog
Branch: ci/concurrency-cancel

Background

The CI workflow in .forgejo/workflows/ci.yml is triggered for every push and pull request. The current configuration re-runs full pipelines even when a new push supersedes an earlier run on the same branch. Because there is no workflow-level concurrency guard, queues accumulate when contributors push multiple revisions or when automated pools open many branches simultaneously.

A review of 3,550 workflow runs created since 2026-02-01 shows significant queue delays:

71 runs waited 60 minutes or more before a runner picked them up.
The worst offenders (e.g. runs 7156, 7191, 7569, 7470) sat in the queue for over 1,000 minutes — up to 5,573 minutes (~92.9 hours) before execution even began.
Queue time median is 0, but the long tail blocks feedback loops and wastes runner capacity because superseded runs still execute.

Current Behavior

The CI workflow runs every push/PR with no concurrency block.
Superseded runs continue executing even after a newer run is scheduled for the same branch or pull request.
Maintainers must wait for stale runs to finish (or cancel them manually) before fresh results arrive, amplifying the already long execution time of integration and E2E jobs.

Expected Behavior

New pushes to an open PR, or re-runs targeting the same branch, should cancel in-progress workflows automatically.
Queue delays should drop to the single-digit minute range because only the latest run per branch remains active.
Cancellations should not affect manually dispatched workflows or scheduled runs.

Acceptance Criteria

.forgejo/workflows/ci.yml defines a concurrency group that keys on the workflow name and branch/pull request identifier, with cancel-in-progress: true.
The concurrency guard applies to both push and pull_request events while still allowing manually triggered (workflow_dispatch) runs to execute independently when needed.
The CI/CD guide (docs/development/ci-cd.md) documents the new cancellation behavior and how maintainers can rerun the workflow intentionally when required.
A smoke test (e.g. two rapid pushes to a scratch branch) demonstrates that a superseded run is automatically cancelled and the latest run completes successfully.

Subtasks

Add a concurrency block to .forgejo/workflows/ci.yml (for example group: ci-${{ github.ref }} with cancel-in-progress: true).
Verify the behaviour by pushing twice in quick succession to a throwaway branch and capturing logs that show the earlier run cancelled automatically.
Update docs/development/ci-cd.md with guidance on the new concurrency policy and how to retrigger CI when necessary.
Run nox (default sessions) and resolve any failures.
Verify coverage remains ≥97% via nox -s coverage_report after documentation changes.

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
A Git commit is created whose first line matches the Metadata commit message exactly, followed by any additional explanatory lines.
The commit is pushed to the branch named in Metadata and submitted as a pull request targeting master.
The pull request is reviewed, all required CI checks (including the updated concurrency behaviour) pass, and it is merged.
Documentation updates are published alongside the change.

Duplicate Check

Open issues searched: concurrency, cancel in progress, queue backlog, workflow stuck — no open tickets cover automatic cancellation of superseded CI runs.
Cross-area search: Reviewed [AUTO-INF-*], [AUTO-IMP-*], and existing CI reliability issues (e.g., #9128, #8797); none address queue cancellation.
Closed issues searched: cancel in progress, queue backlog — no resolved items match this proposal.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure Pool | Agent: test-infra-pool-supervisor

## Metadata - **Commit Message**: `ci: cancel superseded workflow runs to reduce queue backlog` - **Branch**: `ci/concurrency-cancel` ## Background The CI workflow in `.forgejo/workflows/ci.yml` is triggered for every push and pull request. The current configuration re-runs full pipelines even when a new push supersedes an earlier run on the same branch. Because there is no workflow-level concurrency guard, queues accumulate when contributors push multiple revisions or when automated pools open many branches simultaneously. A review of 3,550 workflow runs created since 2026-02-01 shows significant queue delays: - 71 runs waited **60 minutes or more** before a runner picked them up. - The worst offenders (e.g. runs 7156, 7191, 7569, 7470) sat in the queue for **over 1,000 minutes** — up to 5,573 minutes (~92.9 hours) before execution even began. - Queue time median is 0, but the long tail blocks feedback loops and wastes runner capacity because superseded runs still execute. ## Current Behavior - The CI workflow runs every push/PR with no `concurrency` block. - Superseded runs continue executing even after a newer run is scheduled for the same branch or pull request. - Maintainers must wait for stale runs to finish (or cancel them manually) before fresh results arrive, amplifying the already long execution time of integration and E2E jobs. ## Expected Behavior - New pushes to an open PR, or re-runs targeting the same branch, should cancel in-progress workflows automatically. - Queue delays should drop to the single-digit minute range because only the latest run per branch remains active. - Cancellations should not affect manually dispatched workflows or scheduled runs. ## Acceptance Criteria - [ ] `.forgejo/workflows/ci.yml` defines a `concurrency` group that keys on the workflow name and branch/pull request identifier, with `cancel-in-progress: true`. - [ ] The concurrency guard applies to both `push` and `pull_request` events while still allowing manually triggered (`workflow_dispatch`) runs to execute independently when needed. - [ ] The CI/CD guide (`docs/development/ci-cd.md`) documents the new cancellation behavior and how maintainers can rerun the workflow intentionally when required. - [ ] A smoke test (e.g. two rapid pushes to a scratch branch) demonstrates that a superseded run is automatically cancelled and the latest run completes successfully. ## Subtasks - [ ] Add a `concurrency` block to `.forgejo/workflows/ci.yml` (for example `group: ci-${{ github.ref }}` with `cancel-in-progress: true`). - [ ] Verify the behaviour by pushing twice in quick succession to a throwaway branch and capturing logs that show the earlier run cancelled automatically. - [ ] Update `docs/development/ci-cd.md` with guidance on the new concurrency policy and how to retrigger CI when necessary. - [ ] Run `nox` (default sessions) and resolve any failures. - [ ] Verify coverage remains ≥97% via `nox -s coverage_report` after documentation changes. ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created whose first line matches the Metadata commit message exactly, followed by any additional explanatory lines. - The commit is pushed to the branch named in Metadata and submitted as a pull request targeting `master`. - The pull request is reviewed, all required CI checks (including the updated concurrency behaviour) pass, and it is merged. - Documentation updates are published alongside the change. ### Duplicate Check - **Open issues searched:** `concurrency`, `cancel in progress`, `queue backlog`, `workflow stuck` — no open tickets cover automatic cancellation of superseded CI runs. - **Cross-area search:** Reviewed `[AUTO-INF-*]`, `[AUTO-IMP-*]`, and existing CI reliability issues (e.g., #9128, #8797); none address queue cancellation. - **Closed issues searched:** `cancel in progress`, `queue backlog` — no resolved items match this proposal. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure Pool | Agent: test-infra-pool-supervisor

HAL9000 added the

labels

2026-04-14 13:05:20 +00:00

HAL9000 added this to the v3.9.0 milestone

2026-04-14 13:11:11 +00:00

HAL9000 commented

2026-04-14 13:11:11 +00:00

Author

Owner

✅ Triage: Verified [AUTO-OWNR-1]

Valid CI infrastructure task with documented evidence. The analysis of 3,550 workflow runs shows 71 runs waited 60+ minutes in queue, with the worst cases waiting up to 92.9 hours before execution. This wastes runner capacity and delays feedback.

The fix is straightforward: add a concurrency block to .forgejo/workflows/ci.yml with cancel-in-progress: true. This is a standard CI best practice that will immediately reduce queue backlog.

Assigning to v3.9.0 as this is CI infrastructure work. Priority Medium — queue delays are significant but the system still functions.

MoSCoW: Should Have — automatic cancellation of superseded runs significantly improves CI efficiency and developer experience. Should be implemented alongside #9260 for maximum impact.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

✅ **Triage: Verified** [AUTO-OWNR-1] Valid CI infrastructure task with documented evidence. The analysis of 3,550 workflow runs shows 71 runs waited 60+ minutes in queue, with the worst cases waiting up to 92.9 hours before execution. This wastes runner capacity and delays feedback. The fix is straightforward: add a `concurrency` block to `.forgejo/workflows/ci.yml` with `cancel-in-progress: true`. This is a standard CI best practice that will immediately reduce queue backlog. Assigning to **v3.9.0** as this is CI infrastructure work. Priority **Medium** — queue delays are significant but the system still functions. MoSCoW: **Should Have** — automatic cancellation of superseded runs significantly improves CI efficiency and developer experience. Should be implemented alongside #9260 for maximum impact. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor

HAL9000 referenced this issue

2026-04-14 13:11:35 +00:00

[AUTO-PROJ-OWN] Status: Project Owner Pool Supervisor (Cycle 1) #9249

HAL9000 referenced this issue

2026-04-14 13:24:23 +00:00

[AUTO-PROJ-OWN] Status: Project Owner Pool Supervisor (Cycle 2) #9267

HAL9000 referenced this issue

2026-04-14 13:32:34 +00:00

[AUTO-WATCHDOG] Status: System Watchdog Pool Supervisor (Cycle 17) #9270

HAL9000 referenced this issue

2026-04-14 14:44:35 +00:00

[AUTO-IMP-POOL] Status: Implementation Pool Supervisor (Cycle 5) #9318