[AUTO-INF-1] CI Execution Time: instrument CI telemetry for queue and job durations #9148

New issue

Open

opened 2026-04-14 08:36:19 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-14 08:36:19 +00:00

Owner

Problem

Recent Forgejo Actions data shows wide variance in CI wall time (max 131.97 minutes, 75th percentile 13.47 minutes across the latest 50 runs), but we do not persist these measurements anywhere. Once a run scrolls out of the Actions UI, the trend is lost, making it difficult to quantify the impact of optimizations such as #9040 or #8244.
Queue time is currently anecdotal (e.g., docker runners regularly queue for 20–38 minutes), so we cannot tell whether changes like docker job isolation actually improved latency.
Without durable telemetry we cannot answer "which jobs regressed this week?" or "did the benchmark gating change reduce median run time?" — decisions rely on manual log spelunking.

Proposed Solution

Add a lightweight telemetry step that runs after status-check and captures per-run metadata via forgejo/actions/runs/<run_id> (queue duration, total runtime, per-job duration) and writes it to an artifact in NDJSON or CSV format.
Append that artifact to a long-lived object (e.g., S3 bucket or repo-hosted JSON) so we maintain a historical record; optionally expose a simple dashboard (Superset/Grafana or even Google Sheets) fed by the same data.
Document the telemetry pipeline in docs/development/ci-cd.md so future CI changes include instrumentation updates, and add an alert (Slack/webhook) when median wall time crosses a configured threshold.

Duplicate Check

GET /api/v1/repos/cleveragents/cleveragents-core/issues?state=open&q=telemetry → no telemetry/metrics optimisation issues.
GET /api/v1/repos/cleveragents/cleveragents-core/issues?state=open&q="ci telemetry" → no matching issues.
GET /api/v1/repos/cleveragents/cleveragents-core/issues?state=closed&q=telemetry → only status/cleanup tickets (#8687, #8313, #500) unrelated to CI metrics.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure Pool | Agent: test-infra-pool-supervisor

## Problem - Recent Forgejo Actions data shows wide variance in CI wall time (max 131.97 minutes, 75th percentile 13.47 minutes across the latest 50 runs), but we do not persist these measurements anywhere. Once a run scrolls out of the Actions UI, the trend is lost, making it difficult to quantify the impact of optimizations such as #9040 or #8244. - Queue time is currently anecdotal (e.g., docker runners regularly queue for 20–38 minutes), so we cannot tell whether changes like docker job isolation actually improved latency. - Without durable telemetry we cannot answer "which jobs regressed this week?" or "did the benchmark gating change reduce median run time?" — decisions rely on manual log spelunking. ## Proposed Solution 1. Add a lightweight telemetry step that runs after `status-check` and captures per-run metadata via `forgejo/actions/runs/<run_id>` (queue duration, total runtime, per-job duration) and writes it to an artifact in NDJSON or CSV format. 2. Append that artifact to a long-lived object (e.g., S3 bucket or repo-hosted JSON) so we maintain a historical record; optionally expose a simple dashboard (Superset/Grafana or even Google Sheets) fed by the same data. 3. Document the telemetry pipeline in `docs/development/ci-cd.md` so future CI changes include instrumentation updates, and add an alert (Slack/webhook) when median wall time crosses a configured threshold. ### Duplicate Check - `GET /api/v1/repos/cleveragents/cleveragents-core/issues?state=open&q=telemetry` → no telemetry/metrics optimisation issues. - `GET /api/v1/repos/cleveragents/cleveragents-core/issues?state=open&q="ci telemetry"` → no matching issues. - `GET /api/v1/repos/cleveragents/cleveragents-core/issues?state=closed&q=telemetry` → only status/cleanup tickets (#8687, #8313, #500) unrelated to CI metrics. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure Pool | Agent: test-infra-pool-supervisor

HAL9000 added the

MoSCoW

Could have

Rows
Columns