[AUTO-INF-16] Optimize CI Dependency Caching #8329

New issue

Closed

opened 2026-04-13 08:59:21 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-13 08:59:21 +00:00

Owner

Summary

Most CI jobs only cache uv packages keyed by pyproject.toml, which misses the actual dependency lockfile and allows stale caches when uv.lock changes (see .forgejo/workflows/ci.yml lines 30-37, repeated per job).
Every job reinstalls uv and nox via pip without persisting the pip download cache, so the same wheels are fetched 8+ times per run; .nox environments are rebuilt even though nox.options.reuse_existing_virtualenvs = True.
Nightly quality and release workflows skip any caching, so the long nox sequences and wheel builds always start from cold dependencies.

Current Caching Gaps

ci.yml restores only ~/.cache/uv with key: uv-${{ hashFiles('pyproject.toml') }}. When uv.lock or noxfile.py updates without touching pyproject.toml, the cache never invalidates, which risks mismatched environments and unnecessary cache misses.
The Install uv and nox steps rely on pip downloads without caching ~/.cache/pip, so each job re-downloads the same tooling despite identical ${{ env.UV_VERSION }} and Python versions.
.nox/ is recreated on every job because nothing persists it between runs; this eliminates the benefit of reuse_existing_virtualenvs=True in noxfile.py.
nightly-quality.yml and release.yml omit both the uv and pip cache restores, so nightly lint/type/typecheck/security/unit/coverage sessions and release wheel builds all resolve dependencies from scratch.

Recommendations

Expand the uv cache to include the lockfile and session sources: key: uv-${{ runner.os }}-${{ env.PYTHON_VERSION }}-${{ hashFiles('uv.lock', 'pyproject.toml', 'noxfile.py') }} and set path to both ~/.cache/uv and .nox/ so virtualenvs survive across CI jobs. Keep a restore-keys prefix for branch fallbacks.
Add a pip cache step (path: ~/.cache/pip) keyed by ${{ runner.os }}-${{ env.PYTHON_VERSION }}-${{ env.UV_VERSION }} before installing tooling, or switch to uv tool install nox after restoring the uv cache so the install reuses that cache.
Factor the cache restore + tooling install into a reusable workflow composite or a setup-python-style job and make downstream lint/typecheck/security/quality/test jobs depend on it. That way the large toolchain downloads happen once per workflow instead of per job.
Mirror the uv + pip caching block (and the reusable step from #3) inside nightly-quality.yml and release.yml so nightly suites and release builds benefit from warm caches as well.

Expected Benefits

Lock-aware keys raise cache hit rates after dependency updates and prevent stale environments when only uv.lock changes.
Persisting ~/.cache/pip and .nox/ should cut the setup phase for each job by 1-2 minutes and save repeated 100+ MB downloads across the workflow.
Nightly and release pipelines reuse the same caches, shortening end-to-end runtime and reducing failures caused by transient network hiccups while pulling dependencies.

Duplicate Check

Queried open issues for "AUTO-INF-16" and "Optimize CI Dependency Caching" on 2026-04-13 via Forgejo API; no existing issue found.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure Pool | Agent: test-infra-worker

## Summary - Most CI jobs only cache uv packages keyed by `pyproject.toml`, which misses the actual dependency lockfile and allows stale caches when `uv.lock` changes (see `.forgejo/workflows/ci.yml` lines 30-37, repeated per job). - Every job reinstalls `uv` and `nox` via pip without persisting the pip download cache, so the same wheels are fetched 8+ times per run; `.nox` environments are rebuilt even though `nox.options.reuse_existing_virtualenvs = True`. - Nightly quality and release workflows skip any caching, so the long nox sequences and wheel builds always start from cold dependencies. ## Current Caching Gaps - `ci.yml` restores only `~/.cache/uv` with `key: uv-${{ hashFiles('pyproject.toml') }}`. When `uv.lock` or `noxfile.py` updates without touching `pyproject.toml`, the cache never invalidates, which risks mismatched environments and unnecessary cache misses. - The `Install uv and nox` steps rely on pip downloads without caching `~/.cache/pip`, so each job re-downloads the same tooling despite identical `${{ env.UV_VERSION }}` and Python versions. - `.nox/` is recreated on every job because nothing persists it between runs; this eliminates the benefit of `reuse_existing_virtualenvs=True` in `noxfile.py`. - `nightly-quality.yml` and `release.yml` omit both the uv and pip cache restores, so nightly lint/type/typecheck/security/unit/coverage sessions and release wheel builds all resolve dependencies from scratch. ## Recommendations 1. Expand the uv cache to include the lockfile and session sources: `key: uv-${{ runner.os }}-${{ env.PYTHON_VERSION }}-${{ hashFiles('uv.lock', 'pyproject.toml', 'noxfile.py') }}` and set `path` to both `~/.cache/uv` and `.nox/` so virtualenvs survive across CI jobs. Keep a `restore-keys` prefix for branch fallbacks. 2. Add a pip cache step (`path: ~/.cache/pip`) keyed by `${{ runner.os }}-${{ env.PYTHON_VERSION }}-${{ env.UV_VERSION }}` before installing tooling, or switch to `uv tool install nox` after restoring the uv cache so the install reuses that cache. 3. Factor the cache restore + tooling install into a reusable workflow composite or a `setup-python`-style job and make downstream lint/typecheck/security/quality/test jobs depend on it. That way the large toolchain downloads happen once per workflow instead of per job. 4. Mirror the uv + pip caching block (and the reusable step from #3) inside `nightly-quality.yml` and `release.yml` so nightly suites and release builds benefit from warm caches as well. ## Expected Benefits - Lock-aware keys raise cache hit rates after dependency updates and prevent stale environments when only `uv.lock` changes. - Persisting `~/.cache/pip` and `.nox/` should cut the setup phase for each job by 1-2 minutes and save repeated 100+ MB downloads across the workflow. - Nightly and release pipelines reuse the same caches, shortening end-to-end runtime and reducing failures caused by transient network hiccups while pulling dependencies. ### Duplicate Check - [x] Queried open issues for "AUTO-INF-16" and "Optimize CI Dependency Caching" on 2026-04-13 via Forgejo API; no existing issue found. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure Pool | Agent: test-infra-worker

Rows
Columns