TEST-INFRA: [ci-pipeline-design] Implement Docker layer caching #1886

Open
opened 2026-04-03 00:06:32 +00:00 by freemo · 1 comment
Owner

Metadata

  • Branch: task/ci-docker-layer-caching
  • Commit Message: chore(ci): implement Docker layer caching to reduce image build times
  • Milestone: v3.8.0
  • Parent Epic: #1678

Background and Context

The CI pipeline currently builds Docker images from scratch on every run. Each build re-downloads and re-installs all base layers, OS packages, and Python dependencies — even when none of those inputs have changed. This results in unnecessarily long build times, increased runner resource consumption, and slower feedback loops for developers.

Docker layer caching is a well-established technique that stores intermediate image layers between builds. When a layer's inputs (e.g., requirements.txt, pyproject.toml, base image digest) have not changed, the cached layer is reused rather than rebuilt. On a warm cache hit, image build time can drop from several minutes to seconds.

This issue tracks the work to introduce Docker layer caching into the .forgejo/workflows/ci.yml pipeline, specifically for the docker job that builds and optionally pushes the project's container image.

Current Behavior

  • The docker job in .forgejo/workflows/ci.yml builds the Docker image without any cache configuration.
  • Every CI run performs a full image build: base image pull, OS package installation, Python dependency installation, and source copy — regardless of whether any of those layers have changed.
  • No cache-from or cache-to directives are present in the build step.
  • Build times for the Docker image are consistently high, adding unnecessary latency to every CI run.

Expected Behavior

  • The docker job uses Docker's built-in layer cache, backed by the Forgejo Actions cache store (via type=gha) or a registry-based cache (via type=registry).
  • On a cache hit (no changes to Dockerfile, pyproject.toml, uv.lock, or base image), the image build completes in seconds rather than minutes.
  • On a cache miss (any of the above inputs changed), the build proceeds normally and the new layers are stored for future runs.
  • The cache strategy is documented in a comment in ci.yml so future contributors understand the approach.
  • The Dockerfile is structured to maximise cache reuse: dependency installation layers appear before the source-copy layer, so a code-only change does not invalidate the dependency cache.

Acceptance Criteria

  • The docker job in .forgejo/workflows/ci.yml includes cache-from and cache-to directives on the build step.
  • The cache backend is type=gha (GitHub/Forgejo Actions cache) or type=registry with a clearly documented rationale for the choice.
  • The Dockerfile orders layers so that infrequently-changing layers (base image, OS packages, Python dependencies) precede frequently-changing layers (application source code).
  • A comment in ci.yml explains the caching strategy and the cache key inputs.
  • The CI pipeline passes end-to-end with the new caching configuration on a clean runner.
  • A second CI run (with no source changes) demonstrates a measurable reduction in docker job wall-clock time (cache hit path).
  • All existing nox stages continue to pass after the changes.
  • Test coverage remains ≥ 97%.

Supporting Information

  • Parent Epic: #1678 — CI Execution Time Optimization (timeouts, concurrency, artifact sharing)
  • Related: Docker BuildKit cache documentation — https://docs.docker.com/build/cache/backends/
  • Related: Forgejo/GitHub Actions cache backend for Docker Buildx — type=gha
  • The Dockerfile and .forgejo/workflows/ci.yml are the primary files affected.
  • This task is complementary to the concurrency groups, job timeouts, and coverage artifact sharing tracked in the parent Epic.

Subtasks

  • Audit the current Dockerfile layer order; restructure if dependency layers are not before source-copy layers
  • Add cache-from: type=gha and cache-to: type=gha,mode=max (or equivalent registry-based cache) to the Docker build step in .forgejo/workflows/ci.yml
  • Add an explanatory comment in ci.yml describing the cache strategy and which inputs invalidate each layer
  • Update features/ci_workflow_validation.feature to assert that cache-from and cache-to are present in the docker build step
  • Run nox (all default sessions) locally and fix any errors
  • Verify coverage ≥ 97% via nox -s coverage_report
  • Perform a second CI run with no source changes and confirm the docker job is faster (cache hit)

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly (chore(ci): implement Docker layer caching to reduce image build times), followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly (task/ci-docker-layer-caching).
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
  • All nox stages pass.
  • Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: Unknown | Agent: ca-new-issue-creator

## Metadata - **Branch**: `task/ci-docker-layer-caching` - **Commit Message**: `chore(ci): implement Docker layer caching to reduce image build times` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context The CI pipeline currently builds Docker images from scratch on every run. Each build re-downloads and re-installs all base layers, OS packages, and Python dependencies — even when none of those inputs have changed. This results in unnecessarily long build times, increased runner resource consumption, and slower feedback loops for developers. Docker layer caching is a well-established technique that stores intermediate image layers between builds. When a layer's inputs (e.g., `requirements.txt`, `pyproject.toml`, base image digest) have not changed, the cached layer is reused rather than rebuilt. On a warm cache hit, image build time can drop from several minutes to seconds. This issue tracks the work to introduce Docker layer caching into the `.forgejo/workflows/ci.yml` pipeline, specifically for the `docker` job that builds and optionally pushes the project's container image. ## Current Behavior - The `docker` job in `.forgejo/workflows/ci.yml` builds the Docker image without any cache configuration. - Every CI run performs a full image build: base image pull, OS package installation, Python dependency installation, and source copy — regardless of whether any of those layers have changed. - No `cache-from` or `cache-to` directives are present in the build step. - Build times for the Docker image are consistently high, adding unnecessary latency to every CI run. ## Expected Behavior - The `docker` job uses Docker's built-in layer cache, backed by the Forgejo Actions cache store (via `type=gha`) or a registry-based cache (via `type=registry`). - On a cache hit (no changes to `Dockerfile`, `pyproject.toml`, `uv.lock`, or base image), the image build completes in seconds rather than minutes. - On a cache miss (any of the above inputs changed), the build proceeds normally and the new layers are stored for future runs. - The cache strategy is documented in a comment in `ci.yml` so future contributors understand the approach. - The `Dockerfile` is structured to maximise cache reuse: dependency installation layers appear before the source-copy layer, so a code-only change does not invalidate the dependency cache. ## Acceptance Criteria - [ ] The `docker` job in `.forgejo/workflows/ci.yml` includes `cache-from` and `cache-to` directives on the build step. - [ ] The cache backend is `type=gha` (GitHub/Forgejo Actions cache) or `type=registry` with a clearly documented rationale for the choice. - [ ] The `Dockerfile` orders layers so that infrequently-changing layers (base image, OS packages, Python dependencies) precede frequently-changing layers (application source code). - [ ] A comment in `ci.yml` explains the caching strategy and the cache key inputs. - [ ] The CI pipeline passes end-to-end with the new caching configuration on a clean runner. - [ ] A second CI run (with no source changes) demonstrates a measurable reduction in `docker` job wall-clock time (cache hit path). - [ ] All existing nox stages continue to pass after the changes. - [ ] Test coverage remains ≥ 97%. ## Supporting Information - Parent Epic: #1678 — CI Execution Time Optimization (timeouts, concurrency, artifact sharing) - Related: Docker BuildKit cache documentation — https://docs.docker.com/build/cache/backends/ - Related: Forgejo/GitHub Actions cache backend for Docker Buildx — `type=gha` - The `Dockerfile` and `.forgejo/workflows/ci.yml` are the primary files affected. - This task is complementary to the concurrency groups, job timeouts, and coverage artifact sharing tracked in the parent Epic. ## Subtasks - [ ] Audit the current `Dockerfile` layer order; restructure if dependency layers are not before source-copy layers - [ ] Add `cache-from: type=gha` and `cache-to: type=gha,mode=max` (or equivalent registry-based cache) to the Docker build step in `.forgejo/workflows/ci.yml` - [ ] Add an explanatory comment in `ci.yml` describing the cache strategy and which inputs invalidate each layer - [ ] Update `features/ci_workflow_validation.feature` to assert that `cache-from` and `cache-to` are present in the docker build step - [ ] Run `nox` (all default sessions) locally and fix any errors - [ ] Verify coverage ≥ 97% via `nox -s coverage_report` - [ ] Perform a second CI run with no source changes and confirm the `docker` job is faster (cache hit) ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly (`chore(ci): implement Docker layer caching to reduce image build times`), followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly (`task/ci-docker-layer-caching`). - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: Unknown | Agent: ca-new-issue-creator
freemo added this to the v3.8.0 milestone 2026-04-03 00:07:11 +00:00
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • MoSCoW: MoSCoW/Could Have — CI/test infrastructure improvement.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **MoSCoW**: MoSCoW/Could Have — CI/test infrastructure improvement. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#1886
No description provided.