Lab: Reproducible Container Builds — Pinning, Verifying, and Diffing Images

by

Overview

If you build the same Dockerfile twice and get different images, you cannot verify build integrity. A non-reproducible build means you have no way to confirm that the artifact running in production was actually produced from the source code you audited. Attackers can exploit this ambiguity to inject malicious code during the build process without detection.

This lab walks you through the sources of non-reproducibility in container builds, demonstrates techniques to eliminate each one, and shows how to verify reproducibility automatically in CI/CD pipelines. By the end, you will have a fully reproducible Dockerfile and a GitHub Actions workflow that proves it on every commit.

Prerequisites

  • Docker with BuildKit — Docker Desktop 23.0+ has BuildKit enabled by default. Verify with docker buildx version.
  • diffoscope — Install with pip install diffoscope. This tool performs deep, recursive comparison of files and archives.
  • crane — Install from go-containerregistry. Used to inspect and manipulate container images and registries.
  • Cosign — Install from Sigstore. Used for container image signing and verification.
  • A test repository with a Dockerfile (we will create one in the setup step).
  • Go 1.22+ installed locally (optional, for local testing outside Docker).

Environment Setup

Create a fresh test repository with a simple Go application. This gives us a realistic, minimal project to work with throughout the lab.

Step 1: Initialize the project

mkdir repro-build-lab && cd repro-build-lab
git init
go mod init github.com/example/repro-build-lab

Step 2: Create the Go application

Create cmd/app/main.go:

package main

import (
	"fmt"
	"net/http"
	"os"
)

func main() {
	port := os.Getenv("PORT")
	if port == "" {
		port = "8080"
	}

	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintf(w, "Hello from repro-build-lab v1\n")
	})

	http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusOK)
		fmt.Fprintf(w, "ok\n")
	})

	fmt.Printf("Listening on :%s\n", port)
	http.ListenAndServe(":"+port, nil)
}

Step 3: Create the intentionally non-reproducible Dockerfile

This Dockerfile contains every common mistake that leads to non-reproducible builds:

# Intentionally non-reproducible Dockerfile
FROM golang:latest

WORKDIR /src

# Floating package versions
RUN apt-get update && apt-get install -y curl

# Embeds current timestamp into the image
RUN echo "Built at $(date)" > /build-info

COPY . .

RUN go build -o /app ./cmd/app

EXPOSE 8080
CMD ["/app"]

Notice the problems:

  • FROM golang:latest — the base image changes without warning.
  • apt-get install -y curl — no version pin, so the installed version floats.
  • echo "Built at $(date)" — injects a timestamp that is different on every build.
  • No .dockerignore — local files like .git/ leak into the build context, changing layer hashes.

Commit the initial project:

git add -A
git commit -m "Initial non-reproducible project"

Exercise 1: Demonstrate Non-Reproducibility

Before fixing anything, let us prove that the current Dockerfile produces different images on every build.

Step 1: Build the image twice

# First build
docker build --no-cache -t myapp:build1 .

# Wait a moment so the timestamp differs
sleep 2

# Second build
docker build --no-cache -t myapp:build2 .

The --no-cache flag forces Docker to execute every layer from scratch, which is essential for this comparison. In a real CI/CD environment, builds often run on fresh runners with no cache.

Step 2: Compare image digests

docker inspect --format='{{.Id}}' myapp:build1
# sha256:a1b2c3d4e5f6... (example)

docker inspect --format='{{.Id}}' myapp:build2
# sha256:f6e5d4c3b2a1... (different!)

The digests are different even though nothing in the source code changed. This means you cannot verify that a given image was produced from a specific commit.

Step 3: Use diffoscope to identify what differs

# Export both images as tarballs
docker save myapp:build1 -o build1.tar
docker save myapp:build2 -o build2.tar

# Run diffoscope
diffoscope build1.tar build2.tar --html-dir diff-report

Open diff-report/index.html in a browser. The report reveals exactly what differs between the two builds:

  • Timestamps — the /build-info file contains different dates.
  • apt package metadata — package lists and cache files contain timestamps and may pull different micro-versions.
  • Go binary — the compiled binary contains embedded build paths and build IDs.
  • Layer ordering and metadata — Docker embeds creation timestamps in layer metadata.

Each of these is a source of non-reproducibility that we will eliminate in the following exercises.

Exercise 2: Pin the Base Image by Digest

The biggest source of drift is the base image. golang:latest is a moving target — it can change between builds, between CI runs, or even between regions if a registry is eventually consistent.

Step 1: Find the current digest

crane digest golang:1.22
# sha256:d0902bacefdde1cf45079803dc16feeb58f3aa9df52052cc00deb2c3e5de367b

Step 2: Pin the base image

Update the FROM line in the Dockerfile:

FROM golang:1.22@sha256:d0902bacefdde1cf45079803dc16feeb58f3aa9df52052cc00deb2c3e5de367b

The format is image:tag@sha256:digest. Docker will pull by digest, ignoring the tag. The tag is kept for human readability.

Step 3: Rebuild and compare

docker build --no-cache -t myapp:pinned1 .
sleep 2
docker build --no-cache -t myapp:pinned2 .

docker inspect --format='{{.Id}}' myapp:pinned1
docker inspect --format='{{.Id}}' myapp:pinned2

The digests are still different — other sources of non-reproducibility remain. But if you compare layers, the base image layer is now identical between builds. You have eliminated the largest source of drift.

Why this matters

Without digest pinning, a compromised or hijacked tag can silently replace your base image with a malicious one. Digest pinning is a cryptographic guarantee: you get exactly the bytes you expect, or the build fails.

Exercise 3: Pin Package Versions

Floating package versions introduce non-determinism in the dependency layer. Every time apt-get update runs, it fetches the current repository index, which may list different package versions.

Option A: Pin Debian package versions

RUN apt-get update && \
    apt-get install -y --no-install-recommends \
      curl=7.88.1-10+deb12u8 && \
    rm -rf /var/lib/apt/lists/*

To find the current version available in your base image:

docker run --rm golang:1.22 apt-cache policy curl

Option B: Use Alpine with pinned packages

Alpine packages have simpler version strings and smaller images:

FROM golang:1.22-alpine@sha256:<alpine-digest>

RUN apk add --no-cache curl=8.5.0-r0

Option C: Multi-stage build (preferred)

The best approach is to avoid installing packages in the final image entirely. Use a multi-stage build where the build stage has the tools and the runtime stage is minimal:

# Build stage — tools are only needed here
FROM golang:1.22@sha256:d0902bacefdde1cf45079803dc16feeb58f3aa9df52052cc00deb2c3e5de367b AS builder
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go build -o /app ./cmd/app

# Runtime stage — no apt-get, no floating packages
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=builder /app /app
CMD ["/app"]

With this approach, the runtime image has zero package manager calls, which eliminates an entire class of non-reproducibility.

Rebuild and compare

docker build --no-cache -t myapp:pinpkg1 .
sleep 2
docker build --no-cache -t myapp:pinpkg2 .

docker inspect --format='{{.Id}}' myapp:pinpkg1
docker inspect --format='{{.Id}}' myapp:pinpkg2

The package layers are now identical between builds. The remaining differences come from timestamps and the Go binary itself.

Exercise 4: Remove Timestamps and Non-Deterministic Content

Timestamps are the most obvious source of non-reproducibility. Any command that captures the current time produces a different result on every build.

Step 1: Remove explicit timestamps

Delete the line that writes the build time:

# REMOVE this line:
# RUN echo "Built at $(date)" > /build-info

If you need build metadata, pass it as a label with a fixed value derived from the source:

ARG BUILD_COMMIT
LABEL org.opencontainers.image.revision=${BUILD_COMMIT}

Step 2: Set SOURCE_DATE_EPOCH

SOURCE_DATE_EPOCH is a standardized environment variable that tells build tools to use a fixed timestamp instead of the current time. Many tools respect it, including tar, gzip, zip, and Go’s compiler.

ARG SOURCE_DATE_EPOCH
ENV SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH}

Build with the timestamp of the last git commit:

docker build \
  --build-arg SOURCE_DATE_EPOCH=$(git log -1 --format=%ct) \
  --no-cache \
  -t myapp:repro .

This ensures that builds from the same commit always use the same timestamp, regardless of when the build actually runs.

Step 3: Use BuildKit OCI output

BuildKit can produce OCI-format images with more deterministic layer creation:

docker buildx build \
  --build-arg SOURCE_DATE_EPOCH=$(git log -1 --format=%ct) \
  --output type=oci,dest=myapp.tar \
  --no-cache \
  .

The OCI output format avoids some of the non-deterministic metadata that the default Docker image format includes.

Exercise 5: Reproducible Go Builds

Go embeds several pieces of non-deterministic information in compiled binaries by default: local file paths, a unique build ID, and debug symbols that reference the build environment.

Step 1: Use reproducible build flags

RUN CGO_ENABLED=0 go build \
    -trimpath \
    -ldflags="-s -w -buildid=" \
    -o /app ./cmd/app

Here is what each flag does:

Flag Purpose
CGO_ENABLED=0 Disables cgo, producing a statically linked binary. Avoids dependency on system C libraries that may differ between builds.
-trimpath Removes all local file system paths from the compiled binary. Without this, the binary contains paths like /src/cmd/app/main.go from the build environment.
-ldflags="-s -w" Strips the symbol table (-s) and DWARF debug information (-w). These contain build-environment-specific data.
-ldflags="-buildid=" Sets the build ID to empty. Go normally generates a unique build ID that changes between builds even with identical source.

Step 2: Verify binary reproducibility

# Build twice
docker build --no-cache -t myapp:go1 .
docker build --no-cache -t myapp:go2 .

# Extract and hash the binary from each image
docker create --name tmp1 myapp:go1
docker cp tmp1:/app ./app1
docker rm tmp1

docker create --name tmp2 myapp:go2
docker cp tmp2:/app ./app2
docker rm tmp2

sha256sum app1 app2

The SHA-256 hashes of app1 and app2 should be identical. The Go binary is now bit-for-bit reproducible.

Exercise 6: The Fully Reproducible Dockerfile

Now let us combine every technique into a single, fully reproducible Dockerfile.

The complete Dockerfile

# syntax=docker/dockerfile:1

# ---- Build Stage ----
FROM golang:1.22@sha256:d0902bacefdde1cf45079803dc16feeb58f3aa9df52052cc00deb2c3e5de367b AS builder

ARG SOURCE_DATE_EPOCH
ENV SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH}

WORKDIR /src

# Cache dependency downloads
COPY go.mod go.sum ./
RUN go mod download && go mod verify

# Copy source and build
COPY . .
RUN CGO_ENABLED=0 go build \
    -trimpath \
    -ldflags="-s -w -buildid=" \
    -o /app ./cmd/app

# ---- Runtime Stage ----
FROM gcr.io/distroless/static-debian12:nonroot@sha256:6ec5aa99dc335b19f6c2bcb8e09cf92404e56f0db4e2f58cf92c4536e1548415

ARG SOURCE_DATE_EPOCH
ENV SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH}

COPY --from=builder /app /app

USER nonroot:nonroot
EXPOSE 8080
ENTRYPOINT ["/app"]

The complete .dockerignore

.git
.github
.gitignore
*.md
README*
LICENSE
docker-compose*.yml
Makefile
.env
.env.*
*.tar
*.log
tmp/
build/
diff-report/

The .dockerignore is critical. Without it, the .git/ directory leaks into the build context. Since .git/ contains timestamps, lock files, and other changing metadata, it makes every build context unique even when the source is identical.

Build and verify

SOURCE_EPOCH=$(git log -1 --format=%ct)

# Build twice
docker buildx build \
  --build-arg SOURCE_DATE_EPOCH=${SOURCE_EPOCH} \
  --output type=oci,dest=build1.tar \
  --no-cache .

docker buildx build \
  --build-arg SOURCE_DATE_EPOCH=${SOURCE_EPOCH} \
  --output type=oci,dest=build2.tar \
  --no-cache .

# Compare
sha256sum build1.tar build2.tar

With all reproducibility techniques applied, the SHA-256 hashes of the two OCI tarballs should match or be extremely close. Any remaining differences will be in image configuration metadata and can be resolved with BuildKit’s --source-date-epoch flag (available in BuildKit 0.13+):

docker buildx build \
  --build-arg SOURCE_DATE_EPOCH=${SOURCE_EPOCH} \
  --source-date-epoch ${SOURCE_EPOCH} \
  --output type=oci,dest=build-final.tar \
  --no-cache .

Exercise 7: Verify Reproducibility in CI/CD

Reproducibility is only valuable if you verify it continuously. A build that is reproducible today can become non-reproducible tomorrow if someone adds a floating dependency or a timestamp. The solution is to build twice on every CI run and assert that the results are identical.

GitHub Actions workflow

Create .github/workflows/reproducible-build.yml:

name: Verify Reproducible Build

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  verify-reproducibility:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Compute SOURCE_DATE_EPOCH
        id: epoch
        run: echo "value=$(git log -1 --format=%ct)" >> "$GITHUB_OUTPUT"

      - name: Build image (first pass)
        run: |
          docker buildx build \
            --build-arg SOURCE_DATE_EPOCH=${{ steps.epoch.outputs.value }} \
            --output type=oci,dest=build-pass1.tar \
            --no-cache \
            .

      - name: Record first digest
        id: digest1
        run: echo "sha=$(sha256sum build-pass1.tar | awk '{print $1}')" >> "$GITHUB_OUTPUT"

      - name: Build image (second pass)
        run: |
          docker buildx build \
            --build-arg SOURCE_DATE_EPOCH=${{ steps.epoch.outputs.value }} \
            --output type=oci,dest=build-pass2.tar \
            --no-cache \
            .

      - name: Record second digest
        id: digest2
        run: echo "sha=$(sha256sum build-pass2.tar | awk '{print $1}')" >> "$GITHUB_OUTPUT"

      - name: Compare digests
        run: |
          echo "Build 1: ${{ steps.digest1.outputs.sha }}"
          echo "Build 2: ${{ steps.digest2.outputs.sha }}"
          if [ "${{ steps.digest1.outputs.sha }}" != "${{ steps.digest2.outputs.sha }}" ]; then
            echo "::error::Builds are NOT reproducible! Digests differ."
            echo "Running diffoscope to identify differences..."
            pip install diffoscope
            diffoscope build-pass1.tar build-pass2.tar --text diff-output.txt || true
            cat diff-output.txt
            exit 1
          fi
          echo "Builds are reproducible. Digests match."

      - name: Upload diff report on failure
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: reproducibility-diff
          path: diff-output.txt

      - name: Sign the verified image
        if: github.ref == 'refs/heads/main'
        env:
          COSIGN_EXPERIMENTAL: "true"
        run: |
          # Load the OCI image into Docker
          docker load -i build-pass1.tar
          # In production, push to a registry and sign with Cosign:
          # cosign sign --yes $REGISTRY/$IMAGE@$DIGEST
          echo "Image verified as reproducible and ready for signing."

This workflow does the following on every push and pull request:

  1. Checks out the code and sets up BuildKit.
  2. Computes SOURCE_DATE_EPOCH from the last commit timestamp.
  3. Builds the image from scratch (first pass) and records the digest.
  4. Builds the image from scratch again (second pass) and records the digest.
  5. Compares the two digests. If they differ, the job fails and runs diffoscope to produce a detailed diff report.
  6. On success on the main branch, the verified image is ready for signing with Cosign.

This is the strongest guarantee you can have: every CI run proves that your build is reproducible. If a developer introduces non-determinism, the build breaks immediately.

Exercise 8: Diffing Images Between Versions

Reproducible builds also give you the ability to diff between versions and verify that only the expected changes are present. This is critical for auditing releases: you want to confirm that a version bump only changed the application binary, not the base image or system packages.

Step 1: Build version 1

SOURCE_EPOCH=$(git log -1 --format=%ct)

docker buildx build \
  --build-arg SOURCE_DATE_EPOCH=${SOURCE_EPOCH} \
  --output type=oci,dest=image-v1.tar \
  --no-cache .

Step 2: Make a code change

Edit cmd/app/main.go to change the version string:

fmt.Fprintf(w, "Hello from repro-build-lab v2\n")

Commit the change:

git add cmd/app/main.go
git commit -m "Bump to v2"

Step 3: Build version 2

SOURCE_EPOCH=$(git log -1 --format=%ct)

docker buildx build \
  --build-arg SOURCE_DATE_EPOCH=${SOURCE_EPOCH} \
  --output type=oci,dest=image-v2.tar \
  --no-cache .

Step 4: Compare with diffoscope

diffoscope image-v1.tar image-v2.tar --html-dir version-diff-report

Open the report. You should see that the only differences are:

  • The Go application binary — because we changed the source code.
  • The SOURCE_DATE_EPOCH value — because the commit timestamp changed.

The base image layers, the distroless runtime, and all other layers should be completely identical.

Step 5: Compare layers with crane

# Load images and push to a local registry for crane inspection
docker run -d -p 5000:5000 --name registry registry:2

# Load and push v1
docker load -i image-v1.tar
docker tag myapp:latest localhost:5000/myapp:v1
docker push localhost:5000/myapp:v1

# Load and push v2
docker load -i image-v2.tar
docker tag myapp:latest localhost:5000/myapp:v2
docker push localhost:5000/myapp:v2

# List layers for each version
crane manifest localhost:5000/myapp:v1 | jq '.layers[].digest'
crane manifest localhost:5000/myapp:v2 | jq '.layers[].digest'

Compare the layer digests. You will see that all layers are identical except the one containing the Go binary. This is exactly what you want: a version bump should only change the application layer, nothing else.

If you see unexpected layer changes (for example, the base image layer differs), it means something broke reproducibility and needs investigation. This layer-by-layer comparison is a powerful auditing technique that only works when your builds are reproducible.

Cleanup

# Remove test images
docker rmi myapp:build1 myapp:build2 myapp:pinned1 myapp:pinned2 \
  myapp:pinpkg1 myapp:pinpkg2 myapp:go1 myapp:go2 myapp:repro 2>/dev/null

# Remove OCI tarballs
rm -f build1.tar build2.tar build-pass1.tar build-pass2.tar \
  image-v1.tar image-v2.tar myapp.tar

# Remove extracted binaries
rm -f app1 app2

# Remove diff reports
rm -rf diff-report version-diff-report

# Stop and remove the local registry
docker stop registry && docker rm registry 2>/dev/null

# Remove the test project (optional)
cd .. && rm -rf repro-build-lab

Key Takeaways

  • Pin base images by digest, not tag. Tags are mutable pointers. Digests are cryptographic guarantees. Use crane digest to find the current digest and update it deliberately through a PR, not silently during a build.
  • Pin all package versions or avoid package managers in the runtime image. Multi-stage builds with distroless or scratch runtime images eliminate an entire category of non-reproducibility.
  • Eliminate all sources of timestamps. Use SOURCE_DATE_EPOCH derived from the git commit timestamp. Never run date, timestamp, or similar commands in a Dockerfile.
  • Use reproducible compiler flags. For Go: -trimpath, -ldflags="-s -w -buildid=", and CGO_ENABLED=0. Other languages have similar options.
  • Verify reproducibility in CI/CD by building twice and comparing. This is the only way to guarantee that your build remains reproducible as the project evolves. If the digests diverge, fail the build.
  • Use diffoscope to audit changes between versions. Reproducible builds enable meaningful image diffs. You can verify that a release only contains the changes you intended — nothing more.

Next Steps

Now that you can produce reproducible container images, explore how to build a complete integrity and provenance chain around them: