Build Integrity and Reproducible Builds: A Practical Guide for CI/CD

by

Introduction

If you can’t reproduce a build, you can’t verify it. This simple truth sits at the heart of software supply chain security. Build integrity ensures that what you deploy is exactly what you intended to build — nothing added, nothing modified, nothing tampered with between source code and production artifact.

In recent years, supply chain attacks have demonstrated that the build process itself is a high-value target. Attackers who compromise a build pipeline can inject malicious code into trusted software, affecting millions of downstream users. The only reliable defense is to make builds reproducible: given the same inputs, you should always get the same outputs. When that guarantee holds, any deviation becomes detectable.

This guide walks through the principles of build integrity and reproducible builds, explains why they matter for CI/CD security, and provides practical techniques — from pinning dependencies to hermetic build systems — that you can adopt incrementally in your own pipelines.

What Is Build Integrity?

Build integrity is the guarantee that build inputs deterministically produce build outputs. In other words, if you take the same source code, the same dependencies, the same toolchain, and the same build instructions, you should get a bit-for-bit identical artifact every single time, regardless of when or where you run the build.

Why Non-Reproducible Builds Are a Security Risk

When builds are not reproducible, you lose the ability to verify them. If two builds from the same source produce different binaries, how do you know which one is correct? How do you detect if an attacker injected code during the build process? You simply can’t. Non-reproducibility creates a fog of war that attackers exploit.

Consider the verification problem: an auditor wants to confirm that a released binary corresponds to its published source code. They check out the tagged commit, run the build, and compare. If the build is not reproducible, the outputs will differ — and the auditor has no way to distinguish a legitimate difference (caused by a timestamp or random ordering) from a malicious modification.

The Relationship with SLSA Build Levels

The SLSA framework (Supply-chain Levels for Software Artifacts) directly addresses build integrity through its build track levels:

  • SLSA Build L1: The build process is documented and produces provenance metadata.
  • SLSA Build L2: The build runs on a hosted service that generates authenticated provenance.
  • SLSA Build L3: The build environment is hardened, isolated, and resistant to tampering — even by the project maintainers.

Reproducible builds complement SLSA by providing an independent verification mechanism. Even if you trust the build service (L2/L3), reproducibility lets anyone rebuild from source and verify the output matches.

Real-World Examples

SolarWinds (2020): Attackers compromised the SolarWinds build system and injected a backdoor into the Orion platform update. The malicious code was added during the build process, so the source code repository appeared clean. A reproducible build system would have made this detectable — rebuilding from the published source would have produced a different artifact than the one distributed to customers.

XZ Utils (2024): A sophisticated supply chain attack targeted the xz compression library. A malicious maintainer introduced obfuscated backdoor code through the build system’s test infrastructure. The injected code was designed to compromise SSH authentication on affected Linux systems. The attack exploited the complexity of the build process, injecting malicious payload through binary test fixtures that were processed during the build. Reproducible builds and careful review of build-time behavior would have raised red flags much earlier.

Sources of Non-Reproducibility

Understanding why builds differ between runs is the first step toward fixing them. Here are the most common sources of non-determinism in build processes:

Timestamps Embedded in Artifacts

Many build tools embed the current date and time into output files. Java JAR files contain timestamps in their ZIP entries. C/C++ compilers may record __DATE__ and __TIME__ macros. PE executables on Windows include a timestamp in their headers. Every time you build, the timestamp changes, producing a different output.

Non-Deterministic File Ordering

Archive formats like tar and zip do not guarantee a consistent file order. The order in which files are added to an archive may depend on the filesystem’s directory listing order, which can vary between machines or even between runs on the same machine. This produces different archives with identical contents.

Floating Dependency Versions

If your build configuration specifies express: ^4.18.0 instead of an exact version, you might get 4.18.1 today and 4.18.2 tomorrow. Unpinned dependencies are one of the most common and impactful sources of non-reproducibility.

Build Environment Differences

Different operating system versions, compiler versions, system library versions, locale settings, timezone configurations, and even the number of CPU cores can affect build output. A build on Ubuntu 22.04 may differ from one on Ubuntu 24.04, even with the same source and dependencies.

Network Fetches During Build

Builds that download dependencies at build time are inherently non-reproducible. A package registry might serve a different version, a CDN might return cached or updated content, or the network might be unavailable entirely. Any build that requires network access is at the mercy of external systems.

Random Values and Memory Addresses

Some build processes embed random UUIDs, use non-deterministic hash map iteration orders, or include memory addresses in their output. Profiling data, coverage information, and debug symbols can all introduce randomness into build artifacts.

Hermetic Builds: The Gold Standard

A hermetic build is one that is fully self-contained: it has no network access, all inputs are explicitly declared, and the build environment is completely specified. Hermetic builds are the gold standard for reproducibility because they eliminate entire categories of non-determinism by construction.

What a Hermetic Build Means

In a hermetic build:

  • The build process cannot access the network. All dependencies must be pre-fetched and declared.
  • Every input — source code, dependencies, toolchain, configuration — is explicitly listed and versioned.
  • The build environment is defined precisely, down to the operating system, installed packages, and environment variables.
  • The build is sandboxed so it cannot read undeclared files from the host system.

Bazel as a Hermetic Build System

Bazel is designed from the ground up for hermetic, reproducible builds. It sandboxes each build action, declares all inputs and outputs explicitly, and caches results based on input hashes rather than timestamps. Bazel’s remote caching and remote execution features maintain hermeticity even in distributed build environments.

# Bazel WORKSPACE file: all external dependencies declared
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "com_google_protobuf",
    sha256 = "a79d19dcdf9139fa4b81206e318e33d245c4c9da1ffed21c87288f9142c5f4ef",
    strip_prefix = "protobuf-23.2",
    urls = ["https://github.com/protocolbuffers/protobuf/archive/v23.2.tar.gz"],
)

Docker Multi-Stage Builds with Pinned Base Images

Docker multi-stage builds can approximate hermeticity when combined with pinned base images and pre-fetched dependencies:

# Stage 1: Build with all dependencies pre-installed
FROM golang@sha256:2c3f3c4a1f8e4c2b7d5e1a9f8b7c6d5e4f3a2b1c0d9e8f7a6b5c4d3e2f1a0b AS builder

WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download

COPY . .
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
    go build -trimpath -ldflags='-s -w -buildid=' \
    -o /app/server ./cmd/server

# Stage 2: Minimal runtime image
FROM gcr.io/distroless/static@sha256:1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b AS runtime
COPY --from=builder /app/server /server
ENTRYPOINT ["/server"]

Note the use of -trimpath to remove local file paths from the binary, -buildid= to clear the build ID, and -s -w to strip debug information. These flags are essential for reproducibility in Go builds.

Lock Files

Lock files are the most accessible tool for improving reproducibility. They record the exact resolved versions of all dependencies, including transitive ones:

  • Node.js: package-lock.json or yarn.lock — always use npm ci instead of npm install
  • Go: go.sum — records cryptographic hashes of all module versions
  • Python: poetry.lock or pip-compile output — pins every transitive dependency
  • Rust: Cargo.lock — always committed for binary projects

Lock files should always be committed to version control. A build that ignores lock files is a build you cannot reproduce.

Vendoring Dependencies

Vendoring goes further than lock files by storing the actual dependency source code in your repository. This eliminates any reliance on external registries at build time:

# Go: vendor all dependencies
go mod vendor

# Build using vendored dependencies
go build -mod=vendor ./cmd/server

Vendoring trades repository size for reliability and reproducibility. It is especially valuable for builds that must work in air-gapped environments or for projects where long-term reproducibility matters.

Practical Trade-offs: Hermeticity vs Developer Experience

Full hermeticity comes at a cost. Bazel has a steep learning curve. Vendoring increases repository size. Pre-fetching every dependency requires infrastructure. The key is to adopt hermeticity incrementally: start with lock files and pinned images, then add vendoring and sandboxing as your security requirements demand.

Pinning Everything

Pinning is the practice of specifying exact, immutable versions for every component in your build. Tags are mutable — they can be moved to point at different content. Digests and commit SHAs are immutable. Always pin to immutable references.

Pinning Base Images by Digest

Docker image tags like node:20 or python:3.12-slim can change at any time. The registry can push a new image to the same tag. Pin by digest instead:

# BAD: tag can change at any time
FROM node:20-alpine

# GOOD: pinned to an immutable digest
FROM node@sha256:a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2

# BETTER: tag for readability, digest for immutability
FROM node:20-alpine@sha256:a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2

You can find the current digest with:

docker inspect --format='{{index .RepoDigests 0}}' node:20-alpine

Pinning GitHub Actions by SHA

GitHub Actions tags are mutable. A compromised action can push malicious code to an existing tag. Always pin to the full commit SHA:

# BAD: tag can be moved to malicious commit
- uses: actions/checkout@v4

# GOOD: pinned to immutable commit SHA
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1

Use tools like ratchet or pinact to automate SHA pinning across your workflow files and keep comments with the version tag for readability.

Pinning Toolchain Versions

Your build toolchain — compiler, runtime, package manager — should be version-pinned. Use version manager configuration files to declare exact versions:

# .tool-versions (used by asdf version manager)
nodejs 20.11.0
python 3.12.1
golang 1.22.0
rust 1.75.0
# .nvmrc (Node version manager)
20.11.0
# rust-toolchain.toml
[toolchain]
channel = "1.75.0"
components = ["rustfmt", "clippy"]
targets = ["x86_64-unknown-linux-gnu"]

Using Nix for Build Environment Reproducibility

Nix provides the most rigorous approach to environment reproducibility. A Nix flake declares the complete build environment as a function of its inputs:

# flake.nix
{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05";
    flake-utils.url = "github:numtide/flake-utils";
  };
  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let pkgs = nixpkgs.legacyPackages.${system}; in {
        devShells.default = pkgs.mkShell {
          buildInputs = [ pkgs.go_1_22 pkgs.nodejs_20 pkgs.protobuf ];
        };
      });
}

With Nix, every developer and every CI runner gets the exact same versions of every tool, down to the system libraries. The flake.lock file pins the entire dependency tree to specific Git revisions.

Verifying Build Integrity

Achieving reproducibility is only half the battle. You also need to verify it — to confirm that independent builds from the same source actually produce identical artifacts.

Comparing Builds Across Environments

The simplest verification is to build the same artifact in two different environments and compare the outputs:

# Build in CI
sha256sum build/output/myapp.tar.gz
# Output: a1b2c3d4... build/output/myapp.tar.gz

# Rebuild locally from the same commit
git checkout v1.2.3
make build
sha256sum build/output/myapp.tar.gz
# Output should match: a1b2c3d4...

If the hashes match, the build is reproducible. If they don’t, you need to investigate what differs.

Using diffoscope for Deep Analysis

diffoscope is an essential tool for diagnosing reproducibility issues. It recursively unpacks archives, decompiles binaries, and shows you exactly where two builds differ:

# Install diffoscope
pip install diffoscope

# Compare two builds
diffoscope build-1/myapp.tar.gz build-2/myapp.tar.gz --html report.html

# Compare two container images
diffoscope image-1.tar image-2.tar --html report.html

The HTML report shows differences at every level: archive metadata, file contents, binary sections, and embedded resources. This is invaluable for identifying and eliminating sources of non-determinism one by one.

Storing Build Metadata

Even before achieving full reproducibility, recording build metadata provides traceability. Capture and store:

  • Git commit SHA and branch
  • Hashes of all input dependencies
  • Build environment details (OS version, toolchain version, environment variables)
  • Hashes of all output artifacts
  • Build timestamps and duration

SLSA Provenance

SLSA provenance is a standardized format for build metadata. It records what was built, from what source, using what build process, and in what environment. Tools like slsa-github-generator can automatically generate signed provenance for your GitHub Actions builds:

# .github/workflows/release.yml
jobs:
  build:
    outputs:
      digest: ${{ steps.hash.outputs.digest }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
      - run: make build
      - id: hash
        run: echo "digest=$(sha256sum myapp | cut -d' ' -f1)" >> "$GITHUB_OUTPUT"

  provenance:
    needs: build
    permissions:
      actions: read
      id-token: write
      contents: write
    uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v1.9.0
    with:
      base64-subjects: ${{ needs.build.outputs.digest }}

Container Image Scanning

For container images, verify integrity by inspecting the image layers for unexpected content:

# List all layers in an image
docker history myapp:latest --no-trunc

# Export and inspect image contents
docker save myapp:latest -o image.tar
tar -tf image.tar

# Use dive to inspect layer contents interactively
dive myapp:latest

Reproducible Builds in CI/CD

CI/CD platforms introduce their own reproducibility challenges. Runner images change, caches expire, and build environments are ephemeral. Here’s how to achieve reproducibility on major platforms.

GitHub Actions

A reproducible GitHub Actions workflow pins every external component:

name: Reproducible Build
on:
  push:
    branches: [main]

jobs:
  build:
    # Pin the runner image (or use a self-hosted runner with a known image)
    runs-on: ubuntu-22.04

    steps:
      # Pin action by SHA
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1

      # Pin setup action and toolchain version
      - uses: actions/setup-go@0c52d547c9bc32b1aa3301fd7a9cb496313a4491 # v5.0.0
        with:
          go-version: '1.22.0' # Exact version, not '1.22.x'

      # Cache with hash-based key for determinism
      - uses: actions/cache@ab5e6d0c87105b4c9c2047343972218f562e4319 # v4.0.1
        with:
          path: ~/go/pkg/mod
          key: go-mod-${{ hashFiles('go.sum') }}

      # Build with reproducibility flags
      - run: |
          CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
          go build -trimpath -ldflags='-s -w -buildid=' \
          -o myapp ./cmd/server

      # Record artifact hash
      - run: sha256sum myapp >> checksums.txt

      - uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3 # v4.3.1
        with:
          name: build-artifacts
          path: |
            myapp
            checksums.txt

GitLab CI

In GitLab CI, use fixed Docker images for runners and pin all dependencies:

# .gitlab-ci.yml
variables:
  # Use a specific image digest for the build environment
  BUILD_IMAGE: golang@sha256:2c3f3c4a1f8e4c2b7d5e1a9f8b7c6d5e4f3a2b1c0d9e8f7a6b5c4d3e2f1a0b

stages:
  - build
  - verify

build:
  stage: build
  image: $BUILD_IMAGE
  script:
    - go mod download
    - CGO_ENABLED=0 GOOS=linux GOARCH=amd64
      go build -trimpath -ldflags='-s -w -buildid='
      -o myapp ./cmd/server
    - sha256sum myapp | tee checksums.txt
  artifacts:
    paths:
      - myapp
      - checksums.txt
  cache:
    key:
      files:
        - go.sum
    paths:
      - /go/pkg/mod

verify-reproducibility:
  stage: verify
  image: $BUILD_IMAGE
  script:
    - go mod download
    - CGO_ENABLED=0 GOOS=linux GOARCH=amd64
      go build -trimpath -ldflags='-s -w -buildid='
      -o myapp-verify ./cmd/server
    - sha256sum myapp-verify
    - diff <(sha256sum myapp | cut -d' ' -f1) <(sha256sum myapp-verify | cut -d' ' -f1)

Using Nix in CI for Full Reproducibility

Nix provides the strongest reproducibility guarantees in CI by specifying the entire build closure:

# GitHub Actions with Nix
name: Nix Build
on: push

jobs:
  build:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
      - uses: cachix/install-nix-action@ba0dd844c9180cbf77aa557a09b7b0d890fbd0fb # v26
        with:
          nix_path: nixpkgs=channel:nixos-24.05
      - run: nix build .#myapp
      - run: sha256sum result/bin/myapp

With Nix, the flake.lock file pins the exact version of every package in the build closure. Two developers running nix build on different machines will get identical output, because the entire dependency graph — including the C compiler, system libraries, and every transitive dependency — is precisely specified.

Reproducible Container Image Builds

Building reproducible container images requires special care. Traditional docker build embeds timestamps in every layer. Tools like kaniko, BuildKit, and ko offer better reproducibility:

# Using BuildKit with reproducible output
DOCKER_BUILDKIT=1 docker build \
  --build-arg SOURCE_DATE_EPOCH=$(git log -1 --format=%ct) \
  --output type=docker,rewrite-timestamp=true \
  -t myapp:latest .

The SOURCE_DATE_EPOCH environment variable is a standardized mechanism for telling build tools to use a fixed timestamp instead of the current time. Many tools respect it, including GCC, dpkg, tar, and zip.

When Perfect Reproducibility Isn't Feasible

Bit-for-bit reproducibility is the ideal, but it is not always achievable — and that's okay. What matters is understanding where you are on the reproducibility spectrum and applying compensating controls where needed.

Acceptable Non-Reproducibility

Some non-reproducibility is harmless. A build timestamp in a metadata file that is never executed does not affect the security of the artifact. A log entry recording when the build ran is not a security risk. The key distinction is whether the non-reproducible element is in the artifact itself (risky) or only in metadata alongside it (acceptable).

"Good Enough" Reproducibility

For many teams, the practical goal is functional reproducibility: the same inputs produce the same functional output. The binaries may differ in embedded timestamps or debug symbols, but they behave identically. This level of reproducibility is achievable with standard tools and practices:

  • Pin all dependency versions with lock files
  • Pin base images by digest
  • Pin CI actions by SHA
  • Use fixed toolchain versions
  • Strip timestamps where possible

Compensating Controls

When full reproducibility is not feasible, compensating controls provide alternative assurances:

  • Code signing: Cryptographically sign your artifacts so consumers can verify they came from your build system.
  • SLSA provenance: Generate and publish provenance metadata that records the build inputs, environment, and process.
  • Software Bill of Materials (SBOM): Publish a complete list of components in your artifact so consumers know exactly what they're getting.
  • Build log retention: Store complete build logs for forensic analysis if a compromise is suspected.
  • Multi-party builds: Have multiple independent parties build from the same source and compare results.

The Reproducibility Spectrum

Think of reproducibility as a spectrum, not a binary state:

  • Level 0 — Nothing: No version pinning, no lock files, builds depend on whatever is latest. This is where most projects start.
  • Level 1 — Pinned dependencies: Lock files committed, dependencies are fixed versions. Builds are mostly reproducible.
  • Level 2 — Pinned environment: Toolchain versions and base images are pinned. Build environment is controlled.
  • Level 3 — Hermetic builds: No network access during build. All inputs explicitly declared. Strong reproducibility guarantees.
  • Level 4 — Bit-for-bit reproducible: Independent builds produce identical artifacts. Full verification possible.

Each level builds on the previous one. Moving from Level 0 to Level 1 is often the highest-impact improvement you can make, and it requires minimal effort.

Conclusion

Reproducible builds are the foundation of supply chain trust. Without them, you are relying on blind faith that your build system has not been compromised — a faith that SolarWinds customers, XZ Utils users, and countless others have learned is misplaced.

The good news is that you don't need to achieve perfection on day one. Start with the basics:

  • Commit your lock files and use npm ci instead of npm install.
  • Pin your base images by digest in every Dockerfile.
  • Pin your CI actions by SHA in every workflow file.
  • Pin your toolchain versions with .tool-versions, rust-toolchain.toml, or similar.

Then add hermeticity incrementally: vendor your dependencies, use Nix or Bazel for build isolation, strip timestamps from your artifacts, and set up verification jobs that rebuild and compare.

Every step on the reproducibility spectrum makes your builds more trustworthy, your supply chain more auditable, and your software more secure. In a world where build systems are a primary attack vector, reproducible builds are not optional — they are essential.