Your CI/CD pipeline can have airtight security controls—signed commits, pinned dependencies, SAST scans, container image signing—but none of it matters if the deployment process itself is weak. Deployment is the critical junction where pipeline security meets production security. A compromised deployment workflow can bypass every upstream control you have built, pushing malicious code straight into the environment your customers rely on.
This guide covers how to build secure deployment workflows end to end: choosing the right deployment model, enforcing gates and approvals, verifying artifacts at deploy time, rolling out changes progressively, and maintaining a full audit trail from commit to production.
Deployment Models: Push-Based vs Pull-Based (GitOps)
The first architectural decision that shapes your deployment security posture is whether you use a push-based or pull-based model.
Push-Based (CI-Driven) Deployments
In a traditional push-based model, the CI/CD pipeline builds the artifact and then pushes it directly to the target environment. GitHub Actions deploys to Kubernetes via kubectl apply, or a GitLab CI job runs helm upgrade against a cluster. The pipeline itself holds credentials to the production environment.
This model is straightforward but carries inherent risk: the CI runner has direct write access to production. If an attacker compromises the pipeline—through a poisoned dependency, a malicious pull request, or a stolen secret—they inherit that production access immediately.
Pull-Based (GitOps) Deployments
In a pull-based or GitOps model, a dedicated controller running inside the target environment—such as Flux or ArgoCD—watches a Git repository for desired state changes. When a new manifest is committed (typically by the CI pipeline updating an image tag), the controller pulls the change and reconciles the cluster to match.
The security advantage is significant. The CI pipeline never needs direct credentials to the production cluster. The attack surface shrinks because the deployment agent lives inside the cluster and only pulls from a known source. Drift detection is built in: if someone manually modifies a resource, the controller reverts it to match Git.
Recommendation: For production workloads, prefer a GitOps pull-based model. Reserve push-based deployments for development and staging environments where speed matters more than strict access control. Even in push-based setups, apply the principle of least privilege rigorously to deployment credentials.
Deployment Gates: Manual Approvals and Protected Environments
Automated pipelines are fast, but deploying to production should not happen without human verification for high-impact changes. Deployment gates introduce checkpoints that require explicit approval before a release proceeds.
GitHub Environments and Required Reviewers
GitHub Actions supports Environments with protection rules. You can require one or more reviewers to approve a deployment before the job runs. This is configured in repository settings and enforced at the platform level—pipeline code cannot bypass it.
# .github/workflows/deploy.yml
jobs:
deploy-production:
runs-on: ubuntu-latest
environment:
name: production
url: https://app.example.com
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Verify artifact signature
run: |
cosign verify \
--key cosign.pub \
ghcr.io/myorg/myapp:${{ github.sha }}
- name: Deploy to production
run: |
helm upgrade --install myapp ./chart \
--set image.tag=${{ github.sha }} \
--namespace production
With the production environment configured to require reviewers, this job will pause and wait for approval before executing any steps. The approver sees exactly which commit and workflow run triggered the deployment.
GitLab Protected Environments
GitLab offers protected environments that restrict which users or groups can trigger deployments. Combined with manual jobs, this creates a robust approval workflow.
# .gitlab-ci.yml
deploy_production:
stage: deploy
environment:
name: production
url: https://app.example.com
rules:
- if: $CI_COMMIT_BRANCH == "main"
when: manual
script:
- cosign verify --key cosign.pub $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
- helm upgrade --install myapp ./chart
--set image.tag=$CI_COMMIT_SHA
--namespace production
resource_group: production
The when: manual directive requires a user to click “Play” in the GitLab UI. The resource_group ensures only one deployment runs at a time, preventing race conditions.
Slack-Based Approvals and ChatOps
For teams that live in Slack, integrating approval workflows with chat provides visibility and fast response times. Tools like Opsgenie, PagerDuty, or custom Slack bots can post a deployment request to a channel and wait for an authorized user to approve via a button or reaction. The key requirement is that the approval mechanism is auditable and cannot be spoofed—use verified Slack app tokens and log every approval decision.
Artifact Verification at Deploy Time
Signing artifacts during the build phase is only half the equation. You must verify those signatures at deploy time. Otherwise, an attacker who gains access to your registry can replace a signed image with an unsigned or re-signed malicious one.
Cosign Verification Before Deployment
Add an explicit verification step in your deployment pipeline that runs before any deployment command. If verification fails, the pipeline must halt immediately.
# Verify the image signature before deploying
cosign verify \
--certificate-identity "https://github.com/myorg/myapp/.github/workflows/build.yml@refs/heads/main" \
--certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
ghcr.io/myorg/myapp@sha256:abc123...
# Verify SLSA provenance
cosign verify-attestation \
--type slsaprovenance \
--certificate-identity "https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v1.9.0" \
--certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
ghcr.io/myorg/myapp@sha256:abc123...
Admission Controllers: Kyverno and Sigstore Policy Controller
Pipeline-level verification is good, but it can be bypassed if someone deploys directly to the cluster using kubectl. Admission controllers enforce verification at the Kubernetes API server level—no unsigned image can enter the cluster regardless of how it was submitted.
Kyverno is a Kubernetes-native policy engine that can verify image signatures and attestations as part of its admission webhook:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-signed-images
spec:
validationFailureAction: Enforce
background: false
rules:
- name: verify-signature
match:
any:
- resources:
kinds:
- Pod
verifyImages:
- imageReferences:
- "ghcr.io/myorg/*"
attestors:
- entries:
- keyless:
subject: "https://github.com/myorg/*"
issuer: "https://token.actions.githubusercontent.com"
attestations:
- type: https://slsa.dev/provenance/v1
conditions:
- all:
- key: "{{ builder.id }}"
operator: Equals
value: "https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v1.9.0"
The Sigstore Policy Controller (formerly cosigned) provides similar functionality and is maintained by the Sigstore project. It integrates tightly with keyless signing workflows and is a strong choice if your organization has standardized on the Sigstore ecosystem.
The combination of pipeline-level verification and cluster-level admission control creates defense in depth: even if one layer is bypassed, the other catches unauthorized artifacts.
Progressive Rollouts: Canary, Blue-Green, and Feature Flags
Deploying a new version to 100% of traffic instantly is a security and reliability risk. Progressive rollout strategies let you catch problems—including security issues—before they affect all users.
Canary Deployments
A canary deployment routes a small percentage of traffic (for example, 5%) to the new version while the majority continues hitting the stable release. If metrics like error rates, latency, or security signals (unexpected outbound connections, elevated privilege escalations) degrade, the canary is automatically rolled back.
Tools like Flagger (for Kubernetes), AWS App Mesh, and Istio automate canary analysis. Flagger, for example, can be configured to monitor custom Prometheus metrics and promote or roll back automatically:
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
name: myapp
namespace: production
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: myapp
progressDeadlineSeconds: 600
service:
port: 8080
analysis:
interval: 1m
threshold: 5
maxWeight: 50
stepWeight: 10
metrics:
- name: request-success-rate
thresholdRange:
min: 99
interval: 1m
- name: request-duration
thresholdRange:
max: 500
interval: 1m
Blue-Green Deployments
Blue-green deployments maintain two identical environments. The “blue” environment runs the current version; the “green” runs the new one. Traffic is switched all at once (typically via a load balancer or DNS change) after the green environment passes health checks and security validation. If something goes wrong, switching back to blue is instantaneous.
The security benefit is a clean, predictable rollback path. There is no partial state to reason about, and the previous version remains fully operational throughout the deployment.
Feature Flags as Security Controls
Feature flags decouple deployment from release. Code is deployed to production but remains inactive behind a flag. This gives security teams a kill switch: if a newly released feature introduces a vulnerability or behaves unexpectedly, it can be disabled instantly without a full rollback. Tools like LaunchDarkly, Unleash, and OpenFeature provide centralized flag management with audit logs of who toggled what and when.
Rollback Strategies
Every deployment plan must include a rollback plan. When things go wrong—and they will—the speed and reliability of your rollback directly determines the blast radius.
Automated Rollback on Health Check Failure
Kubernetes natively supports rollback through its deployment controller. If new pods fail readiness or liveness probes, the rollout stalls and can be automatically reversed:
# Check rollout status and rollback if needed
kubectl rollout status deployment/myapp --namespace production --timeout=300s
if [ $? -ne 0 ]; then
echo "Rollout failed, initiating rollback"
kubectl rollout undo deployment/myapp --namespace production
exit 1
fi
In a GitOps model, rollback means reverting the Git commit that introduced the change. The controller detects the revert and reconciles the cluster back to the previous state. This preserves the full audit trail in Git.
Immutable Deployments
Immutable deployments treat every release as a new, disposable instance. Rather than updating containers in place, you deploy an entirely new set of resources and decommission the old ones. This eliminates configuration drift and ensures that what was tested is exactly what runs in production. Combined with image digests (rather than mutable tags like latest), immutable deployments guarantee binary reproducibility.
Separation of Build and Deploy Identities
One of the most impactful security improvements you can make is ensuring that the identity used to build artifacts is different from the identity used to deploy them. This limits the blast radius of a compromise in either phase.
Different Credentials
The build pipeline should have credentials to push images to a registry and sign them—but no access to production infrastructure. The deployment pipeline (or GitOps controller) should have credentials to pull images and apply manifests—but no access to source code repositories or signing keys.
In practice, this means using separate service accounts, IAM roles, or OIDC claims for each phase. On AWS, the build role might have permissions for ECR push and KMS signing, while the deploy role has permissions for EKS and Secrets Manager but not ECR push.
Different Runners
Take separation further by running build and deploy jobs on physically different runners. Build jobs run on ephemeral, general-purpose runners. Deploy jobs run on dedicated, hardened runners that sit within a network boundary closer to the production environment. This prevents a compromised build runner from pivoting to production.
For a deeper treatment of identity separation and least-privilege principles in CI/CD, see our guide on Separation of Duties and Least Privilege in CI/CD Pipelines.
Deployment Freezes and Change Windows
Not every moment is a good time to deploy. Deployment freezes—periods during which production changes are prohibited—reduce risk during high-traffic events, holidays, on-call transitions, or active incident response.
Implement freezes at the platform level, not just as a team agreement. GitHub Environments support deployment branch policies and wait timers. GitLab allows deploy freezes configured via the UI or API with cron-style schedules. For Kubernetes-based workflows, you can enforce freezes with an OPA/Gatekeeper or Kyverno policy that rejects deployments during specific time windows.
# Kyverno policy to enforce deployment freeze
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: deployment-freeze
spec:
validationFailureAction: Enforce
background: false
rules:
- name: block-deployments-during-freeze
match:
any:
- resources:
kinds:
- Deployment
namespaces:
- production
preconditions:
all:
- key: "{{ time_now() }}"
operator: GreaterThan
value: "2026-03-27T00:00:00Z" # Freeze start
- key: "{{ time_now() }}"
operator: LessThan
value: "2026-03-30T00:00:00Z" # Freeze end
validate:
message: "Production deployments are frozen until March 30. Contact platform-team for emergency exceptions."
deny: {}
Document an exception process for emergency security patches that need to deploy during a freeze, including who can authorize the exception and how it is logged.
Audit Trail: Linking Deployments to Commits, Approvers, and Pipeline Runs
A secure deployment workflow produces a complete, tamper-evident audit trail. For every production deployment, you should be able to answer: What was deployed? Who approved it? Which pipeline built it? What commit does it trace back to?
Platform-Level Audit Logs
AWS CloudTrail records API calls to EKS, ECS, and Lambda, including who initiated the deployment and from which source. GCP Audit Logs provide similar coverage for GKE and Cloud Run. Ensure these logs are shipped to an immutable, centralized log store (such as a dedicated S3 bucket with object lock or a SIEM) where they cannot be tampered with by an attacker who has compromised the deployment environment.
Pipeline-Level Traceability
Annotate Kubernetes resources with deployment metadata so you can trace back from a running pod to the exact source:
# Include in your Helm chart or Kustomize overlay
metadata:
labels:
app.kubernetes.io/version: "{{ .Values.image.tag }}"
annotations:
deploy.example.com/commit-sha: "{{ .Values.commitSha }}"
deploy.example.com/pipeline-url: "{{ .Values.pipelineUrl }}"
deploy.example.com/approved-by: "{{ .Values.approvedBy }}"
deploy.example.com/deployed-at: "{{ now | date \"2006-01-02T15:04:05Z\" }}"
In GitHub Actions, pass these values through the deployment workflow:
- name: Deploy with traceability
run: |
helm upgrade --install myapp ./chart \
--set image.tag=${{ github.sha }} \
--set commitSha=${{ github.sha }} \
--set pipelineUrl="https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}" \
--set approvedBy="${{ github.actor }}" \
--namespace production
Monitoring Post-Deployment
Deployment does not end when the new version is running. Post-deployment monitoring closes the feedback loop and catches issues that pre-deployment checks missed.
Anomaly Detection
Establish baseline metrics for normal application behavior: request rates, error rates, latency percentiles, CPU/memory usage, and network connection patterns. After each deployment, compare current metrics against the baseline. Tools like Prometheus + Alertmanager, Datadog, and Grafana Alerting can trigger alerts when post-deployment metrics deviate beyond thresholds.
From a security perspective, pay special attention to unexpected outbound network connections, new processes spawned inside containers, elevated system calls, and sudden increases in authentication failures. These can indicate that a compromised artifact made it through the pipeline.
DORA Metrics for Security
The four DORA metrics—deployment frequency, lead time for changes, change failure rate, and mean time to recover—are typically used to measure DevOps performance. They are equally valuable for security:
- Deployment frequency indicates how often you can ship security patches. Higher frequency means faster remediation.
- Lead time for changes measures how quickly a security fix goes from commit to production. Long lead times mean extended exposure windows.
- Change failure rate tracks how often deployments cause incidents. A high rate suggests inadequate testing or verification—a security concern.
- Mean time to recover (MTTR) measures how fast you can roll back or remediate a bad deployment. Low MTTR limits the blast radius of any incident, including a security breach.
Track these metrics per environment and correlate them with security events. If your change failure rate spikes after adopting a new deployment pattern, investigate before it becomes a security liability.
Putting It All Together: A Complete Secure Deployment Pipeline
Here is a complete GitHub Actions workflow that incorporates the practices discussed above—artifact verification, environment-based approvals, deployment traceability, and automated rollback:
# .github/workflows/secure-deploy.yml
name: Secure Deployment
on:
workflow_run:
workflows: ["Build and Sign"]
types: [completed]
branches: [main]
jobs:
verify-and-deploy:
runs-on: ubuntu-latest
if: ${{ github.event.workflow_run.conclusion == 'success' }}
environment:
name: production
url: https://app.example.com
permissions:
id-token: write
contents: read
steps:
- name: Checkout manifests
uses: actions/checkout@v4
- name: Install cosign
uses: sigstore/cosign-installer@v3
- name: Verify image signature (keyless)
run: |
IMAGE="ghcr.io/myorg/myapp@${{ github.event.workflow_run.head_sha }}"
cosign verify \
--certificate-identity "https://github.com/myorg/myapp/.github/workflows/build.yml@refs/heads/main" \
--certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
"$IMAGE"
- name: Verify SLSA provenance
run: |
IMAGE="ghcr.io/myorg/myapp@${{ github.event.workflow_run.head_sha }}"
cosign verify-attestation \
--type slsaprovenance \
--certificate-identity "https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v1.9.0" \
--certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
"$IMAGE"
- name: Configure AWS credentials (deploy role)
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789012:role/deploy-production
aws-region: us-east-1
- name: Deploy to EKS
run: |
aws eks update-kubeconfig --name production-cluster
helm upgrade --install myapp ./chart \
--set image.tag=${{ github.event.workflow_run.head_sha }} \
--set commitSha=${{ github.event.workflow_run.head_sha }} \
--set pipelineUrl="https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}" \
--set approvedBy="${{ github.actor }}" \
--namespace production \
--wait --timeout 300s
- name: Verify rollout
run: |
kubectl rollout status deployment/myapp \
--namespace production --timeout=300s
- name: Rollback on failure
if: failure()
run: |
echo "Deployment failed — initiating rollback"
kubectl rollout undo deployment/myapp --namespace production
echo "::error::Deployment rolled back due to failure"
And the equivalent GitLab CI pipeline with similar controls:
# .gitlab-ci.yml
stages:
- verify
- deploy
- validate
verify_artifact:
stage: verify
image: bitnami/cosign:latest
script:
- cosign verify
--certificate-identity "https://gitlab.com/myorg/myapp//.gitlab-ci.yml@refs/heads/main"
--certificate-oidc-issuer "https://gitlab.com"
$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
rules:
- if: $CI_COMMIT_BRANCH == "main"
deploy_production:
stage: deploy
environment:
name: production
url: https://app.example.com
resource_group: production
needs: [verify_artifact]
rules:
- if: $CI_COMMIT_BRANCH == "main"
when: manual
script:
- aws eks update-kubeconfig --name production-cluster
- helm upgrade --install myapp ./chart
--set image.tag=$CI_COMMIT_SHA
--set commitSha=$CI_COMMIT_SHA
--set pipelineUrl=$CI_PIPELINE_URL
--set approvedBy=$GITLAB_USER_LOGIN
--namespace production
--wait --timeout 300s
validate_deployment:
stage: validate
needs: [deploy_production]
script:
- kubectl rollout status deployment/myapp --namespace production --timeout=300s
after_script:
- |
if [ "$CI_JOB_STATUS" == "failed" ]; then
echo "Rolling back deployment"
kubectl rollout undo deployment/myapp --namespace production
fi
rules:
- if: $CI_COMMIT_BRANCH == "main"
Summary and Related Guides
Secure deployment workflows require defense in depth across every phase: choosing the right deployment model, enforcing gates and approvals, verifying artifacts at the cluster boundary, rolling out changes progressively, maintaining clean rollback paths, separating build and deploy identities, respecting change windows, and logging everything. No single control is sufficient on its own. The combination of pipeline-level verification, admission-controller enforcement, progressive rollouts, and comprehensive audit logging creates a deployment process that is both fast and secure.
Continue building your secure CI/CD knowledge with these related guides:
- Separation of Duties and Least Privilege in CI/CD Pipelines — Deep dive into identity separation, scoped credentials, and the principle of least privilege across your pipeline.
- Defensive Patterns and Mitigations for CI/CD Pipeline Attacks — Practical countermeasures for the most common attack vectors targeting CI/CD systems.
