Skip to main content
In this guide, we’ll walk through adding a lightweight integration testing stage to your Jenkins pipeline after deploying an application to the production namespace. This approach ensures your production release undergoes quick sanity checks before serving real traffic.

​
Overview

Post-deployment testing often includes functional, smoke, black-box, and performance tests. Common tools include JMeter for load testing. Here, we’ll implement a simple Integration Tests – PROD stage in Jenkins that mirrors our dev workflow.
Test TypePurposeExample Tool
SmokeBasic health and response validationcurl, Postman
FunctionalBusiness logic verificationCustom scripts
Black-boxExternal interface testingcurl, Selenium
PerformanceThroughput and latency measurementJMeter
This integration step is a quick sanity check. For full end-to-end or load tests, consider extending with specialized tools or scripts.

​
Jenkins Pipeline Changes

We insert a new Integration Tests – PROD stage immediately after the K8S_Deployment - PROD stage. If any test fails, the deployment rolls back to the previous version automatically. 
stage('Integration Tests - PROD') {
    steps {
        script {
            try {
                withKubeConfig([credentialsId: 'kubeconfig']) {
                    sh "bash integration-test-PROD.sh"
                }
            } catch (e) {
                withKubeConfig([credentialsId: 'kubeconfig']) {
                    sh "kubectl -n prod rollout undo deploy ${deploymentName}"
                }
                error("Integration tests failed; rolled back deployment.")
            }
        }
    }
}

​
Testing via Istio Ingress Gateway

In production, services typically use a ClusterIP. To access your app behind Istio, retrieve the NodePort assigned to port 80 on the Istio Ingress Gateway: 
kubectl -n istio-system get svc istio-ingressgateway -o json \
  | jq '.spec.ports[] | select(.port == 80) | .nodePort'
If the Ingress Gateway service does not expose port 80 as a NodePort, your tests will fail. Ensure your Gateway configuration allows external access.

​
integration-test-PROD.sh

Below is the shell script executed in the Integration Tests – PROD stage. It:
  1. Waits briefly for the deployment to settle.
  2. Retrieves the Istio NodePort for port 80.
  3. Constructs the application URL.
  4. Checks a simple increment endpoint and HTTP status code.
#!/bin/bash
set -euo pipefail
sleep 5s

# Retrieve the Istio Ingress Gateway NodePort for port 80
PORT=$(kubectl -n istio-system get svc istio-ingressgateway -o json \
  | jq '.spec.ports[] | select(.port == 80) | .nodePort')

if [[ -z "$PORT" ]]; then
  echo "Error: Istio Ingress Gateway port 80 is not exposed as a NodePort."
  exit 1
fi

applicationURL="http://devsecops-demo.eastus.cloudapp.azure.com"
applicationURI="/increment/99"

echo "Using port: $PORT"
echo "Calling: $applicationURL:$PORT$applicationURI"

# Perform the increment test
response=$(curl -s "$applicationURL:$PORT$applicationURI")
http_code=$(curl -s -o /dev/null -w "%{http_code}" "$applicationURL:$PORT$applicationURI")

if [[ "$response" == "100" ]]; then
  echo "Increment Test Passed"
else
  echo "Increment Test Failed: expected '100', got '$response'"
  exit 1
fi

# Validate HTTP status code
if [[ "$http_code" == "200" ]]; then
  echo "HTTP Status Code Test Passed"
else
  echo "HTTP Status Code Test Failed: $http_code"
  exit 1
fi

​
Complete Jenkins Pipeline Snippet

Here’s the full context showing how both the K8S_Deployment - PROD and Integration Tests - PROD stages integrate into your Jenkinsfile: 
pipeline {
  agent any
  stages {
    stage('K8S_Deployment - PROD') {
      steps {
        parallel(
          "Apply Deployment": {
            withKubeConfig([credentialsId: 'kubeconfig']) {
              sh """
                sed -i 's#\\\${imageName}#${imageName}#g' k8s_PROD-deployment_service.yaml
                kubectl -n prod apply -f k8s_PROD-deployment_service.yaml
              """
            }
          },
          "Rollout Status": {
            withKubeConfig([credentialsId: 'kubeconfig']) {
              sh "bash k8s-PROD-deployment-rollout-status.sh"
            }
          }
        )
      }
    }

    stage('Integration Tests - PROD') {
      steps {
        script {
          try {
            withKubeConfig([credentialsId: 'kubeconfig']) {
              sh "bash integration-test-PROD.sh"
            }
          } catch (e) {
            withKubeConfig([credentialsId: 'kubeconfig']) {
              sh "kubectl -n prod rollout undo deploy ${deploymentName}"
            }
            error("Rolled back due to integration test failure.")
          }
        }
      }
    }
  }

  post {
    always {
      junit 'target/surefire-reports/*.xml'
      jacoco execPattern: 'target/jacoco.exec'
      pitmutation mutationStatsFile: '**/target/pit-reports/**/*mutations.xml'
      dependencyCheckPublisher pattern: 'target/dependency-check-report.xml'
      publishHTML(
        allowMissing: false,
        alwaysLinkToLastBuild: true,
        keepAll: true,
        reportDir: 'owasp-zap-report',
        reportFiles: ''
      )
    }
  }
}

​
Demo Run

Once pushed, the pipeline runs the integration script: 
$ bash integration-test-PROD.sh
32564
http://devsecops-demo.eastus.cloudapp.azure.com:32564/increment/99
Increment Test Passed
HTTP Status Code Test Passed
With this setup, you have a streamlined integration test step for your production deployments. To expand your testing strategy, integrate JMeter for performance or other specialized tools.

Watch Video