Fault Injection Into Kubernetes

This guide will walk you through injecting network faults into Kubernetes resources such as a service and its pods. You will not need to change any code.

While you may manually deploy fault into Kubernetes, fault comes with a friendly automated fault injection command to simplify the process.

Prerequisites

• Install fault

  If you haven’t installed fault yet, follow the installation instructions.

• Familiar yourself with how fault injects itself into Kubernetes

  If you haven’t read it yet, please explore the related reference for kubernetes platform injection.

Inject Latency Into a Kubernetes Service/Pod

• Create a basic nginx pod and its service

  nginx.yaml

  ---
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: nginx
    labels:
      app: nginx
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: nginx
    template:
      metadata:
        labels:
          app: nginx
      spec:
        containers:
        - name: nginx
          image: nginx
          ports:
        - containerPort: 80
  ---
  apiVersion: v1
  kind: Service
  metadata:
    name: nginx-service
  spec:
    selector:
      app: nginx
    type: NodePort
    ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  

  Let the API server create the resources:

  kubectl apply -f nginx.yaml
  

• Grab the service's IP

  export NGINX_IP=$(kubectl get -o template service/nginx-service --template='{{.spec.clusterIP}}')
  

• Make a first request

  This first request establishes nginx is running:

  curl -w "Connected IP: %{remote_ip}\nTotal time: %{time_total}s\n" http://$NGINX_IP
  Connected IP: 10.43.30.208
  Total time: 0.000680s
  

• Inject fault into the nginx service

  fault inject kubernetes --ns default --service nginx-service --with-latency --latency-mean 300
  

  When you do not explicitly set the service, fault lets you pick up one from the CLI:

  fault inject kubernetes --with-latency --latency-mean 300
  ? Service:  
  grafana
  kubernetes
  > nginx-service
  [↑↓ to move, enter to select, type to filter]
  

• Make a new request

  This second request establishes nginx is running with a latency of 300ms

  curl -w "Connected IP: %{remote_ip}\nTotal time: %{time_total}s\n" http://$NGINX_IP
  Connected IP: 10.43.30.208
  Total time: 0.303097s
  

  The nginx response time is now greater from the client's perspective.