Application Health Check Using Chaos Engineering

Introduction

Chaos Engineering

Chaos engineering is a method of testing distributed software to ensure that it can resist unexpected interruptions is known as chaos engineering. It is based upon concepts from chaos theory, which emphasizes unpredictable and random behavior. Chaos engineering seeks to pinpoint systemic weaknesses through carefully orchestrated experiments that induce random and unpredictable behavior.

Software testing and quality control are methods used in chaos engineering. It is the right match for modern distributed systems and processes. It tests and makes corrections to a system, unlike stress testing and it mainly aims to gain knowledge about the system.

Customized Cloud Solutions to Drive your Business Success

Cloud Migration
Devops
AIML & IoT

Know More

Principles of Chaos Engineering

Build a hypothesis around a steady state behavior

The first rule of chaos engineering is to concentrate on a system’s measurable output rather than its internal characteristics. To determine the system’s steady state by monitoring that output for a brief period.

Vary real word events

Chaos variables represent actual occurrences, particularly failures of hardware and software as well as non-failure events like a spike in traffic. The frequency or possible impact of each event determines its priority.

Run experiments in production

The most exact simulation of the production environment is typically used to conduct chaos experiments. This is a safeguard to avoid the worst-case situation that might happen when testing in a live environment.

Automate experiments to run continuously

It should automate an experiment to examine a wider variety of conditions once it has been set up, with its outputs documented and its range of conditions defined.

Minimize the blast radius

Production experimentation is a risky step small occurrences can be dealt with quickly, but the chaos engineer must make sure that any consequences are controlled and kept to a minimum.

Steps to Perform WordPress Application Health Check

Update package list for upgrade

sudo su
apt-get update -y

1 2	sudo su apt-get update -y

step1

2. Install python3 and python3 virtual environment

apt-get install python3 python3-venv -

1	apt-get install python3 python3-venv -

step2

3. Create virtual environment

python3 -m venv ~/ .venvs/chaosk

1	python3 -m venv ~/ .venvs/chaosk

4. Activate the environment

source ~/ .venvs/chaostk/bin/activate

1	source ~/ .venvs/chaostk/bin/activate

pip install -U chaostoolkit

1	pip install -U chaostoolkit

step4b

5. Install Helm

curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null

1	curl https://baltocdn.com/helm/signing.asc \| gpg --dearmor \| sudo tee /usr/share/keyrings/helm.gpg > /dev/null

step5

sudo apt-get install apt-transport-https –yes

1	sudo apt-get install apt-transport-https –yes

step5b

echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list

1	echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" \| sudo tee /etc/apt/sources.list.d/helm-stable-debian.list

step5c

sudo apt-get update

1	sudo apt-get update

step5d

sudo apt-get install helm

1	sudo apt-get install helm

step5e

Helm -version

1	Helm -version

6. Add repo to nginx-stable

helm repo add nginx-stable https: // helm.nginx.com/stable

1	helm repo add nginx-stable https: // helm.nginx.com/stable

7. Get update from the repository

helm repo update

1	helm repo update

step7

8. Install nginx controller in the namespace ingress-basic

helm install nginx-release nginx-stable/nginx-ingress –create-namespace -n ingress-basic

1	helm install nginx-release nginx-stable/nginx-ingress –create-namespace -n ingress-basic

step8

helm ls -n ingress-basic

1	helm ls -n ingress-basic

step8b

kubectl get all -n ingress-basic

1	kubectl get all -n ingress-basic

step8c

9. To deploy the WordPress application, add a repository that is azure marketplace

helm repo add azure-marketplace https://marketplace.azurecr.io/helm/v1/repo

1	helm repo add azure-marketplace https://marketplace.azurecr.io/helm/v1/repo

step9

10. Get an update from the repository

helm repo update

1	helm repo update

step10

helm show values azure-marketplace/wordpress > /tmp/values.yaml

1	helm show values azure-marketplace/wordpress > /tmp/values.yaml

vim /tmp/values.yaml

1	vim /tmp/values.yaml

Make changes in /tmp/values.yaml

11. Enable autoscaling to true

step11

12. Enable ingress to true

step12

13. Make persistence enabled to false

step13

14. Make persistence enabled to false in MariaDB Primary Persistence parameters

step14

15. Change the service type of WordPress from Load Balancer to ClusterIP since we are running on the nginx controller

step15

16. Add annotations and save the file

step16

17. Install helm chart, to deploy the WordPress application

helm install my-release azure-marketplace/wordpress –values /tmp/values.yaml –create-namespace -n webapp

1	helm install my-release azure-marketplace/wordpress –values /tmp/values.yaml –create-namespace -n webapp

step17

helm ls -n webapp

1	helm ls -n webapp

step17b

kubectl get all -n webapp

1	kubectl get all -n webapp

step17c

18. Check whether pods are running

kubectl get pods -n webapp

1	kubectl get pods -n webapp

step18

kubectl get svc -n ingress-basic

1	kubectl get svc -n ingress-basic

step18b

19. Add ModHeader to Chrome

step19

20. To access the WordPress application from the web copy and paste the External Ip address

step20

step20b

21. To check whether the application is running and deployed properly access using the IP address.

curl -H “Host:wordpress.local” 104.155.148.17

1	curl -H “Host:wordpress.local” 104.155.148.17

step21

create /tmp/health.yaml 
vim /tmp/health.yaml

1 2	create /tmp/health.yaml vim /tmp/health.yaml

step21b

22. Copy and paste the content, which contains the steady state hypothesis: to check whether all the pods running in the namespace web app is in a healthy state.

version: 1.0.0
title: What happens if we terminate an instance of the application?
description: If an instance of the application is terminated, a new instance should be created
tags:
- k8s
- pod
- deployment
steady-state-hypothesis:
  title: The app is healthy
  probes:
  - name: all-apps-are-healthy
    type: probe
    tolerance: true
    provider:
      type: python
      func: all_pods_healthy   ## This will check all the pods running in the namespace webapp.
      module: chaosk8s.probes
      arguments:
        ns: webapp
method:
- type: action
  name: terminate-app-pod
  provider:
    type: python
    module: chaosk8s.pod.actions
    func: terminate_pods
    arguments:
      label_selector: app.kubernetes.io/name=wordpress
      rand: true
      ns: webapp

version: 1.0.0

title: What happens if we terminate an instance of the application?

description: If an instance of the application is terminated, a new instance should be created

tags:

- k8s

- pod

- deployment

steady-state-hypothesis:

title: The app is healthy

probes:

- name: all-apps-are-healthy

type: probe

tolerance: true

provider:

type: python

func: all_pods_healthy ## This will check all the pods running in the namespace webapp.

module: chaosk8s.probes

arguments:

ns: webapp

method:

- type: action

name: terminate-app-pod

provider:

type: python

module: chaosk8s.pod.actions

func: terminate_pods

arguments:

label_selector: app.kubernetes.io/name=wordpress

rand: true

ns: webapp

step22

kubectl get pods -n webapp --show-labels

1	kubectl get pods -n webapp --show-labels

step22b

23. Run the health.yaml file to check whether the app is healthy,

Here, the experiment failed and deviated when it tried to probe whether all the pods are healthy or not within the namespace web app. It found the system is in unhealthy state.

chaos run /tmp/health.yaml

1	chaos run /tmp/health.yaml

step23

24. Kubernetes action is taken time to create the pod

kubectl get pods -n webapp

1	kubectl get pods -n webapp

step24

25. To run the application successfully need to add pause:20 sec in /tmp/health.yaml file, within those 20 seconds deployment will take an action and recreate the pod

step25

step25b

chaos run /tmp/health.yaml

1	chaos run /tmp/health.yaml

step25d

kubectl get pods -n webapp

1	kubectl get pods -n webapp

step25c

Conclusion

Hence, the application ran successfully after adding pause:20 sec in /tmp/health.yaml file, within those 20 seconds deployment will take an action and the pod will be recreated.

Get your new hires billable within 1-60 days. Experience our Capability Development Framework today.

Cloud Training
Customized Training
Experiential Learning

About CloudThat

CloudThat is also the official AWS (Amazon Web Services) Advanced Consulting Partner and Training partner and Microsoft gold partner, helping people develop knowledge of the cloud and help their businesses aim for higher goals using best in industry cloud computing practices and expertise. We are on a mission to build a robust cloud computing ecosystem by disseminating knowledge on technological intricacies within the cloud space. Our blogs, webinars, case studies, and white papers enable all the stakeholders in the cloud computing sphere.

Drop a query if you have any questions regarding Chaos Engineering and I will get back to you quickly.

To get started, go through our Consultancy page and Managed Services Package that is CloudThat’s offerings.

FAQs

1. Why Chaos Engineering is applied in health check?

ANS: – Unlike conventional testing, Chaos Engineering evaluates software applications and systems through experiments and unexpected pairings. By doing this, firms broaden the testing and examine how the program will function in unexpected circumstances.

2. What is the Steady State Hypothesis in Chaos Engineering?

ANS: – For the experiment to reveal information concerning weaknesses when contrasted against the specified “normal” limits of what is measured, a steady state hypothesis (SSH) specifies “what normal looks like” for the system. The Steady State Hypothesis serves two roles in the Chaos Toolkit. Before experimenting, it is used to ensure that the target system is in a recognized normal state. The results offered by the experiment’s report are formed using it as a template for comparison of the state of your system after the experiment has been run.

WRITTEN BY Abhilasha D

Abhilasha D is a Research Associate-DevOps at CloudThat. She is focused on gaining knowledge of Cloud environment and DevOps tools. She has keen interest in learning and researching on emerging technologies.