Simulating AZ Failures in Amazon EKS with AWS Fault Injection Service (FIS)

Introduction

In today’s cloud-native environments, ensuring the resiliency and robustness of your infrastructure is paramount. One critical aspect is understanding how your systems behave under failure conditions. This blog post delves into simulating an Availability Zone (AZ) failure within an Amazon Elastic Kubernetes Service (EKS) cluster using AWS Fault Injection Service (FIS). By deliberately introducing failure scenarios, you can observe and evaluate the response of your EKS cluster, identifying potential weaknesses and ensuring that your system is designed to handle such events effectively. This guide provides a comprehensive, step-by-step approach to performing an AZ failure test, highlighting the necessary prerequisites, detailed steps, and analysis of the results to help you enhance the fault tolerance of your Amazon EKS deployments.

Benefits

1. Enhanced Resiliency and Fault Tolerance: Simulating AZ failures helps identify and address vulnerabilities, ensuring your EKS cluster can maintain operations during zone outages. This enhances the overall fault tolerance and reliability of your infrastructure.
2. Proactive Risk Management and Disaster Recovery: Controlled failure tests allow for the proactive identification and mitigation of risks, verifying the effectiveness of disaster recovery plans. This preparation minimizes unplanned downtime and business disruption.
3. Informed Decision Making and Optimized Resource Utilization: Data from these tests guide architectural and operational improvements, leading to better resource allocation and redundancy strategies. This ensures efficient use of AWS resources and supports informed decision-making.
4. Improved System Understanding and Compliance: Testing provides valuable insights into system behavior under stress, helping to refine automatic scaling and recovery processes. Regular failure testing can also aid in meeting industry compliance standards and achieving higher reliability scores.
5. Increased Customer Satisfaction and Confidence: Ensuring high availability and reliability through regular testing directly contributes to customer satisfaction, as users experience fewer disruptions. This builds confidence in your service’s stability and reliability.

Prerequisites

• AWS account with appropriate permissions to create and manage FIS experiments, Auto Scaling Groups (ASG), and Amazon EC2 instances.
• An Amazon EKS cluster with worker nodes deployed in multiple availability zones.
• AWS CLI or AWS Management Console access.

Step-by-Step Guide

1. Set Up AWS Fault Injection Service (FIS)

• Log in to the AWS Management Console
• Navigate to the AWS Management Console.
• Access the AWS Fault Injection Service.

• Create an Experiment Template
• Go to the “Experiment templates” section.

• Click on “Create experiment template”.
• Define Actions
• Action 1: ASG Insufficient Capacity
• Action name: ASG-Insufficient-Capacity
• Description: Cause the targeted Auto Scaling Groups to receive insufficient instance capacity errors when provisioning new instances.
• Duration: 30 minutes
• Zone: ap-south-1a
• Target: Auto Scaling Group created by EKS
• Action 2: Terminate Instances
• Action name: Terminate-Instance
• Description: Terminate specified EC2 instances.
• Target: All EKS instances present in the ap-south-1a zone.

• Set the duration and sequence of actions to simulate the AZ failure.
• Review and Create
• Review the configuration of the experiment template.
• Click “Create experiment template”.

2. Execute the Experiment

• Start the Experiment
• Navigate to the “Experiments” section.
• Click on “Start experiment” and select the created experiment template.
• Monitor the Experiment
• Monitor the behavior of the EKS cluster during the experiment.
• Observe how the Auto Scaling Group handles the insufficient capacity scenario and the termination of instances in the ap-south-1a zone.

3. Analyze Results

• Check Cluster Resiliency
• Evaluate the impact on the EKS cluster.
• Verify if the cluster spins up new nodes in other availability zones.
• Review Logs and Metrics
• Analyze logs and metrics from CloudWatch or other monitoring tools.
• Ensure there are no unexpected disruptions or failures in the cluster.
• Document Findings
• Document the behavior and resiliency of the EKS cluster during the AZ failure simulation.
• Provide recommendations for improving the cluster’s fault tolerance if necessary.

Conclusion

Regularly conducting such tests helps maintain a robust cloud infrastructure and provides valuable insights into system behavior, enabling proactive risk management and informed decision-making. Ultimately, these efforts increase customer satisfaction and confidence in your service’s stability and reliability. Embrace the practice of fault injection to continuously strengthen your cloud deployments and ensure they are prepared to handle the unexpected.

Drop a query if you have any questions regarding AWS Fault Injection Service and we will get back to you quickly.

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

CloudThat is the first Indian Company to win the prestigious Microsoft Partner 2024 Award and is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 650k+ professionals in 500+ cloud certifications and completed 300+ consulting projects globally, CloudThat is an official AWS Advanced Consulting Partner, Microsoft Gold Partner, AWS Training Partner, AWS Migration Partner, AWS Data and Analytics Partner,AWS DevOps Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner, AWS Microsoft Workload Partners, Amazon EC2 Service Delivery Partner, Amazon ECS Service Delivery Partner, AWS Glue Service Delivery Partner, Amazon Redshift Service Delivery Partner, AWS Control Tower Service Delivery Partner, AWS WAF Service Delivery Partner and many more.

To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.