Automatically scale Amazon RDS storage using Amazon CloudWatch and AWS Lambda

Introduction

Discover the transformative capabilities of Amazon RDS storage autoscaling, a dynamic feature designed to adapt database storage to real-time demands. This article explores its applications in handling fluctuating workloads, mitigating unpredictable growth, and optimizing resource efficiency. Learn how this innovative tool minimizes manual efforts and ensures continuous database availability, ushering in a new era of performance efficiency in database management.

Expanding storage in Amazon RDS has a 6-hour limitation after modifications. Autoscaling increases storage by 10 GiB, 10% of the current allocation, or predicted growth for the next 7 hours, favoring the greatest option. For heavy data loads, this may lead to hours of a full-storage state, impacting performance. Our automated solution, using Amazon CloudWatch and AWS Lambda, lets you customize how much your database scales when low on space. Choose a scaling percentage aligned with your needs to avoid excessive free space. This solution works alongside storage autoscaling, catering to users requiring scaling beyond the default increment.

Optimal Use Cases for Amazon RDS Storage Autoscaling

Amazon RDS storage autoscaling proves particularly beneficial in scenarios where dynamic adjustments to database storage are crucial. Here are key instances when this feature is especially useful:

• Fluctuating Workloads: In environments with varying workloads, where the demand for database storage can change rapidly, autoscaling ensures that storage adapts to the current needs without manual intervention.
• Unpredictable Growth: Autoscaling becomes a valuable tool for databases experiencing unpredictable data growth. The feature intelligently forecasts future storage requirements and adjusts accordingly, preventing potential disruptions.
• Resource Efficiency: Autoscaling is advantageous when aiming for resource optimization. It automatically adjusts storage based on actual usage, preventing unnecessary over-provisioning or under-provisioning of resources.
• Reducing Manual Effort: In situations where constant manual adjustments to database storage are impractical or time-consuming, autoscaling eliminates the need for frequent interventions. This is especially useful for maintaining optimal performance in dynamic environments.
• Preventing Downtime: Autoscaling helps prevent database downtime due to storage limitations by automatically adjusting storage to accommodate growth. This is crucial for applications that require continuous availability.

While Amazon RDS storage autoscaling is versatile and beneficial in various scenarios, evaluating specific use cases is essential to determine its suitability. Regular monitoring and adjustments can ensure optimal performance and resource utilization in diverse database environments.

Steps to automatically create Database Scaling with Amazon CloudWatch and AWS Lambda

This solution leverages Amazon CloudWatch to monitor Amazon RDS database storage and triggers Amazon SNS notifications when space falls below a defined threshold. Amazon SNS invokes an AWS Lambda function, which dynamically adjusts the database storage by adding a user-specified percentage. This approach prevents repeated storage shortages during the 6-hour optimization period, providing a flexible and responsive scaling solution.

Below is the existing database.

1. Create an Amazon CloudWatch alarm to monitor the existing database’s storage.

Navigate to the Amazon CloudWatch console, select “Alarms” from the navigation pane, and click “Create alarm.” Input the database identifier in the search bar, select “RDS > DBInstanceIdentifier” from the options, select FreeStorageSpace as the metric, and choose Select Metric.

2. Set the Amazon CloudWatch alarm to trigger when free storage reaches 15% of the total storage capacity. For instance, if you have a 100 GB database, the alarm should trigger at 15 GB (15,000,000,000 bytes) to initiate scaling before autoscaling takes over.

3. Select an existing Amazon SNS topic if you already have one, or you can create a new one.

4. Once the Amazon CloudWatch alarm is in place, upon reaching the specified free storage threshold, it will notify the Amazon SNS topic, triggering the associated AWS Lambda function to scale up the storage capacity of the Amazon RDS database.

The AWS Lambda function dynamically adjusts the Amazon CloudWatch alarm threshold, aligning with the database storage growth. For instance, if the database expands from 100 GB to 200 GB, the alarm triggers 30 GB of free storage instead of the initial 15 GB.

Maintaining the free storage threshold at 15% of the total database storage enables the continued use of Amazon RDS storage autoscaling at a 10% free space threshold, serving as a fallback in case of any solution failure.

Here is the part of the AWS Lambda code:

In the AWS Lambda code snippet, three variables (ScalingUnder1000GB, Scaling1000GBto5000GB, and ScalingOver5000GB) represent scaling percentages for different database size ranges. The default settings are to scale databases under 1,000 GB by 20%, those between 1,000 and 5,000 GB by 15%, and databases over 5,000 GB by 10%. Feel free to adjust these values in the AWS Lambda function to customize scaling percentages. Note that the minimum allowed scaling percentage for Amazon RDS storage is 10%.

After the Amazon CloudWatch alarm is triggered, the AWS Lambda function is invoked to initiate the scaling process for Amazon RDS storage. The AWS Lambda function dynamically adjusts the storage capacity based on the predefined scaling percentages. As a result, the Amazon RDS instance transitions into the “modifying” state while the storage modifications are applied, ensuring seamless and responsive scaling to meet changing workload demands.

Conclusion

In conclusion, the solution leverages Amazon CloudWatch alarms, Amazon SNS topics, and Lambda functions to proactively monitor and dynamically scale Amazon RDS databases based on customizable storage thresholds.

Drop a query if you have any questions regarding Amazon CloudWatch and we will get back to you quickly.

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

CloudThat is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 650k+ professionals in 500+ cloud certifications and completed 300+ consulting projects globally, CloudThat is an official AWS Advanced Consulting Partner, AWS Training Partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner, Microsoft Gold Partner, AWS Microsoft Workload Partners, Amazon EC2 Service Delivery Partner, and many more.

To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.