Step-by-Step Guide To Store AWS IoT data into S3 Bucket

1. Introduction

Nowadays, real-time IoT (Internet of Things) applications such as food industry, asset tracking, industrial device monitoring, agricultural production climate monitoring, irrigation application, vehicle tracking, home automation, CCTV Camera, and so on have increased dramatically. These industrial and commercial IoT applications generate massive amounts of data. However, keeping the data in the database for an extended period costs a lot of money and causes a severe business impact.

We can significantly reduce costs by storing these massive amounts of data in AWS S3(Simple Storage Service). AWS S3 is an object storage service that provides industrial data storage with high data availability, scalability, security, and low cost.

Here are some references on AWS Cloud, and AWS IoT Services.

• Step-By-Step Guide To Register A New IoT Device in AWS Cloud 
• Introducing AWS IoT 1-Click: New AWS IoT Service To Enable Simple Devices 
• Uploading Files To AWS S3 Bucket From Html-Form Using A Python CGI Script 

Here, I will show how we can use AWS IoT rules to store IoT device data in AWS S3.

2. High-Level Architectural Diagram

3. Step-by-Step Guide:

1. Create S3 Bucket:

Log in to your AWS account, type S3 into the search box in the main menu, and then select S3 from the services menu.

In the left blade of the AWS S3 console, select the Buckets option and then click Create bucket option

Enter the Bucket name as s3-bucket-for-iot-data and select an AWS region as you prefeed to ensure that AWS IoT services and S3 are both in the same region. I have selected as US East (N. Virginia) us-east-1 as my region

Leave the rest of the options alone and click Create bucket at the bottom

We can see that the bucket has been successfully created with the name s3-bucket-for-iot-data

2. Create an IoT rule to send data to the S3:

In the search box, type IoT Core, and then pick IoT Core from the services selection.

In the IoT Core console, on the left blade, click Act, and then on the right blade, click rule. Finally, click Create in the Rules console to create the rule to transfer IoT data to the S3 bucket.

Enter IoT_data_rule_for_S3 as the name and define the rule in the description section.

Select 2016-03-23 as the SQL version in the Rule query statement and input the Rule Query statement as SELECT * FROM ‘iotdevice/+/datas3’, where * denotes reading the whole data sequence from the IoT topic iotdevice/+/datas3 and + denotes a wildcard character.

To add an action to a rule, select Add action

To add an action, click Store a message in an Amazon S3 bucket, select an action blade, and then click Configure action at the bottom page

In the section “Configure action” Choose the name of the S3 generated in the previous step, which is s3-bucket-for-iot-data, for S3 bucket name. In the key, type ${cast(topic(2) AS DECIMAL)}/${timestamp()} to help differentiate the data over devices id and time. Here ${cast(topic(2) AS DECIMAL)} indicates the devices id which is present in the IoT topic and ${timestamp()} is a current time of decimal value.

We need to provide role for AWS IoT rule to send data to S3 to create a new role click Create Role

In the Create a new role console, enter Iotdata_S3_role as the role name and select the Create role option

Brief period Iotdata_S3_role is created with attached policy and click Add action to add the role

Select Create rule at the bottom of the Create a rule page to create the rule. For example, we can see that the IoT_data_rule_for_S3 has been created.

3. Test IoT rule and S3:

To test an IoT rule, go to the AWS IoT Core dashboard, then to the left blade, select Test, and then MQTT test client. Click the publish to a topic option on the MQTT Test Client page and enter iotdevice/55/datas3 as the Topic name and Message payload {“temperature”: 22, “humidity”: 88, “co2”: 455}. To publish the data, click the Publish option. Now the data is published.

To verify the published data, open the S3 console and choose the bucket, creating the first step, s3-bucket-for-iot-data. Next, we can find the created folder named 55 and open that folder to see the published data.

The published data may be found in the objects console; click the time streamed data and Download to see the data.

4. Use Cases:

We require an IoT core with AWS S3 for long-term storage whenever we have many IoT devices that generate a large volume of data or videos from IoT cameras that must be taken as a backup.

5. Conclusion:

As a result, AWS IoT Core with the IoT Rule engine will assist in filtering IoT topics and the storage of data in AWS S3. AWS IoT core can receive and send millions of IoT data at a time, and the AWS IoT Rule engine can filter MQTT topics from IoT devices and send them to other AWS Services and a time stamp. For data backup and archive, AWS S3 will be used.

About CloudThat

CloudThat is the official AWS Advanced Consulting Partner and collaborates with top IT giants to provide cloud strategy, build solutions, and manage infrastructure. We are on a mission to build a robust cloud computing ecosystem by disseminating knowledge on technological intricacies within the cloud space. Our blogs, webinars, case studies, and white papers enable all the stakeholders in the cloud computing sphere.

Feel free to drop a comment or any queries you have regarding AWS S3 bucket, IoT data storage, and any other AWS services we will get back to you quickly. To get started, go through our Expert Advisory page and Managed Services Package that is CloudThat‘s offerings.