Batch Data Processing using AWS Kinesis Firehose to Amazon S3

Introduction

AWS Kinesis Firehose is a fully-managed service provided by Amazon Web Services (AWS) that allows businesses to easily collect, process, and deliver streaming data in real time. It is designed to help users ingest data from various sources, including application logs, clickstreams, IoT devices, and social media platforms.

With Kinesis Firehose, businesses can quickly load data into AWS data stores like Amazon S3, Amazon Redshift, Amazon Elasticsearch, and other third-party destinations like Splunk and Datadog without having to write any custom code. This enables companies to focus on building applications and analyzing data instead of worrying about infrastructure management.

AWS Kinesis Firehose is highly scalable, reliable, and cost-effective. It can handle data streams of any size and automatically scales to meet the demand of data ingestion. With its pay-as-you-go pricing model, businesses only pay for the amount of data they ingest, making it a cost-effective solution for streaming data processing.

What is AWS Kinesis Firehose?

AWS Kinesis Firehose is an Amazon service that allows you to deliver streaming (event) data into destinations such as BI databases, data exploration tools, dashboards, etc. AWS Kinesis Firehose is fully managed with elastic scaling that responds to increased throughputs i.e., as the rate at the data has been sent to your stream increases, the output rate will also increase.

The beauty of this service is that the infrastructure and the capabilities of elastic scaling are all automatically provided therefore, there is no special configuration to worry about. the AWS kinesis firehose service is very easy to navigate, and with very little knowledge, it can be operated easily. This service allows you to batch many events into a single output file (the data is compressed into a single output file it is sent to one of your destinations).

The image above is pulled from AWS kinesis firehose documentation. Let me take you through the above diagram:

1. Input: Any device, website, or server that records data in the real world. It captures the data and sends it to AWS Firehose Kinesis.
2. AWS Kinesis Firehose loads the data and sends it in a batch to your targeted locations, such as Amazon S3 and Redshift. Once either the buffer size is hit or the buffer time
3. The batched data can then be used to visualize or perform operations on it.

Key takeaway terms

1. Buffer size: The maximum size of the data that the Kinesis Firehose can hold until it is sent to a desired location in a batch. Let’s say my Buffer-size is 10 MB. Until the size is reached, Firehose will not send the data to the location. For this condition, we have another factor to remember: Buffer time.
2. Buffer-time: The maximum time the data can stay in AWS Kinesis Firehose. Once the time limit is hit, it will send the data to the targeted location even when the buffer size is not hit.

Real-World Example

Let’s take a scenario of an online store. Daily transactions occur, and let’s further classify them into two transaction types.

In transactions, we have transactionId – The unique serial number for each transaction,

Transaction Amount – The amount of the item being purchased or refunded is of two types and is mentioned below.

Type – “PURCHASE” or “REFUND”

Customer Details – This contains information about the customers.

So, what’s happening here is that people are buying in the real world, so messages are being sent to this transactions topic, and we have a Lambda processor subscribed to that topic.

Thus, for every event that occurs in the transaction, we can have a trigger for this transaction processor, which will be coded in such a way that it performs a PUT operation on the AWS Kinesis Firehose endpoint.

Conclusion

Anytime an event occurs, it will be delivered to your AWS Lambda, and the Lambda will publish the data into the AWS Kinesis Firehose stream. After reaching a certain buffer size or buffer time, the data will be pushed into Amazon S3. Another important feature is that Amazon Kinesis Firehose reads through our data and transforms it accordingly, making it more readable.

About CloudThat

CloudThat is also the official AWS (Amazon Web Services) Advanced Consulting Partner and Training partner and Microsoft gold partner, helping people develop knowledge of the cloud and help their businesses aim for higher goals using best in industry cloud computing practices and expertise. We are on a mission to build a robust cloud computing ecosystem by disseminating knowledge on technological intricacies within the cloud space. Our blogs, webinars, case studies, and white papers enable all the stakeholders in the cloud computing sphere.

Drop a query if you have any questions regarding AWS Kinesis Firehose and I will get back to you quickly.

To get started, go through our Consultancy page and Managed Services Package that is CloudThat’s offerings.