CRIBL Pipeline to Push Logs from Amazon S3 Bucket

Overview

In this digital era, Logs are playing a crucial role in troubleshooting and modernizing our Infrastructure, and it help us to improve our application setup.

Log files are comprehensive documentation of all activities occurring within a system, encompassing various events like transactions, errors, and intrusions. These records can be transmitted in structured, semi-structured, or unstructured formats. Analyzing logs is essential; currently, we have many options like Logscale, OpenSearch, and other tools. But how can we ingest the logs to these tools? There comes the importance of CRIBL. CRIBL is not only an ingestion pipeline. It also helps us compress and filter our records before sending them to any Log search engine tool.

CRIBL

By leveraging Cribl Stream, customers can optimize their observation efforts while minimizing costs, enjoying flexibility and choice, and maintaining full control over their valuable data.

Use Cribl Stream as your universal receiver to collect from any observability data source – receive data from all your agents and push based sources, schedule batch collection from multiple endpoints and APIs, and recall data from low cost storage.

Cribl Stream Pulls Data from Amazon S3 Bucket

Amazon SQS poll message for workers. If messages are accessible, the call will return them; otherwise, it will time out after 1 second if no messages are available.

Amazon S3 distributes the workload to each Worker. S3 returns a limit of one message in a single poll request by default. This default can be changed in Max messages.

Amazon S3 Setup Strategy

The source Amazon S3 bucket must be configured to provide the following data: s3:ObjectCreated:* events can be sent directly to an Amazon SQS queue (the simplest method) or via Amazon SNS (Amazon Simple Notification Service).

Amazon SQS messages are erased after they have been read unless an error occurs, in which case Cribl Stream retries. This implies that, while Cribl Stream will disregard files that do not meet the Filename Filter, Amazon SQS events/notifications from those files will still be read and cleared from the queue (together with those from matching files).

Other Amazon S3 Sources targeting the same Amazon SQS queue will no longer be able to access these ignored files. If you still need to process these files, we recommend the following options:

• Using a separate, dedicated Amazon SQS queue. (This is preferred and encouraged.)
• A broad filter on a single source is used, followed by a pre-processing pipeline and route filters for additional processing.

Step-by-Step Guide

Step 1: Create an Amazon SQS Queue

1. Go to Amazon SQS and click on Create Queue.
2. Select Standard queue and provide a suitable Name.

3. Provide below Access policy,

4. Leave others settings as default and create the queue.

Step 2: Create Event Notification on Source Amazon S3 Bucket

1. We need to create an Amazon S3 event notification from Amazon S3. Go to your AWS Account and select the S3 Bucket, which stores AWS CloudTrail.
2. Then go to Properties, and Click on Create event notification,

Provide a suitable name for the event notification.

3. Select event types as s3:ObjectCreated:*

4. Choose the Amazon SQS queue we created in the last step and save the changes.

Step 3: Create an AWS IAM Role for CRIBL Setup

1. Go to AWS IAM Role, Create Role, Select Custom Trust Policy, and paste the policy below.
2. Change the TrusPrincipal ARN with Cribl Worker ARN
3. To get Worker ARN, Go to the CRIBL Home page and Select Network Settings.

4. Then Go to the Trust option, and you can find the Worker ARN.

5. Create an External ID without any special characters.

Then create a New Policy and paste the below code,

Then, Create the Role.

Step 4: Configuring Cribl Stream to Receive Data from Amazon S3

Login to CRIBL and top nav, click Manage, then select a Worker Group to configure.

1. From the top nav, click Manage, then select a Worker Groupto configure(Select default one)
2. Select the QuickConnect option and select the source option.

3. Click on Add Source, then search for Amazon S3, and click on Add New.

4. Provide a suitable name as Input ID, then provide Amazon SQS queue URL, and leave everything as default.

5. Go to Authentication and select the Auto option.

6. Select the Assume Role option and provide your Account ID in the field of AWS Account ID,

7. Provide AssumeRole ARN as the AWS IAM Role ARN we created in the earlier step.

8. Provide External ID and Duration.

9. Go to Event Breakers, click Add Ruleset, and select AWS Ruleset.

10. Then Go to Connected Destinations, opt for QuickConnect, and Select Destination as Logscale.

11. Now click on Save and click on Commit and Deploy.

12. Now click on Add Destination, provide a suitable name as Output ID, and then provide LogScale Endpoint.

13. Provide LogScale Auth token, select cribl_pipe in the post processing settings.

14. Save the Destination, then commit and deploy.

We have successfully set up a CRIBL pipeline to push the logs from the Amazon S3 bucket to LogScale.

Conclusion

We live in a digital age, and the data collected daily grows. Teams are drowning in vast amounts of data, straining to understand everything. This is becoming increasingly difficult as new tools and capabilities are introduced to the stack. Observability restores control over how data is supplied, its format, and where it should be routed. Decouple the data input layer from data analytics tools to reduce complexity and tool dependencies. CRIBL reduced these complexities and made our life easier.

Drop a query if you have any questions regarding CRIBL and we will get back to you quickly.

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

CloudThat is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 650k+ professionals in 500+ cloud certifications and completed 300+ consulting projects globally, CloudThat is an official AWS Advanced Consulting Partner, AWS Training Partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner, Microsoft Gold Partner, AWS Microsoft Workload Partners, Amazon EC2 Service Delivery Partner, and many more.

To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.