Sending Amazon ECS Container Logs to an Amazon S3 Bucket using Amazon ECS Fire Lens

Introduction

Managing container logs is crucial in the cloud-native ecosystem, and Amazon ECS offers a solution with ECS FireLens. This powerful log router can efficiently send container logs from Amazon ECS to various AWS services, including Amazon S3 buckets. In this blog post, we’ll guide you through setting up Amazon ECS FireLens to streamline container log management by directing logs straight to an Amazon S3 bucket.

Amazon ECS Fire Lens

Amazon ECS FireLens is a log router for Amazon Elastic Container Service (ECS) that lets you easily route logs from your containers to different AWS services or third-party logging platforms. It centralizes log management, offers flexibility in choosing destinations like Amazon CloudWatch Logs or Amazon S3, and allows you to format and filter logs for better analysis. Essentially, Amazon ECS FireLens streamlines collecting and managing logs from containerized applications on Amazon ECS, enhancing monitoring and troubleshooting capabilities in a cloud-native environment.

Pioneers in Cloud Consulting & Migration Services

Reduced infrastructural costs
Accelerated application deployment

Get Started

AWS Fluent Bit

AWS Fluent Bit is an open-source log collector and processor designed to gather, transform, and forward log data efficiently. It’s lightweight, ideal for containerized environments, and offers flexible configurations to define input sources and output destinations. With built-in support for AWS services like Amazon CloudWatch Logs and Amazon S3, Fluent Bit simplifies centralized log management in AWS cloud-native applications.

Key features of AWS Fluent Bit

Lightweight and Efficient: Designed for minimal resource usage, ideal for containerized environments.
Flexible Configuration: Versatile configuration language for tailored log collection and forwarding.
Built-in AWS Integration: Seamless support for Amazon CloudWatch Logs and Amazon S3.
Extensibility with Plugins: Customizable with a wide range of plugins for added functionality.
Multi-platform Support: Compatible with Linux, Windows, and other operating systems.
High Performance: Optimized for scalable and efficient log collection.
Security and Reliability: Supports TLS/SSL encryption and offers reliable log delivery features.

These features highlight the versatility, efficiency, and integration capabilities of AWS Fluent Bit, making it a valuable tool for log management in AWS environments.

Pre-requisite

Amazon ECS Cluster – to deploy the container.
Amazon S3 Bucket – to store the container logs through the Fire Lens Log Router.

Step by Step guide to Send Amazon ECS Container Logs to Amazon S3 Bucket Through Fire Lens

Step 1: Set up an Amazon S3 Bucket Policy to enable Amazon ECS to save logs in Amazon S3, then create a role and link it with this policy.

From the AWS IAM console, choose the “Policies” option, click “Create Policy”, select the “JSON” tab, and paste the provided policy.

step1

step1b

{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": [
"s3:PutObject"
],
"Resource": "*"
}]
}

{

"Version": "2012-10-17",

"Statement": [{

"Effect": "Allow",

"Action": [

"s3:PutObject"

"Resource": "*"

}]

}

Name the policy, then click “Create Policy”.

step1c

From the AWS IAM console, choose “Roles”, click “Create Role”. Select “AWS service” for Trusted entity type, choose “Elastic Container Service” for Use case, and then select “Elastic Container Service Task” in the sub-category.

step1d

step1e

step1f

For the role’s permissions, select the policy we created earlier. Name the role, then create the role.

step1g

step1h

Step 2: Create an Amazon ECS Task definition that includes a Fire Lens configuration for Amazon S3

From the Amazon ECS Management console, choose “Task definitions”, click “Create new task definition” from the dropdown, select “Create new task definition with JSON”, and paste the provided task definition configuration below.

step2

{
    "family": "firelens-example-firehose",
    "containerDefinitions": [
        {
            "name": "log_router",
            "image": "amazon/aws-for-fluent-bit:stable",
            "cpu": 0,
            "memoryReservation": 50,
            "portMappings": [],
            "essential": true,
            "environment": [],
            "mountPoints": [],
            "volumesFrom": [],
            "user": "0",
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "firelens-container",
                    "awslogs-region": "us-east-2",
                    "awslogs-stream-prefix": "firelens"
                }
            },
            "systemControls": [],
            "firelensConfiguration": {
                "type": "fluentbit"
            }
        },
        {
            "name": "app",
            "image": "httpd",
            "cpu": 0,
            "memoryReservation": 100,
            "portMappings": [],
            "essential": true,
            "environment": [],
            "mountPoints": [],
            "volumesFrom": [],
            "logConfiguration": {
                "logDriver": "awsfirelens",
                "options": {
                    "Name": "s3",
                    "bucket": "ecs-test-logs-bucket",   #Replace with your S3 Bucket
                    "region": "us-east-2",
                    "retry_limit": "2",
                    "total_file_size": "1M",
                    "upload_timeout": "1m",
                    "use_put_object": "On"
                }
            },
            "systemControls": []
        }
    ],
    "taskRoleArn": "arn:aws:iam::211125544165:role/ECS-S3-Taskrole",
    "executionRoleArn": "arn:aws:iam::211125544165:role/ecsTaskExecutionRole",
    "networkMode": "awsvpc",
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "256",
    "memory": "512"
}

{

"family": "firelens-example-firehose",

"containerDefinitions": [

{

"name": "log_router",

"image": "amazon/aws-for-fluent-bit:stable",

"cpu": 0,

"memoryReservation": 50,

"portMappings": [],

"essential": true,

"environment": [],

"mountPoints": [],

"volumesFrom": [],

"user": "0",

"logConfiguration": {

"logDriver": "awslogs",

"options": {

"awslogs-create-group": "true",

"awslogs-group": "firelens-container",

"awslogs-region": "us-east-2",

"awslogs-stream-prefix": "firelens"

}

"systemControls": [],

"firelensConfiguration": {

"type": "fluentbit"

}

{

"name": "app",

"image": "httpd",

"cpu": 0,

"memoryReservation": 100,

"portMappings": [],

"essential": true,

"environment": [],

"mountPoints": [],

"volumesFrom": [],

"logConfiguration": {

"logDriver": "awsfirelens",

"options": {

"Name": "s3",

"bucket": "ecs-test-logs-bucket", #Replace with your S3 Bucket

"region": "us-east-2",

"retry_limit": "2",

"total_file_size": "1M",

"upload_timeout": "1m",

"use_put_object": "On"

}

"systemControls": []

}

"taskRoleArn": "arn:aws:iam::211125544165:role/ECS-S3-Taskrole",

"executionRoleArn": "arn:aws:iam::211125544165:role/ecsTaskExecutionRole",

"networkMode": "awsvpc",

"requiresCompatibilities": [

"FARGATE"

"cpu": "256",

"memory": "512"

}

In the Amazon ECS Task definition JSON configuration provided earlier, replace the “bucket name” with your actual bucket name and update the Task Role and Task Execution Role ARN accordingly.

Step 3: Deploying the Container (task) in the Amazon ECS Cluster

From the Amazon ECS Cluster tab, choose “Create service”. Select “task definition family” and “latest revision” in the Deployment configuration, then create the service.

step3

step3b

step3c

After the service is deployed and shows an “active” status with the desired task in the “running” state, check the Amazon S3 bucket where you set it to store container logs. You should see an object named “Fluent-bit-logs” containing the logs.

step3d

You’ve successfully streamed Amazon ECS container logs to an Amazon S3 bucket using Fluentbit and Firelens as a log router.

Conclusion

AWS Fluent Bit offers a lightweight, versatile log collection and processing solution with seamless AWS integration. Its flexible configuration, extensibility, and high performance make it a valuable tool for centralizing log management in cloud-native environments.

By leveraging Fluent Bit, organizations can simplify log monitoring, enhance analytics, and optimize resource utilization within the AWS ecosystem, making it an essential component for efficient log management on AWS.

Drop a query if you have any questions regarding AWS Fluent Bit and we will get back to you quickly.

Making IT Networks Enterprise-ready – Cloud Management Services

Accelerated cloud migration
End-to-end view of the cloud environment

Get Started

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

CloudThat is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 650k+ professionals in 500+ cloud certifications and completed 300+ consulting projects globally, CloudThat is an official AWS Advanced Consulting Partner, Microsoft Gold Partner, AWS Training Partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner, AWS Microsoft Workload Partners, Amazon EC2 Service Delivery Partner, and many more.

To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.

FAQs

1. What is the difference between Fluentd and Fluent Bit?

ANS: – Fluentd is a full-featured log collector and processor, while Fluent Bit is a lightweight data collector focused on log collection and forwarding. Fluentd offers more features and flexibility but may require more resources, whereas Fluent Bit is designed for minimal resource usage, making it ideal for containerized environments and edge computing.

2. How does AWS Fluent Bit handle log security?

ANS: – AWS Fluent Bit supports secure log transmission using TLS/SSL encryption to ensure the confidentiality and integrity of log data during transmission. Additionally, it offers features like retries and buffering to ensure reliable log delivery, enhancing the security and reliability of log management in AWS environments.

WRITTEN BY Mohammad Zubair Saifi

Mohammad Zubair Saifi works as a Research Associate at CloudThat. He has knowledge of AWS Cloud Services and resources and DevOps tools like Jenkins, Docker, K8s, Ansible, and Terraform. He is passionate about improving his skills and learning new tools and technologies.