An Introduction to AWS Centralized logging, Architecture, and Pricing

Introduction to AWS Centralized Logging in AWS

Organizations look to analyze their system logs to obtain valuable insights that can help with troubleshooting, security and compliance, and business insights. This solution will provide a secure infrastructure to automate log aggregation from multiple AWS accounts and regions so that developers and administrators can spend more time on actual log analysis.

If we look at the typical AWS account, there are a lot of things that are happening. Services produce operational metrics, security data, and audit logs, and additionally, applications produce web-level, app-level logs. And with more customers moving towards a multi-account strategy there’s a strong desire to centralize logs and have a single view of what’s happening across your AWS account properties that’s where centralized logging comes in. It captures all activity from across AWS workloads and centralizes it for analysis. Centralizing logs comes increasingly useful for customers who want to manage multiple applications and even multiple accounts. Additionally, customers with audit and compliance requirements find it useful to be able to analyze everything all in one place.

Centralized Logging: Architecture

The architecture is fundamentally a hub and spoke. The spokes represent AWS accounts customer may have many of those, each of those have running applications, that have events, that drive logs, and customers want to view those in one place. That’s why centralized logging aggregates all those logs into the hub.

There are three key components let us look at them one by one.

Log Ingestion – the solution creates a CloudWatch log destination in all the regions by default but if one wishes to deploy that log destination in very specific regions, one can do so by changing the parameter inside the CloudFormation stack. This will need to create CloudWatch logs subscription filter in each of the spoke accounts. Once you create that subscription all the log events will come through directly to the primary account.

Log Indexing – In this component, two log indexing is focused. The Kinesis data streams are used as a target for the CloudWatch log destination. All the log events make it to the Kinesis streams that will invoke a Lambda function with that log event. The functionality of this Lambda function is to transform that into an Amazon OpenSearch service document. Once that is transformed, restore that into Kinesis Data Firehose, that Data Firehose then indexes these events or these documents into the OpenSearch service.

Visualization – Data visualization and exploration support are provided by Kibana and Amazon OpenSearch services. Kibana dashboard helps visualize all the index documents in the OpenSearch service. This dashboard allows users to dig deeper into the log events by providing customized time duration for analyzing HTTP 404 code count, HTTP 200 code count, as well as the number of packets that are being accepted, and those which are rejected. Analyzing all the log events at a specific event level is also possible by providing filters in the search section of the dashboard.

Here, there are a few sample data visualization examples of the Kibana dashboard.

Pricing of AWS Centralized Logging

The cost estimation for running Centralized logging solution depends upon factors, which are as follows:

The cluster size of AWS OpenSearch service is available in sizes like Small, Medium, and Large, and the average data transfer rate for Amazon Kinesis Data Firehose, and Amazon Kinesis Data Streams. The following table represents the cost estimation for the Centralized logging solution where a small size cluster of Amazon OpenSearch services is used:

Source 1: https://docs.aws.amazon.com/solutions/latest/centralized-logging/cost.html

Conclusion

Centralized logging can quickly deploy to start monitoring multi-account environments and position themselves to meet their audit and compliance requirements. And aside from Elasticsearch, this is a serverless solution using native AWS components like CloudWatch, CloudTrail, and VPC Flow Logs, all visualized through the Kibana dashboard. For more research, I encourage the readers to check out centralized logging and other AWS solutions on the official AWS Solutions page.

About CloudThat

CloudThat is the official AWS (Amazon Web Services) Advanced Consulting Partner and Training partner and Microsoft gold partner, helping people develop knowledge on the cloud and help their businesses aim for higher goals using best in industry cloud computing practices and expertise. We are on a mission to build a robust cloud computing ecosystem by disseminating knowledge on technological intricacies within the cloud space. Our blogs, webinars, case studies, and white papers enable all the stakeholders in the cloud computing sphere.

Drop a query if you have any questions regarding AWS Centralized logging, AWS services, or consulting opportunities, and I will get back to you quickly. To get started, go through our Consulting page.