A Guide to Logging and Monitoring in Amazon Managed Grafana

Overview

Effective logging and monitoring are essential for maintaining applications’ health, performance, and security in today’s cloud-driven landscape. AWS provides a suite of services to help monitor cloud resources, collect logs, and visualize performance metrics. Let’s start our blog by understanding the current challenges the organizations face with Logging and Monitoring in AWS. Then, we will look into how we can overcome those challenges.

Introduction

Amazon Managed Grafana is a fully managed service that allows users to deploy Grafana dashboards without managing the underlying infrastructure. Grafana is a well-known open-source analytics tool frequently used to display logs and metrics from many sources. With Amazon Managed Grafana, you can concentrate on insights rather than operations, as AWS handles the setup, scalability, security, and maintenance.

In this section, we’ll walk through the following steps for setting up Amazon EC2 for Grafana:

1. Setting Up Amazon EC2 Instance for Grafana: This involves creating and configuring the Amazon EC2 instance with the necessary AWS IAM roles, policies, and security groups to allow access to CloudWatch logs within the same AWS account. We’ll review the permissions required for Grafana to retrieve data from Amazon CloudWatch and show you how to set up roles and policies for secure and seamless integration.
2. Installing Grafana on Amazon EC2: Next, we will review the steps to install Grafana on your Amazon EC2 instance. This includes updating the instance, downloading the Grafana package, and configuring it to connect with CloudWatch for monitoring.

Note: When using Amazon Managed Grafana (AMG), you don’t need an Amazon EC2 instance for Grafana. AMG is a fully managed service that eliminates the need to set up and maintain Grafana infrastructure on Amazon EC2.

Current Challenges

We all know that Amazon CloudWatch logs provide great information about your AWS environment. Still, the information in these logs is in text format and challenging to visualize. While AWS has a native Amazon CloudWatch dashboard that can visualize the logs, the dashboard is not very dynamic. (With the dashboard not being dynamic, what I mean is suppose you have 5 Amazon EC2 instances running in a region, and you configure an Amazon CloudWatch dashboard for monitoring Amazon EC2 instances, then these 5 Amazon EC2 instances will be displayed in the Amazon CloudWatch dashboard. But suppose afterward two new Amazon EC2 instances are added, then the Amazon CloudWatch dashboard is not dynamic to automatically capture those two new Amazon EC2 instances)

Therefore, organizations across the world, while they have their workloads running on AWS but for logging and monitoring-

• They rely on 3rd party tools for logging and monitoring purposes (e.g., data dog).
• OR they have to integrate multiple AWS native services (Kinesis + Elasticsearch + Kibana) and build a logging and monitoring solution.

All these approaches are expensive and have lots of integrations.

Therefore, to overcome this challenge, Amazon Managed Grafana can be used to provide a simple and cost-effective solution.

Step-by-Step Guide

1. Launch the Amazon EC2 Instance

• Go to the Amazon EC2 Dashboard in the AWS Console.
• Click on Launch Instance and choose the appropriate AMI, such as Amazon Linux 2.
• Select an instance type (We have taken micro).
• Configure instance details and storage as needed, then proceed to the Configure Security Group

2. Configure Security Groups

• Create a security group for the Amazon Grafana instance.

• Add (default port 3000)to the inbound rules of SG to allow access to Grafana:
  • HTTP: Port 80 from your IP or a specific IP range.
  • HTTPS: Port 443, for secure access.
  • Custom TCP Rule: Port 3000 from your IP address or a specific range (used for Grafana).
  • Optional: Allow SSH (port 22) from your IP to access the instance directly.

3. Set Up AWS IAM Role and Policies for Amazon CloudWatch Access

• In the AWS IAM Dashboard, create a new role Grafana-access-role-2024, with the following steps:
  • Choose AWS Service and select Amazon EC2 as the use case.
  • Attach the following policies to allow Grafana access to Amazon CloudWatch logs and metrics:

• Update the Trust Policy of Grafana-access-role-2024:
• Go to AWS IAM Console in AWS.
• Find Grafana-access-role-2024 and select it.
• Under the Trust relationships tab, edit the Trust relationship
• Modify the Trust Policy to allow the assumption by Grafana-Access-Role:

Paste this updated JSON trust policy

Installing Grafana on Amazon EC2 Instance:

• SSH to the Amazon EC2 Instance.
• Update all installed packages-

• Next, we will install a new YUM repository so the operating system knows where to get Amazon Grafana. –

• Add the lines below to grafana.repo. Grafana’s Open Source version will be installed with this configuration.

• Now install Grafana.

• Reload the system to load the new settings. Check the server status-

• Start the Grafana Server –

• Check the status of the Grafana Server. (Status should be active) –

• Run the command below to make sure that Grafana will start upon booting our Amazon Linux 2 instance –

• Login to the Grafana Server by typing the Public IP address of the newly installed Grafana Server on port 3000.
• Below screen should open up:- Login to the screen with user name as admin and password as admin

• Once logged in to the instance, you must attach a data source (CloudWatch log). Click on “Add your first data source” and search for “Cloudwatch”

• Select “AWS SDK Default” and provide the ARN of the role you created in the previous section. Select the Default region as per your requirement. Once you are done, click “Save & Test”. You have successfully integrated your Grafana instance to take data from your Amazon CloudWatch Logs. Now, you can go ahead and create various dashboards per your requirements.

Conclusion

By configuring Grafana on Amazon EC2, Amazon CloudWatch metrics and logs may be used to log and monitor AWS services efficiently.

Regardless of the approach used, centralizing logging and monitoring is crucial for preserving performance and operational effectiveness.

Drop a query if you have any questions regarding Grafana and we will get back to you quickly.

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

CloudThat is the first Indian Company to win the prestigious Microsoft Partner 2024 Award and is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 650k+ professionals in 500+ cloud certifications and completed 300+ consulting projects globally, CloudThat is an official AWS Advanced Consulting Partner, Microsoft Gold Partner, AWS Training Partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, AWS GenAI Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner,  AWS Microsoft Workload Partners, Amazon EC2 Service Delivery Partner, Amazon ECS Service Delivery Partner, AWS Glue Service Delivery Partner, Amazon Redshift Service Delivery Partner, AWS Control Tower Service Delivery Partner, AWS WAF Service Delivery Partner and many more.

To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.