Lagging Behind Because of Logs? ELK Stack to the Rescue!

One of the common mistakes done by most of the professionals is not using valuable data called ‘Logs’. Because of the quantity of logs generated, the chances of using them becomes very less. Logs are used only to debug in case of failure or issues, but it can be used for much more

For Example:

• Monitor processes
• Finding the root cause of the issue being faced
• Analyze flow and performance of processes and many more

The collections and analyzing of the log becomes extremely difficult because of the diversity generated. For example we have access logs, error logs, application logs etc. which are associated with an application or a server.

In this blog, I will be demonstrating how to install and configure ELK Stack.
ELK stands for: Elasticsearch, Logstash and Kibana.

Before we begin, let’s have a quick overview of the overall architecture with their components, followed by the implementation procedure.
 

Architecture of ELK Stack:

1. ElasticSearch:
   • It is an Indexing, Storage and Retrieval engine
   • Powerful open-source full-text search library
   • A Document is the unit of search and index
   • Fast search against large volumes
   • De-normalized document storage: Fast, direct access to the data
   • Broadly distributed and highly scalable
2. Logstash:
   • Log input slicer and dicer and output writer
   • Centralize Data Processing of all types
   • Normalize Varying Schema
   • Extend to Custom Log Formats
3. Kibana:
   • Data Visualizer
   • Kibana is an open source data visualization plugin for ElasticSearch
   • Smooth integration with ElasticSearch
   • Give shape to the artifacts
   • Sophisticated Analytics
   • Flexible Interface
   • Visualize Data from different sources

Working:
The ELK stack architecture is very simple and clearly specifies the flow of the process.Various logs from different locations will be pulled by the Logstash (If you install Nginx for allowing external access then the logs will go to Nginx first), it will process the logs.
Logstash is the center where all the logs are processed and differentiated. Logs are then pushed to ElasticSearch, which is a Retrieval engine, it will index all the logs as per index pattern and will store it to be further accessed by Kibana.
Kibana is a Web UI through which we will do all the activities such as visualizing and analyzing, creating index patterns, etc.

Prerequisites:

• • OS: Ubuntu 14.04
  • RAM: 4GB
  • CPU: 2

Making ELK Stack Up and Running:

Step 1: Launching EC2 Instance and all Installations

• • Go to AWS console and launch a t2.medium(recommended) type of instance so that all three services can run in same instance
  • Login to the instance (or) if you are not going with AWS EC2, then you can do it in your local machine as well

 Install Java 8

• Install ElasticSearch, Logstash and Kibana on it

Install ElasticSearch 

 Install Logstash 

Install Kibana 

Step 2: Configurations
 Configure Logstash:

• • We need to redirect the logs to logstash, such as system logs or any other logs
  • Here, we will redirect the system logs here
  • Create a file where we will do write the configurations at the location /etc/logstash/conf.d/demo-logs.conf
  • Put the following code to it and save it

• Now, save the file and restart all the services

NOTE: This will make Kibana accessible to instance_ip only. If we want to allow external access, then need to use Nginx as reverse proxy.

To allow external access following are the steps to configure with Nginx

• Install Nginx

• Create an admin user to access Kibana dashboard

This will prompt for a password that you will need to access Kibana dashboard along with kibadmin user

• Open the nginx default server block and replace the whole content with the following code

This configuration will make nginx to direct the server’s HTTP traffic to kibana which is listening on localhost:5601. This will enable to access kibana dashboard with elasticsearch server’s public ip.
Restart nginx to apply changes that we made

Step 3: Access Kibana Dashboard

• If the configuration is for localhost then type instance-ip-address:5601 on the web browser, this will open the kibana dashboard.

• And if the configuration is for external access then type the public IP of elasticsearch server such as https://elasticsearch-server-public_ip/

This is the dashboard that we will get.

In this way, we get the logs.But, there are many options to view logs in different formats and to filter them.

Here, you can see the logs of nginx. There are many such options on the Kibana dashboard that you can explore.

Conclusion:

Implementing ELK Stack will provide you with the following benefits:

1. Simple and quick way to manage logs
2. Easy analysis of logs
3. Deep dive into logs (based on timestamp)
4. Various types of forms to view logs (bar chart, pie diagram, etc.)

You just need to create index pattern as per your need and you are ready to go.

Feel free to ask your questions below and I will get back to you on them.

Need professional assistance or consulting services for your ELK Stack project? Kindly visit here  Please comment and share if you liked the article.

WRITTEN BY CloudThat

CloudThat is a leading provider of cloud training and consulting services, empowering individuals and organizations to leverage the full potential of cloud computing. With a commitment to delivering cutting-edge expertise, CloudThat equips professionals with the skills needed to thrive in the digital era.