Login
March 13, 2023|
Voiced by Amazon Polly |
Overview
This blog is about getting logs and metrics from EKS Cluster and displaying them in the Kibana Dashboard by setting up the EFK (ElasticSearch, Fluentbit, and Kibana). The setup mainly concentrates on the Kubernetes cluster, which is created with the help of AWS EKS and monitoring and generating logs and metrics of the Kubernetes cluster components. EFK Stack and Metric Beat are used to generate logs and metrics, respectively.
Pioneers in Cloud Consulting & Migration Services
- Reduced infrastructural costs
- Accelerated application deployment
Introduction
Usually, we can get logs in the Kubernetes Dashboard/terminal itself, but it will get generated with the command kubectl logs. But it is likely got only one pod logs at a time. It will be most difficult to get the logs of all pods. EFK Stack will get all the metrics and logs integration at a time.
Steps for Creating of EKS Cluster
- Create IAM Role for EKS Cluster with AmazonEKSClusterPolicy and name it as
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 |
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "autoscaling:DescribeAutoScalingGroups", "autoscaling:UpdateAutoScalingGroup", "ec2:AttachVolume", "ec2:AuthorizeSecurityGroupIngress", "ec2:CreateRoute", "ec2:CreateSecurityGroup", "ec2:CreateTags", "ec2:CreateVolume", "ec2:DeleteRoute", "ec2:DeleteSecurityGroup", "ec2:DeleteVolume", "ec2:DescribeInstances", "ec2:DescribeRouteTables", "ec2:DescribeSecurityGroups", "ec2:DescribeSubnets", "ec2:DescribeVolumes", "ec2:DescribeVolumesModifications", "ec2:DescribeVpcs", "ec2:DescribeDhcpOptions", "ec2:DescribeNetworkInterfaces", "ec2:DetachVolume", "ec2:ModifyInstanceAttribute", "ec2:ModifyVolume", "ec2:RevokeSecurityGroupIngress", "ec2:DescribeAccountAttributes", "ec2:DescribeAddresses", "ec2:DescribeInternetGateways", "elasticloadbalancing:AddTags", "elasticloadbalancing:ApplySecurityGroupsToLoadBalancer", "elasticloadbalancing:AttachLoadBalancerToSubnets", "elasticloadbalancing:ConfigureHealthCheck", "elasticloadbalancing:CreateListener", "elasticloadbalancing:CreateLoadBalancer", "elasticloadbalancing:CreateLoadBalancerListeners", "elasticloadbalancing:CreateLoadBalancerPolicy", "elasticloadbalancing:CreateTargetGroup", "elasticloadbalancing:DeleteListener", "elasticloadbalancing:DeleteLoadBalancer", "elasticloadbalancing:DeleteLoadBalancerListeners", "elasticloadbalancing:DeleteTargetGroup", "elasticloadbalancing:DeregisterInstancesFromLoadBalancer", "elasticloadbalancing:DeregisterTargets", "elasticloadbalancing:DescribeListeners", "elasticloadbalancing:DescribeLoadBalancerAttributes", "elasticloadbalancing:DescribeLoadBalancerPolicies", "elasticloadbalancing:DescribeLoadBalancers", "elasticloadbalancing:DescribeTargetGroupAttributes", "elasticloadbalancing:DescribeTargetGroups", "elasticloadbalancing:DescribeTargetHealth", "elasticloadbalancing:DetachLoadBalancerFromSubnets", "elasticloadbalancing:ModifyListener", "elasticloadbalancing:ModifyLoadBalancerAttributes", "elasticloadbalancing:ModifyTargetGroup", "elasticloadbalancing:ModifyTargetGroupAttributes", "elasticloadbalancing:RegisterInstancesWithLoadBalancer", "elasticloadbalancing:RegisterTargets", "elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer", "elasticloadbalancing:SetLoadBalancerPoliciesOfListener", "kms:DescribeKey" ], "Resource": "*" }, { "Effect": "Allow", "Action": "iam:CreateServiceLinkedRole", "Resource": "*", "Condition": { "StringEquals": { "iam:AWSServiceName": "elasticloadbalancing.amazonaws.com" } } } ] } |
- Create IAM Role for Worker node (EKSNG_Role) with AmazonEKSWorkerNodePolicy, AmazonEC2ContainerRegistryReadOnly, AmazonEKS_CNI_Policy.
- After that, go to the search bar and search for EKS, then you will get the landing page of EKS Cluster.
- Then click on Add cluster button and choose to create option.
- Give the cluster name (EKS_POC) and attach the role created earlier, i.e., EKSC_Role and choose Next.
6. You need to Select VPC and Subnet selections in this Section and Choose the Cluster End Point Access, where you need to access your cluster and choose Next.
7. No need to enable logging, as we will be using other tools for logging. Click on Next.
8. Then Review and create.
9. The cluster will get created.
10. Then need to create a Node Group. For that, navigate to Compute tab, as shown in the above screenshot.
11. Choose Add Node Group.
12. Enter the Node Group name and the role (EKSNG_Role) for the node group created earlier. And select Next.
13. Select the AMI type, Instance Type of your choice, and capacity type as OnDemand. Select the no. of nodes you need in the Node Group Scaling configuration.
14. Specify the Network configuration in which subnets the Nodes are to be launched, and if needed, you can go with enable the SSH section. For that, you need to create a KeyPair in EC2.
15. Then Review and Create.
16. We need to update kubeconfig file on the host machine
|
1 2 3 4 5 |
aws eks --region us-east-1 update-kubeconfig --name POC_Cluster aws eks --region us-east-1 describe-cluster --name POC_Cluster --query cluster.status cAdvisor Setup |
17. We need to set up Metric Server and cAdvisor for monitoring clusters.
|
1 |
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml |
cAdvisor Setup:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 |
apiVersion: apps/v1 kind: DaemonSet metadata: name: vrops-cadvisor namespace: kube-system labels: app: vrops-cadvisor annotations: seccomp.security.alpha.kubernetes.io/pod: 'docker/default' spec: selector: matchLabels: app: vrops-cadvisor template: metadata: labels: app: vrops-cadvisor version: v0.33.0 spec: tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule containers: - name: vrops-cadvisor image: google/cadvisor:v0.33.0 resources: requests: memory: 250Mi cpu: 250m limits: cpu: 400m volumeMounts: - name: rootfs mountPath: /rootfs readOnly: true - name: var-run mountPath: /var/run readOnly: true - name: sys mountPath: /sys readOnly: true - name: docker mountPath: /var/lib/docker #Mouting Docker volume readOnly: true - name: disk mountPath: /dev/disk readOnly: true ports: - name: http containerPort: 8080 #Port exposed hostPort : 31194 #Host's port - Port to expose your cAdvisor DaemonSet on each node protocol: TCP automountServiceAccountToken: false terminationGracePeriodSeconds: 30 volumes: - name: rootfs hostPath: path: / - name: var-run hostPath: path: /var/run - name: sys hostPath: path: /sys - name: docker hostPath: path: /var/lib/docker #Docker path in Host System - name: disk hostPath: path: /dev/disk |
EFK Stack Setup
18. Install elastic search on the cluster using helm repository (Need to install helm)
|
1 2 3 4 5 |
helm repo add fluent https://fluent.github.io/helm-charts helm repo update helm upgrade --install fluent-bit fluent/fluent-bit |
19. Install Kibana on the cluster using helm
|
1 |
helm upgrade --install kibana elastic/kibana |
20. Make both Kibana and ElasticSearch service as Loadbalancer
MetricBeat Setup
21. Install MetricBeat Server (Need to install MetricBeat after Kubestate metrics server and Kibana got installed)
|
1 |
helm upgrade --install metricbeat elastic/metricbeat |
22. After the Successful installation of MetricBeat, check with the cluster once.
23. We need to install dashboards in Metricbeat pods.
|
1 2 |
./metricbeat setup --dashboards -E setup.kibana.host=<hostip/url/externalip>:port |
24. Copy the Service External-IP of both Kibana and Elastic Search and paste it in the browser as:
<External-IP>:Port
25. Elastic-Search dashboard
<External-IP>:Port/_cat/indic
26. Kibana Dashboard with elastic search logs:
To get logs in Kibana, go to Stack Management, navigate to Kibana, click on Index Patterns, and Select Index Pattern from the list where you want to get logs and place * at the end. Choose @timestamp for getting logs based on time.
Afterward, go to Discover, and the Index logs will be displayed below.
27. Kibana Dashboard with MetricBeat
- To display the Metrics of Pods and Nodes, we need to customize the Kibana dashboard.
- In the Navigation pane in the Observability section, select the Metrics tab, and you will navigate to the Kibana dashboard configured with Metricbeat Metrics there, you can see the logs of the Kubernetes cluster.
Metrics of Elastic_Search_POD
Metrics of Own_POD
Conclusion
So, finally, to get logs and metrics from the Kubernetes cluster, we have successfully set up the EFK stack. As mentioned earlier, the EFK stack refers to Elasticsearch, Fluent Bit, and Kibana. The process is completely done based on Elasticsearch, FluentBit, and Kibana. In addition, Metricbeat is used to get the metrics and make them observable in the observability section of the Kibana dashboard.
Get your new hires billable within 1-60 days. Experience our Capability Development Framework today.
- Cloud Training
- Customized Training
- Experiential Learning
About CloudThat
CloudThat is also the official AWS (Amazon Web Services) Advanced Consulting Partner and Training partner and Microsoft gold partner, helping people develop knowledge of the cloud and help their businesses aim for higher goals using best in industry cloud computing practices and expertise. We are on a mission to build a robust cloud computing ecosystem by disseminating knowledge on technological intricacies within the cloud space. Our blogs, webinars, case studies, and white papers enable all the stakeholders in the cloud computing sphere.
Drop a query if you have any questions regarding Kubernetes, EKF Stack and I will get back to you quickly.
To get started, go through our Consultancy page and Managed Services Package that is CloudThat’s offerings.
FAQs
1. What is an Elasticsearch index?
ANS: – An Elasticsearch index is a group of documents mostly related among them. Elasticsearch usually stores the data in the format of JSON documents. The document is associated with a set of keys usually having the properties or names of fields, probably having matching values like Booleans, numbers, arrays of values, geolocations, dates, strings, etc.
2. What is Kibana used for?
ANS: – Kibana is a data management and visualization tool for Elasticsearch that offers real-time maps, line graphs, pie charts, and histograms. Kibana also provides advanced tools like Elastic Maps for visualizing geographical data and Canvas, which lets users build unique dynamic infographics based on their data.
3. Is Elasticsearch free?
ANS: – Elasticsearch is free, and open features are freely used under the SSPL or the Elastic License. The Elastic License offers additional free services, and paying subscriptions grant access to support and more sophisticated capabilities like alerts and machine learning.
4. What is Metricbeat used for?
ANS: – Installing Metricbeat, a lightweight shipper will let you regularly gather metrics from the servers operating system and active services.
WRITTEN BY Bhanu Prakash K
K Bhanu Prakash is working as a Subject Matter Expert in CloudThat. He is proficient in Managing and configuring AWS Infrastructure as well as on Kubernetes and DevOps tools like Terraform, ansible, Jenkins, and Git. He is very keen on learning new technologies and publishing blogs for the tech community.
PREV
Comments