Continuous System Monitoring Using Prometheus and Grafana on top of Kubernetes
Problem Statement
Integrate Prometheus and Grafana and perform in the following way:
- Deploy them as pods on top of Kubernetes by creating resources Deployment, ReplicaSet, Pods, or Services.
- make their data to remain persistent.
- both of them should be exposed to the outside world.
Prerequisites
Minikube and kubectl program should be installed in the system and it should be added to the path
ConfigMap :
ConfigMap is used to store the configurations of the application running in Pod. For example, the httpd.conf for httpd. You could modify the httpd.conf file in Pod but everything will be gone once Pod is replaced. So we can write the desired configurations in ConfigMap and tell Pod to use such configuration whenever Pod is replaced. Therefore Pod is always running with how we wanted it to be.
PVC :
when implementing Prometheus and Grafana on K8S. Everything will work fine but as we know that Pod is stateless and could be recycled anytime, therefore any settings configured on Prometheus and Grafana would be gone when Pod is replaced.
Therefore, to solve this problem, we need to use Persistent Volumes(PV), PersistentVolumeClaims(PVC), and ConfigMap.
PVs as named the provisioned storage space in clusters to store data. Data is stored in PV rather than Pod itself, so data is retained when Pod is replaced. PVC as named is used to claim the provisioned storage space for use.
Service :
This is used to expose the pod's services with a port number so that it can be accessible from the outside world.
The default port range for Kubernetes services is between 30000–32767, you can select any between them or Kubernetes will provide automatically any random port in that range.
Create a file called Prometheus.yml
And add the following to the file
1.Prometheus ConfigMap :
Through ConfigMap, we will define our own configurations in Prometheus. Since we are going to monitor a particular node/System so we will be using Node Exporter. For this, In the Prometheus config file we have to add the target ( IP of the system which we are want to monitor). For this I have created a job with the name “node1” and under this, I have specified the IP of that system.
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-config
data:
prometheus.yml: |-
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'prometheus'
static_configs:- targets: ['localhost:9090']
2.Prometheus Service :
To expose the Prometheus Pod so that it can be accessed from anywhere and we will be able to access the Prometheus WebUI at http://<minikube ip>:30001.
apiVersion: v1
kind: Service
metadata:
name: prometheus
labels:
app: prometheus
spec:
ports:
- port: 9090
nodePort: 30001
selector:
app: prometheus
tier: backendtype: NodePort
3.Prometheus PVC :
- to store data
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: prometheus-pvc
labels:
app: prometheus
spec:
accessModes:
- ReadWriteOnce
resources:
requests:storage: 20Gi
4.Prometheus Deployment :
To launch the pod and connect it with all the previously created resources. Now we have to create the deployment for Prometheus. It will pull the Prometheus image from the docker hub, configure it, then create and mount the persistent volume at the destination folder.
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus
spec:
replicas: 1
selector:
matchLabels:
app: prometheus
tier: backend
strategy:
type: Recreate
template:
metadata:
labels:
app: prometheus
tier: backend
spec:
containers:
- name: prometheus
image: prom/prometheus
volumeMounts:
- name: config-volume
mountPath: /etc/prometheus
- name: data
mountPath: /prometheus
ports:
- containerPort: 80
securityContext:
runAsUser: 0
volumes:
- name: config-volume
configMap:
name: prometheus-config
- name: data
persistentVolumeClaim:claimName: prometheus-pvc
Create a file called Grafana.yml and in similar way create resources.
1.Grafana Service :
we will be able to access the Grafana WebUI at http://<minikube ip>:30002.
apiVersion: v1
kind: Service
metadata:
name: grafana
labels:
app: grafana
spec:
ports:
- port: 3000
nodePort: 30002
selector:
app: grafana
tier: frontendtype: NodePort
2.Grafana Deployment :
apiVersion: apps/v1
kind: Deployment
metadata:
name: grafana
labels:
app: grafana
tier: frontend
spec:
selector:
matchLabels:
app: grafana
tier: frontend
strategy:
type: Recreate
template:
metadata:
labels:
app: grafana
tier: frontend
spec:
containers:
- image: grafana/grafana:latest
name: grafana
ports:
- containerPort: 3000
name: grafana
volumeMounts:
- name: grafana-storage
mountPath: /var/lib/grafana
volumes:
- name: grafana-storage
persistentVolumeClaim:claimName: grafana-pvc
3.Grafana PVC :
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: grafana-pvc
labels:
app: grafana
spec:
accessModes:
- ReadWriteOnce
resources:
requests:storage: 20Gi
Kustomization.yaml
Which is used to bind Prometheus and grafana YAML files
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- prometheus-deployment.yaml
- grafana-deployment.yamlNow we can set the entire environment just by running the following command
kubectl create -k .
To view config maps
kubectl get cm
kubectl get deployments
kubectl get svc
kubectl get pvc
kubectl get rs
You can check the status of the deployment using the command
kubectl get all
to know your minikube ip
minikube ip
Prometheus WebUI :
To access the WebUI you need to go to the minikube ip with the provided port as mentioned earlier.
http://<minikube ip>:30001
and this will be the first page you will see
Grafana WebUI :
To access the WebUI you need to go to the minikube ip with the provided port as mentioned earlier.
http://<minikube ip>:30002
when logged in for the first time the user and password will be admin, then after you can change the password of it.
this will be the Dashboard you will see after you have logged in to the Grafana.
