In this article, we will see how you can check logs of multiple pods in a Kubernetes Cluster. In any Kubernetes environment troubleshooting a critical issue requires you to check the logs of the pods in a quick manner to find the root cause of the problem. For this purpose Kubernetes provides kubectl logs <pod> command to check the logs of any running pod in a Cluster.
But in a critical production environment where you might have number of pods running for each of the application stream, it might not be feasible to run kubectl command for each of the pod individually to filter the logs and find out the error. In those cases what you actually need is a command or a simple script that can query the logs of all the pods in an automated way and show you the output.
How to Check Logs of Multiple Pods in a Kubernetes Cluster
Also Read: How to Check Logs of Failed CronJobs in Kubernetes Cluster
For the demo purpose we are going to use below three pods that are currently running in our two node Kubernetes cluster.
[node1 ~]$ kubectl get pods
NAME READY STATUS RESTARTS AGE
my-nginx-66b6c48dd5-dhb8p 1/1 Running 0 2m34s
my-nginx-66b6c48dd5-l6qxz 1/1 Running 0 2m34s
my-nginx-66b6c48dd5-wg75z 1/1 Running 0 2m34s
It is always easy to check the logs of a single pod by just using kubectl logs <pod> syntax. For example, here we are checking logs of my-nginx-66b6c48dd5-dhb8p pod by using kubectl logs my-nginx-66b6c48dd5-dhb8p command as shown below.
[node1 ~]$ kubectl logs my-nginx-66b6c48dd5-dhb8p
But the real challenge comes when you are having some issue and you don't know the pod on which you need to check the error. In a real scenario you might have more than 5 replica pod running for an application stream. With the priority issue in hand, usually you won't have much time to check the logs of each of the pod one by one. So it is always a better idea to run some quick command or a small script to check the logs of multiple pods at once. This will be beneficial in multiple ways:-
- First you will be able to figure out the problematic pod where the error is happening.
- Second, you will be able to take the error output and save it in a file for reference purposes.
- Third, it will save your precious time of running single command every time for each of the pod.
- And, finally you can run almost the same command every time to find multiple errors.
I always uses one-liner command to get the error I would like to check on multiple pods. But before that you need to first get the list of all the pods on which the error needs to be checked. For example, here we are checking the logs of all the pods that starts with the name my-nginx using below command.
[node1 ~]$ kubectl get pods | grep my-nginx | awk {'print $1'}
my-nginx-66b6c48dd5-dhb8p
my-nginx-66b6c48dd5-l6qxz
my-nginx-66b6c48dd5-wg75z
Then you can simply run a for loop on above command to print the entire logs of all the pods that starts with the name my-nginx.
[node1 ~]$ for i in $(kubectl get pods | grep my-nginx | awk {'print $1'});do echo "******$i******\n";kubectl logs $i;done
******my-nginx-66b6c48dd5-dhb8p******
******my-nginx-66b6c48dd5-l6qxz******
******my-nginx-66b6c48dd5-wg75z******
If you are looking for some specific error from the logs then you can grep the keyword like below. For example, here I am looking for any error from the pods by just grepping the "error" keyword.
[node1 ~]$ for i in $(kubectl get pods | grep my-nginx | awk {'print $1'});do echo "******$i******\n";kubectl logs $i | grep -i error;done
******my-nginx-66b6c48dd5-dhb8p******
******my-nginx-66b6c48dd5-l6qxz******
******my-nginx-66b6c48dd5-wg75z******
If you want to save all the above output logs to a file then you can use the redirection operator(>>) and append all the output to a file say error_log as shown below. After saving the output, you can verify the the contents of the file by using cat error_log command as shown below.
[node1 ~]$ for i in $(kubectl get pods | grep my-nginx | awk {'print $1'});do echo "******$i******\n";kubectl logs $i | grep -i error;done >> error_log [node1 ~]$ cat error_log ******my-nginx-66b6c48dd5-dhb8p****** ******my-nginx-66b6c48dd5-l6qxz****** ******my-nginx-66b6c48dd5-wg75z******