In this article, we will look into how to check logs of failed cronjobs in Kubernetes Cluster but before that let's first try to understand more about CronJobs. What is CronJob ? Why we need to run CronJobs ? So as you probably know a CronJob is basically a Job scheduled to perform a specific task. If you have ever used crontab in Linux System then you probably know how to schedule a cron job. If you haven't then let me tell you a cronjob object is like a one line of a crontab file written in cron format to run a job periodically on a given schedule.
CronJobs are used for performing regular scheduled actions such as backups, report generation, and so on. Each of those tasks should be configured to recur indefinitely (for example: once a day / week / month). You can define the point in time within that interval when the job should start. Now that we discussed about the cronjob, let's see through an example how to check the logs of a failed CronJob.
How to Check Logs of Failed CronJobs in Kubernetes Cluster
Also Read: How to Access Kubernetes Cluster Metrics Using Port Forwarding
You might want to argue that cronjob logs can be checked through kubectl logs -n <namespace> cronjob.batch/<cronjob_name> command but as of Kubernetes v1.20, it does not work as you can see below.
[node1 ~]$ kubectl logs -n default cronjob.batch/hello
error: cannot get the logs from *v1beta1.CronJob: selector for *v1beta1.CronJob not implemented
First of all, it is important to note that there is no direct command to check the logs of failed cronjobs. It is because cronjobs are usually responsible for starting, running and deleting the pods so anything specific to a failed cronjob can only be checked through pod logs in case the pod was successfully launched by the cron job or through events in case the pod did not got launched or got destroyed abruptly. So here I am going to explain through an example how we can identify the reason for failed cron jobs. To start with, I have the below example cronjob running in my k8s cluster.
[node1 ~]$ kubectl get cronjobs
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
hello */1 * * * * False 5 20s 4m31s
The manifest file for above cronjob is taken from the GitHub repo. As you can see, below Job is scheduled to run every 1 minute.
[node1 ~]$ cat cronjob.yaml apiVersion: batch/v1beta1 kind: CronJob metadata: name: hello spec: schedule: "*/1 * * * *" jobTemplate: spec: template: spec: containers: - name: hello image: busybox args: - /bin/sh - -c - date; echo hello from the K8s cluster restartPolicy: OnFailure
To know more about the above hello cronjob, you can describe it by running kubectl describe cronjob hello command as shown below. If you have some other cronjob then you can describe that cronjob by using kubectl describe cronjob <cronjob_name> syntax.
[node1 ~]$ kubectl describe cronjob hello
Name: hello
Namespace: default
Labels: <none>
Annotations: <none>
Schedule: */1 * * * *
Concurrency Policy: Allow
Suspend: False
Successful Job History Limit: 3
Failed Job History Limit: 1
Starting Deadline Seconds: <unset>
Selector: <unset>
Parallelism: <unset>
Completions: <unset>
Pod Template:
Labels: <none>
Containers:
hello:
Image: busybox
Port: <none>
Host Port: <none>
Args:
/bin/sh
-c
date; echo hello from the K8s cluster
Environment: <none>
Mounts: <none>
Volumes: <none>
Last Schedule Time: Sun, 31 Jul 2022 16:19:00 +0000
Active Jobs: hello-1659283920, hello-1659283980, hello-1659284040, hello-1659284100, hello-1659284160, hello-1659284220, hello-1659284280, hello-1659284340
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 7m12s cronjob-controller Created job hello-1659283920
Normal SuccessfulCreate 6m11s cronjob-controller Created job hello-1659283980
Normal SuccessfulCreate 5m11s cronjob-controller Created job hello-1659284040
Normal SuccessfulCreate 4m11s cronjob-controller Created job hello-1659284100
Normal SuccessfulCreate 3m11s cronjob-controller Created job hello-1659284160
Normal SuccessfulCreate 2m11s cronjob-controller Created job hello-1659284220
Normal SuccessfulCreate 70s cronjob-controller Created job hello-1659284280
Normal SuccessfulCreate 10s cronjob-controller Created job hello-1659284340
To check the status of the pods created by the Scheduled CronJob, you can check the events by using kubectl get events command as shown below. From here you can check the reason of your cronjob failure. For example, in our case one of the pod hello-1659283920-xm7tp did not got scheduled because one of the node in the Cluster is not available. Similarly, you can check and verify the reason in your case.
[node1 ~]$ kubectl get events
LAST SEEN TYPE REASON OBJECT MESSAGE
29s Warning FailedScheduling pod/hello-1659283920-xm7tp 0/1 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
114s Normal SuccessfulCreate job/hello-1659283920 Created pod: hello-1659283920-xm7tp
53s Warning FailedScheduling pod/hello-1659283980-54rdq 0/1 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
53s Normal SuccessfulCreate job/hello-1659283980 Created pod: hello-1659283980-54rdq
114s Normal SuccessfulCreate cronjob/hello Created job hello-1659283920
53s Normal SuccessfulCreate cronjob/hello Created job hello-1659283980
If you have all the pods started and running fine. Then you can check the logs of your pod by using kubectl logs <pod_name> syntax. Here we are checking the logs of hello-1659283920-xm7tp pod by using kubectl logs hello-1659283920-xm7tp command as shown below. If you have multiple containers in a pod then use -c <container_name> option.
[node1 ~]$ kubectl logs hello-1659283920-xm7tp