In this article we will see how to solve Multi-Attach error for Volume pvc-xx Volume is already used by pod xx. This error basically means that the PVC volume which the pod is trying to attach during initialization is already attached to another pod. While in few of the cases, it is easy to understand and fix the problem but in some cases like in mine when the PVC volume is showing attached to the same pod which I am trying to bring up then it becomes little tricky to understand.
However, after few moments I did realize the problem and managed to fix it and then later I thought to write an article about this so that it will help you guys also in case you are also facing the same problem.
Solved: Multi-Attach error for Volume "pvc-xx" Volume is already Used
Also Read: How to Disable or Suspend CronJobs in Kubernetes
In some of the cases where you see that the PVC attached to another pod is throwing Multi-Attach error, there you can free up the volume from that pod and attach to the pod where you would like to but what happen in those cases where PVC is showing attached to the same pod which you are trying to reinitialize. In those cases, usually the problem is because PVC attached to the pod is configured as ReadWriteOnce(RWO) and the strategy is configured as RollingUpdate.
As you may or may not know, when the PVC volume is configured as ReadWriteOnce(RWO) then it essentially means that this volume cannot be shared with other pods and configuring the restart strategy to RollingUpdate means that old pod would only be deleted when the new pod comes up in running state. So this is basically the root cause of the multi-attach error for volume pvc-xx volume is already used by pod xx.
The older pod won't release the PVC volume in this type of strategy and hence new pod will never come up as the PVC volume is still attached to the older running pod. So how do we solve this problem. This problem can be solved by multiple ways but let's divide the solution in two categories i.e workaround solution and permanent solution.
Workaround Solution
In workaround solution, you can immediately fix the problem by rescaling your pod to 0 and then scale it up again. As soon as you scale it down to 0, the older pod will get terminated and then you can scale it up again to bring up the new pod. So to scale down, you need to use below command.
kubectl -n <namespace> scale deployment/<deployment_name> --replicas=0
After scaling the replica to 0, you will notice that the pod which is showing in init state will be terminated and then you can scale it back to the desired replica using below command.
kubectl -n <namespace> scale deployment/<deployment_name> --replicas=<desired_replica>
After running above command, you will notice that now the pod will be initialized and running fine.
Permanent Solution
To fix this problem permanently, you can choose to apply any of the below three solutions.
a) Change strategy to Recreate
If you are currently using RollingUpdate strategy then you can switch to Recreate strategy to fix this problem permanently so that older pod will get terminated first and then only new pod will come up.
.spec.strategy.type==Recreate
However, it is important to note that in this strategy there will be some downtime depending on the application you are trying to bring up. So analyze this solution carefully before applying in a production environment.
b) Use StatefulSet instead of Deployment
If you are looking to get rid of the multi attach error altogether then you can think of deploying your application as StatefulSet instead as Deployment where RollingUpdate is the default strategy. So that you don't have to worry about setting strategy type at all. You can perform rolling update n number of times. However, in this solution it will cost you an extra PVC volume. So choose this solution accordingly.
c) Set Volume to ReadWriteMany(RWX)
The final solution that might suit you best is to configure your PVC Volume as ReadWriteMany(RWX) instead of ReadWriteOnce(RWO). This will allow the volume to be shared between the deployment pods and hence you won't be getting Multi-Attach error for Volume pvc-xx Volume is already used by pod xx again at all.