In this article, we will try to understand how you can determine the maximum number of kafka consumers that you can scale up to in a kubernetes cluster at any point of time. To understand this problem, you need to have an understanding on Kafka Architecture. You can always check Popular Apache Kafka Architecture Explained Using 4 Basic Components article to understand more about the Kafka components. Once you have the complete understanding then you can go through below section to understand how to find the maximum number of kafka consumers that you can scale up to at any point of time.
How to Find the Maximum Number of Kafka Consumers You can Scale Up To
Also Read: How to Solve Kubernetes Pod CreateContainerConfigError
It is often very important to understand the number of kafka consumers that you may require at any point of time to consume the messages from the subscribed topic. Well, this mostly depends on the number of messages you are expecting a Kafka producer to produce but in general it also depends on other factors such as cost, resources and the capabilities that you currently have. But in any case, the minimum number of consumers must be kept at a level so that it prevents building any consumer lag that could adversely affect the data at the downstream systems.
But to answer this important question of how many maximum number of kafka consumers one can scale up to in a kubernetes cluster, it actually depends on the total number of partitions in a topic where producer are publishing the messages and consumers are subscribed to consume those messages. So the total number of consumers cannot be greater than the total number of partitions in a consumer group. It is simply because as per the Kafka rule one consumer in a consumer group can consume messages from one partition only. No two partitions can be assigned to a consumer in consumer group.
So, let's say if you have total number of five partitions and the producer is publishing messages to just three partitions then ideally your consumer replica will have three pods running at that time. If you run more than three pods then the extra consumer replicas will end up sitting idle.
If you are expecting producer to publish messages in all five partitions and if you are still running three consumer replicas then you will certainly have consumer lag as there are no consumer running from the said consumer group to consume messages from the other two partitions. Additionally, this will also depend upon the Kafka producer on how often it is producing messages to all the five partitions.
But to deal with this problem you need certainly scale up the consumer replicas to five and then it should be able to consume messages from all the partition as expected given that there is no special situation with the payload of your messages. For example - if you have three replicas running for fscm-consumer deployment as shown below.
[cyberithub@node1]$ kubectl get pods
NAME READY STATUS RESTARTS AGE
fscm-consumer-66b6c48dd5-8n9sh 1/1 Running 0 2h
fscm-consumer-66b6c48dd5-q72cf 1/1 Running 0 2h
fscm-consumer-66b6c48dd5-z8tgd 1/1 Running 0 2h
Then to scale this to five replicas, you need to run kubectl scale deployments/fscm-consumer --replicas=5 command as shown below. You will see that two new pod will start as shown below.
[cyberithub@node1]$ kubectl scale deployments/fscm-consumer --replicas=5 NAME READY STATUS RESTARTS AGE fscm-consumer-66b6c48dd5-8n9sh 1/1 Running 0 2h fscm-consumer-66b6c48dd5-q72cf 1/1 Running 0 2h fscm-consumer-66b6c48dd5-z8tgd 1/1 Running 0 2h fscm-consumer-66b6c48dd5-k47ac 1/1 Running 0 5s fscm-consumer-66b6c48dd5-a84bj 1/1 Running 0 5s
The new consumers will start consuming messages from the other two partitions and hence will prevent any consumer lag.