Is Your Kafka 'Truly' Load Balancing? Pro Tips for Leader Partition Distribution with Network Usage

One of the primary goals of distributing leader partitions across all brokers in a Kafka cluster is load balancing. This practice helps prevent overloading any single broker and optimizes the throughput of the entire cluster. The distribution of these leader partitions is mainly determined by the preferred leaders set during the creation of topic partitions.  

However, the built-in CLI or API-provided leader election functionality does not effectively achieve the core objective of distributing leader partitions while considering network usage.

  Anticipating future usage when creating topic partitions is challenging, and usage patterns can change over time.  

Therefore, while maintaining the balance of the number of leader partitions in a Kafka cluster, significant fluctuations in actual network usage among leader brokers can still occur.

  To address these issues, I aim to introduce methods for evenly distributing leaders while taking network usage into account.