The Performance of Kafka's New Consumer Rebalance Protocol

KIP-848 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol) proposed a new paradigm of the rebalance protocol for Kafka consumers. This prompted a large architectural change in the group coordinator, primarily moving to an event-driven architecture.

  My co-speaker and I have been working on the KIP-848 implementation for some time, and we have started testing the performance of the new architecture.

  At this session, we will highlight the major internal design changes in the group coordinator and the rationalization behind these decisions.  

  Next, we will discuss the primary load dimensions we wanted to explore as part of the performance testing. We will also describe use cases that KIP-848 proposed to improve upon, such as the time to complete a rebalance.

  Lastly, We will share the intermediary and final results. We will discuss the findings from the intermediary results and the improvements we have made to confirm that KIP-848 brings no regressions and instead shows substantial improvements when compared against the old group coordinator.

  If time allows, we will share how we have tested the correctness of the new architecture.