Self-Hosting Kafka at Scale: Netflix's Journey and Challenges

At Netflix, our usage of Kafka has significantly grown and evolved over the years, paralleling our product development and expansion. This talk aims to share our unique journey with Kafka, tracing our path from using it behind the scenes for building Keystone to now offering Kafka as a Service.  

  This journey has led to huge growth in our Kafka deployment and the use cases that are powered by Kafka within Netflix. Managing this large footprint of Kafka clusters has necessitated us to be agile, innovative, and efficient, especially given our lean team.  

  In this presentation, we will delve into the details of this product evolution, shedding light on how we successfully manage hundreds of Kafka clusters at Netflix. We will discuss the operational strategies we employ and the solutions we have developed to keep our Kafka infrastructure running smoothly and efficiently.  

  Furthermore, operating Kafka at the scale of Netflix is not without its challenges. We will share some of these hurdles, along with our solutions and learnings, providing valuable insights for those looking to scale their own Kafka operations.  

  Join us as we narrate our Kafka story, sharing insights and lessons that could help shape your own Kafka journey.