Robinhood’s Kafka Journey from EC2 to Kubernetes

Robinhood relies heavily on Apache Kafka for many functions that power the application’s features. From stock trading to crypto, Kafka is involved in almost every mission-critical step of Robinhood’s functionality.  

  Historically, Robinhood has maintained its own set of Kafka Clusters, leveraging Confluent’s CP offering and running them on top of EC2. These servers were historically provisioned using Saltstack, which was heavily leveraged by Robinhood before its migration towards Kubernetes. However, as the footprint of Kubernetes grew and that of Salt provisioned EC2 boxes shrank, Robinhood’s Kafka clusters were due for a migration.  

  Consequently, the talk will largely focus on:  

- Robinhood’s historical set-up of our Kafka clusters and how they were provisioned and maintained  

- Why Robinhood decided to invest in the new Kafka Kubernetes operator that was built in-house to accommodate the needs of the migration and operational maintenance of Kafka clusters here at Robinhood  

- The safeguards put in place to make sure that there are no performance and reliability regressions between running mission critical Kafka clusters on EC2 vs Kubernetes  

- The migration process of moving large, mission-critical Kafka clusters from EC2 to Kubernetes with minimum downtime and quick replication.  

  If your organization currently runs Kafka outside of Kubernetes and is looking to move clusters over, please attend this talk to see how Robinhood has done it to see if there are any learning points that can be applied to your own organization.