Mastering Kafka at Scale: Unleashing the Power of Temporal at Stripe

Stripe is at the forefront of the payment processing game, handling an astonishing 700 terabytes in Kafka publish throughput daily – a feat that is only possible with the open-source powerhouse Kafka. However, managing 50 Kafka clusters across three environments and multiple regions can be a Herculean task. That's where Stripe's Stream Infrastructure team stepped in, leveraging the incredible capabilities of Temporal to build a state-of-the-art Kafka Control Plane.  

Welcome to a talk where we'll do a deep dive into Stripe's Kafka Control Plane, exploring:  

1. The extensive functionality of the Control Plane, ranging from broker replacement and cluster rebalancing, to topic configuration and health management.  

2. The reasons behind choosing Temporal as the foundation for this advanced platform and the benefits it offers for managing Kafka at scale.  

3. How we tackled challenges during development, like ensuring the safety of long-running workflows that make state changes to our system, crucial for preventing payment-processing downtime!  

4. Innovative features we've developed to ensure all system state changes remain secure, providing a reliable and efficient experience.  

5. The challenges that we faced when building our control plane in Temporal.  

Don't miss this intriguing dive into how Stripe conquers Kafka management at scale using Temporal. Whether you're simply curious or actively managing Kafka clusters, we invite you to join us and uncover the secrets behind one of the world's leading payment processors!