Building Stream Processing Platform at OpenAI

Curious about how OpenAI leverages Apache Flink for real-time data processing? In this session, we will dive into the technical intricacies of building the Flink platform at OpenAI. We’ll walk you through our Flink infrastructure setup—including deployment strategies, integration with Kafka, and our multi-region architecture. Additionally, we’ll explore how we’ve enhanced PyFlink to operate effectively at our scale. Finally, we’ll discuss the challenges we face, share our strategies for overcoming them, and outline our future roadmap.