Current London 2025
Session Archive
Check out our session archive to catch up on anything you missed or rewatch your favorites to make sure you hear all of the industry-changing insights from the best minds in data streaming.
Unified Schema Registry: Ensure schema consistency across data systems
Schema Registry is the backbone of safe schema evolution and efficient data transfer in the Kafka ecosystem. Our data infrastructure spans online APIs (gRPC/Rest.li), databases (MySQL/Oracle/Espresso/TiDB), and powerful streaming and ingestion frameworks built on Kafka. From real-time ETL to OLAP systems like Pinot, from tracking and metrics pipelines feeding into Hadoop, to offline jobs pushing insights into derived stores like Venice; Kafka is the key cog in our data flywheel. As data moves through LinkedIn’s ecosystem, it traverses multiple schema languages and serialization formats, evolving—sometimes seamlessly, sometimes with transformation. But what happens when an upstream schema change inadvertently breaks downstream systems? Traditional siloed validation falls short, leading to data corruption, operational disruptions, and painful debugging. Join us as we dive into LinkedIn’s Universal Schema Registry (USR) solution for seamless, large-scale schema validation. 1. End-to-End Schema Compatibility – USR validates schemas holistically across RPC, Kafka, Espresso, Venice, and more, safeguarding the entire data lineage. 2. CI-Integrated Early Validation – A shift-left approach catches issues at schema authoring time, saving thousands of developer hours and avoiding costly data migrations. 3. Multi-Format Schema Mapping – Supports complex transformations across Proto, Avro, and more, enabling automated migrations like LinkedIn’s move from Rest.li to gRPC for online RPCs. Join us as we share LinkedIn’s journey of building and scaling USR to handle millions of validations every week, the challenges faced, and the best practices that keep our data ecosystem resilient.
Souman Mandal, Sarthak Jain
Democratising Stream Processing: How Netflix Empowers Teams with Data Mesh and Streaming SQL
As data volume and velocity continue to grow at unprecedented rates, organisations are increasingly turning to stream processing to unlock real-time insights and fuel data-driven decision-making. This presentation will explore how Netflix has evolved its Data Mesh platform, a distributed data architecture, to embrace the power of Streaming SQL. The session will showcase the journey from the initial implementation of Data Mesh as a data movement platform to its transformation into a comprehensive stream processing powerhouse with the integration of the Data Mesh SQL Processor. Attendees will learn about the challenges Netflix faced with its earlier system, which relied on pre-built processors and the low-level Flink DataStream API, leading to limitations in expressiveness and a steep learning curve for developers. The presentation will then unveil the innovative solution: the Data Mesh SQL Processor, a platform-managed Flink job that harnesses the familiarity and versatility of SQL to simplify stream processing. Through practical examples, attendees will discover how the SQL Processor interacts with Flink's Table API, converting data streams into dynamic tables for seamless SQL processing. Moreover, the session will spotlight the platform's user-friendly SQL-centric features, including an interactive query mode for live data sampling, real-time query validation, and automated schema inference.
Sujay Jain
Migrating Kafka from ZooKeeper to KRaft: adventures in operations
From version 4.0 of Kafka, clusters will not support ZooKeeper anymore. This is a change that has been in the works for a while, and Kafka now has its own consensus protocol named KRaft. If you want to create a new Kafka cluster, using KRaft is pretty straightforward, but what if you already have existing Kafka clusters? Since Kafka 3.6, you can migrate ZooKeeper based clusters to KRaft. The process has a few intricacies about it and even more so if you want to automate it, rather than doing it manually for each cluster. In this session we will take you along for the ride of migrating a ZooKeeper based Kafka cluster, to a KRaft one. We’ll share the pitfalls we discovered while implementing an automated process in the open source Kafka operator Strimzi to handle this operation. It was certainly an adventure, so come along to see how you can navigate this migration as smoothly as possible, by avoiding a lot of error-prone manual steps on your production clusters.
Kate Stanley, Paolo Patierno
Async Processing: Get any Kafka Streams app into the throughput Olympics
Since the very dawn of Kafka Streams it has been haunted by the possibility of high latencies in a topology. Custom logic with heavy processing, RPCs, and remote state stores were more or less considered out of the question, and those who bravely tried anyways ran into endless problems. At the same time, even simple applications were getting in trouble as companies grew and so did their workloads, yet apps could only scale up to the partition count. Are slow and/or heavy applications just not a good fit for Kafka Streams? The answer of course is no! With Kafka Streams anything is possible. In this talk we’ll introduce Responsive’s Async Processor, a lightweight wrapper that turns your slowest applications into medal winners. With just a few lines of code you can easily convert any Kafka Streams app to async. By injecting an async thread pool to hand over the actual processing work, records can be executed in parallel, even within a partition — all without sacrificing any of the usual correctness guarantees like same-key ordering or exactly-one semantics. This feature is production-ready and available now through the Responsive SDK, but we’ll end by discussing our vision for native async processing in open source Kafka Streams. We’ll also go over some neat features we added to Kafka Streams like the ProcessorWrapper, which is used by the async framework but has the potential for so much more. Join us to hear all the gory details of async processing and how it reimagines the limits of Kafka Streams itself. Bring your slowest application and find out if it can make the team!
A. Sophie Blee-Goldman
Flink & Metered Billing - a reprocessing story
Join us to explore how a metered billing system was built and utilized Apache Flink to achieve accurate numbers. We’ll cover session window aggregation, ordering, idempotency, late enrichment, metered billing, reconciliation, watermark alignment, and data accuracy. Learn how we iteratively modified and expanded our business logic, idempotently reprocessing over 500M events more than ten times to ensure precision for our customers. We’ll dive into the architectural decisions, operational strategies, and challenges faced, providing valuable insights into building robust, real-time data processing systems with Flink.
Pedro Mázala
Kafka Connection Chaos: Surviving the Storm
It is 9 AM, support team began the maintenance to renew Kafka Broker's certificates. At 9:30 AM half of the cluster has been updated correctly, but, the liveness probe metric seems unstable. We check connectivity — everything looks fine. Our monitoring stack shows it is able to consume and produce from/to all brokers. Connections are a bit higher than usual but still within limits. 9:40 AM: some teams start complaining that they can neither consume nor produce. What is happening? Suddenly, we discover the acceptor metric indicating that brokers are blocking 80% of connections. What is an acceptor, and why is it blocking our connections? The previous paragraph describes an incident where our Kafka platform experienced a connection storm, leading to significant degradation. This event highlighted the crucial need for effective connection management and exposed our gaps in understanding Kafka’s connection handling, especially with new connections. In this talk, we will share our journey and insights with platform teams maintaining Kafka. You’ll learn how Kafka on Linux servers manages connections and the challenges you might encounter. We will dive into the metrics and mechanisms Kafka offers to detect and protect against connection storms. And last but not least, we’ll share tips from our experience to help you avoid the mistakes we made.
Javier Hortal, Rafael García Ortega