Current London 2025
Session Archive
Check out our session archive to catch up on anything you missed or rewatch your favorites to make sure you hear all of the industry-changing insights from the best minds in data streaming.


How Kafka's new KRaft protocol replaces ZK for partition management, and improves data performance
Partitions and Data Performance" delves into significant changes introduced in Apache Kafka with the introduction of KRaft mode, which stands for Kafka Raft Metadata mode. Traditionally, Apache Kafka, a popular distributed event streaming platform, has relied on Apache ZooKeeper for managing and coordinating Kafka brokers and clusters. However, the dependency on ZooKeeper posed several limitations and complexities, particularly in the areas of scalability, operational simplicity, and performance. In an ambitious move to address these challenges, the Kafka has developed the KRaft mode, essentially removing the dependency on ZooKeeper. we will discuss how KRaft mode simplifies the architecture by integrating the metadata management directly into Kafka, thereby making the system more straightforward to manage and potentially enhancing overall performance. Key points highlighted: 1. Introduction of KRaft Mode: The motivation behind moving Kafka to KRaft mode, emphasizing the desire to eliminate external dependencies and streamline the operation of Kafka clusters. 2. Performance Impacts: It explores the potential impacts of KRaft mode on partitions and data performance. Early benchmarking and testing suggest that KRaft could lead to performance improvements, particularly in reducing latency and enhancing throughput. However, the performance gains can vary based on different deployment scenarios and workloads. 3. Operational Simplicity: By removing ZooKeeper, Kafka strives to reduce the operational burden. This simplification is anticipated to make it easier to deploy, manage, and scale Kafka clusters, which is particularly beneficial in large-scale environments. 4. Migration Considerations: This touches upon considerations for users planning to migrate from ZooKeeper to KRaft mode. It highlights the importance of a thoughtful migration strategy to ensure system stability and data integrity.
Amit Nijhawan


CLI tools you need to know for your everyday Kafka
Apache Kafka is all over the place! Now you can begin using brokers, topics, and clusters. If you, like many other users, prefer the terminal to the graphical interface or web consoles, you should be familiar with these CLI tools to increase your productivity. It is a collection of applications that can assist you with everything from creating a cluster to managing your Kafka Connect connectors or Kafka users. Join us as we go over some of the most practical CLIs for Kafka-related tasks as well as some of the fundamental commands that will help you out. Starting with the scripts that are part of the Apache Kafka distribution, we'll move on to more general tools like kcat for Kafka and kcctl for Kafka Connect. Last but not least, if you are using Kubernetes, we will discuss tools for managing customer resources, such as kubectl and strimzi-kafka-cli.
Hugo Guerrero


How Do You Architect a Kafka Streams Topology?
Struggled with the complexity of designing Kafka Streams applications? Without sufficient up-front architecture work, it’s all too easy to stumble into misunderstandings, rework, or outright failure. Although standards like UML and C4 model have guided software designs for years, stream processing has lacked a visual framework - until now. KSTD (Kafka Streams Topology Design) introduces an open standard and component library for describing and visualising Kafka Stream Topologies with Excalidraw. Simple principles ensure teams can keep diagrams simple yet include important details, build trust in their designs, and streamline the development lifecycle. You will learn how standardised diagrams support team alignment, and how KSTD fosters consistent and clear communication for Kafka Streams. Design up-front, avoid mistakes, save time, and build trust.
Hartmut Armbruster


Don't Judge the Consumer by Its Lag: Uncovering the Metrics That Truly Matter
In today’s fast-paced world of real-time data processing, Apache Kafka has become essential for managing massive streams of information. A key performance metric is consumer lag—the number of messages waiting unprocessed in a consumer group. At first glance, rising lag appears to signal that consumers are falling behind. Yet, this metric alone can be misleading. Imagine a busy restaurant where orders pile up on the counter. It might be tempting to blame the chefs, but delays could also stem from late ingredient deliveries or a malfunctioning oven. Similarly, spikes in consumer lag might not indicate a failing consumer at all; they can result from external factors like sluggish downstream systems, temporary bottlenecks in external services, or sudden surges in data volume. This presentation challenges the conventional reliance on consumer lag as the sole indicator of performance. We will explore how integrating additional metrics—such as message ingestion rates, processing throughput, and the health of interconnected services—provides a more holistic view of your Kafka ecosystem. Through real-world case studies and practical insights, you’ll learn to diagnose issues more accurately and uncover hidden bottlenecks that might otherwise go unnoticed. Join us as we peel back the layers of Kafka’s consumer dynamics and move beyond a single metric. Discover strategies to optimize your data pipelines, ensuring they remain robust and agile amid evolving challenges.
Jungwook Song


Tick-Tock with Kafka: Building Scalable Timer Functionality for Microservices
Timers are a cornerstone of any software system, yet traditional implementations often rely on in-memory solutions or RDBMS dependencies. In this talk, I’ll present a unique approach that leverages Kafka alone to power timer functionality—eliminating the need for RDBMS and embracing a distributed architecture. Using Kafka Streams, I’ll demonstrate how to efficiently schedule delayed work at web scale, enabling resilient and scalable microservices.
Sergey Zyrianov


Global Schema Registry: Door to Data Catalog
Data sharing across cloud service providers is emerging as a mission-critical need for large-scale enterprises and for those looking for cloud agnostic event streaming solutions. While this can be achieved with Kafka multi-region architecture for high availability of data, it still remains a challenge for clients to establish data contracts and evolve their schemas to be in-sync across Kafka clients. In this talk, we will discuss how Fidelity Investments designed a multi-cloud global registry for schemas using schema registry as a centralized repository for managing schemas enterprise-wide. We will also deep dive into the topology of our global schema registry service and demonstrate how it remains resilient over different failure scenarios (region/CSP). We will review metrics that are monitored for deeper observability and benefits such as the simplification of data contracts between producers and consumers and the untangling of data sharing channels across organizational units. Whether you are an analyst or an architect, this session will improve your ability to discover, manage, and correlate event schemas across a wide range of personas.
Nagashree B