Current London 2025
Session Archive
Check out our session archive to catch up on anything you missed or rewatch your favorites to make sure you hear all of the industry-changing insights from the best minds in data streaming.
Ensuring Client Continuity in Kafka: High Availability in Confluent Kafka
Managing large-scale Kafka clusters is both a technical challenge and an art. At Trendyol, our Data Streaming team operates Kafka as the backbone of a vast event-driven ecosystem, ensuring stability and seamless client experiences. However, we faced recurring issues during broker restarts—applications experienced connectivity errors due to misconfigured topics and improper bootstrap server configurations. To address this, we leveraged Confluent Stretch Kafka across multiple data centers, enabling automatic leader elections without service disruptions. Additionally, we enforced topic creation and alter policies and built a custom Prometheus exporter to detect misconfigured topics in real time, allowing us to notify owners and take corrective actions proactively. Through rigorous alerting mechanisms and enforcement via our Internal Development Platform (IDP), we have successfully eliminated disruptions during broker restarts, enabling smooth cluster upgrades and chaos testing. This session will provide practical insights into architecting resilient Kafka deployments, enforcing best practices, and ensuring high availability in a production environment handling thousands of clients. Attendees will learn: How multi-DC Kafka clusters ensure client continuity The impact of misconfigured replication factors and how to prevent them How real-time monitoring and alerts reduce operational risks Practical strategies to enforce resilient topic configurations
Yalın Doğu Şahin, Mehmetcan Güleşçi
Simplifying Real-Time Vector Store Ingestion with Apache Flink
Retrieval-Augmented Generation (RAG) has become a foundational paradigm that augments the capabilities of language models—small or large—by attaching information stored in vector databases to provide grounding data. While the concept is straightforward, maintaining up-to-date embeddings as data constantly evolves across various source systems remains a persistent challenge. This lighting talk explores how to build a real-time vector ingestion pipeline on top of Apache Flink and its extensive connector ecosystem to keep vector stores fresh at all times seamlessly. To eliminate the need for custom code while still preserving a reasonable level of configurability, a handful of composable user-defined functions (UDFs) are discussed to address loading, parsing, chunking, and embedding of data directly from within Flink's Table API or Flink SQL jobs. Easy-to-follow examples demonstrate how the discussed approach helps to significantly lower the entry barrier for RAG adoption, ensuring that retrieval remains consistent with your latest knowledge.
Hans-Peter Grahsl
How Kafka's new KRaft protocol replaces ZK for partition management, and improves data performance
Partitions and Data Performance" delves into significant changes introduced in Apache Kafka with the introduction of KRaft mode, which stands for Kafka Raft Metadata mode. Traditionally, Apache Kafka, a popular distributed event streaming platform, has relied on Apache ZooKeeper for managing and coordinating Kafka brokers and clusters. However, the dependency on ZooKeeper posed several limitations and complexities, particularly in the areas of scalability, operational simplicity, and performance. In an ambitious move to address these challenges, the Kafka has developed the KRaft mode, essentially removing the dependency on ZooKeeper. we will discuss how KRaft mode simplifies the architecture by integrating the metadata management directly into Kafka, thereby making the system more straightforward to manage and potentially enhancing overall performance. Key points highlighted: 1. Introduction of KRaft Mode: The motivation behind moving Kafka to KRaft mode, emphasizing the desire to eliminate external dependencies and streamline the operation of Kafka clusters. 2. Performance Impacts: It explores the potential impacts of KRaft mode on partitions and data performance. Early benchmarking and testing suggest that KRaft could lead to performance improvements, particularly in reducing latency and enhancing throughput. However, the performance gains can vary based on different deployment scenarios and workloads. 3. Operational Simplicity: By removing ZooKeeper, Kafka strives to reduce the operational burden. This simplification is anticipated to make it easier to deploy, manage, and scale Kafka clusters, which is particularly beneficial in large-scale environments. 4. Migration Considerations: This touches upon considerations for users planning to migrate from ZooKeeper to KRaft mode. It highlights the importance of a thoughtful migration strategy to ensure system stability and data integrity.
Amit Nijhawan
CLI tools you need to know for your everyday Kafka
Apache Kafka is all over the place! Now you can begin using brokers, topics, and clusters. If you, like many other users, prefer the terminal to the graphical interface or web consoles, you should be familiar with these CLI tools to increase your productivity. It is a collection of applications that can assist you with everything from creating a cluster to managing your Kafka Connect connectors or Kafka users. Join us as we go over some of the most practical CLIs for Kafka-related tasks as well as some of the fundamental commands that will help you out. Starting with the scripts that are part of the Apache Kafka distribution, we'll move on to more general tools like kcat for Kafka and kcctl for Kafka Connect. Last but not least, if you are using Kubernetes, we will discuss tools for managing customer resources, such as kubectl and strimzi-kafka-cli.
Hugo Guerrero
How Do You Architect a Kafka Streams Topology?
Struggled with the complexity of designing Kafka Streams applications? Without sufficient up-front architecture work, it’s all too easy to stumble into misunderstandings, rework, or outright failure. Although standards like UML and C4 model have guided software designs for years, stream processing has lacked a visual framework - until now. KSTD (Kafka Streams Topology Design) introduces an open standard and component library for describing and visualising Kafka Stream Topologies with Excalidraw. Simple principles ensure teams can keep diagrams simple yet include important details, build trust in their designs, and streamline the development lifecycle. You will learn how standardised diagrams support team alignment, and how KSTD fosters consistent and clear communication for Kafka Streams. Design up-front, avoid mistakes, save time, and build trust.
Hartmut Armbruster
Don't Judge the Consumer by Its Lag: Uncovering the Metrics That Truly Matter
In today’s fast-paced world of real-time data processing, Apache Kafka has become essential for managing massive streams of information. A key performance metric is consumer lag—the number of messages waiting unprocessed in a consumer group. At first glance, rising lag appears to signal that consumers are falling behind. Yet, this metric alone can be misleading. Imagine a busy restaurant where orders pile up on the counter. It might be tempting to blame the chefs, but delays could also stem from late ingredient deliveries or a malfunctioning oven. Similarly, spikes in consumer lag might not indicate a failing consumer at all; they can result from external factors like sluggish downstream systems, temporary bottlenecks in external services, or sudden surges in data volume. This presentation challenges the conventional reliance on consumer lag as the sole indicator of performance. We will explore how integrating additional metrics—such as message ingestion rates, processing throughput, and the health of interconnected services—provides a more holistic view of your Kafka ecosystem. Through real-world case studies and practical insights, you’ll learn to diagnose issues more accurately and uncover hidden bottlenecks that might otherwise go unnoticed. Join us as we peel back the layers of Kafka’s consumer dynamics and move beyond a single metric. Discover strategies to optimize your data pipelines, ensuring they remain robust and agile amid evolving challenges.
Jungwook Song