Events and Entities, a Dichotomy for the Streaming World

The dichotomy of OLTP versus OLAP is very well known, but perhaps less well known is that the terms were coined circa 1993, i.e. it's only 30 years old. About 15 years ago, NoSQL was all the rage, as the boom of the internet meant that technology companies all started fighting trying to get horizontal scale out of systems built for vertical scale. This has given way to a wide expanse (perhaps it's even fair to call it an overwhelming expanse) of technology choices: Key-Value Stores, Document Stores, Search Engines, Streaming PubSub, Stream Processors, HTAP, Real-time Analytics, Cloud Data Warehouses, etc. etc.

The Enterprise Data Supply Chain

In the era of digital transformation, effective data management is crucial for maintaining transparency, consistency, and governance. In their session, "The Enterprise Data Supply Chain," Adam Tybor and Tom Mahaffey, AI and Data Architects for Accenture Global IT, will presentAccenture's blueprint for managing data from source to consumption.They will highlight the importance of source-aligned data products reusable across various channels, including Generative AI and RAG models. Key principles include broadening data producers and consumers, leveraging metadata for automation and traceability, and utilizing cloud-native capabilities and real-time data movement for optimization.

Powering Intelligent AI Apps with Confluent Cloud on Azure

Discover how the Native ISV Offering on Azure can simplify your AI and cloud projects with features like centralized billing, seamless integration, and automated data pipelines. In this session, we'll tackle common AI challenges, including data processing and transformation, and demonstrate how our tools can accelerate your work. We’ll wrap up with a live demo and share self-service video tutorials and hands-on code samples, so you can get started quickly with your Azure account. Perfect for developers and cloud architects looking to enhance their AI capabilities.

Immediate Infinite Zoom: Revolutionizing Real-time Data Interaction with Imply

The data coming into Confluent and Apache Kafka® is fresh and highly dynamic. Traditionally, gaining real-time insights has been challenging, often requiring multiple tools and resulting in a fragmented experience as users switch between different levels of detail.Imply transforms this process by allowing users to instantly explore data at varying levels of detail—much like navigating a digital map. Users can seamlessly transition from high-level overviews, spotting trends and patterns, to detailed views that examine specific events closely. This capability preserves the full context of real-time data exploration, ensuring that insights remain intact as users shift perspectives.Join this session to discover how Imply is helping companies to build real-time analytics applications with immediate, infinite zoom capabilities. We’ll showcase real-world examples of companies using Imply to bridge the gap between strategic insights and detailed analysis, offering a unified solution for real-time analytics.

Integrating MongoDB data into the Kafka and Confluent Cloud ecosystem through Atlas Stream Processing

Are you streaming data between MongoDB and Kafka and looking to enhance your setup? This talk will dive into various methods for integrations between MongoDB and the Kafka ecosystem. First, we’ll examine the traditional approach with Kafka Connect. Then, we’ll cover MongoDB’s new stream processing solution, Atlas Stream Processing. This fully managed service uses the document model and MongoDB Query API to offer a user-friendly, cost-effective, and high-performance way to handle your data. You’ll leave with a clear understanding of Atlas Stream Processing’s capabilities and how to apply them to your streaming needs with Kafka and Confluent Cloud.

Powering Real-Time Generative AI with Amazon Bedrock and Confluent Streaming

Generative AI thrives on real-time data access for relevant responses. This session covers integrating Confluent's scalable data streaming platform with Amazon Bedrock, a fully-managed Generative AI AWS service, to build innovative solutions. We will explore patterns for ingesting, processing, and serving data streams at scale and how to leverage Confluent's pre-built connectors, stream processing, and low-latency inference to contextualize AI models with real-time data and stream governance for trustworthy generative AI. Learn to create real-time knowledge bases, accelerate innovation cycles, and build secure AI applications. Enable real-time context, scalability, flexibility, and exceptional customer experiences.

Leveraging LLMs for SQL Query Automation and Real-Time Data Streaming with Cardinal Health Sonexus, Google Cloud Vertex AI, and Confluent

In this breakout session, we will explore how to harness the power of Large Language Models (LLMs) to automate SQL query generation and seamlessly stream results into Confluent using Vertex AI. This session will offer an in-depth exploration of how to integrate advanced AI models into data pipelines, facilitating real-time analytics and decision-making in healthcare.Attendees will learn how to leverage LLMs to simplify SQL query creation, enhance data processing workflows, and utilize Vertex AI for scalable and efficient machine learning operations. Join us to discover innovative ways to optimize your data infrastructure and drive actionable insights with cutting-edge AI and data streaming technologies.

Getting Gory with Active-Active Multi-Region Disaster Recovery

Kafka is one of the most durable and reliable data systems in existence. Thousands of mission critical businesses have been ironing out the wrinkles over the last decade. Once you move your event driven apps or streaming pipelines out of the confines of a single region the route is not that well understood. Asynchronous replication and eventual consistency add several challenges irrespective of the technology you are using. Businesses have a strong desire to employ an active-active strategy to gain the benefits of data locality and to mitigate the blast radius of regional failures. When you sum this all up, you have yourself a complex challenge. In this talk, we will cover:

Addressing Streaming ETL Pipelines Challenges: Delving into Flink CDC

Data freshness significantly impacts the value of data insights, particularly for business data housed within databases. Establishing a more real-time synchronization pipeline is crucial for deriving actionable insights from this data. However, constructing such a pipeline encounters various challenges. Common issues include frequent table schema changes in the upstream database and the need to add or delete tables according to evolving business requirements. Moreover, flexibly scaling resources during historical reading and log reading stages proves difficult, while synchronizing multiple tables or the entire database consumes substantial resources. Additionally, constructing job workflows using the DataStream API poses its own complexities.

Kafka Unlocks Scalable Vector Search in Milvus

Powered by the popularity of ChatGPT and other large autoregressive language models, we've seen a huge surge in interest for vector search in 2023. In applications that leverage [RAG](https://zilliz.com/use-cases/llm-retrieval-augmented-generation), the Milvus vector database is commonly used as a way to retrieve relevant document chucks and other short-form text. The applicability of text-based RAG has led to widespread adoption, from single-person tech startups to Fortune 500 fintech companies.

How the World's Largest Options Clearinghouse Moved From Batch to Stream Processing to Manage Risk

In this session, we will review how the OCC performs complex analytics using Kafka and Flink on a data stream containing prices for over 2 million financial instruments per second and billions of open option positions. We will also review how Ness and the OCC satisfied the SEC's requirement to build isolated Kafka + Flink infrastructure in both AWS and an on-premises data-center for disaster recovery purposes. Using Kafka + Flink, the OCC radically improved performance for their core margin calculation system, going from an 8+ hour, overnight batch process, to a 5 minute intraday operation.

Kafka Streams as a Data Store for a Workflow Engine

At LittleHorse, we are building a high-performance workflow engine for microservice orchestration. We have a long list of performance-related requirements, including low latency, horizontal scalability for throughput, transactional semantics, and high availability. Long story short: we decided to build our own data store on top of Kafka Streams.

Trust and Safety at Indeed: Detecting and Preventing Fraud in Real-time

At Indeed, we help people get jobs. The Trust & Safety team at Indeed contributes to our mission by developing, enforcing, and advocating for quality, creates policies to govern content, and evaluates many different fraud vectors including job postings, messaging, data export, and more. This ensures the safety of jobseekers and that all employers are able to have a robust pool of qualified applicants to meet their hiring needs.

Building a Scalable Flink Platform: A Tale of 15,000 Jobs at Netflix

At Netflix, we operate over 15,000 Apache Flink jobs, processing over 60 PB of data per day. Apache Flink is widely used across many engineering organizations within Netflix to power stream processing use cases such as data movement, personalization, messaging, and finance. This diverse usage of Flink manifests itself in different job characteristics, such as cluster size (up to 2000 parallelism), state size (stateless to 4 TB), and complexity of job graphs.

The Performance of Kafka's New Consumer Rebalance Protocol

KIP-848 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol) proposed a new paradigm of the rebalance protocol for Kafka consumers. This prompted a large architectural change in the group coordinator, primarily moving to an event-driven architecture.

Effective Data Lineage Strategies for Real-Time Systems

Data in motion is data on a journey, and understanding that journey is a crucial part of streaming governance. Data lineage enables organizations to unravel the intricacies of real-time data flows, enhance transparency, and build trust across an event-driven landscape. Data lands on a topic, consumed and transformed, then produced to another topic, repeat. In this way, data travels through any number of topics and intermediary systems in real-time before reaching its final destination. This poses a significant hurdle; without detailed message-level tracking, chain of custody and transformations done on that data can be lost.

Apache Kafka's Common Pitfalls & Intricacies: A Customer Support Perspective

As Apache Kafka gains widespread adoption, an increasing number of people face its pitfalls. Despite completing courses and reading documentation, many encounter hurdles navigating Kafka's subtle complexities.

Streaming Queries Without Compromise

Modern databases excel at analysis when data changes infrequently. However, for rapidly changing, many custom systems have been built under the guise of "streaming systems". To offer near-real time answers, streaming systems compromise on the expressiveness of the computations they perform.

Change Data Capture & Kafka: How Slack Transitioned to CDC with Debezium & Kafka Connect

Change Data Capture (CDC) has emerged as a critical component in modern data processing architectures, enabling organizations to capture and process real-time data changes efficiently. This talk presents a detailed case study of Slack's transition to CDC, highlighting the utilization of Vitess, Kafka, Debezium, and Kafka Connect in revolutionizing data processing workflows.

Unbundle Your Cake and Eat It Too: Redefining the Limits of Kafka Streams

Kafka Streams powers mission critical applications across a variety of use cases at some of the biggest companies in the world. What are the properties that make Kafka Streams so popular amongst developers looking to build event driven applications on top of Kafka? What are the operational risks associated with using Kafka Streams in your mission critical applications? Most importantly, what can you do about those risks?

Atlassian's Lithium Platform: Dynamic, Self Hosted, and Distributed Ephemeral Streaming Pipelines

Atlassian has numerous use cases that require moving large amounts of data around, validating and transforming it in flight. Platforms such as Apache Flink can be excellent choices for moving and transforming data at scale - effectively streaming ETL. However our uses cases present some challenges - they require stream processing pipelines to be entirely provisioned at runtime, including dedicated Kafka topics, parallelism, selection of stream processors with appropriate and available compute resources, and our stream processors must be hostable directly in product and platform services to enable in-process access to service context in order to meet throughput and concurrency goals. In addition, our pipelines require coordination amongst sources, transforms, and sinks that exist in different product and infrastructure services. This led us to build the Lithium Platform - designed to meet these needs. It is 100% event driven and is built on Kafka and Kafka Streams.

Unlocking the Power of Flink's Table API: A Deep Dive into Advanced Capabilities and Use Cases

In today's fast-paced digital world, the ability to process and analyze real-time data streams sets the foundation for innovative applications across industries. Apache Flink's Table API enables developers to write complex data processing queries in a simple, declarative manner using Java, Python or even combined with SQL. This talk aims to unravel all the capabilities of the Flink's Table API, providing attendees with a thorough understanding of its architecture, functionalities, and the immense possibilities it unlocks for real-time data analytics.

Atomic Dual-write Recipes with Kafka Two Phase Commit (KIP-939)

Kafka is the central nervous system of organizational data that connects heterogenous data sources and sinks together. KIP-98 introduced exactly-once semantics for data processing within Kafka. KIP-939 brings exactly-once semantics to the next level and enables exactly-once semantics between Kafka and a database.

Optimizing Apache Kafka with GraalVM: Faster, Leaner, and Performant

Docker, a widely adopted container runtime, has revolutionised the way we deploy applications. With the implementation of KIP-975, Apache Kafka 3.7.0 and subsequent versions now come with a docker image.

CDC Pipelines to the Data Lakehouse Using Apache Hudi and Apache Flink

Change Data Capture (CDC) is crucial in capturing and disseminating real-time changes from upstream OLTP databases. Data Lakehouses provide scalability for data exploration on open data formats, but many struggle with record level updates in CDC workloads. Apache Hudi® has emerged as a comprehensive lakehouse platform purpose-built for managing mutable data, facilitating near real-time insertions, updates, and deletes with a robust incremental processing framework. Hudi’s integration with Apache Flink® bridges the gap between the streaming and data lakehouse communities, enabling low-latency, real-time data applications for various use cases such as change log reconcilation, ETL pipelines, streaming joins, and constructing materialized views.

Billion Dollar Frauds - Unleash the Power of Confluent’s Streaming A.I. to Combat Voice Vishing

In the wake of a startling scam where a branch manager in Hong Kong was deceived into transferring $35 million by a deepfake voice, the imperative to fortify against such frauds has escalated. This incident underscores the critical need for advanced defences against voice vishing, particularly as criminals leverage easily accessible generative A.I. to create highly realistic cloned voices. In response, we're running the inference of our Streaming Voice Vishing A.I. model in Confluent and Flink and directly integrating with call center solutions like Genesys to offer Fortune 500 companies a robust shield against these ferocious threats with billions of dollars in losses globally.

Consistency and Streaming: Why You Care Should About It

Streaming systems are often evaluated by a single metric: performance. We look for systems that can compute up-to-date results over volumes of changing data as promptly as possible. But if we stop our evaluation there, we ignore a pretty critical question: do the results we’re computing make any sense?

Real-Time Entity Resolution At Scale

Case management systems (insurance claims systems, benefits administration, claimsprocessing, etc.) are typically forms-based and have long processing times. Learn how real-timeentity resolution – the deduplication of similar data – can drastically help with creating acomprehensive view of your data. In case management, you’ll need to understand who filed forwhich benefits so that you can accurately assess fraud cases, eligibility requirements, and manyothers.

Mastering Kafka at Scale: Unleashing the Power of Temporal at Stripe

Stripe is at the forefront of the payment processing game, handling an astonishing 700 terabytes in Kafka publish throughput daily – a feat that is only possible with the open-source powerhouse Kafka. However, managing 50 Kafka clusters across three environments and multiple regions can be a Herculean task. That's where Stripe's Stream Infrastructure team stepped in, leveraging the incredible capabilities of Temporal to build a state-of-the-art Kafka Control Plane.

The Joy of JARs (and Other Flink SQL Troubleshooting Tales)

Apache Flink® is a powerful thing to work with but, my, it does test one's patience sometimes ;-) Particularly if you're new to using it, understanding how to troubleshoot problems that arise is an important survival skill to acquire.

Bridging the Kafka/Iceberg Divide

Apache Kafka® has long been the de-facto standard for managing and distributing data in motion, and Apache Iceberg has been on the rise in a similar role for data at rest. A common pattern is to collect streaming data from Kafka into Iceberg so that it can be efficiently queried. However, this process can be complex, expensive, and error-prone. It entails a sequence of connectors, streaming, and batch jobs. These data pipelines consist of multiple steps where each step needs to be manually configured, managed, and scaled. This also results in multiple copies of the data without a source of truth. As the scale grows, so does complexity, cost, and toil.

Topic Federation: Enhance Kafka Availabilty with Sharded Topics Across Clusters

Powering various business verticals (e.g. ride sharing, eats, and delivery) at Uber, the Kafka messaging system is one of the most critical components of the real-time data infrastructure and its availability is of utmost importance. Intuitive approaches such as scaling (vertical or horizontal) out a cluster, and adding more replicas for topics can enhance Kafka availability to a certain degree. However, it fails to address the fundamental issue of cluster-level degradation.

Which Data Format Should I Use in Apache Kafka®?

You're just starting your data streaming journey or are extending it to include Schema Registry and you learned that multiple serialization formats are supported. How do you know which one(s) to choose? What if the choice is not yours? If you are offering the data streaming service as a platform to your internal customers, what kind of guidance should you offer them when choosing a data format? Is everything which is possible advisable? What implications does mixing data formats have? How do different formats differ from each other in terms of limitations, evolution and linking options? How does this affect consumers?

Pinterest’s Journey to an Automated, Efficient, and Low-Maintenance PubSub Platform

At Pinterest, Apache Kafka® runs on ~2000 brokers across 50+ clusters, moving data at petabyte scale every day. Through investing in automation and focusing on cost efficiency, we have seen positive results in dev velocity, scalability, stability, and cost footprint, enabling everyone to quickly and easily launch data streaming use cases that power business growth, while keeping our platform low-maintenance.

Generating Deterministic, Synthetic Workloads for Stream Processing

Do you want to kick the tires on Apache Flink®? Do you want to make sure it can perform well for your target use case? How can you do that?

Modernized Trading: Managing Market Data with Confluent Cloud at Vanguard

Vanguard is one of the world’s most respected investment management companies, offering a broad selection of investments, advice, retirement services, and insights to individual investors, institutions, and financial professionals.

Beyond Tiered Storage: Deep dive into WarpStream's Storage Engine

In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka® protocol compatible system that has zero local disks.In this talk we'll cover how to design a storage engine from the ground up around commodity object storage.

Flinking Enrichment: Shouldn't This Be Easier?

A common pattern in stream processing applications is to enrich incoming events with contextual information. For example, a fraud detection pipeline might enrich incoming transactions with information about the customer, the merchant, previous transactions, etc, as part of building up a feature vector to hand off to a fraud detection model.

Real-time Adaptive Controls for Kafka Consumers

Modern Kafka consumers are equipped with hundreds of tunables. There are a lot of these tunables such as worker pool sizes, autoscaling policies, throttlers and circuit breakers that directly effect the consumer resilience. Finding ideal initial values for these tunables requires deep technical expertise. Also, these workloads change over time, requiring regular effort to re-tune stale parameters. As a consequence, configuration errors have become a source of operational toil and one of the major causes of overload, cascading service and system failures across the industry. Consumers should aim to expose a minimal configuration surface by dynamically adjusting parameters based on observations. Praveen will provide a deep-dive into how CrowdStrike is using real-time Adaptive Controls(inspired from TCP congestion control) to dynamically tune these parameters for improved resiliency using real-time sampling of errors and latencies, removing the need for periodic adjustment. He will also discuss lessons learned deploying the feature to CrowdStrike’s massive production systems that handles multiple trillions of events per day without causing any incidents.

Highly Reliable Streaming Ingestion from Kafka to Iceberg Tables with Seamless Schema Evolution

Streaming data ingestion often has to deal with evolving data formats. Schema of incoming records can change over time as new attributes get added by producers. Even when schemas evolve or records are invalid, the stream processing should continue without interruptions. Graceful handling of schema changes and data errors is critical for zero-downtime in modern real time data platforms.

MirrorMaker 2’s Offset Translation Isn’t Exactly Once (and That’s Okay)

Is your streaming application ready for a cluster failover event? MirrorMaker 2 offers replication of topic data and consumer offsets between Kafka clusters, and allows applications to recover from the loss of a Kafka cluster. MirrorMaker2 also includes support for exactly-once data replication with KIP-618, preventing errors from duplicating data. But turning on exactly-once doesn’t prevent data from being re-delivered to your application when using translated consumer offsets. Why?

Conquering the Hard Bits of Event Processing with Restate & Kafka

We demonstrate how to use Kafka and Restate together to easily build event-driven apps with properties that are normally considered rather hard to do

Protobuf Support in Uber's Real-Time Data Stack

A Real-time data platform (Apache Kafka, Apache Flink and Apache Pinot) plays a crucial role in Uber’s serving stack which primarily uses Avro as serialization format. At the same time, Protobuf is the primary data format at Uber for online applications and services. The discrepancy make engineers to take additional effort in order to convert the Protobuf data model into the Avro data model to tap into the real-time data stack.

Events at the (API) Horizon: How to synthesize domain events & changes from your HTTP/REST APIs

REST-based Request-response APIs are the lifeblood of most architectures. API transactions,specifically involving the CUDs (Creates, Updates, Deletes) are great sources of events and/orchange data. This is also true of various RPC style APIs.However, because of the client-server nature, such events need to be published by the service,using an "outboxing" mechanism and subsequently disseminated to pub-sub event broker,most often Apache Kafka. This involves code changes to the application and in addition, a moreexpansive solution like CDC (change-data-capture) to collect those changes from the outboxtable.

Understanding Stream Table Duality Using Kafka, Flink SQL and Debezium Format

Kafka Streams, ksqlDB or Flink SQL are popular processing engines that enable us to run SQL queries on top of streaming data. Isn't it fascinating that we can run SQL queries on top of streaming data as if they were relational tables, or convert a table into a stream of changelog events? This is known as the stream-table duality.

Speed of Apache Pinot at the Cost of Cloud Object Storage with Tiered Storage in StarTree Cloud

For real-time analytics, you need systems that can provide ultra low latency (ms) and extremely high throughput (1000s of qps). One such system is Apache Pinot, which is excellent for real-time analytics use cases like user-facing analytics and personalization. The users of Pinot love the speed of Pinot and want to use Pinot for all their use cases - internal analytics, ad hoc analytics, reporting. For such use cases, you typically need to store really long retention data.

From 📎 to 🧠: Building Gen AI Apps & Copilots on Streaming Data with Flink SQL

Gen AI apps such as intelligent business-specific Copilots can serve as a vital link between foundational models and data streaming, enhancing developer and employee productivity. Enabling users to ask questions about streaming data in natural language can provide business and engineering teams with dynamic, real-time access to organizational knowledge, enhancing employee productivity and improving workflows across enterprise operations.

Vector Databases (pg_vector) and Real-Time Analytics

This session introduces the basics of vector indexes and vector databases, providing essential knowledge on their use. We will guide you through setting up a Real-Time RAG (Retrieval-Augmented Generation) data pipeline in Kafka and Pinot, a real-time OLAP and vector store. We'll demonstrate how to load documents from a website and perform question-answering on that content. I'll also talk about how AI and Real-Time analytics are converging. Key technical details covered include vector indexes, Apache Pinot, RAG, and creating embeddings for similarity searches. By the end of the session, you will understand how similarity search can be applied in real-time scenarios. This session is ideal for anyone interested in RAG, AI, vector databases, and real-time analytics.

The Kafka Consumer: An Unexpected Journey of Data Consumption

Once your data is stored on your Apache Kafka® cluster, the next step is to consume that data and do something interesting with it. Enter: Kafka Consumers. We all know how to set up a Kafka Consumer to poll data… but do you know how a consumer fetches the data from the cluster? Let’s find out!

Centralized Monitoring and Troubleshooting for Kafka Clients

Kafka clients collect a lot of metrics that can be used for troubleshooting. When you have ten clients in a cluster, it's already hard to coordinate collection of the metrics. When you have thousands of clients, it's really hard. Many cluster operators do not have control over the clients, and the lack of consistent telemetry across clients is an operational gap.

A Tale of Two Platforms: Routing Events at the Edge

Embark on a transformative journey from legacy operations to cloud innovation. A small team began a mission that would evolve into the event-driven backbone of a global Enterprise, modernizing its architecture. We'll explore the construction of a robust event streaming platform that tackled two core challenges: interfacing with legacy systems constrained by limited hardware and isolated data silos and innovating a solution that allowed edge systems to run in the most remote locations without internet connectivity, ensuring seamless data integration with Kafka and our new cloud platform upon reconnection.

Kafka Streams at Scale: Avoiding Common Pitfalls

At LittleHorse, we use Kafka Streams extensively for a use-case that requires transactional semantics, high throughput, low latency, large amounts of state storage, and high availability. Seems like a tall ask, eh?

Unlocking Real-Time Insights: Uber Freight's Evolution from Batch to Streaming Analytics

With Uber Freight overseeing a staggering billion loads annually, the demand for real-time analytics is paramount. Delve into our journey as we transitioned from traditional data aggregation methods reliant on stored procedures to a dynamic real-time application powered by Apache Kafka and Flink. Explore on how this innovative architecture revolutionized our operations, slashing data aggregation latency from fifteen minutes to mere seconds.

Self-Hosting Kafka at Scale: Netflix's Journey and Challenges

At Netflix, our usage of Kafka has significantly grown and evolved over the years, paralleling our product development and expansion. This talk aims to share our unique journey with Kafka, tracing our path from using it behind the scenes for building Keystone to now offering Kafka as a Service.

Fortifying Your Transactions with KIP-890

Transactions are crucial to exactly once semantics in Apache Kafka®. But over the years we've found some gaps in the protocol. Enter KIP-890: Transactions Server-Side Defense! The goal of this KIP is to address some known issues like hanging transactions and to strengthen the protocol to make sure we write our messages not zero times or two times but exactly once! This session will cover the rationale of the KIP, the new changes introduced by the KIP, and future work made possible by the KIP. It will also cover how to set up your Kafka cluster with the improved transactional protocol.

Enabling Flink's Cloud-Native Future: Introducing Disaggregated State in Flink 2.0

In this presentation, I will introduce disaggregated state, a novel architecture we propose in Flink 2.0. It leverages Distributed File Systems (DFS) as the primary storage solution to address critical challenges faced by Flink in the cloud-native era

Test Data Streaming Applications With Confidence!

Testing real-time data streaming applications is hard but without robust integration tests you cannot ship your code with confidence. In this session we explore how Testcontainers libraries allow you programmatically create, manage lifecycle, and configure ephemeral instances of popular data steaming technologies. From spinning up individual Apache Kafka services to creating complex topologies your tests are in control of the environment they require. You can also run the same setup in your local development environment or in CI pipelines, which makes reproducing issues locally so much easier and fixes - faster. Testcontainers approach gives your flexibility and confidence to reliably test your code, and after this session you'll learn enough to start testing your new and existing data streaming applications.

Streamlining History: ClickHouse & Flink's Fast-Track for Data Backfills

Integrating historical data with real-time streams is a common challenge in data streaming applications. Our approach leverages Apache Flink's Hybrid Source capabilities and ClickHouse's storage efficiency to selectively synchronize only the data that matters. This strategy streamlines the backfill process by leveraging true random access capabilities, unattainable with Kafka alone, dramatically reducing the time and resources required for effective data integration in streaming environments.

Modernizing Systems with Kafka Connect

In 2021, we presented “Kafka Connect for Near Real-time Data Moving Pipelines” at Kafka summit. Since then, Kafka Connect has evolved and matured, supporting businesses move data efficiently in near-real time. In today's rapidly evolving technological landscape, the need to migrate data from traditional databases to modern databases is important for business efficiency and staying competitive.

Taking Charge of Tables with OpenHouse

In the world of Big Data Management, there's always a tussle between control and flexibility! Cloud data warehouse solutions maintain tight control over the systems, which allows data infrastructure teams to ensure proper governance, data integrity, security, and performance. However, these systems often suffer from a lack of flexibility and scale. That's where open source data lake(house) systems come into play.

Kafka - No Rocks, Please : Using Kafka Streams With Alternative State Stores

A popular data transformation library in the Kafka world - Kafka Streams - is bundled with a high-performance key/value store - RocksDB - with many advantages regarding near real-time, high throughput, and low latency processing. But what if RocksDB key/value store semantics are just not enough? What if the use case requires features of other storage engines - document indexing, graph processing, or simply multiple secondary indexes? Can we employ other embeddable storage engines, for example, Lucene, Neo4j, or extend RocksDB-based state stores?

Data Contracts In Practice With Debezium and Apache Flink

Log-based change data capture (CDC) is an invaluable part of the data engineering toolbox: it enables a variety of use cases such as real-time analytics, full-text search, or cache invalidation by publishing data change events from your database. But when publishing change event streams across context or team boundaries, aren’t you tieing external consumers to your application’s data model, thus limiting yourself in evolving the same?

Tracking Locomotive to Train Associations in Kafka

A deep dive into the iterative design process of creating a highly resilient enterprise service for tracking locomotive associations for over a thousand trains all at once.

Timing is Everything: Understanding Event-Time Processing in Flink SQL

In the stream processing context, event-time processing means the events are processed based on when the events occurred, rather than when the events are observed (processing-time) in the system. Apache Flink has a powerful framework for event-time processing, which plays a pivotal role in ensuring temporal order and result accuracy.

A Survey of Fake Data Generation Tools for Data Streaming Applications: Stop Writing Your Own

Dummy data is commonly used for testing applications. For small, functional tests, developers can quickly and easily mock up something close to what they expect. When it comes to data streaming applications, however, generating realistic streams of data that test all the functionality in an application means choosing from one of a crop of data stream generating tools, each with their own nuances, strengths, and limitations. So how does one choose?

Tapping into the Stream: Streamlining Real-time Data Acquisition and Analysis

Robinhood’s Kafka Journey from EC2 to Kubernetes

Using Kafka Streams to Build a Data Pipeline for the Hospitality Industry

Real-time Platform Engineering: Scale Kafka Adoption Through a Kafka-as-APIs Program

Apache Kafka is an integral part of the data infrastructure of organizations aiming to build real-time applications, event-driven architectures, and microservices. However, the true potential of Kafka is only unlocked when Engineering teams across the organization are equipped with seamless access to building on top of Kafka data streams.

Sentiment Analysis in Action with Apache Flink: Building Your Real-time Pipeline

Introducing Apache Flink: 5 Things You Need to Know to Get Started

Breaking the Bottleneck: Decentralizing Data to Power Library Innovation

Otter Romp: Shallow Dives into Kafka Streams

Multi-Cloud Data Sharing: Make the Data Move for You Across CSPs Using Cluster Linking Description

Visualization in Motion: How to Create Effective Data Visualization with Real-Time Data

Scaling Data Ingestion: Overcoming Challenges with Cell Architecture

Streamlining Entry into Streaming Analytics with JupyterHub and Apache Flink

The Architecture of Open Source Message Brokers

Empowering Teams: Learning Strategies That Fuel Kafka Adoption

Speed Wins: The Basics of how to push more queries through each CPU core

Simplifying Apache Kafka® : Easy Configurations for Immediate Impact

Leveling Up Customer Experience with Kafka: From Modeling to Activation

Keeping your State at Bay: Patterns to Limit Kafka Streams Store Sizes

The SQL Ecosystem: Powering the Instant World with 40-Year-Old Legacy?

Folks, THIS is Apache Kafka

3 Lessons We Learned Running Stateful Streaming Pipelines with Apache Flink and Kafka

At Bloomberg, we deal with tremendously high volumes of data. Financial markets move significantly from millisecond to millisecond. Clients must react to these immense amounts of market data and trust Bloomberg to synthesize and analyze the information in real time.

Your Swiss Army Knife for Kafka-based Applications

Imagine you need to prove that a certain record is actually present in a topic. Or you need to quickly copy one topic to another cluster. Or you need to fix a corrupted topic where the schema information is missing. Or one of your colleagues from analytics would like to have a topic in Parquet format, or a manager in Excel... To do these jobs, so far, you would often resort to scripting with the existing command-line tools from the Apache Kafka distribution, or, in more complicated cases, to writing your own Java or Python application. This can be so tedious and time-intensive that it can bring entire Kafka-based projects down. In this lightning talk, you will be thrown into the midst of a typical Kafka nightmare of corrupt topics and schemas, where the deadline for the qualification test looms and there is no time to spare. And we will show you how Kafi, your Open Source Swiss army knife for Kafka, can help you to get out of this horror trip quickly and easily, with just a few lines of Python code.

Treat Data Like APIs

Microservices encapsulate code and data of a domain and establish standard interfaces to access the domain via APIs, but service owners sometimes forget that service data goes into a data lake for analysis or reporting. Traditionally, this data lands in a data lake via change data capture (CDC), and data teams use transformation jobs on this “raw” data to build curated datasets. But this breaks the microservice encapsulation and exposes internal data layout to external offline consumers. Making changes to the database for microservice becomes hard because numerous other data pipelines depend on it downstream. Data bugs are only discovered days after they first occur, because those data assets are separated by a couple of degrees from the team that owns the source of truth microservice database schemas.

“Failure to Connect” Bugs and Issues in Kafka Connect to Watch Out for a Healthier Environment

Kafka Connect is a great product at connecting systems together and ensuring messages are produced and consumed from topics to endpoints. In order to have the highest availability on Connect, you need to understand the “failures to connect” in regard to any bugs and issues to watch out for in Kafka connect.

So You Want to Write a User-Defined Function (UDF) for Flink?

Apache Flink gained a lot of momentum during the past couple of years and is well-known across the data industry for its versatility. Developers and data engineers alike preferably pick higher-level programming abstractions such as Flink's Table API and Flink SQL whenever reasonably possible. And yet, specific requirements might cause the need for writing custom code and integrating with 3rd party libraries - Flink UDFs to the rescue!

SketchDB: Reducing Your AWS Analytics Cost by 10x Using Principled Approximation

Streaming analytics serve a variety of downstream applications. With rising volumes of data and an increased need for real-time analytics, analytics costs are ballooning out of control. Many downstream applications are amenable to working with approximate analytics and thus, methods like sampling are commonly used by practitioners to reduce costs. Unfortunately, in doing so, practitioners are faced with a cost-accuracy dichotomy, having to choose between low cost and high accuracy.

Data State to Business Events: Application of Set Theory to Event Streaming

At ACERTUS, we are using Apache Kafka to deliver real business value to our customers by automating order lifecycle and providing near-real-time business events. One of the tools we created is an elegant architecture for collaboration between business operations and software developers to efficiently transform data events into business events.

The Availa-blitz: Kafka's Scaling Saga for High-Speed Trading

In the fast-paced world of high-speed trading, every millisecond counts. Join us for a lightning talk as we unveil the remarkable journey of scaling Kafka for high-speed trading at Charles Schwab. Discover how Kafka's event streaming architecture revolutionizes real-time data processing, enabling Charles Schwab to handle 2 million messages per second with millisecond latency.

Is Your Kafka 'Truly' Load Balancing? Pro Tips for Leader Partition Distribution with Network Usage

One of the primary goals of distributing leader partitions across all brokers in a Kafka cluster is load balancing. This practice helps prevent overloading any single broker and optimizes the throughput of the entire cluster. The distribution of these leader partitions is mainly determined by the preferred leaders set during the creation of topic partitions.

Streaming Analytics vs. Streaming Database - An Unbiased Comparison

Event stream infrastructure has become an important part of modern data stack in many organizations enabling them to rapidly ingest and process streaming data to gain real-time insight and take actions accordingly. Streaming storage systems such as Apache Kafka provide a storage layer to ingest and make data streams available in real-time. Streaming analytics(stream processing) systems and streaming databases are two platforms enabling users to gain real-time insight from their data streams. However, each of these systems is suitable for different use cases and one system alone may not be the right choice for all of your stream processing needs. In this talk we introduce both streaming analytics(also known as stream processing) and streaming databases and provide an overview of capabilities in each system. We also explore different categories of use cases and discuss which system is more appropriate for these use cases.

Disaster-Proof Data: Navigating Real-Time Streaming with Zero Downtime

Dive into the realm of zero downtime for real-time data streaming apps with a focus on Disaster Recovery best practices. Join me as I share practical insights and experiences from real-world production use cases that leverage multi-regional clusters. Discover how to architect resilient event-driven systems and ensure seamless operation even in the face of unforeseen challenges. Unleash the power of continuous data streaming with confidence!

Breaking Kafka Immutability: Are you mad?

Who hasn't had to reset the offset of a consumer group to skip a defective record? You must stop your applications, identify the malformed records, adjust the offsets of the consumer groups reading this topic using some tools, and then restart everything. That's quite cumbersome, right?Have you ever dreamed of breaking Apache Kafka's immutability to delete or fix a record instead of juggling consumer offsets? What if I tell you that it's possible?

Streaming Entity Resolution for Kafka with Quine

"These two records are really the same thing!" Whether it's misspelled company names, or users who sign up twice with different email addresses, or bank accounts controlled by the same shadowy figure, the data received in a Kafka stream doesn't always resolve to reality we're trying to represent. This problem has been slowing down streaming adoption across the industry—from the world's largest banks, to the small teams trying to deploy their first streaming data pipeline. But cleaning that data quickly has gotten easier recently with new open source streaming graph tools which can perform powerful entity resolution on streaming data at scale while it's still in motion, and even if it's out of order.

Debezium vs. the World: An Overview of the CDC Ecosystem

Change Data Capture (CDC) has become a commodity in data engineering, much in part due to the ever-rising success of Debezium [1]. But is that all there is? In this lightning talk, we’ll outline the current state of the CDC ecosystem, and understand why adopting a Debezium alternative is still a hard sell. If you’ve ever wondered what else is out there, but can’t keep up with the sprawling of new tools in the ecosystem; we’ll wrap it up for you!

Client Cannot Communicate - Lessons From a Decade of Kafka Consulting

We rightly celebrate the successes of high-profile Kafka projects, but what about the less visible failures? Let's delve into the invaluable lessons learned from these disappointments. Allow me to unveil the hidden truths of avoidable failures—strategies that can compromise your project's success, sabotage your engineering endeavors, and subtly diminish your team's productivity. Drawing from extensive experience across numerous Kafka projects in various sectors such as banking, transportation, retail and product development, I've collected my fair set of scars and lessons from the field that everyone should know

Robinhood’s Flink Deployment Practices

Robinhood leverages Flink for many of the stream processing applications that power the platform’s features, including shareholder’s positions, fraud detection, data ingestion, etc. With Kafka involved in almost every mission-critical step of Robinhood’s functionality, Flink is crucial for building scalable, maintainable and reliable stream processing applications.

CLI Tools You Need to Know for Your Everyday Kafka

Apache Kafka is all over the place! Now you can begin using brokers, topics, and clusters. If you, like many other users, prefer the terminal to the graphical interface or web consoles, you should be familiar with these CLI tools to increase your productivity. It is a collection of applications that can assist you with everything from creating a cluster to managing your Kafka Connect connectors or Kafka users.