Building Reliable CDC and Kafka Mirroring Pipelines at Trillion-Message Scale

At large scale, reliability is unforgiving. When a data platform processes trillions of events per day, even small delays or inconsistencies can ripple across analytics, AI systems, and customer-facing products. In these environments, Change Data Capture (CDC) pipelines are no longer just ingestion tools — they become core production infrastructure with strict latency and correctness requirements.

In this talk, we’ll share lessons from operating Brooklin, an open-source data streaming platform used at LinkedIn to run reliable CDC and Kafka mirroring pipelines at massive scale. Brooklin processes over 7 trillion messages per day across 50+ clusters, mirrors 100k+ Kafka topics, and supports sub-minute SLAs for critical workloads spanning multiple teams and use cases.

Rather than focusing on how to build CDC systems from scratch, this session emphasizes how platform teams can adopt proven patterns to operate CDC and Kafka mirroring reliably in real-world environments. We’ll discuss common CDC use cases across database-heavy organizations, including capturing changes from systems such as MySQL, Oracle, and TiDB, streaming them into Apache Kafka, and mirroring data across Kafka clusters for isolation, multi-region deployments, and organizational boundaries.

This session is aimed at intermediate to advanced data engineers and platform teams. Rather than diving into low-level internals or how to build CDC from scratch, we’ll focus on practical design and operational strategies for adopting and operating CDC and Kafka mirroring platforms at scale: partitioning and throughput considerations, handling schema evolution, managing backpressure, and supporting differentiated SLAs—from near-real-time (≈1 minute) to relaxed latency (≈30 minutes)—across shared infrastructure.

Brooklin, an open-source data streaming platform, will be presented as a reference implementation that demonstrates how these patterns work in practice. We’ll share how similar approaches can be adopted by other organizations building CDC and Kafka mirroring pipelines across diverse databases and environments.

Attendees will leave with concrete insights into designing reliable CDC architectures, understanding real-world failure modes, and applying proven patterns to build production-grade streaming systems.