Modernizing Systems with Kafka Connect

In 2021, we presented “Kafka Connect for Near Real-time Data Moving Pipelines” at Kafka summit. Since then, Kafka Connect has evolved and matured, supporting businesses move data efficiently in near-real time.  

  In today's rapidly evolving technological landscape, the need to migrate data from traditional databases to modern databases is important for business efficiency and staying competitive.  

  Sybase Exit is one of the key deliverables for JPMC across multiple line of businesses for the year 2023 & 2024. Most critical requirement for Sybase Exit is to support transporting data to various downstream systems. Kafka Connect is key facilitator of this data migration pipeline.  

  By deploying Kafka connect with CDC connector for SQL Server, real-time database changes are produced to the kafka topic, and subsequently consumed by one-or-many consumers like different JDBC compliant databases using sink connectors.  

  In Asset and wealth management, multiple complex applications involving thousands of database objects and multitude of replication flows are already live using kafka connect solution, with many more in pipeline for year 2024.  

  This talk delves into the intricacies of implementing a successful data migration strategy using Kafka Connect. Traditionally, it has been operationally challenging to adopt different replication tools for each data source and target, leading to a fragmented ecosystem. These tools are expensive, support a single system and typically have a learning curve. Kafka connect with its rich library of connectors, seamlessly integrates with various systems, and provides a unified approach to data streaming.  

  Attendees will gain insights into the solution devised during migration process, from ensuring data integrity and consistency to optimizing performance and scalability. We will talk about role of kafka connect, design considerations for efficient data movement, monitoring and troubleshooting and best practices.