Debezium, Apache Kafka, and an Acyclic Synchronization Algorithm

How do you migrate millions of user accounts from a live legacy platform to a new one with zero downtime, while both systems remain readable, writable, and in sync?

In a previous role at a major classifieds platform in Germany, I worked on a user-migration system based on Apache Kafka and Debezium that kept two live production systems synchronised in real time during a multi-month platform migration. Users could update their data on either platform while records gradually progressed through a dedicated migration state machine.

In this talk, I’ll walk through the architecture and design decisions behind the migration system: why we chose a Debezium-based change data capture (CDC) approach over a hand-rolled transactional outbox, how it was integrated with an existing legacy MySQL cluster, and how forward sync and backsync between the legacy and new systems were designed. I’ll then dive into the acyclic synchronisation algorithm that uses the legacy system’s optimistic-locking version column as a logical clock to break infinite update loops while still allowing controlled updates on both sides.

Participants can expect to learn practical architectural patterns for large-scale data migrations: modelling migration states, running Debezium in production, and using logical clocks to keep two production systems aligned in real time with Apache Kafka.

This session targets intermediate to advanced engineers and architects familiar with Apache Kafka fundamentals.