MirrorMaker 2’s Offset Translation Isn’t Exactly Once (and That’s Okay)

Is your streaming application ready for a cluster failover event? MirrorMaker 2 offers replication of topic data and consumer offsets between Kafka clusters, and allows applications to recover from the loss of a Kafka cluster. MirrorMaker2 also includes support for exactly-once data replication with KIP-618, preventing errors from duplicating data. But turning on exactly-once doesn’t prevent data from being re-delivered to your application when using translated consumer offsets. Why?  

  In this talk, we’ll be discussing the history of offset translation across the MirrorMaker2 versions, the existing translation algorithm, and its weaknesses. We’ll explain the design constraints surrounding the algorithm, and how MirrorMaker2 tries to strike a balance between memory usage, offset accuracy, offset precision, and translation availability. Like anything involving these kinds of trade-offs, the result isn’t perfect. So, we’ll also describe what you can expect from your offset translation in practice and explain how to get the best results from offset translation in your environment. Be prepared for your next Kafka failover by learning about the offset translation system and its limitations.