Reducing Kafka Producer Cross Network Costs with RackAwareStickyPartitioner

Dream11, the world’s largest fantasy sports platform, manages unparalleled scale, with RPM surpassing 300 million during flagship events like IPL 2024. With Kafka producers forming the backbone of real-time data pipelines, Dream11 faced a significant challenge: soaring cross-availability-zone (AZ) network costs due to the indiscriminate partitioning strategy of regular producer partitioners. To address this, Dream11 engineering developed the RackAwareStickyPartitioner, a custom solution for Kafka producers that achieved a 70% reduction in cross-AZ network costs. By intelligently routing producer batches to Kafka partitions within the same AZ, this innovation minimized cross-AZ traffic while preserving high throughput. A 10-day controlled experiment demonstrated a dramatic cost reduction in “DataTransfer-Regional-Bytes” by over 30%. This optimization is tailored for high-throughput scenarios, with careful consideration required for low-volume applications to avoid partition skew. Join this session to explore how Dream11 engineered a cost-efficient solution for Kafka producers at scale, sharing insights on architecture, challenges, and real-world impact.