From Zero to Hero: petabyte-scale Tiered Storage lessons

Whether you’re running mission critical applications or just shipping logs in real time, Tiered Storage can make your Kafka Cluster cheaper, easier to manage and faster. To understand the benefits, tradeoffs and the development history join this talk where we’ll uncover KIP-405 and showcase how the community backed up this important feature for Apache Kafka.

We’ll rollback the KIP history, starting from 2018, to understand the major milestones and share details on how major industry leaders like Apple, Datadog and Slack helped out and tested both the Tiered Storage functionality and the first AWS S3 open source plugin. Furthermore, we’ll share details, gotchas, and tradeoffs of users successfully adopting Tiered Storage in production at scale, surpassing 150 GB/s of throughput.

If you want to optimize your Apache Kafka cluster for performance, cost, and overall health, this session is for you.