Beyond Tiered Storage: Deep dive into WarpStream's Storage Engine

In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka® protocol compatible system that has zero local disks.In this talk we'll cover how to design a storage engine from the ground up around commodity object storage. We'll dive deep into WarpStream's storage engine implementation and cover topics like: ingestion, metadata, compaction, fetch, and service discovery.Finally, at the end we'll see how the concept of tiered storage can be re-introduced even in a system that has zero local disks to make different trade-offs around cost and performance.