Democratising Stream Processing: How Netflix Empowers Teams with Data Mesh and Streaming SQL

As data volume and velocity continue to grow at unprecedented rates, organisations are increasingly turning to stream processing to unlock real-time insights and fuel data-driven decision-making. This presentation will explore how Netflix has evolved its Data Mesh platform, a distributed data architecture, to embrace the power of Streaming SQL. The session will showcase the journey from the initial implementation of Data Mesh as a data movement platform to its transformation into a comprehensive stream processing powerhouse with the integration of the Data Mesh SQL Processor.

Attendees will learn about the challenges Netflix faced with its earlier system, which relied on pre-built processors and the low-level Flink DataStream API, leading to limitations in expressiveness and a steep learning curve for developers. The presentation will then unveil the innovative solution: the Data Mesh SQL Processor, a platform-managed Flink job that harnesses the familiarity and versatility of SQL to simplify stream processing. Through practical examples, attendees will discover how the SQL Processor interacts with Flink's Table API, converting data streams into dynamic tables for seamless SQL processing. Moreover, the session will spotlight the platform's user-friendly SQL-centric features, including an interactive query mode for live data sampling, real-time query validation, and automated schema inference.