Streaming Queries Without Compromise

Modern databases excel at analysis when data changes infrequently. However, for rapidly changing, many custom systems have been built under the guise of "streaming systems". To offer near-real time answers, streaming systems compromise on the expressiveness of the computations they perform.    

 We argue that such compromise is unnecessary. Based on a new theoretical foundation [1] we have built Feldera, a streaming query engine which can execute any traditional database computation in streaming mode, analyzing data as it flows in, and producing near-instantaneous results. The core of our system is an algorithm that converts an arbitrary query on data tables into a query that computes on change streams.. We describe the core ideas behind technology and give a demo of our open-source implementation.    

 [1] DBSP: Automatic Incremental View Maintenance for Rich Query Languages, best paper award at the 2023 conference of Very Large Databases (VLDB)