Conquering the Hard Bits of Event Processing with Restate & Kafka

We demonstrate how to use Kafka and Restate together to easily build event-driven apps with properties that are normally considered rather hard to do, such as:  

 - highly burst-y loads  

 - delaying/postponing certain events  

 - complex state semantics and interactions  

 - high-fan-out routing to consumer APIs  

 - long-running API calls on events (without head-of-the-line waiting)  

 - state machines that combine Kafka events with webhooks and other signals  

 - or full-fledged event-driven workflows.  

All while seamlessly moving those apps between servers and serverless without code changes.  

  On top of Kafka, Restate is our tool for building those resilient applications, from workflows and to async tasks, to event-driven applications or even microservices. While Restate is an event-driven system (sharing some similarities with a Kafka/Flink/KStreams crossover), it has a very simple programming model: Durable Execution (for RPC and event-handlers), Durable Promises (for reliable signaling), and Virtual Objects. Those three constructs are the resilient encapsulations of the core ingredients of every distributed application: compute, communication, and state.  

  Besides practical examples, this talk will also take a conceptual look at similarities, synergies, and differences between Restate and Kafka.  

 - How do event- and stream processing differ from Restate's programming model and are yet related and complementary?  

 - How do both systems internally organize and handle event logs and durability and how does that lead to complementary sweet spots?  

  The audience will gain an understanding of what in particular makes the above mentioned patterns and use cases hard nuts to crack, and how those requirements are tackled with a system built for transactional event-driven applications, where complex control flow, complicated state semantics, and many interacting systems/services are common challenges.