Tapping into the Stream: Streamlining Real-time Data Acquisition and Analysis

Real-time information about customer and systems behavior offers a wealth of opportunities for modern companies, from enabling real-time, business-aware monitoring and reacting to the creation of novel customer experiences. Despite the consensus around this, tapping into real-time data sources is challenging for various reasons, e.g., different systems can use different technologies, different event backbones and heterogenous data formats. All these hinder the efficient sharing of real-time information which, in turn, reduces the ability to harness it.

In this talk, we’ll explore FanDuel's journey in addressing this problem both from the technological and the human factor. First, you’ll hear about the multi-year strategy that aimed at standardizing event-driven communication across the company, followed by the architecture of the platform that enabled it. We will delve into the architectural decisions made and why a centralized streaming platform was adopted, its key components and their technology. Finally, we will discuss ways to improve developer experience by means of software frameworks and abstractions built on top of these components.

In the second part of the talk, we will focus on the actions taken to enable the smooth adoption of the strategy, combining business and engineering buy-in, and the benefits they brought. We conclude with its cultural implications and future plans.

The lessons learned presented in this talk are industry-agnostic and can be easily transferred to different contexts. You will gain a good understanding of important components and principles around designing successful streaming platforms and insights into supporting company-wide changes and their adoption.