Data State to Business Events: Application of Set Theory to Event Streaming

At ACERTUS, we are using Apache Kafka to deliver real business value to our customers by automating order lifecycle and providing near-real-time business events. One of the tools we created is an elegant architecture for collaboration between business operations and software developers to efficiently transform data events into business events.

  One implementation of this architecture uses a “source-of-truth” Postgres database, Debezium CDC, Apache Kafka, and Kafka Streams. We apply logical predicates of set membership to derive events from relational database state and suppress “duplicate” events. Entry into, departure from, or an update within a set implies event occurrence

  We'll discuss two examples of this architecture applied to a legacy software product in two critical ways:  - Replace an error-prone long-polling ETL process, reducing event delays from 30 minutes to near-real-time.  - Extend the product capability to produce near-real-time updates to customers on their orders.