Tracking Locomotive to Train Associations in Kafka

A deep dive into the iterative design process of creating a highly resilient enterprise service for tracking locomotive associations for over a thousand trains all at once.  

  By highlighting some of the less efficient and outright erroneous decisions made throughout the process of building the complex event processor, we hope to provide a thorough understanding of the motivations and philosophies behind the solution. A problem composed of classic Kafka conundrums ranging from copartitioned state-stores with 2 different keys requiring repartitioning between processes, to state aggregations and retry mechanisms to ensure data quality during periods of high consumer group lag. Our "lta-cep" is a (very) stateful combination of the Kafka Streams DSL and the processor API ingesting nearly 10 different protobuffed input topics each with unique business logic for state manipulations.  

  What more is there to say? If you're interested in a deep-dive into a modern implementation of one of the railroads most challenging problems you know where to be.