Real-time Platform Engineering: Scale Kafka Adoption Through a Kafka-as-APIs Program

Apache Kafka is an integral part of the data infrastructure of organizations aiming to build real-time applications, event-driven architectures, and microservices. However, the true potential of Kafka is only unlocked when Engineering teams across the organization are equipped with seamless access to building on top of Kafka data streams.

This will be the job of the modern Platform Engineering team. And the best way forward is a scalable “Kafka-as-APIs” program.

In this talk, we’ll run through best practices for building this program as a part of a larger Platform Engineering strategy, focusing on:

-The Role of APIs in Platform Engineering: We'll explore the fundamentals of API management and design, emphasizing how APIs act as the glue that connects disparate systems and services, including Kafka

-The Kafka Developer Experience: we’ll explore how to improve the Developer Experience for both internal and external Developers as Kafka consumers

-Strategies for API and Kafka Integration: Learn about best practices for designing and implementing APIs that interact seamlessly with Kafka, ensuring efficient data exchange and streamlining platform operations

-Self-service Kafka API products: how to leverage modern API Management to enable self-service access to Kafka via APIs

-Architectural Patterns and Use Cases: Discover how combining Kafka with APIs supports architectural patterns like event sourcing, real-time stream monetization, and more

-Challenges and Solutions: Addressing the common challenges faced when integrating Kafka via APIs, such as data consistency, security, and scalability, and providing practical solutions to overcome these hurdles

‍

This presentation is designed for technology professionals, including developers, architects, and business leaders, who seek to deepen their understanding of Kafka and APIs' synergistic role in modern platform engineering.