Kroxylicious: Taking a bite out of the Kafka Protocol

As Apache Kafka usage continues to grow, it gets deployed in increasingly sensitive and regulated environments. At the same time, data engineering teams have more and more requirements to satisfy the needs of businesses to support AIs and provide real time business intelligence. Unfortunately, for historical or design reasons, Apache Kafka is not able to provide all the features everybody needs.

One solution gaining traction over the last couple of years is to proxy Apache Kafka. This session introduces Kroxylicious, an open source Kafka protocol aware transparent proxy (part of the Commonhaus foundation). Kroxylicious offers developers a standardised Filter API to allow them to customize the messages passing through the proxy as well as a plug-in based extension mechanism to allow them to interact with remote resources, such as Key Management Systems or Schema registries. All this is completely invisible to clients and clusters and does not require updating them.

Out of the box Kroxylicious provides a customizable record encryption to ensure data at rest is safe even if you use a cloud provider. It also integrates with a schema registry so you can ensure that records sent to specific topics match the configured schemas. As it fully understands the Kafka protocol, it opens the possibility for building a wide range of features such as automatic cluster failover, offloading authentication, multitenancy, etc.

At the end of this talk, attendees will understand the core principle and out-of-the-box functionalities of Kroxylicious. They will also know how to run and operate it, as well as know how to incorporate custom business logic.