How to write your own partition assignor in Kafka’s KIP-848 Era

Apache Kafka 4.0 makes the new consumer group rebalance protocol (KIP‑848) generally available, shifting assignment computation from clients to the broker-side coordinator and eliminating the classic leader-driven, stop-the-world rebalances. This change simplifies consumers and improves stability and time-to-recover during membership or metadata churn, but it also removes support for client-side partition assignors on standard consumers in favor of pluggable server-side assignors. For teams that previously relied on custom assignment strategies, the good news is that you can still customize behavior by implementing a broker-side assignor via the new server assignor SPI and enabling it based on group-level and client configurations.

This talk demystifies how assignment works in the KIP‑848 era and provides a practical, end-to-end guide to writing, testing, and safely rolling out a custom broker-side assignor. We’ll cover the coordinator’s target-assignment model and incremental reconciliation, as well as the constraints and contracts your assignor must respect (stickiness, determinism, payload limits). We’ll share a working example: real code and configs to write and use our very own custom assignor.

We’ll also share migration tips for teams moving off classic client-side strategies, how to observe and debug assignments with the new metrics, and guardrails for compatibility and rollback. You’ll leave with a clear checklist and reference implementation path to bring your own assignor to production without sacrificing the resilience and operability benefits of KIP‑848.

Who is it for

Platform and application engineers running Kafka at scale

Developers who need custom partition strategies

Engineers upgrading client applications to Kafka 4.x, migrating from from classic client-side assignors to broker-side assignors under KIP‑848