Header-aware state stores for Kafka Streams

Kafka record headers are increasingly used to carry critical metadata such as schema identifiers, correlation IDs, tracing information, and feature flags. In modern event-driven architectures, these headers are also the primary vehicle for propagating distributed tracing context (trace IDs, span IDs, and causality metadata) across asynchronous boundaries. Today, Kafka Streams state stores ignore this metadata and only persist key and value bytes, which means any header-based semantics and tracing context are lost as soon as a record passes through a stateful operator and gets materialized. This makes it hard to use header-aware serdes consistently, breaks end-to-end traces at stateful boundaries, limits downstream processors that rely on headers to drive behavior, and prevents Interactive Queries from exposing header-level observability when headers are part of an application’s protocol.

This talk introduces KIP‑1271, which proposes header-aware state stores for Kafka Streams as a building block for robust end-to-end tracing in Streams applications. We’ll look at the new header-preserving store types and how they embed serialized headers alongside values while keeping the existing key/value abstraction intact. You’ll see how applications can opt into these stores via the new *WithHeaders factories, how header-aware serdes plug in so that tracing and other metadata survive stateful operations and replays, and how upgrade is handled through a single rolling bounce with lazy migration of existing state. Attendees will leave with a clear understanding of when and how to adopt header-aware state stores to keep tracing context and other critical metadata intact end to end.