Event Driven Views With Ingest-Materialize-Index Stream Topology

At Indeed, we help people get jobs.

Our team supports this mission by ingesting data about employers’ hiring needs, enabling them to manage this data, and transforming it into searchable job advertisements—commonly known as job posts. Creating job posts is a complex and I/O-bound process, requiring enrichment from multiple bounded contexts. Adding to this complexity, many downstream systems must be notified in real-time when any part of a job post’s data changes.

To meet these demands, we implemented a system that produces and maintains an Event Driven View (EDV)—a materialized, denormalized representation of job posts that stays up-to-date as changes occur across the business. This view is powered by a novel Ingest–Materialize–Index (IMI) stream topology, which enables us to scale processing while preserving strong observability and reliability guarantees.

This talk will explore how we build and operate EDVs using the IMI architecture:

- We’ll break down each IMI stage and show how clean separation of concerns enables performance and observability.

- We’ll show how we use micro-batching, structured concurrency, and I/O pipelining to manage I/O-bound enrichment at scale.

- We’ll share strategies for recycling failed view materializations through rate-limited retry streams.

- We’ll cover how we incorporate Change Data Capture (CDC) to reliably notify our downstream clients about changes in our persistent EDVs.