From Data Pipelines to Context Streams: Building Infrastructure for the Agent Era

For decades, data engineers have built infrastructure optimized for human consumers—analysts running queries, scientists training models, executives viewing dashboards. Batch processing, overnight refreshes, and query-friendly schemas made perfect sense. But in 2026, a new primary consumer is emerging: AI agents. And they have radically different requirements.

Agents don't wait for nightly batch jobs. They need fresh context delivered in milliseconds. They don't browse dashboards—they consume structured context windows that must be assembled on-demand from multiple streaming sources. They don't tolerate stale data gracefully; outdated context leads to hallucinations, incorrect actions, and compounding failures across multi-agent workflows.

This talk introduces "context engineering" as the discipline of building data infrastructure for agent-facing applications. We'll explore how streaming platforms like Apache Kafka become the foundational layer for real-time context delivery, and why the patterns that served human analytics fall apart when agents are your consumers.

We'll cover three core challenges through production examples: First, context assembly—how to join, filter, and enrich multiple event streams into coherent context windows with sub-100ms latency using Kafka Streams and Flink. Second, state management for agents—leveraging event sourcing patterns so agents can access not just current state but temporal context ("what did this customer do in the last hour?"). Third, observability for agent-consumed data—why traditional data quality metrics miss the failure modes that matter for agents, and how to build context delivery SLOs.

Throughout, we'll examine real architecture decisions: when to push context to agents versus let them pull, how to handle context window limits as a backpressure signal, and patterns for graceful degradation when upstream data sources lag.

The underlying principles of good data engineering—reliability, freshness, correctness—remain constant. But the application layer is transforming. Data teams that recognize agents as first-class consumers, not afterthoughts, will build the infrastructure that powers the next generation of AI applications.

Attendees will leave with concrete architectural patterns for agent-facing data infrastructure, an understanding of how streaming primitives map to agent context requirements, and a framework for evaluating whether their current data platforms are ready for AI-native workloads.

By attending this session, your contact information may be shared with the sponsor for relevant follow up for this event only.