Batch Is Just a Slow Stream: Designing Event-First Pipelines Without Going All-In on Real-Time

Most data platforms still think in batches. Daily jobs, hourly micro-batches, and carefully tuned schedules dominate, even in organizations already running Kafka. The result is familiar: complex backfills, fragile dependencies, hard-to-debug pipelines, and endless debates about whether “real-time” is worth the cost and complexity.

This talk argues that the real problem isn’t batch versus streaming, it’s the mindset behind them.

Batch and streaming are not fundamentally different architectures. They are the same model operating at different speeds. When teams design pipelines around intervals instead of events, they lock themselves into unnecessary complexity and make future change expensive. By contrast, designing workloads as streams first allows processing speed to become a configuration choice rather than an architectural constraint.

In this session, we explore how to transition batch workloads by shifting to stream-first, event-based thinking without committing to always-on, low-latency systems on day one. We’ll show how to model data as a sequence of events, reason about state and correctness over time, and decouple business logic from scheduling. From there, the same pipeline can safely run daily, hourly, or continuously, depending on cost, operational maturity, and business value.

We’ll also discuss when event-driven architecture naturally emerges as the simplest solution not because “real-time” is a goal, but because making change explicit removes the need for artificial intervals. Backfills become replays, late data becomes a first-class concern, and debugging shifts from job-centric to time-centric reasoning.

Attendees will leave with a practical mental model for evolving batch pipelines using Kafka, guidance on choosing processing speed deliberately, and a clear path toward event-based systems that scales with their organization not against it.