Uber Technologies
Uber is an American multinational transportation company that provides ride-hailing services, courier services, food delivery, and freight transport
.jpg)
At Uber, Data is a continuous, real-time flow of events, for insights and responsive applications
Uber is a two time Data Streaming Awards winner! See their previous winner case study from 2023.
Data Streaming Technology Used:
- Apache Kafka®
- Apache Flink®
- Apache Pinot®
What problem were they looking to solve with Data Streaming Technology?
Uber operates one of the world's largest data streaming platforms, built on Apache Kafka and Flink, which processes trillions of messages and dozens of petabytes of data every day. This platform is the central nervous system for the company, powering everything from the pub-sub message bus for rider and driver apps to real-time analytics, database change data capture, and data ingestion into its Apache Hadoop® data lake.
Operating at this immense scale presented a confluence of significant challenges. The core challenges included:
- Availability: As the platform underpins business-critical services, any loss of availability could directly result in revenue loss for Uber.
- Scalability: Uber's real-time data volume has been growing exponentially year over year. The platform must seamlessly handle this growth while maintaining strict SLAs for data freshness and end-to-end latency.
- Efficiency: As a low-margin business, maintaining low costs for data processing and high operational efficiency is paramount and drives key architectural decisions.
- Complexity: The platform must reliably serve a vast and growing number of use cases across a heterogeneous, multi-cloud environment
The ultimate challenge was to engineer a unified and standardized platform capable of satisfying these diverse and demanding requirements at scale
How did they solve the problem?
To address its monumental challenges, Uber engineered a multi-faceted data streaming platform with innovations spanning infrastructure, data quality, and user-facing analytics.
Core Infrastructure:
- Maximum availability and cost efficiency: Uber implemented an All-active, multi-region, cluster deployment to ensure business continuity during regional outages. To further enhance stability, they developed and contributed.
- Kafka Canary Isolation (KIP-1095), a technique for safely testing new software on a small portion of live traffic.
- Tiered Storage for Kafka (KIP-405), which drastically reduces storage costs by moving older data to cheaper object stores.
- Advanced Messaging: The uForwarder Consumer Proxy to overcome the inherent limitations of Kafka for asynchronous queuing (like head-of-line blocking), Uber developed and open-sourced the Consumer Proxy - uForwarder. This service acts as an intelligent intermediary, fetching messages from Kafka and pushing them to consumer services via gRPC. This decouples consumption from processing, enabling true parallel message handling within a single partition, out-of-order commits, and a robust Dead Letter Queue (DLQ) mechanism to prevent ""poison pill"" messages from stalling the system.
- Analytics and Ingestion at Scale: Uber built a unified real-time analytics platform on Apache Flink and Apache Pinot. Flink jobs process and transform raw Kafka streams, and the results are immediately ingested into Pinot, a real-time OLAP datastore that powers hundreds of sub-second latency dashboards for partners and internal teams. As part of its ""IngestionNext"" initiative, Uber standardized on Flink for ingestion due to its superior performance and higher throughput, ensuring near real-time data freshness across the company.
What was the positive outcome?
The platform successfully operates at a massive scale, processing trillions of messages and dozens of petabytes of data daily, demonstrating a complete solution to the core scalability challenge. It empowers a vast array of use cases, from the real-time event data from rider and driver apps to sophisticated streaming analytics and the ingestion of all company data into its data lake. The innovations in infrastructure and tooling have resulted in a platform that is simultaneously highly available, massively scalable, and cost-efficient, meeting the foundational business goals.