Pinterest’s Journey to an Automated, Efficient, and Low-Maintenance PubSub Platform

At Pinterest, Apache Kafka® runs on ~2000 brokers across 50+ clusters, moving data at petabyte scale every day. Through investing in automation and focusing on cost efficiency, we have seen positive results in dev velocity, scalability, stability, and cost footprint, enabling everyone to quickly and easily launch data streaming use cases that power business growth, while keeping our platform low-maintenance.  

  We believe that sharing the innovations along our journey to a highly-automated, efficient, and low-maintenance PubSub platform can provide immense value for industry partners. This talk will cover the following topics in-depth, bringing inspiration and actionable takeaways for each attendee who might face the same problems operating PubSub platforms at scale:

1. Automation: >80% reduction in overall onboarding time for PubSub applications

• An Automated endpoint discovery with PubSub Client (PSC) via 100% completion of Java migrations to PSC, available standalone and as Flink connector (open-sourced)
• Automated resource provisioning for PubSub-based applications via drag-and-drop UI, reducing setup time from 30 mins to 5 mins
• Automated Kafka broker & cluster management and recovery via Orion (open-sourced)
• Automated client-side error handling via PSC (open-sourced)

2. Efficiency: >25% reduction in platform cost

• Reducing PubSub infrastructure cost via migrating large pipelines from Kafka to MemQ (open-sourced)
• Reducing Kafka cluster sizes and optimizing efficiency via using heterogeneous hardware
• Reducing Kafka broker footprint via storage tiering
• Reducing cross-region KafkaMirror resource utilization via Shallow Mirror (KIP-712)

3. Low-Maintenance: simplifying business processes

• Generic CDC framework via Debezium / Kafka Connect
• Improving cost visibility and attribution for multi-tenant environment, enabling cost-driven decision making
• Moving towards service tiering to further simplify customer onboarding and platform operation