Apache Pinot and Real-Time Analytics with Neha Pawar

Open||Source||Data

English - May 25, 2022 14:45 - 40 minutes - 37.5 MB - ★★★★★ - 16 ratings
Technology Homepage Download Apple Podcasts Google Podcasts Overcast Castro Pocket Casts RSS feed

Previous Episode: Real-Time Data, Enabling Developers, and User Experience with DeVaris Brown

Next Episode: Data Observability with Barr Moses, Einat Orr, and Shinji Kim

This episode features an interview with Neha Pawar, a Founding Engineer at StarTree. StarTree is a software development company that focuses on democratizing data for all users by providing real-time, user-facing analytics.

Prior to her time at StarTree, Neha was a Senior Software Engineer on LinkedIn’s Data Analytics team where she spent five years working on Apache Pinot. Neha has provided countless contributions to Pinot over the years, focusing on real-time streaming integrations, ingestion, and storage.

In this episode, Sam sits down with Neha to discuss Apache Pinot’s impact on the data community and how LinkedIn popularized real-time analytics.

-------------------

"Many people do think that a batch is good enough, real-time infra is expensive anyway. And what difference is it going to make if the data shown in this application is a day ago or an hour ago, and it's not real-time to the nearest second? And while that is true, in some cases, but in many other cases, not having real-time data can be super expensive and can affect the business badly and also make them irrelevant. You need the real-time data and then you also need to be able to analyze that data at the speed of your thought. For example, if you are having fraudulent activity somewhere, you can't wait for, ‘Hey, my model is going to learn about this.’ And then the next time, be able to tell me that that was a fraudulent activity. You need to be able to analyze all that data right now. So, it's not just a nice-to-have, it's a must-have.” – Neha Pawar

-------------------

Episode Timestamps:

(01:58): What open source data means to Neha

(06:04): Neha’s learnings from the LinkedIn Data Analytics Team

(07:07): What peaked Neha’s interest in real-time data analytics

(08:30): Neha’s first experiences working on Apache Pinot

(11:40): How the work of real-time data spread from LinkedIn to other companies

(17:30): How the Apache community has grown

(24:04): Neha’s focus at StarTree

(30:41): Neha’s motivation for tiered storage at StarTree

(37:07): Neha’s advice for open source data folks

-------------------