Episode 112: Distributed Systems Research, The Philosophy of Computational Complexity, and Modern Streaming Database with Arjun Narayan

Datacast

English - April 07, 2023 05:00 - 1 hour - 100 MB - ★★★★★ - 4 ratings
Technology Business Careers research data engineering data science artificial intelligence machine learning statistics technology startup computer science venture capital Homepage Download Apple Podcasts Google Podcasts Overcast Castro Pocket Casts RSS feed

Previous Episode: Episode 111: Astrophysics, Visualization Recommendation, and Scalable Data Science with Doris Lee

Next Episode: Episode 113: Data Applications, Real-Time Analytics, and Cloud Product Management with Shruti Bhat

Show Notes(01:18) Arjun shared formative experiences of his upbringing - growing up in Bangalore, India; going to UWC Mahindra College for high school; and pursuing a liberal arts education in the US.(04:45) Arjun described his overall academic experience at Willams College - where he studied Computer Science and Economics and did a one-year stint at the Computer Lab at the University of Cambridge.(11:19) Arjun talked about his specialization within academic computer science: distributed systems.(14:17) Arjun unpacked the arc of his Ph.D. experience at the University of Pennsylvania, advised by Professor Andreas Haeberlen.(19:25) Arjun dissected the technical challenges and novelty of his Ph.D. dissertation on distributed systems that computed differentially private things.(23:20) Arjun shared his love for teaching which benefits his industry career.(25:55) Arjun walked through his decision to join Cockroach Labs as a software engineer.(32:25) Arjun unpacked the CockroachDB Performance Guide and a RocksDB deep-dive on the Cockroach Labs blog.(37:24) Arjun shared valuable lessons learned from his scaling journey with Cockroach.(41:36) Arjun mentioned how his writing practice benefited his day-to-day work designing database systems in a production setting (Check out his posts on database transaction isolation semantics and the history of log-structured merge trees).(45:46) Arjun unpacked his 2019 blog post titled "The Philosophy of Computational Complexity."(52:52) Arjun emphasized the importance of writing evergreen and authoritative long-form content that attracts a small amount of audience.(55:54) Arjun shared the story behind the founding of Materialize, which builds a SQL streaming database on top of Timely Dataflow and Differential Dataflow, two research projects created by his co-founder Frank McSherry.(01:00:04) Arjun unpacked the architecture design of Materialize at a high level.(01:04:36) Arjun explained a core capability of Materialize called Streaming SQL.(01:07:37) Arjun discussed successful tactics to raise the adoption and contribution to Materialize's open-source project.(01:11:23) Arjun walked through the major enterprise-grade features baked into Materialize Cloud.(01:15:54) Arjun dissected a blog post about Materialize’s unbundled cloud architecture detailing the shift from the Materialize single binary to Materialize Cloud.(01:21:13) Arjun envisioned how Materialize fits into the quickly evolving modern data stack.(01:25:07) Arjun shared valuable hiring lessons to attract the right people who are excited about Materialize's mission.(01:27:59) Arjun shared his brief take on building a high-performance company culture.(01:29:19) Arjun discussed the challenges for his team to find the early design partners.(01:31:17) Arjun walked through notable use cases of Materialize.(01:34:24) Arjun shared fundraising advice with founders who are seeking the right investors for their startups.(01:41:21) Arjun highlighted the similarities and differences between being a researcher and a founder.(01:42:46) Closing segment.Arjun's Contact InfoLinkedIn Twitter GitHub Google ScholarMaterialize's ResourcesWebsite | Twitter | LinkedIn | Slack Docs | GitHub Blog | Events | Guides CareersMentioned ContentResearch + ArticlesDistributed Differential Privacy and Applications (2015)Performance Report: Benchmarking CockroachDB's TPC-C Performance Why We Built CockroachDB on top of RocksDB (2019)A History of Transaction Histories (2018)A Brief History of Log Structured Merge Trees (2018)PeopleKyle Kingsbury Bob Muglia Frank McSherryBookZero To One (by Peter Thiel)Notes

My conversation with Arjun was recorded back in May 2022. Since then, a lot has happened. I recommend looking at the resources below:

About Materialize webpage (which shows the team building Materialize as well as the pedigree)Guide: What is a Streaming Database (which walks through why Materialize is important and different from a normal database)Case Study: Real-time Delivery Tracking UI in a Single Sprint at Onward Tech Demo: CI/CD Workflows for dbt+Materialize (March 2023)Announcing The Next Generation of Materialize (Oct 2022)About the show

Datacast features long-form, in-depth conversations with practitioners and researchers in the data community to walk through their professional journeys and unpack the lessons learned along the way. I invite guests coming from a wide range of career paths — from scientists and analysts to founders and investors — to analyze the case for using data in the real world and extract their mental models (“the WHY and the HOW”) behind their pursuits. Hopefully, these conversations can serve as valuable tools for early-stage data professionals as they navigate their own careers in the exciting data universe.

Datacast is produced and edited by James Le. For inquiries about sponsoring the podcast, email [email protected].

Subscribe by searching for Datacast wherever you get podcasts, or click one of the links below:

Listen on Spotify Listen on Apple Podcasts Listen on Google Podcasts

If you’re new, see the podcast homepage for the most recent episodes to listen to, or browse the full guest list.

About the show