The Data Stack Show - The PRQL: Open Source and the Evolution of Data Systems with Andrew Lamb of InfluxData

Episode Date: April 22, 2024

The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

Welcome to the Data Stack Show prequel. This is a short bonus episode where we preview the upcoming show. You'll get to meet our guest and hear about the topics we're going to cover. If they're interesting to you, you can catch the full-length show when it drops on Wednesday. Welcome back to the Data Stack Show. We're here with Andrew Lamb, who's a staff engineer at Influx. And Andrew, Costas and I have a million questions for you that we're going to have to squeeze into an hour. You worked on the guts of some pretty amazing database systems. But before

we dig in, just give us a brief background on yourself. Hello. Yes. Thank you. I'm Andrew. Obviously, I've worked on lots of low-level database systems. I started my career at Oracle for a while. Then I worked on an embedded compiler for a while in a startup. And then I spent six years at a company called Vertica, which built one of the first sort of a phase of big distributed, shared nothing, massively parallel databases. I then worked in some various machine learning capacity startups, which was fun, but not really related to data stuff so much. And then for the last year, I've been working with Paul Dixon and the co-founder and CTO of Influx Data

on the new storage engine for Influx DB 3.0. So I've been down working on building a new sort of analytic engine for focused on time series. And you've also been working a lot with data fusion right the open source projects and they're like a lot of things we can chat about as eric mentioned but something that i'm really interested to hear from like your experience and perspective of andrew because you've been in this space for a very long time is that it feels like we are almost like at a inflation point when it comes like to data systems

and how they are built something that's probably happening for a long time but data systems are complex systems and very important systems so there's always like let's say risk is not exactly like something that people want to take when it comes to their data, right? So like the evolution usually is like a little bit slower compared to other systems. But it seems like we are reaching that point right now where the way that we build the systems is going to like radically change. So I'd love to hear from you how we go to this point, what are, let's say, the milestones that are leading to that? And what the future is going to be looking like based on what you've seen and what you're seeing out there? So that's the part

that I'm really interested to hear. What about you? What are some things that you would like to talk about? Yeah, I would love to talk about that. And I'd love to talk about sort of the role open source software plays in that evolution and how we got there. That was something that Influx, I think, always as a company has understood and really valued is open source and how that's both evolving and then also how you leverage open source to build the next generation products. I think it'll fit beautifully. And I think we can illustrate the story of sort of what we're doing with Influx DD 3.0 as part of that longer term trend. And I think there's lots of interesting things to talk about there. Sounds great. What do you think, Eric?

Let's go and do it. I'm ready. I'm ready. We need to hit the ground running here because we have so much to cover. Yeah, let's do it. All right. That's a wrap for the prequel. The full length episode will drop Wednesday morning.

Subscribe now so you don't miss it.