The Data Stack Show - The PRQL: Making the Data Stack Serverless in the Cloud with Mike Driscoll of Rill Data

Episode Date: March 11, 2024

The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

Hi, Data Stack Show listeners. I'm Pete Soderling, and I'd like to personally invite you to Data Council Austin this March 26 to 28, where I'll play host to hundreds of attendees, 100 plus top speakers, and dozens of hot startups on the cutting edge of data science, engineering, and AI. If you're sick and tired of salesy data conferences like I was, you'll understand exactly why I started Data Council and how it's become known for being the best vendor-neutral, no BS, technical data conference around. The community that attends Data Council are some of the smartest founders, data engineers, and scientists, CTOs, heads of data, lead engineers, investors, and community organizers. We're all working together to build the future of data and

AI. And as a listener to the Data Stack Show, you can join us at the event at a special price. Get 20% discount off tickets by using promo code DATASTACK20. That's DATASTACK20. But don't just take my word that it's the best data event out there. Our attendees refer to Data Council as Spring Break for Data Geeks. So come on down to Austin and join us for an amazing time with the data community. I can't wait to see you there. Welcome to the Data Stack Show prequel. This is a short bonus episode where we preview the upcoming show. You'll get to meet our guest and hear about the topics we're going to cover. If they're interesting to you, you can catch the full-length show when it drops on Wednesday. We are here with Michael Driscoll from Real Data.

Michael, thank you so much for joining us on the show today. Great to be here, Eric. All right, well, give us your brief background. How did you originally get into data and what are you doing at Real today? Yeah, thanks. My background is actually probably not that dissimilar from a few of your guests you've had over the years. I actually started my career as a software developer working for the Human Genome Project a couple decades back. And naturally, there's a lot of data in the human genome project. And that was really the beginning of a multi-decade love affair, working with data at scale, heterogeneous data.

And since then, I've started a few companies. My first startup was an e-tailer called customink.com. We sell t-shirts on the internet. I later started a consultancy called Dataspora. We did a lot of consultant work for banks and folks in the big data era. I then went on to start a company called Metamarkets, which was acquired by Snapchat or Snap, the makers of Snapchat that did analytics for advertising.

And now I've got Rill data. We're a few years into that journey and focused on an operational business intelligence product with Rill. All right, that's quite a journey, Michael. And I know that part of this journey also includes like some very interesting technologies like Druid and from the conversation we had earlier like I've learned a few things that I wasn't aware about Druid and the relationship it had like with BI and what were like the initial ideas behind it and I'm super excited excited to get into that and learn more

about how you started building Druid while you did that and how you ended up today, actually, with real data that has Druids on the backend, but it's more than a query engine, right?

So I'm super excited to get into the details. What about you? What are you excited to talk about today? Yeah, well, I think there's a few big macro trends that we're seeing in the data world today. I would say I would be delighted to talk about some of the emerging data engines that are out there for powering fast analytics at scale, really at any scale.

So Druid and ClickHouse, also DuckDB, we for me is particularly exciting is the trend towards serverless frameworks. there's a lot of new frameworks out there for really taking not just, you know, data technologies to the cloud, but making them serverless in the cloud. And so I look at, yeah, you know, almost any area of the data stack I think is being remade to be truly

serverless at scale in the cloud. And that's a pretty exciting area that's going to take several years to play out. Yeah, 100%. We'll have a lot to talk about that. So, Eric, what do you think? Should we dive in? Let's do it. All right, that's a wrap for the prequel.

The full-length episode will drop Wednesday morning. Subscribe now so you don't miss it.