The Data Stack Show - The PRQL: Does Putting All Your Data in One Place Create More Problems Than it Solves?

Episode Date: April 15, 2022

Eric and Kostas preview their upcoming conversation with Kaycee Lai of Promethium.

Welcome to the Data Stack Show prequel. We just talked with Casey from Prometheum, and it was a really interesting conversation. Costas, I loved the counterintuitive advice that he gave around the data stack and especially how you build an analytics practice. So Casey said, the advice that you, sort of the advice that you get, you know, in the industry today is, you know, get a data warehouse, get a data lake, like

collect all your data in one place. And he said, don't, that it actually creates more problems. So do you agree or disagree with that? I, I'm sure that I agree. I find his approach, first of all, it's an indication that Casey has a very tangible experience working with data and doing all the engineering. I think one of the lessons learned in engineering all these years and moving, let's say, from the waterfall model towards like the more original model to Jackie Lannert's. Because these problems and these projects that we are trying to start have so many unknowns, making all these assumptions and first building everything and then testing the assumptions at the end is adding more and more, let's say, risk and friction. And I think especially with data, because we start, let's say, from, I don't know,

like knowing pretty much nothing, right? Like we just have an idea, like we only have assumptions. I think at the end, like what he says is that test your assumptions, turn them into actions, and then stop building your foundations, like to, to serve all the systems that you need, which I think makes a lot of sense. By the way, just to, to, to make myself, myself clear, there are definitely engineering problems out there where you need to follow a waterfall approach. A hundred percent. Right? It's not like Agile is like the solution to everything.

And there are like projects where you need to be, let's say, very, very thoughtful and very methodical and beforehand. An example, I don't think that like we could be really Agile and send like the Webb telescope in the sky, right? Like, because yeah, like when we will figure out that we made the mistake, it's going to be too late. But when it comes to data, I think that it's like, there's a lot of value in assuming

a more of like agile approach and first learn and then implement and not do the other way around. Yeah, I agree. Fascinating conversation. We also talked about a self-optimizing data governance system based on user feedback, which was super interesting, and tons of other fun stuff. So definitely check out this episode with Casey from Prometheum, and we'll catch you on that show.