The Data Stack Show - The PRQL: Be Careful, Young Padawan, When Comparing Software Observability and Data Observability

Episode Date: May 2, 2022

Eric and Kostas preview their upcoming conversation with Barr Moses of Monte Carlo.

Welcome to the Data Stack Show prequel, where we talk about the show that we just recorded. Costas, what a treat. We had a bar from Monte Carlo, one of the co-founders and CEO of the company on the show, Data Observability Company. And one of the things, this is maybe giving away a little too much, but hopefully it's enticing for the audience. But we talked a little bit about standardization, actually a lot about standardization. And it was amazing to me that they invested so much time and resources into actually going to companies who are trying to solve these problems and who had built bespoke solutions at scale. We've talked about this a lot on the show, right? Netflix and Facebook and LinkedIn

and Uber solving problems that no companies faced before because they haven't hit that level of scale with that business model. And they sort of did a massive survey of all those companies who built solutions and sort of codified them, which was absolutely fascinating. So definitely check out the episode if that's at all interesting. But one question I have for you, because we talked about this a lot on the show. So you come from a software engineering background. And we talked a lot about how the principles around observability, you know, are sort of metaphors borrowed from software. Does it feel like a natural shift for you for sort of the observability side of things to come into the data space?

Is that a natural thing for you? Or is that, you know, sort of forcing principles on data that don't necessarily apply in the same way? Yeah, that's an excellent question. I think that applying or like using the language that we use as part of like observability in software engineering is something like very useful. Like we have to do that. Like we need to use like metaphors to create new products and new ways of solving problems, right? Like otherwise it won't work, right? But I think we are also at the stage right now where we should also, let's say, stop a little bit and start reflecting on how we use these terms and make sure that we refine these terms to make them reflect, let's say, the context and the reality of what it means to work with data instead of servers, right? Like it's one thing to talk about the availability of a service or the availability of a server.

And it's another thing to talk about what it means to trust data, right? Or what it means to have like data that are available. And the data set might be there, but might not be fresh enough, for example. So there are different concepts that need to be introduced and make sure that we refine these ideas. And we don't just copy them directly and use them in the same way that they are used in a different domain. Because this is like a disaster, right? And that's, I think, a very interesting part of the conversation that we had with with Barr. We talked about SLAs and what it means to have an SLA around data, compared to an SLA for the availability of a service or a server.

Right. So yeah, I think we are still in the process of like, defining these terms. And people like Barr, I think they're doing like an amazing job on spreading, like starting actually the right type of conversations that will lead us to come up with the right definitions at the end. I agree. And here is a teaser that will entice you. We also got the background on where the name Monte Carlo came from. And it is related to college-level, university-level mathematics. So if you want the backstory on the company name, definitely join us for the next show, and we will catch you on the next one.