The Data Stack Show - 95: How the Metrics Layer Bridges the Gap Between Data & Business with Nick Handel of Transform

Episode Date: July 13, 2022

Highlights from this week’s conversation include:Nick’s background and career journey (2:40)What Transform does (5:53)Metrics layer vs. metrics store (8:04)Signals vs. metrics (13:24)The user of a metric layer (14:34)Using Transform within an organization (17:05)How to fuse two sources into a metric (23:54)Currently supported databases (28:46)Community engagement (31:33)Optimizing for queries, metrics, and use cases (35:33)Technology and the human factor (40:49)Managing metrics amidst fast-paced change (46:53)Out-of-the-box metrics store (49:26)The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

Welcome to the Data Stack Show. Each week we explore the world of data by talking to the people shaping its future. You'll learn about new data technology and trends and how data teams and processes are run at top companies. The Data Stack Show is brought to you by Rudderstack, the CDP for developers. You can learn more at rudderstack.com. Welcome to the Data Stack Show. Today we are going to talk with Nick from Transform Costas. You and I have actually chatted offline a whole lot about this company and the concept in general. They build a tool that does metric store. Metrics layer is also a really common term for what they do. DBT kind of has used that terminology as well.

And so my burning question for Nick is actually just defining the metrics layer. It's a concept that sounds great, but it's one of those terms that we run across all the time in the show where if I asked you to define that, it's probably kind of hard just because the term is a little bit ambiguous in terms of what does that mean for me day-to-day as someone who works with data. What are your thoughts? 100%. Yeah, I totally agree with you. And we definitely have to spend some time on that. I really want to go through like also a bit of like the history behind this technology, because, okay, we start hearing about all these variations of the semantic layer, the metric layer, the metric store, the headless BI, all these different names of like

pretty much the same thing, but obviously like people were struggling with the same problems like in the past, right? And that's also like, for example, how Looker became successful because they provided like a solution to that. So I'd love to hear like what we, from Nick, like what we did in the past, what has changed today and why we need to keep innovating on that. Hopefully like this together with like what you have in your mind will help like our listeners understand the value and how these technologies are used today.

All right. Well, let's go figure out what the metrics layer is. Let's do it. Nick, welcome to the Data Stack Show. We are so excited to chat with you. Great to be here. Thanks for having me. Of course. Okay, let's start where we always do. Give us your background and what led you to starting Transform. Yeah, definitely. So started off studying math. I was always really into math, did math competitions as a kid, kind of was a nerd and was proud of that.

And yeah, so I studied math in college, moved over to kind of applied math. So first to sciences, and then I started working on economics and kind of got more interested in the behavioral pieces. Out of school, so it was 2012. We were kind of coming out of this double dip recession, which I thought was really interesting in school and studying economics and math. I was like, just wanted to pay attention to that. And so I ended up going to BlackRock as a research analyst. I was working on a team that basically did macroeconomic research and produce signals for various investment funds really enjoyed that.

But I also found that the feedback loops were quite long. Macroeconomics kind of moved slow and post all the kind of global meltdown, just found that things were kind of calm in 2013, 2014. And so I wanted something that would give me faster feedback loops. So I started kind of looking around, didn't know exactly what I wanted to do, but figured that it would probably be somewhere in tech. And luckily, I, in 2014, landed on the data science team at Airbnb. I was on the growth team, worked a ton on kind of product experimentation, lots of different analysis, machine learning, pretty broad at the time that the kind of data science team wasn't super specialized. And yeah, eventually I had been kind of working closely with the data tools, data infra team, and eventually decided to move over to the product team in 2017.

Did about a year as a product manager building data tools for people like me and then realized I kind of wanted to go back to data, which is kind of a trend in my career, just kind of veer back and forth between product work and data work. And so went and joined a startup called Branch, which is a micro lender as their head of data. And micro lender, tons of machine learning applications for loans and fraud and all kinds of different things. But that ended up being really fun. Lots of interesting kind of NLP applications. And on the side, on the side, like head of data was also responsible for a lot of the reporting and kind of figuring out how's the loan book doing.

And so ended up going back to a lot of the analytical work that I was doing earlier in my career and just really enjoyed that. But realized that, you know, some of the stuff I was working on at Airbnb was one really valuable. I think we were, you know, a bit ahead of the curve just because we had, well, one, you know, data scale problems. And two, just a bunch of resources to go and try and build various tools. And yeah, realized that this metric space was really interesting and decided to start working on Transform kind of at the end of 2019 and just have been doing that since. Very cool.

And tell us at a high level, I mean, we have so much to dig into, but tell us at a high level, what does Transform do? Yeah. So Transform is working on a piece of technology called a metric store. And that is a very, I think, loaded term. I don't think people really know what that means. So hopefully we'll hash that out. We definitely will. At a very, very high level, there are kind of two really big problems with how companies and how people, how data analysts and business users interact with metrics today. One is that, you know, especially on larger teams, there tends to be this kind of challenge of defining metrics consistently. You know, that means, like, is my SQL the same as yours? That means, you know, I tried to do this analysis, and it's slightly different than the way that you

talk about, you know, this like revenue thing or new user signups or whatever it is. And so there's this, basically this trust problem. The second kind of, you know, big piece of what the metric store is aimed at solving is around productivity. So it should be very, very easy to go and ask a question like, you know, what's the search to check out conversion rate? And I now I want to know search to, you know, check out conversion rate by country now I want to know search to, you know, check out conversion rate by country. Like those are questions that should not require knowing how to join, you know, three different tables

and knowing how, you know, to filter various things or aggregate various things. Those definitions should be baked and people should be able to kind of reliably consume them around an organization. Totally. Okay. So I want to dig a little bit deeper. So there are sort of two terms. So you said metric store, and then also this paradigm is described as a metrics layer as well, right? And so, and what's interesting about the terminology is that you think of a layer as sort of spanning

a stack or sort of spanning a lot of different things and a store as sort of a distinct entity. Can you help, you know, sort of parse that out for us? Like, what's the difference? Are they different? Are they the same thing? Okay. So I'm going to start with semantic layers and I'm going to move my way up to a metric store. Love it.

I think that there is in this like inner, inner bubble, let's imagine kind of like three circles that are encompassed with each other. You know, I think of like the, those three circles that show like machine learning AI, I'm forgetting what the the NLP or something is in the middle. So in the middle, the most narrow definition in this space is a semantic layer. And so what is a semantic layer? It's basically a business representation of some kind of company's data. And another word could be an ontology. I think that when people talk about semantic layers, you know, they think about most recently, they probably think about look ML what's within looker. I think, you know, there are other tools in this space. There's at scale is, you know, talks about themselves as a semantic layer historically, you know, business objects was probably kind of the the biggest i mean tons of people still use it

yeah sorry historically maybe that's a yeah i shouldn't say yeah i mean no but it does feel that way but it is amazing how many people still use it yeah and so like what does this thing do basically you know if you think about a entity relationship diagram a bunch of different data sets a bunch of different objects they, a bunch of different objects. They, you know, within them, they have measures and dimensions, things that are aggregatable, and then things that are, you know, the various kind of like group bys and filter bys, the dimensions, and they have relationships between them. And so, you know, that right there is basically the kind of semantics of what exists within a data warehouse. And it's kind of the building block that we do a lot of our analysis on top of.

Whatever, you know, whatever kind of data modeling techniques you use, there's still some kind of expression of semantics there. Sure. So my definition of a metrics layer is, I think that they can come with kind of very weak concepts of semantics all the kind of very weak concepts of semantics all the way to very strong concepts of semantics. So a very weak concept of semantics in a metrics layer would be to just define some SQL, right? It's not very structured.

You're not really kind of creating these like reusable abstractions. You're just writing a SQL query. On the other end of it, you have something where you are providing these measures and dimensions and relations in a very structured format, and then expecting the semantic layer to do a lot of the work for you to define the SQL and build it up. And the way that I think about the metrics layer is basically just taking whatever that semantic construct is and extending it to have a concept of a metric. So an object that is a metric.

And I think that there are some interesting conversations around what makes a metric different from a measure. So the way that I think about that is measures are basically aggregatable expressions. And a metric could be as simple as an aggregatable expression. So like a count star from the transactions table is transactions. And that could be a metric. But then there are some metrics that are just a whole lot more complicated than that.

And they require like a conversion metric or something like that. It requires two events. It requires an understanding of the relationship between those two events, an understanding of the, like, difference between these metrics and measures. Metrics are basically a, you know, a more kind of complicated construct. So metrics layer basically is like, you know, the semantics of what's in the warehouse, some kind of concept of like, how do I take that semantics, turn it into some logic that can be executed to actually pull out a measurable object, a metric, and then, you know, some kind of querying interface. And I think if you look across the board, that's kind of what these various tools are. Semantics, some kind of performance, and then querying interfaces. MetricStore basically just takes that and extends it with organizational governance.

Beyond technical governance, how do you get a bunch of people to collaborate and agree on the definition of a metric? And then how do you basically involve the various people that need to be in those conversations and kind of manage that lifecycle of the metric? Okay. There are many interesting topics and questions that I want to ask and all that stuff. But before we go there, I want to ask you something that came up in my mind, like as you were making the introduction of yourself. So you mentioned like two important terms as part of your work experience, right? Like you mentioned metrics. That's something that, I mean, you build a business around that. And you also mentioned signals, right? Like you said that you made it like very clear that when you were like in BlackRock,

that your work was like to generate signals, not metrics. So what's the difference between a signal and metric? Yeah, so I kind of skipped over this part of my career. But so signals in my mind are features for machine learning applications. So that's what I was talking about. Yeah. And when I was a product manager at Airbnb, what I was working on was Airbnb's feature store, which is actually quite similar to a metric store, but also quite different in a few interesting ways. So there are just different requirements for these two tools that make them. I think, you know, there's, there's quite a long way between those two tools being the same. But yeah, I'm talking about kind of machine learning applications when I say signals. Okay. Okay.

Got it. That's, that's a very interesting distinction. And I, it's very interesting. Like that mentioned the feature stores there. Okay, cool. So let's go back to metrics. Can we start a little bit like from a product perspective, like who is today like the user of a metric layer?

Like who is the primary user who will gate it, transform as a platform? Yeah. So, I mean, this is, this I think is part of the interesting thing about what the metrics layer could become. There is this gap between the way that data people talk about data and the way that business people talk about data. When you get a data request as a data person, it's almost always in the format of, you know, I would like these various metrics. I would like them, you know, filtered by these various dimensions. I'd like to have these various dimensions

for me to slice and dice and kind of ask some questions. And then what you do with that is, you know, you go and you translate that request of, I would like these metrics at these granularities filtered in these ways to SQL on top of tables in the warehouse. that they can use for their analysis. And the data person is this separation between the tables and all of the kind of SQL logic that needs to be expressed to actually pull the data out and the way that people talk about data. And so I think that the point of the metrics layer is to try to bridge that gap

because it is a common language. It's kind of the way that we translate a business user's needs into something that can actually be expressed as SQL, as logic on top of a data warehouse. And so ultimately, I think that the end user of a metrics layer is the data analyst, like the analytics engineer, the data engineer. They need to be able to define the logic on top of these tables as these objects, these concepts that business people can understand and can consume in downstream applications. I think that the metric store extends that a little bit. And it actually adds this piece of organizational governance where there are business owners

to metrics and they should be involved in that process. And so in some ways, there is actually a second user of a metric store, which is the person who is responsible for a metric or partakes in the conversation of how it should be defined. Okay. So people using the Transfer Platform can collaborate, right? You have the analyst, the data engineer, and also like the business user there, and they engage in the process of defining a metric. Can you like help us, like take us through like this experience? Like how, how, how, how would you, how you have seen it like

working inside an organization? Yeah. So I think a good example is actually, you know, I think you had Paige from Netlify on the show a few months back. And I think that, you know, her explanation is probably better than my explanation because it's a lived experience from an actual user. But, you know, just to kind of reiterate it from our perspective, data analysts know the logic of how to express these things. Business users don't necessarily know the nuances of how to join various tables, how to filter in the correct ways, how to produce these metrics. And so

the data analyst has this problem of, it's a good problem to have, right? Business users are interested. They want to know the answers. And so they're basically going and asking various questions. And that is time consuming for the data analyst. It takes a lot to kind of resolve all those questions. I think that, you know, there's a good book from Tamash Tungus and Frank Pien from Looker that, you know, basically coins the word data breadlines. And like the gist of that is business users are kind of getting in line, waiting for their data. And so what the, you know, what the semantic layer, what this like metrics layer on top

of that enables is it enables the business user to have a metric concept that they can understand, that they can request, and allows them to interact with the tool in this kind of more self-serve way. And that really was kind of the special innovation of LookML and Looker in general, on top of historical tools that have done similar things in the data space. And so what that enables is basically just, I have questions about metrics. I would like to get a CSV. I'd like to get that data in Excel or in Google Sheets.

Or I just want to see a line chart of the metric. And I want to know, is this thing going up? Can I slice it in a few different ways and figure out why it's going up? That is, that's really valuable for the end business user. And it solves a problem for the data analyst of having to go and kind of rewrite the same SQL over and over again, you know, try and manage that SQL, try and hash out the definitions between, you know, SQL that wasn't written by them, but it's kind of presented to them as like the right number or something like that. Okay.

So if I understand correctly, like you have the, you have the business user, the business user comes, says like, okay, I want, let's say to measure MRR. Okay. Yep. And we start from there. And through like going back and forth between them and the analyst and maybe also like a data engineer, we ended up in some SQL code that can calculate the amount, right? And what happens after that?

Like what's... let's say we've done that. Okay. We have stored it like in the store, right? Like we have the definition there and we have an agreement between these three, too, because my next question is going to be what happens like when there are

multiple different versions of what MRR is. But what next? Like how this information is then like reused inside the organization and where it is consumed from there? Yeah, so I think that

there should be many options. And, you know, there's this word of headless BI, right? That's kind of a word that's thrown around. I think that people often conflate it with the metrics layer. I actually don't think that the metrics layer should be headless. I think that it should be able to show you the metrics. It should be able to show you the value. But I think that the interesting piece is that, and so in some ways you could say that the metrics layer, like, you know, if it's not headless BI, it's, it's kind of overlaps with BI and in some like, you know, small ways, but it's basically just the metric that you've defined presented on a graph.

It's not the kind of free form. Let me write some SQL and then produce a hundred different graph types kind of experience that people expect out of BI. But so that's the first thing. I think that this tool should be able to show metrics. And I think that it, you know, you should be able to kind of build a really, really great experience for the business user to come and ask a question. And for the, you know, the data analyst to know that if that question that gets asked gets copy and pasted as a PNG and thrown in a deck to the board, that it's going to be the right number and it's going to be well received and not going to lead to chaos down the road. But beyond that, I think that the difference between what we have with Looker today is that this should be open.

And I think Looker's kind of moved in this direction a little bit more recently with the promise of an integration with Tableau. I think that there are some integrations with Gsheets, etc. But I think that it shouldn't just be a limited set of integrations. It should be generic APIs that are open that anyone can consume these data sets from. And so I think that the kind of format that those APIs should expose is metrics by dimensions. That's typically the way that the end business user asks. And I'm not saying that the business user shouldn't have to interact with an API. But I think that these tools ultimately will be pressured to build better first class experiences with these centralized metric definitions. And so some tools actually already have done this.

I think Tablo has been very open to this. I think Mode has done a good job. You know, there's a transform JDBC that allows Mode to directly connect to transform and allows you to ask for, you know, revenue by country by day. And it, you know, sends that request to transform and transform compiles that into their, you know, the correct SQL query and then returns that result. So that's kind of the experience that I think that we're moving towards. Okay.

So let's get a little bit closer to the technical details. And we have, let's say, the semantics that they are brought by the business user, the expert, let's say, in this case. Yep. Then we have also a SQL query, which is created by the analyst. So how do we represent these two sources? How do we fuse these two sources of information into a metric? And I mean that on a technical level right like is this like set of metadata together with the sql query or how it looks like yep so this i mean this is my favorite part of

what i work on and this is the kind of piece of product that i you know refuse to stop thinking about and it'll probably true be true for a long time. So the really interesting thing is, how do you create these reusable abstractions, these kind of core components that allow you to define the logic for a dimension, for a relationship between two tables, for a measure, and be able to kind of use those as building blocks that enable you to define a metric in a dry way, so don't repeat yourself, in a non-duplicative way. And then once you have those nice, clean abstractions,

how do you then allow a user to express them? And so I'm going to talk about Metric Flow, which is Transform's open source metrics framework, because this is the piece. This is what Metric Flow is. And I would maybe just kind of preface that with it's a component within the broader Transform set of products. And so what the spec within metric flow looks like is, well, anyone can go look at it and we have some examples of defining your Salesforce data in it and whatnot, but it's a YAML file that basically has some

SQL expressed in it. So you can point it at a table. You basically say, you know, core.transactions is the name of the table. And I would like to define a measure that is, you know, the sum of one or like a count star, like a count of how many transactions have occurred. And then I would like to define some dimensions, which are, you know, maybe the payment type, the store that the transaction occurred in, et cetera. And so I've now kind of captured the semantics of one table. We then take various identifiers that exist in different tables, and that allows us to do join resolution between two different data sets.

So if I have a transactions table and a users table, I can go in to find various dimensions that belong to a user, like maybe their signup method, various kind of dimensions that might exist in that table. And then when an API request comes into transform, it says, okay, I have a transactions table. I have a users table. This is the user ID, which is a foreign key in the transactions table. This is the user ID, which is a primary key in the user's table. And so if I ask for how many transactions occurred by the user's signup method, then we would basically go and construct a query, which is just SQL. And it's just using the kind of logic that's defined in these two YAML files

to say, I need to take the transactions table, I need to join the users table, and I need to aggregate transactions to the level of the user signup method. And then I need to return that to the end user. And so that whole process of the spec, the like construction of various kind of, you know, joins and all of the logic, and then the rendering of that logic down into a specific SQL dialect is what is metric flow. Okay. So if there's some regular show, the user does not really have to define SQL, right? This is something that's pretty much, let's say, generated

by the system, correct? Yeah. So one thing that you can kind of imagine is the ideal input into a framework like this is nice, clean, normalized tables. That being said, we have done so much work to make it so that you can put in a raw event log or a partially denormalized aggregated table. You can put in a very wide range of tables, but basically the end user is just telling us what is in the data warehouse. And there might be a little bit of SQL, right? Like maybe you want to take a number of transactions that a user has had and define them as a power

user if they've had more than 10 transactions. You could go and write a case when a number of transactions is greater than 10 than power user else regular user. And so you can express this SQL. And that SQL will then be reused across every single time that you go and request that dimension. Okay. Okay.

That's interesting. And okay. So what dialects you support right now? I mean, with what kind of like databases that like users can interact with the framework? Yeah. like, data bases that, like, users can interact with the framework? Yeah, so within MetricFlow, you know, this is a big reason why we open sourced it. One is, what I'm talking about is, I mean, it's a huge engineering project. Like, one, you know, I think that we've been thinking about this for most of the last eight years. And so, you know, it felt right to kind of put that technology out in the world.

But two, it's really ambitious. The idea that you can take a bunch of different data sources and then be able to do multiple hops of join resolution to determine what's on the left side and the right side of these joins to be able to kind of freely traverse the warehouse is a huge project. And so what can you do with it? Well, you can basically ask for any metric and dimension where there is a joined path between them.

And so if you think about it, like, what would you use that for? Well, I mean, you can ahead of time construct data marts in a programmatic way on top of a, you know a well-defined kind of set of abstractions, set of logic, where you've defined your logic in a very dry way and you've then programmatically generated all of the different data marts that you would want to build. So the dream here is that you can take

what would be typically a very deep DAG, like a DAG that has many layers of data transformation, and actually flatten that to, I'm going to use my data transformation framework, something like dbt or just expression of SQL or Spark or whatever you want to use to clean up my data sets. And then from there, I am going to basically just expose these nice clean objects to a metrics layer to this kind of historically Airbnb has sometimes referred to Minerva as a denormalization engine, such that you can define that logic in one place, programmatically generate all of your data marts and even dynamically generate data marts

from downstream querying interfaces. So that's, I mean, that's like a huge, huge, you know, step from the world that we exist in today. But it's actually the world that, you know, a few large companies are existing in right now. So this is kind of what LinkedIn's unified metrics platform does. And this is absolutely what Minerva does at Airbnb. Okay. And all right. So we have like a huge project here with amazing potential. It's open

source. So how do you hope that that people are going to engage with it? What kind of contribution from the community out there are you looking for? Yeah, so I think there are lots of ways to engage. Frankly, what we're trying to solve is denormalization, which is a non-trivial task. And when you think about what is involved there, you need to be able to input any kind of underlying, you know, data structures. So you need to be able to support SCD tables and, you know, partially aggregated tables and, you know, raw event logs. And a lot of the work has been done to be able to support those types of data sets. But there's, you there's always new modeling techniques.

There's always kind of nuances there. There are lots of kind of features where we'll be opening more and more GitHub issues, but also generally we would love more issues. And then beyond that, I think that there are different support, there's support for different data warehouses. So today, Metric Flow, we started off with Snowflake, Redshift, BigQuery.

But next on the list is Databricks, Spark, Presto, Trino, MySQL, Postgres. These are all the key ones that we really need to support soon. And then there's a long tail of really interesting data warehouses really need to support soon. And then there's kind of, you know, a long tail of really interesting data warehouses that you could support with metric flow. The other thing is new metric types. And I think that this is, this is like where metric flow really shines. I think that previous semantic layers have kind of struggled to define complicated metric types. So, you know, there are, there are like various strained ways of kind of defining complicated metric types in something like LookML. And it should be very, very easy to basically define a function within metric flow to support some new metric type. So, you know,

an example is like a conversion window, you basically need to point it at two datasets, you need to say, or a conversion metric, you need to basically need to point it at two data sets. You need to say or a conversion metric. You need to be able to point it at two data sets. Say these are the timestamps in these two data sets. This is the conversion window. And then, you know, true, false, do I want multiple conversions to be able to happen within a window? That's like effectively the inputs into a function that could be a conversion metric. And that's kind of how the framework is designed. That's actually not a metric that we support today. But it's obvious that we should support that. So those are the kinds of conversions, or, you know, features that I think

would be really helpful. There's also optimizations. So we're trying to render both very legible and very performant SQL. And so we need to be able to basically optimize the queries as much as possible to make them as efficient and readable as possible. And we've done some work there, but there's more that can be done. Yeah, that's one of the things that I wanted to ask you, which is about performance, right? Like a lot of work is getting done like by, from like DBAs to analysts to data engineers, like pretty much everyone like trying to optimize the queries that they

have in order like to either reduce latencies or reduce the resources that are needed. Right. But this is part of like working with, let's say like they're all SQL query that exists out there. And it's a pretty complicated job, but in many times like requires like a lot of like deep knowledge of the database system, right? So how do you see like solving this problem? Because, and that's a feeling that I have, right?

I mean, there are like two extremes. One is like, okay, let's let the user control completely the query and do whatever they want, which is what they are doing today, right? There's the other extreme, which says, let's make the whole experience completely opaque, auto-generate the queries. You don't have any way as a user to do something with it, but we will do the best we can to optimize. And my feeling is that usually the optimal solutions, there's somewhere in between. You can't really be a system that optimizes in the best possible way for every possible query and

metric and use case out there. But at the same time, like you want to add more like usability there by automating as much as you can do. So how can this, how did you see like from a, like from the developer experience, like this being delivered at the end? Well, this might be the optimist in me, but I think I, I think I might disagree with that, you know, that you can't create something that's like perfectly optimized. And I can't take credit for this. Like this is my co-founder, Paul. But I think that the design of metric flows is really built around the idea that we can build both very optimal and very legible SQL. And the way that we do that is historically, all of the versions of these metrics frameworks that I've worked on in the past

have been these kinds of like template-based, you know, like let me define some macro and then let me, you know, basically wrap some other macro where I'm consuming from some like SQL that's, you know, within kind of a from statement and I'm doing some additional transformation on it. And there are kind of two conflated problems there. One is, how do I express the logic? And the other is, how do I build, like, the SQL? And so

the way that Metriflow works is, you basically create a query plan, which takes in the semantics and the API requests. So what's in the warehouse and what does the end user want? It then creates a data flow plan, which is, you know, read from this table, filter this thing, aggregate to this level, read from this other table, join these two things, you know, aggregate to this other level, etc. And you can create, you know, complicated query plans from that. And so you can really build like any metric type, you can build quite complicated queries off of this, which is, you know, when I talk about

trying to solve the problem of denormalization, that's roughly what I mean. And from that, you can then optimize that query plan. So if you have, you know, overlapping pieces of read from this table, filter this thing, aggregate this thing, well, I can pull from two different parts of, you know, a kind of a data flow plan, like this DAG of operations, and put that into a CTE or something like that, and then read from that in the rest of those operations. So you start with this very flexible way of building a structure of logic that needs to be expressed to get to what the user is requesting. And then you go and you optimize that. And then you take a DB-specific render that can render to the nuances of each of the different databases. And you render this optimized query plan into that database dialect. And so with that,

you can create pretty darn legible and pretty performance equal. I think it's similar in some ways to what Apache Calcite is. But written in Python and a fairly different approach. Okay. That's super interesting, actually.

I'm just spending a lot of time, like, with the team at Trino right now talking about performance and optimizations. And it's an interesting problem. It's a big problem. And it's something that, I mean, there are always trade-offs, right? And actually, it's kind of interesting because us as like a query engine, like we approach the problem a little bit from the different side. We are trying to figure out, for example, what workloads we should consider as more important right now and try to optimize the whole system to deliver more performance there.

So yeah, anyway, I'd love to chat more about the performance part, to be honest, but I think we can do that also offline. Yeah, me too. We can find another time. Yeah. All right. So do you see any kind of like, I mean, I understand that inside the company, we want

to define a very good metric about MRR, for example. We all agree upon this and we use it. Right? First of all, how easy it is for this to happen inside the organizations? Because humans are involved, right? And humans have different ways that they interpret information and semantics come mainly from them. Right?

So let's talk a little bit about that. Do you see like cases where MRR might be multiple times like defined inside the same organization? Do you see like consensus like at the end? And where are the limits between like technology and where like the human factor starts? Yeah. I like this question a lot, especially based, you know, on what we've been chatting about for the last 10, 15 minutes, because ultimately all the things that we just chatted about, very cool, very interesting, very fun. can agree on a metric and actually connect it to the places where people want to do analysis or people want to consume it.

A business user really only cares about rendering legible or performance SQL if their metric comes back to them really, really fast and it's the correct metric, right? And so this is kind of the fun part where the technology should just disappear under the hood. And I think that, you know, there are a few different pieces around lifecycle management, but maybe just at a high level, like what is the lifecycle of a metric look like? So you have the definition of a metric, right? Here is the technical definition. Can we get everyone to agree on, you know, philosophically,

like, what is this thing? Just ignoring the like SQL expression, you know, is this what MRR means? Are we filtering the right things? Are we including the right things? Does everyone know what that is? And in my mind, you know, there is a long, long way to go there. I don't think that we have solved that problem. And I think that if anything, it's a little bit organization specific. But I think that the way that we're doing it right now, which is either nothing, maybe an email at best, and well, maybe at best like a Google sheet or something like that, that has all of the human readable definitions of some of these metrics is not enough. And so I think that the important piece there is kind of, I think this is like an open conversation. I think more people should talk about how to get this done. Airbnb, I was kind of, I saw how it was done there. And I think that that worked at that organization. Airbnb has talked about this system that they've built out now called Midas, which is like,

how do we create a metric to find a business owner, to find a technical owner, put a stamp on it, let everyone know that this is a Midas approved metric. This is tier one, everyone should trust it. Everyone has agreed on the definition and here's the form that was filled out that has the human readable description as well as the technical logic. But I don like the form that was filled out that has the human readable description as well as the technical logic. But I think that, you know, I don't think that that's going to work at every company. Like, I think that generalizing human process is way, way harder than generalizing technical,

you know, technical solutions. And we have some, we have some opinions here and we've tried to express those in our product around kind of ownership and business and non-technical owners, tiering this like approval process to kind of mark a metric as stamped and accurate. But frankly, there's a lot more work that needs to be done there. You know, beyond that, you get to iteration. How are you tracking versions of metrics? Can you construct historical versions of metrics? Or do they kind of disappear once you evolve because the underlying tables are wiped away? You know, do you save a snapshot of that metric before you wipe away the table and kind of keep that in an

archive somewhere? There's a lot of kind of process there that I would say we haven't even scratched the surface of. Or like archival, you know, Hey, this metric's not useful. People should not come to this in seven months and say, and like try and do some analysis on it. Cause nobody's looked at this table. It's not even supported anymore. You know, this isn't the right definition. The, the one technical piece that I would say is how do you agree on an MRR definition? Well, you know, getting business people and getting technical people to all, you know to get in a room and talk about it is probably the best way to get that started. But at least technically, our framework doesn't allow you to define two metrics with the same

name. And every metric has a name in the code, which is some snake case like MRR underscore new users or something like that. But you can't redefine that thing. So there's at least protections against defining something with the exact same name. So, you know, I could have MRR new users and I could have MRR, you know, first time purchasers or something like that. And ultimately, you know, those two things will show up as probably different tiers and different owners. And like, hopefully we'll start a conversation where we can at least see that there are two different definitions. Yep. 100%. All right. That's all from my side for now. I mean,

I wish we had another hour chatting, but I completely monopolize the conversation and I have to give this page to Eric. So Eric, all yours. No, this is such a great conversation. So question for you, Nick. Yeah. Following along on what we just said, I'm thinking about our listeners who are maybe environments where metrics are changing really fast, right? And so maybe let's just imagine a world

where you can easily get agreement on MRR, right? Even if you agree on it, like everyone agrees that it sort of changes like next quarter because the business model changes, right? Sure, sure. Whatever, right? And so, I mean, of course,

that happens a lot with sort of early stage, you know, earlier stage companies. Sure, sure. having all of that. There are so many companies where the business model is changing, they're acquiring companies, adding new product lines, etc. I just want to know the way that you think about that. Because if you think about the singular case of a single business unit with a single metric for MRR, great, that's fine. But if that's changing a lot, how do you think about managing that? Okay. Yeah. So I worked at Airbnb when it was a little bit smaller. 2014 product team was maybe a hundred people and I saw it evolve to product team was probably, I don't know, 1500 something people. So I saw a lot of

growth there, a lot of evolution, lots of definitions of new metrics, lots of iteration. But there is a set of metrics which remain relatively stable, right? Like you're not going to completely, unless you completely pivot your business right which is truly a startup move and you know and i worked at a startup afterwards and i saw like new products get launched and we only had a few kind of stable metrics and now i you know work at a startup like we have our own metrics defined and transported they're evolving very quickly uh they evolve very quickly so i do live this but the thing I would say is that there are some metrics which are stable and they're very important.

So, you know, an example for us is the number of queries that are issued against the metrics framework from any source, from, you know, from our Tableau integration, from our mode integration, from Google Sheets, from within our UI,

from within the Python API, et cetera. That's a metric that we track. And it's very unlikely that that metric is going to change no matter what we ship or whatever we're working on. And we consider that a tier one metric, which basically means that it is stable and everyone in the company should feel safe consuming it. At the same time, we launch you know, launch new product features and we want to understand, you know, how people are, you know, consuming those features. And so, you know, we'll get some, you know, some new set of events and we'll define a tier three metric on top of that, which basically says, hey, no idea if this is good, but like

we wanted to put it on a graph. We just wanted to look at it, right? And maybe we'll see a dip in weird ways or we'll do some querying in the future that tells us it's wrong in some way. But basically, that allows us to kind of track the new things with the old things and keep them separate. I think that tiering is probably the best solution to this

versus just throwing everything in a big folder keep them separate. I think that tiering is probably the best solution to this versus just throwing everything in a big folder and sorting through it every time you have a question. No, it makes total sense. Okay. So that leads me to my next and last question. I actually lie a lot when I say that. So there may be one more, but we're close to the buzzer here. So this has been a concept that's fascinated me for a long time. So on some level, like if you take, say, an e-commerce business, right? The set of core metrics that don't change are very, very similar across them, right? I mean, the user flow can essentially be broken down into a handful of touch points that are essentially the same, even if they have different names.

Do you envision a future where you can essentially have a metric store that maps to your business model almost out of the box, right? And so when you adjust metrics in that context, you're really almost adjusting sort of components of the semantic layer in the way that you name things, right? But in reality, the semantics that you need to define are pretty much mostly known quantities, right? I mean, you could argue the same for say like a B2B SaaS that has a freemium model or whatever, right? Yeah. I mean, these are known business models, known touch points, generally known metrics. Do you

sort of envision a world where you sort of have out of the box, like this is sort of your metric store and like you don't really have to do a lot of definition. So I really do like that idea. And, you know, I like, unfortunately the answer is, you know, as like most of the time is with questions like this, like it kind of depends, right? Like, unfortunately, and what it depends on is how you want to measure your business, right? Like how you want to understand it. And I think that every business is unique and that leads to various nuances that lead to different definitions of MRR and et cetera. But I do think that at the core, there is a set of metrics where if somebody says, hey, I'm an e-commerce business, then we should

be able to say, okay, here are 20 metrics that you probably want to track. And you probably have some tables that look like this. And so let's put them in and let's kind of have this like boilerplate template for e-commerce businesses. I think that would be very cool. And I think that that's an interesting way, especially with what I said about, I think that the metrics layer should be able to visualize data at like a very basic level. It should be able to show the value of

the metric. I think that's a very interesting thing where you could imagine, you know, an e-commerce business just getting off the ground and exporting a bunch of data to the warehouse and then basically having the 20 most common metrics that they should be tracking, you know, visualize for them. Like, that's a great way to start off as a business versus kind of the situation that we have now of like, dump it all in, you know, munch it around. Like, it's a lot of work. And even kind of beyond that, like if you're using something like a, you know, Shopify, Salesforce, like, et cetera, they produce some kind of nice clean data sets. So I also like the idea of basically predefining those metrics on top of those data sets.

And basically, you know, having all of the kind of getting started work done, done for you. Yeah, super interesting. I, for a while, I've had this sort of, I wouldn't say like a dream, but like imagining basically running a Terraform script and it spits out like not only like a stack with all of the tools that you need for a business model, like the metric and sort of the table structures and all that sort of stuff. I mean, that's, I guess, technically possible, but it's super interesting to think about that, right? It's almost like stack in a box with the metrics layer and you're off to the races. How do you go and define all of these metrics on top of the tables that are landing in the warehouse and then programmatically generate all of the data marks? That, in my mind, is very similar to Terraform, Pulumi, et cetera, but for your data warehouse. And the cool thing there is you make a change to a single metric and it's showing up in five different data marks.

Boom, that change cascades into all of the places where it's being consumed, which I think is the really powerful part about thinking about the metric slayer as a denormalization engine to tie it back to the technical talk earlier. Yeah, absolutely. All right. Well, we are at the buzzer. Nick, this has been so wonderful. We learned so much. I wish we had another hour, so we're going to have to get you back on the show. Cool. Can't wait. Thanks so much for having me. Costas, I love talking to all these smart people

that we have on the show. And one of my big takeaways was the discussion around, and I would love your thoughts on this too, sort of the legibility and performance of the SQL queries that Transform generates. And he was very explicit about the trade-off there and how it's a pretty hard problem to balance both legibility and performance. And I just appreciated that he multiple times sort of stopped and, you know, almost reflected on the difficulty of thinking through that problem, you know, and whenever anyone is, you know, acknowledges upfront the difficulty of a problem and sort of reflects on it, I know that they're really thinking deeply about it, but that's actually more your area of

expertise. So what did you think? The thinking. Well, I didn't necessarily mean that I meant performing queries on the, on, on data infrastructure, not in your own head. Yeah. Yeah. Well, I have to add something to what you said about Nick though.

I mean, I totally agree with you about like getting in contact with all these like smart people and obviously like the people, the team transforms like very talented and smart group of people. But what I really loved with our discussions with him is also how passionate he is about the problems that he's going after, Which I think is like, let's say, when you have this passion together with this amount of smart people, I think something good will come out of it for sure. Having said that, I mean, yeah, there are many, many trade-offs that these technologies are going to face and have to decide. And that's what makes the problem that Nick is going after really, really hard. There are trade-offs that have to do with how people interact with each other,

because there is a very strong human factor there. And many different personas, for example, involved in using the product. And obviously, many trade doves, the technology itself. Balancing all this and finding and doing the right trade doves is what I think is going to decide who's going to win in this space. And it's one of the reasons that I'm really looking into these technologies because it's really fascinating to see how people can build experiences and products around such complex problems. I agree.

Well, thank you so much for joining us. Tell a friend if you enjoy the show. Ask them to subscribe. We always like new listeners and we will catch you on the next Data Stack Show. We hope you enjoyed this episode of the Data Stack Show. Be sure to subscribe on your favorite podcast app to get notified about new episodes every week. We'd also love your feedback.

You can email me, ericdodds, at eric at datastackshow.com. That's E-R-I-C at datastackshow.com. The show is brought to you by Rudderstack, the CDP for developers. Learn how to build a CDP on your data warehouse at rudderstack.com.