Drill to Detail - Drill to Detail Ep.93 'Lightdash, Looker and dbt as the BI Tool Metrics Layer' with Special Guest Katie Hindson

Episode Date: March 9, 2022

In this first episode of a new series, Mark Rittman is joined by Katie Hindson to talk about Lightdash, a new open-source alternative to Looker that uses dbt as its metrics layer and semantic model.Lightdash.comWhat is the dbt Metrics Layer? Lightdash, Looker and dbt as the BI Tool Metrics Layer

Hello and welcome back to a new series of Drill to Detail and I'm your host Mark Whitman. So I'm very pleased to be joined today by Katie Hinson, Head of Product and Data at Lightdash. So welcome to the show, Katie, and it's great to have you join us. Thanks so much for having me. So Katie, maybe just start by telling us a bit about yourself, maybe your background, how you ended up at Lightdash, and I suppose a little bit of kind of the backstory as to why you're involved in this industry. Yeah, for sure. So I kind of had, I guess, my first introduction to data analytics in university.

I studied something called computational ecology. There could be a whole podcast in and of that. But yeah, I learned pretty quickly that I didn't want to be in academia. But the biggest thing I learned there was that if I was going to make a big difference in anything, I needed to like properly learn how to code. So I was in London. And as you do in London, I got my first job at an insurtech company, which is where I met the co founders of light dash Hamza and Oliver. Yeah yeah and moving to the industry was like definitely kind of a baptism of fire it was so different to working in a lab and I think the biggest thing I learned there was that it it's so important to be able to make business decisions using data so in academia you know it's you're never thinking about a company and business impacts.

Whereas at Cytora, it's really where I learned like data analytics drives business impact. And how do your analyses impact the company's kind of goals and opportunities? After that, I moved on to Monzo, which is for those people who don't know,'s an app only bank in the UK and Monzo is really where I got good at using data to make business decisions and it's also where I got introduced to the first BI tool I'd ever used which is Looker and I was obsessed to say the least. I actually, I made the onboarding series for Monzo. It was called looking good. I had an emoji called Katie, the looker lady. Like I was, I just, I loved it. And I think that was really the time where I realized that there was this really kind of human part to BI. Like data is only as good as the people you've gotten to use it and appreciate it. So if people don't trust your data or don't trust the BI or

know how to use it, then it's kind of a useless tool. And it was also a place where there was this feeling of access to data for everyone so this democratization of data that really kind of stuck with me um and it's something that we're bringing a lot to light dash fantastic i mean so monzo you're working at monzo um is is a whole they could have an episode to itself i think there and and so it's interesting that that you were using dbt there, using Looker. So I suppose your role there was, I suppose in the Corales bio that you put together for a presentation you did recently, you were talking about there part of the great migration from scattered untesting SQL files to a beautiful souped up dbt project. So maybe just tell us a bit about what you're doing with dbt there and looker and and generally why you say you're obsessed by looker what what what was it about the role and

the technology and so on that that led to that obsession and i say that as someone who's also equally obsessed by those products as well yeah totally so um i think that there's like a couple of things in there the the first one is really Monzo's data team, which is led by someone called Dimitri. It's incredible. It's this idea of like empathetic systems thinking. So data is a system, but it's being used by people. So it has to be functional. Otherwise, it's useless. And so all of the tools that were implemented right? You solve the fact that you don't have testing.

You solve the speed of generating data tables, like the output of SQL models. You enable version control of your SQL modeling and you enable documentation. There's just, there's so many things that dbt allows you to do with data modeling and that was so beneficial to the data team because it allowed us to give better i mean faster more reliable data to the end users so dbt was like a no-brainer when it came out and we did this huge migration and it was a huge team effort and it took so much time, but it was so worth it. And then another example of that kind of building tools for end users was the implementation of Looker, where Looker allowed people to explore data using this kind of modeling layer. So they get building blocks that they can use to

build their own questions about the underlying data that we had at Monzo. So those are kind of two examples of the general data team's approach to giving access to data to users and how we kind of built out the data infrastructure at Monzo. Okay okay so so I imagine with the role you're doing there and Monzo itself and just in general you must have been fairly kind of hot property in some respects and you're in the center of you're at the center of a fairly interesting sort of industry and a very respected company there but you know you chose to move into product and you chose to move on to Lightdash so maybe just give us a kind of an elevator pitch

for what Lightdash is and why were you interested in joining them and taking on that particular role in product there? Yeah, so in one sentence, Lightdash is an open source analytics tool for your dbt project. So we enable everyone to answer their own questions using your company's data um and actually i joined hamza and oliver when we were a company called hubble

and we were doing automated data validation and it was not it wasn't the thing that we were kind of made to do so we ended up consulting for a bit uh and we were building out data infrastructures for other companies and we kind of kept running into this problem which was there were a lot of like existing tools for bi but but none of them really fit exactly what we were looking for. So we kind of found dbt and we'd always implement dbt. But then it was like, okay, now we have to pick a BI tool. And I mean, Looker would have been amazing. But honestly, it's just too expensive for a lot of companies. And it was really annoying to keep copy and pasting everything from dbt and then we had metabase which was super cool but it didn't have like this modeling or semantic layer approach to

bi and so then we were like oh why don't we just build a bi tool that uses this modeling layer approach plugs directly into dbt and so this is what we ended up doing and that's kind of why we're light dash today uh we kind of saw we saw this problem in the market of copying and pasting all of this business logic it's like where do i put my business logic is it's in the bi tool do i put it in dbt and then there was this other problem of you know know, DBT has all this metadata about data transformation. So things like freshness and accuracy. And a lot of the time, the people kind of consuming the data are like, can I trust this? So we can incorporate all the data from the data transformations, give more context to consumers, more trust in the data. So all good things. And that's kind of how we ended up at Lightdash. And for me, my role at Lightdash is I'm the high and lofty head of product and data, but in a team of six. So it sounds really fancy. But what ends up happening, right, is you start as a team of three. And it's sort of like, well, you were a data scientist at one point,

so I guess you're the head of data now. No, I'm sure it was more scientific than that. But yeah, I've moved into more of a product role, which I was working as a product data scientist at Monzo. So I have some experience kind of thinking about the impacts of data on product. But really, the reason was that I was our end user. So I was able to empathize the most with the users and really take an initial direction of pushing where we wanted the product to go. Okay, okay. So for anybody that hasn't seen Lightdash so far, maybe sort of paint a visual picture of what it would be like to use the tool and what was the user experience like? And also, who do you see as being the kind of the user persona or the typical user of Lightdash? So who would that

person be and what would the experience be like when they use it, really? Yeah, for sure. So we kind of have these two user personas that we talk a lot about. The first one are these data builders, who are the people that are maintaining and building the content in your BI tool. And then we have data consumers who are the people that are consuming the content. So looking at the charts, looking at the dashboards and making decisions from those. And sometimes people can be both and I think there's an overlap between these which is quite important but yeah so when you initially set up LightDash this would normally be a data builder doing this and as of right now you need

to have a dbt project so you have your dbt project which has the definitions of all of your models and your dimensions or the columns within those models and uh descriptions of those things tests everything like that and like with light dash you literally just plug in your dbt model so you'd open light dash you define the dbt model location in github um and you'd tell light dash what data warehouse and project to point to to look for the data uh and then you would compile light dash and you would see all of your dbt columns basically all the models from your dbt project and all the columns that you've defined of your dbt columns, basically all the models from your dbt project and all the columns that you've defined in your dbt project. So light dash relies on dbt schema files. So these are these YAML files, they're called, which is where you define all of the

properties of your models and your columns and all those things. So Lightdash just looks for that and says, okay, you've defined all these models and all these columns. We're going to show you that in your BI tool. And that's the amazing thing. Like that's a magic kind of bit of Lightdash where as a data builder, you're just kind of plugging things in. And then the other fancier thing is that you also define your metrics in your dbt project. So within your dbt models, let's say you have a users table, you would create a metric called count user ID. So under the user ID column, you'd say create a metric called count and then that would also pop up in light dash. So as a data consumer I'm able to look at the charts that other people have built and we're building out chart some more kind of dashboard and chart level interactivity coming soon. But yeah I'd be able to click on these

dimensions and metrics that are already in my light dash project and be able to click on these dimensions and metrics that are already in my LightDash project and be able to build insights from that to kind of answer my own questions. Okay. Okay. So obviously there are other open source BI tools out there, like say Superset, for example. That would, yeah, they wouldn't have the same integration with dbt, but I suppose, you know, an argument there would be that they would have you know if you've built your dimensional model out well enough in in dbt then there wouldn't be a huge amount more to do i mean what would what's the extra problem do you think that that that

light dash is solving that you wouldn't get without looking at particularly picking on say sort of superset but against a tool that doesn't have a metadata layer yeah Yeah, for sure. So we think that really sharing data through BI and an org is the most successful when it has both a great developer experience for these data builders and a great user experience for the consumers. And the existing tools have a variety of user experiences for the consumers, but across the board, it was kind of a bad dev experience for the builders. And with Lightdash, we wanted to build this best-in-breed user experience

for the consumers, but still offer the kind of first BI tool that worked with the modern data tools that analysts and builders love to use, so dbt. And that was kind of where we found a gap in the market okay okay um and okay it's interesting so so maybe let's go into a bit of detail you mentioned about bit about adding metrics into the into the dbt model and so on and i think and i suppose an interesting bit of context here is is is um i suppose how that works um and also maybe talk about let's talk

about um how that relates to the existing initiative that's going on within within dbt itself to add metrics in there so i mean so first of all maybe just go into a bit more detail about what do you need what would you need to add to a dbt project beyond what is already in there to make it work with uh with light dash And how does that kind of work in terms of additional metadata and the thinking you have to do for it really? Yeah, so right now metrics are defined in your DBT model schema files, which is where you define all the properties, you define all the columns, and you add all your descriptions

and testing in your dbt schema files. And light dash requires you to add metrics within your column name. So, for example, if I wanted to aggregate the revenue by some, I don't know, by some date or city or something, I would add a metric to the revenue column, which creates a sum of that metric. So similar to any other BI tool that has this modeling layer, you have to think about what aggregations of data your users are going to do. So this is the human part still of a lot of BI. And it can be automated in a lot of ways. Like you can just create count sums and whatever of continuous and discrete numeric values. But the idea is that you're defining your metrics within the context of the BI, the kind of BI transformation layer

that you already have. So these metrics are defined under your columns and they're aggregations of those columns. And you can use a custom SQL, or you can use these predefined metric types that we have, like sum, max, min, average. And then what happens is it gets compiled when you do a dbt compilation and it appears in lightdash automatically and we also allow you to join models together so for example if my users model um a lot of times i have questions about companies that users are in then i can actually join my users model to my company's model so in lightdash the metrics and dimensions from both your company's and your users' tables would appear in the same explore space so that you can kind of ask questions using both of those data sets.

Okay. Okay. So I suppose one thing, I think that immediately, I think that appealed to me about what you're doing with Lightdash is a problem that we face when we, my company is developing, I suppose, full stack stack modern data stack applications for our customers is is you know the duplication of i suppose thinking and development time and metadata between you know the dbt project and and look ml you know so the looker layer there um it was i mean you said earlier on about about the developer experience what are the what are the benefits or i suppose what feedback have you had from people about the benefit of making everything go into one model rather than splitting it over

both the LookML model and DBT? I mean, what feedback have you had on that and the value of that really? Yeah, I mean, you said it yourself. It kind of sucks to copy paste everything over. And I think just the fact that you have to ask yourself as a data analyst or analytics engineer, where should this go? That's just like the definition of a broken user experience. And so we've had a lot of feedback and that was actually the biggest

unique selling point for us. And it is the fact that we just seamlessly integrate into this thing that is your modeling layer um and i and i think that's what's so amazing about dbt is it's pushing the capabilities of this transformation layer and look ml was that tool for a really long time and they are still that tool for a lot of enterprise scale companies. And they are pushing more and more with this kind of LookML layer that they're developing. But I think there's so many people that when you tell them, oh, you don't have to decide where to put your kind of data transformations they're like oh finally like i totally understand that problem um so yeah we've had a lot of feedback about how it's helping

people i think the the impact of that it's hard to measure um it's it's something that a lot of people do yeah yeah yeah certainly for me that was the interesting thing i mean it seemed that seems such a a kind of um an obvious win really where where you have it in in one place there yeah so but but is there more of a conceptual thing going on as well i think i think it was that there was a twitter thread i think joseph moon um from hyper query was talking about well what you're doing and what they're doing and what other tools are doing where you have got this maybe sort of disaggregation or the separation of the the bi tool and and the metrics layer you know is there a wider kind of

i suppose philosophy and thinking behind this saying that you know what we do is we build a bi tool we don't need to be actually owning the definition of metrics and so on if they're already there is is there a wider philosophy behind this at all yes i think there is and i think that it's it's funny because we kind of accidentally ended up on this metrics wave and this hype of the metrics layer. And from the beginning, we very consciously decided that we didn't want to become the metrics layer. We didn't want to build LookML. dbt does an amazing job at building this data transformation layer. And there's a bunch of other companies

that are building up these metrics layers. So Transform, Metricual, MetricQL, I can never remember how to pronounce that one. And the idea that BI should also be your data transformation layer, it just doesn't sit right with me. But I think that there are still data transformations that happen in a BI tool. And I think completely separating data transformation from BI is just, it doesn't, it's not feasible. So you still have table calculations. You know, when you're looking at a visualization, sometimes you see things that you don't see when you're looking at the code. So it's like, hmm, I wonder if I sum that over some weird 35-day date aggregate. And then this new pattern pops up. So you still need to have this ability to kind of manually change some of the information. But data transformations that are consistently used by everyone at your company

should be in a place that's not before the end of your data funnel, which is really where BI is used to kind of look for insights in things that are well-defined pieces of business intelligence and like business insights. Okay. Okay. So what about, I suppose, what about unique thing not unique things but certainly signature things from from looker with with look them out so for example symmetric aggregates and so on you know is the intention to to try and get kind of feature parity on those or is maybe it may be going a different direction thinking about maybe the relationship between metrics i mean is there any sort of thoughts from you about what the roadmap would be around maybe where you might want to see you taking the metrics layer for dbt and maybe where dbt are taking it? Yeah, it's a really good question.

And we've thought I've actually spent a couple of weeks thinking about symmetric aggregates and kind of I think it was like November. I read a lot of articles about it. And there's everyone deals with it kind of differently. And I think Looker's approach to it is very unique. So for people who don't know what a symmetric aggregate is, it is basically where you join two things together and if you try to aggregate something where it's not uh where it's a one to many or many to one join or a one to many join i forget which way it is uh you can basically get false uh or are un not the results you were expecting to get so you can get sums or averages of things basically anything that kind of sums or sums things.

They become too big or just incorrect because you've joined things many times that shouldn't have been joined. And so Looker does this using some SQL kind of magic. It's pretty impressive the way that they figured it out. And there are some questions around how you actually want to think about these aggregates. Do you want them to be incorrect? Do you want to flag to the user that this is actually not the way that you should be aggregating data? And so the approach we have right now is we don't actually do anything with it but the approach we're going to have soon is to flag to the user that this is not the way that you should aggregate your data so telling them you know you've made a symmetric aggregate that that's not you're going to get the wrong In the longer term, I think that end users typically expect symmetric aggregates to be aggregated to the quote unquote right answer, which is normally the non-symmetric aggregation.

So it's kind of like they expect the join to work, just work, and not to give them this, not to make them question how they've joined things together or how their metrics are related to each other. So in the long-term, we'll probably take more of the looker's approach, which is to aggregate things the right way by adding kind of unique characteristics to the join values. But for now, it's not something that we really prioritized okay okay okay so come back to the metrics layer in a sec but one other thing i

suppose that i was um very curious about was the the business model of of of your company so so the company behind light dash so you've gone for the the i suppose the open core model is one way is that way it's described now where you've got a sort of a core kind of product which is open source and got contributions from people and so on and then I think if you look at the website there's a hosted version as well but just tell us about I suppose the business model and why you chose that particular model. Yeah so we are an open source tool which means that all of our code that builds Lightdash is available publicly to everyone on GitHub. And we've been involved in open source communities and we use open source tools.

But I think there's a couple more reasons why I chose to be open source. For me personally, at Monzo, I saw this transparency by default in action. And building transparently means that you're able to build with a community and users will have great ideas. And it's kind of our job to just prioritize them effectively. So being public by default means that we're able to really use all of our user input and build with them along the way as much as we can, which means that at the end of the day, we'll have a better product. The other big thing, though, is that open source really drives innovation. Like open source software is modifiable, extensible, and reusable because you

have the source code so you can extend functionality. And Lightdash actually started by extending the meta key in dbt, which is a community contributed feature. So we're like heavily rooted in this open source building and extending. And I think the other big thing is that open source development, it enables tech to stay more relevant over time. So you get an input from a diverse group of people with these diverse problems. And, you know, the project becomes more robust as different people find different uses for it. So yeah, all of that is kind of like the soul of why we became open source or why we built open source in the beginning. And then the other part of this is why we decided to build a core cloud, kind of like a paid for version of the product.

And I mean, realistically, not everyone has an engineer on their team that can dedicate all of their time to running a light dash instance for them at their company. Some people might have that, and some people might have an engineer who has a time, who has the knowledge in Docker and things like that to be able to actually launch and host a Lightdash instance for their company themselves. But the reality is most companies don't have that. So for us to be able to offer Lightdash

and offer people to host an instance for them, we had to charge for it because it's not a viable business model to spend our time and our resources hosting these instances and not charging people for them. So the idea is that, yes, we have these cloud and eventually we'll have an enterprise product as well. But ultimately, all of those, the open source product will always be able to kind of reach our mission, which is the same for every tier of the product.

And that's really like enabling everyone to answer their own data questions. So we're never going to limit the open source tier to the point that you can't do this anymore. But there are some features that will have to be charged for, like hosting an instance. Okay. Okay. So I'm curious as well, as the product manager for the product, how that works when you're an open source model

that can receive pull requests from anybody, really, when it's a scratch i remember when i i mean i went from consulting to product for a while when someone said to me the difference between being a product manager and a consultant is a consultant says yes to everything and a product manager has to say no to everything i'm kind of curious how do you how do you get your strategy delivered on when you are at the not the mercy but certainly you have a lot of very kind of uh very kind of vocal people who want to your their feature to go in not necessarily the one that you want how does

it work as a pm yeah it's a great question i think we use it to our advantage more than we do uh it we see it as an advantage more than a disadvantage so i think there's a couple of things that uh really help us to prioritize our work effectively, which is the first thing is as a team, we kind of decide on based on user feedback and based on user feature requests, what we want to build for the next sort of six weeks. Let's say six weeks tends to be our maximum time that we look ahead at the moment. It may get longer over time but uh the decision of what we want to work on is heavily based on user feedback so we're constantly

collecting feedback from people that are using like dash and looking at github issues um and i actually wrote about this in our github discussions because it was something we were talking about recently and the summary of really how we deal with feature requests is the first thing is we really spend the time to understand the user's request and give them a thoughtful response like do we understand the problem that they have are there any outstanding questions we'd have for actually building this or is there a way that we could guide them a bit so they could actually build it themselves? And then we add all of our user feedback to this tool called product board so that we know how popular a request is from users. And kind of that feedback comes from anywhere from Slack, from, you know, DBT Slack, from

GitHub. And then there's kind of this prioritization of requests from there. So if a request is breaking like Dash, then we prioritize it quickly. Like this is more for bugs than for feature requests, but if your product's broken, then you fix it. And then kind of the next level of prioritization is if tons of people are asking for it, or if people are saying, this is the reason why I'm not switching to your tool, then we seriously consider building it sooner rather than later. And we also ask ourselves, you know, is it more important than the work that we've lined up for the next few weeks?

Reprioritization, especially at our stage of a company is like, it's going to happen. So we kind of embrace it and take it on as it's amazing if we get feedback and people are interested enough that they want something and we find out that they want something. And yeah, I guess the last thing is like if it's already related to what we're working on, we'll probably just chuck it in with the other features we're building, especially if it's a small change. But we definitely use this influx of feedback as a benefit rather than kind of seeing it as a curse um so far i'll say like four months excellent excellent so so there's also but actually the first way i got to know about you you guys was the dbt to look at um um sort of get get repo you've got as well or certainly that that that utility you've got is that something i mean i guess it's it's a side thing for you and i can see where it came from but but what what is that something that people use a lot and and you know what was the back what was the background to that

really in the past yeah so dbt to looker is actually how like dash started uh we were kind of so we've done as i said we've done a bit of this consulting uh and we were in of, so we'd done, as I said, we'd done a bit of this consulting and we were in this ideation phase of building a company. Like basically it's a nice way of saying we weren't too sure what we're going to build next, but we were thinking a lot about the problems that we'd run into consulting. And this was obviously one of them, which is it's really annoying to copy paste all of my code from dbt to looker so oliver who's the cto at like dash went away and he built dbt to looker and then you know because you're kind of like ideating on stuff and we were just like you know what why don't we actually build looker

like why are we building this translation layer why not just build the bi that plugs in and that was where light dash came out okay okay interesting so so something else i'm interested in is is more from the strategic side you've got i think there's two relations i imagine there's two relationships that you need to manage well and give thought to and and kind of make sure that um i suppose you've got you've got a strategy on you've got the relationship with dbt labs and i'd imagine also a relationship in some form with looker as well or certainly you know there would be an awareness from them of what you're doing so maybe talk about dbt labs first of all um so as you said you mentioned

earlier on the the um the wave of interest in in in kind of met in metrics layers that's out there and you know you and other vendor other other kind of companies have come up with your own takes on what that might be um and now there's an official one what's your what's the so i suppose the the strategy around the fact that there's one emerge is that there's an approach to metrics layers coming from dbt labs that is not necessarily as fully featured as yours but maybe what might become a standard you know what what's the what's the strategy around around that and managing your i suppose your strategy around metadata and and the metrics library in relation to that yeah so um the dbt metrics layer is great we are very happy that they're building it uh because we're you

know we're connected to dbt and we consciously didn't build a much like we consciously didn't want to build a metrics layer so we consciously said you know we're going to build metrics into dbt's transformation layer if dbt wants to build it they will build it way better than we will because we kind of built it as this part that it was a requirement for us to be able to build the product that we wanted to build and so ideally we'll be able to kind of plug into any metrics layer that we want so some people are using dbt but then let's say they have transform for their metrics layer so we want. So some people are using dbt, but then let's say they have transform for their metrics layer. So we want to be able to plug into your metrics layer and your transformation layer

and allow you to explore your company's data. So right now you can kind of achieve a lot more using like dash's own metrics, but for simple cases, you can use dbt's metrics. But as the ecosystem develops around dbt's metrics. But as the ecosystem develops around dbt's metrics, I'm sure we'll see other tools and functionality built from using these metric definitions, like seeing your metrics rendering your dbt docs or accessing your metrics in other apps. And this is where we're really excited to see how the metrics layer evolves. And we're happy to kind of evolve with it. but we're definitely not trying to build a metrics layer ourself um yeah and i mean right okay so it was true it was more through so it was more

through um through need rather than a strategy really yeah that you built your own really yeah i mean eventually we the ideal scenario for us is having like an open source standard for defining metrics. Wouldn't that be so cool if everyone, all of these applications had different way of services, like serving you metrics, but they were all defined in the same way, then kind of the explosion of third-party apps that could be built on that would be incredible. So that's really what I'm hoping for. And I hopebt runs that charge but we'll see okay and so i suppose the elephant in the room the relationship with looker i mean so it's it's obviously i'd imagine it's a great um compliment to what they built um but uh how do you but how do you i mean how do you run that kind of i suppose tightrope

between between you know making it clear what this product is about but also not bringing the you know the uh the too much attention on yourselves how are you managing that relationship really yeah so there's I mean there's no direct relationship that we have with Looker as like a business um I mean if if they got to a point of thinking that we were a problem, then I think that we've, we've done some really great things. So, uh, I'm not too worried about that for right now. I think that they're doing pretty okay to, if they're competing against someone like us, but, but initially the idea was, um, we, we, we basically, we wanted to build this developer first approach that's kind of unique to Looker because of LookML. And this was wanted, this was something that we really wanted to

associate ourselves with so that we could resonate with the right audience. So the setup plan, it was easy, you know, for people to be like, oh, it's Looker, but it's open source or like, oh, it's Looker, but dbt instead of LookMLML so it was really easy for us to resonate with the right people and get people being like oh that's what you do um longer term obviously we're building our own brand and identity and and we want to build this best in breed ux for consumers that's that's also built to be used with modern data tools that analysts and builders love to use like dbt and i think that looker is incredible in what they've done for the bi tooling and the way that they've kind of approached this semantic layer approach to bi but i think there's a lot of other tools that do amazing things as well like metabase and their their the drill down functionality in metabase or tableau and their visualization uh abilities like there's so many things that other bi tools

may do better than looker and eventually that light dash might do better as well but yeah initially it was very much like a an easy for people to understand approach to building a tool yeah i guess i guess a couple of things on that for me i mean you've got first of all it it's probably fair to say that look ml it has been in in maintenance mode i suppose to some extent for a few years now i mean there's nothing really apart from maybe um think about this maybe support for aggregates for example and and the aggregation layer they're putting that recently it hasn't really been extended much further and i think you know there are probably areas like for example maybe the relationship between between metrics and there's maybe many many things that you know that you

could even go in a direction further than look have gone really um so i guess you know i probably i'd imagine you're thinking it's great to kind of start from that that's a clear clear reference point people have but you must have lots of ideas to go well beyond what they've done really um as if like Looker was being developed now although it's not obviously you know thinking where I take it further really I mean things like hierarchies between you know the relationship the concept of hierarchies it doesn't really exist in Looker Mel um is that something that again putting on the spot as such but is that one of the areas, for example, that you're thinking, you know, we could extend the concept further,

maybe in conjunction with the dbt team, beyond what Looker have done? Yeah, so I think that there's a lot to do within the modeling layer that we can use, we can not take advantage of, but kind of take advantage of the rich metadata that DBT gives to us using the transformation layer. So things like data quality, freshness, accuracy, these are all things that we have access to by being plugged into your transformation layer. In terms of how we communicate the relationships between your data, that could be very useful. I think there's a way bigger chunk though missing for data consumers, which is context. If I'm looking at a dashboard, I don't really care how users are related to companies. I care about finding the content that I was looking for about a company,

or, you know, looking at what other people have asked with that dashboard. Or if I have a question, like, who do I ask? And so I think there's a lot around context that we're really keen on pushing in BI tools and that's what we want to build in Lightdash. So things like suggested queries, this is actually something that Looker more recently built, but finding content, like how do we enable you to find and answer the question that you're looking for? And also you have these other tools like Amplitude, which are building these kind of product analytics use cases or like Mixpanel. And it's crazy to me that you need two BI tools to do that. Like, that just feels so wrong and and that's the thing it's mixed panel and amplitude are so good at what

they do that people are willing to pay for looker and amplitude which is it's it's unbelievable so so i think there's so much more to do for data consumers actually um that a lot of bi tools aren't doing and and they're i mean eventually there's also a lot of things to do with kind of data hierarchy and how you look how you're able to visualize the underlying data in your transformation layer as like uh as structures as opposed to like visual charts to consume the data but but i think that's something that's less pressing to me than kind of the giving and consumers more context okay okay well i mean it's been great speaking to you um that is katie and i'm you know i wish you all the best for the for the product i

mean what what's you mentioned about your six week you know your six week sort of i suppose yeah what you're doing the next six weeks and roadmap and so on yeah what what would be maybe give us a little kind of flavor of what might be coming down the road here with light dash in in the near future that would be uh even more of a kind of like incentive for people to look at the product now and keep an eye on it. Yeah. So one of the biggest pieces of feedback we've had recently, or not recently, kind of since it started on kind of chart configurations, dashboard interactivity, how users are able to interact with the visual aspect of LightDash. So there will be a lot more development and things coming that way. And then kind of the longer term, we want to build out a lot more functionality around building trust in your data.

So, again, bringing in more of the dbt metadata around data quality and freshness into the BI tool, thinking more about like end to end analytics. So using the fact that we're an open source project to make things like dbt packages with light dash charts and dashboards out of the box. So like, you plug into, you know, your five tram, I don't know, Twitter thing, package with dbt. And there's also light dash metrics in there. So really using the fact that using what we have to our strengths, which is being open source, and kind of being able to think about what our community wants. And yeah, also dabbling in these like product analytics use cases. So looking more at how we can kind of integrate this amplitude and mixed channel type functionality into a BI tool and making it easier to use for everyone.

Fantastic. So last thing, really, how do people find out more about the product? How do they get to do a trial or get to experience what it's like? And what's the next step, really, if people are interested after listening to this episode? Yeah, so you can find us on github that tends to be where we're hanging out the most often so it's the project is called light dash uh it's you can star it and we'll plant a tree for every star that we get uh so we're building a forest slowly but surely uh and then if you want to kind of just play around with the project, we actually have a demo project that's at demo.lightdash.com. Uh, and you

can go to the LightDash repo and you can see all the information about how to install it yourself. If you're interested, or you can sign up to our cloud wait list. Or the last place you can find us is in dbt slack. So in the tools dash light dash channel, we hang out there quite a bit. So there's a lot of places you can find us. Fantastic. And so as opposed to the ultimate endorsement, I spent last weekend actually putting together light dash against our own corporate data warehouse, our own kind of company one. And yeah, yeah fantastic i got it working over about 10 hours or something managed to create the metrics layer um get it plugged in run it in heroku and it's now you know running against a i think 150 model dbt sort of package um yeah

fantastic it worked really well and um you can see obviously how it's yeah it's early days with certain the visualization side it takes a lot to say to make a product it takes it takes a lot to for you to say and then look is quite good at visualizations um you know so there's there's kind of work to do there but certainly the basic principle of being a connected like you say connected to your dbt dbt package and then instantly use those data models you know with a little bit of kind of additional definition of metrics it's it's very interesting and um you know certainly we'll be i'll be keeping an eye on it in future and probably running it internally for some of our reporting as well nice that's so exciting to hear yeah it's it's it's kind of crazy because when you're like oh we're gonna build a bi tool you realize how

much work it actually takes and uh yeah we're we're a strong team of six now but i think about 80 of the product was built by four people. So it's, yeah, we still have ways to go, but it's good to have room to improve. Okay, fantastic. Well, great to have you on the show. And thank you very much and good luck and stay in touch. Thanks so much for having me

and have a good night. Thank you. you