Drill to Detail - Drill to Detail Ep.92 'dbt, Spectacles and Making Testing Sexy' with Special Guests Dylan Baker and Josh Temple

Episode Date: June 23, 2021

We're joined on this episode of Drill to Detail by Dylan Baker and Josh Temple to talk about developing multi-layer modern data stack projects, Spectacles’ recently released integration with dbt and what's now possible with their new public APIExtending Spectacles with API-Triggered RunsSynchronizing Looker and dbt™ with SpectaclesDrill to Detail Ep.82 'Looker Development, Automated Testing and Spectacles' with Special Guest Josh Temple Adding Looker Regression Tests to the dbtCloud CI/CD Test Pipeline using SpectaclesDrill to Detail Ep.67 'Thinking Probabilistically and Scaling-Out the Modern BI Stack' with Special Guest Dylan Baker

so hello and welcome to another episode of drill to detail and i'm your host mark rittman so i'm very pleased to be joined today by not one, but two returning guests, Josh Temple and Dylan Baker. So good to see you both again. Thanks for having us, Mark. Yeah, great to be here. Brilliant. So why don't the two of you introduce yourself then to the audience? Maybe start with Josh first of all. Yeah, sure. So my name is Josh Temple. My full-time job, I am an analytics engineer at Spotify. And on the side, I co-founded Spectacles with Dylan, which we'll probably talk a little bit more about. But I'm a self-taught data engineer, analytics engineer. I've worked at companies as small as a data team of one at a bakery called Milk Bar, all the way up to larger data orgs like Spotify.

Yeah, Josh and I had a very similar background, I think, when he was at Milk Bar. I think probably the last time I was on the podcast, I was leading an analytics team at a company in the UK called Growth Street. Subsequent to leaving Growth Street, I decided to take my learnings, particularly early on with the dbt and Looker stack, which I was very fortunate to use um very early on in both their existences and went freelance and so i've spent a chunk of the last three years doing analytics engineering consulting both with larger multinationals who are putting these tools in place as well as kind of seed and series a startup so kind of running the gamut typically people putting in stacks involving tools like looker dbt snowflake bigflake, BigQuery, those types of things. And as Josh said, as of about two years ago, we started working on Spectacles, and I'm

sure we'll talk more about that in a second. But really about a year ago, we started pivoting more of our time towards working on the commercial offering of Spectacles, which is a big, big chunk of what I do today. Fantastic. So as you mentioned, I had josh on recently and we talked about spectacles on there so um you know question is why are you back on again and i think really for me um i there's been a new release of spectacles and for me it's been a game changer really in terms of usefulness of it um because like most um like many people in the in our sort of space and you're working with tools like um i suppose looker and dbt and so on i mean i'm always really interested in how we can test the i suppose the end-to-end pipeline

how we can test dbt and looker and the whole development work we're doing as one thing and so one of the new features we'll get onto a moment with spectacles is this new ability new integration you've got with um with dbt cloud so i was particularly interested in getting you on the show and i think that might be a second instance of launching your product before uh before you do actually on there um but um but yeah so the other thing really was i want to talk to you about i suppose the wider topic which is i suppose how do people test the modern data stack um you know in general and in particular when you're doing development across multiple products or multiple layers. So I'm interested to understand with you later on, you know, what are you hearing on the

market and what are you seeing around that? And also, I suppose, what's going on around APIs as well, because again, third time launching a product for you, there's some interesting work you've been doing around APIs. But maybe just first of all, sort of set the scene really. Josh, maybe talk about, I suppose, maybe just outline what the modern data stack is again and why testing it is kind of really important. It's a fairly big topic, but maybe just start off with that. Yeah. So it's kind of funny that we've named it the modern data stack because in order for it to stay the modern data stack, it has to

continue to change to be modern at each stage of evaluation. But I think generally what people are referring to when they talk about is what we've seen is of this modularization of the way that we approach analytics. And so previously you had these enterprise legacy BI tools that were pretty monolithic and they tried to do everything all at once and they were really expensive and they were clunky. And so there's been this rise of new tools that handle transformation, new tools that handle loading data into your warehouse. And then all of that really on the back of the cloud data warehouse

of BigQuery, Snowflake, Redshift. And so when you consider all that together and you kind of think about it as the modern data stack, generally you have some service that's loading data into your warehouse, that's Fivetran, that's Stitch, et cetera. Generally you have something that's handling transformations within the warehouse that could be dbt, that could be custom SQL that you're running in an orchestration layer like Airflow or Prefect. You've got a BI tool sitting on top like Looker, Tableau, Mode, Periscope, et cetera. And then you have, of course, the warehouse itself. So I think when people say the modern data stack, typically they're talking about some combination of those tools. Okay. Okay. And I think you have the honor, really,

I think, of making testing sexy, really, in some respects. I mean, you're about the only person I know who can write an article on the importance of doing testing and make it sound like it's as cool as really anything you could do with, you know, technology. I'm putting that on my resume. But, I mean, I suppose, where did your interest in testing come from and why is it something that that's really important to do, you know, with the kind of the easy to use stacks that we use today? Yeah, for me, it was really out of necessity. So when I was working at Milkbar, I was the data team of one per se. I didn't have a senior engineer who was reviewing my code.

I didn't have a complicated CI, CD pipeline setup or anything like that. So it became very clear to me that being a fallible human being, I needed to be able to check my work. And testing made a lot of sense to me because I could write tests to validate the assumptions that I was making as I was going about modeling the data. And I could very concretely ask questions of my work like um you know do the things that i've set up make sense are they cohesive do the numbers all add up and it saved me a ton of trouble and dylan i remember you from i think i i think i saw you once uh presenting or first time i saw you presenting was a looker meetup and you were talking about cicd and and so on there so maybe talk about suppose, some of your interest in this

and I suppose where you're interested in kind of deployment and that sort of thing came from. Yeah, I think as Josh said, I, at the time, I think definitely when we met for the first time, I was at a high growth startup that was doing a bunch of this stuff. And I think as Josh said, a lot of this is one checking your work. I think the other is you in these, uh, in these environments want to move very, very quickly. And so you want to be able to have this stuff such that you get notified very quickly if there's a failure, not just when you're doing the work, but at some point in the future, cause you don't want a sprint or whatever it is to be totally derailed. And I think for me, that was a big, big part of this. It was the idea of being able to put

stuff in place such that our work wouldn't get derailed by some bug that we hadn't tested or whatever it might be. And so that was certainly part of it. I think the talk that I think I was giving at the time was I was at the time working at a... Growth Street was a fintech lender, peer-to-peer lender. So for all intents and purposes, a financial institution. And we had some regulatory requirements around knowing who had access to what, when, and what we had built there at the time was something that was built on top of the Looker API that was kind of configuration as code, kind of something Terraform-esque for Looker that said, okay, I want to actually define all of my

roles, groups, permissions, permission sets, model sets, whatever it might be for Looker that said, okay, I want to actually define all of my roles, groups, permissions, permission sets, model sets, whatever it might be in Looker as YAML. So that when I push that to master, it gets, uh, sent back up to Looker so that we have a, uh, historic source of truth around who had access to what, when, um, in the, uh, in effectively our, our Looker repo. And that for me was one of the first big, I mean, I was certainly writing dbt tests at the time and doing a bunch of the stuff that, you know, I think a lot of organizations do today. But I think that was the first for me and particularly, obviously, building into Spectacles was, I suppose, like a seminal moment. It was the

instance where I got, oh, there is some really cool stuff that, in this case, the Looker API enables you to do that makes running your organization much more efficient. And so I think for me, I've always been, I was a guy who did VBA macro consulting when he was 16, because, or it wasn't even that I was doing VBA macro consulting, I was doing Excel consulting. And my way of doing it efficiently was to write VBA macros. And that was one of my first jobs. And I think for me, it's just like stuff that saves you time. I think I'm inherently lazy and that is why I devote a lot of attention and time to CICD. Okay. It's interesting. It's interesting. So, so tell us, maybe tell us one

of you how this led into spectacles or what was the kind of the, the origin story of spectacles? So, um, so Josh and I, I think knew each other. I think Mark, because of that article that you, you referenced that Josh wrote, I think that's probably when we first met, uh, I felt the need to compliment him on it and probably got in touch in dbt Slack. Um, and now the dbt Slack has its full history. I could probably go back and look, but anyway, um, I got invited, uh, through a mutual contact of ours, um, to speak at an event in New York while I was in North America traveling and just basically on holiday and was at a Looker meetup in New York. I was part of a panel. And at the end of the panel, I made a very flippant comment, which was something like,

there's this big problem with Looker in that I love it, but it never, particularly when you're doing a ton of work in DBT, it never actually checks that contract between the data warehouse and Looker. And so it's really easy for a database error to end up somewhere in development across either tool. And I said something to the effect of, look, if someone would build this thing, I would pay to use it and my clients would pay you to use it. Please, someone go and do this. And we went to the bar after everyone, or the subset of the people, I suppose, at the meetup and Josh and I got to talking and started talking about what that solution could look like and what it might kind of possibly end up looking like and being. And then a couple of weeks later over Slack,

he messaged me and said, look, this thing we talked about, I think we should go do it. I think we should build an open source version of it. I think we should pitch it to go speak at Looker's conference later this year. And that was really the kind of catalyst for everything that has come since. And so we built an open source version of the tool in late 2019, presented it at Looker Conference in November 2019, it joined 2019. And that was really from there, we chose kind of shortly thereafter to build this commercial solution. But that kind of first six months was where entirely remotely, having only met once, Josh

and I collaborated on this bit of software and really how Spectacles came to be. And it had a couple of really bad names early on. And I think ultimately Spectacles was a far better naming choice than anything that had come previously. You have to tell us what the names are now after that. You can't just leave us hanging there. So I'm just trying to remember them. I think one was we were focused on the idea that it was kind of managing contracts.

So one way was it was managing a contract between Looker and your data warehouse. And so I think one name that it was briefly called was Looker Lawyer, which is just a truly, truly terrible name. And then I think the other one, because the other way, the other kind of mental model we had for it was it was like a mechanic keeping things like greased and moving smoothly between all parts. And so I think for a while it was also called Fonz, named after the Happy Days character who was a mechanic. both my suggestions. Josh, who is much more creative and much more like diligent around the kind of creative and design aspects of the business, shot both of them down totally justifiably. And we ended up with Spectacles kind of playing on the like looker, it helps you look good type kind of play on words. There was another piece of the story there, which is we started looking at package names

on pip that were available and wrote, this is just going to make us sound like such nerds, but we wrote some code to all the, all the pip is data is on BigQuery and you can see what, what packages have been claimed already. And we started thinking about like, oh, in the same way that you want to try to get the best Instagram handle, like what are some common words that are not already taken as Python package names. And we started cross referencing like the first, you know, the most hundred common words in the English language with that list. And just looking through what made sense. I think bless was one that came up that we thought was kind of interesting, because it's sort of like you're giving a blessing to LookML.

And then Spectacles we obviously liked because of the kind of pun on Looker and this idea of like, oh, if you're using Looker and you want to look better, you can put on your Spectacles and improve your eyesight. I think you should have called it Looker Lawyer, though. I can't think of anything more likely to attract their attention than that or all the or the funds or the i mean you couldn't have picked you couldn't have picked two two two names that are more likely to get you in legal trouble so uh that's uh luckily for sure i'm not sure we could afford afford the domain for either those of either of those either yeah yes exactly so so um i mean so josh i mean you you've had quite a bit i mean

both of you actually but you have had i suppose other open source projects um that i suppose inspired you or led to to this i mean so so how how did how did you contribute to this what was your what was the technical part you brought into it really yeah so i i love open source i love the idea that anybody can put code up, and people can contribute to it and make it better. And so I think we were also, both Dylan and I are heavily influenced by watching Fishtown and the DBT journey.

And so we thought that was such a great model. It's like, OK, have this open source CLI. And we were seeing this being very, very popular business model of the core functionality being open source, but all of the annoyance of deploying and maintaining and everything being something you pay for. And we just thought that made a lot of sense. So we also thought open source would be a great way

for us to test for product market fit and see if we actually have built something that people care about. And so I think the minute when we knew that that was the case was when we presented at Join, no offense to whatever person was organizing Join, but I don't think that the session on like continuous integration and LookML probably sounded like it would be a big crowd pleaser. So they put us on the smallest stage and it was like, you know, very much an afterthought in the

way that our talk was marketed and everything. And then we showed up to the talk and the room was packed out like four rows deep. There were people standing in the back in the hallway, like trying to listen to this. And that was when we knew, okay, people care a lot actually about the software development lifecycle. And they care a lot about shipping quality code.

And people in data are starting to understand what CI is and why it matters. So I think at that point we realized, okay, this is something we should build, not just the open source tool. And of course, we're continuing to maintain and add features to that, but build out a paid service that people can plug directly into their teams. Okay. Okay. So we haven't actually talked about what Spectacles is yet. So maybe one of you, Dylan, maybe tell us what Spectacles is, and then let's talk about what are the new features in the new release, which is what we're here for, really.

So why don't you talk about what is actually Spectacles? So Spectacles is, we use a different term depending a little bit on the audience we use, but it's either an automated testing tool or a continuous integration tool for Looker. And I think partly we use that language because there are certainly Looker users who don't come from a software engineering background for whom continuous integration isn't a term where they immediately know what we'd be referring to. So we use those two interchangeably. But fundamentally, it provides automated testing for your Looker instance and your LookML. And at its core, it tests three things.

It tests whether all of the SQL that is written in your LookML is compatible with your data warehouse. So it's really easy when developing across your data warehouse in Looker to have a mismatch between column names or table names or something, a column disappearing in the warehouse, but still being referenced in Looker that results in a database error. So fundamentally, that is the first thing, what we call our SQL validator. The second is the content validator, which is built on top of Looker's content validator, but provides a couple other value ads that we think are really useful to people. And so helps you as you're developing kind of proactively be aware whether you are breaking content across your instance and helps you manage

them and kind of be aware of them and fix them before you merge a new code. And then the third is we piggyback on some of the LookML data tests to make sure that all of your metrics in Looker, so the first two don't really cover metrics, the first two make sure that everything keeps running, and then the third, the assert validator, the data test validator, checks that all of the metrics that are in Looker are actually correct. And that's one where there's a lot of that functionality in Looker, but we found the blocker to adoption of them was really the workflow around them and kind of making the results of these tests transparent and making them visible and collaborative for everyone on a pull request in a data team.

And so those are the three validators that we have, the SQL validator, the content validator, and the assert validator, the data test validator and the assert validated the data test data test validator people kick those off uh before the the upcoming or the most recent release that josh is going to talk about in a second people kick those off in a couple different ways they either ran them on pull requests to make sure that their pull requests were never breaking anything they ran them on a schedule to make sure let's say their overnight etl job on their warehouse didn't break anything and then reasonably regularly they'd run them manually and so you kind of had a couple options of how to run these tests by a couple different triggers. Okay, okay.

Before Josh gets into his bit, I just want to kind of talk, something that I suppose isn't always clear to people or it certainly warrants just going back into again is so people, most developers know they can run the validator in kind of the Looker IDE and they can validate the Looker mail, but it doesn't validate the SQL

does it? What does that really mean? What does that statement really mean? So when you click that big blue validate LookML button in the top right corner of your Looker IDE, really all that's doing is checking for compiler errors. It's checking that your LookML is legitimate LookML. It's kind of the equivalent of, you know, a YAML linter telling you that, oh, you've got all your indentations the right way. But it's never, and this is where that kind of contract conversation comes in, it never actually tests the contract with the data warehouse. It never really forces you to say, oh, actually, all the SQL here is runnable. And I think that's, I think people eventually figure it out, but I think it's a little counterintuitive when people first use Looker,

that they assume that, oh, I've clicked this blue button, it's returned a green tick, I'm probably getting these colors wrong. But that fundamentally, like everything with my LookML is okay. And that's not totally true. And so we kind of think of, and we still think that that button is really useful. Like you need to validate your LookML, you need to, the bare minimum is that you need to ensure that you're writing accurate Look ML. But Spectacles just provides a bunch of additional stuff on top of that, that we think for data teams who are taking testing seriously in Looker, make sure that they really, really have... I think it probably takes you from, let's say, 30% confidence to, we'd love to say 100%, maybe

it's only 90% or 95% confidence, but a much higher margin that all of your Look ML is actually doing everything you want, not breaking anything in Looker. And Mark, just to add to that and to really, really talk about why this matters. If you put yourselves in the shoes of a company that has embedded Looker in their product. So I have a product and it has an analytics page where I can go in and see a dashboard. And that dashboard is powered by Looker. It is crazy to me how easy it is to push SQL that doesn't work out to that dashboard, which could affect every single customer that you have. Because all you have to do is introduce a typo into the SQL that you've written and push that to production. And so for someone who's not just internal analytics, but actual product is hinging

on the accuracy of their Looker instance, that is just a non-starter. And so with Spectacle's being able to say, hey, we've tested your LookML, we know the SQL will run in your data warehouse, gives teams a ton of confidence to ship code, and they can sleep better at night knowing that what they've shipped is going to work. The final bit I would add to that is, and Josh and I talk about this a fair bit because we kind of straddle this analytics engineering world and the software world, is a bug in software is kind of reasonably glaring, but you're not going to make any... So one, you'll discover it quickly, but two, you're not going to make any huge decisions on the back of a bug. It's not going to derail your business most of the time. A bad metric, you know, people look

at a report once they go, okay, here's this trend. We're going to go and kick off this initiative to resolve that trend. And if that trend isn't actually there, the data's wrong. Actually, I think in our view, the ramifications of something bad in your data is so much bigger, like of a bug in your data is so much bigger than a bug in your product. And that's the other bit. So there's one is like, if you're using this data, as Josh said, operationally as part of an application, but two, if you're a data driven business and you're making big decisions off the back of your data, having a bug in that data can be incredibly costly, like incredibly costly. And that's one of the other things that I think we think about a lot as we build Spectacles. Okay. Okay. So on that point then, so, so obviously Spectacles can test that the SQL will run and you go

through and it checks every, every kind of column individually, but so maybe talk about just very, very briefly how you leverage data tests then to do what you're just saying there. So Dylan, so how, how would you actually go about checking the actual data is logically sort of like correct as well as it actually retrieves from the database. Okay. Yeah. So there's a couple of different different ways i think we see people doing this one is a thing that people sometimes used to do in dbt and i do think for some of these tests looker is a better place for them is just saying look i know what my revenue number should have been last january or my active customers number should have been last january and i want to make sure that that's

always the case that if i change my look, such that that number retrospectively changes, which it shouldn't, that I'm notified because it means that I've done something incorrectly in my code. And so that's one of the ways that we see people doing this a lot. They decide on, let's say their five or like top five, top 10, top 20 metrics, and they will write one or multiple tests for each one, specifying an exact value that it should be equal to for some timeframe. So that is one that we see a lot. Some people did that in dbt. I think to some degree it makes sense in dbt, but if the test requires business logic of look, what is my conversion rate?

That's not typically something you would test at the database layer. That's typically something that is going to be manifested in the BI layer. And so we think that that is something that you should test it there effectively. So that's one. And then the other is, I think, a variation on that, which is that people may know what the bounds of something is in real time. So they're not looking retrospectively, looking at the current month or upcoming months, and they know, okay, look, I don't know exactly what my conversion rate is going to be this month. I'm not magic. But I know that it's always going to be greater than 0% and less than 6%.

And if something goes way out of whack, you know that it's at least worth looking into. Maybe your conversion rate has improved significantly, but at the very least, you should be notified to check what that looks like. And then the third way that we see people doing some of this in Looker, and this is, I think, particularly if they don't have as much modeling built out in their data warehouse, is some people do uniqueness and not null checks and things like that in Looker with data tests. And so some people do move that elsewhere, but we certainly see Spectacle's customers doing some of that where they are doing more of their data modeling in LookML than they are in some SQL-based tool external of Looker. Okay. Okay. So Josh, maybe talk about what's new in the new release then. What are the new features in there that you've just announced? Sure. So up to this point, we've been

very much focused on Looker and LookML. But one of the things I've been thinking about a lot and that we're recognizing at Spectacles is that a typical analytics engineer is using a few different tools and those tools don't really talk to each other. Of course, the interface is the data warehouse, but if something changes in your upstream data, that means that it has to change in your dbt model. And if it changes in your dbt model, that means it probably also needs

to change in your lookML, and so on and so forth. And so what we're trying to do is make analytics engineers' lives easier by catching some of these inconsistencies before it ends up being an actual SQL error that someone is experiencing in a dashboard. So at a high level, what the feature does is we integrate either with dbt Cloud or via our API with a kind of custom way that you might be running dbt in CI, CD.

And when you, so we'll take the dbt Cloud example to start. When you open a pull request to modify a model in dbt, dbt Cloud does the first step, which is it will build those dbt models in a temporary or a staging schema. So now we have data that reflects that change to the dbt model. And what Spectacles will do is Spectacles will, via a user attribute, and we

can kind of get into the technical details more later, but essentially Spectacles will say to Looker, hey, I want you to use production LookML, but I don't want you to point to production data. I actually want you to point to this staging data that was just generated by dbt cloud. And then Spectacles goes and runs the SQL validation and checks, does the lookML work with that staging data?

So as a hard example, if you were to change the data type of a column in your dbt model, spectacles would catch that that might result in an error in LookML before you even merge the dbt change in the first place. And so the effect of that is that you can catch downstream changes or downstream errors that might happen in Looker based on changes that you're about to make in dbt before those changes go live. Yeah, of course. So dbt cloud does uh a lot of the heavy lifting for us here so what's great about dbt cloud and this has been the way really for as long as it's existed is when you open if you're on github and i think other version control systems now but certainly github um and you open up dbt pull request and you've configured your jobs the right way what dbt will do is it will run

a series of tests uh typically including dbt runs or generating all of your models such that you can make sure that they all run and it does that into a temporary data set so it'll typically be something like dbt cloud pr and then it's a number that is your job number and then it's your a number that is the pull request number. And we know those exact things because we need to know exactly how that's structured so that we can use it. But fundamentally, dbt is going to run these tests and generate a full replica dataset, staging dataset of your data

based on the code in that branch on that pull request. And what's great about that and really unlocks a lot of the stuff that we do here is that it creates this data set that you can test for downstream processes. So we now have something that is close, at least in schema, maybe not in the entire data, but at least in schema, close to what would be in production when this code is being merged in. And so we can then run a bunch of tests off the back of that. And so what happens kind of practically is dbt cloud

writes a status update back to GitHub and says, yep, look, all the dbt tests passed. You're good to go here. Spectacle sees that we get a web hook from GitHub saying, Hey, dbt cloud just told GitHub we're good to go. Do you want to do anything about that? And we go, yes, we'd love to do something about that. We know where your schema is because you tell us what your job idea is. And we know what the pull request number is. And what we can do is go and kick off a Spectacles run. But instead of looking to your default production data set, we're now going to point it to this other data set. So dbt has built this data in dbt cloud PR 8358 underscore nine. And we tell Looker, go and update this user attribute that controls the

schema in the customer's LookML code. You tell us what that user attribute is and you need to put that in place in your LookML to leverage this if you haven't to begin with. And then we go and run all the same tests against that staging information. And that is kind of how they dovetail together. So dbt runs all of its stuff. We wait for it to give us the all clear. And then we step in and go, cool, let's check if dbt tells you whether that all works within dbt itself. And then we tell you a big explore, lots of reports and so on, you're talking about a lot of tests going on. And potentially then you've got either test everything or you just fail on the first one that goes wrong. What's the strategy that you recommend around the degree to which you test the look of content at the end of this process? Do you mean content or the SQL tests?

Well, I suppose both really. I mean, so I had my test configured, sorry, tested a couple of explores and it went and tested the content validator and so on and either took ages to run or it kind of failed quite quickly. I mean, is there any sort of strategy you recommend about how intensively to test the look of content

on how to do that really? Yeah, so there are a few things. One, the content is usually tested very quickly. What takes longer is running the queries in your warehouse. And we do a lot of little cheats to make those queries run faster, like we add a limit zero and things like that. But when you're talking about a very, very large explorer

with hundreds and hundreds of dimensions, of course, it can still take some time. So there's a few things you could do. One in Spectacles, you can actually restrict to only test certain models or explorers. So you can also exclude things as well. So if there are things that aren't really necessary, you can exclude them.

Another thing that we are thinking a lot about is this idea of not just testing the whole project, but actually testing the diff of the change that's being proposed. That's a little bit complicated. It requires us to do some things like parsing the look ML and comparing to the branch you're merging into and things like that. So it's still a work in progress on our end, but in an ideal world, we would love to only test the things that you have actually affected via your pull request. Okay. Okay. So I suppose another thing that you launched, I noticed was being able to do all of this via API or certainly an API for Spectacles. I mean,

maybe one of you talk about what the API is and does and what you envisage it for really in terms of problem it's going to solve. Yeah. So interestingly enough, when we first built Spectacles, I thought there's no reason why we would need an API, but it really just goes to show that in the modern world of software development, everything can benefit from having an API. And we've already touched on some of the use cases. I'll actually throw another one into the mix for the API, which is that some people wanted to trigger their tests in a more dynamic way. They wanted to actually kick off Spectacles tests against their production in LookML,

but they wanted to do it at the end of their ETL jobs. And if you imagine your ETL is controlled in an Airflow DAG or something like that, it makes a ton of sense to be able to, at the end of your ETL DAG, kick off a Spectacles test and make sure that new data hasn't affected anything in Looker. So we just started to run into all of these different reasons why you might want to use the API. And it became clear that we need to stand something like that up. It's fairly basic right

now. Right now you can kick off a run and you can get the results of that run back. That's about it. As far as configuration of your test suites and things like that, it still needs to be done through the Spectacles UI. But we find that's really the main thing people want to be able to do is they want to control the timing and where each run is pointed to. Okay. So is there a role for it, do you think, where certainly I've seen a lot of Looker sales deals recently that there'll be maybe a couple of environments thrown in there by Looker, maybe a test and dev environment and so on. And testing your code and deployments in that kind of a situation can get quite complicated. Is that something that you saw the API being used for?

It's definitely something we see a lot. And particularly for the larger enterprise customers that we work with, as you said, Looker increasingly, I think for a while, they were a little reluctant to give you a prod instance and a staging instance. And I think they've acknowledged that for a lot of people that is going to now be best practice. And as you said, the pipelines that test those where you, you know, maybe move content between the two instances, you promote code between the two instances, you're probably using advanced deploy on the Looker side. Those pipelines can get very involved. You're going to want to maybe trigger spectacles against multiple different instances, against multiple different projects kind of in sequence with some conditional logic in between. That is certainly one of the use cases that we see for the API and we have some of our larger customers using. I think an extension of that and just becomes part

of these larger, more involved pipelines is people who are auto-generating their LookML. So maybe they've got something where they're using PyLookML in one of the other packages, or they've just written it themselves, where they're doing some auto-gen LookML and want to be able to push that automatically. So maybe a dbt run finishes, they auto-generate some LookML based on the artifacts that are produced by dbt. They push that to a Looker branch, and then they want to test it right away and be able to push it to production right away if tests pass. That is another one where we've been working more and more with customers to support them in those use cases and see exactly how that works. And those two,

I think, are certainly use cases, as Josh said, where when we started this project, certainly as an open source tool, we didn't imagine would be the things that people do with this. And I think certainly some of that is just Looker has evolved over the past two years. But as those deployments become more mature, there are larger organizations using Looker in much, much more complex and sophisticated ways. We're finding that that API just empowers them to do all the stuff they want. And we get the, I'm sure they're happiest saying this, we get the feedback from, you know, customer Looker sales engineers that say, you know, there are some of these enterprise deals we could not have closed without Spectacles because these more sophisticated users are coming

to them and wanting to do this stuff. And, you know, not every time, but a lot of the time, Spectacles, as well as other tools, you know, LAMS and LDeploy and other things that are built in-house at Looker are, you know, together make up this really nice mature stack for these sophisticated data users to use on top of Looker. That's interesting. And actually that's, I mean, that is a topic of interest to me at the moment in that we, so when we develop sort of solutions for customers, we build them in dbt and Looker them and build them as a, we build them as a, a single project. Um, and we also have a, a, a look at look at our auto generator that generates code. I mean, what, what are you seeing out? I suppose, what are you seeing out there around, you know, beyond what you said there around people developing these stacks and

particularly working with dbt and look at together, you mentioned a couple of projects there as well. What, what state of the art moment do you think really around this development process? I was just going to plus one on the auto-generated LookML piece. I'm definitely seeing more and more of that. And it makes a lot of sense because there's a lot of admin work that goes around, oh, schema change in the database, oh, update the LookML, oh, schema change in the database, update the LookML. And so to the extent that you have the ability to write a pipeline that can make those changes to LookML or generate new LookML when you have new data sources, I definitely think that is becoming more common.

I've seen projects around, oh, we'll convert your DBT model into LookML and things like that. I think that's only going to become more popular in the larger companies that have the capability to connect the dots with the transformation layer and LookML. I think they will get more and more integrated. I think one of the things that I would add is for a long time, I think refinements in particular, which were added to LookML, I guess, a year ago, maybe longer, unlocked a lot of this. Because I think one of the problems for a long time was, and extension solved some of this, was with the auto-generated LookML, you can't, it's hard to update something where some user has had some input to the code. Anything that

is auto-generated historically, you need it to kind of leave because if a user updated it, the next time that the auto-generation ran, it would overwrite it. And I think there's been two changes there. The first is kind of refinements and extensions to some degree, such that it's easier now to have these files that are untouched by users that are just auto-generated. And then you build a ton of stuff on top of that.

So I think that's one thing that has unlocked this a little. I think the other, and Josh may not have said it himself, but I will big him up in this area. Josh has written like a truly fantastic looker parser in Python that allows you to update instead of just overwriting, actually parse it and update the records in place. And I think that is a thing that we are certainly going to be building stuff on top of or with at Spectacles,

but I think is a big game changer in how some of that works. And so I think one of the things that we've talked about for a long time, and I know we've seen kind of like come up in the dbt and Looker communities every probably quarter, is how do I push my dbt documentation to Looker? And I certainly thought that a year ago, pre the parser and pre a couple other things, that that was a pretty big problem, just like, or not a big problem. It was certainly a big one, but also a very hard problem. And I think it is a big problem, but less hard to do now. And we have not built a solution yet for that, but it's certainly one of the many ideas that we think about a lot day to day.

Yeah. Interesting. So we use refinements in our projects and we, we, we also generate what we call a base layer, which becomes the thing that we can then ship out. We can ship out then updates to that over time to the customer with the customer, then customizing the layers above that. So it allows them to, to customize stuff and further, further refine it as such.

But actually, you know, we can push out changes to the underlying base kind of definitions of tables. So you mentioned there about, about parsing the SQL parsing, the look ML and the minute plating that, that sounds an interesting way that you can probably do the last thing i'm interested in you doing which is to fix the problems for me in the first place so you know

i go and i run i run spectacles it tells me all these problems there and then i have to go and fix them could you imagine i suppose is it in your thought process that potentially um you know spectacles might be able to actually at least offer suggestions to how to correct some of the common issues and common errors you find? Yeah, I mean, that would be awesome. To some extent, we are a little bit limited by what comes back from the data warehouse. So if you think about the process, we would essentially need to read the error that comes from BigQuery or Snowflake or wherever and be able to figure out what to do with that. It's possible that, at least on those three warehouses,

we could just maintain a direct mapping of, oh, the error message looks like this. And sometimes I think some warehouses will even make the suggestion for you, oh, you said this. Did you mean this? And perhaps we could pull that directly out. It's certainly possible. I think the question is just, to what extent can we interpret those error messages and

know what to do with them? When I think there's two bits to it, which is we could certainly, I think an easier solution, which is probably more likely to be on our roadmap in the short term, is at the very least parsing the error and giving a suggestion just in text in Spectacles and saying, look, when it's this type of error, typically you're going to want to look for this type of thing is probably an order of magnitude easier than making a suggestion, but also like doing that in the code. So that's, I think there's like two steps. One is what is the fix? And the two is, all right, are we able to programmatically build that fix? And I think frankly, the first one for

a lot of people would be a big, big help in just helping them figure this out. But one of the other things that we've talked about a lot, and I think will be something that you'll see from us later this year, is a lot of... I think one of the issues with BI generally is this kind of purely additive process. You add new views, you add new explorers, you add new constant. There's rarely this hoovering up process underneath. And I use the word hoover because it's kind of what something like Henry, which is under this open source tool on top of Looker does.

But what Henry does is it tells you, look, here's this explorer that could be deleted because it hasn't been used. Here's this join that could be deleted because it hasn't been used. But you still then have to go in and do that. And typically, you know, you find an intern and they go in and, you know, take a week

and do that work. What's really cool about the parser is that what we've gotten actually is like an MVP of internally right now, which we're playing with, is the ability to open a pull request that just deletes an explorer for you and tells you whether there's any problems with it. Because we can do some tests around the LookML validator through the API and all of our normal spectacles tests, we imagine a world where instead of someone having to run a report every month can do some tests around the LookML validator through the API and all of our normal Spectacles tests, we imagine a world where instead of someone having to run a report every month and seeing what explorers haven't been used and speaking to the stakeholders and figuring out, you know, can we remove them and then running a bunch of manual tests, we just, we run it every

day. We open a pull request for you when we think that there's a view, a join, an explorer that isn't needed anymore. And it just kind of, you review the PR and you say, yeah, you know what, you're right. That isn't needed anymore. Merge it in and we're done. And I think we want to make that cleaning up process easier for people so that they can do all the kind of moving forward stuff, which is really what you want to be focused on. You don't want to be doing this admin bit. Interesting. So one last thing I want to mention was I've seen a trend, I suppose, recently with some kind of tools like High Touch, for example, to start in a way using the dbt Git repo to store their own kind of metadata and their own kind of definitions of projects. Is that something that you kind of see going on elsewhere?

And is that something, can you maybe imagine the version of Spectacles that stored its configuration inside the dbt repo? I mean, is that something you'd be thinking about? It's not something we're thinking about right now. I think it makes sense. And I'm always trying to think, what's the parallel with software engineering? And when you think about Python development, it's quite common to see a lot of metadata files stored in your Python project. You've got your setup files.

Often your CI config is in there. So I think them deciding to put the config in dbt totally makes sense. I think for our purposes, because we're targeting looker analysts and often looker analysts who like Spectacles or looker analysts who don't have a ton of experience with things like CI and config files and repos, We're trying to keep things very easy to use. And so just being able to click over into the Spectacles app and make changes is definitely what we're optimizing toward. I would say, though, it is certainly a thing that we're seeing more.

Like, you know, Fivetran have released their dbt scheduler. All of that is defined in a YAML file. I know a lot of people are using tools like Permafrost to set their configurations in Snowflake. A lot of that is config as code. Not a direct analogy, but SQL Fluff, the linter, certainly has all of its config. Granted, it's pure open source, so there's a reason for this, but it has all of its configuration as code. So I think there is certainly, as analytics engineering as a domain becomes larger, as Josh said, it's not something that we are thinking about that much today, but it is certainly a thing that we are seeing generally across the board in

the space. Okay. Okay. And one last thing that I suppose you mentioned SQL fluff. Can we imagine, can we maybe see SQL linting sort of testing for that being, being spectacles at some point? Yeah,

I think linting is something I think we still on our open source repo have a, I think linting is something, I think we still on our open source repo have a line that says linting is coming soon. And I think whether that's true or not is just a question of judgment as to how soon you think soon is. But there is certainly linting is something we think about a lot. And I think there's two bits, as you said, Mark, there is the linting the SQL, if you've written a derived table and making sure the SQL is written correctly. And then there's linting all of the kind of standard look ML and making sure that that conforms to some standard.

And there is certainly a world where we do one, the other, or both. And I do think SQL fluff, if we were to do the SQL bit, would certainly be one of the tools we consider using heavily. I've used it a ton recently and really enjoy it

as a framework. Fantastic. Fantastic. Well, how do you, okay, so how would people get hold of maybe the open source version or the version that's hosted? How do they get hold of Spectacles and get to kick the tires as you say? Yeah. So the great thing about an open source CLI is you could go and install it. It's a Python package. You could set it up with a demo and a free trial for a couple of weeks so you can walk through and really try it in detail and make sure that it's the right thing. Okay, fantastic. So it's been great having you both on the show. Thank you very much for coming on. And yeah, great to speak to you and good luck with the product. Thanks, Mark. Thank you very much. you