The Data Stack Show - 197: Deep Dive: How to Build AI Features and Why it is So Dang Hard with Barry McCardle of Hex

Episode Date: July 10, 2024

Highlights from this week’s conversation include:Overview of Hex and its Purpose (0:51)Discussion on AI and Data Collaboration (1:42)Product Updates in Hex (2:14)Challenges of Building AI Features (13:29)Magic Features and AI Context (15:22)Chatbots and UI (17:31)Benchmarking AI Models (19:06)AI as a Judge Pattern (23:32)Challenges in AI Development (25:31)AI in Production and Product Integration (28:43)Difficulties in AI Feature Prediction (33:38)Deterministic template selection and AI model uncertainty (36:21)Infrastructure for AI experimentation and evaluation (40:11)Consolidation and competition in the data stack industry (42:27)Data gravity, integration, and market dynamics (47:12)Enterprise adoption and the bundling and unbundling of platforms (51:03)The open source databases and the middle ground (53:18)Building successful open source businesses (57:00)The fun approach to product launch video (1:01:14)Final thoughts and takeaways (01:03:15)The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

Hi, I'm Eric Dotz. And I'm John Wessel. Welcome to the Data Stack Show. The Data Stack Show is a podcast where we talk about the technical, business, and human challenges involved in data work. Join our casual conversations with innovators and data professionals to learn about new data technologies and how data teams are run at top companies. With one of my favorite guests we've had on, Barry McArdle from Hex. It has been a while, Barry, since you've been on the show. How long ago were you on?

I don't know, a couple years ago, maybe? Yeah, that's crazy. It's been way too long. Well, for those who didn't catch the first episode, give us a quick overview of who you are and what Hex is. Yeah. Well, thanks for having me back on. Hex is a collaborative platform for data science and analytics. We built it largely just to, it's an incredibly selfish company, really. It's just, we built the product we always wish we had. I've been a builder and user of data tools my whole career and longest stint was at palantir where i met my co-founders and a bunch of our team and got to sort of you know just built building a lot of different data solutions for a lot of different data problems and you know hex

is built to be this product you we kind of call it this multimodal product that is able to bring together teams and workflows and sort of unite them in one place around being able to work with data in a much more flexible fast and integrated way than the tools that we had sort of struggled with before we can go deeper into that but that's the high level all right so one of the topics i'm excited about talking about is AI and BI together. So let's make sure we get to that topic. What other topics do you want to cover? We talk about a ton of things. I think this is a very interesting time in the data stack as we're recording this, the Snowflake and Databricks conferences for just over

the last few weeks. So it was very interesting just being there and sort of good chance to check in on where data is going. I think the AI topic is super interesting and rich. Tons we could cover. Yeah. Yeah. Awesome. Well, let's do it. Yeah. Sounds good. Okay, Barry, there's, I mean, a million things I want to cover, but can you give me, let's just say, I didn't even look this up, shame on me. I didn't even look at when we recorded our last episode, but let's just say, I didn't even look this up, shame on me. I didn't even look at when we recorded our last episode, but let's just say it was a year and a half ago, which is probably directionally accurate.

What are the biggest product updates in Hex in that time period? Can you go rapid fire? Wow. You guys ship so often that was a very unfair question. At least give it a chance to pull up the release notes.

No, it's good. I mean, I've got it all in my head somewhere in there. Yeah, look, I think longitudinally, we started very deliberately wanting to build a fast, flexible product for a data work that sort of filled this gap that we felt was really around, you know, you go into most companies to see a data stack. Most companies at a certain size will have, you know, a sort of BI tool that's very dashboard

centric. Yep. And that's all well and good. But I think every data team I've been on or been sort of exposed to, it's like 80, 90% of the data work was happening outside of that. You know, it's in some menagerie of like random SQL snippets and Python notebooks and spreadsheets and screenshots of charts and PDFs of decks sent in emails is like the way to communicate. Yeah. And I just felt that pain so viscerally. I sort of built Hex to help. Yeah. And I just felt that pain so viscerally. I sort of built Hex

to help solve that. But the long-term vision was, you know, increasingly unify and sort of integrate a lot of the workflows. The way we've already talked about internally

is we want to build that sort of front end of the data stack where you don't have to jump around between tools. And both as an individual, you can bring workflows together or your workflow

can be more sensible because it's together. It makes it easier to collaborate as a team and then expose this to more people in the org. So we talked a year and a half ago. I think we were pretty far down that path, you know, just in terms of maybe the first era of that. Yep. I think people thought of Hex and maybe still largely do think of Hex as like a notebook product, you know, almost like a super flexible, fast product for that. I think as we've expanded, we've grown, there's a bunch of things. So about a year and a half ago, we introduced our AI assist features.

I think we were pretty early to that. We'll dig in a ton on sort of how we've done that and where we think that's going, but that's been awesome. They're called our magic features. And that was a huge thing. We'll go deeper on that. We've built a whole suite of no-code tools in Hex. I mentioned this word earlier, multimodal. It's kind of a buzzword in the AI space right now. But it really just means like being able to mix these different modalities together.

So, you know, in Hex, for a long time, you've been able to mix like Python and SQL. Like that was actually like, it maybe sounds prosaic, but it's actually, it's like pretty profound in terms of a lot of people's photos yeah definitely first people to be able to bring that together really fluidly but since then we've integrated just one more so we have a whole suite of no code cells like charts pivots filters write back cells and now you know a few weeks ago we launched a bunch of stuff that's effectively being like spreadsheet functions into our table cells. So you can sort of work. You can actually do a full end-to-end workflow in Hex now, fully no code.

We built a bunch of tools for governance, whether it's data governance or sort of reviews, like Git style reviews on projects, endorsements. I don't know. It's a long list. Yeah. reviews on projects, endorsements. I don't know. It's a long list. But effectively, the focus has been how do we expand the number of people who can participate in these workflows while continuing to make the product more and more powerful for sort of that core data team that's really at the center of the decisions companies are making. Yep. Love it. Yeah, I actually didn't, I had this thought when you were saying screenshots of screen passive dashboards and I was like how often is Slack

literally a screenshot in Slack like it's like someone's main source of data is literally just it's one of those things where actually I wonder I think majority of data is probably consumed via static screenshots

yeah if you just back up and you think about like charts index charts and slack yeah it's gotta be totally it's you know in some ways that's a very old problem in some ways yeah yeah so like eminently unsolved yeah yeah totally but i think there's actually a reason behind that i think part of the reason is like let's say it's for an executive they want like a signature behind it right they want like this is certified correct per x analyst as of this date you know like they want some kind of like yeah yeah that's i think it's actually a really interesting segue into some of the ai stuff because i think that you know you just said they want like a signature

from an analyst on it right and it's like I think it's something I've thought a lot about with how AI is going to show up. I think that really reductive, like, oh my gosh, I just saw GPT for the first time and I saw that it can write SQL. It's going to be like an AI data analyst. There's a bunch of reasons that is not going to happen or it's not going to happen overnight.

But my favorite sort of weird reason is like culpability. Yeah. I live in San Francisco and there's self-driving cars like Waymos all over the place. And I take them. I love them. But I think it's very interesting to me that they still live in this like regulatory purgatory. Yeah. Which is like every day all across America, like legions of eager distractible 16 year olds get driver's

licenses yeah and like self-driving cars it's like a multi-year slog to get them like approved even though like right any objective observer would look and be like a waymo is less likely to hit and kill someone than like a 16 year old right right but like i think there's i think it's like a societal justice thing actually that's like we know what to do with a 16 year old who like hits and kills someone. And it's like some version of like our justice system is set up around that. Yeah. It's like, what do you do when a self-driving car inevitably just like, you know, they drive enough miles, there's going to be accidents.

What do you do with a self-driving car? Totally. Do you put it in like robot prison? Yeah. And I bring it back to data like i i think about this with like ai like data analysts say ai data scientists there's there's certainly a lot of companies that market themselves that way it's like if i'm like hey should we increase prices and i'm gonna do an analysis on that and i ask the ai to go analyze a bunch of

data on that and i get charts like my my, who's behind, who's standing behind that analysis? If the price increase doesn't work or whatever the decision is, like, you know, I think with a human analyst

or a human data scientist, like someone's credibility is on the line and there's like a reporting chain of people whose credibility is like on the line. Right, totally.

And with an AI, like what do you do? You like fire the bot? I guess, you know, you'd like churn off whatever AI software you were using. It's just a funny thing. And I think our society is actually going to have to learn how to develop this tolerance or way to handle the defect rate

or the inevitable stochasticity of these AI systems in a way we're just not well-equipped to do today. So AI culpability in the context of like data and analytics. If you have an end user who's like engaging with data, who is not an analyst, right? They don't know SQL. They don't know Python. Mythical business user. Exactly.

Yes. Yes. And this mythical business user in this like mythical company where everyone uses data to drive, you know, every decision that they're making. But do you see that creating a lot of extra work for data teams at companies where, you know, you sort of implement, because I mean, there's, and maybe it's not even, maybe these things aren't even like fully real, right? As much as the marketing would lead you to believe. Like, believe. You're saying the marketing is selling something out of... Startup marketing is selling capabilities? Because I went on this website and it says

there's an AI data scientist. Are you telling me there's no... Surely you jest. Is that going to create more work for data teams? Or is it like... I don't know. Does that make sense as a question? Yeah, it does. Especially for for critical stuff if it's fuzzy who cares right like it's directionally accurate but hallucination can be catastrophic in certain well i i think i just there's a few things in some ways this is a really new topic with ai in some ways it's like a very old topic of like self-serve generally as a pattern in sort of like the analytics market is like okay if you set a bunch of people loose there's going to be a defect right like you know it's going to

be like well you didn't use that metric correctly and that's where people talk about semantic layers and governance and we've certainly done a bunch there is more we're going to do but it's kind of the same i think as it boils down effectively the same problem with AI, which is like AI will also sometimes misuse the data. It's like, how much context can you give it? And, you know, whether it's semantic layers or like data catalogs or whatever, like the kind of pitch of those products like pre-AI is like, well, this is a place where you can put a bunch of context and guardrails around how humans can use this data. I think it's effectively the same pitch for with ai is like how you you do that and like it's why we built a bunch of those tools into our magic ai features with hex which is like you can incorporate like automatically sync your dbt docs or your column descriptions from

snowflake or bigquery or you can enrich it within hex itself to give context to the model now does that mean it's 100% accurate? No. When we do our own evals, our own experimentations, our own internal stuff, it does dramatically increase the level of accuracy in it being able to use that. Is it perfect? No. And so the question you're asking is like,

when those defects happen, does that work on the data team? I think there's maybe some version of like equilibrium, of like you're saving them a bunch of time answering random QQs and then they're having to do that. The way I see it is, I think the best version of self-serve of any type, whether it's augmented or not, which I really think is in some ways very profound and in other ways pretty incremental, is you're taking a lot of... The best version of this is you're not replacing the data team. You're taking a bunch of the best version of this is you're not replacing the data team you're taking a bunch of the sort of like low level stuff out of their queue so they can focus on the really intensive deep dive stuff and in many ways you know that self-serving of course i'm ceo of a company that

builds a product i want to sell to people but like that's really what we built hux to do which is like the deep dive work we want to be the best way for data teams to get to the complex answers. And I think it's an interesting question of like, does self-serve AI augmented or not give them more time for that? Or does it just create more work trying to govern all of that? And I actually think there's no one right answer. I think it comes down to a lot of like the way team structure stuff, the style they want, what their level of comfort is on people sort of coming up with their own answers. Different people have different approaches. I don't think there's one right or wrong. Okay, I have a...

John, I know you have a ton of questions, but can we get down and dirty in talking about how you built your magic features and maybe other stuff that you built? Because John and I have actually prototyped a bunch of different LLM applications, AI apps, if you want to call it. And it's just way more difficult than I think, than the startup marketing.

We talked before the show, there's this typical like, hey, I got this working in development, now let's productionize it. There's that typical workflow. And with AI, I'd say it's order of magnitudes more between like, hey, this kind of works versus productionizing. Yeah. And more context is, this morning, I knew we had a podcast recording. I forgot that it was with you. But coincidentally, I literally pulled up He up hex and i was just doing some experimentation on our own rudder sack data uh with the intention of using magic and playing around with it and it was great like it was glad to hear that yeah i had a target table i had like a set of questions

and i was like okay i'm gonna go like i'm gonna go fight with this AI thing to see, you know, to like man versus machine. And you won. Or it won. I don't know. It was great. It was great. I mean, down to like the fifth decimal point on one thing. I was like, this is awesome.

Awesome. I'm really glad to hear that. Makes me shame. It was very cool. But I also know that that is non-trivial from a product. Like it's so, it is in fact, like, I don't mean, we haven't talked much about this, John, but like when you do that, it's so simple that it, the, it's almost an unhealthy level of obfuscation on what's actually happening under the hood to make it good, which is part

of the, you know, magic, I guess, not to be too cheesy. Yeah, that's right. Yeah, I mean, well, again, thanks for saying that. We've put a lot of hard work into that. So, yeah, we should have a very practical conversation about this because I think that the big thing over the last couple of years is these models have been working. They've been working really well on doing a bunch of things. And's easy to look at them and it's easy to set up a

quick demo with open ai api but like actually building a production ai system is a much different thing it's turning out that it's really well suited to certain tasks and not others i mean there's a bunch of stuff here i think that the pattern we're using under the hood is something i assume people a lot of people are probably familiar with, which is retrieval augmented generation. As far as we can tell, 98%, some really high number of AI apps today are using RAG, as opposed to going and spending a bunch of time on fine tuning. Right. And under the hood, we're using base models from OpenAI and Anthropic and kind of built it to be pretty modular models wise. And the magic really is in the context you're providing it.

So it's one of those things like right place, right time. And I didn't have like Gen AI on my bingo card when we started the product. It turns out the product format of Hex is actually pretty sweet for this, which is it's that sort of notebook UI, the core UI that we built Hex around. We're kind of big believers in that. We don't think of the product just as a notebook, but the UI pattern works really well. And it works really well for AI, which is that you kind of have a bunch of upstream context. I mean, you can just start de novo using AI to generate stuff, and that's great. And the context we're going to use there is context we have about your schemas,

you know, column names and sure, but also we can go and look at things like all the column descriptions. We can increasingly, we're tapping into other queries you've written and trying to pull up like more on text based on your actual workspace. But, you know, as you're actually working through a project,

it's a tremendous opportunity for us to sort of pull in all of that upstream context. If you've been querying into a certain table already or you've been using certain columns or you've been joining things in a certain way, that's all context we can provide the model. And there's some really nice UI things we've done as well, like being able. And this is the other hard part about AI is like getting the UI right. I think there's a lot of we're in this very chatbot centric phase of UI design right now or like the default way to build an AI product is like basically copy paste chat GPT

yeah I'm a believer and pretty insistent that is not going to be like all of SAS is not just going to converge to chat bots and pressing in terms of like the UI Google had it right with their just single field

it's all back to that right yeah but even yeah I don't know in terms of the UI. Google had it right with their just single field. It's all back to that, right? Yeah. You guys caught the Apple intelligence stuff. I think it's actually really interesting that Apple's done a ton of Gen AI work and none of it is chat. It's like the other thoughtful ways to incorporate this into your product.

Anyway, even things like being able to at mention tables or data frames, there's a bunch of stuff that I think is like helps give context and also helps give confidence to the user of like, I can instruct this thing. Right. I think one of the interesting things designing the UX for AI is it's not just like an intuitive UX. There's a really subtle thing of helping the user form a world model of what the AI knows. And this sounds so anthropomorphized.

Yeah. But I think when you're using an AI product, a lot of times you are kind of in the back of your mind kind of thinking like, what instruction can I give this? What is it capable of? Right. What does it know about? And I think being able to expose more of that, you mentioned obfuscating it, Eric. I agree. I actually think we want to expose more of that to users to help fill out that world model of like, what does this model know? What can it do? What can't it do? How might you want to modify your instruction to get the type of response you want? That stuff's all really hard

from a prompt perspective. It's also, I think, really tough to get right or a lot of hard work to get right from a UX perspective. So I have a question. How do you benchmark? So all these models are changing all the time. So say that you're like, all right, we want to use the latest, you know, Claude model. How do you benchmark between models? I feel like that's a pretty difficult problem. That's, it's an incredibly difficult problem. And it's actually, I'm glad you brought the exact example up because we're literally doing that now we're testing out the newest cloud model yeah 3.5 or whatever yeah 3.5 summit i think it's but you know you read the like announcements of the blog posts about these models they all sound like they're gonna you know god's gift to me you know it's right You know, the benchmarks look great and all that stuff.

You have to benchmark it yourself. And this is a term that's called evals, which is kind of just a very specialized form of tests. And we've built a pretty extensive eval harness. So there's open source stuff. So there's like the spider SQL benchmarks, which is sort of an open source set of benchmarks around text to SQL. And then there's a lot of our own evals we've written as well. And for us, our magic tools don't just generate SQL, they generate Python code, they generate charts, they generate chains of cells. You ask a certain question, you want to get a SQL query and then a chart after that. And so we have evals built for a pretty

broad set of this. We've had a lot of internal mechanical turkeying of building these eval sets out. And what it lets us do is quickly run experiments in our experimentation framework we've built internally called Spellgrounds, which is basically we can very quickly say, okay, great, I want to test this new model out, point Spellgrounds at that model, have it run a sample or all of the evals, and then get back a result. So we actually see based on different types of generation tasks, whether it's SQL generation, Python generation, chart generation, chain generation, text generation, whatever the task at hand is, retrieval stuff, how good is it at those? And what's really interesting, even upgrading from

GPT-4 Turbo to 4, you actually saw certain things, it was much faster, but you also saw tasks where it got better, you also see tasks where it got worse. Then you start thinking, like, okay, do we have to do some prompt tuning for this? And you can get down these permutation loops of

like, okay, wow, it got worse at ChartGen. Is there something about the model to do that and it's just taking a step back i mean as somebody's been building software for a while this all is so nuts like it is such a mild and primitive way to program i think we're like increasingly used to it here i am on a podcast like talking with authority about how we're doing evals and prompt and you actually look at what you're doing when you're looking at this shit. And it's like, okay, I'm yelling at the model this way. And I need to yell at it. And I need to add another line that says you will not respond with markdown. You're like talking at it. Like it's a child. You will not include Python in your response when I ask for SQL. Like, yeah, it's very threatening. Yeah.

I'm so instructed. It's like you're like, it's that scene from Zoolander where he's like getting brainwashed to go like kill the prime minister of Malaysia. It's like, you know, you're like brainwashing these models. You're like, yes, you're a world-class analyst. Totally. It's like you write CTEs for your queries yeah yeah it's weird man and i you know here we are building a bunch of like expertise and infrastructure and tradecraft on

how to do this but i can't i can't avoid the feeling of like we're gonna look back on this whole era of using ai and being like wow that, that was weird. Yeah, the point where I had a very similar, you know, you sort of step back and try to objectively look at what you're doing was when, do you remember we were working on a content generation project or something? And John was hammering on this prompt, hammering on this prompt.

And I was like, okay, this is crazy. And then it became like a multi uh multiple steps of prompts to generate prompts which was i was like this is like that was the moment for me where it was like okay this is like you're using an llm to generate a prompt oh yeah you know or to create a persona to create a persona that can then generate like a better prompt. We go even crazier. I'll take you another level of sort of craziness on this.

Like meta. We have adopted a strategy. We'll publish a blog post about it soon for one of our AI operations around editing, which is an AI as a judge pattern, which is basically you have the LLM generate which is basically you have the lm generate

something and then you have the hello another model call look at that response and judge whether it's an accurate totally what like tree what is that like tree of thought is that what that's called i don't know there's something which is also is different but it is also a really interesting and weird thing which is if you tell the model, like, think step by step, you get better reasoning abilities. And it's like, why? It's very spooky. There's technical explanations for it. I listened to this talk by one of the guys who co-authored that paper, the original paper about that. And there's people who have theories on why this works, which is like by telling it that

it will actually spend more tokens, which basically makes it like think more. And you're kind of forcing it to break things down. And it's not that it actually has like reasoning per se, but it's by like forcing it to spend more tokens on the task.

You're basically making it think more. And like, then that opens up this whole thing around different model architectures that beyond transformers, which are undergirding what we think of as large language models today which are basically spend the same amount of thinking reasoning compute whatever you however you want to frame it across the whole generation versus other model architectures that are sort of still in the R&D phase

that can more selectively apply that. And so then you can think about, are there patterns down the road where you could even, as part of the prompt, tell it what parts to think carefully about or be able to steer its reasoning more. And it kind of gets in this very weird hall of mirrors, but we use this AI as a judge thing, which is separate, which is almost like calling the model in again to look at the work yeah of

another model car yeah sure yeah and it gets really weird and then it's like what what if the judge doesn't like it it sends it back to do another generation right yeah along with comments from the judge about what it didn't like and it's like we're kind of facilitating a conversation between like two total yeah calls it's like we're kind of facilitating a conversation between like two protocols. It's like, wait, what are we doing? You take a step back. Totally. For the entire history of software engineering, you're basically used to these

very deterministic things. Totally. In fact, any non-determinism is like a bug. It's like, okay, I'm going to write this test. Yeah. It is a unit test when this function is called with these arguments it should respond with these responses yeah every time and if it doesn't do that something's broken and now it's like oh you know yeah sometimes it dreams up new things

yeah right i had a moment like that this isn't related to ai but very similar feeling when i realized like like we're you know we're we have these models talking to AI, but very similar feeling when I realized we have these models talking to each other. But I was taking my son somewhere, doctor's appointment or something. We get in the car. My phone automatically connects to Bluetooth. And I don't know whatever setting, but it just automatically starts playing whatever audio I had been playing.

And so, of course, that morning at the gym, I had been listening to gong, you know, sales calls on gong at on 1.75. And so it starts playing and these, you know, it's like rapid fire discussion. Yeah. And my son was like, are you on a call, daddy? And I was like, oh, no, like, you know, and I pause it and he's like, oh, then what was that? And I was like, oh, no. Like, you know, and I pause it and he's like, oh, then what was that? And I was like, I was listening to a recording of other people that daddy works with on a call. And he was like, I mean, you just pause for a second. And then he was like, do they talk that fast in real life? And I was like, no, I speed it up so that I can listen to like a lot of calls in a row. And so he was like, you are listening to other people that you work with

on a call that you weren't on really fast. And I'm thinking, I was just like, yeah. And he's like, that is so weird. Yes, son. That's product marketing. That's how I know you're a product marketer because you're listening to sped up gong calls at the gym yeah 10x 10x product marketer or 1.7 1.75 1.75 and make him a 10x product marketer that's the key right there i don't think the same thing

where like you step back and like just like with an objective view of like looking what you're doing you're like this is weird when you say it in English. Yeah, when you think about that, it's weird. Yeah, I mean, I agree. And I think going back to the topic of like getting things into prod, it's like, I think we learned really fast after chat GPT that it's not hard to get cool AI demos working. Like you had like a couple of YC batches in a row that were basically like reasonably set up.

Cool AI demos around GPT. It's much harder. And I think what's interesting to me is to look at some of these apps as they're growing up. And I'm fortunate to know the founders of a bunch of these both within data, but outside as well. Where is like an AI first thing, you know, where you're going to say, great, we're going to build this just a wrapper around GPT and then evolve the thing versus not?

There's a bunch of these now that are AI support companies, right? Not within data, just customer support. Yeah, yeah. You get into it and you're like, yep, there's clearly a huge opportunity to change the way customer support is done. But a lot of these companies may well have to rebuild a lot of what like the last generation of support companies did like it's not clear to me that these startups won't have to effectively rebuild intercom or yeah right yep and you know

it's a question of like can the intercoms and zendesks get good at this faster than the startups can get totally bad and that you know we can talk about that in the data space too. But what I've really come to learn and appreciate is how much, how hard it is to get this AI stuff working in prod and how much it is dependent on having the rest of the product. Because we could not get the quality of what we get with Magic without having a lot of the other infrastructure and product we built. Not to mention millions and millions of lines of code. And I had to go look at hundreds of thousands of projects that people have built over the years. We're not trained, just to be very clear, we're not training these.

There's no IP leakage issue. But even just having the product and the context that organizations have built up over time is really the thing that makes this work well. So there's a bunch of AI-specific tradecraft, but then it's like, how are you incorporating in the rest of the platform is really important. What have you tried that didn't work? You know, it's just like, okay, we have to stop going down this road of productizing. A bunch of things.. A bunch of things. And a bunch of things.

And I think one of the interesting things on those moments is you step back and you say, does this not work because of the models? And like maybe GPT-5 or 6 will be better at this. Does it not work because of we're not, don't have the prompt, you know, just right. Maybe one amazing prompt away from cracking this not hearing the judge yeah is there you need an ai jury too i don't know yeah yeah that's it yeah i execution are you know shut down um i have vigilantes come about justice system yes bounty hunters story. Justice system. Bounty hunters.

Give it the context of literally an entire justice system. Yeah. Okay, we'll pull out that one. But do we not have the infrastructure ready? There's things we've tried where it's like, okay, we actually need to go build up some expertise or infrastructure

and being able to do a different type of retrieval to have this work so like as an example our mad there's a feature we have in magic that lets you do generate basically chains of cells and we were the first version of it that we've sort of beta tested let just told the model just generate any number of cells you think necessary to complete this task. And it would go and do that, but you would have this drift down the chain of cells. So just conceptually, you can imagine like asking that question and having a general, like a SQL cell that retrieves data and then having to do Python cell that operates on that data. And then like a SQL cell that reshapes that data using SQL. And then like a chart cell that visualizes that. And you'd see it make some really weird decisions.

And we were getting down this like path of like prompt engineering at being like, oh no, like don't make a bad decision. Like think harder about it. And like it would get better at certain things and worse at others. And eventually we fell back to like, okay, actually the pattern,

the predominant thing is like actually one of a few templates. And we're actually going to like almost like hard code those templates and have the AI select amongst those templates. Yep. Now, when you look at that,

when I think about that, I go, ah, you know, the template thing feels brittle. It feels limited. And obviously to me, the longterm should be unconstrained shade generation.

Like that, that is, I think like a almost obvious thing that should work. Yep. The question we grapple with is like, you know, maybe Cloud 3.5 will be good at it. Maybe GPT-5 will be good at it. Maybe we just weren't doing enough. And even since we worked on that, there's a bunch of new techniques people talk about,

like self-reflection. We could pull in our AI as a judge strategy. And we think that there's some ways that could work better. And so it's really hard to know. And again, like I think we've been, my team and I have been building software, data software for a long time. We can stare at most features and be like,

few sprints, you know, a quarter, whatever. You might get the estimate pretty quick. I've got like, you know, just enough data points. I can be an RFC and if an engineer is going to be, is like, this is going to take six months. I'll be like, no, it's not. No, it's not. These AI features, there's things that we've looked at

and like, that's going to be hard and it basically works overnight. And there's things that we've looked at and be like, that should totally work. And it's like, this might be theoretically impossible. Like, it's difficult to know. Another example is diff-based editing. So like when you ask to edit in Hex today,

like if you have a SQL query and you ask to edit it, we will go out and generate, basically pass the current query or the code block or whatever with your edit instruction and some other context to the model and say, hey, edit this code block to reflect this. Great, we'll get a new block back.

But we stream back effectively the whole new generation. We show it to you side by side. Now, that feels slow because if you've got like a hundred line Python code or query, you know, we're streaming the whole thing back. So it's like, well, can we ask it just for the diffs? That all at once is something that I think anyone who's spent any time with Alams can like very easily imagine like, okay, yeah, we'd send just the lines you want edited and actually getting it to respond with the right lines and having a certain right place is really hard and it's like okay but we almost gave up on that

we actually did for a bit and we came back to it with a new technique and then it worked and so it's really hard to know what is actually going to crack these things open. And the space is moving so quick. And it's not like the models even just get monotonically better at things. As I mentioned with GPT-4.0, it actually got worse at some things. And it's like, where are the lines where you're betting that you're just going to get this natural tailwind and rising tide from the models getting better? And you can just kind of skate to where the puck is going and it'll catch up versus like the other hard work you have to do. That is, I think the,

what makes building products in this space like really tough today. I don't think people are talking about that enough. Yeah. Maybe sometimes I wonder, is everyone else having a much easier time with this? Yeah. The answer is no. John and I.

Yeah. Back to the template thing. I think there's an interesting concept here. You were mentioning like, Oh, ideally like it can just produce all these cells and it will just work well but i think there's kind of an interesting interface thing here where it's like okay is there something trivial i can ask of the human that drastically helps like on the ai so maybe the human is like setting context for

the cell is like sql or python like that's super trivial but then with that context the cell is like SQL or Python. That's super trivial. But then with that context, the AI is like, oh, I'm doing Python. Are there other examples of that? If you use magic today and you generate me a Python cell to do something versus generate me a SQL cell to do something, it will follow your instruction and it will dutifully do that because it's passed in as part of the context.

Yes. And this gets where it's not just a single shot GPT wrapper. Like we have several different agent responses happening as part of that. When you type that prompt off, what's happening under the hood? Like there's a step where we're selecting

the template to use. And so that context you provided will help with that. Is that deterministic? Like selecting the template? It's non-deterministic and that is AI response. We have the temperature turned on. Okay, got it.

We pass it a list of templates. We pass it your prompt and we effectively say our AI engineer is going to be like, it's a lot more complicated than that, Barry, but we effectively say like... Right, yeah. For the listeners, you adjust the temperature of the way I understand it. in that period but we effectively say like right yeah yeah which is for the list like for the listeners like you adjust the temperature of like the way i understand like more creative more varied

responses like yeah that's tighter more exact responses yeah yeah that's right so it's set to more exact responses because it's like one of a few templates right and because we have that temperature turned on it's one of the things that when we do evals it's you know if you think about it just from first principles like if you're going to do evals it's you know if you think about it just from first principles like if you're going to do evals you you actually want to make sure you're getting the same response effectively every time so you yes i wouldn't call it deterministic but it should be stable yes okay a certain model version is the way to look at it um but yeah like and then it's like okay well like what if the model's uncertain what if you haven't given enough context? Can it respond to asking for a clarification? Well,

it's like, that's a very interesting question. How do you, are models good at expressing uncertainty? Some aren't, there's other techniques you use to do that. You can have a judge weigh in, you can, it's like, there's different ways to do this and these are really unsolved. And so, you know, we look and think about this all the time. And the nice part for us coming out this, again, I think just kind of right place, right time is like, we're a product that is really our core focus. It'll always be our core focus, our being incredible for data practitioners, we want to be, you know, this awesome tool for people who are asking and answering questions, trying to do deep dive, you know, get to insights, things that aren't just churning out dashboards. And in many ways, we can incorporate AI into that product, into our product.

And it's okay for us to be a little wrong because we're targeting people who are otherwise B and hex writing SQL and Python, right? So like, it's the same as like a copilot, right? Like, it is not perfect. We use it, a lot of people use it or people use other cursor. There's a lot of other AI coding tools now. Those are not perfect. They can be a little wrong,

but the assumption is the user knows well enough to, or even writing tools, right? Like you use a, you know, ask AI to help you write a blog post. Like it's actually perfect, but enough what good looks like. Yeah, exactly. Right.

And so this has been really good for us because we can iterate on these things. Like we can be a little wrong and still be really good for users. Yeah. And get better and better at this stuff. And I think the question is like, how tight are you making those feedback loops? How good are you judging whether you're right or wrong? How are you in the trade craft we're building? Like everything we're building, you know, we run our own vector database. We didn't write the vector database ourself, but we self-host

a vector database. And setting up the pipelines for it and doing retrieval over it and scaling that and figuring out deployment to our single tenants. Like that's really hard, but that's something we're good at now. And we can just compound as the models get better, as the techniques get better, as new papers are published. And that's been really fun to see that sort of those learnings stack on each other. Yeah. For listeners who maybe were tuning out at the drive-thru, just a reminder, we are on with Barry McArdle from Hex. We're talking all about how hard it is to build AI stuff. So I want to change gears. And because you had a question about AI and BI, and I think that's a great opportunity

to zoom out and just look at the landscape. Barry, you were just at Snowflake and Databricks. But before we leave the AI topic, one very practical question, and this is, you know, mainly for me and John. He started a selfish company. Asking for a friend. Yeah. Yeah. No, but also for the listeners.

Are there any heuristics that you have developed through that process? And I'll just give you one example. You know, have you found that if you have to keep hammering on the prompt and getting like really wild with that, is that, you know, okay, when that happens, like we tap the brakes because we need to step back and evaluate or, or other things, right? Where you've sort of learned how to increase the cycle of experimentation by knowing when to stop or change your strategy? I don't know that I have a simple answer to that. I think we've got a pretty amazing AI engineering team that I think have developed a lot of heuristics and tradecraft around that.

I do think there's a point we've seen certainly in real life, a point of diminishing returns on prompt hacking. Yep. But one thing we've done that I think is important is really focused on setting the infrastructure for experimentation and evaluation up. I think that running headlong into... You can do all the prompt engineering in the world in a playground. Where the rubber meets the road is like, how does that work over a larger set of data points?

And I think that's the fundamental problem of like, you know, getting something working for a demo data set on your machine is one thing, getting it working in the real world for real user problems, whatever the space, data, customer support, writing, whatever it is, is much different. And your ability to build quickly,

to iterate quickly, to understand and shorten your cycle time of experimentation is really predicated on that infrastructure you build up. So that's where I was talking earlier. Like we've developed this thing we call spell grounds. We've built our own evals. Like that helps us learn much faster and see like, are we actually descending a gradient on improving by incremental prompt improvements. Like we can edit a prompt and run it over a subset of our evals and just get a feedback. Like, okay, is this actually working for real? And get that feedback. And there's times where it's improving things and there's times where it's not. And so I don't think there's a clean heuristic other than to say that like the only way to really

tell is to have that infrastructure for experimentation in good shape. Love it. All right. AI and BI, John. Yeah. Yeah. So I think coming off of conference season here, we've got a lot of players that are consolidating. I think you mentioned before the show how we have these nice lanes, like say two years

ago, like, all right, we're in the visualization lane, we're in the storage lane, et cetera. So maybe talk a little bit about the consolidation you're seeing and, you know, any interesting observations or predictions on, does that continue? What does that look like? Yeah, totally. I mean, it's such an interesting time and we just spent a bunch of time talking about AI, but even if you set that aside, I do think we're coming through a cycle in the data stack here on the data stack show. Right. Yeah. You guys have actually had a really interesting longitudinal look at the evolution of it. Because you've had a lot of guests on. You've had people on over time. Pay attention to the space, obviously. You know, what happens when a lot of capital flows into an ecosystem, you get a bunch of flowers that bloom. And I think that's a beautiful thing. I mean, I think people can look at it cynically, but I know a lot of really smart

people who built some really cool stuff in the data space over the last few years. And some of those things will continue on, some won't, but it's been very interesting to see. I do think we're coming into an era of consolidation, and it's not just like interest rates are going up or whatever. I think it's just like we've learned a lot. Like people have tried different patterns. There's things that work, there's things that didn't. And I do think that there's a couple dimensions of this. There's sort of like a horizontal dimension, which is at each layer of the stack, who's emerging as winners and losers? And what are the actual subcategories at that layer?

Like we kind of sit at the front end, the collaboration layer, you know, what are the actual divisions there? And, you know, if you look at like the metadata governance cataloging layer, what are the divisions there? Is there a standalone orchestration and transformation layer? You know, who wins there and how does that look? And then at the bottom, you know, the company's running these big conferences and running the infrastructure, you know, you have the cloud scalers, so Microsoft, Amazon, Google, and then Snowflake, Databricks are the big players. And then you have a long tail of

other independent, you know, data, you have like the Starbursts of the world, the Dremios of the world. And it's just this question of like, how does this actually consolidate? Are some of these sectors categories winner take all? Are they, can you have multiple winners and what do you need to do? And I think it was very interesting being at the conference and seeing, just walking the floor, talking to people, companies that two years ago, let's say at Summit, like Snowflake Summit 2022, were advertising like partnerships or talking about how well they work together or they were next to each other, you know, Boots were next to each other and they're co-hosting parties together

are now like, oh, actually, we're going to compete more than we had thought because you're kind of being pushed to do that. And this is happening at every layer of the stack. I think there's a really interesting question around like data catalogs, governance, metadata. Like, does that all become one thing? Like DBT has tests. DBT has their explorer view. You know, how far do they go with that? Where do standalone data catalogs live?

Where do standalone data observability platforms live like and i don't think anyone really knows the answer other than that there will likely be just by count like less account distinct you know less players but also perhaps like the actual categories will be more of an amalgamation than like these really fine subdivisions that we had in sort of the cambrian explosion of the modern data stack. It's always interesting to me to think about like Salesforce, for example, like really dominant in CRM. And then, you know, HubSpot's got some of that market too. It's interesting for me to think like in this space, is there going to be just somebody that's so dominant, like a Salesforce and then like maybe like a number two and then

like a sales force and then like maybe like a number two and then like a long tail or do you think it'll be a little bit more like because historically like oracle was really strong sql server was really strong yeah you know open my you know my sql postgres like do you think it's more like that or more like a sales force winner take all you know smaller well let's ask a question why is sales force so uh sticky and durable like i don't know if you use the salesforce ui i do sometimes i try to avoid it try not to i know no shade to my friends at salesforce but like i i think they would probably also say that the ux design of like the salesforce crm app is probably not the like thing that everyone's. Well, it's like a class moving from classic to lightning

was like a giant mutiny. Their own users were like, we're not, we are not changing. Like a decade long project, right? Yeah. Yeah. And so anyway, I mean, why is it so sticky? I had the chance to ask a very senior Salesforce

success this recently. And they were telling me, because I was curious about this, and they were telling me, it's the data gravity. It's the fact you can have a much, there's a lot of startups that have a nicer UI, but it's the fact that the data lives there

and there's all these integrations with it. And the industry standard, all the way from other tools to systems integrators and consultants, have all standardized around that thing. If you kind of look at the data stack, like maybe the closest thing to that in terms of like just sort of singular dominance has been Snowflake in the warehousing world over the last few years.

And it's a question of like how durable and sticky is that data gravity and governance? It's like, well, the data is there, you have other stuff there, but like this is why there's an interesting conversation on Iceberg and open formats is like, I think a lot of buyers see that. A lot of buyers have experienced being on the sharp end of this with Salesforce. With Salesforce, yeah, absolutely. And they're like, wait, hang on. I want more modularity.

And a pattern we're seeing a lot of customers do is actually stand up like a data lake alongside. And I asked a customer at Snowflake Summit this. Oh, interesting. Why is that the pattern? Because with Hex, they're excited about, you know, we can have one project and it can query from Salesforce or excuse me, from Snowflake and our, our data lake. They were, I think they were using Trino for it. Yeah.

And they were like, well, you know, it helps us get some of these costly workloads out of Snowflake, but it also like we tell Snowflake that we're doing it and then that helps our negotiating. Oh yeah, sure. Yeah. It was less about the actual like, we moved this workload from here to here and it's this much cheaper and more like by proving that we can move a workload, we've established some leverage in our negotiation or Snowflake rep. And so it was an interesting question, right? So you can see that the vendors of that layer, the last thing they want to be is commodity query engine

on open source format, data on commodity blob storage. That's a bad situation for them. And so then you start building a lot of other stuff to try to lock people in. And that sounds like a dirty word, lock in. I think the more generous way to say it you know you're trying to provide more integrated value for customers yeah but customers see that coming a mile away and so there's a question like what the market wants and how much that power of like integration will will pull and it's a question for everyone in the data stack right now so my my theory on it then is you're

right is there are enough people and it's not just salesforce so you got salesforce but then you also have like an sap or an oracle and the you know erp side of things so my theory is there's enough people out there in enterprise that want to avoid the you know huge lock-in and the implementation fees and the re-implementation fees and the upgrade fees and you know contract all of the money they have to spend around these systems to where I don't know if that model works again right now. As far as where a Snowflake or somebody could be as big as a Salesforce. We'll see. I don't know. Look at Microsoft. So Microsoft famously was on the sharp end of an antitrust lawsuit 20 years ago or i but but they're good at the same things if you look at it now yeah they've got you can go as an enterprise buyer as a cio and spend one contract with microsoft one overall agreement and have leverage in your pricing because it's

bundled on everything from windows laptop to your word license to getting teams thrown in for free to increasingly they're trying to leverage that over now into AI. So you're also going to have that same thing, not just your Azure compute, but you know, you get a GPT-4 endpoint on your own VPC that you get nice economics on. And then while we're at it, let's throw in Fabric. Yeah. Yeah. Fabric.

Cool. It's this one thing. It's, you know, it's better than Snowflake on this. It's better than Fabric. Yeah. Yeah. Fabric, cool. It's this one thing. It's, you know, it's better than Snowflake on this. It's better than Tableau with this. It's, you know, whatever. And you can bundle all that together. That is really attractive

to an enterprise buyer. It's also really scary to an enterprise buyer because you are all in with one vendor. I think it's a very real question of like how that tension plays out. And that's not new.

Like, I think it was like, you know, I'm not, I don't consider myself super young anymore but it is funny i was catching with my uncle who was like a really successful software executive you know in the sort of last generation and we were talking about this and he was explaining like how microsoft like had snuffed out markets in the 90s that I had never even heard of. Like these are companies I had never heard of. To him, it was like, it'd be the equivalent of them snuffing out, yeah, I don't know, like, you know, Snowflake or something. It's a company that's very prominent. I'd never even heard of it. I wasn't even aware of it. This pattern has been going for decades and it's going to continue. And there's this bundling and unbundling and

it's a very interesting time. Yeah. One thing that is interesting, though, and so this is a question for both of you. In terms of the platforms, I think, you know, there's sort of a classic, like Snowflake Databricks, you know, Battle Royale, and that's sort of been, you know, sort of played out in a number of ways, that a lot of companies, a lot of large enterprises,

like, run multiple different, they run several of the big vendors. Right, like per division or per business unit. Right, exactly. But at the same time, I think back on that a couple of years ago, and that probably was more real where it's like, okay, we run these workloads on this vendor and these workloads on this other vendor. But every vendor is building on the infrastructure to run like all of those workloads. And so I think it's more possible now from a baseline technological standpoint for there to be like a winner take all than it was previously.

Right. Because they, but that's more of a question. Like, do you think that's true? Or do you think that the different flavors of the cloud platforms mean that, like large enterprises will probably always run, you know, multiple of the major vendors?

Well, if history is any guide, large enterprises will have a lot of things for it. Like, I asked ask a customer, like, great, what do you guys use for data warehousing? And they're like, everything. It's like a fortune 500. Literally everything, yeah. And not just one of everything,

like multiple of everything. Yeah, yeah. And why is that? Well, maybe the company's the result of mergers or maybe different divisions have chosen different things. Increasingly, I think it's very strategic, which is like, we want to run different vendors because we want, we don't want to be locked in.

And multi-cloud is a very real strategy. And there's a lot of enterprises that are very bought into that path. In fact, I have not talked to many enterprise, like CIOs, CDOs who are like, yeah, our strategy is we're all in on one thing. So I don't know that it's how it'll face out. And I think you can look at even like the current markets that are like reasonably mature. Like I would argue that data warehousing

is like a reasonably mature market. It is very interesting to observe that the two players that wind up getting the most airtime, I mean, if you set BigQuery aside for a moment, or Snowflake and now Databricks, neither of them run their own metal. Yeah, right. They're both running on AWS or Azure or GCP.

Yeah. And it's actually just like observing that for a moment. Like Snowflake and AWS have an incredible partnership. They go to market together. I was at Summit in this like exec track event with, you know, like the senior AWS guy and the senior Snowflake guys and they're all there. But meanwhile, there's teams in each company

that are competing on deals. AWS makes Redshift. They make Athena. And I think, so this is not even a new pattern in the data stack. And I think when I talk to folks at Snowflake as an example,

they're aware that they are increasingly building things because they're worried about being commodity query engine that will compete with partners. But I think what's interesting talking to them is like, yeah, we're on both ends of this ourselves. I mean, famously, Databricks and Microsoft had a really great partnership over the last few years. Like Databricks on Azure was like a really big deal. And many ways that Databricks what it is.

Fabric is like ripping off a lot of Databricks. Yeah, sure really big deal. And many of us thought Databricks was what it is. Fabric is ripping off a lot of Databricks. Yeah, sure. And so I don't know. I think these people don't see it as winner-take-all. I don't think so either. And I don't think the data world has ever really been that way. But we'll see. In my opinion in the past, maybe 20, 30 years ago,

compared to, let's say 15 years ago you did have a big market emerge in the open source like postgres and mysql right because prior to that it was mainly closed source databases is oracle and oracle yeah of course yeah oracle sql server ibm dbt yeah so that was all closed source and And then you had a huge surgence with Facebook, for example, right? They went open source. So you have all these companies that prove like, oh, we're going to open source operating systems,

open source databases. So that's a major change. Now it's like, it's hard to see people swing all the way back. It seems like there's got to be some kind of middle ground where people aren't going to go all the way back like yeah we'll just do closed source like data like we're just going to go all in snowflake we're not going to think

about iceberg so I think because that won't go all the way back it's less likely to have like a winner they call it well I mean open source is a whole really interesting topic of like how and where do you build successful open source businesses

you know I've got a thesis personally and informed by a lot of conversations with smart people this isn't entirely my authorship but of like you can build a successful open source business can be successful when it's a pain in the ass to scale the technology yourself right like it's spark like yeah we i lived through as a palantir we used an enormous amount of spark we scaled itself it was a nightmare yeah and that's why databricks exists scaling kafka really hard so confident scaling any type of database typically is hard it's why the database vendors you know elastic at all exist i think when you look at certain open source technology, and there are some in the data space that is not hard to scale yourself. Right. Well, okay, you got to, why are we going to make money on this? And it's hard to make money then

on the actual open source tech itself. You have to make money on adjacencies. And without naming names, you know, I think that you see some vendors in the data space doing this. And I think you look at Iceberg as an example now post-acquisition. By the way, side note, the announce about them being acquired by Databricks went out during Snowflake Summit keynote. And our booth was at a kiddie corner to theirs. The vibe at the tabular booth. Sorry, it was tabular that was acquired, not tabular. Yeah, yeah, yeah.

Right. Ryan, yeah, yeah. But the vibe at the tabular booth was just so funny to watch. Oh, man. The employees were like, what do we do? Are we supposed to talk to each other? Should we pack up? We're kind of like in enemy territory now.

Are we like Databricks partisans now? Very funny situation. Yeah. Just really hilarious. That's hilarious. Was Ryan there? I didn't see him. Are we like Databricks like partisans now? Like, you know, very funny situation. Yeah. Just really hilarious. That's hilarious. Was Ryan there? I didn't see him.

Um, but it was funny to see him and like customers are going by their booth and like congratulating them, but the team doesn't know what to say. It was like, yeah, yeah, yeah. It's classic. But like, you know, it was like for them, like they were not, it's, I don't think it's a secret. Like they were not printing money themselves as a standalone. It was like, was an iceberg by itself technology you could make money on or is the money made on the like query engine around it? So, you know, I think you've got good points on the open source thing.

I think it has to flow from like, where do you actually, where does value accrue in open source? Right. where do you actually where does value accrue in open source right and what does it mean for like people's willingness to self-host or self-run things or that modularity and i think the story with iceberg and the reason databricks bought them is like you want control over that open format i mean it is there's like it's very telling that they want control yeah yeah sure because the more And they've done such a good job of this with Spark, right?

Anyone can ostensibly host Spark. The networking part of Spark historically was an absolute nightmare. That was the thing that made it really hard. And what Databricks did

was very clever. They wrapped Spark up in an open source governance model where they, it was open, but they, I'm using air quotes,

people can't see, but they controlled it, you know, Databricks is in full control of the Spark roadmap. And the networking part, they made modular. And then Databricks, like hosted Databricks Spark as like a superior networking module. Right.

And so you basically have an open source thing that's way harder to use. They've made their version of it, you know, much more scalable. And I think, you know, you can see this coming from a mile away with this sort of like table format stuff of like, clearly, whoever feels like they are able to control that technology is going to do a bunch of stuff to make it no less open on paper, but less possible for people to run and scale and utilize it themselves. And I think that's the thing to keep an eye on.

Yep. Well, I literally can't believe we've been recording for over the time limit, by the way, because we started early. So this is what we get to do when Brooks is out of town. I know, it's the best. But okay, Barry, I have a burning question that's unrelated to anything data that is related to Hex. So you've put out some amazing marketing material one launch video in particular and so can you how did like who wrote the script for that like how did was there a team where were you involved in

that how did it come about uh i just need a little bit of backstory on that because and we were talking before the show we like in the office, we have, you know, one physical office at Ruddersack, you know, and we all gathered around the computer and we watch it like three or four times. And so can we get a little bit of backstory on it? That's very, I'm very flattered. Yeah, I think you're referring to our spring launch video, which you put out a few weeks ago. People can find it on our site. Yeah, so we had a lot of fun with it. And yeah, the backstory is like, when we do these things, I think there's a very standard sort of like Apple keynote homage style

of product launch video that is very easy to sort of default to. And with everything we do, we just try to have some fun with it. People can see but on the video here you have like a boxed version of rudder stack yes the like a 1990s software box yeah our coalesce booth last year so we just kind of approach these things we're like can we just have more fun with this and not take ourselves so seriously while still being very serious about the software we're making and the value we want to provide so yeah yeah, the video was really fun. I was involved very closely. I really enjoy that stuff. I have a bit of a background in it.

I had a brief dalliance with thinking I wanted to be in like some version of film production earlier in my life. Oh, cool. Nice, I got that. But we've got a great team internally that we just have a lot of fun. There's kind of a core brain trust of a few of us that jam on these ideas all the time

and throw out pretty unhinged stuff in Slack almost every day. And it gets refined down to a few things. And actually, the video, the skit we did, the sort of office style skit was like, we were struggling with what to name this. We're like, is it 4.0? We were like, we literally had that internal, like, what do we call this release? The beginning of the launch video has this kind of dramatized version of us struggling to come up with a better name than spring release, which is very boring.

Yeah. I was kind of leaning into that. We got some more fun stuff coming. Yeah, that's great. That's great. I mean, you can have more screenings in the next few months. Yes. No, I'm super excited.

No, it was incredible work. Barry, thanks so much. We need to get you on the show few months. Yes. No, I'm super excited. No, it was incredible work. Barry, thanks so much. We need to get you on the show more often. Let's try not to wait a year and a half until the next time. Whenever you want, let me know. You know how to reach me. I'll see you guys.

Thanks for having me on. The Data Stack Show is brought to you by Rudderstack, the warehouse-native customer data platform. Rudderstack is purpose-built to help data teams turn customer data into competitive advantage. Learn more at rutterSAC.com.