The Data Stack Show - 222: The Future of Data Modeling: Breaking Free from Tables with Best-Selling Author, Joe Reis of Ternary Data

Episode Date: December 31, 2024

Highlights from this week’s conversation include:Joe’s Recent Projects and Work (0:55)Joe’s New Book and Inspiration for Writing It (4:39)Challenges in Data Education (7:00)Internal Training Programs (10:02)Creative Problem Solving (17:46)Evaluating Candidates' Skills (21:18)Market Value and Career Growth (24:03)AI's Impact on Hiring (27:47)Content Production and Quality (31:56)The Evolution of AI and Data (34:00)Challenges of Automation (36:12)Convergence of Data Fields (40:26)Shortcomings of Relational Models (42:09)Inefficiencies of Poor Data Modeling (47:10)Discussion on Resource Constraints (51:50)The Role of Language Models (53:13)AI in Migration Projects (57:00)Joe’s Teaser for a New Project (59:05)  Final Thoughts and Closing Remarks (1:00:07)The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

Hi, I'm Eric Dotz. And I'm John Wessel. Welcome to the Data Stack Show. The Data Stack Show is a podcast where we talk about the technical, business, and human challenges involved in data work. Join our casual conversations with innovators and data professionals to learn about new data technologies and how data teams are run at top companies. All right, welcome back to the Data Stack Show. We're here today with Joe Reese, a second time guest. Joe, welcome to the show.

What's up? How are you guys doing? Good. Also, Eric is out today and we've got the cynical data guy, Matt, here as co-host. Just sliding on over from the couch. Glad to have you here. So Joe, catch us up a little bit on

what you've been up to the last few months since we last spoke. Not traveling as much, which is good. So I've been non-stop globetrotting, which happens in the spring and fall. So I'm just back home in Salt Lake City, working on some projects right now. And that's about it. It's just been nice to just,

I mean, definitely thankful to travel a lot and see some cool places and meet awesome people, but it's good to be back for a bit. Yeah. Yeah. It sounds, that sounds really nice. Okay. Joe, we spent a few minutes chatting before the show. I'm excited to dig into a little bit about the book you're writing and just maybe get into some cynical takes on what you're seeing out there in the data world. Yes. Well, yeah, I don't think it's any secret. I'm working on a new book right now. It's on data modeling. And I can get into why that is. But what the book is about is it's an end-to-end treatment of data modeling across different use cases

whether we're talking applications, analytics, machine learning across different modalities of data, whether we're talking structured data, semi-structured, unstructured. The goal of the book is to really equip practitioners with an understanding of data modeling end-to-end. And so I think it's what I consider to be sort of the next phase of where data modeling is going. It's not just about tables anymore. It's much more than that.

We're working with different types of data across many different use cases. And so the goal of this book is obviously to equip practitioners with a body of knowledge of the existing techniques as well as hopefully introducing some new ones as well. The working title is Mixed Model Arts, which is sort of a plan words of mixed martial arts. So I can kind of understand where the threat is coming from. And I think the inspiration comes back. I grew up in the 80s. I grew up watching really trashy TV, like Kung Fu Theater and wrestling and all this stuff and boxing and you know and i think back in the day combat sports fighting was very one-dimensional you'd be a boxer or a pro wrestler and you're speedo or kung fu master in the uh mountains in china or something but there's always this notion that you know the you know the questions are always like who would win a fight

like would bruce lee beat mike t Lee beat Mike Tyson in a boxing match or under some set of rules? But UFC, that came around in the early 90s. There were obviously other things before, like Vale Tudo in Brazil, which is early mixed martial arts. But UFC, I think, was the mainstream thing. It blew the lid off the notion of being a one-dimensional martial artist. Fast forward to today, and you couldn't tell me

that the best boxer in the world, if that person gets into the ring in UFC, that they would do very well, or any one-dimensional sport. So I think, but if you take sort of the parallels to this with data, we've been stuck in the past. We've been stuck with these notions that,

you know, there's one true way to model data. You know, there's one technique to rule them all. You know, I think we've been, like I said, stuck in a table-centric view of the world, and sort of, you know, it's almost technique to rule them all. I think we've been, like I said, stuck in a table centric view of the world and it's almost akin to thinking the universe revolves around the sun. And the world's moved on. We have endless amounts of different ways of storing and querying data. We have different ways of moving data. Streaming is becoming increasingly popular and has been for a long time machine learning is everywhere and now it's ai and so you know but i feel like hopefully the world of data modeling and and data in general starts catching up to it to where we are so it's

part of the effort of the book yeah awesome all right joe so i gotta ask before we dig in a little bit more on the book, what inspired you? Like, at what point did you decide, like, I want to write, I want to write books? Because you were, if I remember, a data scientist a while ago, then you kind of evolved from there. But at what point were, like, books part of the equation for you? I mean, I've always wanted to write a book. I've been writing, you know, blogs forever. But I think during COVID, everyone had their

little project. We learned to brew beer or bake bread or become a gardener or something, knitting or whatever you want to do. Sourdough, that's another one. That's a big one. Everyone's way into sourdough. But I think Matt Housley, my co-author and business partner, we decided to write a book on data engineering. And why would we do that? I think that when we looked at a lot of the books out there, we realized there wasn't a book that really gave data engineering fundamental treatment from first principles. You know, it was typically let's approach data engineering from the perspective of, you know, teaching data engineering, which hadn't really been defined, using tools like platforms. And so we felt like take a step back,

maybe try and give a sense of order and a framework to think about the field of data engineering, which I don't think had existed before. And so I think the book is just a primer, right? If you're going to do data engineering, we just want to write a book where if you read it, you could at least be equipped, I think, with a pretty basic knowledge of what it would be like to operate as a data engineer but we wrote it in a way that was agnostic of technology or tools

because i don't my personal opinion is i don't believe that in this day and age you need to write books on technologies or tools i think perhaps use them as examples but the right things change those are better off being courses. At this point, I just think that books, it's a fundamentally different, it's a difficult exercise to write a book, especially a book that's actually simple, right? I think one of the criticisms of fundamentals of data engineering is like, well, I did, I knew all this stuff before. I was like, well, that's great. Good for you. But to write a book that's-

But are you doing it all too? That's the other question. That's the other part. And I could guarantee most most these people aren't um so you know and i think but to write something simple to bring complexity to bring simplicity to complexity i think it's very difficult to do yeah i think that was a challenge of it um but yeah anyway that's why we wrote the book um i think also to kind of go to your point there, there's a lot of stuff, especially for kind of technology specific that it has an expiration date on it. Those things are going to change over time. And the principles are more important for you to learn so that it doesn't matter what the specific tech is, you can always adopt to it. Oh, absolutely. Like right before we hopped on, I was advising a university on their data curriculum and, you know, they had things like data mining and big data with Hadoop and

all this. I was like, why are you teaching this in this day and age? This is very antiquated. You know, so, but yeah, the underlying principles are still there. They're still widely used, but, you know, technology's come and go, I would say. You know, the Hadoop, right? If you were to teach that as a class in Hadoop these days, I would say you know the hadoop right if you're to teach that as a as if you're a class in hadoop these days i would say you're probably yeah students out of their money but so teach as a historical artifact but it's sort of like teaching i don't

know how to churn butter manually or something right but sure it has a place but anyway well it's a funny it's a funny space right because you've got like commercial sass companies that like if they were to produce literature they're going to produce like how to use their tool courses on how to use their tool if you're in academia then you have like we want to keep this really theoretical and then often you have people that like learned r or learned hadoop or something and they want to keep teaching that class over and over again they don't want to reinvent the class every year necessarily because that's a lot of work.

But you can have it when I was in grad school and they taught us SAS. Oh, sure, yeah, or you have a partnership. But this is a tension in academia especially, right, where tenured professors don't want to revamp their courses because it takes away from their research time.

But then, as I was telling this university, I said, that's great and all. Congrats on your research papers, probably of which 10 people are going to read it. Meanwhile, you have students, especially international students, who don't get any discounts on their tuition who are paying top dollar for an education. And my impression is if I'm a student paying this kind of money for this education from this institution, give me the best education that's relevant to helping me, you know, get a good start in my career, right? Yeah. Which is also interesting because so much of the data programs that you see out there, I remember when I first started being a manager and I had to hire for this.

And that was when the data science programs first came out. Yeah. to be a manager and I had to hire for this. And that was when the data science programs first came out. And they were so applied based to like, this is how you call this library and then run this code. And there was almost no theoretical understanding behind it. Like a bootcamp style. Yeah, but like a bootcamp,

but for like 10 times the price. Interesting. It's crazy, right? Data science is a big one. That was a big one. And everyone wanted to jump into it because it was the sexiest job of the 21st century. If you weren't doing data science, you're just going to be left behind. Yeah.

The curriculum's bad. Yeah. And that was, like when I was hiring, we actually made the decision after interviewing a bunch of people for data analyst roles that we said we weren't going to take anyone from a master's program because we would have to unteach them so much information. It was easier to take someone just out of undergrad and teach them how to do it right. Wow. That's crazy. So did you have your own like apprenticeship program then in place? Oh yeah. So we had had a whole i was a crazy first year manager i developed the entire like analyst training program with the kind of the top two people on my team so we had it for we had a data scientist who could teach you a lot of stuff on he was in

what was a time called nlp like i had a data engineer who was a you know former dba and full stack developer so he really understood and he could come at a lot of stuff with data from like set theory and things like that. So he was really strong at teaching that. And then I basically sold people on it by saying, I'm going to teach you how to think and I'm going to teach you more about how business works. And so I had a, I actually had a like 24 book reading curriculum. I took through like a year wow damn it was intense but that's what it takes i think right and in school you did that i don't think a lot of companies or managers have that initiative or or insanity to do something like that in a good way

uh you know but but what i what you often see as sort of the the if you don't have a standard body of knowledge and standard expectations i think what you find is you probably know what happens. People do all kinds of crazy stuff on their own and they make up, they fill in the blanks, right? And you can't blame them. They hadn't been trained otherwise, you know, and the manager has only themselves to blame at the end of the day. So, but standardization is hard and skills and knowledge and teaching is, it's commendable.

So, and I think there's a little bit of a curse of knowledge in that for a lot of people, they've been doing it for a while and they've built up that kind of background knowledge in a lot of things. And then they come back to it and they say like, oh, well, you don't need to do all this stuff because, you know, it's really just these simple things. And you don't realize that like, you know, you have a lot of guardrails and a lot of ideas that keep you in the right path

because you had all this other stuff that you went through and all this other training. Right. Yep. How did you, question for you, Matt,

how did you pitch that internally? Because that sounds like a pretty heavy investment from the company and these people and that seems like a challenge.

I kind of just didn't tell them yes i said we were going to just train people internally and everyone went okay yeah because it was keeping kind of blind to the details i was also saving us like ninety thousand dollars in the process on our budget so once people saw that they just kind of stopped yeah it's not a bad hires and stuff and retrains? Well, so I initially pitched it when we had a certain budget for, long story short, I was on a team that was like four people.

I got promoted and two other people got promoted elsewhere. So I had people that were like fairly expensive that I was going to be able to backfill with, along with adding a few new positions. And like one of them originally was I pitched as a junior data scientist. And then we started interviewing people, and I went, oh, no, no, that does not exist. Okay, forget about that.

So we downgraded two of the positions to data analysts, which meant I didn't have to pay as much for them. But titles don't matter, right, Matt? Titles don't matter. Not at all. Sorry. So because of that, and there had been some internal pressure to like, well, like the two people I hired, two really strong senior people, and I actually had HR trying to talk me out of it. Because they're like, well, yeah, they're really good, but, you know, maybe it would be better if we hired someone, like, a little less skilled, easier, maybe a little longer.

And I'm like, that's insane. I'm not doing that. Right. So it was kind of that tradeoff that I went with, and I just kind of generically told him I'm going to do stuff. And I told my boss, hey, you're going to see some expenses from me for some books periodically. Yeah. And he went, okay. And he was a new VP, so he was so busy.

Yeah. He wasn went, okay. And he was a new VP. So he was so busy. He didn't really look at me that hard. You're saying you hired at the higher end of like the data analyst, like band, essentially. Yeah. So we had some people that, that had some very, like they were at the very top of the data analyst band and they got promoted into other places. So then I hired someone out of college who I could pay like half the price for. And I hired another person who was actually a woman who had not worked in a while

because she had raised her kids. She had gone back and she had gotten an associate's degree in programming, was a database person, and was the strongest data analyst I think I've worked with ever since. Nice.

That's cool. But we literally pitched it. We changed the whole job posting to be like, here's the 10 things you're going to do in this job. It was like writing documentation and stuff like that just to make sure that everyone was very clear of what you were going to be doing here. So Joe, I'm curious,

you've worked in at least a couple services type businesses. Like how did you guys approach and think about hiring? I mean, it's an interesting one, right? I think hiring is one of these things where I've hired a lot in the past, you know, outside of my own company, right? So, but I feel like you never know.

I think your example of this person who was, you know, a stay-at-home mom, which is, I think, a very difficult job, probably harder than most jobs. Definitely. Harder than most of them, yeah. and your ability to continuously learn. And, you know, I think, are you an add to the team in that regard? I very rarely look at your credentials in terms of what, you know, what school you went to, what big logos or companies you happen to have on your resume.

To me, those are great signals, but I've just known too many people who have gone to the finest universities in the world and have had impressive titles at the biggest companies in the world who I think are just, yeah, they're kind of douchebags. So, you know, I just don't think that's, they're not the kind of people I would have hired.

I think there's a lot of grift and I can swear on your podcast, right? Well, there's a lot, I would say there's a lot of bad behavior, right? So the only thing I really screen for is do you have the ability to i think be a good person with your teammates you know add value and continuously learn and are you a person of good integrity and character things that really can't be taught how do i assess this i just get to know

you i don't know i mean it's pretty it's probably not scalable but but to me, it's, you know, I'll look at your social media. I'll kind of see what you're about. I'll, you know, I think that if I hear words a lot, like if the conversations are generally about, you know, yourself and kind of what you're trying to get out of stuff, then I assume that you're very driven, you know, to look out for yourself. Sure. And not contribute to the team. Or if it's motivations like, you know, trying to climb the, you know, the corporate ladder and stuff, and you have a history of stabbing people in the back to get there. That's quite a kind of person that I would want to work with.

And there's, I'm sure there's plenty of fine outstanding companies out there that'll hire you where you can, where that behavior is institutionalized at this point that's how i hire i'm pretty old school in that regard where i again i look at things that you can't teach right like you the type of person who continuously learns that's hard to teach i can't teach you to do that right yeah no matter how hard it you can try you you can fake it but ultimately self-motivation and character and integrity are the things I look for. Yeah, I mean, especially on the self-motivation. I mean, at this point, I've had just a bunch of people that I worked with who would even ask like, hey, how can I try to learn more about data or try to break into it? Gladly give them stuff.

And one out of maybe every 10 would actually do and do the work. Yeah. Yep. Yeah. That's just it. So then you have your answer, right? Because the thing is business moves fast and, you know, the nature of a business is to solve problems. Yeah. Preferably for your customers.

And so you're always trying to solve new problems, which means if there was a standard playbook for solving problems, I mean, I could just write a program to do that or use an AI to do this. Exactly. Yeah. So by definition, you're expected to creatively solve problems in a continuous fashion. Yeah. Yeah, Matt, I've tried to do something similar. Find a way to gauge like willingness and ability to learn in that like, will they like read an article read an article like like like take any sort of like you know i say hey like this book is good on the subject or like hey this article this podcast like take any sort of interest in any sort of learning right like i found has been good and here's one that i'll ask both of you to have i i have seen i feel like this used to be a thing and people

don't do it anymore maybe just because they're lazy. Do you ever call references or ask for references with people in interviews, like when you've done that in the past? Because it used to be a bigger thing and I haven't heard people do that as much anymore. Well, I mean, I think it's still out there, but it's do they matter?

Because I mean, I used to work at places where HR insisted they had to do it. And I was like, okay, tell me if they say something terrible because then that says they're idiots. Other than that, I don't really know what to make of it. It can be game so easy. If any of us were asked

for references, I'm sure we'd call all references and give them a heads up and just say, hey, just make sure you put in a good word for me. I don't think, I don't believe in references. I'm sure that maybe depending on the company that might be required or just part of the playbook. My limit of status is if somebody refers you to me, I take that pretty heavy,

especially somebody who I respect. That's a good point. But if it's, you know, but I'll say, are you willing to risk our friendship or your job on this person? That's a good question. That is a good question. I mean, I've done kind of a similar thing

where I've seen people who worked at companies of friends of mine. And like that saved me one where I said, hey, you know, I'm seeing this person. Do you know who they are? And they went, oh yeah, he is quite possibly the laziest person I have ever worked with. Oh yeah.

Wow. And I was like, okay, scratch him off the list. Yeah. Buy you a beer next time. Yep. Yeah. And I was like, okay, scratch him off the list. Yeah. Buy you a beer next time. Yep. Yeah. And that's just it. I mean,

and these days it's so easy to find stuff on people. I mean, everyone's got a social media footprint. Or if they don't, then that's also a big red flag. They're probably a serial killer

or something. You know, in which case, like that person's almost weirder. So, like if they're not on LinkedIn, for example,

right? Like that's the first thing. Yeah. If they're not on linkedin for example right like that's the first thing like yeah not on linkedin i'm like either you're amish or you don't understand how networking and hiring right in today's age right right um maybe you don't care maybe you're a hipster maybe you still have an aol account or something too that's cool but we work in tech yeah yeah right you know and i and do the things like, who are you connected to? Right. It's because I want to understand,

okay, so who's, at least visibly, who's your network? Right. Yeah. Yeah. It's,

you know, people recommending you, what are they recommending you for? I mean, that to me is more of a reference. Yeah. Recommendations used to be gamed pretty easy.

Like, I think one day, me and a friend, we recommended one of our coworkers for stuff like ballet and horse training and stuff like that. So I don't know.

One of my friends he got a recommendation or endorsement for crying. That's brutal though. I have used in the past though specifically for some roles

where they may not be a total tech role but there's like a combo you know marketing analyst or something like that where you just go and you look at what skills have they tagged for themselves because if they don't have any tech skills tagged that's a red flag to me if all they have is hard work or communicate or marketing strategy, whatever it is,

like for whatever the role is, but they don't have any specific or like excels the most technical it is, that's generally a red flag that I would

look for. Now, of course, it was interesting. Like I was talking to a friend of mine last night who works in a non-technical industry who wants to become,

you know, so this person works in product management and you know, so this person works in product management and, you know, the health and fitness space, right? Yeah. And is really awesome at it. But it's kind of bored with health and fitness because it's super saturated in health and fitness.

Yep. It's the same stuff year after year. And, you know, this person wants to branch into tech. I'm like, okay, so how are you going to do this, right? If you're to go through an applicant tracking system with your resume, it's probably not going to go very far. Yeah, it's going to get kicked out.

Yeah, so go to meetups, meet people, start learning about tech, right? And start, I would say, gain proficiency, not just at a conversational level, but dive into it. But you're going to have to demonstrate competence in an area where all the odds are stacked against you. But if there's something you want to do, then there's ways to into it. But you're going to have to demonstrate competence in an area where all the odds are stacked against you. But if there's something you want to do, then there's

ways to do it. I think people do transition. Data and tech are notorious for people transitioning into it from adjacent fields. It happens all the time. You may have to take a little bit of a step back when you first go in, depending on what level you're coming from. I have a friend who actually reached out to me about that.

I told him in his situation, his best chance was to get to know someone who could hire him yeah because i said it's going to be the personal relationship is going to be the thing that helps you break in the fact that someone knows you and says i can see them and i kind of trust their ability to learn this because if you're just doing it blindly off of an application, you know, it's extremely difficult. Oh, difficult. That's polite. Nearly impossible. Yeah, because I mean, you know, think of how many people, resumes people are shooting out these days. I mean, the job market in tech and data ain't great.

So, I mean, I know people that are, you know, people with resumes too. Like, well, that sounds stupid. They have an established resume. Right, right. But, you know, like, you know, sounds stupid. They have an established resume. Right, right. But you know, like, you know, they've been looking for work for like 18 months now. Yeah. And part of it, I think that you point out, Matt,

is like the network too, where that's the other part of it, where it's great to have all these skills. And I wrote about this over the weekend in my sub stack, where it was like, I think it's titled, what's your path? Like, what's your brand? What's your network? And the whole point of it was,

you could be in a, this nugget resonated with people too, where it's like, What's Your Path? Like, what's your brand? What's your network? And the whole point of it was, you could be in a, this nugget resonated with people too, where it's like, you could be really well known inside your company. You could be awesome inside your company. But the thing is, if nobody else knows about you outside of your company. Yeah, right.

You know, what, is it really worth that much at the end of the day? Because, you know, especially if layoffs happen, everyone that just got laid off is looking out for themselves. You know, they might try and help each other out, but, you know, there's a bunch of people on the market now. And that's a paradox.

What you see is a lot of people put a lot of effort into being just the biggest rock star in their company. But the thing is, it's self-contained. And so how public are you in terms of getting out there? I think it's increasingly a big ingredient to success. Whereas in the past, that wasn't the case because you had more job security and so forth. There was more of an expectation like, hey, if I just work hard and do a good job

and just keep my nose to the grindstone, I'm going to get recognized someday. And that doesn't happen anymore. If you're waiting on that, I would say good luck. Yeah, I mean, I think how much that really worked, I think in the past sometimes it was tough too, that you've got to be good at something. But people have to know that you're good at it. It's not just going to leak out into the public.

Yeah. I always have this three circle model. If you can imagine like three circles overlapping one circle. And then I'd use this like with employees to try to explain, like somebody was frustrated, like I feel stuck career wise. Like I want to get promoted. I want to make more money. So like, all right, think about these three circles.

Circle number one is your market value. Circle number two is your value to the company. And those two things overlap in a spot. And there's areas that they don't overlap because their market value things that you might be able to do that we don't care about. And then there's things you can do for us that's super valuable that nobody else is going to care about. And then the third circle is what do you actually like doing? So I tell people like, you want to try to optimize, like, what do you actually like to do?

We're like, what's the market value? And then you got to provide value in your current job. And if you can figure that out, like that is typically like the right, like intersection of three things for most people when it comes to like, to career. Yeah. but i would say even now though because we've seen this flood into the market in the last i don't know five-ish years or so and now that we've had this a little bit of a contraction and everything that actually you know has become a lot more important sure oh yeah yeah i think even long term like especially if you're going to go into more of a leadership role or something i think an undervalued part of what you bring is your connections is your network whether it comes to hiring or influencing or whatever like that is something that is there to help you get a job but

also within those jobs oh yeah yeah and any sort of leadership level, right? If you have to like, bring in people, like certainly helps if you aren't, especially if you're doing a turnaround, like if you're turning a team around, it sure does. It helps you a lot to have a network of people you can pull from to be like, all right, we're going to turn this around in 18 months. Like, like, let's go do it. Versus like cold interviews and high, like, that's just. That is actually a red flag for me. If I go look at an executive at some place, if I see they've been there for a year or two and they haven't brought anyone over from a previous job.

From any previous. From any previous job. They've only, you know, done the normal recruiting process. Cause that tells me you have either burned all your bridges or like you don't do a good job of relationship building in the first part. And that's a problem. Interesting. Happens. It happens.

Yeah. Yeah, and it only takes a few of those things to happen. But yeah, I don't know. It's an interesting market, and I'm curious to see what happens, especially, I'm sure we wanted to throw these two letters out, AI. I'm curious to see what happens with AI

and how it affects hiring and work and so forth well what's well I want to talk about that what what is your take on that because I have a perception and we talked about this a little bit before the show I have a perception that there are plenty of companies are going to do like the wait and see thing of course of like yeah like let's see like at what point it's like do we stay really lean? Just get as much as we can out of the existing people. We'll throw you a bell and like, here's some money to spend on AI tools. Like at what point does that potentially get exhausted? And people say like, all right, we just need to hire more people. Like we can't just throw AI on it and like squeeze an extra,

like 10% out of this team. Or maybe that's not happening, but it's just kind of a perceived thing I have from some companies I've worked with at least what gives you that perception that like the companies i've worked with it's all like oh like well we can be more efficient now like here's an ai tool like developers like this makes you 30 more effective microsoft said so like so you only need three on the team instead of five you know or whatever the math is there that's more like wish casting than an actual plan but of course no it's not a plan it's just like an executive reddit headline that co-pilot or whatever i'm just like saying co-pilot but whatever tool makes developers x more percent effective therefore like we need that less people we need 30 percent less

people i i think it's it's the wish of every executive that they could just have a company run by conceivably no people and make billions of dollars a year and so forth. And I think there's a lot of people inspired by Sam Altman's prediction that somebody's going to build the world's first one-person billion-dollar company. And we'll see if it happens. I mean, I'm starting a new company right now and I'm trying to leverage as much automation and where it makes sense. AI is

as humanly possible. Like why not? But I'm starting from nothing too. I have no entrenched processes or resistance from anybody because there's nobody here, right? It's just me and my friends. So that would be a fun like thought experiment. So say you're starting a company right now, say you were doing this 10 years prior to now. What do you think your differential is on people roughly between if it was like five or 10 years ago and now? Given the state of technology back then? Yeah, exactly. Yeah.

I mean, it's an interesting one. I mean, I was starting companies back then. So I have to think of an opinion on that and working at startups. I think back then, you know, you had it it was, it was basically when, when SaaS was just getting hot. Right. So, but what you saw was you, you could definitely sign up for a lot of services, you know, like it would handle your expenses.

Payroll got a lot easier than it was. I mean, I don't know if you remember payroll before SaaS sucked, still sucks, but it's easier. Yeah. You know, what else? Just HR management, all the kind of the stuff you don't want to think about, but you have to like, that's gotten a's easier. You know, what else? Just HR management. All the kind of the stuff you don't want to think about but you have to

like that's gotten a ton easier. There's still a lot of friction involved because you're dealing with people at the end of the day but stuff like documents workflow tools are easier.

I would say workflow management and task management is conceivably easier except for the fact again, you have people running it and sometimes

you don't even use the tools so I don't care if you're using Jira or Asana or any other great product they're all great. Can't blame them. So, you have people running it, and sometimes they don't even use the tools. So I don't care if you're using Jira or Asana or any other great product. They're all great. Can't blame them. So, you know, I mean, back then it was, okay, so how would you, you know, so there's a plethora of great SaaS tools. And then, you know, if you're hiring developers, it's, you know, back then it was easier-ish to find developers.

It's still hard to find good ones. That hasn't changed. But, you know, software development life cycles are what they are. I don't think that's changed that much. We're still doing basically the same stuff that we've done in the past. Now we just happen to have AI co-pilots. And I think to the degree they're effective, I think depends

from what I'm seeing in my own experience, it depends on the type of problem you're trying to solve and the language you're using. It works really good in Python, for example. I think it does. It doesn't work so well in other languages, according to my friends who work in more esoteric stuff like, I don't know, Elixir, for example. But I would never write an app in Elixir, so I don't really care. I have no reason to do that.

Shout out to all the Elixir people out there. So yeah, but I think it's, and then with content, it's interesting because on one hand, LLMs have made content production conceivably easier. I could say it's also made it conceivably worse depending on the type of content you're producing. It's kind of like a lot of technology. It hollows out the middle.

So the mediocre stuff now gets a lot easier to do. But that just means we get flooded with a lot of mediocre to below average stuff. Oh, yeah. Look at any social media right now and you can see it. You can tell when you listen to the person speak with the re-note the script. Or if you read the copy, a lot of it's just super generic looking. There's no personality.

So on one hand, I think the rote tasks are going to get easier. I mean, I think it's still early days of the agentic workflows. I still think that it's... It feels like maybe another year and that's going to be a bit more baked in, I think, and useful. But, you know, I mean, I use LMs

all the time. I have pro subscriptions to all of them. Why? Because I at least want to, you know, use them where I can and experiment in areas where I think they think probably in a year or two they're going to be there. But I don't think it's going away. What does this do for hiring? What does it do for jobs? I don't know. I think that's TBD.

You read about Klarna that says they put a hiring freeze in recently and they're just going to run the company with AI. We'll see if that works. I think they said they're also, correct me if I'm wrong, I thought they were gutting Salesforce and trying to build their own AIs on top of that. There's speculation too. This is an interesting thread where, okay, so the nature of software is going to change, right? Where instead of application workflows,

now you have agent workflows. I don't know if this is marketing speak or if it's real. But this is an exciting time because you get to try this stuff out and see where it works at a ground level in your business. I think that's super cool so yeah one of the things i wonder about that you kind of touched on there is like we had all these sass tools came up and it was supposed to change everything but a lot of times it was well you have a people are a process problem we've created technology it's going to fix it and it's like okay but if no one fills in the stuff it doesn't

matter what your technology is exactly so a lot of this kind of AI agentic stuff, do you think, are we getting to where there's more of the people, you know, we're handling more of those like people process things? Or is it just like, here's a souped up technology that if people don't use, it still doesn't matter? I think it's definitely the latter for now but at the rate these things are changing either agentic AI is going to be a flash in the pan or if it continues I think it'll be great if you look at something like Devin

I think it's super early days for that kind of workflow where you can have a quote junior software engineer that happens to be a bot I don't know i mean i've i was around when people said the internet was a fad as well and that you know that would have been one of the dumbest things you could say now so who knows right i just i even blockchain

right i've i don't see any utility in it but who knows maybe there will be at some point you know people make a lot of money in it but that doesn't that that's not the same as utility you can speculate on it. That's all speculation at this point. Yeah, right. So I don't know. I mean, but I've just learned I don't discount it. I don't write anything off, you know, out of hand.

You know, I'm more of the kind of person who looks... But ChatGPT was interesting, right? So I think unlike, you know, Bitcoin, I mean, what did that come out? That paper came out with 2007, 2008 or something like that, right? I think we're still waiting for the use case where, apart from coming up with meme coins, we're going to change the world with it. Or money laundering for illegal activity.

Sure, right? I mean, yeah. I mean, that probably revolutionized money laundering. And that accounts for a lot of money. It's not discounted. I'm not advocating it, but it did probably streamline that industry or that way of transacting. Is that the mainstream? No. But contrast that with ChatGPT where when that came out was like two years and 20 or 19 days ago, that changed the face of a lot of things.

Like I could hand my child ChatGPT and they knew exactly what to do with it out of the gate. That's different. So, and then, you know, you have every CEO in the world who's using chat GPT and their kids are using it like, okay, so how can I do this in my business? Like, what can I, you know? Yeah. And that, and that seems to be one of the problems. I mean, we've talked about this before, the idea of, well, look, I can go on chat GPT and I can do this thing. Why can't you do it? It's like, yes, you can make it work once. Now make it work a million times without variation. Yeah.

At scale for a million different people. Yeah. And that's just it too. It's got, I think, you know, that's where the edges of utility, at least for me, is like, where do I need something that's deterministic and where do I need something that's kind of, you know, it can be more fuzzy with, right? And so if it's deterministic, I'll just write code because I know that's going to work every single time.

But then I can also use a copilot that can, in theory, speed up my development. I'm still at about a 60% or 70% success rate with that. I have no idea why you suggested this code at all. But it definitely helps speed things up so i look at more of an accelerant but the crux of it is you have to know what you're doing right so this comes back to what you were doing with your training you know and god bless you for doing that i think that's awesome because if you don't know what to look for

now you just do dumb things extremely fast yes yeah. Yeah. Yes. It can accelerate pain for a lot of people because the, you know, because you can invest a problem, in my opinion, with whether you're coding for apps or data pipelines or whatever, you can make things work and do it in some really terrible ways. And Copilot and other similar technology can absolutely just be fuel on the fire to do that and then end up with bigger messes than if it was all done by hand poorly. Yeah. I mean, I worked at a place where we did, we were doing loan auto decisioning and that was one where like you had to be careful on that stuff, especially because it was a regulated industry and all that stuff. And and i that was a unique one because i didn't make the models but i owned the code base that did all

the decisioning on it so we had to be that was an interesting problem especially when it came to how do we test to make sure stuff is working right but those are the types of things that like we had defaults that if anything failed you know you would do a certain action because the amount of money it would cost to just do bad stuff was huge. Yeah. That's interesting. Auto loans.

That was it, right? Yeah. Yeah. That's an interesting business. Yeah. I ran into a guy once. This is back late 2000s I think I think he's trying to hire me for something but he was doing subprime auto loans I know that space

a bit but he had his spreadsheet and I have an actuarial background so I was fascinated by this it basically is risk pooling is all you're doing it was interesting he managed to make a lot of money i think then the other sub no this is actually during the subprime crisis as well and he he was still doing well so

yeah well the key the key on the subprime stuff is generally you're gonna you have to just account for can i make a profit if someone doesn't pay this back? That's literally what it is. And then a lot of it, if you can correctly kind of rank risk, and if you can make a model where people will take the loan and you can make money back and, you know, you can make money in a year or two of them making payments and you keep your expenses low, then you can do it. And so that was very, you know, for a lot of it was really easy. As interest rates went up, that started, that was actually the big thing that started squeezing some of those

was interest rates went up, cost of capital went up. That was a huge squeeze on those types of profits. So I want to make sure we get to some data modeling here. So Joe, give us like a, just kind of a brief on your book you're working on now. And then curious from you, Matt, too, after we talk through that, on what modeling stuff you've done in the past. But yeah, give us a brief on the book. I mean, so yeah, I think as I hit on earlier, it's going over, I think, a lot of the established well-trodden concepts,

but revisiting them, I think, from, again, kind of a first principle standpoint. So at first, the book starts off with, like, what is data modeling? Why do we even bother doing it? Why is this important in today's age? Why don't we just ignore it, right? Yeah. And then we go through the history of what I call the convergence, which is, you know, the fields of computing, analytics, and data, and then AI, right? And in the past, these were sort of all separate fields. Maybe there was some

overlap in some, like using a computer to run an AI program or so forth. But, you know, the fields of study had been very, I'd say, isolated. But over the decades, what you see is, you know, all these tend to converge. And this is where the notion of mixed model arts comes from. It's data modeling around the notion of the convergence of different use cases of data across different types of data. This is reality right now. If you use any app out there, see Uber, Netflix, whatever, this isn't just some janky Ruby on Rails app. It's a very data-intensive, robust analytics

and ML-powered application. This is where the world is and the world is going. And so, you know, then the book basically gets into some of the building blocks of data modeling. So if we look at the notion of an entity, right, that's pretty well-trodden and tabular data. But now, how do we extend this into things like semi-structured and unstructured data? Especially when you talk about unstructured data, like an image or text, it gets very interesting.

An entity could mean a lot of things, actually. Entity resolution could be, well, anything that's in the body of unstructured data. Or it could just be the file itself, right?

And so this is, so I think it's helping, I think, expand people's thinking. And then obviously, what I'm working on right now, which I hope to publish this week, is the exploration to the relational data model. It came out in 1869, 1970.

And I would say that is the underpinning of how you work with tabular data. Everything is derivative of the relational model. It was the first model that really took into account how do you, you know, if you take a step back, what I've been writing today is basically the underlying math mathematical principles of the relational model what is relational theory how does this work so how do we translate this into tables and what are the shortcomings of this approach right this is something that actually doesn't get discussed is when we talk about things like tables and sql this actually doesn't map correctly back to the relational model there's a a lot of flaws, namely duplicates, nulls, ordering of data, which if you look back at basic set theory, you can't have nulls.

You can't have a null element. That doesn't make any sense. But we allow nulls in tables. That violates the relational model. You can't have duplicates in sets by definition, but you can have duplicates in tables all day. So it's exploring these things, right? And I think just establishing, I think a theoretical and then a practical example baseline of each different use case, but, and giving treatment to, again, the big ideas. If you're talking analytics, obviously, Kimball, modeling for

data marts, you could argue that kimball is data mark modeling at least as linda would describe it then data vault and one big table which is popular these days right but then why would you use and i think and also assessing the trade-offs like if you're going to choose one big table why would you choose this versus another approach what are the trade-offs i'm not going to give you an opinion one way or the other the goal is to make you just cognizant of okay if i'm going to take this approach and i know all these other techniques just like in mixed martial arts if i'm going to fight like i'm not going to be orthodox about i need to throw jabs only this is how i'm going to win this fight and maybe a hook or something like that would be

if any of us were in like an actual fight that would be the most idiotic approach i've actually done this in jets where i said i I'm only going to try and win by armbar. But, you know, with machine learning, right? This is where things are going. So how does data work? We're taking tabular data, right? What's the mental framework to use? Tabular data with machine learning, right?

Basics of feature engineering and just the big model approaches. And then what types of models are appropriate for different types of data? And then kind of closing out with sort of a future looking view of data modeling as it stands of, you know, today, which will probably change. But that's really the kind of flow of the book is that there's a lot to cover. But, yeah. So I'm curious, Matt, I want to get your take on this too. I'm looking at this, I guess, maybe in two ways.

One of like, there is sometimes like an ideal model that we should use for this problem, like when it comes to data modeling. Most of the time, there's like, oh, we could do it a couple of different ways. It probably doesn't ultimately matter. I'm curious about- Until it does.

Yeah, so, but I'm curious, exactly. I'm curious about that third topic of like, what, maybe some real life examples examples if you guys can think of it i just thought of one of where the data was modeled wrong like a project you've worked on and it just haunted the team or the project for like for a long time because it was like a fundamental data model problem you want to go first matt oh i have one right off the top of my head this was back in the heyday of the SQL versus NoSQL days. Ooh, yeah.

And so this was, I've talked about this one on the show before, but it had to do with scraping prices off the internet. And the executive who was in charge of this project insisted on putting it all into a Mongo NoSQL database. Oh, that was great back in the day yeah yeah and when pushed on why the response i got back from the team was because it's the future of data to which i said that doesn't mean anything okay but the problem was is that every use case we had for it the first thing we had to do was turn it into tabular data yeah every freaking time and so and if you've ever looked at like the mongo querying language, I have the way I described it at the time was it's like Martian sequel.

So it was like, we had to go through all of these things of where they were so proud of this database they created. And we're like, and we have to basically break it off and do all these weird things every time we need to do something with it. Yeah. Oh, and by the way, the whole point of this is to get this data into a SQL database. A table, yeah. Like that's where it's all going.

So why are we doing this in between stuff? Yeah, absolutely. What about you, Dan? Something similar to that, right? I mean, Mongo's great if you know the use case you're using it for and I think know how to use it and why you're using it and have a good reason. I don't think what Matt described as the reason would be a valid reason in my opinion, but you know, whatever, not my problem. So. Yeah, that was the thing. It was my problem.

It's your problem. Yes. And you, you know, and what I often see, you know, if you're talking apps, relational databases, right? Talking varieties of Postgres, My know, and I often see, you know, if you're talking apps, relational databases, right? Talking varieties of Postgres, MySQL, whatever. How many times have you seen a relational database, tables, for an application not modeled in any real relational form, right? It's probably first normal form. All the time, in my opinion. You know, you may start off with good intentions or no intentions, but it's just, there's tables. You can put data into tables, so let's put data into tables.

This is, you know, and our app will just run off this. That's great, except when you look at, and this is why I kind of go over the why of why these techniques were created in the first place. Relational model was the notion is to reduce

data redundancy and dependencies and update anomalies right if so if you have redundancies in your data guess what happens if you try and update it or delete it now you have to deal with all these other places you get to do it and you'll probably make an error right just understanding simple set theory and thinking about your data from first principles would solve that problem. It's just a tiny bit of thinking, not even that hard. Just, is this data dependent upon this other thing? How could I split this apart where I don't need to duplicate my data?

I just have a row of, you know, I have a table of IDs over here. They relate over here, you know. And so that's just the notion. But if you just put like an ounce of thought into what you're doing, it's not even that difficult. It would save you so much time down the road because what inevitably happens is, again, you have all kinds of update anomalies. And at some point, your database starts creaking under its own load because now it's doing

unnecessary work, right? It's operating extremely inefficiently. And I've seen this happen. I had one client where they were trying to run this very data intensive application, doing lots of analytical workloads in the app database, right? So there's an

OLTP, I think there's Aurora, but every single time, because they're trying to do these analytical workflows and these kind of quasi intelligent workflows, as a transaction occurs, it kicks off all these other stored procedures and it kicks off all this other stuff and then they're like our database is creaking under a lot of load and like yeah i

can tell you why like that shouldn't those workloads need to be somewhere else like right here but yeah like it was at the point where they had to actually start pulling off features off of their app and reducing the functionality of it in order to not completely crumble under the weight of this database. I've seen that too, where in one place I worked, they had, well, were basically foreign keys, but weren't in there as foreign keys in every table. But you couldn't connect them directly so you had to go through my joke was it was a snake schema because you had to go through this whole lifting yeah and it was literally 14 joins to connect and you had to have a distinct on your select statement yeah like that's a problem yeah yeah we i, I had an app. So you referenced Ruby on Rails. This was an app that was groovy on Grails. Oh, I remember that stuff, yeah.

Yep, Java. So this was an app. And one of the user features was a completely no-limits, build-your-own-search thing that essentially you pick as many columns for as many tables with like whatever where clause that you want and then it builds arbitrary SQL and executes it on the database. What could go wrong?

What reason was this done? It was like it was part of like it was a transportation app and it was part of like hey we want to be able to identify loads via a bunch of different characteristics and pull in all these like hey we want to be able to identify um loads via a bunch of different characteristics and pull in all these like different fields where is it going to where is it going from who last touched it like all sorts of different things and we went through a

mongo db phase we went through a solar phase if you remember that we went oh yeah church phase of this we went through a like it was postgres in the back and we went through a like let's spin up several read replicas and send this stuff to the read replicas of the postgres so we got through all these phases and the thing that we were constrained by this because this is almost 15 years ago we were constrained we're actually on physical hardware so like teams now like you can buy your way out of a lot of these situations if you really want to by just continuing to throw money into larger and larger instances. I mean, I've even seen that on, like, Google BigQuery. It is far too powerful for the user's own good. I mean, one place I worked, we had that,

and there was one query that people were writing that literally had seven levels of nested subqueries. Wait, what? Seven levels. And somehow BigQuery was able to optimize that to run. I mean, it took a while to run, but they were able to do it. I tried teasing it out when I got there. I'm like, this is crazy.

Let me figure it out. And I got to about three levels down and went, I give up. And I told them, this is where if you had to do this on a on-prem system, it would break and you would learn how to do this better. So I'm really curious on your takes on this because this is, because like when you, I think what all three of us learned, there was a practical like constraint on resources where like, you know, it's a big pain to like go procure more servers.

So you have this constraint. Then that constraint comes off, say, 10 years ago or whatever, and now it's like, we don't know what to do. I don't know, just size up the instance. What do you think that's going to do to the developers? What is that going to do to people? Because that just reinforces bad behavior and inefficiency. That's also one of the problems of when people say just teach finance SQL and they can, you know, we're going to make everyone can be an analyst type thing.

Right. You know, because I've seen it where it was an on-prem server and queries took literally like two second queries were taking five minutes. Okay. I was like, this is the worst hardware I've ever seen. They migrated over to aws and what we found out was that finance was running queries that the first step was get 12 billion rows and then start telling these were theories that they would literally start running at the beginning of the day and they wouldn't finish until after lunch yeah they literally like just have it spinning at

their desk like go get lunch lunch. Yeah, it would be they'd turn around at 8 a.m. and then do whatever work you had to do and then around after lunch it would finally work. And that was what was crippling the system.

Of course, yeah. Crazy. Well, it kind of goes back to LLMs, though. I think that that might be one of the utilities is to say that like seven-layer hellscape

of subqueries or whatever. It sounds like a bad taco bell burrito um seven layer dip seven layer is a sequel diarrhea so you know but that might be a use case where an lm could you know be helpful just like throw it to that and say i don't know what to do with this figure it out like it might because at that point you're at a hail mary where you're not going to do it anyway. So I don't know, can robots fix it? The other one I see this a lot

is just SSIS workflows. I mean, this is like the game that puts a lot of companies together. And sort of procs in general and all this other stuff. There's all this code right now that sort of runs companies

and maybe the team that wrote it's still there. Maybe there code right now that sort of runs companies you know and maybe the team that wrote it's still there maybe there's comments i don't know but probably not if you breathe on it wrong it might break yeah so i'm like you know this is where i see that potential for large language models in particular is like go into these code bases and just like try and figure it out nobody else is going to do. This is a job for humans to do. I mean, humans caused it, but I don't know. It's gross. I mean, once you get that far away from it,

I mean, I've had friends who work at banks that they have a 70-year-old developer who's on a $500,000 retainer because that cobalt code breaks once a year and they need to come in and fix it. And otherwise, they just live out on the on like uh key west that's what they do yeah yeah i think we i guess we missed the boat on that one but maybe that'll be ssis for us but i mean i think the interesting part about those gooey based like ssis alteryx like there's other gooey based tools where like somebody

that's not like officially in IT, maybe is on a data team, maybe they're just kind of on the line in no man's land, they can build fairly sophisticated workloads, put them in an SSIS job or Alteryx or whatever, one of these GUI-based tools, and you end up with lots of business logic, often dicks into these tools and they end up in critical processes.

And then that's terrible if on the technical the technical side you got to manage it like you can't it's not version control it's not documented like but on the business side like that was like their best solution because they didn't because a lot of times they didn't want to mess with it they didn't want to get on it's roadmap they thought it would take forever like what do you think what do you think the right answer for that tension is? I don't know that there's a right answer for that. I mean, that's one

that like, I mean, part of it is you got to get into there. There isn't a roadmap for that. You got to actually get in and figure out what's going on with that company. Sure. I think there's also you know, these tools can be effective if contained. The problem is once they do a little thing good, they say, let value, average order value, and they're all different. Right.

And they're all hidden. Right. And everyone, once it goes out of sight, everyone assumes it's right, or they just kind of implicitly trust that. And now we get into these arguments over, well, what about, you know, which number is the right number?

And what's this and that? And, well, this, we're seeing this. This is what the numbers say. And it's like, like well what is the definition of that nobody knows anymore because it's hidden then it's all like oh let's just agree to disagree we don't agree but we'll we work together we'll just have we'll just have four metrics there's marketing sales we'll just average all four of them and that'll be the number all the time but you know i be the number. All the time.

But, you know, I'm hoping this is like the killer. I keep telling everybody if they're working on AI problems or technologies, like migrations and fixing legacy code, like this is the toil that, you know, at least bring it to light. Yeah. You know, I mean, transformers are made to translate stuff. Literally made to translate. I think for Google Translate, like It's pretty good at this stuff. I totally agree.

I mean, Amazon had their whole study with it. They had their whole press release in August where they said they migrated from Java 8 to 17. They saved like 4,500 years of work. And I'm like, that's awesome. That is awesome.

Yeah. And that's one where there's actually a lot of the stuff that AI is currently doing for people really well is lower value work. So it's hard to kind of for lack of a better term, like monetize that up for the cost of making these models.

Migrations could actually pay for a lot of this stuff. Sure. Yeah. Nobody's going to join a company or very few people are going to join a company saying I want to work on the migration project. That sounds like a lot of fun and a way to boost my career.

That's why everyone just says, we'll completely build it ourselves. We'll just build a new thing. Yeah. And how often does it work, though? It doesn't work. I mean, it doesn't work, but they view that as a better shot than trying to migrate. It's a temptation. Like, you know, I can go to therapy or I can just, you know, I can just change my friends or my spouse.

It's fine. I'll just find some fine people that reinforce, you know, I don't want to change. I'll get a new job. I don't need this one anymore. This place sucks. Even though I'm like total cause of all my problems exactly it's amazing how everywhere i go the people are just terrible yeah yeah i guess i just have bad luck this is so unlucky yeah i've heard that one yeah all right guys i think we're

coming up on time here yeah we're gonna get real cynical in a second on this one so yeah that's what we're here. That's my job. Yeah. Joe, can you give us a teaser about the new project, new company you're working on? I don't know if you've had any official announcements. Do you want to give us a teaser on that? I'll be announcing something end of January. Yeah, it'll be fun.

It's education related. Okay. It's not like a mafia guy trying to describe it. I work in garbage. Waste disposal. No. It's a waste disposal company coming out at the end of January. Awesome. Waste disposal.

Alright. You got any takes for us? You got anything cynical you want to add with Matt? I mean, I think we kind of hit at the end there with some cynical things. We're going to depress a bunch of people if we keep talking. I know, yeah. We just need to come up for a good source. I mean, that's just what I kind of just am, just sitting right there. to cut it off for a good source. I mean, that's just what I kind of just, that's my jam just sitting right there.

But I think we're good for now. All right, awesome. Joe, thanks for coming on. We'll definitely have to have you back after you officially launch your next thing. And Matt, I think you guys are in the pod too sometime. I got a new live show coming up soon and you guys are fun and curmogeny enough.

It'd be good to have you guys on the pod as well. Excellent. This is beyond good behavior. This is pretty good behavior for you. You got like an ankle bracelet on or something like yesterday? We'll let you get a beer at the bar if you behave today.

Alright, awesome. Thanks, Matt. Thanks, Joe. Thanks. See you guys. The Data Stack Show is brought to you by Rudderstack, the warehouse-native customer data platform. Rudderstack is purpose-built to help data teams turn customer data into competitive advantage. Learn more at Rudderstack.com.