The Data Stack Show - 223: End-of-Year Product Trends: The Cost of Rushing Features with The Cynical Data Guy

Episode Date: January 8, 2025

Highlights from this week’s conversation include:Christmas and New Year Edition of Cynical Data Guy (0:28)Discussion on AI (0:42)12 Days of Shipments (1:04)Attention-Grabbing Strategies (2:01)Founder Mode vs. Manager Mode (3:11)Technical Debt Remediation (5:03)LinkedIn Posts Discussion (6:05)Cultural Impact on Roles (8:03)Investment in Modernization (12:07)Reflection on Company Strategies (15:03)Gratitude for Data Trends (16:18)Future of Data Access (19:14)Looking Forward to 2025 in Data (21:45)Final Thoughts and Takeaways (22:11)The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

Hi, I'm Eric Dotz. And I'm John Wessel. Welcome to the Data Stack Show. The Data Stack Show is a podcast where we talk about the technical, business, and human challenges involved in data work. Join our casual conversations with innovators and data professionals to learn about new data technologies and how data teams are run at top companies. Welcome back to the Data Snack Show.

We're here for a very special show, a Christmas edition of the Cynical Data Guy. So I'm here with Matt. Matt, welcome back to the show. Welcome back. Here to bring the festivus to your Christmas cheer. Excellent. Yeah, unfortunately, Eric can't be with us today. So you're stuck with myself and Matt, but we've got some fun topics.

We are going to start out the show, which is a record for us, by the way. We're going to start out the show talking about AI. It's unusual. It's Matt's favorite topic. Oh, so much. All right. So one of the things that I've seen a lot, Matt, I think you've seen it as well, is we keep seeing these various forms that OpenAI calls it the 12 days of ship mess. And there's basically there's several different software companies doing this,

where they're attempting to brand pushing really hard at the end of the year, essentially, to get product out. What do you think about that, Matt? It sounds like someone made promises for the end of the year and they have not kept up with them at this point. Then some people that are like, crap, we promised investors we were going to do something. We haven't done it.

Yeah. And it's in such a juxtaposition to like the other the what i would be used to i think what you're used to too is like we're gonna do like we're gonna do code freezes so we don't break anything we'll be on vacation and keep and things are very stable at the end of the year so what how do you think this is gonna work out i i mean i think i think it partially just garners attention is one of the big things but the unfortunate thing is once one person does it and they get attention for it yeah then everybody's going to try to do it next year and it's and it just becomes the new standard and nobody really gets any credit for it right it just becomes oh now we have to ship 12 features every December. Why are we

doing this? Here's an article. OpenAI, let's see, their 12 days of shipments, they shipped the O1 reasoning model, which if you'd like to spend $200 a month on chat GPT, you can. I will pass.

What else did they ship? Apple Intelligence was day five with ChatGPT. My iPhone isn't good enough for that. It won't support it. That's too bad. And a couple of other features, day four. Oh, and then Sora was the other big one. That's their, it can create video.

I still don't have a use case for that in my personal life or professionally at this point. So good for them. Not something I'm going to be taking advantage of. Yeah, that's fair. Okay, so we've got the 12 days of Stripmas. And Matt and I were talking before the show about, and we've talked about this topic before, about this founder mode versus, I don't actually have a good word for the opposite. What's the about this founder mode versus I don't

actually have a good word for the ops. What's the opposite of founder mode? I think they called it normal. Okay. Okay. Manager mode. So we're talking about like this, things like the 12 days of shipments.

It's like, well, we don't want people to like slack off at the end of the year, or maybe we're late on some deadlines, but mainly we don't want people to slack off at the end of the year. or maybe we're late on some deadlines. But mainly we don't want people to slack off at the end of the year, so we're going to set a bunch of deadlines. Doesn't it feel a little bit like the dad who hasn't been paying attention and then he sees his kids kind of slacking off or something,

and he goes, that's it. We're all buckled down. You guys are going to be locked in your rooms. Never coming out until you get an A or something. It had that little bit of feeling of like we're detached and then you came back and you're overcorrected. Yeah. Well, it actually reminds me,

I had this guy I worked for years and years ago and he would intentionally, it was one of these things where we, you know, pretty much a 24-7 operation, but he would intentionally come in like Christmas Day, New Year's Eve, New Year's Day, and just to like, you know, check and see who's working type of thing. And, you know, I don't know. This kind of reminds me of that.

Of like, all right, let's make sure. Let's make sure we got everybody putting in the extra hours. Yeah. The only thing with this is it kind of it's it risks eliminating one of these unknown like not unknown but under the radar things that kind of keep a lot of these places going which is when everything kind of slows down suddenly you can sneak in there between christmas and new year's and all that technical debt you can actually make a dent in it nobody was letting you touch right we're going to 12 days of shit mess when is anyone going to secretly do all the tech debt remediation i mean you never get rid of all of it but you're at least going to pay it down a little

bit yeah now we're just throwing everything on the credit card yeah till next year just gonna keep throwing it on there we gotta pay it No. I just let the interest keep rolling. Yeah. No, I mean, that was something that I actually scheduled. I'd never been on a team that did this, but we actually scheduled quarterly clean stuff up, pay tech down stuff. It was like a week quarter or something like that. We'd do security reviews that nobody got around to. We'd clean stuff up. We'd do security reviews that nobody got around to. We'd clean stuff up. We'd actually pay down some debt

or just review stuff, stop, for a minute. And I don't think many teams do that. That's unique, being able to do that. Most of the time, you get very hung up in the, but I have another data request. Why can't I just do that? Right. For sure.

All right. So it wouldn't be a good episode without some LinkedIn posts, right? Of course. That is what I'm very cheerful for this past season. All the LinkedIn posts that we get to then bring on here. All right. I think Matt's got one for us here. Maybe even a couple.

So this one was out there and it's a very short one. It just says, data engineers can do what analysts do. Analysts can't do what data engineers do. Some fighting word. Man. Tell me about some of the comments first. That does not sound like a dangerous place to be in that comment section. That got some fiery comments, though.

My favorite one was a person who, not in the thread, but separately just wrote, a data analyst can't do what a data engineer can do because it's boring. What? A data analyst can't do what a data engineer can't do because it's boring yeah well honestly i think if you are an analyst a lot of times you look at what they're doing as a data engineer oh i never want to do that yeah it's just the it's the plumbing like that's boring yeah i mean it's different personalities for usually sure but i, as a former data analyst, those are fighting words right there.

Same, yeah. I spent a number of years as a data analyst. Well, and also, actually, here's a really interesting thing. Depending on the company I worked for, I'd be curious if this was true for you, the value of a data analyst was drastically different than engineering. So I remember one of the first companies I worked for,

the analyst, I would say kind of in the middle. And then actually the people above analyst tended to be project managers, which is a little unique. Another company that I worked for, analyst was kind of a lower tier thing. And then there was kind of the traditional IT hierarchy, and analysts were kind of a lower tier thing and then there was like kind of a traditional it heart and the analysts were kind of lower and then a third company that i worked for there was it was just a bigger company so it was like more split out and there were kind of like levels of analysts versus business intelligence you know so it really i think it really depends

on the culture because there is that like there's some cultures where it's like well these guys actually like drive the business forward. They have the business knowledge. Like we value them the most. Like we can just replace the IT people. And then other people it's like, well, I actually like the IT, you know, data engineering skill set. It's really hard to find somebody good.

And, you know, we pay them a lot of money. So we value them a lot. I mean, what have you seen? Well, so we're going to go back to the old, the olden times theater world when i first started i think the most common thing you saw was people would hire data analysts yes i want whatever it is i need an excel monkey i want someone to go just find me a nerd to do whatever all those types of things and then there was this brief error where they said, the first data hire shouldn't be an analyst.

It should be a data engineer. The problem with that was you would hire a data engineer and they would come in here and they'd be like, I'm going to go make the plumbing or whatever. And then they would immediately start getting data analyst requests. Yeah, of course. So you'd have this data engineer who's now spending over half of his time trying to do data handling requests which to be honest most of them were not very good at

sure that was not their thing i have i've worked with some very good data engineers i have also worked with data engineers that literally didn't know what we do as a company sure i mean like you would sit in a meeting they'd be like why are we doing this why do we have all of this information on them on these customers because we're giving out loans do you not understand like why do we even need this yeah why is this all secured because it's personal information about so that i felt like was always a problem i think there's always this temptation to say well we got to do it sequentially right data engineering is the first step we got to do that first and we'll like build up but it doesn't really work because people want

something tangible from it that's why anwra typically you were the first hire right so you've got to kind of it's like this cold start problem you have to figure out well and i think we i think it was the the episode we did with the team from zyletic we talked we went way, way deep on housing and plumbing and analogies between that and data. And I think that applies again here where it's like, okay, we hire a data engineer, the first hire. You can end up with a house with seven bathrooms

all plumbed with three sinks each. Yes. Multiple showers and all the bathtubs in weird places. And the hard part with that is the better the data engineer is, the more they're susceptible to this idea when you say, hey, we're doing this data migration. We need to get this data from a place. And we can set up a staging area and we'll do this.

This is going to last four months. Right. I don't need, this is not a full-term thing right i don't you know you're wanting to lay down railroad track as we're just gonna pick it up behind right it's not gonna be over right now i mean that's actually a really interesting topic because again if we're talking homes like there are situations where like hey this is an rv situation right we literally want to park this here for a couple weeks.

It needs to be livable. It doesn't need to be perfect. And then we're going to move it. And then in data, the category is typically either fully overbuilt, like we're going to build an empire and we want this to last a lifetime. Or a tent. There's not

much in between as far as philosophy right there's the one end which is the we have to eat it right at front yeah and it needs to be the taj mahal that we're doing and then there's the other one that's like there's a canvas over there and i can hang a string and we can do it and and it's neither of those because you need to have something that's a little sturdier most of the time i mean there's also there's different situations required from things but you generally need something that's going to be sturdier but that can evolve right and sometimes that means you have to redo things and that pisses off a lot of people sure who built it like i don't want to build this i'm just going to have to redo things. And that pisses off a lot of people who build it.

I don't want to build this. I'm just going to have to rebuild it two years. I get that. Also, that's the best way to go about this. Yeah, I mean, I think, I mean, the rework thing is challenging, right? Because especially when you're pitching projects or talking to your projects, like that, that'll come up like, okay, we have to rework this later. In the honest, if you keep saying no,

and you're going through a complex project, that's a little bit of a red flag over any large time frame. Because the reality is, if you're going to invest in such modularity and flexibility that you will never have to rework anything, then that's not necessarily the right answer. Nor even if you do invest in that, there's always going to be some amount of rework. And the worst one is when you're sitting on that edge between an old system and the possible new system. I remember one place I worked,

there were clear things that we knew would make our ability to track user data and stuff better. We're going to sunset that out. We're going to have a new app. And like, okay, so when is it going to come? Three to five years. Five years later, they were three to five years from it. And all of these problems had piled up to the point Three to five years. Five years later, they were three to five years from it.

And we were in all of these problems and piled up to the point where it was causing actually customer problems. And it was one of those, like, if we had just done the work, not just little bits here and there over that, we wouldn't be in this situation. Right. Where instead they had to fire a whole team just to modernize this app that they were still working on trying to place. Totally. Well, and the interesting part there, too, is it can be a good strategy. There's a company that I was speaking to recently where essentially the company had been around 20 or 30 years. They got acquired by a much larger company. And they made it.

They made it on sticks and stones and on older technology and this, that, and the other. And sold. And whatever owners were part of that company, was it the right decision for them to just make it happen and keep the lights on for 20 years with bare minimum? And maybe now it's also easy to think through, well, could they have done more if they'd invested more here and there? Maybe too. They may have missed out on some things. It's hard to quantify.

It's always hard to quantify, which makes all of these things tough to justify. Because these modernization efforts usually happen after a hard loss not like an opportunity loss yeah that's yeah that's very true it's usually once you there's a perception you get a wall of what you can do yes or that there is some feeling that's holding you back or you get new leadership in and they have that moment of where they're horrified you're doing what no we have to stop this right right but it always has those risks it never goes smoothly and sometimes the right end the right answer is not keep it or get rid of it it's kind of like we talked about a previous episode follow it out right and just use the frame of it and just put everything else together on it.

Yes, definitely. All right, so we've got to talk about the festivus and the cheer and the Christmas. So we've got the 12 days of Shabbos. Yeah. What are 12 things? I'll do that for you. What are 12 things that you're grateful for this year no what are one or two things that you can look back on um that you're as festivus or chair

whichever category you'd like to choose but i'll say one thing that i'm happy for is the final kind of break away from all of these sass walled gardens and putting the warehouse kind of like in the center yeah i always hated working on it when it was like oh we got all this data and it's in a sas application and they won't let it out i hated that yeah so move it to the warehouse i'm very grateful for it comes with a whole bunch of other challenges but i'd rather deal with those held hostage yeah by the vendors that's a good one other thing i am you know we're now a couple years away from the peak insanity of the COVID, tech evaluations, stuff like that.

We're seeing more and more tech companies are having to act more like real companies now that money is free. And I would just be like to show my appreciation for all this, just say, welcome to my world where money doesn't just fall out of the vents every time you want to do something. Well, I mean, if you're an AI, I would argue that it's still kind of falling out of the sky.

But other than that, yes. Yes. But I try to pretend that's not real. No, you pretend that's not real. I know you pretend that's not real alright my number one I would have to agree and even kind of expand on there's an awesome

trend and data of A companies like centering what they're doing around a warehouse but like broader scope than that just seeing all the growth in open source data formats like Iceberg, for example. AWS released some cool things with S3 and Iceberg at the conference this year. And then all the other major vendors are playing with it as well. But I think that's

a really exciting trend where if there's some future where essentially all of a company's data can live in some commodity storage. And then applications that get access to it, there is a flip of like, I have my data and I allow application access to the data versus like I store all of my data with an application. And they're required by law to give me the data if I leave. But it can be one CSV file at a time if they want it to be. Or put them all in an XML. I hope that philosophy keeps trending in that direction where it really is more about we have all of our company data stored together and we're allowing these software vendors to to quote use it or to be part of their product

and then when we're ready to leave we just cut off access we're not like quote migrating necessarily i think that would be a hugely positive thing yeah i think the big key for that will be having the open formats not be kind of co-opted yeah i'm sure you know oh we've got 17 vintages of iceberg depending on your different flavors that are not really compatible with each other right you know well they have it but you can only use their catalog it doesn't play nice with anybody else and those types of things yeah this will all be great as we kind of decouple the warehouse up until we reach the absurdity point of that and then someone sells a completely coupled warehouse right right or or just a coupled you know solution that includes you know x y and z you know other solution that includes, you know, X, Y, and Z, you know, other things with the warehouse.

Right. So which is the trend, right? Like this is the type of thing that will get decoupled, further decoupled than like regrouped together. Which my absolute pet peeve frustration of the example of this is cable TV and streaming. Like there was such a hard sell of like cut the you know cut the cord save money move to like streaming platform and then essentially we're at the point where everybody's

actually paying more than they ever would have and you have to pay one per thing and it's insane we're seeing consolidation and there's attempts at more of it so they will have to pay for a streaming service you will have to pay more money than you want to and it'll come with a hundred channels or content you don't want you don't want or right back to cable we just eliminated the cable box right right right which is wild like it's just not not something i mean it makes sense retrospectively but it's like man we really man, we really failed on that initial promise. You know, pay for what you want to use, et cetera. All in part, it's going to be great.

Oh, wait, no, it doesn't work. It doesn't work. And yeah, my fear is that some version of that very well may happen with a lot of this data space deconsolidation. But I don't know, maybe it'll be different this time. Maybe. I think as we look forward to another year of commentary on the ridiculous things that happen, the fun stuff and the ridiculous things

that we get out of data, I think it'll be an interesting time all around, especially with all the stuff going on in the world. As they say, may you live in interesting times. Well, that's not a problem. That won't be a problem. All right, thanks for joining us. Merry Christmas.

Happy New Year to everybody. Happy Festivus. Stay cynical. See ya. The Data Stack Show is brought to you by Rudderstack, the warehouse-native customer data platform. Rudderstack is purpose-built to help data teams turn customer data into competitive advantage.

Learn more at ruddersack.com.