The Data Stack Show - 209: Storytime with Cynical Data Guy: Data Projects, $50K Web Scraping Fails, and the Role of CDOs

Episode Date: October 2, 2024

Highlights from this week’s conversation include:Previewing the Next Cynical Data Guy Episode (0:13)Story Time: Coolest Data Project You’ve Worked On (1:13)Failed Web Scraping Project (3:40)Building a Neural Net for Matching (5:22)Rebuilding the Project Strategy (7:04)Project Completion and Politics (9:35)Agreeable Data Guy's Pricing Story (11:00)Balancing Advanced and Simple Solutions (14:15)Insights from Pricing Team Meetings (16:19)Building for Scale vs. Immediate Needs (18:29)Open Source Data Formats (19:46)Disaster Recovery Experiences (22:34)Reflections on Chief Data Officers (25:01)Cynicism in Data Projects (28:19)Final Thoughts and Takeaways (30:20)The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

Hi, I'm Eric Dotz. And I'm John Wessel. Welcome to the Data Stack Show. The Data Stack Show is a podcast where we talk about the technical, business, and human challenges involved in data work. Join our casual conversations with innovators and data professionals to learn about new data technologies and how data teams are run at top companies. Welcome back to the Data Stack Show. It is officially October, and so we're going to start out Halloween month with a Cynical Data Guy episode, but a

special edition Cynical Data Guy episode, which we're calling Story Time with Cynical Data Guy and actually Agreeable Data Guy. I got some positive stories to add to the text. Yeah, yeah, yeah. We have to counterbalance. We have to counterbalance. And then, of course, we'll end with actually just a single round of LinkedIn lightning round.

And I've got a good one. So, all right. So let's start deep in the bowels of corporate America, which is usually where we like to start on these shows. Okay, here's story time. Here's the story time question. What's the coolest data project you've ever worked on? Okay, so I'll go first.

So probably the coolest project I ever worked on also entered in me no longer being at that company. Well, this is like a therapy session where you're understanding, you're starting to understand some of the like formative experiences that shaped the synthesis. Exactly. So the really short background on this without going into many details is I was running a team that was kind of doing like internal consulting data science work. And then we also had, they brought in someone to be the VP of innovation, which by the way, if you ever go to a company that has a department of innovation run away, that's how innovation works. Yeah. And he had early on tried to recruit me into his group,

but then I saw what he was telling the executives and his timelines and requirements for how we would have to build, which were like 10 years old, and I rebuffed them. He did not like that. So without going too much into the details, we had this company.

One of the big things we did was we collected a lot of pricing data. It was all manually done at the time. So it was literally a team of like 12 people who would either call up suppliers or would go online and search for specific things. That is brutal. It was really bad. Caused a lot of problems. It made some of our claims a little questionable that we told customers, stuff like that.

So there had been a project in the innovation team to turn all that into web scraping. Yep. Okay. You've said enough with web scraping. You can stop right there. So they had made this big promise. Yeah, because going from manual to web scraping isn't

that doesn't go any problems the manual with more complexity well yeah so there's so many things that were wrong with this but so he made a promise if he was going to put in you know it was some ridiculous number like a hundred thousand prices into our database in like six months six months later i get a phone call from my boss hey can you help us and it turns out that in that time they had though they had claimed oh we've scraped two million prices they had matched and gotten into our internal database exactly five oh my wow that's like yes and i'm like and this wasn't a project where it was like oh they had a little bit like they had gotten this whole group had gotten like over a million dollars to fund them type thing so that's one of those you know you that's one of the i call these moments oh yeah

like movies are based on you know like reality movies are based on, you know, like reality. Movies are based on reality. You know, like it's just hard to believe that. So we did some work, some really preliminary work to just get the number up because there was some KPI that they were kind of under the gun to hit at that point. And then it became a, this project isn't working over here. We want you and your team to take it over, right? So my half a million dollar team in total was taking it over from this seven million dollar team. We got into the details of it and it was one of these things where they weren't targeting prices. They were literally paying a company

to scrape entire websites, like every price from the website. And the person that kind of played it off is like, look, I've price from the website. And the person that kind of played it off is like, look, I've done the hard part. I have a million prices. Now all we have to do is match into our database. Which, if you've ever done this before, is like

web scraping is relatively simple. Matching them is insanely hard, especially when you don't have good names in your database. So we had gone through a couple, this was one of these where like every turn they like kind of tried to cancel us and stop the project

or the guy who had had it before tried to convince the executive that our method would never work. So we had started with, this was pre-GPT and all this stuff. We were going to build a unsupervised language model using you know like just neural nets and stuff like that we then had to prove that our method would work

which ended up being we had a we got a group you know we had a labeled set so we built a neural net a pretty simple one not huge that could match and got to superhuman ability and matching on this stuff. Nice. A couple of turns later, we're still trying to figure out, and I'm like negotiating with the web scraping company we have because they were spending, I mean, they were literally spending

like $50,000 per site, per scrape. Wow. Because everyone was a custom project. Right. Jeez. They're probably still getting great margins. Yeah.

I'm sure. Yeah. I mean, we're on the cynical data guy show. Right. Jeez. They're probably still getting great margins. Yeah. Sure. I mean, we're on the cynical data guy show. So I was talking with the guy and I had talked with other people and it was one of those things where like our use case really isn't what web scraping is for. You're supposed to know the product, know the page and we're like, we want to discover

more stuff that we know is out there but no one's found it yet. Right. And so it was one of those i'm in a meeting with this guy and he's trying to convince me to stay with them because i'm getting ready to fire him yeah and he's like well you know what you could do is search for it and pull the top like five or ten results and it was one of those moments where you go that's useless unless i had some way to like automatically match it like a model that's superhuman and matching this stuff and it clicked at that point and so we rebuilt like our whole thing around this idea we also

had some kpis it was one of those like you know this was an okr at the beginning of the year so therefore it had to be done even though we've got eight months of information that says this is really stupid and we shouldn't do this. We still had to hit it once. So we had convinced the team to basically, in the process, we also proved this team was super inefficient because they claimed it took like four months to do all this research. We got them to redo all of their online research in like six weeks. And because we were like, we need the web page,

we need a better description, blah, blah, blah. And that was all just going to feed into this thing but it was also one that we realized we could use the apis for them that when we wanted to say hey what because you could set up agents on websites basically right that we could say hey we've got this new thing we need to get pricing data on okay what is it oh it falls into x category, now we can hit these APIs on them, right? And now we can have it then automatically use this. Because it's like computationally expensive to try to match it against everything in your database.

But it's still faster than doing it by hand. Right. And there were things for me to move from Windows machines to Linux machines because Windows doesn't fork memory and all sorts of stuff like this, which was a big deal at that because they were a Microsoft shop through and through. But we got it to the point where we were on the cusp

of basically like we could go find stuff on demand. And it was more robust than web scraping because by searching, yeah, products are going to drop off of these websites, but other things are going to come back on. So we're not dealing with broken links because we're always researching and scraping. That's researching. Oh, yeah.

That's a great point, yeah. Because broken links are huge. When you say searching, are you actually using the search engine of the website? Yeah. The thing that's made for finding products? Yeah. Brilliant.

Yeah, and the scraping, the broken links thing, it's gone. You don't have to worry about it. And on top of that was the thing of every time we did it, if it got accepted, it would go into our database of like acceptable descriptions of these different products because they had slightly different names. Yeah, sure. We're dealing with stuff with like, you know, that had measurements to them and we could kind of group them all there. And so it was the type of thing that when you put it in place, it's going to get better over time.

Right. And we were going to and we were planning on basically having it where if it was in a certain threshold, it gets kicked over to one of the people to actually say, okay, is this a match? And so it should be robust. It should get better over time. It had all these great things to it. So I got the project from completely off track to we hit every KPI by the end of that year. And then two months later, that VP got me pushed out.

They eliminated my position, disbanded my team, and he took back over the project. But he took it over two months too soon, so it wasn't ready yet. Oh man, timing's everything. Yeah, so I got pushed out, which was an interesting

soul-searching experience to go through that. So why are you cynical? Please explain. I don't understand why you're so cynical. You do have to give the guy credit, though. That guy was kind of innovative, just in a very nefarious way. What do they call it? The coda or whatever the epilogue of that story is.

Because I ended up going to another place and learned some important lessons for myself just on my own ego and things like that. So it made me a better, the experience made me better in the long run. That particular individual was unceremoniously fired almost one year later. Oh, okay.

With cause, if the story I heard was correct. Wow. Yeah. So it was, that's probably, it was correct. Wow. Yeah. Yeah. And so it was, that's probably, it was one of the coolest things. We didn't get it completely to production, but that was mostly because of politics pushing me out. Which is probably true way more than any of us would like to believe. Yeah.

And then when he took over that project, because all the people who were on my team who were still doing things in there, one of them got placed under him, they would send me updates and I'd be like, yeah, he completely scraped everything he did and he's trying to do it the old way that doesn't work Oh no I don't think that project ever got off the ground again So I actually have my own pricing

story, this wasn't what I was going to say, so I'll get you like a cool project because there are two. You have the embittered cynical data guy and story time with the agreeable data guy. So there's two categories of cool projects. Number one category,

which is probably the most important, is like, hey, we have this outsized business impact. Number two category is like, this tech is really cool. So I got one of each. Okay, great. My first one is a pricing project. Ironically enough, we didn't talk about this beforehand.

But round one, so it was an e-commerce company managing tens of thousands of SKUs. And I was like, okay, we need a way to stay up with the competition. So a lot of it was distribution, which means you have multiple people selling the same thing. So your price is like the thing that you went on or lose on. It's like, okay, so we go down that road and we're like, oh, look, there's like this company that does it. And this company that does it, that can scrape the price and like sync it to your database.

And we're like, okay, great. So we like go down the line with several of those companies. We trial with one and then was like, all right, that didn't work at all. We trial with another one and see some early signs of working. But it was one of those nebulous, we take in all of this signal from your page to help guide the price.

And I was like, okay. We got some early results, rolled it out far broader than was wise help guide the price and we did it i was like okay yeah we like got some early results like rolled it out like far broader than what then was wise and then really screwed up our pricing from that one and then went with a more like like so that was more like signal based more like a like web scraper based thing yeah and that only got like marginally. So I was like, oh, what are we going to do? This is a problem. So what we ended up doing was just very simple of hired an analyst for pricing and then basically had three or four sources.

Amazon was a source. We had a web script thing that was a source and we had a couple other sources. And then we'd have basically reviews of a couple like thousands at a time we kind of we had like queries and things that ran to like produce these data sets and we made it very efficient but an analyst would like scan through it and then decide like to set prices and like it worked like fairly well like to the like i think we ended up increasing there's a

couple other factors but this was a big driver to increasing margin, like 11 or 12% over the course of a year or two. That's crazy. Yeah. It was crazy. Some of it was a product mix thing where we're moving to more private label, which helped. And then some of it was just pricing because it was, you know, because it was not systematically managed. And when you get to like 20 and 30,000 plus SKUs, I mean, you're going to have a huge impact yeah so that's my cool business outcome like story i'm like cool on the tech side which is just as fun working with a client right now it's not as fun for that analyst no i mean the poor

analyst it was somebody that it was their first job out of school so they didn't have a lot of like i don't know what working's like. Maybe this is what all jobs are like. We were smart with the hire. Yeah, yeah, yeah. I actually hired somebody with a math degree, which was interesting, and they weren't there for like a super long time. I think that is a good example there.

Even just thinking about those two stories, that's a really nice, those stories contrast well to highlight the fact that sometimes you do need to build something pretty advanced. Yeah. And sometimes you just need a really simple solution for some subset that's going to have the highest impact. Yeah. And a lot of people overlook that, the simple solution. Yeah.

Can we just have a person look at this? Yeah. At least at first. Let's just like figure out how this works first yep if it's brute force work that is hard to scale from the human capital standpoint because it's hard but yeah that was interesting that was a good but the very least if you do that even if it's something you're like we'll never be able to scale this you will learn intelligence that's a great about the process about what is, rather than just doing the blind. We're going to throw data in this pool of machine learning

and stir it around and see what comes out. Because that's when you get like, oh, it worked for a month. And I think it's important too. There was a push. I mean, this is from even the start of my career around automation, which is a big thing. But I feel like now a lot of times projects get scoped around like,

hey, let's do this thing. And nobody even knows what the manual process is. And that is a tough place to start if you want to start right into like, oh, we want this ML thing. We wouldn't even know how to do it in a spreadsheet. Yeah, how are you going to do that if you've never done it in a spreadsheet? And I think that's also depending on the analyst or like the data scientist or whoever you have, there can be a little bit of where they want to jump right into that. They're like, oh, let's make a decision tree or a neural net. And it's like, let's how about we actually go through what this process is and what it looks like to try to do this right i mean when i was you know i ran a pricing team at an auto finance place and one of the things we would do every month was we would

actually sit in a room and you would look at loan app you know like so you'd see the application you'd see what the actual you know what they're buying the car is what they're buying it for and you had to use that to try to predict like we were like very deep subprime so we're predicting how many payments have you made the total that you yeah but it does help give you some intuition over like okay these are things that matter these are things that don't and it does make a difference compared to we hit people who were on you know more of the they were the modeling side and they would do things and we would look at them and go like like, you can't use that.

Like, that variable you're using is something that gets backfilled later. You can't do that, right? Or like the idea where we would say, I don't, you know, the things you're looking at are not available at the time you're making this loan decision. But they don't know that because they're not sitting in these meetings to see that. Yeah. I was, I'm working on an old car car and i had a new windshield installed in it and i don't think they did it correctly and i thought that i had a slight leak but i wasn't it was hard

to tell so i thought okay well i'll just find someone who will pull it back out and like you know sort of reseal it or whatever. Anyway, so I called, it's an old car and so not a lot of people want to work on it, right? Because they're like, you can't get the warranty, blah, blah, blah. So I need to find someone who can do this. Anyways, I called this guy who's, you know, local and he, it was a very short conversation because I explained like, hey, I don't think this is right. Can you pull it out and seal it? And he's like, are you getting water in your car? And I was like, not really, but I don't think this is done correctly. And he, this is great. I love it when he's like, when people say like, son, you know, it's like, I'm about to get some good

advice. He's like, son, if you don't have a problem, don make a problem and i was like what i like was quiet for a second i was like oh yeah you know what that's exactly right you know what like i feel like that with tech right where it's like if you don't have it like you can invent a problem by throwing technology at it unnecessarily right When the simple solution in many ways can show you whether or not you actually have a problem. Yeah, or you can do the thing where you're trying to build it for scale when you're like,

hey, this is for an implementation of an ERP system, so we need it to work for three months. Yeah, yeah, totally. So I don't need all your fault tolerances. I don't need the pipeline. Yeah, yeah. I just need you to take this spreadsheet and turn it into this table

that we're then going to match against another table and be done with. Yeah, I know. Sorry, we interrupted another... Okay, cool tech. Cool tech. That's a cool tech segment. Yeah, so one of the cool projects I'm working on right now, Quiet has

almost all their data in Snowflake. Most of their data in Snowflake. most of their data in Snowflake. And going through, they have some fairly rigorous requirements around disaster recovery, one of which is if you have data in a SaaS system, you can't just trust whatever the SaaS system's DR is. It needs to get out of that SaaS system, which as a policy is probably reasonable. But most people would be like, I don't know, like Snowflake probably has that covered.

Yeah. Right? Yeah, yeah, yeah. Or you just buy the enterprise plan and you get like, you know, the extra stuff. So I'm sure they do

have it covered, but for their internal requirements, like this is what they do. So like, okay, so we're taking backups from Snowflake to Iceberg, pressing an S3. So the data's landing there.

And then if you've been following the space, Snowflake has a data catalog that sits on top of Iceberg. Yeah. Which, if you're familiar with databases, the way this works is a traditional database comes with storage, tables, a schema, which is a catalog, and then all the security and stuff wrapped around that.

So like as you get into these like newer open source formats, like everything's kind of broken apart. So you've got your, like for Iceberg, you've got Iceberg as a table, but the files are actually stored underneath in Parquet usually. And then you got your table, Iceberg table, and then the layer above that is the catalog layer. So we're doing all this, which has been really cool.

Snowflake has some neat support where you can sync tables with the Polaris catalog and then basically access the data through Polaris from Spark or from Trino or like the thing we're trying is the hosted version of Trino, which is Starburst has support. So in essence,

like you've got this like read, potentially read, right. But at least for our case, a read path that goes from the, let's say Starburst, Starburst, like all the way into the data and the S3 bucket,

you either using the Polaris hosted catalog, or I think Starburst is the one that, that supports you querying the tables just directly. So A, like for DR, it's kind of cool. B, for data sharing, say you're on like Snowflake, somebody else has some other like Databricks or some other technology. It's kind of neat. And that's an application we're going to look into. Oh, yeah.

Create these catalogs. Yeah, where somebody can just use Spark or some other dialect that they want to use to create the data. But yeah, that's been one that's been really interesting. And I see like a future where people go to like, oh, how about we just park the data here and government has access to it instead of just moving it around all the time.

Yeah. Okay, two interesting comments on that. One, Andrew Lam from Influx talked about this. I was, I can't remember. Brooks, we can put it in the show notes, but this is several months ago. But he talked about this exact thing where you are going,

he was saying that you're going to see this shift and all sorts of interesting technology proliferate around it because of the advantages of flexibility, the cost, everything. Right. Also, trivia, Costas, the former co-host, I believe helped name Starburst's cloud product. Oh, Galaxy. Galaxy. Yeah. That's cool. Galaxy. Starburst Galaxy. Yeah.

That's cool. I think that's true. Costas, I know you're listening because I know you still listen to every episode. So we'll have you back on to verify the truth of it. So one of the things that just made me think of, because you're talking about like kind of a data recovery or like disaster recovery.

So my dad worked in state government for about 30 years. And he did like, I mean, so to give you some background, when my dad got his computer science degree, it was punch cards. He lived through punch cards. He moved to mainframes. By the end of his time, because they weren't, state governments aren't going cloud, right, at the time. He was doing, you know, he was in charge of like provisioning VM servers,

which was also databases and all that stuff. So they had to do disaster recovery every year and test it in case you don't know it most states is a nightmare i'm sure because it all involves if there's a disaster you basically dump everything into a third-party site so it's not like we're proactively doing it because i mean this is you know it's tax returns social're proactively doing it. Cause I mean, this is, you know, it's tax returns, social security numbers, um, all sorts of things, you know, unemployment benefits, stuff like that. So they have a whole process of where it's, you know, at the time, at least where it was like, all right, you know, they'd have to, this is up in like Massachusetts. They would have to drive out to like Connecticut. It would be all right, we're going to go through

this idea of some disasters hit and all this stuff has to basically be pushed to Connecticut and then rebuilt again. They didn't like back up to tape, store it in a vault and then drive the tape

to Connecticut? Nope. Okay. Because I've heard those stories where people, even like more recently, people still like backing up to tape

and then like driving to a vault. Yeah. And that's where their backup is. No, I'll... Which the joke's on us if something major happens. Those people have their data.

One other fun fact I'll say about this just to brag on my own dad here. So when he retired a couple years ago, they had hired two people to shadow him for a year. So two people are replacing him in this job. So each of them had about half of his job.

Within three months, one of them quit because the stress was too much to do half of his job. Wow. And he literally would do the crossword puzzle in the middle of his work day. Amazing. That's awesome. Okay, I think we have time.

That was a great story, Todd. Maybe we did that to kick off Halloween, but maybe we need to do that more often. Okay, LinkedIn hot take. I'm just going to read, since we are close to time here, I'm just going to read an excerpt from the article that this LinkedIn post linked from. Author will go unnamed. And I'm

just going to dive in here. Considering that as recently as five years ago, the chief data officer role was hailed as a critical addition to the C-suite, what went wrong? How about that for a spice? This issue goes beyond budget cuts and raises important questions about the definition of the CDO role, the management of data within organizations, the future of executive committees, and the impact of generative AI. As former chief data officers ourselves, we aim to explore these topics through a series of articles with the goal of catalyzing a broader discussion, given the significance of the shift. Clearly, a key driver has been the financial pressure organizations have faced over the last 12 to 18 months. As executive teams have scrutinized

budgets, frustration is mounted regarding the lack of impact of data initiatives on operating margins. This is concerning given that companies have invested tens if not hundreds of millions into building data capabilities and acquiring platforms. As one retail executive remarked to Ryan, all we have to show for our efforts are prettier dashboards. Sorry, I couldn't get that out without laughing. All we have to show for our efforts are prettier dashboards. While few chief data officers have considered profitability one of their KPIs, they are certainly being challenged on it now. Okay, time out.

Few considered profitability one of their KPIs. What did you think your job was? Hi, we'd like you to spend $20 million as fast as possible. I'm sorry. That's really the attitude of a bunch of CDOs. That position deserves to go away. Well, that's what Edward talked about recently on the show, right? Yeah, with the P&L.

Yeah. He was just like, if you don't own a P&L, if you don't own something that&L, you know, if you're not, if you don't own something that's directly contributing to the bottom line, I think his words were, you're in the kiddie pool. Like,

your job, it's a fake executive job. Yeah. And then they, then you try to make these, you know, okay, well,

we're not contributing to profit or anything, but we're slowing the growth of our cloud computing bills and stuff like that. It's, I don't know. Like,

I, early on because one of the one of the kind of like chief data officers i worked for was a guy who came from the sales and product side it's probably the most effective cdo i ever worked for so i have it in my head in when i'm here like how am i contributing how am i contributing where you're going and so i've been in that situation where you're sitting there and you're looking around and you're like, I don't see where this is going towards the bottom line.

And I'm getting nervous about this. The answers I'm getting from the people above me are not making me feel good right now. Well, that's a tough situation to be in, especially if you're a couple like rungs down, because you can see that. And especially if you're a couple rungs down. Because you can see that. And especially if you're young, right?

It's like, I probably just don't see it. Like, I don't think this is working. I don't think this makes sense. But like, I don't really know. Like, maybe I just don't have the full picture. And then like when you've been around long enough, you're like, oh yeah, no, that's a thing. Like, yeah.

You came up much more humbly than I did. I was in my first job and I turned and I went, this all looks stupid. Why are we doing this? Which is why we call you the cynical guy. Yeah. I will say probably it's easy to be cynical about these CDOs, but my guess would be a lot of them were brought in to do some major data projects.

They got a ton of funding based on a number of factors, right? Like low interest rates, easy money, a lot of margin and budget to like do cool things and take advantage of new technology. So in essence, they were kind of like innovation officers. Coming full circle. Wow. But they most likely, the company did not actually have a plan. It's more like, hey, we have this pool of money over here. It's data. It's the like, hey, we have this pool of money over here. It's data. It's the underpants dome problem, right?

Okay, we've got $50 million for data, question mark, big profits, right? Like then they're like, oh, you're going to help. But I will say, I think the other thing that hurts them in that situation is a lot of times in order to do those jobs effectively, you're going to need someone who's going to like drive a certain amount of change. But the problem is a lot of times you order to do those jobs effectively, you're going to need someone who's going to drive a certain amount of change. But the problem is a lot of times you get in there and everyone says, we really want to change. We really want to do this.

Okay, cool. You get in here and you're like, okay, we need to change X, Y, or Z, or we need to not do these types of things. And then you get that, yeah, we really, I'm totally with you on changing, but don't actually change anything. And it's tough.

I mean, I almost feel like CDOs should probably get like six-year guaranteed contracts the way that college football coaches do. Because at some point, I'm going to push back on something and you're going to be like, well, I don't really want to do that. Yeah, that's a great point. That's a great point.

All right. So we've heard sad stories, happy stories. The story was not sad. Cynical stories. I'll call that. That is true. That is true. Although you did tell a great story about your dad.

That was awesome. Yeah. It's a happy story. We talked about disaster recovery. Yeah. All day's work. All right. Happy Halloween early from the Data Stack Show.

Plenty more great episodes coming your way. Subscribe if you haven't so you get notified about new episodes and we'll catch you on the next one. Stay cynical. The Data Stack Show is brought to you by Rudderstack, the warehouse native customer data platform. Rudderstack is purpose built to help data teams turn customer data into competitive advantage. Learn more at RudderSack.com.