Freakonomics Radio - 440. Does Advertising Actually Work? (Part 1: TV)

Episode Date: November 19, 2020

Companies around the world spend more than half-a-trillion dollars each year on ads. The ad industry swears by its efficacy — but a massive new study tells a different story.

Have you ever been puzzled by something that's supposed to be true, but you didn't quite believe it, and you didn't have the evidence to challenge it? But then, one day, the evidence appears? Today is that day. This is what you've been waiting for. Today is the day the long-standing puzzle is finally solved. It's amazing. It's a puzzle about something you encounter all the time. Liberty, liberty, liberty, liberty. Every day, we are each exposed to hundreds,

even thousands of advertisements. Liberty, liberty, liberty, liberty. It's a number that's grown exponentially thanks to the internet. Liberty, liberty, liberty, liberty. It's a number that's grown exponentially thanks to the internet. Liberty, liberty, liberty, liberty. In the U.S., more than $250 billion a year is spent on advertising. Liberty, liberty, liberty, liberty. Globally, the figure is more than half a trillion dollars. Liberty, liberty, liberty, liberty. So it would seem there's a basic question worth asking.

Does all that advertising actually work? From Stitcher and Dubner Productions, this is Freakonomics Radio, the podcast that explores the hidden side of everything. Here's your host, Stephen Dubner. Steve Levitt, my Freakonomics friend and co-author, is an economist at the University of Chicago. Occasionally, he will get a call from a company that wants his help solving a problem. Exactly. He once got such a call from a big box retailer. I ended up flying to the headquarters of this company and sitting down with them.

I said, OK, so what's the problem? And they said, the problem is that we spend almost a billion dollars a year on advertising and we don't know whether it works or not. I said, OK, what do you know? And they put up these PowerPoint slides and they were some of the most beautiful PowerPoint slides you've ever seen. These slides seem to show the value of the firm's advertising, but Levitt was skeptical. I had thought a fair amount and failed a number of times in my academic studies to really understand a causal impact of ads on sales, because typically there's nothing like a randomized experiment going on. So teasing out the causal part, the sales that wouldn't have happened absent the advertising, it's just a really hard problem. The executives told Levitt that there was one thing they knew to be true,

that the TV ads they ran were much more effective dollar for dollar than their newspaper ads. They also said they had been advertising in every big Sunday newspaper in the U.S. every week for the past 15 years. And I said, I suppose you advertise every day of the year on national TV as well? He said, oh, no, no. We've really only advertised three times a year on TV. That's because TV advertising is much more expensive than newspaper advertising. We have a really big push right before Father's Day, and then we advertise a bunch on Black Friday right after Thanksgiving. And then in the lead up to Christmas, of course, we do an enormous split. So, of course, there was a correlation between the TV advertising and store sales.

But it's not necessarily or even primarily because of the ad. It's because the company knows when the big selling days are and they target the ads around it. Levitt did try to analyze the data the firm gave him, but because the company only ran TV ads exactly when customers were already planning to buy a lot of stuff, it was impossible to disentangle. I went back to this company and I said, I'm really sorry to say, but with the data you have, with nothing like a randomized experiment, it's just possible that the return on investment could be anywhere

from zero to infinity. Levitt did offer to help the company run a randomized experiment. Their newspaper advertising would be perfect for that. Since those ads ran everywhere, every week, they could stop running them in certain markets and measure the effect on sales. And they said to me, are you crazy? We can't turn off the newspaper ads. One time we hired this summer intern and his job was to do the newspaper inserts for Pittsburgh. And the guy was so incompetent that he just didn't do it. And when the CEO found that out, he said, if you ever do that again, you're all fired. The Pittsburgh blackout lasted an entire month. So I said to them, well, okay, but when you looked at the results, what happened to the sales in

Pittsburgh when you were dark for a month? And they called me back about a week later and they said, you're not going to believe it. We looked at the data in Pittsburgh and we saw no impact on sales when they didn't do any inserts for a month. So I said, oh my God, that's amazing. Okay, so when can we get started? Started, that is, with a wide-scale experiment to replicate the Pittsburgh accident. They said, are you crazy? It was almost, if they found out they didn't work,

it was far worse for these people than it was not finding out it didn't work. Because then they had to explain why for the last 15 years, they had been wasting $200 million a year. So they were happy to just live in a world in which as long as there were ads in every market every Sunday, life was good. So economists like you are always telling the rest of us that firms are, if nothing else, profit maximmaximizing animals, that they really know how to spend money that's going to help make more money and to not spend money that's wasted. So any economist that tells you that firms are profit-maximizing

has not ever worked with firms. That's a simple model we use when we teach beginning economics because it's easy to solve mathematically. But the realistic picture is that firms are composed of people and all of the foibles and shortcomings that people exhibit in their everyday life, they bring those to work with them. Now, why should any of us care that a company like this was spending so much money on something that was apparently ineffective? After all, it helped support all those newspapers, and goodness knows they need every ad dollar they can get these days. But if you are, say, a retiree who owns stock in this company, well, you can't be very happy

about this. And if you are a customer who buys stuff from this company, you probably aren't very happy either. Because who do you think ultimately pays for all this advertising? That's right, you do. So don't get me wrong. I'm not implying that advertising doesn't work. I'm implying that we don't have a very good idea about how well it works. So let's try to figure out how well advertising does work. Let's start with the most generous assumption possible, that it's 100% effective. Well, you can't be 100% effective with anything.

I'd like you to meet Keith Weed. But companies would not be spending the money they're spending on advertising if they didn't, first of all, believe it worked, and secondly, could quantify its measure. Weed sits on the board of several companies. Including WPP, the world's largest advertising and communication agency. And until last year, I was the chief marketing officer of Unilever, the second largest advertiser in the world.

Even if you are not familiar with Unilever per se, you are surely familiar with their products. Every day, 2.5 billion people use a Unilever per se. You are surely familiar with their products. Every day, 2.5 billion people use a Unilever product, and they're in about 197 countries. Very much a mass market consumer goods business with brands like Dove Soap, Lipton Tea, Ben & Jerry's Ice Cream. And I see you're also the president of something called the Advertising Association, yes? Oh yes, I'm that as well. Yes, I'm president of the Advertising Association, which gives me some credibility to be talking to you today.

So not much of a spoiler alert here, but Keith Weed is unabashedly pro-advertising. The fact that Coke and Dove and Ford have been around for decades, and the fact that companies like Unilever spend billions suggests that maybe advertising does work. The fact that some companies have been around a long time and spend a lot of money on advertising may suggest that advertising works, but does that constitute proof? I asked Weed what share of advertising dollars currently go to TV. What advertising wants to do is engage with people where they're spending their time. And it's hugely different in countries around the world. So

in the US, you would find more than half of advertising dollars going on digital. But if I took you to another market around the world, you would still find your TV being very strong. So things are changing. But TV is still hugely important in building broad mass reach propositions. Can you give an example of a brand or product that, let's say it's brand new right now, that TV is a must? If you're going after a very targeted audience, let's say you have a premium wax for surfboards, going on television, that would be a waste of money because you'll be advertising your premium wax to a whole load of people who aren't at all interested in surfing. What digital enables you to do, of course, is not only go after surfers, but go after people who are interested in a premium

wax on your surfboard. The sort of products that TV is still very powerful for are indeed consumer goods and things. Things that everyone uses. Yeah, exactly. Cars and soft drinks and foods, etc. So it's where you're looking for broad engagement. There's a famous old quote that I'm sure you've heard attributed to John Wanamaker, the department store merchant, who maybe said, half the money I spend on advertising is wasted. The trouble is, I don't know which half. I'm guessing that's not the kind of message that you as the chief marketing officer of Unilever would have wanted your CEO to think about. Well, I think the time when that great expression was said, there was probably a

lot more truth to it. With the amount of money in advertising, the quality of measuring advertising has gone up every single year. Is that measurement usually done internally or externally? In other words, if I see that, do I feel, oh, that's an audited estimate of efficacy? Or is this the chief marketing officer telling me that his crew has analysed this and determined that the efficacy is relatively high? So the answer is both. There are things that you can get from panel data and measures that are accepted by the industry. Having said that, what every company would like is a level of differentiation, which makes them that little bit more competitive. And there's been a lot of science being put into that over the years.

Now, one of the things that I spent a lot of time developing was the whole area about brands with purpose, brands that matter, whether that be Dove and Real Beauty and challenging the beauty industry or Ben & Jerry's and the work done around social justice and climate justice. And building brands with purpose, we believe, gave us a competitive advantage against other brands. So if I were to say to you, Keith, that a trio of academic researchers in the States did this massive analysis of consumer packaged goods, and they found that, quote, the vast majority of brands over-invest in advertising and could increase profits by reducing their advertising spending. If I were to say that to you, you would then say what?

Well, I would say that I'm sure you've got exactly those group of people because I think everyone loves to have a theory about advertising. We do have exactly that group of people, one of them at least. Her name is Anna Tuckman. Hi. Yeah, I hear you great. Tuckman is an associate professor of marketing at Northwestern University's Kellogg School of Management. And I study the effect of television advertising, as well as research questions that lie at the intersection of marketing and public policy. Considering how long advertising has been around, you would think

we would know pretty much everything about advertising there is to know. How wrong am I? I mean, you're exactly right. Advertising has been around for a long time and researchers have been interested in studying the effects of advertising for a long time. There's actually some really nice work on the psychology side of how advertisements work. But in terms of linking ads to actual purchases, that's going to require data. When a researcher like Anna Tuchman talks about using data to answer a question, she doesn't mean the same thing that advertisers mean when they talk about data. And she ran into the same measurement problem that Steve Levitt ran into with the big box chain he worked with. Let's say that we're thinking of like a lotion manufacturer. Okay. So this firm may know that demand for lotion just naturally happens to

be higher during winter months. So this firm, they may say, well, that seems like the best time for me to advertise. So what are we going to find now? We're going to see that we sell more lotion in the winter when we also advertise more. So yes, there is a correlation between advertising and increased sales. But to what degree is the advertising causing the increased sales? Isn't that what you really want to know? Exactly. So one of the challenges with measuring

the effects of advertising is that firms aren't out there assigning their advertising randomly across geographies and across time periods. Since an outside researcher like Anna Tuchman would have a hard time getting a bunch of firms to suddenly embrace this kind of experimentation, she thought about another method to measure advertising, what's called a border strategy. The way that advertising is purchased, there are about 200 television markets in the U.S., these local markets that tend to be centered around large cities. In the ad industry, these are known as designated market areas or DMAs. So what we want to do is think about neighboring

television markets as almost like a natural experiment, where we get to see two markets that may be quite similar on observables. Similar on observables, meaning that people living on one side of the DMA border aren't very different from the people on the other side in terms of socioeconomic and demographic markers. But because individuals living on the borders are exposed to potentially different levels of advertising. That is, the two populations are getting different ads on their TV stations. Then we get to see similar people exposed to different amounts of advertising and then trace out the variation in the purchases they make over time and how that relates to the variation in ads that they're exposed to. This was the strategy Tuchman used in a study last year to investigate the impact of TV

advertising for e-cigarettes. This was a particularly interesting product to choose. TV advertising for traditional cigarettes has been banned since 1971. E-cigarettes were considered, at least by some people, to be a safer alternative, although that is a complicated issue. If you want to learn more about that, you can check out our episode number 398. It's called The Truth About the Vaping Crisis. Anyway, Tuchman looked at e-cigarette advertising data from 2010 to 2015 in more than 200 border markets across the U.S. What'd she find? So I find, as we would expect, that e-cigarette advertising is leading to an increase in sales of e-cigarettes.

We should say e-cigarettes were still relatively new at the time. We might think that for a variety of reasons, advertising for new products may have larger effects than advertising for established products. Okay, that seems pretty clear. But the less clear question was how e-cigarette advertising would affect demand for tobacco cigarettes. So there's some debate as to whether these e-cigarette ads could lead to an increase in sales of tobacco cigarettes when e-cigarette sales increase. So that sounds like an argument in favor of thecigarette advertising, like some policy groups are calling for? What would be the impact on sales of tobacco cigarettes?

And so I find that approximately 130 million more packs of cigarettes would have been sold in the U.S. in the absence of e-cigarette advertising. And that's each year. So again, that's an argument that advertising does, quote, work, at least to some degree. It accomplished, it sounds like two goals there, right? It sold more e-cigarettes and fewer combustible cigarettes, right? Yes. But what can you tell us about ROI, the return on investment of that advertising for e-cigarettes? How much did it cost? Was it, quote, worth it? So unfortunately, in this paper, I didn't have information on

advertising costs, so I am not able to measure the ROI. And when presenting the research, I get a lot of questions like, well, do these ad effects make sense? Is this within the realm of reason of how large or small we expect these effects to be? So people looking for there to be a benchmark to evaluate the efficacy of your analysis. Are you are you missing something? Tuckman's e-cigarette study and the response to it gave her an appetite to go find a general benchmark for the effectiveness of TV advertising. Coming up after the break, you will hear how she did it and what she learned. But first, we are working on an episode about asking sensitive questions, and we need your help.

Is there a question that you would really like to ask a coworker or friend or family member, but you just can't bring yourself to ask it? If so, use your phone and make a voice memo. Send it to radio at Freakonomics.com. Maybe we will use your question in our episode. We usually ask people to include their name and where they're from. But in this case, if that'll get you in too much trouble, you can leave it out. Again, send your voice memo to radio at Freakonomics.com. You are listening to Freakonomics

Radio. I'm Stephen Dubner, and we'll be right back after what I'm sure will be the most effective ad in the history of advertising. Thank you. about the effectiveness of advertising might be exaggerated. Then she learned that a pair of economists at the University of Chicago had been thinking the same thing. Yeah, so I started chatting with Brad Shapiro and Gunter Hitsch, and independently we had all worked on different projects related to advertising and encountered similar sorts of questions. Shapiro and Hitsch had done studies similar to Tuchman's on e-cigarettes, but their topics were, respectively, antidepressants and frozen entrees, like Lean Cuisine and Stouffer's

Pizza. And this led us to start discussing, hey, it'd be really nice if we had a good source that we could use as a benchmark for the effectiveness of television advertisements. There was, in fact, an existing benchmark in the marketing literature based on a series of earlier papers. Yeah, some of these papers come back with benchmarks of around like the average ad elasticity is 0.15 or 0.2. An average ad elasticity meaning what exactly? An ad elasticity measures the percent change in quantity sales that result from a given percent change in advertising. Let's assume the percent

change in advertising is 100%. In other words, you double your ad spending. An ad elasticity of 0.15 or 0.2 indicates that sales would increase by 15 or 20 percent, which is a pretty substantial increase, which would suggest that advertising spending is quite effective. At least that's what the existing benchmark said. But when Tuckman, Shapiro, and Hitch calculated the ad elasticity in their own research. They found a much smaller number, 0.01. That would mean if you were to increase your ad spending by 100% or double your ad spending, this would lead to an increase in sales of 1%.

So you're saying that your experience with e-cigarettes and your colleagues with frozen pizza and antidepressants was seeming to make the argument that TV advertising is about 15 to 20 times less effective than the literature said, yes? That's right. Was most of the research that had made claims about ROI in advertising, was most of it done by the advertising and marketing industries themselves? Not necessarily. So there's plenty of independent academic research that's been done on advertising effectiveness over the years. Many of these earlier research projects were individual case studies that measured just a single product. But the effects for a single product may not generalize to other types of products or other product categories. Earlier researchers tried to address this problem by creating what's called a meta-analysis, that is, pooling together hundreds of existing studies across a variety of categories. But as we started digging into those meta-analyses, we started to worry that the majority of data collection comes from published studies of advertising effectiveness.

And in our own experiences presenting our work on advertising, we experienced a lot of feedback that people expect that advertising must be effective, that it must be profitable, because we observe firms spending billions of dollars on television advertising every year. And does that mean that if you end up doing a study that shows that the advertising is not effective, that it just gets put in a drawer instead? That's exactly our concern, that if you start analyzing some data and find a null result, you may be worried that it would just be really hard to get that published. So you may stick it in the file drawer, that's called the file drawer problem. Or if you actually decide that you want to take on this battle and try to publish the paper, you may face resistance in the review process at academic journals of skepticism from others who say, hey, this isn't what we would expect. If we see these

firms spending millions of dollars on ads, it must be profitable. So there must be something wrong in your analysis. In other words, a conventional wisdom, once established, can be terribly hard to dislodge. Remember what Keith Weed told us earlier. The fact that companies like Unilever spend billions suggests that maybe advertising does work. So Tuchman and her colleagues faced a dilemma. They could accept that the results they had gotten measuring the ad effectiveness of individual categories like e-cigarettes and antidepressants and frozen entrees were outliers or maybe even wrong. Or they could try to do a massive empirical analysis of many products across many categories using better data than the previous researchers who did individual case studies had had available.

This is one of the benefits of being an academic researcher in such a data-rich era. I mean, it's not by chance the popularity of these individual case studies. It's hard enough to go out and collect data on one firm and their advertising and sales spending, let alone do this for hundreds of products at once. And so this was a challenge, but we had a really great opportunity, which was to work with Nielsen data that's made available to academic researchers. Remember, you need two distinct sets of data to measure ad efficacy. How much money advertisers spend and when and where they spend it, as well as how much change there is in product sales and, again, when and where. So the data that we work with is from 2010 to 2014. The sales data is collected by Nielsen in partnership with many different retail chains. So this is going to be grocery stores, drug stores, convenience stores,

mass merchant stores, etc. So the data contains sales for more than 300,000 different brands, which are typically going to be consumer packaged goods that are sold at these more traditional retail outlets. To narrow it down, Tuchman and her co-authors focused on the top 500 brands as measured by dollar sales. Brands like Coca-Cola and Pampers and Folgers and Bud Light. So these sales data represent half of the data equation. But of course, we're interested in measuring ad effectiveness. So then we need to take this sales data and merge it up with the advertising data that is also collected by Nielsen. And ultimately, we were left with 288 out of those initial 500 brands.

Does this mean that 212 of the top 500 consumer packaged goods brands don't routinely advertise on TV? There are a few brands that advertise very, very few weeks where we wouldn't have enough variation in the data to measure anything. So it's not all 212. But yes, there are many brands that are choosing not to advertise on TV. Brands like Crisco and Bumblebee Tuna and Naked Fruit Juice. So plainly, there are plenty of successful companies who don't think TV advertising is as worthwhile as the ad industry seems to think. Now, keep in mind that Tuchman's analysis covered only consumer packaged goods, no automobile advertising, no ads for insurance or financial services. So how representative would this analysis be of the whole advertising picture?

I don't want to say that it's representative of the whole picture. And as I described the selection of products, we focus here on the top 500 brands that advertise or the 288 that advertise. And so we're naturally going to be selecting more established products. Once they merged the gigantic dataset of product sales with the gigantic dataset of ad spending, Tuchman and her colleagues used an analytical method similar to the one she had used to study e-cigarette sales. That is, they looked at bordering advertising markets that received different ads. And then, using a few different models, they compared product sales. What'd they

find? This might sound familiar. We find that the median brand in our data has an ad elasticity of around 0.01. Ouch. So this means that doubling the amount of advertising would lead to about a 1% increase in sales for these brands. I can hear marketing directors across the country having their brains implode and praying that their CEOs are not hearing this. I mean, a 1% increase in sales on a huge base of sales could still be a meaningful increase. And so that's sort of the next step in our analysis is to try to estimate the ROI of this advertising once we take into account the cost of buying those ads. Okay, so what'd they find when they calculated

the return on investment of advertising dollars across their entire sample? We find that almost all brands seem to be over advertising and that they are earning a negative ROI from advertising in an average week. And if they were to instead decide not to advertise in a given week, they would earn higher profits. Things look slightly more rosy when we look at the overall ROI calculation. There, a larger fraction of brands do seem to be better off with their observed level of advertising relative to the alternative of not advertising at all.

Is there a lot of variance between different types of products or categories? Maybe beer ads work great, but ads for laundry detergent are a dud. So we try to look at this and we don't find any statistically significant differences across product categories. Okay. You conclude in the paper that, the vast majority of brands over-invest in advertising and could increase profits by reducing their advertising spending. So my following question is as much a philosophical one as an empirical one. Why are so many modern capitalist companies, which are theoretically devoted to optimizing their profits,

making such a fundamental economic mistake? This is very much the billion-dollar question that we would love to have a great answer to. Unfortunately, our empirical analysis can't fully explain why, but we have a few different hypotheses as to what could be going on. One hypothesis involves what economists call the principal agent problem. So the manager that's in charge of setting the television advertising spending and working with the advertising agency, their incentives may not be aligned with the profit-maximizing goals of the firm. They don't want to put themselves out of a job by doing a lot of digging and showing that,

oh, it turns out our TV ads are unprofitable. Another reason firms may be spending more than they should on advertising is simply because, as both Tuchman and Levitt found, it's really hard to measure ad effectiveness. So it may be that advertising managers do want to ensure that advertising is profitable, but they may not be using the sophisticated methods or tools to properly account for this endogeneity problem. Endogeneity meaning it's hard to tease apart different variables that may travel together, but may not be causally related, like running TV ads for hand lotion only in winter. It may make sense to advertise more during periods of high demand.

That doesn't have to arise because a manager is just trying to pad their numbers. It could actually be the right decision to do that. But if we don't account for the fact that demand would naturally be higher during certain periods when we tend to advertise more, and we falsely attribute that increase in sales to the causal effect of advertising, that could lead us to overstate the effect of ads. That said, I'm still going to assume that you and your colleagues are not going to be invited to many advertising conferences in the near future? Well, we'll have to see. I presented this work at an advertising conference back in January in Australia when we could all still travel. And folks from academia definitely had very different perspectives compared to the folks

that were there from industry. Yes, I'm a little bit skeptical, as you can hear in my voice. That, again, is the longtime advertising executive Keith Weed. And I think it's great that academics look at it. If they have found the magic about how all these companies could improve their profit, I'm sure that everyone would love to see that solution. I could see consumers hearing that analysis and thinking, well, a lot of the cost of advertising, maybe all of it for all I know, is passed on to consumers. And in many instances, generic products are shown to be demonstrably as good as or identical to brand name products. So what do you say to those consumers who maybe feel like they're being fleeced a bit?

So first of all, advertising funds a huge amount of things that we see around us. So all our free entertainment, Google searches, Google Maps, posting on Facebook or tweeting, or indeed a large amount of TV, is all for free. In fact, the free press, the backbone of democracy, is funded by advertising. So I would start with saying that advertising does a lot of good. This answer may strike you as a moving of the goalposts. That is, when presented with evidence that advertising doesn't work as advertised, the advertising executive counters with all the ancillary benefits of advertising.

That said, Keith Weed has another more practical objection to the research we've been discussing. No business wants to be wasteful in resources, and whether that be water or natural resources, or indeed human or advertising. So the hypothesis you put forward is attractive. If you could reduce the cost, why wouldn't you? Yeah, I mean, the conclusion in our paper is really that many firms are over-advertising, and this will hopefully serve as an incentive for them to more carefully estimate the effectiveness of their own advertising with the goals that they can optimize that ad budget and spend money on ads where it's effective, but not spend that money that's not effective. So we've been talking about advertising on TV, which has been a wildly successful and remunerative industry for a long, long time. But of course, the world has changed a lot and digital advertising is

different in a number of ways. The reason Facebook and Google are collectively worth nearly $2 trillion is because they can easily target advertising because their consumers willingly tell them exactly what they like and don't like. So I would think that digital advertising would be many times more effective than the TV advertising you've been studying, yes? If done effectively, then yes, we might find that on average, certain types of online ads have the potential to be much more effective than TV advertising. There are also great examples of online ads that prove to be not super effective. So I don't know if you've come across this paper that was looking at the effect of Google search advertising by Blake Nosko and Tadellas. Yes, my name is Steve Tadellas. I'm a professor at Berkeley's Haas School of Business. Just as Anna Tuchman told us today about the efficacy, or lack thereof, of TV advertising,

Steve Tadellas is dying to tell us what he's learned about the efficacy of digital advertising and how the whole digital ad ecosystem works. But that will have to wait till next week. The wait, I assure you, will be worth it. Oh, ads definitely work, but we can't tell you how or why or give you any evidence for it. That's next time on Freakonomics Radio. Until then, take care of yourself and, if you can, someone else too. Freakonomics Radio is produced by Stitcher and Dubner Productions. This episode was produced by Daphne Chen. Thank you. The theme song is Mr. Fortune by the Hitchhikers. All the other music was composed by Luis Guerra. You can get Freakonomics Radio on any podcast app.

If you'd like to hear the entire 10-year back catalog, use the Stitcher app or go to Freakonomics.com. Freakonomics Radio can also be heard on many NPR stations around the country. As always, thanks for listening. How much money would you estimate that you personally have spent on golf over the past maybe five years? And how much of that was driven by some form of advertising? Maybe $25,000. And I tell you, advertising was very effective on me because I more or less bought one of everything.