Today in Digital Marketing - The New Metric That Can Predict Product Sales

Episode Date: February 4, 2025

Tod chats with Jeremy Yang — co-author of a new paper that proposes a new metric for influencer marketing. His paper's "PE Score" measures how well a product is integrated into videos and its potential impact on sales. In this episode, they explore the methodology behind the study, the significance of engagement metrics, and the emotional factors that drive viewer interaction..📰 Get our free daily newsletter🌍 Follow us on social media or contact us📈 Advertising: Reach Thousands of Marketing Decision-Makers.GO PREMIUM!Get these exclusive benefits when you upgrade:✅ Listen ad-free✅ Back catalog of 20+ marketing science interviews✅ Get the show earlier than the free version✅ Member-only monthly livestreams with TodAnd a lot more! Check it out: todayindigital.com/premium✨ Premium tools: Update Credit Card • Cancel.MORE🆘 Need help with your social media? Check us out: engageQ digital🌟 Rate and Review Us🤝 Our Slack.UPGRADE YOUR SKILLSGoogle Ads for Beginners with Jyll Saskin GalesInside Google Ads: Advanced with Jyll Saskin GalesFoxwell Slack Group and Courses.Today in Digital Marketing is hosted by Tod Maffin and produced by engageQ digital on the traditional territories of the Snuneymuxw First Nation on Vancouver Island, Canada. Associate producer: Steph Gunn.Some links in these show notes may provide affiliate revenue to us.Our Sponsors:* Check out Kinsta: https://kinsta.comPrivacy & Opt-Out: https://redcircle.com/privacy

Influencers on TikTok aren't just entertainers anymore. They're also sellers, blending engagement with product promotion. But marketers face a big question. Does high engagement on these influencer ads actually lead to increased sales? This is where a new study published in the Journal of Marketing Science steps in. The authors developed a new metric called the Product Engagement Score, which measures how well a product is integrated into a video and predicts its sales impact.

They've tested this with thousands of short-form videos linked to real sales data. With me today is one of the co-authors, Jeremy Yang, who teaches business administration at Harvard, and he joins me from his office in Cambridge, Massachusetts. Welcome. Thank you, Todd. Before I begin, I should mention that we are recording this mid-January. This is after TikTok was in front of the Supreme Court, but before we got a ruling on that.

So as we record this today, the Chinese app Red Note is gaining in popularity. And there are rumors that, God help us all, Elon Musk is interested in buying the app. So just as a precursor, people of the future, that's the world where we sit as we record this right now. So, Dr. Yang, I guess I should ask as a precursor to our discussion, do you think your findings, which we'll talk about in a moment, do you think they would be similar across other short form video platforms like Instagram Reels or YouTube Shorts? Yeah, absolutely.

Okay. All right. Well, let's continue then. Tell me about this PE score you developed and how does it work? for the study, which is, you know, when we look at these short videos on TikTok, but also across social platforms, people typically think about engagement as the North Star for how well the video is performing, right? Because it can be challenging to connect the video with sales data. So very often you see marketers using engagement, such as number of likes, comments, and shares as indicator or proxy, noisy proxy, I would say, for how well the video would perform in driving sales. So starting from there, we use data to test if that hypothesis is actually true. It turns out in our data set, it's also collaborated with the result is verified by a few previous study that it is not the case that engagement always lead to higher sales. So that's why we want to dive deeper into if the overall engagement is not a good indicator

of sales, what exactly about the video that can tell us about the sales impact? You know, you're right. That has been the North Star, right? And, you know, when you go on these influencer marketing platforms and you're trying to select a creator, usually the score that they give you, I mean, they give you two things in most cases. They give you the number of followers,

but they also give you engagement rate, you know, because that's sort of the, has been, like you say, the proxy. Did that surprise you? It was a bit surprising before we think deeply about this topic but um after we we find the result we report in the paper if you think about it it's also not that surprising after the fact right because um you know in many cases what makes the video engaging actually have nothing to do with the product itself. So that's why you see tons of engagement.

But we call those engagement ineffective engagement. Because if the goal is to drive sales, it's not enough just to have the video being very engaging, but for reasons unrelated to the product. Right. That makes sense. Can we talk about your data set that you had? These were videos that you correlated to real sales data. What was the video source? What was the sales data? We need to combine two data sources. So one is the video data set that also contains engagement. So we got this from the Chinese version of TikTok or Douyin. And then for the other part of the data, which is sales,

because at the moment when the paper was written, so the project started in 2019. So at that time in the chinese market what happens is one uh when the advertiser put an ad on doing or chinese tiktok to advertise for the product um the platform does not host the product so they would link the product to external website, usually e-commerce aggregator, something like Taobao or Alibaba or JD.com, right?

So in order to test our hypothesis, what we did was essentially linking the influencer ads on Douyin or TikTok with the corresponding sales of the product being advertised from taobao.com so that we can actually look at what's the sales impact after the video is posted.

You know, I've seen a lot of sort of studies that have studied TikTok actually do use Douyin data, the Chinese app. I'm not sure whether it's just much easier to get a hold of or whether, you know, I'm not quite sure why. Do you have a sense sure whether it's just much easier to get a hold of um or whether you know i'm not quite quite sure why do you have a sense of why it's easier to use doyin data so i think the key reason is um e-commerce on doing is much more advanced and mature than uh e-commerce on tiktok right so uh t TikTok recently, they started to really prioritize their e-commerce in the international market. So they call this TikTok shop, right?

So before, I think it really started probably the last year. Before that, really it's not an option to buy stuff on TikTok. It's purely entertainment platform. But if you look at Douyin, it's a much developed e-commerce ecosystem, I would say. So you would very easily be seeing product ads

that contains a link to purchase. Very much like what you would see on e-commerce platform. Is there anything different about engagement Very much like what you would see on e-commerce platform. Is there anything different about engagement from Chinese consumers on that app versus engagement from quote-unquote Western nations on the TikTok app? It's a good question. So in the paper, I think the good thing about our method is that we don't decide a priori what elements in the video are driving engagement. We let the data tell us which part of video are engaging. And the way of doing that is essentially we have a lot of data on doing video ads and their corresponding engagement measured by number of likes, comments, and shares generated on that video, right? So we didn't do the analysis with international TikTok data or data on other platforms.

So I cannot tell you the result if what makes, what kind of content would engage Chinese consumers would be the same type of content that would engage international consumers. I'm not even thinking content though. I'm thinking like just sort of the mechanics of it. Like do Chinese consumers tend to have a higher click-through rate? Are they more likely to hit the heart button? Or maybe there's not been a study on that.

I don't know. Not to my knowledge. Not to my knowledge not to my knowledge but you can very easily compare that if you have both data from doing and tiktok all right back to your score the the pe score yes can this score that you've developed can it predict sales before a video is even posted uh yes yes exactly and that's the whole intention of the of the score right because it's already too late after you you post a video so our intention is to compute a score prior to the release in an ad think about you're the marketer and then you might have a few different options of what kind of ad you might post on the platform and then you can use the algorithm to

generate the score and you can use the score to select the winner right the best performing video to post so exactly we want to do this prior to the release of the video and can it be used to help or can can marketers use it to select the right influencer or is this just useful after the video is made the video currently sorry the algorithm is currently designed to take a video into um as the input so the score is made? The video currently, sorry, the algorithm is currently designed to take a video into as the input. So the score is a video specific. So even for the same influencer, say the influencer has 10 different videos, the scores will be different for each video.

Right. However, that being said, you can still compute a score at the influencer level if you were to just aggregate the scores for different videos posted by the same influencer. so can i just sort of rephrase that and see if i have this right so if i am trying to select if i have access to this pe uh algorithm yep i would go i could go on to a tiktok account holder or you know reels or whatever um apply this score, generate an average PE score, and then use that. Theoretically, anyway, that average PE score per influencer would reflect their likely sales potential.

Is that fair? That's exactly right. Yeah. Your study highlighted that human presence and certain emotions can boost engagement. Can you share some examples of what worked best? So one key component in our algorithm is the engagement heat map, which is to use the algorithm to output a heat map overlaid on the original video that will highlight the key moment and regions in the video that are more engaging than other parts of the video. And then once we have that, we can try to ask the algorithm,

okay, you see these regions on the video in these moments to be more engaging. What's happening in those moments? So what objects, what kind of activity are there driving the engagement? So it's hard to, there are a wide range of activities because as you can imagine, there are all sorts of content on TikTok. So that's why we stay with, I guess, these more generic terms to capture. Because I think overall, the thing that drives engagement in our data is really what we call these high intensity, high energy type of things. So like dancing, for example, that's the first thing people would think about when they think about TikTok video. So that would be one good example. And is it as simple as saying if they

are dancing, the PE score is higher, therefore there is more likelihood of sales? It can't be that direct or simple. I'm glad you asked. That's a great question. So when we think about engagement, we talk about engagement map. I didn't mention the product at all, right? So what's driving engagement? That's just one part of the equation. The other part is the product, right? So it is indeed not the case that, you know, if you just dance, you know, you have high engagement, you necessarily would have high product engagement score. That's because we have to consider product and engagement together. It's not enough just to make the video engaging, right? Back to our motivating example of, you know, many videos that are being engaging, but not driving sales. So the key really is

that you would want to feature the product in the most engaging parts of the video. So it's not enough just to be engaging. You have to be engaging in a way that introduce and present the product. Between juggling client meetings and keeping up with what's changed in the marketing world, who has time to stress about website security? But security is important. What with bots pummeling your brand site by the hundreds every hour looking for a way in. Kinsta provides managed hosting for WordPress sites, offering lightning fast load times, top tier security and human only customer support.

And thanks to their unlimited free expert led migrations, Kinsta ensures a smooth transition from other hosting providers so you won't experience any downtime. Ready to experience Kinsta's hosting for yourself? Get your first month free when you sign up at kinsta.com today. It's the perfect opportunity to see why Kinsta is trusted by thousands of businesses worldwide to power their websites. Visit kinsta.com to get this limited time offer for new customers on select plans.

Don't miss out. Get started for free today. You know, I don't want to pick it apart too much too much but let me see what does your gut tell you i mean i don't think you studied this level but what does your gut tell you on something like this 60 second video you have the influencer be fairly you know light i don't know lightweight engaging but then when the product is when they hold up the product, when they go for the jugular, then they turn on the energy. Is it as easy

as that? Can we manipulate it that much? So in the video, we want the high engagement part to overlap with the part where the product pairs. So that's really the key and intuition behind the algorithm. So if you can find a way to make the product appearance engaging,

then based on our algorithm, you would have a higher PE score. Does your heat map include things like facial expressions? It does. Oh, okay. And what did you learn from that? So facial expression. So we try to analyze the emotion of the face of the influencer. So what we find is happy. I think happy and sad faces tend to generate more engagement. Happy or sad? Happy or sad. Yeah.

So is it just a sort of extreme emotions? It doesn't really matter what emotion is just recognizable? Or is it literally happy or sad and no other emotions? Yeah, we did test other strong emotions like anger, right? Anger, disgust. We didn't see those emotions generating higher engagement. It wasn't moving the needle on it. Yeah. The sad one surprises me. Did that surprise you?

It was surprising. Because typically people tend to think about, you know, you have to be very positive being the influencer promoting a product. But I guess sadness is a way to also captivate attention. People might be curious, why are you so sad? And you mentioned that anger and other emotions didn't necessarily move the needle positively. Was it that it just didn't move the needle at all or did it have a negative effect on sales?

So based on our result, it didn't move the needle at all. In either direction. Not in either direction, yeah. Do you think that TikTok's format, the short videos, fast-paced videos, makes it harder to integrate products effectively? I think it probably depends on what type of product. If it's a product that people really need to, for example, laptop or headphone, right?

There's some specifications you might need to look into, like how big is the screen? What is the resolution? What is the storage? How fast is the CPU and stuff like that, right? So basically, you need to communicate a lot of information in that ad, and consumers need to process and think about this information. So if that's the case, then probably it's not very well suited for a very fast-paced video ad

versus where, you know, for products, people don't really need to deliberate that much before making a purchase. Something very fast and quick, then it's suited. Yeah. Can we get our hands on this PE calculation? Yeah. I know your paper is open source.

Like your paper is published in its entirety. It's not just a, and we have a link to it. I'll mention at the end, But is that in the paper? Can we just rip that off and start using it? Yes, you can. But with a caveat. So we have a licensing agreement

with a startup that is productizing the algorithm. So you can totally access and use the code, but only for academic purpose. Oh, well, that only for academic purpose. Oh, well, that's not much fun. What did you have in mind?

For profit purposes. Yeah. I mean, I suppose that is sort of culturally appropriate in the academic world, but does it frustrate you at all? I mean, I certainly know as a marketer and someone who's married to a scientist that these things are done in sort of theoretical levels, but then don't get into the sort of the practical application. That gap must be frustrating for you as a scientist. Yeah, absolutely. Absolutely. So we definitely want to study something that is managerially relevant. Right. We don't want to lock ourselves in the office just right on the whiteboard.

So, you know, exactly. So overall, it's my intention to really close that gap. So finding ways to find practical use cases of the topics we study. Yeah. So this example would be one good example. Yeah. All right. So bottom line here for marketers who are working with influencers, what should they be doing differently based on your findings? I would say two things, right? So think about the process of marketers working with influencers. I

would say there are two big steps. So one is selection, right? Finding the right influencer to work with. The second thing is after you've found an influencer you want to work with, then deciding what's the optimal content to post out there, right? So for the first step,

I would come back to Todd, the earlier question you asked, which is, can we compute something like a score for an influencer? And indeed, you proposed the way to do that. So if you compute

a P score for all the videos the influencer has posted in the past, then you can have a influencer level score, right? And then marketers can use the score to select which influencer has the best, I guess, conversion potential, right? So that's the first step.

And after the marketer has found the right influencer to work with, you know, when the influencer is thinking about what kind of content to post, the brand, the advertiser can use the algorithm to essentially evaluate the sales potential of this video. And then if they're not happy with the result, they can tell the influencer, hey, based on our evaluation, it's not likely that this video is going to drive a lot of sales.

So maybe we can tweak this and that part a bit and see, you know, maybe it will have a better score, something like that. So it's like running it through a fortune machine, fortune telling machine. Yeah, that's right. You had co-authors on your paper. Who were they? Yes, I have two co-authors. One is Junjun Zhang, who is a professor at MIT. She is also my dissertation advisor. So this paper is one chapter in my thesis when I was at MIT. Oh, okay. And then the other person is Yuhan Zhang, who was at that point a visiting student at MIT.

She was doing her PhD at Tsinghua in Beijing, China. And then now I think she is a professor at the Harbing Technology Institute. Well, the paper is very interesting. I think it gives us a lot to think about, especially as TikTok evolves. You know, as you and I sit here, as I mentioned at the start, we don't know which way it ended up evolving between when we're recording this and when this runs. But it certainly will be interesting. Jeremy, thank you.

Thank you, Todd. Jeremy Yang is an assistant professor of business administration at Harvard University. His paper is called engagement that sells influencer video advertising on TikTok. As I mentioned, the full paper is available for free online.

That's sometimes a bit unusual. You can find it at the shortcut that we created, which is b.link slash ttstudy. I'm Todd Maffin. Thank you for listening. See you Friday for our wrap-up of the week's news in digital marketing.