Magic: The Gathering Drive to Work Podcast - #605: Market Research

Episode Date: January 25, 2019

In this podcast, I talk about the many ways we learn about what the players like and don't like.

I'm pulling in my driveway. We all know what that means. It's time for another drive to work. Okay, so today's topic came from my blog. I talk a lot when answering questions about market research. I talk a lot about why we will or won't do something because all of you are or aren't interested in it. And so someone said, hey, can you talk about market research? So yes, I can. And that is today's topic. So let me, when I say market research, by the way, I'm using a broader term. And what I mean by that is the collection of any data that we can use to then make decisions about how we make the game. So when I talk about market research today, I'm going a bit broader than, I mean, I will

talk about questionnaires and things like that, which are more traditional market research, but I'm talking in the broader sense of finding out what you all like and acting accordingly. Okay, so today's podcast breaks into three parts. The first part, and probably the lengthiest part, is about gathering the data. How do we gather data? What data do we gather? I'll talk all about that. The second part is analyzing data. How do we analyze the data? And a big part of doing market research is not just collecting the data. It is understanding the data and using it to realize things. The third part is using the information in design. It's like, okay, we've done market research, we've learned things. How do we use that? How exactly does it affect the game?

So that is my topic today. I want to talk all about sort of the gathering of the data, the analyzing of the data, and the use of the data. So let's start with the gathering of the data. Okay, so let's begin in the most traditional sense of market research, which is actual, like technical surveys and things. Okay, so there's two different ways to do surveys. One is online and one is in person. So let's talk about each of those. So online is we often will do surveys, where what we do is we make a link to the survey on our website, on our social media channels, and it's like, hey, come let us know what you think about Thing X. Now surveys can have a lot of different forms.

They can be about a particular set. They can be about a particular product that's not a set. They could be about some general things. Sometimes we're trying to learn about how people feel about categories of things. Sometimes we're trying out new ideas and want to test what people think of new ideas before we do them. We've gotten more and more into do market testing of ideas that we're thinking of doing. Now, surveys can be done online or they can be done in person. And a person is where we

bring the people in personally, show them information. Sometimes we let them interact in a way that we can't online. And then we ask people about it. Online surveys Oh, the other thing we always do whenever we do a survey And then we ask people about it. Online surveys. Oh, the other thing we always do whenever we do a survey, this is just normal market research, is we will ask questions about the person answering the questions, biographical data.

Why is that important? Because part of what we're trying to do is, our audience is not a singular audience. It is not as if everybody who plays Magic wants the exact same thing. Different people want different things. And so one of the ways to help chop up the information is to get biographical information for all you. You know, who are you? How long have you played magic? How often do you play magic? How much money do you spend on magic? What is your age, your gender? You know, where do you live? You know, we'll do a lot of things to sort of be able to chop up information so we can use it in bite-sizeable chunks, if you will. A lot of the important

parts of gathering data is having the means by which to analyze it. And so the biographical information is very important. It also allows us, if we have similar information between sources, we can start organizing that information together. And so when you take a survey, now, there are many different kinds of surveys you can take. They range from surveys that are very short and just a few questions to things that are pretty long and can take a while. The example of long things is sometimes if you take surveys on our sets,

we'll ask you about specific mechanics, about specific cards. We'll go in depth about what do you feel about things. Oh, the one thing we always do, by the way, on our written surveys is most of the survey is multiple choice. It's pick the range or whatever of your answer. But we do have write-in boxes on most of our surveys, and we do read that information. That is probably,

if you want to get a message across to us, that is the cleanest way to get a particular message across. There are other means like social media, which I'll get to, but if you're taking our surveys, please, there's a box, sometimes there's more than one box. Usually there's one box near the end where it's like, okay, do you have anything else you want to say?

We read that. So if you have something you want to say and it's not communicated through a lot of the other part of the survey, please use that as an opportunity to let us know. One of the themes of today is the reason we do all the market research is we want to learn what you guys want. Like my job, or all of our jobs, is to make a game that you all want to buy. That's something that you enjoy. And in order to do that, well, one of the easiest ways to do that is to actually ask you guys what you do and don't like and use that information to constantly

adapt. Like I talk about how magic design is iteration. Well, iteration is about making something, getting feedback, and then adapting to the feedback. Well, the feedback, some of the feedback, is this kind of research that we're getting. Anything that we learn from you is used as a means to try to understand what it is you like and don't like. Okay, so that is an online survey. Online surveys, the problem with online surveys is that

they're self-selecting based on where you put the survey. Meaning, if we put it on our website, we can only get Magic players that come to our website. And that is not a cross-section of Magic players. That tends to skew in certain directions. For example, the more enfranchised you are, which means the longer you play, the more often you play, the more you spend on, you know, the more involved you are with Magic, the likelier

you are to be involved, like to come to our website, to be looking at our social media. So when we put out a survey where it's sort of like, hey, people who are already paying attention to us, now that's our motion franchise crowd. We care what the motion franchise crowd thinks. You guys spend a lot of money. We definitely care what you guys think. But it's not necessarily a cross-section of magic. So one of the reasons that we do non-ridden surveys is the non-ridden surveys is us trying to track you down.

Meaning, a written survey is like we just put it in places where Magic Players will be and say, please take the survey. We don't select you, you select us. You choose to take the survey. In a live, what we do is we usually hire a service that will go out and find people that

meet a certain criteria. It depends on what the survey is. Sometimes we want Magic players. Sometimes we want Magic adjacent players. Sometimes we want people that have never even heard of Magic. Depending on what it is we're doing, they'll get different kinds of audiences. And when we sort of, the difference there is we're seeking out the audience we want. We tell them the criteria we care about and then they, in their, you know, they will usually they go to malls or sometimes they go to the telephone.

I'm not sure where they find people. But these services, they have means to go find people and then they ask the right questions they need to to narrow down the audience such that the people they bring in are of an audience we're looking for or an audience we're interested in getting information from. So the difference there from the printed surveys online is that's us selecting the audience, not the audience selecting the tasks. Both of them are valuable. It's not like the online surveys aren't very valuable. We do them. It's just that the online surveys are probably the easiest way. And like I said, we tend to ask our most enfranchised players the most questions because you guys

are the easiest to talk to. Now, on the flip side, enfranchised players tend to spend the most money, are the most invested. I mean, there's a reason to pay attention to the enfranchised players, obviously. But when we're trying to learn things about beginners or stuff in which it's less about the franchise player, we more often have to go out. Every couple of years we do what we call a deep dive. So what a deep dive is, is we go out and we get a large section of the public

and we start asking them questions. And the idea of a deep dive is to understand things like what percentage of the audience, usually Americans, but we also do market research in other countries, what portion of the audience is aware of magic? What portion knows someone who plays magic?

What portion plays magic? And that allows us to get some biographical information from a big picture. Like one of the things when we have you take the test online is because it's self-selecting, the data we get is a little bit limited, meaning it doesn't always tell us some of the broader information we need about some of the biographical information. The deep dive and stuff lets us have a better sense of sort of magic awareness and magic adjacency and magic play and stuff like that. We also do these things called focus tests.

So normally we bring people in, we ask them questions, and then they collect the information. What a focus test is, is people come in and then they are either being interviewed or they are playing something. A very common thing to do in a focus group might be then they are either being interviewed or they are playing something. A very common thing to do in a focus group might be to give them material of a new game and let them play without us interfering with them all, seeing what do they do with, you know. Like, for example, when we're trying to learn how to do magic starter stuff, we bring people that don't know magic and then say, okay, we're not going to do it.

Here's the material, and we watch them to see how they do. The idea of a focus test is that we are watching them either through a two-way mirror or cameras, most often both. And then it's an idea to see, okay, let's see how the players interact without us, you know, sort of, we get to see it directly. Sometimes focus testing can also be interviews and stuff. And the reason you would bring those in is there's some questions that are a little more complex, that are hard to do in a written thing, where you get, in a focus group, you have an interview, and usually it's a professional, someone not tied to the company, who is, you know, we want to learn certain things,

and they're trained in what it is that we're trying to learn. And so they can ask questions and try to draw information out of people in a way that is really hard to do in a written test. So we do, so market research spans, traditional market research spans a bunch of different areas. We in Wizards have a market research group. We have people who are in charge of overseeing it. Like I said, they work with a lot of external groups to do a lot of the people collecting and stuff. Okay, that is just one. That is the most, often when people think of market research, they think of that traditional market research. Like we are asking people directly. We are asking them questions, you know. And while that is an

important part of market research and, you know, definitely the sort of most market research-y of our market research, there are other things that we do to gather information. Let me explain. Okay, so next, sales data. So, we sell our game. One of the things as a business is we care very much sort of how the game sells. So the short version is we sell the product to distributors, which are kind of like middlemen or middle people. And they sell it to the stores directly.

And then what we do is we gather data on how sales go out to the distributors and how the distributor sell to the stores and then we have something we call sell through which is how long does it take a product before it is gone um and depending on the product some products have a very very short amount of sell through time there's some products we make in small you know like high demand products we make in small, you know, like high-demand products we make in small number that sell quickly. Sometimes we make something, like our standard legal sets, that are meant to be out for quite a while. And so we print to demand,

so, you know, if they sell it, we'll print more. Often when we talk about sell-through, if we have a product that has multiple printings, we'll look at sell-through per printing, like how long it took for the first printing to sell through. And the reason this is important, I mean, first off, we're a business. Products selling is important.

We monitor it, A, because we need to know how our business is doing. But from a market research standpoint, sales are a good indicator of player happiness with a product. Maybe that's the wrong thing. Happiness is the wrong word, actually. Players' willingness to buy the product. Meaning that if we make something and it sells well, that means there are players that want to buy it. That doesn't always mean they're happy with every choice we made. We can make a product where players dislike some aspect of the product but they still want the product and they buy it. So sales is not the be-all end-all to understanding whether players like or

dislike something, but it is an important factor. Like one of the things that we really want to care about is, you know, if we make something, let's say we make something and players go, this is awesome, this is amazing, and we get nothing but rave reviews and excitement online, and then it doesn't sell. Well, that tells us something interesting. Like, it tells us the players in concept liked it, but if they really liked it, they would probably buy it, you know? So if something struggles in the sales, there's something about that product that is a problem.

Now, it might not be the content. Sometimes, for example, something suffers not because people don't like the content, but the price point is wrong or some aspect of it is wrong or it doesn't match the format it's made for or whatever. There's different reasons why people might like the product but not buy the product. That's something else we need to learn. But a product is not successful if we don't sell it. A product that doesn't sell well is not a successful product. No matter how much people say they like it, if people don't buy it, we consider that a failure. That doesn't mean we might not learn from that and try to do something similar

that addresses the issue players had, but it does mean that that product in its current form is a problem. So another thing that we do with things like sales is we are interested, and when I get to the analyze part, I'll explain this in a little more detail, but we really think it's important to look over time. And so we like to gather data from the same thing at the same time. So, for example, with sales, one of the things we'll do is we will look at the first week of sales, second week of sales,

third week of sales, fourth week of sales, every week, but we compare them to other sales of the same size set at the same time. Oh, well, this is a large fall set. Well, let's compare it against other large fall sets. Well, how did it do in its first week? And usually by comparing sales versus other sales, it also gives us a sense of momentum. Like one of the things that sales is really good for is our players growing tired of it, our players excited by it. You know, watching the, Are players growing tired of it? Are players excited by it?

You know, watching sort of its sales pattern can tell you a lot about the momentum of the product, the direction of the product, whether people are getting more excited about it or less excited about it. It gives us a lot of information that we then can use later on about sort of what it's doing right and doing wrong. To give you an example of one of the kind of places where something like this becomes really important is, I use Unstable as my example. So, we made Unglued back in 1998. We made Unhinged in 2004. So,

we had not made, when this product came out in 2017, it was, yeah, 2017, it was 14 years since we had made, or 13 years, 13 years, 13 years since we made the previous unset. Well, I believe that there was an audience for the early unsets and that we overprinted it, which is why we had stuff we had to destroy.

A lot of other people felt that it was a failed product, that it didn't have an audience. So there was a big uphill battle to get Unstable made. And so one of the things when we were measuring things, it's very important for us to sort of see how things do. Because, for example, I was saying there was an audience. Well, how do I prove there is an audience? Sales was a big part of it. That what I had to say is, hey, look, people were buying the product.

If no one was buying the product, well, then there wouldn't be an audience. But if people are buying the product, well, then I can make my argument there's an audience. There's somebody buying the product. And Unstable, not only did people buy the product, we reprinted it numerous times. So people liked it so much that we had to make more and do that numerous times. So now when I'm making my argument for the fourth unset, I can look at sales data and say, hey look this demonstrates there's an audience there. There's people who bought the product. Okay next, organized play data. So we run millions of tournaments all around the world and

we gather that data. That data is important for a couple ways. One is it tells us about formats. It tells us about what formats people are playing, what they enjoy playing, and we can compare formats against each other. The one note I'll make here is that certain formats lend themselves more toward tournaments, so the organized play data is a little geared more toward tournament play. While we do have sanctioned stuff like commander play, a lot of people play commander aren't playing at their store necessarily.

I mean, some are. But whereas if you're talking about a standard tournament or a modern tournament, more of that's happening in stores. I mean, there are people playing standard outside of stores and modern outside of stores. But it gives us a little better gauge. So A, organized play data tells us a lot about the formats being played. It can tell us information about how cards are used. I know play design cares a lot about sort of what gets played at what tournament and what's doing well.

A lot of we can use the data of the metagame as it means to understand sort of what isn't isn't working and you know what cards are hitting and what cards are not hitting um another thing you can tell us is it gives us we track dci numbers so we know when new players join we know how often people play. You know, it lets us look at sort of the trends of the people who are involved in our organized play system. And I should note, the majority of Magikers aren't involved in our organized play system. But the ones that are, more enfranchised, more invested, and, you know, spend more money. So we care quite a bit about who's playing our organized play system. And we have a whole system that we do.

Just like Magic is trying to optimize itself, our organized play system is trying to optimize itself as well. And this data helps do that. We also can learn about sort of what sets are bringing in new players, what sets are bringing back players.

We can look geographically to try to understand is certain play more popular in certain regions? And so the organized play is a lot of very interesting data that gives us a better understanding of who is playing and how they're playing and where they're playing and how often they're playing. There's a lot of very valuable information that comes from there. Okay, next.

Digital data. So right now we have two digital magic games. We have Magic the Gathering Online and we have Magic the Gathering Arena. So the neat thing about digital is it is a data sponge. Anything that happens online or most things that happen online, can get tracked. In fact, digital data is kind of opposite. Most of the time, we're struggling to get as much data as we can. That if you ask us, you know, if we're talking about sales data, or we're talking about organized play data, our answer

is always like, oh, we would love to get more data. You know, we're always trying to get as much data as we can. Digital data is the one that's the opposite. It's like, whoa, whoa, whoa, firehose of data, firehose of data. The problem with digital data is not getting the data. It's being able, like, it's trying to find the needles in the haystack, if you will, in that we have to figure out from the data we have, what do we want to learn? And so a lot of the data online is us specifically trying to figure out what to look for. That there's so much raw data that it's intimidating. And so what we want

is to figure out how we can use this data to get interesting information. Now the one thing about online play is there's certain kinds of data that it's hard for us to get in tabletop magic. For example, we can track in digital magic how often cards get played. Like, literally, we can say, okay, how often did that card get put in a deck? How often did that card get cast? How often did that card,

you know, did someone turn a wild card into that card in a magic arena? You know, how we can look and get really interesting data about gameplay now as with anything digital versions of magic especially magic online for example skews toward a much more established player

when we look at data online we have to understand that it's not a complete cross section of magic that everyone who plays Magic plays online. But of the people who play, it gives us a lot of information. And it helps us understand... Like, for example, we look at tournament data, so we can really get a sense of metagames. It's probably the place that's the most thorough when we're trying to study metagames, because we have literally all the

data. So, you know, it is you know, when we look at real world stuff, tabletop, we have data, like, we can get deck listed things. So we have some of that, but we don't know, for example, you know, like in casual play, something we

can get on the online stuff is we can see what people are playing for fun. And learn like, oh, well this card that is not at all a tournament card, but people are playing it and they're having fun. One of the things we always look for is things that are popular without being powerful. It is very easy for powerful things to be popular. Winning is fun, and so people will play cards that help them win. winning is fun, and so people will play cards that help them win. What is more interesting is when people play something that isn't powerful,

because that's a really good indicator that it's fun. Because why would you play something that isn't powerful if you're not enjoying playing it? And so that kind of information can really kind of inform us on what things are fun outside of things that are powerful. Next, I will have what I call miscellaneous. Essentially, I'll put it this way. Anything that is put into numbers or put into a clean, simple way to categorize, we will collect all data we can collect, any and all data. For example, I do a thing called Head to Head on my Twitter

where every three weeks I take a topic, there's 16 things in the topic, and they go up against each other. And we'll get down to it's like a sweet 16 and they go to there's one. And now given that's people on my Twitter, somewhere between like 2,500 and 5,000 people tend to vote on a thing, it is not a cross-section of Magic players. Obviously, people who are on my Twitter lean toward the more serious Magic players, or more franchise, not necessarily serious, but more franchise.

I mean serious in how often they play, not in whether they play for fun or not. And, for example, the very first one I did was on creature types, and Wizards did better than we expected. You know, we normally seed them, and wizards did pretty well, and so when I was making Dominaria, not too much later, I was looking for a tribal component for Dominaria, and I remembered, oh, wow, wizards did better than I thought. Hey, maybe we should do wizards here.

There's some evidence that people like wizards. Wizards really thematically fits what I'm trying to do. Hey, okay, let's do wizards here. There's some evidence that people like wizards. Wizards really thematically fits what I'm trying to do. Hey, okay, let's do wizards. And so, like I said, any place we can collect data, we will. Any place where people are giving information, like, we will soak up any information we can. And I just say miscellaneous because there's lots of different kinds of information that are out there.

Some gathered by other people. You know, some gathered, you know, anytime people gather information we're interested and we'll look at it and we will incorporate whatever data we can get our hands on as I get to the analyze part. The more data, the better for us. The final thing is what I would call anecdotal data and that is mostly social media and in-person contact.

For example, I have a blog. I answer questions every day on the blog. I've answered like 10,000 plus questions. Oh, I'm sorry, 100,000 plus questions. And I get approached on all my different social media. People ask me questions or just tell me things

and not always questions. Plus, I go out and I read different magic threads. I read different magic articles. I will follow conversations on social media. You know, when on a day-to-day basis, things come up and happen. And, you know, there's always something that people... There's always a topic of the day. Sometimes more than one.

Sometimes, like I said, spurred by an article or spurred by something that happened in an event or whatever. And I and the rest of R&D are involved in those conversations. Well, we observe them. We tend not to, depending. We don't get involved too often. Mostly we listen to what you guys have to say. But it's something that we do. And like I said, it's anecdotal in the sense that it's not. A lot of what we do with our official data collecting is,

the science behind it is, you don't need a lot of something, what they call sampling, that you can learn about large groups by sampling a small percentage of them, assuming you correctly have a cross-section of the kind of sampling you're doing. And so one of the things we're trying to do when we collect data is make sure that we

get enough data that we can use it for sampling. Different data has different elements to it. I mean, there's part of what we try to do is get enough data that we can wash a little bit of the noise between the data we collect by the fact that we're getting from different places and listening to different people. The reason anecdotal data is important is not that it is, we're not analyzing it quite in the same way we do sort of the heart-crunchy data, if you will. But it does put a human face on things. It puts emotion on things. For example,

let's say we do something and it upsets players. Well, players will write to me and tell me why they're upset. And we'll explain they're upset in more human, emotional terms sometimes, not always. And I think one of the things that's important is it's very easy when something comes up to get detached from how it affects people on a personal level

and just being connected to the people and hearing that from them helps you understand sort of contextually how things affect people. Like one of the things I care a lot about if you listen to, you know, in my podcast is I'm big on emotional response, right? Like I'm trying to evoke some emotion out of you. I want to know what that emotion is. And that is hard to collect in hard data form. It's hard in a survey

to say, well, what emotions were you feeling? This is kind of hard to collect. So a lot of that kind of information collecting is more anecdotal talking to players and getting a sense by listening to enough players communicate to get a general sense of how something makes them feel. And so the anecdotal stuff, like the reason that I continue to answer questions and interact with the public is that anecdotal data is very, very important. It tells me things that other data we collect can't tell me. And it also is a good barometer for me to understand

just what people are thinking about, what people are caring about. If something's a hot-button issue, guess what? I will get asked a lot about it. If people don't care about something, I won't get asked a lot about it. So, you know, it's a good way to gauge how invested and how much people care about a particular topic based on sort of how much they're asking me about it. Once again, my audience that asks me questions is not representative of a larger magic audience. I'm aware of that.

That is why on my blog sometimes a lot of people will bug me for one particular thing and I come back with, oh, well research shows that this is not something that's very popular and people are like, but wait, everybody here seems to like it. I'm like, well, this is not a cross-section. And there is a dynamic that can happen on social media where when people know they're being heard, we'll push agendas they care about, and it's easier to rally other people with the same agenda. It's the nature of social media. So people

being loud on social media, it's not that we don't listen to it, but we always compare that against the other data we have. And just because a loud minority might want something doesn't necessarily mean that's what the majority of players want. Anyway, as I get to how we use the data, I'll explain a little bit. Like there's different kinds of data we can and can't use in different ways. Okay, so that was all about collecting the data. So let's talk about analyzing the data. So we have a couple teams at Wizards whose job it is to analyze data. Two major teams. One is what we call BMI. And I'll be honest, I don't know what BMI stands for.

I know it's not Body Mass Index. But they are the team that do the hard, crunchy data analysis. And they're making use of computers and, you know, the data nerds, I think is what they call them, that really, really get into sort of how to process the information. And that there are a lot of people that get very excited by that. And this team is able to sort of take the flood of information we have and then be able to get really usable, hard, crunchy things out of it so that we can understand things. And that team works with R&D.

Part of what they say is, can you explain to us how we can figure things out? And we will say, oh, well if you look at these factors and these factors, and then we will give them some guidance on how we can sort of figure out what players do and don't like. For example, I know we spend a lot of time with Play Design, spend some time with them, trying to analyze what gets played and how it gets played and try to get a good sense of using the data we have to have the best understanding

of how the metagame is working and stuff like that. There's another team I don't know what the other team's name is, they keep changing names. It's I don't know what the other team's name is. They keep changing names. It's, I don't know what the team's called. Like Player Analytics or something. And that team is more about, is less about crunching the data than it is about understanding the data. That they, like for example, one of the things they do is they create some

of the surveys that we do to gather kind of the branching information. A lot of times what we want to do is understand how our information correlates with one another. And so sometimes we'll make some surveys that act as sort of the glue, if you will, to sort of understand different components and how they come together. And that group is very much about understanding player insights. Like BMI is about crunching the data and making the tools necessary for us to be able to produce content that we then can analyze and understand. And the consumer insights or player insights is more about a helping provide the right kind of

information so it can be crunched and then understanding what it means and having larger sense of who our audience is and what they care about. The reason that's very important is from a marketing standpoint, from a selling the game standpoint, we want to know, like R&D wants to know what people like so we can replicate things and we can, you know, make the game better and make it more enjoyable. You know, brand wants to know what players like so that they can better sort of cater things toward them, both in product mix and in how we advertise, how we market our product.

Like, part of marketing is who wants to buy this product? Where are they? How do we communicate with them? And a lot of this information that we do can help us. Like, I talk a lot about how Magic is many different games to many different people, right? So not every product is for every magic player. And so part of what we need to understand is, okay, who is this product for? And where would I find these magic players?

I want to talk to these magic players. Let's say, for example, we're making commander decks. Okay, who's the audience for this? Well, commander players, that would be. Okay, well, we want to advertise to commander players. Where do we go? Okay, well, maybe there are certain podcasts and different content creation

that's about that format that's popular. Maybe we go there. Maybe there's certain places they're likely to look. Maybe there's certain places that commander players go to read or something. Maybe we advertise there. Or maybe in our larger thing, we figure out the kind of players that play Commander and maybe some of our broader Magic advertising, we just skew it a little bit in where we

put it on social media or on wherever our ad buys go so that we kind of can hit the kind of player that would like Commander. The other thing that the information tells us is what about the product they like? Why do they like it? Part of good advertising is not just making you aware the product exists, but making sure the product is what you want the product to be. And so a lot of times we'll work with the player insights as we... Like one of the things that happens is R&D and brand work together so we understand what are the

products we are making. We want to make sure we're making the best products we can make. And that part of that is understanding audience need and oh, there's players that want thing X or thing Y and if we made a product that did thing X we think we could sell well. So you can analyze stuff like that. Also, as I mentioned before, we do have a market research team. Um, they also do some analyzing. Um, mostly what they tend to do is they do the surveys and they correlate the surveys and they can say, um, you know, based on the various questions of how they interconnect with each other, they can make some conclusions. One of the things that R&D enjoys, I know, is whenever we do surveys,

R&D loves seeing the hard data because R&D will analyze some of the data itself. One of the things that's funny also is the market research people are experts in market research. They're not experts in magic. You know, that's R&D. And so it's funny sometimes when they're collecting data. So the story I always tell, I might have told this story before, but it's a funny story. So we were doing market research on Galoo, the very first unset.

And the two worst-ranked cards in the set was Blacker Lotus and Chaos Confetti. Well, the thing they had in common, as the person who made the set, is, oh, they were the two cards in the set that you ripped up when you used. And so when I saw those were the two worst cards, I immediately knew, oh, well, I guess players don't like ripping up their cards. But the funny thing was, the way that the market research people do their analysis is they take all the questions they ask and try to find extrapolative

data from what they're doing. So what they did is, oh, these two cars were by far the least popular, but we can't find any correlative evidence. Because they're looking at how people rated names and flavor checks and art and all this different stuff. And from the data they asked, they go, wow, we can't find any correlation between these two cards. But wow, these are by far the most hated cards in the set. And I laugh because the thing that's funny about it is

they're using the tools they have at hand. The thing they didn't ask, they didn't do a question of, do you like ripping up cards? And people go, no, I don't. Oh, 99.9% don't like ripping up their cards. That's not a question they asked. And so it was instantly apparent to me what the problem was,

but it wasn't instantly apparent to them because they're analyzing the questions they asked. And from the questions they asked, there's no way to understand why those two cards are the most hated. But I find that funny. Okay, so we have a lot of different teams that do a lot of analysis and are trying to look at everything from what is selling well, what makes people play a lot, what is seeing,

what kind of cards showing up in the metagame, all the different vectors that we can do, everything about a set. Oh, another thing, like I said before, is a very, very important point of data analysis is having data over time. And so one of the things we do is once we decide to measure something, we keep measuring it. So, for example, if we want to understand what players think of, let's say, different worlds, well, we can do surveys and ask them against each other of how do these worlds do, and we also have the data that every time we ask, we ask the same questions, so we can look

across time and say, when this app came out, and we asked them about the world or the names or the art or the cats, whatever, we can look and see. If you ever see me do my Storm Scale or my rabia scale articles um i talk a lot about what people thought of it and that is just me straight lifting information out of market research of what players say how do i know how popular mechanic is well people rate it on a scale and we compare that scale to the same scale to the rate of other mechanics and so i can go oh well over time you know over the last whatever 15 some years we've been measuring this oh, well this fell in the top 10% of mechanics

that's a pretty good sign this fell in the bottom 10%, that's not a great sign and so we can use that there's a lot of comparative stuff that we do as well magic has become more and more data driven as far as using data as a tool to understand

the other thing we've started doing more of is using market research, not on finished products, but on ideas that we're experimenting with to see what people think. Because one of the things that we're trying to do is, if we're trying to make something you guys like, some of it might be early in the process, develop things a little bit so we can get some ideas from audiences and get some general, hey, first impression, is this exciting to you? Is this something you want to do?

And so that also is a very useful tool. Okay. So now let's talk about how we use the data. First and foremost, there's a couple different ways. Number one is just popularity. Okay, we made mechanics. People rated the mechanics. And it's not just people rated the mechanics.

I talk to people. People give feedback. There's a lot of feedback on what people think about things. So we have a good sense of, was this mechanic beloved? Was it liked but not, you know, liked it, but it wasn't their favorite. Was it something they thought was okay? It wasn't hated, but it wasn't, you know, it's okay. Or did they really dislike it?

They really not liked the mechanic. And popularity teaches us a bunch of things. In general, the more people like something, the more likely they are to bring it back. In fact, the more they like it, the more specifically it might return. Let's take mechanics as an example. If we make a mechanic and players really love it,

assuming that it can fit other worlds and there's design space, we'll bring back the mechanics that people like. We often bring back a mechanic in a set, and we bring back mechanics where there's evidence that players liked it. So popularity is one thing. Note that it's possible for players not to like something, and us to

believe that there's certain execution of it that we somehow could change. Chroma being the poster child here. Chroma was a mechanic that we hinted at in Future Sight, and showed it for the first time. Eventide, it didn't go over that well. It wasn't disliked, it was just kind of eh. And I decided there was a mechanic that I thought had a lot more potential

than it sort of showed. So in Theros, I brought it back. I dressed it up. I tweaked it a little bit, gave it a better flavor, and it became Devotion. And Devotion was very popular. So it was just an example of sometimes we do something, and we can use the information to say,

okay, something about this isn't quite right. If we're going to do it again, let's figure out how to do it right. And there's a lot of using the market research to try to get a better understanding of why things did or didn't work. Another thing that we can do is sometimes players show interest. Like sometimes there's a thing that was just a minor theme, but that players showed interest in it. Or it was just a single card, but players showed interest in it. You know, a lot of magic mechanics have been designed because we made a single card and players liked the single card and were like, hmm, maybe we can do more with this. A classic example would be Mistform Ultimis.

Mistform Ultimis was a legendary creature in, I think, Betrayers of Kamigawa. And the Mistforms could change their creature type. Mistform Ultimis said, eh, I'm just every creature type. Don't gotta change me, I'm everything. And later in Lorwyn, when we needed a way to sort of make glue, to fix things up, we ended up using that mechanic because I said, oh, well, I know people like it. We put it on a single card and people really like the individual card.

Okay, you know, like in some ways, I find that some of our field testing is individual magic cards. That sometimes we want to do something that's a little out there. Eh, try one card. Maybe put it in a supplemental set if it's really out there

or in an unset if it's really, really out there. Engage. What do people think? What do people think of this? And, you know, that is a neat and interesting way to sort of do some research while in that process of making the game. Another thing that we will do is we will sort of figure out what elements players liked and disliked about something.

So one of the things is even the best set in the world has things that could have been better. Even the worst set in the world has things that did well. So another reason to use the data is to try to understand within things we did, where do the successes and failures lie? And where was the room and potential to do something different. That's one of the reasons why the open-ended comments and the social media are so important is

nowhere else do we really ask your opinion about what could we do. And those are the two places where we gather information about what would you like us to do. That is very important. Like on my blog, I make a conscious effort when people ask me questions that I think are interesting questions

that I don't know the answer to, then I'll turn the question back on the audience and say, you know, I call people who read my blog question marks. So say question marks. What do you think? Do you like this? Do you not like this? Should we do more of this in the future? Should we do less of it? And then I will use engage to get

a general sense of what people feel about an aspect. And that helps guide me in the future when I'm making new sets to figure out where I can push and where are there areas of player happiness and areas of player unhappiness. And obviously lean toward the former and lean away from the latter. Another way that we use market research is, like I said, it's for marketing. I mean, I would say that marketing,

so one of the things that I have to do when I work on vision design, so we have something we call KSPs, which means key selling points. And that's just, I don't know, an insider way to talk about what's the compelling part of the set. One of the things that I get asked

about any set that I make is, why will players want this set? What about this set's exciting? What will make players go, ooh, I want this set? And if I can't answer that question, if I can't say, here is why players will want this set, my bosses will say, well, keep working on it, get back to us, tell us what, you know,

there needs to be a reason people want to buy the set, usually multiple reasons. And Envision, as somebody who's setting sort of the guideline of what makes this set exciting, I have to figure that out. I got to figure out, you know. And the reason the market research is so important is that a lot of me finding new spaces to play in is me looking at things we did that got a positive response. So, for example, when I was trying to sell Bill Rose on OnSlot having a major tribal theme, Bill Rose on Onslaught having a major tribal theme,

a lot of the evidence I was using with him was the data we had that showed players at the time were playing tribal decks even though they sucked. They were horrible. Early tribal decks were not powerful, yet they were popular. And so I was able to sort of say to Bill, look, once again, anytime something is popular without being powerful, that's a big indicator that there's something about a player's life. And I was able to use that data as a means to convince Bill to let me use that topic as a theme. And I'm always on lookout for new themes that I think are exciting. Well, the only way to, well,

I'm always on lookout for new themes that I think are exciting. Well, the only way to, well, not the only way. The major way to find that is to do sampling in our sets and then look at the sampling and see what people think. And one of the cool things is, because we're constantly making new cards, we have lots of opportunities to sort of try out new things, and then we can gauge from that what players do and don't like. So it's a perpetuating system, which is good, which is we make cards, players like some of the cards,

we use the cards they like to make a new thing, which now creates an impetus to make more cards, and that's ongoing. The other thing that I personally use market research for is I like to communicate to all of you why we do the things we do. And that one of the things that's very important for us is, um, when we are like, not every magic player is the same, not every magic player wants the same thing. And so where I can, for example, it depends on the decisions, what elements of the game they fill up. So let's say we're talking on a card level. So

for example, the psychic graphics, I like to make Timmy and Tammy cards, Johnny and Jenny cards, and Spike cards. I want to make sure that all of the psychographics have stuff that makes them happy. And traditionally, for example, the Johnny and Jenny cards, when we do our rare poll, don't tend to score well. And the reason is, the things that Johnny and or Jenny like about those cards

is they make you build around them in weird, interesting ways. There's something about it that's quirky that says, hey, maybe there's something neat you can do with me. But, if it's too straightforward, they tend not to like it. So what they most like is here's a weird thing. What are you going to do with this? And for somebody

in a vacuum that isn't excited by what are you going to do with this, they look at it and they go, I don't know, this is weird. Why would I use this card? And they rate it low. But because I know there's players out there that really value those cards, that those cards can be the thing that makes magic magic for them, we make sure

to keep those. But that's on a card-by-card level basis. That's easier to do. And that, one of the things that when we gather information is we spread out to understand that, like I said, there's not a singular magic audience. There's many magic audiences that want different things. So part of making people happy and part of using this information is not like just making one singular decision. It's making a lot of small

decisions. Oh, certain players like these kind of cards. Let's make those. Other players like these kind of cards. Let's make those. And we'll look at formats. We'll look at style of second graphic. We'll look at the aesthetics, you know, the Nel and Vorthos. We'll look at all the different aspects of why people like things and make sure that we address

all those different aspects. You know, some people are really about art. We want to focus on that. Some people are very about story. We want to focus on that. Some people are very about world building. There's a lot of different ways to enjoy magic. There's a lot of different formats to enjoy magic in There's a lot of different formats to enjoy magic in.

In each one of those, we have to be very specific. The area where we get into a little bit of trouble is where we have to make a unified decision for the whole set. So, for example, the setting, the plane that we're on. One of the things I get is a vocal minority will want us to do something. Kamigawa being a very classic example on my blog, where there's a lot of people that are fans of Kamigawa.

A lot might be the wrong word. There's a very invested group that is excited by Kamigawa. They tend to be in franchise players, and they tend to be on my social media. So whenever I ask about it, I get a lot of people jumping in about how much they love Kamigawa. And then I have to say, oh, well, we have a lot of data, and the data is saying that while there's a small group that is really excited by it, the larger group is not excited by it, and there's other things that they're more excited by.

And when I point that out, I'm just trying to explain, like, one of the things I know that's hardest for players is that you play Magic, and you feel about things strongly, and then we do something that seems to contradict what you feel strongly about. Now, if it's something where we can deliver on a small amount, like on a single

card, then we do that. Now, there are ways to deliver on planes. One of the things we try is, in supplemental sets, to do individual cards that might visit those planes and things. You know, we've definitely done a bunch of different sets, Commander especially, where like, hey, here's some cards set on Kamigawa that are Kamigawa things. You know, and we do that from time to time because we understand there's an audience that likes Kamigawa. It's not a large enough audience necessarily to do a whole set dedicated to Kamigawa,

but at least it's something we can sort of do at a smaller level. But anyway, players are very invested in how they see the game, and whenever the message is, what the game is to you isn't necessarily that to the majority of players, it's a little disheartening. Like, for example, let's say you love Kamigawa, and you want us to go back there. Me constantly saying, I don't know, you know, on a scale of one to ten, with ten being, I doubt we're doing it, it's an eight, does not make people happy, because they want me to say,

oh, well, okay, the thing you love for sure will do that, you know, and so one of the tricky things is, as someone who communicates to the audience, is the reason I communicate is I want you to understand our decision-making process. I'm big on transparency. I'm big on you understanding why we do or don't make decisions. And a big part of that is the market research. And the reason I did today's podcast is I think there's a lot of misunderstanding how we collect the data, or how we use the data, or why we care about the data, and part of what I'm trying to say is, we want to make Magic the game that you guys want it to be, and it is a tricky thing to do, there's a lot of different requests, there's a lot of contradicting requests.

But we do the best we can with what we can understand. And the reason we gather all the data we can and the reason we analyze it the way we do and the reason we use it in our decision-making is we truly believe it ends up with a game that is closer to what you, the audience, want the game to be. And the reason I'm spending all this time today talking about this is we do not do market research as a means to not give players what they want.

It is not a means to somehow create evidence to not do the thing players are asking for. The reason we do market research is to make the best magic product we can to make people the happiest that we can and well i know it can be frustrating when i explain that what market research is telling us might not be exactly what you want um it is it is not a uh it is done out of a means of trying to make magic the best it can be not we never want to ignore the voice of the audience. The audience wanting something doesn't always mean it happens, partly because sometimes the audience doesn't represent the full audience,

or sometimes there's other factors at hand. There's things the players might want us to do that other factors keep us from doing. There's things that players might go, this would be awesome, and we're like, okay, there's business reasons why we can't do that, or logistical reasons, or whatever. I remember

when I first did Innistrad, I talked about the way I wanted to do the double-faced cards, where you had a single-faced card that then got the double-faced cards, and when I explained that, a lot of people got mad, like, why didn't you do that? And I'm like, well, we couldn't. Logistically, we couldn't do it. The printer wasn't able to do it, or not at the accuracy rate that was acceptable to us. So anyway,

just because we can't do something doesn't necessarily mean that, I mean, it can mean it's not what the majority wants. It can mean that it's something we can't do, or we choose not to do. But anyway, I just wanted to understand that data collecting is very important to us. Oh, and I see today, I had a lot of traffic today. So you guys got an extra long podcast

all about market research. Who wouldn't want to learn as much as they can about market research? So anyway, I hope you guys enjoyed today's podcast and that it illuminated

some things and told you a little bit about how we function. But I'm now parked, so we all know what that means. It means it's the end of my drive to work. So instead of talking magic,

it's time for me to be making magic. I'll see you guys next time.