Magic: The Gathering Drive to Work Podcast - #1114: Playtesting in 2024

Episode Date: February 23, 2024

Ten years ago, I released a podcast on playtesting, but a lot has changed. In this podcast, I look at how we playtest today.

I'm pulling down the driveway. We all know what that means. It's time for another drive to work. Okay, so today's topic is playtesting. So, whenever I do a topic, normally in the morning I figure out what my topic is, and then I search and look at the previous drive to works just to make sure that I haven't done the topic before, because I'm 1100 in, so I've done a lot of topics. And I had, in fact, done playtesting. In fact, I've done it twice. But both were in the 100s, and we're now in the 1100s. So that's like 10 years ago.

So I decided that every 10 years, I'm allowed to talk about playtesting. Things have changed. 10 years ago, we were back in the design and development era. Now we're in exploratory design, vision design, set design, play design. So anyway, I'm going to design, play design. So anyway, I'm going to talk about playtesting now. I'm sure there's a little bit of overlap from 10 years ago, but enough differences that I thought it was worth doing a podcast on. And the reason I'm doing this is a couple of people have asked me recently on my blog about they want more details of how we playtest. So I thought I would get into how we playtest. Okay, so let me start by talking about the most important tool for playtesting.

So in order to keep track of all the cards, we have a database that we call Drake. And the idea is every card that exists in Magic is in Drake. And not just every card that exists, every card that we are currently working on, like in order when you, when you make a file and you work off a file, the file is in Drake. And the idea is, it's so that everybody can interconnect and work with it. And so if I want to, for example, see how a certain set is going, I can go up and look in the file and look through the cards. And I can leave notes. They're called dev notes, developer notes. We can leave notes.

Anyway, now not only does it have every card that does exist, it has cards that could exist. So each designer is given what we call a sandbox. And basically what that means is you have a little area of the database that is yours where you can just make cards that you want to make because you want to test them. And so each person has a little, because normally all the cards go in a file, right? Normally when you make cards, they go somewhere. Oh, well, this card goes in this file. And so we wanted a place where we could experiment

with things. So each person got their own little sandbox and then we can make cards in our sandbox. And so the idea is, um, then there's a tool in the database called, um, a deck builder, I think is what the tool is called. So the idea is I can go into the deck builder. I can choose any number of cards depending on what format I'm playing. And, uh, all the cards exist in the database. I can pick whatever cards I want, including cards I've made, you know, that are in my sandbox, and then it'll make a deck for me. It'll print up a deck. Oh, let me explain the print-up-a-deck technology.

So when I first started at Magic long, long ago, the playtests were done on cardboard. That's how Richard... Sorry, Rachel's my daughter. That is how Richard did Alpha, that he printed on little bits of cardboard. When I got there, we were still printing on cardboard. The cardboard was a little closer to the shape of Magic cards. If you've ever seen an Alpha playtest card, they're much smaller than the normal cards.

We were playing with cards that were not exactly the magic shape, but closer to the magic shape. And they used to be, like Mirage was on green. All the cards were green. Anyway, so eventually we got the technology of making stickers. So we would print on stickers and then we had to cut the stickers and then we would sticker on top of old cards. And so like you make a sticker, then if it was a red card, it would go on a red card. Over the years, we got better and better with our sticker technology. Eventually, we added color. Eventually, in the early days, they just had text.

Eventually, we could put in sketches or final art. And they started making it so it looked like the frame. So we got advanced with our sticker technology. But eventually we went one further. And so how we do it now is we have Blink Magic Cards. We have the ability to get Blink Magic Cards. And then we just print directly from a color printer. We print directly on the cards.

So when I say if I want a deck, I can print a deck, I literally can go in the system and go, okay, here's my 40, 60, 100 cards, depending on what format you're playing. Okay, print up, and it'll just print the cards off the printer, and then you have a playtest deck. And that is how all playtesting, or mostly all playtesting, is done. Back in the day, in the early days of Magic, we used to get, for every set, we would get some number of cards from that set. What do we call them? But the idea is we'd get 100 of each card or something.

And then when we went to build decks, we literally would go get actual magic cards. That got kind of unwieldy as more and more magic sets got made. And also, over time, like, you know, people would make decks but not put things back in the right place. It would get messy over time. The database is a nice, clean, you can just print whatever you want, anytime you want. Also, not only can it print decks, but it can print boosters. So let's say we want to do a draft.

We can print up boosters and it will, for example, we now have play boosters. And play boosters are it will, like, for example, we now have play boosters. And play boosters are a little more complicated. Before, back when we had draft boosters, draft boosters were, I mean, not that there weren't exceptions to this, but it would be like one land, ten commons, three uncommons, one rare that won every eighth time. It would be a mythic rare. And so nowadays with play boosters, there's a higher variance on what slots can be meaning, oh, this slot has some percent chance of being a common or uncommon or rare or mythic rare. So we can program such that it makes the booster and it matches

how the play boosters work. We even can mimicils. Um, and the person, the lead designer has flexibility when printing to say how it wants to print it. Um, for example, um, let's say you were doing Dominaria, for example. Dominaria had, uh, every pack had a legend. Well, we could program that. So every pack pulls from the legends and every pack has a legend. Well, we could program that. So every pack pulls from the legends and every pack has a legend. If you're doing,

for the industry, you could have a slot that's a double-faced card. So there's a lot of, I will say that double-faced cards are the one thing the printer, when we did stickers, we would just sticker front and back. We can only print on one side of the cards.

I'm trying to think. I think what they do is when they... I think when they do double-faced cards, they'll print two versions, like the front and the back, and then they'll put them in sleeves. And so it has a front and a back, but it's physically two cards. I don't think they can print on the front and the back of the card. Partly because there's a back on the card,

but also because the printer just prints on one side. But anyway, okay, so we have this tool so we can, if we need to print magic decks or magic boosters, we can do all that. Let me just set up. That is a tool we have at our disposal. A very important tool. Okay. So what I'm going to do is I'm going to walk through the four stages of design and talk about how each stage uses playtesting because it will vary. I'm more familiar with exploratory design and vision design because those are the parts I run.

I have some awareness of set design and play design. I did do set design for Infinity, so I actually have led one set design. Okay, so let's go to the very beginning, because that's the best place to start. Exploratory design. So exploratory design, the whole premise of is we just want to test a lot of different things. We don't build a file yet. It's not until vision design that you build a file. So playtesting exploratory mostly is making sort of a pre-constructed deck.

And the way that'll work is, let's say we're testing Mechanic X. You'll make some number of cards in Mechanic X, usually somewhere like six to ten, maybe a little bit more if needed. But most of the deck will just be reprints. We can pull them from anywhere. Usually what you want to do when you're testing something new is you want the rest of the deck to be a little on the plainer side. You don't want a lot of complication outside of what you're testing because it muddies the water a little bit.

So a lot of the stuff we put in tends to be like corset style cards. But if you're trying to play into a certain theme, you know, Mechanic X is a graveyard mechanic. Well, maybe I put milling or, you know, I might put something in that helps me get stuff in my graveyard. So you can put cards in that help you do what you need to do to test off a mechanic. But usually everything surrounding the things you're testing, you want to be a little on the simpler side. Another thing is, while normally you, I mean, the new mechanics, you have to make those cards. They don't exist yet. You tend to fill it in with cards reprints if you can, but let's say you need something and it doesn't exist. You could also make other cards.

Sometimes you'll make the support cards because there's just not a clean version of the support card. So rather than use a complicated card that has extra tech for no reason, maybe we'll just make the cleaner version of it. Once again, with the sandbox, you can make whatever cards you need. So in Exploratory, the idea is you are making a deck that just represents the mechanic you're trying to play. You tend to make the ASVAN a little higher than normal because you want to make sure that you test off the thing you're testing.

The big thing about Exploratory design, some extent vision design is it's more about sampling than anything else. That you want to just, you want to get experience playing something and see how it feels. And so what that means is you want your ass fan to be a little bit high just to make sure you get there. We have done enough exploratory design that we understand that things are a little bit high. We understand that it's not representative yet of the environment that it'll be in. We're not worrying about that yet. It's just

showing what it can do. Now normally in exploratory design, each designer can bring can make their own deck. Different people will make different number of decks. And we tend to save decks. Oh, so in Exploratory, you end up with three buckets. You playtest a mechanic, one of three things happen.

So bucket number one is that's good. That means you don't need to playtest it anymore. Write it on the list of mechanics for vision design to look at and move on. Bucket number two is this is no good. Usually bucket number two is like, wow, this didn't work at all. Normally when that happens, you scrap them. Okay, let's tell vision design we did it so they know stay away from this area. It's a dud. And then bucket number three is there's something about it.

It wasn't a dud, but it wasn't a success. It's kind of in the middle saying, well, there's something there, but we haven't quite found exactly what. Bucket three is most often the bucket that you will then redesign. Bucket one, you don't need to redesign it. It's good. You're done. Bucket two, usually, if it's in bucket two, it's like's like wow this isn't very salvageable sometimes maybe you try to salvage it normally you rework stuff in bucket 3 just to try to get

versions that you think are good every once in a while bucket 8 you find something you like and then you might try riffs on it just to see if the riffs that go in different directions will also work but anyway as you find successes, you write them down. Sort of what exploratory hands-off to vision design is, here's a list of all the mechanics we tried. And here's what we liked, here's what we didn't like,

and here's what I think shows promise, but we didn't find the right execution of it. Okay, so most of the playtesting in exploratory design is just testing things. What'll happen is you'll come to the meeting, you will have built zero to four decks, let's say. And then, between

everybody, usually there's enough decks that you have some stuff to play. Not everybody brings a deck to every single playtest, maybe you have other stuff going on. And some people when they build playtest decks build more than one. Also, it's very common sometimes when you build a deck to print more than one. So even though it's the same deck,

maybe I print two so somebody else can play that deck. And usually what happens is you tend to play different things against each other. We also have what we call a control deck, which is sort of just a normal deck with normal cards that sometimes we'll play the new deck against the control deck, which is just, here's a normal, simple magic deck. What we've discovered is that it's just

faster to play two decks with two new things in them against each other. You just learn more quicker. But sometimes we'll use the control decks if we only had, only one deck got made that day or something. Sometimes we'll use the control decks. Anyway, Explorator Design is mostly playing pre-made decks against pre-made decks. You are testing to find out whether a specific thing works. Okay, now we move on to Vision Design. Okay, Vision Design, we are now starting to make a file. And what that means is we're going to make commons, we're going to make uncommons. We're going to make rares and mythic rares. Now, the commons and uncommons are the most important when you're building out the file. So for those that don't know my metaphor,

vision design, you're building a house. Vision design is the architect. You're making blueprints. Set design is the builder. You're building the house. So what you are making with your file is what we call proof of concept, which means we're trying to demonstrate how we think the file will work. When it gets to set design, they might keep some of the things you've done. Some of the cards we make in vision design end up getting printed, but they'll rework a lot of things. And so it's, once again, it's not that any one card will necessarily say some can. It's more sort of demonstrating what you're trying to do big picture. When we hand off the file, we always have a full set of commons and a full set of uncommons.

We normally have a full set of rares, but not always. The rares are less important to hand off. Usually we want enough rares that we can sort of have a draft and have rares in it later on in the process. But rares are not quite as important as getting the comms and uncommons done. So anyway, you start building the file. So the earliest playtests, well, when you're first testing on mechanics early in vision, sometimes you'll do what we do in exploratory. We'll make pre-constructed decks.

And sometimes midway along the way, when you're figuring out archetypes, sometimes every once in a while you'll build the archetype as a deck. But most of the time in Vision, you are playing sealed. And, well, I should say this, you're playing limited. The early part of Vision, you tend to play sealed. So let me explain again in a little more depth. So in Vision, early Vision is all about sampling. part of vision you tend to play sealed so let me explain again um in a little more depth so in vision when you're in early vision it's all about sampling i have ideas i want to play them i want

to see how the ideas play i want to see how different mechanics intermingle i want to see how themes are working i'm just sort of sampling things it is not till set design when they actually build the real file um and not even until well set design starts sort of adjusting cost correctly in vision design everything is costed to be playable um meaning you have a flat power level what that means is in actual magic um not every card is pushed some cards are pushed for constructed some are pushed for limited. In order to craft the environment, play design along with set design will figure out what cards they want to be stronger. As a general rule, we don't make every card as strong as it can be because that's power creep. We pick and choose where we push cards based on what we think the best play patterns are.

In vision design, we're not yet testing the environment. So what that means is we just cost everything at an equal power level so that everything can be played because once again, we're sampling. So early vision design, we do sealed. Sometimes in very early vision design, I will assign colors to people. You're playing black red today. You're playing white green. The reason I would assign colors in an early playtest is just to make sure all the colors get played. One of the things that can happen early on is you haven't really done a lot of balance yet,

so it's quite possible that certain colors are just better. And if everybody just makes the best possible deck they can, you might not test colors. And once again, we're trying to sample stuff, so early on you want colors. Sometimes when I assign colors, I don't even give people cards that aren't that color. I just give them cards of the color of the playing. Normally in Sealed, we kind of mimic

what you guys do in the real world. The equivalent of six boosters is normally what we do. And I would say, so Vision Design, Fortune Design is about two months long. Vision Design is about four months long, and the first two months is usually just sealed. Somewhere in the middle of the third, somewhere during the third month, usually once we have our draft themes, I mean, once again, set design will change some of them,

but have a rough first guess at our draft themes, it makes sense to start drafting. Once you have enough of an environment where people can pick and choose cards and you can see how people fight over cards and stuff, that's when we start drafting. Normally, we like to playtest in Exploratory. In Exploratory, we playtest almost every time. In Vision Design, usually for the first two months, it's roughly like two weeks of design

and then you playtest. About every two weeks, you playtest early on. And then in the second half of vision design, I like to playtest every week. The way our meetings are set up for vision design is you have two one-hour meetings every week

and one two-hour meeting every week. Usually we use the two-hour meeting to do our playtesting. Now, that, by the way, is internal playtesting to the team. So that is you use team time with the team drafting. Now, once we're drafting, our teams will have four to six people. We normally will try to recruit people to get up to eight. We can draft with four. We can draft with six.

I mean, but eight is the best experience because in the real world, eight is what people are going to draft. So we try to get eight if possible. If we can't, we draft with less. Now, the other thing I should point out is there are, there is playtesting opportunities that go on beyond the team. So every morning, I think, yeah, every morning, I'm not sure whether there's playtests on Friday, but every morning at like at 9.30, there's a playtest. And the idea is different sets get put in the playtest. It's usually different sets that are in some form of being made. And then there's a rotation with the schedule. Usually the leader of the set is there. And it's an opportunity for people outside the team to draft. You get fresh eyes.

You get people, because most of R&D is working on some number of sets, but they are completely out of other sets. So the average person in R&D has blind spots. Oh, I'm working on set A and D and F, but B, C, and E I'm not working on. So you're good for playtesting those because you're not familiar with them. And it's nice to get fresh eyes because when you work on a set for a long time, you get used to things and it's hard to remember what it's like when you first approach it. So having fresh eyes is really important. The outside playtest, usually Vision Design only does that maybe in the fourth month of playing. We don't want outsiders to draft until it's in a form where it's useful to get the feedback.

And so the outside playtest happens for all of set design and play design, but usually just the very end of vision design. Okay, so that vision design does seal. Later it'll do draft. It can do pre-construct every once in a while. It has the outside drafts that it can do. Okay, let's move on to set design. So set design is starting to build the real file.

And, oh, one last thing I should mention about vision design. Because vision design is not testing environment, it is just sampling. Usually in vision design, we're a lot looser. Like if you have a bad hand, don't mulligan, just draw a new hand. There's not a lot of value when you're not testing environment to test stuff like mulligans because the environment isn't even there to test. Once you start getting into set design, it starts getting more real. They're starting to do mulligans and a more real, like the, they more mimic how people will play in the real world.

Okay, so in set design, usually early on they do a lot of drafting. Drafting is a great way. Set design does have to learn the set, and drafting is a good way to play a lot more cards. When you play constructed, there's only so many cards that make sense in constructed. cards. When you play Constructed, there's only so many cards that make sense in Constructed. Way back when, we used to do

there used to be a format called Block Constructed, back when we made blocks, and we actually would have Protors on it. There was a period of time where we would test Block Constructed as a way to test stuff. It's not really a format anymore, so we don't do that. But anyway, Set Design will do a lot

of drafting. I don't think they do much anyway, set design will do a lot of drafting. I don't think they do much of any sealed. They mostly just draft. But they'll do a lot of drafting. They'll even have days that they go off-site and draft. Or I guess maybe that's play design that goes off-site to draft. But anyway, there are days to do that.

And set design will also start doing constructed playtesting. Constructed playtesting means, like I said early on, you can take a deck, you make a deck, and then you play it against people. And so the later on you get in set design, the more constructed playtesting that happens. And they continue doing draft. Draft gets done the whole way through. I don't think set design does as much pre-constructed stuff other than sometimes when they need people external to the team to sample it, the architects, people that are like, they're checking sometimes with other people external to the set. And one of the things that you'll do sometimes

is you'll build pre-constructed decks for those teams to play. So then they can just sample the themes you want to do. Oh, I did mention, by the way, at the very end of Vision, or the beginning of the fourth month, we have what's called the Vision Summit. I did a whole podcast on it. That is usually a draft. And that's another checkpoint.

There's a lot of checkpoints that happen along the way. So set design, like I said, sometimes we'll make pre-constructed decks so that other teams outside of R&D can play decks and understand. The marketing team, for example, at some point wants to come and understand what they're marketing. So we tend to make these pre-constructed decks for teams like marketing, which is, hey, here's the main themes of a set built into deck form. Just play them and you'll get a sense of what the set is about.

Now, play design will start... There's an overlap between the start of play design and the end of set design. So play design is something they call the future-future league. So I've told this story before, but it's a fun story. So once upon a time, many years ago, R&D created what they called the Future League. And the Future League, I think, played six months ahead of time. So it allowed them to see where the standard would be in six months' time. Or at least, having a guess, you would play a standard as it would exist in six months' time.

The problem they learned was they would learn things from it, but it was too late to make changes in existing cards. And so all it really did is just kind of taught them what might happen in the real world, but it wasn't useful in the sense that it was too late to make changes. And they're like, okay, that's silly. If we're going to test things, we at least want to test things in a way where we can make changes. So they made what they called the Future Future League, what now is called the FFL. So they could test, I think, about a year out.

And the idea is a set will enter the Future Future League. This is play design. And that means you're allowed to use any cards that are in standard at the time of the set your testing is released. And then you just play decks from that standard period. Depending on what you're making, like if you're making Modern Masters, we'll have people in to test Modern and stuff like that.

There also is a casual play design. There's a competitive play design team that's testing stuff like Standard and Draft, and there's a casual play design team. They're testing stuff more like Commander. And so I think competitive play design starts a smidgen earlier than casual play design. But both of them, they will do constructed play tests.

Constructed will, let's say they're testing standard. They'll build standard decks if they're testing draft. They'll make draft boosters. Yeah, I believe it's play design that does the offsites every once in a while. They do an offsite to test draft environments. And what they'll do is they'll go offsite and just draft all day long.

And then take rapid notes. And a lot of trying to fine-tune drafting is figuring out, you know, what are the strongest colors? What are the weakest colors? You know, where do we need to adjust to sort of balance things? Once again, I

always say this when talking about play design, we're trying to make a system complex enough that the audience doesn't crack at day one, meaning we're making something more complex than the team can necessarily understand. They push in certain directions. They have certain percentages.

So they have an idea of, we think these cards have a greater chance of making it. These cards have some chance. Like, there's sort of percentages of how likely the cards are to be constructed and in various formats. The most testing gets done in standard and draft by competitive,

depending on the format. Sometimes old, you know, Pioneer, Modern. You know, a lot of the larger formats, we can track and get data from how they're doing in the real world. And we use some of that data. For stuff like Modern Masters, where we're making a set specific for Modern, we do bring in pros to do playtesting on modern. So set design and play design mostly are using the similar tools. I mean, they're mostly using decks that they form that may make for constructed and boosters that they make to play draft in. The other thing that happens is as set design, well, vision design is trying to do its job of guessing where things are.

And a lot of times we'll, we'll take big picture stabs at things. Um, for example, usually if we do a one of pack thing, as fan one, like meaning there's a slot in every pack, uh. Dominaria was legendary creature. In March of the Machine, it was a battle. Usually vision design will try to figure that out. But one of the things set design has to do is vision design is working a little bit more. I mean, we'll work with the people

that do collation and stuff to figure it out. But the hard, usually set design has to like finalize that stuff and figure out, oh, how are we doing this? What is, because like when you add a slot, let's say there's a double-faced slot, that changes the math of things. You want to understand how often things show up. And so vision design will do the start of this, but set design has to do the finish

of it to figure out. Oh, another thing that we do, we have what we call play days. I did a whole podcast on play days, if you want more detail, but the short version is about once a month, everybody takes off from their normal duties to have a day where we play one of the sets near completion all day long. The way play days work is we will have drafts in the morning and afternoon, we will have play days work is we will have drafts in the morning and afternoon, we will have sealed available, and we will have commander decks available. And so you can choose to do the same thing morning and afternoon, you can do different things morning and afternoon.

People tend to do the playtesting where, A, they're most interested, and B, they have the most sort of information to add, you add, the most feedback they can give. And the idea is it just lets, as I said before, there's a lot of blind spots. A lot of people don't get to see the sets. And R&D does a lot of the morning playtests. The design developers do a lot of the morning playtests. But there's a lot of other people on the Studio X team, the tabletop team.

And the playdays give all of them a chance as well. So, once again, you get more fresh eyes. Also, R&D, by the nature of the people who make the game, we tend to be a little more hardcore players. It's nice. A lot of Studio X has a lot more casual players, so we just get

a little bit more experience on the set that we're making. Normally for Play Day, it's far enough along that we're printing so you can see art. And like it's the cards, they're still printed cards off our printer, but we've made it such that they look pretty good. I mean, you know, they're not real magic cards, but they look a lot. They have a frame and they have art and it really feels like a lot closer to the Magic cards than stickers ever got, for example. So we have the play days. We have the Vision Summits.

We'll have check-ins where sometimes we will play. We'll bring certain people in to play with them. Like I said, with the marketing team sometimes we'll make decks and play with the marketing team.

They're just people downstream that need to understand the set so they can do their part of the job with the set. And so there's definitely playtesting with them so that they can get up to speed. And it's also possible because we have weekly playtests and because we have play days, sometimes other people will come to some of those. For example, often in vision design, I will have my art director come play in a playtest and I will have the creative lead come play in playtests so they get a sense of how the set plays because that might affect how they do the art and how they do the creative. And so you want to have people that are intimately involved early on actually playing the set and just getting a sense of what you're doing and how you're making it work. Anyway, that, my friends, is Playtesting 2024.

I did not go back and listen to my podcast from 10 years ago some of what I'm saying there definitely is some amount of early design is early design and the things we care about well the technology has changed a little bit we've definitely learned about how to get better at making sets at different periods, like the technology improves,

the basics are still the same. You know, vision design or early design is all about big picture stuff and understanding structure, where late design is about more fine-tuning of cards and, you know,

making each card the best that it can be. And so how you do the playtesting varies greatly on what it is you care about. But anyway, guys, I'm now at work. I'm in the parking lot. So I hope you guys appreciated an updated look at how we do playtesting.

But as I'm at work, we all know what that means. This is the end of my drive to work. So instead of talking magic, it's time for me to be making magic. I'll see you guys next time. Bye-bye.