Game Theory - Did Dream FAKE His Speedrun? A Final Analysis.

Episode Date: June 6, 2024

Join former Game Theory Host MatPat as he dives into Dream's Minecraft Speedrun and investigates to see if it was faked or not. Credits: Writers: Matthew Patrick and Justin Kuiper Editors: Dan "Cybert" Seibert, Alex "Sedge" Sedgwick, Forrest Lee, Tyler Mascola, Koen Verhagen and Shannon (Bomb0i) Assistant Editor: AlyssaBeCrazy Sound Editor: Yosi Berman

Oh, hey Matt. Oh, hey! You? It's me, Robert. We sat next to each other for like three semesters in college. Yeah, Richard. Robert. Robert, yep. Have you been? What are you up to these days? Done with college, I presume? Uh, technically no, since I'm a professor now. Ha ha ha ha. Teacher humor. Anyway, what are you up to?

Well, uh, my life definitely took an unexpected turn. I'm, uh, I'm actually a professional YouTuber now. Oh, God. Don't tell me you're setting couches on fire in your backyard. No, no. Those people are the worst. No, I have a show where I actually talk about video games. Well, someone got lucky. So what do you do all day? It's about what you'd expect. Play video games. Make fun of video games. Read 29-page papers about binomial distributions in a series of Minecraft speed runs.

Wait, really? Ha! No, no, I'm just kidding. Only part of the paper deals with binomial distribution. The other sections describe modular arithmetic and the properties of linear congruential genera... Hello, Internet! Welcome to Game Theory. The show that speeds through scientific topics for the benefit of those with short attention spans, yet still manages to be longer than a Minecraft speed run.

It's true. As of November 3rd, the world record for speed running Minecraft 1.16 is 14 minutes, 39 seconds, and 520 milliseconds. Millisseconds! That is how granular we're getting here, people. And that should show you that Minecraft speed runs are a serious business, which is probably why a Minecraft speed running controversy says, the internet on fire all throughout December.

Because what better way to capstone a year full of crazy headlines than people taking the world's most relaxing game and turning it into a war zone? Dream, SMP excluded, of course, that thing is always a war zone in the best way possible. But let's just say when the Minecraft speed running team puts together a 29-page paper claiming that Dream cheated while speed-running Minecraft, and Dream in turn hires an anonymous astrophysicist to offer a 19-page rebuttal, all of it dealing with sampling bias and binomial distribution and computer, modeling and statistics. It's kind of a big deal. When people are busting out latex, you know it's serious. I'm glad that maybe one of you got that reference. For everyone else, I'm not talking about latex plastic. I'm talking about latex a document preparation software that allows academics to get

all those weird mathematical symbols into their docs. Because if you've ever tried to use word, it don't work. So why then am I here? Well, honestly, I wasn't sure I wanted to cover this. I don't want to insert myself needlessly in drama. Also, this is future Matt Pat dropping in. We're finalizing this episode now to release in about a week, but Dream just issued his final word on matter on Twitter. I don't want this video to be seen as beating a dead horse or reopening old wounds. Just took me a long time to work on the math, all right? And these videos take forever to edit. And while I could have just shelved the episode and taken the L on this one, now that the discussion is officially over, I think us being late to the party is actually a good thing, since there are a

lot of interesting takeaways in all of this, which I'll get to later in the script. Anyway, just wanted to drop in, let you know that I recognize this is coming way late. And I think you this video, especially the conclusion, is still very valid. Now back to the episode as scripted. Personally, all of my experiences with Dream have been very positive. Dream has been fantastic to work with on both our collab video and as a guest on our charity livestream last year. Plus, I really admire the work he and the crew are doing on the Dream SMP. But... It's not every day that people are excited to talk about statistics.

And a lot of these discussions have been very technical and complex in a way that's pretty difficult to understand if you're not comfortable with math. And honestly, that's the foundational reason that I created game theory. This show was originally designed to be a place where I could break down a lot of big and seemingly complex topics and fun, more digestible ways. These days, it's mostly big complex topics like the FNAF timeline, but also things like physics and history, and yes, even math. Do you want to know more about the games you love without having to put in any work of your own?

Well, now you can play and learn. We have created game theory, gaming's tangential. learning experience. Oh, the cringe. It is so bad. It's so bad. It's blinding me. Anyway, all joking aside, I was lucky enough to have teachers that got me excited to learn, who were good at explaining things, but I know other people aren't so lucky. So I always wanted this channel to be that place for whoever found it. To explain why these sorts of topics are actually a lot more fascinating than you might think. And this is absolutely one of those situations. It's also one of those situations where I think we all might be missing the bigger point in question, But I'm gonna get to that one a bit later. Understand, we all okay here. I'm not trying to attack anyone. This is purely meant for education. There's a lot of misinformation circulating here.

And as that cringy old promo of game theory said, knowledge is power. Oh, I played the cringy promo again. What was I thinking? So I think it's worth starting from the beginning. Dream is a Minecraft speed runner. That's a bit reductive since these days he does a lot more than just speed running,

but the first time he appeared here on Game Theory, he introduced himself by saying, I'm Dream, I'm a Minecraft speed runner. So yeah, I'm gonna go with that. Anyway, Dream, at various points in the history of Minecraft's speed running has held world records. Back in March 2020, he completed a run of Minecraft 1.14 in 32 minutes and 21 seconds, which at the time was a world record by 16 seconds.

It was beaten several days later, so Dream came back to reclaim the crown with a time of 26 minutes and 44 seconds. In June, he once again claimed the title of World Record Holder with a run of Minecraft 1.15 in 22 minutes, 4 seconds. Yes! In fact, to this day, Dream's run is actually the fastest this 1.15 run on record. Though in an interesting twist, he's not actually number one on the leaderboard, since there are five people that have actually completed Minecraft 1.14 with faster times.

Why would that matter? Well, 1.14 and 1.15 are similar enough that they're grouped together as a single category. Though fun fact, 1.15 is actually considered to be the harder for speed running due to the removal of a trick that lets you infinitely restock villagers. And even amid the current scandal, dreams 1.15 remains on the leaderboard, without any disputes over its authenticity or validity. And this leads us to the controversy, which didn't start until last October when Dream started playing Minecraft 1.16. That's the one with the Nether Update, an update that was big enough to core mechanics to get it listed as an entirely separate category of speed run. In the best of these runs, Dream was able to complete the game with an impressive time of 19 minutes and 24 seconds, putting him at fourth place on leaderboard. However, it also involved him getting very lucky. In fact, during his October speed running live streams, Dream got so lucky that it attracted the attention of several people who started to question whether his live streams were proof of more than just good luck. Had Dream actually modified the game? Now, a lot of people might be asking at this point, why scrutinized Dream in particular? After all, while his 19-minute 24-second run was certainly impressive and very lucky,

it's far from the fastest Minecraft speed run on the books. In fact, he's several minutes behind the 15-minute 12-second run that was the previous world record at the time. If you're looking for suspiciously good luck, why wouldn't you focus on the fastest times? Well, as the Minecraft speed running team explained in their 29-page report, the problem wasn't with any one of Dream's speed runs in particular, rather it was abnormally high drop rates of key items across multiple consecutive live streams. What kinds of key items? So, one of the biggest bottlenecks in any Minecraft speed run is crafting

eyes of Ender to use the End portal to reach the end, which requires both Ender Pearl and Blaze Powder to craft, hence the importance of these two particular items. And Dream, across these six live streams in question, was getting them at suspiciously high rates. Specifically, the investigation team noted that, quote, 42 of the 262 Piglin barters performed throughout these streams yielded Ender Pearls. And 211 of the 305 killed blazes dropped blaze rods. This means that for this specific set of speed runs,

Dream had an Ender Pearl barter success rate of 16% when the actual rate is actually much closer to 4.7%. and a blaze rod drop rate of 69.2% when the actual drop rate is 50. That might not seem like a big deal. After all, a 16% drop rate is obviously higher than 4.7, but are the numbers really that different to merit a cheating investigation? I mean, these are random events, after all. Sometimes you just get lucky.

So how does this all apply to Minecraft? Well, for example, you only have a 4.7% chance of getting an Ender Pearl from a Piglin barter. You might get lucky and get an Ender Pearl after, say, four attempts, leaving you with a 25% drop rate, but the more times you try, the closer and closer you should be getting to the true drop rate of 4.7%. And when you're talking about over 20 hours of live streams and hundreds of barter attempts, you'd expect the number to be pretty much closer to that expected value, which, and I'm just going to call this fact out, means that if cheating was involved in this situation, It was the fact that Dream showed his work that got him flagged.

It wasn't the individual record-setting run itself that set off the alarms. It was that, plus, all the other 20-plus hours of run attempts aired online around it. An individual run wouldn't have had enough drops to show any sort of subtle tampering or bias. In other words, if there was cheating involved, and this was just an individual run that happened offline, completely removed from any context, and then was submitted for consideration, none of this would have happened. There'd be no way to verify the validity of the run, unless, of course, I'm missing something here.

Just thought it was an interesting loophole that this whole situation brought up that probably should be closed up. Now, let's think about an extreme example of luck, winning the lottery. The chance of you buying a lottery ticket and winning the mega millions jackpot are roughly one in 300 million. Those are some incredibly low odds, but obviously some people do get lucky enough to win the lottery. I won a lottery! That's great! and it wouldn't make sense to track down that lottery winner and question that person about cheating. On the other hand, if the same person were to win the jackpot multiple times,

well, that's when fraud investigators are going to start getting involved. This example illustrates one of Dream's early responses to being singled out for the review, that he just happened to win the lottery. He got extremely lucky, and by looking only at those instances of extreme luck, the review team was seeing skewed numbers, quote from his response. Rather than taking a truly random sample, the sample starts when the luck seems off, And the sample ends as soon as I hit a personal best goal of 19 minutes. Although this can be touted as a random set of data, it is most definitely not.

End quote. This is an issue known as sampling bias. Referring to the fact that dreams speed running streams weren't randomly chosen for review, people spent hours pouring over the data specifically because someone noticed that it seemed to be getting luckier than normal. And obviously, if you scrutinize someone specifically because they seem to be getting lucky, when you look at the results, well, they're going to seem luckier than normal. So to combat the bias, what everyone has been keeping, calculating and arguing about in their massive research papers are the odds of drops this lucky happening.

Not specifically to dream, but to literally any Minecraft player in existence. You know the old saying about how if a monkey is hitting keys at random on a typewriter for an infinite amount of time, it'll eventually write the complete works as Shakespeare. It's based on the idea that something super improbable will almost surely happen given enough time and enough attempts. And it's an actual theory, a statistical theory, called the Infinite Monkey theorem. Anyway, same thing applies here. Dream's luck was extreme, but how extreme was it? Both the Minecraft speed running team and the anonymous astrophysicist that Dream hired to write a rebuttal have submitted their numbers, but people seem to be getting those numbers confused.

To directly quote from the original report, the Minecraft speed running team presented a number of 1 in 7.5 trillion. Well, that number is them basically saying, hey, let's assume that there are 1,000 Minecraft speedrunner monkeys at 1,000 keyboards. There's still only a 1 in 7.5 trillion chance of recreating the luck that we saw on Dream's streams. Conversely, Dreams astrophysicist determine the odds to be 1 in 100 million. The number 1 in 10 million gets thrown around a lot from Dreams Rebuttal, but that's actually a different number. Comparing the two is actually like comparing apples to oranges. If you're looking for the best comparison, it's the speed running team's 1 in 7.5 trillion versus the astrophysicists' 1 in 100 million. But still, that leaves us with a massive difference in calculations.

Why is that? Well, as much as I'd like it to be, it isn't something that has one big answer, but instead, a lot of smaller answers, which, you know, is why these arguments are happening in 20-plus page documents. It includes things like differences in the application of the stopping criterion, different estimates for the total number of live streams, other differences to account for sampling bias, and differences in accounting for P-hacking bias.

We've already covered the sampling bias discussion in broad strokes, and we're about to touch on that last one, P-hacking, but as you can see, it's a lot. much more than I actually want to get into here. If you want to get into the nitty gritty of it, I highly recommend Mathemaniac's video breaking down all the arguments point by point from a mathematical, deeply statistical perspective.

Good job Mathematiac, keep doing the good work. But all of this, I just hope, illustrates the wider point here, that this is a complex topic with a lot of different questions to consider, all using numbers that, at the very best, are rough estimates that are very flexible. Which leads us to the last point I want to talk about P-hacking bias. P-hacking is, sometimes, sometimes also referred to as data dredging.

It's basically the practice of looking for patterns and selectively presenting data to come to a conclusion. It's practically the principle that this channel was founded on. For instance, look at our Mario Is Mental series of episodes from way, way back in the channel's history. In these episodes, I look through hundreds of Mario games to present you, the viewer, with selected data points to lead you to the conclusion that, yeah, Mario is indeed a sociopath. And to get there, I ignore decades of him, acting heroically and selflessly to save the mushroom kingdom. Though, to be honest, this is actually to this day one of the theories that I still believe.

I just can't ignore the fact that Mario whips D.K. in his early games. Or he grinds Luigi's foot when he wins in the tennis match. Anyway, this process is sometimes described as post-dicting. You get it? It's the opposite of pre-dicting. Instead of coming up with a hypothesis and then doing an experiment that confirms the hypothesis, you start with a pile of data, and then you use the data to come up with a hypothesis to explain it after the fact. So how does this apply to our Minecraft speed running controversy? Well, in any game of Minecraft, there are lots of sources of randomness, but thus far we've only been looking at two,

pearl trades and blaze rod drops. But what about other sources of luck? Like the frequency of blaze spawns or the number of piglins that you're actually able to barter with? These are all random too. Just because two variables happen to be higher than you'd expect, how do we know that we're not just selectively choosing the data to come to the conclusion that cheating happened? Well, it all depends on the number of variables. If there are only 10 factors to consider and two are unusual, well, then yeah, it's hard to deny that something suspicious is going on. But if there are, say, a thousand possible variables, well, then it's very easy to cherry pick two of them and come up with a false correlation. In fact, there is a really fun website dedicated to exactly this. For instance,

did you know that the declining divorce rates in Maine are actually tied to the decline in people eating margarine? It's true. There's a 99% correlation between the data point. The chart doesn't lie. But obviously our conclusion does. Divorce and Margarine are completely unrelated, unless of course marriages are suffering due to poor butter substitute choices. But when you're looking at a huge pool of data, you can come to practically any conclusion. And that's the other major point of contention between Dream and the speed running team.

How many sources of RNG are there that could be manipulated as possible sources of cheating? How large is this pool of data? The Minecraft speed running team made a quote-unquote generous estimate of 10 different factors. Their words not mine. Dream claimed it was closer to 37. And honestly, the frustrating thing about this is that there's really no clear-cut answer because it's not just about how many variables there are, but also how measurable those variables are. I bring all of this up, one, to explain it a bit more clearly, and two, to point out that some of these calculations and differences between the numbers are based on numbers coming from assumptions that can feel kind of arbitrary, and that's where there's room for debate here. But all of this, for as great as it is to talk about statistics, I think is missing the point. Yes, it is essential to uphold integrity on speed run boards. A hundred percent absolutely super important.

But are we sure that this is a game that should have a speed run board in the first place? For all the back and forth on this topic, one thing that's not in dispute here is the dependence of luck in Minecraft 1.16 speed running. Well, the Nether Update was much beloved by many Minecraft players outside of the speed running community. Myself included, given the huge amount of theory fodder that it added to the game, the speed running community hasn't exactly welcomed it with open arms, and it's led to a bit of a divide. With Minecraft 1.16 considered so different from previous versions as to warrant its own leaderboard. The update substantially changes the amount of time needed to complete Minecraft under ideal conditions.

As I'm recording this, the fastest pre-1.16 Minecraft speed run has a completion time of 19 minutes 36 seconds. Whereas with version 1.16, the fastest time is a mere 14 minutes and 36 seconds. So why would 1.16 be so divisive? Well, it all comes down to how you go fast in 1.16. The thing that makes 1.16 speed runs so much faster, saving as much as 5 minutes compared to previous versions of Minecraft, don't have to do with tricks like bunny hopping or diagonal bridging or other methods that Minecraft speedrunners have spent years perfecting.

It all has to do with what we've been talking about this entire time, random events. And most specifically, the piglin barter. That Ender Pearl trade is where the extra speed of Minecraft 1.16 comes from compared to previous versions. And unlike bunny hopping or bridging, this isn't the kind of thing that you can get better at by practicing. Whether the bartering system gives you the items you need mostly comes down to pure randomness, and bad luck can ruin the most skilled of runs. Now obviously, RNG has been a big part of speed running, and especially Minecraft speed running since the start. After all, the most popular speed running category involves playing on worlds generated from a random seed.

But that sort of randomness is seen by many runners as the interesting kind, the kind that forces you to improvise along the way. To me, speed runs like that test the most important skill that you can have as a gamer, something that I call sight reading a game. Having enough fundamental skill and knowledge of the game's mechanics that you can just be dropped into it and know what to do in order to put down a solid time, no matter what curveballs the game may throw your way. But barter RNG doesn't really challenge the player that way.

It's more like a slot machine, where you just shove gold at an NPC in, hope for the best. And the swing in times that it can create is massive, larger than many other RNG elements from the past, maybe even speed run breaking, which merits the question, does it defeat the underlying goal of what being a speed runner is? It's not surprising that Dreams exceptional luck is certainly one of the most attention-getting and controversial parts of this whole incident. After all, he's a YouTuber with 15 plus million subscribers and even more who aren't subscribed to his channel. But that doesn't change the fact that with version 1.16, Minecraft has become a game where your ability to set a world record is most up to random chance. It's partly a measure of your skill, absolutely, but arguably the biggest thing that's being tested is your patience. Do you have the persistence to keep playing for the hundreds or thousands of hours it takes for the RNG to bless you with the luck that you need to get the best time? Is it even worthwhile to invest those hundreds or thousands of hours when the most important factor in your success is something that's completely out of your control? So then, what would the alternative be? Perhaps a more accurate yearly indicator of the best speed runner for that version of the game would be a tournament of speed run races, dropping two or more players into the same random world seed, and seeing who can roll

with the punches the best, who can use their knowledge of the game to get to the end of the end the fastest. You would probably have to make Piglin bartering off limits, but would that be any different from a Smash Brothers tournament where certain characters aren't able to be used because of imbalance or being broken? I don't know, it's just a thought that I wanted to bring up. And heck, I just really enjoy watching a competitive version of Minecraft, because let's face it, if that comes out of all this controversy, well, I'd definitely call that a silver. Well, that and everyone being forced to learn statistics in order to talk about it. But hey, that's just a theory.

A game theory. Thanks for watching.