PurePerformance - 072 Monitoring Minecraft – Why and What to Learn from it with Mike Villiger

Episode Date: October 8, 2018

Minecraft – the hugely popular sandbox video game – might not be your traditional software to monitor with an APM (Application Performance Management) tool. But Mike Villiger did it anyway in order to learn some advanced concepts in application monitoring such as custom entry points, thread diagnostics, method hotspots or simply to figure out why his mod’ed Minecraft sometimes couldn’t keep up with processing all the changes and skipped cycles. By using Dynatrace – first AppMon now Dynatrace SaaS – he learned more about the internals of Minecraft, how the single threaded architecture calls each mod and why a single mod must not take longer than 50ms to process. Mike gives us insights into which problems he found within Minecraft but more importantly what he takes away for his daily job as a performance advocate and evangelist. If you have any interesting side projects where you use APM tools let us know – there is always something to learn from every project!https://minecraft.net/https://twitter.com/mikevilliger

It's time for Pure Performance. Get your stopwatches ready. It's time for Pure Performance with Andy Grabner and Brian Wilson. Hello everybody and welcome to another episode of Pure Performance. My name is Brian Wilson and as always we have Andy Grabner with me. Hello Andy. Hello Brian. I still have to chuckle a little bit.

You still have the chuckles. Yeah. Hey, our guest is chuckling. We have a fun show today, don't we? Usually they're interesting. This one I think is going to be a little bit more fun a little a little i don't want to say off topic right but it's a little a little bit different today yeah but can you is this is it what what is it good to say it's going to be a fun show if we don't even know if the show is going to be really fun it's like a comedian and it starts it's going to be a good night well that's that's

why we gotta that's why we got to prep the audience. So who is the guest? Mike, are you there? How are you? Yes, I'm here. And this is your second time now, Mike, right? Or is this your third? I think it's the second.

It might be third. I don't really recall. It's hard for me to search by guest name, but I tried doing a Google search of pure performance and Mike Villager. I only had one hit, but it doesn't mean it's the truth. Do you expect me to believe you? So Mike, tell us a little bit about yourself first

and then tell us a little bit about the topic. But first about who you are for those people that don't know you. I want to ask you a bunch of questions. I got some questions too. Well, so I'm Mike Villager. I actually work for Andy. So Andy perhaps knows me a little too well. I used to work with Brian. So Brian knows me too well also.

So, you know, one of the things that I'm doing as part of Andy's team is, you know, kind of running our technical relationship with one of our partners, Pivotal. But, you know, I've been kind of in the APM space for quite some time. And, you know, one of the things that I'm going to talk about today is kind of a perhaps a little bit of a off the beaten path use of APM tooling. It's probably not something that folks would necessarily be able to do unless they had, you know, easy access to the tooling like I do, because it would probably be cost prohibitive most of the time. But I've been using Donatrace to instrument my modded Minecraft instances. And this is kind of something that I've been doing,

that I started doing back in the Atman days. And as we transitioned to Dynatrace, it actually became a lot easier to do this kind of in my basement lab. So first of all, I got a question. I got to admit, first of all, I have

never played Minecraft. Okay. I tried it once when it came out. Yeah. I'm not a gamer, but yeah. Well, it is a game. I know. I'm not a gamer, I out. Yeah. It's not a gamer, but yeah. Well, it is a game. I know. I'm not a gamer, I said, so. Oh, you're not a gamer.

So, Mike, can you, for those people like me that have, I mean, I know what it is and I've seen it, but for those people that have never heard about Minecraft, can you quickly explain what Minecraft is? All right. So, if there are a small handful of people in the world who have never heard of Minecraft, so Minecraft is basically a sandbox, right? You are presented with a world and you have nothing and you have to make yourself tools and you have to make yourself a home and you have to kill zombies and cows and sheep and chickens and dragons and like just all sorts of shenanigans. And one of the things that that I've been doing now for the past, I don't even know how long is, you know, kind of progressing past have cropped up around it to create modifications to the core Minecraft gameplay. They add things like special machines that do certain things, you know, add things like

questing and stuff like that. And basically, there's just a whole lot of functionality that's been added that's kind of taken things well beyond what the original ideas were. And it's kind of all possible because the Minecraft that everybody is using for this particular set of functionality is kind of like the OG Java-based Minecraft. So, you know, a while back after Microsoft bought Mojang, who's the developer in, I believe, Sweden, who actually developed Minecraft, they bought Mojang for like over a billion dollars, I think. I remember that. And the original Minecraft is actually Java based. And, you know, continued development takes place on that particular offshoot.

But Microsoft, after they bought Mojang, they're like, OK, well, you guys have to make a version of that that's like native Windows, native Xbox, you know, kind of C++ based. So everything that I'm talking about today is actually on the Java based version of Minecraft, which is the one that's actually supported in the community, you know, kind of with with modifications. And I did want to I did want to point out, Andy, that the question you asked Mike about what is Minecraft is kind of akin to if you go back to the early 80s, during the height of Pac-Man fever, when there was even a song about asking somebody to explain what Pac-Man is. But, Mike, you handled it very well i have to tell you and i know there's a lot of pressure because andy's your boss and if you didn't get if you didn't answer a question even even though once he starts realizing that you're doing this all during the day he's going to probably fire you anyway but so so what you're saying though

is there's two there's basically kind of like two branches, for lack of better word, going on with Minecraft. The kind of the home kit, the Java version or the Java or JavaScript. You said Java, I think it's Java. Right. So that's the one everyone's modding. And then there's the official version, which is owned and run by Microsoft. So but there's no. No. So let's let's say that they're both they're both totally official. OK. Right. So they're both totally official. They're both, They're both continuing to get updated and supported by Mojang and Microsoft. It's just you kind of have these parallel branches. One of them is available in the Microsoft Store.

Right. Right? And that's the one that's available in mobile. That's the one that's available on consoles. That's the one that they ensure works 100%. Yeah. And then the Java version is the one that has, you know, kind of goes back to the early days.

And then that's the one that actually has modding support just kind of due to the nature of how Java operates. Alright, so then if people want to know more about Minecraft, I think the website is mine.net, I believe. Correct.

So now, Mike, fill us in. Why would somebody like you install an APM product on Minecraft? Did you want to find out if Minecraft itself is kind of like there are potentials, as we call them, potential opportunities to make it faster or if the modified versions or whatever, the add-ons, the extensions. What was your initial thought on that? So that's a great question. So number one, I've done a lot of research into how Minecraft operated and so on to kind of, you know, figure out if that was something to even bother doing. This was something that I started doing about six years ago, even before I was a Dynatrace employee, when I was starting to kind of figure out,

you know, what Atman could do. I thought, okay, this is an interesting exercise in figuring out what APM tooling can do around applications that are not web-based, right? So it was a good opportunity to figure out how to, you know, define custom entry points and things like that. So as I moved beyond that and I started to get really deep into modded Minecraft, I started to experience a lot of performance problems. Performance problems are something that's actually very common in the modded Minecraft world. Right. So nobody's writing these things with performance in mind. No. Well, so so it's not necessarily that they're not writing things with performance in mind, which is very true. You know, a lot of people actually use Minecraft

as an excuse to learn Java. So there's a lot of like just really terrible code out there. But the interesting thing is the architecture of Minecraft itself lends itself to performance problems. Absolutely, yeah. So it's actually really interesting because minecraft basically is single threaded for the most part everything that's happening in your minecraft world is happening in what's called the master tick thread and that master tick thread has

to iterate over everything in the world that has to have something happen to it. So that's going to handle, you know, figuring out where all the monsters in the game are going to move to. It's going to figure out what all of your machines are doing. Is that kind of like a turn-based operation sort of? Well, it actually, yeah, it sort of ends up, you know, being that way. And then basically what Minecraft will do is everything in Minecraft is predicated around ticks per second. So it's basically everything has to happen in 50 milliseconds or less, or else Minecraft will start to skip ticks.

So that means those behaviors that you were expecting to occur, they might not occur. This is obviously though, having something single threaded like a main worker thread and then working on items, that's something not unusual. So why is this bound then for performance problems? Because if you put something in that takes longer than the 50 milliseconds, obviously, now I get it. If I explain it to myself. Yeah, exactly. So basically what you're doing here with modded Minecraft is you're adding a lot of different things to that master tick thread.

You know, on top of that, Minecraft is now a, you know, basically it's multiplayer. Right. So not only does it have to figure out everything that's happening to you in the space around you, it has to figure out everything that's happening to all the other players in the space around them. And it does not always do that in the most efficient way. And then you start adding modifications and these modifications start to add even more things

to that master tick thread. And then you just start to have problems. So you've developed a solution to that master tick thread problem by looking you just start to have problems. So you've developed a solution to that master tick thread problem by looking at all the APM data, correct? And you're going to present that, you're going to announce that at the end of the show, correct?

Sure, sure. I mean, I don't know what I have to announce there. Other than, so it- You've rewritten the engine as a multi-threaded, right? It's more, well, actually the funny thing is, is that there's various projects that have cropped up in the community over the years, and none of them have actually borne any fruit because syncing threads is not easy. That's one of the reasons why it is single-threaded. But one of the interesting things here, so when you start trying to look at deploying an APM solution

on top of something like this, one of the things that they do that makes it just really, really difficult is the core Minecraft jars are all obfuscated. So all the function and method names are complete garbage. So they, the, the actual names don't make any sense. So luckily there's an organization out there that creates something called

MCP. And what the, the it's like the, the modded core pack or something like that basically it's they create a spreadsheet that gets updated with every release of Minecraft that de-obfuscates the method and function names so you can pull down a spreadsheet and say okay well I want to find this method what is it actually called right and it might you know it might be you know might be you know because it's going to get some sort of like really completely bizarre name

and then you can take that information and then hop in the dynatrace and create a custom service right which is which is what i did to actually light everything up in the first place and then you know one of the things that I did then is since I know 50 milliseconds is a hard, hard, hard limit. Now I've got an alert that generates, you know, every time the master tick thread takes more than 50 milliseconds. Interesting. So the, now let me, quick topic, the de-obfuscation or the, I side isn't there a standard where you can have mapping files from obfuscated to non-obfuscated functions and then we can upload that to dynatrace i don't actually know how to do that so that's a that's a great that's a great question yeah

i think maybe maybe maybe i'm confusing this a little bit with – we definitely have it on the mobile side, right? For mobile development, for mobile monitoring. Right. We can upload these MADPIN files and then especially for crash information, this is where you would then see the stack traces of crashes and exceptions with deobfuscation. I'm just wondering if there would be something – I'm sure they've tried. then see the stack traces of crashes and exceptions with deobfuscation. I'm just wondering if there would be something. I'm sure they've tried on the irregular instrumented code as well on the pure paths. I've never heard of it.

Yeah. Yeah. So that means a big. So first of all, what you learned, which is a great exercise, it's definitely better than taking a sample console application and finding the entry point. Or I think we always used some, what was it called? There's a drawing app, a Java drawing app, J-something that we used in some of the tutorials to find custom entry points. As you said earlier, it's a great way to learn how to monitor a non-web application, because for web applications,

we always have all the entry points by default. And for these apps, but that's a cool thing. So that means you went through these files and then found the right entry points. Did you then, and maybe you said this already, did you instrument interfaces because these extensions had to implement a certain interface or how does it work again? No, no, I was able to just actually find the, you know, the actual method that starts the master tick thread. And I just dropped that as a, as basically a custom entry point. And then when things start to go wrong, the cool thing then is, you know, the method hotspots in Dynatrace allow me to immediately see which mod is actually causing the problem.

Oh, wow. Yeah. Right? Because generally speaking, the code that the modders create is not obfuscated. So it's usually very easy to do, you know, based on the method name and the package to determine where that came from. And then, you know, I'm able to kind of figure out what's going on. And then sometimes now, so here's a, here's a problem that I ran into two weeks ago. So I found that the master tick thread was gradually slowing down i really wasn't sure why

and quite frankly i i still don't necessarily understand why it's a gradual slowdown but you know with donatrace i was able to find that the problem was being caused by a specific method inside of a specific mod and then what i was able to do is i was able to hop into minecraft and then use a client-side mod called lag goggles and what lag goggles does is lag goggles allows you to basically attach a profiler to your world oh wow right and obviously you know profiling is extremely heavy overhead. So that's why like, you know, to just kind of figure out where to look, something like APM tooling is much superior. But when I use lag goggles, then I can actually see

which item it is in the world that is causing the problem. It's kind you know kind of sort of like imagine if you know the the end state of augmented reality apm support you know some of the things that we're doing with the hololens imagine if the end state of that was allowing us to you know walk into a data center and find out which server it was that was causing the problem it's exactly like that you you hop into the server and the tile entity, which is the block in the Minecraft world, will be highlighted in red. That's amazing.

You know what would be cool? Sorry, Brian, do you want to? No, I was just going to ask if the thing that was the problem was your Thriller-style Michael Jackson jacket you got for your character in the world. Was that causing the... No, no, no sadly sadly It was part of a mod called bomb craft

which is in a very very very beta state and There's an item in thumb craft that actually fills your cauldron with water And I do believe that the problem is caused by that just ticking too frequently. And have you shared that then with the developer? In the process. In the process. Well, probably by the time this airs, I'm sure you will. That's really awesome.

Andy, I'm sorry. I just wanted to find that, but that led to another question. Yeah, no, it's good. So as I've never played Minecraft, but I saw it. So objects always have obviously relationships to other objects, I would assume, right? Mm-hmm. Yeah. So wouldn't it be cool to have a completely modified Minecraft where we use the OpenKit

and then every item in Minecraft becomes a monitored entity in Dynatrace? And then we would see SmartScape. SmartScape would then be a different visualization of your Minecraft world with all the dependencies. And then if the tick thread for a particular element or entity then would exceed the 50 milliseconds, it would immediately light up the smartscape and say, here is the problem.

That would be really cool. What I would do to try and make something like that happen is the Minecraft world itself has what are called chunks. And a chunk is a 16 block by 16 block square that then extends, I believe, from the sky to bedrock. And then you would actually have an entity inside of SmartScape for each chunk. The reason why I say that is because I think otherwise it would be too cluttered because there would be billions of tile entities. Yeah, or we use tags in the management zones. But, I mean, obviously, it would be, obviously, a theoretical but interesting use. Yeah, that would be interesting.

Absolutely, yeah. Yeah. But I have another question for you now. So I assume when you were starting with this project, you had AppMon. And so the approach with AppMon, obviously, would be to find the method, define the entry point sensor, and so on and so forth. Now, sometimes what I've used in the AppMon days

is just the CPU sampling. And then based on the CPU sample, figure out what the potential entry point could be. In our new data choice with the one agent, wouldn't you have been able to just say show me the cpu hotspots because we automatically do cpu sampling besides tracing transactions as well and then trying to figure out where your hotspots are and based on that define your entry points that that would have that would have made a lot of sense, but I think in this case, it might have had me going a little bit too deep.

When I knew exactly what I wanted to track, you know, so this kind of is analogous to, you know, when you're part of the performance team and you have easy access to the developers, you can basically say, hey, what is it that we want to define as our entry point? So that's effectively the sort of knowledge that I had by consulting that MCP spreadsheet, because I knew what I wanted was the entry point to the master tick thread, because I knew the master tick thread and everything contained in there was what really mattered to me. That's why I dropped the sensor there. And I think if I would have done CPU sampling, they might've actually had me a level deeper. Yeah, no, that makes sense.

I mean, but just theoretically, if I would, because I have, you know, let's assume I would not have found this spreadsheet, then maybe, or if you are tasked with any type of application where you have no clue about where the entry points are. But a similar app, like in your case, that the core engine is obfuscated and then you know there's some packages that come in from a certain extension. Then one option would have been to say, well, let's search for one of these extensions in the CPU sample and then walk up the stack until we hit the first obfuscated one because then we know this is kind of the,

what do you call it, the interface from the core engine to the module. But obviously you're much faster if you have the developer insights. Exactly, exactly. And what you stated is the, at this point, kind of like the well-established workflow when you have no access to anything. And in this case, I was just able to kind of leapfrog that a little bit and save myself some time.

Yeah, cool. Hey, now another question for you. Obviously, the big question that always comes up with people that use monitoring tools like dynatrace is what's the overhead and then obviously this is now a game and you are uh you know it's the tick thread if the tick thread is slower than 50 milliseconds and it starts dropping have you experienced do you know what the overhead is of dynatrace now have you so so i haven't really experienced much of anything now in the app the Atman days, I did. Because in the Atman days, I followed what was best practice then to sprinkle sensors. And by sprinkling sensors, sometimes I would overwhelm things. Because I'm talking about something that needs to be completing in 50 milliseconds constantly and if i sprinkled too many sensors sometimes you know i could overwhelm you know atman itself or i could

overwhelm my my instrumentation now things have simplified to the point where you know the only thing that i've changed is actually adding the entry point and i allow the method hotspots to do everything else for me without having to make any more changes yeah that's that's awesome I mean obviously we know the benefits of the of our new diamond trace it's just so much so much easier hey um so Minecraft you said it runs in the sandbox is it a complete sandbox what can the individual modules or extensions can they call to the outside world and maybe fetching I don't know some they call to the outside world and maybe fetching, I don't know, they have to fetch something, right? It's actually really funny too because it's one of the other problems that I've

uncovered is the fact that when you authenticate to a Minecraft instance, it's actually phoning home to actually look up your email address and password against some master database service that actually maintains all that. Because Minecraft is not free. It's free to run a server, but it's not free to run a client. So it actually does that lookup every time. And every once in a while, that lookup can take tens of seconds, if not more. So having Dynatrace dropped on there, you know, sometimes makes it easy to know that, hey,

I'm having problems logging into my Minecraft server, not because there's something wrong with my Minecraft server, but because that authentication service is having trouble. So it's kind of that same, like, you know, third-party dependency kind of issue that we talk about. And then the other cool other cool thing too is that um you know in the atman days actually instrumenting you know getting the agent in the minecraft involved a lot of you know messing about with start parameters and things like that

what i'm doing now that's really interesting is you know i have a big server in my basement that's running up to like a dozen minecraft instances at any in time. And each one of them is in a container, right? So I have, I have some automation around, uh, creating Docker instances of specific modded versions of Minecraft with specific mod packs and so on. So, uh, you know, so I was able to light up all of them, you know, just based on having the one agent, uh, the one agent on the server in my basement. That's pretty cool.

That's awesome. And to add another feature to it, do you see anything with log analytics? Is Minecraft writing any logs? It is. It is. I haven't needed to dive into that just yet, but I'm starting to envision a couple of use cases there. One of the things that I have done is there are situations in Minecraft when Minecraft is doing world gen because the world of Minecraft is procedurally generated. And the idea is, is what it should be doing in an ideal scenario is only generating.

Remember how I said that it's arrayed in the 16 by 16 chunks? Ideally, it's only supposed to be creating chunks that you can see. Okay. Right. So that it doesn't go off on a wild goose chase, creating the whole world all at once, because obviously that would fall apart very quickly now what happens in modded minecraft and actually even in vanilla minecraft which is like the og minecraft sometimes a tile entity is created on the edge of a chunk that has dependencies on something in the chunk next to it so what you end up having is something called

cascading world gen where generating one chunk ends up resulting in like 1500 chunks getting generated and now with the versions of minded minecraft that i use forge and ftb they create log entries when that happens it'll'll say like, you know, mod XYZ is resulting in cascading world gen. So I've also set up an alert for that so that I know when that's happening. That's cool. Okay, Dad, who would have thought that APM can be such? I mean, this is, I mean, honestly,

it sounds stupid maybe, but maybe we should ask our engineering teams to think about Minecraft as another test app that they should use internally. Because obviously it seems it has a lot of things to offer. Yeah. I think it's really interesting, too, because Minecraft has become such a phenomenon. I remember I said I played with it a little bit in the very beginning when it came out. And back then, it was kind of like in the world of geeks only it seemed right it was you know i was reading about it on wired you'd see some people

playing it and it wasn't as widespread and you know i i just as i said not much of a gamer but then maybe about a year later or a year and a half later i saw my like seven-year-old nephews playing it on their ipad or something you know, and it was like, Holy, what kids are like, it was everywhere. Now you see merch everywhere, shirts everywhere. Um, and it's incredible how it picked up and it's also just, it's become so pervasive. So I think at 8 PM, putting 8 PM on it is kind of the natural extension then because it is such a big world thing now mike had

a question i had a question regarding that so obviously you mentioned that goggle thing that you can do some profiling on which i'm sure a lot of people use when they're making their mods but have you run into anybody else who's doing anything like you're doing taking an apm approach to this with with anything any other apm tools i i have not and and quite frankly a lot of that is is just likely the fact that some of these folks don't have the same. Excuse me. Some of these folks don't have the same access to the tooling that we do. Free trials. Right. So I think it would have been great with, you know, I think it's a it's a great free trial application. You know, the downside, of course, is like this is the mother of all monoliths. Yeah.

So so when you talk about a, you know, a average like Minecraft and modded Minecraft installation to support like a small handful of players, you know, you're likely looking at like something that's going to be in the, you know, six gig or more heap range. OK. that's going to be in the, you know, six gig or more heap range. Okay. So, you know, it's, you know, like, for example, my server in the basement is, you know, it's a 64 gig host with plans to go up to, I'm actually hoping to take it to half a terabyte soon, but that's just because I'm a nerd. So, so that's just, you know, so it, it, it tends to be one of those. It's one of the interesting things about actually running something like this in the public cloud. Yeah. Is it actually, it's not the kind of thing that's going to run well on like a T2 micro.

So, you know, you end up, you end up needing to have a, you know, pretty beefy host to, to run it. So, you know, now folks do often take a profiler approach to figuring out what's going on here. And as I'm sure you all, you know, the three of us all know that profilers are a great way to, to get that, you know, nitty gritty detail level when you know what you're looking for. Well, even, even when you don't necessarily know what you're looking for, the problem is like we run without much overhead. But if you're running a profiler on a Minecraft instance that might have a dozen players on it, they're all going to have a bad time. All right, that's what I meant.

Like when you know what you're looking for because you're kind of doing it specifically because you know there's an issue. So you go on and turn it on to deal with that as opposed to just keeping it running. Yep. and turn it on to deal with that as opposed to just keeping it running. And then even something like that lag goggles application, I've seen that actually crash the server process. Awesome. Have you tried monitoring lag goggles with APM? I don't know what sense that would make, but I'm sure I could. To see why it's crashing it.

Well, that's your standard kind of memory profiling use case because that's why it crashes is because it's actually trying to persist the object that represents that profiling data before sending it to the client and then you just run out of heap.

Cool. Well, Mike, what I learned today, I finally know what you're doing. No, I'm just kidding. Mike, when you were using Atman, did you ever do this free trial and then did share your pure path with Andy back before you started working with us?

Oh, I should have. He would have been like, what the heck is this crap? That's awesome. Any other cool tidbits you wanted to get out there with this, Mike? No, no. That was really the gist of what I was doing there and why I thought it was kind of interesting. I'm glad Andy brought up the log analytics aspect of it because that was actually something that I had forgotten that I did. Well, I'm glad that

we had the chat, was it two weeks ago or three weeks ago in Minneapolis and we were all together and said, hey, wouldn't this be cool to actually do an episode on that because it's something different. Yes. Awesome. And maybe this encourages artists too to come forward and talk

about the dirty little secrets on what else to do with their APM product. Yeah. All sorts of crazy nefarious things that you can do. I'm going to put it on Quake Arena. Yeah. Or Chips Challenge. Well, the new Quake Arena is what, HTML5?

So I think... I didn't even know it was still in existence. Well, there's like a new version of it that's actually like browser-based. Ah, there you go. So, quick live. Quick live, it's called. Browser Diagnostics.

Yeah. So, Andy. Yeah. Do you want to bring out the Summaryator? Sure. Do it now. Well, I learned that some of my colleagues are spending valuable time in obviously exploring capabilities of the tool on software that it was not initially meant for, but obviously it shows a lot of great value. No, seriously, I think it's a great way to actually use tools and learn them. As you mentioned, finding custom entry points

and figuring out how to monitor a non-traditional application is a great way. As I said earlier, I've never really used Minecraft, so it's great to learn a little bit more, especially interesting how it all works internally when it walks through the main tick thread and it basically asks every extension, every mod to then do its work.

So that's pretty cool. Thanks, Fyke, for bringing this up. And hopefully, again, this encourages others to come up with maybe more esoteric topics like this one. Yes, yes. I just want to clarify for everybody listening at home that this is definitely a, you know a evenings and weekends type of use case.

Cool. No, thank you so much. Yes, thank you. I got nothing to add except for learning that if I ever join Andy's team, I'll get to play video games. Exactly. There you go. All right. Thank you, Mike. there you go alright thank you Mike if anybody's got any questions or comments

send them over to pure underscore DT if you want to contact Mike about anything what is it Mike you're at what are you on twitter these days yeah so the easiest thing would be to you know DM me and follow me on twitter it's at Mike Villager my last name is V-I-L-L-I

G-E-R. It's surprising how many people just assume that it's spelled like the word village. I was messing that up today. When I was setting up the session. I was like, wait, I got it wrong again. So yeah, you can get hire Mike to consult on your mod

now and he'll do it for free. And yeah, thanks for listening everybody and we'll see it for free. And yeah, thanks for listening, everybody. And we'll see you in two weeks. Bye-bye. Thanks. Bye.