CoRecursive: Coding Stories - Story: One Million Checkboxes - The Viral Game That Defied Convention
Episode Date: February 3, 2025What if internet trolls could become your greatest collaborators? Nolen Royalty discovered this unexpected truth when his simple checkbox game went viral. It began with a school email prank that crash...ed servers but sparked a philosophy: creative constraints breed innovation. From "Flappy Dird" to "One Million Checkboxes," Nolen built games that turned limitations into playgrounds. But when his checkbox project became a battleground of organized chaos, Nolen faced a choice: suppress the chaos or lean into it. Discover how systematic constraints can channel creative anarchy - and why one developer now believes the best user experiences emerge when you code for controlled chaos of strangers meeting on the internet. Episode Page Support The Show Subscribe To The Podcast Join The Newsletter Â
Transcript
Discussion (0)
Hello, and welcome to Code Recursive. I'm Adam Gordon-Bell.
Back in high school, during the dial-up era, my family had a computer,
and so did most of my friends, and we all connected through this chat site that was popular in our little circle.
One of my friends had this paper with HTML, like handwritten on it in pencil.
And if you typed it into the chat app correctly, which was hard because it wasn't clear what all the symbols were,
it would launch a bunch of pop-ups and it would crash your browser.
It was like this secret hack where you could take down the chat site.
And it was handed around from person to person.
Back then, nothing on the computer felt very real or important. It was just like a fun playground for teenage mischief. There was once a fight that broke out at school over something
that someone said in chat, but mostly it was just playing and a little bit of trolling,
occasionally crashing people's browsers. But now the internet is just life, right? And when the
internet became serious business, when it became part of reality, did we lose that fun? Today's
show is with Nolan Royalty. We'll dive into his wild encounters with the internet's chaos, how his creation
sometimes got misused, and also his worry about being hacked and data breaches.
This was happening on Monday evening, and my girlfriend had made dinner. I thought that I
had a data validation error that let people stuff strings into this bit set. Because I wasn't,
I hadn't really even thought about what it was
that my script did at this point. And I had kind of like gotten re obsessed with like, what is
going on here, right as dinner finished. And she was like, Can we like sit down and eat? You've
been very busy recently. And I was panicking about like, there are these URLs I have been hacked,
I have been hacked. But before we dive into the hack, let's rewind to a moment in his past that reveals something
crucial about him. A moment where he pushed the boundaries, expected punishment, but got
something entirely different. Something that might just explain why he's drawn to the chaos
of the internet in the first place. Nolan calls himself a deranged game builder. He dropped out
of college, he spent some time at AWS, and then Jane Street, a trading firm. But today is not
about that, right? It's about the deranged game building part of Nolan's career, and his exploration
of the bounds of computer systems, and what you can do. And it all started back when he was in
boarding school, where they had something called the student-wide information system. It was like email, and it was
chat. But point being, it was, you know, an important part of student life. And also,
it was the way that you turned in your homework. It had like a weirdly powerful mail rule system. So you could write rules in the same way that like in Gmail, you can filter
your mail or whatever, right? You could write rules that said like, when I get a message from
Richard, apply the Richard tag. The rules could take a lot of actions. And in particular, they
could send emails. And one other quirk, the reason that i kind of got into doing the thing that i ended up doing
was that when you got a new email it made a pretty loud beep and i had this friend broady
who always had headphones on during our study hall we had study hall for two hours a night
um sunday through thursday and whenever he got an email uh he would like jump because the noise
was loud and he had headphones
on i thought it was funny so it's kind of fun to like send him a message and see him jump
then nolan started tinkering with the email's rule system so what i did is i wrote a rule that said
whenever i get a message from myself with an empty subject one send a message to my friend braddy
two delete the email that i just got from myself. And three, send myself two more
emails that also have empty subjects. And so then I would enable this rule and I would send myself
a message. And then it would send Brody two and then four and then eight and then 16 and, you
know, pretty quickly a lot of messages. And he would get like ding, ding, ding, ding, ding, ding,
right? And like jump out his chair, fall over. And it was very funny to me.
Brody fought back, setting up rules that would delete Nolan's emails before they could cause a beep.
And then he made rules that would send things back.
Nolan tweaked his emails to slip past Brody's defenses.
And so the nerdiest battle began.
Each of them writing rules to outsmart the other.
And that was all totally fine and fun.
And we did this for like, you know, a week or something. And then one day, both of us just because whatever our defenses were good enough, forgot to disable our rules. So normally, you would like enable this for like three minutes and send a lot of emails and then it was fine. But instead, we just totally forgot and we like went to bed and we woke up the next day and the school email server was done. No one could send emails. They got the
system back up briefly, but the rules were still active, causing it to crash repeatedly over a
couple of days. Eventually they traced the problem to the rules and informed the dorm supervisor
that Nolan and Brody's battle was behind the email chaos. And he came to me and was like,
what's going on here? And I explained, and he was like, that's hilarious. That's so funny. And he made me a t-shirt, but I was kind of proud of finding this thing. And I think it certainly
made me, I don't know, there's like some through line from that to plenty of the stuff that I do
now in terms of like, how can I misuse this thing in a way that's kind of fun? And so being
encouraged then I think was pretty nice. Imagine expecting detention, expulsion even,
and instead being met with laughter and encouragement.
This supervisor didn't just forgive Nolan's digital mischief,
he seemed to celebrate it.
That moment was a hint of what was to come for Nolan.
The New York Times interviews, the media buzz,
the Washington Post, the speaking gigs, they all trace back to his knack for bending things
outside their normal use.
So fast forward,
Nolan is an engineer, then a manager at Jane Street, the quant firm. He builds up a bank role that's large enough to quit and do something that he's always wanted to do, make video games.
And I had this big list I had spent all of 2022. Every time I had a game idea,
I would like put it in the list. And so I had this big list of ideas that I thought
I would like be working from for a while. The first thing I made was Tetris, just to get an understanding of
how to put objects on the screen in Gido. And a thing I decided pretty early was that the goal was
to build lots and lots and lots of small games. I understood that one common failure mode for people
who try to make games is that
they're like, oh, I really love whatever. This is a silly example, probably, but like World of
Warcraft. So I'm going to build a big MMO. And don't think about how that takes years for a team
of, you know, hundreds. And it's probably not a good starter project. And you learn so much by
finishing a game and seeing people play it or play test it. And the thing you want to do is get a lot of reps in of that.
So it's like, I'm going to make games that take me like a week or two as fast as I can and blog
about them. And there's a sign right above me that says the plan, one, make games as fast as you can,
two, blog a lot that I put on my wall to kind of, I don't know, give myself some goal.
That sign is still there.
And so Nolan builds games, but it's harder than he thought.
Like, it's actually really hard to just like go to your little, I don't know, second bedroom
and sit at the computer all day and write code and not talk to anybody about it.
And you can tell your friends about it.
But like, I don't know, your friends aren't working on it with you and you probably can't
talk to them as much as you would talk to your coworkers about what you're working about.
So Nolan decided to apply to the Recurse Center, a place that's kind of like a writer's retreat,
but for coding. You go there and you write code for 12 weeks or six weeks if you do a half batch.
And when you join, there are a bunch of people who've been there for six weeks to kind of show you the ropes. Then they leave six weeks in,
and now you're showing people the ropes. And the goal is just like, write as much code as you can
every day, according to whatever interests you. Right? So like when you apply, I think they do
like a basic coding test thing, but mostly they're looking for like, can you write code? And do you have an idea
of what you want to work on in a very self-directed way? So I applied and I told them I was going to
make eight games in 12 weeks. And I showed up and I started making games. The Recurse Center wasn't
just about solo coding marathons either. They had workshops. Nolan joined one on pair programming,
where his task was to create a small game.
We were writing a solver for the game Mastermind.
Yeah, with the mines?
No, not Minesweeper.
Oh, yeah, Mastermind is like the colors?
Yeah, the colors.
So it's actually very much Wordle.
Like Wordle is Mastermind,
but there's only one player and you're guessing words,
and Mastermind is a two-player version of Wordle. Wordle is mastermind, but there's only one player and you're guessing words. And mastermind
is a two-player version of Wordle where you're guessing the colors of some pegs that somebody
has hidden from you, but it has kind of similar mechanics. But they told us we were going to write
a solver for mastermind or a backend for mastermind or something. And I said that I wanted to work in
SQL. And they matched me with someone else that could work in SQL. And so we wrote a mastermind thing that worked entirely as like a big SQL query.
And it was really fun. It was like a huge blast. And we showed it to a bunch of people. And
everyone was like, yes, this is like, this is funny. We're glad that you did this.
And that was very exciting to me. Nolan got inspired to create a game using open search.
That is the way that your computer knows how to
search a specific site. So if you've ever gone to like, you've typed Wikipedia into your search bar
and hit tab and then typed a query and it's like searched directly over Wikipedia, right? The way
that works is that under the hood, when your computer visits Wikipedia, Wikipedia serves it
a document that says, if the user wants to search my site, hit this end point. And the spec happens to include this crazy thing where optionally browsers can choose to support search suggestions.
So just like when you type into the Google search bar, it guesses what you want to search for.
Your website can offer those suggestions. So you say, hey, whenever the user types a new character, hit this endpoint, and I'll return some ideas of what they might be searching for, along with URLs to directly visit that page.
So the whole bit was that you can return whatever you want from that.
So why don't you return Wordle answers based on the Wordle guests that they're typing into the URL bar, if that makes sense.
So Nolan took this idea and
created a Wordle-inspired project. Then he wrote a blog post to break down how it all worked.
It's nice to have a website where you can point to everything you've made. But I think it's also
just a really nice way to codify what it was that you learned from a project and explain what you
were thinking. And so for this, I made a blog. Nolan demoed this at a Friday presentation at the Recur Center,
where anyone could grab a slot and show what they were working on.
And I think that was the first time that I saw that people enjoyed something that I made,
as I presented it, and the response was pretty good.
Getting positive feedback on that was really fun.
Someone at the Recur Center shared Nolan's blog post on Hacker News,
and then it gained traction.
Encouraged by this response, Nolan kept going to the Reeker Center and he just kept making games.
I made several more kind of more traditional games, you know, games with like high scores
and guys that you controlled and you like shot an enemies. I remember taking a game of mine,
a more normal game of mine to a playtest and And being there with some friends from Recurse who
had brought games that I thought were much more interesting than what I had made. And being at
this playtest event and getting feedback on my game from some people and just being embarrassed
to be showing the game, being like, oh, I really... Not like it was a terrible thing,
but just that it so clearly wasn't what I wanted to be doing. I wanted to be one of my friends showing these much weirder, more experimental things. And I was getting feedback on how the numbers in my game could be adjusted and like, oh man, I just really don't want to think about balancing numbers in games. That's not what I'm interested in, what I'm passionate about. And so I think the big thing there for me was learning that what
I wanted to be doing was this kind of weird stuff. And also that I was just better at it.
I'm very, very into taking a weird bit of technology and pushing it as far as I can.
I have zero patience for picking nice numbers that make a game feel balanced. I don't want
to work on games that need nice numbers to feel balanced.
I want to work on like weird experimental stuff.
He wanted to do the stuff that bent the rules,
that surprised people,
that maybe even broke things a little bit in a fun way.
Just like that original boarding school email prank.
But now he was channeling his creation powers into this.
And after 12 weeks, Nolan's batch ended.
But he kept showing up at the Reeker Center every day.
I came up with this idea for making a game inside macOS Finder by renaming files and doing a bunch of other hacky stuff,
which turned into a version of Flappy Bird that you ran inside macOS Finder.
That, I think, was the first time that I remember making something and like realizing after, like I demoed it at Recurse. I was no longer
in batch, but you can still show up to demos whenever you want. And like people like lost
their minds. Like it was, you know, like the reception was really, really, really good.
And a bunch of people wanted to talk to me about it after. And I had also just loved making
this thing. I thought it was so, so fun. And I loved talking about it, getting to present it,
and the story of how to make it faster was really fun. I got a bunch of advice from some friends on
the problems, and some people helped me solve that. But that was the first thing I think I made
that I was like, oh, wow, like this is this is I think actually good.
Meanwhile, Nolan is wrestling with his direction.
I like having a plan.
And when I don't feel like I have a plan or a goal, objective function, it's hard to know that I'm making progress.
And I do need to be making progress.
I think was really, really challenging at points because I would wake up sometimes and be like,
what am I doing?
Like, what in the world?
What's the goal?
Like, what am I trying to do?
Does anyone like me in any way?
Like what?
And I sometimes still feel that way
and find that really stressful.
But then Flappy Durr was released
and it went to the top of Hacker News
and then it was on Daring Fireball
and then GitHub.
They have a YouTube channel and they covered it on that.
From this experience, Nolan learns to refine his taste a bit.
There is an appetite out there for these sorts of conceptual weird games.
And so he keeps building.
And then I made this game called Stranger Video, which I think is one of the things that I'm still probably the proudest of.
And Stranger Video was a website that showed you live video of a stranger's face until one of you
blinked. And that's all it did. It used MediaPipe, which can do, it's a library from Google that has
face detection, to crop video to only show faces and prevent you from showing anything else.
And also to detect whether your eyes were opened or closed. That was it. Like it was just a website with silent video
of somebody else on the website where all the restrictions were kind of coming from this place
of like I've used chat roulette. I understand what happens when you just have strangers look
at each other on the internet and what they choose to show and I don't want them to be able
to show those things. I want just only faces. I was also kind of thinking at the time about like some social anxieties I have
around like, when do I make eye contact with people that I've noticed far away and kind of
wanted to channel that into a game. So Nolan releases this game. And for some reason,
it went viral in Japan, like really viral for a couple days. And then even some of his friends
were very touched by it. Martin, one of his friends, said he never had so many people look at him and smile in his life.
And that was like this profound like, oh, wow, like that is I've delivered.
I've given Martin this like, I think, genuinely cool novel experience and kind of opened my eyes to like a new thing type of thing that I could be building.
People playing with the restrictions was really interesting. So your eyes needed to be opened, but you could put pieces of paper over your eyes that had little
eyes drawn on them and that tricked the camera. And if this was a competitive game, that'd be a
problem. But for a weird social experiment, that's actually just funny and fine, I think.
I made this thing and I tried to figure out why it worked and why I liked it so much
and how to do more things like it.
And that was certainly a moment of validation and of like, okay, like I'm proud of this.
I want to make more things like this and people like it when I make these things.
And I think it has artistic value.
So the next constrained collaboration game was talk paper scissors the phone version of rock
paper scissors but with strangers so you like call a phone number it asks you to say rock paper
scissors use some voice detection so that you can't say anything but those words plays your
audio to someone else plays their audio to you and that wasn't as much of a hit a stranger video
but again got i don't know like 50 000 calls or something like enough calls that i those kind of
validated again like this is people like to play with strangers. There are not a lot of environments
that are obviously restricted enough that you feel comfortable doing it. Both games thrive on
constraints, right? They connect you to strangers, but they have limits on how you can interact.
And I think in particular had this theory of like the trick to this is constraints.
And the cool things that come from this are how people subvert those constraints.
That like you add constraints to constrain the assholes, but also that gives people material for like funny things to do that wouldn't be possible with more of a sandbox.
But since kind of the thesis here is like simple interaction with constraints that prevent jerks.
It didn't really feel like something I could force. You need to come up with an interaction
that you think is going to be compelling and fun and maybe silly. And you also need to come up with
a way to constrain the obvious ways of abusing the action. Stranger Video and Talk, Paper,
Scissors weren't just games. They were experiments in constrained chaos.
Nolan was almost designing for playful subversion,
for people to find the edges and to push against them,
just like he had in the early days.
Meanwhile, he craved community.
He craved that interaction that comes with building things together.
So he went to NYU's ITP summer camp,
which stands for like, I actually don't know, interactive telecommunications program or something.
It's pretty old, so they probably use the word telecommunications.
But it's a school that's focused on like interesting artistic mixed media stuff, tech art things.
It's like a real, you know, you can get like a master's there.
I don't want to get a master's.
That's a lot of time and money. But they have a summer camp that's like hang out there for three weeks with people that think it would be fun to hang out
there for three weeks, which like, I don't know, I probably am going to get along with most of
those people. So I did that this year because I didn't want to do Recurse again. At the summer
camp, Nolan dove into face tracking tech. It was fun, but it was a lot of work and he couldn't
release anything. And it left him feeling eager to just get something out the door.
I think I feel really good when I'm regularly releasing stuff.
And when I'm not, I get kind of down on myself.
And I got lunch with my friend Neil, who makes the website Neil.fun,
which is a really lovely collection of fun, different games.
Infinite Craft is a very popular game of his
from from this year feels like you two should know each other yeah we're both in brooklyn uh
i think like a surprising number of people that build funny stuff like this uh are from brooklyn
and uh but and at the same time like that scene is not that big like it's you know feasible to
know a lot of those people but i think we originally met via a friend of
mine from Recurse. And then like, I don't know, we've got common interests and sometimes get
together to bounce ideas off each other. And I think explicitly the goal was we were bouncing
ideas off of each other. We had originally planned to do kind of a Stranger Videos style thing
together. And then he made Infinite Craft and that went crazy. Like that went completely
insane. And so he was busy for months and I was doing other stuff. And then that had come down.
We're like, okay, like we'll come back and we'll collaborate on this thing. And I pitched 1 million
checkboxes just as kind of a joke. The pitch to Neil for 1 million checkboxes is simple. It's a
website with a million checkboxes on them.
If you check one, it's checked for everybody else. That's it. And we chatted about it for a bit.
And we're kind of like, eh, this is funny, but I don't think there's that much there. And I don't think either of us thought it was like a bad idea, but we were both kind of like, this is not the
thing that we should collaborate on.
But I thought it was funny and kind of interesting.
And I knew I could build it really fast.
Like I knew it was just not that much work.
I know how WebSockets work.
I know how to put checkboxes on a web page.
And I was super hungry for like making something quick, right?
Like I said, I hadn't shipped something in a while. And so near the end of the conversation, I was like, hey, do you mind if I just go build this?
I think it'll take me a few days and then we can collaborate on one of these other ideas when that's done.
And that was fine.
After that Friday lunch, Nolan spent the weekend figuring out how to build 1 million checkboxes in a way that would feel responsive in real time.
It's a lot of checkboxes, a million of them, and they need to stay all in sync across all the users.
How do I only render the check boxes that you're looking at?
I've never built an application that uses WebSockets of the scale.
Like, can I sketch out an architecture that will handle the couple hundred people that
I think will play and that I could scale up if it was a little more popular than that?
And so I kind of came out of that weekend with a plan and then I built the site in two days. I actually wrote the code that the next Monday and Tuesday.
So on Wednesday, he launched it, the million checkboxes. And that's when things got interesting.
I was excited about it. I thought it was funny. I was proud of like the constraints I had added
in particular in ways we can talk about later made it difficult for people to draw because
I understand what happens when you let people draw on the internet. And so I was proud
of all of that and thought, you know, it was evidence that I had kind of learned how to make
websites, which wasn't true at the start of the year. So I was proud of that too. Nolan thought
it was just a curiosity. Unlike Stranger Video, all you could do is check a box. There wasn't a
lot to do, but somehow it took off.
I think it was first Mastodon.
My theory is like it was Mastodon and then some big names on Mastodon posted about it.
And that got several people to post it to Hacker News.
And Hacker News collects votes if you repeatedly submit the same link in a quick succession.
And that got it onto the front page there. And then I think once it was kind of popular, the fact that it was kind of popular drove the popularity. You went to the site and, oh my god, all these boxes are changing all the time. The
site is so much more compelling because it's weirdly popular and that kept it more popular.
I don't know. I think the first three hours were me noticing,
wow, there's some activity on the site. And then, oh my god, there's really activity on the site.
And then, oh my god, my servers are going to... This is going somewhere. And if I don't start
working, the server is going to fall over and I need to stop checking the load and start writing
code. And that was, I don't know, that was the morning. But it was a very fun, if still slightly confusing thing to wake up and see.
That week marked Nolan's third at summer camp and it was time to show off what he had made,
his new spin on Pac-Man. And the camp show was that Wednesday night. Like you're supposed to
be presenting something that you had worked on there. I was planning on presenting my Pac-Man game. And instead the game had gone like
pretty viral by that point. And so I was manning my booth, trying to show people my game,
but mostly like spinning up more servers, trying to keep the website online.
How many people did you reach?
So the numbers here are a little fuzzy for two different reasons. The first is that I originally, so I logged everything. But when I wrote the login code, I was like, there's no way that more than a million boxes are checked in a day.
So my logs will be truncated at a million lines a day, because like, that's an insanely high number. Like, that's stupid. And I didn't think about that until 5 p.m. or something.
So I lost all my logs of what boxes people checked for the first several hours, which is a bummer.
So my estimate of players from the first few days is from looking at my NGINX logs, which I have IPs, etc.
And I think there's something like 500,000 unique IPs there.
It's hard to know, you know,
sometimes people are going to be like,
many users are buying the same static IP
at a school or whatever.
In the first few days, on average,
we checked something like 50 million boxes a day,
which is just totally, totally nuts
and well beyond expectations.
50 million check boxes, half a million people.
These numbers are wild right
picture the the largest stadium or music venue you've ever been to it's more people than that
the largest stadium in the u.s per wikipedia is michigan's big house and that's a hundred
thousand people we're talking 500 here that's that's five times that many people over the
course of 48 hours and these half a million people fall into two camps, right?
There's the checkers who like to check things and watch the score go up.
But then there's also the uncheckers, unchecking the boxes and driving the numbers down.
Like, I think you can look at like media coverage of the site and plenty of the time it'll say
something like, it's a website where the goal is to check all the boxes.
And I'm like, I didn't say that.
Like I just said, here's some check boxes,
do whatever you want.
And it's fine that that's your goal,
but like that's your own goal.
But I think that that was part of the appeal here
is that I think in general,
people are much more attached to goals
that they come up with themselves
and saying, here's the thing,
do whatever you want with it.
And you deciding my goal is to
prevent all the boxes from getting checked. Or my goal is to prevent box number two from being
checked ever. Or I want to check every box. Pretty quickly, people start having their own stories
about how they're using the site. So an example of that is I would often hear from someone who
would find a place where someone was like checking a bunch of boxes and they would follow that person
and uncheck the boxes and they'd be like yeah and like they were getting so mad right like i was
trolling them and they're getting so mad and again it's like you don't actually know that right like
you have no idea whether they're they're noticing what you're doing like you have no idea what's
going on with that person maybe they find it funny but because you have such limited information
it's really easy for you to like play pretend in this way. And that's just super fun. Like you get
really into that. So I think it really let people tell those stories in fun ways.
Okay, let's get into the weeds for a minute. Imagine managing a million pieces of shared
mutable state with tens of thousands of reads and writes per second. It's a scaling challenge. It needs to
feel real time to engage users. And plus, it has some kind of aggregate data, you need to have a
count of the number of things each user has flipped, and then a total. Seriously, if you've
ever had to stop and scale a data store or a database, stop and think about how you would
handle this for a minute. What's the hard part? To me, the issue is the number of writes. If you're going to use a traditional database,
it's a lot of concurrent writes. And some of the rows, depending on how you organize data,
we're going to be really hot. Those first hundred checkboxes, I'm sure are where a lot of people are
clicking. So then I start thinking about something in memory, but how do you keep all this state in lockstep? Yeah, yeah, the architecture
was like, Redis has a bit set, just roughly a million bits, with some nice operations for
flipping individual bits. So Redis has that. Redis has a count of the number of boxes checked,
stored separately, just so that we don't have to iterate through a million things to determine how
many are checked, and the website's supposed to display that. Side note,
Redis also, you can write Lua and invoke it via Redis atomically, which is really sick. So like,
you really don't have to worry about races as long as you write your logic in Lua scripts.
So it was really easy to say like, oh, we're storing this count separately from the number
of checkboxes. There's an obvious race there kind of, you know, depending on the primitives you have. But like, let's not even
think about it at all. Redis has a pub sub feature, you can flip a bit, bump a counter,
and then broadcast the change all in one Lua script. And the rough architecture was,
you had a bunch of workers whose job were to accept WebSocket connections from people on the website.
They offered an API that let clients flip bits along with some basic rate limiting.
There was also some client rate limiting, but that's obviously trivial to avoid.
And then whenever a client flipped a bit, you would write that to the PubSub.
And then every WebSocket worker process was also reading from that PubSub. And then every WebSocket worker process was also reading from that PubSub. And whenever
something got written there, would update all of the clients that were connected to that individual
instance. This is smart, right? State management is atomic and fully outsourced to Redis. But
scaling the workers to handle the WebSockets, that's still a challenge. And then there are
the things that Nolan didn't even think about. I think it was night one when I realized I hadn't thought about bandwidth cost.
I was managing all my servers myself.
And so I had originally decided, okay, like my costs are capped.
Like I know how much my servers cost.
I'm spinning up a bunch of servers.
Like this is probably gonna cost me like a couple hundred bucks.
I'm getting some donations.
That's probably fine.
And then remember to the, you know, get charged for bandwidth. And I'm sending like, a weirdly large amount of
data, especially at this point, I hadn't thought about how to reduce the size of these updates that
I was sending clients. And so I was pushing out, you know, hundreds of megabits a second or
something. So Nolan started doing the math and estimating costs, and the math gave him an estimate of $10,000 for his bandwidth so far. I started working on like, I can vastly reduce the amount of data I'm sending. That's going to
help a lot. And two, I happened to be pretty comfortable using the Linux command line utility
TC, which stands for traffic control and is an obscure utility for limiting bandwidth on
instances. My job back at Amazon Bud Services a decade ago involved using it a lot. And so I was
like, oh, I can just like add a rule that drops all traffic, like all outbound traffic above, I think it was 250 megabits a second. And that let me calculate
an upper bound, right? And that was what really mattered to me was like, the upper bound was kind
of high, but DigitalOcean's bandwidth pricing is actually quite sane. So that was the one time that
I was really scared. Beyond that, I was delighted and baffled and excited and super motivated to keep the site
up and keep working on it and improve it in different ways.
I had made a mistake that I often make when I make things like this, which was I had very
little attribution to myself or any self-promotion when I made it.
I don't even know that there was a link to my website.
There was certainly not a link to my mailing list or a way to donate me money or anything like that.
Nolan added links to his website and his mailing list. And just like that, the media came calling.
And I did the interview with the Washington Post, which was fun. They asked me how I was
monetizing it. I was like, not really. But I think in the question,
they were like, are you going to sell this to the New York Times like Wordle? And I made a joke
about like, I will sell it to the New York Times for $1 a checkbox. And like 12 hours later, I got
an email from the New York Times, which quickly came with a follow up email saying, we're so
sorry, this is not an offer to buy the site, but was just someone there who also wanted to talk to
me.
And I did all that.
And that was super fun.
That was the first time I'd ever done any media like that.
And pretty validating.
Like it was very convenient for the family in that they don't really understand what like Hacker News is or whatever.
But like, hey, I'm in the New York Times is like that.
That is meaningful to them.
Next up, Nolan needs to boost the streaming server's performance.
So I had a friend from the startup that I used to work at back in Michigan,
who these days is like a performance engineer at Facebook, who is very good at making things fast.
And we've been wanting to work on something together for years and hadn't had the time.
And I was like, one, you really want to work on this problem. And two,
like, I actually really, really need help here.
So I called him up and went over to his apartment at like 2 p.m. on a Sunday.
And we worked until 2 a.m. on a Sunday.
Part of the big rewrite was moving from an untyped Python flask app to Go.
During the rewrite, we noticed at several points we screwed up our rate limits in one way or another.
There were a couple of pretty embarrassing bugs. It was, you know, like we'd just been working on
this thing for a long time. It was getting really late. And so with one of the first versions of
the Go rewrite we launched, the site went totally crazy because we'd broken our rate limits and like
Redis briefly broke, like we were sending Redis too many updates. That was crazy.
Normally, you could only send a limited number of requests per IP. Otherwise, you could just
open developer tools and write a little for loop and start looping through and checking all the
boxes. But were people actually hitting that limit? Were people writing bots and hitting the API?
You know, sometimes you'd go somewhere and like 30 boxes would clear all at the same
time and you'd be like, that's not a human. But I think the scale of it wasn't clear until
we removed the rate limits for a little bit and the bots went crazy. And during the rewrite,
I had noticed briefly some data in Redis that looked weird. And by looked weird, I mean it contained URLs. Nolan had spotted some weird
data before. In fact, he had had the site crash due to weird data on the first big night. And the
way that it crashed is that I wasn't validating indices. So I had an API that the clients used
to checker and check a box. And it took an int. And I didn't validate that the int was in the range
zero to 1 million. And someone was like, what if I manually call this and send 10 million and check
that box? And that totally worked. And Redis's behavior around the data structure that I was
using was like, oh, I was told to check bit number 10 million. I have a million bits. I guess I need
to add 9 million bits of padding to check bit number 10 million, right have a million bits. I guess I need to add 9 million bits of padding to check
bit number 10 million, right? So that happened, the site crashed, they did all sorts of weird things.
And I woke up to this and quickly figured out what was going wrong, but made a couple of utilities
to dump the database. And the goal, you know, my goal then was like, I will bring the site back up
and I will just truncate
the bit set to the first million bits. So I had these scripts that dumped the data in
the database. And they happened to, just based on what Reddit does when you ask it to display
this data structure, convert that data to ASCII. So you have 125,000 bytes, right? A
million over eight. And each of those bytes is a number and we look it up and decide what ASCII character is. And like, that is what Redis prints, unless you're asking it to print a character not in the ASCII character range, because ASCII is only what, like 127? Otherwise, you get like a hex digit, approximately. While running the script to dump checkboxes, Nolan and his friend Elliot spot the URL in the data, but they have bigger priorities to fix. And I saw that while Elliot
and I were working on it. And we were in the middle of this rewrite from Python to go. My
Python had no types. It was wildly unsafe in all of these ways. We noticed it briefly and we're
like, oh, you have another bug around not validating user input.
But I thought that I had a data validation error that let people stuff strings into this
bit set because I hadn't really even thought about what it was that my script did at this
point.
And so we noticed this and we're like, well, we're rewriting this in a much more type safe
way.
Like this bug will just be fixed so like don't
we don't we don't have much to worry about right with the new service up and running they breathe
a sigh of relief and they call it a night and then it got like hilariously faster with the the go
rewrite i have these cpu graphs that were pegged at 100 and then like we launched the go version
and they're pegged at you know one like there's just nothing you know it's it's ridiculous and i woke up the next day and i
ran the script again i guess just because like it was still kind of bugging me like i had noticed
these urls it seemed like maybe i had been not quite hacked necessarily but like something bad
was going on and i ran the script again. And I noticed
that the URLs were still there. And I was like messaging him being like, they're still there.
We're going back and forth. And we're like, this shouldn't like we're type safe. Like there's no
like obvious, like you're just injecting strings into my database bug here. What is going on?
And then I remembered something, which is that I had gotten a bunch of DMs on Twitter about OMCB.
And many of them were just kind of like, hey, I like this thing.
But one had been from someone who was like, hey, can I help you rewrite the site to make it faster?
I am friends with a bunch of people who are good at writing performant code.
And I was like, oh, maybe, I don't know.
I like collaborating with online strangers and giving you access to my code base. It's a little
sketchy, but that is a very sweet offer. And then he had sent me this other message that I had no
idea what to make of at the time that was like, look at the end of the site or something. A friend
of mine left you a message. And that like that message popped back into my head.
And I had no like, I think I had sent him like question mark, question mark, question mark at
the time. Like, what in the world are you talking about? That's like such a weird thing to say.
But realized like, okay, maybe there is something there. This all happened on Monday evening. And
Nolan had started to connect the dots in his head, but also his girlfriend had made him dinner.
And I had kind of like gotten re-obsessed with like what is going on here right as dinner finished.
And she was like, can we like sit down and eat? You've been very busy recently.
And I was panicking about like, there are these URLs. I have been hacked. I have been hacked.
And had like decided, okay, like I'll stop thinking about this for a second.
And sat down and then had this thought and like, you know, I'll stop thinking about this for a second, and sat down and then had this
thought, and it all came together, and then rushed back to my computer to confirm what was going on.
And I think all the pieces kind of came into place, started looking at the actual site,
oh my god, there is a pattern here, oh holy shit, they're not... The data is in my database because
what they are doing is checking and unchecking boxes to make the binary representation of this when interpreted as ASCII spell out this URL.
Like this isn't hacked.
This isn't, I have a bug.
This is like someone who is being very persistent about writing a weird message into my database.
The only way this could happen is with a bot, right? Any visitor can check
or uncheck a checkbox, but this URL is always there. And encoding that URL takes a lot of
checkboxes in a row, right? 192 for this specific URL. That implies a bot is enforcing the data in
this very specific part of the checkbox grid. And the URL linked to a Discord. It took me like five minutes
to convince myself that I could click it since it was catgirls.win slash OMCB. And that seemed
super sketchy, but I did everything I could to convince myself that it was a real website,
that it was safe to visit, and then decided that I couldn't convince myself of that,
but that I would regret it forever if I didn't click the link. So I just had to click the link
anyway. And I clicked the link, went to a Discord, Discord is called Checkin' Boxes,
and it was full of teens talking about botting the site who had been gathering because they were all
looking at this data. They had all figured out how my API worked, they were writing tools that hit
the API and the command line, looking at either
the base64 or ASCII representation of the data for one reason or another, and had all noticed these
URLs and had joined the Discord of, I think, one of the first people to start botting the site.
That's awesome. How many people were in the Discord?
I think there were about 20 when I joined. And then by the time I shut the site down about 60, which is just like mind-blowingly cool.
You know, not remotely what I expected.
Like it's kind of impossible to convey
the feeling of going from,
I have been hacked, this data is still here, right?
Like I think it felt worse
because I thought that I understood,
like, yeah, I was kind of owned,
someone had stuffed these strings in,
but then we fixed it.
And then, oh my God, I haven't fixed it. Like maybe somebody has, you know, root-aligned boxes,
like something is really bad. To wait, there's this whole community and they're doing something
that is like insanely cool. Nolan created 1 million checkboxes without a fixed grid
because he wanted to stop people from drawing offensive images. The number of checkboxes wide
was device specific, So you could draw
whatever you wanted, but most people wouldn't see it. This tied into his idea of curbing bad
behavior. But if you mass check and uncheck boxes and build a CLI app and somehow write to the bit
set as ASCII, you find a URL and it leads you to a discord. just like that community forms and their big goal in addition to
gathering more people was drawing on the site which they were doing by um treating it as a
thousand by thousand canvas so they would have a bot that like it would take in an image and
coordinates and a size and from that determine the set of check boxes that would need to be
checked to like
form that image on this hypothetical thousand by thousand canvas that they were drawing onto.
And then listen to events and check and uncheck boxes to make sure that that image stayed drawn
on the canvas, which was just insane and so, so cool.
So does it farm out the work to multiple of them or are they each?
I think within the Discord, people had a sense of who is drawing what, where. So they solved the problem of like, obviously, if you run two of
these bots on the same territory, you just check and uncheck boxes forever and nothing ever gets
drawn and my server crashes. But for the most part, almost everyone who was drawing stuff was
in the server. There was one exception of some people that are trying to draw the logo for Osu,
the game, who weren't in the server and were fighting over territory with the people on the server, and that was pretty funny to see.
And then the other big problem for them was I had rate limits, right?
Like, I rate limited people based on IPs, and you could work around them by getting more IPs.
And so the people that were, like, the fastest and able to draw the biggest things were the people that could come up with the best ways to get a bunch of IPs to send their traffic from. And these are like kids mostly. They're like
mostly, you know, like 18. So they don't necessarily just have like a bunch of money to like
rent a bunch of servers. They're coming up with clever, like there are ways to route your traffic
through residential IPs. They were using a bunch of stuff like that. So that was kind of the big
constraint for them was having enough IPs and coordinating
which check events you farm out to which IPs.
You need whatever, a central server that's like listening for those events and then farming
out the work to a bunch of different workers, which it's like kind of an impressive amount
of infrastructure to spin up for something like this.
So what were they drawing?
When I joined, they were drawing really simple stuff.
So they had, I think the first thing they wrote was Be Gay, Do Crime.
And then after that, when I joined, they had a QR code that linked to the Discord,
which I thought was really funny.
But they started getting more and more complicated over time.
And so one way that they got more complicated is they started coming up with like big detailed
images to draw uh they did a windows blue screen and death which i thought was really funny
but all sorts of things that they would try to draw over the vast majority of the grid which
just like took more technical work and i say vast majority there because they were very careful to
not touch the top few or bottom few rows because they figured that's where most people were playing
and they knew they were kind of degrading the experience, which I thought was pretty sweet of them. But then the
big innovation later was, oh, we should start doing GIFs and what are animated images that we
could put on? So obviously, we got some rickrolls at some points. We got a lot of Jake Gyllenhaal
GIFs. Do you know bad apple no um so there's this thing i don't quite understand this phenomenon
to be honest but there is a video called bad apple it is a cool interesting like anime video thing
but the point is it's a meme to make bad apple run inside stuff kind of like how with computers
it's always like can i run doom on this thing it's like can i run bad apple run inside stuff. Kind of like how with computers, it's always like, can I run Doom on this thing?
It's like, can I run bad apple inside this thing?
So there was a project bad apple
that a lot of people were devoted to.
And I actually hadn't thought that they had succeeded.
And then when I looked back through all the data
after building some of my own tools to inspect it,
found that someone had successfully run bad apple on it.
So that's cool and exciting.
Now, all that's being concerned is figuring out
the right way to connect with this community. don't want to like ruin it right like
i think it was very fun that i was there and they they like were excited they had a lot of questions
for me they had found bugs whatever they wanted to talk to me but at the same time i did think like
this is your fun little hangout for you and you're mostly like teen friends and i am like
a 33 year old software developer
and like we're in very different life stages and like you should have your own space.
So I did kind of try to like pop in, answer questions. I would pop in and make announcements
like, hey, an interview with me is running in the Washington Post. There will be new users. Please
like limit your botting for a little bit, things like that. And they were happy to listen to that,
which was great. But they had a bunch of questions about the site, about how I was solving things,
and just kind of like about my deal. Like I think they're just like interested and I was happy to
talk about that. In a few cases, people have gotten in touch with me somehow to ask whether
any of the teens in the Discord are looking for jobs. And so I have relayed some job offers to
them, a couple of things like that. But yeah, we've chatted a fair amount and they're great. I don't know. I'm so proud that
they decided to point their talents at my website. And I think they're just crazy talented.
For Nolan, just knowing this group existed changed things. This felt like it really
validated this thesis that I had had from back when I made like Stranger Video and other things.
I had had this belief for a while based on these smaller but still moderately successful
stranger interaction things that the way to approach a problem like this, or a way that
I think works well, is to have a small, fun, novel interaction with enough constraints
that people can't, you know, trivially ruin it. And that those constraints
produce really fun, interesting behavior. And this just felt like the purest possible validation of
that thesis. That like, I believed this. I had given some talks where I believed this and I said
it in the talk and I think the talks were okay, but like I didn't have enough
proof. And this was like all of the proof I would ever need. This was so much of a reason to continue
pursuing projects with this flavor. Yeah, it was validating to kind of like this core thing that I
knew to be true, but needed to prove in a bigger way. And that was also really, really
thrilling. And I don't know, we've talked a couple of times about clarity of what to pursue and what
I know how to do and what the goal is. And I think that was another step in that direction.
It's funny how this direction that Nolan has refined and this direction that he's still on,
it all started off with that playful exploration and boundary pushing around email rules and how they were handled back in high
school. Because back then he was the one who was having the trolling behavior that other people
saw potential in. And it wasn't just his dorm supervisor person who created him a t-shirt
because he also emailed his high school computer science teacher about the rules he was creating.
Like, I don't think I knew what Turing completeness was, and I'm not sure that the
rules were actually Turing complete, but I was kind of like, this is way too powerful for a
school email system. But like my, you know, memory of it is basically him being like,
don't mess everything up. Like, don't, you know, like, that's pretty funny. Like,
I'm sure he was like, kind of proud. Yeah, I don't know. I think back to that sometimes in terms of like, wow, I really, you could easily imagine
a world where the response by the school was like, you're expelled.
You like broke a bunch of stuff.
And the fact that instead I was kind of mischievous, but it was kind of funny.
And it was like ultimately pretty low stakes.
And people just kind of were encouraging was really nice.
There are a lot of versions of trolling that are really fun. If you go back to the version of the
internet that we had in 2008, I think there are lots of places where people did some kind of like
large-scale collective trolling that is remembered very fondly, like 4chan rigging polls to spell out
funny things. The thing that was the most
emotionally satisfying to me was certainly discovering this and then just being so proud
and excited of them and of myself for making something that they thought was worth their time.
Getting to be proud of them and not mad at them was so meaningful to me. I cried a lot about it,
which is not something I normally do. I think that when people think about trolls now,
this is more like, I don't know, people being really mean on Twitter. But providing people
a very low stakes way to troll where the worst thing you can do is uncheck a checkbox is great,
right? It's fun to do that. It's fun to prank somebody in some small way. And it's totally okay.
And I don't know.
I think people really enjoy it when you give them a chance to do that.
That's it.
That's the show.
Nolan's had multiple projects spreading around the web since we last
talked. He's really good at this. Regex Bad Apple is that bad apple animation he previously mentioned,
but playing inside Vim using regular expressions. Everyuuid.com is a website with everyuuid on it
somehow. So thank you, Nolan, for pushing the boundaries. You're clearly one of the world's
experts at twisting things beyond their intended use for our amusement. Go to his website,
eieio.games, and he can send you a newsletter whenever his new creations come out.
And thank you, especially, to the supporters who keep me at this. The main benefit of being a
supporter is literally showing
your support. And I do appreciate that. But there is also a Slack channel and some bonus episodes.
Really just thank you everybody who joins the Patreon supporters group just to show
that they support what I'm doing here. And yeah, we also have a pretty awesome Slack channel that
you can find on the website.
So I hope you liked this one. And until next time, thank you so much for listening.