Programming Throwdown - Scaling Web Sites with Daniel Moore

Episode Date: September 27, 2016

This show is an interview with Daniel Moore about scaling web sites. Show notes: http://www.programmingthrowdown.com/2016/09/episode-58-scaling-web-sites.html ★ Support this podcast on Patreon ★

programming throwdown episode 58 scaling websites take it away jason hey everyone we are doing an interview show and we're super excited to have Daniel Moore on the show and he is an expert at scaling websites, he works at Fog Creek Software So Daniel, why don't you tell the audience a little bit about yourself Yeah, I'm a software developer, I've been developing software for, I guess, about 10 years now, always messing around on computers. And my recent project, HyperDev, is something we started at Fog Creek to really make development easier for most people, like broader than just developers. And we're hoping that it will be a tremendous success requiring more than one database. All right.

That's awesome. So, cool. So, a lot of people, you know, we have a lot of people on this show who are taking CS 101 or maybe Electrical Engineering 101 and they're working on Arduinos and things like that.

I've never built a website. I took, I actually hadn't built a website in a really long, until much later in my career when I got into the sort of web. And so for someone who's never done web, you know, what is, you know, you hear about the web stack, right? You know, like, what is that?

What are some examples? And what does that really mean? Yeah, I guess these days it's kind of weird actually because it used to be i guess like five or ten years ago you'd say like the lamp stack which basically would be linux apache my sql and php so you'd have these like discrete components like an operating system layer, a web server like Apache, a programming language like PHP even have a web server under your control, much less like a database. You could use all kinds of different things. So it's actually getting pretty weird today where it used to be like, you could say,

this is the stack. Now there's so many different choices and so many different ways you can combine things that even drawing the line between like web versus mobile or APIs versus sort of just purely client applications, it gets a little blurrier. So, so you mentioned the like LAMP stack, the Linux, Apache, MySQL, PHP stack. I mean, is that, do people still use that or is that i mean that was super popular you know

a while ago is that kind of fading out and are other things kind of taking it or is that is that still have the lion's share uh i mean it's definitely there are a lot of newer things but i think there's still much like there's still a ton of java and still a ton of cobalt from decades ago. Like the things don't just disappear if they were popular. They usually even gain popularity. It's just they get eclipsed by bigger things that are either trendier or newer.

Yeah, that makes sense. That makes sense. My personal favorite web stack now is the like pure JavaScript web stack. And I've said this on past episodes that the thing I like the most about it is the fact that you can duplicate your validation code. So if you have some code on the client that says this person's name can only be three letters and someone sort of hacks their browser and gets a two-letter name in, on the client that says this person's name can only be three letters and someone sort of hacks their browser and gets a two-letter name in on the server side you can have the same exact code

to block it and you don't have to you know kind of duplicate all of those sanity checks so that's that's my favorite what's what's your favorite kind of web stack that's out there right now let's see, I've actually been experimenting with some stuff using pure client-side applications. It's like just static HTML and JavaScript. But instead of building them by hand and putting them up on a server, I'd build it on a GitHub Pages repository, where the first application lets you load any GitHub repository and then edit it and publish it to its GitHub pages. So it's kind of like a self-contained browser-based editor. And it's purely client-side. So there's no OS, no database.

And it's just kind of a weird experiment, I guess, that actually led me into a lot of the stuff I've been doing recently. Yeah, I think you just melted my brain. I mean, if you don't have a server, then how do you store the user data and all of that stuff? Yeah, so on these type of applications, I haven't done much of the user data stuff.

You can do stuff with Amazon DynamoDB, and you can actually do some really crazy stuff with that. A lot of the Amazon stuff is weird because it's really powerful, but it's hard to get into. It's actually like, I would say, imagine going into a giant alien spaceship and nothing makes sense,

but it's all very powerful stuff. It was not designed for humans, but you can like decipher these cryptic glyphs and then figure something out and bring this powerful technology to others. And that's Amazon stuff. And so you can use stuff like DynamoDB and all these cognito identity provider things.

And you can even do like a pure client side web app using Amazon services as the equivalent to a database backend using GitHub pages or S3 as the equivalent of like hosting a web server like Apache or

Nginx. And it's like really weird to kind of think about this stuff because it's not clear as like do we need web servers? It's like well someone to kind of think about this stuff because it's not clear it's like do we need web servers it's like well someone needs to serve the web page there's definitely a server somewhere but do you need to manage your own web server anymore and the answer is not often oh i see so what you're saying is you so just to explain it for people who have never built a website, usually you have a server and the server has a database running and it also has some server side code. And what happens is, you know, you serve sort of some web pages and the client comes back and says, you know, give me this picture. You give it to them. Give me this, you know, paragraph, you give it to them. But then it says, you know, give me this information about, you know, myself, like some demographics or something. And you say,

oh, that's not part of the website that's on the database. And so the server will make a call to the database, you know, get the person's information that they're asking for, and then send it back to the client. And Dan, what you're saying is, well, why don't you just skip the middleman and just have the client connect directly to the database and get information? I guess it's true. I never really thought about it. I mean, I guess maybe people could abuse that, but you could just rate limit certain clients and things like this you can

have like the client can when they talk to your server they basically get credentials that scope their permissions of what they can interact with on the database resource so you can get like using amazon again as an example you could get an s3 upload policy and give a user an upload policy that's scoped like a subfolder in a bucket, and then they can upload directly to that. And if they try to upload outside of that, Amazon checks their credentials and says, oh, they aren't allowed, so they block it.

So then instead of saying, I'm going to upload one gigabyte straight to the server, and then the server's going to upload it to Amazon, and then we're going to go back and tell the user, okay, it finished like an hour later then the server is going to upload it to amazon and then we're going to go back and tell the user okay it finished like an hour later the user can upload it directly and uh just by passing in these uh credentials around which are very small and much easier cool that's amazing yeah it's really cool i never really thought about it but yeah it sounds i i first one of you had mentioned i thought you were talking about Meteor.js

where you're passing a bunch of data over the wire and building that sort of client-side database. That's cool in its own right, but actually what you're describing is even better because you don't have to even have the server then. Yeah, it's like definitely sort of an evolution because Meteor came out a little early, before it was so clear that you could do this type of thing.

They're probably one of the pioneers, especially in the JavaScript across the whole stack trend and the client-side validation and server-side validation and sort of blurring it so you don't have to see the distinction necessarily between your server side code versus your client side code. I think that's interesting, but I actually feel you kind of have to know the distinction of where your data ends up. Because if it's in the database on the same machine as

your web server, that has certain performance characteristics and implications. And if it's in an S3 bucket and Amazon hosts it in a region, that has different implications. So if you don't know where your data actually ends up, you might make some bad decisions. But it's usually not too bad. Even with Meteor and Amazon, most people don't know. Even professional software engineers don't know even professional software engineers don't really know where their data ends up or what they're doing but we get by somehow

gotcha so the the new stack i've been reading out which apparently has been uh popular for a little while but i'm not that into web development and i just started recently for some reason catching my eye is the serverless architecture, which to your question, Jason, is kind of weird because there's still a server, but this is where you run lambdas on a hosting provider. So there isn't this notion of a machine listening for sockets that come in

or requests that come in for some information. Instead, it's listening for a bunch of websites, a bunch of clients. And when a request comes in, it spins up a process with a bit of code, almost like a function, a lambda, executes that lambda, and then goes away. So you don't have an ongoing concern on a server. And these just kind of pop up, and you just have them on a hosting provider. You're not on a specific machine. Have either of you guys seen that before?

I think that's, isn't that the App Engine is like that, right, or no? I think they, all the providers probably have some sort of equivalent. I know AWS has one, they named Lambda, and even like

there's other ones like Google Container Engine has something container focused which is a little different but they're kind of filling similar roles in that instead of saying there's one server and you put your whole app on it, everything is kind of broken down into smaller pieces. You just put one function on one little service

and then that's the unit you can deal with which has different trade-offs. They're making things really small, make some things better, but some things worse. Yeah, right. That makes sense. So do front-end engineers have to, you know, why do they sort of have to care about the back-end? Like, why can't someone just do everything in JavaScript on the client? I guess these days you can, except you still have to... You talk to some service, even if it's using a jQuery AJAX request or something, you still have to say,

okay, if I want the client to have data on more than one device or to talk to another person or to interact with this at some point in the future i need to take this data and put it somewhere on the internet it doesn't necessarily need to be on my server but it needs to be some kind of storage provider some kind of database equivalent some kind of uh thing where i can store information and then get it back later and so you at least need to know like how to make those jQuery AJAX calls to say, okay, put this image somewhere and then later we can get it back.

Or even put this user's login info somewhere and then later when they come back, we can verify it's the same person and give them their same account. So what if you have to do like, let's say I'm making a site just for Patrick memes. And so, you know, I have to do some computation. I actually have to take the text someone writes and I have to draw it, you know, on an image of Patrick, like, you know, wearing a suit or, you know, whatever.

Like you have to do some kind of computationally intensive thing how would you how does that sort of fit into this whole framework i mean i guess i guess you would like you would have other servers that talk to the same database that do number crunching or something yeah like if it's something like adding text on an image you could actually do that all client-side in the browser using HTML5, using the Canvas API, and things like that. And so there are some things that are fairly simple

that you then can distribute to all the clients and say, okay, you can do this computation locally and then just upload the finished image. On mobile, it's a little harder because they're more resource constrained. So you have to have some idea of your audience and what's the limit of the task that you need to do on a server.

But if you have to do a giant... Say you're making a search engine that crawls all of the internet and compiles these giant databases, you can't necessarily expect to do all of that on someone's client device there are some computations that are uh you would kind of like to have internally although it would be interesting to imagine what would it be if you made a search engine where each client just updated a little

tiny piece of the database and i don't know that's weird and cool and i like it yeah i mean if i think if uh if google.com ran like a little bit of the SETI program, we'd be talking to aliens right now. We would have found them. That's interesting. So if you went to go search something, you would have to contribute some portion of work before you could get your result. Yeah, and I think people are working on stuff like that,

sort of similar, I guess, like with Bitcoin and all this weird blockchain stuff and distributed trust, basically. And I don't know, I guess we're just at the beginning of that, too. And we'll see if it ends up working on a large scale for things like a distributed search engine. I think mathematically it seems like it's possible, but I guess we'll see how it's implemented that would be pretty cool like you uh yeah you get access to some service but only if you you know let your computer crunch numbers for a little bit there's something cool about that especially if it was some type of i mean seti at home kind of thing um you know like you can access your medical records, but while you're doing it,

they're like curing, you know, detecting cancer or something in JavaScript. That's weird because it brings in an externality. I like that you need to do a portion of the work necessary for the service. Because in this idea of like the cost to run the service, I may not have to run as many ads against you. Well, maybe they still want to make money.

But you could in theory run like a not- for profit something like archive.org but you need to go crawl some website for archive.org if you're going to use archive.org yeah it's kind of like bittorrent with the ratios and all of that yeah yeah and the hard part though then is then how do you validate what the person archived is the right thing and how do you prevent groups of colluding bad actors from corrupting your thing and it gets into a lot of kind of weird talked about zero knowledge proofs we had about zero knowledge proofs last episode that's true that's part of the answer yeah it could be it could be relevant yeah that's right the so you know we have we've actually tried to explain docker twice and both times we've gotten hate mail for people who uh tell us we don't know what we're talking about so

so i'm not going to try and do it a third time i'm going to leave this up to daniel uh and i'm going to forward all the hate mail to you so can you try to explain docker to uh to us and to other people who uh who don't you know i mean we know vms pretty well virtual box and all of that um but sort of how is docker kind of different than that different than your os things like that all right i'll probably uh make some mistakes so uh i'll get my hate mail inbox warmed up all right achievement unlocked yeah uh so i guess as i understand it we are actually using docker on part of hyperdev uh and so the way i see it is it's sort of all the things that docker does have been done before in various pieces. So it provides certain things about resource constraints that you can have on a machine. It provides things like a sandboxed file system

that's isolated from the rest of the machine. It provides things with limited permissions and access to the host machine's resources. But it lets these containers in Docker parlance run as if they were their own entire machine. So they don't necessarily know that they're just a sandboxed portion of a larger machine. And it also has some things with like Docker files to specify configuration and how to set up the requirements for your programs and applications. So it basically ties together many things that have existed,

but with like a nicer API and ties them all together into a coherent unit that you can then use as part of your tool belt to solve some of your problems cool hopefully that made a little bit of sense yeah totally yeah totally so so what are some reasons why someone would use docker well let's see i can tell you our reasons of why we're using docker uh it's a little bit different than maybe many people's applications, but I'll start with that. So we use Docker on HyperDev.

I guess for those who don't know, HyperDev, it's a website you can go to. It has an editor in the browser. And in two seconds after you get there at hyperdev.com, we've already launched an application running your code. And your code's in the editor,

you can type in changes and it immediately reloads in your application. It's live on the internet. Anyone else can go to it and see what you've written. So in order to launch something like that very quickly, we use a Docker container image,

a Docker image to launch the container. And the image basically has Node.js and all these operating system level tools that your apps might need. And that image is sort of like the blueprint for the application. And we just take that image, put in a little tiny layer on top that's

your application code, and then it runs your application. And it can come up in just a couple of seconds. Whereas if you want to spin up a VM or even a virtual server or even a physical server, those things would take a lot longer to spin up because they're sort of larger chunks of the stack that they're trying to simulate. And so Docker sort of is simulating kind of the smallest chunk that still gives the appearance of having a full machine. Gotcha.

And so that's how, if you go to HyperDev, which we'll talk about definitely in much more detail later but if you go to hyperdev.com it gives you this machine and if you delete all the files like try to like you know you know bork the machine up the worst you could do is break this docker container which then goes into the abyss somewhere is ignored yep. So the theory is that the security provided by Docker is strong enough that you can't escape from your sandbox and mess up

other people's applications running on the same node or on the same machine. It's always possible that there could be security exploits so we have to stay on top of all of that. But just like with VMs or even on shared hosts, it's always been an issue is staying on top of security.

And I think, you know, it's just a different, like I said earlier, the abstraction with Docker is now sort of the lightest weight thing that gives you the experience of imagining your app has the full machine to itself yeah very cool so uh we talked about you know doing doing client only uh websites but uh if we revert back to the mean and talk about you know most uh websites where you have

a server you know you you're maybe buying a, you know, instance on Amazon, where you're running my SQL database. And, you know, let's say the canonical LAMP stack, right? And you have some website where you're doing, you know, you're doing betting on different events happening in the world. This is like the latest rage, right? There are all these sites where you can go and bet on just random things. Yeah, prediction markets. That's right, exactly. So you're running your own prediction market server,

and it has a few users, your mom, your family, your friends. Yeah, don't do that in the United States. Yeah, yeah, if you do this, you're going to jail. But you're not, you're in Sweden or something. But then all of a sudden, everyone finds out about your server, and it just starts blowing up. People are writing you tons of emails saying, what's going on, I can't get on the website.

You go there and you find the disk is just maxed out and it's just falling apart. How do you sort of scale that up? you go there and you find like the disc is just maxed out and it's just falling apart like how do you sort of scale that up is that something you can do quickly you know if you're in that kind of emergency is that something you have to plan for that takes you know weeks or months and and what how can you actually make that happen uh let's see i guess my advice on it is it's not really easy to do it quickly. Usually what happens is there's no little switch you can throw to say, okay, my server is now scaled up because I pushed that button.

Sometimes you can pay more money and get a bigger instance or bigger database, and that'll go for a little while. But if you're just doubling things, that's still not that much more than if your traffic is like 100 times or 1,000 times. It'll definitely swamp you on the spikes. But I think the overnight success is pretty rare, even, well, that's not true. There are launch days that have a big impact and things.

But usually the site just blows up. Then you recover a little bit and fix the biggest problems. Scaling isn't a single event. It's a process where you're at a certain scale and you have a certain traffic expectation. Every once in a while, you maybe get a big spike or a big thing that comes in. It might knock your site

down temporarily but it will generally return kind of back to what your level was before but maybe a little bit higher and so it's not like a one-time event where you start your site and now it's popular and has a million users like those people had to come from somewhere and they usually came from the users you already have so you grow it a little at a time. And as you hit problems, you can try and solve them one by one usually as you're growing. Like you can profile your application

and you can see, oh, we're writing a lot of data to the disk and we're writing this much every day and it's still growing. So you can count how much disk space you have left and have a pretty good idea that you have to fix it at least you know by the next week uh yeah that's a really good point i think i think sometimes we underestimate or we we under announce really how much time you spend looking at graphs like people think you're a software engineer you're just like banging away code and you know they they watch the movies and they think you're constantly like SSH

and the Matrix is this neon green text going flying by. But the reality is you spend a lot of time looking at data and saying, oh, this is how my website is growing. This is sort of how I have to plan for the future and things like that. Yeah, just like last week i looked at the disk space usage on some hyperdev servers and it's like at 50 i'm like okay i've got a little time i looked at them today and it's like at 80 i'm like okay this is i better start doing something about this you know maybe tomorrow that's awesome so um so there's you know the the sort of lamp stack where you have you know one you know beefy machine um that has like one large database on it and then you also have kind of as you suggested these sort of um you know these

these kind of platforms a platform as a service type of architecture, you know, these kind of platforms, platform as a service type of architecture, where, you know, one sort of person's hosting your server, another person's, it could be a Docker container, or it could be through AWS, through Dynamo and all these things. What are sort of the pros and cons? Like is running a MySQL server on your, on, you know, just kind of obsolete or is there still an advantage to that? You know, should everyone move to one or the other?

What are sort of the trade-offs there? I think running a database server with your application will probably never be obsolete. Like there's a certain simplicity as to why it was popular in the first place. Like you can just get set up on your local development machine you can install MySQL or Postgres you can install Apache you can install PHP

they even have one-click installers that set it all up and it's sort of getting there's some complexity that comes with that that you can move around a little bit by adding Docker or other services. But if you just swap out Postgres for a Dockerized Postgres, you still have one thing you're installing.

You still now need to install Docker instead of Postgres. It really only pays off if you have many things that you're swapping out for fewer things uh and so web servers are and web applications are still like a node app or a ruby on rails or php apps like they're still simple enough that putting everything in a VM or putting everything into Docker, you don't always come out ahead for apps if you're just getting started. So I'd say don't worry too much about it and just do what's the easiest thing to get started. And then later, you can solve those problems one at a time as you encounter them. That makes sense. So for somebody who's maybe uh in high school they

they're taking ap computer science they're on week three they wrote hello world and uh they want to build you know a website uh where they can talk about minecraft or or something like that how how do you get started like from from from zero? Yeah, I guess it depends on if your goal is to just have a website, like a forum or something. A lot of shared hosts, you can just get a domain, get some hosting, and click a button, and then boom, you've got a forum on the internet. And that doesn't really teach you how it all works, but it sort of gets your thing built up.

You can kind of customize the styles. Yeah, and as long as you have access to the source code, that's a good point. You could start off with PHP BB or one of these one-click install, and then invariably you'll run into, oh, I really want Minecraft badges on my forum. Let me go and edit some code. And that can kind of get you into it.

Yeah, definitely. It's actually getting kind of tough these days because everything is like more complicated than it used to be. Like websites are bigger, applications are bigger. There's more layers and there's more components. And it's like hard to find things that are as simple and as hackable although maybe there was never a time when it was simple and it's just like looking back it's like oh yeah the good old days and things are simple but i don't know

it just seems like it's harder to open stuff up now and like dig into it and make a little change and see that it works no i agree i mean people always write and ask you know hey i want to build the next best-selling video game you know how do i get started uh and it's like well writing a best-selling video game is very like you're kind of pointing it's very complicated you know we think kind of much more than it used to be uh i don't know if that's actually true or not but it feels there's many more technologies you need to learn, you know, to write a 3D first person shooter or a massively multiplayer game. And it seems kind of like what you're saying, like, even if you I want to run a forum, that's easy. You can just run one off the shelf.

But if you say I want to learn about how to write these apps, building an easy app. I mean, I guess if you're trying to build something that is the equivalent of another thing, the amount of technologies you have to learn does seem to be growing or you have to be able to do more because people expect more from a web service. I wonder if I wonder if this is sort of just part of progress. And I wonder if maybe the answer is that we have to do more encapsulation right like for example i have no idea um like how like you know assembly you know stuff works and branch prediction i mean i could guess but but i mean i if if you told me to write a compiler that actually produced an executable that could run in windows like i have no idea how to do that and uh um it's just it that has become like a very specialized skill and and compilers have become so incredibly complex that if you wanted to make big improvements to gcc or something you have to be a total wizard but but

then what happened is people kind of moved up in the stack. And I feel like with games, it's the same thing. Like the games, you could say, oh, you know, if I was back in the Atari days, I could make a whole game by myself, including writing all the source code, building the whole engine. Like, you know, Pitfall and Asteroids are written by one or two people, you know with the engine and everything um but now you still have you know one person two person game development companies but they're just not making the engine they're just using unity that's a good point yeah and so so now it's like all the one person game studios are strictly focused on content and uh and and and showing like a new type of pattern that's appealing. To phrase your original question a little differently, I mean, I guess when you don't know, you don't know what you don't know.

Oh, that was profound. Unconscious ignorance, that's what it's called. And so when you're starting out, you're in AP computer science, you aren't in the field. It's very difficult to know what's hard and what's easy. And that's kind of cool because you don't know what you aren't in the field, it's very difficult to know what's hard and what's easy. And that's kind of cool, because you don't know what you shouldn't be tackling. Like you said, you may not know that a compiler is really hard. And so you can go tackle it. And there's a certain kind of awesomeness in there. But I guess like what as a group do we think is a good recommendation for, hey, this is an appropriately sized web project to work on, where you can get your hands dirty at an appropriate level for starting out

well let's see i think the trick is just to not worry about it because if you start trying to figure out what's like the right thing like you're just over analyzing it you have something you want to do just do it and try and do it as simply as you can like find whatever the first thing is that seems like it might work and just try it out and really experiment, dig into it like it is going to be totally overwhelming and confusing at first

but it might not be, you might just luck out and it will work out but if you analyze what's the perfect piece of work, it kind of distracts you from whatever your goal is. Like you're trying to use computers to solve a problem. I think there is no glory in using computers

just to be good at using computers. Like that's kind of a, it might help you accomplish other things, but it's those other things that matter. Yeah, it's a good point. That's a good way to look at it. Yeah, I think that, you know, if people get into computers, you know, I mean, there are some people who get into,

let's say, let's say video game development. There's people who get into that because they like video games. They like the outcome. But there are a lot of people who get into software engineering or development just because they like building things and they like seeing code run really fast and things like that. And so what happens is when they do a personal project, they try to optimize what they like and not sort of what they want, which are kind of two different things like i mean if you write a really awesome game engine that might that feels kind of good in like this visceral way like you see oh look at these triangles and they're so fast and and all these

things um but if but if you if you set out to make a video game and you say, I'm going to build this really fast engine, then you're kind of serving yourself, which is fine. But that's often not what people think is going to happen. I mean, people set out to make some game because they want to deliver a certain experience. And then they end up with three years of building something they think is cool. And then three years later, they haven't even started delivering the experience. And so I think to your point, like, it's, you know, it kind of is innate to people who are analytical to say, Oh, I need to find the best possible game engine. I mean, there's so many websites dedicated to what's the best game engine, or, you know, what's the best database database there's like so many wars over mongo db

couch db and all these people just writing paragraph after paragraph but really just like pick something and just you know go with it and then only if you feel like you've made just a grave error should you really turn around and and if you're picking something popular that's going to be very rare uh that that you pick something that just completely fails you yeah definitely i think uh it's important when you're starting to do something especially if you think oh my goal is to build a game and then you spend years on it like you may have committed too early to building a game you just say oh if my goal is unexplore using Unity, and just how does it work?

And you start messing around with it, and then you can kind of get a feeling for what you actually want to do. Whereas if you commit from day one, knowing nothing about it, I'm going to build the best first-person shooter. That can commit you to a path that is very challenging and possibly not as rewarding as you thought.

Or if you just go with an open mind, think, I want to explore this. What can I do in this system? What is the world like today? What does this tool enable me to do? And kind of follow it. It'll lead you to things. And then maybe once you know a little more,

you can have some idea about what type of game or the scope of the project you can undertake. And just being open to being lost like being a beginner which is why kids can learn so fast actually they don't necessarily have a strict goal in mind or if they do they can change their mind very often to explore new things yeah that's a really good point because often you know what motivates uh people is is when they see something and they think they could do better. You know, there's a lot of people who have a very good eye for, let's say, a very good eye for, a very good ear for music, and they hear a song and they say, oh, you know, I could make that song better. And you might have

a good eye for game design, and you might play, you might play the new Doom, and you might say, oh, I could do this better. But then if you make that your goal, that's going to be very hard. Take this AAA title and make it better. Unless you work there, that's's very unattainable goal i think it's much better to um kind of as daniel suggested um just kind of just kind of let yourself be creative and don't uh try to focus on that kind of incremental uh you know advances yeah a lot of games actually have a pretty big modding community and some of them are still easy to mod like the developers design them in a way to be more easily modifiable and so if you do like you see a game you're like i could make this better

then you can like look for a mod that maybe does the same thing or something similar and like crack open the mod and change it a little bit and like you can make it better and it can be better for you and better for people who share your aesthetic yeah that's a good point it's a good point yeah and if if the mod you know if you if you have to make you know your your my little pony video game with with doom characters then you know that's what you do to to get the prototype out and get the message out yeah and like as you uh expand your skills and like get more and more ambitious you'll naturally run to the limits of your tool and you'll have to look farther like bigger things more powerful tools

learn more knowledge and it's like a natural process yeah patrick yeah did you oh i was just i was gonna side rail the topic i was gonna say that uh you're talking about modding have you guys seen the crazy mods people do in the grand theft auto and the like photorealistic graphics mods oh no i saw the mod where you can fly which which makes the game actually like very different because you just fly from checkpoint checkpoint just playing the missions oh yeah i need to say someone where they yeah yeah that just seems like a different game yeah okay but that was a complete side topic

yeah it's okay actually so not to spend too much on this but there's there is the flying mod which basically breaks the game or changes it dramatically but there's a hulk mod where you can turn into the hulk pick up cars and throw them at people and that also breaks the game, but it is extremely entertaining.

If you don't want to go through the trouble of installing it, just watch it on YouTube. It's very funny. It'd be cool if they incorporate that Hulk mod into the next game. It's like there's a science lab or whatever, and it's just like a temporary thing for one level. Oh, yeah, that'd be awesome. Yeah, that's amazing yeah the uh i'm waiting for the i want a mod for minecraft where it would take random people on the internet it would take like things they've built and put it into my game

and vice versa i think the dark souls for minecraft yeah yeah exactly what are you waiting for you you are a computer engineer dude go do this i know i need to build this you know how to write code yes that's true um so uh so you talked about hyperdev a little bit um and you mentioned you work at fog creek so well first like what is what is fog creek um what do you guys what's sort of your mission uh you work on hyperdev what other things uh does your company work on and what's what's it all about yeah so at fog creek we try and make uh life much better for developers like really give developers great tools to do their jobs. So we have Fogbugs for issue tracking.

We actually were the company where Stack Overflow started out. It's now split out to its own company. And we also were the company where Trello started, and Trello is now also split out to its own company. So our main goal is helping the world's best developers to do the best work they can uh and hyper dev is trying to expand that a little further to not like just necessarily software developers who work as software developers but just anybody who has a problem

they need to solve using a computer because everything's on the internet, everything's in various APIs and services, and you just want the computer to help you solve your problem. You don't want to necessarily mess around with, oh, I need to install Node, but the operating system version of Node

is incompatible with this library, and now i got to install a different version and you spend all day trying to set up your environment when you just want to like get this api and then post a cat photo somewhere so um so like why would someone um why would someone use hyperdev Which I think is, we've already covered that a little bit, but just and also why would someone, what are sort of the limitations of HyperDev? Like when would you say, okay, I can't use this anymore? Yeah, I think HyperDev would be a great

first step for people who either know a little bit of JavaScript or are just starting to learn. Say if you have trouble figuring out all these arcane command line things to install and set up your environment, but you've always wanted to have your app run in, you can go to HyperDev. It's running

in two seconds. You can start getting into this feedback loop where you change your file, you see how it changes your app, and you can change your file a little more and see how it changes you change it again it breaks this time you just undo it and now it's back to where it was and so like once you get into that loop you can really start exploring like just the coding part which is hard enough without having to spend all day just trying to understand crazy insane stuff that developers thought was really cool at the time and now we're stuck with it from like 40 years ago because it's too late to change it

yeah that's how i feel about uh um like so many like vi keys or uh you going to start hate mail. Most of Bash. Like, most of Bash commands. Like, the other day, I needed to output, but it was a big, like, I didn't want to stream it all into one file because it was 11 gig. I wanted to create, you know, a bunch of different files. And there's a Unix command called split, which you just do do split and then you do a file size like in bytes and whenever it gets to that many it creates a new file like a rolling file which is like something i wish i knew you know 20 years ago but it's just like basically there's it made me realize like one how powerful you know unix is and all of that and the core utils but also like how like how poor the

discoverability is that it took 20 years for something that probably would have been useful like my entire career you know for me to find that command and so yeah it's almost impossible it's crazy the way we expect people to learn computers. We don't teach anyone these things. You just have to study these arcane tomes and figure it out somehow. But computers now are a billion times more powerful in almost any

dimension. Yet you can't just go to a webpage and then have it like interactive documentation where you type it in and see it work people are starting to do it but it's like still very rare whereas i don't know it just it always makes me kind of uh weird and angry thinking about how hard we make things for ourselves and for other people like on wikipedia so you want to learn learn Fourier transforms where it's like

transforming a sequence of data points from a time series into like a frequency spectrum. And they've got all these little like animated GIFs with little waves

and they like stack them up and you can see how the waves stack up to make the same data points. But like why is it an animated gif why can't i like grab those waves and mess around with them so like really see how it changes the equation and changes the waves and changes the data and just mess with any part of it like it's kind of uh kind of insane that that's not commonplace yeah i mean the problem is i think it's so hard to democratize really anything i mean this is why

we've had the same gif uh format for so many years right is it's you know and that's why we use email the reason why email well one of the reasons why email is so popular is because i could have some bash script and i could create some output and i could pipe that output to send mail. And I just, I get an email. It's like, you know, it's like baked into Unix for a lot of people. And, and, uh, it's just like the things that have been democratized are so old and it just doesn't seem like there's this winner take all phenomena where if you come up with something better, there's just no way to get any traction. Wait, did you just claim email was popular because you could do it from Unix?

Well, one of the reasons. I think email is popular because you can do it totally wrong and it still works. You can just mess it up and just send, you can spam everybody, you can have malformed headers and whatever, and it will basically kind of work and so you can misconfigure your server you don't have to do it right and it somehow still works yeah i guess i guess you know email is popular for a lot of reasons but especially in the work environment where it's it's not hard to move everyone to something else um the email sticks around because we have so many servers at work that are

sending emails by just piping things to send mail yeah and there's like definite trade-offs too like even if there is a better interface or better like usability if it doesn't interoperate with email or if it is like your data is walled up somewhere you can't get to then it comes with real trade-offs so what's the difference between hyper dev and like js fiddle for example so for people don't know js fiddle is this website um you can go there you can write some html and some client-side javascript let's say you want to um Let's say you want someone to teach you how to rotate an image on the browser. Someone has a fiddle for that.

They send you a link, and you can just see this image rotating and see the source code that generated it. Yeah, so JSFiddle is great. It's a purely client-side tool, though. I guess they publish the HTMLtml and javascript so you can get it back later but its capability is only running in the browser so it's basically a aside from serving up their code snippets it's basically serverless the user's browser does all

the work to execute the commands uh in hyperDev, you actually get your own server and your own URL that you can post data to. You can get data from other APIs and get it into your application. And so HyperDev gives you a full stack application. You can have the front-end components, the back-end components, and you have your own web server that sits and runs for a little bit.

So if I... What's HyperDev used for the database? I guess you use Dynamo or one of these things? Yeah, so right now you can use whatever database you want, but we don't really provide something out of the box. We're thinking of adding maybe SQLite just in the file system later,

or possibly maybe some Postgres, but what I've done is I've spun up a Heroku-free database and then just use its database URL as an environment variable so HyperDev can talk to it. But you can find any provider like Mongo or Heroku

or any of these little databases or service providers and just plug that straight into your HyperDev app. So if I make a HyperDev app and I start getting 10,000 or let's say 100,000 QPS on my app, what happens? Like, do you, does your electricity bill go up? Do you shut the site down? I mean, what happens when something, when a hyperdive site gets really popular? Yeah. So the specific app that it is will probably slow down a bit. It won't be able to serve all of the requests,

but it shouldn't impact the other apps too much because each container has a limited amount of resource allocation. So it'll basically be your app won't be able to serve as much, but all the other ones will only be

slightly hindered. And it might add stress on our like proxy server as well but uh well I mean what if the person like what if the person came to you and said hey my hyper dev site is all of a sudden is amazing you know what do I do I mean like they don't want their app to be slow like do you would you work with them to I mean what sort of the next step there, right? Yeah, so since we're starting with basic Node.js and Express apps, there are a lot of ways to speed things up, like using CDN if you're serving images

instead of serving it through Express from the file system. Maybe making sure the database is probably located in the same aws region if you have another database provider we can try and figure out ways to make sure your stuff is close together and also we're still in beta right now but we want to add paid plans soon so you can upgrade the amount of resources your app has. And I think that should be good for a lot of people because our free tier is, I guess, pretty narrow in what sort of resources you can do.

It's probably great if you have dozens or hundreds or maybe thousands of requests. But if you have tens of thousands or something with extremely high traffic, it will definitely not be enough. So we would like to add ways where you can just upgrade your app without having to worry about it too much. Just pay more money and your problems go away. Oh, man, that sounds awesome.

Yeah. Yeah, that could apply to so many things. Our plan is that HyperDev is really easy to get started and great for prototyping. And our secret trick is once your app works and then your boss sees it, he'll say, great, ship it to production. You try and tell him, no, it's just a prototype,

but he won't listen, and then it gets popular. He's like, well, we could just pay him money and it'll still work. So that's our plan. That's awesome. Yeah, we could just pay them money and it'll still work. That's our plan. That's awesome. Yeah, I mean, that plan worked very well for GitHub and Dropbox and a lot of other companies.

So I think you're in good company. No pun intended. Yeah, we're stealing from the best. Yeah, that's right. Steal from the forgotten masters. Cool. Yeah, that's awesome. I tried it, actually.

I made a website, added some of the Node.js code, added some dependencies. It works really, really well, well, he's my kid's older now, too. But he used to like, you know, just vomit on me and I have to clean his poop and stuff like that. But before that, I used to be kind of like a perfectionist, like everything had to be kind of perfect. And especially with computers, if you know, there was when I first started getting into Linux, I wouldn't I was installing Mandrake, which doesn't even exist anymore. I think they forked it and it's Mandrake now. But I was installing mandrake which doesn't even exist anymore i think they forked it and it's mandriva now but i was installing mandrake linux and if i if i got into like the

smallest snag i reinstalled the os so i mean i was just reinstalling the os every day and uh um and i because it's it's part of it is you feel like you have no control and so you know you starting from scratch is the only thing that you can do. And so it's much harder to sort of get yourself out of a ditch. And it's also kind of like more educational, just keep starting from scratch and trying different avenues and things like that. It's like a choose-your-own-adventure book. And HyperDev allows you to do that with web programming. If you're just starting out, you make a website and you start dabbling.

You end up kind of in a bad spot and you don't really know what's going on. You haven't spent too much time. You can just blow it away and start over. You never have to worry about blowing up your computer or anything like that because you're not the one serving. You're going to blow up Daniel's computer. Yeah, and we know how to deal with that. Yeah, that's right.

So definitely check it out if you're starting out. Even if you're not, I found it akin to JSFiddle, but with all the server-side capability, which is really cool. And yeah, thanks for coming on the show. Super interesting. And yeah, definitely everyone tried out. It only takes a minute. Cool.

Thanks, guys. The intro music is Axo by Binar Pilot. Programming Throwdown is distributed under a Creative Commons Attribution ShareAlike 2.0 license. You're free to share, copy, distribute, transmit the work, to remix, adapt the work, but you must provide attribution to Patrick and I and share alike in kind.