The Changelog: Software Development, Open Source - The future of the web is HTML over the wire (Interview)

Episode Date: April 5, 2021

This week we're joined by long-time web developer Matt Patterson. Earlier this year Matt wrote an evocative article for A List Apart called The Future of Web Software Is HTML-over-WebSockets. In this episode Matt sits down with Jerod to discuss, in-detail, why he believes the future of the web is server-rendered (again) and how Ruby on Rails is well positioned to bring that future to us today.

This week on The Changelog, we're joined by longtime web developer Matt Peterson. Earlier this year, Matt wrote an evocative article for a list of parts called The Future Web Software is HTML over WebSockets. And in this episode, he sits down with Jared to discuss in detail why he believes the future web is server-rendered again and how Ruby on Rails is well-positioned to bring that future to us today. Big thanks to our partners Linode, Fastly, and LaunchDarkly. We love Linode because they keep it fast and simple.

Check them out at linode.com slash changelog. Our bandwidth is provided by Fastly. Learn more at fastly.com. And get your feature flags powered by LaunchDarkly. Check them out at launchdarkly.com. What up, friends? You might not be aware, but we've been partnering with Linode since 2016. That's a long time ago.

Way back when we first launched our open source platform that you now see at changelog.com, Linode was there to help us, and we are so grateful. Fast forward several years now, and Linode is still in our corner, behind the scenes helping us to ensure we're running on the very best cloud infrastructure out there we trust linode they keep it fast and they keep it simple check them out at linode.com slash changelog so i'm here with matt patterson m.e patterson if you're a fan of highly graded horror and science fiction novels you may know him as m.e, thanks so much for coming on The Change Log. Hey, thanks for having me. Happy to have you.

And today we're here to talk about a fascinating article which you wrote for a list apart recently, which is sort of predicting where you think this whole web dev thing is going, which is a common topic we like to take up from time to time. We recently had Amal Hussain and Guillermo Rauch on the show talking about the future of the web. And I think your opinion differs from them. And so I was like, oh, this is a very interesting opinion. A lot of those topics are about pre-rendering, about Jamstack, about the edge and CDNs and where they think things are headed. But you have kind of a different idea, which has been percolating for a while.

Happy to hear about it. First, tell us some of your personal history in web development. You've been doing this a very long time. Is that right? Yeah, it's been a while. I actually, this will date me, but I graduated college during the dot-com bust, which was a fun job market to jump right into. But I actually started out my career doing stuff all the way back to Cold

Fusion. I did about six years of PHP development. I bounced around from company to company, different little startups that flamed out that were doing all sorts of different things from publishing to, you know, hardware to video games. And then I've been in the education space for quite a while to actually at two different higher ed institutions. And so,'ve been in the education space for quite a while too, actually, at two different higher ed institutions. And so, you know, I took up Rails around about 2006, which I think puts it right at about the 1.0 release. And I've been a Rubyist and a Rails developer since. I've also done a fair amount of Python, some machine learning AI stuff, some of which I do for my current job with the Texas State Technical College. So yeah, I've been building web software for quite a while.

Yeah, and have you seen a lot of tools and techniques come and go and the trend cycles and all those kind of things? So an interesting perspective to share for sure. Yeah, yeah. So this post you wrote is called The Future of Web Software is HTML over Web Sockets. And in that, you do give a brief history of some of the web development techniques over the years.

You start in 2005. So we have your personal history. But give that brief rundown of what you have seen happen over the last, I guess it's been 10, 15 years, of people trying to improve our processes of making web apps. Yeah, I mean, I guess you could start at like the two biggest sort of sides, right? Which is, we started with the notion that everybody's computers were, you know, too weak to do anything of real use.

And so we had to have big servers and you connected to these big mainframes. I mean, you can go all the way back to the quote unquote microcomputer days in which everything was served from the server and all your computer really did was just show stuff on screen. You know, today where there's been an increased push to move everything into JavaScript and effectively push all of that onto the memory and CPU of the client computer to do pretty much as much computation as we can get away with up there with the server really relegated to doing what's necessary to maintain persistent data and push things up via APIs or REST, GraphQL, whatever you want to do.

And so along that time, a lot of the stuff that I've seen sort of changes how much we sort of waffle back and forth between those two sides. How much do we push to the front? How much do we keep in the back end? And I've seen that pendulum kind of swing back and forth a little bit. Yeah, it's funny. Whenever I hear about servers or not server side about static site generators, I remember that when I first got into the game, which was in the 2004, 2005 time range, blog engines were static site generators.

The very first one was a Perl engine that basically took your stuff and splatted out some pages. And everybody thought that was so lame because dynamism was so lame. And then because dynamism was like the deal, right? So when I went from, I think it was called Blogsome or Blossom, this Perl engine that could generate a blog. And I found WordPress.

And I was like, wow, WordPress has like dynamism, right? Dynamic backend, CMS, all this stuff you could do. And I could change the way my page rendered on the fly depending on logic, right? Depending on the client, depending on the time of day, signed in, signed out, all these things. And everything went server side to a certain degree, server side rendering. And this was probably in the timeframe that Rails emerged

and said, here's a way of doing that well, right? Let's do convention over configuration. Let's do these things, make it as easy as possible on the web developers so they can be productive, etc. And then over time, that became passé and SSGs have lots of benefits. Static sites have lots of benefits, especially around peace of mind of your site not going down

with at least the pre rendered HTML. And so people started heading back in that direction. And single page apps became a thing because now we have all these tools, we have all this functionality inside the browser. And our clients are getting smarter. Like you said, I mean, smartphones came about every year, the iPhones, CPU got better and better and Android, you know, shortly behind it. And people started putting more and more stuff into the clients. And so what did we find though? Because one of the things that you say in this article is like, we do a lot of work to get that single page application going. And so what we found was what? Was dragons? I mean, some people

still think SPAs are the way to go. And I think in some circumstances they may be. Everybody's situation is different. But what do you think goes on with a single page app that maybe people don't expect or what they find as they get going into it? Sure, sure, yeah. And I've never been, caveat here,

I've never been the sort of guy to say there's like one tool to rule them all. Sure. Or, you know, one web framework and no one should ever look at anything else. I've certainly done a couple of my time. I've built SPAs. I've done React. I've done Ember. You know, one of the lenses that I look at this through a lot, especially at the point that I'm at in my career, is, you know, beyond just what's good for an individual

developer building some software by themselves, you know, what's good for a team of developers that's working for a business or in my case, a nonprofit, you know, that have a goal that they're trying to meet, whether that's a revenue goal, whether that's getting their unicorn off the ground, whether that's, you know, as in the case of Skills Engine, actually getting more people better jobs. And the faster you can get to that goal, the faster you can, you know, outrun your competition, the faster that you can solve the societal problem that you're hoping to solve. And if you're leaning on technologies simply for the sake of leaning on that technology, and you're not taking a hard look at how does this enable the people that I've hired to move as fast and as powerfully as they're capable of, then I think you're missing an opportunity. Now, that said, you know, there's plenty of developers out there who can take, you know, the current SBA frameworks and churn out some amazing products really fast with them.

I'm certainly not trying to say that like, oh, this is a failed, you know, technique that can't be used to do anything useful. Obviously it's great, but I think it's worth sometimes taking a step back, especially if you're a startup, especially if you're trying to run lean and look at, well, you know, do we need to be building to effectively two apps, oftentimes in two repos, and then gluing them together with, you know, JSON being serialized and deserialized back and forth. And then all of the state management issues in terms of managing how you keep your front end state data store in sync with your backend database. Do we need to bite off all of that to achieve what we're actually trying to achieve as a piece of software? In some cases, sure. Yeah. But I think in a lot of cases, I think what you're seeing now and, you know, this you see this throughout the years right in this industry, which is the problems that Google and Facebook and others say, here's our problem and here's how we solved it. Then the next thing you know, there's a three person startup solving their problems that way. And maybe that's not necessarily the right choice for that startup, you know, and I think that's where you're starting to see some of this, you know, this interest in things like the JAMthe-wire, over-web-socket approach provide us a way to go back to the quick prototyping of server-side rendered apps while still keeping the power of a really dynamic user interface. Yeah, there's an interesting debate or spectrum that people talk about with regard to

what they call developer ergonomics versus user experience or thinking about the developer first or end user first. And there's a lot of push and pull. And I will admit that there's definitely some trade-offs there. Who am I optimizing this for? Is it for the developer to move fast and be happy and all these things, which is really the essence of Ruby was developer happiness from the very beginning that Matt started. And Rails, of course, was all about productivity from the very start. But then also thinking about the user first

and putting their priorities first over the developers. And while I do see a tension there at times, where if you're merely thinking about yourself as a developer and you aren't thinking as much about the user, you can sacrifice them for yourself, for example. It should be a whole bunch of JavaScript because it's just easier to pull in one more library than it is to maybe write three or four functions yourself

that are difficult to write or whatever it is, just for one example. But there's no doubt that to a certain degree, developer happiness and ergonomics and productivity produces user experience, produces features, right? Because like you said, speed is important in both contexts, both the speed of the application, but also the speed of how fast you can put new ideas into the world really does matter. And a lot of these things don't go from, well, I can do it as a developer, but I'm not happy to I can do it happily. It's like the difference

between I can accomplish this or I cannot accomplish this. And at the bottom line, that affects your end users more than anything else. Like, well, I don't have the skills to build this particular thing. I'm not going to be able to do it versus I can sacrifice a little bit here and pull it in. So definitely trade-offs to be made. Yeah. And, you know, one of the things that got me so excited when I started digging in to using Stimulus Reflex on Rails and this new HTML over the wire approach was that I realized almost immediately I can build features that I have purposely put off building in numerous

prior apps, because I thought, gosh, that's going to be a big, complicated mess to make that work. You know, especially when you get into the realm of things like multi user collaboration, totally, you know, Google Docs are amazing in that you could sit there and edit a document with someone else. I've never had a desire to actually try to build, you know, a multi-user document collaboration tool because I understood just how complicated that was and how to shake out all the bugs and the edge cases around that. But with this approach, I can actually see a way to that that's much more straightforward. And that's exciting. Totally.

So lay out the approach for us. Web sockets, HTML over web sockets. You're not using JSON. You're saying HTML. You have a web socket connection. Explain exactly how that works. And then, you know, as part of that, trickle out why that's what's good about that.

Sure. And, you know, let me start with a kind of anecdote and an analogy. A couple of years ago, I was having beers with a friend of mine who's a really excellent Rails developer, as well as a really excellent Ember developer. And he had been spending most of his career recently, you know, working in both with Rails as his API and Ember as his front end. And, you know, we were just talking about this sort of topic, the future of what the web's going to look like. And I posited because, you know, I spent a couple of years in the video game industry that, you know, if you look at game companies that are doubling down now on this notion

of a sort of thin terminal streaming future, right? And say whatever you will about cyberpunks uh issues right you know the fact that it actually runs reasonably well on stadia is nothing short of astounding and you know the game industry would much rather push towards that than asking everyone to buy an 800 monstrous gaming rig in a tiny box that you put in your living room. Right. And so I posited with my friend, well, wouldn't web apps ultimately go the same route as broadband, you know, penetration increases across the country and the world? Wouldn't the right approach be to ultimately get to where

if I'm a business that's serving you a website, I'm paying for all the cost of serving you that site. I'm not without your consent jamming megs and megs of JavaScript onto your device and asking your device to give up, you know, 60% of its computing power in order to look at my website. And so, you know, at what point do we say, hey, you know what, there's a way that we can give users on this sort of thin terminal approach just as rich of an experience, but everything is constantly just being pushed to them, streamed to them, if you want to use the video game approach from the server side. And so, you know, he thought, well, that's an interesting idea. And we kind of dropped it at that. Fast forward a few

years, I discovered Stimulus Reflex. And of course, you know, that kind of was born out of Chris McCord's live view over in the Phoenix framework. And I thought, gosh, this is kind of that. So, you know, to your actual question, you know, the general approach and whether you're really using WebSockets or like Basecamp's new Hotwire library can even sort of fall back to just using AJAX to essentially accomplish the same thing. But the idea is that

rather than have a server that's presenting an API of raw data, right? Your list of users from the database or your list of widgets. Instead, just like in the old days of a server rendered app, where you would push a whole HTML page to the front end on every, you know, click, every action, every page change. Instead, what we're going to do is we're going to send you the full page to start. And then any action that you take, just like if, just like if you're in an Ember or Angular app, we're

going to have a seamless interface because what we're actually doing under the hood is sending that request down WebSocket and telling the backend, okay, this is what the user wanted to change. But unlike the HTML request response cycle, all we're really doing is telling the backend to do this, right? So it changes the database, changes state, whatever. And then it on its own terms, because you have an open full duplex socket connection can then just tell your client, hey, you know what, I need you to change this chunk of the DOM. So the silly example is you create a button and a little counter. And when you click that button, it just tells the DOM. So the silly example is you create a button and a little counter. And when you click

that button, it just tells the DOM to replace that number with another number. But the point there is that the backend knows what number it just told the DOM to change to, right? It's not a, the front end has a data store, the backend has a data store, and gosh, we hope that they're in sync. Which is where, you know, a lot of the complexity comes in, right? It's not that it's an impossible problem to keep those in sync, but the complexity of what happens when the backend just flakes out on

what the user just told it to do from the front end. So now you've already updated the front end to make it look really slick and fast, but now the backend didn't actually do the thing. So then the front end has to have some code to then figure out how to roll back the visual change it just made. Right.

Right, and vice versa. What if the back end did something, but the front end never got the response for some reason because you're on an unreliable internet connection? And every hour you spend working on that problem is an hour you're not working on pushing the application forward, right? That's exactly right, yeah.

That's an hour, you know, I really pushing the application forward, right? That's exactly right. Yeah, that's an hour. You know, I really liked the phrase that I heard in one of the Discord channels I hang out on. It's, you know, use tools that give you a surplus to do awesome. So if you're using your time and your energy and your brainpower to do mundane things, then you're not doing awesome things. And, you know, you could argue that that goes all the way back to DHH's approach with Rails, right? Convention over configuration. Hey, let's get all the boring, you know, bland stuff out of the way so you can

immediately start building the features. our friends at retool help you to build internal tools remarkably fast stop wrestling with ui library stop hacking together data sources, and stop trying to figure out those access controls. Start shipping apps that move your business forward. Learn more and try it out for free today at retool.com slash changelog. Again, retool.com slash changelog. so before we continue down the technical road of what this all looks like i want to take you back to that conversation you had because i think that was a pretty good analogy with regards to the streaming of games or that direction and the

streaming of the application, kind of its final state down the wire versus making the client do a whole bunch of computing. And I wonder if there's a fly in the ointment. Some of these things in the industry, you just don't see them coming. I think back to the iPhone. And prior to the iPhone, I remember John Syracuse was making these blog posts about how Apple had to replace Objective-C like yesterday. They needed a higher level language in order to compete with other platforms. And then the iPhone dropped and the App Store dropped and the iPhone's computing power compared to desktops was just minuscule. And Objective-C, because of its low foundations and its manual memory management,

and all these things that make it not a very desirable high-level language, actually gave Apple this advantage for a few years, where their phones and the apps on their phones could outperform a lot of the competitors because they didn't have GC and these other things that Java has on Android, for example. And that bought Apple probably five years before they had to come out with Swift. And so that's just an example of like, well, nobody saw that or maybe Apple saw it coming, but those

of us watching said, you really need to do this. And it turned out, something changed. They didn't really have to do it for a while at least, and then Swift came. I'm not sure how many years later Swift came, at least five, maybe ten years later. And now they're competing at that higher level. Well, one thing that's changed with regards to apps is privacy and security.

And I think about this with regards to the client-server relationship. I think about this with regards to the client-server relationship. I think today we are less likely to trust the server than we are our own clients. We see pushes for offline apps. We see pushes for phone-only apps, where the database is only local. These are things people are trying to build. We see the ad blockers, the tracker blocking,

this whole movement towards privacy because our privacy has been abused by people running servers for a while now and we're waking up to it. Has perhaps pushed people away from the idea of, well, just stream everything down from this server that I can trust because I'm not sure I can trust that server. I just wonder what your thoughts on that fly, maybe fly in the ointment of the idea of as much server side as possible. Yeah, I mean, it's an interesting point, right?

I mean, part of my answer would be, well, you're probably never going to get away from the server entirely unless the app that you're building is uniquely suited to running entirely on the client side. And that's where I've seen some really great uses of frameworks like Ember and Angular, where they just don't really need a database or they need an ephemeral data store that they can use browser storage for. Yeah, like I'm tracking a budget or my taxes or something. This is not a collaborative thing. This is not something that I'm going to put into the cloud.

Or I'm playing a game. I'm playing Solitaire on my phone. I do not need a network connection for Solitaire on my phone. We see this a lot, even in the gaming industry. One of the things that I don't enjoy about games today is that when I'm playing offline on my Switch and the game requires a network connection. And it's like, no, I'm just playing

single player mode on my Switch at my house. I do not want to be, have to be connected to the network. And so those things of course are often business concerns and not technical concerns in that sense. But this constant connection to the server, I think some people, and it's probably a minuscule amount, we're rowing weary of that to a certain degree. But I'm with you. I mean, some things you absolutely do need to have cloud storage.

Yeah. And I mean, you know, I'll use the startup that I work for as an example. You know, we do very complex things with the data in the backend, right? And that complexity extends to, you know, doing machine learning and AI and NLP work with it, as well as just really complex data relationships and event tracking and things like that. And when I say tracking, I don't mean personal tracking, but like tracking how the app is working, what's happening whenever you change a thing. And trying to push that off on the client is incredibly complicated. And I know because we have for at least one of our apps.

And it's a really hard set of problems to solve there when you're trying to have that complexity exist on a front end data store and properly mirror what's actually going on in the real database where the data is actually going to persist past you closing your browser window. And so, you know, I don't think that we're going to get away from needing persistent data stores and, you know, heavy computation. And that said, it's amazing the kind of machine learning stuff you can now actually do on phones and things like that. But I don't know that I see JavaScript

adding in a browser-based machine learning model too soon. I don't know, they're shoving a lot of stuff in there. I'm sure there's a GitHub repo out there where somebody's built an implementation of TensorFlow that runs in JavaScript. There's TensorFlow.js for sure. That's Atwood's law. Anything that can be written in JavaScript eventually will be written in JavaScript.

But my point still being that I think when you're talking about an app that deals with large amounts of data across large amounts of people and accounts and users, and you want to be able to do really complex and sophisticated things with that data, I think having it on a server in a database is going to continue to make a lot of sense. And, you know, and then there's a, and I kind of brought it up once before, but there's a little bit of a consent argument too.

Like, did the user actually consent to having us jam so much into their memory and CPU? I think I would argue that, you know, my mom doesn't know how much stuff her phone is actually running computationally when she accesses certain websites. She's not aware of that. And so it's, it's an interesting point, because yeah, the privacy stuff is really concerning. But I think a lot of folks don't realize the degree to which when you go to websites, they're making but i think a lot of folks don't realize the degree to which when you go to websites they're making your phone do a lot of stuff or making a browser on your on your laptop do a lot of stuff that you don't know it's doing right especially with the rise of

crypto miners.js oh yeah and not to mention the fact that you know one of the harder parts in building any anything that runs purely client side is if you want to do any kind of integration right with third party whether you're you're integrating with some sort of IAS or some sort of other cloud-based service well there's not a great way to store client credentials in the front end and so you end up probably having to build something on the server side anyway to protect credentials to your third-party integrations. Yeah, there's lots of challenges there. I like the idea of really the user's phone,

their network plan, all these things as user concerns that people, I think they are waking up to them more and more because they are more tangible costs than privacy costs. Like trading in my privacy is very hard to intellectualize exactly what am I doing here. But if I'm on, especially if I'm on a capped connection or I'm paying per megabyte or my battery is dying in the middle of the day

and I'm seeing like actually using this application is causing me real hard costs, right? Right there in my wallet. And I think people are starting to make decisions around that. We see it happening with the phone operating system starting to percolate up to the user. What is draining my battery right now? Well, it's this application. Ah, you know what? I'm sick of that application. I'm not going to use it anymore. So I think the caring for your user, part of that is for their resources and their devices. And the more you can do on the server,

especially when you're talking about heavy computation, the more you're actually caring about them in that way. So good point. Yeah, I mean, the number one use of all computational power on my laptop is Chrome. And the number one use of all computational power and battery power on my laptop is Chrome, and the number one use of all computational power and battery power on my iPhone is Safari. That tells you right there who's doing all the work.

Let's talk about WebSockets then, because this is the backbone of this new way, the old new way of doing web applications. This way, sending HTML over WebSockets. Now, I guess the first question is WebSockets. Now, I guess the first question is WebSockets, can I use? Because it seems like they've been flaky for a very long time. And a lot of people maybe tried WebSockets a decade ago. And you know, there's all these fallbacks, and it rarely ever worked. And you had NAT problems, you had firewall problems, all sorts

of problems, at least that was my experience. and it always ended up falling back to polling anyways but is it relatively well supported i mean websocket's like standard technology in all browsers today i mean it seems to me that as long as you're running on any of the modern browsers you know there doesn't seem to be significant problems with that i mean enough so that you know the base camp guys have doubled down on it with Hotwire. Now, they even have, as I mentioned before, a fallback in there. So if you're really concerned about that sort of thing, there's no reason you couldn't also implement a fallback to Ajax in the case where you couldn't get some data. And I think some of that comes down to what your app does, what you need it to do, what you care about deeply, right? Do you want to build your trading, stock trading app using, you know, a WebSocket?

Well, you probably want to have a fallback there to ensure that, you know, a trade that's trying to go through in millisecond time actually gets through in case something flakes out. That said, you know, the apps that I've built so far with this approach, I've had almost no issues in that respect. Like with modern browsers, they work just fine. And if anything, most of the time, the issue is either something I created myself with bad code or, you know, an internet problem. Like I actually lost my connection at which point, well, it doesn't matter. None of your stuff. Right. So the major advantage of WebSockets over AJAX is that you have a persistent connection with the server, right? And so all you have to send then is the exact information that you are sending back or that you are interested in, right? The exact HTML elements that need to go into the DOM are getting

sent down the wire. You do not have to set up a TCP connection. You do not have to send HTTP requests. You can just send push and pull commands. And that's a much faster, more efficient way of communicating with the server versus Ajax, which is ultimately just an asynchronous HTTP request, right? Right. I mean, the overhead on your average HTTP request is, you know, quite a bit higher, right? Then, then once you already have a persistent full duplex connection. So, you know, we're talking, you know, a button click sends a tiny bit of information down the web socket that says they clicked this button, right? And some, depending on the framework, if you're using

stimulus reflex or hotwire, you know, there's going to be some other stuff wrapped around that to tell the backend what's going on. But it's, you know, a minor amount of data. which is it literally just lets Rails re-render the whole page and send that whole page HTML back up and then has the front end use the morphdom library in JavaScript to just surgically alter the DOM to match the new HTML, which makes it look really slick and fast without you having to worry on the server side about figuring out what you need to change. You just re-render and it's like you're building

an old-school server app and yet it feels new school, so to speak, on the front end. But you can also fall down all the way to selector modes and things like that, where you say, you know what, actually, I'm just going to send this, you know, node in the DOM changed because I know that that's all that I need to change for this particular user request. And so that can get down to, you know, tiny amounts of data. It's just a little HTML framework, right? And there's nothing, as I pointed out in the article, there's really nothing stopping you from also just sending up text, right?

I mean, it's a web socket. You can send up any data you want. You don't have to send HTML. You could literally just send a number back if you know that that's all the front end needs to change. So let's take a very specific example, because one of the things that you do say with WebSockets and this way of writing apps is good is you can do form validation. And one of the pains of my life is writing the same form validations in two places because it just hurts, you know. So I have it on my server side because you cannot trust your client so you must have

your input validation server side if you're not doing that you should stop listening right now and go change your code and do that server side before somebody some client you can't trust sends you things that you don't want but then you also want that in the form right because it's slow to like well I'm going to fill out the form

I get so mad even to this day when I fill out a form and I submit it and it round trips and comes back and there's things that were wrong. It's like, you should have told me that while I was filling it out. So this is a classic in the browser thing you want to do. So there's been solutions over the years of like, well, I can write it on my server side,

but then I can matriculate that logic via some reflection in JavaScript and this linking between, you know, inputs and form fields. And I can have it written in one place and kind of just like matriculate it over into my JavaScript. And that's been, you know, moderately successful. You have a lot of people that just write it in both places, you know, it's like, well, I'll just do it in the front end. I'll do it in the back end. And sometimes that's different people on different teams. And so that logic, like you said, the logic gets bifurcated at some point. But speed has always been a problem when it comes time to doing that. So some things are easy, even if you're doing like the, well, I'll just generate the JavaScript to do the same validation. Email addresses, right? This has to be a number. Those kind of things are easy. But the hard one is uniqueness

because uniqueness has to live where the source of truth is and the source of truth is the database, right? So you always have to kind of round trip for a uniqueness check. And so what I was reading into this article, you're saying, well, you can do form validation over WebSockets.

And I'm wondering, like, it's always been too slow to do that round trip in a way that's interactive, right? I'm waiting for the server. I remember even Twitter used to have a username uniqueness validator, right? You type it in and it would sit there with a loading spinner and it'd be like, I'm sorry, this username is taken. Well, that's because they're checking their whole database of users to see if that handle's available, right? Does WebSockets provide enough of an efficiency improvement over AJAX-style requests

to where those kind of checks can also be just server-side only and you're just asking the server, like, is this valid? Is this valid? Is this valid? Does that feel pretty fluid? Or is that a thing where you kind of have to throw a loading spinner in there and wait? I mean, for the most part, I think yes.

Right now in the Discord channel that I like to hang out with, with some of the folks that created Stimulus Reflex, and we talk about Hotwire and a bunch of these other approaches, one of the caveats we'll always throw out there is, well, do you really need to do that? Right? Because sometimes you really could probably just get away with a good old fashioned, like just send a new page, you know, render back up the old fashioned way and show the, you know, the error fields. And imagine, I don't know, some part of your app that's very rarely used, but has a form in it that the user can form it, you know, can type in. Do we really do it? Does that user really need to know that that fast? Are they okay with, you know, it's got three fields and if they didn't fill one of them out,

it tells them that it's not there. The uniqueness check, you're right, is an interesting example because that's where you then can get into issues around, well, how fast can you make a uniqueness check on your backend? Irrespective of what front-end technology you're using, can your database look through 10 million rows and figure out that this thing isn't unique quickly enough? And there's tricks and you can get all the way into database partitioning and do all sorts of things in order to try to speed that up. Right. Yeah. I mean, there's already gems in the Rails ecosystem.

There's a gem called Optimism that basically does this in real time. So you put a debounced key up event on every field in the form. And every single time you stop typing for some number of milliseconds, it just sends everything that the form currently has in it down the wire. It instantiates a new Ruby object on the back end, runs it against the validator. And if it says false, it just sends the HTML backup to, to change that form with all the errors written out. And you can say exactly what was wrong, right? The uniqueness one is interesting because that's always been the sticky one, even in the rails back. I've seen plenty of apps crash years after they were built because somebody put in a uniqueness check and didn't realize that's not going to work great when you have a million rows. So that one, you know, maybe for your example with the Twitter one, maybe that's a field you exclude from the, you know,

the debounced key up. And that one has its own little spinner and its own little way of interacting with the backend, you know, since we do have to do a big long check to figure that out. Maybe, or maybe you can get away with it. Maybe you can throw them all in something like Elasticsearch and you can use that for your uniqueness check in, you know, a ridiculously fast amount of time. Or you can just ship your entire users table

into the browser and then it'll be instantaneous. Hey, why not? There's no security issues there. We have 10 million users and you have them all inside local storage. Congratulations. This episode is brought to you by our friends at O'Reilly.

Many of you know O'Reilly for their animal tech books and their conferences, but you may not know they have an online learning platform as well. The platform has all their books, all their videos, and all their conference talks. Plus, you can learn by doing with live online training courses and virtual conferences, certification practice exams, and interactive sandboxes and scenarios to practice coding alongside what you're learning. They cover a ton of technology topics, machine learning, AI, programming languages, DevOps, data science, cloud, containers, security, and even soft skills like business management and presentation skills.

You name it, it is all in there. If you need to keep your team or yourself up to speed on their tech skills, then check out O'Reilly's online learning platform. Learn more and keep your team skills sharp at O'Reilly.com slash changelog. Again, O'Reilly.com slash changelog. Again, O'Reilly.com slash changelog. So we've been talking a lot about Ruby on Rails, and to be fair, we haven't talked a lot about Ruby on Rails on the changelog recently. It's just not something that comes up very recently.

And you're given some of your history and how long you've been using the framework. And so for this last part, I'd like to just focus in on Rails a little bit because, of course, this HTML over WebSockets paradigm or technique can be used in any technology, right? It's just a web thing, right? WebSockets are not language specific. Yeah, even Basecamp's Hotwire framework that they just dropped in December is usable in any language.

I mean, obviously, it integrates well with Rails because that's their framework. But you can drop it into any of the other frameworks. Yeah, exactly. That being said, in the post you do, and I like the way you said this, you put your contender for best framework in a leading role. Maybe you're getting ready for the Oscars,

which are still upcoming now, was Rails. And that was kind of a refreshing new take on an old, I mean, Rails has become, what's the word I'm looking for? Maybe boring is the right word. Most people- Established.

Yeah, established. It's kind of grown up and matured, and it's still used by many, many people around the world, but it's just not exciting to many folks today. And it's not usually the framework that they say, you know what's the best for this? It's Ruby on Rails. Not very many people say that anymore. And so I kind of like seeing that because it's like, oh, here's a guy who's like, you know what? This is a great way to make web apps.

And the best way to do that is with Ruby on Rails. So please tell us more. Sure. Well, you know, I want to throw out a statistic. There was an article that went around recently where they, I think it was on Y Combinator, where they were looking at the revenue brought in by various startups. And they enumerated the technology stacks that each of these startups did. And they came up with about $92 billion in revenue were driven by Ruby and or Rails-based startups. Now, obviously, you know, you've got the

old example of Twitter and eventually they, you know, pulled a lot of the Ruby stuff out, replaced it with things that were faster when they needed that kind of performance, right? And so, you know, you're going to have examples of people using, you know, nobody's using only one technology once they get big enough that they start running into two major problems. But would Twitter have ever existed if it was from, if it was scale off or Scala from the very beginning, right. Or like these, I mean, who knows, right. Exactly. But, uh, you know, I've been a rails guy, like I said earlier from about 1.0. Uh, so obviously I'm biased, right?

But I've also built things in Python. My company does both Rails and Ember. And I've worked on other frameworks. I've done some Elixir here and there. And so partially it's a personal bias. But I think, at least looking back over my career, one of the things that I've always appreciated about Rails is that it's simultaneously highly opinionated and a very welcoming community,

which is an interesting mix, right? It's opinionated in the sense of the original convention over configuration philosophy, and the community will often work pretty hard to not fragment out into a million different ways of doing the same thing, like you see in some other languages. Sometimes that's good. Sometimes that's, you know, not so good. But I think what it allows a lot of smaller cash-strapped startups and, you know, idea kind of folks to do is grab a framework like Rails and immediately start building something and not be worrying so much about getting it all set up and architecting how we're going to build this. And I think that's partially why I reacted somewhat strongly, reacted to, you know, the addition of this heavy JavaScript layer on the front end.

And like I said before, you know, it has its place, I think. But the ability to get in quickly and prototype things with Rails to get an idea off the ground, I personally don't think that there's a framework out there that is quite as competitive. Laravel's amazing. Phoenix does some incredible things, especially if you need to build an app with high reliability because you're sitting on Elixir's beam. But for the ecosystem and the welcoming community and the interest in continuing to push the envelope forward while still sticking with the things that have worked and continue to work well, I think the Rails community is hard to beat. So tell us about Rails' WebSockets story.

What does it look like to use these technologies inside of Rails today? Well, I mean, I think like anywhere that's starting to tinker with this HTML over the wire approach, you know, people are trying different things simultaneously, right? So Nate Hopkins put out Stimulus Reflex, or he started on it a couple of years ago. I don't know the exact date, but it really started to take off last year, which is when I discovered it. And now it's up to version 3.4, something like that. And then in December, Basecamp dropped Hotwire, which is their own sort of take on it. And it's not necessarily a competitor so much as just a different approach to it, depending on which you like better and what you need.

You can actually use both of those frameworks together if you want to get the goodness of what Hotwire does really easily. And then Stimulus Reflex is sort of you're more of a Swiss army knife for your more complicated things. You know, that's my take on it, at least different people have different takes, but they're both really great frameworks. And so your options there are starting to expand. There's another one that I'm forgetting the name of right now that also came out that kind of does a similar thing. So there's a couple of different libraries in the community, at least on the Rails side, that do this HTML over the wire approach. And some are kind of starting with the notion of like the model itself informs the front end directly of anything that needs to change.

Others like Reflex kind of are more of an alternative to the controller when you want to have Reflex-based actions. So there's a wealth of options appearing. and MIRB collapsed years and years ago, or if they're just going to sort of, like user authentication, just going to sort of hang around with there being some that are kind of the big ones that most people use, and then some that are more niche for if you have certain kinds of feature needs or certain kind of problems

or you just prefer their style. Hotwire is interesting because it seems like in the past, DHH would have just put this into Rails, right? He would have been like, this is how Rails works now. And he didn't do that. They released it as its own deal.

As I understand it, I believe Rails 7, they're planning to have it be built in. And I believe I just heard that Rails 7, they're planning a release of it this year, 2021. Now I may just be repeating rumor, but... But now we're repeating to a bunch of people. So it might as well be true at this point.

Make it so, Rails people. It makes a lot of sense, right? I mean, they've rolled out Hotwire. They're using it in their own Rails apps. You know, he's said that they see it as kind of a replacement for UJS, which is the sort of old school way that Rails kind of does these JavaScript partial swapping. Right. So it might get in there.

It makes some sense that Rails 7 or at least somewhere beyond Rails 7 would start building in hardware. My guess is like Stimulus.js, it will be one of those things that you don't have to use if you don't want to. But if you want to use it, it probably will be really easy to just

integrate. It'll be part of the Rails way. Yeah. I mean, you've already got Action Cable and all that stuff built into the framework, right? So you've already got all of the WebSocket cabling stuff is already there. So it's just a matter of, do you want to use this way of adjusting your DOM on the fly? Don't call it a comeback

because Rails has been here for years. I'm not quoting you to you, but I like that line as well as an old LL Cool J fan. Have you used this approach? Have you built anything? You know, have you kicked the tires on what this actually feels like in practice? Yeah, absolutely. The first, the sort of starter app that I built kind of going back to last year in the middle of this pandemic, I have a dad's D&D group that includes my's-in-law and a couple of other dads. And we get together, you know, and play on Zoom. And I was frustrated with some of the different tooling out there.

Things like Roll20 are amazing, but I don't really need all of the stuff that it does. I don't really need battle maps and all that kind of stuff. So I wanted to build something that was sort of more tailored towards my group and what we needed to, you know, sort of track our gameplay while we're playing virtually. And so I said, well, I'm just going to build my own app. And so I started building it using Stimulus Reflex on Rails 6.1. And that bought me, you know, easy chat integration, which, you know, pretty much works as well as, you know, Slack and other things like that in terms of, you know, just being able to quickly chat and you have emojis and all that sort of stuff. Built-in dice rolling, built-in multi-user notifications. So, you know, when you're tracking things like combat, if I take an action, everybody else that's currently looking at the

table sees those actions happen. You know, these are not amazingly new things, but the difficulty of building those things with prior approaches was always really daunting. And I built most of this stuff in a couple of weeks, only working nights, like after the kids had gone to bed. So, you know, anybody that has kids knows that, you know, you get like a good two hours maybe before you fall asleep. And so, you know, I built this in record time compared to apps that I've built in the past with similar kinds of functionality. And so that app's called Chaos Mage or chaosmage.app. startup I work for at Texas State Technical College, we're actually, you know, taking a hard look at using this same approach for some new stuff that we're building as well. So we're going to be rolling it out as, as, you know, some really advanced interfaces for the kinds of problems that

we're trying to help solve in society. Yeah. Any hiccups along the way or challenges or things that aren't quite there yet that you've run into well to uh to fall back on the old quote one of the hardest things in programming is cache invalidation right and i saw somebody on twitter mentioned that um they thought they read my article and thought well that sounds like a really interesting approach i wonder if there would be problems with caching and it's an astute point In order to get things to be really fast, you know, you want to fall back on caching so that you're not constantly hitting up your database for the same thing that the user just asked about a split second ago when they last clicked that button. And you know that thing hasn't changed.

So you throw it into Rails cache, and then you end up in a situation where the user's looking at the page and they can't figure out why, you know, something hasn't changed on the page when the DOM is supposed to be getting surgically modified. And it's because, oh, well, the backend's just sending the same data back up the socket because you didn't bust the cache properly. If you're falling back on the classic Rails ways of doing those things, you're not getting the whole controller request response cycle. And so you may not be busting the cache the way you thought you were, and you may have to do it yourself. But that wasn't so much a limitation of the approach as much as my bad code

and not thinking about when I cache something and when I needed to bust it before I send stuff back up the wire. I think that was about the only really major hiccup I ran into outside of some issues of integrating some JavaScript stuff together that was really weird. Some of the dice rolling graphics libraries are pretty insane. Gotcha. Well, I'm sure Chaos Mage hasn't had to scale quite yet, but with Rails, the old adage was, can it scale, you know?

And so I wonder with Action Cable and with WebSockets over Rails, surely it can scale at this point. Is there any concurrent connection limitations? I mean, I'm sure there are limitations, but where do you start hitting up where you're like, oh, I need multiple nodes or that kind of thing? Yeah, I mean, I'll back up to my video game days i worked for a triple a game

developer out of dallas um and worked on a really hotly anticipated game and we had a primarily ruby based uh server infrastructure running all of like the data collection and things like that from you know people playing the game and i mean we're talking like terabytes and terabytes of data that would come in constantly. And we were able to scale all those things up with only a couple of instances running on the cloud to handle all the traffic on launch night of a AAA title. So this notion that you can't get Rails or Ruby to scale, I've always thought it's kind of silly. Like, well, maybe you can't, but, you always thought it's kind of silly. Like, well, maybe you can't.

But, you know, it's more of a question of, well, looking at what part of it is you're having trouble scaling, and then what do you need to adjust in terms of its normal conventions in order to handle that level of traffic? And it's not that big of a deal, I think. But, you know, when we get to the WebSocket approach, you know, anecdotally, it seems as though the standard WebSocket server that Rails spins up as part of its web server process, you know, and you're not interested in horizontally scaling out, then you could swap out the built-in WebSocket server for any cable, which is written in Go, I think. Something compilable, so it's super fast. And I think they've measured it up to like 10,000 concurrent connection, WebSocket connections without any real issues. And then, of course, like you can just horizontally scale beyond that, right? If you've got that much traffic coming in.

It probably depends to some extent on what you're doing on these sockets, right? If you're relying on entire full page reflexes, for example, it's going to be, you know, you're going to be doing a lot more work than if you're doing a lot of little surgical DOM changes. Well, Matt, we have just a couple of minutes left. You want to tell us about your books real quick? Why not? Squeeze it in there. Sure. Yeah, I mean, sure. I love talking about the books. Yeah, I don't know. I, you know, I actually got an English degree from college, so I have no business doing any of the software stuff.

But, you know, fiction writing was always a hobby of mine. And I decided a number of years ago to sit down and start writing some books. And this was sort of back when like the TV show Supernatural was really hot and like demons and angels and supernatural stuff was kind of a hot thing. And so I've written two novels kind of in that vein, centering around a guy in Las Vegas who who his sort of supernatural ability if you will is that he's possessed of just horribly bad luck for everyone else around him and that gets him sort of wrapped in a whole world ending plot with with demons and angels and so there's two books in in that devil's hand and burning cards and then i also have kind of been working on my own science fiction universe for years and years, like anybody who plays role-playing games probably does. and features sort of a young kid in sort of a space opera setting

trying to find out the mystery of where his long-lost father went and how that ties into the greater fate of the universe. And so that's a fun one. I've got a lot of people that have asked me to write more in that series. You going to take them up on that? Tinkering with some outlines right now. It's like good software. You're never done writing it.

It's like, you know, good. No, they're always living documents to some extent. Awesome. We'll link up your Amazon in the show notes for people who want to check out those writings of yours. Yeah, they're affordable. They're fun reads. People seem to like them. So if you're interested in that kind of stuff, give it a shot. Well, Matt, thanks for coming on the show. Thanks for beating the drum on this new slash old, but still new style to build web apps, HTML over WebSockets, sharing your enthusiasm for this

and for getting more people to think about it. Like I said, everybody has to make their own choices and decisions. Everybody has their own context. It depends on the most popular phrase among software developers. And so this is another approach that may fit your needs. And so we appreciate you sharing it with us and talking through some of the details and the pros and the cons. It's been lots of fun. Yeah, thanks for having me. And I want to tell everybody again, if you're out there writing software and you're thinking about what technologies to choose,

choose to do stuff that lets you do awesome. Because then you can finish doing awesome, deliver that to people that it matters to, and then spend your time on your family and the things that you love instead of spending your time wrestling with obscure technical details. Well said. All right. Thanks. We'll talk to you all later. That's it for this episode of The Change Law. Thanks for tuning in. We'll talk to you all later. and launch darkly. When we need music, we summon the beat freak Breakmaster Cylinder. Huge thanks to Breakmaster for all their awesome work.

And last but not least, subscribe to our master feed at changelog.com slash master. Get all our podcasts in a single feed. That's it for this week. We'll see you next week. Thank you. Game on!