The Changelog: Software Development, Open Source - JSON API and API Design (Interview)

Episode Date: January 1, 2016

Yehuda Katz joined the show to talk about JSON.API — where the spec came from, who's involved, compliance, API design, the future, and more. We also finally got Yehuda on the show alone, so we were ...able to talk with him about his origins, how he got started as a programmer, and his thoughts on struggle vs aptitude.

Transcript
Discussion (0)
Starting point is 00:00:00 I'm Yehuda Katz, and you're listening to The Change Log. Welcome back, everyone. This is The Change Log, and I'm your host, Adam Stachowiak. This is episode 189, and what a show it is. The first show of the year, kicking it off with Yehuda Katz talking about JSON API. We talked about Yehuda's past. Because this is the first time he's been on the show solo. So we had to dive deep with him. We talked about JSON API. JSON API compliance.
Starting point is 00:00:34 Resource discovery. Supporting extensions. Future proofing API design. And so much more. We had three awesome sponsors. CodeShip. TopTile. And also DigitalOcean. Our first sponsor of the sponsors, CodeShip, TopTile, and also DigitalOcean. Our first sponsor
Starting point is 00:00:47 of the show is CodeShip. Head to resources.codeship.com. Once again, resources.codeship.com, and there you'll find all sorts of free resources from CodeShip, eBooks, webinars, email courses, guides, all talking about continuous delivery, Docker, containers, getting your code to production in a workflow that makes sense for you and your team. You get a couple of guides that are awesome. Why containers and Docker are the future, efficiency and development workflows. They have a couple of webinars coming up here in January this month in 2016. Continuous delivery and feature flagging and introduction to web apps with Docker.
Starting point is 00:01:23 So those are really awesome free webinars you can join. They also have this really awesome five-day email crash course on continuous delivery. Once again, head to resources.codeship.com. Enjoy all those free resources from Codeship. And now on to the show. Hey, everyone. We're here with Yehuda Katz. It is the first of the year, January 2016, so we're excited about that for one.
Starting point is 00:02:00 But Jared, we're here talking to Yehuda today, and we've had this show in particular on the docket for a while, but this recent tweet kicked it off. What was that tweet about? Yeah, that was Yehuda tweeting about JSON API, which is our conversation we'll be about today, where he says that you can think of JSONAPI.org as ASM.JSON, a low-level, consistent serialization format and protocol you can build on. So I'm sure we will tee that one up later in the show as a conversation point.
Starting point is 00:02:26 But that was kind of a thought that spurred us to say, hey, we never had to want to talk about JSON API back when it went. I think it was 1.0 was earlier this year. A few months ago, yeah. And so here we are. Here we are. We've had this, Yehuda, we've had this.
Starting point is 00:02:41 So everybody, you've heard Yehuda back there in the wings there. But we've had this in Yehuda, we've had this, so everybody, you've heard Yehuda back there in the wings there, but we've had this in our podcast Trello board for, I don't know, all year, basically, all of last year, technically, since we're actually recording this in 2015, published in 2016. So, you know, there you go. Hopefully we'll make it over the hump. There you go. We had Yehuda and Steve Klabnick on sometime last year talking about Rust, episode 151, which was, yeah, April. And I know at that time we definitely had right next to that one in the queue was JSON API just waiting.
Starting point is 00:03:15 So it's definitely been a while. And Steve also works on JSON API. So, interestingly. So obviously that's the main topic of the show. I'm sure we'll talk a little bit about Ember. We'll talk a little bit how those two things play in. And I'm sure you got some, we have a couple of questions in the wings too about JSON API and Rails 5 and playing nice together and stuff like that, which I'm sure is what you can talk deeply about. But something recently that we've done with our guests,
Starting point is 00:03:45 especially someone like you, you've been on the show probably four or five times. Jared, I didn't even go back and count, honestly. Yeah, four. Four times? So we'll put all the links in the show notes to those past shows that Yehuda's been on, but Yehuda, we haven't had you on the show by yourself,
Starting point is 00:03:59 so I don't know how you feel about that. You always come with some sort of guest or some sort of wingman, so to speak. What do you think about that? Yeah, I mean, the reason for that is that pretty much I only like to work on open source projects with other people because I don't honestly, I just don't trust myself that much. And so there's always at least one other person who I'm like, well, whatever I say, the other person will have another perspective that will be useful. And I think if you like if you listen to me talking with almost any other person, there's usually valuable insights that I would have just missed. So
Starting point is 00:04:28 that's why I like to do that. It's just my style. Jared, should we kick off the show with the mention from Brian Lyles? I know I got a couple questions as well, but that was an interesting takeaway from Beyond Code up in GopherCon. Yeah. So I think, you know, we're gonna spend a little spend a little time getting to know you from a historic perspective. I think we all have our own interesting roots, and it's fun to find out more about the people behind the awesome open source software that we all use and love. shoot at conference after parties. And season three was at GopherCon, which is still TBD on the release date on that one. But during that, we had Brian Lyles on Beyond Code. And we do ask everybody in that series who their programming hero was. And he mentioned you.
Starting point is 00:05:17 And so I actually pulled the quote out because I think he gave some interesting reasons as to why. And he also seems to be a friend of yours because he kind of takes a couple cheap shots at the end there. So I was going to read this to you and have you kind of just react to it. So he said, Yehuda Katz is a programming hero. He says, the reason why is because I remember
Starting point is 00:05:35 when Yehuda first started writing code and he wasn't good. And then he started writing more code and he wasn't good. But then he started writing even more code. And now look at him. Not only has he mastered Ruby, he mastered JavaScript. Now he runs his own company. And he's like this tall. And he kind of looks his hand out. And he's kind of weird looking. Yes, I'm very short. But he's an amazing person to be around. So high praise from Brian along with
Starting point is 00:06:01 a couple of cheap shots. But could you just react to that? Yeah, I think that's all accurate. I'm short. That is true. I think I'm a pretty good programmer these days. I also was a very bad programmer at the beginning, like literally every other human being who ever starts programming. And I also think that the process of persevering through struggling with code is a thing that made me better. And I think that I try to tell that to every new person I meet who's struggling is like basically everyone. That's everyone's story. Everybody struggles with code. And if you think, oh, that person's cool, they must have had it must have been easy.
Starting point is 00:06:41 That's just like it's just like a mythos. It's not a real thing. I mean, maybe there are some people, but I don't know them. That's a good dovetail right into that tweet stream, Jerry, we have mentioned there and Yehuda, you probably know this because it was recent, but you'd mentioned the gulf between using programming to automate some repetitive tasks and being able to work on a Linux kernel.
Starting point is 00:07:01 And you kind of dove a little bit deeper into struggle versus aptitude. Can you speak to those tweets? We have some links. We'll put the show notes, but I'm sure you can recall that. Yeah, I remember. I've been doing a few. I actually haven't blogged a lot recently because I sort of came to the conclusion that almost anything I have to say could be constructed as like a series of at most 10 tweets.
Starting point is 00:07:23 So there you go. So, yeah, I guess. 1400 characters, yeah. Yeah, confirmed. So I guess what I was saying in that, in that tweets stream is that there's, I've met a lot of people over my career who have tried to program.
Starting point is 00:07:39 Some people end up being programmers. Other people end up, you know, getting to have some aptitude with programming, but not necessarily being quote unquote programmers. And I think it's pretty damaging that we get, we give off this sense of programming as like a wizardry level thing. Like you have to be a wizard to be useful at all. And I often like to tell people like so, so many people are very productive in Excel.
Starting point is 00:08:06 And Excel is a very simple programming language. Obviously, it's very simple. But there are functions and variables and some of the more abstract concepts that you need to do to be a programmer are already present. There's, like, some amount of variables and things like that. And I like to think that if you could program in Excel, that's probably not the end of the story. You could probably do a little bit of scripting. Um, and, uh, one of the things I like to think about is, uh, when I, one of the first jobs that Leah had, um, who I work with now at, at Tilda and I am married to, uh, I met her in college.
Starting point is 00:08:41 Uh, but one of the first jobs that she did, um did was a marketing job at a company. And she was basically told, like, every single day you're going to get an email from one automated system. And what you're going to do with it is you're going to open it up in Excel, make some changes to it, and then email it to another automated system. She came home and told me this story. And I said, first of all, can you find out, like, why the person in the first automated system can't just email the second automated system and change the format? And so she came back like the next day and she said, I asked them and they told me like, they can do it, but it will be like six months to schedule it or whatever. And I was like, okay, that seems terrible. But then I said, well, why don't we just use Ruby to pull the mail in the mailbox,
Starting point is 00:09:26 get the CSV file, use the CSV library to make the translation, and then just use the Ruby 132 API to send an email to the second system. And we wrote that code. And that saved her some non-trivial amount of time every day. Every day she was supposed to do this very annoying repetitive task, and we were able to script it away. And I guess I just think that there is a lot more of that. I think like Ruby and Perl and Bash are scripting languages. And really what that originally meant was like you could use them to write little scripts, like little 20 line scripts.
Starting point is 00:10:00 Actually, another example of this is every year Tilda sends out like a compensation report that says what your bonus is and things like that. And also me and Leah wrote a thing that it's just like a giant, it's like a 50 line file that sets up a bunch of variables and then evals and ERB basically. And every year we go in and we make some changes. We load a CSV file that she does in Excel, right? And these are all tasks that like a lot of people have. They have nothing to do with programming per se, but they're very good candidates for automation. And there's no real reason why we should tell people, like, if you can't see yourself being a full-time professional Rails developer, you can't, you shouldn't even think about
Starting point is 00:10:37 programming. Programming, there's a big slope between, you know, not knowing anything, knowing Excel, knowing a little bit of scripting, and then working your way up. And I think a lot of people who became programmers actually entered through that path. And I think we should be more happy about it, about the existence of it. I mean, I think I just want to give my amen to that. I think automation is kind of a gateway drug into more formal programming things. I know when I first started in college, I was studying MIT Management Information. Not MIT.
Starting point is 00:11:13 I was not at MIT. I was studying MIS, Management Information Systems. And I never considered myself a programmer or developer. I just thought of myself as someone who likes to automate things. And I was using Perl and just scripting little things away. And when you see the benefit of that, especially when it saves you five minutes a day and it can save you somebody half an hour a day, that's something that you think, hmm, I can take this to the next level, and you start having new ideas.
Starting point is 00:11:40 And before you know it, you have an app. Yeah, exactly. Eventually, you have a big thing you need to make. As long as you're not, I mean, as long as you're not convinced that the thing you have done is divorce from programming, which when we get to my story, eventually I will still explain the connection. Yeah. Well, let's get to that because you, you said that you were writing programs when you were
Starting point is 00:11:57 eight years old, but you were convinced that you didn't have the aptitude for it. Can you tell us that story? Yeah. So I guess it's pretty connected to this other thing, which is, so I was a kid. My father had done submarine simulators for the Air Force, like in the 60s. So, and he, I would say that that was not considered
Starting point is 00:12:15 like a programming job. It was more of like a hardware work with your hands, but like, you know, building a computer or anything like that involves a certain amount of technical acumen. And so my father in the 80s, which is after I was born, I'm very old, I know. My father in the 80s had his own hardware shop where he would basically repair people's computers. And because of that, when I was born in 82, I already had a computer.
Starting point is 00:12:46 I had an 8088, and that was the computer. We didn't have so much money, so that was the computer we had for some number of years, well after its theoretical shelf life. And I don't know if people remember this, but DOS, like some middle versions of DOS started coming with QBasic. So there was GWBasic, which was like this thing where you have to type the line numbers. So you'd be like 10, print whatever, 20, print whatever. And then you always would like, the rule was that you would always leave
Starting point is 00:13:15 some amount of space between your line numbers in case you want to go back later and add more things. So if you're like, oh, I want to put something in between the first line and the second line, you better have enough numbers left so you can go 15. And like, if you became like a serious business GW Basic programmer, you would do like 100 or 1,000. So you always would have enough space to put stuff into.
Starting point is 00:13:34 That's funny. Yep. So GW Basic was a thing. But I would say because of the line number thing, it was not real. It felt, and there was no, you couldn't like really, most GW Basic environments were like a REPL there. You couldn't really save it. It was, or it was no, you couldn't like really, most GW Basic environments were like a REPL. You couldn't really save it.
Starting point is 00:13:47 It was, or it was hard to save it and load it back again. So GW Basic was more of like a thing people would get a, if you had a Commodore 64, they would like get some magazine and they would type in the game and then they would play the game. Right. That was, that's sort of like the mythos of GW Basic. But QBasic was uh stands for quick basic came with dos and you could like write your file and save your file and you have five and you
Starting point is 00:14:11 were now in like a full screen thing that ran your program so you could and it had like so it didn't have a lot of control structures but it had go to which is actually sufficient to do most things and it also had like a few graphics routines you you could make circles and squares. And I would say it was probably at a similar level of abstraction as like canvas, the canvas tag, you could draw some basic shapes. And if you want to draw anything complicated, you had to animate by wiping the screen and writing drawing the next screen. But as canvas shows that is, in fact, sufficient to do all the things. And I guess when I was eight years old, I found this fascinating. And I made like some very basic games. The one I remember the most was called
Starting point is 00:14:50 Carnival. And it was like the most the thing I was most proud of was I had a game where like something would move back and forth along the bottom and you had to like hit space bar to get it to go into like a basketball hoop that was painted at the other side of the screen. And like my family would play it so i in retrospect the fact that i was using go to and like effectively canvas primitives to draw a animated graph game when i was eight seems like probably a good indication that i know how to program but at the time i was like okay well q basic has first of all it's interpreted so there i didn't know that but i knew you have to hit f5 and it was annoying. I didn't know how to make the game not require
Starting point is 00:15:29 that. Although eventually I discovered that there was like a QBasic compiler, but I didn't really know how to reliably have access to it. And I was like, oh, I want to learn more things. This seems like a little, it's like pretty limited. And somebody gave me a book on C, which was like, oh, I want to learn more things. This seems like it's pretty limited. And somebody gave me a book on C, which was like, oh, here's the next step. After QBasic, what you should do is learn C. And I was like, I literally have no idea anything that's going on here. But what I've been told is
Starting point is 00:15:55 QBasic is a toy programming language. C is like the real programming language. And obviously, since I have no idea what's going on with the C book, which I think the book I was given was the KNRC book, which pretty much is not an appropriate book for a small child. I was I basically just came to the conclusion in my head that like, oh, I can I'm good enough to like program toy things. But I basically do not have the aptitude to program anything serious. There was no like good bridge between the stuff that I already knew how to program and quote unquote real programming. So at that point I gave up programming for like a while. Um, and
Starting point is 00:16:30 then I went back, I would say like in my teens, I programmed again, and this is like embarrassing, but the thing I was programming was a star day calculator for Star Trek. Um, I've like found online. I was online pretty early I found online like a fact that someone had done all the work to reverse engineer all the stardate algorithm for every single era of Star Trek and I was like oh that seems cool
Starting point is 00:16:55 let me like make software out of it and I use Visual Basic 6 which a lot of people still have fond memories of and Visual Basic 6 is basically like a WYSIWYG programming environment, but you can double-click on any
Starting point is 00:17:09 part of any UI element, and then you can say, effectively attach an event handler to that UI element by just visually, right? So you have to write the code to say what it does, but you don't have to do the work of wiring up the event handler to the visual element and the actual visual work is done using a
Starting point is 00:17:31 UI, right? So you could sort of think of it as interface builder, but in interface builder, you still have to do like write the code in the text part and then you have to like drag the arrow or drag the line from the UI element to the text part, and then you have to like, draw, drag the arrow, or drag the line from the UI element to the text part. Whereas in Visual Basic, you would write do the UI part and then instead of having to like, also write the text part, you would just double click on it and it would effectively scaffold out the part that interface builder makes you do
Starting point is 00:17:59 manually. So it was it was very, I would say it was very easy to use. And I was able to like, the algorithm, as you can imagine, is not very complicated. So I was able to write it in Visual Basic. And at that point, it felt pretty similar to my QBasic time, right? It was like, it's pretty clear that it's like a toy environment. But now the toy environment has gotten better. So I can do, like, I can basically get more done with the amount of work that I already, with the simple programming that I already know how to do.
Starting point is 00:18:31 But then equivalently, I eventually was like, this seems pretty annoying. I want to do more things. And what I was told at the time was like, well, the next step is to learn the Win32 API. And I went to look at the Win32 API and if anybody wants to look at it, it is horrifically terrible.
Starting point is 00:18:46 Well, it's not terrible. It's just equivalently very low level. Very dense. Hard to understand. Similar to your C, to the experience you had in your A with C compared to GW. Exactly. And the Win32 API is basically like you have this top-level switch statement, and every action that happens is basically running,
Starting point is 00:19:05 like telling that switch statement what to do. And it has to do low level things like drawing boxes. And it's just, it's basically a massive loop between, uh, like the nice environment, visual basic and the quote unquote serious programming that you do with one 32. So again, when in my teens, I was like, well, that's, that was fun and all, but it doesn't really, I don't seem to have the aptitude. I'm no longer eight years old. Basically the same thing is happening. So I'm pretty sure I just should not do this.
Starting point is 00:19:33 So I guess when did the professional time happen for you then? Was it a struggle for you? I know you talked about struggle and aptitude there. And you said that the idea of struggle doesn't mean that you don't have the aptitude. It's anybody who's learning the program is going to have some sort of struggle. That's a good thing. So what was the, how did things progress for you to get into professional, I guess, programming? So what basically happened is I just gave up. I was like, okay, I like basically know the equivalent of like very, very light scripting, but I can't really program. So
Starting point is 00:20:02 I, and in college I could have been a CS major, but I was like, it doesn't really seem like it's for me. And I ended up doing a bunch of other stuff, accounting and journalism. But journalism in college ended up teaching me design. So I learned, I basically did a newspaper design in college. And then I got my first job out of college as a web designer from a person who didn't really understand that web design and print design are not the same thing. And I also didn't know that at the time, as it turns out. So I was basically given a job to do what I thought was a like pretty straightforward design task that ended up actually being a programming job. And I was like, oh shit, if I want to keep my job, I actually need to learn programming. So I had no choice. And I was, the the way the reason i ended up succeeding was because i
Starting point is 00:20:45 was given a couple of applications that people ever had already written um in cold fusion and php and i was told like please update this application for this year so it was a non-profit and the non-profit had like these dinners and they were like okay we will we basically have an application that lets you pay for your dinner ticket and perhaps ask for additional items. It was like not totally trivial. And they were like, please update it for the changes that we want to make for this year. So, of course, I was able to go in there. And either in ColdFusion or PHP, I was able to make the changes because that kind of thing is not so complicated.
Starting point is 00:21:18 But often when I was doing that, I was like, oh, well, it seems annoying that you have this giant form. And like every time you click next, you have to go back to the server. So maybe I could find a way to make it more interactive. Again, I was like thinking about it as a designer. And I was like, this was like right around the time Ajax had become a thing. I was like, oh, there's this Ajax thing. I should probably learn that. Maybe that will help me do it. I was like very enamored at the time of like the idea of constructing forms that progressively revealed themselves based on the previous information. And the thing that I really was most angry about that still annoys me is the fact that you would have to put in your city, state and zip code a lot of
Starting point is 00:21:56 times. And I was like, if you put in your zip code, you can derive the city and state. Why are you making me type all these three things? And I basically didn't really know how to do it. But I was like, I'm pretty confident that this is not actually necessary was that the first library you wrote then um i wouldn't say that it was a library but i figured out how to like download at the time there was no libraries really for any of the languages i was writing but i was able to like download an access database of the city state and zips and i was able to figure out how to get it to reveal its secrets to me um and not force me to do the thing and then i learned ajax so i didn't have to reload the page and things like that so that was sort of how i got started with programming
Starting point is 00:22:34 um but then i was like well i can't like learning ajax quote unquote ajax all by myself seems very hard i'm i the internet was not so good at the time. There was experts exchange was like the most popular thing. And it was like constantly asking me to pay. I just didn't feel like I was getting a lot of traction. So I took this class by Thomas Fuchs, which is like about Ajax. And I effectively, by the time I took that course, I knew most of the things that was in the course. It was actually a course on prototype. But at the very end of the course, he was like, by the way, if you want this to be easier, you could use Rails. Like Rails will be easier for you. So you might want, I didn't want to force you to use Rails in this course, but probably be a good thing to investigate.
Starting point is 00:23:14 And I don't know how many people actually did in that course, but I was like, well, if it's easier, sounds better to me. So I'll investigate that. And then I did. And I would say that was basically the beginning of me being an actual programmer. I was like, oh, well, now I was able to basically build a CMS system for that company without really knowing how to program, which I don't know how true that is of Rails today. I think Rails has become a trickier starting target. But at the time, there weren't even migrations. You just go into a WYSIWYG editor, change your database schema.
Starting point is 00:23:44 You'd make whatever changes. And then it was just like you have to know HTML and like a very tiny bit of Ruby to do things. And I was and I sort of knew Access. So I had like the right proximate mental model for it. And I was I was able to do that. And that basically made me feel like massively empowered. Like in the past, every time I had done this toy thing, it was like basically not a real application. I wasn't getting paid for it. And it was like, whether or not,
Starting point is 00:24:09 you know, the CMS I built was like serious programming or not. The fact of the matter is somebody who's running a real business. Right. You built something real. Somebody used it. They put it in production. They made money from it. Yep. And so that's awesome. So that basically made me feel like, well, whether like I don't, I didn't, it didn't make me have to think a lot about whether or not the programming I was writing was real or not, because it was being used for something real. Does that tailor into your opening the open source box, so to speak? I mean, you're pretty prolific in open source now. It sounds like prototype Thomas and Rails was sort of an entry point to you.
Starting point is 00:24:44 What was the very first thing you did in open source that started to take you down the road you've been going down? So the first thing actually was not rails. The first, uh, I basically learned rails and jQuery at the same time. Um, the, the, I, I found jQuery because I was looking, there was some, uh, something that would like have been the equivalent of hacker news, but I can never remember. It might've been like, um, it might've been slashed off, but I actually don't remember it being slashed out.
Starting point is 00:25:08 So I don't really know. There was some website that was like, you should check out this UI. You can like, you can construct a thing that is like the finder that lets you select icons from a grid. And I was like, wow,
Starting point is 00:25:18 that's pretty cool. I didn't know you could do that on the web. And I looked at it and it was built using this library called interface, which was built on top of jQuery at the time. It's like the precursor to jQuery UI. And I was like, wow, jQuery can do that. That seems pretty cool. So I looked more into jQuery. And the problem I had at the time was that jQuery, while very cool, was basically documented by John's writing documentation on the wiki at his leisure, which as you can imagine, is not very likely to be up to date, especially
Starting point is 00:25:46 since jQuery itself was pre 1.0. It was like, not only was it pre 1.0, it just had no versions. Like every so often, John would like put a tarball or a zip file on his blog, like here's a new version of jQuery. And I was like, I really think we could do better than this. What if instead of you writing things on the wiki, what if you gave me at the time, like this was 2005, so don't laugh. But what if you gave me an XML dump? Like what if you gave me an XML dump of the documentation? There already were like tools that would automatically generate inline docs. If you give me the XML dump, I will basically do the work to make a like automatic website.
Starting point is 00:26:21 And I used XSLT at the time to like convert it into something visually appealing. And so I was basically, I like kind of owned the documentation effort for a little while in jQuery, just because there was no one else doing it. And I was, I was feeling like, I guess I know, I have no idea how I, why I felt like this, but I felt like this should be easier. There should be basically everything is in place. Someone just has to like glue it all together. And then if you glue it all together, then you have automatically up-to-date documentation for every version of jQuery ever, just because someone wrote inline docs. And I felt like that was cool. So I would say that that was probably the first real open source thing I did. And the reason, like the gateway drug for me was just, that thing was
Starting point is 00:27:03 massively high leverage right like before people would come and they would look at jQuery and they would bounce right off because they couldn't figure out what the what the rules were but then as soon as there was any documentation on the internet all of a sudden tons of people stopped bouncing off of it yeah I can recall those days actually I mean I remember just there was a couple other different things happening around then obviously product that was around but it felt as as someone who came from a front-end perspective which you could probably lament a little bit about this with an html css background sass wasn't even invented yet or it might have just been invented but it wasn't it was only a ruby you know very different than it is today it you know there was no easy
Starting point is 00:27:39 path that i could find at the time into this javascript world and uh but i remember when jquery had docs, it was actually quite a bit easier to get into it. And I was like, okay, now I feel like I can actually do something with this. There was this library called Rico at the time, which I actually was like, Rico actually seems pretty cool. I want to use it. Like the demos are really good.
Starting point is 00:27:58 And every single time I would try to use it, I was like, I literally have no idea how you're, it seems cool and I can look at the source, but I do not know how you're supposed to actually use this. And I bounced off of it a bunch of times. And I guess that was sort of the epiphany that I had as a person in the world. Like if there are things that are so close, like think about it. I basically didn't know how to program at all at this point.
Starting point is 00:28:19 I would say that I effectively had like the level of programming that you have when you're writing Excel at this point. But I was able to figure out how to take documentation that other people had already written and tools that other people had already written to generate the XML version and then XSLT, which was pretty declarative and didn't require a lot of programming skill. And I could take all that and put it together into a thing that enabled a ton of people to use jQuery. And I felt like, wow, that is massively high leverage. You can change a lot of people's lives easily, quickly with open source.
Starting point is 00:28:51 And that's sort of been my guiding worldview ever since. How do you take people who basically don't necessarily, who are bouncing off of things, and how do you enable them to do more? It's rough. It's an interesting thing too. Hearing your background on automation and it seems like you have a true passion
Starting point is 00:29:13 slash addiction with it. I'm not really sure how to phrase that, but it totally makes sense that you wrote Thor. Yeah. Yeah, Thor was definitely, I wrote Thor like after writing a bunch of CLIs and saying, why is this? I guess one of my things about open source in general is there's so many things that are like so close.
Starting point is 00:29:32 Like somebody just has to, the difference between where we are now, which is like almost nobody can use it. And it being usable by a huge amount of people is often such a tiny gap. Someone just has to spend the time. And so many people just like, there's such a failure of imagination so often about what the gap looks like. So a lot of times people will try something and they'll have that experience for some reason or other. And they'll just say, this entire idea is a bad idea. This entire idea is dumb. And really all that is needed to make it feel better is like small things that me 10 years ago could have done. Right.
Starting point is 00:30:06 Someone has to just do it. So one more thing we have to mention, obviously, we can go on and on. You have a deep past with programming and open source. And obviously, we'd love to hear all about your past. One thing I want to talk about before we go into our first break is what I'm calling, I think might be your brand purpose, your personal brand purpose. And at November this year,
Starting point is 00:30:30 I think it was either this year or last year, I'm not sure which was, but you said sometime during your talk, and I took this note down, I don't think you made it explicit that this was your brand purpose, but I want to ask you what you think about this before we transition into this break.
Starting point is 00:30:42 But you said build things that empower people. And it seems like, you know know rewinding back to your beginning uh through to your initial comfortable feeling of being a programmer and then moving into open source like you've been doing that that's been your mantra that's been what you do what do you think about building things that empower people i think that's i would agree i also agree that I was it was this year. I also agree it was not so explicit as a like me waving the flag around it. But I I think that open source does is does a lot of things for a lot of people. And I think it's good. But for me, the main thing to do with open source is to find places where the gap between someone being empowered and not being empowered is small and just closing it. And unfortunately, at this point in my career, the gaps that are still a good cost benefit for me are usually much bigger, but they're still, like, I'm never going to be,
Starting point is 00:31:36 I mean, maybe someday, but right now I'm never going to invent the Rust programming language because it's like the level of low levelness is too much for me. But the gap between Rust being a good thing for servo, like a web browser and being good for Ruby programmers is actually much smaller than the gap between like,
Starting point is 00:31:55 like not having Rust in the world and having it at all. And so I, a lot of what I try to do in my, like with my open source work is to try to find things where either, like the fact that something hasn't been invented yet is for a stupid reason, like Thor or Ember, or, um, there's something that exists in the world, but it's just inaccessible for some reason, like the jQuery doc story or rust, like thinking about, okay, what, what can we do to make this a thing that a lot of people that a lot more people can use to do something
Starting point is 00:32:26 powerful? And I just think not enough people think about it. And more concerningly, and this was also in my November talk, a lot of people are just pretty cynical about the whole thing. A lot of people are pretty cynical about the level of complexity in programming. A lot of people are pretty cynical about how likely you are to find a good solution for these things. A lot of people are pretty cynical about how likely you are to find a good solution for these things. A lot of people are pretty cynical about the power of these things in the first place to change the nature of what people can do. And I'm very optimistic about being able to change the nature of what a particular person can do at a particular time. But that requires believing in it. And I think, unfortunately, a lot of people don't always believe in it. Well, I think on the other side of the break, we want to take
Starting point is 00:33:14 these thoughts, the empowering people and the finding the gaps between empowerment and non-empowerment, and use those to cast a light on your work with JSON API. So we'll ask you about that on the other side. We'll be right back. We know you listen to other podcasts. Don't worry. We're not upset. We know the Change Log isn't the only podcast in your list. And you know what?
Starting point is 00:33:36 You may have heard advertisements on other shows about other hiring platforms and other places like our friends at TopTow. But let me tell you, there is no match to how invested TopTow is to both sides of the equation. You have companies out there who need great engineers, great designers, and you have great designers and great engineers out there needing great opportunities. And TopTow plays both sides of that fence very well, helping make sure the right kind of opportunities are there and the right kind of developers and designers are there. And if you're a CTO listening to this or someone on a team who knows you need to expand and add more people to help reach your goals, TopTow is the best place to go and find the best designers and those best engineers or best designers looking for great places to do great work and freelance and travel the world and have a lot of freedom, but also have the same kind of support you would want from working somewhere, TopTal is that place. You can blog with them.
Starting point is 00:34:35 You can travel the world with them. They have meetups all around the world. They absolutely love to encourage and to support and help their developers and the designers that are in their network all around the world grow and be better and do better head to toptal.com to find out more that's t-o-p-t-a-l.com once again t-o-p-t-a-l.com but if you want a personal introduction on either side of the fence whether it's an introduction to find the best designers and developers, or if you're a designer developer looking for great opportunities, get in touch with me. I'd be glad to give you a personal introduction. Email me at adam at changelog.com. And now back to the show. All right, we are back talking with Yehuda Katz about JSON API. And before the break, we laid out, Yehuda, how you just try to do two things. The main purpose is to empower people with open source.
Starting point is 00:35:32 And kind of the way you go about that is by finding these gaps between empowerment and non-empowerment and hopefully find ones that are easy to bridge. Probably they're getting harder and harder to bridge as you get more and more advanced. And we wanted to take those thoughts and apply them to your work behind JSON API, because this is something you've been involved in since, I think, May of 2013. So can you cast that light into JSON API and why it's something that you think needs to exist? Sure. So before I do that, I think it's important to understand, like, what does empowerment actually mean? And what I would say, one thing that I think it's important to keep in mind is that if somebody already knows how to do something, making it ways in which that kind of thing can be empowering. But the most empowering things are taking something that somebody does not know how to do at all, and then making them be able to do it. And fundamentally, in order to do that, you have to use abstractions that hide some of the details. Because pretty much nobody is going
Starting point is 00:36:43 to say like, oh, I would like to build web applications. So before I do that, let me learn HTTP and RAC and all this stuff, right? They're going to, they want to, the shortest path to empowering them to be successful, like I was when I started building that CMS for my company, is to hide a lot of those details. And this is actually a fundamental conflict in open source and software zeitgeist is whether abstracting away details is a good idea. So, for example, Joel Spolsky has this famous blog post from many years ago called The Law of Leaky Abstractions.
Starting point is 00:37:19 And the idea behind this blog post is like you can never successfully abstract anything. It will always leak. It will always leak. So don't even bother. Everybody should just learn everything. I think this is just a silly worldview in part because we have so – over the past 30 to 50 years, we have succeeded at abstracting so many details that it just seems like self-evidently incorrect. Like whether or not TCP, which was his example in his blog post, one of them was TCP, which I think is a hilarious example. The fact that TCP might occasionally leak doesn't necessarily mean that TCP is a bad abstraction. It just means that sometimes,
Starting point is 00:37:55 if you're doing sufficiently advanced things, you will need to learn TCP to get the sufficiently advanced things done. But that doesn't mean that we should not have TCP and everyone should just use UDP and build their own thing. Although there's a good website called notcp.io, which is basically making a joke about this entire phenomenon of people not believing in abstractions. And I guess for me,
Starting point is 00:38:18 if you're looking to empower somebody, you have to figure out what parts of the everyday, the 80-20 job that someone does, what parts of it are about things that are not very important. And interestingly, what I've also found, what I found is that when you do that analysis, quite often, it's a lot of the thing, right? It's a lot of the stuff people do, like on a day-to-day basis, if you look at like what someone does when they write a backbone application, a huge amount, a huge percentage of that is something that you, that Ember doesn't make you think about, or even React
Starting point is 00:38:49 or Angular doesn't make you think about, right? And I think, so I think when you're first looking at empowering people, what you often discover is that's sort of the natural zeitgeist, the natural worldview of how people think about whatever task it is that they're setting out to do. And what people think is like core and fundamental to the tasks they're trying to do is really pretty abstractable. I think if you're if you're going down this path it's important to remember that when you abstract something it's you have to leave escape valve so people can go and break out into the into the level below if you're an advanced user and you discover things are too limiting
Starting point is 00:39:24 and I think Rails did a pretty good job at this and Ember does it with things into the level below if you're an advanced user and you discover things are too limiting. And I think Rails did a pretty good job at this. And Ember does it with things like didInsertElement, which allows you to basically write jQuery code in the middle of your component, right? And React has a similar hook. And I think those kinds of things are very important. But if you make your escape valves reasonably well-designed and designed to be very, very broad, what ends up happening is that people who are new can get a lot done without having to think about the escape valves, but people who are
Starting point is 00:39:49 a little bit more advanced are able to use the escape valves to get the job done. So I guess this is how I think about empowerment in general, is that your job is to take away a large percentage of the thing that people think is core to the experience of doing it. And I've been giving talks about what became JSON API for a long time. I gave a talk just about this in 2012 at RailsConf, which I think was called Rails the Next Five Years or something like that. And with JSON APIs, I think there's so many details about it that have to do with format and they have to do with protocol. But somewhat interestingly, the format and the protocol cannot easily be decoupled.
Starting point is 00:40:31 So if you say, well, you know, we're going to make a format. We're going to, the way we're going to make it, we're going to empower people to build JSON APIs more effectively without as much hassle. And we will allow many, many more people to build JSON APIs than can do it now. If you think that you're going to do that with a format, the problem is that the format interacts in non-trivial ways with the protocol. If you do the opposite, if you say, we're going to do the protocol, which usually people are referring to when they talk about REST, they say, we're going to say, we're going to define the protocol. The problem is the protocol interacts in non-trivial ways with the format, right? And additionally, a lot of times people will say things like, oh, just use REST.
Starting point is 00:41:08 But they don't actually spend any time describing what it is. It is assumed that if you're a programmer, you can figure it out. And number one, a lot of the people we're trying to empower in the first place cannot. But number two, the actual definition of something like REST is sufficiently ill-defined that it doesn't help even more advanced developers who say use REST, they have no idea what you're
Starting point is 00:41:30 talking about. And I think a good example of this whole attempt failing was active resource in Rails where DHH basically thought, and a lot of people thought, like, oh, well, like Rails is pretty conventional. So what we'll do is we'll say you can connect to a quote-unquote conventional Rails server. And actually, I made the same mistake in the beginning of Ember Data that led me down this path, is that Rails is sufficiently conventional that you should just be able to connect to literally any Rails server and get, like, roughly enough information to do the job. The problem is that it turns out that the specification of sufficiently similar
Starting point is 00:42:05 Rails application is impossible. And people do all kinds of random stuff when they're building Rails servers. So we also, because the same reason that caused DHH to make the mistake about ActiveResource caused us to make a similar mistake in the beginning of Ember Data. We said, you know, what Ember Data will do is it will hard code to quote unquote Rails. And of course, that doesn't stop you from, I was overly, I made too strong of a statement there. It doesn't stop you from using the same conventions in any other system. So, you know, there's like the REST framework for Django and Node, of course, lets you do whatever you want. So it was more like, let's say that the Rails conventions are the Ember Data conventions, and we thought that that would get us all the way. But in practice, what ended up happening is that
Starting point is 00:42:48 questions came up, like, should I have a root or not? Should the name of my resource be in the root or not? That's a simple one. How do I include associated resources? What if I want to include just the links to the associated resources or IDs for them, and then lazily load the rest ones. And these things come up very quickly, like within a very short amount of time. It doesn't require you to be an advanced user to encounter some of these questions. It just involves you trying to do anything, even pretty simple with your UI. So we ended up realizing pretty quickly by 2013 that treating Rails as a spec was not working. And so at RailsConf that year, I was bemoaning this. I give it a couple of talks about it. And Brian Ford from Rubinius, his name is now Brian, I don't know how to pronounce his last name, but it's
Starting point is 00:43:42 like something like Sheree, came over to me and he said, like, the world needs to have a better specification. Could you write one down? And I said, well, what I'll try to do is I'll try to extract the implicit specification that Ember Data had. Like, basically, even though Rails has no spec, Ember Data has to have said something. So Ember Data has implicit spec. Let me try to write it down. And so I wrote down the first version of JSON API was just effectively an extracted Ember Data spec.
Starting point is 00:44:10 And actually, people were pretty excited about it. But one thing that we noticed after that was that even Ember Data was pretty wishy-washy about a whole bunch of details because of the fact that we were trying to work against a pretty wishy-washy server. So over time, what we realized was that we needed to get pretty rigorous. We needed to have opinions about all the details. And I actually learned something important about JSON APIs and wishy-washiness, which is that a big part of the reason it turns out that people are so wishy-washy in the way they define JSON API is that JSON APIs are maximal bike-shedding opportunity. So basically everyone looks at a particular JSON API and they
Starting point is 00:44:52 have a visceral response to the aesthetics of it. So if you are trying to get a lot of adoption for your JSON API description in some way, you're trying to figure out like, okay, how do I make something that a lot of people will not respond negatively to? And that basically means being pretty wishy-washy. You say, oh, you could do this, or you could do that. If you don't like this, you can do this other thing. But the problem with the head is that now the clients have to be willing to accept a lot of different things, which makes it harder to write clients. And what we discovered somewhere in the middle of the JSON API journey is that no matter how much people claim that they're willing to adopt something because it has the right aesthetics,
Starting point is 00:45:32 the real value prop of something like JSON API is having good servers and clients. So we were getting a lot of people to not whine a lot about the exact aesthetics of JSON API by being pretty wishy-washy. But then every person who was actually trying to write clients would say, you know, the spec seems fine, but I don't know what I'm supposed to do here. You know, the server says you're allowed to use whatever status code you want here.
Starting point is 00:45:54 Is that actually, like, what does that mean? What if the user gives me, like, a 422? What should I do? Yeah. Let me just real quick state something that's perhaps obvious to you and I, but just to be super clear with everybody, that the JSON API, and I'm using the word just to limit its scope, not its usefulness, but it's JSON API, the project that you've been working on that we're talking about. It is just a specification of how you ought to build JSON APIs, correct?
Starting point is 00:46:26 Yes, confirmed. So just to be clear, it's the only specification that I know of that tells you how to build JSON APIs, both the format and the protocol. In other words, it tells you the HTTP semantics, but it also tells you what the shape of the inputs and outputs is. Of the data. Okay.
Starting point is 00:46:40 And as far as I know, I'm probably wrong. There's probably some other thing that exists in the world. But as far as I know, I'm probably wrong. There's probably some other thing that exists in the world. But as far as I know, the popular alternatives that people say, why, you know, how or something like that, those things are all either about a protocol or they are about a format and not both. OK. But yes, I agree. In retrospect, probably the name JSON API has some confusing ambiguity here. And I apologize in retrospect, in hindsight. It's very Google-able, though.
Starting point is 00:47:04 I mean, you can Google it and find it. So that's a good thing. Yes. I mean, perhaps surprisingly, but it is true that it is very Google-able. So anyway, so we basically went through this back and forth where at first we were pretty responsive to the aesthetics people. And eventually we realized that the aesthetics people were not actually adopting JSON API, but the tool builders were important for the adoption of JSON API. And they were telling us about all these issues. So about a year ago, me and Dan Gebhardt, who was probably the main, like the main editor now of the spec, sat down and in a few face-to-face sessions, we hashed out what ambiguities people
Starting point is 00:47:40 had reported that were making it hard to build tools. And we just made calls about all of them. We just said, the answer is this, the answer is that, the answer is X. And the consequence of doing that is that now JSON API is very trollable from an aesthetics perspective. It's very easy for someone to look at it and say, I don't like that. A very common thing people say is, is this WSDL all over again? Actually, it is not WSDL because it does not have a big schema type thing and has nothing
Starting point is 00:48:03 to do with that. But it's very easy to look at it and say, you know, I thought the point of JSON was to make this a very like human consumable format, but now you have like this attributes key that has a hash of attributes. Why do you need that? Well, it turns out that the reason you need that
Starting point is 00:48:16 is that tools need to know the difference between attributes and associations. And saying that you'll do that by hard coding in every client is not very good for tools, right? So there are some things that we did to try to make ambiguities less. We tried very, very hard when we made the calls to not gratuitously add things for tools that just, you know, that basically would make it a binary format or something like the ugly XML formats that people are used to seeing. But at the end of the day, there really are some ambiguities
Starting point is 00:48:47 that you would like to get out in the spec. And that's sort of why I tweeted that as in.json thing, was that JSON API is not really fundamentally about being a, it's not about being a bespoke hand-rollable format. It's about being, which I think is how a lot of people think about JSON APIs, right? They look at them, they try to construct the most optimal, most
Starting point is 00:49:09 aesthetically pleasing shape of their JSON. I think that the JSON, the API, the hand-rolled artisanal versus XML and WSDL and these things, I think it's this kind of a yin and yang that we go back and forth between,
Starting point is 00:49:26 similar to static versus dynamic type languages where you want the freedom, but then you lose the tooling and these conventions. And I think it's tricky ground because everybody has opinions on it and it seems like there's no black or white answers either way. So I think the interesting thing for Rails is that DHH seems to really come down on the artisanal side on Jason.
Starting point is 00:49:50 But I think I, as a programmer learned the right answer to this question by looking at active record, which had a fairly similar question early on. It basically said, okay, DHH has figured out that the way you could produce the best programming experience is by not allowing you to choose whether or not your primary key is called ID and what the foreign keys are called and all this stuff, right? We're going to tell you what the answer is. And maybe a DBA is going to look at that and say, oh, that's so disgusting. I can't believe the SQL that's being emitted. And you're supposed to say in Rails, I just don't care. I'm not interested in that level of abstraction. And the story with JSON API is pretty similar, right?
Starting point is 00:50:25 It says, what I want to do is I want to describe my models. And yes, there's going to be some impedance mismatch, just like there is with SQL. But I mostly want to think at the model level of abstraction. I mostly want to think about the model in something like Ember is pretty similar to like the resource concept in Rails. It's not about database tables. I want to think about models.
Starting point is 00:50:45 And the fact that there's maybe, you know, maybe an extra attributes hash or extra URLs, or maybe the fact that every single thing has to have a type and an ID, right? All those questions are just not, certainly you want to make sure that the format is debuggable. I think that's one of the things people don't like
Starting point is 00:51:03 about more binary formats. It's like you look at it and you're like, as a human, I cannot figure out what this is even trying to say. And I think that's very bad for debugging. But how many, you know, do I have a type associated with every single one of my records that I have sent over this format? Those are questions that are legitimate aesthetic questions, but they are analogous to like, I would really not, I don't want ID to be my primary key name. They're just not very important. And the more you hold on to saying that it's very important that you care about that, the more we push back, we leave open the big gap. So this is the empowerment thing, right? There are people who want to build stuff on the web and making them learn about handcrafting their JSON and HTTP status codes and all this stuff.
Starting point is 00:51:45 Even me, when I'm trying to write apps, I don't want to be thinking about this stuff. And the fact that this is just a part, the normal part of the work of building applications means that there's just a bunch of people who can't do it or can't do it at any particular point. It's a hard weekend project. It's a hard side project for a job. And basically the job of JSON API and my philosophy in general is to say how much of that actually matters. And in fact, that was also the point of Rails, right?
Starting point is 00:52:11 Is to say how much of the stuff that people are spending all their time on actually matters when it comes down to what your application is doing. Maybe performance really matters. Maybe you need to handcraft this thing and you can't use JSON API, but why don't you figure that out after you figure out if the
Starting point is 00:52:26 thing you're building even matters? I think that there are other aspects. We talked about aesthetics. You mentioned one, performance. The other thing is how well your client libraries, whether it's Ember or an iOS library or whatever it happens to be, how well that
Starting point is 00:52:42 maps on top of what JSON API is speaking both in protocol and in format. So has there been a lot of give and take between not just suiting Ember's needs, but suiting an iOS library, suiting React, or these types of things? Yes, absolutely. So the evolution of JSON API was that it started off being pretty extracted from Ember data. And I agree with DHH,
Starting point is 00:53:07 extracting these things is way better than trying to invent them out of your head. So it started off already being pretty useful, even just to point people at, as if you're trying to build an Ember data thing, it is now documented better. So that's where it started from. Bonus, yeah.
Starting point is 00:53:23 I mean, it was sort of documented, but mostly in the source, mostly in my comments, right? So that's where it started from. Bonus. Yeah. I mean, it was like sort of documented, but mostly in the source, mostly in my comments, right? So that was already helpful. But we started getting a lot of people who weren't using Ember to say, oh, I'm trying to write an iOS client library. We already have iOS client libraries. There's already like a Java server
Starting point is 00:53:40 project that lets you build JSON APIs. There's a Node project. And so increasingly as people try to use JSON API for things that were not the Rails and Ember combo, people ask for more things. And really, the philosophy of JSON API at the end of the day, like today, is that JSON API is just describing an arbitrary graph of objects serialized over the wire. And what that just means is that you have a bunch of objects, and they are linked together in some way. In order to link them in a serialized form, you need some identifier. And we chose to use the type and ID identifier as the key, in large part because servers pretty much already have that information
Starting point is 00:54:16 readily available. So a big part of JSON API is making it possible to build a JSON API server with existing infrastructure, like Express or Rails. It's like not trying to say that you need to start over from scratch and build entirely new infrastructure. But the fact that the type and ID are really just talking about linkage between different objects means that you could have like one type and a UUID as your linkage. If you're trying to serialize a completely arbitrary graph of objects, you could just have like the type is always object and the ID is always a UUID that you generated. That would be just fine. And so the way I think about JSON API these days is that. It's a graph of object serialization,
Starting point is 00:54:56 which really is the most general. It's hard to imagine things that you're trying to serialize that you cannot express using that serialization mechanism because it's really quite general so who all was involved in this process and from you know May of 2013 is when you first kind of pulled it out of that RailsConf conversations and Ember data at the time to now where like we said earlier it shipped stable 1.0 in May of 2015. We're wondering where was W3C in all of this? Who was involved in the specification? Was it just yourself, Steve, and the other person mentioned?
Starting point is 00:55:34 Or were you able to get kind of a groundswell of support? Because I was a person who, when I realized you guys were working on this, I was like, yes, this is the kind of thing that needs to exist. I'm glad that smart people are working on it. And when it's done, I'm just going to use it. And that's one last thing I have to think about as I build my application. So that's where I fell down on it. But were there other people that thought, ooh, this needs to exist and we have different needs than Ember. And so we're going to get involved. Was there a lot of discourse? Yeah. I mean, I think Dan has some number, but there were like, there were threads about seemingly trivial things that went on and
Starting point is 00:56:09 on and on for hundreds of comments that occasionally I had to weigh in on as like the final arbiter of some of these things. I would say that basically it had like a sort of on again, off again story until pretty recently. So I put out the thing. I kind of just said, okay, here's the thing. I'm going to keep quiet for a while, let people try to implement it. And we got like probably like around a year later, we got a good enough amount of initial feedback from people who were trying to put it into production,
Starting point is 00:56:39 which actually was pretty cool, right? So we had like, you know, maybe like a dozen people who were legitimately trying to make JSON API, the original draft, a production thing, and all of them hit various issues that they reported. So about a year later, we said, okay, we, I think we have enough information now to sort of build the second draft. And the second draft was largely driven by these ambiguity reducing ideas, right? So we were trying to find places where we unnecessarily said this is a should or a may. And in fact, there's really no reason for that. The cost of the ambiguity is much higher than the flexibility benefits. So let's just make that a
Starting point is 00:57:17 must, or let's just remove that and say that's a potentially a future direction for an associated spec. So we did that. I was actually part of the W3C tag for a lot of this process. And my participation in standards in general gave me a pretty good sense of like what the process of building a standard should look like. But there was no real standard organization, quote unquote real. There's a funny thing, which is that the WhatWG is sort of perceived to be a standards organization. But all the real standards people are like, oh, these kids making standards. You don't really know what it's like to make a standard.
Starting point is 00:57:51 And the JSON API process was sort of like that. It was a pretty serious attempt to get pretty rigorous. It uses the RFC story for how you say things are required, not required. It tried to get pretty clear about what the requirements were. It uses the media type extension mechanism that the other specs we're working with use. We did a good job of deferring things and leaving open extensibility points.
Starting point is 00:58:22 We've said that it's a living standard, but it's always going to be compatible. So JSON API 1.1 is only going to be additive. It's never going to break existing implementation. So we've done a lot of the things that real standards like to do. But no, there was no, there were no, I mean, I guess I'm a standards person, right? Well, you're saying the word real standards there,
Starting point is 00:58:43 and I'm kind of piggybacking off of something you said in the talk that seems to be some way for us to weave back into this story, which is it seemed like you had some sort of not so much a negative thing against asking for permission from a central authority, which tends to be WC3 or what you're calling real standards. It seemed like you and Steve and these other people involved in this were, like as Jared said before, like anyone else who had heard you working on this were totally excited because you're smart people, you're able to do it. But it seemed like you didn't feel like you had to ask permission to create this JSON API spec that would become the de facto standard should the community bless it and adopt it. So the first thing we actually did was file an IANA request for the media type. So I would say that we did go through the minimum viable standards process, which was
Starting point is 00:59:35 to express what it is that we're doing and ask for permission to have a centrally registered media type. And I think that that was good. I think that I think we did. We did a few things that people who care about standards being standards would have requested. And I think that was the right thing. But I also feel like we're sort of there was there's sort of been this pendulum swing where historically people said we things should be standardized unless there are two interoperable implementations, which sort of implies that the implementations go first. But then like somewhere in the, you know, somewhere after 2000, a bunch of standards work happened that was attempting to lead by standards. And I think there's still some of that left, but I just don't think that works at all. And a big part of the pendulum swing that I've been part of in both the TC39 process and to whatever
Starting point is 01:00:26 extent I've been involved in the W3C has been an attempt to remind the standards bodies that they should not try to lead because that's not what standards are about, right? Standards cannot, standards are about, you know, acquiring social consensus from implementers. It's about having a nice space where you can work for W3C. It's very important that the space is not patent encumbered, right? So any spec that's approved by the W3C, by definition, it's a promise that all the participants in the W3C, all the members, say that they will not sue anybody for patents because of the technologies that are in the specs. That's pretty important, right? So the standards bodies are useful, but they really do need to be facilitators, specifically facilitators and implementers and users, and not try to have standards people do the work. That's a really good point. I mean, because that's what you're saying here, too, is that, you know, going back to Rails, it was extracted from Basecamp.
Starting point is 01:01:21 This was extracted from what you were doing in Ember's, in Ember data. So it seems like people who are actually doing, not so much that standards aren't doing, but I like your idea of how they're providing this safe environment to come to consensus of what's out there, what's working, why is it working, who trusts it, why do they trust it, and bring it all into one thing that is under some sort of governance that's sort of community bound in a way. And I think like, so there always are implementers that are participants. I mean, like TC39 is like a ton of implementers. And the real thing that you get out of, like the thing that you get out of that process is not that the implementers are sitting around in a room talking to each other about what to implement, but that the implementers understand
Starting point is 01:01:59 that they can, that when they implement something or when they propose something, they have acquired the consensus necessary to feel confident that they can move that when they implement something or when they propose something, they have acquired the consensus necessary to feel confident that they can move forward with it. And that process, like some people think of that as being an inherently slow process. But first of all, I just don't agree. I think if you look at like the ES6 process or even the HTML5 process, we've made pretty good progress on features despite the need for consensus. But I think people forget that there's a massive time cost in not acquiring social consensus, right? If you, you know, Chrome maybe could ship something a little faster or Firefox could ship something a little faster,
Starting point is 01:02:33 but whatever gains you get from shipping something a little faster will be eaten up immediately by the cost of convincing the other browsers to implement it. True. So, and sometimes you have to go first, right? We always talk about this on TC39. Sometimes someone has to go first. And sometimes you have to go first, right? We always talk about this on TC39. Sometimes someone has to go first. And usually what will happen is somebody will propose something and everyone will say, I'm not really sure what the value of that is, but I'm not necessarily opposed to it. I will implement it if it turns out. But why don't you go first? And then usually the person will go first and quite often it will happen.
Starting point is 01:03:01 They just draw straws or what? No, it's usually the people who are the most confident that it's a good idea. Well, that's a good place to take a pause. We have one more break before the show comes to a close. However, don't worry. We have lots more to cover. Compliance, resource discovery, some experimental things, future-proofing APIs, just to kind of touch on a couple of topics coming up. But let's take a break. We'll come back and we'll dive deeper with you on JSON API and hopefully you'll love it.
Starting point is 01:03:28 So we'll be right back. Our friends at DigitalOcean, simple cloud hosting built for developers, launched a new feature recently we absolutely love. It's called Droplet Multi-Create for easily launching up to 10 servers at once. You can deploy multiple droplets when you create your droplets
Starting point is 01:03:46 with the exact same configuration. And this is a huge feature. We absolutely love it. Head to digitalocean.com. And when you sign up, use our code CHANGELOCK to get a $10 hosting credit when you sign up. All right, everybody, we're back
Starting point is 01:04:01 and we are diving deep on JSON API. Yehuda, you've mentioned a few features offhand, but I thought we'd take a moment to kind of go through them one by one you're doing is you're serializing a graph of objects on your server. What I mean by graph is just that it's possible for one object to be related to multiple of them, which is like sort of like a hasn't belongs to many relationship. Or you could even have like a, you know, a post has many comments and the comments belongs to a user, but that user might be referenced from other places. So when I say graph, I just mean that people often imagine in their head that the data structure is a tree. They imagine like, well, there's a post and it has many comments, so that's nested inside
Starting point is 01:04:57 of there. And then the comments have users, so you just nest the users inside of there. But in the real world, those objects could be related to yet other objects. So you don't necessarily want to continue nesting. And this is actually a core value of JSON API. It's something that is sort of counterintuitive to a lot of people. But trying to make it a tree has pretty serious implications on both efficiency and your ability to properly deserialize it. Like what if you have the same user in multiple places and they don't match? What does that actually mean?
Starting point is 01:05:27 Do you have to check that? So one of the core values of JSON API is that we're talking about a graph of objects, not a tree of objects. And there's often, not always, but quite often a primary document. So if you ask for post number one, you want to know that you might get back a whole bunch of stuff.
Starting point is 01:05:46 You might even get like a summary of how many documents there are in the entire system. But there's a primary response, which is the post object. And then it might have links to comments. And those links to comments could either be referring to other things that are inside the same response. And that's commonly what people do. But they could also be URLs that link to other documents that you can use. So maybe you're downloading a list of posts just because you want to print the front page of your blog and you don't need the comments until you actually click on a comment. You still want to say, like, if you ever need the comments, here's the URL for those comments.
Starting point is 01:06:22 So the idea is that what we're talking about is a bunch of linked objects. But unlike maybe the more pure rest or the pure like linked data philosophy, the assumption is that you will quite often want to include a lot of linked objects together. And there should be a very canonical standard way to do that. And it shouldn't always have to be URL. So one thing that's a little problematic about URLs, first of all, is that it implies that there's a way to get individual, first of all, is that it implies that there's a way to get individual pieces. But maybe, for example, there's no way to get comment number 17. The only way to ever get comment 17 is by going through the post. And that's a totally reasonable thing to want to be allowed to do. But if you only link things through URLs, you have to
Starting point is 01:06:58 construct a URL for that comment. And maybe like a very URL person would say, that's fine, you can make a fake URL. But this stuff kind of adds up in complexity. And what we want to be able to do is just express connected data in ways that are most natural for your implementation. That doesn't mean that implementation leaks, it just should mean that it's natural for the implementation that people have. So we have, you know, we have this post that has many comments, maybe the comments are directly embedded in the document, maybe they're linked through URLs, maybe they're not linked through URLs, they're linked in some way. But there's very canonical ways of doing all the different ways
Starting point is 01:07:36 of describing things, and they're composable. And then, so all the ways of interacting with that data is done using normal HTTP, the HTTP verb. So you get something using the get request. And when you get something, you just might happen to get back other related things. If you want to add a new thing, you post to it. And there's some rules for exactly how that works. There's also delete, to delete something, and there's patch, which allows you to, there's
Starting point is 01:08:10 sort of a historical sad ambiguity about what put exactly means, but patch very canonically means the right thing, so you can patch to make some partial changes to an existing document. Patch is an update to an existing document and puts like a, it's supposed to replace the entire thing,
Starting point is 01:08:26 right? So I will just tell you, having now spent many years writing this spec, that people feel a lot more confident about what put means than I think they should. But people do have religious opinions about what put means, and because of the fact that you're interacting with infrastructure on the internet, you should probably not run
Starting point is 01:08:41 afoul of people's religious opinions about what things mean. You're just avoiding put altogether. You're just like, we're just going to patch. Yeah. And we may, there are use cases for put in JSON API that I think everyone would accept. But we effectively punted on them for 1.0 because of the fact that what people really usually want is patch. So I would say that that's the main thing.
Starting point is 01:09:06 We basically use normal HTTP verbs to describe interactions. And I say that and people probably are nodding their head and saying, OK, I basically know what that means. I assure you, you basically don't know what it means. Why is that? It's just like the exact, like, OK, I'm going to make a, you know, I'm going to make a patch this document. First of all, what is the format? What is the response format? And then what are like all the
Starting point is 01:09:28 status codes? What are they supposed to mean? If you have errors, how do you format that? There's like a billion, like there's so many questions that you have to answer, even to answer like, what is a 204? What is a 200? What is a 404? What is a 500? Like what kinds of information should come along with each of those things? Some of them, the spec says something like 204. You're supposed to not include everything, anything. Great. Okay. Problem solved.
Starting point is 01:09:50 But what about like every other status code? What about errors? And you really do want all these questions to be not left up to somebody's whims. You really, if somebody doesn't have a better idea, you would really like to tell them what the answer is. If someone has a better idea and it's a really good reason, escape valves are a good idea. And we have this metadata. And actually, that's another core value is that every place in the document has a metadata key that you can use to put arbitrary information that you think is like the spec doesn't talk about, but is useful information to tell your clients.
Starting point is 01:10:23 First of all, that allows us to reserve the top level, like anything, any top level key is reserved. So we can add more stuff later without breaking existing clients. But second of all, it, it means that people can do pretty aggressive experimentation with things like pagination or filtering, um, some of which are in the specs, some of which are not with, uh, without having to worry about interoperability. So it's, it's sort of a place where you say, I, you know, some of which are not, without having to worry about interoperability. So it's sort of a place where you say, I know for my application,
Starting point is 01:10:49 I would like to include this extra information. JSON API is not locking you out of the ability to do that. Does it have pagination and filters? Yep, so there's pagination and filtering and sorting. I would say that the specification for each of these things is not as rigorous. It's basically clear, and the word must and should are used, so it tells you what keys you have to use and, in some cases, what the semantics should be. But for sure, over time, the specification will get clearer about exactly what you should expect.
Starting point is 01:11:24 So, for example, we tell you that there's the offset and limit pagination, and there's a number and size pagination story, there's the cursor story, and we tell you how each of those things should work. And we tell you that you, you know, how you should encode the characters and all these things. But I think we're like, basically, people are using these tools right now to build pagination solutions. And we're getting a lot of feedback from people about what the best way to rigorously define how to do it. Filtering and sorting are actually easier specs. And those are already more rigorous.
Starting point is 01:12:02 But we've effectively reserved the pagination space and given you enough tools to work with, plus metadata stories so that you can figure out something we're waiting for things like ember data and other uh and other clients to figure out like what it really looks like to use the metadata solution and then the idea is that you take the things that people put in metadata and make them first class once we figure out what it is so you mentioned that the meta one of the purposes of the metadata is so you can reserve uh the top level for future changes because you want to be backwards compatible. Can you expand on your future-proof API design? Is that kind of a core value as well? Yes. So one thing that I've learned from specifications is that people underestimate the cost of backwards incompatible changes. And the reason for that is that people underestimate network effects in general. So if you look at something like NPM or RubyGems or APIs on the web, there's a lot more
Starting point is 01:12:52 people involved than you think, and they all connect to each other. They don't all just connect to the central thing. There may be intermediate things. So there may be libraries or other APIs that work with the other third-party API, right? And the implementation details can leak across these abstractions. So one of the things that the web has decided is if you're willing to be creative and you're willing to be pretty careful about what you expose, how much arbitrary goop you allow people to throw in, then you can pretty much keep adding stuff. You could deprecate stuff, but you can deprecate stuff without removing it. And you can keep adding things over time and mostly have things continue to work. And I'm just generally a big believer in the idea that it is almost never the case that breaking changes are worth it. I think in things like Ember, especially when you're talking about the very first version of something, there's probably like one moment in time where breaking changes become important.
Starting point is 01:13:50 I think like Linux had a moment like this. Ruby had a moment like this. And even when you do those breaking changes, you have to be extremely careful. I think you can compare Ruby to Python. You see Ruby did a much better job of being tasteful and careful about the breaking changes than Python did. But I think you can do a pretty okay job of keeping things permanently compatible. And I think that people underestimate the, they overestimate the value and underestimate the cost of being willing to break things. We just talked, the Rust core team just had a conversation a few times about this. And there's like full consensus on this yet, but I think most people think that Rust 2.0 doesn't really make sense. That we should continue trying to, maybe we'll have a Rust 1.40, and we think we could continue to make changes and even add things like garbage collection we think we can do compatibly. And I think people just aren't – they aren't very creative about – it's just much easier to say, oh, the world is changing. We've got to change everything.
Starting point is 01:14:51 And I think it's just not usually true. And I basically have put my money where my mouth is a few times, and I think people should try, and I think it will usually work out. So one of the ways we kicked off the show was actually kind of deriving where this came from. It started with a tweet from you which was comparing JSON API to ASM.js. You said you can think of JSONAPI.org, which is obviously JSON API, as
Starting point is 01:15:16 ASM.json, which is a low-level consistent serialization format and protocol that you can build on. Can you compare the two? Like you said earlier about having a de facto standard. It seems like, obviously,
Starting point is 01:15:32 this is what you're trying to do with it, but then you also have this. Are they competing standards? What is the differences and do they play together? So Asm.js is this thing that Mozilla invented, Mozilla Research, which is basically just, they noticed that there's a subset of JavaScript that if you squint closely enough is a fully typed thing
Starting point is 01:15:51 that is sufficient to represent basically all of C. And so they basically defined the subset of JavaScript that they will validate. You basically tell them that you're using it, then they validate it, and then it will compile into native code. It's like somewhere between 20 and 50 percent slower than just like straight-up compiled C code. And the reason I compared it to that is that if you look
Starting point is 01:16:11 at Asm.js, what you will see is that no human being wants to write this. It's actually significantly worse than JSON API. It's like a horrible format to write in. But the point of Asm.js is that that's not the point. That's not the goal of Asm.js. The goal of Asm.js is to be fast and also to be a thing that you could build on. You could, you know, you could compile gzip to Asm.js and now you have a fast gzip that you could run in a browser. And so the idea is you shouldn't, you shouldn't be too, people have said on Hacker News a lot of times, like Asm.js is so ugly. It's like such a hack. Why you know bytecode? And the point is that that doesn't really, like the exact syntax of Asm.js doesn't actually matter. The thing that
Starting point is 01:16:50 matters is that we have now constructed a thing that you could write that ships on the web that is pretty close to being as fast as C. And that is pretty important and effectively supersedes any aesthetic considerations that you might have. And I was sort of trying to get across a similar thing about JSON API, that the goal of JSON API is not to allow you to build artisanal APIs. The goal of JSON API is to create a nice wire protocol and format for communicating graphs of objects between arbitrary servers and clients. And obviously, there's some constraints there, like you should be able to do it on existing servers. So Rails, Express, Java should all be able to do it. You shouldn't have to build your own whole new server
Starting point is 01:17:31 with whole new server infrastructure. Shouldn't require things like WebSockets to work, right? So it should work with regular HTTP infrastructure people have already deployed. So those things are all very important. But the exact, you know, whether it's 20% better or worse than the aesthetics that you would have personally designed for a given project is actually just much, much less important
Starting point is 01:17:50 than the interoperability. And so that's sort of what I was getting at was that it's more of a low-level protocol. It's a way of getting everyone on the same page about something that you can share and build on than it is about being a, and again, it needs to not be a format that's like impossible for humans to understand. It needs to be a format that a human can look at. But whether or not there's a top level attributes key or you leave it off and put all the attributes at the top level is really just not something that needs to be a reason for non-interoperability.
Starting point is 01:18:20 That is a bad excuse for non-interoperability. So one last question before we close up the conversation around JSON API is, I think 2015 has kind of been the year of Facebook and Netflix kind of launching out these new API, I'm going to call them transport patterns or styles, I'm not sure exactly what you call them, but we have GraphQL, we have Falcor, and these seem to be taking a little bit different approach than the traditional REST JSON-based APIs. Can you compare and contrast kind of advantage, disadvantage of the JSON API style versus these new styles?
Starting point is 01:18:59 Yeah, but I think I've been sort of getting at it, but I'll be very explicit. Okay. I think that what GraphQL and Falcor are both getting at it, but I'll be very explicit. I think that what GraphQL and Falcor are both getting at is the fact that very purist ideological REST is pretty hostile to bulk transport, and it's pretty hostile to customized transport. People don't want you to say, give me this resource, but don't include these attributes. And they don't want you to say, give me this resource and include these other related resources. In fact, the very
Starting point is 01:19:27 purist REST people always talk about HTTP2 as allowing you to just continue down the path of having one URL per resource and just, you know, firehose them with one HTTP socket, but now multiple requests. And JSON API and Falcor and GraphQL all share a philosophy that a client should be able to, you should be able to construct payloads that are appropriate for the access patterns that your client actually has. If you're about to look at a blog and you know that people are likely to click on a comment, you should be allowed to include all the comments at once. So you should be, certainly you should be allowed to include all the comments at once. Certainly you should be allowed to include the count of comments per post so that you can put in the UI.
Starting point is 01:20:10 GraphQL is on a pretty one far end of the spectrum, which is that graph, the idea behind GraphQL is that you derive the exact thing that you need out of the consumption patterns. So you say, I need a post. So every component basically says what it wants. It says, I want a post. The other guy says you say, I need a post. So every component basically says what it wants. It says, I want a post. The other guy says, oh, I want a comment.
Starting point is 01:20:29 I need the comment count. And the GraphQL library takes all those things, combines it into one request, and it asks for very precisely what the client has asked for. And Falcor, I would say, is also pretty similar to that, although Falcore is a more flexible... I don't know. Whoever's listening to me who works on these projects is going to
Starting point is 01:20:50 yell at me no matter what. But I would say that both Filecore and GraphQL are designed for consumption patterns. They're designed for building a request for exactly the thing that you need for what the client is asking for. And first of all, one of the reasons JSON API did not go down this path is that it requires a server that's capable of pretty sophisticated querying. And I think, again, JSON API has as a goal that you should be able to use
Starting point is 01:21:14 existing server infrastructure. But perhaps more importantly, that kind of stuff is pretty uncacheable. So if you ask for a very specific subset of the data, it's very hard to say, oh, I basically know what you're asking for. You're asking for all the posts.
Starting point is 01:21:28 No problem. I already cached the whole payload. It's very hard to do that when you're asking for very specific pieces. It's also not really a good fit for what the GraphQL people pejoratively call overfetching and I positively call overfetching. Sometimes you have to think about when you're building an application, if you're showing a particular page, what are the
Starting point is 01:21:52 likely next things the user is going to do? And if it's very likely that within a few seconds or half a minute, let's say 10 or 15 seconds, the user is going to click on something, it's actually a reasonable thing to give them the thing that they're going to click on together with the original payload sometimes. You have to think about it, obviously, but it's sometimes worth sending extra data out of the gate
Starting point is 01:22:14 so that not the first click will be fast, but then the second, third, and fourth click will be fast. And so the Ember philosophy is a little bit more, be a little more liberal about what information you download so that when you click on the second or third thing, you didn't tightly scope the data request to exactly what was on the screen the first time. That a lot of the times the second click will have opportunistically gotten enough information to render it without making new XHRs.
Starting point is 01:22:39 That's the Ember philosophy. And the JSON API philosophy is sort of agnostic, but it is pretty compatible with the idea of sending extra information out of the gate so that your subsequent navigations are reasonably fast. So again, I would say JSON APIs, like the core, the things that I think are powerful and special about JSON API are that it's both pretty good about, it shares the philosophy of wanting to be able to have a payload that's not just talking about one thing, but it's like all the things I need for this particular screen. But it does not share the
Starting point is 01:23:17 philosophy of needing a new infrastructure to make that work. It wants to be able to piggyback on all the infrastructure that people already have to make it work. And it also is pretty positive towards the idea of opportunistically including extra information that you think the user might want. Again, I'm sure that any GraphQL person listening knows how to make that work with GraphQL. But again, the GraphQL consumption story is that you don't really think about the query that you're constructing. The query is constructed automatically based on what your components have said they want. And I just personally don't buy that. I think it's bad to tightly scope the data to what happens to be on the screen at a particular time. Yeah, you mentioned infrastructure, and I got to thinking about HTTP2 as you were talking. And it seems like all of these are focusing on optimizing that for the
Starting point is 01:24:06 minimum number of requests in order to get the exact data you need for the screen or for the particular widget or what have you. It seems like the HTTP2 features, the pipelining, the multiplexing, so on, keeping that one connection open, doesn't that mitigate a lot of the problems that we're trying to solve? So people have that hunch, and there's a sense in which that hunch is correct. But I think people have to remember. No, I think you can construct examples where it helps. I think people have to remember that there's still a round trip if you use an open HTTP 2 socket to make a request.
Starting point is 01:24:41 So if you're basically letting consumption drive your requests, and yes, it is true that when you click on a page, you can fire off the request much faster if there's an open socket just sitting there. And if you're also able to do eager pushes, you could try to be clever about what you're doing. But there's still, the speed of light is still, has only a limited speed,
Starting point is 01:25:04 and people in New Zealand and Australia But there's still like the speed of light is still has only a limited speed. And people in New Zealand and Australia experience this pretty heavily where people just assume that your internet connection is fast. And the like the New Zealand and Australia problem has not helped a lot by HTTP2. I mean, it's helped a certain degree, but it doesn't make the connection to the server instantaneous just because it's an open socket. So I think people are over-optimistic about the ability to take a thing that used to be one document and break it up into a thousand documents because they have in their head the idea that HTTP2 makes the round trips instantaneous, but that is not correct. So in other words, there's still a certain amount of serialization that happens. Let's imagine that you make the idea is you make a request for document A, and it has
Starting point is 01:25:46 a dependency on document B, and it has a dependency on document C. And it's like, oh, no problem. Now we don't have to include those into one payload because we'll just instantaneously get document A and see that we need B and C and C. In fact, of course, that is not instantaneous. There's still some time that it takes to request and download when a smart server could have bundled together A, B, and C into one thing. And HTTP2 people will say, well, a smart server could also push A, B, and C together.
Starting point is 01:26:13 But that smart server is not really like what GraphQL is talking about or what most people are talking about. You need to basically, the server has to figure out the consumption pattern and say like, oh, I've noticed that a lot of people have asked for B after A, so I will start sending it. Or you need to have something that is morally equivalent to bundling where you are able to construct manifests that say, whenever you see A, that basically means you need B, using basically the same logic as bundling. So I guess what I'm saying is bundling still works very effectively. It's still reliable. It works on old connections and new connections.
Starting point is 01:26:47 And the exact story of making things work without bundling ends up getting to be pretty domain-specific, whereas bundling just works. If you take all the things that you know you're going to need and put them into one payload and send them at once, you don't have to think anymore. You're done. And I think that's why people like it,
Starting point is 01:27:04 and I think it's also why HTTP2... The story about HTTP2 sells better in the realm of the abstract than it does in the real world. The real world is not magically solving the kinds of issues that cause people to like concatenate their JavaScript now that HTTP2 exists.
Starting point is 01:27:21 People are in fact noticing that it doesn't work as well as they had hoped. Well, I guess we'll all just have to wait for HTTP3, which I hear will support quantum computing. I mean, I think, honestly, the TLDR is that if you... We just got to get rid of that
Starting point is 01:27:35 speed of light problem, right? Yes. So, I think the TLDR of all this is that if you... Are you opening the box to get him to talk about quantum computing? No, I'm not going to. I think if you do something that is morally equivalent to bundling, like you basically construct the bundle, but then you use
Starting point is 01:27:52 HTTP2 to push the bundled components, that's probably fine. Like that probably will end up being pretty similar to bundling, except that that's... At that point, you are not really doing the things that people are excited about about HTTP2, right? It didn't really, it will probably happen, right?
Starting point is 01:28:09 It'll probably happen when HTTP2 is very ubiquitously rolled out. People will have some format that lets you say, like, here is the quote-unquote bundle. Someone asked for this entry point, give them all the things, but it doesn't really buy you that much. Well, it's really been a fun, deep, long conversation about
Starting point is 01:28:25 JSON API. Its origins, obviously, you know, the first thing to call was, you know, focus on you personally and your origins in programming and when you became a professional and obviously open source. But this has been a fantastic call. Yehudi, you are a man of many hats,
Starting point is 01:28:41 though, so we couldn't leave this show without at least touching on Ember. We here at the channel have been tracking isemberfastyet.com all of 2015. It's not 2016, so congrats on that actually being fast. But is there anything you can share today about the current state of EmberJS, what's going on, and whatnot there? Sure. So I've been working on a project called Glimmer 2 for the past three or so months. So Is Ember Fast Yet?
Starting point is 01:29:07 was basically a, was about Glimmer 1, the first version of Glimmer, which we actually shipped a while ago now, like maybe May, and then spent the next few months like making it really polished. And basically what Glimmer 1 did
Starting point is 01:29:20 was it made re-renders very fast. So the idea is if you have a, you know, you have a loop, you have an each loop in your template and you want to update it with a totally new array in old Ember and Ember 1.12, that would have to blow away all the DOM
Starting point is 01:29:36 and replace it. And React showed us that we don't need to do that. There are better strategies that allow you to cleverly update only the parts of the DOM that you really need to update. And so what Glimmer 1 was about was about taking those strategies that React pioneered and making them work well in Ember for cases where you have lists of things
Starting point is 01:29:52 that maybe the data structure has completely changed, but the things it's talking about have not. So maybe you have a list of people and you're constructing a completely new array, but the items inside the array are the same, or maybe there are completely new objects, but the items inside the array are the same, or maybe there are completely new objects, but the properties of all those objects are exactly the same. And making it so that basically without having to think about it,
Starting point is 01:30:11 all those scenarios end up updating fast. And Gliver 1 did that. I think we showed it at Ember Conf a demo that we then repeated when we actually shipped Ember 2.0 that showed a speedup of probably more than 10x for the updating case. It was to the point where Ember 1.x, the demo, basically froze. And Ember 2.x, it effectively shows fast updates.
Starting point is 01:30:34 It shows everything updating many times per second, basically, instead of being frozen. So, very big improvement. However, Glimmer 1 did not really do much for initial render performance. And in fact, for various and sundry reasons, it regressed initial rendering performance. And unfortunately, because re-rendering was so slow before Glimmer, almost nobody had applications that were doing a lot of re-rendering, which meant that all the work we did to improve re-rendering didn't affect existing apps much. It did let you write new apps better
Starting point is 01:31:05 that would take advantage of it, but it didn't affect existing apps. But all existing apps got slower because the initial render got slower. So that made people rightly sad. And we've done a lot of work since then to close the gap just by picking away at little paper cuts that we had introduced
Starting point is 01:31:23 for various reasons. A lot of them were compatibility, but not all of them. And the idea behind Glimmer 2 is to say, okay, now that we have integrated Glimmer into Ember, what have we learned about what the requirements are, right? So one of the reasons why things were so slow the first time, I'm being very rough on myself. The reason why performance regressed after Glimimmer one was that we effectively had to learn what the requirements for ember compatibility were as we were integrating it and as you can imagine like as a programmer sometimes the way you solve problems like that is you just do whatever you do whatever you have to do to get past the problem and we did
Starting point is 01:32:00 that enough times that things didn't compose very performant. But now that we're done, now that we've done all the integration against the more React style of internals model, we kind of know what all the requirements are. And so the idea behind Glimmer 2 is to rebuild the primitive layer against the new requirements. And we have some benchmarks now that we've been running for a while that show many, many X performance improvement on initial render and things like beating React on equivalent templates. So like beating React by a decent amount on equivalent templates. So we're definitely pretty excited about that. One of the things that is like, one of the things that I've been working towards that we haven't quite done yet,
Starting point is 01:32:48 but it's going to be pretty awesome is, so Ember's story about templating the whole time has been that we give you a declarative templating layer. So you just write your HTML and you write, you don't write JavaScript in there. You write basically direct what I guess Angular called directives, but I mean the English form. You write directives that say, this is a loop, this is an if statement, whatever. And historically, that whole system has been pretty dynamic. Like we sort of compiled it and then we said, we don't really know what this if statement is talking about, so just figure it out at runtime. But we noticed a pattern, which is that a lot of times people will make components. So there's this project called Font Awesome, which is pretty cool. And people write components that wrap Font Awesome. So that instead of having to say like I class equals FA space, FA bug or whatever,
Starting point is 01:33:36 they'll make an Ember component called FA icon. And they'll say like FA icon, icon equals bug. And it will basically emit the correct code. But if you look at that invocation, icon equals bug, you'll see that there's nothing dynamic about that at all. They're literally using a string bug. So in an ideal world, you'd be able to take the template that is talking about the layout for that component and you would be able to notice,
Starting point is 01:33:57 ah, that is actually not talking about a dynamic thing and compile it into the static thing that it is really talking about. And one of the goals of Glimmer, too, is to have a sufficiently flexible compilation architecture that we can effectively do these specializations at runtime, do them only for hot code. So, like, you probably don't need to do that
Starting point is 01:34:16 if the FA icon only shows up one time. But if it's in a loop, it probably matters more. So basically to have a flexible architecture that lets us say, you know, we notice that this thing is a lot more static than the code would imply. So let's actually compile it into the static form. And this is all owed to the fact that the Glimmer and like Ember templating system is like a pretty nice programming language for optimizations in general. So it's like more like a referentially transparent programming language. It doesn't allow you to do a lot of the dubious things that make it hard to
Starting point is 01:34:48 do optimizations. So we can apply like the beginner level optimizations that you learn in school against this very simplified programming language and have, and be able to do that reliably. So, so that's basically the story behind GlimmerTube is that we're restructuring the architecture so that it's a flexible compilation architecture. And then we can make things that are static behave as if they were static instead of behaving as if they were dynamic. And so far, the work that we've done on this kind of stuff has shown very significant improvements. So we're pretty happy about that. I'm probably one of the people that's putting my hand up saying, I don't know how you do all you do. So we're pretty happy about that. and a consistent API and a future-proofing in a way for the future, but also the work you're doing at Ember and your company,
Starting point is 01:35:48 all that you do there and the different products, I guess, you're working on as well would play into that as well. But Yehuda, we're about 10 minutes past our time, so we're going to close the show today. We did have a couple more questions we want to ask you, but we'll save those for a different day. I'm sure you'll be back on. This is not your last time on the show, I'm sure.
Starting point is 01:36:06 So let's close there. We want to thank our listeners for coming and listening to the show religiously. It is the new year. We have a great year planned ahead of us. And if you're not a member yet, go check that out. Change.com slash membership for 20 bucks a year. You can support this show and do what we do around here and get an all-access pass back to our members only slack channel hang out with us get behind the scenes
Starting point is 01:36:31 we share lots of stuff with our members before we even share the rest of the world we also had some awesome sponsors sponsored the show so thanks to those guys and uh if you have a second you can throw it throw a note on this if you'd like to. But our next show is actually about 0DB, an end-to-end encrypted database protocol with McLean Wilkinson and Michael, I'm not sure how you say his last name, but that was a really interesting find recently. And that's the next show we're doing. So that's it. Any notes you have on that show particularly?
Starting point is 01:37:02 That seems cool i'm very generally very uh pleased about the fact that our our community and our ecosystem is pretty encryption friendly and i think it's uh the government would like it to not be so right unfortunately people like encryption and i'm happy that that's true that's cool all right well everybody thanks for listening. That is the end of the show. So everybody on the show, let's, let's say goodbye.
Starting point is 01:37:29 Bye. Thanks. No problem. I'm out.

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.