The Changelog: Software Development, Open Source - JSON API and API Design (Interview)
Episode Date: January 1, 2016Yehuda Katz joined the show to talk about JSON.API — where the spec came from, who's involved, compliance, API design, the future, and more. We also finally got Yehuda on the show alone, so we were ...able to talk with him about his origins, how he got started as a programmer, and his thoughts on struggle vs aptitude.
Transcript
Discussion (0)
I'm Yehuda Katz, and you're listening to The Change Log.
Welcome back, everyone. This is The Change Log, and I'm your host, Adam Stachowiak. This
is episode 189, and what a show it is. The first show of the year, kicking it off with Yehuda Katz
talking about JSON API. We talked about Yehuda's past.
Because this is the first time he's been on the show solo.
So we had to dive deep with him.
We talked about JSON API.
JSON API compliance.
Resource discovery.
Supporting extensions.
Future proofing API design.
And so much more.
We had three awesome sponsors.
CodeShip.
TopTile.
And also DigitalOcean. Our first sponsor of the sponsors, CodeShip, TopTile, and also DigitalOcean. Our first sponsor
of the show is CodeShip. Head to resources.codeship.com. Once again, resources.codeship.com,
and there you'll find all sorts of free resources from CodeShip, eBooks, webinars, email courses,
guides, all talking about continuous delivery, Docker, containers,
getting your code to production in a workflow that makes sense for you and your team.
You get a couple of guides that are awesome.
Why containers and Docker are the future, efficiency and development workflows.
They have a couple of webinars coming up here in January this month in 2016.
Continuous delivery and feature flagging and introduction to web apps with Docker.
So those are really awesome free webinars you can join.
They also have this really awesome five-day email crash course
on continuous delivery.
Once again, head to resources.codeship.com.
Enjoy all those free resources from Codeship.
And now on to the show.
Hey, everyone. We're here with Yehuda Katz.
It is the first of the year, January 2016, so we're excited about that for one.
But Jared, we're here talking to Yehuda today, and we've had this show in particular on the docket for a while, but this recent tweet kicked it off.
What was that tweet about?
Yeah, that was Yehuda tweeting about JSON API,
which is our conversation we'll be about today,
where he says that you can think of JSONAPI.org as ASM.JSON, a low-level, consistent
serialization format and protocol you can build on.
So I'm sure we will tee that one up
later in the show as a conversation point.
But that was kind of a thought that spurred us to say,
hey, we never had to want to talk about JSON API
back when it went.
I think it was 1.0 was earlier this year.
A few months ago, yeah.
And so here we are.
Here we are.
We've had this, Yehuda, we've had this.
So everybody, you've heard Yehuda back there
in the wings there.
But we've had this in Yehuda, we've had this, so everybody, you've heard Yehuda back there in the wings there, but we've had this in our podcast Trello board for, I don't know, all year, basically, all of last year, technically, since we're actually recording this in 2015, published in 2016.
So, you know, there you go.
Hopefully we'll make it over the hump.
There you go.
We had Yehuda and Steve Klabnick on sometime last year talking about Rust, episode 151, which was, yeah, April.
And I know at that time we definitely had right next to that one in the queue was JSON API just waiting.
So it's definitely been a while.
And Steve also works on JSON API.
So, interestingly.
So obviously that's the main topic of the show.
I'm sure we'll talk a little bit about Ember.
We'll talk a little bit how those two things play in.
And I'm sure you got some, we have a couple of questions in the wings too about JSON API and Rails 5 and playing nice together and stuff like that, which I'm sure is what you can talk deeply about.
But something recently that we've done with our guests,
especially someone like you,
you've been on the show probably four or five times.
Jared, I didn't even go back and count, honestly.
Yeah, four.
Four times?
So we'll put all the links in the show notes
to those past shows that Yehuda's been on,
but Yehuda, we haven't had you on the show by yourself,
so I don't know how you feel about that.
You always come with some sort of guest
or some sort of wingman, so to speak.
What do you think about that? Yeah, I mean, the reason for that is that pretty much I only like
to work on open source projects with other people because I don't honestly, I just don't trust
myself that much. And so there's always at least one other person who I'm like, well,
whatever I say, the other person will have another perspective that will be useful. And
I think if you like if you listen to me talking with almost any other person, there's usually valuable insights that I would have just missed. So
that's why I like to do that. It's just my style.
Jared, should we kick off the show with the mention from Brian Lyles? I know I got a couple
questions as well, but that was an interesting takeaway from Beyond Code up in GopherCon.
Yeah. So I think, you know, we're gonna spend a little spend a little time getting to know you from a historic perspective. I think we all have our own interesting roots, and it's fun to find out more about the people behind the awesome open source software that we all use and love. shoot at conference after parties. And season three was at GopherCon, which is still TBD on the release date on that one.
But during that, we had Brian Lyles on Beyond Code.
And we do ask everybody in that series
who their programming hero was.
And he mentioned you.
And so I actually pulled the quote out
because I think he gave some interesting reasons as to why.
And he also seems to be a friend of yours
because he kind of takes a couple cheap shots at the end there.
So I was going to read this to you
and have you kind of just react to it.
So he said, Yehuda Katz is a programming hero.
He says, the reason why is because I remember
when Yehuda first started writing code and he wasn't good.
And then he started writing more code and he wasn't good.
But then he started writing even more code.
And now look at him.
Not only has he mastered Ruby, he mastered JavaScript. Now he runs his own company.
And he's like this tall. And he kind of looks his hand out. And he's kind of weird looking.
Yes, I'm very short.
But he's an amazing person to be around. So high praise from Brian along with
a couple of cheap shots. But could you just react to that?
Yeah, I think that's all accurate. I'm short. That is true. I think I'm a pretty good programmer
these days. I also was a very bad programmer at the beginning, like literally every other human
being who ever starts programming. And I also think that the process of persevering through struggling with code is a thing that made me better.
And I think that I try to tell that to every new person I meet who's struggling is like basically everyone.
That's everyone's story.
Everybody struggles with code.
And if you think, oh, that person's cool, they must have had it must have been easy.
That's just like it's just like a mythos.
It's not a real thing.
I mean, maybe there are some people,
but I don't know them.
That's a good dovetail right into that tweet stream, Jerry,
we have mentioned there and Yehuda,
you probably know this because it was recent,
but you'd mentioned the gulf between using programming to automate some repetitive tasks and being able to work on a Linux kernel.
And you kind of dove a little bit deeper into struggle versus aptitude.
Can you speak to those tweets?
We have some links.
We'll put the show notes, but I'm sure you can recall that.
Yeah, I remember.
I've been doing a few.
I actually haven't blogged a lot recently because I sort of came to the conclusion that
almost anything I have to say could be constructed as like a series of at most 10 tweets.
So there you go.
So, yeah, I guess.
1400 characters, yeah.
Yeah, confirmed.
So I guess what I was saying in that,
in that tweets stream is that there's,
I've met a lot of people over my career
who have tried to program.
Some people end up being programmers.
Other people end up, you know,
getting to have some aptitude with programming, but not
necessarily being quote unquote programmers.
And I think it's pretty damaging that we get, we give off this sense of programming as like
a wizardry level thing.
Like you have to be a wizard to be useful at all.
And I often like to tell people like so, so many people are very productive in Excel.
And Excel is a very simple programming language.
Obviously, it's very simple. But there are functions and variables and some of the more abstract concepts that you need to do to be a programmer are already present.
There's, like, some amount of variables and things like that.
And I like to think that if you could program in Excel, that's probably not the end of the story.
You could probably do a little bit of scripting.
Um, and, uh, one of the things I like to think about is, uh, when I, one of the first jobs
that Leah had, um, who I work with now at, at Tilda and I am married to, uh, I met her
in college.
Uh, but one of the first jobs that she did, um did was a marketing job at a company.
And she was basically told, like, every single day you're going to get an email from one automated system.
And what you're going to do with it is you're going to open it up in Excel, make some changes to it, and then email it to another automated system.
She came home and told me this story.
And I said, first of all, can you find out, like, why the person in the first automated system can't just email the second automated system and change the format? And so she came back like the next day
and she said, I asked them and they told me like, they can do it, but it will be like six months
to schedule it or whatever. And I was like, okay, that seems terrible. But then I said, well,
why don't we just use Ruby to pull the mail in the mailbox,
get the CSV file, use the CSV library to make the translation,
and then just use the Ruby 132 API to send an email to the second system.
And we wrote that code.
And that saved her some non-trivial amount of time every day.
Every day she was supposed to do this very annoying repetitive task,
and we were able to script it away. And I guess I just think that there is a
lot more of that. I think like Ruby and Perl and Bash are scripting languages. And really what that
originally meant was like you could use them to write little scripts, like little 20 line scripts.
Actually, another example of this is every year Tilda sends out like a compensation report
that says what your bonus is and things like that. And also me and Leah wrote a thing that
it's just like a giant, it's like a 50 line file that sets up a bunch of variables and then evals
and ERB basically. And every year we go in and we make some changes. We load a CSV file that she
does in Excel, right? And these are all tasks that like a lot of people
have. They have nothing to do with programming per se, but they're very good candidates for
automation. And there's no real reason why we should tell people, like, if you can't see yourself
being a full-time professional Rails developer, you can't, you shouldn't even think about
programming. Programming, there's a big slope between, you know, not knowing anything, knowing
Excel, knowing a little bit of scripting, and then working your way up.
And I think a lot of people who became programmers actually entered through that path.
And I think we should be more happy about it, about the existence of it.
I mean, I think I just want to give my amen to that.
I think automation is kind of a gateway drug into more formal programming things.
I know when I first started in college, I was studying MIT Management Information.
Not MIT.
I was not at MIT.
I was studying MIS, Management Information Systems.
And I never considered myself a programmer or developer.
I just thought of myself as someone who likes to automate things.
And I was using Perl and just scripting little things away.
And when you see the benefit of that, especially when it saves you five minutes a day and it
can save you somebody half an hour a day, that's something that you think, hmm, I can
take this to the next level, and you start having new ideas.
And before you know it, you have an app.
Yeah, exactly.
Eventually, you have a big thing you need to make.
As long as you're not, I mean, as long as you're not convinced that the thing you have
done is divorce from programming, which when we get to my story, eventually I will still
explain the connection.
Yeah.
Well, let's get to that because you, you said that you were writing programs when you were
eight years old, but you were convinced that you didn't have the aptitude for it.
Can you tell us that story?
Yeah.
So I guess it's pretty connected to this other thing,
which is, so I was a kid.
My father had done submarine simulators
for the Air Force, like in the 60s.
So, and he, I would say that that was not considered
like a programming job.
It was more of like a hardware work with your hands,
but like, you know, building a computer
or anything like that involves a certain amount
of technical acumen.
And so my father in the 80s, which is after I was born, I'm very old, I know.
My father in the 80s had his own hardware shop where he would basically repair people's computers.
And because of that, when I was born in 82, I already had a computer.
I had an 8088,
and that was the computer. We didn't have so much money, so that was the computer we had for some number of years, well after its theoretical shelf life.
And I don't know if people remember this, but DOS, like some middle versions of DOS started
coming with QBasic. So there was GWBasic, which was like this thing
where you have to type the line numbers.
So you'd be like 10, print whatever, 20, print whatever.
And then you always would like,
the rule was that you would always leave
some amount of space between your line numbers
in case you want to go back later and add more things.
So if you're like, oh, I want to put something
in between the first line and the second line,
you better have enough numbers left so you can go 15.
And like, if you became like a serious business GW Basic programmer,
you would do like 100 or 1,000.
So you always would have enough space to put stuff into.
That's funny.
Yep.
So GW Basic was a thing.
But I would say because of the line number thing,
it was not real.
It felt, and there was no, you couldn't like really,
most GW Basic environments were like a REPL there. You couldn't really save it. It was, or it was no, you couldn't like really, most GW Basic environments were like a REPL.
You couldn't really save it.
It was, or it was hard to save it and load it back again.
So GW Basic was more of like a thing people would get a,
if you had a Commodore 64, they would like get some magazine
and they would type in the game and then they would play the game.
Right.
That was, that's sort of like the mythos of GW Basic.
But QBasic was uh stands for quick
basic came with dos and you could like write your file and save your file and you have five and you
were now in like a full screen thing that ran your program so you could and it had like so it didn't
have a lot of control structures but it had go to which is actually sufficient to do most things
and it also had like a few graphics routines you you could make circles and squares. And I would say it was probably at a similar level of abstraction as
like canvas, the canvas tag, you could draw some basic shapes. And if you want to draw anything
complicated, you had to animate by wiping the screen and writing drawing the next screen. But
as canvas shows that is, in fact, sufficient to do all the things. And I guess when I was eight
years old, I found this
fascinating. And I made like some very basic games. The one I remember the most was called
Carnival. And it was like the most the thing I was most proud of was I had a game where like
something would move back and forth along the bottom and you had to like hit space bar to get
it to go into like a basketball hoop that was painted at the other side of the screen. And like
my family would play it
so i in retrospect the fact that i was using go to and like effectively canvas primitives to draw a
animated graph game when i was eight seems like probably a good indication that i know how to
program but at the time i was like okay well q basic has first of all it's interpreted so there
i didn't know that but i knew you have to hit f5 and it was annoying. I didn't know how to make the game not require
that. Although eventually I discovered that there was like a QBasic compiler, but I didn't really
know how to reliably have access to it. And I was like, oh, I want to learn more things. This seems
like a little, it's like pretty limited. And somebody gave me a book on C, which was like, oh, I want to learn more things. This seems like it's pretty limited. And somebody gave me a book on
C, which was like, oh,
here's the next step. After QBasic, what you should do is
learn C. And I was like, I literally
have no idea anything that's going on here.
But what I've been told is
QBasic is a toy programming language. C
is like the real programming language. And
obviously, since I have no idea what's going on with the C
book, which I think the book I was given was the
KNRC book, which pretty much is not an appropriate book for a small child.
I was I basically just came to the conclusion in my head that like, oh, I can I'm good enough to like program toy things.
But I basically do not have the aptitude to program anything serious.
There was no like good bridge between the stuff that I already knew how to program and quote unquote real programming. So at that point I gave up programming for like a while. Um, and
then I went back, I would say like in my teens, I programmed again, and this is like embarrassing,
but the thing I was programming was a star day calculator for Star Trek. Um, I've like found
online. I was online pretty early I found online
like a fact that
someone had done all the work to reverse engineer
all the stardate
algorithm for every single
era of Star Trek and I was like oh that seems cool
let me like make software out of it and I use
Visual Basic 6 which a lot of people
still have fond memories of
and Visual Basic 6
is basically like
a WYSIWYG
programming environment,
but you can double-click on any
part of any UI element,
and then you can say, effectively attach an event
handler to that
UI element by just
visually, right? So you
have to write the code to say what it does,
but you don't have to do the work of
wiring up the event handler to the visual element and the actual visual work is done using a
UI, right?
So you could sort of think of it as interface builder, but in interface builder, you still
have to do like write the code in the text part and then you have to like drag the arrow
or drag the line from the UI element to the text part, and then you have to like, draw, drag the arrow, or drag the line from the UI element to the text part.
Whereas in Visual Basic, you would write do the UI part and
then instead of having to like, also write the text part, you
would just double click on it and it would effectively
scaffold out the part that interface builder makes you do
manually. So it was it was very, I would say it was very easy to
use. And I was able to like, the algorithm, as you can imagine, is not very complicated.
So I was able to write it in Visual Basic.
And at that point, it felt pretty similar to my QBasic time, right?
It was like, it's pretty clear that it's like a toy environment.
But now the toy environment has gotten better. So I can do, like, I can basically get more done
with the amount of work that I already,
with the simple programming that I already know how to do.
But then equivalently, I eventually was like,
this seems pretty annoying.
I want to do more things.
And what I was told at the time was like,
well, the next step is to learn the Win32 API.
And I went to look at the Win32 API
and if anybody wants to look at it,
it is horrifically terrible.
Well, it's not terrible.
It's just equivalently very low level.
Very dense.
Hard to understand.
Similar to your C, to the experience you had in your A with C compared to GW.
Exactly.
And the Win32 API is basically like you have this top-level switch statement,
and every action that happens is basically running,
like telling that switch statement what to do. And it has to do low level things like
drawing boxes. And it's just, it's basically a massive loop between, uh, like the nice
environment, visual basic and the quote unquote serious programming that you do with one 32.
So again, when in my teens, I was like, well, that's, that was fun and all, but it doesn't
really, I don't seem to have the aptitude.
I'm no longer eight years old.
Basically the same thing is happening.
So I'm pretty sure I just should not do this.
So I guess when did the professional time happen for you then?
Was it a struggle for you?
I know you talked about struggle and aptitude there.
And you said that the idea of struggle doesn't mean that you don't have the aptitude.
It's anybody who's learning the program is going to have some sort of struggle. That's a
good thing. So what was the, how did things progress for you to get into professional,
I guess, programming? So what basically happened is I just gave up. I was like, okay, I like
basically know the equivalent of like very, very light scripting, but I can't really program. So
I, and in college I could have been a CS major, but I was like, it doesn't really seem like it's for me. And I ended up doing a bunch of other
stuff, accounting and journalism. But journalism in college ended up teaching me design. So I
learned, I basically did a newspaper design in college. And then I got my first job out of
college as a web designer from a person who didn't really understand that web design and print design
are not the same thing. And I also didn't know that at the time, as it turns out. So I was basically
given a job to do what I thought was a like pretty straightforward design task that ended up actually
being a programming job. And I was like, oh shit, if I want to keep my job, I actually need to learn
programming. So I had no choice. And I was, the the way the reason i ended up succeeding was because i
was given a couple of applications that people ever had already written um in cold fusion and
php and i was told like please update this application for this year so it was a non-profit
and the non-profit had like these dinners and they were like okay we will we basically have
an application that lets you pay for your dinner ticket and perhaps ask for additional items.
It was like not totally trivial.
And they were like, please update it for the changes that we want to make for this year.
So, of course, I was able to go in there.
And either in ColdFusion or PHP, I was able to make the changes because that kind of thing is not so complicated.
But often when I was doing that, I was like, oh, well, it seems annoying that you have this giant form.
And like every time you click next, you have to go back to the server. So maybe I could find a way to make it
more interactive. Again, I was like thinking about it as a designer. And I was like, this was like
right around the time Ajax had become a thing. I was like, oh, there's this Ajax thing. I should
probably learn that. Maybe that will help me do it. I was like very enamored at the time of like
the idea of constructing forms that progressively revealed
themselves based on the previous information. And the thing that I really was most angry about that
still annoys me is the fact that you would have to put in your city, state and zip code a lot of
times. And I was like, if you put in your zip code, you can derive the city and state. Why are you
making me type all these three things? And I basically didn't really know how to do it. But I
was like, I'm pretty confident that this is not actually necessary was that the first
library you wrote then um i wouldn't say that it was a library but i figured out how to like
download at the time there was no libraries really for any of the languages i was writing
but i was able to like download an access database of the city state and zips and i was able to figure out how to get it to
reveal its secrets to me um and not force me to do the thing and then i learned ajax so i didn't
have to reload the page and things like that so that was sort of how i got started with programming
um but then i was like well i can't like learning ajax quote unquote ajax all by myself seems very
hard i'm i the internet was not so good at the time. There was experts exchange was like the
most popular thing. And it was like constantly asking me to pay. I just didn't feel like I was
getting a lot of traction. So I took this class by Thomas Fuchs, which is like about Ajax. And
I effectively, by the time I took that course, I knew most of the things that was in the course.
It was actually a course on prototype. But at the very end of the course, he was like, by the way, if you want this to be easier,
you could use Rails. Like Rails will be easier for you. So you might want, I didn't want to
force you to use Rails in this course, but probably be a good thing to investigate.
And I don't know how many people actually did in that course, but I was like, well,
if it's easier, sounds better to me. So I'll investigate that. And then I did. And I would
say that was basically the beginning of me being an actual programmer.
I was like, oh, well, now I was able to basically build a CMS system for that company
without really knowing how to program, which I don't know how true that is of Rails today.
I think Rails has become a trickier starting target.
But at the time, there weren't even migrations.
You just go into a WYSIWYG editor, change your database schema.
You'd make whatever changes.
And then it was just like you have to know HTML and like a very tiny bit of Ruby to do things.
And I was and I sort of knew Access.
So I had like the right proximate mental model for it.
And I was I was able to do that.
And that basically made me feel like massively empowered.
Like in the past, every time I had done this toy thing, it was like
basically not a real application. I wasn't getting paid for it. And it was like, whether or not,
you know, the CMS I built was like serious programming or not. The fact of the matter
is somebody who's running a real business. Right. You built something real. Somebody used it. They
put it in production. They made money from it. Yep. And so that's awesome. So that basically
made me feel like, well, whether like I don't, I didn't, it didn't make me have to think a lot
about whether or not the programming I was writing was real or not, because it was being used for something real.
Does that tailor into your opening the open source box, so to speak?
I mean, you're pretty prolific in open source now.
It sounds like prototype Thomas and Rails was sort of an entry point to you.
What was the very first thing you
did in open source that started to take you down the road you've been going down?
So the first thing actually was not rails. The first, uh, I basically learned rails and
jQuery at the same time. Um, the, the, I, I found jQuery because I was looking, there was some,
uh, something that would like have been the equivalent of hacker news, but I can never
remember. It might've been like, um,
it might've been slashed off,
but I actually don't remember it being slashed out.
So I don't really know.
There was some website that was like,
you should check out this UI.
You can like,
you can construct a thing that is like the finder that lets you select icons
from a grid.
And I was like,
wow,
that's pretty cool.
I didn't know you could do that on the web.
And I looked at it and it was built using this library called interface,
which was built on top of jQuery at the time.
It's like the precursor to jQuery UI. And I was like, wow, jQuery can do that. That seems pretty cool. So I looked more into jQuery. And the
problem I had at the time was that jQuery, while very cool, was basically documented by John's
writing documentation on the wiki at his leisure, which as you can imagine, is not very likely to
be up to date, especially
since jQuery itself was pre 1.0. It was like, not only was it pre 1.0, it just had no versions.
Like every so often, John would like put a tarball or a zip file on his blog, like here's
a new version of jQuery. And I was like, I really think we could do better than this.
What if instead of you writing things on the wiki, what if you gave me at the time, like this was 2005, so don't laugh.
But what if you gave me an XML dump?
Like what if you gave me an XML dump of the documentation?
There already were like tools that would automatically generate inline docs.
If you give me the XML dump, I will basically do the work to make a like automatic website.
And I used XSLT at the time to like convert it into something visually appealing.
And so I was basically, I like kind of owned the documentation effort for a little while in
jQuery, just because there was no one else doing it. And I was, I was feeling like, I guess I know,
I have no idea how I, why I felt like this, but I felt like this should be easier. There should be
basically everything is in place. Someone just has to like glue it all together. And then if you glue it all together, then you
have automatically up-to-date documentation for every version of jQuery ever, just because someone
wrote inline docs. And I felt like that was cool. So I would say that that was probably the first
real open source thing I did. And the reason, like the gateway drug for me was just, that thing was
massively high leverage right like before
people would come and they would look at jQuery and they would bounce right off because they
couldn't figure out what the what the rules were but then as soon as there was any documentation
on the internet all of a sudden tons of people stopped bouncing off of it yeah I can recall
those days actually I mean I remember just there was a couple other different things happening
around then obviously product that was around but it felt as as someone who came from a front-end perspective which you could probably lament a little bit about this with
an html css background sass wasn't even invented yet or it might have just been invented but it
wasn't it was only a ruby you know very different than it is today it you know there was no easy
path that i could find at the time into this javascript world and uh but i remember when
jquery had docs,
it was actually quite a bit easier to get into it.
And I was like, okay, now I feel like I can actually do something with this.
There was this library called Rico at the time,
which I actually was like, Rico actually seems pretty cool.
I want to use it.
Like the demos are really good.
And every single time I would try to use it,
I was like, I literally have no idea how you're,
it seems cool and I can look at the source,
but I do not know how you're supposed to actually use this.
And I bounced off of it a bunch of times.
And I guess that was sort of the epiphany that I had as a person in the world.
Like if there are things that are so close, like think about it.
I basically didn't know how to program at all at this point.
I would say that I effectively had like the level of programming that you have when you're
writing Excel at this point. But I was able to figure out how to take documentation that other people had
already written and tools that other people had already written to generate the XML version and
then XSLT, which was pretty declarative and didn't require a lot of programming skill. And I could
take all that and put it together into a thing that enabled a ton of people to use jQuery.
And I felt like, wow, that is massively high leverage.
You can change a lot of people's lives easily,
quickly with open source.
And that's sort of been my guiding worldview ever since.
How do you take people who basically don't necessarily,
who are bouncing off of things,
and how do you enable them to do more?
It's rough.
It's an interesting thing too.
Hearing your background on automation
and it seems like you have a true passion
slash addiction with it.
I'm not really sure how to phrase that,
but it totally makes sense that you wrote Thor.
Yeah.
Yeah, Thor was definitely,
I wrote Thor like after writing a bunch of CLIs
and saying, why is this?
I guess one of my things about open source in general is there's so many things that are like so close.
Like somebody just has to, the difference between where we are now, which is like almost nobody can use it.
And it being usable by a huge amount of people is often such a tiny gap.
Someone just has to spend the time.
And so many people just like, there's such a failure of imagination so often about what the gap looks like. So a lot of times
people will try something and they'll have that experience for some reason or other. And they'll
just say, this entire idea is a bad idea. This entire idea is dumb. And really all that is
needed to make it feel better is like small things that me 10 years ago could have done.
Right.
Someone has to just do it.
So one more thing we have to mention, obviously, we can go on and on.
You have a deep past with programming and open source.
And obviously, we'd love to hear all about your past.
One thing I want to talk about before we go into our first break is what I'm calling,
I think might be your brand purpose,
your personal brand purpose.
And at November this year,
I think it was either this year or last year,
I'm not sure which was,
but you said sometime during your talk,
and I took this note down,
I don't think you made it explicit
that this was your brand purpose,
but I want to ask you what you think about this
before we transition into this break.
But you said build things that empower people.
And it seems like, you know know rewinding back to your beginning uh through to your initial
comfortable feeling of being a programmer and then moving into open source like you've been doing
that that's been your mantra that's been what you do what do you think about building things
that empower people i think that's i would agree i also agree that I was it was this year. I also agree it was not so explicit as a like me waving the flag around it. But I I think that open source does is does a lot of things for a lot of people. And I think it's good. But for me, the main thing to do with open source is to find places where the gap between someone being empowered and not being empowered is small and just closing it. And unfortunately, at this point in my career,
the gaps that are still a good cost benefit for me
are usually much bigger,
but they're still, like, I'm never going to be,
I mean, maybe someday,
but right now I'm never going to invent
the Rust programming language
because it's like the level of low levelness
is too much for me.
But the gap between Rust being a good thing for servo,
like a web browser and being good for Ruby programmers
is actually much smaller than the gap between like,
like not having Rust in the world and having it at all.
And so I, a lot of what I try to do in my,
like with my open source work is to try to find things where either,
like the fact that
something hasn't been invented yet is for a stupid reason, like Thor or Ember, or, um,
there's something that exists in the world, but it's just inaccessible for some reason,
like the jQuery doc story or rust, like thinking about, okay, what, what can we do
to make this a thing that a lot of people that a lot more people can use to do something
powerful? And I just think not enough people think about it. And more concerningly, and this was also
in my November talk, a lot of people are just pretty cynical about the whole thing. A lot of
people are pretty cynical about the level of complexity in programming. A lot of people are
pretty cynical about how likely you are to find a good solution for these things. A lot of people are pretty cynical about how likely you are to find a good
solution for these things. A lot of people are pretty cynical about the power of these things
in the first place to change the nature of what people can do. And I'm very optimistic about
being able to change the nature of what a particular person can do at a particular time. But that requires believing in it. And I think, unfortunately, a lot of people
don't always believe in it. Well, I think on the other side of the break, we want to take
these thoughts, the empowering people and the finding the gaps between empowerment and
non-empowerment, and use those to cast a light on your work with JSON API.
So we'll ask you about that on the other side.
We'll be right back.
We know you listen to other podcasts.
Don't worry.
We're not upset.
We know the Change Log isn't the only podcast in your list. And you know what?
You may have heard advertisements on other shows about other hiring platforms and other
places like our friends at TopTow. But let me tell you, there is no match
to how invested TopTow is to both sides of the equation. You have companies out there who need
great engineers, great designers, and you have great designers and great engineers out there
needing great opportunities. And TopTow plays both sides of that fence very well, helping
make sure the right kind of opportunities are there and the right kind of developers and designers are there.
And if you're a CTO listening to this or someone on a team who knows you need to expand and add more people to help reach your goals, TopTow is the best place to go and find the best designers and those best engineers or best designers looking for great places to do great work and freelance and travel the world and have a lot of freedom, but also have the same kind of support you would want from working somewhere, TopTal is that place.
You can blog with them.
You can travel the world with them.
They have meetups all around the world.
They absolutely love to encourage and to support and help their developers and the designers that are in their network all around the world grow and be better and do better head to toptal.com to find out more
that's t-o-p-t-a-l.com once again t-o-p-t-a-l.com but if you want a personal introduction on either
side of the fence whether it's an introduction to find the best designers and developers, or if you're a designer developer looking for great opportunities, get in touch with
me. I'd be glad to give you a personal introduction. Email me at adam at changelog.com. And now back to
the show. All right, we are back talking with Yehuda Katz about JSON API. And before the break, we laid out, Yehuda, how you just try to do two things.
The main purpose is to empower people with open source.
And kind of the way you go about that is by finding these gaps between empowerment and non-empowerment and hopefully find ones that are easy to bridge.
Probably they're getting harder and harder to bridge as you get more and more advanced. And we wanted to take those thoughts and apply them to your work behind JSON API,
because this is something you've been involved in since, I think, May of 2013.
So can you cast that light into JSON API and why it's something that you think needs to exist?
Sure. So before I do that, I think it's important to understand, like, what does empowerment actually mean? And what I would say, one thing that I think it's important to keep in mind is that if somebody already knows how to do something, making it ways in which that kind of thing can be empowering.
But the most empowering things are taking something that somebody does not know how to do at all,
and then making them be able to do it. And fundamentally, in order to do that,
you have to use abstractions that hide some of the details. Because pretty much nobody is going
to say like, oh, I would like to build web
applications.
So before I do that, let me learn HTTP and RAC and all this stuff, right?
They're going to, they want to, the shortest path to empowering them to be successful,
like I was when I started building that CMS for my company, is to hide a lot of those
details.
And this is actually a fundamental conflict in open source and software zeitgeist is whether abstracting away details is a good idea.
So, for example, Joel Spolsky has this famous blog post from many years ago called The Law of Leaky Abstractions.
And the idea behind this blog post is like you can never successfully abstract anything.
It will always leak. It will always leak.
So don't even bother.
Everybody should just learn everything.
I think this is just a silly worldview in part because we have so – over the past 30 to 50 years, we have succeeded at abstracting so many details that it just seems like self-evidently incorrect.
Like whether or not TCP, which was his example in his blog post, one of
them was TCP, which I think is a hilarious example. The fact that TCP might occasionally leak
doesn't necessarily mean that TCP is a bad abstraction. It just means that sometimes,
if you're doing sufficiently advanced things, you will need to learn TCP to get the sufficiently
advanced things done. But that doesn't mean that we should not have TCP and everyone should just use UDP
and build their own thing.
Although there's a good website called notcp.io,
which is basically making a joke
about this entire phenomenon
of people not believing in abstractions.
And I guess for me,
if you're looking to empower somebody,
you have to figure out what parts of the everyday,
the 80-20 job that someone does,
what parts of it are about things that are not very important. And interestingly,
what I've also found, what I found is that when you do that analysis, quite often,
it's a lot of the thing, right? It's a lot of the stuff people do, like on a day-to-day basis,
if you look at like what someone does when they write a backbone application,
a huge amount, a huge percentage of that is something that you, that Ember doesn't make you think about, or even React
or Angular doesn't make you think about, right? And I think, so I think when you're first looking
at empowering people, what you often discover is that's sort of the natural zeitgeist, the natural
worldview of how people think about whatever task it is that they're setting out to do.
And what people think is like core and fundamental to the tasks they're trying
to do is really pretty abstractable. I think if you're if you're going down
this path it's important to remember that when you abstract something it's
you have to leave escape valve so people can go and break out into the into the
level below if you're an advanced user and you discover things are too limiting
and I think Rails did a pretty good job at this and Ember does it with things into the level below if you're an advanced user and you discover things are too limiting.
And I think Rails did a pretty good job at this.
And Ember does it with things like didInsertElement,
which allows you to basically write jQuery code in the middle of your component, right?
And React has a similar hook.
And I think those kinds of things are very important.
But if you make your escape valves reasonably well-designed and designed to be very, very broad,
what ends up happening is that people who are new can get a lot done without having to think about the escape valves, but people who are
a little bit more advanced are able to use the escape valves to get the job done. So I guess
this is how I think about empowerment in general, is that your job is to take away a large percentage
of the thing that people think is core to the experience of doing it. And I've been giving talks about what became JSON API for a long time.
I gave a talk just about this in 2012 at RailsConf,
which I think was called Rails the Next Five Years or something like that.
And with JSON APIs, I think there's so many details about it
that have to do with format and they have to do with protocol.
But somewhat interestingly, the format and the protocol cannot easily be decoupled.
So if you say, well, you know, we're going to make a format.
We're going to, the way we're going to make it, we're going to empower people to build JSON APIs more effectively without as much hassle.
And we will allow many, many more people to build JSON APIs than can do it now. If you think that you're going to do that with a format, the problem is that the format
interacts in non-trivial ways with the protocol. If you do the opposite, if you say, we're going
to do the protocol, which usually people are referring to when they talk about REST, they say,
we're going to say, we're going to define the protocol. The problem is the protocol interacts
in non-trivial ways with the format, right? And additionally, a lot of times people will say things like,
oh, just use REST.
But they don't actually spend any time describing what it is.
It is assumed that if you're a programmer,
you can figure it out.
And number one, a lot of the people
we're trying to empower in the first place cannot.
But number two, the actual definition of something like REST
is sufficiently ill-defined
that it doesn't help even more advanced developers who say use REST, they have no idea what you're
talking about. And I think a good example of this whole attempt failing was active resource in Rails
where DHH basically thought, and a lot of people thought, like, oh, well, like Rails is pretty
conventional.
So what we'll do is we'll say you can connect to a quote-unquote conventional Rails server.
And actually, I made the same mistake in the beginning of Ember Data that led me down this
path, is that Rails is sufficiently conventional that you should just be able to connect to
literally any Rails server and get, like, roughly enough information to do the job.
The problem is that it turns out that the specification of sufficiently similar
Rails application is impossible. And people do all kinds of random stuff when they're building
Rails servers. So we also, because the same reason that caused DHH to make the mistake about
ActiveResource caused us to make a similar mistake in the beginning of Ember Data. We said,
you know, what Ember Data will do is it will hard code to quote unquote Rails. And of course, that doesn't stop you from, I was overly,
I made too strong of a statement there. It doesn't stop you from using the same conventions in any
other system. So, you know, there's like the REST framework for Django and Node, of course,
lets you do whatever you want. So it was more like, let's say that the Rails conventions are
the Ember Data conventions, and we thought that that would get us all the way. But in practice, what ended up happening is that
questions came up, like, should I have a root or not? Should the name of my resource be in the
root or not? That's a simple one. How do I include associated resources? What if I want to include
just the links to the associated resources or IDs for them, and then lazily load the rest
ones. And these things come up very quickly, like within a very short amount of time. It doesn't
require you to be an advanced user to encounter some of these questions. It just involves you
trying to do anything, even pretty simple with your UI. So we ended up realizing pretty quickly by 2013 that treating Rails as a spec was not working. And so at RailsConf
that year, I was bemoaning this. I give it a couple of talks about it. And Brian Ford
from Rubinius, his name is now Brian, I don't know how to pronounce his last name, but it's
like something like Sheree, came over to me and he said, like, the world needs to have a better specification.
Could you write one down?
And I said, well, what I'll try to do is I'll try to extract the implicit specification
that Ember Data had.
Like, basically, even though Rails has no spec, Ember Data has to have said something.
So Ember Data has implicit spec.
Let me try to write it down.
And so I wrote down the first version of JSON API was just effectively an extracted Ember Data spec.
And actually, people were pretty excited about it.
But one thing that we noticed after that was that even Ember Data was pretty wishy-washy about a whole bunch of details because of the fact that we were trying to work against a pretty wishy-washy server.
So over time, what we realized was that we needed to get pretty rigorous.
We needed to have opinions about all the details.
And I actually learned something important about JSON APIs and wishy-washiness, which
is that a big part of the reason it turns out that people are so wishy-washy in the
way they define JSON API is that JSON APIs
are maximal bike-shedding opportunity. So basically everyone looks at a particular JSON API and they
have a visceral response to the aesthetics of it. So if you are trying to get a lot of adoption for
your JSON API description in some way, you're trying to figure out like, okay, how do I make something that a lot of people will not respond negatively to? And that basically means being
pretty wishy-washy. You say, oh, you could do this, or you could do that. If you don't like this,
you can do this other thing. But the problem with the head is that now the clients have to be willing
to accept a lot of different things, which makes it harder to write clients. And what we discovered
somewhere in the middle of the JSON API journey
is that no matter how much people claim that they're willing to adopt something
because it has the right aesthetics,
the real value prop of something like JSON API is having good servers and clients.
So we were getting a lot of people to not whine a lot
about the exact aesthetics of JSON API by being pretty wishy-washy.
But then every person who was actually trying to write clients
would say, you know, the spec seems fine,
but I don't know what I'm supposed to do here.
You know, the server says you're allowed to use
whatever status code you want here.
Is that actually, like, what does that mean?
What if the user gives me, like, a 422?
What should I do?
Yeah.
Let me just real quick state something
that's perhaps obvious to you and I,
but just to be super clear with everybody, that the JSON API, and I'm using the word just to limit its scope, not its usefulness, but it's JSON API, the project that you've been working on that we're talking about.
It is just a specification of how you ought to build JSON APIs, correct?
Yes, confirmed. So just to be clear,
it's the only specification that I know of that tells you how to build JSON APIs,
both the format and the protocol.
In other words, it tells you the HTTP semantics,
but it also tells you
what the shape of the inputs and outputs is.
Of the data.
Okay.
And as far as I know, I'm probably wrong.
There's probably some other thing that exists in the world.
But as far as I know, I'm probably wrong. There's probably some other thing that exists in the world. But as far as I know, the popular alternatives that people say, why, you know, how or something like that, those things are all either about a protocol or they are about a format and not both.
OK.
But yes, I agree.
In retrospect, probably the name JSON API has some confusing ambiguity here.
And I apologize in retrospect, in hindsight.
It's very Google-able, though.
I mean, you can Google it
and find it. So that's a good thing. Yes. I mean, perhaps surprisingly, but it is true that it is
very Google-able. So anyway, so we basically went through this back and forth where at first we were
pretty responsive to the aesthetics people. And eventually we realized that the aesthetics people
were not actually adopting JSON API, but the tool builders were important for the adoption of JSON API.
And they were telling us about all these issues.
So about a year ago, me and Dan Gebhardt, who was probably the main, like the main editor
now of the spec, sat down and in a few face-to-face sessions, we hashed out what ambiguities people
had reported that were making it hard to build tools.
And we just made calls about all of them.
We just said, the answer is this, the answer is that, the answer is X.
And the consequence of doing that is that now JSON API is very trollable from an aesthetics
perspective.
It's very easy for someone to look at it and say, I don't like that.
A very common thing people say is, is this WSDL all over again?
Actually, it is not WSDL because it does not have a big schema type thing and has nothing
to do with that.
But it's very easy to look at it and say,
you know, I thought the point of JSON
was to make this a very like human consumable format,
but now you have like this attributes key
that has a hash of attributes.
Why do you need that?
Well, it turns out that the reason you need that
is that tools need to know the difference
between attributes and associations.
And saying that you'll do that by hard coding
in every client is not very good for tools, right?
So there are some things that we did to try to make ambiguities less.
We tried very, very hard when we made the calls to not gratuitously add things for tools that just, you know,
that basically would make it a binary format or something like the ugly XML formats that people are used to seeing.
But at the end of the day, there really are some ambiguities
that you would like to get out in the spec.
And that's sort of why I tweeted that as in.json thing,
was that JSON API is not really fundamentally about being a,
it's not about being a bespoke hand-rollable format.
It's about being,
which I think is how a lot of people think about JSON APIs, right? They look
at them, they try to construct the
most optimal, most
aesthetically pleasing shape of their JSON.
I think that
the JSON, the
API, the hand-rolled
artisanal versus XML
and WSDL and these things, I think
it's this
kind of a yin and yang that we go back and forth between,
similar to static versus dynamic type languages
where you want the freedom,
but then you lose the tooling and these conventions.
And I think it's tricky ground
because everybody has opinions on it
and it seems like there's no black or white answers either way.
So I think the interesting thing for Rails is that DHH seems to really come
down on the artisanal side on Jason.
But I think I, as a programmer learned the right answer to this question by
looking at active record, which had a fairly similar question early on.
It basically said, okay, DHH has figured out that the way you could produce the
best programming experience is by not allowing you to choose whether or not your primary key is called ID and what the foreign keys are called
and all this stuff, right? We're going to tell you what the answer is. And maybe a DBA is going
to look at that and say, oh, that's so disgusting. I can't believe the SQL that's being emitted.
And you're supposed to say in Rails, I just don't care. I'm not interested in that level
of abstraction. And the story with JSON API is pretty similar, right?
It says, what I want to do is I want to describe my models.
And yes, there's going to be some impedance mismatch,
just like there is with SQL.
But I mostly want to think at the model level of abstraction.
I mostly want to think about the model in something like Ember
is pretty similar to like the resource concept in Rails.
It's not about database tables.
I want to think about models.
And the fact that there's maybe, you know,
maybe an extra attributes hash or extra URLs,
or maybe the fact that every single thing
has to have a type and an ID, right?
All those questions are just not,
certainly you want to make sure
that the format is debuggable.
I think that's one of the things people don't like
about more binary formats. It's like you look at it and you're like, as a human,
I cannot figure out what this is even trying to say. And I think that's very bad for debugging.
But how many, you know, do I have a type associated with every single one of my
records that I have sent over this format? Those are questions that are legitimate aesthetic
questions, but they are analogous to like, I would really not, I don't want ID to be my primary key name. They're just not very important. And the more you hold on to saying
that it's very important that you care about that, the more we push back, we leave open the big gap.
So this is the empowerment thing, right? There are people who want to build stuff on the web
and making them learn about handcrafting their JSON and HTTP status codes and all this stuff.
Even me, when I'm trying to write apps, I don't want to be thinking about this stuff.
And the fact that this is just a part, the normal part of the work of building applications means that there's just a bunch of people who can't do it or can't do it at any particular point.
It's a hard weekend project.
It's a hard side project for a job.
And basically the job of JSON API
and my philosophy in general
is to say how much of that actually matters.
And in fact, that was also the point of Rails, right?
Is to say how much of the stuff
that people are spending all their time on
actually matters when it comes down
to what your application is doing.
Maybe performance really matters.
Maybe you need to handcraft this thing
and you can't use JSON API,
but why don't you figure that out after you figure out if the
thing you're building even matters?
I think that
there are other aspects. We talked about
aesthetics. You mentioned one, performance.
The other thing is how well your
client libraries, whether
it's Ember or an iOS library or whatever
it happens to be, how well that
maps on top of
what JSON API is speaking both in protocol
and in format.
So has there been a lot of give and take between not just suiting Ember's needs, but suiting
an iOS library, suiting React, or these types of things?
Yes, absolutely.
So the evolution of JSON API was that it started off being pretty extracted from Ember data.
And I agree with DHH,
extracting these things is way better
than trying to invent them out of your head.
So it started off already being pretty useful,
even just to point people at,
as if you're trying to build an Ember data thing,
it is now documented better.
So that's where it started from.
Bonus, yeah.
I mean, it was sort of documented,
but mostly in the source, mostly in my comments, right? So that's where it started from. Bonus. Yeah. I mean, it was like sort of documented, but mostly in the source,
mostly in my comments, right?
So that was already helpful. But we started getting a lot of people who weren't
using Ember to say, oh,
I'm trying to write an iOS client library. We already have
iOS client libraries. There's already
like a Java server
project that lets you build JSON
APIs. There's a Node project. And so
increasingly as people try
to use JSON API for things that were not the Rails and Ember combo, people ask for more things. And
really, the philosophy of JSON API at the end of the day, like today, is that JSON API is just
describing an arbitrary graph of objects serialized over the wire. And what that just means is that
you have a bunch of objects, and they are linked together in some way. In order to link them in a serialized form, you need some identifier. And we chose to use the type and ID
identifier as the key, in large part because servers pretty much already have that information
readily available. So a big part of JSON API is making it possible to build a JSON API server
with existing infrastructure, like Express or Rails. It's like not trying to say that you need to start over from scratch and build entirely
new infrastructure.
But the fact that the type and ID are really just talking about linkage between different
objects means that you could have like one type and a UUID as your linkage.
If you're trying to serialize a completely arbitrary graph of objects, you could just
have like the type is always object and the ID is always a UUID that you generated. That would be just fine.
And so the way I think about JSON API these days is that. It's a graph of object serialization,
which really is the most general. It's hard to imagine things that you're trying to serialize
that you cannot express using that serialization mechanism because it's really quite general so who all was
involved in this process and from you know May of 2013 is when you first kind
of pulled it out of that RailsConf conversations and Ember data at the time
to now where like we said earlier it shipped stable 1.0 in May of 2015.
We're wondering where was W3C in all of this?
Who was involved in the specification?
Was it just yourself, Steve, and the other person mentioned?
Or were you able to get kind of a groundswell of support?
Because I was a person who, when I realized you guys were working on this,
I was like, yes, this is the kind of thing that needs to exist.
I'm glad that smart people are working on it. And when it's done, I'm just going to use it.
And that's one last thing I have to think about as I build my application. So
that's where I fell down on it. But were there other people that thought,
ooh, this needs to exist and we have different needs than Ember. And so we're going to get
involved. Was there a lot of discourse? Yeah. I mean, I think Dan has some number, but there were like, there were threads about seemingly trivial things that went on and
on and on for hundreds of comments that occasionally I had to weigh in on as like the
final arbiter of some of these things. I would say that basically it had like a sort of on again,
off again story until pretty recently. So I put out the thing.
I kind of just said, okay, here's the thing.
I'm going to keep quiet for a while,
let people try to implement it.
And we got like probably like around a year later,
we got a good enough amount of initial feedback from people who were trying to put it into production,
which actually was pretty cool, right?
So we had like, you know, maybe like a dozen people
who were legitimately trying to make JSON API, the original draft, a production thing, and all of them hit various
issues that they reported. So about a year later, we said, okay, we, I think we have enough
information now to sort of build the second draft. And the second draft was largely driven by
these ambiguity reducing ideas, right? So we were trying to find places where we
unnecessarily said this is a should or a may. And in fact, there's really no reason for that. The
cost of the ambiguity is much higher than the flexibility benefits. So let's just make that a
must, or let's just remove that and say that's a potentially a future direction for an associated
spec. So we did that. I was actually part of the W3C tag for a lot of this process.
And my participation in standards in general gave me a pretty good sense of like what the
process of building a standard should look like.
But there was no real standard organization, quote unquote real.
There's a funny thing, which is that the WhatWG is sort of perceived to be a standards organization.
But all the real standards people are like, oh, these kids making standards.
You don't really know what it's like to make a standard.
And the JSON API process was sort of like that.
It was a pretty serious attempt to get pretty rigorous.
It uses the RFC story for how you say things are required, not required.
It tried to get pretty clear about what the requirements were.
It uses the media type extension mechanism
that the other specs we're working with use.
We did a good job of deferring things
and leaving open extensibility points.
We've said that it's a living standard,
but it's always going to be compatible.
So JSON API 1.1 is only going to be additive.
It's never going to break existing implementation.
So we've done a lot of the things that real standards like to do.
But no, there was no, there were no,
I mean, I guess I'm a standards person, right?
Well, you're saying the word real standards there,
and I'm kind of piggybacking off of something you said in the talk that seems to be some way for us to weave back into this story, which is it seemed like you had some sort of not so much a negative thing against asking for permission from a central authority, which tends to be WC3 or what you're calling real standards. It seemed like you and Steve and these other people involved in this were,
like as Jared said before, like anyone else who had heard you working on this
were totally excited because you're smart people, you're able to do it.
But it seemed like you didn't feel like you had to ask permission
to create this JSON API spec that would become the de facto standard
should the community bless it and adopt it.
So the first thing we actually did was file an IANA request for the media type.
So I would say that we did go through the minimum viable standards process, which was
to express what it is that we're doing and ask for permission to have a centrally registered
media type.
And I think that that was good.
I think that I think we did.
We did a few things that people who care about standards being standards would have requested. And I think that was the right thing. But I also feel like we're sort of there was there's sort of been this pendulum swing where historically people said we things should be standardized unless there are two interoperable implementations, which sort of implies that the implementations go first. But then like somewhere in the, you know, somewhere after 2000, a bunch
of standards work happened that was attempting to lead by standards. And I think there's still
some of that left, but I just don't think that works at all. And a big part of the pendulum
swing that I've been part of in both the TC39 process and to whatever
extent I've been involved in the W3C has been an attempt to remind the standards bodies that they
should not try to lead because that's not what standards are about, right? Standards cannot,
standards are about, you know, acquiring social consensus from implementers.
It's about having a nice space where you can work for W3C. It's very important that the space is not patent encumbered, right?
So any spec that's approved by the W3C, by definition, it's a promise that all the participants in the W3C, all the members, say that they will not sue anybody for patents because of the technologies that are in the specs.
That's pretty important, right? So the standards bodies are useful, but they really do need to be facilitators, specifically facilitators and implementers and users, and not try to have standards people do the work.
That's a really good point.
I mean, because that's what you're saying here, too, is that, you know, going back to Rails, it was extracted from Basecamp.
This was extracted from what you were doing in Ember's, in Ember data. So it seems
like people who are actually doing, not so much that standards aren't doing, but I like your idea
of how they're providing this safe environment to come to consensus of what's out there, what's
working, why is it working, who trusts it, why do they trust it, and bring it all into one thing
that is under some sort of governance that's sort of community bound in a way. And I think like, so there always are implementers that are participants. I mean,
like TC39 is like a ton of implementers. And the real thing that you get out of,
like the thing that you get out of that process is not that the implementers are sitting around
in a room talking to each other about what to implement, but that the implementers understand
that they can, that when they implement something or when they propose something,
they have acquired the consensus necessary to feel confident that they can move that when they implement something or when they propose something, they have acquired
the consensus necessary to feel confident that they can move forward with it. And that process,
like some people think of that as being an inherently slow process. But first of all,
I just don't agree. I think if you look at like the ES6 process or even the HTML5 process,
we've made pretty good progress on features despite the need for consensus. But I think
people forget that there's a massive time cost in not acquiring social consensus, right? If you, you know, Chrome
maybe could ship something a little faster or Firefox could ship something a little faster,
but whatever gains you get from shipping something a little faster will be eaten up immediately by
the cost of convincing the other browsers to implement it. True. So, and sometimes you have
to go first, right? We always talk about this on TC39. Sometimes someone has to go first. And sometimes you have to go first, right? We always talk about this on TC39.
Sometimes someone has to go first.
And usually what will happen is somebody will propose something and everyone will say, I'm not really sure what the value of that is, but I'm not necessarily opposed to it.
I will implement it if it turns out.
But why don't you go first?
And then usually the person will go first and quite often it will happen.
They just draw straws or what?
No, it's usually the people who are the most confident that it's a good idea.
Well, that's a good place to take a pause.
We have one more break before the show comes to a close.
However, don't worry.
We have lots more to cover.
Compliance, resource discovery, some experimental things, future-proofing APIs, just to kind of touch on a couple of topics coming up. But let's take a break.
We'll come back and we'll dive deeper with you on JSON API and hopefully you'll love it.
So we'll be right back.
Our friends at DigitalOcean,
simple cloud hosting built for developers,
launched a new feature recently we absolutely love.
It's called Droplet Multi-Create
for easily launching up to 10 servers at once.
You can deploy multiple droplets
when you create your droplets
with the exact same configuration.
And this is a huge feature.
We absolutely love it.
Head to digitalocean.com.
And when you sign up,
use our code CHANGELOCK
to get a $10 hosting credit when you sign up.
All right, everybody, we're back
and we are diving deep on JSON API.
Yehuda, you've mentioned a few features offhand, but I thought we'd take a moment to kind of go through them one by one you're doing is you're serializing a graph of objects on your server.
What I mean by graph is just that it's possible for one object to be related to multiple of them, which is like sort of like a hasn't belongs to many relationship.
Or you could even have like a, you know, a post has many comments and the comments belongs to a user,
but that user might be referenced from other places.
So when I say graph, I just mean that people often imagine in their head that the data
structure is a tree.
They imagine like, well, there's a post and it has many comments, so that's nested inside
of there.
And then the comments have users, so you just nest the users inside of there.
But in the real world, those objects could be related to yet other objects.
So you don't necessarily want to continue nesting. And this is actually a core value of JSON API.
It's something that is sort of counterintuitive to a lot of people. But trying to make it a tree
has pretty serious implications on both efficiency and your ability to properly deserialize it. Like
what if you have the same user in multiple places and they don't match?
What does that actually mean?
Do you have to check that?
So one of the core values of JSON API
is that we're talking about a graph of objects,
not a tree of objects.
And there's often, not always,
but quite often a primary document.
So if you ask for post number one,
you want to know that you might get back a whole bunch of stuff.
You might even get like a summary of how many documents there are in the entire system.
But there's a primary response, which is the post object.
And then it might have links to comments.
And those links to comments could either be referring to other things that are inside the same response.
And that's commonly what people do.
But they could also be URLs that link to other documents that you can use.
So maybe you're downloading a list of posts just because you want to print the front page of your blog and you don't need the comments until you actually click on a comment.
You still want to say, like, if you ever need the comments, here's the URL for those comments.
So the idea is that what we're talking about is a bunch of linked objects. But unlike maybe the more pure rest or the pure like linked data
philosophy, the assumption is that you will quite often want to include a lot of linked objects
together. And there should be a very canonical standard way to do that. And it shouldn't always
have to be URL. So one thing that's a little problematic about URLs, first of all, is that
it implies that there's a way to get individual, first of all, is that it implies that
there's a way to get individual pieces. But maybe, for example, there's no way to get comment number
17. The only way to ever get comment 17 is by going through the post. And that's a totally
reasonable thing to want to be allowed to do. But if you only link things through URLs, you have to
construct a URL for that comment. And maybe like a very URL person would say, that's fine,
you can make a fake URL. But this stuff kind of adds up in complexity. And what we want to be
able to do is just express connected data in ways that are most natural for your implementation.
That doesn't mean that implementation leaks, it just should mean that it's natural for the
implementation that people have. So we have, you know, we have this post that has many comments,
maybe the comments are directly embedded
in the document, maybe they're linked through URLs, maybe they're not linked through URLs,
they're linked in some way. But there's very canonical ways of doing all the different ways
of describing things, and they're composable. And then, so all the ways of interacting with that data is done using normal HTTP, the HTTP verb.
So you get something using the get request.
And when you get something, you just might happen to get back other related things.
If you want to add a new thing, you post to it.
And there's some rules for exactly how that works.
There's also delete, to delete something, and there's
patch, which allows
you to, there's
sort of a historical sad
ambiguity about what put exactly means,
but patch very canonically
means the right thing, so you can
patch to make some partial changes
to an existing document. Patch is an update
to an existing document and puts like a,
it's supposed to replace the entire thing,
right? So I will just tell you, having now spent
many years writing this spec, that people
feel a lot more confident about what
put means than I think they should.
But people do have religious opinions
about what put means, and because of the fact that
you're interacting with infrastructure on the internet,
you should probably not run
afoul of people's religious opinions about
what things mean.
You're just avoiding put altogether.
You're just like, we're just going to patch.
Yeah.
And we may, there are use cases for put in JSON API that I think everyone would accept.
But we effectively punted on them for 1.0 because of the fact that what people really usually want is patch.
So I would say that that's the main thing.
We basically use normal HTTP verbs to describe interactions.
And I say that and people probably are nodding their head and saying, OK, I basically know
what that means.
I assure you, you basically don't know what it means.
Why is that?
It's just like the exact, like, OK, I'm going to make a, you know, I'm going to make a patch
this document.
First of all, what is the format? What is the response format? And then what are like all the
status codes? What are they supposed to mean? If you have errors, how do you format that? There's
like a billion, like there's so many questions that you have to answer, even to answer like,
what is a 204? What is a 200? What is a 404? What is a 500? Like what kinds of information should
come along with each of those things? Some of them, the spec says something like 204.
You're supposed to not include everything, anything.
Great.
Okay.
Problem solved.
But what about like every other status code?
What about errors?
And you really do want all these questions to be not left up to somebody's whims.
You really, if somebody doesn't have a better idea, you would really like to tell them what
the answer is.
If someone has a better idea and it's a really good reason, escape valves are a good idea.
And we have this metadata.
And actually, that's another core value is that every place in the document has a metadata key that you can use to put arbitrary information that you think is like the spec doesn't talk about, but is useful information to tell your clients.
First of all, that allows us to
reserve the top level, like anything, any top level key is reserved. So we can add more stuff
later without breaking existing clients. But second of all, it, it means that people can do
pretty aggressive experimentation with things like pagination or filtering, um, some of which
are in the specs, some of which are not with, uh, without having to worry about interoperability.
So it's, it's sort of a place where you say, I, you know, some of which are not, without having to worry about interoperability.
So it's sort of a place where you say,
I know for my application,
I would like to include this extra information.
JSON API is not locking you out of the ability to do that.
Does it have pagination and filters?
Yep, so there's pagination and filtering and sorting.
I would say that the specification for each of these things is not as rigorous.
It's basically clear, and the word must and should are used,
so it tells you what keys you have to use and, in some cases, what the semantics should be.
But for sure, over time, the specification will get clearer about exactly what you should expect.
So, for example, we tell you that there's the
offset and limit pagination, and there's a number and size pagination story, there's the cursor
story, and we tell you how each of those things should work. And we tell you that you, you know,
how you should encode the characters and all these things. But I think we're like, basically,
people are using these tools right now to build pagination solutions.
And we're getting a lot of feedback from people about what the best way to rigorously define how to do it.
Filtering and sorting are actually easier specs.
And those are already more rigorous.
But we've effectively reserved the pagination space and given you enough tools to work with, plus metadata stories so that you can figure out something we're waiting for things like ember data and other uh and other clients
to figure out like what it really looks like to use the metadata solution and then the idea is
that you take the things that people put in metadata and make them first class once we figure
out what it is so you mentioned that the meta one of the purposes of the metadata is so you can
reserve uh the top level for future changes because you want to be backwards compatible.
Can you expand on your future-proof API design? Is that kind of a core value as well?
Yes. So one thing that I've learned from specifications is that people underestimate the cost of backwards incompatible changes.
And the reason for that is that people underestimate network effects in general. So if you look at something like NPM or RubyGems or APIs on the web, there's a lot more
people involved than you think, and they all connect to each other. They don't all just connect
to the central thing. There may be intermediate things. So there may be libraries or other APIs
that work with the other third-party API, right? And the implementation details can leak across these abstractions.
So one of the things that the web has decided is if you're willing to be creative and you're willing to be pretty careful about what you expose, how much arbitrary goop you allow people to throw in, then you can pretty much keep adding stuff. You could deprecate stuff, but you can deprecate stuff without removing it. And you can keep adding
things over time and mostly have things continue to work. And I'm just generally a big believer in
the idea that it is almost never the case that breaking changes are worth it.
I think in things like Ember, especially when you're talking about the very first version of
something, there's probably like one moment in time where breaking changes become important.
I think like Linux had a moment like this. Ruby had a moment like this. And even when you do those
breaking changes, you have to be extremely careful. I think you can compare Ruby to Python.
You see Ruby did a much better job of being tasteful and careful about the breaking changes than Python did. But I think you can do a pretty okay job of keeping things permanently
compatible. And I think that people underestimate the, they overestimate the value and underestimate
the cost of being willing to break things. We just talked, the Rust core team just had a
conversation a few times about this. And there's like full consensus on this yet, but I think most people think that Rust 2.0 doesn't really make sense.
That we should continue trying to, maybe we'll have a Rust 1.40, and we think we could continue to make changes and even add things like garbage collection we think we can do compatibly. And I think people just aren't – they aren't very creative about – it's just much easier to say, oh, the world is changing.
We've got to change everything.
And I think it's just not usually true.
And I basically have put my money where my mouth is a few times, and I think people should try, and I think it will usually work out.
So one of the ways we kicked off the show was actually kind of deriving where
this came from. It started with a tweet from you
which was comparing JSON API
to ASM.js.
You said you can think of JSONAPI.org, which is
obviously JSON API, as
ASM.json,
which is a low-level consistent
serialization
format and protocol that you
can build on.
Can you compare the two?
Like you said earlier about having a de facto standard.
It seems like, obviously,
this is what you're trying to do with it,
but then you also have this.
Are they competing standards?
What is the differences and do they play together?
So Asm.js is this thing that Mozilla invented,
Mozilla Research, which is basically just, they noticed that there's a subset of JavaScript
that if you squint closely enough
is a fully typed thing
that is sufficient to represent basically all of C.
And so they basically defined the subset of JavaScript
that they will validate.
You basically tell them that you're using it,
then they validate it,
and then it will compile into native code.
It's like somewhere between 20 and 50 percent slower than just like straight-up
compiled C code. And the reason I compared it to that is that if you look
at Asm.js, what you will see is that no human being wants to write this. It's
actually significantly worse than JSON API. It's like a horrible
format to write in. But the point of Asm.js is that that's not the point.
That's not the goal of Asm.js. The goal of Asm.js is to be fast and also to be a thing that you could build on. You could, you know,
you could compile gzip to Asm.js and now you have a fast gzip that you could run in a browser.
And so the idea is you shouldn't, you shouldn't be too, people have said on Hacker News a lot of
times, like Asm.js is so ugly. It's like such a hack. Why you know bytecode? And the point is
that that doesn't really, like the exact syntax of Asm.js doesn't actually matter. The thing that
matters is that we have now constructed a thing that you could write that ships on the web that
is pretty close to being as fast as C. And that is pretty important and effectively supersedes any
aesthetic considerations that you might have. And I was sort of trying to get across a similar thing about JSON API, that the goal of JSON API is not to allow you to build artisanal
APIs. The goal of JSON API is to create a nice wire protocol and format for communicating
graphs of objects between arbitrary servers and clients. And obviously, there's some constraints
there, like you should be able to do it on existing servers.
So Rails, Express, Java should all be able to do it.
You shouldn't have to build your own whole new server
with whole new server infrastructure.
Shouldn't require things like WebSockets to work, right?
So it should work with regular HTTP infrastructure
people have already deployed.
So those things are all very important.
But the exact, you know,
whether it's 20% better or worse than the aesthetics that you
would have personally designed for a given project is actually just much, much less important
than the interoperability.
And so that's sort of what I was getting at was that it's more of a low-level protocol.
It's a way of getting everyone on the same page about something that you can share and
build on than it is about being a, and again, it needs to not be a format
that's like impossible for humans to understand.
It needs to be a format that a human can look at.
But whether or not there's a top level attributes key or you leave it off and put all the attributes
at the top level is really just not something that needs to be a reason for non-interoperability.
That is a bad excuse for non-interoperability.
So one last question before we close up the conversation around JSON API is, I think 2015
has kind of been the year of Facebook and Netflix kind of launching out these new API,
I'm going to call them transport patterns or styles, I'm not sure exactly what you call
them, but we have GraphQL, we have Falcor,
and these seem to be taking a little bit different approach than the traditional REST JSON-based APIs.
Can you compare and contrast kind of advantage, disadvantage
of the JSON API style versus these new styles?
Yeah, but I think I've been sort of getting at it,
but I'll be very explicit.
Okay.
I think that what GraphQL and Falcor are both getting at it, but I'll be very explicit. I think that what GraphQL and Falcor are both
getting at is the fact that very purist ideological REST is pretty hostile to
bulk transport, and it's pretty hostile to customized transport. People don't want you to
say, give me this resource, but don't include these attributes. And they don't want you to say,
give me this resource and include these other related resources. In fact, the very
purist REST people always talk about HTTP2 as allowing you to just continue down the path of
having one URL per resource and just, you know, firehose them with one HTTP socket, but now multiple requests. And JSON API and Falcor and GraphQL all share a philosophy that a client
should be able to, you should be able to construct payloads that are appropriate for the access
patterns that your client actually has. If you're about to look at a blog and you know that people
are likely to click on a comment, you should be allowed to include all the comments at once.
So you should be, certainly you should be allowed to include all the comments at once. Certainly you should be allowed to include
the count of comments per post
so that you can put in the UI.
GraphQL is on a pretty one far end of the spectrum,
which is that graph, the idea behind GraphQL
is that you derive the exact thing that you need
out of the consumption patterns.
So you say, I need a post.
So every component basically says what it wants. It says, I want a post. The other guy says you say, I need a post. So every component basically says what it wants.
It says, I want a post.
The other guy says, oh, I want a comment.
I need the comment count.
And the GraphQL library takes all those things,
combines it into one request,
and it asks for very precisely
what the client has asked for.
And Falcor, I would say, is also pretty similar to that,
although Falcore is a
more flexible... I don't know. Whoever's listening to me who works on these projects is going to
yell at me no matter what. But I would say that both Filecore and GraphQL are designed for
consumption patterns. They're designed for building a request for exactly the thing that you need for
what the client is asking for. And first of all, one of the reasons JSON API did not go down this path
is that it requires a server
that's capable of pretty sophisticated querying.
And I think, again,
JSON API has as a goal
that you should be able to use
existing server infrastructure.
But perhaps more importantly,
that kind of stuff is pretty uncacheable.
So if you ask for a very specific subset
of the data,
it's very hard to say,
oh, I basically know what you're asking for.
You're asking for all the posts.
No problem.
I already cached the whole payload.
It's very hard to do that
when you're asking for very specific pieces.
It's also not really a good fit
for what the GraphQL people pejoratively call overfetching
and I positively call overfetching.
Sometimes you have to think about when you're building an application, if you're showing a particular page, what are the
likely next things the user is going to do? And if it's very likely that within a few seconds or
half a minute, let's say 10 or 15 seconds, the user is going to click on something,
it's actually a reasonable thing to give them the thing
that they're going to click on
together with the original payload sometimes.
You have to think about it, obviously,
but it's sometimes worth
sending extra data out of the gate
so that not the first click will be fast,
but then the second, third,
and fourth click will be fast.
And so the Ember philosophy
is a little bit more,
be a little more liberal
about what information you download so that when you click on the second or third thing, you didn't tightly scope the data request to exactly what was on the screen the first time.
That a lot of the times the second click will have opportunistically gotten enough information to render it without making new XHRs.
That's the Ember philosophy. And the JSON API philosophy is sort of agnostic, but it is pretty compatible with the idea
of sending extra information out of the gate
so that your subsequent navigations are reasonably fast.
So again, I would say JSON APIs,
like the core, the things that I think are powerful
and special about JSON API are that it's both pretty good about, it shares
the philosophy of wanting to be able to have a payload that's not just talking about one thing,
but it's like all the things I need for this particular screen. But it does not share the
philosophy of needing a new infrastructure to make that work. It wants to be able to piggyback on all
the infrastructure that people already have to make it work. And it also is pretty positive towards the idea of opportunistically including extra
information that you think the user might want. Again, I'm sure that any GraphQL person listening
knows how to make that work with GraphQL. But again, the GraphQL consumption story is that
you don't really think about the query that you're constructing. The query is constructed automatically based on what your components have said they want. And I just
personally don't buy that. I think it's bad to tightly scope the data to what happens to be on
the screen at a particular time. Yeah, you mentioned infrastructure, and I got to thinking
about HTTP2 as you were talking. And it seems like all of these are focusing on optimizing that for the
minimum number of requests in order to get the exact data you need for the screen or for the
particular widget or what have you. It seems like the HTTP2 features, the pipelining, the multiplexing,
so on, keeping that one connection open, doesn't that mitigate a lot of the problems that we're
trying to solve? So people have that hunch, and there's a sense in which that hunch is correct.
But I think people have to remember.
No, I think you can construct examples where it helps.
I think people have to remember that there's still a round trip if you use an open HTTP
2 socket to make a request.
So if you're basically letting consumption drive your requests,
and yes, it is true that when you click on a page,
you can fire off the request much faster
if there's an open socket just sitting there.
And if you're also able to do eager pushes,
you could try to be clever about what you're doing.
But there's still, the speed of light is still,
has only a limited speed,
and people in New Zealand and Australia But there's still like the speed of light is still has only a limited speed.
And people in New Zealand and Australia experience this pretty heavily where people just assume that your internet connection is fast.
And the like the New Zealand and Australia problem has not helped a lot by HTTP2.
I mean, it's helped a certain degree, but it doesn't make the connection to the server instantaneous just because it's an open socket.
So I think people are over-optimistic about the ability to take a thing that used to be one document and break it up into a thousand documents because they have in their head
the idea that HTTP2 makes the round trips instantaneous, but that is not correct.
So in other words, there's still a certain amount of serialization that happens.
Let's imagine that you make the idea is you make a request for document A, and it has
a dependency on document B, and it has a dependency on document C.
And it's like, oh, no problem.
Now we don't have to include those into one payload because we'll just instantaneously
get document A and see that we need B and C and C.
In fact, of course, that is not instantaneous.
There's still some time that it takes to request and download when a smart server could have
bundled together A, B, and C into one thing.
And HTTP2 people will say, well, a smart server could also push A, B, and C together.
But that smart server is not really like what GraphQL is talking about or what most people
are talking about.
You need to basically, the server has to figure out the consumption pattern and say like,
oh, I've noticed that a lot of people have asked for B after A, so I will start sending it.
Or you need to have something that is morally equivalent to bundling where you are able to construct manifests that say, whenever you see A, that basically means you need B, using basically the same logic as bundling.
So I guess what I'm saying is bundling still works very effectively.
It's still reliable.
It works on old connections and new connections.
And the exact story of making things work without bundling
ends up getting to be pretty domain-specific,
whereas bundling just works.
If you take all the things that you know you're going to need
and put them into one payload and send them at once,
you don't have to think anymore.
You're done.
And I think that's why people like it,
and I think it's also why HTTP2...
The story about HTTP2 sells better
in the realm of the abstract
than it does in the real world.
The real world is not magically solving
the kinds of issues that cause people
to like concatenate their JavaScript
now that HTTP2 exists.
People are in fact noticing
that it doesn't work as well as they had hoped.
Well, I guess we'll all just have to
wait for HTTP3, which I hear will
support quantum computing.
I mean, I think, honestly,
the TLDR is that if you...
We just got to get rid of that
speed of light problem, right?
Yes. So, I think the
TLDR of all this is that if you...
Are you opening the box to get him to talk about
quantum computing? No, I'm not going to.
I think if you do something that is morally
equivalent to bundling, like you basically
construct the bundle, but then you use
HTTP2 to push the bundled
components, that's probably
fine. Like that probably will end up
being pretty similar to bundling, except
that that's...
At that point, you are not really doing the
things that people are excited about about HTTP2, right?
It didn't really, it will probably happen, right?
It'll probably happen when HTTP2
is very ubiquitously rolled out.
People will have some format that lets you say,
like, here is the quote-unquote bundle.
Someone asked for this entry point,
give them all the things,
but it doesn't really buy you that much.
Well, it's really been a fun, deep, long conversation about
JSON API. Its origins, obviously,
you know, the first thing to call was,
you know, focus on you personally and
your origins in programming and when you
became a professional and obviously open source.
But this has been
a fantastic call.
Yehudi, you are a man of many hats,
though, so we couldn't leave this
show without at least touching on Ember.
We here at the channel have been tracking isemberfastyet.com all of 2015.
It's not 2016, so congrats on that actually being fast.
But is there anything you can share today about the current state of EmberJS, what's going on, and whatnot there?
Sure.
So I've been working on a project called Glimmer 2 for the past three or so months.
So Is Ember Fast Yet?
was basically a,
was about Glimmer 1,
the first version of Glimmer,
which we actually shipped a while ago now,
like maybe May,
and then spent the next few months
like making it really polished.
And basically what Glimmer 1 did
was it made re-renders very fast.
So the idea is if you have a,
you know, you have a loop,
you have an each loop in your template and
you want to
update it with a totally new array
in old Ember and Ember 1.12, that
would have to blow away all the DOM
and replace it. And React showed us that
we don't need to do that. There are better strategies
that allow you to cleverly
update only the parts of the DOM that you really need to update.
And so what Glimmer 1 was about
was about taking those strategies that React pioneered
and making them work well in Ember
for cases where you have lists of things
that maybe the data structure has completely changed,
but the things it's talking about have not.
So maybe you have a list of people
and you're constructing a completely new array,
but the items inside the array are the same,
or maybe there are completely new objects, but the items inside the array are the same, or maybe there are completely new objects,
but the properties of all those objects are exactly the same.
And making it so that basically without having to think about it,
all those scenarios end up updating fast.
And Gliver 1 did that.
I think we showed it at Ember Conf a demo
that we then repeated when we actually shipped Ember 2.0
that showed a speedup of probably more than 10x
for the updating case.
It was to the point where Ember 1.x, the demo, basically froze.
And Ember 2.x, it effectively shows fast updates.
It shows everything updating many times per second,
basically, instead of being frozen.
So, very big improvement.
However, Glimmer 1 did not really do much for initial
render performance. And in fact, for various and sundry reasons, it regressed initial rendering
performance. And unfortunately, because re-rendering was so slow before Glimmer, almost nobody had
applications that were doing a lot of re-rendering, which meant that all the work we did to improve
re-rendering didn't affect existing apps much. It did let you write new apps better
that would take advantage of it,
but it didn't affect existing apps.
But all existing apps got slower
because the initial render got slower.
So that made people rightly sad.
And we've done a lot of work since then
to close the gap just by picking away
at little paper cuts that we had introduced
for various reasons.
A lot of them were compatibility, but not all of them.
And the idea behind Glimmer 2 is to say, okay, now that we have integrated Glimmer into Ember,
what have we learned about what the requirements are, right?
So one of the reasons why things were so slow the first time, I'm being very rough on myself.
The reason why performance regressed after Glimimmer one was that we effectively had to learn what the requirements for ember compatibility were as we
were integrating it and as you can imagine like as a programmer sometimes the way you solve problems
like that is you just do whatever you do whatever you have to do to get past the problem and we did
that enough times that things didn't compose very performant. But now that we're done,
now that we've done all the integration against the more React style of internals model,
we kind of know what all the requirements are. And so the idea behind Glimmer 2 is to
rebuild the primitive layer against the new requirements. And we have some benchmarks now
that we've been running for a while that show many, many X performance improvement on initial render and things like beating React on equivalent templates.
So like beating React by a decent amount on equivalent templates.
So we're definitely pretty excited about that. One of the things that is like,
one of the things that I've been working towards that we haven't quite done yet,
but it's going to be pretty awesome is, so Ember's story about templating the whole time
has been that we give you a declarative templating layer. So you just write your HTML
and you write, you don't write JavaScript in there. You write basically direct what I guess
Angular called directives, but I mean the English form. You write directives that say, this is a loop, this is an if statement,
whatever. And historically, that whole system has been pretty dynamic. Like we sort of compiled it
and then we said, we don't really know what this if statement is talking about, so just figure it
out at runtime. But we noticed a pattern, which is that a lot of times people will make components.
So there's this project called Font Awesome, which is pretty cool. And people write components that wrap Font Awesome. So that instead of having to say like I class equals FA space, FA bug or whatever,
they'll make an Ember component called FA icon. And they'll say like FA icon, icon equals bug.
And it will basically emit the correct code. But if you look at that invocation, icon equals bug,
you'll see that there's nothing dynamic about that at all.
They're literally using a string bug.
So in an ideal world,
you'd be able to take the template
that is talking about the layout for that component
and you would be able to notice,
ah, that is actually not talking about a dynamic thing
and compile it into the static thing
that it is really talking about.
And one of the goals of Glimmer, too,
is to have a sufficiently flexible compilation architecture
that we can effectively do these specializations at runtime,
do them only for hot code.
So, like, you probably don't need to do that
if the FA icon only shows up one time.
But if it's in a loop, it probably matters more.
So basically to have a flexible architecture that lets us say,
you know, we notice that this thing is a lot more static than the code would imply.
So let's actually compile it into the static form.
And this is all owed to the fact that the Glimmer and like Ember templating system is
like a pretty nice programming language for optimizations in general.
So it's like more like a referentially transparent programming language. It doesn't allow you to do a lot of the dubious things that make it hard to
do optimizations. So we can apply like the beginner level optimizations that you learn in school
against this very simplified programming language and have, and be able to do that reliably.
So, so that's basically the story behind GlimmerTube is that we're restructuring the
architecture so that it's a flexible compilation architecture.
And then we can make things that are static behave as if they were static instead of behaving as if they were dynamic.
And so far, the work that we've done on this kind of stuff has shown very significant improvements.
So we're pretty happy about that.
I'm probably one of the people that's putting my hand up saying, I don't know how you do all you do. So we're pretty happy about that. and a consistent API and a future-proofing in a way for the future, but also the work you're doing at Ember and your company,
all that you do there and the different products, I guess,
you're working on as well would play into that as well.
But Yehuda, we're about 10 minutes past our time,
so we're going to close the show today.
We did have a couple more questions we want to ask you,
but we'll save those for a different day.
I'm sure you'll be back on.
This is not your last time on the show, I'm sure.
So let's close there.
We want to thank our listeners for coming and listening to the show religiously.
It is the new year.
We have a great year planned ahead of us.
And if you're not a member yet, go check that out.
Change.com slash membership for 20 bucks a year.
You can support this show and do what we do around here and get an
all-access pass back to our members only slack channel hang out with us get behind the scenes
we share lots of stuff with our members before we even share the rest of the world
we also had some awesome sponsors sponsored the show so thanks to those guys and uh if you have
a second you can throw it throw a note on this if you'd like to.
But our next show is actually about 0DB, an end-to-end encrypted database protocol with McLean Wilkinson and Michael, I'm not sure how you say his last name, but that was a
really interesting find recently.
And that's the next show we're doing.
So that's it.
Any notes you have on that show particularly?
That seems cool i'm very generally very uh pleased about the fact that our our community and our ecosystem is
pretty encryption friendly and i think it's uh the government would like it to not be so right
unfortunately people like encryption and i'm happy that that's true that's cool all right well
everybody thanks for listening.
That is the end of the show.
So everybody on the show,
let's,
let's say goodbye.
Bye.
Thanks.
No problem. I'm out.