Future of Coding - Magic Ink by Bret Victor

Episode Date: December 8, 2022

Before the time-travelling talks, the programmable rooms, the ladders and rocket launchers, we had the first real Bret Victor essay: Magic Ink. It set the stage for Bret's later explorations, breaking down the very idea of "software" into a few key pieces and interrogating them with his distinct focus, then clearly demoing a way we could all just do it better. All of Bret's works feel simultaneously like an anguished cry and a call to arms, and this essay is no exception. For the next episode, we're reading Programming as Theory Building by Peter Naur, with a little bit of Gilbert Ryle's The Concept of Mind thrown in for good measure. Links Four Hundred of the most Chart-Topping Thoughts of All Time: Inventing On Principal Stop Drawing Dead Fish Drawing Dynamic Visualizations Dynamicland Paper Programs by JP Posma was inspired by Dynamicland. "Computers aren't the thing. They're the thing that gets us to the thing." from Halt and Catch Fire Charticulator is Microsoft Research's take on a _Drawing Dynamic Visualizations_-esq tool. Jimmy's Fender Jazz bass looks like this, but red, but like a decade older, but like $600 at the time. We could probably post parts of this episode as Boyfriend Roleplay on YouTube. Fitts's Law is but one thing we've learned about the industrial design aspect of building good software. The Witness is a game where communicating ideas through (essentially) graphic design is the whole entire point of the game. If you haven't played it, know that it comes highly recommended by plenty of folks in the community. A "red letter Bible" is a Bible in which the words spoken by Jesus are colored red, to make them easier to identify. Toph Tucker has a pretty cool personal website. It's rare to see these sorts of sites nowadays, and they're always made by adventuresome programmers, trendy design agencies, or their clients. In the Flash era, it felt like everyone had a website like this, for better and for worse. tldraw is a beautiful little browser-based drawing tool by Steve Ruiz. What few things it does, it does exceptionally well. John while Henry had had had had had had had had had been my preference. #devlog-together is the channel on our Future of Coding slack community where members post small, frequent updates about what they're working on. The (Not Boring) apps are arguably a counterpoint to Bret's theses about information apps and harmful interaction, where the interaction and graphic design are balanced against being maximally-informative, toward being silly and superfluous, to great effect. Did you know there's a hobby horse, but also a hobby horse? I didn't! There are a few examples of folks doing FoC work that, in Ivan's view, align well with the values Bret outlines in Magic Ink: Szymon Kaliski's projects for Ink & Switch, summarized in his Strange Loop talk, Programmable Ink. Mock Mechanics is an environment for building mechanisms by Felipe Reigosa. PANE by Josh Horowitz inverts the usual node-wire programming pattern by putting data in the nodes and data transformation in the wires. Robot Odyssey was a 1984 game for the Apple II (and some other, lesser systems) in which players would go inside various robots to reprogram them. Music featured in this episode: Wash Machine, from the unfinished 2014 album Sneaky Dances Shaun's Amaj Rebirth, created in November 2022 for a friend named — you guessed it — Shaun. Hey! Send us questions we can answer on the show. Like, "How do you keep bread warm?" Or, "What's so great about concatenative languages?" We'll answer them. Send them here: Jimmy Ivan Or just DM one of us in the FoC Slack. futureofcoding.org/episodes/060Support us on Patreon: https://www.patreon.com/futureofcodingSee omnystudio.com/listener for privacy information.

Yeah, this one's going to be all over the map. It's Brett Victor, right? And people kind of, while I don't necessarily think everyone sat down and read Magic Inc., I think almost everyone listening to this has some familiarity with Brett Victor. Whether it's just one of his talks, or all of his talks, or have heard the name, right? Or been influenced indirectly. Or have participated in the meta discussion around. Because for the handful of things that Brett's done,

there's sure been a lot of conversation about that work that I think people encounter, even if they haven't spent a lot of time with the source material. Yeah, exactly. I mean, Brett Victor's probably the best-known person in the broader ecosystem of future of coding kind of things yeah maybe alan k might be ahead right in terms of things that people know and point to though for a while there it kind of felt like everybody was discovering alan k via brett victor yeah that's a good point because brett's work went viral in the last decade in a way that Alan's work never did

because, you know, Alan was doing most of his work before virality was a thing. Yeah, and I think a lot of people would know Brett Victor's work by name, like by Brett Victor, whereas like Smalltalk, right? They might, people might know Smalltalk without knowing about Alan Kay. Or they might know the story of like Apple taking the GUI from Xerox PARC. Ah, yep. And not know that Alan Kay was at PARC working on that. Yeah, exactly.

Whereas like Brett Victor, there's not any one particular work that I think he's known for, except for maybe Inventing on Princi principle, where he is very front and center there, right? Like it's a talk. And I think that's the one kind of like Hickey's simple made easy. Exactly. That's the one that's the easiest entry point into the canon of Victor's works. Yeah, one of the things I know we said we're diving in and now we're kind of giving, you know, not actually an overview of Brett Victor. So for who, if someone doesn't know Brett Victor, Inventing on Principle was the talk that he gave that became very popular. But he's a designer.

This is not somebody who sees themselves as a software engineer, though he's clearly very capable of doing software engineering work. He's very intelligent. He's very well read. He has been focusing on a lot of things around programming, but doesn't see himself as a programmer, sees himself as a designer. He worked at Apple. He was homeless, jumping on trains, I think, traveling across the country for a little while, like nomadic. Yeah, it's a nomadic. And then started Dynamic Land. He's a very interesting person, kind of looking at these design elements,

not just for programmers, but for scientists, for creatives, for numbers of people, and really wants people to kind of, he wants to elevate design. And that's what this essay, Magic Ink, that we're going to be talking about, is kind of about as well, is design and software. A lot of the criticism that I've seen directed towards Brett's work, and for whatever reason, his work gets a lot of negative reaction, probably just because it's popular and it's throwing stones. And so, you know, that upsets people who have a certain fondness for the way things are currently done. The criticism that I've seen directed at his work, or at least one of the

big ones, is that he will present things like alternative ways of programming or alternative ways of working with dynamic media within the computer. You look at them and you think, wow, I wish I had that. That looks like an amazing tool. A great example of this is the talk Drawing Dynamic Visualizations, which presents a tool that is sort of like something you'd get from the Adobe Creative Suite or some Office package from Microsoft or Apple or whatever. It looks like a tool that would fit into one of those collections. And it's a kind of a hybrid of a drawing tool focused on simple vector graphics,

like rectangles and arrows and that sort of thing. But then also some cool programming-ish features so that those vector graphics can have their drawing characteristics be manipulated by data sources. And so it's a tool that you would use to generate like, here is a style of graph that represents some data. And when you bring in a different collection of data, the graph will update itself to represent the new data. And that's cool. That's

something you can do with spreadsheets right now. but what's special about it is that the way that the graph is constructed is not you know i picked from the template set of choices like line graph bar chart pie chart whatever instead you've built a little graphical system with these relationships and these rules and so the way that the graph will look or the or the the visualization of the data will look is very bespoke. And you can approach it in a very artistic way and kind of construct this visualization that really nicely suits the kind of data that you want to present and presents it with a very distinct style. And the criticism is, you know, hey, I wish I had that tool. I wish that was something that actually existed. and from the

demo it looks like he actually built the tool and was using it and people say well why can't i have that why doesn't he just release it why isn't that something that's been made public that everybody can benefit from and i've seen that criticism again and again i've seen that criticism directed at dynamic land and at stop drawing dead fish and at all the different really high quality looking demos of things that he's built or these prototypes the thing i think that gets missed and he's tried to make this point multiple times throughout his works throughout his presentations is that what he's doing is not trying to invent the new way that we should be doing things, but instead tell everybody that new ways to do things need to be invented. And what he's trying to give us is the set of

modes of thinking and the frame of reference and the feeling of compulsion that will drive everyone to be advancing the way that we work, thinking about new ways of creating and new ways of communicating and doing all of that to really leverage what the computer gives you. And so he's not trying to be the one who comes up with the new thing. He's trying to be the one to kind of wake everybody up from their slumber of complacency with the status quo and say hey everyone you need to be inventing these new things i'm not going to do it for you i'm here as somebody who's aware of how limited things are currently and how much better they could be but i you know brett on his own he's he's just one person and so he's not going to come up with all of the

ways that working with a computer could be better, or working with other people via a computer could be better. Yeah, I do find that response so unsatisfying. Not because I think it's false, but because there's just such an easier response. None of the software he made was production-ready in any way, shape, or form. It was a demo. He's so good at crafting demos, it looks like he has a real application that's really there, but he doesn't.

I guarantee if you do anything outside of the buttons he clicked in the demo, it just doesn't work. I just think it's so much easier to be like, yeah, the thing I made isn't really functional. Someone else should go make it rather than I like, I actually think that that response, while true, has actually stopped people from going and just making what he presented. Because they feel like it'll be cheap if they do that, because he wasn't trying to offer a solution. He was trying to offer an idea.

And if I just go replicate the solution then i'm not really taking his point and so that's why i don't think we have that software today and i would love that software today right and i actually think that would be a really fun um while i'm not a fan of hackathons maybe a jam like uh everyone go recreate your uh your favorite brett victor tool that he's presented. Jam, right? I actually think that would be kind of fun to have little prototype-y versions of those things that you could start playing with. It's worse than that, though.

It's worse than, you know, if he did release these demos, they'd be unsatisfying. A, there are some people out there who have taken especially dynamic land and have created their own like yeah paper programs that project is an attempt at recreating dynamic land without actually knowing the true implementation and without knowing the goals behind it let's just recreate the surface syntax of it so to speak and it's it's interesting and i'm sure it was a lot of fun but i i feel like yeah it's missing the point a little bit but the more insidious thing here is you're saying you know if he if he just released these programs people would find them unsatisfying i worry that it's actually the opposite that if he did release those programs people would say oh

thanks for open sourcing this i'm gonna put in a couple of pull requests or make a fork and shore it up a little bit and then good enough and that we are so used to living with poorly designed poorly conceptualized poorly implemented software that brett victor's you know he's not releasing this stuff because his i imagine that his standards are high and i would say appropriately high because i think the standards that most other people have most other working programmers at least people who are used to software being you know oh it's computers it's software it's like if it doesn't work perfectly all the time if it's not very elegant if it's not very nice to use whatever we just put up with it like right we're still using 80 character wide terminals right so we put up with a lot of really terrible so if he released this stuff people would go wow that's

that's so much better than than what we have today and jump all over it rather than going no these prototypes aren't good enough yes they're better than what we have conceptually and perhaps even in terms of some implementation stuff because brett is actually a pretty good programmer i've read his code his code is good but it's not good enough it's not what we deserve as a species as a thinking working cultural humanity writ large we deserve way more out of this magic that we've captured inside the box the dynamic medium we deserve so much more than than what these prototypes are and i don't think people would realize that if he just released these prototypes they'd think oh wow that's the thing no that's not the thing that's the thing that gets us to the thing but has that strategy worked

do we now after you know so magic ink is is 2006 yeah right have we now really learned those lessons do we now have a better plotting tool than what he showed in dynamic drawings? Like, we don't. I'd say yes. Okay. So this is going to be fun. I would take the position that

the people who were going to be satisfied by just emulating Brett's prototypes and his sort of early explorations have actually like disseminated those ideas widely and those ideas have been implemented and they've taken root and I find rereading Magic Inc and and looking at his other works that a lot of the things that he was holding up as examples of what we could be doing better are now better. And what we don't have is the true full realization of the vision, but that stuff was always sort of nebulous and you had to really hunt to arrive at that level of realization. And so of course we don't have that,

but all the concrete things that he used as examples of things that, well, still not good enough could be better. I think we have, and I think we'll see that as we get into the essay. Do you think that's just true for Magic Ink? Are you saying like dynamic drawings? Like if there is a tool out there that does stuff better than what's in that demo of dynamic drawings, I want it. There's one that's called Charticulator, which is for working with Power BI.

And it does a lot of what Brett showed in that prototype. So what you're saying is we got a better world because it's all in Power BI. And it does a lot of what Brett showed in that prototype. So what you're saying is we got a better world because it's all in Power BI. Yeah, right. Instead of open source from Brett Victor. No, what we have is the people who look at Brett's work and are like, oh, I'm doing something kind of adjacent to that space. And there's some really good ideas in this prototype that he showed i'm gonna you know put those ideas into my product that i'm working on that happened what didn't happen was the like people then saying okay now that we've stolen those direct examples how do we keep going from here how do we follow that vector not just the one step, but repeatedly. What kind of bass do you have?

I have a not American Fender Jazz. Ah, cool. Because I got it before I had money. Yep. It's a nicer one. I can't remember now all the details. But I got that from my first ever being paid to program job I got $600 to write PHP for a big ol insurance form mmm I

did not write PHP I wrote a Python script that wrote the PHP for me and I got paid a flat fee of $600 for the job. It took me an hour. It was great. So best, like I got $600 an hour. Like that was definitely the best pay rate I've ever had. Really? And got that base.

Yeah. Yeah. You should go into music. You perform at a festival or something like that for half an hour and you get like five grand. Oh, nice. And then you sign a contract that says I won't perform at any other festivals within a 200 mile radius for the next six months and then are you serious wow yeah

yep uh anyways yeah we were talking about uh whether or not we've actually achieved any of the things that brett victor has trying to push people towards achieving. And I think as we get into this essay, we will see that some of the things that he was pining for came true pretty much immediately after this essay came out and other things, the broader, the more abstract, the deeper kind of philosophical stuff. I think arguably there's a little bit more design thinking happening these days than there was back then, but it's not always a little bit more design thinking happening these days than there

was back then, but it's not always the good kind of design thinking. Yeah. See, I think this is just going to go back to our disagreement on the JCR Licklider episode where I don't think the things that he wanted have been accomplished and you do. So that's great. I'm all for it. Because I still look at, and I think, you know, I again am going to claim that I'm not being pessimistic. I'm being optimistic. I think that there's still yet more that we need to unlock here. Yeah, you have hope. Yeah, I do.

And I think that, but I also just think like what he, so Magic Inc., you know, I'll maybe give like a little bit of a summary as I see it for what he's offering, just because I think it'll make clear what I think we need to get to. And his point is that a lot of software design has failed to be even as good as print design. That software design really has focused so much on interaction. We kind of treat the computer as this machine that we need to manipulate

and less about displaying information for people to make decisions, to come to conclusions, to learn something. And that by focusing so much on interaction design, by focusing so much on the computer as a mechanical thing, we've really made bad software. And so he wants to say part of the problem here is that the people who design can't make these context sensitive magic ink, right? It's like print design, but now it knows something about your context. And so designers really aren't able to make these rich information things

that are dynamic but not interactive. And so you could really kind of call this interaction considered harmful. And the idea is that our software should only be interactive when it has to be. And really, our software should be context-aware, good, print-like displays of information. For the kinds of applications, and I mean lowercase a applications, where communicating information to the user is the primary purpose. And that's one of the things pretty early on in the paper, he kind of splits, you know, he asks the question, what is software and then answers it by splitting it into several different categories. And one of them, the one that's of most interest at the moment, is called information software. And it's software where the purpose the the software is to facilitate learning by the person using it that the person using it needs to absorb some new information or contextualize some information or or do something where what they really need is they need to learn and so the

software should help them do that learning and that's different from manipulation software, which is software that is for designing something or building something or having some model that you are working on and transforming and changing. And then the third category is communication software, where the software is facilitating communication between two people. The purpose of the software is just to allow people to communicate in a new way or in an old way, but different and better. And that one's kind of self-explanatory. And so the main interest here is information software, software that's, you know, there's tons of examples, but like a, like a phone book directory where you want to look up some information about a person or about a business or a shopping website where you're trying to pick what book do I want to order and how which of these editions or which of these

different books on similar topics should I choose or a website that's showing the weather or a website that's showing and I keep saying website but I application in the broader sense application that's showing you know movie showimes for my local theaters or applications that help me plan a trip. Like all these applications that are about learning something so that you can make a decision based on it. Yeah. And I will make a claim that is not made in the paper that with the exception of communication software, because, you know, obviously we're communicating. I think most software is actually information software. He does make that claim in the paper.

Okay, he does. Cool. Oh yeah, super does. Yeah, I wasn't sure. Maybe I missed it. But I think, for example, even programming interfaces, things where we are ostensibly supposed to be creating that seems like it would be something else, I think even that is actually more information software than we give it credit for being.

Oh, for sure. Yeah, totally agree. Most of the time that we are spent is not in the creation process, but in the question process. We're trying to discover and learn, and our interfaces are not designed at all for that i mean think about trying to navigate code and most text editors uh to like learn how a piece of complicated software works yeah it's it's a painful mess it is so hard to keep in your head the call stack that you're like walking down with the c macros from the c ruby code base being all wonky and weird and you not understanding C syntax at all. So you're like, what in the world is this? And then like, all of a

sudden, there's just like magical macros that are defined by themselves, because somehow a pre processor outside the macro processor, you know, insert something. I mean, this is not specific to me at all. I'm sure it's specific, Jimmy. I did definitely did not do this just uh this last week and be really confused by what in the world c does um so yeah no not not just me i know um thank you ivan for sharing uh in my c pains yeah no absolutely i and and so yeah i maybe he does make this claim i didn't want to put words in his mouth but yeah i think like most of our creation software could actually benefit a lot from having better displays of information rather than just focusing on you know when we're pressing the keys on the keyboard yeah and he he does talk about manipulation software the second category

as being information software plus some extra interactivity where the interactivity is is a good thing or it's at least it's it has a purpose whereas in information software you don't want interactivity or at least that's what he asserts in manipulation software you do need some interactivity but you mostly still need all the good design that information software that good information software should have you know when you're doing a manipulation half of it is or more than half is learning about something and then a little bit of it is making these small tweaks to the thing that you were learning about to the thing you're working on the model that exists in the software and a claim that he doesn't make in the paper at least i didn't see it and cue jimmy jumping in and saying actually he did make that claim um is that i think communication

software is actually mostly information software and manipulation software and it's kind of like an even more sophisticated version of manipulation software and to explain why it is that way i think we have to do a little bit more summarization first so i'm going to put a bookmark in this and say at some point we should come back and talk about how communication software is actually more sophisticated manipulation software and how it bypasses some of the issues that interactivity causes in information software because there's some stuff there yeah this is a long paper yeah and it's it kind of mostly published as a website it has this kind of nice setup where it's side notes instead of like footnotes which i i personally really enjoy like margin notes but i will say the pdf for it did not work for me every page would be cut off like the bottom like three or four lines was cut off

when i tried to render the pdf so i had to like print the html as pdf and in doing so i don't have page numbers yeah so i'm gonna call an audible and say this is table talk i'll probably cut it from the episode oh yeah i think it's good but you can cut it if you want all right well fine it's staying in now hello listener this is important we're having a conversation here other people are going to want to have similar conversations yeah all right listener i'm going to give you 10 seconds of silence you say whatever whatever you want, and then I'll respond to it. Well, that doesn't make any. I'll grant you that. Okay, slow down a minute.

This is a lot for me to take in. Well, no, I have thought about it yeah but don't you think everybody would be mad if we did that no it was a it was a death stranding reference you You might not have played it. Are we doing like boyfriend roleplay on YouTubes? I don't know if you know these, this weird genre of YouTube. No, I don't. Yeah. So I only know this from a commentary YouTuber talking about it. But yeah, there's like videos where people will pretend that they are your boyfriend and they're like jealous and they'll have a conversation with you and you are supposed to like like blues clues style you know be talking back to them whoa so make a call back to a previous episode there did we release that episode i don't know if that one yeah yeah that was uh the dynamic

media oh okay where i mentioned blues clues yeah i don't think i've mentioned blues clues before Yeah, yeah, that was the dynamic media. Oh, okay. Where I mentioned Blue's Clues. Yeah. I don't think I've mentioned Blue's Clues before that. I'm not that big of a Blue's Clues fan, at least not publicly. So the thing about page numbers I was going to say is this essay does the nice thing where each paragraph has a permalink, and so you can permalink to any given paragraph. So it's kind of embracing the medium a little bit. No, I did not read it with page numbers.

I read it just as a webpage. Okay. And if we need to look for a particular quote or something like that, talking to you, Jimmy, not the audience, we can take 30 seconds and do a find on page and scan around and that kind of stuff. And I can judiciously, surgically excise that from the edit and then maybe insert an ad read for totally real services that are good. Do we want to start walking through the argument as presented? How do you want to tackle this one? So here's what I'm going to do. I'm going to scroll down to the very bottom where there's a section called summary.

Okay. Clever. That sentence by sentence. Cheating, but okay. Yeah. So let's see what we've got. We've got software design consists of graphic design, drawing pictures, and industrial design, allowing for mechanical manipulation. So that's the first sentence of the summary.

Good job! What's the second sentence? Yeah, okay, so I mean, we could start there, right? If we really want to. Yeah, probably should. Okay, cool. So I do think this is actually a very interesting point, because I think as somebody who's a software engineer who's done a lot of front-end work but is not a designer at all, I think

this is actually something I don't ever separate these two dimensions. I think he talks in here about how we really have these mechanical metaphors in our software. We have these levers that we're pulling, these buttons that we're pushing. Dropdowns are these drawers that we're opening up, whatever. And so I never really think about the software I'm building as graphical and then interaction. It's kind of just all in one.

And one of the points he makes about this is that when we don't separate these two, what we get is software that over time, more and more things get shoved into tiny little menus. More and more things get put in small little places because we need to keep adding more and more things get shoved into tiny little menus. More and more things get put in small little places because we need to keep adding more and more functionality, and the pages just get to be this cluttered mess where there's no visual hierarchy, there's no information hierarchy, and we just keep wanting to have more and more features

but no place to put them. And I think that this is exactly the kinds of things I see in the software that i work i've worked on and especially in a you know business setting right where we're we're asked for more and more features and so i actually think just this separation of the two is very interesting because if you start thinking about the software and remove all the interactions, you start designing very differently. Like, I really think, you know,

I've done this kind of exercise after reading Magic Inc. the first time about, like, what would it look like if I couldn't interact with this software? How would I start designing it? And I think that's actually a very interesting frame that I think is very important for the start of this and for the examples that we're going to kind of see as he walks through them. The context in which this paper was written is important. And I want to explore that for a second. This came out in 2006 as we mentioned and i want to do a little bit of a sort of a guided tour back to what was software actually like in 2006 because my memory of that

time is colored by you know all of the life i've had since then i was in my 20s when this came out and so i was you know old enough to have been working in computers for quite a long time in a in a you know a professional-ish capacity and doing a lot of serious work with them this is one year before the iphone came out and i think that's a really important milestone that's going to come up a couple of times and it's when the internet was just getting to the point that people could do a lot of really major things on it that we have come to if Google Maps is around them, but there were a couple of other competing map services at the time. Right. And this is, you know, Gmail came out just a couple of years earlier. This is the beginning of web apps as a real thing, like the JavaScript S spa or not even the spa but like the the ajax style like you can have a web page and change some interface elements and without reloading

the page the web page updates itself like that's a brand new thing and so the the kind of software that is being used by a lot of people is moving from native on-computer software to software that lives in the web and the interfaces that that web-based software has are basically built out of the handful of primitive input elements that are available in web browsers at the time and this is not to say that native software is that much better. But the I think the most powerful examples of the ills that this paper is trying to push us away from are happening in the world of web based software. And so this is things like and there's some great, great examples in the paper, but you know, you're not going to read it. So I'm going to describe them to you,

or you're not going to read them, you know, pause the episode and go read it right now so i'll describe it to you there's things like if you want to look for flights from one city to another city you have one drop down menu where you select the city you're departing from and another drop down menu where you select the city you're going to you know your destination and then you have a drop down for the date that you want to depart and that is three separate drop downs one for year one for a month one for day and those drop downs start at like you know 1990 or whatever you gotta scroll all the way down to uh you know whatever the current year is or that kind of thing right like they're they're not contextually sensitive at all and so you you pick your month all right february and then it's like, okay, what day?

Well, it's a dropdown with 31 choices in it because it's not going to be smart enough to go, oh, you picked February. So only show 28 or 29 days in the day dropdown. All you are seeing for the most part, when you're doing an action where you have to put in some input, all you are seeing is these off-the-shelf input elements. You're seeing pull-downs, you're seeing number fields or text fields, you're seeing a submit button.

You're not seeing what we would soon have, you know, maybe four years later or so, like around 2010. You're not seeing things like when you need to pick a date, you click on a little thing and up pops a calendar widget where you can actually click on a day on the calendar. Or what we have now, where it's a calendar widget where you can click on a day and drag to select a range of days. Or something where you can click on a day and then pick, you know, I want plus or minus three days.

So show me all the airplanes flying out of this airport on this day, plus or minus three days, and it colors in the plus or minus three days a little differently. So you can kind of see that right in the calendar. We don't have these rich widgetized inputs on the web, and a lot of native apps didn't have them either, but the web was the worst. And that goes for every kind of data source. Like if you wanted to pick a location, you wouldn't be picking from a map. You'd be picking from a, you know, a text entry field, maybe, and you'd click search and it would load a new page with search results and you'd pick the

one that you meant, or it would have a fixed list of location choices and you'd pick the one that's the closest to you or something like that. If were going to be working with with calendars or working with emails or working with any of these different kinds of data sources they were only aware of their own internal kind of data emails only knew about text and now you know nowadays if you get an email that has an event invite or something there's like an embedded calendar widget and a thing that integrates with you know if you're using gmail and google calendar or whatever a thing that integrates the two of them so you can rsvp for the meeting right in the email and it will put an entry in your calendar at the right

time and you can see you know a little preview of what other events are in your calendar at around the time that this meeting is scheduled so you know if you're busy or if you're going to have enough time to do this meeting. There was none of that. Yeah, so I think you picked the one example that we actually have something new for and all the rest of them look exactly like what he shows in this essay. So, hold on.

So, you gave the flight example which i do think flights flights have gotten better interfaces uh they still don't do what he wants and i don't know if what he wants is good where you just show a map and you tell people to click on it um you know we type in the city names which is probably better uh but like movies i haven't gone to look up a movie. I don't really go to the movies. Yeah, no, me neither. I just did. And apparently Fandango is the answer.

That was the first Google result. And it looks almost exactly like what I see on this page here. And our listener, I'm sure, can assume what it's going to be. It looks modern and more updated. Like there's orange buttons instead of black text and the text isn't all blue and you know the there's more white space for the tables and they're you know whatever right like more modern aesthetic but it's a list of theaters and then for that theater it actually shows me the address of the theater and then it shows me each movie and the times that they're playing and there's nothing that he kind of talks about how this is a very bad graphic for what people actually want to do when

they're picking a movie I would bet I can go to a local flower shop that's another example that he has and I'd get like some drop downs of the kinds of flowers that I might want and not beautiful pictures um and colors that are pictures of flowers of different colors and things like that like i i really think like for the most part his point is that oh in amazon i'll use that's one of the other examples he has here which looks identical um in fact might actually have less useful information yeah uh today than it did when he took a picture of it but his point is like these interfaces at a glance

offer us none of the things that we the question answers to the questions we're asking and instead we have to scroll we have to look around we have to interact with the website in order to figure out the sorts of information we really want to know. And he offers up alternatives to these that don't require us to interact, and yet we can answer all sorts of interesting questions from them. So for his Amazon example, I think this one's really interesting here because I want oftentimes to find good book recommendations and decide whether I want to read a book or not. And it takes me a very long time because with Amazon, I have to go and click on the book and scroll and look at reviews and look at all this information.

Instead here, he has this very nice view of a book, who it's by, how many pages it is, a little blurb, a few little review snippets. And then one of my favorite things is he has the star rating, but he also has these little tick marks under the stars so you can know the distribution of the stars. So maybe it's three stars, but it's actually a bunch of fives and a bunch of ones, not a bunch of threes. And these little tick marks underneath the stars kind of show you. They'll be like three or four.

If it's like most people rate it at five, or a good distribution is five, a good distribution is one. There would be like five tick marks on both. And then we have, so this is like one column for these books. So we got Waiting for Good Dough, The Bun Also Rises, Yeast of Eden, right? And they're like by great, you know, like... Yeah, Samuel Biscuit. Yeah. Ernest Hemingwaffle, John Sconebeck. Yeah.

It's great. And then we have on the right, like some contents, like the table of contents is what it kind of looks like. And then related books that have like a similar little setup for stars and everything and so from this like graphic you get so much information there's a price in there new and used there's so much to go on here that you you can just look and sit at this graphic and that's what's so great is like even with these jokes i'm able to like pull out the little like lord of the pies 100 years of solid food like decline and fall of the ramen empire it's so good and like you couldn't even make these jokes with the original amazon

graphic like it kind of the jokes almost kind of reinforced the point that there's so much more information here so much density of jokes just right on the the page and i guess you know i i agree with you that like we've gotten i actually kind of found your point a little weird because like yes we've gotten more context sensitive inputs i guess would be the way of of putting it um but we really haven't gotten better graphical display of this information which seems to be kind of his like primary initial point here is think about what a good print graphic gives you you look at it you can answer questions about you can answer questions for, you can answer questions

for yourself without poking at the paper, right? Because obviously if you do that, nothing's going to happen. And yet our software systems don't give us that same capability. We have to poke, we have to scroll, we have to click, we have to do all of these things just to find the answers to our questions. Yeah. Like that print test. How good is your software if your way of viewing it is and working with it is via printer if you can print from your software and it still performs its function very well then that is a software that has good graphic design in it. But if your software requires that you click through lots of different things in order to get at the information you're looking for, then you wouldn't

be able to use it with a printer without just printing tons and tons of pages and going back and forth between the printed pages and the actual software on your computer. Because one of the things he gets into, and this is going to be a familiar idea so i'm comfortable jumping ahead to this is this idea of wanting a really tight feedback loop and it's funny that the printer test is like let's artificially lengthen the feedback loop of using the software to demonstrate that interactivity like you know good interactivity will have a tight feedback loop but it can't it will never be as tight as the feedback loop between like photons hitting your eyes and what your brain is able to do with those photons and so if your graphic design if the way that you've laid out your

information visually is giving you all the things that you need to think about and giving them in a really well-structured way, then you're going to do so much better than if it gives you just a little tiny information sent and then you need to click something and then click something else and then click something else to get the information you're looking for about one choice and then go back, back, back, and then click, click, click to get the next choice and then go back and to, back, and then click, click, click to get the next choice, and then go back and to build up this, you know, the set of things that you want to be thinking through and comparing. I have a if you're up for it, I have another great example of like, how software today compares with software in this essay. Yeah. So because you you, you correctly pointed out that the the calendar one is like the one big instance where what we have today is exactly what brett was describing as you know here's the right way to do it just for fun i

wanted to see how does google do at getting movie showtimes because in the past you know i used to use fandango and whatever and then google started adding a nice way of getting movie showtimes where you could just type in your your zip code or your postal code and the word showtimes. And it would give you a nice little smart result with like, here's the different movies that are showing and here's where they're showing. Here's the times like right at the top of the search results page. And so I did that. You know, I put in my postal code and then showtimes and Google came back with here's a list of theaters in your area it has places now it doesn't actually do what it used to do or it would show movie show

times it has a little map widget that shows on the map all the different theaters and it's like okay that's uh better than just search results because it's closer to being relevant to what I'm interested in but it's not what I wanted so I click on one of those theaters in the listing and it takes me to another map view. So on the rightmost side, there's like a map showing my city with pins for each of the theaters. And then floating above the map is a little panel that has here's all the movies showing at that theater I clicked, and what times they're showing. And in the left, there's the list of theaters like a list of all the theaters and so if i click on one on the map or if i click on one in that list on the left then it shows me

a different panel of you know the show times for that theater or whatever so it's like okay it's it's letting me click on each of these theaters to see the movies that are showing at that theater, but it's not doing the thing that Brett was advocating for, which is, what if I care about what movie I want to see and not where I want to see it? So just for funsies, I picked a movie that's in theaters right now called Black Adam. I've never heard of this movie.

I have no idea what it is. You mean you didn't pick Harry Plummer and the Goombas of Doom? No. The Harry Potter Mario mashup that we're definitely getting next year no i did not pick rain man forever or uh what was the other one oh yeah the little schemer an elephant journeys to find lambda the ultimate starring car cutter cons and cons. Yeah. I will say, I just have to make this small note.

Yeah. If you type in show times, one word, your zip code show times, one word versus show times, two words, you get totally different views on Google. Oh, interesting.

So if I do one of them immediately. Oh yeah, I do immediately pops up the, like the closest theater to me and then like the the time you know the little graphic that you were talking about of all these times but if i have like if there's two words if i have one word then it's the map view that you were talking about so if i do type in the name of a movie and then show times i do get a view of like here's the performance times near my location of this movie showing, you know, each theater in its own little section going down the page.

And then all the times that this movie is showing at the theater. It doesn't give me a nice timeline view so that I can kind of see, you know, when are these show times relative to one another. It is just a table of times like there's no spatial relationships being used to express the progression of time it's just a sort of a tabular view uh split up by different like ticket prices and features like you know 4dx avx avx dbox whatever the hell all those things are but in order to even get this you had to narrow down to a particular movie, correct? Well, I had to, in Google, type Black Adam Showtimes. One word. Yeah.

So I had to tell it, I want this particular movie. Yep. And you didn't put in your zip, though. It did use your location. Yes, it did use my location. Okay. So I feel like we can rabbit hole a little too much on this. I want to just put out the alternative, because we're looking at the alternative here that brett victor gives yes uh and and his whole point

is like it's you don't know it are people going to be caring about the movie and i want to see that movie are people going to be caring about a time i want to see a movie at a time or a theater or some combination of these right yeah and so his graphic gives us a list of these movies on the left die hard with more intensity rent and rentability and they have blurbs and star ratings and then on the right it's this graphic of there's colors for every single uh cinema that's around us and then there are times and each movie has its own kind of view here so that there's these colors of which cinemas they're in and then the times are actually these like timeline views so you know 11 25 is here and then there's a big space and then three o'clock and

then 7 15 way further down and that's like at old joe's show house these are the times for rain man forever and so if you're like scanning this and you're interested in like one like little in cinemas it's purple and so i can just look at all the purple times across each movie and be able to see what time that movie is playing at little in cinema or if i want to see a movie at five o'clock ish I just look at that sort of vertical slice of the timeline and I can scan down and see oh well at UAE Z Street Rain Man forever is at five and at AMC Lilliput die hard with more intensities at five but if I wanted to see Harry Plummer and the Goomba of doom I'd have to wait till 5 45 and we

have a like the graphic itself is divided in half with this like more shaded like a kind of grayed out and not grayed out so it's like the present it's like a line of what time is now and what time is the future and so like just from this any sorts of those questions that we want to ask, regardless of if we care about the theater, the time, the movie, we don't have to do any interaction. And in fact, we can't because I printed a PDF. I can't do interaction here. And yet, I can answer all these questions immediately.

And if we were to quiz ourselves right now on Black Adam and I asked, when today at this theater, and I did these different questions, if we compared what Google gives us, this billion dollar company offering us things for movies, versus a graphic here, it would be night and day on the amount of time and effort it takes for us to answer these questions. Yeah. And it's the point that I want to make about saying, you know what, I could actually print out this Black Adams Showtimes page and get all the information that I care about. I don't need to click on anything. It's got the summary. It's got the cast. It's got

audience reviews. It's got, you know, the Fandango percentage rating, the IMDB score out of 10, the Metacritic score. It's got all the theaters, all the showtimes, all the different tiers of ticket prices, that kind of thing. Like there's a ton of information on this one page that I could print out and it would be perfectly useful. The thing that I don't have is all software reconceived in this way i have a handful of specific examples like calendars maps movie show times arguably shopping websites there are some good shopping websites out there and there are some tools you can use to like make amazon better that kind of thing but what i don't have is i don't have this kind of a redesign taken to especially relevant programming and of course you know many many other things but this this this philosophy of

try to design the the way in which software presents information so that it leverages all of the lessons we've been learning for thousands of years about how to do graphic design and how to do that really well and not leaning on interaction which comes from industrial design which is also a really well established field it's just more recent and not even worse not relying on this kind of pigeon ad hoc sort of we're figuring it out as we go along mashup hybrid of industrial design and graphic design that people are kind of self-teaching as they are learning how to make software where, you know, we've now learned the lesson like from from from business metrics. Hey, that minimize the number of clicks that people need to do in order to get it the thing that they want, because we've learned, hey, every time somebody has to make a click, you know, 20% of people will bounce off of that. And so if it's in your sales funnel, get superfluous clicks out of there, right?

Like that wisdom pervades and is kind of, you know, in the cases like Google results or in signup funnels on websites that serve some informative purpose or whatever, like we've learned that lesson, right? If you go to kayak to book a trip, it doesn't make you sign up for an account before it lets you start searching for trips, right? So some of the lesson has been learned, but only in select cases. And what we don't have is we don't have all people making software, coming at software with the awareness that their craft is going to require very good communication skills and very good visual communication skills and that the computer is primarily a visual medium and the screen is

something that is it should give us everything that paper gives us and more and we don't take advantage of that and so that's that's my reason for saying hey yeah we do't take advantage of that. And so that's my reason for saying, hey, yeah, we do have the handful of examples in this paper because they were easy examples. And we don't have the hard examples. We don't have the universal examples. All right, hold on. I have a way, you know, I'm going to make my voice sound really cool. So a well-designed information graphic can almost compel the viewer to ask and answer

questions, make comparisons, draw conclusions. It does so by exploiting the capabilities of the human eye. Instantaneous and effortless movement, high bandwidth and capacity for parallel processing, intrinsic pattern recognition and correlation, macro micro dualro-duality that can skim the whole page or focus on the tiniest detail. Like this is the kind of thing that Brett Victor wants us to be focusing on is using our full capabilities of the human eye. Use the human body. Use it.

Yeah, yeah. I think this work really does, I think in a lot of ways, prefigure what we end up seeing in dynamic land, not just in this kind of ways, prefigure what we end up seeing in DynamicsLine, not just in this kind of focusing on the human capabilities, but also later when we talk about some implementation details, which I think are some of the exciting things I want

to get into for this. Yeah, let's keep going, because we're on the first sentence of the summary, the split between graphic design and industrial design. And we didn't talk about industrial design. I don't think we need to. Just the short summary is like, over the last 100, 200, however many years you want to look at it,

we've been learning how to build mechanisms that are nice for people to use. So it's that like, the quintessential example of a tool is a hammer. And a hammer has one end that is designed to fit the problem. That's, you know, the hard end you use to pound the nail. And the other end of the tool is designed to fit the wielder so that's the comfy grip that you hold in your hand

and for everything that human beings need to do to affect some change in the world industrial design is about how do you make the the thing that the person uses to do that effect as comfortable and as expressive and as good as possible. So that's that whole field. You can look at how we make software and say, wow, we are really not using graphic design to even a sliver of its full potential. The same can also be said of industrial design, but that's not as relevant or interesting to us immediately, at least in this conversation.

Yeah. So do you want to keep going through the summary? I'm happy to, but I'm also happy to, you know, pull out points that we want to make. Well, the second sentence, I'm ready to touch on that. So it's, information software is for learning an internal model. Manipulation software is for creating an external model. Communication software is for communicating a shared model. And I wanted to read that and then bring it up again, because we're now ready to, for me to say my short thing about how I think communication software is actually just more advanced manipulation software. So information software, we know that the pinnacle of that

is something that is as good as it can be when printed that you use all of graphic design to leverage the human's you know eyeball and brain to extract as much information as possible as quickly as possible and the way to do that is to not require that a person do a whole bunch of clicking around to explore an information space that you render that information space very immediately and give them a lot to hook into and you leverage graphic design to do that and the manipulation software is that plus they're actually going to be making some changes to the thing that they're learning about and looking at and so the graphic design is now married with really good industrial design where you know, you need to communicate to the computer how to change this internal model and so all of the things we've learned about you know, like

Like Fitts law, right? like if you put something at the edge or in the corner of a screen that makes it easier to click on because you can Just slam your cursor into the edge and it makes it like an infinitely large clickable area and that you know there's equivalence to that for touch screens and all that kind of stuff but there's all these like things we are learning about how to make really good expressive user interfaces and that's super relevant to the future of coding and i'm sure we'll come back to that at some point but that's not as much of a direct focus in this essay. And then communication software, which is not talked about much at all, is this idea of software that is about communicating

between two people. And the reason that I wanted to talk about it for a second is to say that getting information software, that first category, right, is hard. We barely do it. We're struggling to do it. Getting manipulation software right is even harder because it needs to be good information software and have this extra level of industrial design. Getting communication software to be good is even harder than getting manipulation software to be good because it is multiple people learning and manipulating together but the reason that communication software hasn't stymied us the way information software and manipulation software has is because we can more easily punt on the challenge of making you know that software

really good at expressing information and really good at showing information and allowing the person to shape the information they're trying to convey because it's a human on both ends and so both humans capabilities are brought to bear on this and so it's sort of like oh we can keep the communication software really basic and simple and make it like text messaging, right? Text messaging, we all know how terrible it is, right? Have you ever tried to break up with somebody over text messages? Or have you been broken up with over text messages? Or have you, you know, been fired over text messages?

Any of these circumstances that are like heightening the constraint of that medium really show how narrow a channel text messaging is and all the other kinds of communication software you know zoom calls and uh you know sending audio messages back and forth which is another popular kind of thing or something i do which is i have music friends and we record little demos and send them back and forth right that's using the computer as a as a kind of communication software the only thing that makes this good is that there's a human on both ends. And there's a circumstance that I thought of, which is communication software where you don't have a human on both ends, and that's games. And one of the things that makes games so interesting to me

is it is a kind of communication software where you can't just directly send a message from the designer of the game to each player. The designer of the game needs to come up with a system, and the system is going to express an idea or communicate an idea to the person playing the game. And you have to more deeply involve the computer as an intermediary in this communication from game designer to game player. And that really highlights just how challenging communication software can be when you can't just fall back on the human capabilities. I think a great example of kind of the point you're trying to make here about like communication software,

especially indirect communication software like games, being as important for these graphical aspects would be the witness. I think that's actually almost the whole point of the witness. No spoilers here. But the point is this kind of, it is a question about nonverbal communication, right? And how do we convey things to people? And it kind of goes back to this, I know, you know, looking at Jonathan Blow's conversations about this, it's kind of this like pre-language way of communicating. And we still do it today. We can communicate by mimicry and things like that.

I think that's really an interesting way of looking at video games, is this indirect communication application. And so that this interaction has to be crisp, has to be good, it has to have tight feedback, which is some of the things he says are required for any interaction, any interactive application. Yeah, and when you are learning to make games, you have to be, like to make a good game, you have to be a very good graphic designer.

And you have to be very good at interaction design and at industrial design to make the experience of playing the game feel good. You have to be so good at it that you can strategically choose to make some parts of the game challenging and frustrating and intentionally cumbersome because you can use that as a way to communicate ideas. It's such a mastery of the form compared to other software we work with, where if it's ever frustrating or cumbersome or whatever, you know that that's because of inability on the part of the people who designed it and incompetence on their part, not they were such masters of their field that they intentionally made the software hard to use, because that helps you get the full value out

of the software. Yeah, there's a great fairly popular YouTube video, I can't remember by who, you might know, where it was somebody going through kind of critiquing new Mega Man versus old Mega Man level design. And it's a great, hilarious video. But but you know they walk through like what made early megaman level design so good and why they didn't need a tutorial in these early megaman games and then later on you start getting these like very heavily tutorialized you know megaman games and then they just feel wrong and it's all about this like graphical communication that we have here and the way in which when you interact, your environment responds in this context-aware way that Brett wants us to have. He thinks, in fact, here, Brett Victor is quoting

from Jesus and he says, each interaction can and should result in a discernible change to a context-sensitive information graphic. And it's in red text, so I assume that this was Jesus saying this. That's what I learned growing up in church, the red text Bible. I don't know. He didn't cite the chapter or verse or verse but it's in red text so it must be jesus noted interaction designer jesus christ it's just like literally i know that's such a stupid thought but growing up that way yeah i just as soon as i saw red text i was like wait he's quoting from

jesus oh my goodness my brain immediately went there i even have it in my notes here jesus said this i mean the way we revere brett's work right might as well might as well just make it explicit uh yeah so um, I think this is a very interesting point that like, you know, we when we do this interaction, it has to result in a change in the graphic. I think this having this as a principle or constraint, and looking at how all of our software either, you know, lives up to this or doesn't, is a really interesting way of critiquing a design. And I've seen a lot of things where users get very lost, where I get very lost, or especially more non-technical users.

I'll help them because they can't tell what difference their interaction is making or if it's making a difference. And the only reason I know is because I can kind of mentally peek under the hood and think about how would a programmer program this and what might the invisible interaction be behind the machine. And in fact, this is what Brett Victor says is the reason that we've gotten in here. The reason that we've had this problem where instead of focusing on graphics, we focus on mechanical metaphors, is that we, the software engineers, think about our computers as machines

and about how we manipulate them and we kind of implicitly make software that assumes this almost metaphor. Do you want to summarize what Brett's issue with thinking of software as machines is about? Yes, I would love to do that.

I think we've kind of been talking about this, that you were talking about communication software, there's humans on each end. And it's not just humans, it's that the medium that we're using, we're allowing freeform language. We're letting language itself carry so much of the information there. And so he talks about this distinction between kind of linguistic software and software that is GUI. And he says that it actually forms a language. The GUI language consists of a grammar of menus, buttons, checkboxes, each labeled with a vocabulary of generally decontextualized short phrases. The user speaks by selecting from a tiny, discrete vocabulary with an entirely fixed grammatical structure,

a bizarre pigeon unlike any human language, unexpressive and unnatural. And then there's a little side note here. One might wonder what Sapper and Worf would conclude. Which I just love this idea of thinking of I think we as programmers, especially more this typed-bent

love to think about things as a grammar, love to think about oh, we have this algebra, we have this grammar that we can work on, but he's pointing out that it's such a bad grammar. It doesn't give us the context sensitivity that we actually want as humans. And so some of his interfaces that he's going to

recommend, and I think you see this in kind of the explorable explanations world, is having text instead of a big, huge list of all these drop downs and options you could choose, have a little paragraph. And he really exemplifies this in the example that we haven't quite gotten to the BART, right? So the San Francisco train system, it's an application that he really made for that. And the settings instead of being your very typical checkbox boxes and dialogue buttons and all these you know things is a little paragraph and it says things like do announce trains from castro valley and the do is like red and so it's like you know able to be manipulated and there's yeah it's just there's like very clear contextual sentences that convey so much more information than a big series of check boxes

and things would. I thought it being red meant something else, Jimmy. Yeah, yeah. No, so this is a darker red. Oh, okay. Yeah. So it's dark Jesus. Yeah. This is a goth Jesus. Yeah. You know, when he rose from the dead, he started a metal band, you know, etc. Right? Like, as you know from scriptures. As one does when they rise from the dead, you immediately start a black metal band. Yeah, exactly. That's what- Yeah, when I rise from the dead, I start black metal bands. Yeah, like that, we did talk about that and to tie it together, that idea of, you know,

if you're picking a date from a list of drop downs, that sucks. That's bad. The machininess that I want to get to is, and I'll read this as a quote, stemming off of something we've already talked about but leading us maybe somewhere new. Compared to excellent ink and paper designs, most current software communicates deplorably. This is a problem of surface, but not a superficial problem. The main cause, I believe, is that many software designers feel they are designing a machine. Their foremost concern is behavior, what the software does. They start by asking what functions must the software perform, what commands must it accept, what parameters can be

adjusted. In the case of websites, what pages must there be? How are they linked together? What are the dynamic features? These designers start by specifying functionality, but the essence of information software is the presentation. And I want to branch off of this thought to tie it into the future of coding a little bit but also just how we build software and say when i think about the way that programmers get excited about advances in technology or changes in popular practice or new approaches to building software yes sometimes it's the good shit, like, you know, a new Brett Victor essay or a video drops, and then everybody's like, Oh, my God, I'm going to

spend so much more time thinking about the meaning of my designs as I'm working, right? Like, I've, you know, been following Brett Victor long enough that I've seen these cycles happen, where he'll publish some new thing. And then there's this huge reckoning that, you know, hacker news and, and various other blogs and such do where they kind of realize, oh, yeah, no, we're not, we're not doing it right, we need to do it better. But the other side of it is the things that people get really excited about are like, ah, you know, now we're going to express our user interfaces using streams of events rather than using bindings between some input and some function call right like i'm going to build a web app and instead of using on click i'm going to generate a stream of click events and then have a an observer of that stream or a

consumer of it and then have stream combinators where I can like split those events based on whether it's a mouse down or a mouse up or a mouse move and then I can you know have a filter in there and do all these transformations and it's like the things that excite us about building the software as programmers as engineers are the machiny bits and I mentioned this in the last episode, right? We get excited about bun and we get excited about ES build and SWC because we can make the machine more machiny. And this essay does a really good job of saying for most software, which is information software, and even for the large portion of each piece of manipulation software that is not about doing the manipulation but is about learning the shape of the thing you're manipulating and then also for communication

software when you aren't just relying on humans you know to do the heavy lifting but you actually want to put some more substance into the middle of the communication In all of these cases, thinking about the machine is, it's a vice. It's not a strength. I'm going to say it's both. But I think in this essay, it's definitely considered a vice, right? I do think that there's something very important about the machininess. And it might be it's only important enough to overcome it because I do think a lot of what we aren't able to achieve is because we've kind of ignored the computer as a machine and our software has become so much slower.

And so we aren't able to accomplish certain things because the software we're using is so slow that we have to be super clever or it might seem infeasible for us to implement some of these things in a nice performant feedback critical way. Especially now, I'm much more thinking about the computer as a machine since I'm doing compiler work. But I really do think that we have to hold both as software engineers. Maybe a designer should fully separate out from thinking of the computer as a machine.

But I think as software engineers, we have to be able to kind of hold both that it's a machine and that it's a display of information and not hold one above the other. Whereas right now I feel like, yes, you're right. We have this like primacy of machine. Yeah. We have the primacy of machine like within our culture. Yes. Within the things that we get excited about. Yes. And there's this shame aspect of it too, right? Where a lot of programmers, like if you ask them to design an interface or

something like that like they'll there's this term of like programmer art you you brought this up yourself earlier when talking about interfaces and it's maybe it's just my background in the arts kind of made it easier for me to feel comfortable thinking of myself as a designer first and a programmer second and for people who don't think of themselves that way this this is all foreign or intimidating and the the focus on the machine is is fun and comfortable and exciting and and so this is kind of a a hard change to imagine one's self-making you know hey, hey, I'm actually going to work to eliminate the feeling of machininess in the things that I build. Yeah. Because this goes deeper, right? It's not just like the things that I build for end users, but even like the tools we build for each other,

right? We're in this future of coding community because we want to make a new kind of programming. And there's some people out there who think, oh yeah, everybody should be a programmer. So when you're making a new kind of programming, it's going to be for all people, right? People who don't think of themselves as engineers, people who aren't excited about the machine. But others in the community, myself included, don't have that goal. They don't have the goal of everyone should be a programmer. They have the goal of for people who do programming, whatever subset that is, can we make that programming better? So those are the people who are already comfortable working with machines but even there it's like programming

tools could be so much better at communication and it like expressing information like how do we get there how do we how do we do that if the audience we're serving is technical and the people who are doing that serving are technical how do we get those people to increase their level of comfort with the non-technical parts of the problem yeah i could imagine an earlier me having read this you know like i i read it a little bit later but you know had i read this super earlier on where i would kind of be almost offended by the suggestion yeah right that like i would think this is exactly the opposite of what we need um you know we need more precision we need more exactness we need more clarity and logical structure we need more of these machine-like qualities because it gives us a certain kind of predictability and i can imagine even seeing like you know a critique of this sort of thing

where it's like now what you're asking for is every interface is this bespoke you know work of art quote unquote and there's no predictability of my software there's no common patterns i can't just like open up any software and operate it on the way that i want to and expect it to work i have to you know deal with this designer's viewpoint on things and kind of deal with the fact that they want to lay something out different just to be different, right? And so you see a lot of people who would want kind of this conformity in software and make it more machine-like, make it more automatic, make it more consistent. And I don't know, there's part of me that still feels this pull, especially as we

get into Brett Victor wants to tell us like, how do we get this new information software revolution? How can we go beyond these mechanical metaphors that have dominated and really have better software? And in some ways, his answer kind of is like, kick out the software engineers. You know, that's a very not charitable way of putting it, but it is a reading that you could have of it is that we need to not have people who are interested in computers because they think of it as machine be the ones designing software. There was a moment in history where it almost felt to me like this was happening. And I'm sure this has happened a bunch of times

before my time and since. It was in the early 2000s when Flash was having its heyday. And it was the way that you could do dynamism on the web. And the flash player was installed on 97 percent of computers every website for every restaurant and every hotel and every you know if somebody's making a new game or if it's nike or you know all of these companies at every scale and all of these individuals were using flash to give each

website its own sense of style and flair and there was such a an excess and it was so amateurishly done by people who didn't have a background in formal design and in sort of the great works of visual communication that you'd have you know there was the time where the skip intro button was a really common interface element because every website wanted to have a flashy intro and i like i was making these kind of things at the time i made a a website for a mortgage broker in calgary and like you open the website and there's this like rolling green field with cows and little houses sprouting up and the kind of badly cut out photos of the different agents working at this brokerage, like just kind of popping out from behind the hills. And it was like very comically bad in hindsight. And of course, there was a skip intro button because there was this whole animated intro kind of thing but what what you had there was all of this information software was being designed

kind of like games are designed and i keep coming back to that but yes each one is bespoke in its own kind of way but they can't be so bespoke that they're not usable and there was this kind of the culture went through this learning process where we sort of figured out how to make things unique and interesting and exciting when it was appropriate to do that right like probably a mortgage broker site shouldn't be that but like the jim carrey website should maybe you know a hospital shouldn't have an interface that is trying to make it like look at all of our cool services that we offer like two minute intro video and it should instead have like you know like the cloud for style like are you under attack button in the top right are you having a medical emergency button right like you should understand

the context of use for what you're making, but there's a lot of contexts where having something unique and fun and playful is good. And because Flash started as a graphic design tool, emerging out of Director and HyperCard and that sort of lineage of the first interface that you encounter when you open it is the drawing interface and you have to dig behind that to get to the programming interface people would start using them and whether they thought of themselves as artists or designers or something else or a programmer right no matter how you came into it you would learn and work with the graphical interface first, even in a really primitive way. You'd make something terrible where it's like, oh, I've got this rectangle spinning that changes between three different colors or whatever. And then get to the programming interface where it's like, oh, I can actually put some code on keyframes in my animation so that as my animation runs, it's a little bit interactive. Like it pauses at this frame and then there's two buttons and I click

and it goes to one of two different parts of the timeline depending on which button I clicked, right? Like a classic DVD menu for the people listening to this old enough to remember digital video discs. That art-first approach to the environment that was the dominant way of making media on the internet. Yes, it led to a lot of superfluous interactivity that is bad, but it was all bespoke interactivity. And the cultural learning that we went through there was cut off very abruptly. And I don't see that reflected in

modern website interfaces. I see like people trying to claw some of that back like i see a lot of programmers with really cool personal websites like um i'm gonna totally butcher their name but toph tucker they made a new website i think it's new recently where it's like you load the website toph tucker.com like it's it's a very simple text website but each letter of each word is treated as its own particle and there's this like you know spring physics or something like that blowing the letters of the words around the page and when you move your mouse to a spot any words that are supposed to be laid out in that spot kind of swim back to it and form into the word so that you can read them so you have to read the page by like mousing around it and it's superfluous interactivity it fails the printer

test but it is using software from a graphical first mode of thinking and that's hard to do these days because all the tools we have for making software are systems first, they're machine first, they are, you know, you're gonna build this interactive widget, what's the first thing you think about? Oh, do I want to use async await? Because that's a new API for handling stuff? Do I want to use promises? Do I want to use streams? Do I want to use you know, which which approach to asynchrony or concurrency or uh parallelism or whatever do i want to use in this project because that's gonna bleed through and infect all of my code that's where we start thinking about it where in the heyday and i think games

have this now with with engines being so good you start thinking about the way that it looks first and work from there or the way that you interact with it first and work from there and think, how much interaction do I want? Where is that interaction going to happen? But we also need designers to be able to build that kind of software without requiring code. I think this is definitely a theme that you see through all of Brett Victor's work of kind of letting non-programmers build these sorts of software systems that he wants to see created. Maybe we can start with, I don't want to like maybe go into each of these points, but he does want to kind of give us, I think it's always good when someone gives us a step-by-step

program to kind of read it. He says that there's a way in which we can get to this information software revolution. And I'll just read these steps here, kind of the first statement of them. So the first step is to have a recognition of the need for design. And I think that's one thing that you were talking about there is, you know, not everyone sees that. And then it's going to be finding people with the talent for visual communication. Then it's going to be giving people the skills to use that talent. Then it's going to be making tools and platforms for letting people actually express that talent. And then finally, it's going to be having an environment where experimentation, evolution, and interplay of ideas can thrive. This is the path that he wants to set us on.

And these are kind of the steps that he is kind of setting up. Obviously, some of these are just about people, but ultimately it's really these like fourth and this fourth and fifth step here that he wants to focus on is what can the tools and platforms be, and then what can an environment be to really let these sorts of innovative, interesting, context-sensitive, rich, dynamic, Magic Ink-like applications flourish. He kind of sketches out this first tool.

I don't know if you quite call it a tool. I don't know. But he kind of gives an example of how you might build an interface without code that has this kind of rich context-sensitive dynamicness to it. And he starts with an app that he actually made, which is this BART widget that shows you the times of trains coming and leaving from various stations. And he does an interface that at least seems very familiar to me now, which is these kind of like bars of like there's a timeline and then there's, you know, for each train there's various bars of like when they're departing and leaving and they kind of take up the The width is how long they take,

and there's a little annotated of at 517 it's going to leave, and then at 545 it will complete its journey. It's like a Gantt chart. Yeah, yeah, yeah. And so what he wants to do, and I don't think we have to walk through all the details of how this works. It doesn't make for great uh radio as it were uh but he walks through how we could start from concrete instances of someone drawing this graphic and allow people to make a new instance and then the program

underneath can infer the difference between these and create the dynamic, not interaction, I always want to say interaction, the dynamicness of the application just from these inferences. Yeah, it's dynamic in the sense that it is responding to context, not responding to user input. Exactly, yes. It's responding to the underlying data, which might even include the environment in which somebody's in, right? The time it is, the place they are, etc. And then also other kinds of environment will include things like the history of what the person has done

before. And so the program should be building up a memory of things they've done in the past and using that memory to inform how it's going to do its best to present the most relevant information. And that's getting back to that idea of the purpose of information software is to present information that you can think about without having to click on a whole bunch. And one way to avoid clicking on it a whole bunch is if the program is smart enough to figure out what to show you at any given moment. Yeah, and I look, there's a really small tiny bit of text here that says, and to do that you need FRP.

We have this train interface that we're trying to have the whole context of the user being loaded into the application. We need to look at the history. We need to look at the ways in which the user wants to interact with it, but also the underlying data. And that data, of course, is going to change. And so the graphic needs to be able to change. And what we get is this very simple inference process where you can draw one graphic with, you know, one bit of text and then the same graphic with a different bit of text. And the underlying software is supposed to infer that, oh, we must be assigning, you know, this data to like be placed here. And then we might have a length and we'll have one graphic

with a certain length and a certain amount of time. And then another graphic with a different length and a different amount of time. And another graphic with a different length and a different amount of time and we should infer that this is connecting the length up to the time that we have here. But it gets even more than that. We also might want to have like descriptions of like in two hours and so we might have one that's like you know it tells us the exact amount of time and then we might have like in one minute versus in 119 minutes, we probably want to say in two hours or something like that, right? If it was like 120 minutes.

And so there's even this like interpolating, like what will that look like based on these little stepwise functions? And so what Brett Victor is trying to get us to see is that as programmers, we actually could implement these inference functions, that these are actually not that complicated. They're just unfamiliar. Most people aren't aware of,

or at least weren't at the time aware of, the background work done in programming by demonstration and machine learning. Exactly. That would make this kind of stuff easily doable. Yeah. And so, you know, there's no reason that we couldn't build a tool that let designers do these sorts of things without them having to directly specify these functions.

But ultimately, he even says, like, we could expose these, you know, inferences to the end user, and they might be able to help us out in those cases. There's no reason that these have to be completely hidden behind the scenes. Yeah, when you say end user, you mean the person who's building a piece of software, the end user of the design tool, not the end user of the resulting software. Yeah, and so he walks us through this very logical set of steps to build up an interface that, honestly, if I had to code it from scratch would be pretty complicated. It's non-trivial to even code from scratch, and if I walked you through what the code would look like to create the interface absent this tool,

it would be a much more complicated set of steps than the tool that he's made to let you build this. Now, I do, of course, wonder how much that tool would generalize, and could it only really build this interface, or what the expressiveness of a tool like this. And of course, this is kind of just a sketch of an example process to show that what he's asking for isn't that complicated and is feasible. He says as much in the essay. He does say like, hey, this approach that I'm articulating here really is focused on

how you would build this one program and it's not general. Absolutely. But he gives us kind of a guiding principle that I think is really interesting, that the essence of this process is elimination of abstraction. The designer works with concrete, visible examples. And I think this is really interesting because we as programmers really want to work with abstractions in a lot of ways. Like we kind of pride ourselves in our ability to work with abstractions. And so having a design tool that's whole goal is not having abstractions and yet offering dynamicness is kind of, I don't know. I get what he's saying here that we're not working with abstractions.

At the same time, it feels like we're working with abstractions to me. It's the abstraction of a linear function or this inference that can be had. But I guess from the designer's perspective it probably wouldn't feel like that. I think there's a little bit of a misdistinction going on here. The designer using this tool is still working with abstractions, but the way that they work with those abstractions is in terms of concrete examples whereas the way programmers work with abstractions is in the absence of concrete examples and we have to do a lot to create concrete examples um to make sure we

built the right abstractions so things like writing tests or doing like the kind of um testing of software where you run the software and then go click through it a whole bunch and make sure it works we we have ways that we we can kind of make sure that the abstractions that we've built are the right things and are doing the right job but this is this is an alternative approach where you don't start with the kind of the algebraic like let x be this here's my formula kind of abstract construction you're starting with concrete numbers right like i have a five and i need to get 15 what are the transformations that i can do to a five to get to a 15 and yeah you act out those

transformations you don't require the person to do those transformations and those examples in their head and then go, okay, I figured out what the general principle is here. Now that's the code I'm going to write. You let the design tool be the space where you do that thinking of coming up with the abstraction and you do it collaboratively with the computer rather than outside the computer. And then you come to the computer rather than outside the computer. And then you come to the computer to put in the final abstraction that you've come up with. See, this to me would be a great example for the Brett jam that someone should throw. I would love to see a working version of software that just made this. That would actually be more helpful for me because I thought about even, I just didn't have time,

trying to go starting to make something like this with these little inference algorithms and stuff. But it would be a little bit, I can understand how to do this, but it is a decent amount of work, right? It's non-trivial to go make something that would let me build an interface like this. A lot of the parts are not the

oh, well, I need to make an inference algorithm and that's going to be hard. It's like, I need to be able to draw that circle there. I actually looked at, are there any existing tools where I could just have a nice TL draw? Is there an easy way that I could just get all of its events and then start building something on top of this? Oh, Jimmy, you're hurting my heart what we have that have had had had had that tool john well henry had had

had had had had had had had been my preference we had that tool it was flash flash was that tool flash was the tool where you start with a drawing canvas you have all these really half-assed not very great but good enough ways of putting some text down making that text be a text field having a rectangle that is the same i think i think we're talking past each other because i want editor time code execution so as you're drawing in flash i want to execute code that's how flash worked you could and there's different ways of programming in flash some of them more programmery than others but it's a gradient where or it's like a progression or a ladder where um the first way that people encounter code in flash is you can put a piece of code on a keyframe on the timeline so that as the timeline is running,

when it hits that keyframe, it executes that code. And you can do that in the editor. You don't need to go and run, compile your thing and then test it out separately. You can have that in the editor. Yeah, I think we're still talking about that. I'm saying Ivan is sitting down building an application in Flash and he draws

a rectangle and at that moment when he draws that rectangle, Jimmy's code executes and draws another rectangle on the screen in Ivan's editor. Like, I want to plug in for the Flash editor itself. I'm pretty sure that you could work with Flash

in that way. There were downsides to doing it. It wasn't as good as it could be, and it wasn't built for this, but it was the closest that I've ever seen. Yeah, I looked at tldraw, and I looked at Figma, and I didn't, in my 15 minutes of research in each, I couldn't find just an obvious, like, give me every event that happens

when you draw a rectangle sort of thing, right? So, because that was the harder part to me. I would now need to go build a little graphic-y application to let people draw things and I didn't want it to look completely garbage. I wanted it a little bit better. But I do wish that this just was an open source tool because I would go and dive into it and learn from it and then go use it for something else. I wouldn't be satisfied here. And so if someone was, cool, they got a nice tool out of it. But for, you know, I don't know. I would love if that existed. And he continues on besides just

drawing dynamic graphics here and inferring things to inferring from history and he kind of gives us we don't i don't think we have to go into like the details of how he infers and the weights for different times when people look at it and you know there's some really nice little graphics here and it's a very particular solution and tells us that but ultimately he says that this sort of machine learning we don't need richer and richer, more advanced machine learning algorithms. What we need is to make an abstraction to let anyone put learning into their application in a very straightforward, simple way. And actually, there was a proposal for Swift that would do exactly this that I don't think ever landed. Or if it did, it turned into a much worse version.

And their example is VeryBritVictory, where instead of inferring times, it was inferring how fast you... Is it a 2x speed up for your podcast? Is it a 3x speed up for your podcast? Do you listen to this one at 1x? And they would infer that from your behavior over time and it was like three lines of code to do it was it was really nice actually i would have loved

something like that yeah because that's like brett's proposal is that maybe programming languages should have a learn keyword yeah and that would be really cool and i think there are some things out there you know apple actually has i think it's like Turicreate, I think is the weird name that it is, which is like a very simple machine learning setup where you can do a lot of inference in just a couple lines of Python. Continuing on, Brett Victor gives us even more implementation details around like how we can do this inference to inferring from the environment, not just from history, not just these dynamic graphics, but the environment itself.

And so he gives an example of like, you get an email saying, hey, dude, wanna get some pizza? And then ultimately you should get like a map or something of pizza places that you could go order from right away. And so he kind of walks through the details here. And what I think is most interesting

is that it's supposed to be this very open system that you don't have to go make one algorithm that looks for pizza and then it grabs a map but instead you have a series of different services that offer i think in this case they're like xml yeah they're like x XML output data, which is very of the time. But you have this series of applications that output different entities and things, and they'll be kind of this open system that will know how to respond to this. And I actually think this is really interesting

because this is very similar to how Real Talk works, where it is this very open broadcast system where any application could respond. And you can kind of put things out there that you don't know anyone will respond to. But if someone starts to now, you get more functionality over time. It's this sort of philosophically different approach to what an operating system could be that I've seen come up in quite a few places recently i think programmers are starting to pine for this which is what if

your computer had a really good database inside it and what if there was a category of application that was just about working with that data in that database and enriching it and transforming it and simplifying it and then applications could be written on top of that database and that ecosystem of transformers to look for things in that database and surface them in valuable ways and i think maybe we get a little bit of that um the experience of what that might be like to have if you like fully buy into the google ecosystem or you fully buy into the apple ecosystem and they can use all of their different apps and all of their different data about you and what you do to try and tie these things together but of course and he even mentions this in the paper that like the bad way to do this would be for a giant platform to try and do all of it for there to be like one or two big companies you know

that try to own the entire thing and what you want instead is you want this underlying database abstraction to be so open and good that lots of different individual people will write their plugins for it and their apps that sit on top of it and you get a sort of an emergent non-designed behavior to come out of it. Yeah, and I think this is definitely a tale as old as time in some ways. This isn't something Brett is inventing here. It's definitely asking for things that people wanted. But I think it's really interesting the way in which he's using it here.

And I do remember some systems around this time trying to accomplish these sorts of goals and they always kind of fall into these failure modes of like over complicated like the semantic web yeah was kind of like in this very similar vein and you kind of i assume it's kind of like reciprocal here where like he's probably probably influenced from semantic web things and maybe he you know went on to influence some of the other ones but like i think the general point here of inferring from the environment of inferring from time you know all of these things are things that i don't know i i'm of two minds on them in some ways they sound really good in other ways some of the most frustrating things i have with software is when they try to infer my behavior and they're wrong

and i would rather them often be predictable than be smart uh i'll give an example yeah so you know i just switched to the new iphone from android And on my Android keyboard, for a while, they didn't have predictive text in the way that iPhone does. And so instead, in this little top bar where you might get predictive text, it was just punctuation. And then they started adding predictive text. But then when you push space, instead of it being predictive text, it would go back to punctuation. And so you'd be able to, like the exclamation mark was always in the left, and then the middle was question mark. And so like you could do like, you know, exclamation mark or question marks, you know, very predictively. Or like you knew where

they were, right? They were always right there. Eventually, they decided to start predicting whether it was a question or an exclamation. And so if you typed in like, what is that? It would automatically put the question mark in the left instead of where the exclamation mark used to be. And so when it was right, I would sometimes, you know, click it properly. But sometimes I want to have like, what is that exclamation mark? Not question. It became really, really frustrating the way in which they inferred these punctuation marks

for me because I could never build up a habit. And in fact, right before I switched, they removed them completely. And so now what they put in place there was the insert GIF button. So I would be like, what is that? And then insert GIF, and now I'm in a totally different menu. So yeah, they broke my habits by trying to predict my behavior. And I find that so incredibly frustrating. So I don't want to over-index

on this particular failure mode of this example, and I'll counter it by saying, here's another way that we could get this vision and have, in some ways, gotten this vision that he's sharing here without falling into this failure mode. And that is that the exact way in which you use each of these contexts

isn't the thing to pay attention to in this paper. Instead, it's the idea that there are contexts and that they should be made available both to the people who are designing software and in some way or another to the people using the software. And so an example I'll give is the iPhone, because we keep coming back to that. And in fact, we're going to find out in a second here that the next section of this paper actually is a prediction

that we will eventually get an iPhone-like device that does a lot of iPhone-like things. But the iPhone has been adding more and more and more hooks across the OS so that you can customize its behavior and add your own widgets and add your own shortcuts and that sort of thing. And so one of the things that you can do is set it up so that there are widgets on your home screen that launch certain applications or that launch shortcuts. And those home screens can change based on context awareness. So you can have home screens that come on at certain times of day, or you can have home screens that come on at certain locations, or home screens that come on when a shortcut runs, that kind of thing. And so you can do things like say you know hey when I'm at a grocery store

I want my home screen to change so that the row of widgets that are there one of them is a button that launches me into my grocery list app so that when I'm at the grocery store I don't have to pull out my phone open it up find the app and open that, and then get on the right list, I just have a button right on my home screen that I can tap, and it takes me into that. And the phone isn't doing that on its own. It's not predictively saying, hey, whenever you go to grocery stores, I'm going to put a button for your grocery list app on the home screen. It is instead giving you a way to say, hey, you can tap into location data or time data, like, you know, set something relative to sunrise or sunset, or you can tap into other kind of periphery data that exists. And then you can trigger something to happen as a result when that data matches

certain criteria. And the thing that happens as a result can be changes to the mode that the phone runs in. And those different modes can offer different interfaces that let you do different things with them. And so you as the end user can kind of cobble together the structure of interaction that you want to have and is it good no is the programming experience good no are the kinds of information that you have access to good no it's it's absolutely not living up to the vision but it is interesting in the way that it's sort of like you can feel the beginnings of apple figuring out that this is the way that people want to interact with technology and this is the way that technology wants to interact with people and i'm using interact not in the interactions considered

harmful sense but i mean more like the the relationship that exists between people and their technology should have this flavor of context awareness and that that context awareness should enable the interface or the information being presented to be dynamic and that all of the different ways we have of working with context whether that's explicit rule setting like you see in the design app where you're building the BART widget and setting up explicit relationships between scheduled data and graphical layout, or whether it's the more implicit kind of thing about noticing, oh, hey, we're talking about pizza in this email. Wouldn't it be cool if you could search the web by just typing the word pizza in any text field in your os and your

os would know hmm they're thinking about pizza right now so when you flip over to the map app one of the suggestions it's going to make is do you want to look for pizza or when you flip over to the you know the recipe app one of the suggestions is hey do you want to make pizza which you know gets into the other thing about like how much is this creepy how much has google kind of pissed in the pot by abusing this and how much you know facebook i don't want to just throw google under the bus facebook the f word um how much have they kind of ruined the soup for everyone um and how much has surveillance ad tech this vision up but i think i think that vision is good and we're getting

closer to it little by little yeah so i think there's some really nice things here and i think there's a lot that we could learn from and there's things that i've been trying to think about like i'm i'm trying to maybe design a debugger uh kind of specific for me like a debugger for my work and one of the things you know i even pushed pushed about this in dev log together on Slack, is like, how can I be more context aware in this, like, if I'm actively working on changes, maybe my debugger should show me the code that changed, because that's probably where I'm going to put some breakpoints. Or if I check out a branch that had a failure

in CI, maybe it can go ahead and look at that failure and figure out the stack trace and kind of show something around there, or show me the code that was the get diff. There's things like that that seem like very context aware that's probably going to be right. And if it's not, it's not going to be too hard for me to overcome. At the same time, I think there's, I don't know, part of this essay feels like it throws the baby out with the bathwater in some ways. And the reason I say that is I think about interaction itself.

And I think about there's kind of this underlying principle, implicit or kind of norm or value here, that ultimately what we're offering people is efficiency in learning, right? Like we want them to learn something in the most efficient manner. And I actually think that that can be kind of an anti-goal in a lot of activities that we want to do. Laying out these graphics so that, you know, I can answer these questions might actually not reinforce my learning. Whereas if I have to search and navigate for them, that search and navigate process actually can help me remember things better.

But also there's times where things that I think he would class as information software or would say that we should focus on some of this graphical aspect that actually the interactions themselves are what I enjoy about the software. So I'm thinking of the not boring suite of applications. And if you haven't checked these out, I think they're really interesting. And they're these kind of like three, they're like overly ornate, unnecessary interactions for these applications, right? Like there's a calculator, there's a timer, there's weather app and a habits tracking app.

And I started using habits on my iPad and I really loved it. And it's this like, very, I don't know how to really describe it well, but it's this very kind of meditative habits app where you hold down on this check mark to say, I've accomplished this habit that I wanted to today. And it gives this like beautiful vibration that I never got to feel when I was on the iPad. But now that I have this iPhone, like I get to feel this beautiful vibration, this wonderful sound.

And this like 3D scene is put in place and these birds are flying around. And it's totally unnecessary for the task at hand. And yet, it's what draws me into the app. It's what makes me want to track my habits. And for the weather, I can fidget with this 3D spinning weather and I can hear the thunder roaring. And the interaction itself brings me joy. And it reminds me of like making pour over coffee. Like I don't do it because I want to get coffee in the most efficient manner. I do it because of that enjoyment of sitting there pouring. And, you know, the kind of frustration of not getting the coffee I wanted sometimes

actually makes it even better when I do get that coffee and so I feel like this is an element that's just completely missing if we got rid of interaction I think that there's it's getting rid of it's getting making these interactions more joyful if we're gonna have them and yes I know you could probably read that into him here and you know maybe I'm nitpicking too much. But. I was not. I totally wasn't going to do that.

Yeah, yeah, yeah. No, you definitely weren't. But it definitely isn't something that I think people will come away with the essay. It's not the main thrust of the essay. And I'm sure Brett Victor would agree with it or whatever. But I just want to put out there that interaction considered harmful might be a little too strong. Yeah. Can I tie this into one of my – this will be quick, I promise.

Yeah, absolutely. One of my – how do I say this without using one of the tired metaphors that is about animal abuse? One of the things that I repeatedly talk about – Your hobby horses. Yeah, sure. There's a lot of metaphors to describe a thing that you do over and over again

where the structure of the metaphor is, ah, here's this animal that I'm going to injure repeatedly. It's weird. Yeah, yeah, but a hobby horse is, you know, there's nothing... Sure, yeah. So I've got my little broom with the horse head on it that I'm running around on.

And it's that the problem with essential versus accidental or essential versus inessential complexity, that's a recurring theme in works that are of interest to our field, we'll talk about some of those works at some point, is that incidental complexity is bad and to be minimized, and essential complexity is also bad, but it's essential so you can't get rid of it. And the strive for simplicity means you want as little complexity as possible. And my take on that, spoilers for upcoming episodes, I'm sure at some point, is that incidental complexity is actually good you just need to choose what incidental complexity you take on and what the shape of that complexity is very judiciously you need to approach it as a design thing and these not boring apps are good because the interactivity that they have isn't interactivity

that interferes with your ability to use them as information software. So the weather app doesn't make you do a lot of tapping around to get at the relevant weather info. It shows all the information that you care about immediately and clearly and nicely. It does a good job at nailing that core learning responsibility that the software has as a piece of information software. The interactivity is placed around that as something you can do purely for fun. So it is an ornamentation. Like you said, this is heavily ornamented software. It is what makes the software exquisite it makes it better to use because that interactivity

is chosen very deliberately to be good it's not something that is put in there because of inability or incompetence on the part of the people designing it it's a flex it is we're so good at software design that we can nail the information representation the the the core purpose of the app and then do a whole bunch of dancing around it at the same time you come to it for the dancing around the core information but you stay using it because or yeah you you come to it because of the of the ornamental excesses and the style and the flair and the sound design and the gameness of it but you keep using it because the the core information presentation is so good yeah no i i mean i don't disagree with

you at all i think that you're spot on with this. And they do a wonderful job of this information hierarchy and everything that's in this essay. And I guess that's what I find so interesting is it's kind of like taking all of the things from this essay and then adding back that interaction. And I think that's genius. And I wish more apps kind of felt like this. And that's one of the things that I think we could, as a software engineer who definitely does not see themselves as a designer first in any way, I have built a lot of interfaces. I have decent enough taste.

I can tell you when something's bad, but I'm definitely not someone who's a master of visually laying out information or any of that. I find this really a helpful essay for, honestly, it's very practical. I think of all the essays we've read, of all the papers we've read, this is probably the most practical one. And not even necessarily just in its details of these inference algorithms, but just in how to think about the software I'm making

and how to really orient it less around features and more about questions being asked and how my software can facilitate answering those. And so I would love to see more languages, more programming tools, etc., designed around that from the beginning. I would love to see programming tools that answer questions just from if you printed them off, right? That we could pull up a debugger and see all the information we need at a glance rather than having to click through all these little menus. That we could see a program and find out all sorts of facts about the program

from a glance that go beyond just like, I built a graph, right? Like, yeah, we've all seen a call graph, and that's cool, but there has to be better ways, and I think there are, and I would love for us to explore those more as a community. So one of the reflections that I had on this essay is that this is actually very similar to augmenting human intellect and to me this is an example of like what would a better presentation of augmenting human intellect have looked like and i i see that similarity both in that it is sort of talking about the way that the world is facing a problem and that that problem is going

to get worse and worse and that we are going to need to come up with better ways of grappling with that problem that sort of high level like here's here's a society-wide issue that the advent of technology is bringing about and we as conscientious designers and engineers and thinkers can participate in alleviating that problem and it and it once it sets that groundwork it gets into it in two ways but you know both augmenting human inflect and magic ink do this they look at concrete examples of here are some things that one could build that would alleviate that problem and in magic ink that is you know here's this design tool and it would allow people who are good graphic designers and good people who understand visual communication

to build software that is able to respond to context and is able to afford a little bit of interactivity and is able to surface information really elegantly without them needing to learn how to program and all that nonsense so there's the like concrete example but then there's also the meta level how do you design a design system or what is the substrate on which these sorts of implementations like the design tool that is used for the bart widget or an anglebert like the you know the thing with the two pens and the big desk the the clerk where you can sort of array all of your virtual documents and work through them those sorts of examples are examples. And the real thing in here is the system that can produce those kinds of tools. And so Brett spends a fair bit of time in this essay

towards the end, kind of talking about like how to get to the world that he wants to see. And it's not the concrete, like we need that design tool, but it's the more general, like I'll read a quote here. Today's ubiquitous GUI has its roots in Doug Engelbert's ground shattering research in the mid 60s. The concepts he invented were further developed at Xerox PARC in the 70s and successfully commercialized in the Apple Macintosh in the early 80s, whereupon they essentially froze. 20 years later, despite the thousand-fold improvements along every technological dimension, the concepts behind today's interfaces are almost identical to those in the initial Mac. Similar stories abound. For example, a telephone that could be dialed with a string of digits was the hot new thing 90 years ago.

Today, the phone number is ubiquitous and entrenched, despite countless revolutions in underlying technology. Culture changes much more slowly than technological capability. The lesson is that even today we are designing for tomorrow's technology. Cultural inertia will carry today's design choices to whatever technology comes next. In a world where science can outpace science fiction, predicting future technology can be a Nostradamian challenge. But the responsible designer has no choice.

A successful design will outlive the world it was designed for. With what artifact will the people of tomorrow learn information? I believe that in order for a personal information device to be viable in the long term, it must satisfy two conflicting criteria, portability and readability. And to briefly summarize, portability just means it needs to be basically the shape of an iPhone, and readability means it needs to basically be the shape of an iPad. You know, it needs to be something small enough that you'll carry it in your pocket, but it also needs to be something that's about the size of a sheet of paper

because that is a comfortable size to design something that works with human eyes and bodies and such. And then he goes on in the rest of this section to explain how to unify those two constraints and what influence that will have on the design of software in the future how that will work with graphical output environment history interaction etc mentioning things like because you're carrying

it in your pocket it will know your location and so it will be able to use your current location to do things like when you open a map you know open the map to the area around where you're standing and he basically describes a lot of stuff that we got with the iphone and to me that that directly mirrors engelbart and lick lighter and other earlier works that in my opinion kind of predicted the way things arrived at what they are today and so that that parallel is interesting because magic ink is is a great read i really enjoyed it i've read it a couple of times now and you know throughout the last decade and it holds up and i'm still getting more out of it every time i read it and i'm sure in a decade

when i go back and read augmenting human intellect i'll have maybe a different reaction to it. But I think this essay shows you what you can do with the essay as a medium for expressing that kind of Engelbartian thinking in the modern era. And it excites me to think that we will get more of these essays in the future, and they will serve the same role that Augmenting Human Intellect and Magic Inc have served for generations of programmers and people who want to bring design thinking to programming thus far. So that's kind of my overall kind of impression of the essay. The other thing that I wanted to talk about is how this relates to future of coding. And I tried to sit down and come up as a problem of information communication, of graphic design? The person doing the programming is mostly going to be learning and how do you help them do

that learning? And then there's a little bit where they're going to do some manipulating, but that's not the most important thing. The most important thing is the learning and i did come up with a couple of examples one of them is what shimon kaliske is doing at ink and switch and i have to admit i am biased in that i worked with shimon on crosscut and have kind of been in the periphery of what ink and switch is doing and so i have some you know some uh no no more animal metaphors i i have some uh some vested interest here um but one of the things that i got from shimon when i worked with him on crosscut that he emphatically repeated again and again until it was drilled into my head is that there really is a great value in designing a kind of a programming where you are always working on the concrete real thing and you are not working on the abstract systematized way of

eventually getting something on the screen for the end user but that you when you are doing the programming should be working with real examples always and that we you know future of coding people trying to design programming tools there's not very many people out there not very many of us who are doing the thing where we're going to design a programming from scratch and that programming has to work with concrete examples and if you're going to introduce abstractions you're going to introduce them in terms of concrete things and that is like shimon's whole bent and it is really refreshing and cool to see and it is totally at odds with my own personal opinion of how we should be designing a programming. But it aligns with, I think, what Brett is calling for in this paper

in a really exciting way. Two other examples that I wanted to call out. One of them is Mock Mechanics by Felipe Rigosa. Apologies if I'm butchering name pronunciations. Which is, it's absolutely a programming tool, but it's a programming tool that i think does the wonderful thing that i wish more people did of saying i don't care whether or not this is a tool that's going to appeal to you because you can imagine yourself using it to do the kind of work that

you do as a programmer it is instead a way of showing what programming could be if you completely approached it from a different direction without regard for it being a drop-in replacement for JavaScript or C or whatever. It's like, what if you're working with a different material entirely? What if you're working with a different substrate that the programming sits on top of or interacts with? What if the materiality of it was completely different? And so it is a little 3D environment where you draw simple 3D constructions, and then you wire them together and imbue them with movement and reactivity. It feels a little bit like an interactive animation tool more than a programming tool but it is absolutely programming it like you just look at it for long enough and and you will see that what is taking

place in this tool is a programming the aesthetic of it is kind of cartoonish or like you know very early 90s late 80s kind of 3d like primary colors simple shapes stuff moving with linear motion and it's very machiny like the things you are building with it are it feels kind of like working with lego or one of those sort of like hey simple machine building tool kits for kids but it is i think way more closely aligned with what brett is thinking about then maybe most people would characterize it as at first blush. And I think that if not the example of mock mechanics directly,

the thinking that led to mock mechanics or that would lead to similar projects is a very fresh and invigorating kind of thinking that keeps coming up every so often. You know, there's some games in the 80s for the mac or the the apple 2 that were about sort of programming simple machines and in the game you'd go inside the machine and something's odyssey i think you go inside the machine you'd have to reprogram the machines and that was how you played the game and there's these like sort of

mechanical machine focused uh graphics first programming environments. And I feel like the fact that they look like machines is the thing that we are going to have to get away from to eventually make this generalize. Because machines are limited in what they can do to like rotating a shape or moving a shape or changing the scale of a shape. And you can kind of map that into some other domain like, oh, I want to write text justification. Okay, I'm going to use like physical arrangement of each letter is on its own block and we're going to make some machine that pushes those blocks around. Like that will work.

It just is hard to generalize that. But the thing that this has that is exciting is it is all about working with a concrete thing. You are working with a concrete representation of something. You are not working with an abstraction. And then getting from there to an abstraction is a second step. You start from the concrete and then you go to the abstract. And then the one other example of this that I thought of is pain by Josh Horowitz, which is a kind of an inverted node wire programming tool, where the nodes are data, and the wires are the transformations of that data. It lets you work with concrete

examples, as you are building up this construction that does sort of a more abstracted thing which is the transformation of the data. And to me, it's not doing the full thing that this essay is advocating for about being context aware. Neither is Mach Mechanics or the work that IncanSwitch are doing, Shimon's doing at IncanSwitch. None of these are doing that full thing, but they are doing that full thing but they are doing the thing of approaching programming firstly as a graphic design problem where you should think about how is the programming environment going to be expressing information to the programmer and that is the thing to start with and once you figure that out here's how the programmer is going to be viewing and learning what it is that they're manipulating, and using concreteness is a great way to do that. It's not the only way, but it's a great way. Then build some programming in terms of that visual presentation. why graphical programming hasn't really caught on is kind of the same reason we see these problems in this essay

of lack of beautiful applications. A lot of programmers just don't feel comfortable around it. We're so used to working in text. And I wonder if programmers would feel better about designing applications if the main things we were doing were graphical because we kind of honed those skills a little bit better. I shudder to think what would have happened

if I had learned to program in BASIC instead of in HyperCard and then Director and then Flash. Because I don't think of myself as a programmer first, I think of myself as an artist first. I'm a musician and a 3d artist that's my background and when i approach building software i approach it from the interface and then work backwards to the implementation always always always and i'm very comfortable using graphic design ideas and i cut my teeth in flash building software in flash and flash was that like graphic first thing and so the fact that

that we we had a graphic first programming substrate that was the ubiquitous way of building stuff on the web and we went through the learning of how to do it wrong and then how to do it right and then it got killed and replaced with text-based programming for the sake of efficiency on battery constrained mobile devices i i think we might someday see a re-emergence of a graphics first programming tool whether that's unity or swift ui if somebody makes a really killer front end to swift uUI and it becomes what React Native never quite became. Who knows? But I feel like on a long enough timeline, we will get that again.

I think it's interesting where you put the demise of Flash, kind of like the iPhone killed Flash sort of idea. I definitely think it was the proprietariness of the runtime and all the security vulnerabilities that happened and all the glitches. I mean, Flash websites honestly were horrible to use on my computers because I had very constrained abilities. I had very slow machines. And so I hated using Flash websites.

There were plug-in incompatibilities and all of that. I think you're right that these things will come back. I think maybe WebGL might be the underlying substrate for rendering these sorts of applications. I also hated Flash things from a programming perspective because they were completely not introspectable. I could never learn how they were created. So I think these are the problems that I would want to see people solve if we do go back to that graphics-first thing, is letting them be open in the way that the web was, and letting

them not have some of the performance problems and proprietary plugin problems and all of those things that we had with Flash. It wasn't perfect, but it was different compared to what we have now in a way that makes it interesting to reflect on. I think this is definitely the historical moment that we're at with this essay. I definitely hope to see more and more influence from this, because I don't think we're quite there. Especially, I would say, maybe you think in consumer space we might be a little closer

to where Brett Victor wanted us to be, but in programming tools, we're so far away from this. Hopefully, we can bring us back to this more graphically oriented thing, even for textual tools. Problem with this podcast, Jimmy, is that we always, like the whole point of the podcast is we're reading these essays and these papers about what could have been or what should have been or what could someday be. And that always forces us into the position where at the end of the episode, we're like, damn, it sucks right now and and maybe something out there can help us improve that and maybe

somebody out there will come up with the new thing or the new substrate or will spark the new ecosystem that that leads to all of that we always end on a downer and it's the like opining for greener pastures kind of downer where it's like yeah maybe there's some green grass on the other side and it's nice to think about that but over here you know it's it's mud uh so we always end the episode on uh on a downer note so we're gonna have to come up with something you know like uh like uh how about a yahoo or something like that that we can do at the end that's so interesting that you think of this as a downer note. I think it's the opposite. I mean, we could do it in a much, you know, more upbeat way if you want, but this is the call to

action. Go out and be the program you wish you had. Like that to me is always the answer. Like if we already solved it, if we already are there, if we've already achieved, then what's the point? You know, I only want to talk about these things to the extent that we can improve them and continue to create new things. Otherwise, we're just looking back at the great works and acting as if, you know, kind of having this ancestor worship, right? Like, well, it was all solved in small talk. Like, yeah, I think we can look back at these and say we didn't achieve them, but also we can look back at these and say we don't want to achieve them, right? And some more than others, right? I think like augmenting human intellect, I'm sorry. I think that's, it's, there's awesome things there,

but there's a good reason we didn't achieve it and I'm glad we didn't. And I'm glad that Brett was inspired by it. Yeah, absolutely. Right? Because Brett's work resonates with me in a way that augmenting human we didn't. And I'm glad that Brett was inspired by it. Yeah, absolutely. Right? Because Brett's work resonates with me in a way that Augmenting Human Intellect didn't. And Engelbert's work resonated with Brett, right? Like this can be a thing where we can each find the thing that personally motivates us

and have that be the thing that drives us to do our own explorations of this space and stand on each other's shoulders. Yeah, I think in all of these works, we have to find things where we say, yeah, this work pushed things forward and accomplished things. It didn't accomplish its whole vision,

and some parts of those are sad that we didn't, but now we can go do it. And then other parts, it's pretty good that we didn't accomplish those. So I don't know. I guess, you know, I'm not sure what a work would look like where we just said, huzzah, at the end. I think it would have to be a pretty theoretical work. If you, the listener, know of such a work, send it to us. And if you have questions about the show

or suggestions for topics we could cover, or if you want to know what Jimmy sounds like when he dunks his head into a bucket of water and sings. What's your favorite song, Jimmy? I didn't just dump my head into a bucket of water. That's my favorite song. Yeah, so if you want me to write a song called I didn't just dump my head in a bucket of water and hear Jimmy sing that song right into the show,

because the voice of the listener to this show has been absent thus far.