Future of Coding - Pygmalion by David C. Smith

Episode Date: June 19, 2024

If you're anything like Ivan (oof, sorry), you've heard of Pygmalion but never caught more than the gist. Some sort of project from the early 70s, similar to Sketchpad or Smalltalk or something, yet another promising prototype from the early history of our field that failed to take the world by storm. Our stock-in-trade on this show. But you've probably heard of Programming by Demonstration. And you've certainly heard of icons — you know, those little pictures that have become indelibly part of computing as we know it. Pygmalion is the originator of these concepts… and more! The best introduction to Pygmalion is Mariano Guerra's No-code History: Pygmalion, which includes a clearly articulated summary of the big ideas, motivation, and design, with a video demonstration of the programming interface, key terminology, and links. The most introduction to Pygmalion — or Pig Million, The Millionth Pig, as it'll surely come to be known — is the subject of today's episode: the original paper by David Canfield Smith. Links $ We don't run ads on this show anymore. Sometimes Ivan makes a fake ad for a nonsense product like CarrotGrid or Hest, but those don't pay for the dirt & vapor we grow them in. But what if they could? Gonna just get this one out of the way: Quotation — and I quote, "A crucial semantic distinction between direct and indirect speech is that direct speech purports to report the exact words that were said or written EXACTLY AS THEY WERE SAID OR WRITTEN, LU, whereas indirect speech is a representation of speech in one's own words WHICH IS ALSO TOTALLY FINE, BUT JUST BE COOL ABOUT IT HEY?" @[e-mail]: but wouldn't it be funny... if i quoted those statements on a podcast... and the podcast editor thought... "that doesn't sound right, bret can't have said that"... (he can do no wrong after all)... and so they thought i was just paraphrasing him wrong... and they didn't mark them as quotes like all the other quotes in the show... wouldn't that be funny DrawDeadFish.com Shout out to Brian Hempel who sent us (among other treats) this concise summary of Pig Million from the seminal book Watch What I Do: Programming by Demonstration. Recent FoC Patreon bonus episodes were about the game Baba is You and, on our first ever video episode, the design of a visual representation for machine code. Leda and the Swan. Lenna, a sexist test image that was and to some extent still is widely used in computer graphics. Living Computation Lu: Biscuit Jimmy: Biscuit Ivan: Limp Bizkit Fine, I might as well link to Frege and analogy. Aaron Sloman's INTERACTIONS BETWEEN PHILOSOPHY AND ARTIFICIAL INTELLIGENCE: The Role of Intuition and Non-Logical Reasoning in Intelligence Ivan: Platonism Jimmy: Neoplatonism Lu: Neuplatonism I would never Derrida Nosey words History of the alphabet TodeTode Lu: Conlang Ivan: Conlon Nancarrow, beloved (by Ivan, at least) composer of music for the Player Piano. Here's a baby-faced Adam Neely with the scoop if you're new to Nancarrow. Welcome. Jimmy: Conway Twitty Autological words Heterological words School for Poetic Computation Programming by Demonstration Player vs Environment For the video demonstrating the programming model, check Mariano's post Open Canvas Working Group Lu's project CellPond, and their SPLASH talk StageCast Creator Marcel Goethals makes a lot of cool weird stuff and is a choice follow. Why does it say "Put all the metal back in the ground" at the bottom of the show notes? Music featured in this episode: Various old stuff by Ivan. The music for StageCast Creator is called Between Two Tigers. Conlon Nancarrow's Study No. 47 Wagner, the new Witness haunting every episode. ! Send us email, share your ideas in the Slack, and catch us at these normal places: Ivan: Mastodon • Website Jimmy: Mastodon • Website Lu: Mastodon • Website See you in the future! https://futureofcoding.org/episodes/072Support us on Patreon: https://www.patreon.com/futureofcodingSee omnystudio.com/listener for privacy information.

two three four five great me and jimmy were exactly the same time from my perspective that's so funny not from my perspective it's all relative that would be a fun paper to do what einstein's famous special relativity paper that's brilliant that's brilliant that's not brilliant that would be amazing why would we do that one because it's well wait a sec uh general's the long one and special's the short one right uh-huh yeah so we do special but it's different time special's the one that's like i think it was first and then general was the later one that was more yes encompassing and why would we do that because it would be hilarious i don't know i

think it'd be a great bonus content 24 pages 24 okay that's a bit long i know nothing about physics on the electrodynamics of moving bodies by a einstein it is physics right yeah that's why it would be great is because none of us are qualified to talk about this. I think what we've realized is that the most entertaining episodes is the ones where we struggle the most. Yes. So actually, we should just not do computer science anymore. Only things we're really bad at as opposed to mostly bad at, you know? I don't know because I'm good at everything.

So that's the problem I have. This paper today was very easy to read and I enjoyed it a lot. So I don't know where you're coming from. Now, listen, I have a bone to pick with you, Ivan. Uh-huh, yeah. About the previous episode. Uh-huh. And I don't know if this editor, get rid of that.

And now I know because I said that that you're going to keep it in. All right. Never mind. Hey, this is editor Ivan. First, sorry, I have a cold. Secondly, this next bit is kind of painful to listen to. It is the dissolution of my friendship with Lou. I'm including it here in the show so that you can see how just transparently cruel and manipulative Ivan is. I need you all to hear that

and just understand. On the last episode, there were some bits that you edited wrong. And I need to bring that up right now are we doing this are we actually let's have it are we airing some dirty laundry because i've done some homework pertaining to this really so yes really okay well there are there are some things there are some direct quotes where i directly quote sprat victor when you say some When you say some, how many do you have in mind when you say some? I don't know. I don't know how many. I know there's, and this is the important thing, there was at least one.

There was at least one, yes. How many do you think? There was exactly one. All right, which one was that? So I don't have the notes anymore. I just need to explain yeah yeah yeah yeah yeah because this is all getting cut out so last episode i directly quoted brett victor and ivan usually does some editing to indicate what's a direct quote and

you assumed that this was not a direct quote uh or did you just miss it what do you plead do you plead manslaughter or murder i plead man's laughter what you can't spell manslaughter without laughter i will say ivan's missed a few of my direct quotes if i didn't broadcast the fact that they were direct quotes but in this case it was very important because it sounded like i was paraphrasing Brett Victor wrong. Because then you sort of said, you know, that's not what he's saying. But actually, it was what he was saying. And I think it was this one. I think it was, visual art has always been made with direct manipulation.

Yes, I think it was that one. That sounds like it. Here's the thing about Ivan when he says something like that's not what he's saying. Hey, it's me again. Just to be clear, that's not something Ivan said. That's Jimmy paraphrasing something that Ivan would have said. To help keep that clear, I'm going to introduce a new kind of sound. Whereas when somebody's quoting something, I normally make it sound kind of like a tinny old fashioned radio. If somebody's quoting something that somebody else quoted, or if somebody's paraphrasingfashioned radio. If somebody's quoting something that somebody else quoted, or if somebody's paraphrasing something, or if somebody's quoting something that somebody else quoted, but either the original quote or the quoting of it is wrong, I'm going to use

this kind of a sound. And you can tell. So you can hear that it doesn't sound like an old radio. Here's the thing about Ivan when he says something like that's not what he's saying so ivan does this pretty regularly on these podcasts where he takes a principle of charity to a very far extent he really wants to give people the benefit of the doubt that if they say something that is on the surface maybe a little too reductionist he expands on it and he'll do this by for example saying i'm sure if you had a conversation with so and so they would agree with that criticism or something like this is it's time for ivan's hermeneutic that he applies to all of these texts unless they say something bad about visual programming then he assumes that they're narrow-minded and don't understand what they're

talking about this is in no way forecasting the paper we're going to talk about today just just putting it out there there's no forecasting that just happened that was really important that missed quote changed the course of the entire conversation. And, and, and, and, and this is just, this is classic Brett Victor. People love him, people worship him, you know, but you have to criticize some of those points, you know? They're worth criticizing. I thought there was some more quotes.

I thought, this was another direct quote, and you can edit this how you want, but he said, this is not a coding tool this is a drawing tool and i think i said that and there's something else i said i said this is also a direct quote he said visual art has always wait i already said that one never mind no uh he said no he said this one he said this one great art has always been created by thinking visually did i just not say that on the episode so here's here's what i did i after you surfaced that critique of my editing uh-huh after you put him on blast on social media i did i did and ascribed perhaps certain motives to it such as you know me not

believing that brett victor could have said something so incendiary. Yeah. I went back and I re-listened to that whole segment and I checked everything that you said against transcripts of the talk. This talk, and Brett is very clear about this, this is a talk about art. This talk is about making art on computers. I found one instance where I attributed something as a quote that wasn't a quote. Brett says, this is not a fish because it's a dead fish. And I made that the quote sound and put a big blast of Wagner.

Is it Wagner? I think it's Wagner. I don't know what you're talking about. For dramatic effect. And I stand by that. Sure, sure, sure. Because it was a wonderful moment. But this other case though.

Yes. And I found one instance of where you said something that I didn't mark as a quote, but I should have, absolutely. Visual art has always been made with direct manipulation. Visual art has always been created by direct manipulation of the art. But I also found cases where you were quoting Brett and ascribing a certain amount of emphatic, impassioned presentation to what he was saying? He says that this kind of responsiveness that he has in his fish art... This sort of responsiveness is not something that we've ever had in a visual art meeting before.

...is something we have never had before. We've never had it. We've never had it in a visual art meeting before. It's something we have never had before. We've never had it. We've never had it in a visual art meeting before. If you're quoting a text passage, I think you can have more lenience in terms of like how it's presented. Like if you're quoting text and you want to read it with a dramatic flair, I think that is fine.

But if you're taking something that somebody says as spoken in a talk, and you present it with a completely different emphasis, I think that transcends the realm of being a literal quote. I don't know. All right. So from now on, you have to do your impression of Brett Victor when you quote him. Okay. Well, and the thing I had meant to do was actually replace the instances where we quoted him with clips of him saying those things ah i'm glad i didn't do that because we get all this nice spicy drama out of it well listen i i think you're gonna have to go back again because you know i i had this list of spicy quotes here right like and i i can't remember

which ones you did quote and which ones you didn't. But like, and you can quote all of these. The main thing I want you to get out of this talk is dissatisfaction. I think you did quote that one. Everything you draw is flat and static and lifeless. Maybe you quoted that? I can't remember. There were some of the ones in that list that you didn't say at all. Or if you did say them, they got cut out. Oh, yeah. Like Super Mario Brothers wasn't modeled on anything.

Yeah. I think we need a Lou cut of that episode. Oh, yeah. Yeah, yeah, like Super Mario Brothers wasn't modeled on anything. Yeah. I think we need a Lou cut of that episode. Oh yeah. Yeah, yeah, yeah. Like a director's cut, but now it's a Lou cut. Like I said, this one, I think you disagree with this. Code is inappropriate for making visual art. Brett Victor said that.

You disagree, right? Ivan's going to be on that side. Do I disagree with Brett saying that? Or do I disagree with that being something that you said that was quoted? I'm confused about the question. Do you disagree with Brett? Okay, wait, read it again. Okay, this is the statement. Code, and I quote, code is inappropriate for making visual art. Brett Victor. I believe that code is an inappropriate way of creating visual art i do disagree with that yes yes yes and do you disagree with this one because i'm pretty sure i said this one on the podcast great art has always been created by thinking visually so

that's the more precise reason why I think code is an inappropriate way of creating visual art. Code is algebraic. We think linguistically when we're coding. Great art has always been created by thinking visually. And so I think we need a more geometric approach. I don't entirely disagree with that.

Spoilers for today's episode. Also, do you hear knocking? Yeah, uh, hang on a sec. Great art has always been created by taking off your headphones, walking away from the mic, and saying, hi Bino, we're recording a podcast right now. Be no Be no Be no Be no Be no Be no

Be no Be no Be no Be no look at him squirm look at him squirm oh it's funny it's quite interesting though i guess it's another thing we get from doing videos instead of papers it is interesting for sure to just see the difference and how the these things turned out it seemed to be uh it came out better than i kind of expected it to i still like the papers so much more yeah i think trying different mediums though will be fun

like doing a game i think would be a great thing to do at least for bonus content hey it's me again i'm still sick unfortunately i wanted to just say that the most recent uh bonus episode on the patreon we played the video game baba is you yeah i it was good i think i i liked it anyway here's back to here's back to lou i don't think great art has to be visual or has to be made by thinking visually i think it can be and it's a way of maybe making it easier but doesn't it hasn't always been created that way i'm gonna i'm gonna um put words in brett's mouth oh not again not again there's no shortage of that going on at the moment in our sphere there is absolutely no shortage of putting words in brett's mouth i didn't put any words in

his mouth i think somebody on this podcast whose name does not rhyme with emmy and whose name does not rhyme with maven has been putting certain words in brett vector's mouth what draw dead fish.com uh that's part of it yeah uh-huh uh-huh when he says great art has always been created by thinking visually he's like i know what he's getting at when he says that, right? Do you all imagine what he's getting at when he says that? Like, what should he have said

to better get across the point that he's trying to make? I think he should have said one easy way or easier way of making great art is by thinking visually. Or maybe thinking visually can help you to create great art. I think it's too general to say that great art can only and always be created that way. Yeah, totes. But like,

I think when you're when you're making that kind of point, you have to be careful. Yes. Oh, totally. Yeah. Though I don't know that he expected to have quite as much impact as he ended up having. Not that that excuses anything because... It's great. Like the impact is we get to respond. Yes. You know, that doesn't make it worse. You know, it's just like developing this idea further.

It's like, wait, is that always true? Yeah. Like these talks don't belong to Brett anymore. They belong to the community at large. And we get to... I don't know. Who knows? belong to the community at large and we get to i don't know uh who knows yeah like i think i think they've they've had such a level of impact that people are creating bits of culture works of

culture that riff on these talks for instance there's this new website called drawdeadfish.com oh yeah when he says that great art has always been created by thinking visually, like it's in the context of him saying he wants there to be tools for him to make and others as well to make visual art on the computer. And the kinds of visual art that he wants to make, you can't really do with the tools that are available. And why is that the case? And what could be done to change it so that those sorts of tools were available and and i think we nailed it i i don't know who it was but i'm just gonna say it was jimmy probably was that the point that we made on the last episode god we're like dissecting our previous episode on this episode gross it's the elephant in the room yeah yeah it's us we're the elephant now

what have we done we're we're the elephant now. What have we done? We're the elephant? Our past selves are the elephant in the room. What did past Jimmy say? Past Jimmy said that, now I've forgotten, so. Oh my God. It was that the audience for the talk was not artists. The audience for the talk was people making tools for artists.

And he didn't do a good enough job of like pointing to that audience with his statements. He's sort of like do a good enough job of like pointing to that audience with his statements he's sort of like almost a little bit of like victim blaming not to like you know use that term inappropriately which i totally just am but what the fuck yeah but like the people the artists who are stuck with the bad tools you know everything you draw is so flat and static and lifeless yeah that's not the fault of the people who are doing the drawing. It's the fault of the people who are making the tools for you to do those drawings with that they don't allow you that richer, deeper expressivity that he would like and that I would also like. Like all of his critiques of computer art resonate with me so much and I don't take it personally i take it as a failing of the art tools industry at large well i completely disagree well you would because

you never try to build consensus you always just want things to be your way you make me sick that's just sometimes i'm just up to here and i guess you know what screw you lou screw you ivan you're editing youtube account wrong editing me wrong and you just don't understand you keep this enormous audience and all you do with them is sick them on people like me trying to make something weird and interesting and beautiful in the world and there you are with your beautiful creations that you make that everybody loves and nobody appreciates the things that anybody else does because they're too busy piling on plaudits and lauding you and celebrating your work. Nobody else gets it.

You're thinking, overthinking, over- I don't think we're gonna solve this today I think this should just be the underlying bubbling conflict of this podcast I bet it won't come up again Ivan I'm sure edited out all the screaming and yelling matches that were happening right now back and forth between Lou and Ivan so I've just been sitting here you know innocently sorry did you mean edited in edited in edited out no I mean you all should have heard what was being said but you know innocently sorry did you mean edited in edited in edited out no i mean you all should have heard what was being said but you know you sure they didn't keep drawing dead fish never stop drawing dead fish i think this is the best part about this like conversational

medium and the fact that we always have a text in the broad sense of the word text, meaning video, whatever, a work that we can talk about. Because the disagreement here is not only between us disagreeing on opinions, but also like what was said in the text. And I think the richer the text, the more you get those kind of disagreements and trying to interpret and trying to figure this out and it brings up conversations we wouldn't have normally. Even though I know I ended like I said that the videos didn't work, I think the end product was better than I thought it was going to be

with the conversation. I'd be down with doing more talks or different media in the future. We got him coming, coming back. Wow, the miller video apology tour yeah yeah there's a roller coaster of opinions on papers videos like what's gonna happen next for me i i have to admit i like paper so much more but i saw just how excited about these brett victor talks that we could get i'll say excited instead of you know any other adjective

there i i reached a level of excitation the temperature was high do you think uh pig million pig million pig million two words pig million pig million million two words as in one million dollars as in the millionth pig. Is it? Is it really? Is that how you pronounce it? No. It's how I'm going to pronounce it and spell it. It's not pig-malian?

Pig-malian. It's pig-malian. Pig-malian. I'm sure there's like a better pronunciation if you like know, you know, Greek pronunciation or something. But like anglicized pig-malian. And how do you, how do you pronounce Phrygian? Phrygian.

That was another word that kept coming up. you how do you pronounce phrygian that was another word that kept coming up phrygian phrygian so related to gotlub phryga phrygian phrygian phrygian did you know this research was sponsored by the national institute of mental health and arpa it's right on the first page which is just a great first page yeah this first page whips just to bring back our segment of talking about first pages of papers yeah i've missed that segment though i guess in the last episode we did talk about the ai generated audio cleanup so there's that a little bit you don't get first pages of a talk that's that's a good point you get audience noise you get get first slides yeah you get the like oh wait are they coming out oh they're coming out wearing that oh they're bringing those with them

how are they gonna fit all of those so this first page is not actually like the first page of the paper proper it's like a cover page to organize how this whole paper is right so it's like stanford artificial intelligence laboratory memo aim 260 computer science department report number stan cs 75 499 and then there's like handwritten stuff on here that are notes and there's stamps and there's yeah there's just so much going on and like this whole scan that we have which i'm pretty sure is from brett victor just to bring things back like it's on his worry dream site really yeah yeah that's where i've stolen a lot of these papers from stolen yeah i i scraped the whole thing downloaded them all nice nice is this like

intentional is this cover it's it's just like incidental right it's not part of the aesthetic i think if you scroll to the very bottom you can see some pages that talk about the paperwork that had to be done to get this document declassified are you serious classified what so because this was like arpa right yeah so i think this cover page is like a little bit of contextual material so that the government agency that put this out knows, ah, here's the status of this document. Oh my God. That's my read of it.

Oh my God. Yeah, it's fast. It just shows kind of like this history and kind of reflects the funding and all of these things of like how early computing came to be. And like this paper, regardless of what you all end up

thinking about it uh we'll find out but like it has a lot of of rich history like this is the paper that coins the term icon right right yeah i highlighted that it's just like did it seriously coin the word icon yeah in a recent talk that uh david camfield smith gave talks about the history of coining that term with alan k that's huge yeah yeah so it's it's one of those things where like we kind of take it for granted that icons just oh yeah of course they mean these pictures on screens but that was not the meaning of the word icon until this paper icons were these religious pictures that you would find in like eastern orthodox churches or something okay that's pretty crazy yeah yeah so

this i i picked this paper i'll just give a little you know even though we talked about this great first page i picked this paper because i think it's fantastic but very long and so we i made a kind of abridgment of it it was still very long for the abridgment but i tried to take out uh things that i just thought were a little bit of side notes or things that maybe we wouldn't have as much to discuss i think i still captured the the important bits but it's a it's actually i think was i think it's a dissertation. It's at least the length of a dissertation. I don't know for sure that it actually is, considering how it was funded and all of that stuff.

But it's quite expansive. And it kind of talks about visual programming and small talk and a bunch of interesting things. So this is 1975. So, you know, fairly early on but not too early on in the history of this fur and brian hempel on the slack sent us like a summary written by by the author himself right and funnily enough like it seems like in his summary it is basically the same content just even more cut down as what you picked out so if you had discovered this earlier it may

have saved you a lot of time yeah i think the the summary though misses the fun here right right yeah it's more surface level right yeah sorry by fun did you mean like exhaustion like the the full canyon-esque breadth and depth of the uh enormity of the slog that this paper was and i would like you to know that i've done this twice now because i as you may remember if you if you listen on our patreon patreon.com future of crowding patreon i accidentally read this thinking we were going to do that last time so i've done this twice now spoilers for the one and only video episode we've done titled 32 checkboxes which uh yeah if you want to see what we're like on video and what editor Ivan does with video, then there you go.

So I had to sit through this abomination twice. It is so good because of its imperfections. Imperfections? I will just say, like, it is, of course, not a perfectly tight edited paper that is something where every sentence is just like, oh, yes, this captured me. It is, of course, not a perfectly tight edited paper that is something where every sentence is just like, oh, yes, this captured me.

It goes into a bunch of irrelevant detail and things. Some of that I do think is because it was creating things new and it's time that we kind of take for granted now, right? Like explaining what an icon is feels very like, why am I reading what an icon is? Oh, I love that actually. That part was okay.

It's the whole like analogical versus Phrygian stuff, but we'll get there. You should love that because this is an argument we have had on this podcast already. Just didn't use Phrygian or analogical. Phrygian. Phrygian. We'll get into that. We'll get into that we'll get into that freggle

rock you know what i saw ivan's reactions you know ivan posted in our little dm chat like oh this paper can't stand it and then and then i thought yeah yeah you're completely right and then i looked at your reasons for hating it and you have all the wrong reasons you pasted something that's actually really good like it's one of its huge strengths and you missed the real problems with anyway god we'll get ivan is just mad at this paper because it said everything he wants to say way before you said it oh it has participatory execution it's true it has why graphics is better than language it's just all of ivan's talking points but made in a way that's smarter than what he says and way before he ever said it i mean i agree with that it even

says something like this is not a programming language this is a system or something just like you said with hest many times before he's just sad that he didn't do it first i have to say the introduction is pretty weird though uh-huh yeah so the reasons why lou is going to hate this i'm sure are completely unpredictable and have nothing to do with you know their typical read on papers of this era all right what's the abstract pygmalion is a two-dimensional visual programming system implemented on an interactive computer with graphics display. Communication between human being and computer

is by means of visual entities called icons, subsuming the notion of variable, reference, data structure, function, and picture. The heart of the system is an interactive remembering editor for icons which execute and optionally saves operations for later re-execution the display screen is viewed as a document to be edited programming consists of creating a sequence of display frames the first of which contains the desired information the last of which last of which sorry the last of which this is a wonderful abstract and by the way ivan that was a quote ah great yes i wouldn't have you know if it weren't for

you need to do the quote sound on that sorry carry on jimmy just helping it gets right in to to like what this whole system is and then we get like little things that are are so good like these little pull quotes here like each operation has both visual parentheses aesthetic semantics and internal parentheses mechanical semantics in fact every operation in pygmalion has three responsibilities. A, for accomplishing a given internal machine task, the machine semantic of the operation. B, in display mode, for generating a representative visual action. And C, in remember mode, for adding onto a code list of operations

necessary to reproduce itself. Yes, none of this makes sense yet yet because we haven't read the paper none of us have we're just reading it you know this is a close reading we've never read the papers no uh but like coming with the jokes these are so just like bullet points right out there that are already hinting at all of what's the beautiful stuff to come of a visual programming system that's so vastly different from what we see as most visual programming systems today that it just it captures what what we need and i'm so excited about this paper from from this abstract this is great it's like i think when we talk about

visual programming today like people kind of have an idea of what you mean by that but this is amazing to me because it's really describing it from zero well i mean we did have sketchpad and a couple of other things prior to this that it's standing on i know but like the way it's phrasing it it's like yeah imagining the reader doesn't have any idea what what those things are right so it's i think it's pretty cool like even later on like it's not necessarily relevant to the paper but it it describes what a mouse is and what a cursor is from scratch as well like it really builds this concept up from nothing which is pretty impressive i think it's it's cool for like understanding where this sits in history for sure like how how early this was how you know its vision of a computer is quite different from

what other people might envision as the computer at the time yeah and we'll get into a little bit more about why that uh bugs me when we get to the next section okay are we ready to move on to the intro yet or not yet no no we there's so did you all just skip over this very long abstract here no i didn't skip over it there's so much more in here all right all right come on we're not even on page one yet give the man is abstract pygmalion is a computational extension of the brain's short-term memory it is designed to relieve the load on the short-term memory by providing alternative storage for mental images during thought yeah i do like this bit i actually labeled this bit yeah as i like it that's that's what's so useful about doing stuff visually, right? Like, this makes me think of not just

visual programming, but just like drawing diagrams that aren't even programmed. You know, when I was in school, when I was a 12 year old boy, my math teacher said, if you don't understand, draw a picture, right? And that, that's what I do now at work when I program. So this makes sense to me. I'm on board. I'm ready to discover the visual programming tool of the future of coding. But then we finally get the main innovations of Pygmalion are a dynamic representation for programs, an emphasis on doing rather than telling,

an iconic representation for parameters, data structures, requiring less translation for mental representations, a remembering editor for icons, and descriptions in terms of the concrete, which Pygmalion turns into the abstract. This is so good. I just cannot say. This is why I'm so confused on all the hate

already permeating okay so no i like this i like this before we can say oh this is so good we should we should at least spend 30 seconds describing what these things mean to an audience who might not be familiar with this stuff already like saying a remembering editor for icons is kind of meaningless described in those terms at this point in the essay so if we're going to be calling wait wait wait we'll get to it later right you're not supposed to know at this point sure okay but we shouldn't spend a lot of time like you know calling it so genius if the if the listener is not going to know what this stuff even means yet no the point of this section for me was not to describe what's going on.

It was to emphasize how wrong Ivan's going to be in my view. Right. I just want to put out like Ivan's hate reading this and I'm love reading it. That's that was, it was just to get that affect out there. Yes. Of course, none of this makes sense yet.

And to go back to, uh, when there. Yes, of course, none of this makes sense yet. And to go back to when we had our, on the rules, the rule of law. Oh, yeah, the Lawrence Diver. Yeah. I wanted to jump to the end and explain the conclusion and talk about it. And Ivan was so opposed to it. And he made a point that all these podcasts that go through papers they go through them in order did i and we should stick to going through papers in order rather than jumping around and now here you see ivan flip-flopping wanting to go to the end of the paper i think we should

go through it in order we shouldn't just stop here and say wow it's so brilliant when we don't even know what it is yet yes i was planning on ending the episode right here no i'm just advocating for the listener like we got down to the end of the driveway and it's like okay wait a sec let's stop for a rest break here listener don't worry you don't need to we're just declaring these things these main innovations they're not being assigned to anything they're not being described they're just being named like as signposts you just catch them later we're just making phragian signs oh and go off references to these things right we haven't explained the analogical structure yet is this helpful is this making it better ivan no so here i'm gonna i'm gonna say this i'm gonna say this

there's four things that pig million did that we're gonna talk about i'm gonna put them in terms that the audience can understand which sounds so condescending it really doesn't mean it like that no no programs are dynamic uh-huh right rather than static source there's like dynamic jimmy's making fun of me on video. I was trying to get you to react to something that no one else could see. Sorry. I'm going to slice you off one of these. That was a middle finger. No, it wasn't.

Yeah, it was. It was a video feed of an image of me, but not actually me. Stop drawing dead middle fingers. Is this Phrygian or analogical that's another middle finger neither not everything has to be one or the other oh okay i see somebody did their reading this though would be go off sorry that was a visual joke i won't even repeat what jim just did no yeah that's that's that's disgusting programs are represented dynamically there is no static source text or source

otherwise innovation number one yeah number two is that there are icons so you're using little visual pictures to represent other things uh number three is that there are icons so you're using little visual pictures to represent other things uh number three is that there's this remembering editor like the editor itself remembers what you do in it as you do it and that's you know as opposed to some other ways of working and we'll get into that more in a bit and then the fourth one is that when you are working when you are programming in this environment you are programming with concrete values rather than abstractions so you're working with actual numbers actual instances of the data being manipulated rather than just like symbolic names that are placeholders where the data will be at runtime so that's that's um a more humane representation of the thoughts in this uh this abstract sorry i just had to kill

jimmy before we could keep going and then we can get to the introduction which rightfully can be made fun of so um i'm i'm all down my uh first three two pages have nothing no notes highlighted because i knew that this would be uh a thing that both you all would react to so the paper begins the paper begins with in greek mythology pygmalion was a king of cyprus who fell in love with a statue of aphrodite ovid in the metamorphoses invents a more sophisticated version. Pygmalion, a sculptor, makes an ivory statue representing his ideal of womanhood, then falls in love with his own creation. Venus brings it to life in answer to his prayer. And you might be wondering, what on earth this has got to do with visual programming? As I was reading this two times kind of weird right kind of weird like

it's a little bit weird like let's start with a man sculpting a beautiful woman right that's that's really creepy that's some weird male gaze vibes man yeah i think the pronouns though here are all fine if they stay he because only a dude would try to do would be really trying to do this right right only a dude would not realize how creepy this whole endeavor is and this is when ivan can come in and do his extremely good faith generous read of it which is saying that this is a satirical piece about the male gaze but i don't think it is in fact i highlighted and this is later on this this introduction later on begins the tradition of only referring to the programmer or the artist or the painter as he right it could only possibly be a man and it's often talking about it as if this man is trying to create a beautiful woman

and that's like the metaphor that's throughout this whole thing which is pretty weird i started by highlighting every single occurrence of this he but there were just too many to possibly do there is one place where it doesn't do he it's when it talks about secretaries and office workers finding programming too difficult so that is also messed up yeah i do want to get to that passage because i do think there's uh i i like let me be clear that any paper we've had from these time periods there are constant like sexist things that exist in them uh-huh and i don't think we should make light of that i don't think oh it's just a product of the time like no this this happened right and it's a part of our computing history and continues to happen absolutely i think

the most important thing to take away from this um time that we're going to spend unpacking the rampant sexism in here is that it's not like this is just an artifact of history it's that this is an ongoing problem that if we're going to spend time learning from this historical document and you know trying to extract interesting things from its view of computing at the time and try and carry those forward into our work and our understanding of computing today and use that as a lens we should do the same thing not just limited to the parts that talk about computing but also the parts that are a reflection of the society at the time that are

still present today yeah yeah and i think that's why it's important to not just pass it off right like you know you could just say oh you know don't worry about it but like that's the kind of attitude that means that these things are just going to get propagated and people are going to read this and and like if no one says that it's not okay then people might just assume it's okay absolutely i know and that's part of the cathartic nature i think about looking at back at these odd things and shining a light on it um like and you know we have to call out that the three of us on this podcast you know it's two and a half men maybe two and a quarter men you know like looking through this and it and it probably shouldn't like but here we are whatever um maybe not whatever editor cut that out um hey the net the second page page two uh we have a

beautiful image of leonardo da vinci and later leader later do you know the story later in the swan i actually don't know it's pretty messed up right like i don't know the whole details, but it involves non-consensual sexual activities. Right. From Wikipedia now. The degree of consent by Leda to the... I don't know if I'm pronouncing Leda right. To the relationship seems to vary considerably. So just saying that's this story of like an artist trying to depict later in a really beautiful way that is a very bad thing to compare to creating a programming tool i would say yeah so bad on on several different levels

right like not not just bad in that it is picking a particularly disgusting story from history, not just awkward because it's representing male gaze and not just awkward because it is limiting the conception of programming to, you know, the activity of a man and the program being made to feminine qualities because, you know, that is why it would be desirable for a man to create it is because it's reflecting this heteronormative male-centric view it just goes like on and on and on there's so many different things about this that are cringe it's like no surprise it's like no surprise that programming is heck is-dominated. These are the ideas that have formed a basis of so much of what we do. And I'm sure there's people listening to the podcast from all sorts of different backgrounds, and some of them are probably rolling their eyes saying, oh, it's just nothing. And that's exactly what I think is worth saying. And just before we

move off of this i want to ask is there anything in particular that we want to champion that people should do differently in response to reckoning with this awkward page of history this like tendency that the industry had to make everything male centric to use uh female beauty as like the object of desire as you know motivation for a lot of these kinds of things or like um there's that famous example image in computer graphics of the uh i can't remember the name of the woman um but it's like a playboy centerfold or something like that oh my god no that was used as the example image for so many things and like included with photoshop or whatever early on as like here's an example image you can manipulate is there is there anything like today that we want to say like hey be careful about how you do this

because i have i have one but i'm curious if either of you have any well i don't i here's the thing like we're two and a half men right two and a quarter men whatever like you know there's only I have one, but I'm curious if either of you have any. Well, I don't. Here's the thing. Like, we're two and a half men, right? Two and a quarter men, whatever. Like, you know, there's only so much we can say, like, other than listen to people who are saying that they feel out of place or that they're not welcome. And the other thing I think we should really do is to not be afraid to call this out. You know, like, it can be awkward, it can be embarrassing, it can make you feel vulnerable, but someone's got to call this out when it happens, right? Even if it's in the past, even if it's already happened, doesn't mean it

can't still have an impact. So what we can do, especially with people who are like blabbing away at a podcast is to say something about it because someone will listen maybe one person will hear this and rethink it i i looked it up the reason i couldn't think of the name of the model who is used as this image this example image is because her name is lena or lena something like that which is so close to lita right you thought you were just remembering it wrong it can't be that yeah um god that's creepily close yeah as you know here's my like one cent thought as part of the problem here is there's a continually pervading attitude among people in tech that it's like we need to take tech and separate it from politics and separate it from society and just treat tech and math and science as things on their own

for their own sake and you know keep the political stuff keep the gender stuff out of this discussion of the merits of the technology or the you you know, the technology in any way. I think that opinion is actively harmful. Yeah, that I agree with that. Like if you're building something for a community of people to use, then it is linked and closely related to that community, you know? And I think for a lot of these projects these papers and it's designed for like people at large right like for the whole world the whole tech community

and so it's gonna have some relation it's gonna be related there's no escaping it there is no it should be for all the fish both the male male and female fish, not just male fish. Right. Right. And can you get non-binary fish? I don't know. Yeah. There's,

there's my biases creeping in. I think we all agree that this first page is awful and there's all of this stuff. So like, what is the point why was this you know passage included in here i think the best example we get that doesn't have these problematic elements as much is of leonardo and the flying machine so i'll quote that yeah because i think it's a bet i think it's honestly just actually a better point anyways it's

still about screwing though which is like the author can't escape being puerile even when it comes to inanimate objects they're still today we lead of leonardo's project to build a flying machine but if we look into leonardo's notes we will not find such an expression what he wants to make is a bird that will fly, and once more there is an exultant tone in the master's famous prophecy that the bird would fly. It did not. The claim to be a creator, a maker of things, passed from the painter to the engineer, leaving to the artist only a small consolation of being a maker of dreams. Today, with the advent of inexpensive computers, for the first time, the average person is presented with a powerful medium through which their ideas can attain a life of their own.

Yeah, I think this is, like you say, this is a much better example. This is a great example. This is what it means to build computers, right? This is what it means to make programmable machines. That's what i'm getting from this section they are things that can have a life of their own right they can do things themselves and that's part of the great appeal of computers yeah i think this like giving life to art giving life to our creations through the medium of computing is a, is a strong desire. And it's a very exciting thing. I mean,

I remember, you know, the first time writing a program that like did something that I wasn't directly doing, right. It was continuing to interact without me. And that, that was exciting.

It is something where you want to see your creations, not just be dead fish, but come to life. It is the elephant in the room. Yeah, it's almost like living computation. Like the Living Computation Foundation. Dave Ackley, anyway. Dave Ackley! Yeah, yeah.

There's another part of the introduction that I do quite like but I don't actually see its huge relevance to the paper at large so maybe I shouldn't like it He's talking about like we shouldn't have

such a clear divide between art and science He's talking about what other people have said and he's saying that he agrees with them Like, we shouldn't have such a clear divide between art and science. He's talking about what other people have said, and he's saying that he agrees with them. So he says something like, oh, no, he says, and I directly quote. Pi is succinct in his agreement. If anyone thinks it important to a civilization that a common ground between art and science shall be found, then they had better look for it in front of their nose, for it is ten to one that they will see there something which has been designed. Right, this is saying that art is important to our world, our society.

And I directly quote again. Kostler, Kostler, Kostler contends that there is no difference between scientific and artistic creativity, or even between scientific and comic creativity. Right, so this is setting up Pygmalion, saying that we can bring aspects of art and aspects of science computer science together that's something we can do and this really resonates with me i like that and what do you think yeah i do think this is actually well of course pygmalion doesn't like achieve this like grand thing of of art or anything it's not trying I don't think it's like trying to be uh an art piece the point here I do think is that

we have done this thing where artistic thinking visual thinking is relegated off as somehow not as important as scientific rigor and I that's what I love is kind of underlying all of this work is that these things don't have to be separated. That the computer gives us a way of letting people express something in what we would maybe have before considered not correct because it's not as rigorous and linguistic and mathematical which were all these things at the time and this paper is in a tradition that at this time was kind of questioning that there's a paper cited about ai i can't remember the author right now but that i i went and looked

at and the whole point there as well was that this linguistic tradition that thinking that we can achieve ai by just listing a bunch of rules and logic rather than looking at images and the way in which we have visual thinking was what was holding back AI. So I love that this tradition is being brought together here. The author of that paper is Aaron Sloman, and the paper is called Interactions Between Philosophy and Artificial Intelligence, the Role of Intuition and Non-Logical Reasoning in Intelligence. I have no idea why I know that or why it might be relevant later on. Oh no. Did you, you have quotes from that paper too? I do, I do as well. Oh, what a

surprise. Oh my god. Reading the references? Interesting, interesting interesting just hearing the title of that makes me think of literally the work that people do nowadays with working with large language models where it's like people are just like guessing i don't know how to prompt these things but i didn't read that so who knows if that's relevant at all i guess we'll just have to find out um about this opening section so we're at the end of part one and pig million has three sorry we're at the end of the intro we're not even at part one oh my goodness okay so at the end of the introduction um and this introduction just to summarize it um it's basically saying there's this role of the

creator and the creation and there are these sort of desires that the creator has and these frustrations that they feel with their creation and it's pulling from the history of art and renaissance artists like michelangelo and da vinci in particular to sort of talk about that that dynamic there and it's using it to set up the understanding that the computer as it's you know popularly known at the time in the mid-70s is limiting the expressive potential of people to create the things that they desire to create and i'm curious if you feel like this section was successful at expressing like at making that point and whether or not you feel like this section was successful at expressing like at making that point and whether or not you feel like that point needed to be made so those are my kind of two questions for each of

you about this section my answer is no it was terrible it was very unsuccessful because because of all this the shit that we uh because of all of the shite that we talked about at the start right it's just completely distracting like completely throws you off and just like disgusting um answer your second question i don't think it needs to do that anyway right like i i think there is clear value in having a programmable system that you can program visually. I think this first introduction is just a bit high and mighty. I don't think it's needed. I don't know. They should have cut it from numerous reasons. I do think I'd take a little different opinion on this first section here. You know, obviously, I'm not trying to say there's not problematic elements to it, but let me try to, as much as I can, set those aside for a second and just think about the

historical context that we're in and think about those passages maybe as historical notes that are not being accepted on their face value, right? We can take a skeptical eye to them, even though they are reflections of what happened at the time. So at this time, people would not have thought of computers as having anything to do with graphics whatsoever. The average person would think about computers in terms of pure text if they think about interactive computing at all. They might still have these images of batch processes of non-interactive computing. And so having a dissertation about computers start with an art historical example and start with a statement about art would have been very, I mean, you wouldn't do that today even, right? Like unless if you're in a computer science department, it would be

uncommon to do that. If you're in like a science department it would be uncommon to do that if you're in like a media studies or something you might be kind of get away with this sort of thing but like starting here with this art historical example starting with the fact that we want our creations to come to life and computers are the first thing that can really give us that ability to anyone, not just somebody with an engineering degree or all of that, is a very exciting message. And it is sad that the examples have to be kind of sucked in with things that will kind of give you a weird feeling about it. So I wish those examples had been different.

But I do like the intention here and i think that it would be more likely that the original audience would have felt this feeling in a different way than we feel it and that's a reflection of the culture and all of that i'm not saying that it's not problematic but i i do like that this tries to start with something that isn't hard math because it offers that computers are not just for math and science they are for creative endeavors which is something i think is not a message being heard much at the time yeah fair enough and it's not really being heard a whole lot right now even like I think a lot of the time in my experience just wanting to be creative and make stuff that comes to life is not

always like a real enough use case you know that almost sounds like a toy and this is something that that gets brought up again and again throughout this paper that this is actually a kind of toy right or at least it's compared to toys the funny thing is that uh i this maybe this is just my writing style i wouldn't start with this i would actually want to start with the thing itself with the with the with the programming environment and then take a step back and say look what it lets me do but you know what that's just me that's just me uh that's just me being harsh i'm gonna withhold my criticism of this section to a later point in this podcast oh my god what was what what about doing things chronologically yeah we'll get there um all right all right uh but uh great

points both of you i'm i appreciate i appreciate both of those perspectives and i see where you're each coming from that makes a lot of sense um so then the paper i'm hijacking the summarizing because we're going to be here for a long ass time and yeah go yeah go yeah uh part one aspects of creative thinking uh so this is where we get into uh more preamble before we get to talk about what pig million actually is and this is where it um i'm i'm gonna play the stinger i haven't played the stinger a lot what do you mean play the stinger play the the stinker or the stinger it's time for what are you talking about i know we're heading straight for that what is he talking about editor ivan can you clear this up there's there's something in here for everyone there's

ivan's visual crankery there's like rampant sexism there's like i love philosophical stuff that needs unpacking all right this chapter one some characteristics of thought uh jimmy uh take it away yeah so what we're having here yes you could consider it preamble before we get to pygmalion or you consider it the groundwork to understand where Pygmalion's coming from, right? Because it might be confusing if you don't know what the goal is. Especially at the time. What?

Especially at the time. Like, we're just going to have to keep saying that over and over again. Like, so many of the things that are here that seem kind of redundant or dreary or exhausting or pathetic to a modern era, at the time, this kind of stuff or dreary or exhausting or pathetic to a modern era at the time this kind of stuff was needed yeah so what we get first is kind of this reflection on how thought actually works and it's this contrast between language on the one hand and symbolic thoughts and by that i

mean like pictorial representations of thought, right? So we get here, I'll quote, a traditional theory of psychology is that the mind manipulates words in the process of thinking. The idea is that all of our thoughts are linguistic. And this is a very popular idea in this time period. This is a very popular idea for quite a lot of people. The idea here is that language is the most important aspect of our thinking, that if we don't have words, we can't do logic, we can't do reasoning. This is what is so powerful about the computing medium is that it takes these words and gives them another thing outside the human being to exist in. And he even says in here, you know, quoting on some more stuff, but it says, in other words, language is the only discernible medium available for thought.

This is a popular idea at this time, and kind of a little bit prior to this as well and so the question is is that really true is it that we actually think in words or do we think more in these mental pictures and do these play a bigger role now the great thing is i have aphantasia so i don't connect with this thought at all because i don't have mental pictures in my head and yet i still love this whole little section here that's cool that's cool like how i just every time i hear that i'm like no but really like you see it like you could see a square in your head, right? So, right? Nope, nothing.

All right. So what we have is this, you know, this question of, well, what about these words and how do they map on to reality? So first we have that it can't be about the actual structure of the words themselves words don't reflect reality in their structure if we think about you know the written ink on paper the word dog does not look like a dog the word as i utter it the sounds dog do not somehow represent dog directly no it's a pointer it It's like a reference, right? Absolutely. In fact, that's what we see here. Words are signs, i.e. they stand for content

without representing its structure. This terminology is defined in Section G. The mind must have something structural to manipulate. Right. This is where this Fregean notion comes in in kind of a more indirect sense. Frege gave us, he's a logician and philosopher, German logician and philosopher, late 1800s, early 1900s. And he talks about these signs, and he talks about sense and reference were his terms.

So it's pretty complicated in some ways, but I think it can also be pretty straightforward. So, for example uh clark kent and superman those have two different senses but they have the same reference so when i if i said clark kent uh is superman i'm not saying the same thing as saying clark kent is clark kent one of those sentences is just a tautology the other one actually gives us new information even though both words refer to the same person so this is kind of this sense and reference distinction can i can i read this whole paragraph because i have it like quoted and i think it's it's something we're going to be digging into a lot here as we start introducing

this idea of phrygian um versus analogical representations because the the whole the whole thing we're going to be talking about here for the next little while just to even more frame it is representations and representing things and you're introducing something and you can see how this is building up to talking about icons right because icons are little pictures that represent something right so we're we're trying to justify that, you know, decision in Pygmalion, that we're trying to like shore up that way of thinking about things on a computer is that it's like we have variable names in some languages now. We have this idea of algebraic symbols in math that like stand in for concrete values,

but you abstract away the concrete and just have this little abstract variable name like x that you can pass into functions and whatnot. So we're setting that kind of stuff up. And this paragraph, I think, is kind of crucial to the framing here. So it's a long one, but I'll read it perfectly on my first try. Suppose we remove words from their meanings and limit them to their actual shapes, i.e. examine language as language. The perceptual dimensions of language, its sounds and written appearance, are unstructured. Little information can be communicated to someone who does not know a language just by presenting them with spoken or written words. Though some languages, such as hieroglyphic languages, do use pictures of objects as words.

Language has meaning only insofar as individual words and phrases are understood to have meaning, i.e., only if we understand to what the words refer, both externally in the world and internally within ourselves. And I'm skipping a little bit. Words are signs that as they stand for a content without representing its structure, the mind must have something structural to manipulate. So it's introducing this idea that there's a structure in representations and what that relates to is that when you have some

representation whether it's a word or a picture of something or like an architectural drawing of a house or something like that that representation is made up of usually if it's not a simple representation if it's you know something a little bit more elaborate, like a simple representation might be like a single dot or something like that, where you put a dot down and you say, let's just have this dot represent,

you know, all of the nuclear energy produced in the United States or something like that, right? That's a, that's a, that representation is erasing a lot of detail, right? It's just like

a single little placeholder thing, but a more complex representation like if you had you know a graph or a drawing or a sentence or something like that some some more complex thing it would be made up of smaller parts and those parts have relationships between them and a lot of what we're going to talk about when we get into talking about analogical versus phrygian phrygian is those little parts and the and the relationships between them how well do those relationships between the parts and the parts themselves map to the thing being represented are the little parts and the relationships mirrored in the actual thing being represented or are they completely unrelated

and we'll get into i'm sure a ton of examples of of the differences that crop up but that's that's sort of what we're getting into here right right right right right right in this paragraph there is a lot i could pick apart and actually say this is wrong, that's wrong, this is wrong. But I don't actually think it relates to the core point here. So I don't think it's worth it, but we should do it on another episode. I'm all for doing a philosophy of language episode. That sounds great to me. Or if the thing that me and Dave actually wrote together got accepted, we should do that eventually because that's about a lot of this,

or at least coming from a different angle. But it might be too weird to get accepted. Anyway, I think there's stuff about structure in here. I don't actually think what they're talking about with like spoken language I don't think I don't think that's relevant to the point they're making I think what they're saying is that some things are completely abstract symbols so like some words have no relation at all to what they're referring to when I say the word biscuit you know what i'm referring to but the word itself the letters b i blah blah blah doesn't actually relate to biscuit i love how you chose a

word that refers to different things depending on who is saying them on this podcast when i refer to the word biscuit it means something different than when you refer to the word biscuit right and that's intentional because it's actually a reference to the paper that i wrote with dave actually right okay but anyway whoa whoa whoa whoa whoa no we gotta unpack this when you went to spell biscuit biscuit you didn't actually get far enough into the spelling before you kind of hand waved it away for us to know whether you're talking about the new metal band limp biscuit or not so i'm just gonna have to assume that you're talking about limp biscuit and not any other is that the genre of limp biscuit new u yeah yeah right so like the word biscuit as a symbol it's fray for a gain phrygian phrygian right because because it has no relation but if i held up if i had like a little printed out bit of card with a picture of a biscuit like an icon that is not phrygian because it's okay is it phrygian it's

phrygian phrygian right a picture of a biscuit is not phrygian fake because it has some relation to what it's referring to. So I have written up a brief summary of Aaron Sloman's definitions of these terms that Dave Smith is pulling from in his use of these terms for this. So Dave Smith cites Aaron Sloman's definition of the terms. I'm quoting myself here, so I'm going to have to put in the quote voice, which I'll summarize as follows. These terms, and actually this part, I shouldn't put in the quote voice because this is my summary of Aaron Sloman, not what Aaron Sloman actually says. No, but you could be quoting yourself. Are you quoting your own summary though? I guess it depends if I re-record it as editor Ivan quoting. You just need to have another type of sound, which is for like close enough quotes. Okay, so these terms are about how we represent something.

Pictures, maps, and scale models are largely analogical. And you can tell. And that word analogical, it's related to the word analogy. So when you, if you're familiar with the word analogy... Like I'm making an analogy here, you know, I'm not actually saying an actual thing. I'm like comparing these two things in a way.

This thing is an analogy for that thing. Wow, that made it more confusing. Pictures, maps, and scale models are largely analogical. Predicate calculus, which is what Frege invented. Predicate calculus, programming languages,

and natural languages are largely, though not entirely, Phrygian. the difference is that when you have some representation and it's made up of several parts do those parts or the relationships between them help you understand the thing being represented so in the case of the word biscuit do the parts of that word and the relationships between them help you understand the thing that is being represented by that word? I will ask rhetorically. I don't want to answer that question yet. For things that are analogical, the answer is yes, that the parts and the relations do help

you understand the thing being represented, though it doesn't have to be 100%. Even just like a little bit helps. So if there's a little bit that the parts in their relationships help you understand, that's that representation being analogical. For Phrygian, the answer is a firm no. The only relation between parts is that of functions and arguments. Like if I were to say, I love you, Jimmy. The function is, I love, and the the argument is you, Jimmy. That's the structure. The function is I love with the argument you, Jimmy. There's no relation between that structure and the things being described. At least according to Aaron Sloman, though I find this slightly debatable. At most, Phrygian representations tell you how to find something.

So you might say like a path into some structure or some like complex relationship, like, you know, Ivan's co-hosts, partner's friend at the horse show or whatever. There's a structure in what I just said that helps you find the thing being referenced, but it's not describing a quality of the thing being referenced. It's just describing a kind of a more abstract relationship that exists about that thing. There's like a spatial aspect to it, right? Like it's kind of like referring to something else spatially.

Like, okay, okay, okay. So like if there's a biscuit in the room... I could point at the biscuit and I'm kind of doing a structure there. I'm referring to this thing in relation to other things in a way. It's just like one way of doing it. It's not the only way. It's like one way. Though I think that's even a different kind of relation than the one where it's like, oh, my partner. Because my partner doesn't imply anything spatial. It implies something about our relationship, which is a more abstract notion. Like it doesn't necessarily exist in physical space. Right, right, yeah.

That's why I said sort of it's debatable. Predicate calculus is totally Phrygian. Some math notation, though not all, is Phrygian. Wait, what is? What's predicate calculus? Predicate calculus is the thing where you have functions and arguments. Oh, I know that one.

Yep. Though I'm sure it has a richer definition than that that some listeners of our show are going to be eager to... It's a formalism created by Phryga. Just think of it like the lambda calculus or something like that. It's a, you know, a formal structure. I bet it's super cool and relevant to programming.

I don't know enough about it. I actually do look forward to somebody telling us about it. If you're listening to it, tell me. Don't tell Lou. Maybe also tell Jimmy. So predicate calculus, totally Phrygian because Frege invented it, right? So it's the thing that we like define Phrygian things as being like predicate calculus. It's

like that's definition. Some math notation is Phrygian. Natural language and programming language are at least somewhat analogical. And in the case of programming languages, one example is that the linear order of statements in your source code corresponds to the execution order at runtime. So there is a little bit of an analogy between the source code and the actual running of the program that the source code represents. So it's a little bit analogical. And I will end with this banger, this quote from Aaron Sloman. Devices such as GoTo, which upset this correspondence, the statement order to execution order, are neither Phrygian nor analogical. These two categories are by no

means exhaustive. And that's all taken from interactions between philosophy and artificial intelligence, the role of intuition and non-logical reasoning in intelligence. There'll be a link to that in the show notes. Jimmy, did you have anything you wanted to add about that? No, I think this is great. I think if we make a callback to our Mythical Man Month episode where we were talking about can, or our debate afterwards or whatever,

can you make a visual representation of programs? We talked about like floor plans and how, you know, there's a mapping between the building and the floor plan of the building. This is a great just analogical example. Whereas if I wrote down a bunch of words describing a house to you, there's nothing about the structure of those words that is analogical to the structure of the house. The meaning of those words can describe the structure of the house, but the physical shape of those words themselves and their structure is not somehow analogical to the structure of the home.

We'll see about that. Ding, ding, ding, ding, ding. We have today's lucky contestant, Jimmy Miller, joining Lou Wilson in the studio. Welcome to Phrygian vs. Analogical, the new game show within a podcast. I'm going to read you a list of things and you two are going to tell me whether they are Phrygian or Analogical. Jimmy, are you ready?

And neither can be a proper answer, correct? Because this is not an exhaustive category. He's not ready. He's not ready. Lou, are you ready to play Phrygian vs. Analogical? I'm ready. I you ready to play versus analogical i'm ready i'm gonna win the save icon which looks like a little floppy disk uh-huh phrygian or analogical it is analogical because it was a floppy disk and okay originally but nowadays, can things be analogical with themselves?

Right, right, right, hear me out, hear me out. Like because this floppy disk symbol has become like so embedded in our culture as a save icon that like if you're making an app, then you're going to use that to mean save, you're not going to come up with an abstract thing, you're not going to say like, if you're making an app, then you're going to use that to mean save. You're not going to come up with an abstract thing. You're not going to say like, well, this triangle symbol means save. You're going to use the analogical one. So even if you don't know what a floppy disk is, then it's analogical because floppy disk is, it's analogical.

Ah, but is it representing the action of saving some data? Or is it representing the place where data used to be saved uh it's the same thing trick question ah and uh if it no longer has the original meaning in popular usage if people now look at that and don't know what it is it's just the abstract save icon is that any different from words where maybe once upon a time they were hieroglyphic in nature and now they're become abstracted to the point that we don't know um it is different because it's it's a picture but it's not a picture of a floppy disk it's a picture of another website apps picture of a save can it be a picture of a picture uh yeah if you put a

this is not underneath it right all right uh 10 points to lou a wonderful answer uh jimmy thank you in our slack chat do you have the slack open i have an image that i've pasted that shows an array a triangle the united states and a bicycle represented in the left column as images and in the right column as words love it when we do do you see that image now jimmy visual content on our podcast that image of a triangle next to the word triangle is the image analogical or phrygian why or why not analogical what is it an analogy to triangles what's a triangle a triangle is a shape with three sides and where do whose angles add up to 180 where do triangles exist uh triangles exist or where does triangle exist uh both triangles we can draw representations of triangularity right so this is one triangle but it doesn't stand for it isn't all

trying possible triangles it isn't the essence of track triangle. It is just one concrete implementation, if you will, of the triangle class, if we want to try to be programery speak. Yeah, if you want to be oh-oh about it. We've got some small talk going on in this paper, so it's appropriate. We've got some small talk going on here, and it's cool. One more question for you, contestant Jimmy. Do triangles exist? Yeah. Yeah? Yeah. I'm going'm gonna give have you ever seen one yeah yeah i'm looking at one right now well you're looking at some straight

lines with some dots but to me a triangle is is one of those things that emerges when you like measure the horizontal component of a rolling wheel it's something that doesn't actually have a physical manifestation and that there's nothing to be represented by the image of a triangle are you kidding me have you ever seen a mountain that's a triangle you can represent things that don't have a physical manifestation have you ever seen a bee with a stinger you know that stinger is a triangle i think it's a stinger so are you saying ivan hold on i want to know i'm saying this is not a triangle are you saying you said that you can't represent things that are don't have a physical manifestation uh right with an image are you saying expressionism as an art form was complete

bunk because they're trying to express things that were not visual in a visual medium they're expressing them but they're not analogical they're phrygian because they're inventing a symbol where there is absolutely no relation between the thing being depicted and the actual thing itself oh absolutely not i would say that the the image of the triangle is completely phrygian and the word triangle is analogical what about impressionism negative 10 points to jimmy what about impressionism which was clearly trying to represent something that was visual without representing it directly instead it was representing the impression you got as you looked at it yeah by using lots of little splash i my dad is an impressionist painter and but he

only became an impressionist painter later in life when his vision deteriorated and so it's like he's painting the world as he sees it and so is that analogical or fregian yeah uh that is analogical but the drawing of a triangle is fregian because triangles don't physically exist who does your dad do the best impressions of hold on so we just got ivan to say that he believes in platonism it's awesome no no no new platonism and you platonism to not to be confused with neo-platonism i've realized that i thought i understood what these words meant and then i we've got a lot of rounds of this game show to go through just warning you i i i feel like my understanding is getting worse but i hope that's just like a dip in the middle of the game show that's because ivan's trying to deconstruct the things to say that there's no difference here in a way that's

completely misunderstanding the distinction being made right i would never deride uh this uh this work so um i had nowhere to go with that pun he thinks he's being clever by deconstructing this somehow but he's just missing the point oh okay and it's a point that ivan himself has made over and over and over again that is the basis for his whole justification for graphical programming what i've actually got is i've got a really interesting list of things that i'm not sure where they sit on the realm of analogical versus phrygian that i think it would be fun for us to lou wilson round two is an arrow in tl draw analogical or phrygian it's analogical because it's like pointing with your finger ah okay great yes i had not actually uh taken it to mean that i had taken

it to be like what is the thing in reality that the arrow is like there's nothing in reality that's like the arrow but it's the act of pointing at something that you can do physically or like making reference it's an analogy to that and it's not structurally similar to the sort of like linguistic style reference where i say my left foot or something like that but it is analogical to the finger pointing i didn't take it that way but lots of things can point so i just had to pick one of them and it's a finger you make a good one um i've thought about this a lot 10 points to lou wilson in round two am i winning uh so far jimmy round two snout snot sneeze sniff snuff snort sneer snarl snore snoot snide snivel and snoop loop analogical or phrygian uh so that's a good question um both how so we this is one of the

things that like i said i would love to do a philosophy of language uh episode so there's all sorts of stuff going back to uh plato talking about words and their origins and if they're a reflection of the underlying reality or not and looking at like how did words come to be and so you know onomatopoeias and things like that have some reflection of the structure of reality boom but at the same time if we're thinking about them written down rather than the sounds that they make right those those writing symbols don't have a structural relation to the things that they are representing but you say them in your head right you hear them in your head i do so okay here just

well do you because i might see them in my head i don't see them in my head i do have an internal monologue but it's only my voice yeah if i if i think about like a bass guitar i hear myself miming bass guitar sounds no really actual bass guitar wait ivan do you do you do this too no i can i have like perfect musical like i can imagine exactly how stuff sounds in my head yeah i'm not surprised no no one's like that like what like they imagine themselves singing bass guitar jimmy just like that no no no one's like hat can perfectly hear like a one-to-one replication of a bass guitar in their head yes i i like can exactly imagine a sound in my head exactly as it feels to hear it. You don't have perfect pitch, do you?

I don't have perfect pitch. I do have a good sense of relative pitch when I train it up. I haven't in a long time. You know, in my head, it's like never my own voice. It's like other voices. Do you know what? That sounds really weird. Yeah, if i try to

do like an impression in my head it's me doing the impression or whatever like it's not like the person i can't and i can't remember most of the time what people sound like or any of that stuff but i don't really hear like i don't really get like full words in my head it's more like wishy-washy it's like great art has always been created by thinking visually oh weird huh yeah yeah i don't know like words this is why i hate words you know i'm like the opposite of jimmy i think right and this is why it blows my mind like this is why it blows my mind because i'm like i kind of wish i could do that better like i wish i could have words in my head better. So much of the time when I'm saying a sentence,

I switch to another sentence, right? Because it's like, I just did it. It's a curse. Anyway. That was, yeah, 20 points to Jimmy for that answer. Yes, great job. So you're still a little bit behind lou but uh catching up quickly uh lou in our slack next to the image with the triangle there is an

image of some ancient germanic i believe roots of some english letters of the alphabet we have yad and cap and i'm going to ask and then i'm going to explain the image i'm going to describe it are the letters of the alphabet analogical or phrygian uh okay well here's here's the thing this is something else that i think we need to bring up that this depends on the execution of those symbols, right? The execution of that natural code, right? In a person, because I look at those and I see that they kind of look like arms and hands. So for me in interpreting and executing those symbols, yes, they are analogical. But I could also understand that someone else, like another person's machinery, their mental machinery,

might look at that and not see it. And for them, then it's frigging. Frigging. Yeah, right? So this depends. It's not deterministic execution. Excellent. That is exactly what i was hoping you would say five points five points i'm now going to describe the image that i set you up to

answer the question with by showing you the image first and then asking for your thoughts on it the image is the ancient sort of pictographic symbol called yad which is for arm and it's looks like a sort of a person's arm drawn from a little box representing the shoulder with a little bit sticking straight out and then a sort of a more abstracted version of it that looks kind of like a backwards f with a little hook at the bottom kind of like if you put f and j together and this this letter form eventually was simplified into the letter i and the letter i in english comes from this root meaning the arm and the following letter at the time and we have another letter in between now but is cap which the pictographic representation is of a hand and it's the letter k and so if you think of that no way you know in today's alphabet the letter i is from arm and the letter k is from hand

those letters still kind of to this day look like the things they're representing and what i found in going back to look this up is pretty much all of the letters in English look like the things they were originally sort of meant to represent. Like M came from a word that was meant to describe water. So it looks like this rippled surface of water. No way. That's crazy. All the letters still look like the things that they were originally meant to describe. That's like tote tote.

Yeah. My made up language. Seriously. meant to describe that's like toto yeah um my made-up language seriously this is what i i did a conlang course right you know conlang like like made up artificial languages yeah he like did these beautiful player piano pieces there's probably one playing right now conlon nancaro so yeah this is conlon nancaro i'm so excited to get to share this with any of you who are not familiar with his music uh he was a composer for the player piano which is like a normal you know piano keyboard instrument like you're used to but with a special mechanism where you could feed through a roll of paper that had holes punched in it, and whenever one of the holes would pass over a little sensing mechanism, it would play a specific key on the keyboard.

And so you'd program this instrument kind of like programming a punch card computer. And Conlon was composing music for this instrument his entire life. He started back in the 1930s, I believe. This piece was composed in the 1930s i believe uh this piece was composed in the mid 80s and uh yeah i just i love this stuff what oh my god yeah Yeah, biscuit. Conway Twitty? Sorry, Lou, you were saying Toad Toad. Right, right. I did a Conlang course.

And the advice that this expert in language origins told me was that the easiest way to make a made-up language that seems real is to think through its origins like come up with something concrete to start with and then as you use it more that will become so annoying that you can't and it will have so many problems and inconsistencies that you can't help but like break things down into more abstract forms and if you do that again and again and again and again over hundreds and thousands of years then you'll end up with something like the languages we've got uh today and i didn't realize that's actually how we got our language it was i thought it was just good conlang advice yeah totally they like the the capital letter a if you rotate it 90

degrees looks kind of like the head of an ox because that's what it originally was so uh we've got uh two more rounds oh my god it's uh jimmy's turn jimmy are autological words analogical or friggin i don't know what those words are. An autological word is a word that describes itself, like multisyllabic is a multisyllabic word. Palindrome is not autological because the word is not a palindrome. Oh my God. So there's a whole class of words called autological words

that are words that describe themselves. So autological words are phragian because i'll read the wikipedia definition here of what uh an ontological word is it expresses a property that it also possesses and when we're talking about an analogical versus phragian we're talking about the structure of something not its meaning right because the meaning is like what the thing represents sort of indirectly the thing expresses yeah so these words express a property that they also possess but their structure is not the same so the sounds or the written symbols don't represent anything.

It's the meaning that's representing it, which is phragian, not analogical. That is an excellent point. And it is super relevant to what we're about to get back to talking about, which is Pygmalion, because icons have that property. Like using a triangle to represent something like a subroutine there is not an analogical relationship between the triangle and the subroutine so that's a a really interesting criticism of pig millions use of icons that i i wish somebody would make in about 15 minutes oh my god depends on what the icon is and what the subroutine yeah is. Yeah, totally. Yeah, great point.

That's worth easily 25 points. Are you joking? Super good. Which I believe puts the running total at Jimmy 35, Lou 25. Lou, final question goes to you. All right. Which is kind of unfair because I think you started also. So the game is rigged uh maybe

it'll be a bonus round for jimmy no how about heterological words as in words that do not describe themselves are those analogical or phrygian heterological words so for example, long, and I quote, is a heterological word because it is not a long word. Okay. Well, it depends. That's a silly question. Wrong question. Because that's so many words. What a joke.

The correct answer to this question, as I've written down, is the opposite of love is not hate it's indifference so uh 10 points lou nicely done i'll just say again that those would be fregian because we're talking about the meaning expressed by them not their physical shape or sound or sensory properties but what if it was a like the word was blue and it was in a blue font but it could also be in a red font so it could you see yeah it's just the wrong question is a word written in a color the same as the word in the abstract i do you know what this is too abstract for me i i i don't understand what you're talking about. Thank you for playing Analogical vs. Fredgian, a game show within a podcast. I've always wanted to do one of those.

Did I win? No, you tied. Oh! Unfortunately, there are no winners or losers when it comes to philosophy. The worst kind of outcome. All right, so to get back to the text here, we are told about this distinction.

We're given a reason why visual images are so useful for programming. And I'll summarize here. Images are derived from sense perceptions. They may be visual, auditory, kinesthetic, olfactory, and may even use taste. This thesis concentrates on visual images because a they have powerful representational capabilities b operations on visual images are versatile yet simple and c computer technology is at a stage where visual processing can be effectively augmented i love that may even use taste yeah it's like this paper should have yeah yeah like gonna maybe not i would strongly recommend using taste when it comes to picking images and then maybe we would have got the

introduction to this thing in the first place oh that was good i saw you laugh about that and then i was like you're laughing about that yeah okay the taste right very good okay okay there was one thing that i just want to because i i want to make sure i'm understanding this right right so that there's this concept of fregian what fregian things and analogical things right and i think i understand the difference but they're saying more than this right they're saying that actually and i quote words are signs i.e they stand for content without representing its structure so it's saying that sentence is saying words aregian and i think it's implying that imagery visuals are analogical is that what they're saying many words are phrygian and images can be analogical and are it's it's easier to make

analogical things with images than words would be a way of putting it if you use the image to make an analogy yes you don't have to have an image like you can have an image that's analogical to something and not analogical to something else it depends what the image represents so they're not making the claim that like all images are analogical and all words are fregean no and you're not just being generous here jimmy like you chastised ivan for earlier if you look at the original source that they're quoting from that's part of the whole point is that it doesn't have to be that way. It's just kind of, you can think of it as a tendency. So I think, you know, I kept in some of this stuff,

like we're still in the section one, that there's a lot of stuff here and we're definitely not going to be able to cover it straightforwardly. But here we get a question about like, how do people actually think? This is what this whole part is supposed to be establishing is do we think in words or do we think in pictures and of course the answer is both or whatever but they give a great example here of short-term memory where if i tell you northeast southeast

southwest and you have to remember that sequence and repeat it back so northeast north southeast no north southwest no no yeah yeah uh these were each separate ones but okay so northeast was one no he knows north east south east south west airlines so if i give you that sequence you can probably remember it it's not that long it's not that hard but as i continued to give more and more of these verbal commands and i just asked you to repeat them back it would become difficult instead they draw the picture right and even for me who does think it doesn't have the visual imagery the picture is so much easier to remember than that uh the list of words in fact ivan as he was repeating those things was pointing and making some sort of physical way of remembering them that was visual in nature right got him i would

say it's spatial rather than visual as i was feeling it with my body not seeing it with my eyes absolutely yeah but it wasn't linguistic that wasn't the the main mechanism here right it was something other than linguistic and yeah i was extending the linguistic information like i was mapping the linguistic stuff to physical sensations in my body you had an interpretation of that linguistic thing so that it became easier to remember yeah i i i agree with this i'm on board i think i think it is easier to like hold images in your head for like even you even you jimmy said you found it easier to like memorize yeah i can draw that picture way faster than i can say the and and i would if i couldn't if i wasn't just hearing that sequence i would translate from the picture to the words yeah not the other way around and i feel like it's easier to visually pass it as well you can just you can just see it you know can you see it yes you can you can see it's a picture

flash up that picture for like a brief moment and then ask me the directions where it's fast up those words for a brief moment and then ask me to draw a picture. One of those is way easier than the other.

And I just want to once again defend the thing that this paper is trying to say in the face of what it's actually possibly saying, which is that this is not true of all things, but there are a class of things for which doing a visual representation is more useful than the words right yeah and it says that very explicitly so here is an example of one of those such things yeah great it's like there are plenty of things where that's easier

it would be great to have tooling that lets us do that did Did you see this? Did you see this on page 11? It has this little diagram, I guess, of some code, some pseudocode. And, you know, it says if dot dot dot, then while dot dot dot do, etc. And it's making the point that it actually says, and I directly quote, quote voice please. One of the few characteristics of linear languages that even approaches spatial organization is indentation quite heavily to structure our code so that it's like more analogical it's saying that that's all we do right like one of the few characteristics and i think that's like a good way of understanding one possible way it says says, at best, this is only an indirect indicator of meaning. In other words, it's not a very good analogy, right? It's still very abstract.

Maybe we could go further. And I paraphrase. Maybe we could go further. Maybe we could make something that is a better analogy and more closely linked to the kind of thing we're making yeah i got a great quote around the same thing it says it is ironic that structured programming has dealt with unstructured programming languages they are unstructured in the sense that programs have no spatial or organization and data is designated abstractly without displaying its structure. A structured programming language would use

structure in its semantics. A point that Ivan has made so many times. I love all of these things. You have. You've talked about how graphical programming, the graphical layout should have a meaning to the program. I'll have you know my notes

begin we are currently on page we're about to get on to page 12 my notes for criticism start on page 74 so okay all right all right well it speeds up because there's lots of pictures later yeah and we get a bunch of stuff here about these, you know, analogical, and we, you know, get various examples. Like, in Fortran, mathematical computation is analogical to math notation, right? Because its structure, the visual representation of it, looks similar to how you'd visually represent mathematical things. And so the whole point here is,

I think, kind of dubious in some ways about how we think. It's not a great, you know, it's not a great representation of how we actually think. It sucks. But it's trying to pull out elements of the fact that, like it's trying to bust this myth that all we think of is in terms of words, that the only way to do logical things is in terms of words, and that, you know, there is some of

this visual element to our thought, and especially when it comes to our short-term representations. That's the gist of it. Is it a scientific treatise that's like properly doing that and really exploring it? Nah. Is a lot of this stuff like maybe it was accepted at the time? Is it hokum? Is it complete hogwash? Are there one million hogs being washed? Pigs.

I'm going to read. That's saying it's helpful. I'm going to read and we're going to give the Brett Victor level of taking this at its word. And if it makes an absolute statement, we're going gonna treat it as an absolute and not give any credit we all know how good you are at that this paper primarily deals with conscious thought really primarily as opposed like there's parts of it that deal with unconscious thought or subconscious thought concentrating particularly on visual thinking as a basis for designing a graphical computer interface. The model of visual thinking presented so far may be summarized as follows. Number one,

conscious thought deals with concepts in the short-term memory, which are retrieved in some fashion from the long-term memory. Concepts in the short-term memory are metaphorical images derived from sense perceptions. Words are pointers to full concepts and are used to suppress unnecessary detail. And three, thought operations are transformations of images. And then it goes on to describe how Pygmalion's use of images is specifically helpful because it pertains to short-term memory uh quoting again uh see how i'm cuing myself so that i don't have to do the work in editing of going fuck is that a quote or not i can't tell i have to go reread the whole paper to figure out if that's a quote or not

go find where that page was and read the passage and listen to it and every time i do that it adds 10 minutes to the edit which takes like 8 to 20 hours per episode of time that i could be spending working on hast so when we when we queue our quotes when we actually queue our quotes it helps me so much it makes my life so much easier sounds like an excuse yeah yeah an excuse for wanting some of my life back by patterning and i quote by patterning itself after this model pig million hopes to stimulate and assist such thinking processes in people and it does this in a couple of different places throughout the paper where it like tries to make this connection between the use of images and the way that short-term memory and long-term memory and conscious thought work and i just personally i found all of those

connections extremely tenuous and suspect and it made me think of this entire opening section that we've done so far as something that was like backfilled to try and lend legitimacy to what pig million actually is not to say anything about whether i like pig million or whether i think that what it's doing is good or whether it made a good contribution we'll get to that but this whole opening section feels like like starting with the art history stuff at the beginning where it's like hey we have to teach you that not all thinking is about hard science and engineering that creative thought deserves a seat at the table and this whole section that's trying to like justify it in terms of like this is how people think this is you know this stuff about fregian versus analogical it's like to me this only

detracts from the rest of what's in the paper and like i kept reading this and asking myself like who is the audience for this that wouldn't be better served by not having this and calling the paper something other than pig million a creative programming environment which sounds so artsy fartsy right from the get go that it's like of course you have to do all this work defending the legitimacy of the design of pig million in because you set yourself up to be looked at through that sort of a scant lens of illegitimacy and if you didn't do that like you could condense all of this justification down into a much more succinct form and i don't think that this justification informed the design like i think this is stuff that david smith was probably already aware of and thinking about but i don't think that

there was a like a strict you know design process done where it's like let's understand how thought works and try and generate a programming environment that is like derived from how people think because if that is the case then it's like miraculous that they wound up with what they wound up with because these connections are so flimsy and if it were that we wouldn't have read the paper because it would have been so boring well thankfully we're on page 12 of 70 something so many of which are wait wait wait a second are we only on i'm on page 11 still well i'm looking at the top of page 12 of 70 something so many of which are wait wait wait a second are we only on i'm on page 11 still well i'm looking at the top of page 12 there's like images or short-term memory which

convert to long-term memory it is funny because like i do feel like ivan could have written this whole part one and now you're just like ripping into it but but ivan ivan it's not a psychological model well it's not a site it's this model this this theory is not trying to make a psychological claim this is a model that the pig million program is built upon, right? Whether you think the model is right or wrong, this is still just like preparations. This is how we're thinking about it. And I don't think you need to say

that this is the correct and only way. And I would say that there isn't like the one true way of approaching how people think and construct things creatively. I would say there's lots of different ways you can see it. Yeah, that's mad to try and do that. Yeah, right. And so it doesn't try and do that. And I quote.

The model of visual thinking presented so far may be summarized as follows. It does not say, and I do not quote This is how people think right like this is just the way that we're approaching it so You know, I have I reserve judgment at this point, you know Maybe they'll say something crazy later But at this point I'm still I'm still on board i just like that that point three thought operations are transformations of images that makes me angry it makes you angry i mean it is a bit like it's a bit it's a bit annoyingly vague i found right like it's like i don't really

know what that means because so let's go to why that point would be in there. And let's talk about what the ultimate climax or conclusion of section one is, which is this idea of articulate representations. Okay, so we got, I'll quote. An articulate form for a problem is one which corresponds closely to the form used in the mind in thinking about the problem. This is problem dependent. The mind uses different representations in different situations. Let us define the translation distance between two representations as the number of transformations that one representation must be put through to make it identical to the second representation.

Then we can summarize the results of the previous section in a minimum translation principle. And I'll quote the minimum translation principle. The most articulate representation for a subject is at a minimum translation distance from the mind's representation. And a corollary. Analogical systems are more articulate than phragian systems for extensional problems. So what is the point of all of this? Why did we talk about for a very long time that I even cut out some of it? Or I think I cut out stuff here. I've cut out stuff later. Anyways, but why did we talk about all of this? Because the idea here is we have to take all of our visual

and spatial thoughts and reduce them down to some linguistic thing we are missing out on being able to articulate our thoughts in the computing medium and this i think we could use Brett Victor's language here, would be wrong for Smith. A violation of that principle. And that was a quote. A quote of what? Of Brett Victor. Violation of that principle. He said that at least 20.

And I quote, the. Yeah, that's what i'm getting at here is it's like there's a certain minimum granularity where it's like maybe he did say those three words in that order at some point yeah you're quoting his meaning not his actual language i you know what i'm just gonna have to do this he said a violation of principle i am just gonna have to do the lou edit every single word that brett victor said at some point maybe not in that order yeah i think what we have is a disagreement about the function of the quote voice effect the function of the quote voice effect. The function of the quote voice effect.

No, no, listen, I don't want to muddy my previous quibbles. Like those were much more than just that, but you could do that. All right, and I'll finally end here. With a suitable representation, the problem solving abilities of a person using a computer can be vastly increased. Yeah, do you see the line of the fundamental problem for Pygmalion?

He states, and I quote, The fundamental problem for Pygmalion is to design an articulate language for communication. Problem statement stated, and it's 6 25 hours into the podcast and we are at the like all right we have stated the problem let's go i just want to summarize what i see because we're about to go into section two which is just like actually describing the system uh-huh what this is is a an argument that programming languages have focused purely on the linguistic that these linguistic elements lack certain things that we often use in our thought processes and that the only way we're going to make it so that programming becomes this more accessible thing,

that it becomes less error prone, that it becomes less of a burden, that it is able to be as expressive as we want it to be, that it helps us with creative acts, is to move away from purely linguistic means of communication to something that is graphical and spatial in nature. That is the argument here. And how he tried to establish this is by looking at the ways we think, looking at how language works,

looking at how pictorial representations work, and combining all of these things together to have a system not trying to solve all the world's problems, not trying to solve general, here's how all programming will occur, but to focus in on a particular domain, which is these kind of computations that we do and that need to live in our short-term memory, but we're not great at keeping those in and so now we're gonna we want to focus a system at letting people represent their thoughts as directly as possible in a way that helps their short-term memory and in a way that existing systems aren't able to like to the extent that existing systems let you express your thoughts they're missing a critical kind of thought they're missing some critical ways of doing that not to say that like previous systems or you know contemporary systems at the time made it impossible they were just like excluding a whole category of thought yeah there

was this translation distance i'm i'm on board with that with that premise you know at this point can so we're at part two we're gonna start talking about actual like what pig million is proper um i have an opening statement for this section if that's okay with both of you i just want to call out that we skipped 48 pages here so we go from page 20 to page 68. Nice. I abridged this quite significantly here. It's still very long. And we skipped from like page 13 to like page 18.

We skipped some stuff here that I just, you know, thought was not going to be good conversation for us. So my, and I'm just going to, because I don't want to spend the entire rest of the episode like as we're talking about pig million kind of like having to um sort of dance around the thoughts that i have about it as we go case by case like i just want to set up like here's my overall thoughts about pig million and the paper so that we can look at the examples and contrast that with with how i'm you know how i feel about this so i'm kind of like doing my like end of the episode summary of the thing now um but i think that'll just make it easier to read through i fucking loved this

paper this was awesome this was so cool i have never heard of Pygmalion before. Like I've heard people talk about it. I've never read about it. I've never seen it. I didn't know any of this work. I had no idea what I was in for when I started reading this. After reading through the whole thing and then re-skimming it, this was 1000% my jam. it's extremely cool is there a catch

there is no catch what my fondness for the whole second half talking about pig million makes me want to say yeah that first part with the like inappropriate justification of all that kind of stuff. Yeah, that sucks. But this paper has like so many different little differently textured sections to it, like as a complete work, like there's entire paragraphs of Dave just like talking about like, yeah, I was thinking about this thing and then I was feeling that thing and like, like just kind of like blogging in the middle of here there's bits of like rigorous technical stuff there's art history there's like cognitive psych stuff that isn't for me but it's like there is so much in here it is there's so much breadth and yes it's long and there should be like more accessible ways of getting at this kind of stuff and getting like to the bits that one would personally enjoy like i would have loved a little

bit of a reading guide or a better you know table of contents or some kind of like more structure to it but in terms of like ideas presented and its contributions and the way that it approaches thinking about this stuff and we'll get into this. Like why I wanted to say this now is because as we're going through this, there's going to be so many places where I'm like, this idea right here

is something I have been obsessed with over the past, you know, three months or whatever in this project that I've been working on, that project where it's like, oh yeah, there's that idea. There's somebody else talking about that. We don't have language for it. But I love that there's so many things in here that like, are embodying ways that I think programming should

be or at least like, saying, you know, agreeing, hey, programming should do this programming should be that here's why this is the way this is is. All in ways that capture things that I think are super important and super valuable and things that haven't happened yet. So I loved the paper. I thought it was so, so cool. This doesn't feel right. No butts, no catches. Thank you for playing along with the first part of the episode where I had to pretend that i hated it and was hate reading it the whole time good i was i'm glad to hear this twist because like this paper is just like yeah is it is it imperfect of course but it says so much of ivan's talking points like

i said from the beginning yeah i had no idea this was out there like i'm so glad that i can point to this now as being like here's somebody who had it and we lost it and here's somebody who had it can we have this good yes thank you for picking this jimmy for for for me all right nailed it uh good episode goodbye guys got me no no no no no now we have to get into it and you two go ahead and and and tear it a new one wherever you feel like doing so. I have my own criticisms. I'm not going to be defending this the whole time.

I just needed to. Well, you know, I guess we're probably not going to go through bit by bit in detail here because there are a lot of examples. And I guess we don't have time. We shouldn't have time. I think we should start with an example. We should at least get people an example of what this system is like.

Yeah, so I was wondering if you had thoughts on what to pick out, Jimmy. Like the factorial example or something more general? Because the factorial example is what they picked out in their summary too right yeah i think there is a lot here and i do think we couldn't walk through every and again i abridged quite a lot out of this and yet there's a lot here so i think that it's a good to get a sense of what uh pygmalion is and i have a quote here on what it's not. I want to emphasize that Pygmalion is not a graphical programming language in the traditional sense. Graphical programming languages have all attempted to find two-dimensional ways to tell programs what to do. This inherently involves manipulation of formal representations of data.

Pygmalion has no representation for telling a program anything. Pygmalion is an environment for doing computations. So this is the key here. Pygmalion is not a nodes and wire interface. It's not anything like that where you can look at the screen and see the program and what it's doing is instead a place where you can manipulate icons and walk through computations with concrete values. So the example that they give is this factorial, and you can have a little exclamation mark that means factorial and a little number beside it. And the program lets you walk through and build up that program from concrete values. You pull out an if statement and it executes and goes down one path. And then it asks you what the value is here and it does this recursion. And so at each point you're working

with like concrete values, like factorial of five and walking through and putting the data where they belong in little boxes. And at the end, you have kind of a movie that could play of that computation. And underneath the hood, we see a little like Lisp-like representation of how all of these, the program runs. And so that's kind of this idea is that we don't work with trying to show the whole structure of our program at once like node and wire interfaces do. We don't have any sort of like direct graphical thing. Instead, we're doing participatory execution of our programs.

And our programs can not be fully specified. And all sorts of interesting features come out of this idea that we actually do the computation along with the computer first. Okay, I find this hilarious that so many visual programming languages claim to not be visual programming languages. Like, I don't know, this tickles me so much. It's the most common trope I ever see. And it's like, to me, it just sounds stuck up. I'm sorry. I'm sorry. and it's like to me it just sounds stuck up i'm sorry i'm sorry i'm sorry guys but it sounds stuck up it's like oh no this isn't a programming language oh no this is a this is a system right like it's like they do it here that you just read it out

brett victor does it and i quote he says this is not a coding tool this is a drawing tool referring to dead fish ivan said it a lot i see it all over the place you you know like what is it about the tech world is like this is no that's not a programming language css or whatever like it's not a programming language such a precious thing like um even like there's you know the school for poetic computation right it's like no no we don't do demos we don't no we we don't do creative coding we do poetic computation and i'm not making a value judgment about this but it is just like as this i find it slightly funny that everyone is so afraid of saying their thing is coding. Why do people say that? Because I think they're trying to big up what they're doing, right?

They're trying to say, no, it's so much more. Let go of your existing ideas of what that thing is. We don't want to be weighed down by the baggage of that and i understand that but i just think it takes even more effort to do that than to just say well this is just slightly different and the thing that i find funny is like, it's like champagne, you know, it's like, no, it's not technically champagne. No, it's sparkling wine because it doesn't come from that region of the dead fish or whatever, you know, like it's funny. It's like, no, no, no, no, no, no. I'm not coding. I'm just writing symbols that make the computer do something it's not coding

right i'm not eating a biscuit i'm merely putting it in my mouth and chomping and swallowing it's not eating i'm not i'm simply there's there's one one specific example you use that i want to flag the rest of it i like broadly agree with what you're saying but when people specifically say that css isn't programming that's i think a little bit different they mean it in a sort of demeaning way where it's like css is something less than programming it's just styling it's just like visual design it's it is lesser than programming programming Programming is harder. And this is an extension of a whole bunch of bigotry. And I just wanted to say that that is, to me, a different category, a different reason that people say that blank is not programming than what Dave Smith is doing here, what I do when I say that something is or is not visual programming, like saying that something is not visual programming or not graphical programming, is trying to unpack a term that has a certain colloquial meaning and trying to like point at that colloquial meaning and say, the words don't match the meaning and I need to point at that difference like when people talk about visual

programming or graphical programming it's not visual in an open-ended sense it's visual only in a very specific sense and if you are doing something that is not visual in that same specific sense you need a term to refer to it and you might want to say that it is visual but it's like outside the set of things that people mean when they talk about visual programming so you need to point at that and say that label is suggesting that the things being described by it are actually more inclusive than they tend to be in common usage so we need a different term or we need to unpack that term a little bit which to me is not about excluding a certain kind of programming from being like good or worthy it's more about like

understanding the way that these designs relate to one another as designs i think the context here though i will say is a little different than even ivan's context or the like trying to make yourself sound cooler than you are. I think the context here was actually trying to diminish the aims of the paper, because what Smith is saying here is that he hasn't accomplished the task of being able to represent a program in its entirety in a 2D way it isn't a graphical program that tries to have that ambition and he later says the part of the reason is because he doesn't think we can uh at least now right he doesn't think yet he doesn't say it's impossible because it's not inherently spatial yeah exactly um but he hasn't he just says he doesn't think we are at that point

and so what he's saying is all we've done is given people an environment to do computation, not to represent it visually. And that the representation ends up being textual. That's why we see like the underlying Lisp code that it is. It remembers what you did and makes a textual representation of that memory. But the actual program, they haven't represented in any way visually.

It's just the doing of computation that's visual. Though some of that is, I think, for a reason that is touched on either later on in this paper or in a different paper about Pygmalion, I can't remember, where it's basically like Pygmalion is triply interpreted. There's the top-level version, and then it gets turned into this bytecode and then into this you know whatever and then down into machine instructions at the end there's like several passes through an interpreter that happen and that the ultimate thing you would want to do is just have a machine that more directly understands what's going on here so i would imagine that the fact that this turns into a textual representation is just a necessary implementation detail and not necessarily part of the actual meaning of the system as an environment for expressing programs through action rather than through source text. Yeah, I think the point is there is no representation that's visual that you can look at at once.

Yes. You can only look at it through time yeah though i think that's not being played as a uh you know like for modesty or whatever played as a as a failing i think that's like very concretely deliberate um and maybe dave feels differently about that now i've i've heard some like criticism of this system being like, yeah, there is no static representation. The only representation is this dynamic running environment. You can't print out a Pygmalion program and have it be meaningful. Screenshots don't do it justice.

It's like you have to be in there using it to actually understand the program. Yeah, the proper representation of a Pygmalion program. Is a movie. That's a quote. Was that a quote? That was a quote. It was. So can I say. I know they're saying it's not.

A graphical programming language. And I think. Part of that. I think is kind of like. There's a certain timidness going on saying you know like oh no but we haven't really done all of these things etc and and I do get the sense that part of it is trying to indicate that it is more than that as as well okay I would say this to me fulfills my definition for this is a visual programming language right and and my

understanding and this is what I was surprised about that it's actually a lot more than that as well to me this seems like just like a graphical canvas tool right like a lot of like there's a toolbar there's a toolbar on the side it reminds me of like photoshop or teal draw but like some of but a lot of the tools are programming tools but similarly it goes into a lot of detail about how things can be placed in 2D space. And it gets into this concept of icons. And I have a section here, a diagram, I say they should have opened with this bit. Programming in Pygmalion, and I quote, is a process of designing and editing icons. The Pygmalion programmer is an iconographer, a maker or designer of figures or drawings,

which is the Webster definition. They quote, quote, quote, and I quote, and I directly quote, quote voice please. Like you can use Pygmalion and not even do any programming. You could just do it to draw a diagram. And that was very surprising. I think that's really cool like it's it's identifying that like a big part of i guess creating things and programming is just visualizing stuff on your screen or on a piece of paper or on a sticky note or whatever so i don't know like this was very surprising to me yeah i i didn't expect to see a canvas laid out in detail like this.

I love the next sentence right after what you read. The programmer of the future will be as skilled in design as architects and artists today. This is the vision of programming that Smith is giving us, is one in which we aren't focused so much on... We are focused on the aesthetic elements of our programs just as much as we are focused on their functionality. And that this is a skill. This is one of the things Ivan said he wants,

is that a programming language where like your artistic ability makes you better at it. And this is the aim here. Can I expand on what an icon is? Because like, I think the way that we use icons typically when when someone says an icon i would think oh like a small little emoji like thing like a little svg that i might put on my website you know to denote something a disket say a disket, say. A disket, right? But an icon in this context, in Pygmalion, it could be like any shape on your canvas. So we have some examples here, like a very wide rectangular box

that says icon in. We have a square icon. We have a rectangle icon that's serving as like a container of other icons. But we also have an arrow. An arrow is an icon. Or like a little circuit diagram symbols. And it can be anything. And like this is the creation of this term icon. And it has a really general use. In To or Draw, things on the canvas, we call them shapes. They're not really all shapes.

But there wasn't really another word for them that popped up like you can have like a link on your page that's shape you could have a text shape text it's not really a shape but we we call it a shape just just because that's the word we chose and to me it's clear that icon is being used to mean that here so don't just think that icon is little pictures it could be anything on the canvas yeah and we can draw these ourselves right the exact way in which they're drawn is not kind of specified here but you can draw a custom icon these aren't predefined and so we get an example of like a spaceship program and there's different parts of the the spaceship being drawn out and you can imagine these as all sorts of different things like oh well if you want to construct a spaceship

you might need these parts and so this is like the interface for spaceships that you now have to meet and instead of seeing a bunch of methods listed in text you see a small little box with some pictures of the different parts. And now these are the parts that you have to meet for that interface. This is something where one of the things that's pointed out here is exactly what you were talking about, Lou, where you don't have to have the full breadth of how the functionality works.

You can just start drawing, and you can start partially filling in functionality to these icons over time. And in fact, if you do a computation and with a concrete value, and it only goes down, you know, the right-hand side of some branch, the left-hand side is just a trap is what they call it, where if that program ever executes and hits that left-hand side, the user is then asked to fill in the computation there. All these programs can be partially specified and yet still executable, something that we don't at all have today. These are all live things that we're interacting with. Yes, can like invoke an icon and the computation begins by playing the movie through and then we can be asked to come in and yeah i know some closure some lisp systems have execution you know exceptions like this right or whatever but it's not something

that we're used to in a programming system like this yeah that idea that you you write a little little bit of the program, then you start it running, and when the program hits a spot where it doesn't know what to do next, it stops the execution and says, hey, we're here. Here's the state of all the data. And that's like that, this way of like allowing you to execute a partial program couples beautifully with the fact that it's all working in terms of concrete values because when it stops execution what you have is a screen full of concrete values and icons that you've drawn so it's all familiar stuff that is the stuff that you work with when you program whether you're running the program or not as opposed to a debugger as we experience them for languages that have a source text and then a separate runtime environment where when you hop into the debugger you see your live values but it is an environment that is foreign to the experience

you have when you're writing the source text that the the debugger with its values it puts the values over in a little side panel with each of the variable names and the current value, and you don't have that same set of tools available that you had when you were writing the program. And I know that this is not universally true, that REPLs are different, and that there's tools out there like replay.io and that sort of thing that are different still from what I'm describing, but i'm just describing the mainstream experience that people have using a step debugger as a separate tool from their editor as this sort of intermediary between the source text and the running code under execution and so that that like fusion of a live environment where you're programming in terms of concrete values and you can execute the code at any point and when it hits a spot where it can't proceed you can just like make adjustments

to the arrangement of objects on the screen and that is how you tell the computer what to do next because that's another thing we haven't talked about at all is that like when you program what you are doing is you act out the thing the computer should do so it's like put this value from here in that box over there so it's like oh what's the return value from this function well take this input and put it in this little thing that's going to transform it and then run that and then get the transform value back out and then take that transform value and say put it here to signify that this is the return value i'm not speaking literally i'm borrowing some language but it's like you pantomime the the way that the data should flow through your system you act it out

you move the data around and you say oh if i see this value here and that value there here's the value that should be the result of this computation. And you demonstrate, right? Because this is the, we've talked a lot about this system being the originator of icons. It's also the originator of programming by demonstration or programming by example, which is another. Is it actually like the OG?

Yes. For that? Yes. Oh my. OMG. Yeah, this is the OG of pbd slash pbe which i can use as acronyms now because i've already said the full versions of those terms pve player versus environment no no enemy yeah yeah the compiler and static representations of text are the uh the enemy in this in this system is it a good time

to talk about like the factorial example because one i have one more thing before we get there but um okay i can i just say then like i i don't completely on in fact not not at all i don't understand at all like actually how to program stuff. I don't know, like, there are screenshots and things of, like, showing what you can do. Hey, like, click here or drag here or whatever. And I don't actually understand how stuff gets programmed. I wouldn't have understood it either if I hadn't watched a video of david smith talking through the factorial example yeah that will be in our show notes if you want to follow the next part of the conversation uh listener i suggest watching that video if you haven't seen pig million what like right now no

i don't mean you lou you're screwed no no no no but like has the listener got to pause and watch that now yes pull over your car to the side of the road stop the dishes get up out of your kids room as you're snuggling them to sleep whatever you do when you listen to podcasts stop it right now and go to your 75 foot television screen and put this on yeah we will do our best to describe what's going on here but obviously we're talking about something visual in an auditory medium and it's a terrible idea how do we keep doing this to ourselves yeah i thought this was going to be a paper one it's almost like visual stuff is bad it's almost like this should be a video podcast and we should be video recording

all of them never again it's actually faster to edit but it's worse because you can't edit out as much yes yeah i can't you can just do the youtube thing we cut cut cut no but i would be cutting us each at different times that's fine no it would show my hand oh that's the thing i can't show my hand that's the rule that's the trick people don't even know and that would so what did you want to talk about before we get to the factorial example this should probably come after the factorial example but i'm doing it now right there's a right there's so we've been talking about what icons are and there's like there's somewhere in this paper i can't find it right now but it basically describes like

icons are these qualities there's like an image like a visual representation that you get to design there's a meaning to the execution model to the like to the runtime there's this like meaning when the icon is encountered while you're executing the program but there's this third thing um jimmy do you remember what the third thing is how to describe that really well he didn't my best recollection is that it's like there's a meaning to your program so there's like the execution meaning which is a little bit different from like here's a meaning to your program so there's like the execution meaning which is a little bit different from like here's what the image represents so i might draw an icon that represents the number six or that represents like the operation like add six to something i

could make icons of those and those are slightly different from the meaning when it comes time to actually execute the code that what the what the program's going to do like oh take this value in this register here and put it in that register there or whatever is a different meaning from you know add six that operation is a different uh thing from you know the little icon that i've drawn of like you know the little die with the six dots on it or whatever. And those three things come together. And that's what makes an icon and that icons. And this is the part that I really wanted to talk about is they have these attributes. They have a name that is a string.

So you can give it a name. So if I've got my little die with a little six on it, that represents my add six operation, let's say I can give it a name. I might call it add six. That's a little six on it that represents my add six operation let's say i can give it a name i might call it add six that's a little textual label then there's a value that this icon holds which is like a reference to some object then there's a shape which is the like little graphic the little die shape with the little six dots on it then there's a body which is if this icon is a function what is the body of that function then there's a display and this is the one that i wanted to talk about which is one of either false name value or shape so when you are looking at the icons you can choose do you want to not see this particular icon or some icon you know being able to show and hide icons depending on context is

awesome uh do i want to see their names as text do i want to see the value that they're pointing to so if i have some icon that represents just the number six do i want to see the number six or do i want to see you know the the name s i x or do i want to see, you know, the name S-I-X? Or do I want to see the shape? That's the last option, like the little thing that I drew. And I love this because it captures something that's important to me and it comes up a lot in what we do at Ink and Switch, which is the difference i draw is my human voice being reflected and and and brought into the system and the value that the thing represents kind of often has to be a computer voice thing because that's like where you put in like this is what this means to the computer the value six as part of a calculation that i'm performing that has to be a computer

voicey kind of thing because it's the side of this representation that faces the computer and then there's the name the string take it or leave it whether you want to have that or whether the shape suffices or whether the name can just be inside the shape but that choice of like you know do i want to see the human voice version of this icon or the computer voice version of this icon is a distinction that is huge to me. And I really love seeing that already here in this work because I don't see it in a lot of other programming systems that have this, like, draw your own representation of things. Do you know what's interesting? Like, this feels so familiar to working on Teal Draw. Every shape has its props, but shapes also have their hard-coded properties, which are different.

And it can have a combination of different attributes like this. Some are hard coded, like all shapes have them. Some are like regularly reused by us and other people can use them as well. And some are like very specific to each shape in particular. Like an arrowhead property, like what kind of arrowhead does your arrow have? Obviously you won't have that on a text shape and there's this open canvas working group that is trying to find like a shared format for all different canvas applications mainly because I think the company behind obsidian came up with their own version and it's widely agreed that it's not very good it's like over opinionated in some ways and under opinionated in other ways and i love how diplomatic you are now that it's about your job and not just like your personal i mean that's a

pretty i think the obsidian one sucks i think the obsidian one is trash right so like it's it's interesting because people have been having these discussions about like, what's important to a shared canvas format. And do you know what? This one listed here is pretty good, I'd say. Like, it's kind of remarkable, like how this is the best open canvas format I've ever seen, right? There hasn't been a lot of work on it yet.

So it's, you know, fair enough. But like, it's so weird to me that this has existed all this time. What am I doing here? I should just give up. And there's more properties than the ones that I listed. And they're also good. Like, if you're going to design this kind of a system, it makes sense that these are the properties you would want to have.

There's something like, it's not one for one, maybe, you know, the most essential core set, but it's darn close. I mean, this,

I think this is good because it's so general. Like it kind of, it indicates to me the scope of what a pig million is. And it's supposed to be very general purpose. An icon can be many many many things it's not narrowly defined the things that are here are the things that you need to get a graphic on screen so the other properties are like position with height right because it's assume they're all rectangles which is like that's a choice but if you're going to make that choice

this is not a bad set of things to have as x y with height it's got display right which is like that's a choice but if you're going to make that choice this is not a bad set of things to have is x y with height it's got display right which is like do i want to see this or not that's in there and there's also do i want to see the name the value or the shape which i you know just finished saying i love that choice is here and that that thought is there there's also like if this is a function what's its body um if you want to run this as a bit of code what code gets executed when you tell this to run which is a little different from saying it's a function with a body this is like if this represents an atomic operation in the system what atomic operation does it represent also awesome that they have that in there in that way and then there's some like stuff to bind it to small talk some stuff to like deal with fetching and storing

whatever i i see that as kind of implementation detail stuff there's container as well right yeah right parent right we call that parent these days the the thing that i would say is missing is like bindings. Does this icon have a relationship with another icon? And it has one kind of binding, it has container. And some, it could like, you could hard code in some other bindings. Like arrows, I'm assuming you mean like arrows moving with the object. Right, exactly. Or in like a node wire situation you could have you know like some specific specific holes to connect different icons together holes you would call those things that connect those holes what would you call them pipes yeah and the things that are being

connected you would call them junk uh if nobody subscribes to our Patreon, shame on you. Go subscribe to patreon.com slash future of coding. There's fire and heat there. Heat and fire. Factorial.

I'm going to explain what we see and leave out some irrelevant details for factorial that are not going to help us here but on the left hand side we see a menu so this is like our little example of how we would compute factorial on the left hand side we see a menu with some like things about icons some op codes some control flow and some other stuff and i'm not going to list all of those but think of this as like a tool palette in photoshop or something like that like it's your you know what would be your move tool your scale your paintbrush that kind of thing but because this is a programming system it's things like tools for creating an icon deleting an icon

duplicating inserting commonly used math operations that kind of stuff that's your your stand unchanged to this day it's the same exact stuff in in max msp etc etc this is like yeah at the bottom we see some metadata that i'm going to ignore kind of what those exactly being here because i don't think they're super relevant to our example and then in the middle in the canvas we have the our factorial which is a box a rectangle with uh in this case number six in it and a drawn exclamation point. So it's kind of like bubble letter exclamation mark. So I just want to comment here, we actually get like proper notation already just by the

fact that these are iconography, you know, icons instead of being text, because you would usually have to write factorial of six instead of being like six bang right but now here we can do six bang we don't have to worry about having a parser that can do post-fix notation and so i like that already and also it's like bubble right yeah the explanation it's like actually bubble writing yeah it's so cute it's a picture it's drawn just yeah very cute and so this is like like our, the box on the left here is the argument to this factorial function, right? And it's six.

And so how do we compute factorial? Well, we need to start with a conditional. And I think you probably just click once because I don't think they had like nice drag out. But you can imagine from this left-hand side menu bar, we drag out and they have their imagine from this left hand side menu bar, we drag out and they have their conditional operator is just a question mark. So this is like an if statement. And just before we go ahead and start putting the conditional in, we're going to tell Pygmalion,

hey, the thing we're about to do is what you need to know Pygmalion to execute the factorial icon. So what we're about to do is demonstrating how when you see a factorial icon next to a six, here's what to do. Now let's act it out. So the first action we take is bring in a conditional, you're going to need a conditional Pygmalion. So let's bring one of those in. And what we see is this little box with a question mark and then arrows pointing to true and arrow pointing to false. And so we have this conditional with each branch going on, but the box where we're going to put the conditional is just a question mark. So now we can drag out the equals op code and we get a little box with two boxes inside and an equal sign in the middle. And so this is like how we're starting to construct our program. Now we can drag that six that we had next to our exclamation mark into our box on the left

hand side. And then on the right hand side, we can type one and we can say, does six equal one? That's what we're doing. And then we'll execute that as false and we'll drag it up to our little question mark here. We're going to say, hey, we are taking the false branch. There's a couple of super cool things in here. One of them, this is painfully obvious and super pedestrian, but it can't be said enough. The fact that you get to define the way that things look and how they're represented in a free canvas and assign meaning by moving things around and performing action frees you from the bullshit of equality having to be double equals. You can use the equal sign to mean equality because assignment is an action that you perform, not a little logographic symbol that you put down on the canvas. So you can use math operators properly.

You don't have to have equals meaning assignment or equals colon meaning assignment and then double equals meaning equality or whatever, or like an EQ function. I mean, you can already do that. You can already do that in dream bird true it's been implemented it's not hypothetical there's lots of languages where you can do that but separating out um a certain class of thing into this is an action that is performed rather than a visual token on the canvas, frees up expressive space by separating out the things that are like, like it, it, when designing visual programming, a lot of the times you have to concern yourself with like, oh, but if I use, or even in like graphic design tools, if I use a certain shade of blue to mean selection, would that get confusing? If I used that shade of blue in my drawing how do you mitigate that it's like being able to say

this kind of stuff is freed up for you to use for your own expressive purposes because we're not using it for our system to express itself love that love it goes one step further and that that equal sign is an uninterpreted symbol right it's not there's nothing about that that is the text ascii character equals it could have been anything and they just happen to choose two lines with boxes on either side as what they wanted to do for equals here so that's a little different than the dream bird example where you're it's not that you could choose any symbol you want for equality that's interesting it could have been biscuit yeah it could have been biscuit it could have been biscuit it could have been biscuit it could have been biscuit um so and another thing is as soon as you put in an uh operation like equals and it has these two boxes

on either side waiting for a value to come in because that equals operation is already fully programmed as soon as it has two valid values it immediately executes and that's your next step in the process of creating this program so we immediately go from saying 6 equals 1, boom, false. And now we can take that false and put it in the predicate box for the conditional. And so now instead of a little question mark for our predicate box, we get false.

And so we are taking the false path down this conditional. Yeah, taking the false path. Taking the false path. And so we are now in this the false side of this conditional and we can start dragging things out into our canvas to say hey when this conditional is false what should i do right and since it's factorial well the answer is we need to do what factorial does which is is do some multiplication and some subtraction, right?

So we're going to start by, in this case, they show us a picture of two different boxes with the 6 in each one. So it's 6 times something. It's empty right now. And then 6 minus 1. And when we do the 6 minus 1, we're going to end up getting 5. Immediately, because we've satisfied the minus. So it immediately executes and gives us a 5.

And then what we do is we drag that 5 out to a new factorial icon that we've pulled into the canvas here. It's so cool. We have a new instance of factorial. And now instead of 6 in the box, which it's been sitting up there the whole time we've been showing that we've been representing six we have now a five here down on the like the bottom right and now the computation runs it knows what to do on the left hand side the false side the the false path yeah but it doesn't know what to do all the way down and i think actually in this paper it

isn't properly represented there um yeah we don't see which branch is active when it stops the next time when it hits a halt like in the in the actual demo video it shows the true or the false have like lit up indicating like right now i need you to program this branch. Like, please go and perform what I, computer, should do when in this circumstance. And so it, as soon as you put that five in the factorial box, it runs until it hits 120,

and it should be showing, yeah, it should be showing a one, I think. It should get to the true branch, which we don't ever see in this little listing, which was one thing that was a little hard

to understand here. No, this is crazy. Absolutely crazy. but we'll get to the true branch uh-huh and we can just drag a one in and say that's the the result should be one and then it will keep coming through until it'll go back up until we get to something where it doesn't know what to do which is that you know six times what and we drag our answer from factorial in there and now the whole computation has been defined okay this is crazy so you you've coded the factorial function by just playing out the function by dragging numbers where they need to go into the little operations they need to do and they could they can be anywhere on the campus and the crazy part is as you're

coding parts of it you could actually use that like partially coded code to do things for you because like factorials recursive that's that's bonkers it whips yeah it absolutely rules and i guess i see what people say about like you can't get like a static representation yes that code because you like it's something that happened over time and there's a description where they're describing like why is this the way it is well david noticed when you put two mathematicians at a whiteboard i'm probably butchering this example so i won't use quote voice because i'm just recalling from memory just use paraphrase voice paraphrase voice which sounds a lot like normal ivan talking voice and you can tell when you put two mathematicians at a whiteboard and give them an interesting problem and come

back, you know, 15 minutes later, the whiteboard will be covered with scrawls that make no sense because what's recorded on the whiteboard is not their thinking. They did their thinking performatively, like by talking as they're drawing and it's that interactive use of the whiteboard as an aid to thinking that is the externalized thought process not the actual marks recorded on the whiteboard the marks recorded on the whiteboard are like uh you know they're like the tire tracks of the car having sped away not the actual car speeding away yeah people used to make fun of me for this in my first programming job really oh because i always use whiteboards not as like a record of artifacts which is what most people were using them for it was to think through

and my handwriting is also awful but you know at the end of people would always want to take pictures at the end for whiteboarding sessions or whatever. And then they would look at mine and be like, wait, none of this makes any sense. I'm like, yes, because it wasn't the end artifact we were trying to do. Like all these arrows and the things I'm moving around, it's in the process as we're communicating. It should have been a movie. They should have recorded a movie of you doing this. And then they would know how to recover the thinking process and that is what dave smith attempted to do with pig million is make a blackboard that you like have this

dialogue with the computer in this blackboard environment and that is the programming activity and any still frame of it is just like a still snapshot of a moment in that conversation between those two mathematicians it's not the full record of the program or a full record of that conversation so so this this is why it like translates so badly to a paper right like this is why i was looking at it and i couldn't understand what's going on but like you're talking me through it i can see it's like this process. And you know what? I like I realize now that this is very similar to StageCast Creator.

The history of StageCast Creator And I quote... I quote this history from the academic institution Wikipedia. Wiki... Wikipedia? Wikipedia. wiki wikipedia wikipedia the software known as creator originally started as a project by alan cypher and david canfield smith in apple's advanced technology group stageast Creator is a visual programming language based on the concept of programming by example.

It was intended to teach children how to program. Smith believed programming was not inherently hard. The true problem was that computer scientists had failed to create an easier way to learn and do it. Creator was a solution to this problem. Creator is a continuation of Smith's KidSim project at Apple. But in 1997, Steve Jobs eliminated the Advanced Technology Group. This meant that KidSim would no longer receive funding. However, Apple gave Smith and about 12 other employees of the former Advanced Technology Group permission to continue the project on their own.

In 1997, Smith and his fellow employees created a startup. Soon after founding the company, KidSim was renamed Creator. Creator is based on the idea of independent characters that have a graphical appearance and non-graphical properties. Each character icon has a list of rules that determine how it behaves. The rules are created by demonstrating what the character does in a specific situation. Each graphical rewrite rule is a before-after rule, stating that when the before conditions of the rule are met, the after actions of the rule are performed.

However, StageCast Inc. operated went out of business in 2002 due to a lack of funding from Venture Capitalists in Silicon Valley. In his own words, Smith said several Venture Capitalists told his company, We love your software, and could we please have a copy for our kids? But we aren't going to fund you because we've never made money on educational software. Smith called the disbandment of StageCast Inc. the second greatest disappointment of his career I don't know

what the greatest disappointment of his career was and many years later I did a talk at Splash it was all based on StageCast Creator. So, however much shit I give David Canfield Smith, his work is the basis of almost everything I do.

That was a quote as well. The basis of almost everything I do, That was a quote as well. The basis of almost everything I do, that's a quote. And what it does is it creates this before and after picture, like of what the state was at the start of your demonstration and what the state was at the start of your demonstration and what the state was at the end but the problem is is that like the process of demonstrating this creates this long line of operations and i showed this in the the presentation i gave at live i showed you that the actual representation of what happens is this list of operations. And it's kind of like this imperative code. And that was one of the weaknesses that I actually

tried to get rid of with cell pond, that it's not the process that you create, it is the before and after picture, right? Because one weakness i can see of this approach now which is true for stagecraft creator and i think it's true for this is that if you want to go back and change one of those steps like what if you realize like oh no i took away two instead of one by accident is there a way of like changing something or do you have to go through this whole thing from the start to finish again yeah so the way that you do that is that there's a giant slider on the left hand side of the screen that you pull down and it lets you rewind to get you back and you see those values moving back to their earlier

forms. You're talking about HEST. But there's no reason that this can't be that, right? Like, that is a, it's an interface detail. Right. Yeah, like that's the missing thing. Right. Is that expressed here or is that like just sort of No, that was me taking the piss. No, no, no.

I'm speaking to Jimmy. I don't think that it's talked about here at all. I will say, though, that this also is very reminiscent of the interface we saw in Stop Drawing Dead Fish to specify the fish's behavior. We have these distinct frames, and we're moving objects across these frames to articulate what the computation should be

without having to actually write a bunch of code instead we're demonstrating how these things should move with geometric constructions it's a canvas etc right like there's clearly an influence here of these kind of systems yeah so in stage cast creator this example i brought up you can go back and edit certain steps along the process but it doesn't feel first class you know like it feels like like that's the the false path the sinful you've you've fucked something up path you know like so this this um process of demonstrating behaviors is so nice in StageCast Creator. Like there's a lot of thought put into it. It's the first thing you learn.

But this process of going back and tweaking, it involves like drop down menus. It involves like more traditional kinds of choosing behavior that like I think we see in things like Scratch now, you know. And my solution to that was to just get rid of that requirement, like keep it simpler. And that means that these before and after diagrams, they needed to have a lot more expressive power. So you actually lose some of the ability to just demonstrate things simply. Like there is more complexity that comes from that. But yeah, I can imagine a different way of doing that

where being able to tweak different actions you made in time in a more polished way. Like I don't think I've seen a lot of work into that, but I imagine there's some out there. I know Ivan's trying. Yeah, one way that you could do this in a programmer-y way, just an implementation detail here,

is you could say, hey, just trap at every step. You get a step debugger here. And you could do this without the system, like in the existing system as they've described it here, there would be no reason I couldn't just say, hey, the default behavior for all of these things is to trap at every step.

And then I decide at that point, do I continue or do I now make a new memory? I am editing the code by just articulating in the same way I wanted to. So obviously Ivan's little dragger timeline is much better of an interface than having to step, step, step, step, step, step. But the point is you could reprogram the system just by programming the system. There doesn't have to be an alternative interface.

So just to crystallize a little bit of what's interesting about HAST here and what's deficient about hast aside from the fact that it doesn't exist and never will um oh my god and other you know attempts at this stagecraft creator when you want to go back and make changes to that existing program that you made by acting out what the program should do when it executes the hest approach of like and it has does a bunch of things differently from this that i don't want to get into but i don't want to say that like oh hest is the same as this plus some extra stuff hest is like making very different decisions but when it comes to like going to that moment where you want to introduce a change like

that moment in the course of execution going to that moment and saying instead of it happening this way i want it to happen that way and like performing the change or like having to go in the execution to the place as opposed to going in the source text to the place you go in the flow of execution or you step in the debugger to that place there's a really interesting trade-off that you can make there a really interesting choice and like cell pond in in having its you know representation of the program is these before and after diagrams that are outside the executing environment you're not demonstrating them you're like inscribing them and they are a static representation right you can take a

screenshot of a cell pond program and recreate it from that screenshot asterisk yes yes yes yes if if if if it had no bugs yeah yeah and if the exact um configuration of inputs that you put into the canvas wasn't also meaningful because there's that as well right it's like if i have a screenshot of cell pond i might not be able to recreate the full image that i'm seeing on the canvas just from the rules i also need to know where to click on the canvas at what times with what set of rules there's like state there too but if you were to in pig million go to like trap to a certain spot and then make some changes, there's this really interesting tension between, like, what happens to the other path that this

code used to take? Like, what happens to that other version of the system, that other demonstrated logic, now that I've introduced a different version. It's something that we get from static representations using tools like Git, where it's like, if I make a change in my source text at a certain point, I can say, well, let me do that on a branch. So I keep the previous version of the code and can still execute it and have it do its thing. But let me also keep this new version.

And the thing that's missing from Pygmalion and also HEST, but not missing from Stagecraft Creator. Stagecast. Stagecast, sorry. Cast. Stagecast, sorry. Yes. Is this timeline view, this like movie film strip, this like, here's the sequence of steps.

Pygmalion has it, but it's a machine code textual representation it's hidden away yeah and it's a different form that you interact with differently if you can interact right at all right and there's no like even in pig million having this like you're making a movie of the things and they use that language there's no play pause rewind forward, step back controls. It's very much in the now, and that's a blessing and a curse. And same with Brett's drawing dead fish. There's those frames of the sequence of transformations that you've made, which is, like you flagged, Lou,

it doesn't feel quite the same because it's not the interface through which you did the demonstration. You didn't do the programming in that little timeline view. Right. branching in a performative like programming by demonstration environment without introducing some accessory representation that's outside of it right you know marcel your colleague marcel uh who's i've never pronounced his last name on air because i feel like i would butcher it but i'm gonna say guttal is that it right right yeah marcel he has shown me a prototype of this thing he's working on

that's like comic strips have you seen this uh maybe i don't know i've seen a lot of stuff marcel makes marcel makes a lot of cool weird stuff link in the show notes yeah um he he showed me this idea he had of using a comic strip format to make this kind of demonstrative coding work, right? Like, you don't, it's not just before and after pictures, right? It's, it is before and after pictures, but you might have like seven different pictures that show different parts of a factorial, but there's more. on those pictures you also have like annotations of movement right you could say that this um this six

moves down to here or goes through this little hole but like not in the programming hole like an actual hole it's a bit weird but like that sort of thing feels feels appropriate i think i think the thing that you're like you're you're getting at is is this phrygian phrygian phrygian phrygian versus phrygian phrygian versus analogy all right that's the end of the episode right there listen no no no this is important yeah uh friggin versus analogous uh comparison because you can start to mark out these all of these different approaches on this spectrum right like textual code uh is on one side of the spectrum it's very different from what you're actually coding right like there's a big jump between the source code and the execution. And on the other end of the spectrum, we have PygMillion, which the

source code, your demonstration, is almost exactly the same as the actual execution, because you're doing it. You are doing the execution. That's how you source code it. Something like CellPund is somewhere in the middle because this these paddles that which is where you do the code where you create these diagrams it's not the execution itself it's like a abstract representative representation of of the actual execution it it is symbolic in a way and that is that's different to pygmalion um something like a roost or baba is you this is this is very over indexed on spatial programming because it's what i know about right those things you are actually coding it in the environment itself right so like it's less of a representation more of a simulation where you're coding within

the world you know what i mean yeah like there's there's a question of where does the computer's behavior come from does it come from you using signs and symbols and in a node wire language those are the nodes and wires in a roost they are the little frog eggs with the lines between them um yes in in lisp it's you know the the s expressions that sort of thing like is the is the representation of the program something that you create that representation directly in an abstract way do you create it in a more concrete way or do you actually like act it out which is a different question from like when you're doing those things what are you doing it in terms of are you doing it in terms of concrete values are you doing it in terms of placeholders where concrete values will go at runtime which is a different question from

what form do those representations or those values take are they visual are they text are they you know you could probably do this with sound or you could do this with smell or with taste if we had any there's like design dimensions here along each of these different um uh dimensions that oh it's that point in the recording yeah well okay yeah there's like you can you can you can mix and match any of these things is what i'm saying is it's like this is not a linearized single spectrum of of you know from here through lisp through pygmalion to hast as an endpoint it's like a there's many different ways that we can go to eventually get it hast what do you you think, Jimmy? Yeah, I think that maybe we're doing

concluding thoughts here for me. I think there's some stuff that we haven't covered in here, of course. We took out whole sections. But this, to me, is a visual programming system that I could actually imagine myself using. Of course, a more polished, a finished version of this, a modern version of this or whatever.

It's something that I am interested in like exploring and like our most recent Patreon episode where we talked about like visual representation of machine code. This is where a lot of those thoughts are coming from. Because writing machine code or writing compiler ir by hand is not very fun uh but writing doing demonstrations of them where they remember what i did would be quite direct and quite easy and uh i would find it i would love to have like a compiler where i'm writing the compiler and i have some op code that I need to now compile to machine code. And instead of having to go write out all of this intermediate representation

that's in such a low-level thing, I could just literally have a thing like this pop up and I could start dragging values around. I see my registers and I see all of the data that I'm working with and I start doing a demonstration and it records that IR for me. It would be so much easier to write things that way i i think this to me is like the first like i'm not against visual programming for people who want it but i've never felt the draw and i think this is the first system that has made me feel that way i love all all the irrelevant bits. You know, yes, there's some problematic

elements here, but I love that there was an attempt at bringing in art history. I love that there was an attempt at bringing in psychology. I love there was an attempt at doing all of these little bits. And did they quite work? No, this is a dissertation. This is the first work coming out of, you know, somebody getting their PhD. Is it perfect nah but did i enjoy it yeah absolutely i think this is one of those texts that's like engelbart's where like yes it could have used some more editing it could have been a tighter picture it could have conveyed those ideas much better but there's something just interesting about them, something meaty about these ideas. And it's research that I don't like a lost tradition in programming that would have been really exciting to dive into.

And I hope that more people take this and go make some systems like this, because I'm hoping to do it when I have the time to as well.