Future of Coding - The Edges of Representation: Katherine Ye

Episode Date: December 5, 2018

Katherine Ye is a PhD student at CMU, where she works on representation, including programming languages, visualizations, notations, and interfaces to enable thinking and creating. She's been affiliated with MIT CSAIL, Princeton, Distill at Google Brain, and the Recurse Center. In this conversation we discuss Penrose, her project to _democraize visual intuition_. Katherine envisions "a magical machine where you can dump in a math textbook and out comes a fully-illustrated math textbook, or more specifically a platform where you can simply type mathematical notation in plain text and automatically get many useful and beautiful diagrams out illustrating the notation." It's a fascinating project in the intersection of mathematics, intuition, education, visualization, communication, programming, domain specific languages... basically, all of the interesting topics in one project. As you'd expect in a conversation about the edges of representation, this is a wide-ranging conversation that I can described by a collection of keywords that came up: embodied intuition code as rhetoric asemic language Colorless green ideas sleep furiously. univalence, homotopy, equivalence, equality modeling the notation of mathematics knot notation, dance notation, and the periodic table of juggling notation a studio class on notation design explorable explanations speculative nonfiction the unexpected futures next door Transcript provided by repl.it at https://futureofcoding.org/episodes/34#transcriptSupport us on Patreon: https://www.patreon.com/futureofcodingSee omnystudio.com/listener for privacy information.

Hello, and welcome to the Future of Coding. This is Steve Krause. Today we have Catherine Yee on the podcast. Catherine is a PhD student at CMU, where she works on representation, including programming languages, visualizations, notations, and interfaces to enable thinking and creating. She's been affiliated with MIT CSAIL, Princeton Distill at Google Brain, which is that really cool publication I imagine many of you have heard about doing AI explorables, as well as the Reker Center. I think I found Catherine through Twitter originally and signed up for her speculative nonfiction newsletter, which I'd

highly recommend. But I reached out to get her on this podcast when I found her Penrose project. It's a fascinating project in the intersection of mathematics, intuition, education, visualization, communication, programming, domain-specific languages, basically all of the interesting topics in one project. And now a message from our sponsor. Repl.it is an online repl for over 30 languages. It started out as a code playground,

but now it scales up to a full development environment where you can do everything from deploying web servers to training ML models, all driven by the REPL. They're a small startup in San Francisco, but they reach millions of programmers, students, and teachers. They're looking for hackers interested in the future of coding and making software tools more accessible and enjoyable. So email jobs at repl.it if you're interested in learning more. So with that, welcome, Catherine. It's great to be here, Steve. Yeah, it's really great to have you. So I like to get these conversations started with a brief history of your background, particularly how you got inspired to work on these problems originally,

like who your main influences? Yeah, so from the start, I've always been interested in interdisciplinary programming languages research and in designing and using powerful environments. And so, I mean, PL is funny because it has these two sides of you are very much verifying a thing and your problem statement is pretty set. You just need to figure it out. Or you're synthesizing or designing a thing, which is much more open-ended. And I think it was good for me to get that foundation in rigor and verification and clarity of thought and to bring that then to design and synthesis

as a grad student yeah i'd be curious yeah to talk about penrose specifically um where the inspiration came for for it from uh i know when i was reading the project description it um very much sounded like uh red victory of stuff, particularly his Kill Math projects was that an inspiration or did you find inspiration elsewhere? Yeah, so the seed of the project came from a conversation between Keenan and I Keenan is my co-advisor, he is an assistant professor at CMU in the computer science department and cross-listed with the Robotics Institute. He does graphics and he's

a really awesome mathematician and geometer. So, you know, I came to, I met with Keenan. So the process at CMU is you come in as a grad student and then you meet with tens of faculty to go through this advisor matching process. So Kenan was the one I met with and I was really interested in doing research that was kind of spanning both language and image, didn't know quite what it would be. And Keenan basically proposed the seed for what would become Penrose on the spot, which was, you know, you had this language expertise, and I have this math and graphics expertise. What if we teamed together and we made this thing

where you can simply type math notation in plain text and automatically get these beautiful diagrams out? And I was instantly sold. And then we pitched the idea to our collaborators and advisors, Jonathan and Josh. They were sold, and then we started working on it. Can you describe your own thinking process? Like when you think about mathematical or abstract topics are you thinking in pictures are they are they important are you drawing pictures is that important to your process

yeah absolutely so one thing that is maybe an interesting note is the relationship of the mind's eye to one's personal process and how that may or may not generalize to other people i think it's a little bit of a a pipe theory i might have heard heard this from Michael Nielsen that people or mathematicians who are very strong at creating visualizations often tend to themselves have a very poor mind's eye. So they have to externalize that. Whereas people who,

and people have different strengths of visualization ability, just in their head. So it's going to differ from person to person. And those who are maybe very strong at visualizing these photorealistic images or whatnot might be less likely to actually create visualizations because they imagine that everyone else is able to do so very well. So in terms of how that relates to me, my mind's eye is actually very poor. I think there's this famous study by Bacon or someone.

I can send you the link later. One of the early studies where he just went around buy bacon or someone, I can send you the link later. One of the early studies where he just went around asking a bunch of people of various professions, this systematic questionnaire of, you know, when you visualize the face of someone you love, is it in the distance? Is it really big? How bright is it? Is it centered? Are they moving? And then asking this about, you this about various subjects of personal importance, like your room or your boss, things that you normally would be able to recall instantly.

And then starting to ask about more abstract subjects like a sphere or an imaginary perfect beach or whatnot and seeing how, what kinds of subjects and situations would change the strength of that visualization. Wow. Fascinating. I'm trying to do it right now and it is not easy. Yeah. Right. I mean, imagine the face of a beloved family member. And are they moving? Do you see them wearing a habitual outfit? What kind of stride do they have? Are they speaking? Is it easy to hold the face in your mind? Or does it just disappear in an instant? These are all different for different people yeah i i can't do this very well or it's not resulting in very clear pictures it's very very abstract i think for me yeah exactly so back to your original question you know what do i actually do when visualizing things so because my mind's eye feels very poor like i have to i feel like

i have to physically hold a picture in my head so i think that's why my ability to draw things is stronger than um my my ability to see them and that's why i draw so many things got it That's funny. So I think there are like some people who doubt visualization or like doubt intuition. I feel like you could call them symbol pushers. Symbol pushers. That's great. Yeah.

Pushing algebra on the kids yeah yeah well i feel like a lot of pedagogically a lot of people a lot of like you know elementary school or arithmetic teachers or just people in general who who aren't mathematicians themselves or don't have mathematical thinking or or think that or maybe they have it they don't think other people can have it um they feel like symbol pushing is a reasonable way to teach things. And I feel like it's kind of related to the Copenhagen interpretation of quantum mechanics, the shut up and calculate. Don't worry about what it means. You don't have to have a model for it in your head.

Just be a computer, basically, and push symbols. Do you feel like you know what I'm talking about? Have you seen the same thing when you've explained your project? Like, oh, only some people need visualizations. Other people can just push symbols. Have you heard this quote? Algebra is the offer made by the devil to the mathematician. The devil says, I will give you this powerful machine.

It will answer any question you like. All you need to do is give me your soul, give up geometry, and you'll have this powerful machine it will answer any question you like all you need to do is give me your soul give up geometry and you'll have this marvelous machine atia i think 2004 i don't have this quote memorized i just looked it up because it's so apropos that's very funny uh so that basically summarizes my opinion on this. Sorry, can you spell it out for me? Yeah, so symbol pushing is very powerful. It offers rigor that is essential for many purposes. This is why, incidentally, proof assistants are entirely textual and symbolic. Although there are some visual proof assistants, mostly for

category theory. I'm not familiar. I haven't used those. So, very powerful, but then you lose the understanding for why it's true. And it's, I think, in in math being able to being able to present your results to other humans and have them have it in their heads and be able to contribute to it is actually almost more

important than having a correct result this is like the problem with the proof like Machiz with the proof, like Matsuzuki's proof of the ABC conjecture, where he basically did this totally inexplicable, marvelous thing, but nobody understands it and they can't build on it. And so it's just this kind of monument that people stare up at and can't make any sense of.

Well, so I think it could go either way for that specific example he could either be uh you know just symbol pushing and then and nobody can understand his symbols uh like they can follow the symbol pushes but they don't have an intuition for it or on the other side uh the way he did i don't know you could tell me the way he did this maybe was through like deep intuition mystic thinking but and he like can't complain explain his intuition somehow or like you know it's hard for him to visualize it explain it did you know which is the case no i'm not i haven't been able to and i haven't tried to actually dig through his work so i don't know but i mean it sounds like there are two strands here, the question of the importance of intuition in math and the question of the importance of visualization in math. And those often coincide, but often don't. Yes, yes. That's well said.

So I have this quote here from Brett Victor where he says, The dirty little secret is that the greatest mathematicians don't actually think in symbols. Einstein himself said that he did his thinking with sensations of a kinesthetic or muscular type. So that's not visual. That's like body thinking, which Seymour Papert, I think, called anthropomorphic thinking, where you like imagine yourself as the turtle and you solve the problems, like imagine what you would turtle and you solve the problems, like imagine what you would do with your legs. And I remember when I learned Logo that that was like a transformation for me,

like learning how to leverage my body knowledge to solve mathematical problems. So, but you're right. That's very, very different than visual thinking. Yeah. I, i recall some i think maybe a quote from fine men's you're joking where he is trying to understand some physical process and he is just I think lying down on the hotel room floor, rolling around. And I can't recall exactly what happened. Maybe like people walked in and thought,

Oh, this guy is totally nuts or something. But like he had to like actually lie down and do this kind of ridiculous seeming physical motion to understand what he was trying to derive the symbols for. So that definitely seems like a common habit in people who are trying to get a deep intuition or something beyond the symbols to have, it doesn't have to be visual.

It could be a kinesthetic or embodied understanding of the intuition yeah um yeah they're definitely both very powerful i think um like another brett victor quote is um how we want to we have all these abstract complicated concepts but we're stuck with these plain old monkey brains and these plain old monkey eyes. But we can leverage them through like kind of almost like tricks to like be able to feel or see the abstract thing somehow. And I think Brett Victor references Edward Tufte's work a lot. Have you read some of his books? Yeah, I'm familiar with some Edward Tufte's work a lot. Have you read some of his books? Yeah, I'm familiar with some of Tufte's work. Tufte in particular talks about William Playfair and how his XY data plot,

he was like the first person to put time on the X axis and then like some number on the y-axis things like that um he talks about how you know all of a sudden our monkey monkey eyes which can like judge uh you know change small changes in visual height can all of a sudden be like you know leveraged to judge changes and abstract quantities over time and how like something as simple as this new notation visualization helps spark the scientific revolution, which every journal has, the XY plot. So anyways, I think a lot of people don't realize that that was invented and the impact it had.

Yeah, exactly. I mean, you know, you ask the fish how the water is and they say what water. Yeah. So let's talk about some of the specifics of Penrose. Because I think from just a programming, just like from an implementation perspective, you made a lot of interesting choices from the engineering design and there's a lot of technical challenges that you're working through.

Can you maybe back up and describe the vision for the project and then the summary of the vision is a magical machine where you can dump in a math textbook and out comes a fully illustrated math textbook, or more specifically, a platform where you can simply type mathematical notation in plain text and automatically get many beautiful and useful diagrams out illustrating the notation that's the vision we want to democratize visual intuition so making that happen it motivates just that sentence which has been our sentence from the beginning, that's motivating us, has driven several design choices in the system. For example, one fundamental question is, do we take a language-based approach

or an interaction or image-based approach first? And the latter could be natural. In fact, if you ask most people how they would design diagramming software, they would probably say build a GUI. But thinking about here first, that there already exists a notation that people are familiar with, that people are working with, which is mathematical notation, that we have this one textual language.

How can we design a language that enables expert users to specify how those mathematical elements are translated into visual elements systematically? The essence of diagramming, defining a visual language for your domain. How should you do that? It seems natural for at least the internal representation of that to be text. Maybe eventually it could be GUI, but internally it simply has to be text because if you specified the visual representation otherwise, then that would be inherently visual, and then you wouldn't be able to swap out different kinds of visual representations.

And then thinking about, there's no way we can build this all ourselves for any possible domain. We need to build an extensible system so people can add their own domains as plugins. It needs to be flexible. It needs to have mechanisms designed from the start for this. And that motivated our design of the language called Element,

which is a plugin for style, a little bit like a module systems type signature, where you say, you just tell us what the fundamental objects in your system are as types, and then the functions and predicates that relate these objects, so the objects and their relationships. And this doesn't have to be specific to math. It could be any semantic domain, say chemical reactions. And that, so those three languages, element, substance, and style. So substance is like the HTML of our language and style is like CSS for substance, like pattern matching, but really souped up. And then, so those are the textual front end of our system.

And then it comes with a powerful numerical solver. So this is now motivating the whole graphics design of the system. By the way, in the write-up on the Penrose site, there's a diagram of the pipeline, and that might clarify anyone who's having trouble understanding my linear description of what's going on. Yes. Yeah. I mean, linear by necessity because speech is linear, but there's a diagram. Did you use Penrose to make this diagram? One day. One day.

Very cool. Yeah, that was a great summary. One of the analogies I saw in the write-up and also you used just now is the HTML and CSS analogy. And for me, I feel like this is very much a nitpick, but I feel like a better analogy is how today in web programming, we have a model, like some like JSON-y thing,

and then we have HTML and CSS that's like a pure function from the model. And so to me, that separation of the model and then the view is kind of like your substance and style. Because to me, HTML and CSS are both very, very visual. So I find it hard to make that distinction. Does that make sense?

Or do you disagree with my characterization? Yeah, I think model view is also a way to understand it. The thing is we need to use an analogy that makes our design familiar to people who don't have necessarily a web programming background. So if you say model view and so on, they might not know what we're talking about. Okay, cool. Yeah, that makes sense. I didn't know if you chose that analogy for that reason or because it was a better fit. So one of the things that occurs to me is that this platform, I think it definitely makes sense from the perspective of I have a math textbook

or I'm a mathematician and I want to like add some visualizations. That's like that story makes a lot of sense. But the story from the other direction of like I'm a math student and I'm looking to understand what I'm doing, that it's a little bit harder. It reminds me of Seymour Papert's quote about how like if you taught dance with the notation first, nobody would want to be a dancer. You have to like learn dance with like the movement part first and then maybe you teach a notation later. But with Penrose, you have to start with a notation and then it comes gives you the visualization, not the other way around. Do you see there being any hope or promise of starting with visuals and then maybe giving you notation?

Is that possible at all? I think that is not in the scope of our system. And I think the design as it is has a great deal of promise in educational domains. For example, one thing that Penrose can help with for as a student is you have the notation but you simply don't quite know how to illustrate it and you don't know what the important cases are and you could't know what the important cases are.

And you could simply type that in, and then Penrose could help you find the important cases. For example, if you just typed in a bunch of set notation, that might be given under a specified diagram and then Penrose could find, oh, you have these special cases where A and B are equal to C, but you only wrote subset equal, and that might help you understand it. Or you had some points in set, but it wasn't clear whether P would lie in the set or not. And then it could find these sorts of kind of like fuzzing your specification. And these are often important corner cases in the notation that trip up learners. And in addition to the corner cases, these general cases of what is

the least misleading way to illustrate this notation as generally as possible. So I think I read in the description that you have some maybe direct manipulation tweaks of the visualization output at the end of the process. I was wondering how those were scored or saved. Is it like a bidirectional thing, like sketch and sketch? Does it make its way back into style somehow, or is it another layer? Right.

So we are not aiming for pro pro direct or bidirectional manipulation. So the way that works is you've compiled your three element substance and style programs. You've gotten a diagram that has been generated by our optimizer. And that diagram is specified with some combination of objectives and constraints. And for the objectives and constraints, so the constraints need to be satisfied. They're hard. And the objectives are just make the energy of this function as low as possible. So try and find something that you like. And so when you drag around these things in the GUI,

we re-optimize the diagram right after that in order to preserve the objectives and constraints on it. So concretely, if you had written some point P is in some set A and you like drag that point out of the set it would get pushed back in because that would violate the mathematics or the substance specification that was combined with the style and so this is also ties back to the educational application because for a student to edit this diagram and then to actually do something that violated what the mathematics meant and have that system, have the knowledge of what it means

and automatically preserve it, but also do the minimum possible change to preserve it, because that's how the optimizer works. I think that's very powerful and no other general purpose GUIpaced tool can let you do that. There's certainly systems with the specialized knowledge built in. For example, Cinderella, which is geometry software, where you can drag around things and have the semantics of the math preserved.

But those are all custom-built software for these constraints. I see. So I was curious to talk briefly about constraint-based programming because it's not something that I have a lot of experience with, and it sounds great. It sounds declarative. In particular, I've been curious about how it relates to other paradigms of programming, like functional programming,

for example um i i think um the the downside i've always seen is that it's like kind of unpredictable somehow or like the performance characteristics of it aren't it's like kind of opaque you know you like you specify some things and then who knows if it's like what it'll come up with on the other end yeah how do you think about constraint-based programming yeah so i think the sense in which you're using constraint-based programming is different from the sense in which i'm using it and the sense in which it shows up in penrose so it seems like you're talking about languages like prolog or MiniKanren, which are relational slash logic slash constraint-based languages.

And I think they hook into either SAT solvers or some kind of logical search engine. And I personally don't have very much experience using these systems, so I can't speak to that. In Penrose, what I mean by constraint-based or solving constraints is that in a style program, as a style writer, so as an expert user, you might specify the visual representation of some mathematical objects or relationships in terms of constraints. So you might say, for any vector that's in a vector space, I would like you to draw the vector as an arrow whose root is at the origin of the vector space. Oh, also draw the vector space as a square. And I'd like the head of the arrow to be constrained to be inside the shape of the vector

space. So naturally, you know, as a person, you don't want to draw the vector going outside of whatever visual representation you had for the vector space. So that's just telling your system, apply this constraint function, which is some math that's like, you know, if the tip of the arrow goes outside the square, then penalize it very heavily in the resulting objective function that we optimize with our solver. And so in style, all of these constraints objectives are jointly optimized, there's currently no kind of logical search or proof search or SAT solving happening. I see. And so can I specify the weighting of

my preferences? Yeah, for sure. So you could add, so as a person who is implementing the domain, you can certainly use your objectives and constraints and add a parameter that's like the weight and then just say the weight is some flow. And then in style, you can say the weight is 10 or 100 or 1,000 or 10 to the negative fifth or whatever. Got it. Very cool. And I'm sure you've thought about

making visualizations that are also interactive given your work on explorable explanations at Distill. So has that been built into the Penrose master plan at all yet? We'd love to do that one day. We have more interesting questions

than we know what to do with. But I think it would be pretty... It's in our roadmap eventually, because we like this capability of Mathematica where you can simply pick a parameter and say, okay, that's a slider, and that's automatically parameterized and interactive. So I think that's something we could add with not so much engineering effort that would actually help a great deal. You could just say, like, vary this parameter in the diagram. But with more sophisticated kinds of interactivity it's hard to think of how to make that automatic and i think that's not in our roadmap yeah that makes sense it's complicated problem yeah and other people are tackling it that problem head-on as a main research question it's its own things i'm not sure if we can simply

add it to a roadmap and do it could you um if you know them offhand list some of those people for reference yeah so i think matt conlon's language idle idle have you heard of it? Yeah. So I haven't used it personally, but I believe they're a markdown variant that integrates with web technologies so you can make scrollable, interactive, web-based explanations. And their work seems very cool. Agreed. Another interesting question that I imagine is kind of farther into the future,

or maybe just a speculative question, is so you have the standard math notation to visualization framework. And just to reiterate, you have domain experts specifying the types of objects, and then you have users specifying specific objects in their relationships, and then you have a visualization layer. Part of me is wondering if we could just chop off the visualization layer for now, and we just have a domain-specific language to explain objects and then then specific objects in a different layer and then kind of turn that into like a replacement for pencil and paper math particularly for

students but maybe for others kind of like how caulk is becoming a replacement for proofs. I guess another way to phrase this question is pencil and paper math is great because it's flexible, but it's crappy because the feedback loop is so long for students. It's like unclear if they did something right or wrong until they give a paper to the teacher and the teacher marks it up with red marks

and gives it back to them like a week later. So I'd be curious your thoughts on how to like tighten that feedback loop for students. Yeah. So the Henry Rose substance language is not as expressive by design as many of the other languages for expressing mathematics. And I think those are better suited for your purpose.

And I think your purpose is definitely one that I care about and a noble one. So have you seen this thesis called The Language of Math? No. By Mohan Ganesalingam, Cambridge PhD. Cool. It's very good. It influenced our design of the substance language quite a bit. And he's doing something much more ambitious than we did,

which is model, well, study the full breadth and depth of mathematical language in the wild and try to systematically characterize it and write a parser for it. And that's a really great reference for, you know, written by a mathematician, how to model mathematical language and the ways in which existing proof of systems fall short. So he actually has a section on, I don't know, whole light or MISA or something and how it can't quite do what it needs to do. But yeah, so in general, just executable and precise and intuitive representations of mathematical expressions are something that I care about and that I find to be important. And as a long-time cock user, it is pretty amazing. It's not just for Proust.

It's pretty amazing how fast the feedback loop is for writing something and then immediately seeing, like, oh, I can check this line. I can move on. Or I can't check this line. I did something wrong. It doesn't type check this line. I did something wrong. It doesn't type check. What did I misunderstand about the math?

It's pretty cool. But it's certainly too advanced for an elementary school level. So there's, I mean, there are lots of games that actually try to do this. I think DragonBlock is one of them. And there might be more where it's kind of like a gamified algebra thing where the rules are really rified in the game. So if you try to, I don't know, divide by zero. I have actually played this,

but maybe if you try to divide by zero on both sides, it just doesn't let you, but it'll let you intuitively drag a factor to the other side of the equation and immediately give you feedback on what that does. That's awesome. And yeah, thanks for that thesis or that paper you mentioned about the language of mathematics. That definitely sounds like a good place to follow up with this question. I just want to make a note that a few times in this conversation, you've used like

the phrase modeling the notation of mathematics. And I find that phrase to be delightful because like mathematics is the language of trying to model the world or model like, you know, specific things. And so we're trying to model a language that models other things. I find that wonderfully meta. Oh yeah. Modeling the model. I hope nobody tries to model us. What do you mean? You work for Distill. Isn't that what you guys do? Oh boy. Yeah. Models on models. So I think one resource that you put together that I really enjoy is your list of notations. That's a really fun list.

So maybe you could summarize what it is and what you've gotten from that list so far. Yeah, so it's a syllabus I put together on how notations influence thought. Not necessarily improving it or augmenting it, as one might suspect, but just how it can change the way you think. this so I did this maybe three years ago four years ago I can't remember and I when I made it at the time I was very surprised that no one had already made it that's why I made it. It wasn't like I was teaching a class on it at that time or something. One of the impetuses was this talk I gave at Strange Loop on not notation, Conway's not notation. And after giving this talk, I just was like, you know, a brief summary of this talk. So Conway has this really interesting and wacky notation for knots that involves these

very specific operations around flipping a thing diagonally and then adding it like visually to another knot and then constructing knots in that way and he made these very bold claims about not enumeration like listing all the kinds of knots and how his notation enabled him to do all these powerful things and i was like wow i need to understand how that happened so then i proposed to give a talk on it, and then they accepted my talk to my great dismay, and then I had to actually learn it. But so after I gave this talk, I was like thinking about generally taking a step back and all the other ways that, you know, knots are not literally knots. Like knots are metaphors for things that are hard to describe.

So everything else in this syllabus is about things that are hard to describe and the ways we insist on describing them and ways that are better or worse for describing them. You mentioned dance notation, and dance notation is one of them. You mentioned dance notation and dance notation is one of them. Or for example, I just love Chanda Horowitz's series on sauna kinetography where she makes these beautiful visual notations, paintings for this sculpture performance series that never actually happened and she's she's got these paintings describing that i believe it's like the timing of the lights or the timing of the movements and it's very precise and very beautiful or another one is this wonderful

write-up by colin wright on juggling notation so i mean juggling that seems hard to describe but apparently i'm not juggling myself but apparently there's a periodic table of juggling notations and you can discover new juggles by systematically examining the ways that you might describe these sequences. So it sounds like your motivation here was like the Sapir-Whorf hypothesis as applied to notation instead of spoken language. Is that a way of characterizing it? Well, Sapir-Whorf is... You know, that's a really good question. I haven't thought about how it relates to what I made. But Sapir-Whorf is...

I think it's very specific to natural languages. And the folks involved have not made statements about other kinds of descriptions. And so I'm not sure if their assertion is relevant here. equivalent question for a lot of languages, I guess. There's, does like the spoken language you speak affect the things you think? Does the programming language you program in affect the sorts of programs you write? Do the notations you write in affect, do the visualizations you see or write affect? You know, I feel like there's a common, a common question of representation affecting

something more semantic somehow. Yeah, well, there's your notation affecting the way you think, but there's also the way you think, like the culture affecting the representations that are designed around it. It's a loop. Yeah. though representations that are designed around it it's a loop yeah um

yeah that yeah i'm sure yeah culture plays into it as well um so um one of the things that so in in the syllabus of like studying notation one of the things that at least maybe i I just missed it, but that I really wanted to see there is process notes on if I'm ambitious enough to want to create a notation, how do I

go about it? What are the tricks of notation designers? I'm going to write that down. That's really interesting. Sorry. Well, I mean, the question is what you want to design a notation for. And I think I had, well okay, so the syllabus is not aiming to do that. That's going from a syllabus to say a studio class on notation design, which is a little different. But there is a little bit of a hit there in terms of, I'm thinking of two things. First, the Greens paper on dimensions of cognitive

notations, cognitive dimensions of notations, something like that. And I think they discuss more specifically, and systematically, the elements of a good,-unquote good versus bad notation and different kinds of cognitive tasks that different kinds of notations can help scaffold so i think that's one resource and second there are some slides in my Strangeloop talk where I was, and it's just only a very few at the end, but I was thinking about more abstractly how these descriptions map to objects and how objects map to descriptions, the properties of that mapping, whether it's one-to-one or many-to-one, one-to-many, and what that means

in terms of intuition and enumerability. So those are two starts. I wonder if there are more out there. Well, so that point you made about the mapping, I do really want to talk about that because I saw that in your talk and that's something that I think about a lot, actually. But before we jump to that topic, I just want to say that it feels to me like the creation of notation is very related to the design of programming languages and the design of macros in programming languages or TSLs or even libraries. It's a very related kind of thing. A programming language is very much like a notation. And then it's also kind of related to user interface design or really any kind of design where someone wants to interact with it.

Even Gmail, is it a notation? Is it a programming language? It's a very limited programming language a very limited notation and it's like a very visual notation maybe i guess that's kind of debatable depends on how you define words but anyways i just thought i would say that it feels like the practice of being a notation designer would be related to these other activities. Yeah, absolutely. And I'm not sure if I would say that I am a notation designer, honestly. Like, from scratch, I, you know, it's not like I've designed bracket notation or whatever, Einstein's tensor notation. Like, I haven't designed a notation. And it's not

just me either. Within Penrose, it's the group of us making these language design decisions. But to your point, yeah, it feels like there's certainly a commonality between language design, interface design, visualization design, representation design, API design, all of that. Yeah. Like things I guess mediating the relationship between things seems like the commonality to me. So I want to go back to that topic you brought up of the the course on correspondence between a how do you say like a concept in your head and then a notation like an encoded edition in the world that's how you describe it and so yeah in your

talk you have like these two sets and like and like one set is concepts and the other set below it is notation and you have like arrows between the two have you know you could have concepts that don't have notation you can have concepts that map to multiple notation you know it's like a mapping thing you can have have notations that don't map to any concepts etc um so and you were, I think the dream is that you have a certain notation such that there's a one-to-one mapping. For every concept, there's a notation and for every notation, there's a concept. Is that the dream? I'm not sure. Okay. Yeah, I guess it is debatable. I think that that's certainly an interesting property,

but I think there are many other interesting properties of these mappings. For example, part of the delay of the, of natural language is that it's so from a PL point of view, natural language is not orthogonal. There are many ways to say the same thing. That's where you get ambiguity and that's where you find humor

and delight. I guess there are also many things that you can't talk about or that we don't have the words for. Exactly. That's why there are many mediums and media. There are also many things you can say that don't have any meaning. They don't map to anything.

Oh, exactly. I, yeah, the edges of language are so delightful for me. These kinds of asemic language or asemic is more used for writing, I guess. Are you familiar with that word? Say it again? Asemic? I think that's how you say it. I've never heard anyone say it. A-S-E-M-I-C. No, I don't know that word. I think one of the best known examples is this seraphinous, I don't know how to say it,

this codex where everyone thinks, like there are these beautiful pictures and then the writing is just, nobody knows what language this writing is from. And scholars suspect that it's just someone writing some beautiful script with no actual meaning attached to it. So any attempts to decipher it are stymied by the fact that there is no actual meaning.

Yeah, or like, I mean, back to your point about language that doesn't mean anything. Like, if you take, you know, some of the things that are really at the edges of language, like colorless green ideas sleep furiously. What exactly does that mean, Chomsky? Wait, say it again. What's the phrase?

Colorless green ideas sleep furiously. Have you heard that? No. It's, I think, one of Chomsky's examples for language that is not supposed to mean something. But of course, all these linguists have written papers about how that sentence actually means something. That is funny. I can picture it, actually. Yeah.

Yeah, so anyway, obviously English is great and ambiguity is great um for a lot of things i i think my dream and i think it's a common dream for mathematic computer science types is like wouldn't it be nice if we could be specific uh and like precise for when we want to be um and so one dream i've had or one thought i've had which maybe you could tell me makes no sense is um wouldn't it be neat if for any concept there was one and only one notation for it. So like, I think that's just inherently not going to be possible because there are concepts that are, that span many domains. Yeah. Yeah. Well, so I guess what, what I'm getting at is so like,

so in a given notation, so like, you know, I'm defining a language right now, you know, it's called steve's notation and steve's notation um you can encode certain concepts and each concept you encode in my notation has only one representation so in every represent in every notation just in my notation does that you still think it's impossible? Well, I mean, is their language going to have arithmetic? Is one plus two going to be different from two plus one? What is the equality?

Yeah, so that's like exactly the point. So yeah, so that's actually where I was going to go next. So you would store one plus two. The notation would describe one plus two in such a way that it would be equivalent to 2 plus 1. If that makes sense. So the notion of 1 plus 2 or 2 plus 1, I think would just resolve to the number 3 and that's it. If you mean 1 plus x or x plus 1 and it can't be resolved,

then I think it would just be stored as a... It would be stored not only as 1 plus x, but it would be stored as x plus one and it can't be resolved, then I think it would just be stored as a, it would be stored not only as one plus x, but it would be stored as x plus one. You know, it'd be stored as the pair or maybe it'd be stored as, you know, one plus x, but all, but by the way, addition is, you know, commutative. So don't, you know, or whatever. So it, you know, basically, yes, the answer to your question is yes, it would try and map all concepts to one notation.

Okay, so what I'm hearing is, one, some kind of normal form. Yes. One, some kind of evaluation strategy. Two, maybe several kinds of well-defined normal forms. Yeah, that's a good way to put it. Perhaps, but I guess what is the end goal of this? So, yeah, that's a good question. And it's kind of vague at this point, but my, my thinking is.

So actually this is kind of related to another topic that I've seen you talk about on the internet is it's, it's good to be at like the highest level of abstraction possible. Like encoding your ideas in the highest level of abstraction possible. So you can be as precise as possible about what you mean. Like when you're forced to describe visualizations in the language of like vector shapes, you're not communicating to the computer what these things mean. And so the computer can't like move things automatically for you and help you

with things. If you move a point out of the vector space it's in, it's not going to like complain because it doesn't realize that it's a point of vector space it just thinks it's unrelated shapes and so it's related to that that idea of um and i guess part of it is like refactoring so if if um i describe like an algorithm in one way and you describe the same algorithm in a different way and we're trying to merge our code, it's going to complain, even though it's the same underlying algorithm. So it's like we're not being abstract enough about what we're trying to accomplish somehow. And then it's going to get in the way of other things we want to do. It's very vague, so it's hard for me to communicate in a short way the problem.

Did anything come across there? Yeah. I mean, I have an unrelated complaint, which is that you might be trying to solve the whole thing problem, but also it sounds like you're tackling one of the most fundamental questions in PL, which is what is equality? What is equivalence? Yeah, so, yeah, I didn't realize that was one of the fundamental questions. How does, what are the keywords or like papers, you know, can you give me some related work on this topic? Yeah, so this is definitely more of a theoretical appeal concern

that I heard many people talk about when I was still in the more appeal theory and verification world. But some keywords, I guess, are univalence, homotopy, certainly just equivalence and equality. Let me see if I can find this old paper. There are many papers that are just about definitions of equality. Oh yeah, and many logics that are about different notions of equality. Okay, let's see. Type equality. So, observational equality now is one paper, which I have not read, but I remember the

title. Oh, yeah, like decidability, deciding equivalence with sums on the empty type. I'm just looking at papers from the CWPL group. So one thing I feel like this is related to is type and the recent work on dependent types, stuff like that. The idea of a dependent type is this function doesn't just take a list and gives you another list. It gives you back a sorted list, which is the thing that you need to know about.

That's what a sort is. You're giving the computer more information. The one you're giving here sounds more like a liquid type than an independent type. I think a dependent type is specifically a type that depends on value, but yours is a type that depends on a predicate, which can be richer. Yeah, I think you're right. You're right. I'm talking about liquid types. Yeah, both seem cool and powerful to me. I mean, I program in Haskell. I program in Coq.

I don't spend so much time in Agda or Liquid Haskell, but it sounds like they're doing great work. Yeah. I think I saw you work on this end of history proof assistant paper. Oh, yeah. Yeah, that was one of the things I worked on with Adam. It feels relevant to this. You try to be as high level as possible, and then that that like helps with things is is that related to this

vague conversation or not really yeah the idea that you have liquid types certainly you have this very rich specification and specification only of say a DNS server. And then you use various synthesis tactics in the proof assistant to create a correct by construction and implementation of the DNS server from its high-level specification. That's very cool. Here, I have another try of explaining this random idea.

Okay. And then we can move on. So in imperative programming, if I wanted to sum up the numbers in a list, it's like a few different ways of doing it. Um, like I have an iterator and a loop and I add to the accumulator and then I'd return the accumulated thing. Uh, but in, in functional programming, it seems like there's like one way to do it. You're like folding plus starting at zero over a list.

It seems somehow more like, like that's representing more of the essential nature of what you're trying to accomplish when imperative programming, you're at like too low level of an abstraction. It's like not at all clear what your intent is when you're like creating these random variables. It's like too, too random and free form and so like i'm not suspicious that anytime you can represent the same idea in multiple different ways we're somehow not being abstract enough about what we're trying to accomplish to the computer if we were being more specific at a higher level abstraction then then there'd just be one way to say what we're

trying to do but because we're like somehow too low, then I can describe it this way and you can describe it this way. And they're not equal at that level, but if we picked a higher level, they could be equal. The idea of orthogonality in programming language design seems very salient here. The idea that the elements of your programming language design, like the grammar, the primitives, span, are like orthogonal bases of a vector space.

And you don't have any bases that are, say, almost parallel, because that would mean they didn't have quite the expressive power that they needed. And it's a little bit of a controversial idea, but it seems related. And the second thing that comes to mind is Krista Lopez and her work on exercises and style. It's fun. It's essentially expressing the same data munging operation, but in many, many different ways in Python. And she's fighting Cano's exercice en seal, something like that, from the French, where he rewrites the same scene a bunch of times in French.

That's a fun analog to look at. That's very funny. Yeah, the orthogonality and the basis vector thing is exactly, I hadn't thought of that, but now that you say it, it's very similar to what I'm talking about. That's very cool that that's a thing in programming languages already that people are talking about. Yeah.

I can't remember where I picked up the word. I can't remember reading about it in any paper or anything, but it's definitely a thing that people in the community discuss. Cool. Yeah. I have been struggling yet to find the right keywords for this. So thank you. I'll do some Google exploring and get back to you. Mm-hmm. I'll do some Google exploring and get back to you. Cool. The last thing that I think is kind of relevant to this audience is Distill, to me, seems like a really noble project. And hopefully it'll become more like the sort of thing that people respect. And basically, it seems like in science we reward people who discover things not people who explain things and um but it's it's wonderful when people

take the time to explain things so yeah discovery is fundamentally tied to explanation you can't yeah and and there is plenty of, like, you have to, when you discover something, or invent something, you do write a paper, but going the extra mile and spending so many hours,

like hundreds of hours, to go the extra mile and really convey what you're trying to say in a paper to people, I find that very admirable. Yeah. One pointer here is Chris's essay on research debt. It is a great primer on his thinking behind.

Yeah, I should have started there. Yeah. So, so just to back up, I feel like the story, Brett Victor wrote his essay on explorable explanations, and it feels like Chris kind of took up the mantle when he wrote about this. Chris Olak took up the mantle when he wrote about this research debt problem. And then Distill is kind of like the baby of Chris,

like trying to explain machine learning AI stuff to decrease the research debt so that, you know, we can, we can do better science faster here. That's it. Yeah. So anyways, that's, that's cool. Do you feel like in, in your own projects that don't have to do with AI, you will continue to do explorables about the things you work on? Does that make sense? Is that a relevant question?

I'm not sure if it's about the explorables. It's about the best, or not even the best. It's about representation, the edges of representation, and the best. Not the best. Why do I keep saying that? And just designing different kinds of mediums for concepts. If an explorable is the way to go, then I would do an explorable. If not, then I wouldn't. I don't think it's part of my platform per se.

Interesting. I feel like the way I think of explorables is that like in essay, just like a regular essay or regular paper is just, is an explorable that's just like bad or just, or just very flat. And then like anything you add to it is just making it more and more explorable. But, but I guess, yeah, I guess that that's not quite really what, what, what they,

how the words are used in practice. In practice, you can spot an explorable. It's some text and it's some widgets and papers. It's not quite as much of a spectrum as I thought it was. Right. Well, pragmatically, the thing about explorables is that readers don't want to explore and i think there's some stat from the new york times or something that's like for articles that had interactive components very the engagement for that component tended to be very low compared to when they just surface the insight and move on.

Yeah. Yeah, it's funny because in my own behavior, I've noticed that that's the case. I like the idea of them, but it doesn't actually make me an active reader. Yeah, I would say I'm more interested in a related idea. The idea of software as rhetoric and how an idea can become much more powerful when it's backed up by a prototype, a provocation, an implementation, a demo. And it doesn't have to be an explorable. Yeah, so software's rhetoric is something I'm quoting from the new inquiry. It's not a new idea. And they released this,

I'm not sure if they've done more with it, but they released this software called Bailblock, which if you install it, it sits on your machine and mines cryptocurrency to be used to bail out prisoners who are unfairly in prison. Wow. That's cool. I think that's very powerful. Well, so it's similar to, like, there have been a lot of projects over the last couple

of decades about using your extra compute time to like solve some scientific challenge like protein folding or whatever so i guess this is like a modern incarnation of that with the blockchain and then and it's like a very different social purpose yeah yeah right but it i believe was also accompanied by writing and the combination of the rhetoric and then immediately the real world impact of that rhetoric as embodied in code. It was very powerful. Yeah, that is cool. I'm trying to think of another example of rhetoric and code that went together like that. I would say building blocks is even more so an example of this because they released this big Colab notebook where you can immediately open it and repro the results and change it.

That's also code is rhetoric. Rhetoric is code. Yeah, interesting. As opposed to when you write a paper but you don't release the code at the same time? Yeah, I think artifacts can be

certainly part of it. Yeah. Cool. I guess, I guess it kind of talks about, it's almost like the power of demos are like pretty powerful. Like, I guess Brett Victor's work,

I guess would fall into this category. Right. Yeah. Michael and I had this, Michael Nielsen and I had this very interesting conversation about demos versus prototypes. Maybe it was demos versus products. Let me try and find that thread. So it was something like,

also it wasn't just the two of us. There are other people involved in this conversation, but it was something like trying to tease out when to do something like a sketch pad, so an influential demo, versus when to do something like a Photoshop, which is a widely used tool. And my reaction was, it's not a dichotomy.

One is often necessary to feed the other. This is someone else's example from the thread. There are just so many quotes in here that I can't figure out who was saying what. Or like Windows versus Mother of all demos, or the example that I gave was Pixar versus Computer Animated Hand. So like Computer Animated Hand was an influential demo from early computer graphics and Pixar is a billion dollar company built on widely used tools and those were actually done by the same person. Who is Ed Catmull?

I mean, obviously Pixar is not one person's work, but he's one of them. Yeah, yeah, yeah. He had a great book, by the way. I feel like you probably read it too. I haven't actually, but I think it might be on my bookshelf. Yeah, yeah. I'd recommend it. So anyways, I know you have to run soon so i i mostly just wanted to ask um about your speculative non-fiction newsletter just to you know describe what it is for people and then also um i like to give it at

the end relatedly i like to give at the end a um place for you to talk about you know where you could find your work online email email, Twitter, or whatever, whatever it is that you want to tell people about how to, what your interface is. Okay. Yeah. So speculative nonfiction is a letter for sure. And it's, I mean, it's only been around for a year so right now less than a year so it's actually the latest one was just this is a bunch of stuff i did because i didn't have time

to write anything new for it so it's a combination of this is a bunch of stuff i did please enjoy it An attempt to show you the unexpected futures next door. So to understand the relationship between technical production, cultural production, and design. I don't know. I don't know how to describe it. It's a letter. That's about it. Cool.

I agree. It is, at least I find it like hard to describe, but I also find it very enjoyable. Thanks. Great. Well, thanks again for taking the time. This was really fun talking to you. Yeah, this is a really great conversation and stay in touch.