Future of Coding - Ravi Chugh: Sketch-n-Sketch

Episode Date: March 9, 2021

Ravi Chugh is a (recently-tenured 🎉) prof at the University of Chicago. He’s famous for leading the Sketch-n-Sketch project, an output-directed, bidirectional programming tool that lets you seamlessly jump back and forth between coding and directly manipulating your program’s output. The tool gives you two different projected editing interfaces for the same underlying program, so that you can leverage the different strengths of each. In the interview we talk about the principles of bidirectional editing, the team and history behind the Sketch-n-Sketch project, benchmarks and values that can be used to assess these sorts of novel programming interfaces, possible future directions for Sketch-n-Sketch and the field more broadly, and a bunch more. It’s a long one — almost two and a half hours — but it’s packed with thought and charm. The transcript for this episode was sponsored by Repl.it. Show notes and the full transcript are available here: https://futureofcoding.org/episodes/049Support us on Patreon: https://www.patreon.com/futureofcodingSee omnystudio.com/listener for privacy information.

Welcome back to the future of coding. I'm Ivan Rees and my guest today is Ravi Chug. Ravi is an associate professor at the University of Chicago. He leads the group working on Sketch and Sketch, a programming environment that fuses direct manipulation with text-based code in a paradigm called bidirectional editing. You have a text editor on the left and a graphical canvas on the right. When you draw in the canvas, it generates code. When you edit the code, it updates the drawing. When you manipulate the drawing, it updates the code. You have two representations of the same information, both equal in importance, but different in how you can work with them. Sketch and Sketch is starting out life as an environment for working with SVG graphics and

HTML webpages, but there's a lot of promise to this idea expanding into other domains in the future. I'll let Ravi take it from here. What Sketch and Sketch looks like, what it currently looks like at least, is a pretty traditional two-pane interactive development environment where on the left half of the window, on the left pane is just an ordinary code editor, a text editor, really not much different from what you would normally expect. And the right side, the right pane, is a completely ordinary canvas

on which the output of the program is rendered. Currently, the Sketch & Sketch programming system, interactive programming system, is tailored to the domain of generating SVG graphics or also HTML pages. But so the program and the text editor on the left, at the end of the day, the main expression has type SVG or type HTML. And after the user runs that program,

on the left, the output value, the main expression, the main SVG or main HTML value is rendered on the right pane on the canvas graphically as it would in any other direct manipulation or drawing system. In that sense, it's a completely traditional, ordinary programming environment. You write a program, you run it, and it shows the output. What the main kind of new features that we're exploring revolve around are making the output of the program editable, manipulable. So as opposed to just being the final value that the program spits out and then it's kind of inert

or disconnected from the program that generated it, the main kind of goal in Sketch and Sketch is to allow the user to actually interact to change, to drag around things in the output to make changes, and for the system to infer changes to the program to match what interactions the user has performed. And so one way to kind of think about it, one way that I kind of like describing it is, in addition to the kind of normal forward evaluation process in any normal programming language, Sketch and Sketch is trying to provide this backwards connection, mapping changes back from the output, back to the program that generated it. Just to kind of talk about other kind of interaction modes, just a little

bit to kind of overview the system. So in addition to kind of being a normal programming language in the sense that you start with a program and run it, and then after you've viewed the output, you can start interacting with it, to try to also kind of provide this kind of backward connection in kind of a new and intuitive way. Another kind of feature that the system provides is the ability to actually add new things, new output values to the output of the program that were not even in the output before. So let's say you had a blank program to start, and initially there's actually nothing on your canvas. The Sketch and Sketch Editor also provides kind of a drawing toolbox of primitives

that you might expect in an SVG drawing tool where you can add new shapes to the canvas and the system will kind of auto-generate or insert definitions into your code, which again, if you would run them in the forward direction, would produce hopefully the kinds of values or shapes that you've just added in the output. And so the goal is to kind of allow both directions of both programmatically generating, but also kind of using direct manipulation, drawing tools to kind of move back and forth between these two directions of authoring. And in the in the strange loop talk that you gave back in 2016, you did a really good job of justifying why this tool needs to exist and why this bi-directional

editing needs to happen. And that is, in a normal drawing tool, when you make your drawing, if your drawing has something sort of systematic about it, like you give an example of a Ferris wheel with a number of cars sort of around a circle, if you want to change the number of cars, you have to manually go in and move everything around. Whereas if you're doing procedural graphics, it's very easy to say, oh, I have a number that represents, you know, how many cars are around the circle, but it's hard to do the sort of the quick tweaking and adjusting and getting things to look the way you want if you're having to use code because you have to make a change and then rerun it and a change and rerun it, or you're sort of like iterating your way towards what you want rather than just doing it. And so Sketch and Sketch lets you have to make a change and then rerun it a change and rerun it or you're sort of like um iterating your way towards what you want rather than just doing it and so sketch and sketch lets you have both it lets you do the direct drawing to get the image that you want but the

act of drawing that you go through creates code that is generating the drawing and so you get the benefit of of it being a procedural drawing without having to sacrifice the graphical way of working. That's right. And that is the goal to kind of, you know, mix the best of both interaction modes. Because clearly, you know, direct manipulation, drawing, direct interaction, specifying changes, you know, directly in the context of what you're building is obviously what you want in many kind of situations. But then there's other times when expressing some really fine-grained, repetitive parameters of your design are much easier to specify if you had a little bit of support for programming and procedural abstraction. And so I think this is, you know, a very natural tension that comes up, you know, not just in, you know, in drawing editors or kind of creative media applications. I think this tension really comes up pretty much in any kind of application software

that we use, right? So if we're doing, you know, traditional office kinds of tasks where we're, you know, writing mostly text in a text editor or a word processor, if we're doing data manipulation in a spreadsheet with some visualization at the end, if we're developing a website, if we're doing 3D animation, I think, you know, in all these domains, there are obviously really terrific, really rich, really sophisticated GUI editors that allow you to do tons and tons of useful operations in those domains of work, but oftentimes they don't provide an escape hatch to a more procedural or programmatic view for doing things that are just not necessarily the best to do inside the GUI itself. And so I think this tension between trying to mix the best of both direct manipulation GUI applications with general purpose programming, you know, I think is a very natural combination that, you know, I think ought to, to, to be useful in, you know, a wide variety of domains this generalizes to other domains. But to start with, Sketch and Sketch is focused on graphics. And so

in this current prototype that you have been iterating on over the last, what's it been, six years now that you've been working on this? It's been about almost five years. Oh, okay. Yeah, it was kind of early 2015 where we really started to kind of think about what it would take to achieve this kind of combination in the setting of graphics. And yeah, I think we started writing code in April of 2015, so coming on about five years. And so in the current prototype that you've built up over that time, are you thinking of it more as a graphics tool or more as a programming tool or or if you put those on kind of either end of the spectrum maybe where's the spot along that spectrum that that uh you think of sketch and sketch is kind of fitting in yeah

it's a good question and i don't think we've really planted our flag or tried to plant our flag on either end of the spectrum, really. I don't think we're trying to, at least in the short term, make this the end-all of drawing tools. And certainly this is not the end-all of programming systems because, again, it's really designed for very small-scale programming at this point with a very narrow application domain. Really, we've been using this initial prototype and this initial application domain as kind of a, you know, a laboratory, a playground to try to study the simplest and the kind of first questions, the first challenges in this goal of connecting programs with the output that they generate and allowing this kind of bi-directional

back and forth connection. Because, you know, as we were kind of talking about, this kind of connection seems like it ought to be useful in a wide variety of domains. And so what that means is, you know, for these kinds of techniques to be, you know, really general purpose and reusable, there ought to be kind of a core foundation upon which, you know, this basic connection between programs and their output ought to be studied. And so the way the approach that we take is by picking just kind of an ordinary traditional programming language, happens to be a functional language, it kind of looks like Elm right now. But in terms of like the core ideas, we're basically studying these ideas in

terms of like the Lambda calculus and very kind of core foundations of pretty much any programming language. And we're really studying like, what does it mean to change the output value of a Lambda of a program in the Lambda calculus or in Elm? And, you know, how do we translate changes to the output back to changes in the program? And so in that sense, you know, it's general in the sense that, you know, SVG and graphics didn't factor into that description at all. But then when we, you know, take the algorithms that we're designing and implement them in a particular system, then we need to also think about, okay, what is the user interface for actually making those changes to the output? And in situations where the algorithms provide multiple options

or need more help and ask the user to kind of choose, that's where we have to really think, okay, for this specific application domain, vector graphics, what is the user interface that we draw? And how do we surface the results of the algorithms? And so in that sense, it's kind of saying, okay, well, how do we get the UI to look more like a traditional drawing tool? And so I would say that currently the prototype is kind of like this mix of very, very, very

bare-bones drawing tool and very, very, very bare-bones programming language so that we can kind of explore these initial questions as soon as possible. But then in the longer term, you know, I would think that, you know, if the techniques continue to kind of improve and scale to more and more sophisticated classes of programming tasks and design tasks, you know, I don't see why we wouldn't want to also try to make this, you know, a really full-featured drawing tool that has programming support, and also the opposite of the spectrum, why we wouldn't try to take these ideas and techniques and embed them into traditional general-purpose programming IDEs.

So that if you were working on a different domain other than graphics, you could still have the benefit of bidirectional editing and output directed programming but then also on the other side if you're doing graphics you get the benefit of having a code representation of your drawing extracted out so that you can drop down and work on it on that level if that affords you some extra ability exactly Exactly, that's right. Cool, so you are thinking about sort of growing in both directions, not just using graphics as a test case

and then going back to the programming side and just focusing on that. You're also thinking about going at it the other way as well. Yeah, absolutely. And that's one of the really fun parts of this project, I think. It's got this really wide range of techniques and challenges that all need to be solved in order to really provide this long-term goal of mixing in a fine-grained way the best of what GUIs and direct manipulation offers and what the best of what programming offers. You know, if one wants to try to combine those in a fine-grained way, that's going to require both advances in programming languages and program synthesis.

It's going to require advances in, you know, UI design and taking into account this kind of interactive conversation between the programmer and the, you know, what the system can do. And so if you look at the kind of, you know, work that we've been, the directions that we've been exploring the past, you know, few years, it's a variety of what you could think of as kind of core PL program synthesis kinds of research questions, as well as kind of core user interface questions. And so, we're certainly excited to kind of keep pushing on directions and, you know, and all these fronts, because I think it's, you know, it's pretty clear that one needs to, you know, think about all these kind of in tandem and not just, you know, from the programmer's point of view or from the, the designer, the end user's

point of view. Before you started working on Sketch and Sketch, what did you see in the world that led you down this path? Like, did you play with an existing tool and think, I know how to make this better? Were you chasing a feeling? How did you arrive at the desire to work on this? Yeah, it's a fun question for me to think about because I do kind of think about the original motivations for this project quite often. And I think the moments that I look back to are pretty specifically centered around my time in graduate school, where I was doing research on program analysis and type systems, something really not super related to what I'm working on now. But in the process of grad school, you know, I give, I would give conference talks, and I would give lectures and things like that. And so I would always use PowerPoint and happily

use PowerPoint to create, you know, visual interactive presentations about whatever topics I was talking about. And, you know, I, I'm certainly not, you know, qualified in graphic design or visual arts, but I always enjoyed the kind of challenge of, you know, coming up with really interactive visual ways to kind of explain the, you know, the ideas about whatever, you know, research topic I was talking about. And so I always really enjoyed how, you know, I could do that in PowerPoint because I could really easily try out a bunch of different visual representations and, you know and the built-in animations often provided kind of a good way to stage transitions and sequence the story of my talk. And so I always really enjoyed PowerPoint.

But every time I would use it, I would also think, okay, there are a whole bunch of operations that are really tedious to perform. And once I've kind of figured out what the basic design of my talk and my visual motif is looking like, it would be much easier if I could make certain changes programmatically instead. If instead of, you know, making a change early on in my, let's say, sequence of slides, if I make some fundamental change to the visual motif, instead of having to go back through all the 10, 20, 30, 40 places I copied and pasted and made changes, if I could instead go into a, you know, programmatic

representation of that sequence and make changes in one or two places, it would be so much easier to kind of, you know, build these really complex visual narratives and presentations. And so that desire really came up over and over again in grad school. And I remember talking with my friends at the time about wouldn't it be great if, you know, PowerPoint was completely programmable, if under the hood, there was a general purpose programming language that I could use to help interactively build these presentations. And so I think, you know, that was that, that, that was kind of, I think, the first time that I really thought about this, you know, wanting this combination of, you know, as a programmer, knowing that general purpose programming languages provide all these

abstraction capabilities that make certain things easy, but then also, you know, using terrific direct manipulation tools and realizing how they made kind of a, you know, different set of interactions very easy. And so that's kind of what I look back to as one of the original kind of motivations for wanting this combination. But, you know, I think the same combination comes up in pretty much any other, you know, GUI application software that I use. And I think probably many other people who are programmers, you know, feel the same way. Wouldn't it be great if you could also get some of the kind of abstraction capabilities that programming provides? It's interesting that you, how you sort of, I imagine you conceptualize yourself as a programmer, not an artist, based on what you said about, you know, you sort of qualified your art chops there. I should probably have qualified my programming chops too

i think i'm equally unqualified yeah but it's interesting to me that that you came at this from the angle of wanting to add programming into an experience that wasn't rooted in programming and i've had the exact same experience using um programming tools and very much wanting to add art capabilities into them and so it's it's neat to me to sort of um to think about that there's you know there are many different roads that lead to this intersection you know especially early on when i was kind of thinking about okay what are the kind of like techniques that under the hood would need to be developed in order to kind of support this combination? And I think, you know, there's ways to kind of think about this as like a programming problem and then

adding, you know, support for, you know, direct manipulation and GUIs to it. And then the other approach is, you know, start with a GUI and try to make it more programmable. And so it's funny how early on, especially, and actually even today, you know, if I'm describing, you know, this project and the kind of ideas that we're working on to different people, I'll often kind of, you know, either pick the programming side of the world and start there and kind of describe how it could be connected to the kind of interactive, you know, design and GUI kind of perspective, or whether to start from the opposite side. But as we talked about earlier, I think really, you know, you know, if these kinds of ideas and techniques turn out to be successful and scalable, the goal is to really kind of bridge this kind of gap so that there really is no divide about whether, you know, you're starting with programming and making it more interactive, or

you're starting with a GUI and making it more programmable. The goal is really, you know, for the foundation to be a general purpose, you know, expressive programming system where you can define these kind of composable GUIs for different tasks. And you can really kind of mix and match, you know, GUIs for common kind of like patterns of use, but then also compose them with others and design your own and really kind of eliminate this kind of spectrum of, you know, this system is for programmers or this system is for end users. And on that train of thought, in the strange loop talk, you said that Sketch and Sketch is fundamentally a coding system with a graphics editor added on top, and that there are shortcomings

of systems that start with direct manipulation and then add code generation. And I'll quote you here, you said, no matter how many built-in tools come with a direct manipulation and then add code generation. And I'll quote you here. You said, no matter how many built-in tools come with a direct manipulation system, there are always going to be limits to what the tool can do well automatically. Can you expand on that thought? Yeah. So I think the idea is that, you know, what a user interface provides is a bunch of tools, a bunch of building blocks, and features for combining them, certain kinds of features for organizing them maybe into layers and applying certain operations. But let's say you, the author of some artifact, the user of one of these systems, want to add some new kind of capability

to the system. Right now, there really isn't kind of an easy way to do that. So certainly, you know, many GUI application tools, you know, have plugin architectures and things like that, where an expert programmer could go build some plugin that extends the UI in some specific way. But that, you know, ability to extend the UI with new primitives, I think, ought to be much more a part of the main process of using the tool, as opposed to something that you, the author, would have to employ some expert developer to then build a new tool for you. So some of the examples that I'd like to use, you know, in the strange loop talk in the kind of recent papers that we had at Wist are, think about even the basic kind of drawing tools in your drawing editor, drawing a shape,

drawing a circle, drawing a polygon, even those, you know, you might want to define variations of those kinds of primitives and have them be the kind of tools that are presented in the toolbox. Like what makes those, you know, definitions of shape and polygon the only way that you can imagine a rectangle tool working? There's other variations where maybe, you know, certain properties of the rectangle would be different from the defaults that are provided for you. And so if even these primitive, if these primitive tools are instead defined as, you know, let's say functions in some programming language, then these tools can actually be, you know, libraries that are provided to users, but could be swapped in for other libraries for, you know, use cases that might not be the kind of use case

for every single user of the tool. So let's say I'm in some project where the primitives that I want to use actually have these, I don't know, seven-sided stars and snowmen or something. If the GUI system allows these primitive tools to be defined by user or library-defined functions, if I'm a programmer or if I'm working with my team of designers, I can define these library functions for seven-sided stars and snowman shapes and add those into

the toolbox so that we can use those as direct manipulation drawing tools, as if they were the built-ins that have been provided for us. Clearly, a user interface can't provide every single tool that every user might possibly want. And so at some point, a user interface and application software will have more and more tools, more and more menu items, more and more options. That's at the same time not going to cover every single thing that users might want to do. And it's not the most scalable way for a user to use a system because sometimes you might really only want five of the 15,000 features that are there. And so it seems like you want to be able to make user interfaces much more customizable and much more extensible so that different users and use cases can customize the UI to do different things.

So in the Strange Loop talk, one of the demos you give is that you build the Lambda logo for Sketch and Sketch. Used a handful of geometric attributes that were exposed by the graphics editor like the positions of points the widths of rectangles that sort of thing and then more recently in the whist talk it looks like you've gone much further down the path of exposing geometric attributes like the midpoints of lines that you can use to you know snap another line onto these additional attributes look like they make it easier to build complex relationships between the shapes. But at the end of the talk, Brian references this book about programming by demonstration. And in that book, there are a number of benchmarks that you can use to evaluate

a programming by demonstration system. And when he mentioned those benchmarks, he said, you know, there's four of them that Sketch and Sketch can currently do perfectly. There's two that it can kind of do. And then there's nine more that it can't really do right now. And in order to do some of those nine, a number of features need to be added. One of those features being like attaching the end point of one line to an arbitrary position along another line. That sort of feature addition feels kind of like a game of cat and mouse to me. And it feels different from what you've just talked about, about adding

different kinds of preset shapes and that sort of thing. This feels more like needing to change the underlying representation of the graphics or changing the way that the graphics map to the abstractions. So is that something that you feel like you're working towards making that open-ended as part of the system? Or is that something that's going to be sort of baked in and the end user is not going to be able to add those sorts of additional capabilities to your vector representation? That's a really good question. And the short answer is yes, I think we can and will want to kind of make those choices and those kinds of features also exposed to users or library or tool builders to kind of customize. And so if you think about like, even the kind of simplest

widgets that you might draw into a rectangle, let's say. So, you know, the completely standard features that you might draw and allow the user to manipulate are the corners of that shape and maybe the midpoints in the center. And so it is the case that currently our editor, Sketch and Sketch, draws predefined sets of features or predefined widgets for the different kinds of shapes. But you could certainly imagine even those widgets, even those choices to be defined in a library instead, right? So you could imagine there being a library function that describes what to draw on top of the primitive, unadorned, undecorated rectangle in the SVG output. And that library function could choose to draw SVG circles or SVG rectangles or whatever it is

that happen to be exactly at the corners and midpoints and centers of those shapes. And so you could imagine then, let's say, choosing a layer, choosing kind of a wrapper around rectangles that don't draw any widgets at all. Because let's say you know that certain rectangles in your output are never going to be interacted with. And so why ever even have the user interface clutter that view with extra widgets? And then you can imagine in some other part of the design, you have more kind of knowledge about you're going to want to be interacting with maybe not even the midpoints, but maybe an arbitrary point on the edge. You can imagine overlaying the right widgets on the edges of those polygons or on the edges of those shapes and then kind of hook those up to what the, you know, the algorithms under the hood that

connect the output value to the program know about. And so, you know, I think there's certainly details about, you know, kind of connecting what the user-defined functions have chosen to draw on top of the real, the main values, and how to map those kind of interactions to what the underlying program synthesis, program repair algorithms can do. And so I think there is a little bit of kind of, you know, extra metadata and other kinds of things that you'd have to define there. But I certainly think that this approach would allow, you know, those kind of user interface widgets and actions to certainly be customizable and changed by users and libraries.

So to build something like that, it feels sort of like you'd need to get at the core ideas of whatever your output is, like the fundamental first principles. So in the case of vector graphics, it might be you have to distill everything down to like a point and then the idea that points are connected in lines and then sort of, you know, bootstrap your way up to all of vector graphics from something like some fundamental seed sort of like that. Does that seem like a fair characterization of what would be needed in order to make that idea work? And if so, how would you apply that to other domains? That's a great point. And I do agree that that's kind of a fair way to describe it. And so I think I did, I think I mentioned this in the

strange loop talk as well. So I think the way that we, we see this as, you know, I said that, you know, we want this kind of bidirectional connection between programs and their outputs to work in many domains. And to kind of do that in a scalable way, it seems that there are going to be certain kinds of operations, certain kind of connections, certain kind of changes that ought to be common across whatever application domain you happen to be working on. But then also for any specific application domain, there's going to be custom program analysis, program repair techniques that cater to the kinds certain tools that are only, that are useful to, that are, you know, specifically designed for a certain domain. And so, like you said, you know, for SVG, if one wanted to expose like a completely configurable, reconfigurable, you know, UI for doing vector graphics, you might

want some really, really kind of general representation-like points, like you said, on top of which you could then build, you know, basic structured shapes. But then you would have really the kind of like finest grain access to be able to specify, you know, constraints over individual points, like in your output. So I think it is kind of fair to say that, you know, for each, let's say, different type of value in your application domain, or I guess for each application domain, you identify the primitive values in that domain, and that kind of defines what the users and libraries can operate over. Currently, for SVG, we choose just the normal SVG primitives as our, um, like from the spec. Yeah, exactly. From, you know, the spec is our domain of values. If one wanted to expose like really complex kind of constraints over individual, you know, sub features of these

shapes, you might have something like you described. And that, that, um, disentangling of the things that are common across domains from the things that are unique to each domain and coming up with the level of uh like um sdf or the semantic web or something like category theory or any of these notational systems that are designed to sort of separate out uh the structure of what constitutes a domain as opposed to the instances of things that have those structures like that sounds like a like a really really really big problem have you made a sort of a beachhead on that part of the problem yet uh when building the core of sketch and sketch yeah so certainly like in our current work there are certain things that seem clearly independent of domain.

And so, for example, our programming language really knows nothing about any specific application domain. It's got a built-in set of types as usual. User can define new types. And then you can program with whatever types of values you're working with, at the end of the day, the kind of connection between the program and the specific domain is, you know, the main definition, the main expression that your program computes. And that's the time at which the editor starts to need to translate that, you know, main expression, that main value into the specific domain, in this case, SVG. And that's the point at which the kind of output value of the program starts to be interpreted in terms of domain-specific primitives. And so in the kind of like for the evaluation of

the program, there's really not much specific to application domain going on. But in fact, the execution trace of the program is being recorded so that, you know, within the final output value, different pieces of the output value are kind of tied to different expressions and different intermediate computations. And so when the user interface, you know, displays the output values and intermediate computations as SVG widgets, then each individual tool that operates over SVG gets to kind of look at the evaluation of the program and decide how it's going to make changes to the program or not. And so I guess the kind of way that I describe it is, you know, a lot of the programming and the kind of tracing during the evaluation is domain independent. But then when you,

you know, want to kind of convert the output into something that is drawn on the screen, and then when interactions with the output are transformed by domain-specific transformations, that's when you take the kind of general purpose information about the program and then decide how to use it. Have you thus far in the project done any work on how those transformations from the output domain back to the code and from

the code to the output domain how those are specified like is that something that's data driven right now or is that something is that a part of the project that you're leaving for a later stage so currently the the transformations that Sketch and Sketch can make based on interactions with the output are each coded as arbitrary AST transformations. And so a transformation in Sketch and Sketch gets to look at the original program, gets to look at the final output of the program,

gets to look at the evaluation trace that produced the output, and it gets to look at some set of widgets or interactions that the user has made on the output. But given that knowledge, the transformation gets to do whatever arbitrary AST change it wants to make. There currently isn't a kind of higher level way of describing desired changes to the program based on desired output interactions. And so currently, each one of these transformations is its own standalone transformation. And so, you know, sometimes they don't compose as well or as naturally as you might want them to. And so in the future, we certainly want to define some kind of higher level specification language for defining new transformations. So that let's say, you know, even within the domain of SVG, let's say I am a tool builder and I've got some new transformation I want to build in. I'd like to be able to kind of maybe provide examples of, you know, when the user performs this action and the code matches the structure, transform

the program in this way, so that not every detail about the program execution trace and fiddly details about the lexical structure of the program and the program dependence graph, so that not all those details have to be manually accounted for in the implementation of the tool. But we certainly haven't done any work on making these kind of higher level DSLs for defining new transformations. We haven't done work on that direction yet. Just for my own personal curiosity, I don't know if I'll actually include this in the show or not. When i'm imagining this architecture that you've built i'm thinking of it almost like a bi-directional multi-pass compiler kind of like llvm or something like

that where you can insert your own um uh compiler stages or your own optimization stages into this modular compiler architecture is that a little bit like how these transformations work where you're doing the code representation is like your source code and you're compiling it into this output result. And then that output result is also preserves a structure so that you have another sort of reverse compiler that takes that back to the code representation.

And so the domain mappings that need to be created are very much like compiler stages where they're doing a sort of like a tree to tree transformation or something like that. Yeah, so I'm not sure whether there's many more stages in which one might want to insert, let's say, a new representation or kind of a custom transformation. I think it really is kind of like these two phases, forward evaluation and then in some sense, backward evaluation. And at least in all the kinds of features and transformations and interactions that we've wanted to provide so far, we can basically collect all the information that we want or need to know based on essentially an ordinary forward evaluation where we're tracing dependencies

between expressions and the values that they compute. And so the kind of tracing mechanisms that we've been using are pretty standard mechanisms that are used for things like omniscient debuggers, where you want kind of explanations of how this value was computed, pretty common to how provenance tracking analysis works, where you want to, again, know where did this value come from. And so essentially, like the evaluation of the program has all the information that currently an end user or a final program transformation might want to refer to. In many cases, it only needs kind of a small subset of this evaluation trace. But we so far haven't thought of any other intermediate stages or any

other intermediate representations that we might need to kind of expose for customization sure i i just meant more in terms of of conceptualizing what it's like to write one of those translation layers because as a you know as an outsider to this project knowing that the or one of the ultimate goals is to take it to the point that it can be applied to arbitrary domains, not just vector graphics. I'm here thinking about, let's say I took your core of your system and I wanted to extend it to a new domain. What would it actually feel like to write that domain transformation stage? Would it feel like writing a compiler pass or would it feel like something else? Actually, that's a good question. It makes me realize that. So earlier I described the forward evaluation of programs as a domain independent tracing mechanism where we kind

of record evaluation as usual, and then domain-specific transformations get to look at that trace information when deciding how to transform programs. But actually, there are a couple of situations in which our tracing mechanism is doing SVG, domain-specific tracing. And so one of the ideas in the most recent WIST paper is to expose user interface widgets for manipulation on not just the final output and sub-values of the final output, but also on some of the intermediate computations that didn't necessarily draw something in the final output. And so an example of that is in the current version of Sketch and Sketch, the evaluation of the program looks for expressions of type point. And so even

if an expression that evaluates to a point, even if that point isn't in the final output of the program, that point will be shown on the output pane as something that you can interact with and snap things to and use as guides as kind of a helper widget for whatever happens to be in the final output. And so that's an example of how our forward evaluator actually is looking for, in this case, things of type point and recording some trace information about it in a domain-specific way. And so to do this for other domains, one might expose maybe the evaluator of the programming language to the tool builder to provide places where they can, you know, hook into different parts of the evaluation behavior and instrument it with additional information, additional

things to log that might be of interest to the downstream transformations. And so I think, you know, like you described, it could be kind of, you know, like what a compiler writer might provide when implementing a new, you know, transformation, like you described, it could be kind of an API where the system provides the tool builder, something that looks like the intermediate representation that is produced during evaluation, and the opportunity to kind of change it or or instrument it in some way the talks that i've seen so far the demos have focused on the direct manipulation the gui workflows but not on the generic provenance tracing or the other parts of the engine presumably my guess is that they don't demo very well. To me, those internal

bits seem like they're the most interesting. And I feel like you feel similarly. So do you feel that way? Do you feel like those internal bits are, are the really interesting parts or what, what parts of the project make you think, yeah, that's, that's the good stuff right so i think you know i think the kind of program analysis program synthesis bi-directional programming kind of techniques under the hood are certainly um you know really important obviously uh critical parts of enabling the user interactions and you know when we have uh those techniques that are worked out clearly and cleanly enough for, like we talked about, a very core traditional programming language where it would be easy to imagine how to incorporate that algorithm into any other programming setting, those are the kinds of techniques that we'll publish as a PL paper. But yeah, those aren't the kinds of, you know, things that on their own can be easily demoed. Those are the kinds of things that, you know, are more easily kind of described in terms of like, you know, the math that describes the algorithm and comparing what that algorithm

does compared to others. And that's certainly very cool part of the research and very fun to work on. But then I also think that, you know, then figuring out how to deploy those algorithms into a, you know, at least even a prototype user interface that starts to kind of show what those algorithms enable for certain simple kinds of authoring interactions. Those are also really, you know, I think fun kind of milestones for us as well. And so like in the WIST paper, the idea there is to really kind of see how much we can do with the kind of milestones for us as well. And so like in the WIST paper, the idea there is to really kind of see how much we can do with the kind of techniques that we've built under the hood. And I think, you know, we're, I certainly enjoy when we were able to kind of, you know, hopefully

demonstrate some milestones on both of these kinds of fronts. What are the kind of technical bits under the hood that enable us to then build some kind of system that is starting to provide interesting uh user interactions on top but you're right in the demos you know at least uh it's it's harder to kind of explain the the algorithms that are under the hood you know it's's less enjoyable to watch. Let me rephrase that. It's more enjoyable to watch somebody demo a really interesting about the technology that's sitting underneath it. Whereas, like you said, a paper is probably the right place to get into the nitty gritty of the internals. Right. Or, or I would say, uh, well-crafted slide decks, I think can also be kind of an exciting and interesting way to kind of convey even the technical material as well. And so, you know, I think certainly, you know, in the research

community, I think, you know, lots of people try to spend lots of time to, you know, give presentations that aren't just, you know, summaries and snippets from, you know, figures in a paper. I think oftentimes you, you know, try to figure out, okay, given that you've got 10 minutes to try to convey, you know, the highest level insights about what the new math and the new algorithm is doing, it takes a different vocabulary, both in terms of words, but also in terms of visuals, right? Coming up with some visual notation, some visual motifs, patterns that you can use to kind of explain some of the insights behind the kind of technical ideas to folks that aren't necessarily going to

spend, you know, an hour or two or three or four reading the nitty gritty full details. I think it's also kind of a fun and interesting challenge to try to develop visual representations of complex, you know, mathematical ideas, kind of like I was describing early on with my initial kind of motivation for a tool like this, a system like this, where in grad school, I would spend a lot of time trying to come up with really nice done well, I think it is kind of an interesting combination of, you know, visual representation of more kind of mathematical and complex ideas. And I think those, you know, the most effective talks like that, I think are really, really hard to do, really take really a lot of work to design well. And do you feel like you've done any talks or made any presentations of your research whether that's in a paper or you've done some really nice posters for various

stages of the project also is there anything in that body of work that you would want to point the listeners to as an example of somewhere where you guys put in a lot of effort to come up with those sorts of visual representations of the nuanced ideas that they might want to go and look at? Yeah, so on the kind of presentations on the talks part of my webpage, we have a bunch of presentations that we've given at various conferences and seminars and things like that. And a few months ago, I gave a kind of a basically a summary know summarize all the different you know main give summaries of the main technical ideas under the hood and it's funny like that slide deck that um that i put together actually has been building up over years and years i don't think i've actually started with a new uh slide deck in like five years or six years i think each time it's kind

of like building on the styles and building on the choices that we've made before about, you know, different colors and shapes to use for this concept and different code examples with, you know, different highlighting motifs and things like that. And so it's funny, it's kind of very full circle. It's, you know, the kind of effort that would be really, really painful to kind of go back and create from scratch or to make changes to because so many choices have literally been copied and pasted over into new uh you know modules of this slide deck for many many years and so it's certainly the kind of thing that like i would hope you know these bi-directional tools could make much more pleasant and much more uh effective ways of building these kinds of presentations in the future.

Currently, I just still use PowerPoint. This talk that I gave a few months ago is covering, I would say, four different research directions that we've been pursuing under the hood to kind of enable these kinds of uh bimodal editors and you know that might be a good way to get kind of a cartoon understanding of some of the the main ideas under the hood and uh i'll include a link to that in the show notes um yeah and it's funny this is like exactly the kind of um artifact that i would really hope to produce in a system that really kind of you know allowed programming but yeah i guess i already said this so maybe i'll say it again yeah the sketch and sketch demos that i've watched show a specific workflow with direct manipulation at each step and that workflow is first you draw, and then you add relationships

between the things you've drawn. And then you do graphical grouping, which serves as an abstraction operation, where you extract a function that creates the grouped drawing that you made. And then you tweak your drawing using that abstraction each of those steps creating or refactoring the code is that workflow sort of a gilded path or are there other ways to work within the tool like importing an existing graphic or starting with existing code yeah so a lot of those kind of example demonstrations are, you know, take that workflow to kind of see what can be done, how expressive a design can be built using just, you know, direct manipulation and not also interleaving text-based edits or programmatic edits. But you're right, of course, an authoring workflow might really mix and match these kinds of two modes of use much more freely. Currently, we don't, for the question about importing existing graphics, currently we don't have any tools that we've implemented to make that process easy. You could imagine taking an existing, let's say, SVG definition and in the limit,

just kind of like inlining that literal into your program. But even better would be to try to automatically identify numbers and properties that appear over and over again and suggest those as variables and maybe even function boundaries if there are repetitive patterns in the imported file. You can imagine doing that, but we haven't spent any effort on that yet. But then more directly about this kind of gilded path through the tool. So in an initial version of Sketch and Sketch, in the version that we demoed at Strangeloop, there were many kind of requirements about the syntactic structure of the program that if they weren't satisfied,

certain interactions in the output would no longer be available. So a simple example is, in that initial milestone, the main expression, the main definition of the program, essentially had to be a list literal of shapes, and each of those shapes had to be a top-level definition in your program. And only then could certain interactions be available to users. And so a lot of the work that Brian has done recently has been to relax those kinds of restrictions. And so by doing more general purpose tracing of the program to support more arbitrary programs in the language, while still retaining the connection that, oh, this value in the output came from certain

locations in the program, that's kind of supported in a much more general way. But there are certain times when, let's say you want to create some parameterized function that repeats some design, there are certain times you can actually take multiple paths through the tool. So for instance, you can copy and paste a shape or a design multiple times and then use a tool called merge that looks for syntactic differences in the definitions that generated them and will take the differences between those programs and turn those into arguments to a function. Or you could say, well, given just a single shape or group, you can use a tool called abstract, which turns it into a function like we just described. So that's an example where there's kind of two different ways you could choose to build one of these parameterized drawings.

But there's other times when you kind of are forced to make a choice about, you know, what constraints to add into your program, and then you can't kind of undo that choice later on. And so there certainly are many times where you do have to kind of pick the right path through the current set of tools. That certainly needs to be, you know, addressed in the future where you want to allow maybe multiple kind of choices to be propagated downstream so that later on when you make some subsequent action, maybe then is the right time to decide whether the structure of your program should be one way or another. Currently, you know, there are times at which you have to make a choice that you can't kind of undo or revert later on. It sounds like you, since you're not storing an edit history on the

graphical side, right? Like every change on the graphical side is immediately propagated back to the code. That's right. That's right. So after every interaction, the transformation changes the code and doesn't store the edits that led up to it. And so reasoning about the edits that are being made in the output editor could certainly be a rich source of information for helping to decide or understand what changes to make to the program. And then also keeping track of the history of program edits as well could certainly provide new ways of trying to infer what the user is intending yeah because if you provide a lot of different ways to achieve the same result

it could mean that the structure of the code that you end up with is different depending on which way you went about achieving that result and it's it makes me have uh sort of flashbacks to for instance like in gmail when you're trying to do um whizzy wig editing or not even gmail slack's recent uh text editor change to a whizzy wig editor had this problem a lot of whizzy wig text editors have the problem where when you try to add formatting, there's a representation that is invisible behind the scenes. And you can end up with things like, you know, here's a spot that has, you know, it's surrounded by white space. But if I position my caret in that spot and type, it's bold, even though the text on either side is not bold, because there's an empty bold node at that spot um those

sorts of things and so it feels like like this is a place where i could imagine it being kind of of tricky to get the right balance so that um and i suppose showing the code and if people who are using the tool are expected to be familiar with the the code representation that that gives you a lot of a lot of benefit because then people can see oh when i when i repeat a shape using this approach it creates that change in the code whereas when i you know repeat a shape using that approach it makes a different change and so that sort of alleviates you from the burden of having to use something like fancy edit history tracking as a you know like a crdt or something like that where you try to merge it down to a canonical

representation no matter how you got there it alleviates you from the burden of having to make sure that different changes in the output result in the same change in the code that since people can see the code, it's sort of, you know, you've moved the burden of correctness and consistency over to the user. And I think that that's a good thing. Like, I think that that's, you know, you're, you're giving them leverage rather than foisting complexity on them. Does that feel right to you? I guess I would say yes and no. I guess so certainly, I think having the program, having a program in a general purpose programming language be the ground truth, be the artifact that matters, I think that's a good choice. I think that is maybe not in the long term the best, but I think it's a very good medium in which to at least allow users, and especially expert users, to make this specific choice about what the representation should be. But programs, there are often times where you

write something one way, but you might want to express it in a different way instead. So a very simple example for this specific domain is, let's say I'm writing a program that generates a rectangle. Oftentimes, you'll decide whether the parameterization for this rectangle should include the location, the point of the top left corner, and the width and height of the rectangle. Or sometimes you might decide the parameterization should be the top left corner and the bottom right corner, in which case the width and height would be derived in terms of those two points. And so there are times in which you might prefer the former parameterization. There are times in which you might prefer the latter. But with a program, you have to pick one, right? And then when you've made that choice, all the subsequent code that depends on it, you know, it's not very easy to

change if you want to go back and change the structure of the initial parameterization. And so, you know, I think, you know, exposing a general purpose program is a good way of making explicit exactly what the artifact is, what the representation is. But again, there's times when programs force you to make these kinds of choices that you would ideally like to have even of more, even more kind of deferred control over. So, you know, there's other intermediate representations you can imagine with like program dependence graphs and other kinds of computation and only have to turn it into a program, like turn it into an abstract syntax tree when, uh, when, when, when it makes sense.

Like almost as a rendering step or something like that like you know export code or something right and then let's say that you want to kind of move back to the more general purpose like bag of constraints bag of computations um do more interactions and then at some point want to say okay at this point i know that i want to do some repetitive operation over my data structure, codify it as some AST that looks like normal data structure that I can map and fold over so that I can perform some actions, but then go back to this more general representation of the computation for subsequent edits.

So about that, at the moment in the user interface even though the in the wist demo brian um the the point of the demo was we can do everything that you're about to see entirely in the in the graphical side without touching the code here's you know we pushed ourselves as far as we could to do things with direct manipulation of the output. I noticed in that version of Sketch and Sketch, there's still a run button for the code. And that was something that I saw in the Strange Loop demo earlier on, was there's a run code button

and you have to click that button to update the output representation. And sort of further to what we were just talking about about maybe inverting that where you're you're working directly with the output and it's keeping some sort of bag of constraints or something as a live representation behind the scenes and then only as needed do you render that to code that can be edited how do you how did you arrive at that need for a an explicit run code button and do you see yourself getting to the point where that goes away or where that's inverted or or in terms of that specific slice of the prototype like what are

your thoughts and feelings about it yes i guess there are two things that i would say so so certainly like having a traditional run button was just kind of a simple, lazy choice. There's no reason why it didn't just implement some kind of automatic parse and rerun feature like you would expect in a live programming environment, or going forward, building the kind of programming with holes work that Cyrus and the Hazel project are exploring. There's no reason why we wouldn't want that as well. And so that's just an orthogonal feature that we just haven't worked on or polished on at all. That's not really an explicit choice. But to your second question about

what about not even showing the code until you really need to or want to for some reason, that's something that's also certainly something that we can imagine doing in the future where while you're performing doing in the future where, yeah, while you're, you know, performing interactions through the kind of visual editor, maybe you need to be kind of shown and told and interact and explain, like, what's going on under the hood at every step, or maybe not. Maybe you do have these kinds of choices building up as constraints behind the hood, and only when the user knows that they need to do something with finer gain control that doesn't have

a mechanism in the visual editor could the system then say, okay, based on the program that the user previously saw, and the changes that have been made since, and the interaction that they want to do next, show them, you know, code in a way that makes the most sense for that next step and maybe only the parts of the code they need to know about. You can certainly imagine wanting to do this more selective and dynamic code generation as part of the user experience. Currently, we've just, you know, again, to kind of keep things as simple as possible, as we're trying to push on the expressiveness of

each of these operations is we have to, we choose to make, you know, the changes of the code at every step and display them. But there's no reason that that's you know the the right uh desirable user interface for for all use cases and that reminds me a bit of sort of the first place i feel like i encountered anything remotely like what you're working on is in 3d animation tools like maya or 3d studio max or other obscure tools from back in the 90s when I was getting into this stuff, where when you are working on your 3D scene, every action that you take in the GUI produces a log line that is a little bit of script. And each one of these 3D tools has its own scripting language. One of my favorite tools used TCL as the scripting language.

Those logged out scripts are just a little snippet that represents, you know, here's the object that you had selected and here's the transformation that you applied. And the purpose of having them logged out is to assist you in building reusable scripts that you can run to programmatically edit your scene. It's almost like, you know, a human in the loop programming by demonstration where you do the little bit of the action that you want to do enough to generate the structure of the code that you need. Because most of the people using this aren't programmers, they're artists. And so this is a way of saying, you know, don't worry about having to look up what function to call or what syntax to use,

just do the edit you want and we'll log you out a little bit of code for that. And then you'd grab all those code snippets and you'd put them together into a script and manually parametrize out the parts that you wanted to make in parameters. And then you could run that script against your scene and feed in the data that you wanted that script to act on. That's kind of the first place I encountered anything that feels like this. And of course, what you're doing is far more sophisticated and automatic, and it's doing a lot of, it sort of takes the human out of the loop so that they don't have to be concerned with doing that um that abstraction process manually did you look at any of these sort

of autodesk style 3d animation tools or anything like that when doing background research for sketch and sketch and if so what kind of a relationship do you see between what you're working on and what they're doing right Right, so those tools that you mentioned and these kinds of programming by demonstration tools, like you said, often do record a series of edits that serve as reasonable functional units. Certainly is very, very related to the kinds of interactions that we've been exposing for certain operations.

I haven't played with these tools myself, but we certainly have read a lot about them, and Brian especially has kind of surveyed a bunch of these different programming by demonstration tools. And I think a lot of times, so I guess some kind of more technical differences, maybe not important differences for kind of the vision, but technically technical differences often include the fact that the languages that these systems are generating programs for or scripts for are kind of imperative languages where there's a lot of state manipulation. And we didn't talk about this yet. You know, I think the language to start with here really doesn't, you know, matter as much, you know, I think certainly, you know, functional programs and functional programming languages make certain kinds of reasoning that the tool has to do, you know, easier, there's less, you know, implicit state manipulation. But really, I don't see a fundamental reason why these techniques couldn't

also work well with imperative and object-oriented programs as well. But I think these more classic programming by demonstration systems often tend to use languages that heavily use mutation and first-order combinations of first-order kind of programs, whereas we were kind of interested in thinking about how to couch these interactions in higher-order functional programs where we can then do things like map and fold and kind of higher-order combinations of these individual units. I think that's one thing that we're trying to push on a little bit.

Earlier on, when you were talking about hiding the code representation in some circumstances, and that possibly being of interest to more expert users who would want more control over when they're just working with the graphic, or when they're also moving over to work with the code that was sort of the first mention i've seen in the context of this project of the idea of an expert user and that's something that when i'm evaluating different tools for you know looking into the future of, there's sort of a universal focus on the beginner experience and on novice programmers and on programming being broadened to reach more people

who don't have a technical background and sort of programming for the 99%. And I think that that's a worthwhile goal and it's very interesting, but it's so universally talked about that I think that it's sort of a gimme. And it's one of the goals that was explicitly stated for Sketch and Sketch at the end of,

I think it was the Wist talk brian mentioned that and what i'm curious about is how much thought have you given to people using this tool to develop expertise and how much do you think about what the experience of using a tool like this would be for somebody who is a master programmer or a master artist or um both yeah so sorry i thought i i was i was thinking this might be a good time to talk about um using tools like this as kind of a vehicle for for teaching programming and then kind of developing programming expertise but i think maybe that'll be a different topic. Yeah, well, and just to touch on that,

because we can go down that road if you want, but my personal feeling is that it's a cliche that everybody who is working on future programming tools is concerned with the beginner experience and that in every interview that i've ever heard about people talking about a new programming tool the beginner experience is brought up almost like a point of hygiene and it's the sort of thing where i feel like so much ink has been spilled about that that there's not like there might be interesting things to say about it

as sketch and sketch relates to it like what is sketch and sketch doing specifically to help that but i i feel like it's sort of like anybody who looks at this can obviously see how it would be better for beginners like it's so so plainly a richer window into programmatic behavior and dynamism and it's so transparently like a you know it has all of the benefits of live programming that you want and it has all of the benefits of being graphical or being focused on the product of the thing that you're making not the abstract contortions that you have to work through to get to that it it saves you from having to play compiler in your head. Like it's

doing so much that I feel like anybody who is thinking about that will see it in what you have done. Whereas the expert side, I never hear anybody talking about that. I think really like, you know, from the beginning of this conversation, we basically took as an axiom that a user will want to do some interactions with text edits and some interactions as, you know, direct manipulation interactions. I think right from that point, we're assuming that there is an expert user that is a programmer, right? And that wants to dive into the programmatic representation. And I think when an expert user goes to make something, whether it be, you know, an essay, or whether they're going to do some

data manipulation, whether they're going to create some presentation, whether they're going to create some web application, create some pictures, some visuals, I think the expert user, a user who's trained in programming, always has to make a choice. Are they going to use a GUI application that is developed for those domains and give up the abstraction capabilities that they know that programming provides? Or are they going to pick their favorite programming language or their favorite library that caters to that domain? You know, oftentimes someone, an expert user would might choose to, let's say, use Beamer to generate slides or use this racket library called Slideshow to develop their slides. and reusable artifacts, making simple changes like, you know, drag this thing a little bit to the right, or copy and paste this thing and then change it, are extremely tedious to make when you have to think about where in the program does that operation stem from. And so I think there is this,

you know, pain that expert programmers realize they go through when they are programming, right? There's this tedious, you know, edit, compile, run kind of cycle that, especially when you're trying to, at the end of the day, generate something that is very visual and interactive, and the design process takes lots and lots of iterations, which, you know, is common when you're building something like this, I think expert users run into that pain point and would clearly see how they could benefit from a tool that allows them to do programming, but then also get some of these interactive capabilities for changing the outputs of their programs.

Given that that's the appeal that an expert would see in a tool like this, let's look at the other side. What is Sketch and Sketch offering beginners, newcomers to programming, or even programmers, but it just seems intuitively that tools that make programming more interactive ought to help with the teaching, the understanding of programming concepts. And so one thing that I'm interested in doing is kind of using, you know, Sketch and Sketch and future versions of Sketch and Sketch to teach introductory programming to students that maybe want to learn, you know, simple graphic design or generative art, using it as a tool in which to teach programming. Because, you know, my sense is that students that are interested in, let's say, design or art would, of course, you know, my sense is that students that are interested in, let's say, design or art would, of course, you know, learn to use tools like Illustrator and Photoshop and all of those tools. And then some of those students might then later on learn a programming

language like Processing or p5.js, which kind of cater really well to these domains of programming. Instead, what if you could teach, you know, the kinds of features that Illustrator and Photoshop provide in the same environment in which you can learn about, you know, variables and functions and have those, you know, different concepts and different interaction paradigms just be the same system and not two, you know, disparate systems. And so I've been looking recently at the kind of processing community. It seems like they've done a lot of really cool curriculum development around processing in p5.js. There's folks at NYU in particular that I've been looking at their work, Daniel Schiffman,

Allison Parrish at UCLA, Lauren McCarthy. They're developing a lot of really interesting content for basically teaching programming to students that are interested in design and art. And so one idea that I'm planning to pursue is think about how to teach programming with tools like Sketch and Sketch, where not only do you learn different programming constructs, but then you can interact with the output of these programs. And interacting with the output of the programs can suggest changes to the original program and hopefully motivate and teach why you might want to learn about variables,

why you want to learn about functions and things like that. And so one idea that I've been thinking about is kind of thinking about designing some of these curricular exercises and projects as kind of like a game. So a challenge would be to, you know, create some design and maybe make, you know, several variations of it with, you know, different colors and different sizes and things, and start by using just the direct manipulation tools that you would otherwise learn in Photoshop or Illustrator. And use that as an opportunity to reveal several kinds of operations that involve repetitive and tedious kind of edits. Then when, you know, students learn what variables are used for and how using a variable in multiple places then enables the

system to map, you know, one of your output interactions to other changes in the output as well, you kind of learn to understand why you might want to use variables in your design. And so kind of staging these projects as kind of like challenges where you want to achieve some design task with, let's say, the fewest number of mouse edits or user edits or the time it takes to carry out one of these tasks as kind of, you know, a way to motivate and explain why, uh, different, uh, programming constructs and abstraction, uh, ideas, uh, can be beneficial when creating certain classes of designs. And it, it's almost like you're taking the tedium out of both sides. Like you're taking the tedious parts of graphics editing where you have to repeat a lot of shapes and make little changes to each one of them or you want to make a change to some shape

that's used all over the place and and update all of the places where that's used that's very tedious and then on the coding side you've you've done the same thing where if you um want to make you know precise changes to your output. That's very tedious to do in programming. And I think that's a fun way of looking at it. And that is something that I'm sure would be appreciated by people who are new to this whole game or people who are familiar with one side

and are new to the other side. Right, yeah, that's certainly the long-term goal. Another thing that I'm kind of interested in is, so I think I read somewhere that the name processing originally came to be to kind of describe the importance of the process of creating some artwork and not necessarily just the final artwork as well. And so I kind of think about, you know, the process of creating some artwork and not necessarily just the final artwork as well. And so I kind of think about, you know, the process of programming is not the ideal process that we would want, right? We would want it to be much more interactive. And so it seems like, you know, it seems like it's very much in kind of spirit with, you know, the goal to build tools that are what we would hope to be like you know effective and efficient and kind of

tools that allow us to to build and offer things in the way that that seems kind of like it should be and when you were talking about using sketch and sketch to introduce programming to people who already maybe have a little bit of familiarity with graphics tools. You said that, and I don't know if this was intentional or not, you said that what might happen in Sketch and Sketch is that somebody makes a change to the graphic, and that change might suggest a change in the code. And are you suggesting that it wouldn't be the case that a change to the graphic immediately changes the code but instead presents some sort of an interface saying like ah i see you know clippy pops up i see you want to make that kind of change to the graphic here's the way that you would go about doing that by editing the code and it's sort of like a like a tutelage sort of mode or did you mean suggest as

in it just makes that change to the code and they sort of have to look at um the code and think oh i remember how this was before and i see how it's different now and i'm gonna kind of relate that to what i understand about what i did over on the on the graphic side so i meant i meant the former um but i think both you know are are could be could be you know useful so currently as it is there are you know certain interactions that uh sketch and sketch uh there are certain interactions that lead to a program transformation right away without any subsequent interaction but there are other times when currently the tool will give kind of a menu of options, and you kind of hover over each of the options. And in each case, it'll kind of show you a preview of the changes to the code and a preview of the changes to the output. Because for many of these interactions, there's just a huge amount of

ambiguity in what the change of the user to, the change that the user wants to affect. And so, you know, currently we have this very simple interaction paradigm where we, when we have multiple options, we, you know, sometimes use kind of heuristics to try to rank them according to what we think are most likely to be desirable changes, but then the user has to kind of interact with the different choices and pick one and so in an educational context you would you could really like play up that kind of interaction where um you know maybe it's you could even think about like presenting undesirable options and thinking about multiple kinds of edits and see uh you know visually what the effect would be on the output and have um the user kind of have to like decide whether or not you know, visually what the effect would be on the output and have the user kind of have to like decide whether or not, you know, different code changes actually correspond to what they were

hoping to achieve. And so you can imagine, you know, this kind of interactive, you know, dialogue between the system being used specifically for the purposes of teaching new ideas and new constructs that maybe they haven't yet seen. But then you can also imagine modes where, let's say, once they've learned to use some language feature and learned to use some user interface feature, then they could configure it in a way that as soon as they make this change, it automatically applies the code transformation that they've already learned to reason about. That's extremely cool. I would love to see that as a direction that

Sketch and Sketch goes. So it sounds like you are already doing things in the user interface where the code that you see isn't actually the true underlying representation of what's happening in the graphic, because there's some way that you're accomplishing showing a preview of what the change to the code would be depending on a choice that you make in a pop-up context dialogue or something like that so how are you handling that architecturally like is there a true underlying representation and the code display is a rendering of that and you might adjust that rendering based on what the the user is hovering over or is there like where's the where's the disconnect in the system between what's actually happening in the in the core model and what's

actually happening in the the user interface right so the the the ground truth really is just a single program. That is the main artifact. That is the model. That's what everything is built off of. But when a user makes some interaction with the output and then invokes one of the transformations that the menu provides, at that point, the results of that transformation may have more than one program, may have more than one candidate change to the program. And each of those candidate programs, when the user hovers over, will show you a preview of that program, and also evaluate that program and show you a preview of the output. But in some sense, those are just, you know, saying to the user, do you want program one or program two or program three or program four? The user then commits to

one of those choices and that becomes then the ground truth again. And so it's at each stage, we can suggest multiple possible changes to the program. And so how do you avoid the issue of something like a like a feedback loop or something like that where when you're previewing the different options that changes the output representation which changes the thing that you're interacting with to generate the preview when you're doing the previewing does that mean that you're not currently affecting the output? That's right. So after a transformation has been invoked, a set of candidate programs are generated. And what the user can see is what is the output of those candidate programs,

but can't interact with any of them further. Until they make a choice as to what program to use. At least currently, yeah. And so again, this is a kind of a very simple model right now where yeah each interaction has to be mapped back to a single choice and then you proceed but like we talked about you know to support more kind of non-linear authoring workflows you might want the user to to explore you know multiple different paths and continue to interact with the uh with the the artifact and then only later make uh choices about how those interactions ought to be codified in code currently we don't do any of this kind of um this non-linear kind of multiple path uh interactions yet and so on that topic when there are certain moments in time where you are free

to manipulate the output and then there are certain moments in time where sketch and sketch will say okay the output is temporarily frozen until you make a choice about what the consequence of your last action should be on the program. There are some visual programming environments where, like Sketch and Sketch, there's a code representation on one side and a graphic representation on the other side, or they might even be in entirely separate places, like you might have a text editor and then a browser that's doing a live refresh every time you save in your text editor or something like that there are some cases where the live editing experience is you know code is in one place and

the live preview is in another place and then sketch and sketch brings them close enough together and you get the bi-directional stuff going on where not only are they right next to each other but a change on either side basically instantly updates the other and then there are some projects like i think some of sean mcdermott's work where the canvas contains the code and the code is always represented in the canvas and you could imagine if that code were generating some graphical output that graphical output would live in the same canvas as the code. And the two things would kind of coexist graphically, and you would use the same tools to work on both sides. And so to me, it feels like there's sort of a another spectrum or another

space here where there's how close together are the representations of the code and the output and how unified or how distinct are the tools that you use to work with each and so what are the things that you've sort of thought about when bringing these two sides together as you have in sketch and sketch and sort of how did you decide to stop at the point where you are and not go further and say the canonical representation is in the graphic because you've moved further in that direction recently by adding widgets in the graphic that are a representation of an intermediate execution product in the code but there's still a disconnect there and so i'm wondering if it's sort of like two tectonic plates that are going to gradually move together and one sort of subsumes the other, or is there, do you feel like there's a natural stopping point? I think the tectonic plates are going to crash into each

other. And one is going to, I think there's going to be a big earthquake where it's all going to be a mix. I don't think there's going to be this kind of hard divide between code lives here and output values live somewhere else. I think we chose that approach because it was the simplest one that we could take. But I certainly imagine a much more, let's call it canvas, like a freeform canvas where all of your computations live, and sometimes you're viewing the graphical output that it produces or graphical representations of the execution, and sometimes you're looking at the text view. Yeah, I imagine a much more kind of fluid mix of those two kinds of interfaces in the future.

So one kind of, this will be hard to describe maybe over the radio, but let's do it. One kind of UI that I like thinking about is, so I use PowerPoint a lot. That shows you how great of a designer I am. I do all of my editing and stuff in PowerPoint. So my wife and I make holiday cards at the end of the year, and she usually kind of like comes up with like a sketch on paper of like, you know, idea. She kind of picks out a few images that might work well, and then I'll go in PowerPoint and kind of like prototype some designs. And in PowerPoint, like maybe other design tools as well, you kind of see, you know, the space on

which your slide lives, but then you can also have objects, you know, outside of that range, that focus, right? And so I'll often have like multiple images or other kind of objects, you know, outside of the periphery of the main slide that I'm working with, I'm trying things, and then, you know, only what is in the actual slide is what's in the final, let's call it, main expression of my canvas. And so, you know, I kind of imagine that maybe this kind of, like, freeform canvas kind of approach could be useful, where you could, you know, sprinkle around a bunch of program snippets, graphical representations of what they produce, alongside the main object, the main expression that you're constructing in the middle. That's kind of one toy idea that I imagine, you know,

we might try sometime in the near future where you can have, you know, the kind of helper widgets, the intermediate computations, be close to the object you're creating so that, you know, you don't have to decide whether or not something is, you know, in the final output or not. You have access to all the different kinds of, you know, tools that you're using in kind of a more freeform space. I love that so much. If that ends up being the direction you go, I'm going to delight in seeing it if uh if that turns out to be fruitful and if you pursue it because that's i'm all about that that's uh that's extremely cool to hear that that's something that you've thought about and you can imagine like you know an analogy to kind

of normal um programming where like you know usually you configure your program to generate some some final output but then other times you times you will add in a whole bunch of print statements and other kind of logging things to have extra information as you're working on that main expression. And so in the same way, you can imagine toggling on and off all these intermediate computations, these helper widgets, these pads of programming and design that you were trying out but maybe never made it in the final

you know the final artifact but allowing those to kind of coexist in a way that you can you know yeah explore and kind of go back and forth uh through the through the authoring process and you're already doing that to a certain extent like there's the what did you call it like ghost mode or something like that that's right that's right yeah that was just a very kind of simple layering mechanism you can think of it that way where each object that you're generating can live in its own layer so that you know you can naturally toggle between layers that you want to to show or not in the in the output pane and the layers that you might want to show or not at least in terms of how i've seen you use them are for things like widgets that let you do advanced changes to the code representation that these widgets aren't going to be shown as

part of the final output graphic but they're graphic representations of of the structure of the code or of transformations that the the user might want to make on the code that's right they could be or they could even just be kind of helper shapes that you know the user has put because it makes you know relationships between the main shapes easier to specify right like an arbitrary center of rotation or something like that right yeah so you know i find myself doing that a lot in powerpoint too like temporarily making a shape that has some certain size that I want, because then it's easier to snap my main shapes to it. And so I feel like that pattern obviously comes up over and over again. And so you can naturally keep all these helper shapes in different layers. But rather than having to delete them,

but then maybe insert them back again, if you want to make that change change again you could selectively toggle them on and off in different layers i work with a team of artists and i've seen them do that time and time again they'll use the graphics uh program that they're working in to construct little like scaffolding shapes you know here's a line that's a certain length and it's a certain distance away from this other line and i'm just going to move this little construction all over the place to help me line up everything on a grid they make these sort of little scrap pieces of graphic and then use them to add structure to what they're working on and so it's cool to hear that you've also done that kind of thing working in your art tool of choice, PowerPoint.

Yeah. This podcast has a transcript, which you can find at futureofcoding.org slash episodes slash 49. And that transcript is brought to you by Repl.it, an online REPL that lets you spin up an environment for working with any number of programming languages, taking all of the pain and hassle away from trying a new language or getting a quick project up and running. The Repl.it code editor is a collaborative environment so you can have multiple people working on the same code base all at the same time. They have a thriving community of students and professionals who are doing amazing things with the tool all the time. So go to repl.it, that's R-E-P-L dot I-T, to check out what they're doing and try it out for yourself. My thanks to Replit for sponsoring the transcript and for helping to bring us the future of coding.

I should also make clear that this has really been a big team effort. I actually haven't done hardly any, I've hardly done any programming on Sketch and Sketch in the past year, two years even. I'm looking forward to getting back into it, but really the kind of heavy lifting has been done by this really terrific group of students. Brian Hempel has been doing a great job over many years. He was crazy enough to kind of join me when I was starting this project. Justin Lubin, a really terrific undergraduate who's made tons and tons of contributions to the project.

Mikael Mayer has been doing lots of really awesome work. Cyrus kind of joined my group and tons of contributions to the project. Mikael Mayer has been doing lots of really awesome work. Cyrus kind of joined my group and kind of brought the idea of holes and programming with holes to the group as well. Nick Collins. And so it's really been a really terrific team effort. I've been really lucky to have such great collaborators. Speak to that a little bit more. Like what have each of those people done? It'd be sort

of nice to know for each of those people what they've contributed. Sure. So Brian's focus over the past couple of years has been extending the expressiveness of what you can do purely with direct manipulation interactions. And so he was the driving force behind this most recent Wist paper where, using just direct manipulation interactions, he's able to build complex, reasonable, readable programs for a variety of parameterized drawings, including recursive drawings, including drawings that have shapes and groups of shapes repeated over various geometric dimensions. And to do this has required exposing much more about the execution of the program than just the final value that

it computes. And so recording a lot of information about how intermediate program execution ends up affecting the final output value, all this kind of more general purpose tracing of programs and exposing richer widgets for manipulation, exposing a whole bunch of new tools for transforming programs has really been the kind of focus of his work over the past couple of years. And he's now thinking about how to expose similar kinds of interactions, building programs based on output interactions, for other domains where you don't necessarily have very visual representations of values like you do in vector graphics. So how could you implement more kind of typical general purpose data structures and data structure manipulation functions by example, by demonstration? How can you do that in this kind of style? So that's been, that's his kind of current, uh, focus right

now. Uh, Justin Lubin has made contributions to the project kind of, uh, throughout in, in various aspects. Um, one of the main kind of, uh, projects and features that he, uh, led was the design of, uh, this, uh, uh, text editor interface that we call Deuce, which is something that actually doesn't have to do with the bidirectional programming at all. It's kind of a more traditional feature. It's a feature for more traditional text editors for code editors, where, so obviously, text editors are the kind of you know main uh interface through which programmers uh read and write code but of course a program once it's you know parsed into uh an abstract abstract syntax tree has a lot more structure than just the kind of underlying linear text

buffer that created it and so refactoring tools offer a variety of structure transformations like renaming, extract method, things like that. And then structure editors have kind of lower level AST transformations kind of built into the system so that you don't have to resort to text editing all the time. And so a feature that we built for the text editor in Sketch and Sketch is what we call Deuce, which tries to overlay the structure information of the program, the ASD, on top of the normal flat text representation.

And so Justin led the effort on designing that interface where you can hover over the code box, and as you hover over different parts of the program text, it kind of shows you different nodes in the AST that you can select. So, for example, you might select some variable definition, and then you might select some white space in between two definitions in your program and one of the tools that sketch and sketch will then propose is do you want to move this definition from this part of the program to the other part that you've selected and so

these kinds of refactorings or structure transformations can be made while kind of staying within the text-based editor. And so Justin took the lead on lots of that project. Mikael Mayer has worked on one of these algorithms under the hood that we kind of referred to earlier, which is really the kind of core bidirectional evaluator, which allows changes to be made to the output value in a way that are mapped back to changes of the program. And this bidirectional evaluator has really been developed in a kind of generic domain-independent way, but it's one of the many kind of features under the hood of Sketch & Sketch

that allows you to, for example, make changes to things like colors and positions of things and have those edits be mapped back to corresponding program repairs. He's also been exploring that idea of bidirectional evaluation applied to the domain of HTML applications as well. He's actually pursuing kind of a startup to try to build some of those technologies into more usable tools. Yeah, and then so Cyrus obviously had been working on the Hazel project before he joined my group and hasn't been working on Sketch and Sketch proper, but the idea of programming with holes and running incomplete programs is something certainly that we're going to incorporate into future versions of Sketch and Sketch.

We've recently been working on a project for taking these programs with holes, partially evaluating them, but then taking ideas from bidirectional evaluation, allowing examples of what these incomplete, these partially evaluated programs ought to evaluate to as a way to kind of synthesize expressions for those missing pieces. Yeah, so the idea is to allow a combination of a program sketch and examples of what that program ought to, uh, to do, uh, to help, help, uh, synthesize, help fill

in missing pieces of the program. Um, and that's a project that Justin, uh, and Nick have been, uh, uh, spearheading also that, that algorithm that doesn't really doesn't really appear in either Sketch and Sketch or Hazel yet. But it's sort of a potential future direction. Yeah, it's very much one of these kinds of engines under the hood that I think could be a really useful part of these interactive authoring environments. And then Nick Collins has also been working on aspects of this program sketching plus bidirectional evaluation project, as well as several kind of structure editing kinds of ideas on the Hazel project, and has done a little bit of work on the Deuce code editor as well for Sketch and Sketch. Cool. Yeah, thanks for going through each of their contributions it's um one of my goals in

the interviews i'm doing on this podcast is to both learn about the projects um but also i'm very interested in learning about the process and how people um approach working on these future tools and and the sort of the context around it because all of the things that we approach working on these future tools and, and the sort of the context around it, um, because all of the things that we're working on are evolving bit by bit. And so I think it helps to, uh, to kind of share our, our processes and our insights into how the tools get made. Yeah. And so sort of building on the theme of, of of working on the project and building on the example that Brian brought up in Sketch and Sketch or like a programming by demonstration system could use to evaluate their project. Like if I'm working on my own project and I wanted to evaluate its flexibility and its suitability, here are some standardized tests that

I could put my program through. And i'm curious if if those references were collected in the whist paper or anywhere else that you know of so that the other people listening to this who are working on their own projects can say oh i'm curious to see how my my tool does against those benchmarks because it's like it's tantalizing to know that in a in a a sort of, um, you know, like a, um, a friendly competitive sort of way that sketch and sketch could do four of the benchmarks and then could kind of do two. And then there were nine more that it couldn't do. And that's, that's a really interesting, um, admission. And so it's, it's sort of, um, uh, an interesting idea that i hadn't considered before is almost like a like a to-do mvc for live you know bi-directional program directed output uh tools and so i'm wondering if you've

made that collection of references anywhere if i should just go on to the slack community and nag brian for those uh for those links so certainly the, the Wist paper, um, does kind of refer to the sources from which we drew these benchmarks. And so the one that the ones that you mentioned, I think are from this, uh, watch what I do kind of benchmark suite and it is a, yeah, kind of very well known and, um, available kind of resource. I guess one, you know, one challenge about, you know, doing head to head comparisons is that, you know, there's so many differences among, you know, the specific language that is being used, and some of the tools maybe don't run because they're 20 years old, and some of the goals or the spec benchmarks of output-directed programming, but it certainly is the kind of thing where, like, yeah, you know, we certainly want to, you know, be able to compare different systems on shared examples, if not exactly the same,

you know, example, at least, you know, different, you know, incarnations of the same goal or the same concept across multiple systems, and so I don't know that there's, like, a, you know, incarnations of the same goal or the same concept across multiple systems. And so I don't know that there's like a, you know, single kind of benchmark suite that already exists, but certainly kind of comparing to all these other examples that people are using seems to be, you know, a good first step. For the people who are working on these sorts of tools, which I think is most of the listeners of this podcast um those sorts of benchmarks would have utility in that they might force you to approach your tool in a way that you weren't naturally gravitating towards and so you can sort of use it to test

your assumptions and to test your model and to see where your ideas break down and that's that's the appeal of these sorts of benchmarks to me is the competition thing's more of a like a kind of a joke it's more that um this is a a way of um like a lot of our community are working independently. And so they might not have the resources to do extensive user testing, or they might be at the wrong stage of their, um, process to do user testing. But those, um, benchmarks might serve as ways to help people think about their model and to just read the benchmark and think, is that something that I could even do within my tool? And I was going to, I wanted to reference in this context, Brian Eno, who's a very significant figure in the history of generative music and generative art. He made a deck of cards called oblique strategies that you're supposed to use when you're facing a

creative crisis when you're staring at a blank page and you don't know what to write or when you've made something and it's not finished and you're unhappy with it but you don't know what to do to continue working on it and each of these cards in this deck of cards is sort of an open-ended prompt and many of them are are very unusual in what they ask you to do and they're meant for interpretation and they're sort of meant to shake you out of a creative rut and so i'd love to see and it sounds like these the examples from the watch what i do book are a little bit like this, is I'd love to see somebody put together a collection of just here are intellectual stress tests that you can put your ideas through in order to assess their generality or assess their applicability to

the domain or assess, you know, things that will force you to think through the problem space so that you don't lead yourself into a blind alley. So if you know of anything like that, other than the watch what I do benchmarks, that would be a really interesting thing to share with our audience. Yeah, I agree that it would be kind of valuable to have that sort of thing. I remember at the end of the live 2018 workshop, there was a little bit of discussion about whether there were kind of common benchmarks or reference points that, like you're describing, one could look to as stress tests and things like that. I don't remember if anything came of that discussion. These aren't examples of the kind that you described, but one thing that I would mention is this work on what's called the cognitive dimensions of notation, which

are a set of heuristics for evaluating user interfaces, programming languages, other kinds of abstractions like that. And they include things like a closeness of mapping, how closely does the notation or the user interface represent to the notion of what the user is trying to create, what notion in the world the user is trying to create. Another heuristic is, are there hidden dependencies? Are there other the world the user is trying to create. Another heuristic is, are there hidden dependencies? Are there other things that the system is doing and knows about that are not exposed to the user for understanding and for manipulation? And so these are kind of

like, you know, properties that, you know, generally are good if they hold, they ought to hold. And so some of them are kind of like mutually unsatisfiable, but you want to be able to satisfy as many of these as possible. They're not benchmarks or examples in the kind of sense that we were talking about, but are kind of useful heuristics to try to understand whether the system that one is building or trying to build satisfies multiple of these goals or or not hey so this is ivan from the future just cutting in here um i normally uh adhere to the doctrine that uh you should not reveal the technical details of how a podcast is made on the podcast unless you are doing so intentionally so for instance avoiding letting people hear the glitches that sometimes happen

when skype drops out or you know one person having a really good sound quality and another person having a really bad sound quality that kind of that kind of behind the curtain stuff is to my taste unprofessional but i'm going to break my own rule right here and say that at this moment in the conversation my connection to the internet died um i can't remember why but uh ravi and i reconnected and picked up the conversation where we left off i couldn't salvage this in editing um that's you know i've i've uh had a number of other hiccups like this in my brief time as a podcaster, and I've always managed to pull them together through a very deft use of Ableton Live. But this one was just too jarring, and I couldn't find a way to stitch it together coherently. So you get to hear me blathering on and apologizing and saying, I'm sorry, this isn't how I like

to do it. Don't do me like this. So anyways, that caveat out of the way uh back to the interview end user programming and programming by demonstration other kinds of approaches like that you know have never really kind of succeeded in in in the mainstream or into really kind of usable useful solutions and so i guess one question is like you know will will this you know ability to actually mix programming and direct manipulation will it actually help with like you know novices and more uh less expert users or will this only

be limited to you know experts because this is just another set of tools that need to be used and mastered but i guess you know i guess i. But I guess, you know, I guess I'm kind of hopeful because, you know, the kind of success story that most that many people like to talk about are, you know, how widely used spreadsheets are, and how many people use like formulas and macros and a little bit of programming without really knowing it. And I guess I like to think that, you know, what if like, the standard toolbox that you see in many different GUI applications has a tool that looks like equals X, where you can introduce a name for something, and then you can drag that name onto multiple properties and multiple objects on your canvas. I feel like very simple

notions of giving names to things and very simple relationships between these things seem like they ought to be kind of simple and general enough to be realistically part of you know end user user interfaces for a variety of domains um but again i guess it remains to be see if that's that's really gonna play out or not that's why i brought up um maya and 3d studio max and the other autodesk tools is and i neglected to mention this but another ability that they have that's sort of universally present in 3d animation tools is that um they allow you to make any property on any object in your scene, the result of a function of another property. And the way that you can wire them together is extremely open-ended. And, um, as a, as an early teenager, knowing nothing about programming, knowing frankly, nothing about, uh, computer graphics and not being a very good artist I was still able

to pick up one of these tools and just through playing around with the user interface figure out how to make for example like glitch art by using the incident angle of the camera on the surface which is an xyz position be used as the rgb color attribute and so the color of the surface which is an xyz position uh be used as the rgb color attribute and so the color of the surface is based on the it's based on the angle that you're looking at that surface from and and those sorts of um mappings from one uh data type to another or one representation to another or one property to another and applying a function on that mapping in in those programs is a a really um like it's a core part of the of the ui and a lot of the uis are built in terms of those uh mappings between properties and it's it feels like like when i use other programs that don't have that ability it feels like something's

missing like i've been sort of cheated um when i use photoshop or when i use illustrator and i can't do that kind of um like like it it gave me an early taste of what it feels like to be a programmer before i actually learned to be a programmer because I was programming my scene. I was programming my art. And I, uh, I definitely feel like, um, like what you've just talked about, about, you know, wondering about whether, um, something like sketch and sketch or something like programming by demonstration would find a home in all of these domains. I, think that it would i really think that there are examples of where that has happened in uh existing tools and unfortunately they like if you pick up a program like maya or moto and you open it up it doesn't't look like Excel. It looks like, you know, a nightmare version of

Excel where instead of just the one ribbon across the top with, with way too many tools, it has, you know, user interface controls everywhere. It has thousands of features. If you right click in 3d studio max, it brings up four context menus around your mouse. Um, these things are there like the Starship enterprise. It's, it's enormous and complicated. And yet as a, you know, a 12 year old with no internet at the time and no tutorial books and nothing other than a, than, uh, the, the software on my computer and no expert guidance, I was able to learn how to do it because those interfaces are self-revealing. And so that's why I wanted to ask about the expert experience and about whether you think about the ramp towards expertise is because I think the focus on making

things approachable by beginners, a lot of times people will react to that that desire by making a tool that is sort of meant to be handled with with boxing gloves or with you know it's like safety scissors they sort of um try to distill it down to some sort of simplified core essence so that people can't hurt themselves or be scared away by the complicated user interface and yet i think there's in making these tools that have that programmability behind the scenes there's going to be a need to expose more complexity and expose that complexity in a way that's tractable in a way that's approachable in a way that is self-revealing and and discoverable um to borrow a ted nelson uh term to make the interface explain itself not to make it intuitive but to make it so that somebody who's

playing around with it with no other reference can figure out what it does and i think um sketch and sketch it looks to be a great um approach uh to doing that like it the the ability to see the change that you're making uh graphically or or in whatever output domain turn into the change in code like that that um connection between the complexity of the code and the simplicity of the output, I think gives people that, it gives them that ability to discover the way that the tool works. And so I am optimistic about that being something that we can look forward to in the future. What I am not optimistic about and what I would be interested in hearing your thoughts on, if you have any, is what it takes to make a tool like this have a place in the market. Because we've had, you know, past examples like HyperCard being, you know, dying an early death, even though it enjoyed phenomenal success. Um, there's,

there's sort of, and flash more recently flash was a, was a beloved tool for doing, uh, interactive art that died because, uh, I would argue because of mismanagement by Adobe. Um, but that's a whole other matter. It seems like these tools have a really hard time surviving in the market and so what do you think about that at all and what do you think you might be able to do to help fight that trend yeah so so market i guess there's a couple ways to think about you know getting hopefully getting these kinds of ideas technologies tools into the world and is, you know, can this be built into some, you know, system tool technology that can be marketed, that can be sold? And others, can it be implemented, developed, and at least open source tools that, you know, are used, you know,

by a variety of people in a variety of settings? And so, you know, so far, as, you know, researchers, a small research group, we've kind of chosen, things in a clean slate, prototype, you know, write papers and kind of try to explain the main essence of the idea so that, you know, hopefully someday in the future, folks that are building industrial scale, you know, languages and editors and environments, you know, hopefully these kinds of ideas, you know, make it into those efforts in the long term. But in the kind of shorter term, I guess I would say that, you know, make it into those efforts in the long term. But in the kind of shorter term, I guess I would say that, you know, I think there are certain parts of the techniques, at least, like some of the bidirectional algorithms that are making pretty modest changes to

the existing program, assuming that a programmer has already written a lot of the kind of high-level logic of the program, a lot of those algorithms, I think, you know, could be scaled up to, you know, practical editors for, you know, many languages, you know, in the not too distant future. And so one of the things that Mikael Mayer has been working on is kind of scaling up one of these bidirectional evaluation techniques to JavaScript. And so one way we might do it is actually have like a library that, you know, takes your JavaScript program and can run that JavaScript program in reverse so that when the output of your JavaScript function is changed, it will suggest to you a bunch of changes to your

program that you might want to choose from. And so a library like that, you know, we would hope could be hooked into a bunch of different application domains in which people are writing JavaScript programs to, you know, script various things. Could you provide some of this kind of program repair, bidirectional evaluation functionality somewhat for free so that, you know, a tool builder or an application builder

doesn't have to do anything more than surface the suggestions, the functionality to the user. And so, you know, I think, you know, he and we, and I think hopefully others will want to,

we'll put in the effort to try to see if we can actually kind of make it, you know, a drop-in replacement for, you know, you have eval javascript can you have unavail also without a whole lot of extra work on uh on the user's part what about artist tools or mixed art and programming tools like hypercard or flash yeah so i think in some sense, I mean, the bar is, is higher for building really usable

tools for, you know, for, for artists or programmers or novices that don't have as much programming experience. I think the easiest, you know, realization of these ideas in the, in the world, you know, in the short term, I think are as developer tools, because you can imagine building, you know, plugins for existing languages, you can imagine building plugins for existing editors, for, you know, you can imagine building plugins for like, Chrome and Firefox that kind of expose these, these kind of capabilities. So I think in terms of like the short term, you know, some of these ideas, I think, could appear in kind of developer tools, you know, much more easily than they could for more ambitious kinds of domains. Because I think,

you know, for a system like this to be useful, it really has to provide a lot of the kind of capabilities that so many existing application GUI tools already provide. And so in terms of marketability for these kinds of application domains, I certainly don't have any good kind of answer. It's hard for me to kind of imagine the situations in which this kind of thing could be a useful, marketable tool. I guess one idea is thinking about the kinds of people that maybe are learning programming not because they want to be programmers but because they want to create some artwork

and graphic design and things like that oftentimes will have to learn programming as part of their workflow and their toolchain but clearly the state of the world where you have tools that are good for direct manipulation, you have tools that are good at programming, but nothing that kind of combines them. I guess if there's the right niche application domain, maybe one could focus building this up for a domain in which there doesn't already exist a very good

you know solution for the kind of programming side of things but i guess this is not a really yeah this is not a very helpful answer um it's hard one thing i thought about in in trying to find out the difference between why it seems tractable uh to offer this sort of tooling to programmers and intractable to offer to artists is that programmers are in the business of assembling things and that extends to the tools that they use and so it's it's you know rare is the programmer that is using a stock configuration of emacs or is using a framework with no middleware. We tend to kind of endlessly customize everything we do by assembling other pieces together with a core structure. And artists don't do that. Artists buy a complete workstation off the shelf. If you're a musician, you buy

Ableton Live and Ableton Live lets you use audio unit plugins and VST plugins, but it doesn't let you add in, you know, a new way of, like you can't add in a score notation for, you know, writing sheet music within the Ableton Live environment. It's a much more closed tool, and it's expected that you will use the extension points it offers, and that's it. Whereas programming tools, the same philosophy applies. You can only use the extension points that exist. But programmers are coming with a mindset that they want to customize. So let's make lots and lots and lots of extension points.

Let's make it so that things are modularized and composable. And so I almost feel like what it might take to be able to, as a small independent research team, to be able to make a contribution to artist tools would be to find a domain where the artist tools are more modular and where there is sort of the invitation in that domain for people to come in and add the extra slice of functionality that they can add and it makes me wonder maybe like uh like like scientific visualization or medical imaging or something like that like maybe there's a domain out there where that is more uh common practice and that might be a um sort of a way to get the foot in the door to the arts rather than just um continuing to offer things to other

programmers yeah yeah the thing that The thing that scares me about scientific visualization, medical imaging, also like 3D animation games, all of which I think are certainly domains in which I would hope these techniques will work for and scale to eventually. What makes me scared in the short term is that, you know, the math that goes into the computation of the final artifact is much more complex. And so one of the challenges for this kind of connecting programs to their output in both directions is how well can you invert the operations? When you make a change to the output value,

how do you invert it and map it back to the program that generated it? And the more and more complex math that you have, the more ambiguity, the harder it is to kind of really get meaningful changes back in the, you know, the inputs to the computation. It's certainly not impossible, I don't think, but it's, I think, more challenging than what we've been focusing on so far. And so in the shorter term, I'm still hoping to find more like 2D application domains in which this kind of combination could be beneficial. And so one tool that I haven't used much, but I've heard a lot of good things about it, is Figma.

And I've heard there's lots of plugins that could build, let's say a Figma plugin that provided some amount of, you know, abstraction capabilities via, via code in a kind of, um, relatively lightweight way, whether or not like, you know, that could be a useful part of the kind of design and prototyping and mock-up kind of, that Figma, I think, is often used for. Yep, totally could be. I don't know. I've used it, but only in the most casual way. I don't know if they offer that sort of extension, but that kind of a tool absolutely would be a good fit for this. Another example, not in this kind of like, you know, creativity kind of space, but more in the kind of, again, what I like to think of as like office or everyday kind of spaces. So, you know, products

like Coda and Notion have been kind of exploring interesting new ways of, you know, breaking down the barriers between all these existing, you know, different office applications into more kind of composable and unified ones. But those two tools, at least, don't go and expose, you know, don't expose code to users. So if one could have kind of the kind of nice integrated composable UI features that, you know, Coda and Notion expose, but then also back it up, but also expose the programmatic representation that is behind it and integrate that into the expected user workflow in some way. You know,

that's one way that, you know, I would think is, is worth, is worth trying to go. Thank you for taking so much time. Great. Yeah.

No, thank you for coming on the show and for, uh, yeah, letting me pick your brain about all these different parts of sketch and sketch. It, thank you for coming on the show and for, uh, yeah, letting me pick your brain about all these different parts of sketch and sketch. It's a tool I've had my eyes on since the strange loop talk. And it's very exciting for me to be able to learn where it came from and especially to learn all of the different goals that you have for the project and if not goals, then potential directions that it could go in

the future. I enjoyed hearing about that very much. Yeah, thanks a lot. This was a lot of fun. Thanks for taking the time. And thanks also to you and Steve for, you know, organizing this community and providing so many great resources for all of us. So thanks for that. So that's the end of my interview with Ravi Chug about Sketch and Sketch. I'd like to thank him once again for coming on the podcast. And now that we're at the end of the show, I'm going to go extemporaneous for a little bit and just share what's happening with our community and with myself. And completely off script at this point, so if I get a bit rambly, you'll have to forgive me for that. This interview with Ravi was recorded about a year ago. I had done a whole bunch of interview recording in January and was planning to do an

episode a month for the past year. And of course, that did not pan out as expected. Many things over the past year did not go as expected for people. And if I could to illustrate that point, I am right now sitting in my podcasting studio, which is a sort of a, you know, a padded room with a nice microphone and a nice recording set up at my company's office. But that office doesn't have any heat in it right now. So I'm wearing my coat and a toque because this is Canada and the winters here are cold. And of course, the reason it doesn't have any heat in it is because the province in Canada that I'm in

is back under another lockdown. So nobody's here except me all by myself in this building, in this padded room. There's these strange voices echoing through the halls. But I'm great. In general, life has been really good for me over the past year. I live out in the woods, so I don't see a lot of other people normally anyways. And that's just been a regular part of my life for a number of years now. And the past year, it was just more of that, being out in the woods, having terrible internet, so not really being able to participate in the big move to Zoom and streaming and all of that that a lot of other people are doing, but

still just sort of keeping my head down and programming away, doing my own thing. I'm going to try to keep this podcast feed a little bit more active over the coming year, even though it's not yet clear that we're on our way out of this pandemic. And I'm not sure how often I will be able to get into my podcasting studio where the internet is good and where I can record a decent quality podcast. Because that's a big part of why I'm doing this show and why I'm so enthusiastic about it is I love podcasting as a medium. I listen to probably 50 different podcasts.

I've learned that the podcasts that I enjoy most are the ones that are very high quality. They have a lot of thought and care put into them. The sound is good. Subjects are well chosen. They're often, if not tightly edited, then at least slightly edited so that they're not an incoherent mess. And I've heard a lot of programming podcasts that do not hold themselves to that standard of quality, where the sound is sort of like being on a speaker phone where the guests are chosen because they have a book to promote or because they have a startup or a library that they're trying to draw a lot of attention to and i i find those shows sort of

heartless and and not compelling and so i'm trying to do something better for this community, for this podcast. And so I'm going to be trying to book additional interviews, but that's just a tricky thing for me to pull off just in my living situation right now. As for the Slack community itself, I still go there every day. I still love seeing everybody there. There's still a constant stream of new information coming into that community, links and papers and talks and videos of the projects that people are working on. It's really fascinating and I'm enjoying being there a lot. So that's probably enough rambling for me. I hope to see you again here soon and I hope that you enjoyed hearing my interview with Ravi Chug,

and I look forward to seeing you again in the future of coding. Take care, everyone.